'Bleeding lines' - agents over hearing other agents calls

[kaconley]

The following is frorm one of our administrators (she has done some very extensive research on the problem of agents overhearing each other). I thought this would help identify the issue (and if already fixed we'd LOVE to hear about it- naturally). We haven't done the last upgrade but are going to do it this week. Here is the detail from the admin (apologies for the length but there is some good detail):

Dialer issues that need resolution:
1. Bleeding lines – in some cases, the reps are hearing each other make presentations and can talk to other reps or to two different prospects.
1a. Normally, if a rep logs out and logs back in within a short time frame, they get the same session id that they had previously (rep1 session60).
1b. However, occasionally, the computer gives that session id to another rep. Now, rep2 has session60. Normally, when you sign onto the dialer, you receive a message that says “You are the only one in this conference.” In this case, rep2 does not receive this message. Sometimes the rep has calls on the screen at the time they log in.
1c. When rep1 logs back in session60 is busy so rep1 gets session65. Now, we have bleeding lines. The reps can talk to each other and hear each others calls. I have been told that a rep quit dialing but forgot to hangup her phone. When she went back to use her phone two hours later she could hear another rep making presentations just as if they were on her phone line.
1d. I do not understand why the dialer does not close the session every time an agent logs out – thus providing a new session each time they log in. Giving the rep the same session id does not help us solve bad connections either. If the agent is not logged out, why would the dialer give that session to someone else.
1e. Some of the reps have found that they can make this happen by logging in and out. Eventually, if not already, we will have reps doing this intentionally to steal leads.
1f. We cannot hear this when we monitor because the monitoring does not pickup or record anything from outside the phone conversation. However, we have already submitted you the details from one case that we were able to document by talking to the rep.

2. Customer has hung up. We have lots of reports of reps getting the customer hung up screen when they are still talking to the customer. This screen will appear multiple times on the same call. They have to click ok to get it to disappear and sometimes they lose the call they were on.
2a. I think these messages are from different calls (not the ones they are on). Perhaps this is where we are getting the huge number of dropped calls that are filling up the file and stopping the dialer at times.
2b. This may be related to the logging in and out of sessions. These customer hang up messages may be calls directed to the previous agent who had that session. When the previous agent does not answer, the customer hang up message appears on the screen for that session.
2c. While working with Agent Green (rep 162719) we captured some important information that leads to this conclusion. Friday, June 16, she logged into the dialer at 9:30 CT and took calls until 12:40 CT. During this time she had several calls where she received the customer has hung up message. That was the list I sent to you yesterday. At 12:47 CT she logged out for a few minutes and then logged back in. After logging in the second time, she had NO problems with this issue. Agent Green logs almost everything that happens and I rest assured of the accuracy of the data received. Please note: We checked her caller id records from Friday. Her caller id does not indicate that she received a call from the dialer at 9:30 am CT when she logged in the first time. However, it did indicate that she received a call from the dialer at 12:47 pm which was the same time she logged in the second time.
2d. It appears as if the previous agents phone was still connected to the dialer. However, the dialer let her log into the session. Perhaps the first rep was getting calls during her first session, when the agent did not answer, she gets a hang up call message.
2e. I want to check this out more thoroughly. Unfortunately, Agent Green did not record the session ids. Some of the calls she received during the first session are xxx, yyy, zzz. Please see if there were different session ids. Also, please provide me with the name and user id of the rep who had the first session id prior to Agent Green. I want to call them and find out if they received calls.
2f. Agent Green was on again today 8/17/2006 in session60 at 12:18 p.m. Can you let me know who was on session60 prior to her?

3. The hang up customer button is not registering on some answering machines and thus the agent has to listen to the entire message before the call can end. This is prevalent in cases where the answering machine is asking for a response – for Mary, press one. The rep presses the hang up customer button but the call does not end. Recording attached 231819. You can hear him press the hang up call button prior to my asking him to do so but it does not work. I can’t tell if the call times out or if the hang up customer code just takes longer to work in some cases.

I am also wondering if this log in issue is providing some of the voice quality problems.
(Note on above comment - we seem to have quality issues on the same lines that agents complain of the 'bleeding' or overlapping - that is why she makes this comment)

Thank you for any help!

[mflorell]

The following is frorm one of our administrators (she has done some very extensive research on the problem of agents overhearing each other). I thought this would help identify the issue (and if already fixed we'd LOVE to hear about it- naturally). We haven't done the last upgrade but are going to do it this week. Here is the detail from the admin (apologies for the length but there is some good detail):

Dialer issues that need resolution:
1. Bleeding lines – in some cases, the reps are hearing each other make presentations and can talk to other reps or to two different prospects.
1a. Normally, if a rep logs out and logs back in within a short time frame, they get the same session id that they had previously (rep1 session60).
1b. However, occasionally, the computer gives that session id to another rep. Now, rep2 has session60. Normally, when you sign onto the dialer, you receive a message that says “You are the only one in this conference.” In this case, rep2 does not receive this message. Sometimes the rep has calls on the screen at the time they log in.
1c. When rep1 logs back in session60 is busy so rep1 gets session65. Now, we have bleeding lines. The reps can talk to each other and hear each others calls. I have been told that a rep quit dialing but forgot to hangup her phone. When she went back to use her phone two hours later she could hear another rep making presentations just as if they were on her phone line.
1d. I do not understand why the dialer does not close the session every time an agent logs out – thus providing a new session each time they log in. Giving the rep the same session id does not help us solve bad connections either. If the agent is not logged out, why would the dialer give that session to someone else.
1e. Some of the reps have found that they can make this happen by logging in and out. Eventually, if not already, we will have reps doing this intentionally to steal leads.
1f. We cannot hear this when we monitor because the monitoring does not pickup or record anything from outside the phone conversation. However, we have already submitted you the details from one case that we were able to document by talking to the rep.

This is really just an issue of agent properly logging out and hanging up their phones. There is a way to prevent the sessionID deletion after logout which is to comment out this SQL statement in the vdc_db_query.php code:
DELETE from vicidial_live_agents where server_ip='$server_ip' and user ='$user';

It is not recommended since you may run out of sessionIDs during the day, and you need to make sure the vicidial_live_agents table is cleared at the end of your shift.

2. Customer has hung up. We have lots of reports of reps getting the customer hung up screen when they are still talking to the customer. This screen will appear multiple times on the same call. They have to click ok to get it to disappear and sometimes they lose the call they were on.
2a. I think these messages are from different calls (not the ones they are on). Perhaps this is where we are getting the huge number of dropped calls that are filling up the file and stopping the dialer at times.
2b. This may be related to the logging in and out of sessions. These customer hang up messages may be calls directed to the previous agent who had that session. When the previous agent does not answer, the customer hang up message appears on the screen for that session.
2c. While working with Agent Green (rep 162719) we captured some important information that leads to this conclusion. Friday, June 16, she logged into the dialer at 9:30 CT and took calls until 12:40 CT. During this time she had several calls where she received the customer has hung up message. That was the list I sent to you yesterday. At 12:47 CT she logged out for a few minutes and then logged back in. After logging in the second time, she had NO problems with this issue. Agent Green logs almost everything that happens and I rest assured of the accuracy of the data received. Please note: We checked her caller id records from Friday. Her caller id does not indicate that she received a call from the dialer at 9:30 am CT when she logged in the first time. However, it did indicate that she received a call from the dialer at 12:47 pm which was the same time she logged in the second time.
2d. It appears as if the previous agents phone was still connected to the dialer. However, the dialer let her log into the session. Perhaps the first rep was getting calls during her first session, when the agent did not answer, she gets a hang up call message.
2e. I want to check this out more thoroughly. Unfortunately, Agent Green did not record the session ids. Some of the calls she received during the first session are xxx, yyy, zzz. Please see if there were different session ids. Also, please provide me with the name and user id of the rep who had the first session id prior to Agent Green. I want to call them and find out if they received calls.
2f. Agent Green was on again today 8/17/2006 in session60 at 12:18 p.m. Can you let me know who was on session60 prior to her?

"customer not in session" errors when a customer is present is usually caused by either a misconfiguration in the dialplan or an error with the AST_update.pl script not running. There were also issues with SIP trunks getting this error, but those have been resolved with the snapshot that was released last week.

3. The hang up customer button is not registering on some answering machines and thus the agent has to listen to the entire message before the call can end. This is prevalent in cases where the answering machine is asking for a response – for Mary, press one. The rep presses the hang up customer button but the call does not end. Recording attached 231819. You can hear him press the hang up call button prior to my asking him to do so but it does not work. I can’t tell if the call times out or if the hang up customer code just takes longer to work in some cases.

This can be a system-load issue or if it is a SIP trunk call it may be a bug that was fixed in the most recent snapshot. What is the load on the system when this happens?

I am also wondering if this log in issue is providing some of the voice quality problems.
(Note on above comment - we seem to have quality issues on the same lines that agents complain of the 'bleeding' or overlapping - that is why she makes this comment)

voice quality issues are caused by 3 things:
- poor internet connectivity, latency, inconsistent transmission speeds
- using ztdummy instead of a hardware zaptel timer
- high load on the server

Are you using VOIP trunks? If so what codec?
What is the load on the server when this happens?

[AIRAM]

I've had similar situations as well. Actually even after pressing the hangup button the line is still open and a new call gets to the agent. It has got to the point were they have left conversations with other cutsomers on answering machines from the previous call.

I'm using IAX trunks. I just upgareded to latest snapshot so will test it out and let you know how it goes.

[mflorell]

AIRAM- What is the loadavg on your system when this happens?

[AIRAM]

Honestly I have not checked the load average though this has happened even with only 2 agents logged in since I am yet testing this server.
This is a Dell2650 server with 2.6 Ghz CPU and 2 GB memory so I didn't think that would have been the issue at that point but I will keep an eye on it and keep you posted. Anyway as I said before I already upgraded the software version so hopefully it doesn't happen again.

[kaconley]

I am monitoring the load today, but I don't think that this is the issue. It seems to be something with the logout. Either the user is X'ng out and not 'logging' out and then when they come back in (or another user gets in before them) they get the same session ID as the previous person. (So session ID is being kept for some unspecified period of time and given to someone who logs in closely after someone has disconnected - and maybe not logged out correctly.)

The overlapping (or reused session ID's) seem to cause the common conference problem where agents can talk to each other or multiple customers. (One agent can talk to another agents customers even).

We are using codec iLBC. We are connecting to the agents via PSTN. We are terminating calls from the dialer via SIP trunk to other carriers using iLBC or G711 (depending on the carrier).

More information later today...

Thanks! WONDERFUL PRODUCT!

[kaconley]

This is really just an issue of agent properly logging out and hanging up their phones. There is a way to prevent the sessionID deletion after logout which is to comment out this SQL statement in the vdc_db_query.php code:
DELETE from vicidial_live_agents where server_ip='$server_ip' and user ='$user';

It is not recommended since you may run out of sessionIDs during the day, and you need to make sure the vicidial_live_agents table is cleared at the end of your shift.

We have not changed that code at all. We have the opposite problem. Session ID's being reused - even when they are still assigned to someone. So, someone logs out and then 10 minutes later logs in, and they get the same session ID. (and if someone logs in before them, that person gets their old session ID and even though the original person gets a new Session ID they can hear each other) This is happening quite a lot - to the point where the agents can even talk to each others leads. (also yesterday when they load was quite light)

"customer not in session" errors when a customer is present is usually caused by either a misconfiguration in the dialplan or an error with the AST_update.pl script not running. There were also issues with SIP trunks getting this error, but those have been resolved with the snapshot that was released last week.

We are installing the snapshot this week - so I'll advise on this.

This can be a system-load issue or if it is a SIP trunk call it may be a bug that was fixed in the most recent snapshot. What is the load on the system when this happens?

We can create the situation even with less than 5 agents on. And we are normally running about 20 agents - so I'm sure it isn't a load issue.

voice quality issues are caused by 3 things:
- poor internet connectivity, latency, inconsistent transmission speeds
- using ztdummy instead of a hardware zaptel timer
- high load on the server

Are you using VOIP trunks? If so what codec?
What is the load on the server when this happens?

We are using G711 primarily. Server load light or heavy doesn't seem to change the experience. We are using ztdummy.

We are keen to solve this problem as we want to move to 200 agents over the next month. Our first step was putting in a dedicated link to one of our suppliers, the next step is the upgrade (the next few days), then we'll let you know if it is still happening. I don't know how to insist the the agents log out the 'correct' way. Some of them say they are pressing logout, some of them are (we can assume) pressing X to get out of their browser, and some of them are doing who-knows-what... We have to account for that somehow.

Thanks for your help.

[mflorell]

This is really just an issue of agent properly logging out and hanging up their phones. There is a way to prevent the sessionID deletion after logout which is to comment out this SQL statement in the vdc_db_query.php code:
DELETE from vicidial_live_agents where server_ip='$server_ip' and user ='$user';

It is not recommended since you may run out of sessionIDs during the day, and you need to make sure the vicidial_live_agents table is cleared at the end of your shift.

We have not changed that code at all. We have the opposite problem. Session ID's being reused - even when they are still assigned to someone. So, someone logs out and then 10 minutes later logs in, and they get the same session ID. (and if someone logs in before them, that person gets their old session ID and even though the original person gets a new Session ID they can hear each other) This is happening quite a lot - to the point where the agents can even talk to each others leads. (also yesterday when they load was quite light)

I don't think you understood. You should try commenting out the execution of that query in that script and see if it solves that problem for you.

voice quality issues are caused by 3 things:
- poor internet connectivity, latency, inconsistent transmission speeds
- using ztdummy instead of a hardware zaptel timer
- high load on the server

Are you using VOIP trunks? If so what codec?
What is the load on the server when this happens?

We are using G711 primarily. Server load light or heavy doesn't seem to change the experience. We are using ztdummy.

what are your zttest results for 6 hours of running zttest?

It is generally best to have a zaptel timer in your VICIDIAL servers. You can get them for $20-$40 so they are not that expensive for the quality improvement they can give you.

[roche]

Well I have some sound quality problems.

- poor internet connectivity, latency, inconsistent transmission speeds

I am using mtr and iptraf in order to check the connectivity seems I am having 3% of lost packages to our provider in several hopes, I think 3% is very high for Voip.

BTW, our asterisk machine is on a rack with only one Nic (two ips)
One IP has the gateway to the Internet connection, the virtual IP goes to the network (subnet /vlam) where the phones are .

- using ztdummy instead of a hardware zaptel timer

Dammit I am using ztdummy with out a hardware zaptel, How can I ensure that these is not an Issue ... ?

- high load on the server

Oh well ... last few day in this machine Dell2650 server with 2.6 Ghz CPU and 2 GB memory the load average rose to 9.0

The asterisk was using 37 % of CPU
and Mysql 20% .

How can I now what is causing these behavior?


That description seems worthy to be posted in Reader's Digest

[mflorell]

Looks like you have everything going against you.

I would first recommend getting MySQL onto a separate machine.

Second I would recommend buying a cheap X100P card($24-$40) and installing it in your Asterisk/VICIDIAL server.

Third I would recommend trying different providers to see if the network path to any other provider is better than what you have now. 3% dropped packets is bad, especially for predictive dialing VOIP.

a 9.0 is pretty bad loadavg, what is your average?

How many agents do you have on this single server?