slow web interface (http) but perfectly clear media (rtp)

[praz]

Hi guys,

Goautodial CE 2.0
VERSION: 2.9-441a
BUILD: 140612-1628
Intel(R) Xeon(R) CPU 16 core 2.27GHz 24GB RAM
Number or seats in the call center: 10

I have a tricky situation here but hoping to get some help please . When my call centre agents connect to a remote dialer hosted on another network, sometimes the time on the agent interface freezes or skips a few seconds. The information of the client is also slow to pop up on the agent interface after the call connects. What is strange is that the voice quality is perfectly clear at this point and this issue does not happen to all the call centre agents at the same time but rather one agent at a time.
When we move the dialer server to our call centre, there is no agent screen freeze or time skip experienced any more.My call centre has a dedicated 10Mb connection up and down and the dialer is hosted in a very bandwidth rich data centre with very good internet and hosting other clients.
All the agent pcs have either 3GB or 4GB ram and have core duo at minimum. At the call centre we use a draytek 2830.
The agents connect to port 80 HTTP on the dialer.

If you have experienced an issue like this in the past please share your thought.
Thanks

[bobchaos]

I wouldn`t rule out networking issues just yet: If there's any kind of QoS running on there, RTP would be the last thing to take a hit. Try some packet capture, see if those http packet need to be resent often. You should probably check the average latency with the remote site, just to be sure.

Also, I understand you use different clients when testing internal VS hosted, that could be part of the problem. If the interface freezes it could be the client can`t refresh as fast as they need to. Vicidial is optimized for Firefox, is that what you're using?

[praz]

Thanks for the reply. WIth regards to the networking, qos has been disabled totally at the call centre site on both the draytek vigor 2830n router and by the ISP. Infact, the ISP did confirm no QoS or priority was set at both the call centre site and the data centre site (where the server is hosted). I also believe RTP should be the first to be affected but I see the draytek has an option checked by default 'allow voip priority first'. I asked the ISP to run some pings (10000 packets) between the call centre front end router and the data centre switch and we had no packet loss both ways 'Success rate is 100 percent (10000/10000), round-trip min/avg/max = 4/8/36 ms'.

[praz]

Also, the client agents use Firefox. You mentioned refresh rate. Is this something that can be checked on the dialer and how do I see this please? The average latency looking at the 'qualify' value from the asterisk console is 100 milliseconds.
Would the default management port number of the draytek (port 80) be a reason for this issue? I am thinking of changing the http port number for the dialer but I want to be sure it is a good move.
Is there anything I could try or change on the dialer or draytek router to help this pls?

[bobchaos]

Those sound like great networking stats and they shouldn't be an issue. No QoS makes me uneasy, but there's tons of folks who do it that way and just throw more bandwidth at it if it becomes a problem. I seriously doubt that your router's management interface is causing the interference since it's on a different device and you don't do traffic shaping anyhow.

I have to say I'm a bit stumped here. If I were in your position, I'd break out Wireshark and start following HTTP streams between a slow station and your web server. Sometimes it's the only way to trace back the source. Once I had a major issue with sound quality and Wireshark was clearly showing delayed packets so after spending days poking around the forums and the web I just got angry and gutted the network. When I reinstalled the whole thing using a good old Cisco Router instead of some garbage Sonicwall everything started working fine even tho both devices were configured the same. I guess that's something you could try, setup a quick lab using a sturdier networking device and see if the issue is reproduceable in your lab.

Regarding the refresh rate, I'd didn't mean it as a setting, just "how fast can the client display stuff on screen". It's an unlikely issue, given that the interface is very lightweight, but who knows, maybe your client stations are 486s o.O .

*Edit*

You may want to check out other forums too: Vicidial forums are loaded with Vicidial experts and Asterisk experts, but you have a networking/webserver issue. More expertize to be found elsewhere like expert exchange or the Apache forums.

[mflorell]

Could be your database getting slow, which can cause slow web responses.

How many records are in your vicidial_list, vicidial_log, vicidial_agent_log tables?

Is this an all-in-one vicidial system?

What is the line-to-agent ratio that you are using?

As a side note, since it was mentioned, QoS is always a bad idea with Vicidial use.

[praz]

Thanks for the reply guys. I appreciate the help so far.

Matt the number of records in my tables are as follows: vicidial_list: 826586; vicidial_log: 67408 and vicidial_agent_log: 15577. I would like to say at this point that I only recently moved the client data (campaigns, lists and callbacks) to this dialer from their previous dialer system which was operating on vicidial version 2.7 (an older version).

Asterisk 1.8.23.0, mysql 5.0.95 and httpd Apache/2.2.3 are running on the same server (I assume this is what you mean by all-in-one vicidial system)

I also have a cronjob set to optimize the entire database every morning
40 2 * * * mysqlcheck -u root -pvicidialnow --auto-repair --check --optimize --all-databases

The line to agent ratio that we use on the dialer is 3-to-1 on average (dial method: ratio) so there is probably a max of 24 calls going out at a time.

What do you think about this and do you have any reservations about running a callcenter from a Draytek vigor 2800 router?

[praz]

Apart from the freezing or skipping time on the agent web interface, other symptoms of this issue include the following:

- The web browser shows the round circle on the browser tab 'indicating connecting'. It is like it is waiting for information from the dialer.

- The agent web interface could appear to be in the ready state but the vicidial report interface would either indicate the agent is on a dead call 'Dead' (in most cases) or 'Paused' (in some cases). The quick way to fix this would be for the agent to logout of the interface and login again.

-Sometimes when the call comes through to the agent and the agent is speaking with the called party, the information of the called party could take up to 10 seconds before it comes up on the agent interface.

Taking the dialer to the client site (placing it on the LAN) fixes this issue entirely but we have to host it in our data centre. Also some other dialers are hosted in the data centre and work very well.

Any ideas would be appreciated pls...

[bobchaos]

Did you try getting some data with Wireshark? If it's an all-in-one box and the only factor is the server's physical location there's a solid chance the problem is somewhere along the highway, so to speak. You could start by having a look at your Firewall/router's stats, but if you see nothing blatant and obvious, you'll need packet captures.

[praz]

hi bobchaos. I don't have access to the client pcs at the moment making the wireshark check hard at the moment. I would review the network again but if there is any other I thing I can check pls advise. Thanks again

[bobchaos]

You'd want to monitor the server anyhow, it's easier than installing and running it on all your clients o.O

If you've never used it before, under *Nix shells the package is (usually) called tshark, you use that to grab a capture file, then SCP it over to a workstation where you load it in Wireshark GUI for analysis. I'm pretty sure it's in the default yum repos for CentOS.

[praz]

thanks bob. yes I usually run it from my linux centos server and I have tshark installed. unfortunately there is a holiday here in the UK tomorrow (monday) so I have to wait for Tuesday when the agents login before I can do this. But running a trace on http port 80 is a great idea I have not tried. I would be checking for packet retransmissions and duplicate ACKS which would indicate packet loss using the Analyze -> Expert info. Please advise if you have an alternative way to check for packet losses and I would be happy to try this.

Thanks again

[praz]

Hi guys, this issue is related to Lags and errors on my network (which I can see on wireshark). I know there are a few pages on the forum about the dialer's strict connection and I appreciate the patch by aouyar and matt in my version. Matt please can you explain a bit about this
$conf_check_attempts = '3';
If I increase this value to 15 for instance would this help? What do the "attempts" count? seconds or errors?

Thanks

[mflorell]

You can try increasing that if you want, but we left it at 3 because in our extensive testing on bad networks it really didn't help to put it any higher than that. At some point the extra traffic from the re-attempts actually makes things worse.

[praz]

Hey guys, Issue seem to be fixed now. I enabled KeepAlive on the httpd.conf (etc/httpd/conf/httpd.conf) and set the MaxKeepAliveRequests to 0.
I still see lagged but much less than before so happy days ahead. Thanks to other forum users.
Matt just 1 question please. how many seconds do an agent need to lose web connection before they get the lagged pause status?

[mflorell]

I believe that 9 seconds is the amount of time before an agent is considered lagged. There are a couple of different processes that can lag an agent, and the time are different for each.

[praz]

Thanks Matt... noted