Agent Paused/ Syncing issue between client & server

[callcrazy]

From vicidial.php around line 3684

Code: Select all

                                                if ( (VLAStatuS == 'PAUSED') && (AutoDialWaiting == 1) )
                                                        {
                                                        if (PausENotifYCounTer > 10)
                                                                {
                                                                alert('Your session has been paused: Click OK->AUTOP->Resume');
                                                                AutoDial_ReSume_PauSe('VDADpause');
                                                                PausENotifYCounTer=0;
                                                                }
                                                        else {PausENotifYCounTer++;}
                                                        }
                                                else {PausENotifYCounTer=0;}


I added in the alert so the agent knows they were paused and so they select a pause status we call "AUTOP" for tracking purposes.

We have 20-30 agents on the system daily and receive anywhere from 10-40 of these events per day.

We also notice sometimes agents are stuck in a DISPO status but on the agent screen they see a status that they are ready to take calls.

Appears to be a mismatch between agent browser and the servers state.

I've been watching packets through the switch and noticed that around the time of the AUTOP status being pressed by the agents around 10 seconds prior to this there is a gap in time around 4-6 seconds where no packets are being seen in wireshark from client<-->server.

I've checked apache logs and notice apache IS receiving packets in the window where wireshark doesn't show activity but the datetimes on in the log get out of sync and sometimes are not in chronological order. This indicates to me that the agent system IS sending API calls to the server but apache is not responding back to them in a duration of time and this causes the client to pause the agent.

Apache is running on the same server as mysql. Asterisk is running on a different system.

Running apachetop on access_log I see about 50-60 requests per second

Apache configurations that I believe are relevant:

Code: Select all

Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients    256
MaxRequestsPerChild  10000
</IfModule>

<IfModule worker.c>
StartServers         3
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  10000
</IfModule>


4 threads CPU available with a LA around 2.0@5&15m
10gig RAM
64k in paging and doesn't seem to be used.

QUESTIONS:
1) Is this a congestion issue?

2) Thoughts or ideas on other things to check?

3) It is a guest on an ESXi host that has 16 threads and 32gig ram available. Looking at the utilization of the total host including other guests it runs about 20-50% utilization. Could it be a VMWare issue?

4) At what point is having a second apache server (load balancing) required or beneficial?

[omarrodriguezt]

Hello check the sync issue in the wiki and let us know if thats helped you.
http://wiki.vicidial.org/index.php/VICI ... o_an_agent

[callcrazy]

@omarrodiguezt Time is tip top all around with same hour and TZ on all machines.

All network connectivity is over our LAN and appears to be ok and nowhere close to being maxed out but still looking into this.

I wireshark and see very few dropped or red packets.

My main concern is apache configuration because this is the only place I'm noticing the logs have the times out of chronological order which would imply the requests are taking seconds to fulfill and I'm not seeing anything showing up in the mysql slowquery log with slow query time set to 1 second.

[omarrodriguezt]

@omarrodiguezt Time is tip top all around with same hour and TZ on all machines

What you mean by is tip top?
Is the same?

[callcrazy]

That it is identical on all systems.

[omarrodriguezt]

Ok. can you please posts the exact error?

[callcrazy]

There is no error.

1) The agents find themselves in DISPO status from the server report but their client shows they are in READY state waiting for calls yet no calls will ever be sent to them.

2) The agents press hangup to disposition a call and then are presented with the pause status screen as they have been automatically put on pause because the client has lost sync with the server.

[omarrodriguezt]

Are the agents behind a proxy?

[callcrazy]

Agents are setup on a proxy for outbound traffic but the web servers internal bypass that and go direct. I have thought about a reverse proxy between agents and the web server but the web server is only operating at 25-50% cpu/ram usage so I'm not sure if that would help.

I have made a change to apache configuration that seems to have made a major impact on disk utilization in some way:

Code: Select all

<IfModule prefork.c>
ServerLimit 768
StartServers 450
MinSpareServers 250
MaxSpareServers 500
MaxClients 768
MaxRequestsPerChild 1000
</IfModule>

mflorell recommended this in another thread. I switched our configuration to this and restarted apache.

After midnight this morning our disk i/o dropped to 0 and this may have fixed the problem but am unsure why it didn't take effect until midnight.

[omarrodriguezt]

Sounds good!