All agents automatically paused!

[rajeevpn]

Facing this strange problem the last week or so. All agents go on pause at the same time automatically - there is no response from the server at this point. The following things happen then:
1. The customer is not able to hear the agent
2. All buttons on the VICIDIAL become 'unclickable' - cannot hangup / resume - nothing at all

Sometimes they are able to logout and re-login. Before they re-login we do a AST_resetmysqlvars just to make sure and they can continue calling for a while.

Versions used:
Asterisk : 1.2.17
VICIDIAL: 2.0.3

* Number of agents: 30 - Workstations running Fedora Core with idefisk on gsm codec
*1 asterisk server + 1 DB server - both dual core opteron with 4GB RAM and SATA HDD
* Load average < 3
* Calls going out on 2 E1 PRIs using TE205P
* No recording
* Server been running fine for over 3 months with same specs

What the logs show:
1. VNAK received by chan_iax2.c
2. full:Oct 19 10:19:58 WARNING[2756] res_agi.c: Connect failed with unexpected error: Network is unreachable
3. app_meetme.c: Unable to write frame to channel Local/8600072@default-06c3,2
4. Hangup event logged for IAX2 clients in listen.YYYY-MM-DD

We looked at the switch - didn't see anything unusual but they are regular unmanaged DAX switches.

Any assistance appreciated!

[mflorell]

Do both servers have ntp installed and time syncing to the same NTP time server?

[rajeevpn]

Yes - they both have NTP installed. the DB server is connected to the internet and talks to a public NTP. the asterisk server, the workstations, talk to this server to do a time sync

[mflorell]

Have you tried installing "mtop" to see if there is a database quesry that is sticking and locking everything up?

[rajeevpn]

That was it!! Did an mtop and saw a report gen. query causing the problem. when we don't run reports when agents are on calls, things seem fine.

thanks matt!

-r

[mflorell]

I'm glad I could help, I have seen reports cause issues before at one client. In fact I have the mtop screen up on one manager's station so he can see when one of his room managers is going report-crazy