Exceptionally Long Queue Length & Timesync error

[dgroth02]

Switch is at 10% utilization, 0 port errors or throughput errors, put servers outside the firewall with no NAT
Still same results - agents are getting dead calls with no audio and delayed data/no ability to transfer

I'm watching the DB/web server - its hardly breaking a sweat - no DB errors or mysql-slow log entries

Interesting thing I was told by one of the agents. Manual callbacks work beautifully. No issues at all.

Almost like a problem with the dialing scripts. But again - none of the servers are having CPU utilization issues, interface issues, switch performance issues. I have tried spreading the trunks out, putting them on the same server as the agents. Same issue - no improvement.

The owner informed me today he has had enough - he will be moving to another platform.

I'd like to know what this issue is in case I run into it in the future - anyone care to look before I shut it down? I guess I can say it did in fact solve the exceptionally long queue length - I haven't had to reboot ANY of the servers and haven't gotten this error at all.

[dgroth02]

Well,

I would consider this the last post for this troubleshooting effort.

After 3 days of troubleshooting this issue, and trying all suggested efforts listed here with little or no improvement (removing firewall from loop, replacing switch, etc), the hammer has come down. The call center owner has decided that enough is enough and is going with a different hosted dialing provider instead of his own servers and carrier.

Strangely enough - it is a hosted version of the EXACT version of Vicidial I've installed - however, I spoke with the person on the phone that runs it, and he said that they use some special "patches" and such to "make Vici work" and to "prevent strange behavior" - It does make me wonder what exactly they had to do to make their work properly, but of course because they are selling a hosted service and wish to maintain a properly running product (and a customer) - I was unable to obtain this information. If anyone has had to do such changes, why aren't they part of the standard set of scripts I wonder?

The owner still may use the system as a backup, but will have Vicidial.com folks log into the server and possibly "look at it". Needless to say this has been frustrating.

If I ever do find a solution, I will happily post back to this thread.

Good luck!

[dgroth02]

I should add two tidbits of info I was able to find out:

1. Outbound manual calls and inbound worked just fine. No issues whatsoever. So the autodialing script must be at fault somewhere in there.
2. I NEVER had any issues like with Asterisk 1.8 on Vicibox . Seems like Asterisk 11 & 13 have been absolute shit with Vicidial.

[williamconley]

i'm gonna suggest you try it again and leave them outside the firewall while you work out why individual calls don't have audio.

This is our solution, as a rule: We don't let anything other than iptables make decisions. IF anything other than iptables is making decisions, then the solution will always be somewhere in the firewall, and firewalls are all different. Every manufacturer has their own coined terms and descriptions and confusing menu system.

In fact, some manufacturers have more than one system from one piece of equipment to the next. There are no actual rules.

[rfugate]

In regards to the Exceptionally Long Queue Length and Timesync errors:

I was experiencing them daily, up to about three times a day across various servers, agent or non-agent alike. I replaced all of my network gear, reran cables, turned things off and on again, a whole host of troubleshooting modifications to no avail.

Then I came across a thread in an asterisk forum about this specific problem, and one person recommended editing the /etc/asterisk/rtp.conf file, and increasing the port range. The conf file had the range set from 10k to 20k, but said that 5k to 31k is the default. I modified the rtpstart to 5000, and rtpend to 31000, and made these changes across all of my asterisk systems. Some folks in that asterisk thread said this worked for them, some said it didn't help their issues, but I really needed a win.

That was two weeks ago, and I haven't had a single timesync issue since! If anyone else has this problem, and is absolutely confident it's not ntp related, this could be a potential solution for you.

[mcargile]

I am 99% sure that your server is being hit by a SIP brute force attack. Do you have port 5060 locked down to only communicate with known IP addresses? There is a form of SIP brute force attack that does not normally output any NOTICE or WARNING messages in Asterisk. As such there are very hard to spot. They only show up if you enable the "security" logging in logger.conf but this ends up generating a massive amount of log output as every Vicidial action triggers like 4 security events. These brute force attacks are annoying when using Asterisk 1.4, 1.8, and 13, but to Asterisk 11 they are a nightmare. Asterisk 11 becomes unstable. When it does you get these exceptionally long queue length errors (possibly because Asterisk cannot free an RTP port fast enough) and the AMI interface locks up. The AMI locking up turns around and causes the update script to lock up. The update script locking up causes the time sync errors.

With Asterisk 11 you need to lock port 5060 down so it can only communicate with your carrier and any remote centers where agents are located. If you have at home agents give them IAX2 phones. This is in general a good security practice for all Asterisk versions but it is pretty much a necessity for Asterisk 11.

[williamconley]

... The update script locking up causes the time sync errors.

Interesting. I was thinking we've never had this problem until I reached this part of the comment. We did have an (apparently until now) unexplained RED server with time sync errors that always traced back to the update script. We just restarted the update script and everything went back to normal.

However: This server was whitelisted (they all are). Which leads me to the very possible conclusion that someone they authorized on their IP list was in fact to blame. They have not had this problem in a long time, but if it ever comes back I will have a look.

[dgroth02]

For reference - For my SIP providers and only the allowed SIP phone IPs were whitelisted - EVERYTHING else was not allowed. Web, ssh, everything. I did not try the RTP issue (although if memory serves, I believe I expanded the RTP range in desperation from 10000-20000 to 10000-50000 with no change.

[williamconley]

For reference - For my SIP providers and only the allowed SIP phone IPs were whitelisted - EVERYTHING else was not allowed. Web, ssh, everything. I did not try the RTP issue (although if memory serves, I believe I expanded the RTP range in desperation from 10000-20000 to 10000-50000 with no change.

so the server didn't have any agents or managers on it? was it just for balance dialing?

[dgroth02]

Didn't matter how I structured it - tried keeping the servers with agents on them and dialing more or less on the same server, also tried using agent-only servers and dialing-only servers. Same results.

[williamconley]

Didn't matter how I structured it - tried keeping the servers with agents on them and dialing more or less on the same server, also tried using agent-only servers and dialing-only servers. Same results.

Are you saying that you told the agent to stop registering to that server, or that you actually cut off everyone's access to that server except the carrier and the other vicidial servers?

If agents/managers still had open SIP ports, you could still be experiencing the described issue (which was precisely what I think may have been happening on our server a couple years ago, when it was resolved on its own before we could find the cause).

[dgroth02]

Everyone’s access to the server was cut off except for the agent ip’s and the carrier ip’s everything else was a blacklist (ie zero access on ANY port from ANY othe IP)

[rfugate]

Do you have port 5060 locked down to only communicate with known IP addresses?

I have my sip vendors whitelisted in my rules, but not to a specific port! I'll have to lock that down. Thank you very much!

[geilt]

Hi, I know this post is old, but in regards to the No Audio, make sure you don't have any custom extensions or scripts you are routing with that have AGISIGHUP=no in it.

We spent hours scouring the Vici code to solve an issue with Pauses coming back with no Audio.

We found the issue had to do with a script we were running that was using AGISIGHUP=no before the extension to allow for the PHP script to cleanup on channel hangup by the caller.

The Park Call Script in Vici runs a sleep(360) command which is killed by another process when the call is transferred back to the agent.

But with AGISIGHUP, it would wait the full 360 seconds, and then actually transfer back. It would APPEAR to have no audio, but in fact the script just didn't finish executing and reconnecting the channel. By this time thought, either the agent or the client has hung up the phone which makes it appear like a no-audio issue.

We had to resort to installing pcntl for PHP to then, just at minimum, call pcntl_signal(SIGHUP, 'anyfunction' ); in our scripts to allow for channel cleanup instead of killing PHP on SIGHUP. The function doesn't even have to do anything, just catch the Signal, and then AGI functions as it should properly on channel hangup.

We thought scripts were behaving oddly because the AGI Command for Asterisk 13 is supposed to have DeadAGI functionality.

So check your scripts if you have any!

[williamconley]

Great postback. Is this on the latest SVN version of Vicidial? Do you have a DIFF? Have you seen the "Vicidial Issue Tracker" link at the top of this page? It would be most excellent if you were to post a DIFF against the latest SVN of the file you altered to fix the problem along with an explanation in the Issue Tracker. For one thing, after the fix is approved you would be able to upgrade without ReFixing it.