ViciBox v.7.0.4 Frequent Time Sync issue

[arunvsadnikov]

Hi Team,

Current Setup.

ViciBox v.7.0.4-170113
Asterisk 11.25.1-vici
Vicidial Version: 2.14b0.5
SVN Version: 2659
DB Schema Version: 1485
32 G RAM, 2 TB HDD, 8 Core Xeon Processor

We currently experiencing "Time Synchronization" issue with vicidial. We got ~22-25 agents dialing on an outbound campaign with ratio 6 with answering machine detection and call recording enabled. We got ~100k leads in the vicidial_list table as well, system load will be varying from 2-4. We are seeing this "Time sync" issue now very frequently and I had to stop and start vicidial to recover. Im pretty much sure its not related to Time instead something broken with Asterisk. I see the following logs in asterisk messages. Anyone experienced this issue before? IS this a capacity problem? or a bug ?

[Jul 10 00:14:23] WARNING[28709][C-000229d4] channel.c: Unable to write to alert pipe on Local/58600085@default-000217e5;1 (qlen = 39): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29282][C-00022a43] channel.c: Unable to write to alert pipe on Local/58600075@default-00021846;1 (qlen = 33): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[19966][C-000224ad] channel.c: Unable to write to alert pipe on Local/58600058@default-000212f2;1 (qlen = 35): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29271][C-00022a41] channel.c: Unable to write to alert pipe on Local/58600057@default-00021845;1 (qlen = 51): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29289][C-00022a44] channel.c: Unable to write to alert pipe on Local/58600084@default-00021847;1 (qlen = 81): Resource temporarily unavailable!

FYI, access to the server is very limited and only specific IPs are allowed to connect to SIP(udp 5050) and RTP(udp 10k-20k). I dont see any dos attacks in dmesg as well.


Thanks
~Arun

[arunvsadnikov]

Can someone please help us here? This is impacting our production

[mflorell]

I would suggest upgrading to a more recent svn/trunk revision of VICIdial. Also, look at the Modify Server page and confirm the settings are correct, like the Asterisk Version.

[arunvsadnikov]

Hi, thanks for the suggestion. We have some custom code changes in vicidial and Im afraid to upgrade it. Is this is something with vicidial or asterisk? Can we try upgrade only asterisk in vicibox?

[mflorell]

The version of VICIdial code you are using will not work with the newer version of Asterisk. There were several VICIdial bug fixes since the version you are using, I would strongly suggest upgrading.

[arunvsadnikov]

The version of VICIdial code you are using will not work with the newer version of Asterisk. .

There are many modifications, I will get in touch with the dev guy and plan for an upgrade.

I have attached the server performance report for one day, Im just wondering is this some kind of capacity issue with asterisk, like software limits. Any feedback on this will be appreciated.

[williamconley]

Server load is visible on each server with "uptime" and/or "htop". If the server is a web server, it's also available at the bottom of the agent list in the real time screen. Server load should never exceed the actual number of cores (safest cruising average server load is half the number of cores).

If you're not exceeding the number of cores for your average server load, it's probably a networking issue causing packets to drop between your agent's web browsers and your server. Especially since it sounds like you have a single-server system, time sync errors can NOT be actual "sync" errors since there's only one server. So the odds are that you have a networking issue. Network packets dropping can cause the system to think there's a sync error because those packets update a time field (missed updates = time is out of sync error, just like actually being out of sync on a multiple server cluster).

Additionally, the asterisk error you reported should also be chased down, but you did not respond to matt's suggestion to verify the admin-> server values with any direct information (like ... the settings in admin->servers including, but not limited to, the actual entry for Asterisk Version). It may not be a bad idea to also list the asterisk version from /etc/astguiclient.conf (and if either of those has changed, you have a bit of an issue since changing those values often requires updating some configuration files which is best done by running the Vicidial install package which might overwrite any custom files).

[arunvsadnikov]

Server load is visible on each server with "uptime" and/or "htop". If the server is a web server, it's also available at the bottom of the agent list in the real time screen. Server load should never exceed the actual number of cores (safest cruising average server load is half the number of cores).

Server load varies in between 2-4 depending on the number of agents online, we got a 8 core cpu, so yes it may go up to load 8 until fully utilized.

If you're not exceeding the number of cores for your average server load, it's probably a networking issue causing packets to drop between your agent's web browsers and your server. Especially since it sounds like you have a single-server system, time sync errors can NOT be actual "sync" errors since there's only one server. So the odds are that you have a networking issue. Network packets dropping can cause the system to think there's a sync error because those packets update a time field (missed updates = time is out of sync error, just like actually being out of sync on a multiple server cluster).

I got a chance to take a look at our PRTG graph and couldn't find any packet errors or bandwidth issues. Im working with my service provider for more details on this.

Additionally, the asterisk error you reported should also be chased down, but you did not respond to matt's suggestion to verify the admin-> server values with any direct information (like ... the settings in admin->servers including, but not limited to, the actual entry for Asterisk Version). It may not be a bad idea to also list the asterisk version from /etc/astguiclient.conf (and if either of those has changed, you have a bit of an issue since changing those values often requires updating some configuration files which is best done by running the Vicidial install package which might overwrite any custom files).

Sorry about that Matt, I have verified asterisk version in GUI, its 11.25.1 and in /etc/astguiclient.conf it is 11.

My current plan is to upgrade vicidial code to the latest trunk and work on the custom modifications we have done with the dev guy, btw this may take some time in testing, bug fixing etc. So I decided to add one more telephony server to current setup.

I got a couple of question for you William:-
1. The current installation is vicibox-express, can we convert a vicibox-express installation to cluster without reinstalling the entire stuff? I was trying to add a new telephony server to the existing express installation in my local environment and the installer failed back to Legacy mode. Do you recommend Legacy installation of new Telephony server?

2. Is it possible to checkout a specific revision of vicidial svn during the vicibox-install instead of trunk?

Thanks for all your support. Appreciate that!