Asterisk Locks Up Server Load Averages 50.25 48.49 33.49

[iratik]

We have been through 3 servers in 3 days. Whenever we put agents on the system and begin calling (manual dial). We are able to make calls for about 30-40 minutes. The load average slowly climbs until asterisk becomes unresponsive. We had a Rackable server with dual 2.7 ghz AMD processors and 8 gb ram. We had a 2.4ghz dual core off-the-shell PC. Now we have a custom-built quad core 2.4 ghz AMD athlon box with 2gb ram. The same symptoms happened every time and we thought it was the hardware. It has to be the software.

This is on a base install of vicidialnow 1.3. All we've done is add statuses, agents and leads. We use SIP based ITSPs (no analog cards etc..).

I would have posted the top results but this board will not let me.

[mflorell]

number of agents?

SIP codec?

Have you tried ViciBox?

[iratik]

5-7 agents . Sip codec is ulaw/alaw (no g729). Tell me to try vicibox and i'll try vicibox. Please tell me there is some known issue that causes this for which there is an straightforward fix.

[mflorell]

I have never seen what you describe happen on a ViciDial system before, especially given the specs you describe. It can't hurt to try vicibox.

[iratik]

We just got done with the vicibox install. It was a bit arduous but I've got it up. What are your rates for private consulting / logging via ssh and sorting this all out if this solution does not work?

[mflorell]

just email sales -at- vicidial.com,

[iratik]

We load tested the system by logging up all agent stations and calling a special number we set for MOH to play infinitely. The relation between number of concurrent channels in use and load average was positive however, the amount the load average went up was not nearly as much as expected were the number of concurrent channels the cause behind the high load averages. At 8 concurrent channels after 20 minutes of activity we had load averages around 1.5. It has been suspected that the call recording functionality of vicidial is causing the issues behind load as lame and sox were seen maxing out 2-3 cores of our cpu at a time. This theory was supported by our test results as the determining factor in high load averages seemed to be related to the number of calls ended in a certain period of time. More calls ending means more work for sox to mix the audio and work for lame to encode the audio.

We setup our oreka open-source network sip recorder to monitor calls from vicidial so we could turn the vicidial recording off in case of needing to test this theory. The next morning we ran 7 agents at 30 minutes test intervals in production calling. The load averages began to rise after 15 minutes to 6 and at 20 minutes to 20 and at 25 minutes into the 80's. Lame was observed at the top of our htop reports and iostat was reporting waiting on io due to lame/sox. We shutoff call recording.

The next test ran for 30 minutes with a peak load average of 2. Spikes continued to be seen mostly due to the script /usr/share/AST_update.pl. Upon inspection we observed a large amount of database activity due to system performance monitoring. System performance monitoring was activated on our previous installations and may have contributed to AST_update taking too much CPU.

The next test ran for 30 minutes with a peak load average of 2.0. It should q

We also modified /etc/asterisk/logger.conf and removed debug levels for warning,error and debug. We were getting errors related to "channel deadlock" which we had in the past considered to be the reason why asterisk was taking taking up so much cpu.

Our recommendation is for astguiclient to provide the option to limit , balance or defer processing of audio files so as to provide controls for limiting encoding/mixing cpu usage for sox and lame. May need to look at AST_update and do some performance oriented analysis on that script. We observed the code had some routines dependent on the number of phones. The default installation of vicibox contains far more than the 20-30 phones we needed. We removed the surplus phones but we had very little experience profiling perl based scripts and were unable to quantify any performance improvements.

It may be worth checking out the --single-threaded option in sox to see if that can help limit cpu usage in multi core systems. The switch -f in lame may allow lame to encode to mp3 faster so there is less build up during periods in which large numbers of mixed *-all files are present to be encoded.

[mflorell]

I'm not really sure what is going on with your system, but we do have very large clients that are doing phone-login-load-balancing that are registering 250+ phones to each server, so I know that issue with ASTupdate is not causing a great load problem.

As for recording, we have a client recording inbound IVR as well as agent conversations and they are doing upto 150 recordings concurrently on a single Intel Core2quad with 4GB RAM with no severe load issues.

We have clients using ViciDial for more intensive campaigns on lesser hardware so I don't really know what problem your systems are suffering from.

[iratik]

Not sure if its relevant but we have setup several other clients on vicidial as well and have never seen this issue. All clients we have setup were setup over a year ago on versions pre 2.0.5. (2.04-152). I suspect this has to do with 2.0.5+ versions of astguiclient and their processes for recording and mixing. You may be able to enhance the scalability of vicidial if you use mongodb or couchdb for some of the objects that are less relationally dependent.ou can also use mongodb/couchdb for event queuing.

A memcached layer on top of mysql can reduce the number of queries/second on mysql. (E.g. cache the server setup, pull the results from the cache in all the AST_* scripts instead of the db, expire the cache any time an admin change is made). We also noticed the large number of polling requests astguiclient generates to maintain contact with the server. Though this is backwards compatible with many browsers it may be worth looking into some sort of long-polling or other type of live updating technology (E.g. eventmachine, twisted).

[okli]

Mixing and encoding are just a couple of cron jobs, why don't you set mp3 encoding to take place when server is not busy?
You''d need the mixing script to run often, so your ram drive gets freed up regularly, but the encoding and ftp scripts could be moved after shift.

What type of hard drives are you using and in what configuration/raid level?

[mflorell]

All of our large clients are running 2.2.0 or higher ViciDial and we are not seeing issues like this.

[gardo]

I'd recommend you use the RAM drive for the initial storage of the recordings. Recording all calls creates a bottleneck on the harddrive and causes high system load. Using the RAM drive will remove this bottleneck.

You can also re-schedule your mp3 encoding as Okli suggested when the server is not busy. Encoding to mp3 eats a lot of CPU resources.