How to optimize Vicidial Server for Maximum channels

[anandkumargupta]

Hi Team,

I am running Goautodial 2.0 with asterisk 1.4.27-1 on HP Prolient G7 Server where Database is running on CentOS 64bit and Httpd is running on Goautodial 2.0 on HP Prolient G6 Machine.

Asterisk Server having 24 GB RAM 2 physical 2.66 8 core processor(16 Processor) where Mysql having 4GB RAM and Httpd Server Having 4GB RM.

The server is dedicated for One campaign and 6 inbound groups where daily calls are above 16k.

The example of system load is

\---------- TOTALS, PEAKS and AVERAGES
Total Calls in/out on this server: 361
Total Off-Hook time on this server (min): 177.31
Average/Peak channels in use for server: 429.3764 / 856
Average/Peak load for server: 3110.6074 / 11108
Average USER process cpu percentage: 23.5516 %
Average SYSTEM process cpu percentage: 5.0487 %
Average IDLE process cpu percentage: 71.4032 %

The system works well but some time asterisk server hangs in the load but in the other hand server use to handle more load then the situation when system was hanged.

I am rebooting the server on daily basis and optimizing database on daily basis. I know that the bottleneck is not in database or httpd server.

How I can maximum optimize the asterisk server or architecture for best performance?


The steps which I am following right now.

* I am converting all files in GSM/alaw/ulaw.
* GW on different server and sending call to asterisk server through IAX with GSM.
* Endpoints are useing zoipper with GSM. ( to minimize the conversion of codec)
* Every 2 month I am archiving the data using the database archiving script.
* Clearing server_performance data regularly from db.

The top result is
top - 18:25:08 up 17:40, 2 users, load average: 13.29, 13.88, 15.47
Tasks: 251 total, 3 running, 248 sleeping, 0 stopped, 0 zombie
Cpu0 : 1.7%us, 0.7%sy, 0.0%ni, 78.7%id, 4.0%wa, 0.0%hi, 15.0%si, 0.0%st
Cpu1 : 5.0%us, 1.3%sy, 0.0%ni, 74.4%id, 18.6%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu2 : 4.7%us, 1.0%sy, 0.0%ni, 73.8%id, 20.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 8.0%us, 2.0%sy, 0.0%ni, 88.7%id, 0.7%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu4 : 6.3%us, 1.0%sy, 0.0%ni, 89.0%id, 3.3%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu5 : 16.6%us, 4.0%sy, 0.0%ni, 4.7%id, 73.8%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu6 : 6.6%us, 1.3%sy, 0.0%ni, 86.0%id, 6.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 5.3%us, 1.7%sy, 0.0%ni, 81.7%id, 11.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu8 : 3.0%us, 1.0%sy, 0.0%ni, 83.0%id, 13.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 4.3%us, 2.0%sy, 0.0%ni, 85.7%id, 7.7%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu10 : 8.9%us, 1.3%sy, 0.0%ni, 86.1%id, 3.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 11.4%us, 2.0%sy, 0.0%ni, 79.3%id, 7.4%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 7.6%us, 1.3%sy, 0.0%ni, 74.6%id, 16.2%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu13 : 10.7%us, 2.7%sy, 0.0%ni, 47.0%id, 39.3%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu14 : 44.9%us, 1.3%sy, 0.0%ni, 41.2%id, 12.3%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu15 : 5.0%us, 2.3%sy, 0.0%ni, 84.1%id, 7.9%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 24949400k total, 21037116k used, 3912284k free, 231204k buffers
Swap: 1044216k total, 84k used, 1044132k free, 20032676k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3954 root 15 0 389m 143m 5776 S 97.7 0.6 760:12.43 asterisk
4134 root 18 0 13284 8016 2736 R 19.9 0.0 123:04.42 AST_update.pl
4143 root 15 0 12796 7560 2672 S 2.3 0.0 15:02.44 AST_VDauto_dial
4627 root 19 0 6268 4428 1744 S 2.3 0.0 0:00.07 AST_send_action
4629 root 19 0 6268 4428 1744 S 2.3 0.0 0:00.07 AST_send_action
4631 root 19 0 6268 4464 1764 S 2.3 0.0 0:00.07 AST_send_action
4641 root 19 0 6268 4428 1744 S 2.3 0.0 0:00.07 AST_send_action
914 root 10 -5 0 0 0 D 0.7 0.0 1:24.41 kjournald
4652 root 18 0 3748 1848 612 R 0.7 0.0 0:00.02 lame
779 root 10 -5 0 0 0 S 0.3 0.0 0:03.50 ata/4
4137 root 15 0 10860 5432 2664 S 0.3 0.0 2:51.79 AST_manager_sen
7847 root 15 0 13480 6904 1512 S 0.3 0.0 0:01.89 FastAGI_log.pl
9243 root 15 0 13480 6900 1512 S 0.3 0.0 0:01.82 FastAGI_log.pl
1 root 15 0 2072 628 540 S 0.0 0.0 0:04.68 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.02 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.05 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.02 migration/2
free result is

total used free shared buffers cached
Mem: 24364 20309 4055 0 225 19322
-/+ buffers/cache: 761 23603
Swap: 1019 0 1019

vmstat result is

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
15 1 84 4100548 230584 19799816 0 0 1 127 35 15 6 3 81 10 0

[anandkumargupta]

Maximum user count use be be 130+

[Geil21]

I'm pretty sure that the meetme conference rooms only work under ulaw.

By using GSM, processor usage may increase as a result of the trans-coding.

If you truly want to reduce the load then the first thing I would do is standardize all connections to use ulaw exclusively.

If someone could confirm my thoughts as they are based on experience gained from version 2.0.3. The general rule regarding the proper codec may have changed since then.

[anandkumargupta]

Nice one. Let me confirm.

[Geil21]

Here is the link:
http://www.voip-info.org/wiki/view/Asterisk+cmd+MeetMe

Here is the comment:
Some additional notes on Meetme conferences

The conference bridge runs ulaw codec by default. If you let people connect with GSM or other codecs, Asterisk will use CPU power to convert audio between codecs

[anandkumargupta]

Hi Geil,

I think you are right as the comment was made in may 2004 but still I didn't find any document saying that the meetme is supporting other codec too. Let me change the codec end to end. It will take near about 4 days in my company as a process. I will share the result.

Hi Team,

Please provide me more input to improve the performance

[anandkumargupta]

Hi Team,

Today we implement the ulaw but no profit I think. There is no different.

After Implementing the ulaw.

---------- TOTALS, PEAKS and AVERAGES
Total Calls in/out on this server: 346
Total Off-Hook time on this server (min): 210.71
Average/Peak channels in use for server: 281.3478 / 424
Average/Peak load for server: 1136.4460 / 7675
Average USER process cpu percentage: 8.4447 %
Average SYSTEM process cpu percentage: 2.3845 %
Average IDLE process cpu percentage: 89.1876 %

---------- LINE GRAPH:
rows: 6348 tick: 90 minutes scale: 15 minutes

Before implementing the ulaw.

Server Performance Report 2011-08-11 21:24:09
Time range: 2011-08-10 09:00:00 to 2011-08-10 21:00:00

---------- TOTALS, PEAKS and AVERAGES
Total Calls in/out on this server: 384
Total Off-Hook time on this server (min): 199.32
Average/Peak channels in use for server: 294.0399 / 490
Average/Peak load for server: 1197.2078
/ 7117
Average USER process cpu percentage: 9.2623 %
Average SYSTEM process cpu percentage: 2.3297 %
Average IDLE process cpu percentage: 88.4151 %

---------- LINE GRAPH:
rows: 6093 tick: 90 minutes scale: 15 minutes

[williamconley]

please try to state the problem clearly (it seems mixed in with the data)

asterisk server hangs? describe the problem itself. demonstrate that there is a problem at the operating system level if you can to begin tracing the problem to its root (htop or iostat or mytop) and summarize the findings so we know what to look for (high wait time on a HD? long query wait in mytop? server load of 20 on an 8 processor system?)

be sure to identify the servers being used clearly with something other than the model number (ie: we have ONE database server, TWO web servers, FOUR asterisk servers ... then refer to them by purpose and number such as "Web server 2")

THEN show data to help us track down the issue AFTER clearly stating your issue