vicidial.org

by **iulianm** » Fri Apr 01, 2011 7:37 am

Hello

I have a cluster of 2 servers.

1. Web + db + asterisk
2. Web + asterisk

When I put more then 45-50 agents on the system ( 25 extension on 1st server, 25 extension on 2nd one )

I also balance both to use the both webservers , and in this moment the 2nd server takes to long to respond .. the agent interface loads in 2 minutes. I cannot do anything.

The load on 1st servers is:

load average: 3.13, 2.82, 2.15

The load on 2nd server is:

load average: 0.53, 0.55, 0.36

I tried to put all of the agents with both the extension and web address to the 2nd server and again it gets the same problem. It looks like it tries to connect to the mysql and takes to long to respond.... The 1st server looks pretty loaded .. it but it has Xeon(R) CPU E5640 @ 2.67GHz and the mysql process takes around 50-80% cpu usage .. and spikes going to 120-130% .

What can I do to optimize the situation ?

L.E. Situation improved after some mysql tuning ( http://download.vicidial.com/conf/my-vicidb.cnf )

by **iulianm** » Fri Apr 01, 2011 12:29 pm

the problem is still there... I think the server is overloaded. I will try to put a 3rd server and have only one dedicated for db , one with web + asterisk and another one with asterisk

by **williamconley** » Fri Apr 01, 2011 12:49 pm

remember that it is NOT a requirement (or even helpful for that matter) for an agent to register a phone and log in to web on the same server. the functions are completely seperate and do not interact with each other. so having an agent connect to one for phone and another for web is quite normal.

move your agent web connections from the "3.X" load server to the "0.X" load server until they balance.

by **iulianm** » Fri Apr 01, 2011 1:23 pm

Yes I know... But I think that the server that has now the database ( wich is both asterisk + web ) is overloaded ... Because after I put over 70 agents and web interfaces started to go slow on both servers and ,eventually, asterisk crashed and everything was blocked. I will update to the latest version of vicidial and make the cluster of 3 servers.

I use also full recording for all agents... and gsm codec .. that's why I want more asterisk servers...

1 x db + asterisk ( 25% of operators)
1 x web + asterisk ( 25% of operators )
1 x asterisk ( 50% of operators )

I will try this configuration and I will post here the results. My servers are R410 and all of them will be installed with 64 bit version.

by **williamconley** » Fri Apr 01, 2011 3:11 pm

you don't have to "think" it's overloaded, that's what the "3.X" load was telling you. Unless you have 6 processors on that bot, it should stay at or below 50% of the installed processors (ie: 4 processors should not go above 2.0).

so move agent phone registrations and web logins around until BOTH servers are at or below 50%.

by **iulianm** » Sat Apr 02, 2011 2:53 am

I have xeon quad core with hyper theading activated ; So I see in TOP 8 cores and a load of 3 .. should be ok....

by **williamconley** » Sun Apr 03, 2011 12:21 am

if you have a quad core, you have 4. If you have a Dual Quad Core, you have 8. A load of 3 is past 50% on a Quad. If you are experiencing "issues" as well ... hyperthreading is cool for a desktop in Windows, but I'm not sure it helps AT ALL in a server that is actually fully utilizing the processor (i'm far from authoritative on this subject, of course).

How about this (sort of the clincher for me when watching a system):

When viewing with "htop" (or viewing all processors with "top", individiually with their own bars): each minute several processes fire (each time we pass from :59 to :00). When those processes fire, do any of the bars top out?

If one "barely" tops out, you are near limit. If one or more slam to the right ...

Although, perhaps you should try it without the hyperthreading and see if there is mere compatibility issue.

And another question: are BOTH machines like this (Quad Xeon hyper)? in which case ... you are still pushing a lot more onto one server than the other and could benefit from shifting load to the 0.X machine.

Troubleshooting is fun. LOL

If all else fails: a quick fix is often to simply move the SQL server to another machine. Nice thing about the SQL server: It does not need to have the vicidial scripts running on it. (no screens in screen -list, for instance). It should have the optimizations on it for mysql (to allow for LOTS of connections). But otherwise, just a solid fast machine ... with a fast HD for "forward planning" (15k SAS with 6G write for instance).

by **iulianm** » Sun Apr 03, 2011 2:12 am

I have disabled also hypearthreading and now I have only 4 cpu's . I have made the cluster of 3 machines.

1 x Dell R410 8GB ram = db + asterisk ( VARactive_keepalives => 123468 )
1 x Dell R310 4GB ram = asterisk ( with VARactive_keepalives => 12345678 )
1 x Dell R310 4gb ram = asterisk + web ( VARactive_keepalives => 123468 )

All of them are running Vicibox 3.0.8 . Opensuse has been updated and has the latest kernel , dahdi recompiled for this and vicidial to the SVN version 2-4.306a ; I use predictive, gsm codec and full recording ...

I have around 10 extension on R410 ... around 70 extensions on first R310 and around 30 extensions on R310 that has also the web interface. I will test monday to see how is going to work.

In used for a lot of years gnudialer with success of 160 agents in predictive / 3 servers ( 1 x asteris, 1 x web , 1 x db ) without any big problems. gnudialer indeed uses chan_agent and not so much reply on how fast is the mysql. I hope I can build also a nice stable cluster with vicidial.

by **iulianm** » Mon Apr 04, 2011 2:13 am

same problem

I have the load like this:

R410 ( db + asterisk ): 1.30, 0.85, 0.35
R310 ( asterisk ) : load average: 0.22, 0.26, 0.10
R310 ( asterisk + web ): load average: 0.00, 0.00, 0.00

I looked with HTOP on all servers... the db server is around 30-35% on all cores.

And it's not working.. when I try to access the webpage.. just sits in connectiong... I don't see any mysql queries that takes long time

L.E. I saw a problem and fixed it .. now it works ok.. until this moment. The problem was related to apache regarding MaxClients and ServerLimit wich were set to 150.. so that's why my web interface was jammed and server didn't had any load

by **williamconley** » Mon Apr 04, 2011 9:03 am

now experiment with hyperthreading and see if it helps or hinders

BTW: which directions did you follow to cluster these boxes?

by **iulianm** » Mon Apr 04, 2011 9:48 am

Basically the settings were made automatically when installed with ViciBox.

I have double checked them using the manual Vicidial Multi-Server Manual by Poundteam v1 1 .... I set up the ntp to be the same ... and also tested out the load balancing from

http://download.vicidial.com/vicidial/t ... ANCING.txt

All the server now have HT disabled .. I tested today with around 100 agents. 10 extension on the server wich is also db, and 90 extension on the server that is only asterisk .

the load was around 2.5 on db , 2 on asterisk and 0 on the webserver. Tomorow i will increase more agents and put the extension on the webserver.

by **iulianm** » Fri Aug 26, 2011 10:31 am

I've just wanted to update the post that I had today 150 agents and I'm planning to go to 200 agents in the next weeks.

I have 3 x DELL R410 , xeon quad core, 4gb of ram used as asterisk + web ( one having 8gb of ram is also the web server )

1 x DELL R710 , 2 x xeon quad core , 16gb of ram used as database

1 x virtual machine on a xen enviroment in a mysql master slave configuration for reports.

The DELL R410 has vicibox 3.0.8 installation ; R710 is CentOS 6 and has only MySQL and the VirtualMachine is CentOS 5.6 and also has only MySQL.

The load on the database server is around 1.5 and the rest of the servers has between 0.5 - 1.5 . The 150 agents are split 50 / server and uses also full recording. The campaign maxratio is 4 and I get a waitaverage between calls of 6-7 seconds.

All 3 asterisk servers have sip trunks with a opensips proxy that balance calls to 4 different providers.

I have ordered now some Sangoma USB Timers to see if I get some improvement.

For the next 50 agents I will put another server to act as asterisk + web.

by **williamconley** » Fri Aug 26, 2011 7:10 pm

excellent post. you should also mention that you aren't(?) having any call quality or paused agents or call routing issues, or merely state that your system is running well with no issues.

model numbers of the motherboards and CPUs and HDs (or HD speeds/types) would be helpful for many viewers as well.

by **iulianm** » Sat Aug 27, 2011 5:50 am

I don't have call quality or other problems .. pretty much is running fine.

I am using everynight on the database the command

Code: Select all: mysqlcheck --auto-repair -o -v -A --table

in order to check , repair and optimize the database. Before using this I've got asterisk crash the next day... and the day after 3-4 times / day .. and the day after 8-9 times / day... It's 3 months since I'm using this command and I have never rebooted any server from the cluster.

My hardware is:

Database:

Code: Select all: DELL R710 - CentOS 6.0 RAM: 8 x 2 GB DDR3 1333 Mhz optimized for 2 CPUs CPU: 2 x Xeon E5630 @ 2.53GHz HDD: 2 x 450GB SAS 15K in RAID 1 using PERC H700 controller

Web/Asterisk

Code: Select all: DELL R410 ViciBox 3.0.8 updated ( zypper up, dahdi recompiled ) in June 2011 RAM: 4 x 2 GB DDR3 1333 Mhz CPU: Xeon E5640 @ 2.67GHz HDD: 2 x 250GB SATA 7200RPM in RAID1 using PERC H200 controller

Asterisk Only

Code: Select all: DELL R310 ViciBox 3.0.8 updated ( zypper up, dahdi recompiled ) in June 2011 RAM: 2x 2 GB DDR3 1333 Mhz CPU: Xeon X3470 @ 2.93GHz HDD: 2 x 250GB SATA 7200RPM in RAID1 using PERC H200 controller

Asterisk/Web for some clients

Code: Select all: DELL R310 ViciBox 3.0.8 updated ( zypper up, dahdi recompiled ) in June 2011 RAM: 2x 2 GB DDR3 1333 Mhz CPU: Xeon X3470 @ 2.93GHz HDD: 2 x 250GB SATA 7200RPM in RAID1 using PERC H200 controller

Reports - Only MySQL

Code: Select all: XEN Paravirtualized guest - CentOS 5.6 RAM: 4GB DDR2 CPU: Xeon L5335 @ 2.00GHz ( 4 cores assigned to this guest ) HDD: 100GB 15K SAS in RAID1 using PERC 6/ir controller on the host machine ( LVM storage )

2 weeks ago I have updated to the VERSION: 2.4-329a.

I have to install now a 4th asterisk/web wich is another DELL R410 server to put another 50 agents.

I am using gsm as a codec .. the iax2 trunks between the servers work great to pass the calls from one asterisk to another.. helping to get a waitaverage of 5-6 seconds .
For example today ( saturday ) on the AM shift , we had 41 agents that got 16336 calls with a waitaverage of 0:03 ( talkaverage is 0:20 ). The agents are talking not so much because they pass calls to the closers , and with an audioerror of 2% ( on the disposition page there is a disposition that I put audio-error and the agents click on that disposition when there are problems with the audio ( voice interruption, client not hearing the agent and so on ) ; this can also be related to internet/voip provider and so on )

by **williamconley** » Sat Aug 27, 2011 12:11 pm

Wow. You are a cool poster.

You should submit your Audio Error method for inclusion. Could help everyone track and kill problems.

by **iulianm** » Tue Aug 30, 2011 4:59 am

It is in plan to go to 350 seats.

Do you think that if I add another 16GB on the database server it will do the job ?

For each 50 seats I will add a asterisk server. So I will have 7 x asterisk / web servers , 1 x database , 1 x reports.

Or I could have 7 x asterisk , 1 x database , 1 x webserver the same configuration as the database server ( 2 x xeon + 16gb ram ) and 1 x reports.

by **mflorell** » Tue Aug 30, 2011 8:40 am

For 350 seats you will need more than one webserver.

by **williamconley** » Tue Aug 30, 2011 9:15 am

if you are planning to go to 350 seats ... call The Vicidial Group. They are the professionals at ensuring these types of systems work smoothly. I've seen the setups when complete, and they are tight. Noone else is actually qualified for that sort of buildout. Seriously.

So when Matt says "two web servers", that's not a flippant remark, get another web server. LOL

by **iulianm** » Tue Aug 30, 2011 10:51 am

Thanks. I will test with having web servers on the asterisk servers with a proxy in front or 2 separate webservers..

Anyway this plan to go to 350 seat won't be done over the night. I will increase everymonth and check the load, system stability and other stuff.

I will put here updates.

by **iulianm** » Thu Oct 06, 2011 2:51 am

I have just reached 215 agents yesterday.

25 of them were CLOSERs.

I have now the configuration above + 1 asterisk/web server

so I have 4 x asterisk/web servers , 1 x db , 1 x reports

I want to build now the system with a http load balancer and give to all my agents only 1 ip.

The plan is to go to 350 seats but I think I will build another cluster for the rest of 150 agents because I don't need them to be on the same system.

The system is pretty stable ... I get an asterisk crash once every 2-3 days randomly on one of the 4 servers and I don't know why. I just saw an out of memory error and I was thinking of updating the asterisk. When it will crash again I will post it here.

by **mflorell** » Thu Oct 06, 2011 6:46 am

Thank you very much for posting your experiences.

by **iulianm** » Tue Nov 15, 2011 3:46 am

I have a very strange problem .

When I log in the servers using gsm codec, I get a ticking/clicking sound in the background wich is not constant .. for a couple of seconds is silcence .. and after that again ticking sound ... Is like when you plug a jack in a amplifier in Line In and you put your hand on the other jack and you get that kind of noise....

If I switch to alaw I have no problem ... I tried using sangoma usb timer but there is no difference . ( I have 5 asterisk servers installed with vicibox 3.0.8 )

If I make direct calls from the softphone using gsm codec I have no such problems, so is related directly to meetme conferences... Do you have any idea what I can tune to solve this?

by **williamconley** » Tue Nov 15, 2011 8:54 pm

1) when you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.

this IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

2) What does this have to do with this thread, again? I'm not sure I get the link between this post and the prior subject matter. Maybe it's me 8-)

(Or perhaps you should create a new post with a proper subject line ...?)

3) Have you tested this on all 5 servers?

by **iulianm** » Wed Nov 16, 2011 1:09 am

I posted here because it might be related to my cluster setup wich is explained in the previous posts , so I though it was not required to post again all the configuration/ versions

I created a new topic here for this problem: http://www.vicidial.org/VICIDIALforum/v ... 8941#78941

vicidial.org

Vicibox 3.0.8 cluster

Vicibox 3.0.8 cluster

Who is online