Server Capacity Question

[amjohnson]

Is there any way to tell given my current server load with agents how many more agents the database server will take? Assuming of course I will add the appropriate servers to my cluster to handle the additional workload..

I recently updated my Vicibox servers to Vicibox 5.0.3 using ViciBox5.x86_64-5.0.3.iso
I am running:
Version: 2.8b0.5
SVN Version: 2040
DB Schema Version: 1359
DB Schema Update Date: 2013-11-08 01:40:22

On this Hardware:
ASUS Z8NR-D12(ASMB4-IKVM) Dual LGA 1366 Intel 5500 Tylersburg SSI EEB 3.61 Dual Intel Xeon 5500 and 5600 Series w/ Remote Management Server Motherboard
2 x Intel(R) Xeon(R) CPU E5640 @ 2.67GHz
24 GB Registered Unbuffered DDR3-1333 mhz RAM
LSI MegaRAID SAS 9280-4i4e 6Gb/s PCI-Express 2.0 w/ 512MB onboard memory RAID Controller Card
LSI LSI00292 MegaRAID CacheCade Pro 2.0 Software Pack
8 x Intel 256gb Enterprise SSD

Raid Configuration:
Raid 10, No write caching, no read caching (manufacturer recommendation)


Some misc info:
mtop shows we have few to zero queries that go over 1 second. My leads database typically has between 3 million to 5 million numbers in it. I am currently running 96 agents including 10 inbound only Closers,outbound dialing, ADAPT_AVERAGE dial metod, 250-450 outbound lines.The database server appears to almost be idling at 10-15% processor usage. The database server is setup for telephony/web but only has a handful of manager extensions for monitoring. The web is for IT admin changes only so it see's little use.

output of IOSTAT:
Linux 3.7.10-1.16-default (90vicidial.90vicidial) 11/12/13 _x86_64_ (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
9.70 0.00 3.90 0.02 0.00 86.38
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.01 0.04 0.00 672 0
sdb 80.30 220.41 636.20 4208069 12146092

summary of ioping /dev/sdb2 (os/db location)
--- /dev/sdb2 (device 887.2 Gb) ioping statistics ---
20 requests completed in 19.6 s, 4.1 k iops, 16.1 mb/s
min/avg/max/mdev = 149 us / 243 us / 292 us / 36 us

Nothing even looks like its being stressed... would it be safe to say maybe 200 agents?

[williamconley]

Code: Select all

would it be safe to say maybe 200 agents?

Um. ... No.

htop and uptime show "1 5 10 minute average server load". This is a very useful tool. It also has a bar chart showing instant usage on each processor.

If you are anywhere near 50% cpu usage (ie: with 8 processors, that would be 4.0) at 1,5, or 10 minute load averages, you are AT capacity. If you EVER go over 50% ... you need to be looking to add a 2nd server and making a cluster. IF, however, you are nowhere near 50% usage, that doesn't mean you won't be there soon. These systems do not load "linear", but "Exponential". You will find that once you breech 50%, you will hit 75% pretty quick and the jump to 100% is really fast and then you'll hit 500% in a heartbeat and 2400% my not be shown because the processor is too hammered to discuss it with the htop application.

Or to put it more succinctly, just because the server isn't sweating yet, does not mean it will still hold up when it does. There may also be some limitations in the number of simultaneous Conferences. You should also watch for network backlog in the near future.

A couple points: You listed "2 x Intel(R) Xeon(R) CPU E5640 @ 2.67GHz" but did not mention how many cores total. While this information is certainly available on the ark, it would be simpler to post it as a "Total 8 cores" addendum.
If you have hyperthreading on, consider turning it off or at least paying attention to high load days and consider turning it off Then and seeing if there is a difference (other than a false number of CPUs in htop, LOL). We have had slightly better experience with it off than on, but would certainly be interested in hearing other responses.

Note that when you breach 125 users (approximate!!!) you will enter a new realm of "why did THAT break?". Be sure you have a duplicate system and extra networking architecture to be able to break off into two servers at half capacity (or at least move 25 agents into it ... to reduce load on the primary system) in an emergency. Networking will become extremely important, anything imperfect will kill your dialer. Gigabit switches and routers everywhere! No bottlenecks.

Consider moving your database to a replication server and running reports there. Same with all recordings (an archive server with all recordings pushed via FTP).

Happy Hunting!

[Vince-0]

I've seen multi-server Vicidial systems run over 160 agents comfortably on one database with similar RAID, CPU and RAM specifications as yours but with 4+ telephony servers limited to 50 agents per server and a dedicated web server.

I see a major problem with table locks on agent_log table as mentioned in this thread:
http://vicidial.org/VICIDIALforum/viewtopic.php?f=4&t=29417&p=102794#p102794

This may have to do with time lock functionality enabled but I've yet to update software and test and prove anything on this particular system.

It seems that there will be a software performance bottleneck either with MySQL 5.1 or Asterisk 1.4 respectively when scaling hardware for Vicidial. I am yet to try the newest Vicidial ISO with Asterisk 1.8 and MariaDB but I expect better performance if even only slightly better.

Vin.

[mflorell]

The most common issue with scaling that we have seen is having a disk system on the database server that is too slow. We have clients with over 400 agents across 9 telephony servers that experience no DB load issues at all. But we have seen clients with as few as 100 agents using improper raid cards or even just raid settings and they run into these DB locking and load issues.

So, repeat after me: "LSI Logic, MegaRAID card with RAID-10" for the database server. NOT 3ware, NOT adaptec, NOT RAID-5!

[amjohnson]

William,

The linux server load is usualy 0.59 to 0.8 on an 8 core system. So like I said its idling..

htop sounds interesting I usually just look at it via either top, mtop or a quick snapshot using uptime..

When I said 10-15% I was taking all that into account.


-Andrew

[amjohnson]

Matt,

So, repeat after me: "LSI Logic, MegaRAID card with RAID-10" for the database server. NOT 3ware, NOT adaptec, NOT RAID-5!

Yes I remember the drilling at training class

Which is why it has a LSI MegaRAID SAS 9280-4i4e 6Gb/s PCI-Express 2.0 w/ 512MB onboard memory RAID Controller Card in it...

The 8 SSD's are in RAID 10 and I should have about 200000-300000 iops Based on manufacturer specs but I have not scratched the surface of that yet..

We started with 4 Seagate Chetahs and an LSI MegaRAID 9260-4i but overran their capacity at about 60 agents. We started having long queries when the hopper loaded etc.. mysql-tunning got us up to about 75 and periodically rebuilding the leads database really helped that smoothed out everything most of the time but once in awhile the database with 5 to 6 million numbers would get to fragmented during the day and I would have to shut down and run an optimize on it.

I run this about once a month on the leads database:

Of course I always make a complete backup before running this as it could trash the leads if it fails

Code: Select all

use asterisk;
drop table if exists vicidial_list_old;
drop table if exists vicidial_list_tmp;
CREATE  TABLE IF NOT EXISTS  `asterisk`.`vicidial_list_tmp` LIKE `asterisk`.`vicidial_list`;
INSERT INTO `vicidial_list_tmp` select * from `vicidial_list`
# uncoment the line below to reorder your list helps if you have stale leads
# order by rand()
;
RENAME TABLE `asterisk`.`vicidial_list` TO `asterisk`.`vicidial_list_old`;
RENAME TABLE `asterisk`.`vicidial_list_tmp` TO `asterisk`.`vicidial_list`;


-Andrew

[williamconley]

I wonder if another index or two would remove the need for the dump/reload or if it is related to the way the data is actually stored on the disk.

I also wonder why you don't merely set that to run nightly ... right after the nightly backup.

Sounds like you've got a pretty solid system. We usually manage to get people up above 125 before they begin to melt down and have to rebuild network and DB drives. And no matter how often they are warned in advance they are always surprised. Good that you listened to Matt.

[amjohnson]

I thought about making that a nightly. Just kinda scares me if it fails and it happens to be a day I have off. Nothing worse than a panicked manager calling me at 6:00am "WHERE DID ALL MY LEADS GO!?!?!?!?" Yes I know the rename SHOULDN'T get run if the insert fails.. BUT I always error on the side of caution.


I was thinking of writing a program to do it will some consistency checks so that it only renames the databases if the counts are equal in both databases.

[williamconley]

have you tried a sort or another built-in function to rebuild the table?