Cluster Setup for 600 Agents

[rustynail]

Hi,

I am looking for best practice to setup cluster for a team of 600 agents using vicibox. Can you please suggest optimal setup of how many DB, WEB, DIALLER servers will be required and of what specs. How many should be DB slaves and should I consider DB master<->master replication or no? as I read somewhere in forum that mflorall discourages this, HA proxy for web etc..

Usually we use 4 core servers with 12G ram for stand alone (all in one) dialers for 20-25 seats. and for cluster we use 8 core cpu with 16GB RAM (DB), 1 web and many dialers with 4 core cpu and 12G ram.

Main issue we face is that as vicidial_list table grows, it starts causing DB load which we want to avoid at all costs. Currently vicidial_list table has 5 million records and it starts causing latency which I would like to avoid in new cluster.

Thanks,
Nasir Javaid.

[williamconley]

The Vicidial Manager's Manual has a page at the front discussing server counts for buildouts.

agent count is a tiny piece of information.

how many simultaneous live calls? how many inbound waiting for agents? how many ringing calls? Is AMD on? Are there survey campaigns? Recording all calls? Codecs in use?

system capacity is based on "peak moment", so at the most how many of all these things are going to be happening at the same time.

If you intend to build a 600 agent server, and you do NOT contact a professional Vicidial management company (such as The Vicidial Group or my employer PoundTeam ...) you are crazy. IMHO. It's like starting a nationwide trucking company with 600 employees and just going down to the local Jeep dealership to buy some trucks. It will not end well.

Ever heard the phrase "Penny wise and pound foolish"? Just sayin'.

[rustynail]

Thanks for reverting Willian,

The proposed system is in discussion and planning phase so as per current scenario, as you mentioned system capacity is based on "peak moment", so we can think of all 600 agents doing outbound calls and many customers are dumb enough to keep cps to 10-20 and dial level to 10+ also due to compensate bad leads. So you can imagine the agents and ringing calls. Yes AMD will be on and there are no survey campaigns. All calls will be FORCE recorded and codec will be alaw/ulaw

I will forward your suggestion of contacting professional vicidial company for a 600 seats contact center to my boss. Please share the cost associated with this setup and support period so in case of any issue arises, it can be tackled?

However as I mentioned we haven't faced much issue regarding server resources till 200 agents but main hurdle was DB load as mtop shows most of the queries go to red zone (vicidial_list table) so was thinking for best practice to handle it. I read somewhere that mflorell discourage the use of master-master replication of mysql so what would be best practice to manage it.

Cheers!

[mflorell]

We have a few clients on VICIhost that regularly place over 2 million outbound calls a day per cluster with anywhere from 100 to 850 agents(depending on if they are auto-dialing or manual dialing), so 600 agents is certainly possible given the right hardware, configurations and management. Although dialing at 10:1 with AMD on might actually require two separate clusters of that size, because you're talking over 4 million calls per day at that point(one of our VICIhost clients currently has 3 clusters of that size). A recommended set of hardware for a single cluster of that size would cost around USD$65,000 if you wanted to own and run it yourself.

[rustynail]

Hi florell,

many thanks for your response. By saying "you will require 2 separate clusters of that size", are you suggesting 2 separate clusters of 300 agents each?
Also can you please recommend optimum hardware specs for DB, WEB and Diallers.. Will a single Master DB be able to handle all calls if we go for single cluster of 600 agents?

Cheers!

[mflorell]

You can only have one master DB server for a single cluster, and for the clusters I've described only one is needed, although we do recommend getting an equal sized slave DB for reporting and redundancy.

As for hardware specs, those seem to change monthly with new hardware that comes out. At least a 2 x octal-core CPU motherboard with 64 GB RAM and 2 separate RAID 10 volumes(1 for OS and apps, 1 for MySQL data directory) with very fast drives and a MegaRAID card.

Also, the number of agents is not a fixed restriction point, it depends on the number of calls placed and active at any given time too. 600 agents isn't a problem on a single cluster if it is built correctly and you dial at reasonable levels, but if you want to dial those systems at 10:1 with AMD, then the system will be overloaded.

[rustynail]

Sorry for delay as it was weekend.

Thanks so much for the valuable feedback. I have noted your suggestion regarding DB server specs and drive separation technique, however do you mean a reasonable dial level will be 3-4?
For 10:1, will there be any hardware upgrade or it will not be possible. One big hurdle is also the table size, specially vicidial_list table, when it grows, it causes slow queries to increase and mtop shows lots of red color queries. I don't think this is related to RAM as lot of ram is still available when this happens.

As there is no archive for vicidial_list so its hard to clean it, however we are working on a script to create an archive table and move specific statuses to _archive. hope it will work as out of 10 calls there are 6-7 bad numbers or useless calls so it will reduce table size considerably.

Cheers!.

[mflorell]

Yes, 3-4 would be a more reasonable dial level. As for capacity, you really just have to see what happens, but the DB server is really the most limiting factor.

As for archiving vicidial_list records, we've posted a few times about that, here is an example of how you can set that up:

CREATE TABLE vicidial_list_archive LIKE vicidial_list;
ALTER TABLE vicidial_list_archive MODIFY lead_id INT(9) UNSIGNED NOT NULL;

select count(*) from vicidial_list where list_id IN('8881106');

INSERT INTO vicidial_list_archive SELECT * from vicidial_list where list_id IN('8881106');
select count(*) from vicidial_list_archive where list_id IN('8881106');
DELETE from vicidial_list where list_id IN('8881106');
optimize table vicidial_list;



Also, there are admin pages that are set up to look at the vicidial_list_archive data, like the Lead Search and Lead Modify pages.

[rustynail]

Thanks again, Yes found the forum posts which show list archiving. I fully agree that database is a major limiting factor and that is what I am fearing about. Because we are currently running a 100 seat cluster setup but still their db server takes load intermittently at peak times so don't want to go with 600 agents till I figure out best way to reduce DB load. Below are the details:

Xeon(R) CPU E5630 @ 2.53GHz 16 core CPU
RAM 16GB

LSBLK:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 0 237.9G 0 disk
├─sdc1 8:33 0 1G 0 part /boot
└─sdc2 8:34 0 236.9G 0 part
├─system-root 254:0 0 226.9G 0 lvm /
├─system-swap 254:1 0 4G 0 lvm [SWAP]
└─system-tmpfs 254:2 0 6G 0 lvm /var/spool/asterisk/monitor
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 1024M 0 rom


Server load doesn't go above 1.5 and RAM is also almost 8-10GB free at peak times. Below is the custom MySQL config which I am using (even on standard config we face issue)

mysql -uroot -e "set global query_cache_type='OFF'"
mysql -uroot -e "set global query_cache_size=0"
mysql -uroot -e "set global key_buffer_size = 1073741824"
mysql -uroot -e "set global max_connections = 2000"
mysql -uroot -e "set global max_heap_table_size = 67108864"
mysql -uroot -e "set global table_open_cache = 8192"

vicidial_list table has 3965568 records and it seems whenever hopper is reset or any update/select is performed on this table, there is pause issue... mtop shows queries getting red in color and agents facing pause issue.

Can you please help resolving this issue so I can plan 600 seats accordingly. Will be greatful for your kind cooperation.

Thanks.

[williamconley]

Reducing DB load is done one of two ways:

1) (Not very popular): Use the DB less. EG: Less agents. Less calls.
2) Replication/Reporting server. This allows "select" db requests to run on the slave and "insert/update" to run on the master, dramatically improving throughput.

Altering the db settings in the configuration file is often counterproductive. Be very careful. Vicidial is a high-volume transactional system and this type of system has different "common sense" settings than (for instance) a web server which can accommodate queing and delays.

Thus increasing a value may seem like a good idea, but may in fact slow down the system.

[rustynail]

Hi William,

Of course first option is out of question. Even client refuses to lower cps/dial level and even lists and insists on keeping settings high, even some keep cps=50, trunks 500 and dial level also 50/100 .. but they have liberty of not being technical

And yes we have all reporting on slave server except "realtime report". For DB related changes I do not edit my.cnf as it requires mysql reboot, instead I run it on CLI as you can see from the commands I shared in previous post. This way its easy to revert any settings which seems downgrading performance. Today, in "show processlist" , I noticed a command "Copying to tmp table on disk"...
Suspecting it might have some impact so set value of max_heap_table_size and tmp_table_size to double from their default value. Fingers crossed till next shift and will observe if it has some positive impact.

Thanks very much for your feedback and suggestions.

Cheers!

[williamconley]

report back your findings (preferably with the underlying data) both for us and for when you (some day) want to revisit this. someone else may chime in and improve or clarify as well. we've all made some interesting changes.

one warning: don't assume a change is good unless you can "UN"change it and RE change it several times and essentially prove that your change was really good. I've had far too many technicians brought down by insufficient empirical evidence.

[rustynail]

Sure, I am monitoring closely and will report back any positive findings. Yes that's why I opted for setting mysql variables via command line instead of my.cnf so I can undo any change which seems downgrading performance

Fingers Crossed.

[rustynail]

Hi Guys,

Latest update is that client wants to go ahead with 300 agents on webphone in first phase and then remaining 300 after successfully running the phase 1. My query is whether it is possible to use proxy in case of using webphone in same way as we use in usual soft phone setup?
Because when using webphone, each server must have a wss url in its settings. How can we overcome this?

Thanks.

[williamconley]

use vpn instead of proxy

[rustynail]

Hi William,

sorry for late reply. Can you please explain about which vpn will be best suited for this purpose. a dedicated hardware/mikrotik based or pptpd etc?

[williamconley]

What's missing from all of this is why you don't use use it without proxy or VPN. What creates your "i need proxy/vpn" in the first place? Why do you have issues with wss url?

1) We don't recommend using any of them. Use a whitelist lockdown on the server and allow only the agents through the firewall. Then VPN and Proxy are pointless.

2) VPN and Proxy servers can be hardware or software based. Trust is the issue. Don't use a free one under any circumstances unless it's open-source, free, and widely used (ie: lots of people would have had enough time to find out if it's spying on your or running evil code in some fashion).

[rustynail]

Thanks,

Main need for proxy is that there will be many web servers and client doesn't want to use 2-3 web servers but a single IP/domain. I have had some success with haproxy on test environment and will keep exploring it more.
wss might not be a big issue for now.

[williamconley]

Just use a single domain that "bounces" users among the existing web servers, rotating among them. This in no way requires a "proxy". It's a simple php web page.

[rustynail]

Ok thanks, Will try it also. I have tried with haproxy and seems working fine for now.

[williamconley]

Hey, as long as it works and you're comfortable with it, that was the point. Go with what you know.