Difference in calls on two identical Clusters

[samadsaeed]

Hi,
I am running a press1 type campaign. I have a cluster setup details below in signature. I have two identical clusters and around 24 agents login on both clusters. The leads are divided into parts and distributed on both clusters with a dial level of 60. Please understand that both clusters contain the same configuration method of installation same crontab entries, php.ini configuration is same and hardware is also identical. The problem I'm facing is when both clusters have agents logged in 1 of the cluster dials perfectly fine with below 30 seconds agent available time at the end of the shift and I hardly see any more than 2 agents waiting for call during the whole shift, but at the same time the second cluster is not performing like that. Mostly it has idle agents waiting for calls for more than 1 minute. I am failing to understand what am I missing as both of the setups are identical in every manner even the leads are the same which are distributed in half before upload on each one. Let me know if any further information needs to be added here as this is my first post and please help me fix this issue soon. Thanks a lot in advance!

Regards,
Samad.

[williamconley]

Samad, you are looking at the forest and skipping the trees entirely. It's not about "but they're identical!" it's about "why did this one call fail" or "why did this one call NOT generate".

Are they attempting the same dials? Are the outcomes of the calls similar?

For instance: If one of the servers has 40% CONGESTION (available via Real Time report option show carrier stats), then the problem could be that one cluster is using up all your Carrier Channels.

But there are other possibilities, such as networking and lead quality ... all of which require actual troubleshooting on the server that's not working as expected. You have an advantage, however, over those in a similar situation with a "non-optimal" experience: You have an otherwise identical "optimal experience" server with which to compare. Instead of looking at that as a curse (one works, one doesn't), look at it as a blessing: You can find the difference(s) in configuration and/or experience.

Literally, they can NOT be identical, as they can not have the Same IP address (among other things). While the difference(s) may not be the cause, they may at least point you to the culprit. But your best bet is usually to troubleshoot a single call that did not go as it should have.

Happy Hunting.

PS: Good job posting your specs. Most fail to do that. I would suggest, however, that you should include a link to your installation method instructions since you did not use an .iso installer for Vicidial.

Also worthy of note: The 1T HD may be large, but is it fast? The DB usually should have 6G/sec RAID10 with 15K SAS or good quality SSD. Eventually the DB server will be the choking point (archive and/or prune your log and list tables regularly!)

[samadsaeed]

Thanks William, I am considering the points you have mentioned already! yes you are right I have something to compare from which is good. Sir I forgot to mention that these servers are hosted servers from www.oneprovider.com all of them with same specs and configurations. One thing I have planned during clustering is that i made sure every cluster contains same subnet of IPs so that all latency issues between the cluster communication gets killed. Secondly I have noticed that on DB server I see mysqld using 300% CPU and on dialers I see above 150-250% CPU usage by asterisk while memory is only being used by 0.6-1.0% on all. What can we do for that? I have used this ray solomon's link for the installation method for all servers (http://ray-solomon.com/vicidial.html) my.cnf is also the same as below. Sir please let me know if I can post any other stats which may help me sort out this issue! Thank you.

[mysql.server]
user = mysql
#basedir = /var/lib

[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
datadir = /var/lib/mysql
#tmpdir = /home/mysql_tmp
socket = /var/lib/mysql/mysql.sock
user = mysql
old_passwords = 0
ft_min_word_len = 3
max_connections = 800
max_allowed_packet = 32M
skip-external-locking

log-error = /var/log/mysqld/mysqld.log

query-cache-type = 1
query-cache-size = 32M

long_query_time = 1
#slow_query_log = 1
#slow_query_log_file = /var/log/mysqld/slow-queries.log

tmp_table_size = 128M
table_cache = 1024

join_buffer_size = 1M
key_buffer = 512M
sort_buffer_size = 6M
read_buffer_size = 4M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 64M

max_tmp_tables = 64

thread_cache_size = 8
thread_concurrency = 8

# If using replication, uncomment log-bin below
#log-bin = mysql-bin

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
#log-error = /var/log/mysqld/mysqld.log
#pid-file = /var/run/mysqld/mysqld.pid

Regards,
Samad.

[williamconley]

High CPU Usage is expected. If there are no failed processes, that's not actionable as a rule. It may in fact be a symptom, but until we have a failure somewhere ... troubleshooting is needed to find what is actually broken.

You didn't mention the call outcome percentages. CONGESTION? Other differences in call outcome out-of-line with the other cluster? Are the leads randomly distributed or is it possible that the leads are different quality in one vs the other? Are they both Trying to Dial enough to keep the agents busy, or is one of them NOT dialing enough? Troubleshooting.

[samadsaeed]

Sir, below are the congestion percentage of 3 different clusters.

1 -- 14 Agents: 5%
2 -- 24 agents: 11%
3 -- 27 agents(the perfect cluster): 12%

All of them are dialing at a Dial level of 60. Leads are of same quality as a single set of leads is divided into 3 parts and uploaded on all clusters. The one with 14 agents has been added today to check the performance but it ain't up to the mark like the 3rd one. Sometimes they start performing nice but they aren't stable like the third one which runs perfectly all through the shift. One more thing, all clusters are set to reboot everyday before the start of shift.

[williamconley]

You'll have to find out whether the substandard cluster(s) are not trying as hard or are failing after trying as hard. That's where the logs come in. The logic behind "should I dial more leads right now?" is the second stage, the first is to check all the logs for failures (if it's trying, but failing, there should be errors associated with the failing, usually easier to troubleshoot that).

[samadsaeed]

Sir please can you point me out what logs should I start seeing? or may be I can post them here so that you and all other experts can help me out on this? I have tried to reinstall this cluster for more than 20 times but not getting the same result as that single cluster is constantly giving. I have copied the my.cnf and crontab entries from that one but still not upto the mark. please point me the logs which I can see and also post here too for any possible solution.

Regards,
Samad.

[samadsaeed]

Please let me know which logs to focus on as I'm a bit confused with so many of them. I'm not new to this setup but still facing this type of issue for the first time!

[williamconley]

I think the point here is that this troubleshooting run is how you determine which log(s) are relevant. After you find a log with a difference or an error or incorrect information or ... anything that raises a flag, you'll know where to start.

Goal: Find out whether the "not working" cluster is making the calls it should to keep the agents busy, but those calls are failing, or whether it is NOT making the calls in the first place. At that point you have your first branch in where to look. There are multiple screens (screen -list) running which contain information, but much of that information is also logged.

Find out if the dialer is not making the calls or if it's making them but they are failing. This could be in the autodial screen/log or the adapt screen/log ...

[samadsaeed]

Thanks William, I'll definitely be looking at them. Just a quick question. The perfect cluster which is dialing flawlessly has campaign Minimum Hopper Level set to 10000 where as on the rest of the servers I am unable to find any way to go past 2000. Googled every bit of the internet but couldn't find a way how to do it. Please tell me where can I increase the Minimum Hopper Level to 10000.

Regards,
Samad.

[williamconley]

That would be why I referred you to the Vicidial Manager's Manual. It's FREE and located on the store at EFLO.net. There's also a paid version, but the free one should have reference for hopper maximum settings.

However: I don't know that a number over 2000 is possible without altering code in admin.php.

Perhaps your servers are not as "identical" as you believe.

[samadsaeed]

yes today I noticed this difference. I don't think that any code was altered to increase the hopper level. I thought there might be a setting in system settings as dial level can get increased from there.

[williamconley]

Of course, that's still a "hip shot" attempt and not troubleshooting. If you found that the hopper script was overloading the CPU, that would be different.

Kinda like beginning to take bolts off the rear bumper of your car because it won't start. Eventually you'll find the problem, but ... it could be a while.

[samadsaeed]

Hi,
I think the issue resolved after I did a scratch install on all the servers except the one which is performing good. Only this time I did asterisk 11.22 on all dialers and now they seem to be dialing almost identical. Anyways it seems that all 3 clusters are now performing identically. Thanks William for all the support.

--
Regards,
Samad.

[uncapped_shady]

Thanks William, I'll definitely be looking at them. Just a quick question. The perfect cluster which is dialing flawlessly has campaign Minimum Hopper Level set to 10000 where as on the rest of the servers I am unable to find any way to go past 2000. Googled every bit of the internet but couldn't find a way how to do it. Please tell me where can I increase the Minimum Hopper Level to 10000.

Regards,
Samad.

Hi there, glad to hear you have got the issue resolved. Just one thing to note about the hopper setting.....

You need to set the hopper to an amount that your agents can comfortably dial in around a 3-5 minute period (if your hopper script is still set to run at the default 60 secs interval.) and not to like 2000 or higher if at all possible. One reason for this is simply that, should you be dialing certain geo-locations on a per lead list your hopper will load the first 2000 leads into the hopper and will only dial those geo-locations until the hopper loads the next 2000. So dependent on the amount of leads your agents go through in a day you might find that you end up dialing only one geographic area the whole day. This is true for lower volume call centers but a rule of thumb overall would be to set your hopper level to the amount of leads your agents will go through in a 3-5 minute period.