ViciDial poor performance with no local DNS server
Posted: Wed Dec 04, 2019 2:52 pmHi
[We had this problem for the past 2 days which we solved this afternoon, but I post it here for reference and also for review and advice please.]
ViciDial was performing badly, in particular Auto dialled calls were not being originated quickly enough and agent wait times were high (Dial Level up at 15 on many campaigns). The hopper load that usually finished well within 1 minute was taking as long as 10 minutes to run.
We did a huge DB archiving and optimisation, tried adding indexes, but could get no marked improvement. There were no DB locks longer than 1 second and nothing in slow_query.log set at 3 seconds long query. Confusing!
Fortunately the SysAdmin remembered that he had changed the Vici servers to point to 1.1.1.1 as DNS rather than the local onsite DNS! (There had been problems with the onsite DNS server late last week and it was offline for a day.) Ping to 1.1.1.1 is 3.9ms whereas to local DNS is 0.34ms. When it was corrected to the local DNS server the hopper load immediately started running 100 times (literally) faster than it had been. Problem solved!
I've looked into this and I see that the MySQL server has host_cache_size=87 - so it should have been caching DNS entries on its side. However, I see nowhere that there's any DNS caching on the Telephony servers - Perl DBI does not seem to do it, and Centos does not either (from what I can find). VARDB_server is set to a name rather than an IPv4 address - is that recommended?
Is there anything we can do differently to mitigate this situation in the future?
Thanks
Roger
-----
Vici SVN install VERSION: 2.12-538a BUILD: 160122-1401 - Asterisk 1.8 Centos 6 MySQL 5.6.28
427 concurrent agents on today, 285 000 calls in the day, peaking at 80 calls per second Origination
[We had this problem for the past 2 days which we solved this afternoon, but I post it here for reference and also for review and advice please.]
ViciDial was performing badly, in particular Auto dialled calls were not being originated quickly enough and agent wait times were high (Dial Level up at 15 on many campaigns). The hopper load that usually finished well within 1 minute was taking as long as 10 minutes to run.
We did a huge DB archiving and optimisation, tried adding indexes, but could get no marked improvement. There were no DB locks longer than 1 second and nothing in slow_query.log set at 3 seconds long query. Confusing!
Fortunately the SysAdmin remembered that he had changed the Vici servers to point to 1.1.1.1 as DNS rather than the local onsite DNS! (There had been problems with the onsite DNS server late last week and it was offline for a day.) Ping to 1.1.1.1 is 3.9ms whereas to local DNS is 0.34ms. When it was corrected to the local DNS server the hopper load immediately started running 100 times (literally) faster than it had been. Problem solved!
I've looked into this and I see that the MySQL server has host_cache_size=87 - so it should have been caching DNS entries on its side. However, I see nowhere that there's any DNS caching on the Telephony servers - Perl DBI does not seem to do it, and Centos does not either (from what I can find). VARDB_server is set to a name rather than an IPv4 address - is that recommended?
Is there anything we can do differently to mitigate this situation in the future?
Thanks
Roger
-----
Vici SVN install VERSION: 2.12-538a BUILD: 160122-1401 - Asterisk 1.8 Centos 6 MySQL 5.6.28
427 concurrent agents on today, 285 000 calls in the day, peaking at 80 calls per second Origination