Dropped Calls

[rhunter]

I installed using ViciBox_Redux.x86_64-3.1.14 ISO
VERSION: 2.4-351a
BUILD: 111223-0043
Asterisk version 1.4.21.2
Dell R410s (quad core Xeon) with 8 GB RAM and USB Sangoma Voicetime Wanpipes
I have a cluster of one database server (HA/Failover), 3 Dialer servers, and 4 Webserver VMs (LB)
The database server is Ubuntu with MySQL 5.1.41-3ubuntu12.6-log (Dell R410 Perc5i Raid 1 SATA 7200RPM (yeah I know...working on this), all others are ViciBox
I also use Sangoma CPA software

Hello All,

I was wondering if there is a way to see why a called was dropped or what happened with a dropped call.

Situation:
A few days ago it was brought to my attention that our dialing team was having an issue keeping dropped calls under control. As I watched the dashboard, every so often I would see 10-15 calls queue up and wait, and then all of them would get dropped. I was also told that the agents were getting messages on their screen saying "Your session has been paused". This led me to the database where, as it turns out, there was major locking on the vicidial_list table (according to mtop) correlating to when the calls would get queued up and dropped. I eventually tracked it down to one of our database guys who was inserting fresh leads every 15 minutes in a very non-indexish sort of way so he revised his query and everything seemed smooth for the rest of the day.
The next day, everything seemed to be going fine at first, and when I looked back at the dropped calls around noon, it was already at about 50 which seemed higher than usual. I checked the slow query log, and didn't find any extraneous locking. There's no extra database jobs running in the morning specifically. Yesterday I noticed the same exact thing, near the same exact time, with nothing out of the ordinary in slow query log etc. It only seems to occur in the morning, and then the rest of the day seems okay (some dropped calls here and there but not in excess). There are no cron jobs that run at that time (vicidial or otherwise), back-ups are long done by this point, there aren't any more agents on at that time than any other (about 15-20).

I'm sort of at a loss for why this is occurring, which is why I was wondering if there's a way of seeing why a call gets dropped. I'm also seeing some calls getting dropped with agents available, which concerns me.

Some more info:
Usually about 15-20 agents @ around 2:1
vicidial_list has about 5.5 million records
I did set up the cron script to archive the prescribed tables now (keeping 6 months worth)
Moved most ViciDial built-in reporting to the slave DB
Using an AS5400 series router as a gateway tied to a TW Telecom DS3, with Velocity true VoIP as backup

They stay under the Federal mandatory 3% drop percentage, but it means cutting the dial ratio back and agents waiting for long periods of time. We seem to have, on average, about 100 dropped calls per day, IDK if this is high or not. Here are some stats from Outbound Calling Report for today:

---------- CALL HANGUP REASON STATS
+----------------------+------------+
| HANGUP REASON | CALLS |
+----------------------+------------+
| CUSTOMER | 7746 |
| AGENT | 7974 |
| QUEUETIMEOUT | 76 |
| ABANDON | 2 |
| NO ANSWER | 3593 |
+----------------------+------------+
| TOTAL: | 19391 |
+----------------------+------------+

I assume the 76 QUEUETIMEOUTS + 2 ABANDONS make up my 78 dropped calls for the day. Is this telling me that 76 people were actually dropped because they waited the 2 seconds before being dropped? That would seem high, especially because like I said, I keep seeing the dropped calls count increasing when agents are in ready.

Any help would be greatly appreciated. Thanks!

[williamconley]

you'll need to catch this when it happens, but I have noticed many clients who have their dialers (especially early 2.4 and before) set too high for max calls per second.

if you upgrade to a later version, some of this is alleviated, but here is the scenario:

Your system kicks in and dials a bunch of leads all at once (in one second, seriously all at once ...). As a result, you get a "group" of answers which often overwhelms the available agents (not enough agents, too many answers). You have 2 seconds to get all these to live agents so you lose a few.

The expectation is that this will 'smooth out' as the shift progresses, they won't All happen at once any more. In other words, as a call terminates or an agent gets off a call, those will generate outside of a "land rush" and the calling will be smooth and agent rotation will make agents and call randomize so there should pretty much always be an agent for a call.

But that is not what happens. NA calls (defined as anything that did not get answered other than Busy which has its own SIP response) will "get lost" and must be harvested by a process. Thus "re-bunching" these calls when that process runs. So this scenario (too many calls, not enough agents) will repeat itself all day long, some times more severe than others. This does have a lot to do with NA call percentage. To make matters worse, you will note that most call centers have NA as a dialable status ... so these calls will all be called again after the next reset. Exacerbating the issue and causing it to get steadily worse as the list(s) mature.

One helpful new feature is the limit to the number of times a lead is called. This became available in 2.4. But it's only a slight relief, of course.

The best fix is the latest SVN in 2.6 which had an addition a few months ago that more accurately/properly designates DC calls (disconnected numbers) instead of including them in NA. I believe this also causes the calls to be pruned immediately which means they are immediately redialed instead of "bunching up" when the harvester comes through.

It also helps if you set your max calls per second to 1 for every 10 agents. This has definitely been known to smooth out the calls instead of bunching them up. If you have max 20 agents, set max calls per second to 2.

This is your situation, of course, ONLY if you note that your agents will have too many calls for a few seconds ... then actually NO calls for 45 seconds, then start over. Eventually this may lessen in severity, but it should not be happening at all. The calls should originate based on the agent becoming available OR the termination of another call.

Another point to ponder is that usually these calls terminate, but Vicidial is unaware of it, and often the statistical engine in vicidial takes a while to "catch up" to some of these events in the older systems. So the calls ringing count in Vicidial is often quite higher than the calls actually "ringing" according to Asterisk.

If that is all true ... UPGRADE! And benefit from The Vicidial Groups improvements over the past year that you may be missing. It's not just that there are new features (which there certainly are), but because some deeper logic has been fixed to greatly improve actual usage.

Rant over. LOL

[rhunter]

Thanks for the detailed response williamconley. The original max calls per second for each of our servers was set to 20 for some reason... I changed this to 2 for each server and have been watching it like a hawk over the past couple days. The drop rate seems to have gotten better since the change so thanks for your help! No more mysterious drops woot!

[williamconley]

default is 20, you just never changed them. have a discussion with your provider as to what your limit is and whether this limit is per server or for your entire account.