Agents waiting is almost 50%.

[jessiekidfernando]

Hello Everyone,

Seeking for your assistance on how we can drop the agents waiting to less than 5% or at least 1%. Here's the system/campaign settings:

We have 10 dialers clustered. All of them are load balanced.

Settings of all servers:
MAX TRUNKS:250
MAX CALLS PER SECOND: 7
BALANCE DIALING: Y
BALANCE RANK: 9
BALANCE OFFLIMITS: 0


Could on AWS |VERSION: 2.14-685a BUILD: 180825-2100 © 2018 ViciDial Group | Asterisk 13.21.0-vici | Clustered | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz


Average wait time is: 50%

Agents logged in: 214
Agents waiting: Usually around 100 - 125

DIAL LEVEL: 14.000 TRUNK SHORT/FILL: 2584 / 1612 FILTER: MAXATTEMPTS TIME: 2019-03-01 18:28:06
DIALABLE LEADS: 328893 CALLS TODAY: 232538 AVG AGENTS: 70.92 DIAL METHOD: ADAPT_AVERAGE
HOPPER ( min/auto ): 4000 / 4326 DROPPED / ANSWERED: 1572.000 / 33823 DL DIFF: 41.00 STATUSES: CALLBK, AFTHRS, DROP, TIMEOT, XDROP, LRERR, ERI, HU, PU, NANQUE, NA, N, B, ADCT, ADC, AB, AA, A, NEW
LEADS IN HOPPER: 17814 DROPPED PERCENT: 4.65% DIFF: 57.81% ORDER: RANDOM 4th NEW


Campaign Settings:

Dial Method: ADAPT_AVERAGE
Auto Dial Level: 14
Auto Dial Level Threshold: DISABLED
Available Only Tally: N
Available Only Tally Threshold: NON-PAUSED_AGENTS
Drop Percentage Limit:10%
Maximum Adapt Dial Level: 14
Latest Server Time: 2100
Adapt Intensity Modifier: 0 - Balanced
Dial Level Difference Target:5 --- 5 Calls Waiting for Agents
Dial Level Difference Target Method: ADAPT_CALC_ONLY
Concurrent Transfers: AUTO
Queue Priority: 50 - Higher
Multiple Campaign Drop Rate Group: DISABLED
Inbound Queue No Dial: ALL_SERVERS
Inbound No-Agents No-Dial: --DISABLED--
Inbound No-Agents No-Dial Threshold: 0


Regards,

Jessie Kid Fernando

[williamconley]

Please remember to include your installer with version. it can make a difference.

the percentage of agents waiting for a call is not relevant. that depends on your script.

the average wait time per agent, however, that's a valuable piece of information. no, this is not a value shown. you (human being) will be looking at the real time screen and see agents waiting for XX seconds and get upset because it's too long. of course, this is only valuabe IF you have the Next Agent Call set to "longest wait time".

214 agents on ... how many servers? what's the average server load and core count (those are related) for each of the servers in question?

is this a Press One/Survey Campaign? Or just live agents?

Have you dialed through all your lists several times and begun to collect "NA" status leads in a big pile? Turn those off and see what happens for an hour.

TRUNK SHORT/FILL: 2584 / 1612

you need to have more trunks on each server. spread your agents out evenly among them to reduce this. if you can get it to zero/zero you'll be much happier with production. it may require more servers to avoid overloading each server, but it may also be reduced by increasing the max trunks on each server (beware of overload, beware of exceeding max trunks on your carrier!).

check with your carrier and see what your call per second max is. remember that you have to set it Per Server whereas they will likely have a "system max" instead of "per server", so ask them and work that out so you don't end up with congestion from overloading the carrier.

next up: what are your normal percentages for congestion/cancel/answer/etc when you show carrier stats in the real time screen?

[jessiekidfernando]

CARRIER STATS: HANGUP STATUS 24 HOURS 6 HOURS 1 HOUR 15 MIN 5 MIN 1 MIN
ANSWER 174539 62.6% 96978 56.9% 25544 79.9% 0 0.0% 0 0.0% 0 0.0%
BUSY 14841 5.3% 8234 4.8% 1864 5.8% 0 0.0% 0 0.0% 0 0.0%
CANCEL 62232 22.3% 51415 30.2% 3076 9.6% 0 0.0% 0 0.0% 0 0.0%
CHANUNAVAIL 15815 5.7% 10021 5.9% 1443 4.5% 0 0.0% 0 0.0% 0 0.0%
CONGESTION 11510 4.1% 3846 2.3% 25 0.1% 0 0.0% 0 0.0% 0 0.0%
TOTALS 278937 170494 31952 0 0 0

[jessiekidfernando]

Next Agent Call was set to longest wait time. All of the 214 agents are live agents on a single campaign. These agents are equally logged in on all 10 dialers.

[jessiekidfernando]

Here's the list count per status.

COUNT STATUS
253021 AA
154206 HU
121631 NA
40340 NEW
27384 B
26067 A
13442 PDROP
7765 ADC
5355 DNCL
4256 IN5
3414 DROP
2700 NI
2391 AFTHRS
2097 DNC
1999 PU
1653 IN1
957 BAD
802 ERI
602 CBHOLD
426 ORD
121 CALLBK
55 AB
24 LRERR
22 DCMX
14 INCALL
4 TIMEOT
1 NANQUE

[williamconley]

using amd can overload servers and piss off customers.

each call is "delayed" by two seconds to decide "human vs machine". this is notable. without AMD, there is a roughly 1.5 second delay. add the 2 second AMD delay, now you have almost 4 seconds.

So a prospect says "hello?" .... 1.... 2 .... 3 ...4 "crap, it's a sales call". NOW your agent must climb that mountain and become someone other than the devil. But on the upside, the agent in question does Not risk having to push the hangup customer button for machines, as they never get there.

I am of the opinion that pissing off all the actual people is NOT worth removing the machines. but that's me.

Also, if your dial timeout is too low it will not detect all ADC leads. Some carriers will wait up to 44 seconds to send that signal. Vicidial killing the call before then leaves that disconnnected number as AA and you'll dial it again. So either set your call count limit to something like 8 to remove them after they have been dialed too many times OR set your dial timeout to 45. Or both. Both is good. lol

[jessiekidfernando]

Our campaign has AMD detection and it is using the dialplan below. Also the DIal Timeout was set earlier to 40.

1. Playback(sip-silence) [pbx_config]
2. AGI(agi://127.0.0.1:4577/call_log) [pbx_config]
3. AMD(2000,2000,1000,5000,120,50,4,256) [pbx_config]
4. AGI(VD_amd.agi,${EXTEN}) [pbx_config]
5. AGI(agi-VDAD_ALL_outbound.agi,NORMAL-----LB) [pbx_config]
6. AGI(agi-VDAD_ALL_outbound.agi,NORMAL-----LB) [pbx_config]
7. Hangup() [pbx_config]


And also on System Settings:

Max FILL Calls per Second: 150


Do you think this drops the wait time?

Thanks a lot William for answering my inquiries.

[williamconley]

you want to organize your agents to NOT need fill at all. that will improve your dialer's ability to "keep up". reduce the agent count per server or increase the trunks available. trunk short/fill slows down the server drastically.

also try to increase the calls per second max in admin->servers, but you have to be careful not to overload your account with your carrier.

AMD is why your dial ratio is so high. and it's why you burn through numbers and can't keep your agents on the phone. even a good script will suffer when AMD is on, but a mediocre/bad script ... well, dialing 14:1 is fairly common.

[jessiekidfernando]

Hello William,

Upon changing all mentioned key items here on the entire forum. We are still experiencing 50% of the wait time. Only 50% occupancy. We are aiming to have it at least 95%. Any other aspect that might help us to succeed?

Regards,

Jessie Kid Fernando

[williamconley]

Hello William,

Upon changing all mentioned key items here on the entire forum. We are still experiencing 50% of the wait time. Only 50% occupancy. We are aiming to have it at least 95%. Any other aspect that might help us to succeed?

Regards,

Jessie Kid Fernando

One more time: % on the phone at any given moment is a function of the quality of your script. Nobody else can fix your script.

Wait time, on the other hand, the amount of time each agents waits before getting the next call, *IS* a function of the campaign and dialer settings plus the number of agents on that campaign.

[jessiekidfernando]

DIAL LEVEL: 32.000 TRUNK SHORT/FILL: 1360 / 0 FILTER: GOLD_FILTER TIME: 2019-03-05 15:07:59
DIALABLE LEADS: 3515880 CALLS TODAY: 240871 AVG AGENTS: 44.56 DIAL METHOD: ADAPT_AVERAGE
HOPPER ( min/auto ): 5110 / 64000 DROPPED / ANSWERED: 3847.000 / 38844 DL DIFF: 13.31 STATUSES: A, NA, PDROP, AA, NEW
LEADS IN HOPPER: 47594 DROPPED PERCENT: 9.90% DIFF: 29.86% ORDER: DOWN

Current active calls = 2433
Calls ringing = 2418
Agents logged in = 312
Agent in calls/dials = 91/1
Calls waiting for agents = 26
Agents waiting = 59
Paused agents = 110
Agents in dead calls = 20
Agents in dispo = 20


Having the above setup? What is the recommended settings that we need to apply?

[williamconley]

TRUNK SHORT/FILL: 1360 / 0

1360 short trunks. add more servers or increase your trunks in some fashion. Or find out why your system wants to dial so many lines more than it has (seems a bit high, actually).

[jessiekidfernando]

DIAL LEVEL: 32.000 TRUNK SHORT/FILL: 1323 / 2881 FILTER: GOLD_FILTER TIME: 2019-03-05 15:23:04
DIALABLE LEADS: 3515880 CALLS TODAY: 254708 AVG AGENTS: 44.25 DIAL METHOD: ADAPT_AVERAGE
HOPPER ( min/auto ): 5110 / 64000 DROPPED / ANSWERED: 4469.000 / 40799 DL DIFF: 10.67 STATUSES: A, NA, PDROP, AA, NEW
LEADS IN HOPPER: 69742 DROPPED PERCENT: 10.95% DIFF: 24.12% ORDER: DOWN

Right now it is above. Each of our dialers has 300 trunks. We have 10 dialers.

[williamconley]

and the average wait time for the agents? The most useful metric. The agents being off the phone is wasted money. The rest is all merely interesting.

[jessiekidfernando]

Right now the average wait time of agents is less than 30 seconds.

[williamconley]

if you have that many agents in one campaign and a 10% drop, you should have something more like 10 seconds (or less) wait time. Maybe 15. But under 30 is adequate.

If these agents are spread out among many campaigns, however, each one is its own world and will have its own average wait time. plus they will interfere with each other on the short/fill counts causing variable wait times based on "who won".

[jessiekidfernando]

All of these agents are under single/one campaign.

[williamconley]

Consider investigating those wasted (dropped) calls. If they happen during "bunched" calls (where a lot of calls go out, a lot of answers happen, and of course there aren't enough agents because of so many bunched answers), it may be a good idea to change your drop call seconds to Zero to see if you can "unbunch" the calls. Sometimes it helps.

[jessiekidfernando]

Hello William,

Thanks for the additional inputs. We will try this and let you know if there's an improvement.

[jessiekidfernando]

Hello William,

Additional inputs.

We've seeing a high no. of "Waiting for table level lock" | SELECT count(*) FROM vicidial_live_agents table on our database. Our database is a master-slave setup on AMAZON RDS.

Notes:
MySQL [asterisk]> show variables like 'query_cache%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+

Any recommendations?

Regards,

Jessie Kid Fernando

[williamconley]

those that are waiting are not the problem. a process has locked that table. you need to find the oldest query, which will be your offender. it also often shows in the slow query log.

after you find the offender, you will have a direction to resolve your issue. but it could be almost anything, and creating a list of possibilities would be book-level.