Multiple calls to same phone number.

[mstopkey]

Greetings all. Ran into a problem today and looking for solution. Upfront, I think it might just be database server load but not sure. Here it is. We have some instances where the same lead was called at the same phone number 2 or more times EXACTLY 60 seconds apart according to lead search results. We are running ADAPT_AVERAGE dial method with Auto Alt-Number Dialing set to EXTENDED_ONLY. Since we have several dialer manages, first thought someone changed campaign setting incorrectly. I checked and everything seems fine. I reduced allowed campaign dial statuses to NEW only. Same thing. Servers load is Ok. See below.

Code: Select all

CALLS TO THIS LEAD:
#   DATE/TIME   LENGTH   STATUS   TSR   CAMPAIGN   LIST   LEAD   HANGUP REASON   PHONE   CALLER ID
1   2019-02-22 12:47:46   64   WG   VDAD   USCOLECT   603   76313535   AGENT     XXXXXXXXXX
2   2019-02-22 12:47:21   0   NA   VDAD   USCOLECT   603   76313535   NONE     XXXXXX0535   <--- 2nd DIAL
3   2019-02-22 12:46:21   0   NA   VDAD   USCOLECT   603   76313535   NONE     XXXXXX0535   <--- 1st DIAL

Campaign did not have dial status NA enabled. No Lead Recycling either. I queried vicidial_log by the lead_id. See below.

Code: Select all

uniqueid lead_id list_id call_date start_epoch end_epoch length_in_sec status phone_number user term_reason alt_dial called_count
1550861129.19767 76313535 603 2019-2-22 12:46:21 1550861181 1550861181 0 NA XXXXXX0535 VDAD NONE NONE 1
1550861190.29689 76313535 603 2019-2-22 12:47:21 1550861241 1550861241 0 NA XXXXXX0535 VDAD NONE X1 2
1550861246.19913 76313535 603 2019-2-22 12:47:46 1550861266 1550861270 64 WG XXXXXXXXXX VDAD AGENT X2 3

That number is NOT in the vicidial_list_alt_phones table. All lists are deduped against system when loaded.

In every instance of this, the time between double dialed calls is 60 seconds Exactly.

Any clues?

[mstopkey]

Update: It just now went crazy dialing same phone numbers multiple times. Rebooting cluster in order. Stuff just hit the fan! Oh, and I ran DB check on database.

[mstopkey]

Still happening.

[williamconley]

turn off lead recycling.

[mstopkey]

Hello William. We always have it off. Happening less often now but still going on. Only allowed dial status is NEW

[williamconley]

Hello William. We always have it off. Happening less often now but still going on. Only allowed dial status is NEW

i was with you all the way to "only allowed dial status". at that point I begin to think you believe that allowed dial status is related to recycling ... but it's not.

so to be clear: in the campaign detail view's Lead Recycling top menu: "LEAD RECYCLING WITHIN THIS CAMPAIGN" has zero entries shown below for the campaign in question?

[mstopkey]

That is correct. It is still happening this morning. I have turned any alt number dialing. Only dialing phone number is the list table now until I can fix this. We have always dialed Extended from the vicidial_list_alt_phones table and one phone in the phone fields of the vicidial_list table. Yesterday, this blew up. Look at the below.

uniqueid lead_id list_id call_date start_epoch end_epoch length_in_sec status phone_number user term_reason alt_dial called_count
1550861129.19767 76313535 603 2019-2-22 12:46:21 1550861181 1550861181 0 NA XXXXXX0535 VDAD NONE NONE 1
1550861190.29689 76313535 603 2019-2-22 12:47:21 1550861241 1550861241 0 NA XXXXXX0535 VDAD NONE X1 2
1550861246.19913 76313535 603 2019-2-22 12:47:46 1550861266 1550861270 64 WG XXXXXXXXXX VDAD AGENT X2 3

The first two rows are the same phone number. That phone number is only in the vicidial_list table. For some reason, vicidial thinks it is also in the vicidial_list_alt_phones table as indicated by the X1. It is not in htat table but vicidial dialed it again. Currently , the only allowed dial status on the campaign is NEW. And, we just disabled any alt or extended number dialing for the time being. Only thing I can think of is database corruption but my initial CLI DB check indicatede no problems.

I really need help here. After agents leave at noon today, I am going to run DB check and repair from PhPMyAdmin.

[williamconley]

Information you've implied or left us to infer:

Are you saying that you have changed a setting and stopped the problem? But you need to be able to use that feature you turned off?

[mstopkey]

That is correct. I turned off extended dialing. Now dialing only 1 phone from the vicidial_list.phone field. No longer dialing same number over and over. Even setting allowed campaign dial statuses to only NEW did not stop it from dialing same phone over and over. We never use lead recycling. We have been using EXTENDED_ONLY alt number dialing for many years with no problems.

[williamconley]

OK: Extended and alt number dialing, just like lead recycling is NOT related to the dialable statuses. It's related to the Human Answer setting in Admin->System Statuses. If a Human Answer does not occur, the next number in the chain is dialed. Somebody turned off Human Answer on the statuses in the agent selectable list in the Dispo screen (which is NOT the dialable statuses).

Dialable status: Start of call.
Disposition status: End of call.

Not related to each other.

Happy Hunting!

[mstopkey]

No cigar. After doing extensive database checks this weekend, I turned on EXTENDED_ONLY alt number dialing. Still doing it. William, I checked your suggestion. The example I have this morning was a lead with phone number 5122960447 in the vicidial_list.phone_number field. This lead has 3 other phone numbers in the vicidial_list_alt_phones table. 1st call to this lead was to 5122960447, dispo as WG. That status is set to Human Answer. 4 minutes and 12 seconds later dialer called it again. Dispo as DNC.
All examples I have found, this is only happening on the phone number in vicidial_list table. It does not seem to be double dialing phones that are in the vicidial_list_alt_phones table. I could be wrong on that.
This is really strange. All the years we have been dialing in this manner, I have never ran across anything like this.
This is from the vicidial_log table.

Code: Select all

1551104047.164   76611107   609   2019-02-25 08:14:24   1551104064   1551104107   43   WN   5122960447   MARIA1   CALLER   MAIN   1
1551104160.180   76611107   609   2019-02-25 08:16:51   1551104211   1551104211   0   NA   5127613515   VDAD   NONE   X1   2
1551104224.305   76611107   609   2019-02-25 08:17:56   1551104276   1551104276   0   NA   5129634534   VDAD   NONE   X2   3
1551104300.627   76611107   609   2019-02-25 08:18:36   1551104316   1551104372   56   DNC   5122960447   WILLUR   AGENT   XLAST   4

This is from the lead search in vicidial.

Code: Select all

CALLS TO THIS LEAD:
#   DATE/TIME   LENGTH   STATUS   TSR   CAMPAIGN   LIST   LEAD   HANGUP REASON   PHONE
1   2019-02-25 08:18:36   56   DNC   WILLUR   USCOLECT   609   76611107   AGENT     5122960447
2   2019-02-25 08:17:56   0   NA   VDAD   USCOLECT   609   76611107   NONE     5129634534
3   2019-02-25 08:16:51   0   NA   VDAD   USCOLECT   609   76611107   NONE     5127613515
4   2019-02-25 08:14:24   43   WN   MARIA1   USCOLECT   609   76611107   CALLER     5122960447

Gonna check to make sure cron jobs are running. Only idea I have left.

[mstopkey]

Here is another one. In this instance, phone number 9563198386 we somehow got it loaded into both the vicidial_list_table and vicidial_list_alt_phones table. First time I have seen this. Gotta check out my auto list loading scripts.

Code: Select all

EXTENDED ALTERNATE PHONE NUMBERS FOR THIS LEAD:   (ALT NOTE COMMENTED OUT TO PROTECT THE INNOCENT!)
#   ALT PHONE   ALT NOTE   ALT COUNT   ACTIVE
1   1 9563198686   XXXXXXXXXX   1   Y
2   1 9565683053   XXXXXXXXXX   2   Y
3   1 9566355953   XXXXXXXXXX   3   Y
4   1 9567635318   XXXXXXXXXX   4   Y
5   1 9567756833   XXXXXXXXXX   5   Y


CALLS TO THIS LEAD:
#   DATE/TIME   LENGTH   STATUS   TSR   CAMPAIGN   LIST   LEAD   HANGUP REASON   PHONE   CALLER ID
1   2019-02-25 08:50:45   36   WN   MARIA1   USCOLECT   609   76611755   AGENT     9563198386
2   2019-02-25 08:50:22   0   AB   VDAD   USCOLECT   609   76611755   NONE     9566355953
3   2019-02-25 08:50:19   0   NA   VDAD   USCOLECT   609   76611755   NONE     9565683053
4   2019-02-25 08:48:54   0   AA   VDAD   USCOLECT   609   76611755   CALLER     9563198386
5   2019-02-25 08:47:16   26   WN   LEONAS   USCOLECT   609   76611755   AGENT     9563198386

[mstopkey]

Last example where phone number 9563198386 was called 3 times, my auto list build script was not set to dedupe phones on load leads. That's how the number got into both tables.

[mstopkey]

Fixed the one list leads load script that was not deduping. Cron is running properly on cluster. Cron jobs 5, 6 & E only on 1 master server. Had to turn off EXTENDED_ONLY dialing.
Building a new campaign. Maybe the current one is just BROKE!

[mstopkey]

Update: We have been running on a new campaign with EXTENDED_ONLY enabled for an hour now with no double dialing. All settings for this campaign are the same as the old campaign (used) to be. I guess maybe the old campaign has a problem I just can't find.

Question, in vicidial_list_alt_phones table, field "active", what is it used for and does it ever get changed to "N"?

[mstopkey]

Update: I was wrong, still double dialing even in new campaign. Almost every instance, the number dialed multiple times is in the vicidial_list.phone_number field. The exception was we had one in both tables. Most of them are status NA by VDAD.

[mstopkey]

Currently In every legit instance of double dialing, happens to phone number in vicidial_list.phone_number and dialer thinks it is also in vicidial_list_alt_phones table, it is not. See below examples.

Code: Select all

1551132045.25737   76625711   451   2019-02-25 16:01:36   1551132096   1551132096   0   NA   4092991063   VDAD   NONE   NONE   1
1551132129.35982   76625711   451   2019-02-25 16:02:08   1551132128   1551132128   0   AB   4092991063   VDAD   NONE   X1   2

Code: Select all

1551131984.35855   76625691   451   2019-02-25 16:00:35   1551132035   1551132035   0   NA   4053012722   VDAD   NONE   NONE   1
1551132129.35973   76625691   451   2019-02-25 16:03:02   1551132182   1551132182   0   NA   4053012722   VDAD   NONE   XLAST   3
1551132044.25716   76625691   451   2019-02-25 16:01:36   1551132096   1551132096   0   NA   4053547586   VDAD   NONE   X1   2

Code: Select all

1551132283.36146   76625997   451   2019-02-25 16:05:35   1551132335   1551132335   0   NA   5124652811   VDAD   NONE   NONE   1
1551132389.36345   76625997   451   2019-02-25 16:07:20   1551132440   1551132440   0   NA   5124652811   VDAD   NONE   X2   3
1551132337.25860   76625997   451   2019-02-25 16:06:28   1551132388   1551132388   0   NA   5124652811   VDAD   NONE   X1   2
1551132442.36453   76625997   451   2019-02-25 16:08:15   1551132495   1551132495   0   NA   5124652811   VDAD   NONE   X3   4
1551132537.36558   76625997   451   2019-02-25 16:09:15   1551132555   1551132586   30   DROP   9797785829   VDAD   QUEUETIMEOUT   XLAST   5

[mstopkey]

This thread is very relevant to our issue.

http://www.eflo.net/VICIDIALforum/viewtopic.php?t=9672

[mstopkey]

After thorough diagnostics, trimming down campaign settings to the point of not dialing extended alt numbers then re-enabling extended, getting many examples I have come to this determination. In EVERY instance, the multiple dialed phone number is only in the vicidial_list.phone_number field and for some reason the dialer calls it a 2nd, 3rd, 4th etc time ONLY as an alt number even though it is not in the vicidial_list_alt_phones table. The ONLY time this happened otherwise we had a list that did not dedupe during load. (Scripting mistake).
By querying the archived vicidial_log_XXX tables, I have discovered this has been going on for a while. How long I am not sure.
Pertinent Campaign settings:
No Lead Recycling on ANY campaign on this dialer. /vicidial/admin.php?ADD=35 confirms LEAD RECYLCES = NONE on all campaigns.
Dial Method = ADAPT_AVERAGE
Auto Alt-Number Dialing = EXTENDED_ONLY
Auto ALT DIAL tab on Campaign: We stripped all statuses down to just NA being left. No difference. Of course removing all statuses prevented Auto Alt Dialing.
Only 1 phone in vicidial_list table. All other lead phones loaded into vicidial_list_alt_phones table.

Example of one server carrier dial plan. (Remember, we are clustered, and load balancing)
exten => _91NXXNXXXXXX,1,AGI(agi://127.0.0.1:4577/call_log)
exten => _91NXXNXXXXXX,n,Progress()
exten => _91NXXNXXXXXX,n,Dial(${SIPTRUNK7}/${EXTEN:1},,tTor)
exten => _91NXXNXXXXXX,n,Congestion()
exten => _91NXXNXXXXXX,n,Hangup

Mr. Florell or Mr. Conley, do you have any insight to this issue?

[mstopkey]

Code: Select all

10   76785564   3612272172   USCOLECT   2019-02-26 08:54:26   WN   WILLUR   449   30   MAIN
11   76785564   3612272172   USCOLECT   2019-02-26 08:55:09   DNC   MARIA1   449   18   X1
12   76785564   3612272172   USCOLECT   2019-02-26 08:56:13   HNGP   NESTOR   449   68   X3


Please help. Dialer ignored a DNC also

[mstopkey]

Code: Select all

1551194434.8594   76784554   449   2019-02-26 09:20:42   1551194442   1551194442   0   AA   5059303987   VDAD   CALLER   NONE   4
1551194290.6230   76784554   449   2019-02-26 09:18:18   1551194298         LRERR   5059303987   VDAD   NONE   NONE   1
1551194358.8444   76784554   449   2019-02-26 09:19:25   1551194365   1551194365   0   AA   5059303987   VDAD   CALLER   NONE   2
1551194390.6265   76784554   449   2019-02-26 09:20:00   1551194400         CTR   5125208045   NESTOR   NONE   X2   3

[mstopkey]

In this example, the phone from vicidial_list table was called 5 times. There are 4 alt phones in vicidial_list_alt_phones. All examples are from vicidial_log table.

Code: Select all

uniqueid          lead_id    list_id  call_date          start_epoch  end_epoch length_in_sec  status  phone_number  user  term_reason  alt_dial   called_count   
1551205413.19101   76800604   443   2019-02-26 12:24:24   1551205464   1551205464   0   NA   7015202636   VDAD   NONE   NONE   1
1551205467.19121   76800604   443   2019-02-26 12:25:20   1551205520   1551205520   0   NA   7015202636   VDAD   NONE   X1   2
1551205522.19152   76800604   443   2019-02-26 12:26:13   1551205573   1551205573   0   NA   7015202636   VDAD   NONE   X2   3
1551205627.19176   76800604   443   2019-02-26 12:27:59   1551205679   1551205679   0   NA   7015202636   VDAD   NONE   X3   4
1551205680.19195   76800604   443   2019-02-26 12:28:52   1551205732   1551205732   0   NA   7015202636   VDAD   NONE   XLAST   5