Press 1 Campaign - IVR Doesnt Play on all answered calls

[kailashJagtap]

Hello,

We have installed the VICI Dial on Remote Dedicated Server. I am running press 1 survey campaign. The problem is when customers answer the phone, the IVR is not playing on all the answered calls. Its blank and the calls gets disconnected after 3-4 seconds. I am running 80-100 concurrent calls. Below are the details about the configurations:

IVR File Types: .wav / .gsm
IVR File Size: 415 KB / 43 KB

- ViciBox v.7.0.4-170113
VERSION: 2.14-708a
BUILD: 190414-0924
© 2019 ViciDial Group

Asterisk 11.25.1-vici

Installation Method: .iso file

Server Info: Dell PowerEdge C1100 Intel 24 Thread 32 GB Dedicated Server With 1000 mbps Port

Please help me with some solution.

I apologize if I have missed some details.

Thanks,

[williamconley]

You did a good job posting your specs. You may also consider including the core count (total cores, not "this many cpus with XX cores each", just total cores overall).

With that core count, please compare it to the 'Average Server Load' when this problem is occurring. If the Average server load is more than the core count, your problem is that your server can't handle the load. If the average server load is more than half of the core count, it's still possible that this is the problem.

But I am a little wary of the description: "24 thread" and "dedicated server" plus "with 1000 mbps port" are somewhat indicative of a Cloud or Hosted server. This can often mean the server is virtual (dedicated virtual server is a very nasty way to sell someone that's not real, similar to earlier descriptions of a "VPS" or "Virtual Private Server", but even worse, lol). So the question is whether there is any Virtualization involved at all or if you are using a physical server without any virtualization.

There are several methods to detect virtualization from "inside" a virtual host. Each has its idiosyncrasies but we often use this one:

Code: Select all

grep hyp /proc/cpuinfo; cat /sys/class/dmi/id/product_name

If the word "hypervisor" appears, you're virtualized.

[kailashJagtap]

Hello William,

Thank you for the reply. I apologize for sending incomplete details. This is my first ever post. Here is the response from the command you suggested:

vicibox7:~ # grep hyp /proc/cpuinfo; cat /sys/class/dmi/id/product_name
XS23-TY3

Below are the server details:

vicibox7:~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Model name: Intel(R) Xeon(R) CPU L5639 @ 2.13GHz
Stepping: 2
CPU MHz: 1733.512
BogoMIPS: 3466.67
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-5,12-17
NUMA node1 CPU(s): 6-11,18-23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm arat dtherm tpr_shadow vnmi flexpriority ept vpid

I am sorry if thats not the correct way to get the details.

Also, the problem is even if I make 10-15 concurrent calls we still face the same issue. I am not sure if this is related to server capacity/configuration.

Request you to kindly guide me on this.

Kindly let me know if more details are required.

Thanks

[williamconley]

During one of the times when this is happening, preferably within 30 seconds of an "incident":

Code: Select all

uptime

You have 2 CPUs. Each CPU has 6 cores. Total 12 cores. But you also have hyperthreading on, so each core has 2 threads. In theory this means you should be able to "cruise" at 12.0 average server load with the "uptime" command before you need to be concerned. As long as "uptime" shows less than 12.0/12.0/12.0 in the response, you are not overloading. Which would be a reason to look at the Asterisk CLI for an instance to find out what went wrong.

[kailashJagtap]

Hello William,

Thank you for the response. I have followed the steps you have mentioned. Here are the details:

[May 24 13:22:04] -- SIP/TEST11-0000e046 answered Local/100016822207649@default-0000e61b;2
[May 24 13:22:04] > Channel Local/100016822207649@default-0000e61b;1 was answered
[May 24 13:22:04] -- Executing [8373@default:1] Playback("Local/100016822207649@default-0000e61b;1", "sip-silence") in new stack
[May 24 13:22:04] -- <Local/100016822207649@default-0000e61b;1> Playing 'sip-silence.gsm' (language 'en')
[May 24 13:22:04] -- Executing [h@default:1] AGI("Local/100016822207649@default-0000e61b;2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----17-----0") in new stack
[May 24 13:22:04] -- Executing [8373@default:2] AGI("SIP/TEST11-0000e046", "agi://127.0.0.1:4577/call_log") in new stack
[May 24 13:22:04] -- AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=1001))
[May 24 13:22:04] -- <SIP/TEST11-0000e046>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[May 24 13:22:04] -- Executing [8373@default:3] AMD("SIP/TEST11-0000e046", "2000,2000,1000,5000,120,50,4,256") in new stack
[May 24 13:22:04] -- AMD: SIP/TEST11-0000e046 4096839395 (N/A) (Fmt: slin)
[May 24 13:22:04] -- AMD: initialSilence [2000] greeting [2000] afterGreetingSilence [1000] totalAnalysisTime [5000] minimumWordLength [120] betweenWordsSilence [50] maximumNumberOfWords [4] silenceThreshold [256] maximumWordLength [5000]
[May 24 13:22:04] -- AMD: Channel [SIP/TEST11-0000e046]. Changed state to STATE_IN_SILENCE
[May 24 13:22:04] NOTICE[1598]: chan_sip.c:28490 handle_request_register: Registration from '"818" <sip:818@209.126.74.66>' failed for '212.129.34.134:5144' - Wrong password
[May 24 13:22:05] == Manager 'sendcron' logged off from 127.0.0.1
[May 24 13:22:05] -- <Local/100016822207649@default-0000e61b;2>AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... --17-----0 completed, returning 0
[May 24 13:22:05] == Spawn extension (default, 100016822207649, 2) exited non-zero on 'Local/100016822207649@default-0000e61b;2'
[May 24 13:22:05] NOTICE[1598]: chan_sip.c:28490 handle_request_register: Registration from '<sip:860@209.126.74.66>' failed for '77.234.46.208:8722' - Wrong password
[May 24 13:22:05] -- AMD: Channel [SIP/TEST11-0000e046]. Word detected. iWordsCount:1
[May 24 13:22:05] -- AMD: Channel [SIP/TEST11-0000e046]. Detected Talk, previous silence duration: 1040
[May 24 13:22:05] -- AMD: Channel [SIP/TEST11-0000e046]. Changed state to STATE_IN_SILENCE
[May 24 13:22:06] -- AMD: Channel [SIP/TEST11-0000e046]. HUMAN: silenceDuration:1000 afterGreetingSilence:1000
[May 24 13:22:06] -- Executing [8373@default:4] AGI("SIP/TEST11-0000e046", "VD_amd.agi,8373") in new stack
[May 24 13:22:06] -- Launched AGI Script /usr/share/asterisk/agi-bin/VD_amd.agi
[May 24 13:22:06] == Manager 'sendcron' logged on from 127.0.0.1
[May 24 13:22:06] == Manager 'sendcron' logged off from 127.0.0.1
[May 24 13:22:06] -- <SIP/TEST11-0000e046>AGI Script VD_amd.agi completed, returning 0
[May 24 13:22:06] -- Executing [8373@default:5] AGI("SIP/TEST11-0000e046", "agi-VDAD_ALL_outbound.agi,SURVEYCAMP-----LB") in new stack
[May 24 13:22:06] -- Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_ALL_outbound.agi
[May 24 13:22:06] -- AGI Script Executing Application: (Monitor) Options: (wav,/var/spool/asterisk/monitor/MIX/20190524-132206_6822207649)
[May 24 13:22:06] -- <SIP/TEST11-0000e046> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[May 24 13:22:06] -- <SIP/TEST11-0000e046> Playing 'DTVIVRnew.gsm' (escape_digits=129) (sample_offset 0) (language 'en')

I apologize for sending all of this. i did not know how much to send.

Here is the uptime response:

vicibox7:~ # uptime
13:22pm up 20:14, 3 users, load average: 0.04, 0.25, 0.77

I was sending 1 call at a time to check on two different phone numbers. more than 50% calls were having no recording. They were blank.

Please let me know whats wrong.

Thanks

[williamconley]

Try turning off AMD. I bet that works wonders.

[kailashJagtap]

Thanks for the reply. So I just need to use Routing Extn 8366 instead of 8373 in campaign setting to turnoff AMD?

Or there are others settings I need to change?

[williamconley]

Thanks for the reply. So I just need to use Routing Extn 8366 instead of 8373 in campaign setting to turnoff AMD?

Or there are others settings I need to change?

Yep.

[kailashJagtap]

Ok Thank you so much. Let me test and will let you know the progress.

Thanks again for your valuable time.

Sorry one more question.

Can you please tell me how many concurrent calls this server can handle with press 1 campaign?

[williamconley]

...

Code: Select all

uptime

You have 2 CPUs. Each CPU has 6 cores. Total 12 cores. But you also have hyperthreading on, so each core has 2 threads. In theory this means you should be able to "cruise" at 12.0 average server load with the "uptime" command before you need to be concerned. As long as "uptime" shows less than 12.0/12.0/12.0 in the response, you are not overloading. Which would be a reason to look at the Asterisk CLI for an instance to find out what went wrong.

I'm not psychic enough to know how many calls will hit that barrier. Test it.

[kailashJagtap]

Ok got it. Thanks for your help & sorry for frustrating you.

[williamconley]

Not frustrated: Just being a smartass. Never hurts to ask.

Run "htop" while you're dialing and see how average server load varies while dialing. Try to set it to stick at 12.0 across the board for "smooth" running.

[kailashJagtap]

Awesome. Thank you so much for your guidance. The performance looks better after the changes you suggested.

Sorry but I have one more issue left. Below is the campaign flow details:

1) leads dialed out from Press 1 Survey Campaign = Campaing ID - 1001
2) Agents are logged in to Blended Campaign = Campaing ID 2000
3) When customer press 1, the call goes to Remote Agent (3333) & from remote agent to In_group & to Blended Camapign: 2000

Now, the problem is:

When the first call comes thru remote agent, the remote agent goes on dead & status shows (M) manual. The remote agent stays dead till the time the call is disconnected. Because the remote agent goes on "dead status" the survey campaign 1000 stops auto dialing calls. When the call is disconnected and remote agent becomes "ready" the campaigns starts the calls.

Can you please suggest what can be done to rectify this?

Kindly let me know if more details are required.

Thanks

[williamconley]

WHY would you bounce several times? Survey Action: Agent XFER ...?

[kailashJagtap]

Hello William,

Thanks for the reply. Yes Survey Action: Agent_XFER

The reason I am doing it with Survey=> remote agent => In_group => Blended: I am running multiple suvey campaigns. Each campaign has different data combination and different caller id. Just to generate more calls.

Any other simple way?

Thanks

[williamconley]

I am running multiple suvey campaigns. Each campaign has different data combination and different caller id.

"data combination"? EG: Lists with different demographics? Or you are trying to track the success rate of each list source? Obviously you could run the reports on the lists without the need to segregate campaigns.

"different caller id"? Caller ID can be set Per List as well. (For that matter, Caller ID can be set per LEAD, lol)



NO need to run multiple campaigns with the same agents. By definition, the campaign is how agents share leads. So trying to share leads when the agents are in different campaigns has to be a red flag and indicate you probably could make an adjustment to get everyone in the same campaign.

In the meantime, however, your problem is that you should get to the ingroup directly from a call menu with a tiny timeout for instant transfer instead of through an extension. Extensions run outside the Vicidial Control set. So links and tracking are lost (eg: DEAD call). But bouncing to a Call Menu with a timeout (or call time) bounce to an ingroup stays withing the Vicidial Scripting system. No DEAD calls. No remote agents (except as the final call recipient who actually talks to the prospect, and then ONLY if the agent has no computer ... which is not the best way to run the system).

[kailashJagtap]

Here is my campaign set up:

I have multiple survey campaigns having different calling lists. I have done this to use different combinations of demographics. All survey campaign calls are routed to one single campaign where the agents are logged in.

In the meantime, however, your problem is that you should get to the ingroup directly from a call menu with a tiny timeout for instant transfer instead of through an extension. Extensions run outside the Vicidial Control set. So links and tracking are lost (eg: DEAD call). But bouncing to a Call Menu with a timeout (or call time) bounce to an ingroup stays withing the Vicidial Scripting system. No DEAD calls. No remote agents (except as the final call recipient who actually talks to the prospect, and then ONLY if the agent has no computer ... which is not the best way to run the system).

I get your point & this solutions looks great. However, my question is, if I create call menu , route it to In_group directly instead of using remote agent. However, how the survey campaign dial the leads without having remote agent active?

[williamconley]

If the campaign's "method" is not agent_xfer, the agent is then ONLY used to activate the campaign. No call will route to the agent. The agent's only purpose is to choose the number of lines to dial.

But without the ability to have the agent become "busy" (since calls are routing to a call menu instead of that remote agent) the campaign will never stop dialing. Thus Agent_XFER with all the agents in one campaign allows the system to ramp up or down based on how many agents are still in need of a call. No other method allows this.

[kailashJagtap]

Ok so if i am following you correctly, here are the changes I made. Let me know if this is correct:

1) Campaign => Detail View => Survey => Survey Method: CALLMENU (Have set this to all survey campaigns)
2) Added New Call menu => Menu Timeout: 1 => Route: INGROUP (Selected Appropriate Ingroup and Campaign ID)

Is this correct?

Thanks

[williamconley]

Yep

Be sure the audio is sip-silence.

You can also set the call time for the menu to "1AM-2AM" so you're always outside calling times and it will always take the "outside calling time" action. No delay. No playing a recording.

[kailashJagtap]

Be sure the audio is sip-silence.

Sorry but I did not understand this. Can you please tell me how to do this?

I have kept "Survey Opt-in Audio File" Blank
In CALLMENU => Menu Prompt: Blank

Is there anything else I have to do?

[williamconley]

menus play a sound. "Menu Prompt". Use "silence". Of course, if you set the menu time check to 1 and set the call time to a time your system is not in use ... that won't matter. Otherwise the Menu timeout and Menu Prompt come into play, which can all be avoided by using the time check to skip straight to the ingroup. Then the "Option" that's set to 'TIMECHECK' is used immediately and the call goes to the ingroup with no delay.

[kailashJagtap]

Great. Thank you so very much for all the assistance and your valuable time. I have made all the changes as per your instructions. Will monitor and see how it works. If any issue will bother you again

Thank you so much again.

Have a great weekend.

[kailashJagtap]

Hello William,

Hope you are doing well. Sorry to bother you again. I have followed all the instructions and it looks working great. We do not have call drops and the recording is playing to all the customers.

However, we still have the problem of remote agent going on Dead status after the call hits and campaign stops dialing the leads.

Here is the campaign set up again for your reference:

1) Campaign => Detail View => Survey => Survey Method: CALLMENU (Have set this to all survey campaigns)
2) Added New Call menu => Menu Timeout: 1 => Route: INGROUP (Selected Appropriate Ingroup and Campaign ID)

Please let me know where I am making mistake.

Thanks,

[williamconley]

Question: How did a "remote" agent get involved in this call? You transferred it to an ingroup, which goes to a live agent, it never went to the remote agent that activated the campaign. Unless you are trying to use the same user for a remote agent as you have for a live agent who is logged in ... which you can't do. Each user in the system must be unique when logged in, two of the same agent has bad results. Otherwise, explain which agent is in "DEAD" ... the one who activated the press 1 campaign, or the one who received the call.

[kailashJagtap]

Hello William,

Thanks for the reply. Here are your answers:

We have different remote agents and different users who are logged in as Agents to answer the calls. Remote agent is only used to activate the campaign.

When the call hits from the campaign it goes to the available agents but at the same time the remote agents also goes on Dead. Agent who answers that call is not having any issues. Agent shows on live call but remote agent stays dead unless the call is disposed & stops the campaign.

Please let me know if you need any additional information.

Thanks for your help.

[williamconley]

... When the call hits from the campaign it goes to the available agents but at the same time the remote agents also goes on Dead ...

Did you mean to say the call goes to a call menu, then an ingroup, and then an agent? Or is this a direct-to-agent call which would then NOT need a remote agent to activate the campaign?

[kailashJagtap]

When customer press 1: It goes to Call Menu => Ingroup => agent

Here is the campaign set up again for your reference:

1) Campaign => Detail View => Survey => Survey Method: CALLMENU (Have set this to all survey campaigns)
2) Added New Call menu => Menu Timeout: 1 => Route: INGROUP (Selected Appropriate Ingroup and Campaign ID)

Thanks,

[williamconley]

That raises the question, then, of how the remote agent being used to activate the campaign gets any activity.

Please post the user id and sip phone account for a user who GOT a call and the "User ID Start" and "Number of lines" and "External Extension" for a remote agent who was DEAD as a result of that same call. I'm having a little difficulty believing there's no connection between them. You are sure that the User ID of the Remote Agent is NOT in use anywhere else in the system for anything?

[kailashJagtap]

Remote Agent:

User ID Start: 8881
Number of Lines: 1
External Extension: 3333


User Agent:

Phone Extension: 1015
User Number: garry

Yes, I have checked the entire system thoroughly. All the configurations looks ok to me.

[williamconley]

And these users are in different campaigns?

Change the extension on your remote agent to a real extension that goes to YOUR phone. Something is calling it and then the call is getting lost. You'll find out what is calling it when YOU are the one being called, and you can smack yourself on the side of the head, say "duh" and refuse to talk about it forever after.

[kailashJagtap]

Hello William,

Hope you are doing well. I have a question and wanted to see if you can help. I am currently using Below mentioned Server:

Server Details:

Dell PowerEdge C1100 Intel 24 Thread 64 GB Dedicated Server With 1000 mbps Port with 100 mbps Dedicated Bandwidth.
First Hard Drive :
500 GB SSD Drive
Second Hard Drive:
500 GB SSD Drive

Vicidial Installation Method: Official ISO by Viciboxserver.org

ViciBox v.9.0.0 190913-1108
VERSION: 2.14-719a
BUILD: 190930-2110
© 2019 ViciDial Group

I have mysql database on to 2nd SSD drive.

Campaigns: Press 1 Survey Method

I am able to do around 150 concurrent calls with it. What I want to know is, Is there any way I can add 2 of these kind of servers with same installation and forward their calls to one server where all my agents are logged in? So basically all the agents will login to one server which will have press 1 survey campaign running. Can I have other two servers also doing survey campaign and when people press 1 the calls should go to the different server where agents are logged in. Meaning other two servers will be used to just dial numbers and throw calls to the main server.

Sorry for if I am making this complicated.

I have searched about cluster installation but then the set up cost which people are charging is around $1500 which is too much for me right now.

Kindly let me know if you need any additional information.


Requesting your guidance.