vicidial.org

by **dlasry** » Wed Oct 21, 2009 7:13 am

Hi Guys,

My agents have been getting a relatively high number of “Dead Calls”. The call comes in, gets routed to the agent, but nobody is on the line. This seems to happen to approximately one out of every 5 calls.

Server CPU, and Load Averages are all performing very well and I have used Wireshark to rule out any SIP provider or network congestion issues.

Is this a known issue ? What would you advise I do next to trouble shoot this? Are my campaign settings too aggressive? Is this why people are using Netboarder CPA..?

I am using an all in one configuration, for the time being:

Server Specs:
IBM System 3650
2 XEON Quad Cores
12 GB RAM
HD SCSI on RAID 5

Vici:
-VICBox 2.05 ( Build 90320-0424) on a VmWare ESX architecture ( however, no other servers on or sharing any resources)

Trunks:
-SIP Trunks over a 9.5 Mbit dedicated circuit,.
-No encoding (G711)

Campaign variables:
-Standard built in AMD on (VDAD 8369)
-No Recording.
-Adapt-Average
-Max Dial Level-5

Thanks
David

by **mflorell** » Wed Oct 21, 2009 8:33 am

These are inbound calls?

Have you tried another carrier?

Asterisk CLI output?

by **dlasry** » Wed Oct 21, 2009 10:08 am

Hi Matt,
These are outbound predictive calls. No I have not tried any other carriers, but I also didn't see any errors at the SIP header level. Do you think this could be a carrier issue?

Here is the CLI output for 2 sample calls:

Example 1 (7182586852)::
-- Executing AGI("Local/917182586852@default-ae5e,2", "agi://127.0.0.1:4577/call_log") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
-- Executing Dial("Local/917182586852@default-ae5e,2", "SIP/N2P/17182586852|55|To") in new stack
-- Called N2P/17182586852
-- AMD: Word detected. iWordsCount:2
-- AMD: Word detected. iWordsCount:2
-- AMD: Changed state to STATE_IN_SILENCE
-- Nobody picked up in 55000 ms
-- Executing Hangup("Local/917186773349@default-420b,2", "") in new stack
== Spawn extension (default, 917186773349, 3) exited non-zero on 'Local/917186773349@default-420b,2'
-- Executing DeadAGI("Local/917186773349@default-420b,2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----NOANSWER----------") in new stack
-- Nobody picked up in 55000 ms
-- Executing Hangup("Local/917183694380@default-1d04,2", "") in new stack
== Spawn extension (default, 917183694380, 3) exited non-zero on 'Local/917183694380@default-1d04,2'
-- Executing DeadAGI("Local/917183694380@default-1d04,2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----NOANSWER----------") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... ---------- completed, returning 0
-- AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... ---------- completed, returning 0
== Manager 'sendcron' logged off from 127.0.0.1
== Manager 'sendcron' logged off from 127.0.0.1
-- AMD: ANSWERING MACHINE: voiceDuration:2000 greeting:2000
-- Executing AGI("SIP/N2P-082de0e8", "VD_amd.agi|8369") in new stack
-- Launched AGI Script /var/lib/asterisk/agi-bin/VD_amd.agi
== Spawn extension (default, 8369, 4) exited non-zero on 'SIP/N2P-082de0e8'
-- Executing DeadAGI("SIP/N2P-082de0e8", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0---------------") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... ---------- completed, returning 0
== Parsing '/etc/asterisk/manager.conf': Found
== Manager 'sendcron' logged on from 127.0.0.1
-- Executing AGI("Local/917182591834@default-7462,2", "agi://127.0.0.1:4577/call_log") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
-- Executing Dial("Local/917182591834@default-7462,2", "SIP/N2P/17182591834|55|To") in new stack
-- Called N2P/17182591834
-- SIP/N2P-08339728 answered Local/917182525102@default-f03d,2
> Channel Local/917182525102@default-f03d,1 was answered.
-- Executing Playback("Local/917182525102@default-f03d,1", "sip-silence") in new stack
-- Playing 'sip-silence' (language 'en')
== Manager 'sendcron' logged off from 127.0.0.1
-- Executing AGI("Local/917182525102@default-f03d,1", "agi://127.0.0.1:4577/call_log") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
-- Executing AMD("Local/917182525102@default-f03d,1", "2000|2000|1000|5000|120|50|4|256") in new stack
-- AMD: Local/917182525102@default-f03d,1 16462918907 (null) (Fmt: 64)
-- AMD: initialSilence [2000] greeting [2000] afterGreetingSilence [1000] totalAnalysisTime [5000] minimumWordLength [120] betweenWordsSilence [50] maximumNumberOfWords [4] silenceThreshold [256]
== Spawn extension (default, 917182525102, 2) exited non-zero on 'Local/917182525102@default-f03d,2'
-- Executing DeadAGI("Local/917182525102@default-f03d,2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----25-----1") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... --25-----1 completed, returning 0
-- AMD: Word detected. iWordsCount:1
-- AMD: Word detected. iWordsCount:3
-- SIP/N2P-082d24f0 is making progress passing it to Local/917182586852@default-ae5e,2
-- AMD: Changed state to STATE_IN_SILENCE
-- AMD: Changed state to STATE_IN_SILENCE
-- AMD: Word detected. iWordsCount:2
-- AMD: Changed state to STATE_IN_SILENCE
-- SIP/N2P-082d24f0 answered Local/917182586852@default-ae5e,2
> Channel Local/917182586852@default-ae5e,1 was answered.
-- Executing Playback("Local/917182586852@default-ae5e,1", "sip-silence") in new stack
-- Playing 'sip-silence' (language 'en')
== Manager 'sendcron' logged off from 127.0.0.1
== Spawn extension (default, 917182586852, 2) exited non-zero on 'Local/917182586852@default-ae5e,2'
-- Executing DeadAGI("Local/917182586852@default-ae5e,2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----5-----0") in new stack

Example 2 (7188533530):
-- Executing AGI("Local/917188533530@default-f2df,2", "agi://127.0.0.1:4577/call_log") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
-- Executing Dial("Local/917188533530@default-f2df,2", "SIP/N2P/17188533530|55|To") in new stack
-- Called N2P/17188533530
-- SIP/N2P-082da650 is making progress passing it to Local/917188533530@default-f2df,2
-- SIP/N2P-082da650 answered Local/917188533530@default-f2df,2
> Channel Local/917188533530@default-f2df,1 was answered.
-- Executing Playback("Local/917188533530@default-f2df,1", "sip-silence") in new stack
-- Playing 'sip-silence' (language 'en')
== Manager 'sendcron' logged off from 127.0.0.1
-- Executing AGI("Local/917188533530@default-f2df,1", "agi://127.0.0.1:4577/call_log") in new stack
-- AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
-- Executing AMD("SIP/N2P-082da650", "2000|2000|1000|5000|120|50|4|256") in new stack
-- AMD: SIP/N2P-082da650 16462918907 (null) (Fmt: 64)
-- AMD: initialSilence [2000] greeting [2000] afterGreetingSilence [1000] totalAnalysisTime [5000] minimumWordLength [120] betweenWordsSilence [50] maximumNumberOfWords [4] silenceThreshold [256]
== Spawn extension (default, 917188533530, 2) exited non-zero on 'Local/917188533530@default-f2df,2'
-- Executing DeadAGI("Local/917188533530@default-f2df,2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----9-----0") in new stack

FYI - I turned on call recording today, just to see how it would affect the server performance.
Thanks
David

by **mflorell** » Wed Oct 21, 2009 10:45 pm

How many concurrent calls?

What is the peak loadavg?

by **dlasry** » Thu Oct 22, 2009 2:25 am

Here is data from Tuesday.

From what I saw using the real time reporting, and HTOP, load average never went above 0.3.

VICIDIAL: Server Performance 2009-10-22 10:13:47
Time range: 2009-10-20 00:00:00 to 2009-10-20 23:59:00

---------- TOTALS, PEAKS and AVERAGES
Total Calls in/out on this server: 17442
Total Off-Hook time on this server (min): 6873.18
Average/Peak channels in use for server: 16.7770 / 92
Average/Peak load for server: 26.9242 / 240
Average USER process cpu percentage: 0.9642 %
Average SYSTEM process cpu percentage: 2.3025 %
Average IDLE process cpu percentage: 96.7003 %

---------- LINE GRAPH:
rows: 20130 tick: 120 minutes scale: 30 minutes

Yesterday, we pushed the aggressively up significantly, and the problem seemed to stay at the same ratio.

Here is the data for a one hour slice. Load Average never went above 3.5.

VICIDIAL: Server Performance 2009-10-22 10:19:19
Time range: 2009-10-21 17:00:00 to 2009-10-21 18:00:00

---------- TOTALS, PEAKS and AVERAGES
Total Calls in/out on this server: 3376
Total Off-Hook time on this server (min): 1311.70
Average/Peak channels in use for server: 74.2282 / 216
Average/Peak load for server: 136.0678 / 485
Average USER process cpu percentage: 4.4003 %
Average SYSTEM process cpu percentage: 9.6206 %
Average IDLE process cpu percentage: 85.9661 %

---------- LINE GRAPH:
rows: 767 tick: 5 minutes scale: 1 minute

Any way this could be a timing issue, being that I am using pure SIP and no hardware?

David

by **mflorell** » Fri Oct 23, 2009 6:21 pm

Could be a timer issue, what do your 1-hour zttest results look like?

Have you thought about getting a Sangoma VoiceTime USB module?

by **Op3r** » Sat Oct 24, 2009 11:54 am

Vici:
-VICBox 2.05 ( Build 90320-0424) on a VmWare ESX architecture ( however, no other servers on or sharing any resources)

dlsary

I never heard anyone with more than 5 seats having a good time using VICIDIAL on a virtual server.

by **dlasry** » Sat Oct 24, 2009 12:02 pm

I agree. My problem was that I thought the need for a timing device was only for PRI / T1 type trunks (hardware based). This is what I mistakenly inferred from a previous post of Matt's.

In any case, none of the other machines on the server are on. But I still agree that this would be the reason.

FYI I did a zttest, today, with nobody on the server and here were the results:

--- Results after 2901 passes ---
Best: 100.000000 -- Worst: 99.084473 -- Average: 99.981278

David

by **mflorell** » Mon Oct 26, 2009 9:24 am

You really need to do the zttest during production when you have agents on.

by **dlasry** » Mon Oct 26, 2009 10:55 am

Hi Matt.

Here are the results for a 1 hour period during when the system was in use.

--- Results after 3716 passes ---
Best: 100.000000 -- Worst: 94.555664 -- Average: 99.939947

Do think this implicates Timing as the problem?

David

by **mflorell** » Mon Oct 26, 2009 1:01 pm

Ouch, 94 is really bad. They should really stay around 99.9 or higher for good voice quality.

by **dlasry** » Sun Nov 01, 2009 11:19 am

FYI We put the team on a physical (non vmware) server identical to the first, with just 1 quad core, and only 4GB ECC RAM and the problem seems to have resolved itself.

I am sure that the USB sangoma Timing device would help though.

David

by **mflorell** » Sun Nov 01, 2009 5:24 pm

As it has been mentioned about 100 times here on the forums, you should not be using a virtual machine for your production ViciDial server.

vicidial.org

Dead Calls?

Dead Calls?

Who is online