Dropped Calls While Closer is in a call

[kashinc]

1.)
VERSION: 2.12-479a
BUILD: 150313-0912

2.)
System Load Average: 0.72 1.39 1.19

3.)
cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
stepping : 3
microcode : 0x1c
cpu MHz : 1800.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 6385.14
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
stepping : 3
microcode : 0x1c
cpu MHz : 1700.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 6385.14
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 6385.14
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
stepping : 3
microcode : 0x1c
cpu MHz : 1500.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 6385.14
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:


cat /proc/meminfo
MemTotal: 16255412 kB
MemFree: 11223932 kB
Buffers: 178568 kB
Cached: 3519792 kB
SwapCached: 0 kB
Active: 3299080 kB
Inactive: 1341280 kB
Active(anon): 988504 kB
Inactive(anon): 20776 kB
Active(file): 2310576 kB
Inactive(file): 1320504 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4176892 kB
SwapFree: 4176892 kB
Dirty: 488 kB
Writeback: 0 kB
AnonPages: 941704 kB
Mapped: 56800 kB
Shmem: 67284 kB
Slab: 229500 kB
SReclaimable: 193512 kB
SUnreclaim: 35988 kB
KernelStack: 3936 kB
PageTables: 30992 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12304596 kB
Committed_AS: 2460828 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 373212 kB
VmallocChunk: 34359363647 kB
HardwareCorrupted: 0 kB
AnonHugePages: 407552 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 59460 kB
DirectMap2M: 2873344 kB
DirectMap1G: 13631488 kB

4.)
Codecs used
ulaw, g729

5.)
VOIP

6.) ViciBox 6.0.3
I followed the express settings from the installation manual.

Hardware
Amfeltec X1 PCI Express System Timer
Samsung 850 Pro 256GB – not setup on a raid array.

I do plan on moving over to a cluster setup over the weekend
1.) Database server / Webserver
32 Gigs of Ram
Three Samsung 850 Pro's 256GB Setup on a Raid 1 Array using an LSI Megaraid Card
2.) Dialer – Asterisk Server
16GB Ram
only one Samsung 850 Pro 256GB not setup in a raid array.

My first question is will the dialing server that is not configured in a raid array cause any issues or dropped calls?


My current Issue

I currently have 9 agents on one server with a dial rate of 13 and for the most part everything has been running smoothly until the last few days. We are doing an average of about 40,000 outbound dials a day right now. We have 3 closers setup.

It seems that my closers are dropping calls during sales pitches, I have a transferqueue setup that is handling the transfers to the next available agent based on longest waiting hold times. Sometimes the drops are random but they happen often when one sales agent is on a call and another call gets transferred to the queue. We have dropped a few calls when transfering to the second agent.

Any input would be greatly appreciated.

[bobchaos]

9 agents dialing 13:1 would likely require more than 1 dialing server. There's a limitation in Asterisk where it starts to break down over 250 channels, and Vicidial tends to have multiple channels open per line. Try dialing it down to, say, 4:1 and see if the issue persists.

Not using a RAID array could be in cause, see if it's causing any latency to read/write ops on your MySQL server. That would certainly cause dropped calls. The output of your asterisk console during one such event would also help a lot, hard to troubleshoot without it. More often than not if you read the output carefully you'll see something illogical in there that will point you directly to your root cause (well, if it is a capacity issue probably not, but anything to do with conf...)

[kashinc]

9 agents dialing 13:1 would likely require more than 1 dialing server. There's a limitation in Asterisk where it starts to break down over 250 channels, and Vicidial tends to have multiple channels open per line. Try dialing it down to, say, 4:1 and see if the issue persists.

Not using a RAID array could be in cause, see if it's causing any latency to read/write ops on your MySQL server. That would certainly cause dropped calls. The output of your asterisk console during one such event would also help a lot, hard to troubleshoot without it. More often than not if you read the output carefully you'll see something illogical in there that will point you directly to your root cause (well, if it is a capacity issue probably not, but anything to do with conf...)

Hey Bobchaos,

Thank you for your response, I switched my mysql server and went into a cluster config. My MYSQL/Webserver combo has been performing a lot better, that is setup on a raid 10 using a MegaRaid LSI card. My dropped calls have definitely decreased. I am still dropping 2-3 calls a day when I turn up the ratio because I only have one dialing server. I plan on adding a second dialing server over the weekend to see if this will minimize the dropped calls as well. I also ordered another server and was thinking of having my closers on a server that is not constantly dialing out to decrease the chances of the closers dropping calls while in the middle of a sales pitch. How important is it for my dialing servers to be on a raid configuration? I am using Samsung 850 Pro's on all my machines which is currently rated as one of the best SSD's in the market. I am trying to minimize costs on my end so if I could avoid adding MegaRaid LSI cards to my dialing servers it would help a lot. I just don't know how important this is on the dialing servers.

any input would be much appreciated.

-kashinc

[bobchaos]

Once you're in the wonderful world of clusters, you can consider buying cheap "throw-away" servers. You get the sturdiest MySQL machine you afford, but for dialing and webservers don't bother with RAID or registered RAM or fancy stuff like that. Just get more servers instead. If you run them on SSD that's already way more juice than you'd need, unless you do lots of long recordings. If you do record everything, go with a 10k HDD or an SSD with 8G ram, but again, RAID is overkill in clusters.

Use all the money you save on redundancy to buy an extra web and extra dialing server, it's overall a sturdier solution for limited budgets.

*Edit*

Also, consider getting a slave MySQL server so you can failover to that if/when the Master dies.

[kashinc]

Once you're in the wonderful world of clusters, you can consider buying cheap "throw-away" servers. You get the sturdiest MySQL machine you afford, but for dialing and webservers don't bother with RAID or registered RAM or fancy stuff like that. Just get more servers instead. If you run them on SSD that's already way more juice than you'd need, unless you do lots of long recordings. If you do record everything, go with a 10k HDD or an SSD with 8G ram, but again, RAID is overkill in clusters.

Use all the money you save on redundancy to buy an extra web and extra dialing server, it's overall a sturdier solution for limited budgets.

*Edit*

Also, consider getting a slave MySQL server so you can failover to that if/when the Master dies.

Hey Bobchaos,

Thank you so much for the help, much appreciated !

-kashinc