vicidial.org

Hello all,

we have been using goautodial installed on centos 5 from their repository instaall for quite some time now, we recently just decided to move to vicidial instead of goautodial.

we have setup a test server , with following configuration:

CPU: Intel Xeon Quad-Core E3-1230 v5 3.4GHz
RAM: 32GB DDR4
Hard Drive: 500 GB SSD
Ethernet : 1 GBPS

this is a smart dedicated server , means it has dedicated harware , but its virtualllised for abstraction purposes.

we have installed vicidial server version 8.1.2 from its iso , as available from the vicidial download website [ViciBox_v8_1.x86_64-8.1.2]

while following the installation steps , we did not know that this version came with asterisk 13 , so we also did this step (using command vicibox-ast11) which installed the aasterisk version 11 on it
so now our installation is running with aasterisk 11, if its too much of a problem we could probably d a reinstall with asterisk 13.

our current versions of everything on the sserver are :

openSUSE 42.3 (x86_64), VERSION = 42.3, CODENAME = Malachite
Kernel : Linux version 4.4.176-96-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Fri Mar 22 06:23:26 UTC 2019 (a0dd1b8)
Asterisk : Asterisk 11.25.3-vici built by abuild @ cloud124 on a x86_64 running Linux on 2019-04-05 13:29:14 UTC
DAHDI Tools Version - 2.11.1
libpri version: 1.6.0
vicidial VERSION: 2.14-707a , BUILD: 190406-1437

now about the issue we are having , we have setup two broadcast campaigns(8373 routing extension) , but we are not using the press 1 kind of thing , we are just using survey method hangup , with optout. we have assigned two remote agents , one for each campaign. each having 100 lines. and we have set the dial method ration and dial level 1 in the server , and started the campaigns.
now this setup works fine , we see about 200 calls flowing , everything being ddisposed correctly , we have about 100 DIDs for our own testing purposses , we tested these two campaigns there.
we have the max trunks in server settingss set to 1000 , and max calls per ssecond to 50 , also we have Max FILL Calls per Second in system settings set to 100.
the problem arises when we increase the dial level , from 1 to lets say 2 , ,for each campaign , while our real time campaign summary report shows approx 400 calls flowing , roughly 200 ffor each campaign , as it should be according to number of lines on remote agents and dial levell, but we get only 200 calls on our voip vendor.
wwe haave tried this many ways , using more number of remote agents with 10 lines each , increasing the dial level to 10 , at a point the server was showing 800 calls flowing , but we still only got 200 calls on the voip sserver

now to clarify further , the voip server is capable of handling 1500 calls atleast , and handles about 200 cps atleast , we know because its our own server , and we haave used it in past for similar traffic

the same voip server , and same vicidial server , when installed with goaautodial 3.3 , still is able to make 600-700 calls , and all thosse calls reach the voip server fine

the voip server , and the vicidial server are on same network , with aa 1 ms ping delay between them , there is also a 1 GBPS private switch availablle to us , to which we could connect both the servers.

we have tried this vicidiaal setup , with diff voip vendors (our other vendors we have , where have enough account channels and cps to handle the traffic ) but sstill they only get 200 calls form us

now i am sure this is just a silly mistake on my part , and i am missing some setting , that probably limits the number of outbound calls the system can make , but this is my first time using the vicidial directly without goautodial , so i am kind of lost

i did try to read all the settings pages , and go through eveything , in hopes to find something that might be limiting these number calls , but i was unable to find anything , hence i decided to post here and ask for advice/suggestions.

this is a completely testing kind of server ssetup , if needed we can provide access to check out anything , please guide us on how to do this

Thanks a lot in advance.

we are getting the following error on asterisk CLI , when the call load is going on the higher side of 200 ..

[Apr 11 14:04:05] WARNING[23273][C-0000023a]: pbx.c:7086 ast_pbx_start: Failed to create new channel thread
[Apr 11 14:04:05] NOTICE[23273][C-0000023a]: channel.c:5747 __ast_request_and_dial: Unable to call channel Local/99900***********@default

[Apr 11 14:04:05] WARNING[2754]: tcptls.c:746 ast_tcptls_server_root: Unable to launch helper thread: Resource temporarily unavailable


we are getting a llot of these errorrs , on any calls over 200..

i hope someone can help

[Apr 11 14:04:05] WARNING[2754]: tcptls.c:746 ast_tcptls_server_root: Unable to launch helper thread: Resource temporarily unavailable

Clearly your system is out of resources, Asterisk cant create new channels, you should think on cluster installation or reduce the load on the server

viewtopic.php?f=4&t=37898

http://forums.asterisk.org/viewtopic.php?f=1&t=83900

do you reboot your server daily?

i rebooted the server and ran the test again

now below were the stats before starting the dialing , when the server was just rebooted :

Code: Select all

vicibox81:~ # free -m
             total       used       free     shared    buffers     cached
Mem:         30003       1290      28713         22         33        394
-/+ buffers/cache:        861      29141
Swap:         8190          0       8190

and the load from vicidial report page :

Code: Select all

SERVER +   DESCRIPTION   IP   ACT   LOAD   CHAN   AGNT   DISK   OUTBOUND   INBOUND
vicibox81    Server vicibox81    69.60.110.128    Y / Y / Y    39 - 3%    0    200    4%    LINK    LINK


now after this we started the dialing , and let the call flow reach till 400 , when the errors start showing up on asterisk CLI , the output of above commands at that time is

Code: Select all

vicibox81:~ # free -m
             total       used       free     shared    buffers     cached
Mem:         30003       2687      27315         34         37        473
-/+ buffers/cache:       2176      27827
Swap:         8190          0       8190

and the load from vicidial reports page is

Code: Select all

SERVER +   DESCRIPTION   IP   ACT   LOAD   CHAN   AGNT   DISK   OUTBOUND   INBOUND
vicibox81    Server vicibox81    69.60.110.128    Y / Y / Y    84 - 39%    468    200    4%    LINK    LINK

SERVER +   DESCRIPTION   IP   ACT   LOAD   CHAN   AGNT   DISK   OUTBOUND   INBOUND
vicibox81    Server vicibox81    69.60.110.128    Y / Y / Y    84 - 39%    468    200    4%    LINK    LINK

Clearly your system is out of resources, Asterisk cant create new channels, you should think on cluster installation or reduce the load on the server

i dont understand , which resources it thinks are lacking , because there is sufficient RAM , and processors available , also the bandwidth usage is not an issue either , its a 1 GBPS NIC card , both our vicidial server , and voio server are in same network , same local network

viewtopic.php?f=4&t=37898

http://forums.asterisk.org/viewtopic.php?f=1&t=83900

do you reboot your server daily?

yes wwe have a cron job to reboot the server at middnight , and since currently we are only in testing phase , we reboot before starting any test also.

thanks for the above links @williamconley , but i have already gone through them from my earlieer google search , and we are not facing RAM limitation issues like them, we have the likes of 70% RAM being free

load values from the vicidial reports page requires that you calibrate that system to know what load is good or bad. you don't know that yet, and as a result can't calibrate.

to check the load of the system you use the "uptime" command at the CLI or the htop or top commands.

as a convenience, the average server load values from the updtime command are shown at the bottom of the list of agents in the real time screen.

next up: If you think the value shown in the real time screen for live calls is inaccurate, you must remember that this is a time-limited view and calls move rather rapidly.

compare that value to the asterisk -rx "core show channels" values. remember that many of those results will be agent sessions (in meetme rooms) and recordings. So some fanciful use of grep and wc -l can get you a more realistic number.

however: if you are running a heavy-dialing system, it's quite possible that the "can't keep up" is due to calls terminating rapidly. in this case it's often necessary to dramatically increase your Admin->Servers "max calls per second" value. I've seen places with high-volume dialing that needed this value to be at 300 to keep the system dialing "at speed". of course, the carrier will need to be consulted as many limit the value MUCH lower than 300.

if you think the numbers are wrong, and not merely due to timing of the db request vs terminating calls, then the question is how the dialer could think that calls are in progress when they really aren't. this can happen if some of the numbers do not match a dialplan entry. we generally install an "s" dialplan entry without the dial line (just AGI and Hangup) to catch and kill these rogue numbers immediately. Killing them (with the AGI process already attached) clears them out of the tables immediately and then the system can generate new calls to replace them immediately and that makes the stats accurate even with bad/impossible phone numbers in the mix. We also put a "NoOp()" line with a note about not matching any dialplan entries, so these are easily found in the logs if someone gets curious.

PS: It is useful if Before You Begin the narrative, you clearly state the basic problem in English. Like: Dialer says 200 calls in progress and 300 calls ringing, but carrier says only 400 calls total. Try to make it technical and "jargon-free" if possible.

thanks for the reply @williamconley, i have ordered another server with same configuration , it should be arriving soon , i will install the older version of go-auto-dial 3.3 that we used to run earlier on it , and run the same settings there to analyse what difference is goingon with both.

also i will go through all the things you gave , and get back with results.

okay , so we ran the test , and the cpu is being overloaded, i noticed from the htop command that the asterisk process and mysql process collectively takes about 100 , sometimes more than 100% of CPU usage, in different ratios

so thats figured out , but what i still could not understand is this , i found one of my old machines, which had goautodial 3.3 installed on it , and brought it up, setup the same 2 campaigns there , and replicated the scenerio there .
after starting the campaigns there , it ran almost without any errors , with 400-450 calls flowing well, and even the weirdest thing is , the configuration for that machine is way lower than this vicidial 8.1.2 machine
any idea on why a newer version on a better machine is taking more load than an older version on a not so better machine ??

Could it be the aasterisk 11 and asterisk 13 difference ?

also what i noticed in the cli was that , when a call was not completed due to resources not available error , it did not get cleared , means it was not initiated , but it stayed there in the "core show channels" , then after the "TIMEOUT" as defined in campaign settings , it was hung up , and after that a new call replacing that was placed , so for the whole duration of "timeout seconds" all those 200 or so channels that were not dialled remained in the "core show channels" and preveneted new calls being generaated.

if you think the numbers are wrong, and not merely due to timing of the db request vs terminating calls, then the question is how the dialer could think that calls are in progress when they really aren't. this can happen if some of the numbers do not match a dialplan entry. we generally install an "s" dialplan entry without the dial line (just AGI and Hangup) to catch and kill these rogue numbers immediately. Killing them (with the AGI process already attached) clears them out of the tables immediately and then the system can generate new calls to replace them immediately and that makes the stats accurate even with bad/impossible phone numbers in the mix. We also put a "NoOp()" line with a note about not matching any dialplan entries, so these are easily found in the logs if someone gets curious.

we dont have bad numbers in the system , thesse are verified numbers uploaded for the testing purpose, but i did put up one dialplan s entry just in case , to catch any dropped calls like that

PS: It is useful if Before You Begin the narrative, you clearly state the basic problem in English. Like: Dialer says 200 calls in progress and 300 calls ringing, but carrier says only 400 calls total. Try to make it technical and "jargon-free" if possible.

i am sorry , this was my first time posting here , i did not know how to post it , will keep it in mind from next time.

mysql always takes up the full memory and often takes up 100% of a single cpu for a short time (this 100% usage can move between cpus as more processes run). is there a slow query happening?

but you didn't post the memory usage or the average server load values which require a core count to have meaning.

Code: Select all

free -m
uptime
cat /sys/devices/system/cpu/online

compare those across the two machines during heavy load. then you can begin to troubleshoot (instead of guessing? lol)

it's not about version, nor is it about "flowing well", which is vague. there must be a failure somewhere. a rejected/failed command, call, or process. overload is not the only reason for failure, it's just a possibility at this point as you have not demonstrated an overload.

i am sorry , i agree i was very vague , as it was going on between my work times

i will be running the test again in off-dialing hours again tomorrow , and posting a complete detailed report.

so you're looking for warnings or fails in file-based logs or command line interfaces (asterisk, linux, /var/log and subdirs). you may also get root email with problems in it.

either there is an error blocking capacity, or the system is not trying because of a configuration error or misunderstanding.

overload can cause errors that block capacity, but that's just ONE possibility. begin by "hip shots" to see if any normal problems are obvious and fixable. failing that, it's time to dig through and see if any of the logs jump out as problematic (never a bad idea even when there isn't a problem). research any bad log entries to see if they are problematic. many are merely notices of known situations and are merely annoying in their distraction.

if all that fails, it's troubleshooting time. walk from beginning to end of the process starting with the problem as you see it. and determine the cause. starting with a simple statement of "the problem" in technical terms.

by technical terms: I don't mean a technical cause, but a solid description, nothing vague ... something observable by anyone without ambiguity, even someone who has never been "in on the conversation", based on your description.

EG: NOT "It's not dialing enough"
EG: More Like "The campaign has 5 agents on calls and 12 available agents and is set to 2:1 dial ratio, and claims 24 calls ringing and 5 live calls, but the carrier only shows 10 live calls, which includes the agents' live calls with leads (5) and then 5 more which should be 24 more, not 5 more ... where are my missing 19 calls?"