Hardware Recommendation

[mubeen]

Hi,

I've been using Vicidial for few months now, but limits of my server are being tested as agents have grown to a number of 40 with 30 cps. I'm thinking of Clustering to support 50 users for press - 1 campaign with 500 cps

Hardware I'm thinking of using is

For DB:
Intel Xeon E3-1270V3 (CPU-B 9852), 32 GB ECC Ram, 2x4 TB SATA with 1G NIC

For Telephony:
Intel Xeon E3-1275V2 (CPU-B 9498), 32 GB ECC Ram, 2x300 GB SAS with 1G NIC

For Web:
Intel Xeon E3-1245V2 (CPU-B 9003), 32 GB ECC Ram, 2x2 TB SATA with 1G NIC

For Archive:
Intel Xeon E3-1245V2 (CPU-B 9003), 32 GB ECC Ram, 2x4 TB SATA with 1G NIC

Not sure if Hyper-threading as I think more cores vs Hyper-threading is better

I probably wont be going for higher end models instead will be adding more server if required. Please suggest

[Kumba]

500 calls per second isn't going to happen on two telephony servers. Maybe this is just an english translation error and you mean 500 total calls? That is certainly doable on two telephony servers. Asterisk itself doesn't respond well to CPS above 30. In fact we usually run a CPS of 10 or less which helps your carrier process the calls without throttling and you end up with a better connect rate.

Your hardware for the database is wrong. I'd recommend using 2x1TB enterprise SATA drives in RAID-1 for the OS and then 2x256GB SSD drives in RAID-1 under /srv/mysql for the database portion. The drives you have listed are likely 7200-rpm or slower SATA drives which will never keep up with two dialers running on Press-1 campaigns. RAM and CPU are more or less OK but using a server that allows you to put in a second CPU and more RAM later if you continue to grow would be a better option. That would mean stepping up to an E5-2600 series Xeon or the new Xeon Scalable CPUs. But if budget is a concern then the E3-1270 CPU and 32GB of RAM will be fine given your current configuration.

On the other servers I would use the E3-1245 CPU with 8GB of RAM and 2x1TB Enterprise SATA drives in RAID-1. The only server that might benefit from more then 8GB of RAM is the Telephony server if you are going to be recording calls that are exceptionally long. Usually we only see this issue on inbound customer service call centers where multiple callers might be on the phone for an hour or more. In this scenario what happens is the exceptionally long number of calls that are being recorded into the system ram fills it up. Outbound call centers usually never run into this issue since calls rarely last long then 10-15 minutes.

The only exception to these servers is the archive server. I would probably use 2x4TB drives or maybe even go larger to 6 or 8TB. You can also buy a NAS device like a synology NAS that can fill in as the archive server. The NAS just needs to be able to accept FTP uploads and give you a method of downloading the files you upload via HTTP. Almost all NAS' at this point offer that sort of functionality.

And just to clarify, an Enterprise SATA drive would be something like a SeaGate Enterprise or Western Digital RE drives. We're pretty fond of Seagate drive model ST1000NM0008 for 1TB drives.

[mubeen]

Thank you for detailed reply. Just for testing purpose I went ahead and installed VICIBOX 7.0.4 (downloaded from vicidial archive) on all machines with cluster configuration.
What exactly I did was

Run vicibox and installed OS using os-install, then using vicidial-install I installed DB then Archive, then web and telephony in last. After installation I created campaign and created users. I also added all 3 servers in GUI from add server option in webserver.

Now when I login as an agent, connecting calls come on eyebeam, It does say you are currently only....but after like 10-15 sec session disconnects

[williamconley]

limits of my server are being tested

What limits? I presume there's an error somewhere or long wait times or something is RED? Without that information it's hard to help without writing a book to cover every scenario (which is not likely today, but Kumba Stepped Up which is cool).

Thank you for detailed reply. Just for testing purpose I went ahead and installed VICIBOX 7.0.4 (downloaded from vicidial archive) on all machines with cluster configuration.
What exactly I did was

Run vicibox and installed OS using os-install, then using vicidial-install I installed DB then Archive, then web and telephony in last. After installation I created campaign and created users. I also added all 3 servers in GUI from add server option in webserver.

Now when I login as an agent, connecting calls come on eyebeam, It does say you are currently only....but after like 10-15 sec session disconnects

Use the vicidial manager's manual available on EFLO.net and start at the first page. Post here the first error you bump into, the page/line you're on, and what happened. Also post what you Expected to happen and any relevent configuration information. Don't skip any pages. Start at the front and keep going until everything you need works.

In this case, it would also be very useful to post the asterisk CLI from a single agent login. From inception to termination, one full call, but don't have anything else going on at the same time (there should only be a couple dozen lines of code, not 3000 lines of unrelated output, lol).

Please also remember to Always post your Full Vicidial Version with Build. Essential for troubleshooting both now and months from now when someone else reads this post.

[mubeen]

What limits? I presume there's an error somewhere or long wait times or something is RED? Without that information it's hard to help without writing a book to cover every scenario (which is not likely today, but Kumba Stepped Up which is cool).

That server I was talking about was a single server, and I meant to say it has already reached the limit of agents it can handle.

Use the vicidial manager's manual available on EFLO.net and start at the first page. Post here the first error you bump into, the page/line you're on, and what happened. Also post what you Expected to happen and any relevent configuration information. Don't skip any pages. Start at the front and keep going until everything you need works.

In this case, it would also be very useful to post the asterisk CLI from a single agent login. From inception to termination, one full call, but don't have anything else going on at the same time (there should only be a couple dozen lines of code, not 3000 lines of unrelated output, lol).

Please also remember to Always post your Full Vicidial Version with Build. Essential for troubleshooting both now and months from now when someone else reads this post.

Sorry for not posting complete version

VERSION: 2.14-694a
BUILD: 181005-1738

I did followed manager's manual, but for some reason its not working with clusters. Same configuration is working with express installation.

Pasting CLI of telephony server of clusters

Code: Select all

[Oct 26 17:11:15] WARNING[1441]: chan_sip.c:4101 retrans_pkt: Timeout on 2135780793-357499114-1980619025 on non-critical invite transaction.
[Oct 26 17:11:16] WARNING[1441]: chan_sip.c:4101 retrans_pkt: Timeout on 3734824d544c19b098141eb2033883d9 on non-critical invite transaction.
[Oct 26 17:11:35]   == Manager 'sendcron' logged on from 127.0.0.1
[Oct 26 17:11:35]   == Using SIP RTP CoS mark 5
[Oct 26 17:11:37]        > Channel SIP/1001-0000001a was answered
[Oct 26 17:11:37]     -- Executing [8600052@default:1] MeetMe("SIP/1001-0000001a", "8600052,F") in new stack
[Oct 26 17:11:37]   == Parsing '/etc/asterisk/meetme.conf': Found
[Oct 26 17:11:37]   == Parsing '/etc/asterisk/meetme-vicidial.conf': Found
[Oct 26 17:11:37]     -- Created MeetMe conference 1023 for conference '8600052'
[Oct 26 17:11:37]     -- <SIP/1001-0000001a> Playing 'conf-onlyperson.gsm' (language 'en')
[Oct 26 17:11:37]        > 0x7fe070019f80 -- Probation passed - setting RTP source address to 182.176.118.54:6400
[Oct 26 17:11:38]   == Manager 'sendcron' logged off from 127.0.0.1
[Oct 26 17:11:46] WARNING[1441]: chan_sip.c:4101 retrans_pkt: Timeout on 1a22f1be9318a6608d7624a3b5b9dfad on non-critical invite transaction.
[Oct 26 17:12:01]   == Manager 'sendcron' logged on from 127.0.0.1
[Oct 26 17:12:01]   == Manager 'sendcron' logged on from 127.0.0.1
[Oct 26 17:12:01]   == Manager 'sendcron' logged off from 127.0.0.1
[Oct 26 17:12:06]   == Manager 'sendcron' logged on from 127.0.0.1
[Oct 26 17:12:06]   == Manager 'sendcron' logged off from 127.0.0.1
[Oct 26 17:12:11]   == Manager 'sendcron' logged off from 127.0.0.1
[Oct 26 17:12:15] WARNING[1441]: chan_sip.c:4101 retrans_pkt: Timeout on 597b745fbff2f61523f4d1c495037c56 on non-critical invite transaction.

[williamconley]

it has already reached the limit of agents it can handle

Ok ... how do you know? Is there a popup window saying "You've exceeded the number of agents this server can handle!" or am I missing some information? (I'll bet it's the latter )

Code: Select all

Playing 'conf-onlyperson

I see the start of the call, some retransmission fails, but no termination. How did the call END? Was a reason given?

[mubeen]

Ok ... how do you know? Is there a popup window saying "You've exceeded the number of agents this server can handle!" or am I missing some information? (I'll bet it's the latter )

Actually I'm simultaneously dialing 90-100 calls with approx 50 users, which include inbound, manual outbound and a dialer. Looking at resources from CLI, much of the RAM is being consumed as well as the load is above 3. When this happens I face call issues.

Code: Select all

Playing 'conf-onlyperson

I see the start of the call, some retransmission fails, but no termination. How did the call END? Was a reason given?[/quote]

I got the issue resolved, I made a mistake in configuring server. I didn't add vicidial version when adding servers (beginners mistakes LOL), but yes I'm clustering first time..

[williamconley]

I face call issues.

We're getting closer. But it almost seems like either you're relating this information from someone else who told you "it don't work so good" and nothing else, or you're hiding the actual problem for some reason.

One more time: WHAT "call issues" do you face?

I'm trying. Really hard. lol

[mubeen]

Well I may be wrong here, may be the issues are related to something else not hardware as I'm thinking, but issues like server web interface takes time to reach, Agents are active but being shown paused on admin screen, using press-1 campaign giving issues like for VOIP number, call is being transferred to agents but drops for real number with buzz sound (which I never was able to resolve)

This is the main reason I'm moving for cluster setup. I do have to run the setup (to start with for new campaign) for 10 agents with dial level 100 or so.

[williamconley]

Now we're talking! These are likely individual "enterprise" issues, not necessarily related to each other. They should be handled one at a time.

server web interface takes time to reach

This is often due to the number of apache processes being too low.

Try these prefork settings on the web server (replace the existing prefork section or just change the numbers ... depending on how well this matches your existing prefork section)

Code: Select all

nano /etc/apache2/server-tuning.conf

Code: Select all

# prefork MPM
<IfModule prefork.c>
        # number of server processes to start
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
        StartServers         450
        # minimum number of server processes which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/prefork.html#minspareservers
        MinSpareServers      250
        # maximum number of server processes which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/prefork.html#maxspareservers
        MaxSpareServers     500
        # highest possible MaxClients setting for the lifetime of the Apache process.
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
        ServerLimit        768
        # maximum number of server processes allowed to start
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
        MaxClients         768
        # maximum number of requests a server process serves
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
        MaxRequestsPerChild  1000
</IfModule>

Also seriously consider turning on the apache status module

Code: Select all

nano /etc/apache2/default-server.conf

We usually put this just below the Directory "/srv/www/htdocs" section (not inside the directory section, beneath it as it's not related to that directory at all)

Code: Select all

# Load the status module for http://XXX.poundteam.com/server-status
LoadModule status_module /usr/lib64/apache2/mod_status.so
ExtendedStatus On
<Location /server-status>
  SetHandler server-status
</Location>

After either of these changes, you'll need

Code: Select all

service apache2 restart

That should allow a significantly higher number of apache processes to run and resolve your "waiting to get a connection" problem.

With the status monitor you can now surf to:

Code: Select all

http://xx.xx.xx.xx/server-status

and see the processes available and in use. You'll see the problem in the status monitor when you next run out of processes (if ever).

You could try activating the status monitor first, wait for your next "problem" and see how it changed so you can more clearly understand the issue.

Extras:

/server-status?auto provides a simpler view
/server-status?refresh=10 provides an auto-refreshing view
http://aliso2.poundteam.com/server-stat ... refresh=10 and now you have both!

Agents are active but being shown paused on admin screen

Did you try refreshing the screen? Or changing the refresh rate? Sometimes it can be set for 40 seconds, which obviously gets stale pretty quickly.

using press-1 campaign giving issues like for VOIP number, call is being transferred to agents but drops for real number with buzz sound (which I never was able to resolve)

I didn't get that at all. But it sounds like you may need to change carriers. Please describe in more detail (and try with another carrier to see if your problem magically disappears).

This is the main reason I'm moving for cluster setup. I do have to run the setup (to start with for new campaign) for 10 agents with dial level 100 or so.

It is quite likely you'll need multiple servers when you approach 1000 channels. But I hope you're doing political work so you're legal (or outside the US, of course).

[mubeen]

using press-1 campaign giving issues like for VOIP number, call is being transferred to agents but drops for real number with buzz sound (which I never was able to resolve)

I didn't get that at all. But it sounds like you may need to change carriers. Please describe in more detail (and try with another carrier to see if your problem magically disappears).

Well Thank you for the detailed reply, I did tried multiple carriers, different hardware. Even with cluster setup and this version....same issue.

For detail please read my post

http://vicidial.org/VICIDIALforum/viewtopic.php?f=4&t=38772


There is a guy who is using
VERSION: 2.9-441a
BUILD: 140612-1628
© 2014 ViciDial Group

with same carrier, its working fine. I rented his server therefore dont have configs or vici version. But he gave me remote agents

[williamconley]

1) That post talks about DTMF, which you didn't mention in this post.

2) That post never got answered by me because it was answered by YOU. When you made your second post, even though there were no responses to the first post, the phpBB system marked it as "answered" (as opposed to UnAnswered) and those of us who answer posts ... skipped over it assuming someone else had gotten to you. For future reference when on a phpBB system, if nobody has answered your post and you want to add something, just change the post. Then it won't show as answered.

3) Did you DIFF those two results or compare to see what the difference was?

[mubeen]

I'm not blaming you guyz for not answering, you guyz are real help...and I didn't mentioned DTMF because it was not the main point here


The difference I saw was - AGI Script agi-VDAD_ALL_outbound.agi completed, returning 4 in case of Real number and AGI returning 0 in case of VOIP number

[williamconley]

Perhaps the difference between those two was 4 being opt-in vs 0 being opt-out and opt-out is set to buzz and terminate the call? Seems like those would just be two different button presses.

[mubeen]

No, they both are from same button press. Everything is same except the number. One is test number which is a voip number other is a real lead

[williamconley]

Are you saying they both pressed 4 or they both pressed 0? Are you saying that one of the number's DTMF was incorrectly processes? If so, you should share the DTMF settings for them. Both ends if the one that failed was the VOIP number and you control that configuration as well.

[mubeen]

No, both pressed 1. Pasting the CLI below
First one is actual call, second is voip call

Code: Select all

[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] pbx.c: [Oct  1 09:42:05]     -- Executing [91305xxxxxxx@default:1] AGI("Local/91305xxxxxxx@default-0008257c;2", "agi://127.0.0.1:4577/call_log") in new stack
[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] res_agi.c: [Oct  1 09:42:05]     -- AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=HS_IVR_2))
[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] res_agi.c: [Oct  1 09:42:05]     -- <Local/91305xxxxxxx@default-0008257c;2>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] pbx.c: [Oct  1 09:42:05]     -- Executing [91305xxxxxxx@default:2] Dial("Local/91305xxxxxxx@default-0008257c;2", "sip/1305xxxxxxx@Carrier,55,tTor") in new stack
[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] netsock2.c: [Oct  1 09:42:05]   == Using SIP RTP CoS mark 5
[Oct  1 09:42:05] VERBOSE[27262][C-0009b15f] app_dial.c: [Oct  1 09:42:05]     -- Called sip/1305xxxxxxx@Carrier
[Oct  1 09:42:06] VERBOSE[27262][C-0009b15f] app_dial.c: [Oct  1 09:42:06]     -- SIP/Carrier-000699c0 is ringing
[Oct  1 09:42:06] VERBOSE[20030][C-0009b15f] res_rtp_asterisk.c: [Oct  1 09:42:06]        > 0x7f805000ab30 -- Strict RTP learning after remote address set to: 169.132.xxx.xxx:23666
[Oct  1 09:42:06] VERBOSE[27262][C-0009b15f] app_dial.c: [Oct  1 09:42:06]     -- SIP/Carrier-000699c0 is making progress passing it to Local/91305xxxxxxx@default-0008257c;2
[Oct  1 09:42:06] VERBOSE[27262][C-0009b15f] res_rtp_asterisk.c: [Oct  1 09:42:06]        > 0x7f805000ab30 -- Strict RTP switching to RTP remote address 169.132.xxx.xxx:23666 as source
[Oct  1 09:42:08] VERBOSE[27262][C-0009b15f] res_rtp_asterisk.c: [Oct  1 09:42:08]        > 0x7f805000ab30 -- Strict RTP learning complete - Locking on source address 169.132.xxx.xxx:23666
[Oct  1 09:42:16] VERBOSE[27262][C-0009b15f] app_dial.c: [Oct  1 09:42:16]     -- SIP/Carrier-000699c0 answered Local/91305xxxxxxx@default-0008257c;2
[Oct  1 09:42:16] VERBOSE[27261][C-0009b15f] pbx.c: [Oct  1 09:42:16]        > Channel Local/91305xxxxxxx@default-0008257c;1 was answered
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:16]     -- Executing [8366@default:1] Playback("SIP/Carrier-000699c0", "sip-silence") in new stack
[Oct  1 09:42:16] VERBOSE[27262][C-0009b15f] pbx.c: [Oct  1 09:42:16]     -- Executing [h@default:1] AGI("Local/91305xxxxxxx@default-0008257c;2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----11-----0") in new stack
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] file.c: [Oct  1 09:42:16]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (language 'en')
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:16]     -- Executing [8366@default:2] AGI("SIP/Carrier-000699c0", "agi://127.0.0.1:4577/call_log") in new stack
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=HS_IVR_2))
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- <SIP/Carrier-000699c0>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:16]     -- Executing [8366@default:3] AGI("SIP/Carrier-000699c0", "agi-VDAD_ALL_outbound.agi,SURVEYCAMP-----LB") in new stack
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_ALL_outbound.agi
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- AGI Script Executing Application: (Monitor) Options: (wav,/var/spool/asterisk/monitor/MIX/20181001-094216_305xxxxxxx)
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:16] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:16]     -- <SIP/Carrier-000699c0> Playing 'OUT-8khz.slin' (escape_digits=123) (sample_offset 0) (language 'en')
[Oct  1 09:42:17] VERBOSE[27262][C-0009b15f] res_agi.c: [Oct  1 09:42:17]     -- <Local/91305xxxxxxx@default-0008257c;2>AGI Script agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----11-----0 completed, returning 0
[Oct  1 09:42:17] VERBOSE[27262][C-0009b15f] pbx.c: [Oct  1 09:42:17]   == Spawn extension (default, 91305xxxxxxx, 2) exited non-zero on 'Local/91305xxxxxxx@default-0008257c;2'
[Oct  1 09:42:27] DTMF[27504][C-0009b15f] channel.c: DTMF begin '1' received on SIP/Carrier-000699c0
[Oct  1 09:42:27] DTMF[27504][C-0009b15f] channel.c: DTMF begin ignored '1' on SIP/Carrier-000699c0
[Oct  1 09:42:27] DTMF[27504][C-0009b15f] channel.c: DTMF end '1' received on SIP/Carrier-000699c0, duration 180 ms
[Oct  1 09:42:27] DTMF[27504][C-0009b15f] channel.c: DTMF end passthrough '1' on SIP/Carrier-000699c0
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'buzz.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 4
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:33]   == Spawn extension (default, 8366, 3) exited non-zero on 'SIP/Carrier-000699c0'
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:33]     -- Executing [h@default:1] AGI("SIP/Carrier-000699c0", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0---------------") in new stack
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0>AGI Script agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0--------------- completed, returning 0


Code: Select all

[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] pbx.c: [Oct  1 07:54:32]     -- Executing [91909xxxxxxx@default:1] AGI("Local/91909xxxxxxx@default-000821c6;2", "agi://127.0.0.1:4577/call_log") in new stack
[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] res_agi.c: [Oct  1 07:54:32]     -- AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=HS_IVR_2))
[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] res_agi.c: [Oct  1 07:54:32]     -- <Local/91909xxxxxxx@default-000821c6;2>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] pbx.c: [Oct  1 07:54:32]     -- Executing [91909xxxxxxx@default:2] Dial("Local/91909xxxxxxx@default-000821c6;2", "sip/1909xxxxxxx@Carrier,55,tTor") in new stack
[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] netsock2.c: [Oct  1 07:54:32]   == Using SIP RTP CoS mark 5
[Oct  1 07:54:32] VERBOSE[7553][C-0009ad94] app_dial.c: [Oct  1 07:54:32]     -- Called sip/1909xxxxxxx@Carrier
[Oct  1 07:54:36] VERBOSE[20030][C-0009ad94] res_rtp_asterisk.c: [Oct  1 07:54:36]        > 0x7f8030014880 -- Strict RTP learning after remote address set to: 169.132.xxx.xxx:20896
[Oct  1 07:54:36] VERBOSE[7553][C-0009ad94] app_dial.c: [Oct  1 07:54:36]     -- SIP/Carrier-00069619 is making progress passing it to Local/91909xxxxxxx@default-000821c6;2
[Oct  1 07:54:36] VERBOSE[7553][C-0009ad94] res_rtp_asterisk.c: [Oct  1 07:54:36]        > 0x7f8030014880 -- Strict RTP switching to RTP remote address 169.132.xxx.xxx:20896 as source
[Oct  1 07:54:38] VERBOSE[7553][C-0009ad94] res_rtp_asterisk.c: [Oct  1 07:54:38]        > 0x7f8030014880 -- Strict RTP learning complete - Locking on source address 169.132.xxx.xxx:20896
[Oct  1 07:54:38] VERBOSE[7553][C-0009ad94] app_dial.c: [Oct  1 07:54:38]     -- SIP/Carrier-00069619 answered Local/91909xxxxxxx@default-000821c6;2
[Oct  1 07:54:38] VERBOSE[7552][C-0009ad94] pbx.c: [Oct  1 07:54:38]        > Channel Local/91909xxxxxxx@default-000821c6;1 was answered
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:38]     -- Executing [8366@default:1] Playback("SIP/Carrier-00069619", "sip-silence") in new stack
[Oct  1 07:54:38] VERBOSE[7553][C-0009ad94] pbx.c: [Oct  1 07:54:38]     -- Executing [h@default:1] AGI("Local/91909xxxxxxx@default-000821c6;2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----6-----0") in new stack
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] file.c: [Oct  1 07:54:38]     -- <SIP/Carrier-00069619> Playing 'sip-silence.gsm' (language 'en')
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:38]     -- Executing [8366@default:2] AGI("SIP/Carrier-00069619", "agi://127.0.0.1:4577/call_log") in new stack
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:38]     -- AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=HS_IVR_2))
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:38]     -- <SIP/Carrier-00069619>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:38]     -- Executing [8366@default:3] AGI("SIP/Carrier-00069619", "agi-VDAD_ALL_outbound.agi,SURVEYCAMP-----LB") in new stack
[Oct  1 07:54:38] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:38]     -- Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_ALL_outbound.agi
[Oct  1 07:54:39] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:39]     -- AGI Script Executing Application: (Monitor) Options: (wav,/var/spool/asterisk/monitor/MIX/20181001-075439_909xxxxxxx)
[Oct  1 07:54:39] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:39]     -- <SIP/Carrier-00069619> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 07:54:39] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:39]     -- <SIP/Carrier-00069619> Playing 'OUT-8khz.slin' (escape_digits=123) (sample_offset 0) (language 'en')
[Oct  1 07:54:39] VERBOSE[7553][C-0009ad94] res_agi.c: [Oct  1 07:54:39]     -- <Local/91909xxxxxxx@default-000821c6;2>AGI Script agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----6-----0 completed, returning 0
[Oct  1 07:54:39] VERBOSE[7553][C-0009ad94] pbx.c: [Oct  1 07:54:39]   == Spawn extension (default, 91909xxxxxxx, 2) exited non-zero on 'Local/91909xxxxxxx@default-000821c6;2'
[Oct  1 07:54:43] DTMF[7563][C-0009ad94] channel.c: DTMF begin '1' received on SIP/Carrier-00069619
[Oct  1 07:54:43] DTMF[7563][C-0009ad94] channel.c: DTMF begin ignored '1' on SIP/Carrier-00069619
[Oct  1 07:54:44] DTMF[7563][C-0009ad94] channel.c: DTMF end '1' received on SIP/Carrier-00069619, duration 220 ms
[Oct  1 07:54:44] DTMF[7563][C-0009ad94] channel.c: DTMF end passthrough '1' on SIP/Carrier-00069619
[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:44]     -- <SIP/Carrier-00069619>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 0
[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:44]     -- Executing [144*076*003*247*8600051@default:1] Goto("SIP/Carrier-00069619", "default,8600051,1") in new stack
[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:44]     -- Goto (default,8600051,1)
[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:44]     -- Executing [8600051@default:1] MeetMe("SIP/Carrier-00069619", "8600051,F") in new stack
[Oct  1 07:54:51] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:51]   == Spawn extension (default, 8600051, 1) exited non-zero on 'SIP/Carrier-00069619'
[Oct  1 07:54:51] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:51]     -- Executing [h@default:1] AGI("SIP/Carrier-00069619", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0---------------") in new stack
[Oct  1 07:54:51] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:51]     -- <SIP/Carrier-00069619>AGI Script agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0--------------- completed, returning 0

Difference I saw
Actuall Lead

Code: Select all

<SIP/Carrier-000699c0>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 4

VOIP call

Code: Select all

<SIP/Carrier-00069619>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 0

[williamconley]

... If so, you should share the DTMF settings for them. Both ends if the one that failed was the VOIP number and you control that configuration as well.

[mubeen]

William, I'm not sure which DTMF settings are you referring to. I've tried both DTMf, rfc2833 as well as inband.

Survey settings are:
Survey DTMF Digits: 123
Survey Not Interested Digit: 3
Survey Wait Seconds: 10
Survey Method: AGENT_XFER
Survey No-Response Action: OPTIN
Survey Not Interested Status: DNC
Survey Survey Xfer Extension: 8300

[williamconley]

Well, something's obviously odd. If they both pressed 1, but VoIP returned 4 and real phone returned 0 ... I'd say some thorough testing is in order. Plus a bit of research to be sure you're not misinterpreting the results.

[mubeen]

Actually VoIP returned 0 and real phone returned 4.

4 and 0 are agi status not DTMF.

Furthermore after taking DTMF, for VOIP it runs "AGI Script agi-VDAD_ALL_outbound.agi"

Code: Select all

[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] res_agi.c: [Oct  1 07:54:44]     -- <SIP/Carrier-00069619>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 0
[Oct  1 07:54:44] VERBOSE[7563][C-0009ad94] pbx.c: [Oct  1 07:54:44]     -- Executing [144*076*003*247*8600051@default:1] Goto("SIP/Carrier-00069619", "default,8600051,1") in new stack

where as for real call, it plays silence twice and after playing buzz call is dropped

Code: Select all

[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0> Playing 'buzz.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] res_agi.c: [Oct  1 09:42:33]     -- <SIP/Carrier-000699c0>AGI Script agi-VDAD_ALL_outbound.agi completed, returning 4
[Oct  1 09:42:33] VERBOSE[27504][C-0009b15f] pbx.c: [Oct  1 09:42:33]   == Spawn extension (default, 8366, 3) exited non-zero on 'SIP/Carrier-000699c0'

I did confirmed form couple of Admins running press 1 campaign, they also told me it just dosent work. They have manually edited the asterisk file but reluctant to share exact changes they made.

[williamconley]

Is the voip phone on the same system or some other system?

[mubeen]

Its on different server but same location. All the servers are hosted and are physical servers.

I'm running Asterisk 11.25.1 on Centos 7 for that.

[williamconley]

Try using a Call menu with it to verify DTMF. First try on the same server it's on, then try on the newly built server with an inter-server phone call. Perhaps the DTMF is getting "lost" or mangled during transit.

What do you have for Asterisk version in all the places that Asterisk version is stated?

[mubeen]

I dont think that is the case, I can see the DTMF on CLI that which number the client have pressed (have posted above). I can also see the DTMF in reports.

And I'm using call menus for other campaigns, its working fine for inbound calls as well as outbound with PD

[williamconley]

I never think. I always test and prove it. You should consider that model when troubleshooting.

And I'm using call menus for other campaigns, its working fine for inbound calls as well as outbound with PD

For instance: I don't know that your "successful call menus and campaigns" have ever had this VOIP phone on them. Thus, proving that this phone works for all of the above would be where I was going.

[mubeen]

As I mentioned I am using call menus for other campaigns and I did tested it for both VOIP and real numbers on same server.

I believe you are referring to send the call to call menu instead of AGENT_XFER. Let me try this scenario as well as transferring to extension directly

[williamconley]

call menus for other campaigns and I did tested it for both VOIP and real numbers on same server

So on the same server, voip phones and real numbers both work for call menus and campaigns, but NOT for Survey? And Survey only fails for VOIP phones, but succeeds for real numbers?

And the only notable difference is that for some reason VOIP returns a different code in the agi script than real phones? interesting. Have you tried this at more than one carrier?

[mubeen]

Yes on same server, voip phones and real numbers both work for call menus and campaigns but not for survey. And Survey fails for real number, not VOIP. If it was other way around like you mentioned, honestly I wouldn't be bothering this much. And yes I tried it multiple times, with single server configuration, cluster configuration, different carriers, both inband and RFC DTMF modes.

I dialed like 10,000+ numbers of which I got response form couple of hundreds. I can see which DTMF they have pressed. While testing I grep DTMF live to see the results. As soon as some presses DTMF I go throuh that whole call. I even took tcp dump and tried to find issue upto best of my abilities but invain. I tested it for multiple different VOIP numbers (well on same server) and dialed those VOIP numbers literally 100+ times and 95% calls were successfully transferred to agent. but wen I add real numbers to same list, call drops

[williamconley]

If I were you, I'd upgrade before going any farther. Just in case someone has already found and squished this bug.

[mubeen]

Okay, I'll install vicibox 8.1.4 and test it there. Will let you know after testing

[mubeen]

On further investigation, when I exported the report I found the status of all calls on which 1 was pressed was "Agent not available"

[williamconley]

On further investigation, when I exported the report I found the status of all calls on which 1 was pressed was "Agent not available"

And it was a coincidence that this only manifested itself as problematic with nonVOIP calls? Odd.

[mubeen]

I know how strange it looks but thats what I found. Have installed the latest version, will be testing on it today...

[mubeen]

I was interpreting the output little wrong, actually the agi is not being run for the call which is being dropped. Agi is run after playing silence twice and buzz.

[williamconley]

Agi runs as the result of interpretation of the DTMF. The DTMF "wait" occurs in an AGI. So there is already an AGI running. When DTMF is received, the DTMF causes code to execute. NON response also causes AGI code to execute. But the AGI code has to complete and eventually terminate.

[mubeen]

Sorry for late reply, I downloaded extended report and found:

status: DROP
term_reason: QUEUETIMEOUT
status_name: Agent Not Available
queue_time: 0
hangup_cause: 16
user_group: (Blank)

Whereas I have 3 agents logged in and waiting for calls. Not sure what I'm doing wrong here

[williamconley]

Agent Not Available

3 agents logged in and waiting for calls

Connect those dots: Prove that the logged in agents are in the same Ingroup as these calls.

[mubeen]

How can I check this in logs? That agents are logged in in same Ingroup as these calls? I mean I can see the status in real time but not sure about logs

[williamconley]

How can I check this in logs? That agents are logged in in same Ingroup as these calls? I mean I can see the status in real time but not sure about logs

Attempting to reconstruct the ingroups to which an agent is logged in can be time-consuming. Try it on the next one or on a staged test. But the vicidial agent logs do show this information. It's just difficult to be certain you're looking at the right log for the right agent at the right time if you've never done it before. Live is much simpler.