1 Million Hourly Calls

[chetan mittal]

I need to make 1 million survey calls per hour, where each call plays a 5-second audio and disconnects once the audio finishes. I plan to run 10 campaigns for this.

Here are my questions:

How many servers should I deploy to handle this load efficiently?
What should be the optimal Max Trunks and Max Calls per Second (CPS) settings per server?
What should be the recommended Max FILL Calls per Second?
How many remote agents would be ideal per campaign?
What should be the ideal dial ratio for these campaigns?

I really appreciate the your valuable assistance in helping me plan this!

[carpenox]

First off, depending on where you are calling, if its the US, that is illegal, second, you would need around 100 servers to make that many calls, third, why are you playing an audio sound for 5 seconds? Seems like you are up to something extremely shady and more than likely a scammer. Prove me wrong.

[chetan mittal]

First off, depending on where you are calling, if its the US, that is illegal, second, you would need around 100 servers to make that many calls, third, why are you playing an audio sound for 5 seconds? Seems like you are up to something extremely shady and more than likely a scammer. Prove me wrong.

This in not to call in USA, calling will be in India. It is not any thing like a scam. We need to play multiple recording as per campaign for different offers. Audio length will be between 5 to 15.

[carpenox]

ok fair. sorry for the suspicion but i do not like to help scammers ruin the industry more than it already has been. So yea, you will need around 100 bare metal servers for that call volume

[williamconley]

I need to make 1 million survey calls per hour, where each call plays a 5-second audio and disconnects once the audio finishes. I plan to run 10 campaigns for this.

Here are my questions:

How many servers should I deploy to handle this load efficiently?
What should be the optimal Max Trunks and Max Calls per Second (CPS) settings per server?
What should be the recommended Max FILL Calls per Second?
How many remote agents would be ideal per campaign?
What should be the ideal dial ratio for these campaigns?

I really appreciate the your valuable assistance in helping me plan this!

First of all, it's not necessary to be cryptic here and it feels like you've left off some information. I'll guess at the pieces I view as missing, please tell me if I'm close and/or helpful (LOL):

Guess: You are generating as many calls as humanly possible to get a "press one if interested" response either for the purpose of transferring calls to a live call center or selling "warm" leads to a call center for further processing of the actual surveys.

Your goal would then be to use the equipment you have at hand to make an estimated number of "press ones" which would result in you getting a specific amount of money, and your calculations lead you to "1 million calls per hour" being profitable.

1 million per hour: 16.66k calls per minute: 277.77 calls per second.

If each call lasted an average of 10 seconds, you'd have 2777 calls "live" at any given moment in time IF all your calls are anwered. Plus of course, these numbers in no way reflect time between dials, ring time, No Answer, etc, just a rough shot in the dark number to work with.

We have, in the deep dark past, built special servers that could handle 600 calls (per server). With those old Gentoo Vicidial builds, you'd have needed at least 7 servers.
* ONE DB server (massive, powerful, lots of processors/cores and fast RAID10 HDs).
* SIX Dialers (Not as massive, no need for RAID).
* Optionally a dedicated web server (if you actually have humans involved in the process anywhere and don't want to load the DB server)
* Optionally a replication server (if you want to run reports while the system is live, but this may affect the main DB as replication is not "free" of load)
* Optionally an archive server to offload recordings (however: if you are making recordings, you'll also need extra servers as each recording is technically also a "call" and that could easily add a few hundred calls to the system!)

Today's Vicidial has a lot more overhead. Today's VICIBOX installer puts a lot more bloat into the OS to support all that overhead. Today's servers, as a result, do not handle as many live calls. If you're lucky, you may get 300 live calls per server unless you dedicate some serious hardware.

That being said: Your best bet would be to build the massive DB server and one serious (perhaps massive) dialer and load it up to see where it hits the wall. Then drop it back down a notch or two.

Be sure to have YOUR CELL/Landline in the system every couple thousand calls to be sure call quality is viable as you "load it up". Paying particular attention to "press one succeeds" if that is a requirement, obviously.

[chetan mittal]

ok fair. sorry for the suspicion but i do not like to help scammers ruin the industry more than it already has been. So yea, you will need around 100 bare metal servers for that call volume

No problem carpenox. I understand your concern. We are also suffering from scammers. They have ruin our reputation in international market.

[chetan mittal]

I need to make 1 million survey calls per hour, where each call plays a 5-second audio and disconnects once the audio finishes. I plan to run 10 campaigns for this.

Here are my questions:

How many servers should I deploy to handle this load efficiently?
What should be the optimal Max Trunks and Max Calls per Second (CPS) settings per server?
What should be the recommended Max FILL Calls per Second?
How many remote agents would be ideal per campaign?
What should be the ideal dial ratio for these campaigns?

I really appreciate the your valuable assistance in helping me plan this!

First of all, it's not necessary to be cryptic here and it feels like you've left off some information. I'll guess at the pieces I view as missing, please tell me if I'm close and/or helpful (LOL):

Guess: You are generating as many calls as humanly possible to get a "press one if interested" response either for the purpose of transferring calls to a live call center or selling "warm" leads to a call center for further processing of the actual surveys.

Your goal would then be to use the equipment you have at hand to make an estimated number of "press ones" which would result in you getting a specific amount of money, and your calculations lead you to "1 million calls per hour" being profitable.

1 million per hour: 16.66k calls per minute: 277.77 calls per second.

If each call lasted an average of 10 seconds, you'd have 2777 calls "live" at any given moment in time IF all your calls are anwered. Plus of course, these numbers in no way reflect time between dials, ring time, No Answer, etc, just a rough shot in the dark number to work with.

We have, in the deep dark past, built special servers that could handle 600 calls (per server). With those old Gentoo Vicidial builds, you'd have needed at least 7 servers.
* ONE DB server (massive, powerful, lots of processors/cores and fast RAID10 HDs).
* SIX Dialers (Not as massive, no need for RAID).
* Optionally a dedicated web server (if you actually have humans involved in the process anywhere and don't want to load the DB server)
* Optionally a replication server (if you want to run reports while the system is live, but this may affect the main DB as replication is not "free" of load)
* Optionally an archive server to offload recordings (however: if you are making recordings, you'll also need extra servers as each recording is technically also a "call" and that could easily add a few hundred calls to the system!)

Today's Vicidial has a lot more overhead. Today's VICIBOX installer puts a lot more bloat into the OS to support all that overhead. Today's servers, as a result, do not handle as many live calls. If you're lucky, you may get 300 live calls per server unless you dedicate some serious hardware.

That being said: Your best bet would be to build the massive DB server and one serious (perhaps massive) dialer and load it up to see where it hits the wall. Then drop it back down a notch or two.

Be sure to have YOUR CELL/Landline in the system every couple thousand calls to be sure call quality is viable as you "load it up". Paying particular attention to "press one succeeds" if that is a requirement, obviously.

Thanks williamconley,

you have guess correctly and very helpful as always. Let me explain what have we done till now.

We are currently achieving a call volume of 170,000 per hour using 3 servers with 5-second audio clips. Our setup consists of 1 database server with 20 cores and 128 GB RAM, and 2 Asterisk servers, each with 4 cores and 16 GB RAM. There are no agent logins involved in this configuration. We are planning to add 8 more Asterisk servers with the same specifications.

The "press one" response will collected and handled by a separate, dedicated Vicidial cluster.

We have 2 campaigns with 200 remote agents, operating at a 25:1 ratio, which means 100 remote agents per campaign. However, when we try to increase the ratio, the call volume drops. We have a total trunk capacity of 20,000 and a CPS of 2,400. After checking the ISP CDR, we have confirmed that we never reach the CPS limit.

My question is: how should we distribute the load across the Asterisk servers? Should we assign 100 remote agents to each server with a 25:1 ratio? What would be the best practice for optimizing this setup?

[williamconley]

However, when we try to increase the ratio, the call volume drops.

How much did you increase the ratio?

how should we distribute the load across the Asterisk servers?

First: Turn off Balance Dialing, it tends to slow things down.

Should we assign 100 remote agents to each server with a 25:1 ratio?

Total calls per server is the general bottleneck. How you arrive at that maximum limit isn't (generally) the issue. (EG: 100 agents at 10:1 vs 10 agents at 100:1 should result in the same number of Live Calls Ringing)

What would be the best practice for optimizing this setup?

Bring it up very slowly to find your maximum "live calls/trunks". The maximum would be "call quality is suffering" or "press one fails" or ... the system refuses to go higher.

But if the system refuses to go higher, you will probably want to find out WHY. That would be in the system logs. Note that you may not technically find an error, but instead you just may find that the system has determined a value of "new calls to make" lower than you want, by design. (ie: you misunderstood something ...). Happens a lot, actually.

The live perl screens spew the decisionmaking processes continually, generally into log files to capture the logic for troubleshooting purposes.

Start reading them and see if you can decipher the "total new calls", how that number was reached, and ultimately why you disagree with that number.

Of course, if you can't find/interpret those logs ... ask.

[chetan mittal]

Just to update we are able to achieve 1 Million Hourly calls with 3 cluster with 1 DB/Webserver, 4 Asterisk servers. Average call duration 10 seconds.

DB: 20 cores and 128 GB RAM Raid 1 SATA SSD.
Asterisk: 4 cores and 16 GB RAM single SSD

[carpenox]

nope, theres no way you can reach 1 million calls per hour with a 10 second duration with 4 asterisk servers. It would take 14 asterisk servers to hit that level.

[williamconley]

Just to update we are able to achieve 1 Million Hourly calls with 3 cluster with 1 DB/Webserver, 4 Asterisk servers. Average call duration 10 seconds.

DB: 20 cores and 128 GB RAM Raid 1 SATA SSD.
Asterisk: 4 cores and 16 GB RAM single SSD

with 3 cluster

Do you mean to say you had three clusters? Thus a total of 12 asterisk servers?

[carpenox]

oh good catch Bill, I missed that, 12 servers would possibly do it you are not dropping voicemails, making recordings or using AMD, but id definitely add another server to each cluster to relieve stress