ViciDial Docker Container

[callforamerica]

HI,

I've always been scared to death to try to experiment with "virtualizing" vicidial but I have spent the last year working with Docker and I think it could be a good match. Does anyone have any experience with running vicidial within a Docker container? Any specific reasons why I should avoid doing this all together, or any advice on what problems to look out for and thoroughly test?

My everyday vicidial deployments waste a lot of hardware resources as its all running on bare metal and I find adding cheap asterisk boxes to be a economical but a little daunting.

Any input or advice would be much appreciated.

[bobbymc]

i have also played with the idea of doing so, i agree wasting hardware sucks because the limitation is often how many calls asterisk can handle and its thread limitations that cause deadlocks, i have gotten vicidial working in a docker instance but has to pause the project due to asterisk 11 instabilities. i wouldnt mind collaborating with you if your interested PM me.

cheers

[williamconley]

1) Keep mysql on hardware boxes, this may help with overall stability and reduce what the virtualized system must handle. And remember that mysql response time is crucial, that hardware box should be very very low latency to the virtualized Vici.

2) If you attack this, attack the perl scripting which uses hi-res timing. Challenge: these perl scripts need to run on the same server that the agent phones are on (ie: dialer!) since these are the processes that automate Vicidial. Concept: Agents logging in to a campaign is what "fires up" Vicidial. No managerial involvement is necessary. This is because there are perl scripts running on the dialers to manage the system which are activated by the presence of agents (ok: always active, but when an agent is logged in on their dialer they "do things" necessary to keep those agents busy ...). Since these scripts use hi-res timing, they are NOT OK to virtualize. Removing Hi-Res timing OR overloading the script (Ie: virtualization can not handle the load) results in instability. Pretty quickly in fact. So these scripts are where the battle will be. IMHO

3) Better to have an Image that can be placed directly onto leased hardware in AWS (you CAN get an entire server!) which would allow "bringing up" as many dialers as you need, but not paying for them during downtime. You'd have to adjust IP addresses (since they change) during startup with your carriers. And if you can: best to allow them to share an internal network. Note, we've never had anyone ask us to try this.

4) We HAVE watched LOTS of very professional virtualizers and vicidialers TRY and fail. Several have claimed success, but in the end that success has never been proven (or been hidden to avoid sharing? LOL). And by success I mean 25 agents on a standalone Virtual Vicidial while utilizing a single Virtual Instance without massive Waste (if private cloud). It's not a success if you can only run ONE instance of Vicidial on a virtual host, unless you can run more than one server simultaneously it's a waste of resources. And more than 15 agents is required, too ... although I'd be very happy if someone could build three reliable "5 agent outbound autodial at 2.5:1" vicidials in a single virtual host. That would be impressive. We've priced out what we think would work and found that even in AWS, to get enough power to run a Vicidial cluster 5 days a week would cost as much as hosting at Vicihost.com! LOL (Or, of course at PoundTeam!) But you then do not get the expertise of the hosting facility, you had to do that all yourself instead. A lot of work to not save $$. And we were not even sure that the systems we were looking at would have been capable of being clustered and running Vicidial. No point in testing if it costs just as much as hosting in our facility in real hardware, of course. So we chuckled and dropped the project.

[bobbymc]

1) agree %100 with number 1, althought with ssds so cheap u can still make it into a docker instance without any issues

2) hi res timing will still work in docker as its a LXC and those share the system timer hence why you can still use dahdi for timing, to be honest the hi res timer is perl is not as big of a issue as dahdi since dahdi is used for meetme and iax2 trunks. the perl scripts technically dont need to be on the same server, i tried putting them elsewhere and had no issues whats so ever. i had a nice perl far going on for a while and separating them allowed asterisk to push more calls

3) can be done with something like kubernetes and a few api calls to a custom script

4) you are right, its been hidden because they dont want to help competition, i know for a fact it can be "virtualized" (if u call docker a vm) and it works, i've done it myself but the issue is always timing, with docker that issue is solved, i have done crazy benchmarks and docker has no timing issues because its not a vm, its a linux container very similar to chroot


hope this shines some light on this issue

[mflorell]

One of our clients used OpenVZ about 7 years ago to do "containerized" vicidial, it's not anything new. It is more complex to set up and manage, and it's not anything that we spent a great deal of time on because of the limitations in how it works. A true "virtual" VM with Vicidial will always have timing issues running at anything other than very light loads.

[williamconley]

2) hi res timing will still work in docker as its a LXC and those share the system timer hence why you can still use dahdi for timing, to be honest the hi res timer is perl is not as big of a issue as dahdi since dahdi is used for meetme and iax2 trunks. the perl scripts technically dont need to be on the same server, i tried putting them elsewhere and had no issues whats so ever. i had a nice perl far going on for a while and separating them allowed asterisk to push more calls
4) you are right, its been hidden because they dont want to help competition, i know for a fact it can be "virtualized" (if u call docker a vm) and it works, i've done it myself but the issue is always timing, with docker that issue is solved, i have done crazy benchmarks and docker has no timing issues because its not a vm, its a linux container very similar to chroot

Timing in DAHDI and perl script timing are not the same thing. My Theory: DAHDI integrates directly to the kernel (fact) and perl hi-res timing is not (theory). Thus perl becomes unstable. I also have doubts about mysql remaining stable under these circumstances (thus the suggestion of an external, hardware-based, mysql).

Our research into complex, higher-end virtualization found that the cost savings was ... pretty much nonexistent. Why would someone expend the manpower necessary to use such technology when (in the end), the Amazon or other cloud stockholders end up making money while the end-user still gets NO Vicidial Assistance from the host and no discount? Not exactly a win-win situation unless you're an owner of the cloud container.

In which case, if the cloud people created the container (gathering the necessary Vicidial Expertise along the way) and offered it to the end users with no need for effort on the part of the user except "pay us!", they would be just like OUR COLO (and Vicihost, of course). Argument over who costs more and supports Vicidial could be had at that point. But for now, we have not seen any advantages to this approach (or of course we could retire our hardware pool and move it all to their cloud ...!). So far it just seems to be something that overzealous IT personnel "decide" is a good idea and blow a lot of resources on before giving up. LOL

[bobbymc]

2) hi res timing will still work in docker as its a LXC and those share the system timer hence why you can still use dahdi for timing, to be honest the hi res timer is perl is not as big of a issue as dahdi since dahdi is used for meetme and iax2 trunks. the perl scripts technically dont need to be on the same server, i tried putting them elsewhere and had no issues whats so ever. i had a nice perl far going on for a while and separating them allowed asterisk to push more calls
4) you are right, its been hidden because they dont want to help competition, i know for a fact it can be "virtualized" (if u call docker a vm) and it works, i've done it myself but the issue is always timing, with docker that issue is solved, i have done crazy benchmarks and docker has no timing issues because its not a vm, its a linux container very similar to chroot

Timing in DAHDI and perl script timing are not the same thing. My Theory: DAHDI integrates directly to the kernel (fact) and perl hi-res timing is not (theory). Thus perl becomes unstable. I also have doubts about mysql remaining stable under these circumstances (thus the suggestion of an external, hardware-based, mysql).

Our research into complex, higher-end virtualization found that the cost savings was ... pretty much nonexistent. Why would someone expend the manpower necessary to use such technology when (in the end), the Amazon or other cloud stockholders end up making money while the end-user still gets NO Vicidial Assistance from the host and no discount? Not exactly a win-win situation unless you're an owner of the cloud container.

In which case, if the cloud people created the container (gathering the necessary Vicidial Expertise along the way) and offered it to the end users with no need for effort on the part of the user except "pay us!", they would be just like OUR COLO (and Vicihost, of course). Argument over who costs more and supports Vicidial could be had at that point. But for now, we have not seen any advantages to this approach (or of course we could retire our hardware pool and move it all to their cloud ...!). So far it just seems to be something that overzealous IT personnel "decide" is a good idea and blow a lot of resources on before giving up. LOL

Both perl and dahdi depend on a realtime system bus timing source or some type of timing card, they are not the same but their requirements are

And yes this is worth doing if you manage a hosted platform as a service like me and others do. For example I have a 64 core box and asterisks threading limitations don't allow me to go past load 10 which is about 15 % utilization of the box. The idea is to use around 50 percent and not have to manage a huge farm of servers that are low end CPUs. After all managing thousand of servers vs a hundred is a big difference. My 2 cents

[williamconley]

dahdi uses a kernel module. perl does not. neither uses a card (but dahdi does use a dummy module that simulates a card's existence).

and if you are talking about 1000 servers, of course "the cloud" is the place to be. but when you're talking about ONE (or up to 10), these "low end servers" are the best price breakpoint, and mid-range blades are the next great breakpoint.

but I only have 10 years in this application area. and we only have a few hundred servers online. nice part: If ONE server goes offline (for any reason) it has Zero effect on any other server unless it's the DB for a cluster (and even then, that only takes down ONE cluster). a single 64-core system creates on heck of a failing point. and it was not cheap to pick up. i can definitely see where (after making that 64-core server purchase, without the ability to return it), you may look at at trying to utilize the other 90% of the server. But I'm not going to recommend anyone else go down that tunnel.

on the other hand, if you make it possible to run 10 servers in that $10k machine (each capable of 25 users with standard usage), that would compare nicely to our $1k servers each capable of the same thing. and IF you could actually break it into 20 virtual servers instead of 10, with 10 users each ... THAT would be a marvelous thing (and an improvement over 20 $1k server!). Just haven't seen anyone make that happen yet. Lemme know if you make that happen, I'd love to see it.

[bobbymc]

dahdi is a kernel module correct, but it uses the servers timing source much like the clock does for exmaple, the issue is that a vm does not have access to the servers hardware clock in realtime which at times causes the servers timing source to fall behind which in return causes dahdi not to have a proper time ticking which then in return causes choppy calls in meetme because it cant sync the audio properly, the same issue effects perls hi res timer. but i may be wrong so feel free to correct me, this is after doing tons of tests and looking at the kernel source code and dahdi to understand why it was failing in a vm. this is also why openvz works because its not a real vm, its a container.

well the idea with the 64 core serve is the following.

you set a few of them up strictly for outbound calling in a load balanced setup. and say you have 5 clusters, and these 5 clusters share these 64core boxes and each have a set of call servers on them. the idea would be to create in a way virtual blades on these 64 core boxes and have enough that incase 1 goes down it doesn't effects a cluster, basically clusters would share these boxes between them selfs, thats my idea or goal.

[williamconley]

dahdi is not a problem virtually. freepbx runs just fine virtually. including meetme rooms. and lots of instances. all over the cloud.

we've not had choppy calls on virtual servers with vicidial, perl breaks down first. of course i would expect dahdi/meetme to break down eventually, but our experience has been that the dialer is unstable long before sound breaks up.

the rest is theory because no one has it running. and considering the lack of cost savings ... that's not a surprise.

[bobbymc]

That is very strange because I'm having the complete opposite issue, I use citrix xenserver for the hypervisor, what do you guys use? Also can you be more specific when you say perl breaks down? What exactly breaks and what are the symptoms?

[williamconley]

when vicidial becomes unstable, the "wrong things" happen. agents get extra calls in their meetme rooms, other functions stall. we've received similar reports from others who have tried it. the only ones who report breaking audio are those using FreePBX when they overload those systems (in which case, moving the server to a "larger instance" usually resolves the issue, as every server has a maximum capacity).

these, by the way, are the same symptoms we observe when a hardware-based vicidial system is overloaded. the scripts that are "vicidial" (such as those running in Screens) begin to make the wrong decisions, or no decisions. system load goes crazy (average server load of 38.5 on a server with 8 processors ..,. not good, LOL)

we've only bothered with vSphere at this point. we've had similar reports from others with different virtualization technologies involved. often, the challenge is that each approach is different and it's hard to get a real response about what's virtualized and how. which processes are running in what type of server. Thus the earlier comments about the complexity of things like "docker containers" and "the cloud" in general. I'm sure it can be done. No doubt there.

But the challenge is to do it in a way that's worth doing. i mean, one COULD rewrite Vicidial completely as a group of cloud apps. but then it would not be Vicidial, and someone likely already has. maybe they named it Lynx or something. LOL