vicidial.org

So we have done some testing to try and work through a number of the deadlocks that have been popping up in Asterisk on higher loaded Vicidial systems. We did this by compiling Asterisk with all debugging on and no optimizations and ran vicidial in performance testing mode with 50 agents at 20 to 1 ratio in a virtualized xen machine.

What we have found is that app_meetme is not well written at all and its no wonder that app_conference does so much better. They lock the linked list that holds the configurations for all of the conferences just about every time they are playing a sound file. This means that when a person enters a conference all conferences have their configs locked while it plays the entering sound file. Same on exit. They also lock all of the configs when one conference has some setting changed. On top of that they lock the list while outputting to the log file or when sending a manager event. All of this works out in the end because there is only one lock so when one thread is done using it the next guy gets it, but while it is working its way out more stuff is piling up. The problem gets worse as there are more conferences added to the list.

We are still looking into using Callweaver instead of Asterisk but if that does not pan out we will be looking into improving the locking in app_meetme or fixing the deadlocks in app_conference. We have already done some minor improvements to app_meetme to reduce the amount of time things are locked and removed some possible deadlocks as well (they were locking then calling a function that tries to lock on the same lock) but this was just a drop in the bucket. If we continue with this we plan on making a lock for each of the conferences as well as the linked list and only locking the list on insertion, deletion, and traversal.

I would very much welcome a streamlined conferencing engine to use for VICIDIAL.

I have played around a lot with app_conference and app_meetme and there are problems with both, but I have settled on app_meetme because it is more stable.

I even tried to strip down app_meetme to only what VICIDIAL needed, but it was way too complex of a task for the time I had available.

I have run the load test with 100 fake agents on a high powered machine and did not have many of the deadlock issues that you seem to be experiencing. Were these just the WARNING deadlocks or an actual crash of Asterisk and/or Linux?

These were just warnings. There really is no way for a deadlock in meetme except if it calls one particular function that I do not think would be called during regular Vicidial operation. The reason they cannot deadlock is that there is only one lock so it cannot get into a point where one thread has one lock and another thread has another and both want the others lock as well. It does develop however long lines waiting for the lock which can cause load increases and choppy audio in conferences when it is trying to do something.

The shear fact that it can only play the entering and leaving sounds to one conference at a time I think is a serious problem when you have close to 200 conferences going.

Not sure if you noticed or not, but the enter and leave sounds are actually CODE, not audio files. Do you think there would be an easy way to shorten those sounds, if anything it should help performance at least a little bit. The enter/leave sounds I used for app_conference were shorter and a bit less noticable.

Yes I did notice. It should just be a matter of recording the file in the proper format (were thinking it is slin) and then opening it in a hex editor and copy and pasting the hex. Or it should be not too difficult to programmatically create it either.