Channels not getting hung-up automatically for days

[rustynail]

Vicibox: 7, Vicidial: 2.14b0.5, Asterisk: Asterisk 11.25.1-vici, OpenSuse Leap 42.1, 10.1.20-MariaDB, Perl: v5.18.2, PHP 5.5.14

Hi,

I am new to vicidial and using it in call center environment since 1.5 - 2 months now. Almost everything was going smooth while on a newly created cluster install I noticed the channels once created are not being hungup and there for they keep accumulating there until TRUNK SHORT/FILL is reached and calls suffer badly.

Can someone please help to resolve this issue step by step. Also could it be due to the version difference as shown in attached image?



EDIT: List of channels which are stuck for more than 20 hours even, can be viewed here https://pastebin.com/E9yCQ7AB

Any help would be greatly appreciated.

Cheers!

[rustynail]

Come on guys... 5 days and no single reply

[mflorell]

That used to be one of the reasons we usually do reboots every night. Although, I haven't seen this happen on any of our newer clusters that only do weekly reboots, so I can't say why this is happening.

[rustynail]

Thanks for the much awaited reply. Yes we have rebooted cluster servers many times but with same result. We used to first reboot DB server and once its up, we reboot each TEL and WEB servers then. So my assumption of SVN version difference doesn't seem to be the cause of this?

This doesn't seem to interfere with billing as I can see new channel with same sip extension working fine on main reporting page. Only these channels show stuck on asterisk CLI. Any pointers to any direction which might lead to a resolution, any specific logs entries to check for?

Any help will be much appreciated.

[mflorell]

We usually don't recommend rebooting database servers unless you have to. They can usually run for years without reboots, even on very busy systems.

[rustynail]

I don't have much clue what might be causing this channel stuck issue. Initially I thought it could be due to chanspy but I thin its not the reason. I am not in mood to update svn as its a production server and slight mistake can cause losing client.
Any guide in right direction will be highly helpful...
Thanks

[williamconley]

Come on guys... 5 days and no single reply

Never reply to your own post. It makes your post disappear from the "unanswered" query and much more likely to get ignored (because everyone assumes someone else answered it .. we don't know it was *you* and you didn't help yourself very much! lol). You got lucky, though: Matt was in a good mood!

I *strongly* suspect you have configured your system in a way that causes this failure or you are overloading the system.

Have you modified any .conf files manually?

What is the server load when these begin to pile up?

Have you verified that these are really your calls, and not an attack of some sort?

Is your server's firewall whitelisted?

[rustynail]

Hi William,
Sure I have noted your advice for future and I hope you and matt remain in good mood always - lol ...

I have 3 other clusters running and all are configured in same was and running fine (except this one has svn difference in one machine as pointed). I have checked thoroughly but was unable to find what I have miss configured... any clue to the direction which I need to follow might light the bulb

Manually configured file is only astguiclient.conf which I do modify on each server so as to add details of ftp server for moving recordings to backup server.

It shouldn't be an attack as I have already checked it and also we have only allowed the needed IP addresses through firewall and default policy is DROP.

Really appreciate your efforts and time in answering our stupid queries ... we are like children and won't mind your scold but will definitely learn something from it too..

Cheers.

[williamconley]

Have you verified that these are really your calls, and not an attack of some sort?

Is your server's firewall whitelisted?

Both answered:

Manually configured file is only astguiclient.conf which I do modify on each server so as to add details of ftp server for moving recordings to backup server.

It shouldn't be an attack as I have already checked it and also we have only allowed the needed IP addresses through firewall and default policy is DROP.

What is the server load when these begin to pile up?

Not answered ...?

Average Server Load when the problem begins for all servers in cluster. Does this happen when one/all are more heavily loaded than other times?

MYSQL processes: Are there stuck or slow queries running when this problem seems to begin? (mysql backlog can create cascade failures in many systems)

[rustynail]

Thanks William for the reply...
Server loads remain just fine all the time. Its seems hangup detection is failing for some channels. I searched around some more and found that the revision of one cluster server is actually more recent than others, could this be the issue ?
ADMIN --> SYSTEM SETTINGS shows

DB Schema Version: 1506 WARNING: Code expects different schema: 1484

I can reinstall that server completely if that can fix the issue as its production cluster and I don't want to update DB or such thing. I can distribute agents to other TEL servers and reinstall it fresh.
Also please guide how to avoid svn change while installation ? should I not run zypper update -y script ?

Please find attached image for reference.




Thanks.