Vicibox 7.0.3 requiring vicidial to be restarted

[alo]

Hello Friends. Happy Holidays.

We have a couple new installs of vicibox 7.0.3 running and seem to intermittently need to run 'systemctl restart vicidial' because the agents get a time sync error and the server section on the reports page goes red. I believe asterisk or dahdi is crashing. this happens intermittently. sometimes once a day sometimes more.

We are not experiencing this with vicibox 7.0.1 installs.

Anyone experiencing the same on vicibox 7.0.3?
kernel version 4.1.20-11-default
VERSION: 2.12-565a
BUILD: 160827-0917
Version: 2.12b0.5
SVN Version: 2585
DB Schema Version: 1467
Asterisk version 11.22.0-vici
dahdi version 11.22.0
Dahdi linux version 2.11.1
Dell R610 Raid 1 with a perc 700 raid card and SSD drives
13 agents logged in with dial level 6. Single DB/Web/Asterisk server

Same set ups are running fine when installed using vicibox 7.0.1.
Ex of one running fine is:
Same hardware
kernel version: 4.1.15-8-default
dahdi version 11.21.0
Dahdi linux version 2.11.0
Version: 2.12b0.5
SVN Version: 2488
DB Schema Version: 1455
11.21.0-vici

So the two things I notice different is the kernel version and the asterisk version and the dahdi version.

So I guess I have two questions. Is anyone else having trouble with vicibox 7.0.3 and anyone know if it would be safe for me to try to update kernel or dahdi version?

[mflorell]

Check the crontab, most vicidial installs do reboot every night.

[alo]

These servers are already restarted nightly.

Again. our vicibox 7.0.1 servers do not have this trouble. the vicibox 7.0.3 do. in addition it happens randomly throughout the day.

Is anyone else having trouble with vicibox 7.0.3 and anyone know if it would be safe for me to try to update kernel or dahdi version?

[josecapurro]

I suspect this is a time synchronization problem, because i got the same situation with some ViciDial clusters, being 7.0.2 and 7.0.3 the version of the installers.

Log in via SSH with root credentials and add an entry in root's crontab (with crontab -e):

*/5 * * * * /usr/sbin/ntpdate 0.centos.pool.ntp.org

Of course, you can change 0.centos.pool.ntp.org with the server of your choice.

After that, get sure ntpd is stopped and disabled:
systemctl stop ntpd
systemctl disable ntpd

This is because if ntpd is running, ntpdate will fail to bind the socket (because ntpd is using it) and it will not synchronize the time.

[alo]

Wow! Thank you Thank you!!

I have been suspecting something with NTP. I have done as you instructed and will let us all know how it works out.

Happy Holidays

[alo]

This did not resolve our problem unfortunately

Any other suggestions?

[kashinc]

I am having the same exact issue, servers go red and out of sync and then go back to normal.. This just started happening about two weeks ago.

any luck?

[alo]

I think this might be the same as viewtopic.php?f=4&t=36648

Once we downgraded to asterisk version 1.8 it stopped happening and has been working well.

[ambiorixg12]

It seems there is an issue with Vicibox 7.03, I just downgrade to Vicibox_v.6.0.x86_64-6.0.4.iso / Asterisk 1.8 and all is working fine

[mflorell]

As an update, we have an idea this may have to do with SIP brute-force attacks that aren't showing up in the Asterisk 11 CLI. downgrading to Asteirsk 1.8 on vicibox 7.0.3 seems to fix the issue, although we are working on it and gathering more information every time we run into this happening.

[ambiorixg12]

Thank you for keeping us updated, but I think steps described to downgrade asterisk to version 1.8 are more complicated than just install a vicidial version who already has Asterisk 1.8 included by default, like Vicibox 6.X,So I think we should avoid version 7.0.3 until this issue get fixed, and just use it, if is necessary : for example there is an special feature only included on that version.


Most of the users are just looking to install and set up their system, and start using it. But when you have to install and then run in to a complicated process of tweaking you system in order to make it works. These makes system less atractive to the users.

These is just constructive feedback,trying to make the user experience better and less stressful at the moment to install and use vicidial

[kashinc]

honestly downgrading my entire cluster to 1.8 was not that difficult or time consuming, It hasn't been tested yet and will be tested on Monday morning but the only thing that concerns me is the SVN trunk version is 2656. I don't know if that needs to be downgraded as well. But the Steps to downgrading are very simple.

[mflorell]

You shouldn't need to downgrade VICIdial svn/trunk at all. VICIdial still works with Asterisk going back to 1.4, so shouldn't be a problem. We still have hundreds of clients running Asteirsk 1.4 and 1.8 systems with the latest svn/trunk.

[ambiorixg12]

Hi Matt, Is there any issue tracker related to this issue, so we can follow and know when Vicibox 7.0.3 or higher will works without be needed to downgrade Asterisk to 1.8

[mflorell]

Not yet, this is a very fluid issue still. We are monitoring frequently affected systems and have built an internal IP blocking option to test, but no definitive fixes yet. What this is looking like is that Asterisk 11 can be crashed by some kind of malformed internet traffic, and Asterisk sometimes throws errors when this happens, and sometimes it does not.

[kashinc]

downgrading to 1.8 and 7.0.1 did not solve my problem. I was having zero issues before this.

-kashinc

[alo]

Did anyone find a solution to this?
asterisk 1.8 still works great but I need a newer version of asterisk for webrtc.