Vicibox 7.0.3 Time Sync stuck at the top of the minute

[alo]

Hello Friends. Happy Holidays.

We have a couple new installs of vicibox 7.0.3 running and seem to intermittently need to run 'systemctl restart vicidial' because the agents get a time sync error and the server section on the reports page goes red and shows the server time as the last minute. at the next minute it will sync back up, but go red again after a few seconds. this happens intermittently. sometimes once a day sometimes more.



I tried messing with the NTP settings and even turning the NTP service off. still happening.

If I run /usr/share/astguiclient/ADMIN_keepalive_ALL.pl it syncs the time but doesn't keep moving and turns red after a moment. (I believe that is why the time shows correct at the minute, because the cron runs the script at the minute)

We are not experiencing this with vicibox 7.0.1 installs.

Anyone experiencing the same on vicibox 7.0.3?
kernel version 4.1.20-11-default
VERSION: 2.12-565a
BUILD: 160827-0917
Version: 2.12b0.5
SVN Version: 2585
DB Schema Version: 1467
Asterisk version 11.22.0-vici
dahdi version 11.22.0
Dahdi linux version 2.11.1
Dell R610 Raid 1 with a perc 700 raid card and SSD drives
13 agents logged in with dial level 6. Single DB/Web/Asterisk server

Same set ups are running fine when installed using vicibox 7.0.1.
Ex of one running fine is:
Same hardware
kernel version: 4.1.15-8-default
dahdi version 11.21.0
Dahdi linux version 2.11.0
Version: 2.12b0.5
SVN Version: 2488
DB Schema Version: 1455
11.21.0-vici

So the two things I notice different is the kernel version and the asterisk version and the dahdi version.

So I guess I have two questions. Is anyone else having trouble with vicibox 7.0.3 and anyone know if it would be safe for me to try to update kernel or dahdi version?

[mflorell]

Try running "/usr/share/astguiclient/AST_update.pl --debugX" manually and see if it gives you an error.

[alo]

VERY helpful Thank you!

So heres what I got when the time is not staying synced after running your suggestion.

|UPDATE server_updater set last_update='2016-12-21 21:36:24' where server_ip='External_IP_suppressed'|
command timed-out at /usr/share/astguiclient/AST_update.pl line 491

Line 491 Asks for @list_channels = $t->cmd(String => "Action: Command\nCommand: core show channels concise\n\n", Prompt => '/--END COMMAND-.*/');

I tried to manually run this with

Code: Select all

asterisk -rvx "core show channels concise"

During the times of issue it thinks for a while and doesn't display anything. When the system is running correctly it outputs stuff correctly.

So I come to the assupmtion asterisk is too busy to output the concise info. so I checked asterisk at the time of issues.

And Wabam!!!

The following statements are spamming the console hundreds per minute

[Dec 21 21:56:54] WARNING[16719][C-000019d8]: channel.c:1338 __ast_queue_frame: Unable to write to alert pipe on Local/58600051@default-000019a4;1 (qlen = 92): Resource temporarily unavailable!

Less frequently but still several times per minute:

[Dec 21 21:56:54] WARNING[1450]: chan_sip.c:4277 __sip_autodestruct: Autodestruct on dialog '7ae0cca0066591d319c4c20621fcd9ea@External_IP_suppressed:5060' with owner SIP/AloIn-000017ff in place (Method: BYE). Rescheduling destruction for 10000 ms

And a few times per minute

[Dec 21 22:08:47] WARNING[16496][C-000019c1]: channel.c:1310 __ast_queue_frame: Exceptionally long voice queue length queuing to Local/58600057@default-0000198e;1

And although no agents are in calls the live_sip_channels table shows calls that were open when the server stopped updating and I get the following when checking the channel count from asterisk.

Code: Select all

asterisk -rvx "core show channels count"

189 active channels
100 active calls
11292 calls processed

So after a systemctl restart vicidial the channels are cleared and nothing is spamming the asterisk console at all.

I do not have port 5060 opened, (ips are whitelisted) so I do not think its due to spam hacking (could be wrong)

Do you think these messages in asterisk is the cause for the system to stop updating? it seems some channels are being stuck open?

and probably most important, why am I all of a sudden getting these messages spamming asterisk console.

I will be swapping to a different carrier as a test tomorrow. any other suggestions or things I should check?

Thanks Again!

[mflorell]

Confirm that the Asterisk version is correct in the Admin -> Servers section of the VICIdial web admin. We've noticed some issues with the default value of that lately.

[alo]

I have Triple checked this now.

Both say 11.22.0-vici

[josecapurro]

I used to have that time sync problem.

Add an entry in root's crontab to synchronize time every 5' (when logged in as root via SSH, use crontab -e):

*/5 * * * * /usr/sbin/ntpdate 0.centos.pool.ntp.org

Of course, you can use any NTP server you like.

Configuring the NTP client via YaST doesn't work as well.

After that, get sure ntpd is stopped and disabled:
systemctl stop ntpd
systemctl disable ntpd

This is because if ntpd is running, ntpdate will fail to bind the socket (because ntpd is using it) and it will not synchronize the time.

HTH

[alo]

GREAT recommendation!

I could definitely see how this could affect it, However this did not resolve our problem.

Any other recommendations?

[KeithHBW]

I am having the same problem and it hangs at core show channels concise when I do the debug as well.
Problem is related to keywords:
autodestruct
exceptionally long voice queue

[mflorell]

We have noticed this as an issue recently as well. We don't know what is causing it, it does not happen every day and it does not affect all systems. Some clients that are affected have had the problem solved by downgrading to Asterisk 1.8. There are instructions to do that with vicibox 7 on this thread:
viewtopic.php?f=8&t=35346

[ambiorixg12]

We have noticed this as an issue recently as well. We don't know what is causing it, it does not happen every day and it does not affect all systems. Some clients that are affected have had the problem solved by downgrading to Asterisk 1.8. There are instructions to do that with vicibox 7 on this thread:
viewtopic.php?f=8&t=35346

HI, I running a server with this vicidial version and working fine

Code: Select all

VERSION: 2.12-560a
BUILD: 160617-1427
© 2016 ViciDial Group

But I'm recently installed this version and I have the sync issue

VERSION: 2.14-582a
BUILD: 161222-0841
© 2016 ViciDial Group

So my question is Matt should we downgrade the vicidial version or is there any other updated version with that issue fixed

[alo]

ambiorixg12, Whats your asterisk version?

We were able to successfully downgrade to 1.8 and are waiting to see if it crashes again.

[ambiorixg12]

Both server are runnning Asterisk 11m but the one older one VERSION: 2.12-560a
BUILD: 160617-1427, it is working with no issue at all, I dont want to downgrade to asterisk 1.8 is dead, and it is not longer supported and there is not security fix. so that is not a good deal

[ambiorixg12]

It seems there is an issue with Vicibox 7.03, I just downgrade to Vicibox_v.6.0.x86_64-6.0.4.iso / Asterisk 1.8 and all is working fine

[gequiros]

I am having the same problem and it hangs at core show channels concise when I do the debug as well.
Problem is related to keywords:
autodestruct
exceptionally long voice queue

TO ME ( i am not experienced in ANY ways ) this is related to a BRUTE or any of the SIP ATTACKS to ASTERISK

NONE of my servers behind a hardware firewall get this string, but, i got 7 outside a firewall and this is a common thing to see...

[mflorell]

Yes, this is related to a new form of malformed-packet brute force that we started seeing on the internet in December. When we were able to figure out how this was happening exactly on our hosted platform using the HOMER SIP logging and analysis utility, we were able to filter the affected attacking IPs out and the problems went away for our hosting customers.

We are actually working on building a separate open-source project to do a full HOMER/OpenSIPs/etc... gateway and monitoring ISO installer so that our non-hosted users can also be able to be protected as our hosted clients are now. See this post for more information:
viewtopic.php?f=3&t=36879

[alo]

Did anyone find a solution to this?
asterisk 1.8 still works great but I need a newer version of asterisk for webrtc.

[mflorell]

We're still using our monitoring and filtering solution, and it works very well at this point. Other than that, using whitelists within your Asterisk configuration should help.

[williamconley]

... Other than that, using whitelists within your Asterisk configuration should help.

Or use Dynamic Good Guys from Viciwiki.com. Installation is ONLY necessary if you want to have the "easy IP authorization" page. If you just use the instructions for the pre-install, you will have a solid whitelist, but will need to use the "yast firewall" custom pane to add authorized IPs.

Don't just protect asterisk, protect the entire server.

[alo]

We have port 5060 closed except for whitelisted IPs using Yast2 firewall and we still seem to have asterisk crash once or twice a day one several servers running asterisk 11.25.1-vici

is there something else I should be doing to block these malformed-packet brute force or could my problem be unrelated and asterisk may be crashing for a different reason?

Here are a few entries from before asterisk auto restarted if you are interested in taking a peek or see something funny.

Code: Select all

[Aug 19 14:11:07] VERBOSE[7879] manager.c: [Aug 19 14:11:07]   == Manager 'sendcron' logged on from 127.0.0.1
[Aug 19 14:11:07] VERBOSE[7879] manager.c: [Aug 19 14:11:07]   == Manager 'sendcron' logged off from 127.0.0.1
[Aug 19 14:11:08] VERBOSE[7862] manager.c: [Aug 19 14:11:08]   == Manager 'sendcron' logged off from 127.0.0.1
[Aug 19 14:11:41] WARNING[2773][C-0000007b] res_srtp.c: SRTP unprotect failed with: authentication failure 110
[Aug 19 14:11:54] VERBOSE[1860] chan_sip.c: [Aug 19 14:11:54]     -- Registered SIP 'st_612' at XXX.XXX.XXX.XX:51226
[Aug 19 14:11:57] VERBOSE[1857] chan_sip.c: [Aug 19 14:11:57]     -- Registered SIP 'st_612' at XXX.XXX.XXX.XX:51225
[Aug 19 14:12:07] VERBOSE[7967] manager.c: [Aug 19 14:12:07]   == Manager 'sendcron' logged off from 127.0.0.1
[Aug 19 14:14:01] VERBOSE[8185] manager.c: [Aug 19 14:14:01]   == Manager 'sendcron' logged off from 127.0.0.1
[Aug 19 14:14:02] VERBOSE[8205] manager.c: [Aug 19 14:14:02]   == Manager 'updatecron' logged on from 127.0.0.1
[Aug 19 14:14:06] VERBOSE[8211] manager.c: [Aug 19 14:14:06]   == Manager 'sendcron' logged on from 127.0.0.1
[Aug 19 14:14:48] VERBOSE[1860] chan_sip.c: [Aug 19 14:14:48]     -- Registered SIP 'st_612' at XXX.XXX.XXX.XX:51226
[Aug 19 14:15:01] VERBOSE[8318] manager.c: [Aug 19 14:15:01]   == Manager 'sendcron' logged off from 127.0.0.1
[Aug 19 14:15:01] VERBOSE[8320] manager.c: [Aug 19 14:15:01]   == Manager 'sendcron' logged on from 127.0.0.1
[Aug 19 14:15:02] VERBOSE[8338] manager.c: [Aug 19 14:15:02]   == Manager 'updatecron' logged on from 127.0.0.1
[Aug 19 14:16:02] Asterisk 11.25.1-vici built by abuild @ lamb66 on a x86_64 running Linux on 2017-04-27 02:35:44 UTC

It seems very random when asterisk crashes, and sometimes it will restart itself (auto restart asterisk setting turned on in vicidial server settings page) but sometimes I will manually have to restart vicidial. Past couple days its been auto restarting sometime between around 15:00-16:00 PDT and before that it hadn't crashed for a few days and before that it was crashing in the mornings around 11. Real weird. I could get behind the fact that maybe its when someone is scanning for voip servers and brutforcing, but I see no logs of failed attempts or anything. unless maybe I am not seeing them because the firewall is blocking them, but they are still coming in. Not sure here.

[alo]

I pulled some logs from other places at the same time that I am working on googling to see if any of these could be causing asterisk to crash.

Code: Select all

[b]Firewall Log:[/b]
Aug 19 14:15:28 vicibox7 kernel: SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=XXX:XXX:XXX SRC=46.186.97.217 DST=XXX.XXX.XXX.XX LEN=60 TOS=0x00 PREC=0x20 TTL=47 ID=38988 DF PROTO=TCP SPT=40699 DPT=23 WINDOW=47916 RES=0x00 SYN URGP=0 OPT (020405840402080A5AEF8C1E000000000103030B)
Aug 19 14:15:31 vicibox7 kernel: SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=XXX:XXX:XXX SRC=46.186.97.217 DST=XXX.XXX.XXX.XX LEN=60 TOS=0x00 PREC=0x20 TTL=47 ID=38989 DF PROTO=TCP SPT=40699 DPT=23 WINDOW=47916 RES=0x00 SYN URGP=0 OPT (020405840402080A5AEF97D7000000000103030B)
Aug 19 14:15:41 vicibox7 kernel: SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=XXX:XXX:XXX SRC=122.252.225.179 DST=XXX.XXX.XXX.XX LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=64024 DF PROTO=TCP SPT=57450 DPT=23 WINDOW=5760 RES=0x00 SYN URGP=0 OPT (020405A00402080A0226E2F50000000001030301)
Aug 19 14:15:44 vicibox7 kernel: SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=XXX:XXX:XXX SRC=122.252.225.179 DST=XXX.XXX.XXX.XX LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=64025 DF PROTO=TCP SPT=57450 DPT=23 WINDOW=5760 RES=0x00 SYN URGP=0 OPT (020405A00402080A0226E5E40000000001030301)
Aug 19 14:15:59 vicibox7 kernel: SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=XXX:XXX:XXX SRC=203.178.148.19 DST=XXX.XXX.XXX.XX LEN=32 TOS=0x00 PREC=0x00 TTL=46 ID=51853 DF PROTO=ICMP TYPE=8 CODE=0 ID=11336 SEQ=4619

[b]Messages Log:[/b]
Aug 19 14:15:06 vicibox7 kernel: asterisk[16346]: segfault at 3c ip 00007f7745df39bd sp 00007f758792dc70 error 4 in res_rtp_asterisk.so[7f7745dd2000+51000]

From [b]Mail:[/b]
Aug 19 14:15:06 vicibox7 pickup[1194]: A88AAD60006: uid=0 from=<root>
Aug 19 14:15:06 vicibox7 cleanup[8352]: A88AAD60006: message-id=<20170819211506.A88AAD60006@vicibox7.suse>
Aug 19 14:15:06 vicibox7 qmgr[1624]: A88AAD60006: from=<root@vicibox7.suse>, size=782, nrcpt=1 (queue active)
Aug 19 14:15:06 vicibox7 pickup[1194]: AF0F6D60004: uid=0 from=<root>
Aug 19 14:15:06 vicibox7 cleanup[8352]: AF0F6D60004: message-id=<20170819211506.AF0F6D60004@vicibox7.suse>
Aug 19 14:15:06 vicibox7 qmgr[1624]: AF0F6D60004: from=<root@vicibox7.suse>, size=792, nrcpt=1 (queue active)
Aug 19 14:15:06 vicibox7 local[8356]: A88AAD60006: to=<root@vicibox7.suse>, orig_to=<root>, relay=local, delay=0.07, delays=0.05/0/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Aug 19 14:15:06 vicibox7 qmgr[1624]: A88AAD60006: removed
Aug 19 14:15:07 vicibox7 local[8357]: AF0F6D60004: to=<root@vicibox7.suse>, orig_to=<root>, relay=local, delay=0.88, delays=0.05/0.01/0/0.82, dsn=2.0.0, status=sent (delivered to mailbox)
Aug 19 14:15:07 vicibox7 qmgr[1624]: AF0F6D60004: removed
Aug 19 14:16:01 vicibox7 pickup[1194]: 8A264D60009: uid=0 from=<root>
Aug 19 14:16:01 vicibox7 cleanup[8352]: 8A264D60009: message-id=<20170819211601.8A264D60009@vicibox7.suse>
Aug 19 14:16:01 vicibox7 qmgr[1624]: 8A264D60009: from=<root@vicibox7.suse>, size=820, nrcpt=1 (queue active)
Aug 19 14:16:01 vicibox7 pickup[1194]: 8A8D4D60007: uid=0 from=<root>
Aug 19 14:16:01 vicibox7 cleanup[8352]: 8A8D4D60007: message-id=<20170819211601.8A8D4D60007@vicibox7.suse>
Aug 19 14:16:01 vicibox7 qmgr[1624]: 8A8D4D60007: from=<root@vicibox7.suse>, size=824, nrcpt=1 (queue active)
Aug 19 14:16:01 vicibox7 local[8356]: 8A264D60009: to=<root@vicibox7.suse>, orig_to=<root>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Aug 19 14:16:01 vicibox7 qmgr[1624]: 8A264D60009: removed
Aug 19 14:16:01 vicibox7 local[8357]: 8A8D4D60007: to=<root@vicibox7.suse>, orig_to=<root>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Aug 19 14:16:01 vicibox7 qmgr[1624]: 8A8D4D60007: removed

[williamconley]

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method (7.X.X?) and vicidial version with build (VERSION: 2.X-XXXx ... BUILD: #####-####).

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "manual/from scratch" you must post your operating system with version (and the .iso version from which you installed your original operating system) plus a link to the installation instructions you used. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

3) Closing one port does not stop your server from being attacked. You have to whitelist the entire server to avoid attack. Any open ports will invite the attack because you represent a live server with free money. Whitelist only. Seriously.

4) segfault is not something that can be predicted or caused by any specific item. It's usually an unpredictable result of a flaw in the system. Any flaw. Hardware failure. An updated package with bad code. An included module with bad code. Heavy load causing normally unused resources to be brought into play (such as virtual memory as a single example). Or an attack trying to destabilize the system, which may result in ... bad code executing.

5) Uninstall (or at least disable) any extra modules you may have installed. Be sure you're not running out of memory or overloading the server. Seriously consider a full reinstall (Perhaps on new hardware). We often will replace a single component at a time to see if the problem goes away with a component. Having an identical server with which to switch parts is excellent for this. If the problem follows a memory chip into another server ... chuck that sucker.

6) If you have special software (webrtc?) running on this server and haven't mentioned it ... well, you should hang your head in shame as this may have been Very Relevant. lol

7) Happy Hunting!

[alo]

You are always so helpful!

I included the Information in the first post, but here it is in the recommended format:

Vicibox 7.0.3 from .iso | Vicidial 2.12-565a Build 160827-0917 | Asterisk 11.22.0-vici | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Dell R610 2x Quad core E5620 2.4GHz 32 gb RAM Raid 1 with a perc 700 raid card and SSD drives

Number 3, Regarding closing all pots, My Sip carrier isn't a media proxy, so it handles the signaling but not the Media. So I have been leaving 10000-20000 UDP open for this reason. Other then Finding a new carrier or trying to get all their Media IPs, how do I go about allowing the media without keeping the ports open? Obviously I am not an IPtables expert, but I thought I remembered hearing something about how established connections would be allowed without a port being open, but how would they be established connections?

Number 6, They are using WebRTC web hpone, but other then the web files it doesn't require anything to be installed.

Thanks again.