Time sync issue

[Keyfin]

So everything was running good until...

Under Vicidial Server Stats and Reports the server time will stop counting time while the PHP time and DB time continue in sync. Problem is temporarily fixed with a server reboot, until it stops counting server time again. On the server side, the 'date' command into the console reveals the server indeed has the correct day and time. The 'time stopping' seems to be happening randomly, at different times, whether server is busy or idle. Any suggestions on how to stop this from happening?

[williamconley]

Ignore if it the server continues to run with no other symptoms.

If there are other symptoms, ask yourself why you did not list them. (and then list them, LOL)

[Keyfin]

Haha, no other issues as far as I know, besides the messages announcing the time sync error that constantly pops up while agents are logged in. Annoying, but not detrimental to the conducting of business. I suppose for now, I'll just restart the server when the issue occurs.

[williamconley]

time sync error that constantly pops up while agents are logged in

Funny, I'd consider that a symptom.

But it's not likely related to time service. It's more likely related to dropped packets between the agents and the Vicidial web server.

Those packets are heartbeat packets, and dropping them causes the system to think that time is out of sync because the timing of the tracking of those packets exceeds the maximum allowable limit of the difference between "last update" and "now". This also happens when time is out of sync between two servers ... thus the confusion on the error message.

[Keyfin]

Interesting, the time sync error message only pops up when the report page i mentioned earlier stops counting the server time... after server restart, the error messages don't pop up until, at some random point, may be a week, may be 3 days, may be 2 weeks, the reports page stops counting the server time. Then the time sync error messages are constant, until I restart the server again, and then the time starts counting again and is perfectly in sync... but again, if the reports page server time stops, i can go to the server terminal console use 'date' command and see that the server time is actually fine...

[mflorell]

You might have a corrupted table, I would suggest running the AST_DB_check_tables.pl script to confirm that your database tables are all running properly.

[Keyfin]

Thanks matt, ran the script, but there was no output from the script.

[mflorell]

run it with the --help flag to see options. You might need to run it with debug enabled to see output.

[Keyfin]

ok with debug flag, i got these final results

Check table results: |185|
Corrupted tables: |0|
Non-closed tables: |0|
Memory tables: |8|
FINISHED... EXITING (2 seconds)

[Keyfin]

Twice, yesterday, this problem occured right at the close of the day. The server time stopped counting, then time sync error messages popped up. I rebooted, it was working for ~2 minutes, then stopped again... rebooted again, and it's been perfectly in sync since then, ~ 18 hours now.

[Keyfin]

There's has been no stop yet, at this time. (Check time stamp on posts for time reference)

[williamconley]

Twice, yesterday, this problem occured right at the close of the day. The server time stopped counting, then time sync error messages popped up. I rebooted, it was working for ~2 minutes, then stopped again... rebooted again, and it's been perfectly in sync since then, ~ 18 hours now.

1) Hard drive full? When the HD fills, quite often rebooting will bring it back as temp files are deleted. Usually this results in corrupted tables, though.

2) When this happens, it is entirely possible that your location is losing web contact with the Vicidial web server or the apache process is crashing (still loss of contact, but that would be server-side). Is your vicidial server local or remote? Of course a brute force web attack could feasibly result in this behavior as well, but that's a bit far-fetched unless your firewall is down.

[Keyfin]

Thanks for the response William!

"df" command

Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 8202216 0 8202216 0% /dev
tmpfs 8207424 0 8207424 0% /dev/shm
tmpfs 8207424 329752 7877672 5% /run
tmpfs 8207424 0 8207424 0% /sys/fs/cgroup
/dev/sda2 476454936 6270208 469199976 2% /
tmpfs 6291456 13608 6277848 1% /var/spool/asterisk/monitor

5% so, no, not a full hard drive, and the server is local private IP 192.168.1.100

As far as attacks, there were, but I've blocked the IPs of the SSH login attempts, and the IPs of attempts to register sip phones, and the IPs of non-critical transaction timeouts. So, atm, there are no hack attempts, at all, but the server time is in sync, and has been for days, so it may have been due to brute forces attacks that i have now subdued...

So far so good, since last time it did this. I will continue to watch for hack attempts and watch the time to see if it stops again, and to see if the two are related.

[williamconley]

Blocking individual IPs to slow attacks is like trying to get rid of cockroaches without cleaning the house.

Note that while you may block them, and you'll drop those packets, that does not in any way stop them from pounding your system's firewall. You not seeing the entries does not stop your network and CPU from "dealing" with this network traffic: It just transfers the "work" from asterisk to iptables. And "hopefully" they get bored at some point and stop attacking. But they'll be back, from a new IP address. And they often slam the firewall for an hour even when they are blocked (unattended scripts don't care if you answer).

So Dynamic Good Guys firewall for Vicidial is a good way to go. There are lockdown instructions for Vicibox if you don't feel the need to actually install DGG, but DGG provides two very easy-to-use web pages to manage your whitelist if you like that sort of thing (no more command line firewall management, LOL).

http://viciwiki.com/index.php/DGG

[Keyfin]

will probably look into integrating, i've simply been blocking ranges using
iptables -I INPUT -s 37.0.0.0/38.0.0.0 -j DROP

and i see DGG has been upgraded to work with the new apache, nice!

[williamconley]

Attacks start a "stupid in the morning", when you're not around. DGG has (to date) not even been attacked after installation unless an attack was already in progress. Then you have to let that attack wear off naturally or change your IP *after* installing DGG.

[Keyfin]

thanks, william. That's excellent.

[njujihara]

crontab -e

#HORA
* * * * * /usr/sbin/ntpdate -u yourntp.server.com 2>/dev/null 1>&2


thats work form me .

[Keyfin]

DGG didn't work.... after install, i couldn't access the server internally or remotely. SSH still worked fine, but all I could get from the web browser is the 'please hold while i redirect you' message, then it would time out while redirecting

also caused apache to not start due to multiple listen to port 81 lines at the bottom of the conf file, but that was ofc fixed by deleting the multiple entries.

[njujihara]

Try, to another navegator.

[Keyfin]

and use what? netscape?

[williamconley]

DGG didn't work.... after install, i couldn't access the server internally or remotely. SSH still worked fine, but all I could get from the web browser is the 'please hold while i redirect you' message, then it would time out while redirecting

also caused apache to not start due to multiple listen to port 81 lines at the bottom of the conf file, but that was ofc fixed by deleting the multiple entries.

Were there any errors during installation?

Also: If you turn off the firewall and that does not resolve your issue, your issue is not DGG.

You could also check to see that you did not break any apache files. Restart the apache service and see if it gives you any errors (this is fairly common if you manually edited any conf file in apache at any point).

[Keyfin]

no errors with instruction beside the rm dgg install command, that didn't work obviously since i didn't have dgg installed yet.... apache is fine now that i deleted the extra listen to port 81 lines. dgg isn't currently installed, and everything is working fine since i put the forwarding ports and the http and https services back into the firewall.

[williamconley]

1) If you had repaired instead of removing the port 81 line (which was likely due to an altered configuration file instead of stock Vicibox), you likely would have had a functional DGG system. If the listen 81 instruction was blocking apache from starting, then your firewall wasn't the issue, but apache was just not running.

2) "Forwarding ports" is for a router, not for iptables. If you have "forwarding" happening then you truly have an unusual system. Did you mean to say "open" ports? iptables (as a rule) either blocks packets or allows them. It rarely has an instruction regarding "forwarding".

3) If you followed JUST the lockdown procedure, you can then add any "good" ips to the custom firewall entries tab. Without actually installing DGG, you would then have a whitelisted and safe system. At that stage, installing DGG is performed solely to allow the easy addition of "Good" IPs without having to resort to the command line (that's why it exists). If you follow those instructions without installing DGG, at least your server will be safe. And you can debate DGG installation at your leisure without having an insecure system.

[Keyfin]

yes, meant open, not forwarded, but for some reason, if i take away the open ports and the http and https services from the external zone in the firewall, i am unable to access the agent login screen even from the internal network... and btw thanks much for the advice and help!

[williamconley]

If your server has two network connections (one internal and one public), you should assign them as such in yast firewall. If you have only one connection (private network only), then you'll need to allow the private network in the yast firewall custom tab (two times: once for UDP and once for TCP).

You'll also need to add ALL authorized IPs for carriers and anyone else accessing externally. Then your lockdown is "whitelisted" and the server still works but is safe.

Definition of "whitelist".

[williamconley]

crontab -e

#HORA
* * * * * /usr/sbin/ntpdate -u yourntp.server.com 2>/dev/null 1>&2


thats work form me .

Never do this. Waste of time, and concievably could bring down the cluster (or get you blacklisted from the ntp system in question, which would cause worse problems).

Manually setting the date breaks any previously configured ntp preferences that are designed to sync properly (and almost perfectly, over time). Those statistics are dumped.

Also: This would probably only even work on very old servers with certain OSs. You probably need an upgrade. LOL

Modify the ntp.conf file to have valid servers in your country instead. Look up proper use of ntp.

[Keyfin]

I did it! Thanks so much, William. I'm not running DGG, but that's quite alright with me, I currently have no open ports to my server, no services allowed and everything is working beautifully. I have full access from both my work pc and home pc. Thanks again, William, you're a godsend.

BTW, the brute force attacks were exactly what was causing the original time sync issue.

[williamconley]

Good for you *)

(And DGG is really only for those who require the ability to easily add/remove "Good" IPs. If you have a finite list and it's already solid, you don't have a use for DGG anyway. Yet. )

[njujihara]

crontab -e

#HORA
* * * * * /usr/sbin/ntpdate -u yourntp.server.com 2>/dev/null 1>&2


thats work form me .

Never do this. Waste of time, and concievably could bring down the cluster (or get you blacklisted from the ntp system in question, which would cause worse problems).

Manually setting the date breaks any previously configured ntp preferences that are designed to sync properly (and almost perfectly, over time). Those statistics are dumped.

Also: This would probably only even work on very old servers with certain OSs. You probably need an upgrade. LOL

Modify the ntp.conf file to have valid servers in your country instead. Look up proper use of ntp.

old system centos 5.4 vici 2.2 LOL!!!

[williamconley]

In which case, grab a new HD and install Goautodial on it (if you must have CentOS) and you'll at least be in the realm of "up to date". Then (if you like) you can upgrade using svn and you'll lose the goautodial interface (which is not required anyway, the regular vicidial interface is still present on the Goautodial install, you just have to do a bit of hunting to find the web links). But after the svn upgrade, you'll have the Latest Greatest Vicidial on CentOS without having to scratch install.

With a new HD, you can always just pop the old hd back in, and you're right back where you started (no damage).

The only true drawback is that the Vicibox install now uses asterisk 11 and will shortly (according to the grapevine) have a built-in Web Phone (which requires asterisk 11, of course).

It already has email and chat integrated into the dialer queue, but Web Phone is way cooler than either of those. No soft phone installation required, just microphone and speakers (or headset) and your agents are functional. goautodial does not yet install asterisk 11.

[irfanshafi]

I am facing the same issue the server time does not move a second, though time gets updated when I reboot I believe there is some service which runs while rebooting and then stops, though date command displays the correct time and PHP and DB time are working and synced can anyone please help would appreciate.

[williamconley]

I am facing the same issue the server time does not move a second, though time gets updated when I reboot I believe there is some service which runs while rebooting and then stops, though date command displays the correct time and PHP and DB time are working and synced can anyone please help would appreciate.

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method (7.X.X?) and vicidial version with build (VERSION: 2.X-XXXx ... BUILD: #####-####).

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "manual/from scratch" you must post your operating system with version (and the .iso version from which you installed your original operating system) plus a link to the installation instructions you used. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

3) Output from:

Code: Select all

screen -list

One of those screens is responsible for updating a table with the time in it. If that screen isn't running, the time is not updated. This is NOT "the problem". It is a tiny symptom of a larger problem. That screen continually checks for other screens and if any are missing, it refuses to run. Those other screens can die if asterisk isn't running or for any one of several other issues. All are necessary for a functional system. The command(s) to start those screens are based on various configuration options and are contained in your keepalive perl script (which runs every minute in your crontab for root).

Test-starting them with --debugX will often give a viable troubleshooting path to find your actual problem.