There is a time synchronization problem with your system

[DomeDan]

Hello Vicidial users!

This "There is a time synchronization problem with your system, please tell your system administrator" has been bugging me a while and I haven't had time checking it out till now.

It happen all of a sudden when I shut down my NAS server, I could not understand how my NAS had anything to do with time synchronization of the server, did a reboot of the server and everything was fine.
I start digging to see what times the system was checking and wrote this one-line script to check the 3 times in question (last_update,db_time,php-time):

Code: Select all

source <(sed 's/\ =>\ /=/g' /etc/astguiclient.conf) && \
echo "SELECT server_ip, UNIX_TIMESTAMP(last_update),UNIX_TIMESTAMP(db_time) from server_updater" | mysql -u$VARDB_user -p$VARDB_pass -h$VARDB_server -D asterisk && \
source /etc/sysconfig/clock && \
php -r "date_default_timezone_set('$TIMEZONE'); echo 'php time: '.date('U');" && echo ""

edit: updated the script to read variables

last_update and db_time stops when i shut down my NAS
using grep I found out that the script AST_update.pl was updating the last_update value,

one part of that script also checks disk usage on the server with the df command,
and that command hangs when it tries to look at the disk-usage of a NFS share that cant be reached.

So I got two options, either change the timeo and retrans options for the fstab entry
to make it timeout faster and let the script continue.
Or I can make some changes to the script, this is what I did:

Code: Select all

--- AST_update.pl.orig   2013-04-03 21:59:57.695319259 +0200
+++ AST_update.pl   2013-04-03 22:00:14.910192229 +0200
@@ -643 +643 @@
-               @serverDISK = `$dfbin -B 1048576`;
+               @serverDISK = `$dfbin -B 1048576 /`;

I just made the script look at the root only and thus prevent it from hanging if I happen to shut down the NAS again.

Hope the next person that encounters this problem finds my post!

[williamconley]

What is the result of "looking at root only"? And how does that stop it from accessing NAS? Or was the network drive mounted in a subfolder?

[DomeDan]

What is the result of "looking at root only"?:
The result is that df wont hang and then AST_update.pl wont hang and then there is no "time synchronization problem"

And how does that stop it from accessing NAS? Or was the network drive mounted in a subfolder?:
Yes, you answerd your own question, this is how I set up my filesystems:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p2      64G   16G   45G  26% /
devtmpfs              1.9G  144K  1.9G   1% /dev
tmpfs                 1.9G  4.0K  1.9G   1% /dev/shm
x.x.x.36:/mnt/vol0    917G  159G  686G  19% /mnt/nas
x.x.x.46:/             63G   11G   50G  18% /mnt/ViciSlave
x.x.x.45:/             64G  8.4G   52G  14% /mnt/ViciDB


and btw, this applies to any kind of mounted network filsystem like samba, sshfs, ftpfs etc. that has a long timeout.

A third way to fix this would be to set some kind of timeout in the perl script to kill df if it takes too long.

A forth way would be to put the "df-code" in a separate script that does not affect anything else when it hangs.

I see now that I should have put this thread under the development board ^^

[williamconley]

maybe df can be told to exclude the /mnt/ folder

[DomeDan]

Exclude, good point!
And I found that there's actually a flag for that: -x
so "df -x nfs -x cifs -x sshfs -x ftpfs" would exclude nfs, samba, sshfs and ftpfs -mounts, and it works! tested with nfs and cifs

[williamconley]

Cool. Now post a diff against the latest svn to the vicidial issue tracker (link at the top of the page) and post a link to that entry here. Perhaps Matt will find this a worthy bug to squish.

[DomeDan]

Yeah I know where the issue tracker is http://www.vicidial.org/VICIDIALmantis/view.php?id=664
its not the best fix but at least it solves the problem