vicidial.org

Hello Vicidial users!

This "There is a time synchronization problem with your system, please tell your system administrator" has been bugging me a while and I haven't had time checking it out till now.

It happen all of a sudden when I shut down my NAS server, I could not understand how my NAS had anything to do with time synchronization of the server, did a reboot of the server and everything was fine.
I start digging to see what times the system was checking and wrote this one-line script to check the 3 times in question (last_update,db_time,php-time):

Code: Select all

source <(sed 's/\ =>\ /=/g' /etc/astguiclient.conf) && \
echo "SELECT server_ip, UNIX_TIMESTAMP(last_update),UNIX_TIMESTAMP(db_time) from server_updater" | mysql -u$VARDB_user -p$VARDB_pass -h$VARDB_server -D asterisk && \
source /etc/sysconfig/clock && \
php -r "date_default_timezone_set('$TIMEZONE'); echo 'php time: '.date('U');" && echo ""

edit: updated the script to read variables

last_update and db_time stops when i shut down my NAS
using grep I found out that the script AST_update.pl was updating the last_update value,

one part of that script also checks disk usage on the server with the df command,
and that command hangs when it tries to look at the disk-usage of a NFS share that cant be reached.

So I got two options, either change the timeo and retrans options for the fstab entry
to make it timeout faster and let the script continue.
Or I can make some changes to the script, this is what I did:

Code: Select all

--- AST_update.pl.orig   2013-04-03 21:59:57.695319259 +0200
+++ AST_update.pl   2013-04-03 22:00:14.910192229 +0200
@@ -643 +643 @@
-               @serverDISK = `$dfbin -B 1048576`;
+               @serverDISK = `$dfbin -B 1048576 /`;

I just made the script look at the root only and thus prevent it from hanging if I happen to shut down the NAS again.

Hope the next person that encounters this problem finds my post!

What is the result of "looking at root only"? And how does that stop it from accessing NAS? Or was the network drive mounted in a subfolder?

What is the result of "looking at root only"?:
The result is that df wont hang and then AST_update.pl wont hang and then there is no "time synchronization problem"

And how does that stop it from accessing NAS? Or was the network drive mounted in a subfolder?:
Yes, you answerd your own question, this is how I set up my filesystems:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p2      64G   16G   45G  26% /
devtmpfs              1.9G  144K  1.9G   1% /dev
tmpfs                 1.9G  4.0K  1.9G   1% /dev/shm
x.x.x.36:/mnt/vol0    917G  159G  686G  19% /mnt/nas
x.x.x.46:/             63G   11G   50G  18% /mnt/ViciSlave
x.x.x.45:/             64G  8.4G   52G  14% /mnt/ViciDB


and btw, this applies to any kind of mounted network filsystem like samba, sshfs, ftpfs etc. that has a long timeout.

A third way to fix this would be to set some kind of timeout in the perl script to kill df if it takes too long.

A forth way would be to put the "df-code" in a separate script that does not affect anything else when it hangs.

I see now that I should have put this thread under the development board ^^

maybe df can be told to exclude the /mnt/ folder

Exclude, good point!
And I found that there's actually a flag for that: -x
so "df -x nfs -x cifs -x sshfs -x ftpfs" would exclude nfs, samba, sshfs and ftpfs -mounts, and it works! tested with nfs and cifs

Cool. Now post a diff against the latest svn to the vicidial issue tracker (link at the top of the page) and post a link to that entry here. Perhaps Matt will find this a worthy bug to squish.

Yeah I know where the issue tracker is http://www.vicidial.org/VICIDIALmantis/view.php?id=664
its not the best fix but at least it solves the problem