Time Sync Error

[hansg]

Hi,

I am having an issue where I am repeatedly getting a "Time Synchronization Error Please contact your Administrator."

This was happening every few days and I have rebooted the system and it has started working again, but today it has happened 4 times in the last hour.

Below is an image of what happens:


What can it be that is causing this? What logs do I need to check?

[carpenox]

make sure the timezones are set correctly on the server, the php.ini files at /etc/php7/apache

[hansg]

I have the correct timezone set in that php.ini file

[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = "Europe/London"

Is there any other files I should check for the same thing?

[carpenox]

on the server, type "date" and see what its set to, as well as the main php.ini file in /etc/php7

[hansg]

I have checked these.

They are all displaying the correct time / timezone.

I am still getting frequent crashes. No idea what is causing them.

[hansg]

I think there is something causing a spike which is crashing the database.

Seconds before the it crashes I see the System Load jump from around 0.75 up to about 7.00

Then instant Time Sync Error.

[carpenox]

is this a virtual server?

[hansg]

It is a Cloud Server.

I have loads of other servers all running perfectly using this cloud server method.

The only difference is that the others were installed from scratch on Centos. This one with the issue is a Vicibox Install using OpenSuse.

[carpenox]

perhaps try to set asterisk to auto restart on crash in the server settings on vicidial interface

[hansg]

Worth a shot.

I will try this and give an update.

[hansg]

Unfortunately that never made a difference.

Still can't find the issue.

EDIT: Having automatically restart asterisk on does fix the time sync error after it happens and allows users to log back in. However, it does not prevent it from happening.

Still an issue for me.

[carpenox]

yea thats just a temp fix. When it happens, have you ran a sockets check to see whats running and whats not? you dont get any error messages in your mail on the system?

[hansg]

How would I find out if it is running out of sockets?

Also what mail would you be talking about?

[hansg]

I am getting 2 errors in asterisk alot:

ERROR[2392]: chan_sip.c:4271 __sip_reliable_xmit: Serious Network Trouble; __sip_xmit returns error for pkt data

and

WARNING[2805][C-00000019]: func_hangupcause.c:140 hangupcause_read: Unable to find information for channel.


From what I have been told, these are just errors to do with WebRTC and can be ignored.

[carpenox]

sockets is by running "screen -ls" at the linux cli and checking to see if the ones you chose to run during install.pl are all running. definitely need send and listen which are the ones that tend to disappear during a time sync error. Yes those errors are from the webrtc, not that serious.

[hansg]

I paid someone to take a look at it for me.

They have said that the issue is with my crontab. They have made some changes to it.

I will test tomorrow. Hopefully it sorts the issue.

[hansg]

Nope.

Still crashing frequently. No idea what the problem is

[carpenox]

I can look into it for you. Reach out to me through facebook: chris nox | skype: carpenox_3

[kjburto]

Did you ever find out what the issue is on this? I am having the same issue with one of my servers. I am starting to think it could be something physical with the server memory or something. Is that possible?

[carpenox]

definitely possible, what are the specs? how many agents?

[williamconley]

... I am still getting frequent crashes. No idea what is causing them.

... which is crashing the database

It is a Cloud Server.

Cloud server doesn't answer the question: Is it Virtual or Physical? Both can reside in a colocation facility such as Amazon or elsewhere. Vicidial does not run Virtual. Google it, enjoy the battle, but be prepared for a serious time investment and likely failure. lol.

... Still crashing frequently.

I am having the same issue with one of my servers.

If an application is "crashing", then it would either be "stuck" or no longer running and in most cases it would leave an error trail of some sort.

I'm not seeing any error logs, or any information regarding processes no longer running or processes that are running but no longer functioning (both scenarios are common during this scenario). (IE: no Troubleshooting ... just guesses)

The original poster showed a red server in the "Reports" main page, but kjburto did not show or mention such an entry.

Normally when the server "goes red" it is because one of the "screen -ls" entries is missing, because one of the processes running in one of those screens has exited (abnormally) and the screen auto-closed. In some cases, the script running in the screen simply stops running without exiting, but that missing script turns the server red as the services the script provides are necessary and there's a "watchdog" field that is no longer updated ... thus turning the server red.

So the first thing we need to know is: Are all your processes running? If you're not sure, try this:

Code: Select all

screen -ls

If you do not know which ones *should* be running, reboot and run that command again when everything is operating normally. The scripts are run based on the server's /etc/astguiclient.conf setting for "VARactive_keepalives".

Once we know which script is missing (If that's the case) we can figure out why it's not running by starting it normally using the same command the keepalive script would use to start it. Add one thing to the command: " --debugX" and you may get lucky and find out why it's crashing. You may need to modify the keepalive to execute it with the --debugX directive so you can keep an eye on the screen (leave the screen running in a terminal and wait for it to crash ... the history may be useful). Often this information is in a log for the screen in question (those missing logs I was talking about )

If the script that is missing is the "asterisk" script, and asterisk is crashing, try using the /etc/asterisk/modules.conf file to set "noload" directives for any asterisk modules you are not using (as a hip shot) or check the asterisk logs to find out if the first error is in any way related to a module.

An older server's noload list

Code: Select all

noload => pbx_gtkconsole.so
;load => pbx_gtkconsole.so
;
load => res_musiconhold.so
;
; Load either OSS or ALSA, not both
; By default, load OSS only (automatically) and do not load ALSA
;
noload => chan_alsa.so
;noload => chan_oss.so
noload => chan_woomera.so
noload => chan_capi.so
noload => cdr_csv.so


A newer server's noload list:

Code: Select all

; Load one of: chan_oss, alsa, or console (portaudio).
; By default, load chan_oss only (automatically).
;
noload => chan_alsa.so
noload => chan_oss.so
noload => chan_console.so
;
noload => cdr_csv.so
noload => chan_ooh323.so
noload => chan_woomera.so
noload => chan_capi.so
noload => res_config_sqlite.so
noload => app_cdr.so
noload => cdr_manager.so
noload => cdr_sqlite3_custom.so
noload => func_cdr.so
noload => app_forkcdr.so
noload => cdr_custom.so
noload => cdr_sqlite.so
noload => cdr_syslog.so
noload => res_ael_share.so
noload => pbx_lua.so
noload => res_speech.so
noload => res_jabber.so
noload => res_fax.so
noload => res_smdi.so
noload => pbx_ael.so
noload => app_ices.so
noload => app_festival.so
noload => pbx_realtime.so
noload => func_realtime.so
noload => chan_skinny.so
noload => format_jpeg.so
noload => format_vox.so
noload => app_sms.so
noload => app_talkdetect.so
noload => chan_agent.so
noload => app_zapateller.so
noload => app_nbscat.so
noload => app_queue.so
noload => cel_sqlite3_custom.so
noload => app_disa.so
noload => chan_gtalk.so
noload => app_image.so
noload => app_dictate.so
noload => app_url.so
noload => res_phoneprov.so
noload => func_pitchshift.so
noload => func_blacklist.so
noload => app_page.so
noload => res_http_post.so
noload => app_directory.so
noload => app_test.so
noload => app_flash.so
noload => chan_unistim.so
noload => app_sendtext.so
noload => app_minivm.so
noload => chan_jingle.so


You can also set asterisk to perform a core dump at the crash moment and read the core to find the error, but that's a wee bit involved. Usually disabling modules that aren't needed will solve an asterisk crash scenario.