Intermittent time synchronization problem

[bigbozza]

Vicidial REDUX VERSION: 2.10-449a, BUILD: 140817-0928
openSUSE 13.1
Asterisk 1.8.29.0-vici

Single server setup has been running nicely for almost a year.

10 agents total, very low loads.

Some calls drop out and the agent gets the time sync error
Some calls they cannot hear the customer but the call has not dropped properly, eventually they see the same time sync error but the next call is fine.
If the agent gets a fax machine and they end the call it can make the error show also.

I've trawled the net for answers but most posts are relating to this issue but not in an intermittent fashion.

I also followed this troubleshooting guide to no avail.
striker24x7 blogspot com au /2012/10/time-synchronization-error-vicidial.html?view=sidebar

The servers time is synced off the active directory and so are the workstations. Every time I check, it appears they are all in sync with each other.
The PHP time and DB time match the server also.

[mflorell]

This is usually a network issue between the server and the workstation, or an issue with the workstation itself. Although we did make a small change in the code recently, related to this, that affects some agents in rare cases that should now be fixed. You should try upgrading to the latest svn/trunk version and see if that fixes your issues.

[bigbozza]

Thanks for your response.

I haven't done any SVN upgrades at all since its initial install (if it ain't broken...) and considering it's been running mostly well for a year, I believe it's a switching issue. The issue has cropped up a couple of times before now, but it appears to be happening quite often now which would denote a degrade in network performance.

I left an MTR running from a workstation to the server with no hops except for the switch between them and it returned 1% packet loss over a 20 hour period - more than enough to drop a call here and there IMO.

I will run more tests on Monday but i'm fairly certain the switch is going to be the culprit. Will report back with more findings.

[mflorell]

Thanks for the postback!

Local network switches going bad actually seems to be an increasing problem, we just had a client have that problem this week. They replaced the switch and all was well.

[bigbozza]

OK so the Windows admin has informed me that some of the workstations are off, even though they are meant to be synced to the DC.

At any rate the biggest time difference appears to be about 23 seconds. Is this enough to cause this issue? And could you possibly touch on (just a little) the interaction between the workstations time and vici please.

[williamconley]

It's not about time at the workstations.

There are two basic dynamics involved:

1) Time between servers/processes

All servers in the cluster must be synced (not "set", but "synced" using ntp) so they share time within 6 seconds across all servers and processes. This means that php and mysql time must be the same zone as well.

2) Packet loss to agent screens is unacceptable

Logged In Agent Screens send and receive a packet once per second. The receiving php script changes a value in the database upon receipt/request. If this value is NOT changed, the system will believe it has lost contact with the agent and kill the session with this same error. The reason the error is "time sync" is that the agent may NOT have lost contact, but the process checking the time or setting the time on the watchdog field may be off. So the problem is that we can not be sure if the agent lost a packet (#2) or if the time is not synchronized (#1), both toss the same error.

And before you go there ... No, the packet does not have a timestamp in it. The processes that handle the packet generate time from the server upon which they run. PHP and MySQL are the two processes running on that server which are involved in this handshake.

[bigbozza]

Thanks for clarifying William. It's a single server setup so clustering does not factor in; And in the reports section when I expand the server it shows it's time, db and php all to be in sync exactly.

Packet loss appears to be the problem still.

So you are saying between the error logs and database(s) there isn't going to be a more verbose reason logged?

[williamconley]

Nope. There is no "error". Packets being dropped is merely "interesting" and the system then does what it's designed to do when communications are lost ... kick out the agent to avoid bad data.