vicidial.org

Hi All,

We've been having a strange issue where across 4 machines & installs, the AST_update_AMI2.pl script has been hanging and not updating the server after the nightly reboot. I have tried restarting every service under the sun (vicidial, mariadb, wicked, apache, firewall) and it still refuses to update. But it's all fixed after a manual restart. I've posted the response where it hangs below.

Code: Select all

 perl AST_update_AMI2.pl --debugX

----- DEBUGGING ENABLED -----


----- SUPER-DUPER DEBUGGING -----


|1|SELECT count(*) FROM server_updater where server_ip='REDACTED';|

|1|SHOW TABLES LIKE "cid_channels_recent_103149217090";|
1|cid_channels_recent_103149217090|SHOW TABLES LIKE "cid_channels_recent_103149217090";
----- AMI Version 2.10.5 -----


2021-05-10 09:57:40
|Channel List:
|total_channels|0
0 channels recieved out of 0 reported.|0|
|SELECT extension, protocol FROM phones where server_ip='REDACTED' and phone_type NOT LIKE "%trunk%"|
callin|EXTERNAL
gs102|SIP

UPDATE server_updater set last_update='2021-05-10 09:57:40' where server_ip='REDACTED'

It is affecting a variety of servers in different clusters on different physical machines ranging from OpenSUSE 15.1 SVN 3336 to OpenSUSE 15.2 SVN 3429. It seems to only be the primary dialers which are affected but they are all whitelisted off. Nothing too special to note in boot.log or journalctl either. Any ideas of where to look next?

Just did some further digging. We had a mount which didn't attach as the backup server went down for maintenance, I restarted the update script after just by chance to check and it was all fixed. I have no idea why a single mount failing would cause this but something to keep an eye out for.

Thank you very much for the update on this.

We have noticed weird things happening before on client systems using mounted partitions on external servers over the years, but we really don't run into those very often since we never use them ourselves.

Disk usage subroutine in these scripts using df might hang if some of the mount points are not accessible, leading to entire script hanging, causing complete stall of the dialler.

Although in newer versions of these scripts some file systems are excluded, there is still possibility this to happen.
We commented out that check in both scripts to stay on the safe side, since systems are monitored by other means.