chronyc

[kashinc]

I have been running the new vicibox 9.0.2 and have seen high system usage on a few machines, the one machine that is bothering me most is my master database server. Chronyc is using 100% cpu on this server... When I was running 9.0.0 I always had low load averages, has anyone else had any similar issues or know how to fix this?

[carpenox]

u sure its chrony? mysqld is pretty high load using a lot of Mem%, but not over 50% ever, but i run a little diff, im on asterisk 13.32.0 instead of 29.

[Kumba]

Chronyc is it's own monolithic piece of software. In some random searches google mentions things like network changes (DHCP) as causing issues with Chrony. If you haven't already, you can try running package updates to see if it pulls anything in.

Other then that, I really don't have a clue. I'd have to be able to duplicate the issue and I have never ran across chrony trying to peg a CPU.

[kashinc]

This problem seems to be related to a AMD chips... Both webserver and database server were both maxed out at 100% cpu usage.

I installed the latest Kernel 5.6 and the issue is now gone, hope this helps for anyone else running into this problem.

[Kumba]

Kernel 5.6 isn't a default option on OpenSuSE v.15.1. Were you already running current updates on ViciBox v.9 before doing that update?

Also, what AMD CPUs were they?

[kashinc]

Kernel 5.6 isn't a default option on OpenSuSE v.15.1. Were you already running current updates on ViciBox v.9 before doing that update?

Also, what AMD CPUs were they?

Yes, I was running current updates... it happens on some of the servers its very strange...

it shows as /usr/bin/chronyc -a -n -m tracking

it hogs 100% of system usage... only happens on some installs and is not only AMD specific...

upgrading the kernel to this version 5.7.0-3.gad96a07-default

will fix the chronyc issue, however this causes issues with dahdi no longer working and you will have to manually compile it for it to work properly... not sure if this is the best route... just trying to give you some feedback Kumba so we can figure it out and make progress, I enjoy giving back to the community whenever possible.

[kashinc]

Kernel 5.6 isn't a default option on OpenSuSE v.15.1. Were you already running current updates on ViciBox v.9 before doing that update?

Also, what AMD CPUs were they?

Yes, I was running current updates... it happens on some of the servers its very strange...

it shows as /usr/bin/chronyc -a -n -m tracking

it hogs 100% of system usage... only happens on some installs and is not only AMD specific...

upgrading the kernel to this version 5.7.0-3.gad96a07-default

will fix the chronyc issue, however this causes issues with dahdi no longer working and you will have to manually compile it for it to work properly... not sure if this is the best route... just trying to give you some feedback Kumba so we can figure it out and make progress, I enjoy giving back to the community whenever possible.

also I have noticed that this doesn't happen on install, or even after zypper ref and zypper up reboot. However after i do vicibox-install this starts happening.

[SPAMSAM]

This issue happens to me on install prior to vicibox-install on 9.0.3

The issue seems to have two components, 1.) chronyd crashes on startup with faiure to lock /var/run/chrony-helper and 2.) 100% CPU usage on one core by /usr/bin/chronyc -a -n -m tracking.

Issue 1 can be resolved by running the following:

Code: Select all

rm -f /var/run/chrony-helper/lock
service chronyd restart


Note that if the system reboots the error will reoccur. Also once chronyd is up and running again it keeps using 100% CPU. So not a fix.

Any suggestions are welcome as the issue seems to be hardware-agnostic.

[Kumba]

Whatever is broken with chrony is beyond me. I have found nothing to explain it's behavior. Some servers it works like a champ, some it dies. I think it might be related to network connectivity on start-up but I haven't been able to definitively test that.

My solution is to burn it with fire and install ntp. Here's how you do that:

Code: Select all

service chronyd stop
killall chronyc
zypper rm chrony  (enter y when prompted)
zypper in ntp
systemctl enable ntpd.service
echo "restrict -4 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict -6 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "server 127.127.1.0" >> /etc/ntp.conf
echo "fudge 127.127.1.0  stratum 10" >> /etc/ntp.conf
echo "server 0.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 1.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 2.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 3.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
service ntpd start


That should fix chrony permanently. The down side is the default ntp configuration doesn't allow it to function as a time source, but that's probably not a big issue unless you have large large clusters.

[carpenox]

good information to have, seems to be a problem with a lot of new installs. i havent had this problem on my r240's but i know a lot of people have

[Fares MEHENI]

Whatever is broken with chrony is beyond me. I have found nothing to explain it's behavior. Some servers it works like a champ, some it dies. I think it might be related to network connectivity on start-up but I haven't been able to definitively test that.

My solution is to burn it with fire and install ntp. Here's how you do that:

Code: Select all

service chronyd stop
killall chronyc
zypper rm chrony  (enter y when prompted)
zypper in ntp
systemctl enable ntpd.service
echo "restrict -4 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict -6 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "server 127.127.1.0" >> /etc/ntp.conf
echo "fudge 127.127.1.0  stratum 10" >> /etc/ntp.conf
echo "server 0.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 1.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 2.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
echo "server 3.opensuse.pool.ntp.org  iburst" >> /etc/ntp.conf
service ntpd start


That should fix chrony permanently. The down side is the default ntp configuration doesn't allow it to function as a time source, but that's probably not a big issue unless you have large large clusters.

It's working now, thanks a lot Kumba .