perl command consuming 100% cpu usage on TOP

[richardroi]

Support,

Need some help regarding CPU usage. What could be the reason on PERL COMMAND consuming 100% cup usage? Is this Bad? What's the worst thing can happen? This just hapened today.

25930 daemon 20 0 29452 4216 1308 R 100 0.1 31:45.51 perl
5028 mysql 20 0 2451m 438m 5696 S 45 5.5 26:59.50 mysqld
5163 root 20 0 515m 17m 5664 S 21 0.2 99:53.54 asterisk
2936 root 20 0 15584 3700 1024 S 1 0.0 1:29.40 mtr
5204 root 20 0 48196 9696 2652 S 1 0.1 6:10.07 AST_update.pl
5821 root 20 0 15584 3700 1024 S 1 0.0 8:11.09 mtr
5210 root 20 0 47660 9164 2644 S 0 0.1 0:08.44 AST_manager_lis
5352 root 20 0 18992 1320 936 S 0 0.0 1:52.53 top
10181 daemon 20 0 101m 9m 6536 S 0 0.1 0:06.92 httpd


Please help...

Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[richardroi]

Anyone??? need help here guys... Thanks.


Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[boybawang]

how many agents do you have at the same time?

are you using call recording on all calls?

have you checked the disk space of your server?

try rebooting your server first, if it comes back the same then theres a problem with the vicidial perl scripts or the data itself has some level of corruption

[richardroi]

i have 21 agents, 9 on resume, 12 on auxes( break, training)
we are recording all calls. I noticed it this morning, already rebooted, it disappeared and then around 12mn it came out again.
what can i do to correct this, is this bad?
already checked hdd. lots of free space.


when i ran ps -ax
25930 ? R 130:46 init
hope this helps.


Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[boybawang]

try doing a db repair after your shift, its done by this command: mysqlcheck -p --all-databases -o -r

[richardroi]

Thank you Sir.
Will try it. can you tell me whats the worst thing could happen?
What could be the cause of this error? Agents are not complaining about any issues...



Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[boybawang]

try installing htop to have a much graphical view

it could just be the db is large and your hard drive speed is slow, anyway i suggest you upgrade to vicidial 2.4 and asterisk 1.4

[richardroi]

we are operating 24/7, yes we are currently building a new one now. but i need to resolve this issue for now. hoping your suggestion (mysqlcheck -p --all-databases -o -r)will work.

mysqlcheck asterisk -repair -u cron -p is this the same with the one that you gave me?

Thank you so much Sir.


Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[boybawang]

try to install htop so you can see a better view of what actual perl script is hogging the server

[richardroi]

Installed htop...
1 cpu is on 100 %

Load average: 1:73 sometimes it is reaching 2.44

PID User Command
25930 daemon init

This is the cause of high cpu usage.

Hope you can help me.



Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[boybawang]

what OS are you using?

[richardroi]

Ubuntu Sir.



Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[richardroi]

someone have an idea?
Please help...



Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[IslandCabby]

type t after you have top open to get a tree view and it will tell you what specific processes are running for perl, it's an underlier for various php-based modules so you might have a job running or a process-heavy custom php script. Sometimes the easiest question to ask is "what has changed on the server? Did I install something new/modify anything right before the problem started?" Most often than not that's the culprit.

[richardroi]

25930 daemon 20 0 29452 4220 1308 R 100 0.1 880:10.33 perl
5163 root 20 0 515m 18m 5664 S 13 0.2 208:04.50 asterisk
5028 mysql 20 0 2474m 503m 5712 S 1 6.3 55:07.10 mysqld
5204 root 20 0 48196 9700 2652 S 1 0.1 12:57.08 AST_update.pl
16347 root 20 0 19396 1432 992 S 1 0.0 4:39.03 htop
412 daemon 20 0 100m 7008 3544 S 0 0.1 0:00.82 httpd
707 daemon 20 0 101m 7016 3544 S 0 0.1 0:00.57 httpd
2516 daemon 20 0 100m 6252 2948 S 0 0.1 0:00.03 httpd

here is my top tree view. I can' t remember anything that i made changes on th server.
thank you.



Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram

[mcargile]

run 'ps -ef' and look for the pid of that perl process. This will actually tell you what perl script is running. Post that and we can go from there.

[richardroi]

hello mcargile.
'ran ps -ef'... daemon 8219 1 99 May07 ? 13:27:07 init
this is the pid that is causing my high cpu usage. I was observing it and if i am correct this perl always run every Monday. Because after restarting it last Tuesday it disappear. just showed up again this Monday afternoon. I will restart it again after my shift, and will report to forum for results.

Thank you.

[mcargile]

Please post the complete output of ps -ef and place it inside a code block so everything lines up like this, but the whole thing:

Code: Select all

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 May03 ?        00:00:02 init [5]
root         2     0  0 May03 ?        00:00:00 [kthreadd]
root         3     2  0 May03 ?        00:00:01 [migration/0]
root         4     2  0 May03 ?        00:00:02 [ksoftirqd/0]
root         5     2  0 May03 ?        00:00:00 [watchdog/0]
root         6     2  0 May03 ?        00:00:01 [migration/1]
root         7     2  0 May03 ?        00:01:34 [ksoftirqd/1]
root         8     2  0 May03 ?        00:00:00 [watchdog/1]
root         9     2  0 May03 ?        00:00:01 [migration/2]
root        10     2  0 May03 ?        00:00:03 [ksoftirqd/2]
root        11     2  0 May03 ?        00:00:00 [watchdog/2]
root        12     2  0 May03 ?        00:00:01 [migration/3]
root        13     2  0 May03 ?        00:00:10 [ksoftirqd/3]
root        14     2  0 May03 ?        00:00:00 [watchdog/3]


Also what is this running on? Vicibox, GoAutoDial, or something else?

The reason I ask is because init is the parent process to everything. It should have a PID of 1. The init you are listing seems to have a pid of 8219.

[richardroi]

Code: Select all

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 May01 ?        00:00:03 /sbin/init
root         2     0  0 May01 ?        00:00:00 [kthreadd]
root         3     2  0 May01 ?        00:00:00 [migration/0]
root         4     2  0 May01 ?        00:00:21 [ksoftirqd/0]
root         5     2  0 May01 ?        00:00:00 [watchdog/0]
root         6     2  0 May01 ?        00:00:01 [migration/1]
root         7     2  0 May01 ?        00:00:26 [ksoftirqd/1]
root         8     2  0 May01 ?        00:00:00 [watchdog/1]
root         9     2  0 May01 ?        00:00:00 [migration/2]
root        10     2  0 May01 ?        00:00:20 [ksoftirqd/2]
root        11     2  0 May01 ?        00:00:00 [watchdog/2]
root        12     2  0 May01 ?        00:00:01 [migration/3]
root        13     2  0 May01 ?        00:00:40 [ksoftirqd/3]
root        14     2  0 May01 ?        00:00:00 [watchdog/3]
root        15     2  0 May01 ?        00:00:02 [events/0]
root        16     2  0 May01 ?        00:00:04 [events/1]
root        17     2  0 May01 ?        00:00:02 [events/2]
root        18     2  0 May01 ?        00:00:02 [events/3]
root        19     2  0 May01 ?        00:00:00 [khelper]
root        54     2  0 May01 ?        00:00:01 [kblockd/0]
root        55     2  0 May01 ?        00:00:01 [kblockd/1]
root        56     2  0 May01 ?        00:00:01 [kblockd/2]
root        57     2  0 May01 ?        00:00:01 [kblockd/3]
root        60     2  0 May01 ?        00:00:00 [kacpid]
root        61     2  0 May01 ?        00:00:00 [kacpi_notify]
root       158     2  0 May01 ?        00:00:00 [kseriod]
root       216     2  0 May01 ?        00:00:00 [pdflush]
root       217     2  0 May01 ?        00:00:15 [pdflush]
root       218     2  0 May01 ?        00:00:00 [kswapd0]
root       261     2  0 May01 ?        00:00:00 [aio/0]
root       262     2  0 May01 ?        00:00:00 [aio/1]
root       263     2  0 May01 ?        00:00:00 [aio/2]
root       264     2  0 May01 ?        00:00:00 [aio/3]
root      1474     2  0 May01 ?        00:00:00 [ksuspend_usbd]
root      1475     2  0 May01 ?        00:00:00 [khubd]
root      1553     2  0 May01 ?        00:00:00 [ata/0]
root      1554     2  0 May01 ?        00:00:00 [ata/1]
root      1555     2  0 May01 ?        00:00:00 [ata/2]
root      1556     2  0 May01 ?        00:00:00 [ata/3]
root      1557     2  0 May01 ?        00:00:00 [ata_aux]
root      2282     2  0 May01 ?        00:00:00 [scsi_eh_0]
root      2283     2  0 May01 ?        00:00:00 [scsi_eh_1]
root      2305     2  0 May01 ?        00:00:00 [scsi_eh_2]
root      2306     2  0 May01 ?        00:00:00 [scsi_eh_3]
root      2659     2  0 May01 ?        00:00:55 [kjournald]
root      2815     1  0 May01 ?        00:00:00 /sbin/udevd --daemon
root      3461     2  0 May01 ?        00:00:03 [edac-poller]
root      3488     2  0 May01 ?        00:00:00 [kpsmoused]
daemon    4374  5310  0 04:06 ?        00:00:04 /usr/local/apache2/bin/httpd -k
ntp       4400     1  0 May01 ?        00:00:11 /usr/sbin/ntpd -p /var/run/ntpd.
root      4419     2  0 May01 ?        00:00:23 [kjournald]
root      4420     2  0 May01 ?        00:00:06 [kjournald]
root      4421     2  0 May01 ?        00:00:00 [kjournald]
root      4772     1  0 May01 tty4     00:00:00 /sbin/getty 38400 tty4
root      4773     1  0 May01 tty5     00:00:00 /sbin/getty 38400 tty5
root      4775     1  0 May01 tty2     00:00:00 /sbin/getty 38400 tty2
root      4777     1  0 May01 tty3     00:00:00 /sbin/getty 38400 tty3
root      4779     1  0 May01 tty6     00:00:00 /sbin/getty 38400 tty6
syslog    4825     1  0 May01 ?        00:00:14 /sbin/syslogd -u syslog
root      4844     1  0 May01 ?        00:00:00 /bin/dd bs 1 if /proc/kmsg of /v
klog      4846     1  0 May01 ?        00:00:00 /sbin/klogd -P /var/run/klogd/km
root      4961     1  0 May01 ?        00:00:04 /usr/sbin/sshd
root      5017     1  0 May01 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql     5042  5017  2 May01 ?        04:27:13 /usr/sbin/mysqld --basedir=/usr
root      5043  5017  0 May01 ?        00:00:00 logger -p daemon.err -t mysqld_s
root      5104     1  0 May01 ?        00:00:00 /usr/sbin/vsftpd
root      5162     1  0 May01 ?        00:00:00 /usr/bin/SCREEN -S astshell
root      5163  5162  0 May01 pts/1    00:00:00 /bin/sh
root      5169     1  0 May01 ?        00:00:06 SCREEN -L -S asterisk
root      5170  5169  0 May01 pts/2    00:00:00 /bin/sh
root      5177  5170 11 May01 pts/2    18:25:50 /usr/sbin/asterisk -vvvvvvvvvvvv
root      5187  5177  0 May01 pts/2    00:00:13 mpg123 -q -s --mono -r 8000 -b 2
root      5189  5187  0 May01 pts/2    00:00:00 mpg123 -q -s --mono -r 8000 -b 2
root      5217     1  0 May01 ?        00:00:08 /usr/bin/SCREEN -d -m -S ASTupda
root      5218  5217  0 May01 pts/0    01:08:52 /usr/bin/perl /usr/share/astguic
root      5220     1  0 May01 ?        00:01:08 /usr/bin/SCREEN -d -m -S ASTsend
root      5221  5220  0 May01 pts/3    00:19:06 /usr/bin/perl /usr/share/astguic
root      5223     1  0 May01 ?        00:00:07 /usr/bin/SCREEN -d -m -S ASTlist
root      5224  5223  0 May01 pts/4    00:01:27 /usr/bin/perl /usr/share/astguic
root      5226     1  0 May01 ?        00:00:29 /usr/bin/SCREEN -d -m -S ASTVDau
root      5227  5226  0 May01 pts/5    00:09:55 /usr/bin/perl /usr/share/astguic
root      5229     1  0 May01 ?        00:00:00 /usr/bin/SCREEN -d -m -S ASTVDre
root      5230  5229  0 May01 pts/6    00:03:39 /usr/bin/perl /usr/share/astguic
root      5232     1  0 May01 ?        00:00:13 /usr/bin/SCREEN -d -m -S ASTVDad
root      5233  5232  0 May01 pts/7    00:06:03 /usr/bin/perl /usr/share/astguic
root      5235     1  0 May01 ?        00:00:00 /usr/bin/SCREEN -d -m -S ASTfast
root      5236  5235  0 May01 pts/8    00:00:07 /usr/bin/perl /usr/share/astguic
daemon    5258     1  0 May01 ?        00:00:00 /usr/sbin/atd
root      5269     1  0 May01 ?        00:00:06 /usr/sbin/cron
root      5310     1  0 May01 ?        00:00:12 /usr/local/apache2/bin/httpd -k
root      5316     1  0 May01 tty1     00:00:00 /bin/login --
root      5424  5316  0 May01 tty1     00:00:00 -bash
root      5657  4961  0 May01 ?        00:00:14 sshd: root@pts/9
root      5695  5657  0 May01 pts/9    00:00:00 -bash
root      5733  4961  0 May01 ?        00:00:51 sshd: root@pts/10
root      5759  5733  0 May01 pts/10   00:00:00 -bash
daemon    5848  5310  0 04:13 ?        00:00:04 /usr/local/apache2/bin/httpd -k
daemon    6372  5310  0 04:15 ?        00:00:03 /usr/local/apache2/bin/httpd -k
daemon    8203 24239  0 May07 ?        00:00:00 [sh] <defunct>
daemon    8219     1 99 May07 ?        13:51:19 init
daemon    8855  5310  0 04:28 ?        00:00:03 /usr/local/apache2/bin/httpd -k
root      9123  5177  0 May07 pts/2    00:00:01 mpg123 -q -s --mono -r 8000 -b 2
root      9124  9123  0 May07 pts/2    00:00:00 mpg123 -q -s --mono -r 8000 -b 2
daemon    9943  5310  0 04:34 ?        00:00:03 /usr/local/apache2/bin/httpd -k
daemon   10753  5310  0 04:38 ?        00:00:02 /usr/local/apache2/bin/httpd -k
daemon   10972  5310  0 04:39 ?        00:00:02 /usr/local/apache2/bin/httpd -k
root     11207  5236  0 02:11 pts/8    00:00:00 /usr/bin/perl /usr/share/astguic
daemon   11564  5310  0 04:42 ?        00:00:02 /usr/local/apache2/bin/httpd -k
daemon   13786  5310  0 04:54 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14248  5310  0 04:56 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14249  5310  0 04:56 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14357  5310  0 04:57 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14358  5310  0 04:57 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14769  5310  0 04:59 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   14777  5310  0 04:59 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   15180  5310  0 02:28 ?        00:00:12 /usr/local/apache2/bin/httpd -k
daemon   15648  5310  0 05:03 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   15656  5310  0 05:03 ?        00:00:01 /usr/local/apache2/bin/httpd -k
daemon   15954  5310  0 05:05 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   16091  5310  0 05:05 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   16226  5310  0 05:06 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   16426  5310  0 05:07 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   16429  5310  0 05:07 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   16899  5310  0 05:09 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17010  5310  0 05:09 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17011  5310  0 05:09 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17163  5310  0 05:10 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17829  5310  0 05:14 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17830  5310  0 05:14 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17867  5310  0 05:14 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17982  5310  0 05:14 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   17987  5310  0 05:14 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   18125  5310  0 05:15 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   18882  5310  0 05:19 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   18887  5310  0 05:19 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   19033  5310  0 05:20 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   19036  5310  0 05:20 ?        00:00:00 /usr/local/apache2/bin/httpd -k
daemon   19037  5310  0 05:20 ?        00:00:00 /usr/local/apache2/bin/httpd -k
root     19328  5269  0 05:22 ?        00:00:00 /USR/SBIN/CRON
root     19329 19328  0 05:22 ?        00:00:00 /bin/sh -c /usr/share/astguiclie
root     19330 19329  0 05:22 ?        00:00:00 /usr/bin/perl /usr/share/astguic
root     19389  4961  0 05:22 ?        00:00:00 sshd: root@pts/11
root     19404 19389  0 05:22 pts/11   00:00:00 -bash
root     19421 19404  0 05:22 pts/11   00:00:00 ps -ef
root     21187  5236  0 02:56 pts/8    00:00:00 /usr/bin/perl /usr/share/astguic
root     23067  5236  0 03:05 pts/8    00:00:00 /usr/bin/perl /usr/share/astguic
daemon   23272  5310  0 03:05 ?        00:00:08 /usr/local/apache2/bin/httpd -k
daemon   24239  5310  0 May07 ?        00:00:04 /usr/local/apache2/bin/httpd -k
root     24833  5236  0 03:12 pts/8    00:00:00 /usr/bin/perl /usr/share/astguic
root     26845  5424  0 May01 tty1     00:21:11 top
daemon   28784  5310  0 03:31 ?        00:00:07 /usr/local/apache2/bin/httpd -k
root     29896  5759  0 03:35 pts/10   00:00:57 htop
root     30124  5236  0 01:10 pts/8    00:00:00 /usr/bin/perl /usr/share/astguic
root     31227  5695  0 May07 pts/9    00:00:00 rasterisk r


This is an old vicidial (ubuntu). We've been using it for 3 years now.

[richardroi]

hello... I restarted the server this morning, and everything is back to normal. will continue monitoring it. I remember restarting it Monday morning(last week) and it came back around 12mn that day, then restarted it again Tuesday morning and it went fine the whole week, it just showed up again yesterday (Monday afternoon). The only thing that bothers me is maybe our server is being idle on weekends, because on weekends there are few calls. Before we are restarting it once a month.

[richardroi]

It showed up again this morning. Can someone help me with this issue?