Page 1 of 1
perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 11:28 am
by richardroi
Support,
Need some help regarding CPU usage. What could be the reason on PERL COMMAND consuming 100% cup usage? Is this Bad? What's the worst thing can happen? This just hapened today.
25930 daemon 20 0 29452 4216 1308 R 100 0.1 31:45.51 perl
5028 mysql 20 0 2451m 438m 5696 S 45 5.5 26:59.50 mysqld
5163 root 20 0 515m 17m 5664 S 21 0.2 99:53.54 asterisk
2936 root 20 0 15584 3700 1024 S 1 0.0 1:29.40 mtr
5204 root 20 0 48196 9696 2652 S 1 0.1 6:10.07 AST_update.pl
5821 root 20 0 15584 3700 1024 S 1 0.0 8:11.09 mtr
5210 root 20 0 47660 9164 2644 S 0 0.1 0:08.44 AST_manager_lis
5352 root 20 0 18992 1320 936 S 0 0.0 1:52.53 top
10181 daemon 20 0 101m 9m 6536 S 0 0.1 0:06.92 httpd
Please help...
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 12:46 pm
by richardroi
Anyone??? need help here guys... Thanks.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 12:47 pm
by boybawang
how many agents do you have at the same time?
are you using call recording on all calls?
have you checked the disk space of your server?
try rebooting your server first, if it comes back the same then theres a problem with the vicidial perl scripts or the data itself has some level of corruption
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 12:58 pm
by richardroi
i have 21 agents, 9 on resume, 12 on auxes( break, training)
we are recording all calls. I noticed it this morning, already rebooted, it disappeared and then around 12mn it came out again.
what can i do to correct this, is this bad?
already checked hdd. lots of free space.
when i ran ps -ax
25930 ? R 130:46 init
hope this helps.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 1:32 pm
by boybawang
try doing a db repair after your shift, its done by this command: mysqlcheck -p --all-databases -o -r
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 1:38 pm
by richardroi
Thank you Sir.
Will try it. can you tell me whats the worst thing could happen?
What could be the cause of this error? Agents are not complaining about any issues...
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 1:54 pm
by boybawang
try installing htop to have a much graphical view
it could just be the db is large and your hard drive speed is slow, anyway i suggest you upgrade to vicidial 2.4 and asterisk 1.4
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 2:05 pm
by richardroi
we are operating 24/7, yes we are currently building a new one now. but i need to resolve this issue for now. hoping your suggestion (mysqlcheck -p --all-databases -o -r)will work.
mysqlcheck asterisk -repair -u cron -p is this the same with the one that you gave me?
Thank you so much Sir.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 2:26 pm
by boybawang
try to install htop so you can see a better view of what actual perl script is hogging the server
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 2:39 pm
by richardroi
Installed htop...
1 cpu is on 100 %
Load average: 1:73 sometimes it is reaching 2.44
PID User Command
25930 daemon init
This is the cause of high cpu usage.
Hope you can help me.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 3:15 pm
by boybawang
what OS are you using?
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 3:18 pm
by richardroi
Ubuntu Sir.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 4:51 pm
by richardroi
someone have an idea?
Please help...
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon Apr 30, 2012 9:21 pm
by IslandCabby
type t after you have top open to get a tree view and it will tell you what specific processes are running for perl, it's an underlier for various php-based modules so you might have a job running or a process-heavy custom php script. Sometimes the easiest question to ask is "what has changed on the server? Did I install something new/modify anything right before the problem started?"
Most often than not that's the culprit.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Tue May 01, 2012 1:40 am
by richardroi
25930 daemon 20 0 29452 4220 1308 R 100 0.1 880:10.33 perl
5163 root 20 0 515m 18m 5664 S 13 0.2 208:04.50 asterisk
5028 mysql 20 0 2474m 503m 5712 S 1 6.3 55:07.10 mysqld
5204 root 20 0 48196 9700 2652 S 1 0.1 12:57.08 AST_update.pl
16347 root 20 0 19396 1432 992 S 1 0.0 4:39.03 htop
412 daemon 20 0 100m 7008 3544 S 0 0.1 0:00.82 httpd
707 daemon 20 0 101m 7016 3544 S 0 0.1 0:00.57 httpd
2516 daemon 20 0 100m 6252 2948 S 0 0.1 0:00.03 httpd
here is my top tree view. I can' t remember anything that i made changes on th server.
thank you.
Asterisk version: 1.2.26.2
VERSION: 2.0.5-174
BUILD: 90522-0506
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
8GB Ram
Re: perl command consuming 100% cpu usage on TOP
Posted:
Tue May 01, 2012 9:24 am
by mcargile
run 'ps -ef' and look for the pid of that perl process. This will actually tell you what perl script is running. Post that and we can go from there.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon May 07, 2012 4:05 pm
by richardroi
hello mcargile.
'ran ps -ef'... daemon 8219 1 99 May07 ? 13:27:07 init
this is the pid that is causing my high cpu usage. I was observing it and if i am correct this perl always run every Monday. Because after restarting it last Tuesday it disappear. just showed up again this Monday afternoon. I will restart it again after my shift, and will report to forum for results.
Thank you.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon May 07, 2012 4:20 pm
by mcargile
Please post the complete output of ps -ef and place it inside a code block so everything lines up like this, but the whole thing:
- Code: Select all
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May03 ? 00:00:02 init [5]
root 2 0 0 May03 ? 00:00:00 [kthreadd]
root 3 2 0 May03 ? 00:00:01 [migration/0]
root 4 2 0 May03 ? 00:00:02 [ksoftirqd/0]
root 5 2 0 May03 ? 00:00:00 [watchdog/0]
root 6 2 0 May03 ? 00:00:01 [migration/1]
root 7 2 0 May03 ? 00:01:34 [ksoftirqd/1]
root 8 2 0 May03 ? 00:00:00 [watchdog/1]
root 9 2 0 May03 ? 00:00:01 [migration/2]
root 10 2 0 May03 ? 00:00:03 [ksoftirqd/2]
root 11 2 0 May03 ? 00:00:00 [watchdog/2]
root 12 2 0 May03 ? 00:00:01 [migration/3]
root 13 2 0 May03 ? 00:00:10 [ksoftirqd/3]
root 14 2 0 May03 ? 00:00:00 [watchdog/3]
Also what is this running on? Vicibox, GoAutoDial, or something else?
The reason I ask is because init is the parent process to everything. It should have a PID of 1. The init you are listing seems to have a pid of 8219.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Mon May 07, 2012 4:26 pm
by richardroi
- Code: Select all
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May01 ? 00:00:03 /sbin/init
root 2 0 0 May01 ? 00:00:00 [kthreadd]
root 3 2 0 May01 ? 00:00:00 [migration/0]
root 4 2 0 May01 ? 00:00:21 [ksoftirqd/0]
root 5 2 0 May01 ? 00:00:00 [watchdog/0]
root 6 2 0 May01 ? 00:00:01 [migration/1]
root 7 2 0 May01 ? 00:00:26 [ksoftirqd/1]
root 8 2 0 May01 ? 00:00:00 [watchdog/1]
root 9 2 0 May01 ? 00:00:00 [migration/2]
root 10 2 0 May01 ? 00:00:20 [ksoftirqd/2]
root 11 2 0 May01 ? 00:00:00 [watchdog/2]
root 12 2 0 May01 ? 00:00:01 [migration/3]
root 13 2 0 May01 ? 00:00:40 [ksoftirqd/3]
root 14 2 0 May01 ? 00:00:00 [watchdog/3]
root 15 2 0 May01 ? 00:00:02 [events/0]
root 16 2 0 May01 ? 00:00:04 [events/1]
root 17 2 0 May01 ? 00:00:02 [events/2]
root 18 2 0 May01 ? 00:00:02 [events/3]
root 19 2 0 May01 ? 00:00:00 [khelper]
root 54 2 0 May01 ? 00:00:01 [kblockd/0]
root 55 2 0 May01 ? 00:00:01 [kblockd/1]
root 56 2 0 May01 ? 00:00:01 [kblockd/2]
root 57 2 0 May01 ? 00:00:01 [kblockd/3]
root 60 2 0 May01 ? 00:00:00 [kacpid]
root 61 2 0 May01 ? 00:00:00 [kacpi_notify]
root 158 2 0 May01 ? 00:00:00 [kseriod]
root 216 2 0 May01 ? 00:00:00 [pdflush]
root 217 2 0 May01 ? 00:00:15 [pdflush]
root 218 2 0 May01 ? 00:00:00 [kswapd0]
root 261 2 0 May01 ? 00:00:00 [aio/0]
root 262 2 0 May01 ? 00:00:00 [aio/1]
root 263 2 0 May01 ? 00:00:00 [aio/2]
root 264 2 0 May01 ? 00:00:00 [aio/3]
root 1474 2 0 May01 ? 00:00:00 [ksuspend_usbd]
root 1475 2 0 May01 ? 00:00:00 [khubd]
root 1553 2 0 May01 ? 00:00:00 [ata/0]
root 1554 2 0 May01 ? 00:00:00 [ata/1]
root 1555 2 0 May01 ? 00:00:00 [ata/2]
root 1556 2 0 May01 ? 00:00:00 [ata/3]
root 1557 2 0 May01 ? 00:00:00 [ata_aux]
root 2282 2 0 May01 ? 00:00:00 [scsi_eh_0]
root 2283 2 0 May01 ? 00:00:00 [scsi_eh_1]
root 2305 2 0 May01 ? 00:00:00 [scsi_eh_2]
root 2306 2 0 May01 ? 00:00:00 [scsi_eh_3]
root 2659 2 0 May01 ? 00:00:55 [kjournald]
root 2815 1 0 May01 ? 00:00:00 /sbin/udevd --daemon
root 3461 2 0 May01 ? 00:00:03 [edac-poller]
root 3488 2 0 May01 ? 00:00:00 [kpsmoused]
daemon 4374 5310 0 04:06 ? 00:00:04 /usr/local/apache2/bin/httpd -k
ntp 4400 1 0 May01 ? 00:00:11 /usr/sbin/ntpd -p /var/run/ntpd.
root 4419 2 0 May01 ? 00:00:23 [kjournald]
root 4420 2 0 May01 ? 00:00:06 [kjournald]
root 4421 2 0 May01 ? 00:00:00 [kjournald]
root 4772 1 0 May01 tty4 00:00:00 /sbin/getty 38400 tty4
root 4773 1 0 May01 tty5 00:00:00 /sbin/getty 38400 tty5
root 4775 1 0 May01 tty2 00:00:00 /sbin/getty 38400 tty2
root 4777 1 0 May01 tty3 00:00:00 /sbin/getty 38400 tty3
root 4779 1 0 May01 tty6 00:00:00 /sbin/getty 38400 tty6
syslog 4825 1 0 May01 ? 00:00:14 /sbin/syslogd -u syslog
root 4844 1 0 May01 ? 00:00:00 /bin/dd bs 1 if /proc/kmsg of /v
klog 4846 1 0 May01 ? 00:00:00 /sbin/klogd -P /var/run/klogd/km
root 4961 1 0 May01 ? 00:00:04 /usr/sbin/sshd
root 5017 1 0 May01 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql 5042 5017 2 May01 ? 04:27:13 /usr/sbin/mysqld --basedir=/usr
root 5043 5017 0 May01 ? 00:00:00 logger -p daemon.err -t mysqld_s
root 5104 1 0 May01 ? 00:00:00 /usr/sbin/vsftpd
root 5162 1 0 May01 ? 00:00:00 /usr/bin/SCREEN -S astshell
root 5163 5162 0 May01 pts/1 00:00:00 /bin/sh
root 5169 1 0 May01 ? 00:00:06 SCREEN -L -S asterisk
root 5170 5169 0 May01 pts/2 00:00:00 /bin/sh
root 5177 5170 11 May01 pts/2 18:25:50 /usr/sbin/asterisk -vvvvvvvvvvvv
root 5187 5177 0 May01 pts/2 00:00:13 mpg123 -q -s --mono -r 8000 -b 2
root 5189 5187 0 May01 pts/2 00:00:00 mpg123 -q -s --mono -r 8000 -b 2
root 5217 1 0 May01 ? 00:00:08 /usr/bin/SCREEN -d -m -S ASTupda
root 5218 5217 0 May01 pts/0 01:08:52 /usr/bin/perl /usr/share/astguic
root 5220 1 0 May01 ? 00:01:08 /usr/bin/SCREEN -d -m -S ASTsend
root 5221 5220 0 May01 pts/3 00:19:06 /usr/bin/perl /usr/share/astguic
root 5223 1 0 May01 ? 00:00:07 /usr/bin/SCREEN -d -m -S ASTlist
root 5224 5223 0 May01 pts/4 00:01:27 /usr/bin/perl /usr/share/astguic
root 5226 1 0 May01 ? 00:00:29 /usr/bin/SCREEN -d -m -S ASTVDau
root 5227 5226 0 May01 pts/5 00:09:55 /usr/bin/perl /usr/share/astguic
root 5229 1 0 May01 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTVDre
root 5230 5229 0 May01 pts/6 00:03:39 /usr/bin/perl /usr/share/astguic
root 5232 1 0 May01 ? 00:00:13 /usr/bin/SCREEN -d -m -S ASTVDad
root 5233 5232 0 May01 pts/7 00:06:03 /usr/bin/perl /usr/share/astguic
root 5235 1 0 May01 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTfast
root 5236 5235 0 May01 pts/8 00:00:07 /usr/bin/perl /usr/share/astguic
daemon 5258 1 0 May01 ? 00:00:00 /usr/sbin/atd
root 5269 1 0 May01 ? 00:00:06 /usr/sbin/cron
root 5310 1 0 May01 ? 00:00:12 /usr/local/apache2/bin/httpd -k
root 5316 1 0 May01 tty1 00:00:00 /bin/login --
root 5424 5316 0 May01 tty1 00:00:00 -bash
root 5657 4961 0 May01 ? 00:00:14 sshd: root@pts/9
root 5695 5657 0 May01 pts/9 00:00:00 -bash
root 5733 4961 0 May01 ? 00:00:51 sshd: root@pts/10
root 5759 5733 0 May01 pts/10 00:00:00 -bash
daemon 5848 5310 0 04:13 ? 00:00:04 /usr/local/apache2/bin/httpd -k
daemon 6372 5310 0 04:15 ? 00:00:03 /usr/local/apache2/bin/httpd -k
daemon 8203 24239 0 May07 ? 00:00:00 [sh] <defunct>
daemon 8219 1 99 May07 ? 13:51:19 init
daemon 8855 5310 0 04:28 ? 00:00:03 /usr/local/apache2/bin/httpd -k
root 9123 5177 0 May07 pts/2 00:00:01 mpg123 -q -s --mono -r 8000 -b 2
root 9124 9123 0 May07 pts/2 00:00:00 mpg123 -q -s --mono -r 8000 -b 2
daemon 9943 5310 0 04:34 ? 00:00:03 /usr/local/apache2/bin/httpd -k
daemon 10753 5310 0 04:38 ? 00:00:02 /usr/local/apache2/bin/httpd -k
daemon 10972 5310 0 04:39 ? 00:00:02 /usr/local/apache2/bin/httpd -k
root 11207 5236 0 02:11 pts/8 00:00:00 /usr/bin/perl /usr/share/astguic
daemon 11564 5310 0 04:42 ? 00:00:02 /usr/local/apache2/bin/httpd -k
daemon 13786 5310 0 04:54 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14248 5310 0 04:56 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14249 5310 0 04:56 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14357 5310 0 04:57 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14358 5310 0 04:57 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14769 5310 0 04:59 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 14777 5310 0 04:59 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 15180 5310 0 02:28 ? 00:00:12 /usr/local/apache2/bin/httpd -k
daemon 15648 5310 0 05:03 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 15656 5310 0 05:03 ? 00:00:01 /usr/local/apache2/bin/httpd -k
daemon 15954 5310 0 05:05 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 16091 5310 0 05:05 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 16226 5310 0 05:06 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 16426 5310 0 05:07 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 16429 5310 0 05:07 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 16899 5310 0 05:09 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17010 5310 0 05:09 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17011 5310 0 05:09 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17163 5310 0 05:10 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17829 5310 0 05:14 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17830 5310 0 05:14 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17867 5310 0 05:14 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17982 5310 0 05:14 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 17987 5310 0 05:14 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 18125 5310 0 05:15 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 18882 5310 0 05:19 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 18887 5310 0 05:19 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 19033 5310 0 05:20 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 19036 5310 0 05:20 ? 00:00:00 /usr/local/apache2/bin/httpd -k
daemon 19037 5310 0 05:20 ? 00:00:00 /usr/local/apache2/bin/httpd -k
root 19328 5269 0 05:22 ? 00:00:00 /USR/SBIN/CRON
root 19329 19328 0 05:22 ? 00:00:00 /bin/sh -c /usr/share/astguiclie
root 19330 19329 0 05:22 ? 00:00:00 /usr/bin/perl /usr/share/astguic
root 19389 4961 0 05:22 ? 00:00:00 sshd: root@pts/11
root 19404 19389 0 05:22 pts/11 00:00:00 -bash
root 19421 19404 0 05:22 pts/11 00:00:00 ps -ef
root 21187 5236 0 02:56 pts/8 00:00:00 /usr/bin/perl /usr/share/astguic
root 23067 5236 0 03:05 pts/8 00:00:00 /usr/bin/perl /usr/share/astguic
daemon 23272 5310 0 03:05 ? 00:00:08 /usr/local/apache2/bin/httpd -k
daemon 24239 5310 0 May07 ? 00:00:04 /usr/local/apache2/bin/httpd -k
root 24833 5236 0 03:12 pts/8 00:00:00 /usr/bin/perl /usr/share/astguic
root 26845 5424 0 May01 tty1 00:21:11 top
daemon 28784 5310 0 03:31 ? 00:00:07 /usr/local/apache2/bin/httpd -k
root 29896 5759 0 03:35 pts/10 00:00:57 htop
root 30124 5236 0 01:10 pts/8 00:00:00 /usr/bin/perl /usr/share/astguic
root 31227 5695 0 May07 pts/9 00:00:00 rasterisk r
This is an old vicidial (ubuntu). We've been using it for 3 years now.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Tue May 08, 2012 10:20 am
by richardroi
hello... I restarted the server this morning, and everything is back to normal. will continue monitoring it. I remember restarting it Monday morning(last week) and it came back around 12mn that day, then restarted it again Tuesday morning and it went fine the whole week, it just showed up again yesterday (Monday afternoon). The only thing that bothers me is maybe our server is being idle on weekends, because on weekends there are few calls. Before we are restarting it once a month.
Re: perl command consuming 100% cpu usage on TOP
Posted:
Tue May 08, 2012 6:27 pm
by richardroi
It showed up again this morning. Can someone help me with this issue?