vicidial.org

by **rrb555** » Tue May 08, 2012 8:48 pm

Hi,

vicibox installer 3.1.15
VERSION: 2.6-365a
BUILD: 120420-1620

I am experiencing high cpu usage on wwwrun. I recently upgraded my vicidial version to the latest svn trunk and it began to experience high cpu usage.

some help please?

Code: Select all: UID PID PPID C STIME TTY TIME CMD root 1 0 0 21:08 ? 00:00:01 init [5] root 2 0 0 21:08 ? 00:00:00 [kthreadd] root 3 2 0 21:08 ? 00:00:00 [migration/0] root 4 2 0 21:08 ? 00:00:00 [ksoftirqd/0] root 5 2 0 21:08 ? 00:00:00 [watchdog/0] root 6 2 0 21:08 ? 00:00:00 [migration/1] root 7 2 0 21:08 ? 00:00:00 [ksoftirqd/1] root 8 2 0 21:08 ? 00:00:00 [watchdog/1] root 9 2 0 21:08 ? 00:00:00 [migration/2] root 10 2 0 21:08 ? 00:00:00 [ksoftirqd/2] root 11 2 0 21:08 ? 00:00:00 [watchdog/2] root 12 2 0 21:08 ? 00:00:00 [migration/3] root 13 2 0 21:08 ? 00:00:00 [ksoftirqd/3] root 14 2 0 21:08 ? 00:00:00 [watchdog/3] root 15 2 0 21:08 ? 00:00:00 [migration/4] root 16 2 0 21:08 ? 00:00:00 [ksoftirqd/4] root 17 2 0 21:08 ? 00:00:00 [watchdog/4] root 18 2 0 21:08 ? 00:00:00 [migration/5] root 19 2 0 21:08 ? 00:00:00 [ksoftirqd/5] root 20 2 0 21:08 ? 00:00:00 [watchdog/5] root 21 2 0 21:08 ? 00:00:00 [migration/6] root 22 2 0 21:08 ? 00:00:00 [ksoftirqd/6] root 23 2 0 21:08 ? 00:00:00 [watchdog/6] root 24 2 0 21:08 ? 00:00:00 [migration/7] root 25 2 0 21:08 ? 00:00:00 [ksoftirqd/7] root 26 2 0 21:08 ? 00:00:00 [watchdog/7] root 27 2 0 21:08 ? 00:00:00 [events/0] root 28 2 0 21:08 ? 00:00:00 [events/1] root 29 2 0 21:08 ? 00:00:00 [events/2] root 30 2 0 21:08 ? 00:00:00 [events/3] root 31 2 0 21:08 ? 00:00:00 [events/4] root 32 2 0 21:08 ? 00:00:00 [events/5] root 33 2 0 21:08 ? 00:00:00 [events/6] root 34 2 0 21:08 ? 00:00:00 [events/7] root 35 2 0 21:08 ? 00:00:00 [cpuset] root 36 2 0 21:08 ? 00:00:00 [netns] root 37 2 0 21:08 ? 00:00:00 [async/mgr] root 38 2 0 21:08 ? 00:00:00 [pm] root 39 2 0 21:08 ? 00:00:00 [sync_supers] root 40 2 0 21:08 ? 00:00:00 [bdi-default] root 41 2 0 21:08 ? 00:00:00 [kintegrityd/0] root 42 2 0 21:08 ? 00:00:00 [kintegrityd/1] root 43 2 0 21:08 ? 00:00:00 [kintegrityd/2] root 44 2 0 21:08 ? 00:00:00 [kintegrityd/3] root 45 2 0 21:08 ? 00:00:00 [kintegrityd/4] root 46 2 0 21:08 ? 00:00:00 [kintegrityd/5] root 47 2 0 21:08 ? 00:00:00 [kintegrityd/6] root 48 2 0 21:08 ? 00:00:00 [kintegrityd/7] root 49 2 0 21:08 ? 00:00:00 [kblockd/0] root 50 2 0 21:08 ? 00:00:00 [kblockd/1] root 51 2 0 21:08 ? 00:00:00 [kblockd/2] root 52 2 0 21:08 ? 00:00:00 [kblockd/3] root 53 2 0 21:08 ? 00:00:00 [kblockd/4] root 54 2 0 21:08 ? 00:00:00 [kblockd/5] root 55 2 0 21:08 ? 00:00:00 [kblockd/6] root 56 2 0 21:08 ? 00:00:00 [kblockd/7] root 57 2 0 21:08 ? 00:00:00 [kacpid] root 58 2 0 21:08 ? 00:00:00 [kacpi_notify] root 59 2 0 21:08 ? 00:00:00 [kacpi_hotplug] root 60 2 0 21:08 ? 00:00:00 [kseriod] root 69 2 0 21:08 ? 00:00:00 [kondemand/0] root 70 2 0 21:08 ? 00:00:00 [kondemand/1] root 71 2 0 21:08 ? 00:00:00 [kondemand/2] root 72 2 0 21:08 ? 00:00:00 [kondemand/3] root 73 2 0 21:08 ? 00:00:00 [kondemand/4] root 74 2 0 21:08 ? 00:00:00 [kondemand/5] root 75 2 0 21:08 ? 00:00:00 [kondemand/6] root 76 2 0 21:08 ? 00:00:00 [kondemand/7] root 77 2 0 21:08 ? 00:00:00 [khelper] root 78 2 0 21:08 ? 00:00:00 [khungtaskd] root 79 2 0 21:08 ? 00:00:00 [kswapd0] root 80 2 0 21:08 ? 00:00:00 [ksmd] root 81 2 0 21:08 ? 00:00:00 [aio/0] root 82 2 0 21:08 ? 00:00:00 [aio/1] root 83 2 0 21:08 ? 00:00:00 [aio/2] root 84 2 0 21:08 ? 00:00:00 [aio/3] root 85 2 0 21:08 ? 00:00:00 [aio/4] root 86 2 0 21:08 ? 00:00:00 [aio/5] root 87 2 0 21:08 ? 00:00:00 [aio/6] root 88 2 0 21:08 ? 00:00:00 [aio/7] root 89 2 0 21:08 ? 00:00:00 [crypto/0] root 90 2 0 21:08 ? 00:00:00 [crypto/1] root 91 2 0 21:08 ? 00:00:00 [crypto/2] root 92 2 0 21:08 ? 00:00:00 [crypto/3] root 93 2 0 21:08 ? 00:00:00 [crypto/4] root 94 2 0 21:08 ? 00:00:00 [crypto/5] root 95 2 0 21:08 ? 00:00:00 [crypto/6] root 96 2 0 21:08 ? 00:00:00 [crypto/7] root 98 2 0 21:08 ? 00:00:00 [kpsmoused] root 293 2 0 21:08 ? 00:00:00 [khubd] root 307 2 0 21:08 ? 00:00:00 [ata/0] root 308 2 0 21:08 ? 00:00:00 [ata/1] root 309 2 0 21:08 ? 00:00:00 [ata/2] root 310 2 0 21:08 ? 00:00:00 [ata/3] root 311 2 0 21:08 ? 00:00:00 [ata/4] root 312 2 0 21:08 ? 00:00:00 [ata/5] root 313 2 0 21:08 ? 00:00:00 [ata/6] root 314 2 0 21:08 ? 00:00:00 [ata/7] root 315 2 0 21:08 ? 00:00:00 [ata_aux] root 364 2 0 21:08 ? 00:00:00 [scsi_eh_0] root 365 2 0 21:08 ? 00:00:00 [scsi_eh_1] root 376 2 0 21:08 ? 00:00:00 [scsi_eh_2] root 377 2 0 21:08 ? 00:00:00 [scsi_eh_3] root 381 2 0 21:08 ? 00:00:00 [i915] root 461 2 0 21:08 ? 00:00:00 [kjournald] root 527 1 0 21:09 ? 00:00:00 /sbin/udevd --daemon root 710 2 0 21:09 ? 00:00:00 [hd-audio0] root 711 2 0 21:09 ? 00:00:00 [khpsbpkt] root 729 2 0 21:09 ? 00:00:00 [knodemgrd_0] root 837 2 0 21:09 ? 00:00:00 [flush-8:0] root 851 2 0 21:09 ? 00:00:00 [kstriped] root 1272 1 0 21:09 ? 00:00:00 /sbin/acpid 100 1294 1 0 21:09 ? 00:00:00 /bin/dbus-daemon --system root 2497 1 0 21:09 ? 00:00:00 /sbin/rsyslogd -c 5 -f /etc/rsys root 2524 1 0 21:09 ? 00:00:00 /sbin/rpcbind root 2608 1 0 21:09 ? 00:00:00 /usr/sbin/irqbalance root 2672 1 0 21:09 ? 00:00:00 /usr/sbin/sshd -o PidFile=/var/r root 2693 2 0 21:09 ? 00:00:00 [kconservative/0] root 2694 2 0 21:09 ? 00:00:00 [kconservative/1] root 2695 2 0 21:09 ? 00:00:00 [kconservative/2] root 2696 2 0 21:09 ? 00:00:00 [kconservative/3] root 2697 2 0 21:09 ? 00:00:00 [kconservative/4] root 2698 2 0 21:09 ? 00:00:00 [kconservative/5] root 2699 2 0 21:09 ? 00:00:00 [kconservative/6] root 2700 2 0 21:09 ? 00:00:00 [kconservative/7] 335 2702 1 0 21:09 ? 00:00:00 /usr/sbin/hald --daemon=yes root 2704 1 0 21:09 ? 00:00:00 /usr/sbin/console-kit-daemon --n root 2769 2702 0 21:09 ? 00:00:00 hald-runner root 2798 2769 0 21:09 ? 00:00:00 hald-addon-input: Listening on / root 2813 2769 0 21:09 ? 00:00:00 hald-addon-storage: polling /dev root 2814 2769 0 21:09 ? 00:00:00 /usr/lib/hal/hald-addon-cpufreq 335 2815 2769 0 21:09 ? 00:00:00 hald-addon-acpi: listening on ac root 3198 1 0 21:09 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --m mysql 3370 3198 4 21:09 ? 00:01:30 /usr/sbin/mysqld --basedir=/usr ntp 3417 1 0 21:09 ? 00:00:00 /usr/sbin/ntpd -p /var/run/ntp/n root 3430 1 0 21:09 ? 00:00:00 /usr/sbin/nscd root 3558 527 0 21:09 ? 00:00:00 /sbin/udevd --daemon root 3559 527 0 21:09 ? 00:00:00 /sbin/udevd --daemon root 3564 1 0 21:09 ? 00:00:00 /usr/lib/postfix/master postfix 3581 3564 0 21:09 ? 00:00:00 pickup -l -t fifo -u postfix 3582 3564 0 21:09 ? 00:00:00 qmgr -l -t fifo -u root 3619 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -S astshell20120 root 3620 3619 0 21:09 pts/2 00:00:00 /bin/sh root 3624 1 0 21:09 ? 00:00:00 SCREEN -L -S asterisk root 3625 3624 0 21:09 pts/3 00:00:00 /bin/sh root 3630 3625 1 21:09 pts/3 00:00:34 /usr/sbin/asterisk -vvvvvvvvvvvv root 3709 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTupda root 3711 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTsend root 3713 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTlist root 3715 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTVDau root 3717 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTVDre root 3719 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTVDad root 3721 1 0 21:09 ? 00:00:00 /usr/bin/SCREEN -d -m -S ASTfast root 3722 3709 0 21:09 pts/0 00:00:11 /usr/bin/perl /usr/share/astguic root 3723 3711 0 21:09 pts/4 00:00:04 /usr/bin/perl /usr/share/astguic root 3724 3713 0 21:09 pts/5 00:00:02 /usr/bin/perl /usr/share/astguic root 3727 3715 0 21:09 pts/6 00:00:03 /usr/bin/perl /usr/share/astguic root 3728 3719 0 21:09 pts/8 00:00:15 /usr/bin/perl /usr/share/astguic root 3729 3717 0 21:09 pts/7 00:00:00 /usr/bin/perl /usr/share/astguic root 3730 3721 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3731 1 0 21:09 ? 00:00:00 ip_relay 40569 127.0.0.1 4569 99 root 3732 1 0 21:09 ? 00:00:00 ip_relay 41569 127.0.0.1 4569 99 root 3747 3730 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3748 3730 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3749 3730 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3750 3730 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3751 3730 0 21:09 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic root 3799 1 0 21:09 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc root 3822 1 0 21:09 ? 00:00:00 /usr/sbin/cron root 3974 1 0 21:09 tty1 00:00:00 /sbin/mingetty --noclear tty1 root 3975 1 0 21:09 tty2 00:00:00 /sbin/mingetty tty2 root 3976 1 0 21:09 tty3 00:00:00 /sbin/mingetty tty3 root 3977 1 0 21:09 tty4 00:00:00 /sbin/mingetty tty4 root 3978 1 0 21:09 tty5 00:00:00 /sbin/mingetty tty5 root 3979 1 0 21:09 tty6 00:00:00 /sbin/mingetty tty6 wwwrun 3999 3799 0 21:09 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc root 4016 2 0 21:10 ? 00:00:00 [kauditd] wwwrun 4040 1 97 21:10 ? 00:34:43 vicidialweb root 4084 2672 0 21:10 ? 00:00:00 sshd: root@pts/1 root 4095 4084 0 21:10 pts/1 00:00:00 -bash wwwrun 4699 3799 0 21:14 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 5249 3799 0 21:17 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 5738 3799 0 21:20 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 5739 3799 0 21:20 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 6451 3799 0 21:23 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 6758 3799 0 21:24 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 6923 3799 0 21:25 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc wwwrun 7378 3799 0 21:26 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 7684 3799 0 21:27 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 7687 3799 0 21:27 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc root 7823 3730 0 21:27 pts/9 00:00:00 /usr/bin/perl /usr/share/astguic wwwrun 7876 3799 0 21:28 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 8392 3799 0 21:29 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 8398 3799 0 21:29 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 9531 3799 0 21:32 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 9813 3799 0 21:33 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 9850 3799 0 21:33 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10404 3799 0 21:35 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10436 3799 0 21:35 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10582 3799 0 21:36 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10724 3799 0 21:36 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10970 3799 0 21:37 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10974 3799 0 21:37 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 10975 3799 0 21:37 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 11145 3799 0 21:38 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 11481 3799 0 21:39 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 11489 3799 0 21:39 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 12509 3799 0 21:42 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 12889 3799 0 21:42 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc wwwrun 13652 3799 0 21:44 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc root 13668 3822 0 21:45 ? 00:00:00 /usr/sbin/cron root 13679 13668 0 21:45 ? 00:00:00 /usr/bin/perl /usr/share/astguic root 13913 1 0 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13917 1 0 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13919 1 0 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13987 1 2 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13989 1 3 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13991 1 2 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13996 1 3 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 13998 1 2 21:45 pts/4 00:00:00 /usr/bin/perl /usr/share/astguic root 14004 4095 0 21:45 pts/1 00:00:00 ps -ef

thanks in advance

by **williamconley** » Wed May 09, 2012 2:35 pm

in signature: goautodial 2.1
in statement: vicibox 3.1.15
also in statement: recently upgraded ...

You can't upgrade from GoAutoDial to Vicibox. You can upgrade the Vicidial in a GoAuto to a more recent Vicidial, but this will have nothing to do with Vicibox.

Also of note: Do you realize that you have upgraded to an alpha of Vicidial 2.6? You were likely at 2.4 before, but Vicidial trunk is now on the next version ...you may want to stick with Vicidial 2.4RC1 until someone says 2.6 is stable. Just a thought.

I've never bumped into "vicidialweb". What happens if you kill the process? Is it in crontab somewhere?

by **rrb555** » Wed May 09, 2012 2:59 pm

signature update.

Also of note: Do you realize that you have upgraded to an alpha of Vicidial 2.6? You were likely at 2.4 before, but Vicidial trunk is now on the next version ...you may want to stick with Vicidial 2.4RC1 until someone says 2.6 is stable. Just a thought.

I have just updated my server to the latest SVN thinking using the latest SVN is the best. as far as it goes i didn't get any issues, thankfully

I didn't though of that actually, with the alpha version of VIcidial 2.6. hmm is there's a way to update only to the 2.4 stable version or downgrade? or would safe to say if i just wait for the Vicibox 4.0 installer?

I've never bumped into "vicidialweb". What happens if you kill the process? Is it in crontab somewhere?

never tried killing the process? is it safe? if i encounter any problem would a simple reboot will get that back up again?

by **williamconley** » Fri May 11, 2012 12:25 am

yep. killing any process is resolved by a reboot. processes all get killed during a reboot anyway.

svn has a new branch for 2.4. instead of trunk, use that. it will never accidentally land you in 2.6. but if you're having a good time with 2.6, keep it!

by **Noah** » Fri May 18, 2012 11:20 pm

Top line scrape
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5727 wwwrun 21 1 5528 3220 1060 S 16 0.1 11:12.82 .vicidial

This is taking quite a bit of resource when no one is logged in. Consistently 16 % on a quad core sys

Here's the lsof | grep 5727

lsof | grep 5727
asterisk 2426 root mem REG 8,3 7566 1135727 /usr/lib/asterisk/modules/func_uri.so
.vicidial 5727 wwwrun cwd DIR 8,3 4096 2 /
.vicidial 5727 wwwrun rtd DIR 8,3 4096 2 /
.vicidial 5727 wwwrun txt REG 8,3 1130350 130586 /tmp/.vicidial
.vicidial 5727 wwwrun mem REG 8,3 54467 318386 /lib/libnss_files-2.11.2.so
.vicidial 5727 wwwrun DEL REG 8,3 138724 /tmp/p2xtmp-5719/Socket.so
.vicidial 5727 wwwrun mem REG 8,3 55574 320391 /lib/libcrypt-2.11.2.so
.vicidial 5727 wwwrun mem REG 8,3 1670857 320373 /lib/libc-2.11.2.so
.vicidial 5727 wwwrun mem REG 8,3 125115 318424 /lib/libpthread-2.11.2.so
.vicidial 5727 wwwrun mem REG 8,3 191006 320398 /lib/libm-2.11.2.so
.vicidial 5727 wwwrun mem REG 8,3 17392 318439 /lib/libdl-2.11.2.so
.vicidial 5727 wwwrun mem REG 8,3 107282 320376 /lib/libnsl-2.11.2.so
.vicidial 5727 wwwrun DEL REG 8,3 138723 /tmp/p2xtmp-5719/IO.so
.vicidial 5727 wwwrun mem REG 8,3 143978 320417 /lib/ld-2.11.2.so
.vicidial 5727 wwwrun 0r CHR 1,3 0t0 893 /dev/null
.vicidial 5727 wwwrun 1w CHR 1,3 0t0 893 /dev/null
.vicidial 5727 wwwrun 2w CHR 1,3 0t0 893 /dev/null
.vicidial 5727 wwwrun 3u sock 0,6 94443395 can't identify protocol

Here's the last entry in "tail -F /var/log/apache2/error_log"

[Fri May 18 16:37:08 2012] [error] [client 209.42.35.46] PHP Warning: mysql_num_rows() expects parameter 1 to be resource, boolean given in /srv/www/htdocs/vicidial/admin.php on line 23008, referer:

Nothing relevant to the time of the TOP or the lsof | grep PID

NOT 100% sure this is a bug due to the intermittent nature of the proc usage.
But there is some suspect activity surrounding this process.

Kill the PID and it creeps back in. No doubt a cron is initiating this someplace.
\

by **williamconley** » Sat May 19, 2012 9:30 am

So find the cron and kill it.

Code: Select all: crontab -e nano /etc/crontab

by **rrb555** » Sun May 20, 2012 8:05 am

@Noah

Are u able to resolve this?

let me know

thanks!

by **Noah** » Wed May 23, 2012 3:28 pm

Haven't been able to find the cron that launches this process - Any other thoughts on how to track this down?

by **Noah** » Fri May 25, 2012 11:37 am

strace -p PID

close(3) = 0
stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=884, ...}) = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
send(3, "\2\0\0\0\4\0\0\0\25\0\0\0clouds.themafia.info"..., 33, MSG_NOSIGNAL) = 33
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=3, revents=POLLIN|POLLHUP}])
read(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32

Any ideas?

by **Noah** » Fri May 25, 2012 12:39 pm

Found it !!!
Vulnerability /tmp is drwxrwxrwt this allows files to be placed, not sure how these got here yet. Two scripts were placed in this directory

-rwxr-xr-x 1 wwwrun www 115 May 8 17:28 .vicicheck*
-rwxr-xr-x 1 wwwrun www 1130350 May 8 17:29 .vicidial*

To find it first:
tail -F /var/log/messages
May 25 11:07:01 v5 /usr/sbin/cron[24107]: (root) CMD (/usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3)
May 25 11:07:01 v5 /usr/sbin/cron[24111]: (wwwrun) CMD (/tmp/.vicicheck >/dev/null 2>&1)
May 25 11:07:01 v5 /usr/sbin/cron[24109]: (root) CMD (/usr/share/astguiclient/ADMIN_keepalive_ALL.pl)
May 25 11:07:01 v5 /usr/sbin/cron[24112]: (root) CMD (/usr/share/astguiclient/AST_manager_kill_hung_congested.pl)
May 25 11:08:01 v5 /usr/sbin/cron[11755]: (root) CMD (/usr/share/astguiclient/ADMIN_keepalive_ALL.pl)
May 25 11:08:01 v5 /usr/sbin/cron[11756]: (root) CMD (/usr/share/astguiclient/AST_manager_kill_hung_congested.pl)
May 25 11:08:01 v5 /usr/sbin/cron[11757]: (root) CMD (/usr/share/astguiclient/AST_conf_update.pl)
May 25 11:08:01 v5 /usr/sbin/cron[11761]: (wwwrun) CMD (/tmp/.vicicheck >/dev/null 2>&1)

Second: looks like wwwrun is allowed to create cron I'll be locking that down momentarily
nav to /var/spool/cron/tabs

And if wwwrun has a cron vi wwwrun and edit the cron to block the running of the script by placing a # in front of the * * * * *

htop or top and Kill the PID for this vicidialweb process

then remove the cron for wwwrun
rm wwwrun

Cd to /tmp

rm .vicicheck*
rm .vicidial*

-rwxr-xr-x 1 wwwrun www 115 May 8 17:28 .vicicheck*
-rwxr-xr-x 1 wwwrun www 1130350 May 8 17:29 .vicidial*

Lastly:
Monitor htop to see if the cron executed or hopefully has stopped at this point and hasn't re-executed while you've been making changes.

The contents of the .vicicheck* file
#!/bin/sh
if ps ax | grep -v grep | grep "vicidialweb" > /dev/null
then
echo " "
else
/tmp/.vicidial &
fi

This launches a compiled script that attempts to push traffic out to multiple ipaddress. I have the script in a sterile environment if anyone would like to check it out PM me and I'll zip and email it to you.

by **Noah** » Fri May 25, 2012 12:55 pm

I then recommend editing /etc/cron.deny and adding wwwrun

by **rrb555** » Fri May 25, 2012 2:12 pm

i also have those files to which my other server doesn't.

is it safe to remove .vicicheck* and rm .vicidial*?

anyone?

by **fibres** » Fri May 25, 2012 7:45 pm

This is slightly worrying. How did these files get here and furthermore how was someone able to place things into the crontab?

This seems to be some kind of exploit.

Regards

by **mflorell** » Sat May 26, 2012 6:39 am

This was an exploit that was reported earlier this week, we added code to prevent it yesterday in SVN trunk and branches/2.4.

There is also another exploit that we fixed 3 months ago that aids in this exploit as well, we strongly recommend upgrading your system to the most recent svn trunk or 2.4 branch as well as changing your manager passwords through the web interface.

by **rrb555** » Wed Jul 18, 2012 7:25 am

Hi Matt,

VERSION: 2.6-372a
BUILD: 120713-2123

I updated my server to revision 1840 and seems like there are some exploit arising again. My server is experiencing high CPU process wwwrun again.

When I viewed the logs /var/log/messages i found this
Jul 18 08:22:01 Vicidial /usr/sbin/cron[9674]: (wwwrun) CMD (/tmp/.x/.sh3ll/update >/dev/null 2>&1)

would you happen to know what cause this?

when i viewed the directory i can see this

Code: Select all: Vicidial:/tmp/.x/.sh3ll # l total 332 drwxr-xr-x 2 wwwrun www 4096 Jul 18 08:30 ./ drwxr-xr-x 3 wwwrun www 4096 Jul 18 08:11 ../ -rw-r--r-- 1 wwwrun www 49 Jul 18 08:30 cron.d -rw-r--r-- 1 wwwrun www 15 Jul 18 08:11 dir.dir -rwx--x--x 1 wwwrun www 14679 Feb 6 2011 f4* -rwxr-xr-x 1 wwwrun www 12683 May 6 06:33 hide* -rwx--x--x 1 wwwrun www 152108 Feb 6 2011 init* -rwxr-xr-x 1 wwwrun www 10848 Feb 6 2011 juno* -rw-r--r-- 1 wwwrun www 1064 Jul 18 08:27 mech.levels -rw------- 1 wwwrun www 5 Jul 18 08:29 mech.pid -rw-r--r-- 1 wwwrun www 254 Jul 18 08:27 mech.session -rwx--x--x 1 wwwrun www 462 Jul 17 23:44 mech.set* -rw-r--r-- 1 wwwrun www 0 Jul 18 08:30 pig.seen -rwxr-xr-x 1 wwwrun www 326 Feb 6 2011 run* -rwxr-xr-x 1 wwwrun www 16776 Feb 6 2011 slice* -rwx--x--x 1 wwwrun www 38 Nov 18 2011 start.sh* -rwx--x--x 1 wwwrun www 15195 Feb 6 2011 std* -rwx--x--x 1 wwwrun www 15078 Feb 6 2011 stealth* -rwxr-xr-x 1 wwwrun www 722 Mar 22 21:30 sys* -rwxr-xr-x 1 wwwrun www 15994 Feb 6 2011 talk* -rwxr-xr-x 1 wwwrun www 6204 Jul 17 23:16 timeout* -rwx--x--x 1 wwwrun www 915 Feb 6 2011 udp* -rwxr-xr-x 1 wwwrun www 183 Jul 18 08:11 update* -rwxr-xr-x 1 wwwrun www 81 Jul 18 08:27 usr*

Code: Select all: Vicidial:/tmp/.x/.sh3ll # less update #!/bin/sh if test -r /tmp/.x/.sh3ll/mech.pid; then pid=$(cat /tmp/.x/.sh3ll/mech.pid) if $(kill -CHLD $pid >/dev/null 2>&1) then#exit 0 fi fi cd /tmp/.x/.sh3ll ./start.sh &>/dev/null

is it safe to delete?
my other single server do not have any of this

by **williamconley** » Wed Jul 18, 2012 8:12 am

If you don't already have a whitelist security system set up, it may be pointless ... but whether it is safe to delete depends on the code being executed.

If it is a malicious invasion virus, it will immediately begin deleting things (possible, but unlikely). If it is merely a remote drone code (allowing your computer to become part of an international network of drones hacking into other systems while being controlled remotely), it'll just stop doing that ... but you'll be "probed" pretty soon to try to re-connect.

Safest bet is full backup, wipe it clean, reinstall with IPtables whitelist only access. Then go back to making money.

Remember that you're not even almost clean until you reboot and verify that these files do not magically reappear.

vicidial.org

High CPU Process wwwrun - vicidialweb

High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Re: High CPU Process wwwrun - vicidialweb

Who is online