vicidial.org

While booting, server get stucks at Asterisk strated...screening logging on' for 30 mins and then boots up.

Whats the solutions ?

Any Help ?

[root@go ~]# du -hs /var/log/*
44K /var/log/acpid
72K /var/log/acpid.1
72K /var/log/acpid.2
320K /var/log/anaconda.log
32K /var/log/anaconda.syslog
657M /var/log/asterisk
3.2G /var/log/astguiclient
8.0K /var/log/audit
0 /var/log/boot.log
0 /var/log/boot.log.1
0 /var/log/boot.log.2
0 /var/log/boot.log.3
0 /var/log/boot.log.4
280K /var/log/brcm-iscsi.log
58M /var/log/btmp
8.0K /var/log/conman
8.0K /var/log/conman.old
2.4M /var/log/cron
6.7M /var/log/cron.1
6.4M /var/log/cron.2
6.9M /var/log/cron.3
6.2M /var/log/cron.4
24K /var/log/dkms_autoinstaller
24K /var/log/dmesg
8.0K /var/log/faillog
1.2M /var/log/goautodial
8.3G /var/log/httpd
36K /var/log/lastlog
16K /var/log/mail
1.4M /var/log/maillog
3.8M /var/log/maillog.1
3.7M /var/log/maillog.2
3.9M /var/log/maillog.3
3.5M /var/log/maillog.4
384K /var/log/messages
308K /var/log/messages.1
396K /var/log/messages.2
504K /var/log/messages.3
432K /var/log/messages.4
13M /var/log/munin
5.1M /var/log/mysqld.log
12K /var/log/pm
8.0K /var/log/ppp
28K /var/log/prelink
20K /var/log/rpmpkgs
20K /var/log/rpmpkgs.1
20K /var/log/rpmpkgs.2
20K /var/log/rpmpkgs.3
20K /var/log/rpmpkgs.4
3.4M /var/log/sa
8.0K /var/log/scrollkeeper.log
1.3M /var/log/secure
1.9M /var/log/secure.1
592K /var/log/secure.2
2.2M /var/log/secure.3
976K /var/log/secure.4
0 /var/log/spooler
0 /var/log/spooler.1
0 /var/log/spooler.2
0 /var/log/spooler.3
0 /var/log/spooler.4
4.0K /var/log/tallylog
776K /var/log/wtmp
1.1M /var/log/wtmp.1
0 /var/log/xferlog
0 /var/log/xferlog.1
0 /var/log/xferlog.2
0 /var/log/xferlog.3
0 /var/log/xferlog.4

astguiclient and httpd can be the reason ?

You have some huge log files there. Perhaps you should remove them. Are you not running out of disc space?

No, I am Not... I have 500 GB HDD.

Can safely delete those Log file ?

Yes, you can remove the httpd and astguiclient logs. They will be recreated by their respective services.

What else can be done to reduce boot up time ?

Does it take 30 minutes to boot? If so, there is another issue causing this - perhaps check any RAID configurations in the Dell controller - one of you discs could be giving issues. You can also test which component is taking long to start by restarting Asterisk for instance:

Code: Select all

/etc/init.d/asterisk restart

There will be a log for this process starting in /var/log/asterisk/messages or /var/log/asterisk/full .

Did you try to restart asterisk? I got this once. Try to run this.

/usr/sbin/amportal stop
/usr/sbin/amportal start

If it works, then never do a "service asterisk restart"

We've had issues with DNS slowing the boot if the firewall is searching for allowable IPs but must search for the DNS name to get the IP. IF DNS fails for a name, older versions of iptables in OpenSuSE can get stuck.

However, if you are in CentOS/GoAuto, this may not be related. But I thought it worth mentioning the possibility.

Another question: Have you tried logging in via ssh during this process to see what is happening? Also check logs for the first entry after the delay to see if you can find out what, in the boot order, was happening?

If I format and Reinstall, all looks Ok and Perfect !!

But after few days of running , boot time increases !!!

perhaps you are experiencing an old ubuntu/debian issue with mysql insisting on checking all tables during startup? the older the system, the more records to check ...

It got stuck for 13 minutes in Asterisk Started....Screen Logging On

4 minutes on asterisk.live inbound agents

4 minutes on asterisk.lead log archive and they rest were normal

Vtiger part moves very quickly

you saw something about live inbound agents and a log archive ... where, exactly?

Thats becos i repair all tables on boot up....but even if i remove it...asterisk things is too much time killing....!!

consider only CHECKING to see if repair is needed on boot up (the word crashed will appear in the notes for the column structure). and then only repair if it is actually necessary.

Ok.. how about asterisk thing?takes 15 mins...

I have no idea what an "asterisk thing" is. asterisk starts in under a second. But you should be aware that the last item loaded by the system visually will "just stay there" while the rest of the actual OS (things like "cron" and "iptables") start up on various operating systems. Processes that are not itemized will continue to start even though the last shown entry may be "asterisk". If you are not sure ... modify the system to NOT start that last process, and the previous process will now be "last" ... and will take just as long.

I have had a few client boxes that had an issue with DNS. And in "iptables" security, they had a domain name (to allow a carrier to send packets to the server ...). The iptables system would take 15 minutes to "fail" to look up the IP address of the domain before finally giving up.

It could be any process in the system ... you can try turning off "services" one at a time to see which one it is (be sure you write down which ones are there before you begin so you can put it all back!).

Or just Build A New Server (after all, the software is free ...).

thats what i do when it goes too much...reinstall....

Reg dns, i use 8.8.8.8 as dns.

Where to check for different services which starts on boot up? ..

I will try to put my system on live ip and then will try to boot....as of now its nat!!!

I dont make any changes to iptables.

nat does not mean the IP is not "live". does it have DNS service available that works? (you said dns, but then said something about your ip isn't live ... i'm not sure what that meant).

you could try turning off services one at a time in whatever method your distro uses. if you have webmin it is very easy.

In simple , I have server on Local IP.
DNS(Under Network Configuration) is 8.8.8.8

Have you check your system logs for any unusual error messages? Harddrive errors and etc?

Check out the following log files:

/var/log/messages
/var/log/dmesg
/var/log/boot

If Hardrive Error, then how come all works good after reformat ?

Log files are too big even to download (

Then I recommend checking the system logs. You might find messages their that can give you ideas on what is happening to your system.

Where will i find the system log file?

all logs are in/var/log

system logs depend on configuration. often in "messages" but there are many options.

a good way to check is

Code: Select all

cd /var/log
ls -lth

this will give you a list ordered by date and you can see which files have been modified today. it's a safe bet that the file storing system messages has been modified ... today. LOL

This is what I see :

Code: Select all

[root@go log]# ls -lth
total 18M
-rw------- 1 root     root     367K Mar 18 11:29 cron
-rw------- 1 root     root     197K Mar 18 11:28 maillog
-rw------- 1 root     root     1.3K Mar 18 11:21 secure
-rw-r--r-- 1 root     root     143K Mar 18 11:21 lastlog
-rw-rw-r-- 1 root     utmp     173K Mar 18 11:21 wtmp
-rw------- 1 root     utmp     3.0M Mar 18 11:21 btmp
-rw------- 1 root     root      65K Mar 18 10:53 messages
-rw-r--r-- 1 root     root      18K Mar 18 07:48 rpmpkgs
drwxr-xr-x 3 root     root     4.0K Mar 18 07:47 astguiclient
drwxrwxrwx 2 munin    munin    4.0K Mar 18 07:43 munin
drwxrwxrwx 4 asterisk asterisk 4.0K Mar 18 07:43 asterisk
-rw-r----- 1 mysql    mysql     20K Mar 18 06:17 mysqld.log
-rw-r----- 1 root     root     5.6K Mar 18 06:12 acpid
-rw-r--r-- 1 root     root      955 Mar 18 06:12 dkms_autoinstaller
-rw-r--r-- 1 root     root      11K Mar 18 06:12 brcm-iscsi.log
-rw-r--r-- 1 root     root      22K Mar 18 06:12 dmesg
drwxr-xr-x 2 root     root     4.0K Mar 18 04:58 sa
-rw------- 1 root     root        0 Mar 17 03:56 boot.log
-rw------- 1 root     root        0 Mar 17 03:56 spooler
-rw------- 1 root     root        0 Mar 17 03:56 xferlog
drwx------ 2 root     root     4.0K Mar 17 03:56 httpd
-rw------- 1 root     root     3.5M Mar 17 03:56 maillog.1
-rw------- 1 root     root     6.1M Mar 17 03:56 cron.1
-rw------- 1 root     root     418K Mar 17 03:37 messages.1
-rw------- 1 root     root     2.7M Mar 17 02:45 secure.1
drwxrwxrwx 2 root     root     4.0K Mar 17 01:08 goautodial
-rw-r--r-- 1 root     root      18K Mar 16 04:11 rpmpkgs.1
-rw------- 1 root     root        0 Mar 13 12:06 xferlog.1
drwxr-xr-x 2 root     root     4.0K Mar 11 14:43 prelink
-rw------- 1 root     root        0 Mar 11 12:42 boot.log.1
-rw------- 1 root     root     311K Mar 11 12:39 anaconda.log
-rw------- 1 root     root      27K Mar 11 12:39 anaconda.syslog
drwxr-xr-x 2 root     root     4.0K Mar 11 12:38 pm
-rw------- 1 root     root     2.4K Mar 11 12:38 faillog
drwxr-xr-x 2 root     root     4.0K Mar 11 12:35 mail
-rw------- 1 root     root        0 Mar 11 12:35 spooler.1
-rw------- 1 root     root        0 Mar 11 12:34 tallylog
-rw-r--r-- 1 root     root      763 Mar 11 12:33 scrollkeeper.log
drwxr-x--- 2 root     root     4.0K Mar  6  2011 audit
drwx------ 2 root     root     4.0K Jan 21  2009 ppp
drwxr-xr-x 2 root     root     4.0K Nov 11  2007 conman
drwxr-xr-x 2 root     root     4.0K Nov 11  2



so have a look inside with

Code: Select all

tail -500 filename

on any you believe may contain pertinent information. cron looks promising.

File modified last when server rebooted :

mysqld.log
acpid
dkms_autoinstaller
brcm-iscsi.log
dmesg

if you cannot download them, use the tail command on them and see if any include a message pertinent to the situation.

Pretty Bad and Hectic !!!

Reinstallation is a easyway out !!

and be sure you md5 check the downloaded .iso file and the cd after you burn it. and check the install log (if goauto has one ...) for errors

(( But why its getting stucked after asterisk ( ?

gmcust3 wrote::-((( But why its getting stucked after asterisk ( ?

williamconley wrote:... asterisk starts in under a second. But you should be aware that the last item loaded by the system visually will "just stay there" while the rest of the actual OS (things like "cron" and "iptables") start up on various operating systems. Processes that are not itemized will continue to start even though the last shown entry may be "asterisk". If you are not sure ... modify the system to NOT start that last process, and the previous process will now be "last" ... and will take just as long.

I have had a few client boxes that had an issue with DNS. And in "iptables" security, they had a domain name (to allow a carrier to send packets to the server ...). The iptables system would take 15 minutes to "fail" to look up the IP address of the domain before finally giving up.

It could be any process in the system ... you can try turning off "services" one at a time to see which one it is (be sure you write down which ones are there before you begin so you can put it all back!).

Or just Build A New Server (after all, the software is free ...).

does not make asterisk the problem. asterisk is just the last thing to start in that checklist. the next checklist is not visible, so asterisk remains on the screen and appears to be the problem. but it may not be. it is very likely asterisk started successfully and one of the myriad other items that comes after it dies, leaving asterisk still visible on the screen.

turn off the process that starts asterisk ... see if the problem persists. then you'll know it is OS related and not Asterisk related.

My cron looks like this :

Code: Select all

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

### uncomment If kernel RAM drive is enabled
#mke2fs -m 0 /dev/ram0
#mount /dev/ram0 /var/spool/asterisk/monitor
#mkdir /var/spool/asterisk/monitor/DONE
#mkdir /var/spool/asterisk/monitor/ORIG

# Disable console blanking and powersaving
/usr/bin/setterm -blank
/usr/bin/setterm -powersave off
/usr/bin/setterm -powerdown

### start time server
/usr/sbin/ntpdate -u pool.ntp.org
### ntpd

### start apache, mysql and vsftpd
/etc/init.d/httpd start
/etc/init.d/mysqld start
/etc/init.d/vsftpd start

### roll the Asterisk logs upon reboot
/usr/share/astguiclient/ADMIN_restart_roll_logs.pl

### clear the server-related records from the database
/usr/share/astguiclient/AST_reset_mysql_vars.pl

### load timer source
modprobe zaptel
#modprobe wcfxo
modprobe ztdummy
ztcfg -vvvvvvvvvvvv

### sybsys local login
touch /var/lock/subsys/local

### sleep for 20 seconds before launching Asterisk
sleep 20

### start up asterisk
/usr/share/astguiclient/start_asterisk_boot.pl

## Repair Mysql :
## =============================
mysqlcheck --auto-repair --check --optimize --all-database -pvicidialnow
/usr/share/astguiclient/AST_flush_DBqueue.pl -q
/usr/share/astguiclient/AST_VDhopper.pl -q
/usr/share/astguiclient/AST_reset_mysql_vars.pl
/usr/share/astguiclient/AST_DB_optimize.pl
/usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +2 -print | xargs rm -f
/usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +2 -print | xargs rm -f
/usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +4 -print | xargs rm -f



/usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +4 -print | xargs rm -f

Above line is the Culprit ?

more likely the repair mysql section.

That's bad !!

that's normal. many distros "auto-repair" mysql at boot to avoid tech support for damaged mysql tables from power outages. unfortunately, it seriously bogs down with large tables. turn it off.