Asterisk is crashing after running few days.

[anandkumargupta]

Hi Team,

I am running on Goautodial version 2.0 on HP ProLiant DL360 G6 server with 8GB RAM and 264 GB HD with H/W RAID 5 controller.

The asterisk use to crash after few days. I was unable to find any error on asterisk log or kernel log "message".

Last crash gave error "[Jan 18 15:33:30] ERROR[4813]: astobj2.c:114 INTERNAL_OBJ: bad magic number 0x0 for 0xb7afffd8" which was the last line in the log and CLI in screen named asterisk.

kernel message was clean and having regular error.

top result after crash was

top - 16:27:03 up 3 days, 5:12, 5 users, load average: 3.92, 3.71, 3.13
Tasks: 211 total, 1 running, 210 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.2%us, 4.2%sy, 0.0%ni, 68.9%id, 15.9%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 8303476k total, 7965284k used, 338192k free, 139212k buffers
Swap: 1044216k total, 5524k used, 1038692k free, 7284532k cached

[root@INMUMGODIALS1 log]# free -m
total used free shared buffers cached
Mem: 8108 7815 293 0 136 7145
-/+ buffers/cache: 533 7575
Swap: 1019 5 1014

[root@INMUMGODIALS1 log]# vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 4 5524 297176 138536 7303584 0 0 11 88 2 13 3 2 92 3 0
0 6 5524 291348 138568 7304984 0 0 3 2575 5354 43669 13 6 69 13 0
4 1 5524 290232 138652 7305340 0 0 0 2448 5276 45125 7 3 77 12 0
3 1 5524 284644 138704 7306224 0 0 0 2392 5274 55062 12 6 70 12 0
4 1 5524 305600 138748 7306660 0 0 0 2608 5365 53474 8 6 74 12 0

In first attempt I thought there is some problem in memory and tune the php.ini and increase the memory allocation to php and found that error giving ratio extended by few days.

I am using phpagi.php class for IVR creation.

Please suggest where I should focus to isolate the problem?

[williamconley]

Reboot the server every night or pay attention to the server BEFORE it crashes to see if you can determine if something has begun eating memory or cpu. Stats after a clean reboot ... really don't help tell what went wrong.

You can use lots of tricks (logging memory and cpu usage, and rolling the log on reboot ... then you can check the moment before reboot to find out what may have crashed it).

Check your cron logs and see if a cron job fired as it crashed.

But mostly: reboot nightly at "stupid" early AM and then forget about it.

You can even try Vicibox instead of GoAutoDial and see if that's more stable for you. It could be a simple driver conflict that may not exist on the other distro (packages have subtle differences), or it could just be something you'll never find. If you have ANY reliable downtime nighly, save yourself the effort and reboot nightly.