Linux crash

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

Linux crash

Postby phil_discount » Tue Mar 04, 2014 4:58 am

Hello,

we've got about 10 clusters running vicidial 2.2.0.
Installed Redux 3.0.5 (Opensuse 11.3) und Redux 4.0.3 (Opensuse 12.1).
every server has differnt hardware
sometimes server acting only as Webserver or only as telephonyserver.

every week one server is completely down. no screen on VGA port, keyboard not responding.
we already installed kdump for a crash Dump. but if the server crash the server doesnt create a dump file.
if i crash the server manually, server crates a dump file under /var/crash/DATE/...

i've got no idea to solve the problem. i looked at each log in /var/log.
sometimes a server reports as last message "kernel: hrtimer: interrupt took 141378 ns", but not every time.

if anybody has an idea what i can do to find the problem, please tell me :-)

one server has 8 cores and 10GB RAM, load average 0,18 0,16 0,16 - only telephony about 25 agents.
i think it cannot be to less power.

regards
philip
ViciBox Redux 3.0.5 | Vicidial 2.2.1-260 100527-2211 | Asterisk 1.4.27.1
9xVicidial 8-core 2.5GHz,4GB,SSD
4xWeb 4-core 2.5GHz,4GB,SSD
DB: 24-core-1.9GHz AMD,96GB,8xSSD Raid10
3xDBslave 4-core 2.5GHz,4GB,SSD (for SELECT: Reporting/LIVE)
400 seats
phil_discount
 
Posts: 468
Joined: Thu Jun 18, 2009 8:44 am
Location: Deutschland/Schweiz/Österreich

Re: Linux crash

Postby geoff3dmg » Tue Mar 04, 2014 6:35 am

When a server crashes with no screen output it's usually a hardware issue. I'm assuming the BIOS/Firmware is up to date. Test the hard disk(s) and the memory (I'd suspect the memory). Beyond that you can try and use the serial console to get a kernel dump.
Vicibox 5.03 from .iso | VERSION: 2.10-451a BUILD: 140902-0816 | Asterisk 1.8.28.2-vici | Multi-Server | Amfeltec H/W Timing Cards | No Extra Software After Installation | Dell PowerEdge 1850 | Pentium 4 'Prescott' Xenon Quad @ 3.40GHz
geoff3dmg
 
Posts: 403
Joined: Tue Jan 29, 2013 4:35 am
Location: Lancashire, UK

Re: Linux crash

Postby phil_discount » Tue Mar 04, 2014 6:46 am

we have 10 differnt Servers, cant believe thats an hardware issue. any other idea? ;-)
i will google for console dump.
do u know a good tool for checking ram and harddisk?
ViciBox Redux 3.0.5 | Vicidial 2.2.1-260 100527-2211 | Asterisk 1.4.27.1
9xVicidial 8-core 2.5GHz,4GB,SSD
4xWeb 4-core 2.5GHz,4GB,SSD
DB: 24-core-1.9GHz AMD,96GB,8xSSD Raid10
3xDBslave 4-core 2.5GHz,4GB,SSD (for SELECT: Reporting/LIVE)
400 seats
phil_discount
 
Posts: 468
Joined: Thu Jun 18, 2009 8:44 am
Location: Deutschland/Schweiz/Österreich

Re: Linux crash

Postby geoff3dmg » Tue Mar 04, 2014 6:53 am

You can run a 'memtest86+' memory test from the grub boot menu. You can use the 'badblocks' command from the console on the hard drive.
Vicibox 5.03 from .iso | VERSION: 2.10-451a BUILD: 140902-0816 | Asterisk 1.8.28.2-vici | Multi-Server | Amfeltec H/W Timing Cards | No Extra Software After Installation | Dell PowerEdge 1850 | Pentium 4 'Prescott' Xenon Quad @ 3.40GHz
geoff3dmg
 
Posts: 403
Joined: Tue Jan 29, 2013 4:35 am
Location: Lancashire, UK

Re: Linux crash

Postby phil_discount » Wed Mar 05, 2014 5:28 am

i checked memory and harddisk .. no errors, everything was fine.
now i disabled all RSYNC cronjobs to sync some files (not vicidial related) from networks shares.
perhaps rsync is causing the problems.

i will report
ViciBox Redux 3.0.5 | Vicidial 2.2.1-260 100527-2211 | Asterisk 1.4.27.1
9xVicidial 8-core 2.5GHz,4GB,SSD
4xWeb 4-core 2.5GHz,4GB,SSD
DB: 24-core-1.9GHz AMD,96GB,8xSSD Raid10
3xDBslave 4-core 2.5GHz,4GB,SSD (for SELECT: Reporting/LIVE)
400 seats
phil_discount
 
Posts: 468
Joined: Thu Jun 18, 2009 8:44 am
Location: Deutschland/Schweiz/Österreich

Re: Linux crash

Postby phil_discount » Sat Mar 22, 2014 4:51 pm

since i splitted web and asterisk server, everything works fine.
ViciBox Redux 3.0.5 | Vicidial 2.2.1-260 100527-2211 | Asterisk 1.4.27.1
9xVicidial 8-core 2.5GHz,4GB,SSD
4xWeb 4-core 2.5GHz,4GB,SSD
DB: 24-core-1.9GHz AMD,96GB,8xSSD Raid10
3xDBslave 4-core 2.5GHz,4GB,SSD (for SELECT: Reporting/LIVE)
400 seats
phil_discount
 
Posts: 468
Joined: Thu Jun 18, 2009 8:44 am
Location: Deutschland/Schweiz/Österreich

Re: Linux crash

Postby williamconley » Sat Mar 22, 2014 5:10 pm

another method you could use (next time) is to boot to a CD after hard power down. you may find the server was indeed running but had no networking and since your last reboot didn't have kbd/monitor on it the system refused to interact with them when added. we've had a few systems (still do in fact) that will not allow addition of a video or kbd after boot if one was not present.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20258
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Linux crash

Postby phil_discount » Sun Mar 23, 2014 5:08 am

Each server is connected to a kvm network switch. After a cradh keyboard isnt responding and no pictire on the screen
ViciBox Redux 3.0.5 | Vicidial 2.2.1-260 100527-2211 | Asterisk 1.4.27.1
9xVicidial 8-core 2.5GHz,4GB,SSD
4xWeb 4-core 2.5GHz,4GB,SSD
DB: 24-core-1.9GHz AMD,96GB,8xSSD Raid10
3xDBslave 4-core 2.5GHz,4GB,SSD (for SELECT: Reporting/LIVE)
400 seats
phil_discount
 
Posts: 468
Joined: Thu Jun 18, 2009 8:44 am
Location: Deutschland/Schweiz/Österreich

Re: Linux crash

Postby williamconley » Wed Mar 26, 2014 1:30 pm

perhpaps the kvm is crashing.

at any rate, put in a CD and boot from it. then you can browse the log files at the moment of crash. if you see log entries for AFTER the moment of crash, obviously it was still running since the CD boot will not have made any entries.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20258
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)


Return to Support

Who is online

Users browsing this forum: No registered users and 95 guests