MariaDB crash in Vicibox 6

[ccabrera]

Hello:

I´ve recently run into a problem with a new installation of Vicibox. This setup is 5 server (1 DB, 4 dialer/web) totalling 100 agents.

From time to time, MariaDB crashes. When it crashes, I read this on the mysql.log (full log is pasted in http://pastebin.com/0sBhuRsw):

Code: Select all

141211 14:18:33 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x192)[0x66e522]
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 5.5.33-MariaDB-log
key_buffer_size=1073741824
read_buffer_size=4194304
max_used_connections=386
max_threads=2002
thread_count=51
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17484634 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x3e327b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ff7891cded0 thread_stack 0x48000
/usr/sbin/mysqld(handle_one_connection+0x41)[0x66e5b1]
/lib64/libpthread.so.0(+0x80db)[0x7ff7e8c070db]
/lib64/libc.so.6(clone+0x6d)[0x7ff7e7cd958d]


Has anyone else ran into this problem? Any suggestions?

[Vince-0]

Necro'ing a thread here but I'm seeing this quite often on a new setup I'm doing.
It could have to do with that over sized key buffer at 1000M.

I will gather any info and revert soon.

Vin.

[Vince-0]

This MariaDB is crashing around once a week lately on this particular server. Is anyone else experiencing this?

Here's my mysqld log pastebin for the crash:
http://pastebin.com/raw.php?i=yqfsvnzi

I've installed from Vici 6.03 preload ISO on a Dell R720.

[williamconley]

1) Vicidial version with build is a requirement. Please remember to always post it!

2) Hard drive full? (Rebooting after the crash often clears enough space to run for a while again ... this happens more often than anything else)

3) Hard drive dying? Check

Code: Select all

iostat -c -d -h -z -t -x 1 5

Note that "smart drives" have diagnostics built in ... that never seem to work.

[Vince-0]

Hi again,

1) Vicibox_v.6.0.x86_64-6.0.3.preload.iso | VERSION: 2.12-484a BUILD: 150429-1222 | Asterisk 1.8.32.2-vici | Single Server | No Interface cards | No Extra Software | Dell R720 | Dual Xeon E5-2609

2)Hard drives are at 8% usage on 821GB partition PERC H710P RAID10 on 4x 300Gb SAS 6Gbps 15k and idling on iostat util % at around 0.80.

I doubt it's the disks, they are healthy. Although it looks like a hardware issue, maybe memory - I didn't test that thoroughly before I installed. I will do so at the next opportunity and revert.
Thanks!

[williamconley]

If the problem occurs regularly, take out half your memory. Next time it happens, put that half back in and take out the other half (Unless you can switch out the memory in this machine with another NonCrashing machine ... which would be even better).

Happy Hunting.