DB access issue

[marcin]

We run a cluster of 1 db, 1 web and 2 asterisk servers for total of 50 agents and 4:1 dial ratio
The issue we are having are missing recordings. The astgui logs show normal call flow but Monitor never get executed on the calls without the recordings.
Asterisk servers running at 3-4 average load, Yes I know, but most of it is used to wait for mysql respond from a db server.
Times I can see 300 WAIT mysql connections on each asterisk server.
DB hardware:
2 quad core Intel(R) Xeon(R) X5460 CPUs @3.16 GHz
16 Gig or ram and raid 5 +1 SAS 10K drives.
Software: ViciBox Redux v.5.0.2-130821 on all servers
Db server runs at 1-2 average load at pick time. Again, I know it is too high:

hdparm -Tt /dev/cciss/c0d0p2:
Timing cached reads: 14584 MB in 1.99 seconds = 7311.24 MB/sec
Timing buffered disk reads: 676 MB in 3.00 seconds = 225.05 MB/sec
and
dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output
83886080 bytes (84 MB) copied, 0.108705 s, 772 MB/s

2 asterisk servers and web are very same: software and hardware.

Next thing I will try is to separate mysql traffic to its own private network.

Any advise on lock of recordings and large number of WAIT mysql connections.

[mflorell]

RAID 5 is horrible, you should never use it. Use RAID 10 instead

Also, the kind of RAID card are you using? Most are designed for large archival, not speed.

[marcin]

The server is hp dl360 g5 and the raid card Smart array p400.

[mflorell]

Make sure you have the latest firmware for the p400, and also consider upgrading the RAM on the cache if it isn't already at 512MB. And the most important change, don't use RAID-5, change to RAID 10 if possible. That will most likely mean reformatting and reinstalling, but it will be worth it.

[williamconley]

RAID10, definitely. But to exclude RAID from your problem list, you could remove RAID altogether and just use fast HDs and get a good backup regimen.

That being said: Your HD is 10k instead of 15k SAS. Not good enough. Next up: Have you verified 6G/sec throughput on all components? It's easy to get a 6G card and 6G drives ... and then use a 3G cable and screw up the entire throughput speed. LOL

Then: What settings are you using in your my.cnf?

and please confirm you have Gigabit network between all the servers (not 10/100!) on a private network (not the public IPs, if there are two ports in use). and that there is no firewall on the private network.

[marcin]

my.cnf:

skip-external-locking
key_buffer_size = 1024M
max_allowed_packet = 2M
table_open_cache = 8192
sort_buffer_size = 4M
net_buffer_length = 8K
read_buffer_size = 4M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 128M
thread_cache_size = 50
query_cache_size = 0 #32M
thread_concurrency = 48 #16
skip-name-resolve
connect_timeout=40
long_query_time=5
log_slow_queries
max_connections=2000
open_files_limit=24576
max_heap_table_size = 2G #64M
bulk_insert_buffer_size = 2G #
tmp_table_size = 2G #recommend 1G min
concurrent_insert = 2
expire_logs_days=3
default-storage-engine=MyISAM
table_definition_cache=8192
table_cache=8192
myisam_recover
myisam_repair_threads=1

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

I will verify a cabling and the private network is a Gigabit. No firewall on private network.
After migrating mysql traffic to private network the systems works much better, but I still see lot of WAIT connections to mysql

[williamconley]

How do these configuration settings compare to the Stock Vicidial my.cnf and the my-big-vici.cnf settings? Seeing just the differences would be useful.

[marcin]

The configuration is a default from vicibox install.
The only changes made are showing with # old value

[williamconley]

So you're not using my-big-vici.cnf values? There is more than one default set. You should look at which one you're using.

[marcin]

We have upgraded the database server hardware and we are using my-big-vici.cnf now. Db server runs at 50-70% load and there is very little wait sessions from asterisk server, however there are 1000s of mysql wait sessions from web server, and we see mysql error on agent screens from time to time.
And we still have up to 5 missing recordings per day.
Any ideas?

[mattyou1985]

please try this
set Campaign Recording: ONDEMAND and then

Agent Recording Override: ALLFORCE

that should help

[williamconley]

What is a "mysql wait session from web server"? (code ... just show one or two lines, though).

Also be sure all your dispo_url entries are valid and accessible to all the web servers. Those can choke the system if not valid.