Server Queries per second question..

[amjohnson]

Version: 2.8b0.5
SVN Version: 2052
DB Schema Version: 1362

I have a question. We have a large outbound dialing cluster.. The database has 48 gigs of ram and 8 Enterprise SSD on a Megaraid 9280-24i.. My question is what is the max Queries per second anyone has seen running Vicidial? Running about 120-160 agents throughout the day and 400-600 outbound lines I am seeing via the mysql workbench server status screen a QPS of 15k-20k sustained... Is this normal? mtop shows very few long queries and the ones we do get are related to reports and other stuff run while they are calling... Our average long query is about 3.5-4 seconds when someone does a lead search.

The reason I am asking is we are getting an occasional 3 way call but more often then not what we are getting is multiple beeps in the conference as if multiple calls are being dispositioned in rapid succession .. Server loads are low. We've tried several different switches for the interserver communications, catalyst 3750(48 port 10/100), dlink (16 port GB) and cisco small business (24 port GB).. All with the exact same results.

In one of my previous posts I noted an excessive amount of ports in use compared to previous versions I have run... I am also seeing a ton of aborted mysql clients. It usually starts out low but about mid day takes off for no apparent reason... server loads stay about the same (25% or less) queries seem to be ok (we get 10-15 long queries out of about 300,000,000) but suddenly the aborted clients will go from a few hundred or a few thousand to hundreds of thousands... We managers start calling us saying they are having problems with caller sessions getting 3 way calls.

Anyone have any idea where to good place might be to look? I'm stumped.. All network connectivity appears to be ok..

BTW all station sip traffic is handled by Grandstream 24 port FXS gateways so my stations do not have to handle or deal with the sip traffic at all..

-Andrew

[mflorell]

For a heavy outbound dialer system that is normal. But it really does depend on what kind of call activity is happening on your system. We have a client that has 400 concurrent agents doing only inbound and manual dial that never goes above 10k QPS.

As for the issues you are having, what version of MySQL are you running?

What settings are you using in my.cnf?

[amjohnson]

Its Vicibox 5.0.3 running:

Code: Select all


To change the server IP in the database type:
/usr/share/astguiclient/ADMIN_update_server_ip.pl

Official paid-for ViciDial support is available at
http://www.vicidial.com

Free community-based ViciDial Support is available
at http://www.vicidial.org/VICIDIALforum

- ViciBox Redux v.5.0.2-130821
vicidb1:~ # uname -a
Linux vicidb1 3.7.10-1.16-default #1 SMP Fri May 31 20:21:23 UTC 2013 (97c14ba)                                                                x86_64 x86_64 x86_64 GNU/Linux
vicidb1:~ # uname -r
3.7.10-1.16-default
vicidb1:~ # uname -ra
Linux vicidb1 3.7.10-1.16-default #1 SMP Fri May 31 20:21:23 UTC 2013 (97c14ba)                                                                x86_64 x86_64 x86_64 GNU/Linux
vicidb1:~ # mysql -V
mysql  Ver 15.1 Distrib 5.5.33-MariaDB, for Linux (x86_64) using readline 5.1
vicidb1:~ # cat /etc/my.cnf
# Basic ViciBox my.cnf for a quad-core 8-GB RAM or so
# Use the dedicate my-big.cnf for 16+GB RAM and 8+ Cores
#

# The following options will be passed to all MySQL clients
[client]
#password       = your_password
port            = 3306
socket          = /var/run/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
skip-innodb
tmpdir = /run/mysql
port            = 3306
socket          = /var/run/mysql/mysql.sock
back_log = 1024
#query_cache_type = 1
#query_cache_size = 100M
#query_cache_limit = 4M
datadir = /var/lib/mysql
skip-external-locking
key_buffer_size = 2048M
max_allowed_packet = 16M
table_open_cache = 8192
sort_buffer_size = 4M
net_buffer_length = 8K
read_buffer_size = 4M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 128M
thread_cache_size = 256
query_cache_size = 0
thread_concurrency=16
skip-name-resolve
connect_timeout=60
long_query_time=2
log_slow_queries
max_connections=1800
open_files_limit=8192
#open_files_limit=8192
max_heap_table_size=64M
expire_logs_days=3
default-storage-engine=MyISAM
table_definition_cache=8192
table_cache=8192
myisam_recover
myisam_repair_threads=1

# This will disable networking
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
#log-bin=mysql-bin

# binary logging format - mixed recommended
#binlog_format=mixed

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
#server-id      = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

# The safe_mysqld script
[safe_mysqld]
log-error       = /var/log/mysql/mysqld.log
socket          = /var/run/mysql/mysql.sock

[mysqldump]
socket          = /var/run/mysql/mysql.sock
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

[mysqld_multi]
mysqld     = /usr/bin/mysqld_safe
mysqladmin = /usr/bin/mysqladmin
log        = /var/log/mysqld_multi.log
# user       = multi_admin
# password   = secret

# If you want to use mysqld_multi uncomment 1 or more mysqld sections
# below or add your own ones.

# WARNING
# --------
# If you uncomment mysqld1 than make absolutely sure, that database mysql,
# configured above, is not started.  This may result in corrupted data!
# [mysqld1]
# port       = 3306
# datadir    = /var/lib/mysql
# pid-file   = /var/lib/mysql/mysqld.pid
# socket     = /var/lib/mysql/mysql.sock
# user       = mysql

# [mysqld2]
# port       = 3307
# datadir    = /var/lib/mysql-databases/mysqld2
# pid-file   = /var/lib/mysql-databases/mysqld2/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld2/mysql.sock
# user       = mysql

# [mysqld3]
# port       = 3308
# datadir    = /var/lib/mysql-databases/mysqld3
# pid-file   = /var/lib/mysql-databases/mysqld3/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld3/mysql.sock
# user       = mysql

# [mysqld6]
# port       = 3309
# datadir    = /var/lib/mysql-databases/mysqld6
# pid-file   = /var/lib/mysql-databases/mysqld6/mysql.pid
# socket     = /var/lib/mysql-databases/mysqld6/mysql.sock
# user       = mysql

vicidb1:~ # uptime
 13:00pm  up   6:38,  2 users,  load average: 2.26, 2.24, 2.05
vicidb1:~ #


2.26-2.75 on a dual quad-core Xeon server with hyperthreading off as you suggested in training..

[Vince-0]

Take a look at the number of open sockets on your web server. I had an issue with tcp_recycle once but I'm not sure if you're in the same position. Reference this thread:
http://vicidial.org/VICIDIALforum/viewt ... 900#p94900

[amjohnson]

We have the tcp reuse and tcp recycle turned on on all servers. With this load if we didnt clients would randomly get pause out and all sorts of other crazyness..

-Andrew

[williamconley]

As for the issues you are having, what version of MySQL are you running?
What settings are you using in my.cnf?

While it's likely matt could assume you are using "stock" from that installation, it is usually a good idea to specify the actual version in response to a specific request. Makes it MUCH more likely to get a direct response from matt again when you pointedly answer the question. Technicians (busy ones especially) are like that.
Post the version and post the my.cnf entries that are relevant. If anything raises a red flag, you'll get an answer most likely.

Next: You are sure you have scoured your logs and checked the times of occurrence for patterns and causes? I don't see an "it happens around 11:15 AM daily ..." or "just after XXX event ..." so I'm asking.

Next: You are VERY sure your DB server cannot be "hit" from A. The Net and B. Agent workstations? (firewalled, impossible to hit ... no possible attack vector except THROUGH a dialer or web server which would then show extra load from an attack ...) Not to be paranoid, but enterprise level servers are often a tempting target and "any" means will often be used to gain access. Including "bouncing" through a two-port VPN server from the unused port into a VPN tunnel on the other port to get to the server cluster in the colo. Raised a few eyebrows when we found that one. They never hit the DB server directly, but slowed down a secondary server linked to it enough to cause the DB server stress. We traced the attack back from there. Goes to show that all angles sometimes have to be considered in Enterprise.