AST_flush_DBqueue.pl stops the server from dialing

[jessiekidfernando]

Hello Vicidial Team,

We are running this script on one of the servers within a cluster. We have in total of 10 dialer/agent server. Each server acts as dialer and agent server at the same time. Our DB is on a different server.

Upon checking, this script causes the dialer from stop dialing giving us a minimum 2 minutes wait time. Any other idea when is the best time to run this script based from our setup?

### flush queue DB table every hour for entries older than 1 hour - Not Required for Slave Servers
1-51/30 * * * * /usr/share/astguiclient/AST_flush_DBqueue.pl --seconds=1800

Thanks in advance for your inputs.

Regards,

Jessie Kid Fernando

[williamconley]

why are you not running it at 11 * * * * as the stock install does? what symptom/condition caused you to veer from this path? why are you specifying the seconds?

It would be very useful to know why you are already doing what you're doing rather than some sort of catch-all "what's best", when you've veered from the stock already ... without explanation. (and no: "2 minutes wait time" is too vague for an explanation)

[jessiekidfernando]

Hello Williamconley,

Does it mean we need to retain cron schedule by default 11 * * * * ? Well the reason we have changed it because we can see that the dialers stop dialing the leads every time the script is being executed thus giving the agent high wait time.

Regards,

Jessie Kid Fernando

[williamconley]

In which case you should try to investigate how that script stops dialing on your system but not others. Is it a long-running-query situation? or does it delete something necessary?

It is interesting that you identified this particular script as being "the culprit", but note that this script may be a symptom and not a cause.

[jessiekidfernando]

Hello Williamconley,

When I tried to debug the script, I saw that the MEMORY tables aren't being optimized. It has something to do with the engine. But just wondering. If let's say having 10 agent/dialer server within a cluster and the DB is on a separate server. What kind of engine should the vicidial tables should be? We have adjusted some of the tables engine specifically the vicidial_manager and vicidial_list table since we are taking almost 100k calls per day.

Regards,

Jessie

[newbie]

I have same issue , all I see in mytop besides the other tables being flushed is the cid_channels_recent massively being cleard.
Im running 6 different clusters with 100 agents per cluster on a master/slave db with 4 box of asterisk/vici. Script being run in master DB.

So base on the posts above. I was assuming that the AST_flushDBqueue.pl should only run @ 11 minute past the hour and it should not be moved to something else? Like @ 18minute past the hour or 25 minute past the hour?

DB Master & Slave Specs:
Xeon E5-2680 v4 14-Core 2.4 GHz
64x ECC DDR4 SDRAM (GB)
4x 480 GB Enterprise SSD (RAID 10)
LSI Logic Megaraid
CentOS 7.0 - 64 Bit
1Gbps Private Network

Vicidial Admin:
VERSION: 2.14-735a
BUILD: 200127-1620

System settings:
Version: 2.14b0.5
SVN Version: 3191
DB Schema Version: 1585
DB Schema Update Date: 2020-01-30 11:58:39

Database:
Ver 15.1 Distrib 10.1.44-MariaDB

[williamconley]

how was your system installed? (.iso name and version)

provide your carrier's dialplan entry

have you customized your extensions.conf file at all? if so: do all your contexts have the hangup entry?

The hangup entry is the one that performs cleanup, and if that fails to run because it's not included in a customized context in extensions.conf (or extensions-vicidial.conf) it may leave a lot of garbage in the system which would then need to be cleaned up by this flush DB script.

The time it runs isn't really relevant, it has to run at some point. But when it runs it shouldn't be disruptive. If it's being disruptive when it runs, then something unusual is happening ... like "far too many entries, way above normal" or "deleting things it should not be deleting". You'll have to find out which ...

A good experiment may be to run it every 5-10 minutes instead of hourly. If it's less disruptive, that would likely indicate it has far too many things to delete hourly. Then you'll need to find out why there is so much to delete, another process isn't deleting those entries (thus the question earlier about the hangup script).

Happy Hunting!

[newbie]

Thanks William,

So far when I increase the max_connections to 2000 I don't get the too many connections error anymore. I suspect it could be my my.cnf configuration for 100 agents is not that doable anymore as this is the configuration for the 50 agents that works just fine.

Can you please suggest anything what I need to change in my my.cnf file for 100 agents and base on my hardware?

DB Master & Slave Specs:
Xeon E5-2680 v4 14-Core 2.4 GHz
64x ECC DDR4 SDRAM (GB)
4x 480 GB Enterprise SSD (RAID 10)
LSI Logic Megaraid
CentOS 7.0 - 64 Bit
1Gbps Private Network

Vicidial Admin:
VERSION: 2.14-735a
BUILD: 200127-1620

System settings:
Version: 2.14b0.5
SVN Version: 3191
DB Schema Version: 1585
DB Schema Update Date: 2020-01-30 11:58:39

Database:
Ver 15.1 Distrib 10.1.44-MariaDB

Code: Select all

# The following options will be passed to all MySQL clients
[client]
#password   = your_password
port      = 3306
socket      = /var/lib/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port      = 3306
socket      = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 640M
key_buffer = 1024M
max_allowed_packet = 2M
table_open_cache = 8192
table_cache = 8192
sort_buffer_size = 16M
read_buffer_size = 32M
join_buffer_size = 2M
read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 256M
thread_cache_size = 64
#query_cache_size = 32M
#query_cache_limit = 1M
#query_cache_min_res_unit = 1024 # Use this is you have TONS of small queries
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 28

# ViciDial DB-Only Settings
connect_timeout=60
wait_timeout = 28800
#long_query_time = 3
#slow_query_log
max_connections=2000
open_files_limit=24576
max_heap_table_size = 2G #64M
bulk_insert_buffer_size = 2G    #recommend same size as tmp_table_size
tmp_table_size = 2G    #recommend 1G min
concurrent_insert=2
expire_logs_days=3
#log-queries-not-using-indexes
#skip-locking
#skip-innodb
skip-name-resolve
skip-host-cache

log-bin
server_id=1
replicate-do-db=asterisk
bind-address=10.160.23.45

# This is for databases with an abnormally high amount of small lookups
# In other words, small inefficient lists. Unless your query return data
# is less then 4kb you do not need to worry about this
#query_cache_min_res_unit = 4096

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
#log-bin=mysql-bin

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
#server-id   = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Point the following paths to different dedicated disks
#tmpdir      = /tmp/
#log-update    = /path-to-dedicated-directory/hostname

# Uncomment the following if you are using BDB tables
#bdb_cache_size = 384M
#bdb_max_lock = 100000

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql/
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

# The safe_mysqld script
[safe_mysqld]
log-error=/var/lib/mysql/mysqld.log

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer_size = 4M
write_buffer_size = 4M

[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer_size = 4M
write_buffer_size = 4M

[mysqlhotcopy]
interactive-timeout


[williamconley]

max connections 2000 is a normal upgrade requirement for larger systems.

my-bigvici.cnf is included in some distros. i think you have to answer questions correctly during DB server building/installation to get it, although they may have simply put the settings into the my.cnf during the buildout instead of creating that file in /usr/src/astguiclient/conf/my-bigvici.cnf

Some considerations based on your system usage and available memory and cpu core count:

Code: Select all

key_buffer_size = 1024M
myisam_sort_buffer_size = 128M
thread_concurrency=16
max_connections=2000
max_heap_table_size=64M

Code: Select all

mysql -u cron -p -e "show status like '%Aborted%'";

If this shows thousands of Aborted_clients or Aborted_connects:

File: /etc/systemd/system.conf
Value: DefaultTasksMax=infinity

Happy Hunting!

[newbie]

newbie question like my name :

whats the equivalent of this in Centos 6?

Code: Select all

/etc/systemd/system.conf


[williamconley]

Code: Select all

cd /etc/
grep DefaultTasksMax * -Rin

Happy Hunting

of course ... if this doesn't result in a lot of aborts:

Code: Select all

mysql -u cron -p -e "show status like '%Aborted%'";

You don't need to find it anyway.