Soundboard Delay After about an hour of use

[alo]

Good Morning Friends!

We were very excited to try out the new soundboard feature and have really enjoyed it.
However after about an hour of use it starts delaying the audio file after the button is pressed. for example you might hit the Hello button and it wont play the Hello audio file for several seconds.

When this happens, I quickly Restart vicidial with 'systemctl restart vicidial' and the agents log back in and the sound files play immediately after being pressed. Then after about an hour or so the delay begins again.

Anyone know what might be causing this?

Vicibox 7.0.3
kernel version 4.1.20-11-default
VERSION: 2.12-565a
BUILD: 170114-1356
Version: 2.14b0.5
SVN Version: 2658
DB Schema Version: 1485
Asterisk version 1.8.32.3-vici
Dell R610 Raid 1 with a perc 700 raid card and SSD drives
10 agents logged in with dial level 6. Single DB/Web/Asterisk server

Thanks!

[mflorell]

I would suggest looking at "mtop" when this starts to happen to see if there are any database issues related to this.

Also, you could try looking at the Asterisk CLI to see if there are any errors there.

[alo]

I see a lot of statements for the vicidial_manager table but I don't see any waiting for more then a second. although I assume even half a second of waiting could cause a delay.
Here is an example statement I see

Code: Select all

UPDATE vicidial_manager set status='UPDATED', channel='IAX2/ASTplay-7774', uniqueid = 'Redacted' WHERE server_ip = 'Redacted' and callerid = 'Avatar_Mainpitch'

Anything I should specifically be looking at to optimize this table?

When I monitor top my WA is always at 0, so I assume my disk speed is adequate.
The my.cnf is default vicibox 7.0.3 except I adjusted: thread_concurrency=16 and max_connections=2048

[mflorell]

How many records are in the vicidial_manager table when you are having these issues? (select count(*) from vicidial_manager;)

[alo]

Ah! Interesting! Thank you!

So it seems the vicidial_manager table keeps growing, and once it gets to about 20,000 records the delay starts.

Is that normal or should the vicidial_manager table not grow so big?

[mflorell]

The table is pruned every hour, but you can adjust a crontab entry to make it trim every 15 minutes if you like:

### flush queue DB table every hour for entries older than 1 hour (changed to 15 minutes)
11,26,41,56 * * * * /usr/share/astguiclient/AST_flush_DBqueue.pl --seconds=900 -q

[alo]

Awesome! so far so good.

I appreciate you more then you know. Thank you for your knowledge, dedication, and kindness!

[alo]

its much better, but It seems to be still intermittently delayed.

I moved the flush queue DB table to every 10 minutes. Is their any other suggestions you might have?

[mflorell]

Add more RAM to your DB server, and adjust the my.cnf settings to use it.

[alo]

I'll do that now! thanks again!

Do you have recommended changes for a Dell R610 Raid 1 with a perc 700 raid card and SSD drives with 32GB Ram?

I already have adjusted the following from default:

max_connections=2048
concurrent_insert=2
thread_concurrency=16

[mflorell]

I would recommend ditching the perc card and getting an LSI Logic MegaRAID caching RAID controller.

Other than that, put the database data directory on a separate physical drive partition from the rest of the Linux filesystem.

[alo]

Still having a bit of trouble here.
Do you recommend putting /var/Lib/mysql on a separate raid Drive config?

Also, What about loading this table to RAM or something like that?

[mflorell]

Yes, a separate drive structure for the MySQL/MariaDB data directory is still recommended.

As for a MEMORY table, what table were you thinking of changing to that?

MEMORY tables have limitations, and if you don't know what you are doing you can actually slow things down instead of speeding them up by putting tables into RAM like that.

[alo]

I was thinking maybe the vicidial_manager table since thats the table that appears to be responsible for the key presses and appears to be the table thats getting locked if I am correct.

Also Just to clarify, are you suggesting a separate physical Drive for the SQL or just a separate partition?

[mflorell]

You have to be very careful changing vicidial_manager to a MEMORY table, in many cases it can actually slow the system down because it has a lot of VARCHAR fields in it. Also you can change the AST_flush_DBqueue.pl script in the crontab on your dialers to clear out records older than 900 seconds.

[alo]

Thanks Matt.

I decided to rebuild the system with a lsi 9271-8i mega raid controller with 2 raid 10 configs. (1 for the os and 1 for the MySQL directory.) Does that sound like it should help?

Also regarding the Slave Database Server. used for reports. Obviously it would be great because it wouldn't lock up the dialing when checking reports, but my theory is it slows everything down all the time because it needs to be replicating? am I thinking about it wrong?

I guess my question is here. does using a slave database cause extra load on the main database?

[williamconley]

but my theory is it slows everything down all the time because it needs to be replicating? am I thinking about it wrong?

I guess my question is here. does using a slave database cause extra load on the main database?

It does, in fact, slow down the DB due to replication. But this is a mere technicality.

BUT:

* the DB server responds to millions of queries per minute, this load is tiny by comparison.
* this single extra step for each changed record allows the slave DB server to be queried an unlimited number of times without any affect on the running DB server. So the end is a serious improvement in DB operations. IE: Even if you only have (for instance) one or two Real Time Reports running, it appears to even out almost immediately. Beyond that, running deep reports makes it extremely profitable.
* it's invisible to the end user when set up properly. Reports configured for the replication server are routed to the replication server in the background with no user interface alteration.

[alo]

Awesome William! thank you!

Should my slave DB be exactly the same as my DB or can I save a bit of money by not doing two raid 10 configs and less ram etc?

[williamconley]

No need for RAID at all, actually. That being said: There are those that like to go the extra mile and use this server as a catastrophic failure measure in case of the loss of the primary DB. In that case, you may want it to be beefy, since it's gonna have all the data "live and ready" when the primary goes down. But I'd also say that this would then be a "limp in" moment and you could deal with whatever you have (which beats the hell out of "nothin").