Page 1 of 1

Slave Replication issue

PostPosted: Mon Jul 09, 2018 5:24 pm
by frequency
HI,

This may be a off-topic a bit. We are replicating vicidial database using normal vicibox installation method. The cluster has about a hundred agents. Slave is using LSI MegaRAID Level 0 Crucial MX500 SSD's. The server is on same network with <1ms latency. once more than 40 agents are logged on the dialer, the slave DB starts to show i/o wait time of 10-12% constant at 6000-9000kB/s writes and slave behind master increased each second.

I have tried replacing the raid card and also the ssd's with fresh installation but the issue persists. the temps are normal. We have another cluster with 100 agents with reporting slave server working flawlessly with similar equipment. what could be the issue?

TIA

Re: Slave Replication issue

PostPosted: Tue Jul 10, 2018 10:55 am
by Vince-0
Slave DB is single threaded as far as I know. You may be able to tweak slave config/mariadb versions to help with this.
Otherwise maybe raid write cache needs to be enabled?

Is your "iostat -x 1" util% at 99% ?

Re: Slave Replication issue

PostPosted: Tue Jul 10, 2018 11:46 am
by frequency
%util is hovering between 89% to 100%.

MariaDB is updated to latest version via zypper up command.

I'll check how to turn on the raid write cache on lsi card

EDIT: enabling write-cache over the megaraid resolved the issue.

Thanks a lot lot!

Re: Slave Replication issue

PostPosted: Wed Jul 11, 2018 4:25 pm
by frequency
The server seems to be retrieving "Agent status report" for a single day in 90 seconds now...User stats seems to be opening just fine.

Re: Slave Replication issue

PostPosted: Wed Jul 11, 2018 5:10 pm
by williamconley
We've also found multi-threading and "conservative" parallel mode to help a "not-so-powerful" slave keep up with a more powerful master server.

Code: Select all
Parallel_Mode => conservative
slave_parallel_threads=256
slave_domain_parallel_threads=256
thread_concurrency=512


In this case, a 40-core master and a 4 core slave.

Re: Slave Replication issue

PostPosted: Thu Jul 12, 2018 11:43 am
by frequency
williamconley wrote:We've also found multi-threading and "conservative" parallel mode to help a "not-so-powerful" slave keep up with a more powerful master server.

Code: Select all
Parallel_Mode => conservative
slave_parallel_threads=256
slave_domain_parallel_threads=256
thread_concurrency=512


In this case, a 40-core master and a 4 core slave.


Seems flawless now. Thank you..