Vicibox 8.1 issues

[dgroth02]

Ok. Its been a grueling 48 hours.

Had a perfectly operating cluster on Vicibox 7.

1DB/Web (Dell r410, dual Xeon quad core, 128 GB RAM, Raid 10 SSD, quad NIC)
8 Telephony (Dell r610, dual or single Xeon quad or hex core - raid 1 SSD Drives)
100 Agents spread across 4 servers, dialing at 10 to one, but lines spread across all servers weighted more towards servers with more resources and fewer agents.

Had a minor hardware issue monday (just needed to replace a disk) so figured it was a good time to upgrade to Vicibox 8 to get the new features and abilities (including the ability to use the webphone without a ton of upgrades and so on). I should add at this time that the system was still functioning, but I didn't want the system to operate with what could be a "failing" disk.

Steps I took:
Backed up system as was.
Upgraded to latest build / SVN on all servers (VERSION: 2.14-694a, BUILD: 181005-1738 , SVN version 3051, DB schema 1561)

THEN - backed up new database.

Proceeded to install Vicibox v8 on all servers, starting with the database. And, after a 8 hour painful data restore (huge amount of log files) - the system was up at 7:00 am the next day.

The agents would receive calls that had no one on the other end. They couldn't leave from a transfer. The database appeared to be running at a much higher utilization (8 cores, 500% - never used to get over 250%)
Agents would be automatically paused out

Checked time/NTP - all was good.

Tried adjusting amount of key cache on database, and using old database tuning paramenters. No good. Even checked parameters from forums from others that have run into any issues with Vicibox 8/8.1

Overall the call center manager was PISSED. I'm in the process of reinstall v7 vicibox and going backwards.

Does anyone see in this process something inherent that I missed. I've read the hardware specs. I've been installing/maintaining Vicidial for 10 years and this is the FIRST time I've had to fight it so hard and troubleshoot so many issues after an upgrade.

[thephaseusa]

I waited till 8.1.2 was released, and also upgraded from 7.0.4 to 8.1.2. I thought it was just me or my DB server because I also noticed that the load went up by 25% to 50% on the mysql server. I went back to 7.0.4 for a week and installed 8.1.2 on one box for testing, and the next weekend I upgraded the cluster again. I really wanted to try out viciphone and certbot and the new firewall. I dont know if its the Maria DB version or OpenSuSE Leap 42.3 but I am experiencing an increase in load on the db server also.

[dgroth02]

I was wondering the same. The database usage was MUCH higher. Possibly a combination of new SVN, OS, and database. The old version is now installed - restoring database with new SVN now and will see how it goes tomorrow.

[dspaan]

Interesting post, i only did test installs of vicibox 8.1.2 so i haven't tested it in production yet. This should be investigated. Shouldn't this thread be in the vicibox forum?

[williamconley]

Ok. Its been a grueling 48 hours.

...

Proceeded to install Vicibox v8 on all servers, starting with the database. And, after a 8 hour painful data restore (huge amount of log files) - the system was up at 7:00 am the next day.

1) you did not specify the actual .iso from which you installed. 8.1.0? 8.1.1? 8.1.2? For posterity as well as any ability to tell you "that version has known bugs", this is important information. Plus, of course, it's good Support forum etiquette to provide details when available. The well-thought-out full text description was excellent. Just that piece missing.

2) This is the General Discussion board, and while the lengthy description makes it seem like a discussion, it's really a support request. You're not asking if Vicidial is a good Christmas gift here, lol. Moving to the Support Board.

3) NEVER upgrade an entire cluster at once to a new version of the installer. Part of the beauty of "device/OS independent Open Source software" is that you CAN have Vicibox 6.0.3 on one server, 7.0.4 on another, and 8.1.2 on another ... all in the same cluster. So to avoid this sort of SNAFU: Upgrade the Entire Cluster to the latest version of Vicidial one day. Let it run to be sure nothing's broken (limiting the amount of repair work that may be necessary). Then when you're comfortable either upgrade one existing dialer to v8.1.2 OR add a whole new server running v8.1.2 (so you can just shut it down if there's a problem). Once it has been determined that the dialer is happy, consider (if needed) upgrading all the other dialers. Then repeat the process for each of the other server roles. Takes a long time, but as the good Dr. Charles said: "Do one thing at a time. Do it well. And move on".

4) There are some underlying operating system differences that can (and will) contribute to differences in load and treatment of each of the various packages. For instance, multi-threading has been reworked in MariaDB. So changing one server role at a time would allow you to fight smaller battles instead of an entire war at once (and adding a new server with that role instead of rebuilding an existing one would theoretically allow you to "go back" if needed). Even if the role in question is DB: replication is a cool thing. But replication from "new master" to "old slave" to try to keep the original DB server up to date would likely be problematic (new master old slave is NOT supported and may fail, depending on how different the versions are, which is a bummer).

[dgroth02]

1) 8.1.2
2)I apologize for posting it the wrong board. Noted for future reference.
3) I wasn't sure if this was possible as it was my understanding that the database was the "main" culprit in this upgrade and I wasn't sure that upgrading an individual dialer would be most prodent as I wasn't sure if that would affect the other dialers (vis-a-vis the database setup). I'm not using database replication for the record.

[thephaseusa]

Sorry you didn’t have a good experience with the upgrade. 100 agents at 10:1 you are dialing 1000 concurrent calls on 8 asterisk boxes. You guys are kicking ass over there)). When you decide to upgrade to Vici8 again is it possible that adding a couple more asterisk boxes to the cluster might make it an easier transition?

[williamconley]

3) I wasn't sure if this was possible as it was my understanding that the database was the "main" culprit in this upgrade and I wasn't sure that upgrading an individual dialer would be most prodent as I wasn't sure if that would affect the other dialers (vis-a-vis the database setup). I'm not using database replication for the record.

The beauty of the way The Vicidial Group has designed this system is that each server in a multi-server cluster is independently maintained. As long as the Vicidial scripts are on the same SVN revision level, all will run smoothly. The installer's job is to provide an environment that will allow all the scripts to run properly. That environment can be any linux distro with any version of any of the necessary software. Limitations are few: For instance the Asterisk configuration script generator has been coded to allow for Asterisk versions 1.2 through 13.

[Kumba]

Try disabling the spectre and other VM hosted exploits. These have significant impacts on I/O.

You'd want to add these to the kernel commands: nopti nospectre_v2 nospec

[williamconley]

Try disabling the spectre and other VM hosted exploits. These have significant impacts on I/O.

You'd want to add these to the kernel commands: nopti nospectre_v2 nospec

Um ... is this in the wrong thread?

[alo]

Did you use webphone on this installation?

I have been noticing trouble leaving threeway calls too. and even worse the transfers not making it over to the closers at all. A reboot seems to fix it for a few minutes, but within an hour the agents have trouble again. but this is using vicibox 8.0.1 actually and not 8.1. however I do have asterisk 13 (but the leave threeway issue was happening on asterisk 11 too). But I think its either webphone or something in the newer SVNs. I want to move off of webphones to test but its harder to facilitate that test. The crazy part is spreading the agent load on to more dialing servers seems to resolve this. It doesn't matter if we move all the channels to different dialing servers, it has to be moving the agents to other dialing servers or it will still happen.

[Kumba]

Try disabling the spectre and other VM hosted exploits. These have significant impacts on I/O.

You'd want to add these to the kernel commands: nopti nospectre_v2 nospec

Um ... is this in the wrong thread?

No. The spectre and other VM mitigation patches they put into the kernel can have significant impact on I/O even though the OS isn't being virtualized. They are on by default and have to be disabled through the kernel's command line when it's started.

[williamconley]

So Vicibox doesn't disable these by default?

Perhaps a good idea to ask or check for VM and set those during install. Or is the kernel precompiled in vicibox?

[Kumba]

Somewhere around Linux Kernel 4.4.110 the Specrte patches made it in and are enabled by default if the CPU is found to be vulnerable to Spectre on boot. The patches affect CPU and I/O subsystems dramatically on a virtual machine, but the CPU impact is minimal on real hardware. The I/O impact can still be quite significant even though you are running on dedicated hardware. I/O includes both hard-drive access and RAM access.

[williamconley]

I would like to suggest that to the average user, I/O and CPU impact feel the same. Eventually the system overloads.

[Kumba]

I would like to suggest that to the average user, I/O and CPU impact feel the same. Eventually the system overloads.

True, and if you compare an old version of ViciDial to a new version of ViciDial the new version alone will have more DB activity. So there's lots of moving parts to consider.

[thephaseusa]

Kumba you my friend are a genius. The kernel parameters you suggested seem to have reduced my load by as much as half.

First question, how was I supposed to know that???))))

Second question, what file do i edit to make those boot parameters permanent?

To make permanent i opened yast and went to System, Boot Loader, then Kernel Parameters then Optional Kernel
Command LIne Parameter and added at the end of the line nopti nospectre_v2 nospec
Then clicked ok

John

[alo]

To make permanent i opened yast and went to System, Boot Loader, then Kernel Parameters then Optional Kernel
Command LIne Parameter and added at the end of the line nopti nospectre_v2 nospec
Then clicked ok

Did this work for you? I want to test this change but do not have physical access to the server.

[thephaseusa]

Yes like I said it reduced the load by half.

You dont need physical access as long as you have shell access.

Make the changes in yast and reboot.