Complete Loss, Asterisk LOAD THROUGH The Roof

[Nefariousparity]

Install Disc: ViciBox_v.7.x86_64-7.0.3.iso
VERSION: 2.12-559a
BUILD: 160611-2230
Version: 2.12b0.5
SVN Version: 2552
DB Schema Version: 1461
DB Schema Update Date: 2016-06-13 20:53:04
Asterisk:11.22.0-vici

4x Servers Dell Power Edge 2950 III
1x Web 1x Database 2x Telephony

Database Specs PowerEdge 2950 III
8 SFF 10K Raid 10
32GB memory
dual quad core Intel Xeon X5460 3.16

Telehony Specs
They are the same, except only two drives SSD RAID 0

1X SuperMicro Telephony
2x Sata 7200 RPM 1TB Raid Zero
Intel Xeon E5420 @ 2.50GHZ

Everything is updated via zypper up -y.

We utilize phone aliasing. So agent can do 8000 and it puts them where needed server a b or c. This was a backup restore database migration to upgrade vici to utilize the new asterisk. Something that is pretty routine and have not had issues with yet. Right out the gate on the intial install, asterisk servers won't call agents. So I start complete from scratch again and do a fresh install without database import. Loads across all three servers spike to 20+ as soon as any real dialing starts, I feel I have adjusted everything. The only way to keep the loads down is to turn off recordings, set to no AMD, and set trunks to 50 and cps to 3 on all three servers. I have had much higher settings with less.

Do I need to burn down these Telephonys and restart over? I just don't get it, I'm running out of options. I have good guy white list setup,(Which I have turned off and on), no we are not out of drive space. Internet is 100MBIT, network is gigabit cat 6.

ScreenShot
http://imgur.com/aTHUMHn

[Kumba]

No clue. What were the trunks and CPS set to prior to your adjustment of them?

Also, sounds like on your import you didn't put in the different asterisk version in the servers page. So it was running Asterisk 11 but thought it was Asterisk 1.8, or something like that.

[Nefariousparity]

Hi Kuma, yeah It has the correct asterisk version in the servers page. Its blowing my mind, honestly I am wondering if it id s DISC or OpenSuse Prition thing, its so strange.


Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 9.9M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda3 215G 62G 152G 29% /
tmpfs 6.0G 5.4M 6.0G 1% /var/spool/asterisk/monitor
/dev/sda1 118M 51M 61M 46% /boot

[Nefariousparity]

Interesting enough, we just pulled a recording and it sounds clear?

[Nefariousparity]

I just noiced, that vicidial-conferences has correct conferences. But conferences one server has duplicate.

[Nefariousparity]

Also noticed this

Jun 15 16:37:13 flwtel03 kernel: dahdi: Version: 2.11.1
Jun 15 16:37:13 flwtel03 kernel: dahdi: Telephony Interface Registered on major 196
Jun 15 16:37:13 flwtel03 vicidial[1495]: Loading dahdi_dummy module... failed.
Jun 15 16:37:13 flwtel03 vicidial[1495]: Fallback to dahdi module... done.

Does that seem normal?

[Nefariousparity]

So here is a question, call between agent and customer is choppy, but the recording of said call is clear as day?

[Kumba]

Drive I/O issues will cause phantom load. The kernel will see wait states on the drives and artifically count it as CPU blocking which drives the loadavg up.

Try running 'iostat -dkx 1' and see what kind of Drive utilization you have going on.

[Nefariousparity]

Yeah so incredible strange, I ended up having to foot the bill for a colocated server at high velocity for now.

Express-Install
3.7ghz 32GB 1X 980GBSSD

Seems to run fine, until they push alot of channels but it is working.

I rebuilt the raid arrays in the dells, reparititioned and formated. I am going to be working on it.

[Nefariousparity]

Still no luck after the rebuild. Totally strange. This cluster was working fine running asterisk 1.8 Vicibox 6.03 I believe. Now running asterisk 11. Soon as 30 agents login spread across 3 or 4 telephony you would think based off the call quality we had all 30 on a single Pentium 3 install. What the heck man!

[williamconley]

turn off sqlite in the asterisk modules if the install is very old and asterisk "delays" during startup.

when you say "rebuild", did you bring your database along for the ride? Or did you just load leads? I would point out that bringing your database along for the ride could certainly "infect" the newly built system with an old problem.

what process is showing the load? problems differ based on loading ... apache2, mysql, asterisk ...

I've had a system overload completely from a simple 'dead link' in the dispo_url field. LOL

[Nefariousparity]

Wow sorry for the late reply. So what I normally do is download the latest and greatest Vicibox Install, build the cluster, then drop the the asterisk database on the fresh install. Create a new Database. Then I import previous database, run update Ip, script. Sometimes I have to empty conferences, and vicidial_conferences table and phones table. So I use excel sheet to recreate those with new ips. Phones I use bulk insert.

Oh and I update web link for recordings.

Load would always be asterisk. I could not figure it out. Now we are on new servers which are humming along. Just something that was routine for me just totally exploded.