Update from SVN2122 to SVN2537 - problem with apache server

[ZibiX]

Hi.

Yesterday i have made an update from SVN2122 to SVN2537 on my client cluster (1 x db, 1 x web, 5 x asterisk).

First problem that happen was message on web browser (when couple dozens of agents wanted to log in): MySQL connect ERROR: Cannot assign requested address

I have solved this by applying this on db and web server (according to this topic: viewtopic.php?f=4&t=31137):
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

The second problem is very high load on web server. Also processes on this server consumes almost all RAM (8GB).

Before the update, loadav on the server was low and memory usage was below 4 GB.

Do i need to modify config on apache after the update?

[ZibiX]

UPDATE:

In error_log on web server I have got thousands of this errors:
File does not exist: /srv/www/htdocs/agc/vicidial_stylesheet.css,
File does not exist: /srv/www/htdocs/agc/vicidial_stylesheet.css

and every few minutes:
PHP Warning: mysqli_connect(): (08004/1040): Too many connections in /srv/www/htdocs/agc/dbconnect_mysqli.php

[mflorell]

The vicidial_stylesheet.css issue should be fixed now in svn/trunk

As for the mysql connection issues, yes the newer code(which switched the PHP "mysql" connection functions to "mysqli") does use more resources, but since PHP's "mysql" functions were unsupported and were going to be discontinued, we didn't have much choice in upgrading to "mysqli". So, you will need to both increase the MySQL max connections and increase the maximum open files used on your apache processes.

For most small to medium clients there was no issue, we only noticed this problem with our larger clients.

How many agents do you have logged in at any given time?

What are the hardware specs of your database and web servers?

Are you using any virtualization on any of your servers?

[ZibiX]

I have 120 agents logged in.

Hardware specs:
- database: i7-3770 3.40 GHz, 8 GB RAM, HDD 7200 RPM
- web: i3-3210 3.20 GHz, 8 GB RAM, HDD 7200 RPM

NO virtualization on any server.

One more information:
Database and Web servers are Vicibox 4.0.3

Questions:
- MySQL max connections are: max_connections in /etc/my.cnf ? I have set this to 2048
- Maximum open files used on my apache processes are settings in /etc/apache2/server-tunning.conf in prefork section?

[mflorell]

What kind of RAIDs are you using? Are you really only using one 7200RPM drive for your DB server?

[ZibiX]

On database server I'm using 2 x 250 HDD 7200RPM on software raid mirror (mdadm)

[mflorell]

Given the number of agents you have, I would recommend a LSI Logic MegaRAID card with faster harddrives. You are effectively running on the speed of a single 7200RPM drive, which cannot sustain the data throughput or seek speed necessary for 120 concurrent agents.

[williamconley]

On database server I'm using 2 x 250 HDD 7200RPM on software raid mirror (mdadm)

Software RAID? That's crazy. Counterproductive, slow and will cause instability. This is not a web server.

If you remove the software mirror you may regain enough to run your system. Of course 15k SAS or SSD is better. If you require redundancy, however, RAID 10 as specified by Matt is best (well-tested and proven) including the brand name of the RAID card and 15K SAS or SSD for all drives.

[ZibiX]

Ok, thank you for your suggestion. Today problem still exist: loadav on web server increased 100 times comparison to web server before update.

[ZibiX]

UPDATE:
After increasing values in my.cnf and in apache config web server is more loaded (cpu 100 % all the time and memory usage goes to 8 GB in 15 minutes).

I have also noticed that in htop number of tasks increased from 700 to 900.

Maybe the problem is that apache do not kill "unused tasks" and that is way the server is overwhelmed?

[ZibiX]

UPDATE2:
Strange thing

I have restored apache config to:
<IfModule prefork.c>
ServerLimit 768
StartServers 450
MinSpareServers 250
MaxSpareServers 500
MaxClients 768
MaxRequestsPerChild 800
</IfModule>

And now everything is running OK. MaxRequestsPerChild was 1200 and now 800.

[ZibiX]

Hi.

Unfortunately problem still exist (74 agents calling):

[frequency]

Issue is the software raid. Better switch to Raid-1 or Raid 10 compelely with a card. If you cannot run a card, run without the RAID 0 which i will never ever recommened with a SSD load average shall be low and not 266.00 It should be equivalent almost to the cores of the processor. I am pretty sure your cpu usage would be very low and I/O wait time (%wa) time would be very high which is causing the load average to go up. I am pretty sure your agents would notice LAG in voice or problem in displaying data in the leads too.

[ZibiX]

frequency, this server is a web server without RAID. I have problem only with this server after update from SVN 2122 to SVN 2537. Before update 120 agents were working on this cluster without any problem. Now problems occur even with 50 agents.

I have lower settings in prefork section of apache config:
<IfModule prefork.c>
StartServers 350
MinSpareServers 150
MaxSpareServers 400
ServerLimit 768
MaxClients 768
MaxRequestsPerChild 800
</IfModule>

Then I have restarted apache and suddenly loadav dropped from 266 to 0.5. Tasks remain at 770 but running tasks dropped from 300 to 2-7 in htop.

First question: why this settings have so much impact on server performance?

I have only two big problems with apache:
1. A lot of this error in error log:
[error] [client xx.xx.xx.xx] File does not exist: /srv/www/htdocs/agc/simpletreemenu.js, referer: http://xxx.com:51522/agc/vdc_chat_display.php...

2. Every day at 10:45 apache restarts with this error in error_log:
[warn] child process 2604 still did not exit, sending a SIGTERM
[notice] SIGHUP received. Attempting to restart

[williamconley]

You may want to verify that all your "dispo_url" values are valid and able to be accessed from your server. I've bumped into this problem a couple times when a new server was unable to get through a firewall to access the dispo_url which the new server inherited from a prior server's Campaign entries in the database. Then the apache process attempts to pull the dispo url ... and the timeout problem causes all these unprocessed requests to backlog forever. Overload on CPU, system failure ...

So find out what in the apache application is causing this overload (or verify your dispo_url's specifically .. right now ... maybe you'll get lucky!).

And if that's the problem, put in an Issue Tracker request to have dispo_url with a timeout value to avoid pileups.

[ZibiX]

williamconley, if you are reffering to Dispo Call URL in Campaign Detail View, I have this fields empty.

Lowering prefork settings seems to do the job.

Also I have found the reason to restart apache at 10:45 everyday. It was happening because logrotate. For unknown reason logrotate happen everyday at 10:45. I have set parametr in /etc/sysconfig/cron to:
DAILY_TIME="22:00"
and viola! Now compressing logs happen at night.

[williamconley]

Anther option, of course, is to turn off logging when not actively debugging the system. Asterisk has logging, apache has logging, iptables has logging, Vicidial has logging, mysql has logging ... some of these even have more than one type of logging. Turning them all off (and knowing how to turn them back on again if you need them) is ... handy on major installations.