SYN flood on port 80. Disconnects all agents.

[ccabrera]

Hello,

A client of mine let me know of an event that occurred today at 9:45 AM, in which all agents at a given Vicidial Server got disconnected. I checked the dmesg and found this:

[11201.121850] TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters.

Since the server was rebooted at 6:34 AM, after adding the 3:06 hours into the event the time frame of the problem seems to match with the log record, so I´m assuming that the 2 events are related to each other.

I checked how many connections are established at TCP port 80 by using : netstat -tuna | grep ":80" | wc -l and right now the server is throwing 13,544 results at me. There are currently 80 agents logged into this server.


As far as I know this isn´t supposed to be a Vicidial issue, but rather a Kernel one. However, if anyone else has run into this error, I´d like to know if there´s a way to solve this (without offloading the apache to another server).

Server specs: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz, 12 cores, 54 GB RAM, usual load : 2.39, 2.21, 2.15. Vicibox 4.0.3 with Vicidial VERSION: 2.8-415a BUILD: 131007-1234.


Any ideas?

Regards,

[covarrubiasgg]

¿80 Agents in a Single All-in-one server?

At this point i would break into a cluster setup, but if you don’t want.

I have tried before to use nginx as a reverse proxy and it helps with the apache´s load.

You may also use a proxy cache in your network.

[ccabrera]

Actually, this client has 3x all-in-one servers with 80 agents each. Only this particular server is giving me a hard time, even though I have tweaked the apache config, from time to time it would disconnect all agents.

The top I have ran has been 120 agents in all-in-one with no issues.

I´ll look up nginx for this particular issue. Gracias por el tip Gabriel.

Regards.

[DomeDan]

13544 connections on 80 agents sounds a bit high.

What version of vicidial are you running and what install method?

if you got this server open to the internet then consider using a whitelist http://www.viciwiki.com/index.php/DGG

[geoff3dmg]

If this traffic is genunie the correct thing to do is to alter net.ipv4.tcp_max_syn_backlog until you no longer get this warning. It would also be prudent to tune the TCP stack to your specific load in a more general sense (Specifically net.core.rmem_max, net.ipv4.tcp_rmem and net.ipv4.tcp_wmem).

Otherwise, yes, look at implementing IP Tables.