HUGE amount of MySQL TIME_WAIT connections building up
Posted: Tue Nov 22, 2022 5:13 amHi,
I have a cluster of 2 servers (DB/WEB/ASTERISK + WEB/ASTERISK). The CC runs an autodial camp for about 80 agents and everything is running fine without any noticeable issues currently.
However, running 'netstat -plant' yields the following:
This totals in more than 3000 connections to port 3306 that are in the TIME_WAIT state.
/var/log/mysql/mysqld.log shows the following entries:
Is this normal behavior or something to be concerned about that can cause issues? if so how should i go about mitigating the issue?
VICI DETAILS:
VERSION: 2.14-869a
BUILD: 221021-1038
SVN: 3651
I have a cluster of 2 servers (DB/WEB/ASTERISK + WEB/ASTERISK). The CC runs an autodial camp for about 80 agents and everything is running fine without any noticeable issues currently.
However, running 'netstat -plant' yields the following:
- Code: Select all
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5038 0.0.0.0:* LISTEN 12764/asterisk
tcp 0 0 0.0.0.0:2929 0.0.0.0:* LISTEN 12775/ip_relay
tcp 0 0 0.0.0.0:18421 0.0.0.0:* LISTEN 1595/sshd
tcp 0 0 0.0.0.0:40569 0.0.0.0:* LISTEN 12773/ip_relay
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2376/master
tcp 0 0 0.0.0.0:4577 0.0.0.0:* LISTEN 1622/perl
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 20339/zabbix_agentd
tcp 0 0 0.0.0.0:17543 0.0.0.0:* LISTEN 12774/ip_relay
tcp 0 0 127.0.0.1:5038 127.0.0.1:35598 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26338 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:23950 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25418 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29180 1x9.xxx.xxx.8:3306 ESTABLISHED 16805/httpd-prefork
tcp 0 0 1x9.xxx.xxx.8:26240 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26076 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25538 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29716 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26352 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29532 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28686 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26332 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 18 0 127.0.0.1:5038 127.0.0.1:35226 ESTABLISHED 12764/asterisk
tcp 0 0 127.0.0.1:5038 127.0.0.1:36672 TIME_WAIT -
tcp 0 0 127.0.0.1:5038 127.0.0.1:32760 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24164 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:27668 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25960 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28198 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25710 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24128 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24474 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28448 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24592 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24522 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25602 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28436 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28820 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26254 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28930 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:27926 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25358 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28896 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29280 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:27428 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:27788 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:27602 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26508 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28750 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24324 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24854 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26176 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29512 1x9.xxx.xxx.8:3306 ESTABLISHED 6838/perl
tcp 0 0 1x9.xxx.xxx.8:29186 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 127.0.0.1:5038 127.0.0.1:33526 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29436 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 127.0.0.1:5038 127.0.0.1:30802 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29686 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28514 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26598 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25416 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25406 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24920 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:25766 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24062 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:28774 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29268 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26370 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:23824 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24136 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26160 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:29650 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 127.0.0.1:4577 127.0.0.1:43910 ESTABLISHED 2282/perl
tcp 0 0 127.0.0.1:5038 127.0.0.1:34888 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:26602 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24174 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp 0 0 1x9.xxx.xxx.8:24168 1x9.xxx.xxx.8:3306 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:48924 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:49118 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:48818 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:47554 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:49026 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:48682 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:48072 ESTABLISHED 1590/mysqld
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:46190 ESTABLISHED 1590/mysqld
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:49326 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:47250 TIME_WAIT -
tcp6 0 0 1x9.xxx.xxx.8:3306 1x9.xxx.xxx.20:47272 TIME_WAIT -
This totals in more than 3000 connections to port 3306 that are in the TIME_WAIT state.
/var/log/mysql/mysqld.log shows the following entries:
- Code: Select all
2022-11-22 11:32:25 140376940181248 [Warning] Aborted connection 115749182 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:33 140376945350400 [Warning] Aborted connection 115749816 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:45 140376920262400 [Warning] Aborted connection 115750653 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:32:48 140376923641600 [Warning] Aborted connection 115750833 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:48 140376512866048 [Warning] Aborted connection 115750881 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:00 140375882516224 [Warning] Aborted connection 115751626 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:22 140375905179392 [Warning] Aborted connection 115753233 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:22 140377384380160 [Warning] Aborted connection 115752922 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:26 140375881901824 [Warning] Aborted connection 115753151 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:26 140375843440384 [Warning] Aborted connection 115753176 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:27 140376937260800 [Warning] Aborted connection 115753258 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:28 140376928933632 [Warning] Aborted connection 115753266 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:30 140375863817984 [Warning] Aborted connection 115753407 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:33 140376923948800 [Warning] Aborted connection 115753590 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:33 140376926783232 [Warning] Aborted connection 115753611 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:36 140375902414592 [Warning] Aborted connection 115753749 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:39:18 140375881594624 [Warning] Aborted connection 115778154 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:39:18 140376922105600 [Warning] Aborted connection 115778187 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
Is this normal behavior or something to be concerned about that can cause issues? if so how should i go about mitigating the issue?
VICI DETAILS:
VERSION: 2.14-869a
BUILD: 221021-1038
SVN: 3651