Havent had too many things to post about lately. Got a problem though that has been racking my brain. Relatively low utilized server. Way over spec'd for the job; 1 web/db 2 telephony. Only 25 agents 95% inbound calls. They will get about 20-30 calls that come in and when it does the calls sit in queue and dont distribute to agents. At first I thought the usual DB performance related problems. However when I do a processlist I can see that the DB has no queries locking it or anything. The problem is intermittent too.
I have tons of these in the mysql log(71 and 74 are the telephony servers):
2020-01-02 11:11:28 140424824690432 [Warning] Aborted connection 102294 to db: 'asterisk' user: 'cron' host: 'XX.XX.XX.71' (Got an error reading communication packets)
2020-01-02 11:11:32 140424825296640 [Warning] Aborted connection 104910 to db: 'asterisk' user: 'cron' host: 'XX.XX.XX.74' (Got an error reading communication packets)
2020-01-02 11:11:43 140424827557632 [Warning] Aborted connection 106076 to db: 'asterisk' user: 'cron' host: 'XX.XX.XX.74' (Got an error reading communication packets)
2020-01-02 11:11:43 140427583203072 [Warning] Aborted connection 104128 to db: 'asterisk' user: 'cron' host: 'XX.XX.XX.71' (Got an error reading communication packets)
Checked firewall rules, even added some blanket rules to allow wide open traffic between these servers. Doesnt appear that anything is getting blocked by the firewall.
Netstats:
- Code: Select all
eth1 Link encap:Ethernet HWaddr 00:25:90:56:0D:6B
inet addr:XX.XX.XX.70 Bcast:XX.XX.XX.79 Mask:255.255.255.240
inet6 addr: fe80::225:90ff:fe56:d6b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17867109 errors:0 dropped:0 overruns:0 frame:0
TX packets:13130435 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3669299611 (3499.3 Mb) TX bytes:4775770877 (4554.5 Mb)
Interrupt:18 Memory:fb900000-fb920000
- Code: Select all
Ip:
24914276 total packets received
3 with invalid addresses
0 forwarded
0 incoming packets discarded
24908982 incoming packets delivered
19608811 requests sent out
Icmp:
1826 ICMP messages received
5 input ICMP message failed.
InCsumErrors: 0
ICMP input histogram:
destination unreachable: 168
echo requests: 1648
echo replies: 10
1679 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 21
echo request: 10
echo replies: 1648
IcmpMsg:
InType0: 10
InType3: 168
InType8: 1648
OutType0: 1648
OutType3: 21
OutType8: 10
Tcp:
11615 active connections openings
182345 passive connection openings
210 failed connection attempts
255 connection resets received
89 connections established
24799318 segments received
20085108 segments send out
78517 segments retransmited
1099 bad segments received.
2338 resets sent
InCsumErrors: 1099
Udp:
156081 packets received
21 packets to unknown port received.
0 packet receive errors
156094 packets sent
RcvbufErrors: 0
SndbufErrors: 0
InCsumErrors: 0
IgnoredMulti: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
InCsumErrors: 0
IgnoredMulti: 0
TcpExt:
210 resets received for embryonic SYN_RECV sockets
9 ICMP packets dropped because they were out-of-window
ArpFilter: 0
33189 TCP sockets finished time wait in fast timer
53 packets rejects in established connections because of timestamp
32895 delayed acks sent
2 delayed acks further delayed because of locked socket
Quick ack mode was activated 66304 times
3 SYNs to LISTEN sockets ignored
2667 packets directly queued to recvmsg prequeue.
1897087 packets directly received from backlog
103663 packets directly received from prequeue
10591051 packets header predicted
56296 packets header predicted and directly queued to user
TCPPureAcks: 1388540
TCPHPAcks: 11598264
TCPRenoRecovery: 0
TCPSackRecovery: 661
TCPSACKReneging: 0
TCPFACKReorder: 1
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 232
TCPLossUndo: 358
TCPLostRetransmit: 4
TCPRenoFailures: 0
TCPSackFailures: 33
TCPLossFailures: 1
TCPFastRetrans: 658
TCPForwardRetrans: 19
TCPSlowStartRetrans: 14
TCPTimeouts: 2409
TCPLossProbes: 74305
TCPLossProbeRecovery: 4479
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 52
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 69082
TCPDSACKOfoSent: 947
TCPDSACKRecv: 68052
TCPDSACKOfoRecv: 79
TCPAbortOnData: 24
TCPAbortOnClose: 1
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 32
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 19
TCPDSACKIgnoredNoUndo: 64086
TCPSpuriousRTOs: 5
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 127
TCPSackMerged: 200
TCPSackShiftFallback: 1821
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 75118
IPReversePathFilter: 1
TCPTimeWaitOverflow: 0
TCPReqQFullDoCookies: 0
TCPReqQFullDrop: 0
TCPRetransFail: 0
TCPRcvCoalesce: 1896826
TCPOFOQueue: 4945
TCPOFODrop: 0
TCPOFOMerge: 947
TCPChallengeACK: 693
TCPSYNChallenge: 0
TCPFastOpenActive: 0
TCPFastOpenActiveFail: 0
TCPFastOpenPassive: 0
TCPFastOpenPassiveFail: 0
TCPFastOpenListenOverflow: 0
TCPFastOpenCookieReqd: 0
TCPSpuriousRtxHostQueues: 50
BusyPollRxPackets: 0
TCPAutoCorking: 15510
TCPFromZeroWindowAdv: 0
TCPToZeroWindowAdv: 0
TCPWantZeroWindowAdv: 0
TCPSynRetrans: 1605
TCPOrigDataSent: 16330189
TCPHystartTrainDetect: 7
TCPHystartTrainCwnd: 119
TCPHystartDelayDetect: 8
TCPHystartDelayCwnd: 334
TCPACKSkippedSynRecv: 129
TCPACKSkippedPAWS: 4
TCPACKSkippedSeq: 39
TCPACKSkippedFinWait2: 0
TCPACKSkippedTimeWait: 21
TCPACKSkippedChallenge: 4
TCPWinProbe: 13
TCPKeepAlive: 9
TCPMTUPFail: 0
TCPMTUPSuccess: 0
IpExt:
InNoRoutes: 0
InTruncatedPkts: 0
InMcastPkts: 0
OutMcastPkts: 0
InBcastPkts: 0
OutBcastPkts: 0
InOctets: 3844927566
OutOctets: 5007665001
InMcastOctets: 0
OutMcastOctets: 0
InBcastOctets: 0
OutBcastOctets: 0
InCsumErrors: 0
InNoECTPkts: 24943653
InECT1Pkts: 0
InECT0Pkts: 2690
InCEPkts: 14
Any ideas? I am just plain out of ideas at this point.