Drop% on realt-time Breaks Midway Through the day

[streetboy]

Hey guys,

so this is a cluster of 3 dedicated servers
1 WEB/DB
2 Asterisk

VERSION: 2.14-865a
BUILD: 220831-0850
Version: 2.14b0.5
SVN Version: 3636
DB Schema Version: 1666


Everything is working fine. Only problem is that the drop% on the real time report breaks midway through the day.

In the start of the shift it is working fine, showing correct stats and then in the middle of the day it will start showing wrong %(showing high %), when we generate the Outbound Calling report, it is showing correct stats. and then after 2,3 hours it will again show the correct stats.

Screens on WEB/DB Server:
There are screens on:
1533508.ASTconf3way (Detached)
1533504.ASTVDadapt (Detached)
1277983.ASTVDremote (Detached)
1046289.ASTemail (Detached)
182592.ASTVDauto (Detached)
182578.ASTlisten (Detached)
182574.ASTsend (Detached)
182571.ASTupdate (Detached)
182189.asterisk (Detached)
182024.astshell20221025141003 (Detached)
181961.ASTVDadFILL (Detached)
181953.ASTfastlog (Detached)
12 Sockets in /run/screen/S-root.

Any suggestions?
Thank you

[jamiemurray]

This used to happen at the top of the hour but it was fixed in 3480 and there was another amendment to the dropped call calculation in 3542.

Do you think your issue could actually be what's described on this post?
viewtopic.php?f=4&t=40777&hilit=dropped+calls+on+the+hour

[streetboy]

am already at 3636 and its still happening.
Any other suggestions what should I check...

[jamiemurray]

Is the issue the same though, as in at the top of the hour for a minute or two the stats go wrong or does it last longer than that?

[streetboy]

Yes brother the issue is same. And yes it does last longer, it sometimes goes for more than an hour, than shows the right stats. And then happens again after an hour or so.

First my thought was maybe the DB was being overloaded so separated it from the asterisk. Now 64GB of RAM for WEB/DB. Server load never goes above 40% with 30-40 dial level 600-700 calls dialing at a time.

Already did DB optimization (mysqlcheck -u cron -p'1234' --auto-repair --optimize --all-databases) many times, but it doesn't fix the real-time stats.

[jamiemurray]

This is a bit of a tricky one to find because of the time sensitive nature of it.
It would be ideal to see the output of the adapt script with debugXXX at the time the stats change from correct to incorrect. I believe this should give some clue.

[streetboy]

So I have seen that the vicidial_campaign_stats table is being populated with wrong stats.

These are the problems I am facing:

1) Under Real-time report stats, the Drop/Answered calls are always less than the number in the Outbound Calling Report which is at 150/1500.
2) Sometimes the real-time report stats(Drop/Answered) goes even less than what it was showing 2 minutes ago. for example, if dropped calls/answered was 100/1000 it will suddenly start showing as 60/600.

During all this time am following up with the vicidial_campaign_stats table, it is showing 60/100 at the moment and it was showing 100/1000 earlier and the Outbound Calling Report is showing 150 Dropped and 1500 Answered.

How do I further troubleshoot this and dig into it to get to the bottom of this? Any suggestions, please...