Agent Logs Missing

[mmirasol]

VERSION: 2.12-555a
BUILD: 160508-1948
© 2016 ViciDial Group
Asterisk 1.8
Ubuntu Server 12.04.5 LTS 64bit

Good day everyone.

We noticed something strange with our vici logs just recently. There are leads with vicidial call logs but no agent logs even though the vicidial log shows they're assigned to the agents. Screenshots below.
https://snag.gy/JA0txc.jpg
https://snag.gy/KGpEwD.jpg
https://snag.gy/HUud1P.jpg

The leads just like all the other leads and the users have other call logs that look fine. The logs for these calls in our vicidial_log table are shown below:
https://snag.gy/CeQgFI.jpg
https://snag.gy/f3l2k1.jpg

Please advise why this is and what we should check.

Best Regards,
Mark

[mflorell]

Check for crashed tables:
# /usr/share/astguiclient/AST_DB_check_tables.pl --help
allowed run time options:
[--quick] = run test in QUICK mode, may not catch all issues
[--email-list=test.com:test2.com] = send email results to these addresses
[--email-sender=vicidial] = sender for the email results
[--test] = test
[--quiet] = quiet
[--debug] = verbose debug messages
[--debugX] = extra verbose debug messages

Also, make sure you didn't run out of space on your database drive partition.

[Rogger]

hello,

Here are some steps for you to test.

1) Check you data base like Matt Florell wrote and try this: mysqlcheck -cA -uroot -p --databases stop and start your MySQL and look at the logs for errors.
2) look at your OS logs for errors.
3) look if your dialplan is missing this line [ AGI(agi://127.0.0.1:4577/call_log) ].

Good luck,

Rogger Faioli
gosat.org

[mmirasol]

Thank you, mflorell and Rogger. We confirmed that the tables did get corrupted on 12/20. However, what's strange is that for some reason even after we addressed that, we still see calls missing from the agent log. Is it correct to assume that any call in the vicidial_log table with a user assigned should also be in the vicidial_agent_log table?

We still see several records throughout the day that are missing in our agent logs table on 12/21.
https://snag.gy/z5LxHc.jpg

Please advise.

[mflorell]

The version you are using has known bugs related to this that were fixed last year. I would strongly recommend upgrading to the latest svn/trunk revision.

[mmirasol]

Correction. Our current build is:
VERSION: 2.14-636a
BUILD: 171011-1450
© 2017 ViciDial Group
Asterisk 1.8
Ubuntu Server 12.04.5 LTS 64bit

Please confirm if this would still be an issue.

[mflorell]

That version should be fine, but it is possible for records in vicidial_log with a user set to not have a vicidial_agent_log record in some cases.

To begin to figure out what the issue is, I recommend enabling the Agent Screen Debug Logging in System Settings.

[mmirasol]

Agent Screen Debug Logging -This setting if enabled will log almost all agent screen mouse clicks and AJAX processes triggered by the agent screen. To enable for all agents, set this option to 1. To enable only for one agent on the system, set this option to the user that you want to log. Warning, this feature can log hundreds of entries per phone call, so use with caution. These agent debug records are deleted after 7 days. Default is 0 for disabled.

so this setting will log mouse clicks and ajax processes. this will confirm if for some reason the process fails or timeouts on dispo for each call correct?
thank you for the advise. we will try this and report back. will this cause call delays?

i'm also wondering if this could be a mysql optimization issue. mysql was recently restarted so i can't check the stats, but previously i saw there were aborted clients and failed attempts.
our dialer generally makes around 200 - 300k calls per day so i'm wondering if our setup is not optimized properly for this volume.

[mmirasol]

That version should be fine, but it is possible for records in vicidial_log with a user set to not have a vicidial_agent_log record in some cases.
> can you advise what these cases would be?

The logs that are missing look very similar to the ones that were recorded. We don't see anything particular with leads, users, campaign or status they're in.
What's worse is that we see even SALES calls not recorded so it's a critical issue.

https://snag.gy/MIR5nU.jpg

[mflorell]

It could certainly be a database or network or some other issue. Missing vicidial_agent_log records isn't normal, and it doesn't happen frequently at all for our larger hosted clients. When I looked for examples of this on a few of our larger systems, almost all of them are status INCALL calls, meaning that there was some kind of agent screen issue causing the problem. One of our clients that places 100,000+ calls per day didn't have a single missing agent log record in the last two weeks.

[mmirasol]

Thanks for confirming, Matt. We will check for mysql and network issues and also look at agent screen logging if we are unable to resolve it.
Also could you advise if our db server specs is enough for the volume of calls?

CPU Type Xeon 8 Core E5-2650 2.00GHz
Total RAM 16 GB
Operating System Ubuntu Server 12.04

[mflorell]

What kind of drives/RAID-card are you using on that server?

[mmirasol]

Processor: Xeon 8 Core E5-2650 2.00 GHz
Memory: 16 GB DDR3-1600 ECC SDRAM
Storage: RAID 10 SSD - 400GB X 2
Bandwidth : 20 TB Bandwidth

[mmirasol]

Yesterday (2018-01-02), we made 204043 calls. Our mysql variables looked like this.
https://snag.gy/wdkPih.jpg

Only 1 failed attempt but again comparing our vicidial_logs with our agent logs we still see several logs missing, 2 of them are SALES calls.
Looking at one of them we see that the record is in both the vicidial_log and vicidial_agent_log tables but can't be matched because the record in vicidial_agent_log has no uniqueid.
https://snag.gy/4PStD7.jpg

But reporting-wise we only read data from vicidial_agent_log so it should be ok.
Is it normal for entries in vicidial_agent_log to not have uniqueid's in some cases?

[mmirasol]

UPDATE: we confirmed that the missing logs are in the vicidial_agent_logs table. We just were not able to match them against the vicidial_log table because the records did not have uniqueid's for some reason. As for why, it would be interesting to know but is not critical.

However, we did have corrupt db tables on 12/20 so I would to ask what causes this and how to avoid them in the future. Kindly advise. Thanks.

[mflorell]

corrupt tables can be caused by several different things, from unscheduled shutdowns to issues with the server like bad memory or running out of space on the hard drive.

[mmirasol]

Hello Matt. What's your recommendation for the calls that show as INCALL in vicidial_log and no status in vicidial_agent_log? Would those be failed ajax calls?

[mmirasol]

For the ones with records in vicidial_log and no record in vicidial_agent_log would those be in the same category still?

[mmirasol]

We're missing around 0.3% of actual agent calls in our vicidial_agent_log every day but our dialer seems to be dialing the calls well. However, because of the missing logs we are unable to generate accurate reports so we have to run a script to create the missing logs at the end of the day. Would you say this is normal for a single vicidial cluster with our current call volume of 200-300k per day? Would you recommend splitting up the calls into multiple clusters or perhaps adding more webservers or increasing the specs of the db server to address the issue?

[mflorell]

What is the status of most of the calls that are missing in vicidial_agent_log?

INCALL status calls means something went wrong on the agent screen. That could mean the agent workstation crashed, the browser crashed, the network was disconnected, etc...

0.3% of agent calls missing in logs I've only seen when there are horrible agent network connections, making their sessions go LAGGED frequently, and causing all kinds of other issues.

As for options to mitigate these issues, it's hard to say without figuring out the cause.

[mmirasol]

Out of the 0.3% missing calls,
66% do not have records in vicidial_agent_log
33% have a record but have no matching uniqueid

Out of the 33% (500 calls TOTAL),
Status Count
A 226
B 3
CALLBK 48
DC 1
DNC 23
DNQ 5
DROP 1
HNGTX 1
HNGUP 21
INCALL 41
N 105
NI 4
RDNC 3
SALE 12
SALED3 1
U 3
WN 2
Grand Total 500

[mmirasol]

Hello matt,

Would like to revisit your comment:

It could certainly be a database or network or some other issue. Missing vicidial_agent_log records isn't normal, and it doesn't happen frequently at all for our larger hosted clients. When I looked for examples of this on a few of our larger systems, almost all of them are status INCALL calls, meaning that there was some kind of agent screen issue causing the problem. One of our clients that places 100,000+ calls per day didn't have a single missing agent log record in the last two weeks.

And ask, what should we do this in the case of INCALL calls cos you mentioned you have those in your larger setups too.

[mflorell]

Missing INCALL agent log entries are somewhat normal, because that means that the agent screen lost its connection with the vicidial system. I say only "somewhat" normal because there is nothing to log if the agent screen disappears, since it is the agent screen that triggers the agent logging.

[mmirasol]

we currently have several ERI leads everyday.
https://snag.gy/pj36Vc.jpg

would you say its normal with our daily call volume? we are currently around midday and we have already made 95k calls.
https://snag.gy/D0jyeS.jpg

[mmirasol]

What would be the recommended steps to troubleshoot and eliminate the INCALL records?

Currently, we are:
- checking the lead records which are showing as ERI's vs the agent stats and we see that the agents were logged out every time this happens.
- confirming if there were internet connection issues during these times.
- also checking if there were any issues with agent computer during that time.
- also checking the asterisk console logs to see if anything strange happened before the agent got disconnected.

Please advise if there are any settings we should be checking in vicidial and the server also.

[mflorell]

ERI status leads are what happens when you have INCALL status log entries. In an ideal world, the agents would never close their browsers, and agent workstations would never disconnect from the internet, but it is something that happens. As to how normal the number is, I can't say. Most of our clients have very few of those on a daily basis.

[mmirasol]

and the only way to address ERI calls is to review them manually and correct them?

[mflorell]

The only way an ERI happens is if the agent screen disappears before the call is dispositioned. I'm not sure what other options you have for a situation like that.