about 0.008% to 0.01% of recordings are always missing

[bobbymc]

im not sure if this is normal or if its my system.. but out of all the recordings for the day, every day there is about 0.008 to 1 percent of recording files missing.. the database shows a record for it but the physical file is missing. does anybody have any idea whats going on? this is a svn trunk im using. keep in mind i have had this issue since day one but it wasn't a issue until a customer needed 100% of all recordings to be there for legal reason.

if anybody has any information on this please let me know. thank you

[okli]

Do you convert them to mp3 using lame? In our case lame was the problem, one in every few thousand conversions was failing due to lame crashing, it was only visible looking at the one of system log files, if I recall correctly in /var/log/messages, that's on Debian. Tested a few lame versions but all were having the same problem.
Try something like:

Code: Select all

zgrep lame /var/log/*

You may also temporarily switch to GSW, GSM or OGG recordings which do not use lame for transcoding and confirm if that's the case.
Hope the helps.

[bobbymc]

In don't convert the audio files.. We keep them as wave. I any other idea why this might happen?

[mflorell]

So you are saying that on average less than 1 in 10,000 recordings is missing?

Have you traced this to the Asterisk screenlog.0 log file on the server?

[bobbymc]

i never even knew this log existed. i'll look on google how to enable them. if possible can you give me direction on how to enable this?

[mflorell]

Asterisk is started in a screen session, and if you have logging enabled for the server then this log will be created when Asterisk starts. It's done this for years.

[bobbymc]

will i get the same resutls if i add verbose in my logger?

[bobbymc]

So you are saying that on average less than 1 in 10,000 recordings is missing?

Have you traced this to the Asterisk screenlog.0 log file on the server?

1 in 100 are missing which is 1 percent

[mflorell]

0.01% is 1 in 10,000.
1% is 1 in 100.

1% sounds pretty bad, although I have seen that before usually for clients using hot-keys where agents kill the call and dispo it quickly before the recording even really started.

It also can happen on overloaded servers.

[bobbymc]

so the agent only servers are usual at 5% load.. so its definitely not the load. and when i troubleshoot the records in recording_logs i see the record and the proper seconds the recording was recorded for but the file is simply missing.. its blowing my mind and i cant find out why or generate it on demand. but yea 1% is pretty bad

[mcargile]

This could be a networking issue. If the AJAX command from the Agent web interface never makes it to the web server, the recording will not get started.

Another option is something is wrong with your RAM drive. It might be as simple as you do not have enough RAM. If you are handling a large volume of inbound calls that last a long time, the RAM drive can fill up. When it does recordings will get lost.

As Matt said check the /var/log/astguiclient/screenlog.0 file for that day. You can view it with all the color data by using the -r option to less. Search for the recording filename. If you see asterisk complaining about being unable to write, your RAM drive is probably full.

[bobbymc]

i see the records in recording_logs which means it cant be ajax failing. from the research i have done so far it looks like the AST_manager_send.pl script deosnt even execute the process which means MySQL fails or the entry in vicidial_manager for whatever reasons gets lost or never executed.. i cant imagen where this would possibly break. im changing the flush script that deletes old records in vicidial_manager to dump the data into a tmp table to analyze why that recording process did not properly get proccesses

[bobbymc]

looks like the record in vicidial_manager table for the call recordings that never got recorded are stuck in QUEUE and for some reason never got processed. i looked at AST_manager_send.pl and cant figure out how it would possibly skip this row. the updates process set it to QUEUE and the only reason i can see this nor working is that effected rows varaible was not set to 1 or the SENDNOW variable was not true. SENDNOW could have not been the issue since the status is QUEUE and not DEAD.. any sugestions or ideas where i should look or how to debug this?

[bobbymc]

found the issue.. has a old version of AST_manager_send that was ordering by entry_date vs man_id. this looks like it fixed a rac condition. but my next question would be how does the SVN trunk do recordigns now?

exten => _58600XXX,1,Meetme,${EXTEN:1}|Fmq

seems to be the line but before it was using the asterisk Monitor function.. it deosnt seem to use that anymore.. can you shed some light on how it records the calls now ?

[bobbymc]

stupid me.. its auto generated thats why i dont see it.. sorry for the newb question.