Page 1 of 1
about 0.008% to 0.01% of recordings are always missing
Posted:
Thu Dec 27, 2012 6:41 pm
by bobbymc
im not sure if this is normal or if its my system.. but out of all the recordings for the day, every day there is about 0.008 to 1 percent of recording files missing.. the database shows a record for it but the physical file is missing. does anybody have any idea whats going on? this is a svn trunk im using. keep in mind i have had this issue since day one but it wasn't a issue until a customer needed 100% of all recordings to be there for legal reason.
if anybody has any information on this please let me know. thank you
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Fri Dec 28, 2012 7:04 am
by okli
Do you convert them to mp3 using lame? In our case lame was the problem, one in every few thousand conversions was failing due to lame crashing, it was only visible looking at the one of system log files, if I recall correctly in /var/log/messages, that's on Debian. Tested a few lame versions but all were having the same problem.
Try something like:
- Code: Select all
zgrep lame /var/log/*
You may also temporarily switch to GSW, GSM or OGG recordings which do not use lame for transcoding and confirm if that's the case.
Hope the helps.
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Tue Jan 01, 2013 7:05 pm
by bobbymc
In don't convert the audio files.. We keep them as wave. I any other idea why this might happen?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Tue Jan 01, 2013 7:52 pm
by mflorell
So you are saying that on average less than 1 in 10,000 recordings is missing?
Have you traced this to the Asterisk screenlog.0 log file on the server?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Wed Jan 02, 2013 1:49 am
by bobbymc
i never even knew this log existed. i'll look on google how to enable them. if possible can you give me direction on how to enable this?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Wed Jan 02, 2013 7:37 am
by mflorell
Asterisk is started in a screen session, and if you have logging enabled for the server then this log will be created when Asterisk starts. It's done this for years.
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Wed Jan 02, 2013 9:17 pm
by bobbymc
will i get the same resutls if i add verbose in my logger?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Wed Jan 02, 2013 9:18 pm
by bobbymc
mflorell wrote:So you are saying that on average less than 1 in 10,000 recordings is missing?
Have you traced this to the Asterisk screenlog.0 log file on the server?
1 in 100 are missing which is 1 percent
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Thu Jan 03, 2013 1:10 am
by mflorell
0.01% is 1 in 10,000.
1% is 1 in 100.
1% sounds pretty bad, although I have seen that before usually for clients using hot-keys where agents kill the call and dispo it quickly before the recording even really started.
It also can happen on overloaded servers.
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Thu Jan 03, 2013 6:37 am
by bobbymc
so the agent only servers are usual at 5% load.. so its definitely not the load. and when i troubleshoot the records in recording_logs i see the record and the proper seconds the recording was recorded for but the file is simply missing.. its blowing my mind and i cant find out why or generate it on demand. but yea 1% is pretty bad
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Wed Jan 09, 2013 1:47 pm
by mcargile
This could be a networking issue. If the AJAX command from the Agent web interface never makes it to the web server, the recording will not get started.
Another option is something is wrong with your RAM drive. It might be as simple as you do not have enough RAM. If you are handling a large volume of inbound calls that last a long time, the RAM drive can fill up. When it does recordings will get lost.
As Matt said check the /var/log/astguiclient/screenlog.0 file for that day. You can view it with all the color data by using the -r option to less. Search for the recording filename. If you see asterisk complaining about being unable to write, your RAM drive is probably full.
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Mon Jan 14, 2013 5:29 am
by bobbymc
i see the records in recording_logs which means it cant be ajax failing. from the research i have done so far it looks like the AST_manager_send.pl script deosnt even execute the process which means MySQL fails or the entry in vicidial_manager for whatever reasons gets lost or never executed.. i cant imagen where this would possibly break. im changing the flush script that deletes old records in vicidial_manager to dump the data into a tmp table to analyze why that recording process did not properly get proccesses
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Mon Jan 14, 2013 12:30 pm
by bobbymc
looks like the record in vicidial_manager table for the call recordings that never got recorded are stuck in QUEUE and for some reason never got processed. i looked at AST_manager_send.pl and cant figure out how it would possibly skip this row. the updates process set it to QUEUE and the only reason i can see this nor working is that effected rows varaible was not set to 1 or the SENDNOW variable was not true. SENDNOW could have not been the issue since the status is QUEUE and not DEAD.. any sugestions or ideas where i should look or how to debug this?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Mon Jan 14, 2013 2:58 pm
by bobbymc
found the issue.. has a old version of AST_manager_send that was ordering by entry_date vs man_id. this looks like it fixed a rac condition. but my next question would be how does the SVN trunk do recordigns now?
exten => _58600XXX,1,Meetme,${EXTEN:1}|Fmq
seems to be the line but before it was using the asterisk Monitor function.. it deosnt seem to use that anymore.. can you shed some light on how it records the calls now ?
Re: about 0.008% to 0.01% of recordings are always missing
Posted:
Mon Jan 14, 2013 3:51 pm
by bobbymc
stupid me.. its auto generated thats why i dont see it.. sorry for the newb question.