Recordings on 1 server not moving / getting lost

[perci100]

I have an odd issue with one of my asterisk servers .

I first noticed this issue when the drives were filling up on just this one server. I run small drives on the asterisk servers (72g/raid10) . I found out it was filling up with recordings. They are accumulating in var/spool/asterisk/monitorDONE/. Cron is exactly the same on this server as the others , running each audio script separately with --debugX I dont see any errors but the files still remain in the monitorDONE directory and I am always missing a few recordings from this server. What is odd is its not all of them.

Some differences in this server , it is the first telephone server I ever had. Seems to run a little slow via SSH . its an older hp g5 . It is the first sever hit by inbound calls. It is the voicemail server (no its not filling with voicemails) .

Could you point me in the direction of some information on how the whole recording storage/ ftp process works? Particularly when they are stored in monitorDONE and what moves them from there.

[perci100]

to add to this , it seems like the recordings that get lost end up having a different link in the system afterwards. Its the public IP address of the recording server instead of the fqdn set in astguiclient.conf

[williamconley]

1) you should avoid adding a reply to yur own post that has never had a resonse by someone. It looks (in the system) like someone answered the post and disappears from the "unanswered" posts list. could cause it to get ignored completely (assuming someone else already took care of you). I clicked on it by accident, or I wouldn't be here.

2) My only guess based on your description would be a DNS issue. If the server randomly loses DNS, it may lose connectivity to your target FTP location and not send files as a result. Oddly enough, lack of DNS will also cause slow login via SSH and make the server appear sluggish when in fact it's merely awaiting a DNS resonse before allowing SSH login (because engineers who program know more than us mere mortals and believe that this should be a thing, and DNS should never fail, of course).

3) this description confuses me:

Code: Select all

I dont see any errors but the files still remain in the monitorDONE directory and I am always missing a few recordings from this server.

Either they are still in monitorDONE or they are missing. They really can't be both.

[perci100]

1) I was unaware of the way the system works. Just wanted to include as much info as possible. I guess an edit would have accomplished that. Noted for the future.

2) Would that not also have an effect on Dialing?

3) I should have been more clear, "missing" meaning they are not retrievable from the link that they generate. All of those recordings remain in monitorDONE and never move. Nothing is technically missing , just not in the right place. If the server looses DNS would it not just pick up the files it never transferred on the next run of the script? I was trying to understand a bit better how that system works. I am non stop programming 70+ hours a week so i havent had the time. Just need it to work. Which seems to be a common phrase in production from everyone lol.

I made some adjustments DNS wise lets see how it performs tomorrow.

Thank you for the help. I appreciate it.

[williamconley]

2) Would that not also have an effect on Dialing?

Depends on your settings. If none of your carriers are DNS based (IP addresses only), then ... no.

3) I should have been more clear, "missing" meaning they are not retrievable from the link that they generate. All of those recordings remain in monitorDONE and never move. Nothing is technically missing , just not in the right place.

It's time to discuss where in monitorDONE they are. Each folder has a meaning.

monitor = call in progress
monitorDONE = call completed
ORIG & MP3 (two copies, same file one compressed and the other Not) = mix and compress completed (could be another compression folder, but MP3 is most common)
FTP = pushed to ftp server
FTP2 = pushed to backup ftp server

In theory, the link should ONLY be changed when the file has been confirmed on the FTP location, just before the file is moved to the FTP folder.

If the server looses DNS would it not just pick up the files it never transferred on the next run of the script?

That's a good theory. I'd say yes. Read the scripts you have activated to be sure the order of events. Verify your FTP storage isn't full and your HD isn't full on the Dialer.

Note that in a properly working system, ORIG files are disposable. Some point that to /dev/null or run an hourly rm to keep it clean. Only useful if a server dies in the middle of a compression and the MP3 is corrupted.

Same *should* apply to FTP since FTP means its been copied to the FTP server ... but it's best to have the FTP2 system set up so you have two copies of everything and then only delete from the FTP2 folder.

[perci100]

Im talking specifically about the root of monitorDONE/
none of the other servers store recordings there at least not for any amount of time. for some reason my tel1 server does this and they never move the other 4 dont , same exact cron settings

These are the scripts being run in cron i just cant figure out why the initial recordings are stored in monitorDONE and not moved. ( I checked permissions as well, same as the other servers)

AST_CRON_audio_1_move_mix.pl --MIX
AST_CRON_audio_1_move_VDonly.pl
AST_CRON_audio_2_compress.pl --MP3
AST_CRON_audio_3_ftp.pl --MP3

The archive is an on premise storage array with about 37 Terabytes free the other tel servers have no problem sending recordings to it.

I did just notice there was no ORIG directory inside monitorDONE on that server that is acting up , maybe that was it? Does it need to store them In ORIG before converting them?

I appreciate the help.

[perci100]

Wow , Figured out the ORIG directory was missing on that server , Must have been deleted somehow. It was probably me trying to move recordings while WinSCP was freezing/crashing

Anyhow , as soon as I mkdir ORIG , all the recordings moved from monitorDONE.

[williamconley]

Excellent postback.