Call rec processing problem

[Vince-0]

Hi forum!

I've encountered a problem on every dialer (x8) we have in production across two sites whereby the /var/spool/asterisk/monitorDONE/FTP folder has old random recordings, some months old:

Code: Select all

dialler4:/var/spool/asterisk/monitorDONE/FTP # ls | wc -l
576544

dialler4:/var/spool/asterisk/monitorDONE/FTP # ls
...
-rw-r--r-- 1 root   root     16K 2012-06-06 18:46 004_11428XXXXX_20120606-184330_FCU0767-all.mp3
-rw-r--r-- 1 root   root     31K 2012-05-16 13:23 004_11428XXXXX_20120514-173206_FCU0630-all.mp3
-rw-r--r-- 1 root   root    119K 2012-05-30 21:10 004_11428XXXXX_20120530-210535_FCU0313-all.mp3
-rw-r--r-- 1 root   root     50K 2012-05-17 13:22 004_11428XXXXX_20120517-131741_FCU0032-all.mp3
-rw-r--r-- 1 root   root     73K 2012-05-17 18:55 004_11428XXXXX_20120517-185058_FCU0821-all.mp3
-rw-r--r-- 1 root   root    106K 2012-05-16 14:52 004_11428XXXXX_20120514-145009_FCU0767-all.mp3
...


This is growing slowly in size and in some cases up over 40GB in the FTP directory:

Code: Select all

45G   ./FTP
757M   ./FTP2
4.0K   ./GSM
4.0K   ./GSW
496K   ./MP3
4.0K   ./OGG
1.6G   ./ORIG


I've had to delete the wav files in the ORIG folder to make space but that is not a permanent fix.
I'm concerned that recordings older than 7 days are removed when they could not exist on the FTP server, resulting in lost recordings.

Cron:

Code: Select all

### keepalive script for astguiclient processes
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### remove old vicidial logs and asterisk logs more than 2 days old
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +2 -print | xargs rm -f
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +2 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +4 -print | xargs rm -f
### recording mixing/compressing/ftping scripts
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 * * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3
#Job to catch up recording processing backlog
#45 23 * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3
2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 * * * * /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3

### kill Hangup script for Asterisk updaters
* * * * * /usr/share/astguiclient/AST_manager_kill_hung_congested.pl

### updater for voicemail
* * * * * /usr/share/astguiclient/AST_vm_update.pl

### updater for conference validator
* * * * * /usr/share/astguiclient/AST_conf_update.pl

### flush queue DB table every hour for entries older than 1 hour
11 * * * * /usr/share/astguiclient/AST_flush_DBqueue.pl -q

### reset several temporary-info tables in the database
2 1 * * * /usr/share/astguiclient/AST_reset_mysql_vars.pl

### remove old recordings more than 7 days old
#24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE -maxdepth 2 -type f -mtime +7 -print | xargs rm -f
0 8-20 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG -maxdepth 2 -type f -mmin +120 -print | xargs rm -f
### Reboot nightly to manage asterisk issues and memory leaks - uncomment if issues arise
#0 6 * * * /sbin/reboot

### remove text to speech file more than 4 days old
#20 0 * * * /usr/bin/find /var/lib/asterisk/sounds/tts/ -maxdepth 2 -type f -mtime +4 -print | xargs rm -f

I find that /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3 --debugX processes only current recordings:

Code: Select all

/var/spool/asterisk/monitorDONE/MP3/004_12536XXXXX_20120622-103427_FCU0914-all.mp3 35424
 ...

I did notice a fourth script that is not included in the cron schedule, which does move recordings to FTP but also results in "File size mismatch" and random call recordings left behind on the dialer:

Code: Select all

/usr/share/astguiclient/AST_CRON_audio_4_ftp2.pl --MP3 --debugX --ftp-persistent

Results in some recordings being FTP'd and others in:

Code: Select all

not transfering /var/spool/asterisk/monitorDONE/FTP/004_19378XXXXX_20120404-111306_FCU0010-all.mp3. File size mismatch 164448 !=

As a result, these dialers' disks are filling and the smallest disk needs manual cleaning and processing with that fourth script.

Q: Is it okay to just remove the wavs in the ORIG folder?
Q: Are all the dialer recording files supposed to be moved the the FTP by the end of the day?
Q: Can that fourth script be included in the cron schedule to move all recordings to the FTP regularly so none get left behind and alleviate this problem?

-----
VERSION: 2.6-365a BUILD: 120420-1620 and VERSION: 2.4-351a BUILD: 111223-0043 at different sites.
Asterisk 1.4
Multi-server: 1x DB, 1x Web, 4x Dialers
Sangoma/Digium hardware timing
G729
VoIP
OpenSuse - Vicibox Redux 3.1.5
No extra software after installation

[williamconley]

The wavs in the orig folder are redundant after the mp3s are created and may be deleted at will.

The ftp recordings should move during the average day, at the end of the day they should all have been moved from MP3 to FTP (and copies should now exist on the remote FTP server). BUT: If there is a problem with the transfer, the script should leave the file in the MP3 folder and not modify the vicidial link to the new location. If there were files being "left behind" in the MP3 folder, this would indicate a problem with FTP transfer.

I note you did not show a script that deletes the files in the FTP folder, unless there is one of those your FTP folder will grow forever. FTP2 is designed for redundant backup, not a "cleanup of the FTP folder".

[Vince-0]

Thanks William!

I searched for any documentation regarding call recording processing because call recordings are a vital requirement and the handling of these is sensitive but I didn't find details about the end to end process.

I can verify the recordings in the FTP directory are duplicates of the ones on the FTP(storage) server and that the URL in the DB points to the storage server so I can delete old recordings. I will introduce a job eg:

Code: Select all

24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/FTP/ -maxdepth 2 -type f -mtime +14 -print | xargs rm -f

Thanks for clearing this up.

Vincent.