recording problem

[eze-voip]

Hi! first my server data:

Kernel Version 2.6.18-238.9.1.el5.goPAE (SMP)
Distro Name GoAutoDial CE 2.1
VERSION: 2.4-309a
BUILD: 110430-1642
© 2011 ViciDial Group
DB Schema Version: 1273

Processors 2
Model Intel(R) Pentium(R) D CPU 3.20GHz
CPU Speed 3.2 GHz
Cache Size 2.00 MB
System Bogomips 12800.15
Raid 1


I have 4 campaigns running with a total of 12 agents but never more than 2 campaigns running at the same time. All the campaigns have a different list loaded each day of the week, inbound calls and manual calls from the agent’s page.

the campaign rec filename that I configured is: TINYDATE_OUT_CAMPAIGN_AGENT_CUSTPHONE

A few days ago the team leader ask me for some recordings but I couldn't find them, when I did: find / -name *3408674591* (one of the numbers) I found 2 wav files in/var/spool/asterisk/monitor/MIX 20120927122759_1166324045_3408674591-out.wav and 20120927122759_1166324045_3408674591-in.wav, as I suspected one was the agent and the other the client.

Does anyone know why these two and many others recs that are in the MIX folder didn't were converted to mp3 ??

[williamconley]

i would be willing to bet that your system was overloaded when this happened, but then again it could be another reason entirely (such as not being set to record so it didn't mix or perhaps it was mixed and then deleted by some other process later).

i would check the lead record and see which campaign, etc, the calls are related to and be sure there is nothing "related" about your list of "not mixed" calls. if they are all unrelated, then check the time of day and run your campaign VDAD report to see if those were heavily loaded times. if none of these ever happen during light load ... you could be experiencing your first overload symptom (if so, there will be others, often not-so-subtle).

also pay close attention to your drive space as I have had clients with this observation the day before realizing their HD was full and the entire system bombed.

[eze-voip]

i would be willing to bet that your system was overloaded when this happened, but then again it could be another reason entirely (such as not being set to record so it didn't mix or perhaps it was mixed and then deleted by some other process later).

i would check the lead record and see which campaign, etc, the calls are related to and be sure there is nothing "related" about your list of "not mixed" calls. if they are all unrelated, then check the time of day and run your campaign VDAD report to see if those were heavily loaded times. if none of these ever happen during light load ... you could be experiencing your first overload symptom (if so, there will be others, often not-so-subtle).

also pay close attention to your drive space as I have had clients with this observation the day before realizing their HD was full and the entire system bombed.

now i'll check the server's status at that time, now is

Load Averages 1.03 1.35 1.23
9.38%

Type Percent Capacity Free Used Size
Physical Memory 97% 114.17 MB 3.84 GB 3.95 GB
- Kernel + applications 11% 457.09 MB
- Buffers 3% 120.64 MB
- Cached 83% 3.28 GB
Disk Swap 1% 1006.79 MB 12.95 MB 1019.74 MB

all the calls are configured to be recorded so there is some specific problem with these ones. HD is now at 86%, never more because of the sql crashing.

[eze-voip]

i have just check the recs and find out that it is doing it again, a lot of calls being recorded but remaining separated:

20121122111447_1166324045_2395451686-in.wav
20121122111447_1166324045_2395451686-out.wav

and never converted to MP3!

[williamconley]

What was the load average when this happened (having a load average when it was not happening is "interesting", but not entirely useful for troubleshooting this particular issue). Also 86% is fairly high and could be an indicator that you have at some point filled up (and then rebooting/restarting cleared up some drive space ...). You could also attempt to execute the mix command manually to see if these get mixed. You may need to copy them back to the pre-mix location if they have been moved out. Check out the crontab -e script that mixes, read it ... it has instructions for use and the script itself contains locations.

[eze-voip]

What was the load average when this happened (having a load average when it was not happening is "interesting", but not entirely useful for troubleshooting this particular issue). Also 86% is fairly high and could be an indicator that you have at some point filled up (and then rebooting/restarting cleared up some drive space ...). You could also attempt to execute the mix command manually to see if these get mixed. You may need to copy them back to the pre-mix location if they have been moved out. Check out the crontab -e script that mixes, read it ... it has instructions for use and the script itself contains locations.

thanks for your help

Load Averages 1.11 1.06 1.20
6.03%

i have ran the script manually without succesful results.

i will reboot and see what happend

[GaD]

I don;t know if I'm reading this right but...., is this:

Type Percent Capacity Free Used Size
Physical Memory 97% 114.17 MB 3.84 GB 3.95 GB
..saying that you only have 114.17MB free RAM? I think that is where the problem recides (is this the right spelling of the word?? ...it looks funny...) if this is true. If you do not have enough memory available the beaviour of your system will be unpredictable. In your case, the audios will not be mixed. In my experience I've had quality problems during the calls, dropped calls, stuck Apache sessions.... You're lucky if you're only not able to do the mix process if you have a memory issue.

Check out your available RAM and see if it could be the cause of your headaches! Drink some coffee afterwards..., it will help loose the migraine!

[eze-voip]

I don;t know if I'm reading this right but...., is this:

Type Percent Capacity Free Used Size
Physical Memory 97% 114.17 MB 3.84 GB 3.95 GB
..saying that you only have 114.17MB free RAM? I think that is where the problem recides (is this the right spelling of the word?? ...it looks funny...) if this is true. If you do not have enough memory available the beaviour of your system will be unpredictable. In your case, the audios will not be mixed. In my experience I've had quality problems during the calls, dropped calls, stuck Apache sessions.... You're lucky if you're only not able to do the mix process if you have a memory issue.

Check out your available RAM and see if it could be the cause of your headaches! Drink some coffee afterwards..., it will help loose the migraine!

Thanx a lot for you reply!

I asked to a lot of linux admins and all sais the same, is common that use of phisical memory because linux use the 100% of the phisical memori before start swapping to disk!

I have 4 gb DDR2 ram total

[GaD]

Well.., I don't know which admins you spoke to, but 114MB of free RAM is not good for a linux server that runs an asterisk server with meetme and a database. I would seriusly recommend you to free up some RAM. And yes, I am a linux Sys Admin.

Recomendation: Restart your Vici server (completely) and see your RAM usage. See if this helps.

[DomeDan]

The ram is not the problem because he wrote

Type Percent Capacity Free Used Size
Physical Memory 97% 114.17 MB 3.84 GB 3.95 GB
- Kernel + applications 11% 457.09 MB
- Buffers 3% 120.64 MB
- Cached 83% 3.28 GB
Disk Swap 1% 1006.79 MB 12.95 MB 1019.74 MB

notice "- Cached 83% 3.28 GB"
that means that he is using only about 580Mb ram at that point, the rest of the ram is used for cach and will be dumped if any program needs more ram

if you look in top/htop the cached amount is not displayed
but if you use the command: free
it will tell you the amount of: used, free, cached etc

eze-voip:
are the two files in /var/spool/asterisk/monitor/MIX/ ?
are there more files in that directory?
post your crontab (crontab -l as root) only intressted in the lines under "### recording mixing/compressing/ftping scripts"
if you search in vicidial admin interface for the phone 3408674591 in question, what recordings links and filenames do you see for the lead?

[eze-voip]

The ram is not the problem because he wrote

Type Percent Capacity Free Used Size
Physical Memory 97% 114.17 MB 3.84 GB 3.95 GB
- Kernel + applications 11% 457.09 MB
- Buffers 3% 120.64 MB
- Cached 83% 3.28 GB
Disk Swap 1% 1006.79 MB 12.95 MB 1019.74 MB

notice "- Cached 83% 3.28 GB"
that means that he is using only about 580Mb ram at that point, the rest of the ram is used for cach and will be dumped if any program needs more ram

if you look in top/htop the cached amount is not displayed
but if you use the command: free
it will tell you the amount of: used, free, cached etc

eze-voip:
are the two files in /var/spool/asterisk/monitor/MIX/ ?
are there more files in that directory?
post your crontab (crontab -l as root) only intressted in the lines under "### recording mixing/compressing/ftping scripts"
if you search in vicidial admin interface for the phone 3408674591 in question, what recordings links and filenames do you see for the lead?

yes, the two files are in that directory, one with the customer's voices and one with the agent's voice
there are a lot of files in that directory since june (i installed this server in april)

here is my crontab

### adjust the GMT offset for the leads in the vicidial_list table
1 1 * * * /usr/share/astguiclient/ADMIN_adjust_GMTnow_on_leads.pl --debug

### reset several temporary-info tables in the database
2 1 * * * /usr/share/astguiclient/AST_reset_mysql_vars.pl

### optimize the database tables within the asterisk database
3 1 * * * /usr/share/astguiclient/AST_DB_optimize.pl

## adjust time on the server with ntp
30 * * * * /usr/sbin/ntpdate -u pool.ntp.org 2>/dev/null 1>&2

### VICIDIAL agent time log weekly and daily summary report generation
2 0 * * 0 /usr/share/astguiclient/AST_agent_week.pl
22 0 * * * /usr/share/astguiclient/AST_agent_day.pl

### VICIDIAL campaign export scripts (OPTIONAL)
#32 0 * * * /usr/share/astguiclient/AST_VDsales_export.pl
#42 0 * * * /usr/share/astguiclient/AST_sourceID_summary_export.pl

### remove old recordings more than 14 days old
24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG/ -maxdepth 2 -type f -mtime +14 -print | xargs rm -f

### roll logs monthly on high-volume dialing systems
#30 1 1 * * /usr/share/astguiclient/ADMIN_archive_log_tables.pl

### remove old vicidial logs and asterisk logs more than 2 days old
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +2 -print | xargs rm -f
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +2 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +4 -print | xargs rm -f

### keepalive script for GoAutoDial processes
* * * * * /usr/share/goautodial/keepalive_goautodial.pl

### logs cleanup for GoAutoDial
8 1 * * * /usr/share/goautodial/go_clean.pl

### asterisk logs access for GoAutoDial
* * * * * /usr/share/goautodial/go_astlogs.pl

### asterisk db daily backup
30 4 * * * /usr/bin/mysqldump -u root -pvicidialnow --databases asterisk > /usr/share/goautodial/godbbackup/

whe I search in the leads i found the call,

1 209715 SALE 1166324045 agent008 999 3408674591 moratemprana

then when i click in the rec link:

http://192.168.0.14/RECORDINGS/MP3/1209 ... 91-all.mp3

error:

Not Found

The requested URL /RECORDINGS/MP3/120927122824_IN_MT3_agent008_3408674591-all.mp3 was not found on this server.

Apache/2.2.3 (CentOS) Server at 192.168.0.14 Port 80

if i search in command line:

[root@go MP3]# find / -name *3408674591*
/var/spool/asterisk/monitor/MIX/20120927122759_1166324045_3408674591-in.wav
/var/spool/asterisk/monitor/MIX/20120927122759_1166324045_3408674591-out.wav

[eze-voip]

apparently, the calls dialed from the manual page are not being recorded either

[williamconley]

turn off the cron mixer. put the files back in their pre-mix location. run the perl script with debugX. see the result.

note: memory in a mysql based machine is always maxed out. this is mysql caching at it's job. it fills the available memory with cached queries, but marks the memory available for use by other apps. but those cached entries are still available if not yet overwritten. after a few minutes of full operation, every mysql machine will fill the cache.

[eze-voip]

turn off the cron mixer. put the files back in their pre-mix location. run the perl script with debugX. see the result.

note: memory in a mysql based machine is always maxed out. this is mysql caching at it's job. it fills the available memory with cached queries, but marks the memory available for use by other apps. but those cached entries are still available if not yet overwritten. after a few minutes of full operation, every mysql machine will fill the cache.

sorry to ask but where is their pre-mix location??

[williamconley]

when in doubt, test this when noone is using the system ... and LOOK while you are calling your own cell phone as a Prospect. this is the best, most educational, testing you can do. Watching what happens when you are in a controlled situation (no other traffic ...) and testing a call to yourself to see where files are at what time and when they move. If you turn off the sound handling crons and run them in debug mode it's even better. You'll learn more than I can teach you on the forum, that's for sure. And it won't cost a dime.

[DomeDan]

turn off the cron mixer. put the files back in their pre-mix location. run the perl script with debugX. see the result.

I cant see any cron mixer script in his crontab list,
but it seams like something tried to convert the files to mp3. is the mix-script executed from some other script?

eze-voip:
as william wrote, find and run AST_CRON_audio_1_move_mix.pl with --debugX flag

[GaD]

Although the RAM is cached -not necesarily in use- not all OSs are efficient releasing the unused memory back. It has been my case with Vici. Although I'm not into DB development/admin I do service the actual servers and monitor them. Out of 6 servers running MYSQL services only Vici (the seventh server) maxes out the RAM. ...could be based on usage too, my clustered servers balance out pretty well and the non-clustered are not so DB intensive.

I would wtill recommend a fresh reboot. To make it even more 'interesting' you could try running the script before reboot and running it again after reboot.

[williamconley]

We still reboot all our servers nightly. No matter how clean the system may be this season ... we've not had 1% of the problems we had prior to instituting the reboot. Between reboot AND cleaner systems (Vicibox is Cooool!): Our servers are always up. If there's a problem, it always relates to a city-wide power surge or broken wire/power supply. (Well, that and the occasional client who fills up a HD! LOL).

[GaD]

I could not agree more!!