tmpfs full

[omarrodriguezt]

The tmpfs is getting full and we are unable to keep recording some calls.
I have a couple of days with the issue, searching in the forum and google, so far nothing,



Also I think that the problem is that the deleted files are not being removed from the system based on these posts: http://askubuntu.com/questions/196509/f ... -adding-up and
http://serverfault.com/questions/445445 ... 449#445449

because when i run this command: lsof | grep deleted ()
/var/spool/asterisk/monitor/9732467365_301_wvalentin_20150309-110957-in.wav (deleted)
asterisk 2575 32686 root 2302w REG 0,31 0 531509 /var/spool/asterisk/monitor/filenamefortoday_20150309-110957-out.wav (deleted)

I see a lot of deleted files.



This is a new vicibox server installed 10 days ago:

VERSION: 2.10-475a
BUILD: 150218-0924
© 2015 ViciDial Group

Express Instalation.

Connected to Asterisk 1.8.32.1-vici currently running

These are the specifications of my CPU It has 4 in total.
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5506 @ 2.13GHz




This is a print of the command
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 271G 16G 254G 6% /
devtmpfs 3.9G 40K 3.9G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 9.4M 3.9G 1% /run
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 9.4M 3.9G 1% /var/run
tmpfs 10G 7.2G 2.9G 72% /var/spool/asterisk/monitor
tmpfs 4.0G 9.4M 3.9G 1% /var/lock


crontab -e (recording part)

### recording mixing/compressing/ftping scripts
#0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3
#1 21,22,23,0,1,2,3,4,5,6 * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3
#2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3
#0 1 * /usr/share/astguiclient/AST_CRON_audio_4_ftp2.pl --ftp-server=server.ip --ftp-login=user --ftp-pass=pass --ftp-directory=/ --ftp-persistent --ftp-validate --transfer-limit=100000 --list-limit=100000

cat /etc/fstab

/dev/disk/by-id/scsi-3600508b1001cc985f17ff56c61df4920-part1 swap swap defaults 0 0
/dev/disk/by-id/scsi-3600508b1001cc985f17ff56c61df4920-part2 / ext4 noatime,acl,user_xattr 1 1
tmpfs /var/spool/asterisk/monitor tmpfs rw 0 0



So far I found that the recordings keeps growing even if the call is hangup already.

Similar to this:
http://www.eflo.net/VICIDIALforum/viewtopic.php?p=8616
and this
http://www.eflo.net/VICIDIALforum/viewtopic.php?p=14037

http://www.eflo.net/VICIDIALforum/viewtopic.php?t=5932


Best Regards,
Omar

[omarrodriguezt]

Fixed.
The reason why those files were still in the memory, it is because some calls (channels) remains opens even when the agents hangup; you can do a query like this:

Code: Select all

 select count(0) as channels from live_sip_channels

to confirm it is your case too.
So, for an unknown reason the dialer in manual mode was leaving some channels opened, even when the agent had press the hangup button.
I fixed altering the manager_send.php , using the code posted by Caspar here:http://www.eflo.net/VICIDIALforum/viewtopic.php?p=8616
But this was not enough, I also created a perl script that runs every minute to kill any unwanted ghost call/channel.
See more details and the files attached in the mantis bug tracker here: http://www.vicidial.org/VICIDIALmantis/view.php?id=837

[ruben23]

HI there is this resolved somehow.? i got same issue, any idea, Thanks

[carpenox]

get a bigger drive?

[ruben23]

i already put bigger Drive but still the Tmpfs still get full during the day

[carpenox]

did you try what he put?

Fixed.
The reason why those files were still in the memory, it is because some calls (channels) remains opens even when the agents hangup; you can do a query like this:
CODE: SELECT ALL
select count(0) as channels from live_sip_channels
to confirm it is your case too.
So, for an unknown reason the dialer in manual mode was leaving some channels opened, even when the agent had press the hangup button.
I fixed altering the manager_send.php , using the code posted by Caspar here:http://www.eflo.net/VICIDIALforum/viewtopic.php?p=8616
But this was not enough, I also created a perl script that runs every minute to kill any unwanted ghost call/channel.
See more details and the files attached in the mantis bug tracker here: http://www.vicidial.org/VICIDIALmantis/view.php?id=837

[ruben23]

how to implement it any idea there are no procedure on the thread

[williamconley]

post a df showing the tmpfs full but plenty of space on the HD, plus the fstab entry showing its creation.

make the tmpfs bigger?

i've noticed that "same problem" without posting the same proof often turns out to NOT be the same problem, but lots of time is spent trying to fix it. lol

which audio scripts are you running?

try running the first one in debug mode during the problem, perhaps there's an error?

is your hard drive healthy? (A dying hard drive can drastically reduce throughput)

[ruben23]

@William

this get easy fill up like after 3-5 hours on a daily basis and our option is to reboot the server to make the tmpfs back to zero and of course we lost recordings after that

root@viciast1:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 4.0K 7.9G 1% /dev
tmpfs 1.6G 356K 1.6G 1% /run
/dev/sda1 288G 75G 201G 28% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 7.9G 0 7.9G 0% /run/shm
none 100M 0 100M 0% /run/user
tmpfs 20G 422M 20G 3% /var/spool/asterisk/monitor


Thats is the actual output of df, i already adjusted the tmpfs bigger that's from 10GB to 20GB now. any idea somehow its like everyday we need to reboot like max of 3 times and min of 2 times

how do i run the debug mode.?

[carpenox]

/usr/share/astguiclient/whateverscript.pl --debugX

[ruben23]

@carpenox,

Im still confuse how to implement the FIX

[williamconley]

@William

this get easy fill up like after 3-5 hours on a daily basis and our option is to reboot the server to make the tmpfs back to zero and of course we lost recordings after that

root@viciast1:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 4.0K 7.9G 1% /dev
tmpfs 1.6G 356K 1.6G 1% /run
/dev/sda1 288G 75G 201G 28% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 7.9G 0 7.9G 0% /run/shm
none 100M 0 100M 0% /run/user
tmpfs 20G 422M 20G 3% /var/spool/asterisk/monitor


Thats is the actual output of df, i already adjusted the tmpfs bigger that's from 10GB to 20GB now. any idea somehow its like everyday we need to reboot like max of 3 times and min of 2 times

how do i run the debug mode.?

Run the first "recordings" script in --debugX mode and find out why it's not moving the audio files off the temp folder. Unless you have a LOT of live calls, you shouldn't have 422M.

However: If you are recording agents (from the moment of login to the moment of logout) you could need to change that behavior to recording the campaign calls at all times instead of the agents at all times. Or force your agents to log out hourly.