call goes silent: Cannot allocate memory

[santhuarnold]

Hi,

Our dialer working fine from last 10 month properly and its maintained well with 3 servers clustering. recently we had an issue that call goes blank after connecting to the customer in both manual dial and auto dial.
PRI/SIP was working fine, we found "chan_dahdi.c: Unable to dup channel: Cannot allocate memory".
when we checked ulimit
open files -1024
max user processes - 515511
max memory size - unlimited

Logs:

Nov 28 10:17:17] VERBOSE[133771] pbx.c: [Nov 28 10:17:17] -- Executing [89910575415@default:2] Dial("Local/89910575415@default-0001fcd8;2", "SIP/olx/9910575415,,ToR") in new stack
[Nov 28 10:17:17] VERBOSE[133771] netsock2.c: [Nov 28 10:17:17] == Using SIP RTP CoS mark 5
[Nov 28 10:17:17] VERBOSE[133771] app_dial.c: [Nov 28 10:17:17] -- Called SIP/olx/9910575415
[Nov 28 10:17:18] VERBOSE[133719] app_dial.c: [Nov 28 10:17:18] -- SIP/olx-000176a9 is ringing
[Nov 28 10:17:18] VERBOSE[133776] manager.c: [Nov 28 10:17:18] == Manager 'sendcron' logged on from 127.0.0.1
[Nov 28 10:17:18] VERBOSE[133777] pbx.c: [Nov 28 10:17:18] -- Executing [58600075@default:1] MeetMe("Local/58600075@default-0001fcd9;2", "8600075,Fmq") in new stack
[Nov 28 10:17:18] VERBOSE[133776] pbx.c: [Nov 28 10:17:18] > Channel Local/58600075@default-0001fcd9;1 was answered.
[Nov 28 10:17:18] VERBOSE[133778] pbx.c: [Nov 28 10:17:18] -- Executing [8309@default:1] Answer("Local/58600075@default-0001fcd9;1", "") in new stack
[Nov 28 10:17:18] VERBOSE[133778] pbx.c: [Nov 28 10:17:18] -- Executing [8309@default:2] Monitor("Local/58600075@default-0001fcd9;1", "wav,20161128-101716_9402051353") in new stack
[Nov 28 10:17:18] VERBOSE[133778] pbx.c: [Nov 28 10:17:18] -- Executing [8309@default:3] Wait("Local/58600075@default-0001fcd9;1", "3600") in new stack
[Nov 28 10:17:19] VERBOSE[133765] pbx.c: [Nov 28 10:17:19] > Channel SIP/5044-000176ae was answered.
[Nov 28 10:17:19] VERBOSE[133779] pbx.c: [Nov 28 10:17:19] -- Executing [8600089@default:1] MeetMe("SIP/5044-000176ae", "8600089,F") in new stack
[Nov 28 10:17:19] VERBOSE[133779] config.c: [Nov 28 10:17:19] == Parsing '/etc/asterisk/meetme.conf': [Nov 28 10:17:19] VERBOSE[133779] config.c: [Nov 28 10:17:19] == Found
[Nov 28 10:17:19] VERBOSE[133779] config.c: [Nov 28 10:17:19] == Parsing '/etc/asterisk/meetme-vicidial.conf': [Nov 28 10:17:19] VERBOSE[133779] config.c: [Nov 28 10:17:19] == Found
[Nov 28 10:17:19] WARNING[133779] chan_dahdi.c: Unable to open '/dev/dahdi/pseudo': Cannot allocate memory
[Nov 28 10:17:19] ERROR[133779] chan_dahdi.c: Unable to dup channel: Cannot allocate memory

please help me to find actual reason for this issue and how to avoid this.
thanks advance

VERSION: 2.12-572a | BUILD: 161102-1040 | ViciBox Redux v.6.0.4 | Asterisk 1.8.32.3-vici |

[williamconley]

When was the last time you rebooted?
<pre>uptime</pre>

How much memory do you have? How much swap memory is in use (often a sign that you are short on memory)?
<pre>free</pre>

[santhuarnold]

the last reboot was 10 days back.


mem: 65995888 total, 26528432 used, 39467456 free
swap: 31454201 total, 0 used , 31454204 free

And another servers with the same configuration running past 1 year without any issues.
is there any particular reason, why this issue will occurs?

Thanks.

[williamconley]

Yes: Linux/Asterisk/Vicidial are not perfect applications. Plus any customizations you may be using.

Reboot your dialers nightly and remove the memory problem OR learn the intricacies of memory management, and monitor all the applications in your system to find the memory leak that may occur after several days.

Note that when you install Vicidial, your revision level is different from any prior install. Scripts, applications, utilities, operating system, all are slightly different. Any one of which may introduce a flaw. We've found that rebooting nightly tends to reduce such problems to almost nothing. With hundreds of servers running.

[josecapurro]

I have got some severe memory leaks in the past.

Maybe it is not the best approach to this problem, but it worked for me:
Log in as root via SSH in your telephony nodes and add this line to root's crontab:

*/5 * * * * /bin/echo 3 > /proc/sys/vm/drop_caches

With this cronjob, each 5' those cached objects who resides in memory will be flushed inmediately. You can check this with free -m

HTH.

[williamconley]

I have got some severe memory leaks in the past.

Maybe it is not the best approach to this problem, but it worked for me:
Log in as root via SSH in your telephony nodes and add this line to root's crontab:

*/5 * * * * /bin/echo 3 > /proc/sys/vm/drop_caches

With this cronjob, each 5' those cached objects who resides in memory will be flushed inmediately. You can check this with free -m

I note that you did not indicate what makes you think caches are in any way related to the problem(s) you or anyone else is experiencing.

I strongly suggest that before you implement such a script, you experience a problem that you can demonstrate has been resolved by your "override" of the standard system operations.

In other words: Did you experience a problem with your dialer, and did this solve the problem? Or did you see memory in use in caches, which is *not* a problem, and then jump to the conclusion that "caches are not necessary" so this deletion of caches *should* solve a problem?

There are lots of caches in a Vicidial implementation. You could delete all of them if you like. But of course, then that cached information would require regeneration with each request, degrading operations of those applications.