vicidial.org

Hello. Looking for some guru assistance please. We're stuck after many hours of troubleshooting.

We have a ViciDial cluster with 4 identical dialler servers (plus DB Master, Slave, Archive and Web servers) - versions below. One of the dialler servers is randomly failing to make some call recordings and the other 3 are fine. For these calls the recording_log record is incomplete - end_time is still NULL and length_in_sec=0 and the recording file appears not to have started. Everything works fine for most of the time and then there are periods in the day when a whole lot of recordings fail, but not all. Last week we had a day when we lost 500 recordings out of 18500 agent calls on the one server. Many days have none of these errors. We cannot see a pattern as to when the problem occurs, call volumes are high at these times as a general rule.

These problem calls correspond with groups of these entries in the Asterisk log:
[2023-04-13 15:47:58] WARNING[42292][C-0003c672] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 15:47:58] WARNING[42295][C-0003c674] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 15:47:58] WARNING[42308][C-0003c677] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 15:51:05] WARNING[6008][C-0003ce80] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 15:51:06] WARNING[6022][C-0003ce84] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 15:51:07] WARNING[6037][C-0003ce86] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory

These relate to any event when a channel is added to a MeetMe, not just call recordings, but these are what we are picking up on.

For example:
[2023-04-13 10:53:45] VERBOSE[19237][C-000187c9] pbx.c: Executing [58600106@default:1] MeetMe("Local/58600106@default-0000d58d;2", "8600106,Fmq") in new stack
[2023-04-13 10:53:46] WARNING[19237][C-000187c9] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory
[2023-04-13 10:53:46] VERBOSE[19237][C-000187c9] pbx.c: Spawn extension (default, 58600106, 1) exited non-zero on 'Local/58600106@default-0000d58d;2'
[2023-04-13 10:53:46] WARNING[19237][C-000187c9] func_hangupcause.c: Unable to find information for channel
[2023-04-13 10:53:46] VERBOSE[19237][C-000187c9] pbx.c: Executing [h@default:1] AGI("Local/58600106@default-0000d58d;2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----0--------------------)") in new stack
[2023-04-13 10:53:46] VERBOSE[19237][C-000187c9] res_agi.c: <Local/58600106@default-0000d58d;2>AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... ----------) completed, returning 0

2nd example - a call coming to an agent over IAX:
[2023-04-13 13:25:01] VERBOSE[10901][C-0002990a] pbx.c: Executing [8600124@default:1] MeetMe("IAX2/c4-vcd02-10701", "8600124,F") in new stack
[2023-04-13 13:25:01] WARNING[10901][C-0002990a] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory

Looking at the source, this message comes from the file open of /dev/dahdi/pseudo
app_meetme.c
/* open pseudo in non-blocking mode */
fd = open("/dev/dahdi/pseudo", O_RDWR | O_NONBLOCK);
if (fd < 0) {
ast_log(LOG_WARNING, "Unable to open DAHDI pseudo channel: %s\n", strerror(errno));
goto outrun;
}

Amfeltech USB timer is all installed and dahdi is working and not showing any issues.

We don't believe that this is a ulimit issue:
a) There are no messages in the Asterisk logs like "Too many open files"
b) We have set the ulimit to 1000000:
c4-vcd01:~ # cat /proc/7717/limits
Limit Soft Limit Hard Limit Units
Max open files 1000000 1000000 files

We have also implemented a nightly restart of Asterisk, bit this has not solved the problem.

I welcome any ideas and assistance from the community. I can provide additional info as required.

Thank you

Roger

ViciBox 9.0.3 install VERSION: 2.14-787a BUILD: 210211-1145 SVN 3353
openSUSE Leap 15.1
Asterisk 13.34.0-vici

Could be bad memory, perhaps run memtest86?

Roger,

May not be a solution, but a workaround:

Code: Select all

echo 1024 > /sys/module/dahdi/parameters/max_pseudo_channels


This should increase your max pseudo channels to 1024 (from the default 512). It has helped me before when dealing with the same issue. However, it looks like you are over using your conferences, so lowering the load for this server should help.

Regards,