Vicibox - Problem Meetme, Dahdi etc

[aleyva]

Hi,
I have problems with Vicibox, 2 times by day a have this messages in asterisk Cli.

"WARNING[9997]: app_meetme.c:1820 conf_run: Unable to open DAHDI pseudo channel: Cannot allocate memory"

Many agents can't dial, that forces me to restart the server.
That may be the problem? Load Server? Conferences? I have 70 agents and 45 ghost agents.

Today, when i have this problem, Linux top is:

servervicibos:/root # top
top - 16:04:23 up 1:47, 4 users, load average: 1.41, 1.46, 1.63
Tasks: 161 total, 2 running, 159 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.8%us, 1.3%sy, 0.0%ni, 86.2%id, 1.3%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 8148252k total, 5316200k used, 2832052k free, 80240k buffers
Swap: 4168860k total, 0k used, 4168860k free, 4787952k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1871 root 20 0 4588m 52m 7720 S 25 0.7 68:27.46 /usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgc

The problem was recently presented, some weeks ago I never see this probleme at the server.


Thanks,

---
ViciBox Redux v.4.0.2 released from .iso | VERSION: 2.8-408a BUILD: 130711-2208 | Asterisk 1.4.44-vici | Vicidial Server, DB Server, WebServer, Dialers Servers | Dialers with Sangoma Hardware | No Extra Software After Installation | HP Servers, IBM Servers

[deficit]

You have probably run out of file descriptors.. try increasing your ulimit

Make sure you are starting asterisk with the boot script this should increase ulmit.

/usr/share/astguiclient/start_asterisk_boot.pl

[schernit]

i have the same error: unable to open pseudo channel - cannot allocate memory

Vicibox 5.0.3 64bits
VERSION: 2.10-442a
BUILD: 140612-2153
Asterisk 1.8.23.0-vici

I5 with 8gb mem

On asterisk init script I can see ulimit -n 65565

When this happen, agents listen silnce when calling and if I restart asterisk nothing happens, I have to reboot server to "fix" this.

Any help ? Is this a busybox bug ? Should I change version to 32 bit ?

[williamconley]

i have the same error: unable to open pseudo channel - cannot allocate memory

Vicibox 5.0.3 64bits
VERSION: 2.10-442a
BUILD: 140612-2153
Asterisk 1.8.23.0-vici

I5 with 8gb mem

On asterisk init script I can see ulimit -n 65565

When this happen, agents listen silnce when calling and if I restart asterisk nothing happens, I have to reboot server to "fix" this.

Any help ? Is this a busybox bug ? Should I change version to 32 bit ?

Before rebuilding your system, investigate the error as stated. I note you have not posted your available memory when this happens (htop is a marvelous tool). Also, you have not stated whether you have installed any other software in the server (which could have modified your ulimit setting). Try setting/checking ulimit manually just to be sure. You also didn't state whether this is a multi-server install or how many agents you have on the server when it happens. If you reduce your agents, does it still happen?

[schernit]

Hi william,

This is the scratch install. I didn't install anything else. I have only 17-18 agents in a single box making manual calls (not predictive).

The only change I did is to create a ram disk (2 gb) on fstab to generate recordings on live calls because we are recording every call. When mixing with process on crontab records are moved to /wav directory on disk.

Could it be the problem ?

I didn't do htop when bug happens. I will do it and let you know.

Regards

[schernit]

Just one correction:

Vicibox reduz v.5.0.2-130821 is my version

[williamconley]

When you say "scratch install", what do you mean? Since that means "installed manually from sources", and you listed Vicibox 5.0.3 ... I get the impression you are trying to use "scratch install" to mean "stock install" (as in you are trying to say you installed 5.0.3 and have made no changes). If so: Please use "stock" instead of "scratch". LOL

And next time it happens, post the actual error message, not your shorthand version of it. Possibly a couple lines above it. And the server load, and how much memory shows available in htop. And still: specifically answer "have you added any other software to the server after install?" If you're not sure, you can type "history" at the CLI and find out if someone else has been running commands that would install software.

And since fstab for your RAMDisk is your only change ... get rid of it. You don't need it with manual dialing anyway. (and because it's no longer "stock" after this change! seems reasonable to undo any non-stock items before trying anything else)

Just one correction:

Vicibox reduz v.5.0.2-130821 is my version

There was a bug in 5.0.3 ... kumba forgot to change the splash screen to 5.0.3 so it still says 5.0.2. If your CD shows 5.0.3 ... that's the real version.

[GaD]

Check if you have a faulty memory. These error messages you can find in /var/log. Look in messages or syslog or kern. Depends on your distribution you will find one or the other or all of them....

[schernit]

William. Ok. It was 5.0.3 stock version. I will undo my fstab changes and let you know.

[schernit]

I undo the changes with same result and I make a clean install with same .iso to another server with same CPU and DISK. I get same result. After some hours everything is a mess (agents are logged out or very bad voice, 20 second to make command such as hang up or call). When this happens on /var/log/asterisk I can see this message: "app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory".

I execute:

1 - htop: I can see asterisk on top with 31.4% CPU utilization with enought free memory and many other asterisk process with different PID but same RES and SHR
2 - free -m: total=7778, used 5074, free 2703, shared 0, buffers 150, cached 4601
3 - on asterisk -r, I execute meetme list and we only have 15 meetme users

I don´t understand what is going on.

As I said, This is another server with same hardware.

Any Idea ?

Regards,

Sebastian

[schernit]

Just to add: on start_asterisk_boot.pl there is: ulimit -n 65536

[schernit]

agents are using iax2 with zoiper (i have testes sflphone with same result)

[schernit]

I created a new server on same hardware with VICIBOX 6.0. It has another asterisk (1.8.28.2) and newer DAHDI (2.9.1.1).

I am getting same result. After some hours same bug and asterisk disconnects conferences. On /var/log/asterisk/messages I can see same Warning: "WARNING[XXXXXX] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory".

Today I decided to change start_asterisk_boot.pl ulimit parameters. I added ulimit -l (max locked memory) to unlimited and changed ulimit -n to 200.000. I will let you know if this solves the problem.

Regards,

Sebastian

[GaD]

How many agents are using your system? How many meetme rooms do you use? On the CLI do:

Code: Select all

meetme list

Do this on a normall workload day and then do it again when you start seeing this problem.

Also, what is your system's architecture? Run

Code: Select all

uname -a

and post the response to that.

[schernit]

I have only 17-18 agents at same time. Only Manual Dialing, all recording.

on normal operation:

asterisk02*CLI> meetme list
Conf Num Parties Marked Activity Creation Locked
8600051 0001 N/A 00:53:14 Static No
8600056 0001 N/A 01:03:17 Static No
8600057 0001 N/A 01:58:38 Static No
8600067 0001 N/A 02:00:25 Static No
8600068 0001 N/A 02:00:53 Static No
8600052 0001 N/A 02:00:54 Static No
8600066 0001 N/A 02:01:02 Static No
8600070 0001 N/A 02:01:40 Static No
8600065 0001 N/A 02:02:59 Static No
8600064 0002 N/A 02:03:38 Static No
8600055 0001 N/A 02:07:28 Static No
8600059 0002 N/A 02:18:23 Static No
8600053 0001 N/A 02:31:35 Static No
8600060 0002 N/A 03:23:25 Static No
8600063 0001 N/A 03:27:51 Static No
8600061 0001 N/A 03:29:35 Static No
8600054 0001 N/A 03:31:50 Static No
* Total number of MeetMe users: 20

Arquitecture:

Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz / MemTotal: 7870956 kB (telephony + apache)
Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz / MemTotal: 7870956 kB (dbserver)

I will let you know if error appears how many meetme users I have.

Regards,

Sebastian

[GaD]

That is not really your architecture, that just says that you have an i5 as your cpu... Post your

Code: Select all

uname -a

To see your kernel version and architecture.

[schernit]

asterisk02:/var/log/asterisk # uname -a
Linux asterisk02 3.11.10-11-default #1 SMP Mon May 12 13:37:06 UTC 2014 (3d22b5f) x86_64 x86_64 x86_64 GNU/Linux

It is the stock Vicibox 6.0.

-----------------------------------------------------------------------
It happens again. I think it is a memory leak of dahdi pseudo channel:

lsof | grep "/dev/dahdi/pseudo" | wc -l
15484

This number is increasing and increasing

When happening:

Conf Num Parties Marked Activity Creation Locked
8600052 0001 N/A 00:00:48 Static No
8600053 0003 N/A 00:08:33 Static No
8600065 0001 N/A 00:12:37 Static No
8600061 0002 N/A 00:13:07 Static No
8600067 0002 N/A 00:13:09 Static No
8600064 0002 N/A 00:16:32 Static No
8600066 0001 N/A 00:17:32 Static No
8600054 0003 N/A 00:18:05 Static No
8600063 0003 N/A 00:18:25 Static No
8600051 0001 N/A 00:18:44 Static No
8600060 0003 N/A 00:18:54 Static No
8600062 0001 N/A 00:19:01 Static No
8600059 0003 N/A 00:19:28 Static No
8600058 0003 N/A 00:19:33 Static No
8600057 0001 N/A 00:19:36 Static No
8600056 0003 N/A 00:19:39 Static No
8600055 0001 N/A 00:19:41 Static No

----------------------------------------------------------------------
I don´t know what else to do. It is really very extrange. Do I need to recompile something ?

Regards,

Sebastián

[schernit]

I don´t have any card. I am using dahdi_dummy. Is it normal this:

lsmod|grep dahdi

dahdi_transcode 14291 1 wctc4xxp
dahdi_voicebus 63307 2 wctdm24xxp,wcte12xp
dahdi 232030 213 xpp,dahdi_transcode,wcb4xxp,wctdm,wcfxo,wcaxx,wctdm24xxp,wcte11xp,wct1xxp,wcte13xp,wcte12xp,dahdi_voicebus,wcte43x,wct4xxp,oct612x
crc_ccitt 12707 2 wctdm24xxp,dahdi

regards,

Sebastian

[schernit]

Today I installed on same server vicibox 3.15. Same version I have on a other server. I added a TDM card to get timing. I uodated to same svn trunk I had before. I will tell you if everything works fine or not. Today This server was working fine with 17-18 agents full recording (manual dialing) for 4 hours. I will test tomorrow all day and let you know.

Regards,

Sebastian

[rrb555]

ive got

VIcibox:~ # lsmod|grep dahdi
dahdi 232030 6
crc_ccitt 12707 1 dahdi

[schernit]

Yesterday with vicibox 3.1 (asterisk 1.4.39.2) and vicidial VERSION: 2.10-443a BUILD: 140617-2017 (same as before) it works great. Today It was 12 hours working fine. This is same hardware that I have with vicibox 5 and 6 with only one difference. I added a Tdm440 card for timing.

I will try with same hardware with tdm card for timing and vicibox 6 to see how it works.

Anybody has any idea ?

Regards,

Sebastian

[schernit]

To resume: on exactly same hardware

Vicibox 3.1 - opensuse 11.2 / asterisk 1.4.39.2 works great. Very low load average all day
Vicibox 5 and 6: asterisk 1.8.28.2 or 1.8.23.0 works bad. Low load average all day but a bit higher than vicibox 3.1 and after some hours all calls are muted with an error on log.

Any ideas ?

[DirsaMX]

I hope It's not too late to post here, but im having the exact same issue on Asterisk 1.8.23.0-1_centos5.go (Goautodial 3.3 stock install) no extra hardware.

lsof | grep "/dev/dahdi/pseudo" shows a constantly increasing number of pseudo devices, up until the "app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory". error (Unrelated to free RAM).
ulimits are set correctly in start_asterisk_boot.pl

I managed to delay the error by increasing the value of "/sys/module/dahdi/parameters/max_pseudo_channels" but that is only a temporary Fix

[inthegenes]

I'm amazed no admins responded to this post last year.
Is this an example of memory leak or a dahdi issue?

[mflorell]

Admins did respond a year ago, this is not an issue on the most recent vicibox 6.0.3, and we do not support CentOS on this forum, you would need to go to the GoAutoDial forum for that.

Yes, memory issues can cause problems like this and many others. If you are having memory issues you should run an intensive memory checker like memcheck for several hours to see if there are any specific problems.

[inthegenes]

Hey Matt,

I had a similar issue. Was searching the forum and found the post. In the end I had to reboot the server to resolve the issue.

I'm running the recent VICIdial install.

VERSION: 2.10-452a
BUILD: 141111-0554

VICIbox Redux 6.0.3.141118

But the server was due a reboot. I don't have the nightly reboot via cron job on. I work remotely and everytime I reboot the box some of the services, including ssh don't start until someone logs in at the terminal.

This is an installation via a Preload ISO. Have you ever ran into this issue. Will be tackling it this weekend any assistance would be humbly accepted.

[mflorell]

You should schedule regular reboots for dialer servers, and if the ssh service isn't starting as it should on reboot then there is probably something not configured properly.

[williamconley]

I have NEVER had a system where ssh did not start on reboot. I would either re-install or find out what the "Very Intelligent, Highly Detail-oriented Software Specialist" did and undo it.

Nope, I'm wrong: I'd reinstall. It's safer. Then you don't need to find out what ELSE he did. And it's free. LOL

And upgrade while you're at it. Nightly reboot and there are no problems with a Vicidial server for years. Thanks to Kumba and Matt.

[inthegenes]

I'd take that comment as a compliment if I didn't think you were being sarcastic. I am the only software specialist managing the box in question. I'm not up for a reinstall and I failed to mention that Apache doesn't start on the box at reboot either.

I'm guessing its a firewall or routing table issue. The box has four NICs.

[williamconley]

If neither apache nor ssh start at boot, I'd reinstall without hesitating.

Note that if you fear the reinstall, you're going to have a bad day when this baby dies because of whatever is ailing it.

Try this on for size: Do a few "virtual" installations. VMWare, Virtualbox, vSphere (all free ...). Or just pop in a new HD and dry run on that a few times (put the original HD back in before go-time, of course).

Trust me when I say that getting good at installing the Free software so kindly provided by The Vicidial Group from the Free .iso so kindly provided by Kumba will allow your business to run much more smoothly. You don't usually get a warning when the system dies, but if you have a warning already and you're not practicing the installation ... you'll be smackin' yourself silly when it actually dies.

Did I mention that re-installing is Free? Even if it's fake?

And you DO have a good backup of this database and any customizations pushed nightly to a completely different HD, right?

Just to be funny: (Make the Backup FIRST ... NOW ... then:) Have you verified that both those services are set to auto-start?

[inthegenes]

Yes I have nightly backups of the Db and the Asterisk files running and being pushed elsewhere.
I have done many VICIdial installs, just don't consider re-installing everytime there is an issue.

If the box dies then I would have to go through the re-install process. Nonetheless, I will consider it if I am getting nowhere in my troubleshooting.

[inthegenes]

As expected (first thing I checked) routing table issue.
The default gateway was not being entered in the route table at boot. In my previous attempt to solve the issue I had entered a command for the default route via /etc/bash.bashrc.local

Which is why the ssh and apache would come up after logging into the PC from the terminal.
In the end their was a typograhical error in the Yast network settings for default gateway. An extra and or unneccasry hyphen at the device field

No re-install needed

[williamconley]

Good postback. Maybe you'll help someone else with similar symptoms ...

We've found similar issues if the network isn't physically connected at startup, on some systems the firewall will seize, or dns will fail for certain lookups, etc ... and the client will NOT reboot their server before calling. In several of those situations, the client is actually happy when a reboot resolves the issue even thought they got invoiced for a reboot, LOL.

But we've also had a few times when a simple typo in networking can cause unusual problems.

Challenges like this are often found somewhere in the /var/log folder, but sometimes its actually easier just to reinstall. LOL (Of course: we'd be dealing with a fresh install with a needle-in-a-haystack challenge ... no client using the box yet!)