Channels not closing - autodestruct

[alo]

We currently installed the new vicibox
from ISO ViciBox5.x86_64-5.0.3.iso
SVN Version: 2018
VERSION: 2.8-410a
BUILD: 130824-2319

What happening is the Active channels will keep building until the server is under high load and the agents start getting time Sync errors and bad quality. the CHAN section or the reports screen will raise over 4000. if I restart asterisk all is well again, until the calls start building up.

Looking at the Asterisk CLI I see:

WARNING[2626]: chan_sip.c:4214 __sip_autodestruct: Autodestruct on dialog '0d42b9247c928cc90702775372e13c01@<<Server IP>>:5060' with owner SIP/ in place (Method: BYE). Rescheduling destruction for 10000 ms

This error is displaying every second.

I have been Googleing like crazy for this error code, but not having much luck.

Only thing different between this server and our other office is the other server is on SVN Version: 2016 And a different network.

My thought it perhaps its network related, because I have seen these errors pop up. Less frequently then the auto destruct one. (maybe once a minute. (saw these error on vicibox 3 too but never caused much of an issue)

res_rtp_asterisk.c:2361 ast_rtp_read: Unknown RTP codec 72 received from

channel.c:4237 __ast_read: Dropping incompatible voice frame on SIP/ of format g729 since our native format has changed to 0x4 (ulaw)

chan_sip.c:26623 check_rtp_timeout: Disconnecting call 'SIP/' for lack of RTP activity in 61 seconds

app_meetme.c:3905 conf_run: Unable to write frame to channel

I would appreciate any Insight, and thanks to this awesome community.

[DomeDan]

these two seams strange:
"...with owner SIP/ in place (Method: BYE..."
"chan_sip.c:26623 check_rtp_timeout: Disconnecting call 'SIP/' for lack of RTP activity in 61 seconds"

seams like some variable is empty,
post your carrier dialplan/account entry/registration string/globals string etc.
and if you made any customization to the extensions.conf dialplan, post that too.

[alo]

That office was having a bandwidth problem, but thats been resolved and still having this trouble.

I am concerned that its not the carrier records, because I am using the same one on a different server.

You will see a few entries in the account Entry section because I have different carriers for inbound.
(I removed the host IPs for this post)
Account Entry:

[AloIn]
disallow=all
allow=ulaw
type=friend
host=
dtmfmode=rfc2833
context=trunkinbound
port=5060

[AloIn2]
allow=ulaw
type=friend
host=
dtmfmode=rfc2833
context=trunkinbound
port=5060

[AloIn3]
allow=ulaw
type=friend
host=
dtmfmode=rfc2833
context=trunkinbound
port=5060


[AloIn4]
allow=ulaw
type=friend
host=
dtmfmode=rfc2833
context=trunkinbound
port=5060

[AloIn5]
allow=ulaw
type=friend
host=
dtmfmode=rfc2833
context=trunkinbound
port=5060

Global string:
ALOIN = SIP/AloIn

Dialplan:

exten => _91NXXNXXXXXX,1,AGI(agi://127.0.0.1:4577/call_log)
exten => _91NXXNXXXXXX,2,Dial(${ALOIN}/${EXTEN:1},,To)
exten => _91NXXNXXXXXX,3,Hangup

[alo]

We also have been getting these errors.

ERROR[15344]: utils.c:1281 ast_careful_fwrite: fwrite() returned error: Broken pipe

but if I recall, thats fairly common. so I am thinking it is not the cause of the channels not closing. Although google did seem to indicate that it may be caused by a missing (hangup) in the dialplan. I have made no changes to the dialpan or any of the config files.

[DomeDan]

Yeah that last error is nothing to worry about,
but is this asterisk 1.4? does both boxes run 1.4?

[alo]

Its actually 1.8.23.0-vici Vicibox 5

Well, I ended up moving them to a different server with vicibox 5.02 REV 2016 and all is working.

Wish I would have been able to Reinstall to see if it was just a bad install, but they are happy with their new home.

Only difference I can see was hardware, or REV 2018 verse 2016.

[DomeDan]

I though you had 1.4 because of that ast_careful_fwrite error, I'm using 1.8.19.0-vici-beta and have never seen that since I upgraded

I would guess there was some issue during the install,
but, what hardware did you use on the failing server? (good to have that info here if someone else got this issue)

[ZibiX]

I'm having the same problem as alo.

My asterisk version is 1.8.24.0-vici

After the system has been rebooted i keep getting errors with Autodestruct. The errors keep building until cpu usage goes to 100% and agents canot work anymore. When i rebooted the server all starts from the beginning.

asterisk version is 1.8.24.0-vici has bugs? How to update this version or downgrade to another 1.8 or 1.4?

[ZibiX]

I have updated asterisk version to 1.8.25.0-vici. Problem still exist. After 3 hours of working there are a lot of errors and process AST_update.pl goes to 100% cpu utilization.

Please help me

[DomeDan]

found this: https://issues.asterisk.org/jira/browse/ASTERISK-19455
you can try this as a workaround:

Please note that as I pointed out on 1945, if you set DEFAULT_TRANS_TIMEOUT to 128000 it helps even more than 32000. I know this is a workaround, but it's very useful for our use case until the engineers start looking at it.

but here is a fix that might work.

@Dean, I think I may have (once again) figured out your issue.
In the errorlog.gz attachment, I took a look through the logs, and one line jumped out at me as being a bit odd:
[Mar 1 11:30:01] VERBOSE[3821] app.c: – Took too long, cutting it short...
This message means that when the conference participant was asked for his name, he took longer than 10 seconds to say it. The result is that a 10 second file was recorded for the participant's name. When the participant left the call, his channel thread then played that 10 second file into the conference bridge to let the other participants know that he had left the call (this happens even if he is the only participant in the conference). During the playback of this 10 second file, the SIP transaction timed out (since it times out after 6.4 seconds).
Essentially what you have here is the same situation that is happening in ASTERISK-19425. Something is occupying the channel for longer than the SIP transaction timeout. In your case, it's playing back the participant's name to the conference bridge. I suspect that the patch I have attached on ASTERISK-19425 will help with your problem. Note that even with the patch from ASTERISK-19425, you will see some warning messages about "Auto destruct called with owner in place" but the channels should eventually clear.
The only thing that I'm hesitant about is that you stated on that issue that even with DEFAULT_TRANS_TIMEOUT set as high as 128000, you still have a few stuck channels.
It may be that back when you made that adjustment, you were getting a combination of the problem I just described AND the issue where hangups were dropped that was fixed in revision 357762. What I suggest you do is to apply both the patch on ASTERISK-19425 and apply the change from revision 357762. If, with both of these applied, you still have stuck SIP channels, then please upload a new debug log and let me know which SIP call-ID is the one involved. In addition, if you find that you still have stuck SIP channels, let me know if the corresponding channels also show up in a "core show channels" call.
@Dan: Given that you are already running a new enough version of Asterisk and you have tried the patch at ASTERISK-19425 with no success, I'm beginning to think that you either have a different issue than what Dean has, or you have the same issue that Dean has for a very small percent of Dean's stuck SIP channels. For now anyway, I'm fine with using this issue as a place for you to upload your logs that demonstrate the stuck SIP channels. If you upload a log, be sure to let me know the SIP call-ID of the stuck dialog, and also let me know if the corresponding channel still shows up if you issue a "core show channels" command.

here is the patch he talks about: https://issues.asterisk.org/jira/secure ... 9425.patch
you could try to apply that, compile and test

if it aint working then post more info about your system and logs from asterisk

[ZibiX]

Thank you DomeDan for your help.
Server specs: Core i5 3.1 GHz, 8 GB RAM, 1 TB gard drive 7200 RPM
System was installed from vicibox 5.0.3 cd. No modification.

I have another server with exac specification. System was installed from the same cd. Only different is that it works on asterisk 1.8.23.1-vici. And it works very well.

How can i downgrade to version 1.8.23.1-vici or 1.4 version? A know that i will need to make correction in astguiclient.conf file and wen admin page.

Aboout patching astersik - I dont know how to do that...

[alo]

Good Morning ZibiX.

I actually ended up replacing the drives with solid state drives and this resolved the problem.

Yesterday I installed another server with raid 10 7200k Drives and 64gb ram and am having the same problem. (I also noticed my 'wa' from top higher then it should be. 5%)

I am not sure if this is truly the case but Both times after replacing the drives with solid state the issue was resolved.

[williamconley]

Excellent postback! Thanks for taking the time to let us in on your fix.

[rosnlama001]

I have installed vicibox 6.0.3 in X3250 M5 server. It was working fine about 6 months. But suddenly i see bad file descriptor error on asterisk logs. i have 24 agents who login in the server. when i give dial level more than 4 then bad file descriptor error message coming in the screen and every agent goes to waiting mode. When i reduce the dial level it is working fine. I dont get the issue. I am using asterisk 1.8. and the channels is not closing and when we restart asterisk autodestruct message is coming . My production is getting hamper. Please help .
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files
[May 11 08:34:34] WARNING[12348] tcptls.c: Accept failed: Too many open files


[May 11 14:41:25] WARNING[12360] chan_sip.c: Autodestruct on dialog '728105f71294bb8a2eb97f8d66aaad46@159.253.57.100' with owner SIP/m3st-00002660 in place (Method: BYE). Rescheduling destruction for 10000 ms
[May 11 14:41:43] WARNING[12360] chan_sip.c: Autodestruct on dialog '728105f71294bb8a2eb97f8d66aaad46@159.253.57.100' with owner SIP/m3st-00002660 in place (Method: BYE). Rescheduling destruction for 10000 ms
[May 11 14:42:01] WARNING[12360] chan_sip.c: Autodestruct on dialog '728105f71294bb8a2eb97f8d66aaad46@159.253.57.100' with owner SIP/m3st-00002660 in place (Method: BYE). Rescheduling destruction for 10000 ms