Congestions, SITs, and other Crap. (handling)

[mcargile]

So we have run into an interesting problem. One of our customers has a full DS3's worth of T1s through Quest. Quest is doing some funny things. They are not sending the correct D channels message when we dial a disconnected number but they send an in band alert and on the B channel play the original switch audio with SIT and announcement which is not detected by Dial(). Through the whole time Dial() stays in ringing. This is a stretch of the ISDN protocol however is not technically a violation. Asterisk is seeing the call as still ringing and will timeout the Dial() sending the call into the hang up extension. Vicidial is thinking that there is another line available however the CO does not release the B Channel for another few seconds and we can still hear the audio on the B channel using ZapScan(). This causes multiple problems:
A. The disconnected number is not detected and ends up as a NoAnswer being dialed over and over again.
B. Vicidial is trying to dial out on the channel when it is not available yet.

Our current bandage for this problem is to decrease the number of trunks the server can dial out on and to put a wait in the h extension to slow down the reusing of the lines. This however is not ideal as it means our client is not dialing at max capacity and it has not necessarily fixed the issue it has just made it appear far less. We are looking to put some checks in the dial plan that would check for available lines before dialing and would wait till one is available. We plan to put this check after the call to the logging agi but before the dial so that Vicidial thinks that the line is in use.

This does not fix the overall problem however because the disconnects are not being handled properly and we are basically fooling Vicidial in order to get proper results.

We are hoping to get Quest to change this behavior to be more in line with the ISDN standard, but this might end up being like running head first into a brick wall. We are going to try other things to force Asterisk to do the tone detection even though these are PRI circuits.

This does however point out a problem with the current implementation of Vicidial which is the the assumption that if a call is hung up it's channel is available. We can either add a check to the hang up agi to see if the channel is really released, which has the problem that the channel might already be in use again by the time the check is run. Or we can add a check on the dialing agi to see if channels are available before we dial though there is a delay between the check and the actual dial.

[mflorell]

I have noticed this through some providers as well, maybe it would be a good idea to add a sleep() command only for NA calls in one of the DeadAGI scripts to deal with this.

[mcargile]

Question is how much of a sleep? We have had audio on the line for 15 seconds after the hangup before. And with this client in particular they have at least 1/2 NAs.

It would be better to block until the channel frees.

[mflorell]

That's a tough one, I have not yet found an absolutely guaranteed way to determin when a Zap channel is free, especially when Asterisk says it's free. I guess I would start by gathering all of the NAs that do this and dial them manually keeping track of the times on each and see if there is some kind of pattern or predictable delay.

[explidous]

Asterisk can Tell us how many channels are really available for dialing, why don't we check before dialing if there are enough "spare" channels for incomming calls instead of just keeping track of the number of calls in progress. On PRI calls in Progress and available channels can differ because of the nature protocol. This might as well help to automatically adapt the dialing if a T1 goes down...

Frank

[explidous]

From what we saw it seems to be related more to the switch that the number is not available on, or at least the provider...
And 15 sec seem to be like FOREVER in predictive dialing... don't have to tell you that, I suppose....

[mflorell]

The number of free Zap channels that Asterisk says are free do not always match the number of B-channels that are ready to place calls on. Is there any more accurate way of checking a B-channel for it's status?

[explidous]

On a purely outbound scenario it would be possible to track the availability of the B channels separately but it would have to be done in Asterisk or use the D channel debugging output (not the most performant way of doing it).
However the Problem remain if a line is used for inbound and outbound because B channel association can be switch by the CO side during the call establishing phase (collision) of the ISDN/PRI protocol.
To fix both problems I would sugest the following enhancement:
For situation where Lines are used for inbound/PBX and outbound calls define Max Vicidial Trunks to full number of trunks but implement an extra Server Variable of "dynamic spare channels" which will always leave that number of channels available no matter for what the other channels are used. This way that number can be set fairly low ( 1-3 should suffice in most settings) but still a higher number of lines can be used for inbound or PBX use without unnecessarily blocking lines for outbound dialing.
Implementing a "Trunks used" variable the will allow to automatically check these Trunks/Trunk groups for number of available channels. Maybe only checking if Max Trunks is set to 0 and only calculate it then.
With not too much work one can create an Asterisk Module that checks the available channels at that time (with a technology dependent option that checks actually available B-channels in case of PRI/ISDN.
The second half is to treat Congestion the way that it should be...
A call that reported Congestion should for all knowledge be treated as not dialed or busy, because the call did not make it to the receiving party, maybe an extra status for Congestion would do as well.

Please let me know what you think before I start working on that...

[mflorell]

That should be possible, but the trunk reservation code is pretty complex due to the per-campaign-reservations for vicidial trunks that you can do.

I usually just solve this problem, on systems that have this problem, by just setting the Max VICIDIAL trunk number 1-4 lines lower than what is available.

[mflorell]

That should be possible, but the trunk reservation code is pretty complex due to the per-campaign-reservations for vicidial trunks that you can do.

I usually just solve this problem, on systems that have this problem, by just setting the Max VICIDIAL trunk number 1-4 lines lower than what is available.

[explidous]

Without using the constant wait in the Hangup extension we have to reduce the number of available lines from 161 to 135... that is about a whole T worth that we have to leave for these cases. By going to a general wait after each connection of 2 sec we get down/up to a 145 max usable lines, still not that good especially since adding 2sec to each call limits the thruput as well, the conditional wait might be better however I am not sure on which connections I all have to put the wait in since the only messages that tell me that we need to wait are not provided by asterisk unless in d-channelö debug

[mflorell]

The conditional "sleep" should be easy to add to the FASTagi_log.pl script in the ^VD_hangup section. Putting in something that checks the dial status for CANCEL|BUSY|CHANUNAVAIL|CONGESTION|etc... and only adds the sleep to those calls should work.

[explidous]

That information is not going to help to much, since the original problem includes not getting propper messages on disconnected lines... they just show as ringing... and then timeout...

[mflorell]

What is the ringing-then-timeout dialstatus variable value? (Every call termination has something as a variable)

Usually it is CANCEL that I have seen.

[mcargile]

we think the dialstatus on timeout is CANCEL normal clearing 16. However most of the calls timeout anyways. And as far as we can tell they look exactly the same.

[mflorell]

What kind of NA percentage are you getting?

[mcargile]

1/2 - 2/3

they are dialing a lot of older leads that there is a good portion are disconnects (why this is such an issue).

They are in collections which means most of the people have probably not been able to pay their phone bill.

[mflorell]

Maybe you should email to Matt Fredrickson(sp) at Digium about this, he is the one who maintains the zaptel release and he might have some more information on this.

On an unrelated question, Does this client have multiple phone numbers per lead for these leads?

[mcargile]

not exactly sure but when i mentioned that 2.0.3 will be able to do alternate number dialing they like the sound of that.