queue problem

[kpanik]

Hi,

I have a big problem with queue. I use astguiclient_snapshot_2006-11-02, asterik, zaptel, libpri e addons svn branches ....

some time the queue is blocked and the call they do not come transferred at agent.

Any suggestion ?

[mflorell]

do you have the following in your extensions.conf:
exten => h,1, DeadAGI...
exten => h,2, DeadAGI...

exten => 91NXXNXXXXXX,1,AGI...

exten => 8365,1,AGI...

[kpanik]

Yes ...

I've :

default h 1 DeadAGI agi://127.0.0.1:4577/call_log
default h 2 DeadAGI agi://127.0.0.1:4577/VD_hangup--HVcause ... EBUG-----${HANGUPCAUSE}-----${DIALSTATUS}-----${DIALEDTIME}-----${ANSWEREDTIME}

and

default _X. 1 AGI agi://127.0.0.1:4577/call_log

and


default 8365 1 AGI agi://127.0.0.1:4577/call_log

[mflorell]

What is the loadavg on the server when this happens?

How exactly is the call getting locked?

What is it's status in vicidial_auto_calls?

do you have any output of this happening?

[kpanik]

Hi matt,

Loadavg on the server asterisk is very low... 0.96 to max 3.50

120 concurrent calls with zap line ( te410p digium card ) with 4 PRI E1
codec a-law ~60.000 calls for day

this is my campaign configuration:

predictive : for 10 agents logged on is 12.0

all balanced



> How exactly is the call getting locked?

ten minutes ....but from the last call that enters in the queue ...

the system continue to call and all the calls taken from 8365 pass directly on queue but all agents are in status ready...

not remain that to cancel all calls with status LIVE from table vicidial_autocalls on the database and all it returns OK work fine.. !

this always does not happen !! random ~ 2 o 3 times to the week

suggestion ??


>What is it's status in vicidial_auto_calls?


all the blocked calls have like status 'LIVE' ...

>do you have any output of this happening?

yes ....

to which it would serve you ??

Tnx in advance

[mflorell]

On what kind of system is 0.96 to 3.5 loadavg low? What hardware is this on?

With the increase of issues like this I think that I will need to build in another method for clearing these calls out automatically, or at least a way of validating that LIVE calls are still even active channels on the server. This would probably be built into VDauto_dial.

[kpanik]

my asterisk + vicidial and apache server hardware

i've a dual xeon cpu nocona EMT64 HT enable 3,2 Ghz the supermicro mainboard chipset e7520 intel a digium te410p whith module echo cancellation ... 2 hdd scsi 73 gb on raid 1 3 gb ram ecc registered ..

on digium board i attached 4 PRI E1 ...


my mysql hardware is
a dual xeon cpu nocona EMT64 HT enable 3,2 Ghz the supermicro mainboard chipset e7520 intel .. 2 hdd sata 250 gb on raid 1 3 gb ram ecc registered ..


ok ok ...

we wanted to know if this happens to someone ??


with regard to as to eliminate the problem I had thought to inside insert query of Perl file that controls the time of one called with live statuses… if the time exceeds the calls it comes assigned a various status and it comes eliminated from the table auto_calls..

this is ok ??


my doubt was knowledge if I were the only one to having this problem..

it's possible that problem is relative to that it has to that to make with the synchronization of the time ???

tnx

[kpanik]

i'm sorry ...

120 concurrent calls with 120 lines and predictive adaptive_average "12.0"

60.000 call a day

10 agent logged on

60 % NA

32 % IVR with 15 seconds of message to LISTENS

8% customer to speak with agents ...

[mflorell]

I have seen this happen one to two times a month at the 140 seat center I monitor. The problem seems to be that a call will die right after setting the vicidial_auto_calls record to LIVE, but it will not delete when the call is hungup for some reason.

This seems to happen more often on loaded servers.

The way I plan to deal with this is to do something similar to what is done for vicidial_live_agents and have an updated column that will be updated every time the transfer AGI runs through it's loop and the AST_VDauto_dial script will check to see that none of those scripts have died and if they are not updating after s set number of seconds those LIVE records will be deleted.

This is also something that will eventually allow for better management of the calls that are waiting for agents.

[mflorell]

I just posted a whole bunch of changes to SVN-2 that address this problem. The code seems to be running fine in production so you shoul dbe OK giving it a try.

You need to make sure you run the new upgrade_2.0.2.sql queries for your database and you should update all of your transfer AGI scripts and your VDauto_dial script(assuming you were running the latest snapshot before)

Let me know if you need any help with this or if you run into any other issues.

[kpanik]

thanks for all... Matt

I try the new code on the new snapshot...

to more soon I will make to have turns out you to you !!!

[mflorell]

The code is not in the snapshot, it's on the SVN server, here are instructions to pull code down from the SVN server. Make sure you grab SVN 2-X:

http://www.eflo.net/VICIDIALwiki/index.php/SVN:howto

[kpanik]

sorry matt...

I have mistaken to express to me.....


I wanted to say the last one release in the svn.. !!!

[kpanik]

hi matt


no error in a week of job in the queue, this makes me to think that the problem is resolved definitively !!

thanks for all ...

[chrisls]

We are having this problem with our system. Intermittently, the system will stop transfering calls in the vicidial_auto_calls table with a status of LIVE to agents. At this point in time its tough to say what is actually causing this problem and I would like to find out.

Our setup is the following:

- version 2.03
- CentOS
- Dual Duo Core Xeon Processors
- 2 GB memory

We usually have between 10 - 25 agents on the system and load average never gets past 2.0. We are handling a mix of inbound/outbound calls in the center but really see this problem when we handle a burst of inbound calls and the queue fills up.

We have seen this issue manifest itself on the 2.01 and 2.02 code base. In the 2.03 code base I noticed there was code for detecting "jammed" calls and logging the event.

In AST_VDauto_dial.pl change log # 61113-1625 says code was added. I can see in that script at line 1175 the code runs.

I see the jammd event log on my system. The problem is that I don't know if this code is actually unjamming the queue. The SQL statement:

SELECT * FROM vicidial_auto_calls where server_ip='$server_ip' and last_update_time < '$BDtsSQLdate' and status IN('LIVE')

should be grabbing the LIVE records past a certain time, then it iterates through them and does its deletes.

Typically, once the jamming starts, it takes over 10 minutes for the system to be able to transfer calls to agents again.

Are there any other people out there still seeing this issue?

What output could I post to the forum which may help determining what the root cause of this issue is?

I can recreate this issue in a controlled environment. Thanks for any direction.

[mflorell]

responded to this one off-list:

This is an Asterisk AGI bug that I worked around in outbound by adding
the "concurrent transfers" field to vicidial_campaigns. Since I do not
do a significant amount of inbound I had not noticed the issue with
inbound calls as well, but apparently it exists there to. A quick fix
should be to modify the code in the inbound AGI script that you are
using to allow more than one transfer through at a time so that the
system could ignore a call that gets stuck.

Change this: if ($rec_countWAIT < 1)
to this: if ($rec_countWAIT < 2)

[chrisls]

We tried making that change but still the jam occurs. I am going to recreate the issue in the testing lab and I will post the content of vicidial_auto_calls and live_channels tables during the jam.

The version of asterisk is:

Asterisk SVN-branch-1.2-r42783M, Vicidial 2.0.3

The load avg on the server is typically under 2. The jam doesn't seem to be performance related.

In order to flush the queue to clear the jam we run a:

delete from vicidial_auto_calls where status = 'LIVE'

the original jam clear which cleared the oldest live call for a particular campaign didn't clear the jam so we experimented with clearing all LIVE calls and that worked.

Posting some other questions you asked offline to update anyone else who may be watching this thread:

How many calls will they get in one hour?
The amount of calls varies greatly...I have seen them get up to 157 calls in a fifteen minute period. Yesterday, I saw the jam happen when there were on 44 calls in the queue. We recreated the jam when there was only 23 calls in the queue.


What versions of Asterisk and astguiclient are you using?
Asterisk SVN-branch-1.2-r42783M, vicidial 2.0.3

We have updated both to the newest release version in July.

How many times a day will the queue fill up?
This depends on the amount of call volume they are getting. There are usually some calls in the hold queue, but the normal number is under 10. The jam condition only happen under number over twenty (this is a speculation, as I have never seen it happen under this number).


What is the loadavg of the server when this happens?
System load is below 2.0


What exactly are you doing to "emergency flush"?
We originally oped that the "Emergency VDAC Clear" feature would clear the jam, but it did not. We modified the clear feature to clear all "live" call from to auto_calls table and this clears the jam.

[seaq]

Hi, i´m having this same issue with inbound calls, i´m trying to debug but the description in this thread is close to what´s happening

i´m using

vicidial 2.04
Asterisk 1.2.24

I´m testing with 3 agents and it seems to happen when a call is hanged too early, i´m trying to reproduce to take the right logs.

What information is needed in order to help debug and solve this issue?

Thanks

[seaq]

I could reproduce the issue, just hanging up while i was waiting on the line

i´ve got some debug data

1. extensions.conf

only this DeadAGI line.. is this correct??

exten => h,1,DeadAGI(agi://127.0.0.1:4577/call_log--HVcauses ... EBUG-----${HANGUPCAUSE}-----${DIALSTATUS}-----${DIALEDTIME}-----${ANSWEREDTIME}))


2. show channels

asterisk0*CLI> show channels
Channel Location State Application(Data)
Zap/pseudo-107156634 s@from-zaptel:1 Rsrvd (None)
SIP/cc125-08833a90 8600052@default:1 Up MeetMe(8600052)
2 active channels
1 active call


at this moment there are 2 fake calls waiting (there are not in asterisk)
and 1 agent waiting for call.

3. the vicidial_auto_calls table

show both fake calls as LIVE. Both of them here hanged up when listening MoH.


4. i´m using agi-VDAD_ALL_inbound.agi

it has a if ($rec_countWAIT < 1)

i´ve modified it but it still present the problem

5. steps to reproduce (at least in my case)

- put your agent in pause
- call with 2 lines and wait both for get in hold
- hangup while in hold (this would be the fake call)
- call again and get in hold
- put your agent in ready


in the test i´ve made the callers got in hold and no calls are passed to the agents... it lasts for about 6 minutes almsot 10 in hold, after that the fake call is cleared and it seems that the calls are being passed again and get sticked again whenever someones hangup again in hold...

6. manual solution

the manual solution i´ve found is to delete the LIVE fake record in vicidial_auto_calls (as pointed previously) , once is deleted the calls start flowing again.

what else should i paste here?

thanks

[mflorell]

Do you have multiple contexts in your dialplan?

[chrisls]

This problem for us is now resolved. It was happening due to a misconfigured dial plan. For us, once we made sure the VD Hangup extension is being called in the Asterisk dialplan after all inbound calls, the issue went away. This is just one of the many issues which will manifest itself if the VD Hangup extension isn't being called after inbound call completion.

[mflorell]

Thanks for posting the followup!

[seaq]

ok, i'll check this and will follow up

thanks

[seaq]

Hi, thanks for all your help!!

In fact my extensions.conf and my context were a mess.

I've regenerated and now it works just fine.

Now is working.

Thanks!