found a potential bug with AST_VDauto_dial_FILL.pl

Discussions about development of VICIDIAL and astGUIclient

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Wed Sep 05, 2012 8:38 pm

using svn Revision: 1842

there are two queries that calculate how many calls should be placed for the load balance setup. Here are the original queries:



Code: Select all
                                                $stmtA = "SELECT count(*) FROM vicidial_auto_calls where server_ip='$DB_camp_server_server_ip[$server_CIPct]' and campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE';";
                                                $sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
                                                $sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
                                                $sthArows=$sthA->rows;
                                                if ($sthArows > 0)
                                                        {
                                                        @aryA = $sthA->fetchrow_array;
                                                        $VAC_server_BALcamp =   $aryA[0];
                                                        }
                                                $sthA->finish();



both should consider XFER and CLOSER calls not as dialed calls like this:

Code: Select all
                                                $stmtA = "SELECT count(*) FROM vicidial_auto_calls where server_ip='$DB_camp_server_server_ip[$server_CIPct]' and campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE' and status NOT IN('XFER','CLOSER');";
                                                $sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
                                                $sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
                                                $sthArows=$sthA->rows;
                                                if ($sthArows > 0)
                                                        {
                                                        @aryA = $sthA->fetchrow_array;
                                                        $VAC_server_BALcamp =   $aryA[0];
                                                        }
                                                $sthA->finish();



basicly add and status NOT IN('XFER','CLOSER'); at the end of the query.. there might be more statuses we should add but im not sure if this will break anything or not. Let me know what your thoughts are and if this is correct i'll add a patch to the issue tracker
Last edited by bobbymc on Thu Sep 06, 2012 5:31 pm, edited 1 time in total.
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Thu Sep 06, 2012 7:02 am

Please post a diff file in a new issue tracker ticket and we will take a look at it.

Also let us know if you have run your patch in production for any length of time.
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Thu Sep 06, 2012 7:13 am

been running this for a week now and it solved our issue. So far i dont see any BUGS from this change. i will create a diff on it asap
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Thu Sep 06, 2012 3:33 pm

today i found a small issue with it i belive.. there is a race condition where once in a while its not fast enough to detect the agent is in a call and it places another set of dials for that agent. any suggestions on how to overcome that?
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Thu Sep 06, 2012 5:31 pm

one of the 2 places above should not be updated witht he XFER and CLOSER exclusion.. i updated the first post to only show which area should be fixed
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Tue Sep 25, 2012 10:10 pm

mflorell wrote:Please post a diff file in a new issue tracker ticket and we will take a look at it.

Also let us know if you have run your patch in production for any length of time.



matt after a few weeks of testing i finally got this bug totally figured out.. i was right the first time that we need to exclude XFER and CLOSER statuses on both places in the script that check vicidial_auto_calls

Code: Select all
$stmtA = "SELECT count(*) FROM vicidial_auto_calls where campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE'";

$stmtA = "SELECT count(*) FROM vicidial_auto_calls where server_ip='$DB_camp_server_server_ip[$server_CIPct]' and campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE'";


BUT there was a downside.. the issue was campaigns that had avaliable tally on would have a race condition where if the calls got trasnfered at that split second the system will try to fill in the calls missing in the dial level and hence causing dropped calls by placing more calls then the system should.

to solve this wat i did in the FILL scritp is also get the available_only_ratio_tally settigns from the vicidial_campaigns table and do a check if the campaign has it enabled do not exclude those statuses from the calculation:


here is a sample code of what i mean:

Here i also get the available_only_ratio_tally settings and apply it to the query
Code: Select all

                                ### grab the dial_level and multiply by active agents to get your goalcalls
                                $DBIPadlevel[$camp_CIPct]=0;
                                $stmtA = "SELECT dial_timeout,dial_prefix,campaign_cid,active,campaign_vdad_exten,omit_phone_code,auto_alt_dial,queue_priority,use_custom_cid,available_only_ratio_tally FROM vicidial_campaigns where campaign_id='$DBfill_campaign[$camp_CIPct]';";
                                $sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
                                $sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
                                $sthArows=$sthA->rows;
                                $rec_count=0;
                                $active_only=0;
                                while ($sthArows > $rec_count)
                                        {
                                        @aryA = $sthA->fetchrow_array;
                                        $DBIPdialtimeout[$camp_CIPct] =         $aryA[0];
                                        $DBIPdialprefix[$camp_CIPct] =          $aryA[1];
                                        $DBIPcampaigncid[$camp_CIPct] =         $aryA[2];
                                        $DBIPactive[$camp_CIPct] =                      $aryA[3];
                                        $DBIPvdadexten[$camp_CIPct] =           $aryA[4];
                                        $omit_phone_code =                                      $aryA[5];
                                        $DBIPautoaltdial[$camp_CIPct] =         $aryA[6];
                                        $DBIPqueue_priority[$camp_CIPct] =      $aryA[7];
                                        $DBIPuse_custom_cid[$camp_CIPct] =      $aryA[8];
                                        $DBIPavailable_only_tally[$camp_CIPct] =                $aryA[9];

                                        if ($omit_phone_code =~ /Y/) {$DBIPomitcode[$camp_CIPct] = 1;}
                                        else {$DBIPomitcode[$camp_CIPct] = 0;}

                                        $rec_count++;
                                        }
                                $sthA->finish();



here i set the sql string addition:

Code: Select all
                                $available_only_tallySQL = "";
                                if ($DBIPavailable_only_tally[$camp_CIPct] =~ /N/) {
                                        $available_only_tallySQL = " and status NOT IN('XFER','CLOSER')";
                                }


here is where i apply the addition
Code: Select all

                                $stmtA = "SELECT count(*) FROM vicidial_auto_calls where campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE' $available_only_tallySQL;";
                                $sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
                                $sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
                                $sthArows=$sthA->rows;
                                if ($sthArows > 0)
                                        {
                                        @aryA = $sthA->fetchrow_array;
                                        $VAC_balance_fill =     $aryA[0];
                                        }
                                $sthA->finish();
                                $DBfill_current_balance[$camp_CIPct] = "$VAC_balance_fill";





and also here


Code: Select all
                                                $stmtA = "SELECT count(*) FROM vicidial_auto_calls where server_ip='$DB_camp_server_server_ip[$server_CIPct]' and campaign_id='$DBfill_campaign[$camp_CIPct]' and call_type='OUTBALANCE' $available_only_tallySQL;";
                                                $sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
                                                $sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
                                                $sthArows=$sthA->rows;
                                                if ($sthArows > 0)
                                                        {
                                                        @aryA = $sthA->fetchrow_array;
                                                        $VAC_server_BALcamp =   $aryA[0];
                                                        }
                                                $sthA->finish();





This is fully tested.. now the only worry i have is how will this effect the tally thredhold feature and how will that possibly change things.. please let me know what you think and if the tally threshold feature needs to also be considered and if so in what way so i can code it..

The reason why i make such heavy use of the load balance system is to lower the load on the servers agents are registered on in order to handle more agents on a single server and send all calls places to so called "call servers".

i like to finish this BUG completely before i submit a patch..
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby williamconley » Tue Sep 25, 2012 10:54 pm

You may want to mention how much "production time" this patch has seen and with how many agents/servers/channels. Makes a big difference if you have 27 agents and 3 dialers with 150 channels and ran it for 6 hours vs 174 agents, 8 dialers, 2 webs, 1 DB and 800 channels and ran it for a full week ... Just a thought. Even if you did not run it in your entire enterprise, knowing where you DID run it would be helpful.

And if you want Matt to show interest ... I would strongly suggest posting the DIFF (now, in its present state) so he can possibly apply it and see what happens instead of reading code in a forum to see what happens.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20253
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Tue Sep 25, 2012 11:29 pm

this was on on a system with 300 agents and a total of 6 asterisk boxes. i will be more then happy to create a diff for the latest svn trunk but before i do so i like to finish the fix if the tally threshold needs to be considered in this BUG
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Wed Sep 26, 2012 5:31 am

You should test the tally threshold feature to make sure there are no adverse effects.
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Thu Sep 27, 2012 12:46 am

matt im going crazy.. disabling this "FIX" for tally only is not a good idea
(by fix meaning excluding XFER and CLOSER statuses) .. what happens is because the FILL script does not exclude XFER and CLOSER status in auto_calls a scenario can happen where agents are ready and its not going to dial any calls for them. Here is why (this bug happens when you setup a load balanced setup)

lets say there are 6 agents logged in with method ratio with dial level of 2. that means its suppose to call 4 calls total for the 2 agents in READY status. Because there are 4 other agents in a phone call with a record in auto_calls with status XFER the calculation will not place ANY calls for the 2 agents waiting because the total call goal is 4 and there are 4 total calls in XFER mode and those two numbers cancel each other.

$VAC_balance_fill is flawed because witht he SVN repo right now it does not exclude XFER or CLOSER statuses.
Code: Select all
$DBfill_needed[$camp_CIPct] = ($DBfill_shortage[$camp_CIPct] - $VAC_balance_fill);



No on the flip side this issue happens on avaliable tally if we do exclude the XFER and CLOSER status:

It looks like its a race condition. Lets say there is one agent logged in and at a dial level 2. As soon as one of the calls pick up and transfer to the agent another call gets placed right away to make it a total of 2 calls placed at all times. This happens randomly and causes dropped calls, because the agent is in call now and no more calls should be placed. I cant think of any way to beat this race condition between the AST_VDauto_dial.pl and AST_VDauto_dial_FILL.pl file.

The idea behind using the load balance system is so all campaigns can share the servers and there aren't servers sitting there idle when it could be used to place more calls. Im more then happy to program any solution but i CANT think of how to solve this race condition =(

matt thank you so much for reading this and i hope you can help me solve this issue.
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Thu Sep 27, 2012 9:17 am

I wish I had the time to dedicate to troubleshooting this, but that would take several hours at least, and we are in the middle of a couple of very complex paid development projects. I should have some time in about a month or so.
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Thu Sep 27, 2012 2:51 pm

mflorell wrote:I wish I had the time to dedicate to troubleshooting this, but that would take several hours at least, and we are in the middle of a couple of very complex paid development projects. I should have some time in about a month or so.


if it possible to pay to get this bug fixed?
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby williamconley » Fri Sep 28, 2012 12:42 am

If you want to Pay the Vicidial Group to get a bug resolved, contact them directly at their sales email or phone. But be prepared, this is the busy season ... it will not be inexpensive.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20253
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Sat Sep 29, 2012 4:14 pm

The problem with a development project like this is that it is very little development and a whole lot of testing, and one small change can have big effects. So it is difficult to even estimate the amount of time needed. Also, we are swamped with development projects right now, including two big ones that are requiring a lot of time to complete. We can usually do small development projects within a week or two, but one like this needs more time, and we are currently quoting November as the time we could get to projects like this.

As far as figuring out how much time to quote, I can only guess at how long it would take, 10 hours would be the minimum. For a quote just email sales@vicidial.com.
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Sun Sep 30, 2012 10:26 pm

Would a 2-3k bounty do the trick ?
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Mon Oct 01, 2012 8:20 am

probably would work
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Mon Oct 01, 2012 1:05 pm

K I can pay right away.. I send a email to the vicidial group. Can you please let them know so they can start on it already ?
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby williamconley » Sun Oct 07, 2012 10:57 pm

I would venture to guess that the payment must precede the "begin". Send your email to sales@vicidial.com to find out how best to pay them.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20253
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby bobbymc » Sat Oct 13, 2012 10:47 pm

i contacted them but i have not gotten a quote yet. what can i do to speed that up?
bobbymc
 
Posts: 425
Joined: Fri Jan 05, 2007 12:26 am

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby williamconley » Sun Oct 14, 2012 4:15 pm

During this season, your best bet would be to offer A LOT more money. Or wait until the rush wears off. If they are working on projects in the realm of $10k+ and you want to "discuss" a 2-3K project ... it could even be sped up (most likely) by PAYING $3k and then asking them when it will be done and being patient. Otherwise (honestly) you're just another person who wants to talk about a project (which does not represent revenue ...).
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20253
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: found a potential bug with AST_VDauto_dial_FILL.pl

Postby mflorell » Thu Oct 18, 2012 6:07 am

The project we are working on is huge, we've been working on it since May, and we are pretty close to finishing it. But we have several projects that have been waiting for this huge project to finish, some for over 2 months. It's not a case of money, it's place-in-line in our development queue. Small projects can usually be done quickly in a few hours when our daily schedules allow, but everything else has to wait, unless it is a critical issue, and this is not a critical issue since I have not observed this issue on any of our client systems, and nobody else has complained about it.
mflorell
Site Admin
 
Posts: 18383
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida


Return to Development

Who is online

Users browsing this forum: No registered users and 17 guests