Force Hopper Reset + Run Hopper Immediately

[martinch]

Hey guys, hope everyone is well.

I was wondering if anybody would be interested in a new feature that couples with the "Force Reset of Hopper" admin option called "Run Hopper Now". It does what it says on the tin, it can run the hopper immediately from the Campaign Basic + Campaign Detailed pages on the admin panel. I proffered the suggestion to a ViCiDial administrator recently and he was very eager to have such a feature. I wonder if other administrators feel the same way.

The benefits to having this option;

- If you use this option with "Force Reset of Hopper", you can have a completely fresh hopper within seconds without having to wait for the hopper to run via cron. This can increase productivity for a campaign.

- If you get fresh, time-sensitive data loads often throughout the day, this option can be very useful to get the fresh data into the hopper faster ready to be dialled by your agents.

- It can allow dialler management teams to run the hopper manually so relieves pressure on ViCiDial administrators who historically would do it in their place.

Here is a screenshot of how the option might look;



And here is a screenshot of the basic debug;



Interested to know what you think. I'm still working on the feature and will submit on Mantis when finished

Pending development items;

Add the admin panel option to Campaign Basic ✅

Add the admin panel option to Campaign Detailed ⏳

Add basic debug to page success area (Basic). ✅

Add basic debug to page success area (Detailed). ⏳

Add verbose debug to page success area (Basic). ⏳

Add verbosedebug to page success area (Detailed). ⏳

Add help for new option. ⏳

Allow running the hopper on All-In-One ViCiDial installation (like ViCiBox). ✅

Allow running the hopper on remote ViCiDial database box. ⏳

Cheers guys

[mflorell]

We have quite a few clients with specific CLI flags enabled on their hopper script, will this new feature allow for those?

[martinch]

We have quite a few clients with specific CLI flags enabled on their hopper script, will this new feature allow for those?

Ah I see mflorell. I didn't code that but I have now...although many of the hopper options don't seem to have context with this feature. Could I get your steer on this list please if you have time as you know better than I do;

• [--help] <- terminal debug which doesn't have context here.

• [--version] <- terminal debug which doesn't have context here.

• [--count-only] <- terminal debug which doesn't have context here.

• [--debug] <- terminal debug which doesn't have context here.

• [--debugX] <- terminal debug which doesn't have context here.

• [--dbgmt] <- terminal debug which doesn't have context here.

• [--dbdetail] <- added.

• [--allow-inactive-list-leads] <- added.

• [--level=XXX] <- added.

• [--campaign=XXX] <- this is included by default.

• [--wipe-hopper-clean] <- redundant in context of campaign as Force Reset of Hopper will delete the leads. This option will also wipe other campaigns which is risky.

Do you like this design;



Or this one;



Thanks

[jamiemurray]

Personally I wouldn't be comfortable adding the options on that screen, too many managers could end up tinkering with it thinking it's going to magically fix a different problem and potentially cause the hopper script to fail if they screw the syntax.

I see potential in this feature for some of my clients who are regularly changing lists, lists are split into geographical areas in one particular case and when they fill the diary of appointments in one area, they move onto the next and they achieve that with changing active lists and resetting the hopper which can cause up to 60s of no calls being placed, for a small team it's not much time, but when you're looking at a campaign of 240 agents, everytime you do that, that's up to 2 hours of dialing lost in just 60 seconds effectively.

What about getting the hopper options from a settings container at run time? This would enable administrators to lock access to them relatively easy.

[martinch]

Personally I wouldn't be comfortable adding the options on that screen, too many managers could end up tinkering with it thinking it's going to magically fix a different problem and potentially cause the hopper script to fail if they screw the syntax.

I see potential in this feature for some of my clients who are regularly changing lists, lists are split into geographical areas in one particular case and when they fill the diary of appointments in one area, they move onto the next and they achieve that with changing active lists and resetting the hopper which can cause up to 60s of no calls being placed, for a small team it's not much time, but when you're looking at a campaign of 240 agents, everytime you do that, that's up to 2 hours of dialing lost in just 60 seconds effectively.

What about getting the hopper options from a settings container at run time? This would enable administrators to lock access to them relatively easy.

Hey Jamie,

I completely forget settings containers were a thing. Yeah it would make sense to store the specific hopper settings in there...thanks

[mflorell]

It's not a bad idea, but you should build in some way to prevent the hopper from being launched if it is already running or from the launching being triggered multiple times consecutively. We've had clients with very underpowered systems where the hopper took a long time to run, and I can see a feature like this being used when managers are trying to "speed things up" but instead make everything worse. Probably making this a disabled-by-default system setting would be good as well.

As for implementation, perhaps making it somehow use the existing "AST_manager_send.pl" script(which has a very short loop time and it already triggers another perl script to run) might be the best option.

[jamiemurray]

Good point Matt, if the hopper takes too long to run you could end up with call delivery and agent actions (like manual dial, dispo) being blocked until it completes. Multiple times a minute could bring a system to its knees.

The hopper runtime could be stored and referenced at page load, optionally blocking this feature if it's too high.

This could likely be expanded on to notify administrators when the run time of the hopper gets to a level that impacts dialing negatively.

[martinch]

It's not a bad idea, but you should build in some way to prevent the hopper from being launched if it is already running or from the launching being triggered multiple times consecutively. We've had clients with very underpowered systems where the hopper took a long time to run, and I can see a feature like this being used when managers are trying to "speed things up" but instead make everything worse. Probably making this a disabled-by-default system setting would be good as well.

As for implementation, perhaps making it somehow use the existing "AST_manager_send.pl" script(which has a very short loop time and it already triggers another perl script to run) might be the best option.

Thanks Matt for the feedback.

Sidenote...is it worth introducing a concurrency check for the hopper? Ideally, the hopper should run only be running one instance at any given time, right? Overlapping hopper runs would get messy.

Potentially something like this;

Code: Select all

my $lock_file = "$script.pl";

# Open a file with an exclusive lock
open(my $lock_file, '>', '$lock_file.lock') or die "Could not open lock file: $!";

unless(flock($lock_file, 2 | 4)) {
    # Exclusively lock file, non-blocking print "Another instance of this script is already running.";
    exit 1;
}

# Unlock the lock file
flock($lock_file, 8);

# Close the lock file handle
close($lock_file);

With that said, this might be the case for other scripts...such as the CRON scripts and various other backend scripts in /bin.

Thanks.

[mflorell]

I think there are over 20 perl scripts in VICIdial at this point that have concurrency checking built-in to them(it is optional on some scripts). It tends to be simpler using 'ps' to detect the other running processes:

Code: Select all

### concurrency check (SCREEN uses script path, so check for more than 2 entries)
if ($run_check > 0)
   {
   my $grepout = `/bin/ps ax | grep $0 | grep -v grep | grep -v '/bin/sh'`;
   my $grepnum=0;
   $grepnum++ while ($grepout =~ m/\n/g);
   if ($grepnum > 2)
      {
      if ($DB) {print "I am not alone! Another $0 is running! Exiting...\n";}
      $event_string = "I am not alone! Another $0 is running! Exiting...";
      &event_logger;
      exit;
      }
   }


[martinch]

Thank you @mflorell.

A quick update on development of this piece;

• I've add a concurrency check to the hopper by porting code from the FTP script. You can run the hopper with or without the --run-check flag. I've decreased the check number from 2 to 1 as you wouldn't want 2 hopper instances running simultaneously as that could get rather messy. ✅
• I've added a note beside the setting that will prompt administrators that wish to run custom arguments with their hopper runs to setup a settings container for it. It must be named HOPPER for it to work. ✅
• I've added two outcomes when attempting to run the hopper via the admin page;
  
  • If the concurrency check reveals that the hopper is already running, the administrator will see this message;
    
  • If the concurrency check reveals that the hopper is not running, the administrator will see this message;
    
• I've also added a toggle to the System Settings page so administrators can control this feature as they see fit. It's turned off by default though. ✅
  
• Support for running on remote database servers (pending)⏳

All in all, it's very close to being done. I hope to submit the code on Mantis soon thanks guys.

[martinch]

I finished the feature now and I've uploaded it to Mantis. The link is here -> https://www.vicidial.org/VICIDIALmantis ... hp?id=1470

Thank you all

[mflorell]

Thank you very much Martin!

I look forward to reviewing your code. Might be another week or two though, I'm in the middle of two larger paid projects at the moment: Agent Latency Log gap alerts and Demographic Quotas. But as soon as those two are done I'll start working through the back-log of submitted code changes.

[martinch]

Thank you very much Martin!

I look forward to reviewing your code. Might be another week or two though, I'm in the middle of two larger paid projects at the moment: Agent Latency Log gap alerts and Demographic Quotas. But as soon as those two are done I'll start working through the back-log of submitted code changes.

You're most welcome Matt and no worries...these coding pieces are not urgent so it's not a problem at all.

Agent Latency Log Gap Alert? Is that related to the piece you wrote recently with all the lag logging? Sounds like that would be an interesting feature as it gives ViCiDial administrators a "heads up" and can tend to problems proactively. Perhaps an email notification to a distro or something like that?

Thanks for your support as always

[mflorell]

Yes, alerts every minute by email to multiple email addresses when latency gaps pass a threshold for current logged-in agents. Can also be segregated by User Group, and there is an end-of-day option with all gaps for the day.

We've been working on this for a big client with internal network issues for the last couple of weeks, and it has been instrumental in their IT group finding out where their network issues are in real-time.

[martinch]

Sounds like fun and games

Yeah when I had this idea...I was thinking of the big corporate clients using ViCi or businesses with dedicated IT departments...far too often given the fast paced nature of the call center world...problems with LAGGED stuff may not appear until the next day or so which is no good really because the moment has passed...this feature will give techs and admins a fighting chance in the moment

[jkidd]

Curious if this was ever built? We run email alerts from a bunch of things using a separate system. An alert that says if the hopper script fails would be unreal. Basically someone breaks a filter and it stop working

Yes, alerts every minute by email to multiple email addresses when latency gaps pass a threshold for current logged-in agents. Can also be segregated by User Group, and there is an end-of-day option with all gaps for the day.

We've been working on this for a big client with internal network issues for the last couple of weeks, and it has been instrumental in their IT group finding out where their network issues are in real-time.

[mflorell]

Yes, the agent network monitoring emails project was completed and committed 6 months ago, you can see the details here:
https://www.vicidial.org/docs/AGENT_SCREEN_LOGGING.txt

It doesn't involve the hopper at all, but yes it would be useful to have an email alert when the hopper script fails, no clients have ever asked us to add that as a feature to this point though.

[carpenox]

damn thats pretty useful for other things as well, great addition matt