Google Chrome and Javascript Throttling

[mflorell]

For Google Chrome users, the newest version(57) starts throttling Javascript in background tabs, This can cause issues for some agents on some VICIdial configurations.

Here are instructions for how you can disable the throttling:


Users may opt out of the extra tab throttling right now by loading Chrome with the --disable-background-timer-throttling flag.

This is done in the following way on Windows machines:

- Right-click on the Chrome icon in the taskbar.
- Right-click on Chrome in the menu that opens, and select properties from it.
- Add --disable-background-timer-throttling to the end of the target field. Make sure there is a space between the path and the flag, e.g.
"C:\Users\Martin\AppData\Local\Google\Chrome SxS\Application\chrome.exe" --disable-background-timer-throttling

Google created the flag for "use cases like running test suites and other sanctioned heavy computations", but it is available to all users of the browser.


Source:
https://www.ghacks.net/2017/03/15/chrom ... bs-begins/

[mattyou1985]

dam good job i use firefox thanks for the heads up

[thephaseusa]

This isn’t a possible reason for agent browsers locking up is it (pause button turns gray, hang up button gray, cant press anything, requiring a re-login)?

John M

[mflorell]

I have not seen it cause that. Usually that is caused by something else, and looking at the browsers console for errors and the Agent Debug Log can help identify the problem.

[thephaseusa]

Thank you Matt. I use fairly old boxes for my work stations, some have windows xp on them some have centos 6.9. Vicidial works perfectly on about half of them using Firefox. But some of them have had issues with Firefox locking up, requiring a re-login and at times a reboot on the windows boxes. After seeing this thread and your response I tried changing out the browser to see if that would remedy the problem. I installed google chrome on the windows boxes and opera on the centos boxes. So far so good; yesterday these boxes went all day without locking up.

[areon]

--disable-background-timer-throttling flag was removed from Chrome version 78.0.3904.70.

Do you have any idea how to prevent throttling in background tabs?

Thank you.

[ccabrera]

--disable-background-timer-throttling flag was removed from Chrome version 78.0.3904.70.

Do you have any idea how to prevent throttling in background tabs?

Thank you.

I'm very interested in knowing alternatives to this. From a week ago I got sudden spike from several clients which end up with Vicidial disconnecting their agents due to "lagged" issues, but there aren't any network problems, its just they are switching to another tabs for CRM work. The Chrome update timing matches perfectly with these new reports, and switching to another browser (like Brave), seems to me as just a matter of time before their Chromium core reaches to this version 78.

So, any ideas would be appreciated too.

Regards,

[williamconley]

Possibly related:

https://www.theverge.com/2019/11/15/209 ... -it-admins

[IgorG]

We have the same issue, Chrome forced throttling in all cases. That cause Vicidial agents goes to DEAD status frequently. Is there any recommended browsers of tips that would make vicidial agent interface usage stable?

Is there any roadmap to use websockets in agent interface operation?

[williamconley]

We have the same issue, Chrome forced throttling in all cases. That cause Vicidial agents goes to DEAD status frequently. Is there any recommended browsers of tips that would make vicidial agent interface usage stable?

Is there any roadmap to use websockets in agent interface operation?

Websockets has a published API. Vicidial code is open source. The Roadmaps are there. You have a lot of work ahead of you to avoid just switching to a browser that doesn't throttle background processes ... but in theory the article quoted should provide guidance to how to unbreak chrome, or at least a starting point to begin the search for how to revert this feature. We just recommend Firefox instead.

[bbakirtas]

a little suggestion
im using phpdesktop (php 5.5) for agents
https://github.com/cztomczak/phpdesktop

"Introduction
PHP Desktop is an open source project founded by Czarek Tomczak in 2012 to provide a way for developing native desktop GUI applications using web technologies such as PHP, HTML5, JavaScript and SQLite. Think of it as Electron for PHP. It is a convienient tool for converting PHP web apps and PHP CLI tools to desktop applications with little effort. The development workflow you are used to while creating web applications remains the same, there is no new framework / API to learn. The process of turning an existing website into a desktop application is basically a matter of copying it to the "phpdesktop/www/" directory."

[williamconley]

a little suggestion
im using phpdesktop (php 5.5) for agents
https://github.com/cztomczak/phpdesktop

"Introduction
PHP Desktop is an open source project founded by Czarek Tomczak in 2012 to provide a way for developing native desktop GUI applications using web technologies such as PHP, HTML5, JavaScript and SQLite. Think of it as Electron for PHP. It is a convienient tool for converting PHP web apps and PHP CLI tools to desktop applications with little effort. The development workflow you are used to while creating web applications remains the same, there is no new framework / API to learn. The process of turning an existing website into a desktop application is basically a matter of copying it to the "phpdesktop/www/" directory."

OK, I'll bite ... Why?

[bbakirtas]

i don't want my agents surfing internet
and firefox eating too much cpu & ram

[williamconley]

Set up your own DHCP server. Set managers to use a different DNS server than the agents. Set the DNS server for the agents to ONLY pass DNS results for business sites you actually want the agents to have access to. Nothing to install on agent workstations.

Or get an ISA server. Or require agents to use a web proxy And otherwise turn OFF internet inside the office (only the proxy server would have internet). Both offer full control over surfing habits.

[mflorell]

Since IgorG asked, and we've recent written one, here is a link to the document on how we get WebSockets to work with the VICIdial agent screen, from the VICIdial side of things:
http://vicidial.org/docs/WEBSOCKETS_SUPPORT.txt

[mariusmarais]

I'm tangentially looking at this again, and if I'm reading this part of the original Google announcement correctly, it looks like using ViciPhone might work around this limitation?

I've just come across this, so haven't tested, but by my reasoning being on a call via ViciPhone should qualify as "playing audible audio", which should exempt the tab from (some) throttling? A few seconds after audio stops the page is throttled, but should come back as soon as audio starts playing again.

Can anyone experiencing this problem test and report back?

[mflorell]

Well, that is the problem with Google and the chrome dev team, they can change their minds with little to no warning and cause a lot of pain to those who had followed their earlier guidance. We had a client several years ago that built their platform using one of Chrome's plugins(which was even developed by Google directly) then they end-of-life'd it with less than a year notice causing our client to scramble to figure out a viable replacement for that plugin.

What you are saying appears to be true in our testing, but Google could change that behavior in the next build, so who knows what the future may hold.

[mjohn425]

What you are saying appears to be true in our testing, but Google could change that behavior in the next build, so who knows what the future may hold.

It appears to be that time. We have recently been getting many users getting lagged during calls on chrome. In the past we have disabled the chrome flag for intensive background timer throttling but this doesn't seem to be cutting the cake at the moment.

I have just seen this update from chrome which I suspect may be the culprit.

https://developer.chrome.com/blog/timer ... chrome-88/

Does anyone have a better browser recommendations for Vicidial?

[carpenox]

mjohn,

I am going to guess you do not use WebRTC phones then?

[mjohn425]

mjohn,

I am going to guess you do not use WebRTC phones then?

Not at the moment, about to make a switch but we are still a couple of weeks away from that as we have a few big projects nearing completion. More so it's just annoying dealing with the chrome devs making changes on a whim.

[carpenox]

ok so your falling under the intense throttling catagory i think then

[mflorell]

We've just added some new features to mitigate this Javascript Throttling issue in Chrome.

The new System Settings for "Agent Hidden Browser Sound" can be set to play a quiet soundfile on the agent web browser every 20 seconds that the agent screen is hidden from view, which will deactivate the Chrome browser Javascript Throttling.

If you would like to try this, upgrade to svn/trunk revision 3390 and configure "Agent Hidden Browser Sound Seconds" to '20', then log in as an agent and move to a different browser tab to test.

[martinch]

Hey guys,

Just in the process of switching agents over to Chrome myself. I've had success keeping a Chrome tab working in the background by using a custom Chrome Extension and setting the persistent flag to true. This may work well for your setups (small operation with say 10 agents) but in my case, we have many hundreds of agents and installing a custom Chrome Extension would be a logistical challenge and something I have no experience with automating.

It's is a kinda invasive and clunky solution but it seems to work. I'm currently trying to figure out how it can be done at the code level without modifying the ViCiDial base too much. I can post code sample code if anybody would like to see some but it's basically a skeleton Chrome Extension and add persistent flag to the manifest and URL match to
match vicidial.php.

Hope this helps someone out,
Martin.

EDIT I just read your comment Matt and this sounds like a much cleaner solution! Will try to test at some point. Thanks again!

SECOND EDIT this may not be to everyone's taste but I will offer these things to overcome without modifying the system at all but rather the agent actions themselves;

- If using Windows or Linux with DE that supports it, snap ViCiDial to one side of the screen and other webpage to the other side. This will keep the tab visible and active on screen and still allow you to use both. This will work great if you have a large display but not so much if you're on a low resolution like me (1024 x 768...RIP)

- You can have ViCiDial in the background and web page in the foreground but in windowed mode. This should also keep JS running at full speed too. Not ideal especially if the other webpage is displaying full screen content.

- You can flick back and forth between ViCiDial and the other page. Tedious I know and requires a lot of patience and discipline...making sure you flick back and forth and puts agents under a lot of pressure given that they are on a call as well! But only putting that here for completeness. You could use xdotool on Debian or AHK on Windows to automate this I guess but the agent must be aware of it.

I like Matt's solution but not to put a downer on things but, it's only a matter of time before Chrome devs patch his fix out. Let's ride the wave anyway though!

[mflorell]

Thanks for the detailed post martinch!

When I read the Google Chrome Dev post that mjoch245 posted earlier this month that specifically mentioned audio playing within 30 seconds on the hidden web page as a way to prevent Intensive Javascript Throttling, I knew that would be the best quick option for a fix for VICIdial since we had added the campaign and in-group "Browser Alert" features less than a year ago, which would mean we had the framework to play browser audio already in our agent code, and it would be the only non-major-change option that would not require anything to be done on the agent side. It also has the added bonus that we are now logging agent screen visibility(coming to an agent report near you soon!) since the new Hidden Alert feature is only active if the agent screen is hidden from view.

As for the Goggle Chrome devs moving the goal posts again in the future, the new features are configurable, so anyone could easily increase the volume or decrease the interval with simple web admin config changes, so hopefully that will last us for a while without having to confront this issue again with Chrome.

[mjohn425]

@martinch

Definitely please post your code here regarding the chrome extension if you are up to it (a github would probably be best), would love to do some testing. We have multiple commercial call centres but we also have our own call centre for sales based activities so we can do testing and improve for the rest of our customers without it being too interrupting to normal flow. I too am concerned regarding the goalposts, my feeling is that the vibe of the initial article I posted is that the dev who posted it is full of "well here's a better way to do it and you should be doing it this way otherwise we will make it harder in the name of performance" without realising that there a whole bunch of open source and self funded projects which can't implement these changes overnight without the backing of AlphabetTM.

One thing which I think I initially misattributed to intensive throttling when I made my first post is that I actually think that the chrome flags (chrome://flags) disable intensive throttling does work. When I view agent screen debugs for these, I can see clear 1 minute gaps in the log which disappear when I disable that flag. (Note there is a semi okay way to automate this through a batch script, it's a damn pain though). Though I still seem to get occasional lagged across the agents, their only crime, moving to a separate tab. I'm not certain whether this falls under the first category listed in the article or it is attributed to some other reason but it's definitely strange. Firefox just about halves/quarters this issue. I'm looking for a solution to avoid having to tell our customers "sorry firefox only" because that would annoy me, let alone clients. In transparency, we are running vici in an iframe as described in http://vicidial.org/docs/CRM_EXAMPLE_SKIN.txt for our customers, we also hit the api/db with about twice as many hits as usual (to keep our app responsive to state without modification to vicidial base code/cross origin issues), the DB/Web should be more than capable of handling the load but I'm concerned that the delay is coming from the client side especially since Firefox seems to perform about twice as well as chrome. Obviously that isn't in the scope of the forum to figure out but I feel it's definitely linked.

Looking at moving to WebRTC shortly through a custom SIP phone and I see that being a bit more of a permanent solution than playing sounds as it would be really difficult to limit timers while RTC is in use.

@mflorell (or others)

For my information, what would be an expected number of lagged agent records a day for say 100 agents (should it be zero, 5 -10?)

[alo]

We only get lagged agent with internet issues. otherwise 0 lagged agent records if every agent is working as they should.

[martinch]

@martinch

Definitely please post your code here regarding the chrome extension if you are up to it (a github would probably be best), would love to do some testing. We have multiple commercial call centres but we also have our own call centre for sales based activities so we can do testing and improve for the rest of our customers without it being too interrupting to normal flow. I too am concerned regarding the goalposts, my feeling is that the vibe of the initial article I posted is that the dev who posted it is full of "well here's a better way to do it and you should be doing it this way otherwise we will make it harder in the name of performance" without realising that there a whole bunch of open source and self funded projects which can't implement these changes overnight without the backing of AlphabetTM.

One thing which I think I initially misattributed to intensive throttling when I made my first post is that I actually think that the chrome flags (chrome://flags) disable intensive throttling does work. When I view agent screen debugs for these, I can see clear 1 minute gaps in the log which disappear when I disable that flag. (Note there is a semi okay way to automate this through a batch script, it's a damn pain though). Though I still seem to get occasional lagged across the agents, their only crime, moving to a separate tab. I'm not certain whether this falls under the first category listed in the article or it is attributed to some other reason but it's definitely strange. Firefox just about halves/quarters this issue. I'm looking for a solution to avoid having to tell our customers "sorry firefox only" because that would annoy me, let alone clients. In transparency, we are running vici in an iframe as described in http://vicidial.org/docs/CRM_EXAMPLE_SKIN.txt for our customers, we also hit the api/db with about twice as many hits as usual (to keep our app responsive to state without modification to vicidial base code/cross origin issues), the DB/Web should be more than capable of handling the load but I'm concerned that the delay is coming from the client side especially since Firefox seems to perform about twice as well as chrome. Obviously that isn't in the scope of the forum to figure out but I feel it's definitely linked.

Looking at moving to WebRTC shortly through a custom SIP phone and I see that being a bit more of a permanent solution than playing sounds as it would be really difficult to limit timers while RTC is in use.

@mflorell (or others)

For my information, what would be an expected number of lagged agent records a day for say 100 agents (should it be zero, 5 -10?)

My apologies. The Chrome Extension discovery may have been a bit of a red herring so apologies for misleading anyone there. I did notice my code didn't freeze but there might have been other factors at play.

I have taken a different approach by kinda calling

Code: Select all

all_refresh

and inside of a web worker, which apparently runs the JavaScript in a different thread in the browser. Here's a mockup I did as I'm not able to test on ViCiDial itself since my home laptop doesn't have a working setup;

Code: Select all

var counter = 1;
var vicidial = 30;
// Initialise refresh interval
var refresh_interval = 1000;

// Call web worker logic
chromeKeepAlive();

/*
 * Function name: chromeKeepAlive
 * Function description: this function will create a web worker that will keep the ViCiDial heartbeat going in a different execution thread
 * even if the Chrome tab becomes inactive
 * Date: 21/03/21
 */
function chromeKeepAlive() {
    // Create new worker object
    function createWorker(main) {
        // New blob
        var blob = new Blob(
            ["(" + main.toString() + ")(self)"],
            { type: "text/javascript" }
        );

        // Return object
        return new Worker(window.URL.createObjectURL(blob));
    }

    // Create worker
    var worker = createWorker(function(self) {
        // Initialise refresh interval
        var refresh_interval = 1000;
        // Intialise counter
        var counter = 0;

        setInterval(function() {
            // Increment counter
            counter++;

            self.postMessage(counter);
        }, refresh_interval);
    });

    // Listen for messages
    worker.onmessage = function(e) {
        // Main ViCiDial heartbeat
        setTimeout(function() {
            all_refresh(e.data);
        }, 1000);
    }
}

// All refresh function call
function all_refresh(counter) {
    console.log(counter);
    console.log(vicidial);
}

It would be great if someone could test this on ViCiDial itself as it could be a nice fix. It may involve more heavy lifting of code though but it's a start.

[ChrisDague]

After days of pulling pcap files from the ViCiweb server and some end users I stumbled on this topic.
We added the --disable_background_timer_throttling and in chrome://flags we disabled Throttle javascript timers for hidden tabs and no more issues.

[roger.milligan]

Just to add to the above. This "Intensive throttling of Javascript timer wake ups" in Google Chrome is active from version M88: Enabled by default - released Jan 2021. One customer's LAGGED agents percentage suddenly shot up this year - 25% of vicidial_agent_log records were LAGGED! Glad to find the solution after 24hrs of deep troubleshooting.

See: https://www.chromestatus.com/feature/4718288976216064
Disable it here: chrome://flags/#intensive-wake-up-throttling

It is interesting that to start with the conf_exten_check calls happen every 1 second (as expected), but as soon as you switch to another browser tab in Chrome, it drops to every 2 seconds. Then, once the tab has not been accessed for 6 minutes, a delay of 1 minute kicks in and the agent gets LAGGED. Disabling the throttling did not change the 2 second refresh, but made the 6 minute LAGGED thing stop. The lagging problem does not happen in Firefox.

For some reason, whenever I tested this, the delayed calls ALWAYS happened at 22 seconds past the minute! Not sure why. Perhaps an internal timer in Chrome starts when I opened Chrome browser at 22 seconds past the minute??

Code: Select all

10.171.60.170 - - [24/Mar/2021:13:25:22 +0200] "POST /agc/conf_exten_check.php?usr=6969&Dte=2021-03-24%2013:24:22 HTTP/1.1" 200 342 T:19479 KA:0+ 5192 "http://10.171.80.10/agc/vicidial3.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
10.171.60.170 - - [24/Mar/2021:13:26:22 +0200] "POST /agc/conf_exten_check.php?usr=6969&Dte=2021-03-24%2013:25:22 HTTP/1.1" 200 308 T:17695 KA:0+ 4514 "http://10.171.80.10/agc/vicidial3.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
10.171.60.170 - - [24/Mar/2021:13:27:22 +0200] "POST /agc/conf_exten_check.php?usr=6969&Dte=2021-03-24%2013:26:22 HTTP/1.1" 200 308 T:17482 KA:0+ 6415 "http://10.171.80.10/agc/vicidial3.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"

[mflorell]

For some reason, whenever I tested this, the delayed calls ALWAYS happened at 22 seconds past the minute! Not sure why. Perhaps an internal timer in Chrome starts when I opened Chrome browser at 22 seconds past the minute??

For me it was 52 seconds after the minute, every time.

As an added note on this subject, I discovered a bug yesterday related to the new VICIdial feature we added to mitigate this issue without having to use Chrome flags: Prior to VICIdial svn/trunk revision 3399, if the System Setting "Agent Browser Call Alerts" was not enabled, then the "Agent Hidden Browser Sound" features would not work properly. This has been fixed in svn/trunk revision 3399.

[mflorell]

We've just added a document to summarize this issue, and the current methods of disabling it,
http://vicidial.org/docs/WEB_BROWSER_JA ... TTLING.txt

[martinch]

We've just added a document to summarize this issue, and the current methods of disabling it,
http://vicidial.org/docs/WEB_BROWSER_JA ... TTLING.txt

Thanks Matt. The document looks great.

For those not able to immediately upgrade to the bleeding edge version of ViCiDial due to company policies, resource issues or anything else, I have written a Chrome Extension to tide you over until you can upgrade. It uses a simlar method as Matt is using for "Agent Hidden Sounds".

It's not published on the Chrome store (don't see any point) so you'll have to load it as an unpacked extension. Here are some instructions on how to use it;

Code: Select all

1) Download the Extension as a zip file (or clone the repo) from [url]https://github.com/TheBlode/ViCiDial-Keep-Alive[/url].

2) Extract contents to a folder.

3) Go to URL chrome://extensions in Chrome or navigate to the Extensions menu through the interface.

4) Enable "Developer Mode".

5) Click "Load Unpacked".

6) Select the ViCiDial Keep Alive Chrome Extension folder (ViCiDial-Keep-Alive-main\ViCiDial_Keep_Alive).


And you're done. Make sure you copy the `quiet.mp3` file from the Extension folder to your web server in the agc/sounds folder to make the sound play.

If you find any issues, of course, you can post on here and I'll help out.

NB: I would ALWAYS recommend parsing the code first to check it for any obvious security issues...but what you'll find is a barebones Chrome Extension with mediocre code...!

Hope this helps someone out! All the best guys!

[fperdomo]

I created a simple and easy to implement solution for this issue you can follow the guide on this link:
https://gist.github.com/masterfermin02/ ... 1c2642b31e

[mflorell]

Wow fperdomo!

I had no idea that just using a simple EventSource Server-Sent Event to trigger a timed event would completely avoid Chrome JavaScript throttling.

It appears to work properly in all web browsers I've tried it in as far as being logged in and keeping the connection active, although it doesn't work well with any of the auto-logout features of VICIdial(Ready-Max, Pause-Max, API-Logout, etc...), generating errors every second because the connection isn't stopped after the logout is triggered, so we'll have to work on that before I can add this to the codebase.

Thanks!

[fperdomo]

Thank you mflorell my pleasure!

If you need any help fixing the issue with the logout just let me know and I would be more than happy to help!

[mflorell]

Thanks, I think I've figured it out, just needed to put a couple of conditional lines in there to close the EventSource connection if the Logout() function has been run. Seems to be working well in my tests so far, I just want to run some more tests before I commit it to svn/trunk.

Thanks again!

[mflorell]

I have added a new System Settings option and committed it to svn/trunk revision 3407:

Agent Screen Timer -

This will select the method of JavaScript timer to be used by the Agent Screen. The original method is setTimeout, which maintains the timer directly in JavaScript. An alternative method is EventSource, which uses a script on the server to trigger the timer. You may want to use EventSource if your agents are using a Chromium-based web browser that has limitations on how native JavaScript timers work, although using this method may possibly cause issues if you have a poor network connection to the server or are on an older network or proxy that does not handle streaming web services well. Default is setTimeout.


I've also updated the WEB_BROWSER_JAVASCRIPT_THROTTLING.txt document with new details on this feature,
http://vicidial.org/docs/WEB_BROWSER_JA ... TTLING.txt

[carpenox]

great job @fermin / @matt - great addition that will fix so many issues around the world these past few weeks....

[martinch]

Nice job @fperdomo! And thanks Matt for including that solution in ViCiDial.

So as it stands, we have the sound hack, this new PHP server side solution and well, my previous suggestion is essentially not a hack per say, it's a replacement for the standalone setInterval by wrapping all_refresh in a web worker, which executes in a seperate thread in the browser. I tested it today on a ViCiDial instance and worked pretty well.

Great work everyone. I'm going to try out fperdomo's fix. Just looking at it, it's looks very clean.

Thanks again!

[martinch]

Upgraded Chrome today. Now sitting on Version 89.0.4389.114 (Official Build) (64-bit). Anybody noticed this is back?

Code: Select all

chrome://flags/#intensive-wake-up-throttling

Throttle Javascript timers in background.
When enabled, wake ups from DOM Timers are limited to 1 per minute in a page that has been hidden for 5 minutes. For additional details, see https://www.chromestatus.com/feature/4718288976216064. – Mac, Windows, Linux, Chrome OS, Android

#intensive-wake-up-throttling


The option has been put back into Chrome. When this option is set to "Disabled", there are no more throttling issues. Handy if you're on this version.