System Delay, Please try again

[fibres]

Hi guys

I am getting this error on a manual dialing campaign.

If I either use hotkeys or disposition and then click dial next number too quickly I get this error.

System Delay, please try again
code: 1498079-1

I have seen a couple of posts about this seeming to suggest that it is something to do with database load. However I have tried this with no users on our system and getting the same problem.

Any ideas?

Regards

[mflorell]

It's either database delay or networking delay that can cause this.

[fibres]

Hi Matt

Thanks for replying, how do I track down exactly where the problem is?

I have tried this from a windows pc on the same subnet as the dialer server connected on a gigabit network.

I have no other agents on system and am just logged into this one manual campaign, my mtop is below


load average: 0.02, 0.02, 0.01 mysqld 5.1.46-log up 8 day(s), 8:58 hrs
21 threads: 1 running, 12 cached. Queries/slow: 5/0 Cache Hit: 99.50%
Opened tables: 0 RRN: 255 TLW: 9.4M SFJ: 0 SMP: 0 QPS: 0


Where do you suggest I go from here?

Regards

[mflorell]

Do you have slow query logging enabled?

What are the specs for your DB server(CPU/RAM/Drives[speed/raid])?

[fibres]

Hi Matt

Is the slow query logging enabled by default on vicibox?

If not how do I enable it and monitor it?

The server specs are as follows

 Server Model: Dell PowerEdge R210 Carbon-Neutral Server
 Processor: Quad Core Intel® Xeon® X3450 2.66GHz
 Memory: 8GB ECC DDR3 Memory
 Hard disk 1: 250GB SATA 7,200RPM Drive
 Hard disk 2: 250GB SATA 7,200RPM Drive
 RAID Controller: H200 Hardware Raid - Raid 1
 Remote Management Card: DRAC6 Enterprise Server Management Card


Regards

[fibres]

Well I have looked at my.cnf and it seems that slow query logging is enabled.

I have changed it to write to a file in /var/log/mysql/slow.log

I see no file therem does that mean there have been no slow queries?

Any ideas?

Regards

[mflorell]

What kind of network switch are you using?

What kind of workstation?

Something is slowing down your database or data traffic in some way. It could be a hardware configuration issue, it isn't always easy to tell, but we have seen this issue before and there is always something that causes a bottleneck somewhere.

[fibres]

Hi Matt

Thanks for the reply.

All our servers are connected into a Layer 3 Cisco switch.

The workstation is a winxp Virtual Machine running on a similar server to the dialer server.

We have the same issues from our usual agent workstations running on dell GX280 workstations.

These are connected to the internet and in turn our hosted servers via a 2mb SDSL line.

This happens every time we try and use hotkeys. It is not intermittent.

Regards

[mflorell]

I am able to replicate this with hotkeys only over a delayed network connection(though not on a local HP Procurve network), this is probably something that should be fixed in the code to put another second or two between hotkey press and dialing of next number. If you can, would you please post this to the Issue Tracker and then link to that issue here?

As for your network, you should check and see if Cisco's deep packet inspection is enabled, or if any kind of filtering or packet manipulation is going on.

[fibres]

Hi Matt

I have posted the issue to the bug tracker.
http://www.eflo.net/VICIDIALmantis/view.php?id=563

As for the network switches, I am pretty sure they are not running any form of SPI, however these are managed by our server host so not 100%

I will try installing a local copy of vicidial at our offices and see if I can reproduce.

Regards

[mflorell]

I believe I fixed this issue in svn/trunk, please upgrade and let me know if it fixes the problem for you.

[fibres]

Hi Matt.

I upgraded last night and made a few test calls and it worked 10 times in a row.

What did you change? Was it just the delay between dispo and dial next?

Thanks for fixing it whatever you did.

Regards

[mflorell]

I added one second delay between dispo and dial-next, which ended up moving a bunch of things around in the code, but it appears to fix the issue in all of my testing as well as yours so looks like it worked

[omarrodriguezt]

Mat, hello.
I'm having same issue.
Can you please tell me what file you modified or were can I found the patch?

[omarrodriguezt]

I found it.
I downloaded the project from http://sourceforge.net/projects/astguiclient/
then unzip it and replace the file vicidial.php with the latest one.
Thank you anyway.

[fibres]

Hi guys

Ok this problem has reared it's ugly head again.

We recently upgraded to latest svn.
VERSION: 2.6-393a
BUILD: 130124-1721

All of a sudden on our manual campaign where we use hotkeys. We can ring 3 leads and fine, then we get system delay.

Same happens when we normal hangup without hotkeys. We can hangup about 3 times then we get system delay and have to retry 3-10 times to make next call.

Regards

[mflorell]

I just tested this by going through 10 manual dial calls all terminated by a HotKey press in a row, no problems.

This can also be caused by network issues.

[williamconley]

perhaps there is a driver conflict? slow software RAID?

in other words: maybe it has something to do with the equipment fibres is using ...?

[fibres]

We are not seeing issues in any other areas on our dialer.

Everything else seems to be working fine. Auto calls are coming through ok.

Regards

[williamconley]

Problems with the virtual workstations or their browsers? What browser are you using? Have you rebooted your workstations recently?

[fibres]

The workstations are same as we have always used.

I have tried rebooting them.

It is weird, It is not just hotkeys its all manual calls. I have tried making manual calls using the normal hangup button and I get this issue.

I can keep trying clicking dialing next number and it keeps coming up with the error. It takes about 10 seconds to clear or more.

Eventually I can click next call and it will dial.

Im not sure where to look next to find this issue.

Im not sure that a 10 sec issue can be caused by network delay.

Regards

[williamconley]

Perhaps you should define "system delay" a bit more precisely. Are you sure you're not just running out of leads? LOL

[fibres]

Hi william.

Lol, Im not quite that green.

There are plenty of leads in the hopper and dialable leads.

I am attaching a photo of the error im getting.

I am seeing this irrespective of if I use hotkeys or normal manual dialing.

http://www.tele-magic.co.uk/vicidialsystemdelayerror.png

Regards

[williamconley]

Wow. A new error I have never seen. Perhaps you should search the source code for the agent file (perhaps that entire folder) to see what was checked before that was "tossed".

[fibres]

Ok I have searched that and it is only referenced in vicidial.php

This is the section that is throwing up the error.

// ################################################################################
// Send the Manual Dial Next Number request
function ManualDialNext(mdnCBid,mdnBDleadid,mdnDiaLCodE,mdnPhonENumbeR,mdnStagE,mdVendorid,mdgroupalias,mdtype)
{
if (waiting_on_dispo > 0)
{
alert_box("System Delay, Please try again - Manual<BR><font size=1>code:" + agent_log_id + " - " + waiting_on_dispo + "</font>");
}
else


I added the "- Manual" bit just to confirm it was that line. So it is being caused by waiting_on_dispo not being reset back to 0 which leads me to think that it is an issue with the dispo not being updated quick enough.

Just need to establish why now.

Now I see in the changelog at the top the following

# 110528-1033 - Added waiting_on_dispo manual dial check

How do I see when that change was made? Also is there a link between that change and a bug report in mantis?

Any ideas?

Regards

[williamconley]

It says if waiting_on_dispo ... so find out how this gets set to a number above zero (ie: where is this set?)

[fibres]

It seems to be being set here from line 10262 in vicidial.php



// ################################################################################
// Update vicidial_list lead record with disposition selection
function DispoSelect_submit()
{
if (VDCL_group_id.length > 1)
{var group = VDCL_group_id;}
else
{var group = campaign;}
leaving_threeway=0;
blind_transfer=0;
CheckDEADcallON=0;
currently_in_email=0;
customer_3way_hangup_counter=0;
customer_3way_hangup_counter_trigger=0;
waiting_on_dispo=1;
document.getElementById("callchannel").innerHTML = '';
document.vicidial_form.callserverip.value = '';
document.vicidial_form.xferchannel.value = '';



It then appears that it should be set back here

var check_dispo = null;
check_dispo = xmlhttp.responseText;
var check_DS_array=check_dispo.split("\n");
if (check_DS_array[1] == 'Next agent_log_id:')
{
agent_log_id = check_DS_array[2];
}
}
waiting_on_dispo=0;
}
}
delete xmlhttp;
}


thisbeing line 10343

However all the xhtml and jscript code inbetween makes no sense to me.

My programming skills are not high enough to make sense of the rest.

Regards

[fibres]

Ok interesting.

I have commented out the line

if (waiting_on_dispo > 0)
{
alert_box("System Delay, Please try again - Manual<BR><font size=1>code:" + agent_log_id + " - " + waiting_on_dispo + "</font>");
}
else

so it no longer does the check before making the manual call.

I can now make calls without the system poping up that message. However now every 6-8 calls it freezes for about 20 secs saying "waiting for ring" and "no live call" in top right.

However I am watching the asterisk log and the call is answered immediatly. there is no further output in asterisk log.

The agent screen sits there for about 20 secs and then shows the call as active.

The call is immediatly connected to the agent just the agent interface does not update.

I cannot see why this is happeneing. There is no load on the server as I have just a single test agent logged on.

The server appears to work fine in autodial mode which all agents use execpt for one who is on a seperate manaul dial campaign who is having the issues.

Regards

[williamconley]

is it a virtual server? LOL

[fibres]

Nooooooo. Lol

Don't use vps for vicidial servers.

It is a hp bl460c G5 blade with 2 x Dual Quad core 3.0ghz Xeon processors.
16gb ram

2 x 146gb sas drives in raid 1 array

Regards

[williamconley]

You may want to have a look at your sql system and see if something is causing a blockage of some sort. Is this the only campaign running?

[mflorell]

This is usually either a network issue or a database issue. You should also make sure you are running a recent svn/trunk version.

[fibres]

Hi guys.

I looked into different tools for monitoring mysql last night. I not had a chance to try them yet.

There were no other campaigns on the system. The load average was 0.00

I cannot see how it is network issues, not when it is taking 20 secs to clear. That is hell of contention.

Matt could you offer a basic insight I to what is going on at the above point in the script? From my basic understanding I suspect that bit if code is run when an agent selects a dispo. I also guess that it continues to run that procedure even after the agent has gone back to main page. It sets the waiting on dispo to 1 runs a jdcropt or XHTML external procedure which does the database updates to set the disposition and then when they come back to say they have finished currectly it resets the waiting on dispo back to 0. Therefore until they have finished running the agent cannot make a further call?

Assuming I'm fairly right there, would nothin JavaScript and XHTML code launched be run by the agents web browser or by the server?

Regards

[mflorell]

Yes, the "A" in AJAX is asynchronous, meaning it can still be going on after it has been launched, and it is a non-blocking process. But since the logging will get messed up if a new manual call is allowed to be placed before the Dispo process completes, I had to add a block if that happens. That's what the system delay message is for. We have many remote clients on our hosted platform that use manual dialing with hotkeys, and the system delay is not an issue for them unless they have network issues, which is why I mentioned that as a likely cause.

[Framercy]

Did you check the time synchronisation in your network?
Are using servers and clients the same time source?
Is the time synchronised successfully on every system start - and after that in hourly intervals or so?

Best regards
Fram

[ed123]

Hi,

I have the same issue "System Delay.Please try again" only in manual dialing. For autodial it works fine. We encountered this issue in goautodial 3.0.
We don't have issue with networks. For server goautodial 2.1 we don't face this kind of issue.Please advice.


VERSION: 2.7-401a
BUILD: 130508-2256
© 2013 ViciDial Group
Server1: Asterisk/Web
Server2: DB only

[SPAMSAM]

I've seen a couple of times how mysql optimizes to the point where it has consumed most of the system resources.

Running the following has worked in small to medium call centers:

Code: Select all

sync && echo 2 > /proc/sys/vm/drop_caches

[gjorge]

good morning

Estimados, in the file vicidial.php can I add a function of delay in the execution to avoid that this alert appears?

for example the sleep function

it would be by conditional if, that takes as parameter the waiting_on_dispo and executes the sleep function for 5 seconds if it is greater than 0.

waiting for your answers!!! thanks