GO Autodial 3 automatic routines

[cbsys]

Does anyone know if there are some scripts or backup routine which Go Autodial runs on a weekly basis on a sunday?

Every week our client's dialler becomes unsuable as the CPU usage of mysqld shoots up to 300%
(quad core xeon hence 300%)
However I cant see anything major running in cron or elsewhere which could cause this

It seems that the dialler must be running a cronjob or other weekly routine such as a backup which is sending the mysql usage sky high
Its a large database, but Im not sure why it should always happen at the same time every single week!

[williamconley]

htop will show the process using the cpu.

crontab -e should show all processes running as root unless there are some in /etc/crontab

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

[gardo]

Only the scripts running in crontab are the "regular" scripts running. Have you checked with "top -c" what commands are being run when your mysld process is shooting up to 300%?

[cbsys]

Here's a copy of the htop screen taken yesterday.

The system is a HP ML110 g7 with a quad-core Xeon & 12Gb Ram running goautodial version 3 (vicidial 2.7rc1)
On a sunday the system is idle, there are no agents and no calls in or out.

This problem only happens on a sunday, but there is nothing in crontab or /var/spool/cron/root scheduled to run on a sunday at all!

[gardo]

Whoah! Just look at that load average. Way too high. Have you checked via mtop what mysql processes are running?

[cbsys]

Exactly! Thats why I was asking if anything else runs automatically on a sunday which I dont know about!

The system is fine on monday-saturday, its only on sundays that this occurs!
In fact the system is currently dialling and has lots of agents on it today and the load average is only 0.69, 2.13, 2.75

I didnt have mtop installed on the system, but its now on there and I'll run it next sunday when the problem occurs again!

[cbsys]

CPU and load average sky high again this sunday

Heres the results of mtop:

load average: 648.55, 645.13, 636.10 mysqld 5.0.95 up 6 day(s), 20:38 hrs
688 threads: 664 running, 12 cached. Queries/slow: 68/0 Cache Hit: 99.53%
Opened tables: 0 RRN: 8.3K TLW: 19.7M SFJ: 0 SMP: 0 QPS: 0

ID USER HOST DB TIME COMMAND STATE INFO
2354190 cron 192168.1.204:598 asterisk 20325 Query Locked UPDATE vicidial_list set status='AFTHRS' where lead_id = '16565924'
2354217 cron 192168.1.204:598 asterisk 20290 Query Locked INSERT INTO vicidial_list (entry_date,modify_date,status,user,vendor_lead_code,source_id,list_id,calle
2354229 cron 192168.1.204:598 asterisk 20244 Query Locked INSERT INTO vicidial_list (entry_date,modify_date,status,user,vendor_lead_code,source_id,list_id,calle
2354821 cron 192168.1.204:600 asterisk 19132 Query Locked INSERT INTO vicidial_list (entry_date,modify_date,status,user,vendor_lead_code,source_id,list_id,calle
2354831 cron 192168.1.204:600 asterisk 19096 Query Locked INSERT INTO vicidial_list (entry_date,modify_date,status,user,vendor_lead_code,source_id,list_id,calle
2355014 cron 192168.1.204:600 asterisk 18768 Query Locked INSERT INTO vicidial_list (entry_date,modify_date,status,user,vendor_lead_code,source_id,list_id,calle
2348740 cron localhost asterisk 18451 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348518 cron localhost asterisk 18245 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348860 cron localhost asterisk 17545 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348774 cron localhost asterisk 17475 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348705 cron localhost asterisk 17432 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349433 cron localhost asterisk 17395 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348919 cron localhost asterisk 17321 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348996 cron localhost asterisk 16751 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349366 cron localhost asterisk 16720 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348887 cron localhost asterisk 16684 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349082 cron localhost asterisk 16613 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2348810 cron localhost asterisk 16451 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349608 cron localhost asterisk 16438 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349151 cron localhost asterisk 16322 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349518 cron localhost asterisk 16093 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349332 cron localhost asterisk 16003 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349184 cron localhost asterisk 15963 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349449 cron localhost asterisk 15938 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349296 cron localhost asterisk 15572 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349217 cron localhost asterisk 15374 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349799 cron localhost asterisk 15120 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349987 cron localhost asterisk 15113 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350400 cron localhost asterisk 15065 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349593 cron localhost asterisk 14941 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349122 cron localhost asterisk 14909 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349959 cron localhost asterisk 14882 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350241 cron localhost asterisk 14798 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349558 cron localhost asterisk 14763 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349235 cron localhost asterisk 14584 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349832 cron localhost asterisk 14537 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350017 cron localhost asterisk 14495 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349686 cron localhost asterisk 14421 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350292 cron localhost asterisk 14030 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349928 cron localhost asterisk 13862 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349487 cron localhost asterisk 13844 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350439 cron localhost asterisk 13828 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350055 cron localhost asterisk 13794 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350088 cron localhost asterisk 13685 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350142 cron localhost asterisk 13666 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349747 cron localhost asterisk 13632 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349646 cron localhost asterisk 13298 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2350689 cron localhost asterisk 13161 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2349860 cron localhost asterisk 13150 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')
2353267 cron localhost asterisk 13149 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (
2350120 cron localhost asterisk 13131 Query Sorting resu SELECT ... FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA')

[williamconley]

you could start with a reboot daily
next find out why there were items in htop that were blank
then go to "show full processlist" in mysql and find the oldest running query to see if you can find out what is locking up the mysql queue

[cbsys]

A daily reboot wont solve the issue though, it will just mask the problem
I can usually resolve it by restarting mysqld and crond, which is what made me think it was an automatic routine in the first place!

I checked yesterday (saturday) and all was well
Today (sunday) we have the problem again, there are lots of these showing in my running processes (a lot more than this Ive just pasted a few)

17853 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18019 ? S 0:00 crond
18020 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18386 ? S 0:00 crond
18388 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18543 ? S 0:00 crond
18546 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18706 ? S 0:00 crond
18709 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18863 ? S 0:00 crond
18865 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19018 ? S 0:00 crond
19020 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19234 ? S 0:00 crond
19239 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19389 ? S 0:00 crond


I get quite a few of these when I "show full processlist", the queries seem to be HUGE!

| 2743693 | cron | localhost | asterisk | Query | 342 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:05:46") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:35:46") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:05:46") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:35:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:35:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:05:46") ) ) ) ) and list_id IN('3370') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2743718 | cron | localhost | asterisk | Query | 191 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:08:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:38:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:08:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:38:27") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:38:27") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:08:17") ) ) ) ) and list_id IN('3370') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2743754 | cron | localhost | asterisk | Query | 111 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:39:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:39:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:39:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:37") ) ) ) ) and list_id IN('3370') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONC
AT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2743780 | cron | localhost | asterisk | Query | 139 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:09") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:39:09") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:09") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:39:19") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:39:19") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:09") ) ) ) ) and list_id IN('3370') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2743807 | cron | localhost | asterisk | Query | 783 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:28:25") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:28:25") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:58:25") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:28:35") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:28:35") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:58:25") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:28:25") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2743840 | cron | localhost | asterisk | Query | 764 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:28:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:28:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:58:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:28:54") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:28:54") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:58:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:28:44") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2743873 | cron | localhost | asterisk | Query | 30 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:10:58") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:40:58") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:10:58") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:41:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:41:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:10:58") ) ) ) ) and list_id IN('3370') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2743897 | cron | localhost | asterisk | Query | 686 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:30:02") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:30:02") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:00:02") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:30:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:30:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:00:02") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:30:02") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2743928 | cron | localhost | asterisk | Query | 594 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:34") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:31:34") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:34") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:44") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:34") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:34") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2743954 | cron | localhost | asterisk | Query | 542 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:32
:26") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:32:26") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:02:26") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:36") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:36") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:02:26") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:32:26") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2743989 | cron | localhost | asterisk | Query | 607 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:21") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:31:21") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:21") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:31") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:31") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:21") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:21") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744019 | cron | localhost | asterisk | Query | 611 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:31:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:27") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:27") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:01:17") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:31:17") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744047 | cron | localhost | asterisk | Query | 440 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:34:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:34:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:04:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:34:18") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:34:18") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:04:08") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:34:08") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744072 | cron | localhost | asterisk | Query | 567 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:32:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:32:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:02:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:02:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:32:01") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744106 | cron | localhost | asterisk | Query | 185 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:38:23") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:38:23") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:08:23") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:38:33") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:38:33") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:08:23") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:38:23") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744136 | cron | localhost | asterisk | Query | 221 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:37:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:37:47") ) ) ) or ( (called_s
ince_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:07:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:37:57") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:37:57") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:07:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:37:47") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744166 | cron | localhost | asterisk | Query | 254 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:37:14") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:37:14") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:07:14") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:37:24") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:37:24") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:07:14") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:37:14") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744196 | cron | localhost | asterisk | Query | 51 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:40:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:40:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:10:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:40:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:40:47") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:10:37") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:40:37") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744210 | cron | localhost | asterisk | Query | 927 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:26:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:26:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:26:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:26:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:26:11") ) ) ) ) and list_id IN('4148') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2744235 | cron | localhost | asterisk | Query | 92 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:39:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:39:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:40:06") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:40:06") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:09:56") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:39:56") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744264 | cron | localhost | asterisk | Query | 16 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:41:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:41:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:11:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:41:22") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:41:22") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='AB') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:11:12") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='N') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 00:41:12") ) ) ) ) and list_id IN('3369') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 210 |
| 2744297 | cron | localhost | asterisk | Query | 747 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:29:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:29:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:29:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:29:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:29:11") ) ) ) ) and list_id IN('4148') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDRO
P')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2744317 | cron | localhost | asterisk | Query | 687 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:30:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:30:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:30:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:30:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:30:11") ) ) ) ) and list_id IN('4148') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2744351 | cron | localhost | asterisk | Query | 627 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:31:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:31:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:31:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:31:11") ) ) ) ) and list_id IN('4148') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |
| 2744383 | cron | localhost | asterisk | Query | 567 | Sorting result | SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where ( ( (called_since_last_reset IN('Y','Y1')) and (status='AA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:11") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='B') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:32:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1','Y2')) and (status='NAT') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 23:32:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='NA') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-14 01:32:01") ) ) ) or ( (called_since_last_reset IN('Y','Y1')) and (status='A') and (( (gmt_offset_now='') and (last_local_call_time < "2013-12-13 14:32:11") ) ) ) ) and list_id IN('4148') and lead_id NOT IN('0') and ((gmt_offset_now IN('99') ) ) and ( ( (status IN('DROP','XDROP')) and (last_local_call_time < CONCAT(DATE_ADD(NOW(), INTERVAL -259200 SECOND),' ',CURTIME()) ) ) or (status NOT IN('DROP','XDROP')) ) order by lead_id asc limit 1 |

[williamconley]

A daily reboot wont solve the issue though, it will just mask the problem

We may have to disagree on this point. In my opinion if you can reboot daily and the problem never arises again, it has been resolved. If that means you do not need to spend the time to dig deep and solve it, then the problem is resolved and you saved all that production time (or your employer/client saved it, depending on the point of view).

I do see where it could be annoying to have an unexplained issue such as this, but I can say quite simply that it if does not raise it's ugly head simply because you reboot daily, it can't be that important and will likely "go away" in a later release without your fixing anything. Also why we use Vicibox instead of GoAutoDial ... more often released and upgraded and more users (to get rid of strange little annoyances like this). But we do not allow that temptation to make us NOT reboot daily because they probably don't need it ... because they MAY need it and it's free. Why not reboot daily?

Oh, and here's another one for you: If your hard drive begins to fail, you'll never know until it dies in the middle of a (heavy usage) shift because disk checks only occur on reboot and only when the system is set to do so. If you do not reboot, you never disk check. And if you do not configure for disk check, you don't even when you reboot. So reboot daily, disk check monthly. You'll have a more reliable system that does not crash.

As for the multiple hopper processes and sql queries, those are symptoms of the same problem. The sql system gets "hammered" by a process and backs up other sql queries in the line. They stall and wait for that original process to complete. Since the hopper fires every minute, those will stack up as well awaiting the completion of the original task.

So find the OLDEST query still running and see if you can identify it and why it is "jammin' up the works". Been there many times. Usually turns out to be some sort of customization that was not well-written.

Happy Hunting!

[gardo]

There's a Linux utility called "smartctl" which is part of the smartmontools. Quite handy.

Here's the description from their website:

The smartmontools package contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA and SCSI harddisks. In many cases, these utilities will provide advanced warning of disk degradation and failure.

Reboot is not an option when you're running a 24/7 campaign on GOautodial/Vicidial. It's proper systems administration and maintenance that's the key to a healthy and properly functioning system.

Oh, and here's another one for you: If your hard drive begins to fail, you'll never know until it dies in the middle of a (heavy usage) shift because disk checks only occur on reboot and only when the system is set to do so. If you do not reboot, you never disk check. And if you do not configure for disk check, you don't even when you reboot. So reboot daily, disk check monthly. You'll have a more reliable system that does not crash.

[gardo]

The AST_VDhopper.pl script runs every minute. This loads the leads into the hopper. It looks like yours are overlapping which means that the previous instance did not get to finish. If you have millions of leads in the system, it can take a while for the AST_VDhopper.pl to finish.

Try running AST_VDhopper.pl in the commandline with the debug option enabled.

Code: Select all

/usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl  --debugX

I checked yesterday (saturday) and all was well
Today (sunday) we have the problem again, there are lots of these showing in my running processes (a lot more than this Ive just pasted a few)

17853 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18019 ? S 0:00 crond
18020 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18386 ? S 0:00 crond
18388 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18543 ? S 0:00 crond
18546 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18706 ? S 0:00 crond
18709 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
18863 ? S 0:00 crond
18865 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19018 ? S 0:00 crond
19020 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19234 ? S 0:00 crond
19239 ? Ss 0:00 /usr/bin/perl /usr/share/astguiclient/AST_VDhopper.pl -q
19389 ? S 0:00 crond

[cbsys]

yes I agree its something like that, there are indeed millions of leads ......
but its fine on every day we are actually calling, its only on a sunday where no calls take place that this happens!

sundays are ouside of the call times we have specified, so no leads should be loaded into the hopper anyway!

[gardo]

That's right. It shouldn't load leads at all if it's not in your call times. Have you checked the campaign settings and see if their calling hours are not set to 24 hours? And try running the leads loader script with the debug option enabled on Sunday. Let's see the output.

[cbsys]

Sunday call times are start 2400 and stop 2400 and no calls are being made - it doesnt stop the hopper script running though as its set to run in cron every minute
(I suppose I could the hopper script to run on a sunday...)

All other times are fine, 810 to 1750 and everything works well monday to saturday

I'll run "AST_VDhopper.pl --debugX" this Sunday as see what we get!

[williamconley]

There's a Linux utility called "smartctl" which is part of the smartmontools. Quite handy.

Here's the description from their website:

The smartmontools package contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology System (SMART) built into most modern ATA and SCSI harddisks. In many cases, these utilities will provide advanced warning of disk degradation and failure.

Reboot is not an option when you're running a 24/7 campaign on GOautodial/Vicidial. It's proper systems administration and maintenance that's the key to a healthy and properly functioning system.

Oh, and here's another one for you: If your hard drive begins to fail, you'll never know until it dies in the middle of a (heavy usage) shift because disk checks only occur on reboot and only when the system is set to do so. If you do not reboot, you never disk check. And if you do not configure for disk check, you don't even when you reboot. So reboot daily, disk check monthly. You'll have a more reliable system that does not crash.

To date I have *never* had any monitor utility predict a hard drive failure before it fails. The exception to that rule is if the utility in question relies on a disk check showing bad sectors that have been repaired (ie: "there are more repaired sectors today than there were yesterday, apparently something is amiss!"). But in Linux if you do not reboot, bad sectors are never checked and that warning never arrives.

Are you saying you've had that utility actually predict a drive death? Or just that it sounds cool and you're using it?

[williamconley]

yes I agree its something like that, there are indeed millions of leads ......
but its fine on every day we are actually calling, its only on a sunday where no calls take place that this happens!

sundays are ouside of the call times we have specified, so no leads should be loaded into the hopper anyway!

So you likely have a massive archive script or db copy or something that runs on Sunday? (when you're not using the dialer ...?)

Remember that even if the hopper does not find any active campaigns, it requires mysql to run a query to determine ... no campaigns. And if MySQL is hammered (for whatever reason), it'll back up the mysql queue.

Just hope you aren't being DOS attacked on weekends because they know you're idle and not paying attention.

[cbsys]

There are no scripts or routines scheduled to run on a sunday whatsoever, checked crontab, /etc/cron.weekly and /var/spool/cron/root
It does seem to me that it might be some kind of backup routine, but its certainly not scheduled in cron in the usual places!

The system is very strictly locked down with a firewall, no outside access is permitted whatsoever!

[williamconley]

There are no scripts or routines scheduled to run on a sunday whatsoever, checked crontab, /etc/cron.weekly and /var/spool/cron/root
It does seem to me that it might be some kind of backup routine, but its certainly not scheduled in cron in the usual places!

The system is very strictly locked down with a firewall, no outside access is permitted whatsoever!

Did you check in both:

command:

Code: Select all

crontab -e

file:

Code: Select all

/etc/crontab

Also, as I may have mentioned earlier: If it is Indeed a backup cause by mysql, the oldest mysql process ("show full processlist") is likely to be your culprit.

Check /var/log (all areas, it could be anywhere) for any sort of errors (or processes that fire on Sunday, of course, especially right around or before your system death occurs).

And ... do you have replication set up?

[gardo]

Yep! That utility is quite handy. Can help you predict a problematic harddrive. Along with syslog. No need really to reboot. Specially running 24/7 call centers.

Are you saying you've had that utility actually predict a drive death? Or just that it sounds cool and you're using it?

[williamconley]

Yep! That utility is quite handy. Can help you predict a problematic harddrive. Along with syslog. No need really to reboot. Specially running 24/7 call centers.

Are you saying you've had that utility actually predict a drive death? Or just that it sounds cool and you're using it?

Excellent. Thanks for the tip. OpenSuSE supports it and apparently our drives are smart enabled.

Now let's see if smartctl finds the next failure before monthly disk check does.