Cluster Mysql Sync Problem

[silent_ukr]

Hi,

I have stumbled upon a problem which I hope experts could shed some light on....
I am setting up a high-volume cluster on the following hardware/versions:
Versions:
GoAutoDial CE 3.0 on CentOS 5.9(Final)

Vicidial:
VERSION: 2.7-401a
BUILD: 130508-2256

Cluster Setup:
server1: Web and DB roles (2 x Quad-code Xeon 3.00GHz, 300GB Raid 1 SSD, 16 GB RAM) - mysql Ver 14.12 (x86_64)
server2: dial node1 (Asterisk 1.4.39.1-vici, 2 x Quad-code Xeon 3.00GHz, 300GB Raid 1 SSD hdd, 16 GB RAM )
server3: dial node2 (Asterisk 1.4.39.1-vici, 2 x 2 x Quad-code Xeon 3.66GHz, 73G Raid 1 15K hdd, 16 GB RAM)

In the long run, I am intending to place 2x 20 agents on each dialer with the following campaign:
AMD --> Survey --> AGENT_XFER (with in-group queue overflow for DROP action)

I am in the phase of stress testing at the smaller loads. however for now only about 12-15 agents logins during testing.
Currently dialing ratio is 30/1 to keep the agents busy.
All calls/agents are in g729 and being recorded, load balancing is enabled and working between the dial nodes.

Once I get about 15 agents logged in to a single node, it runs fine for about 1 hour (average calls being dialed out at about 350)
After an average of 1 hour, the campaign stalls...
Real-time screen shows leads being dialed and keeps updating with refresh rate, however if I go to asterisk console, there are no attempts to dial anything.. no verbose output...

I understand that the first comment that I would receive I am using too many channels per dialer max is 200 channels due to channel locks etc. etc..
However I am confident that this is an issue with Mysql, NOT asterisk channel locks, because:
1) During the "outage" state, I am able to make direct calls from a register sip endpoint directly (by passing agent screen), so asterisk does NOT crash
2) Restart of the asterisk service, nor full restart of node1 does not help to resolve it

Only the restart of mysql service on the DB server solves this lock up, and the system carries on to work at the same volume for about another hour until the next crash...

So far,
I checked the mysqld.log on the db server and there is no output about any errors, just basic log of service stop/start
Check the transfer rates between servers with iperf -- 1 GPBs, all servers connected to the same switch
Checked the NTP, all servers sync up with the same NTP server (pool ntp org)
Checked all servers vitals - system loads are not even hitting 25% on cpu, IO or RAM

Where else I could turn too to further troubleshoot this issue? Maybe agi logs? or Vicidial has some mysql logs? Any other debugging mode of any vicidial services?
As I am new to vicidial, any suggestions would be greatly appreciated..

Thanks in advance.

[williamconley]

Your primary problem is that you are using a "consumer-level" installer for an Enterprise Level system. Switch to Vicibox. Let Kumba's cluster installation do most of your work. You will find that the high-volume settings for Vicibox servers will help dramatically.

If you don't want to switch over (because you believe that it would be a waste of your prior effort, which is a common flaw in IT technicians we've learned to cope with), you could perform this Vicibox installation in a virtual system and borrow settings.

Of course, you could also troubleshoot your "have to restart it" mysql system directly and find out WHY you must restart it and perhaps actually resolve the original issue. It will certainly not hurt you at this point in your project to become a bit more involved in how a mysql server operates. There are such things as slow-query-logs and various other forms of troubleshooting available.

For instance "show processlist" will rat out a "stingy" mysql query that may lock up your system once per hour. Be aware that once a system like that has a hiccup, recover is not simple and a restart may be necessary if you are unable to identify and kill the offending mysql process. However, show processlist can give you the mysql procid which can be used to kill the process in question ... thus freeing up all the "stuck" processes behind that one. Identifying the actual flaw could be quite enlightening. Our last client with a similar problem found out that it was his own simple, unobtrusive, agent report that was actually overloading mysql during heavy load and shutting down his system.



Happy Hunting!

[silent_ukr]

Thank you for your reply, William. I will definitely try enabling logging for slow queries.

Additionally, I happen to have a Vicibox installed as single instance in VM enviroment.
Just to confirm, what particular settings should I be looking at? my.cnf ?

Is there anything in Vicidial itself that might be handy?
I was thinking that it might be the lead hopper load process that is the bottle neck. Currently, there is about 1 million leads in the list (however the target is to have about 250 million eventually). The largest minimum hopper level that can setup is 2000, at this dial rate, 2000 leads go by very fast.

Are there any tricks to run preload the hopper for larger number of leads (maybe 100,000), but less frequently then default of 1 minute..

Alex

[williamconley]

I often find myself discussing the difference between a Technician (read: IT professional) and a "computer person" (read: Amateur). A technician will find the problem through troubleshooting. While a technician may indeed indulge him/herself in a quick round of guessing which may involve a google hit or two, ultimately they will stop guessing and get down to tracing the problem.

In this case, you should be finding the running processes when you experience your "issue" and seeing if you can see the FIRST (oldest) process that is running in mysql which would appear (obviously) to be causing the bottleneck. If this is happening, "show processlist" will have a large (and steadily growing) list of mysql processes. Each will have an age (in seconds) which will continually age with each check.

Find your offending process ... then you can find out what executes it and why.

Find the process stopping your dialer, you'll likely be able to get it going again. Good place to start (instead of "hey! what if I change XX? or YY?).

[gardo]

Sounds like you have congested channels in the database. What's the output of this mysql query when your campaign stalls:

Code: Select all

mysql> select * from vicidial_auto_calls where channel is null and last_update_time>=100;

[silent_ukr]

Hi Gents,

post back on this issue:

Today had another stress testing session. Without any changes made, the campaign stalled in about 40 minutes. I ran the vicidial_auto_calls query during the 'outage' state (select * from vicidial_auto_calls where channel is null and last_update_time>=100); and got 390 entries -- Interesting find that when I would run the query a few times, I would get different value, which means to me that mysql service was indeed operational... (not sure if it is expected)

After the first stall happened, I changed a few parameters on the mysql config and rebooted mysql:

Code: Select all

max_connections=16000
max_heap_table_size=4096M
query_cache_size = 0


At this point, the campaign was running for a good 3-4 hours averaging about 400 channels on the upstream without any issues, however after a few hours, it started hiccups again.
It was not a full crash of mysql, however asterisk node would stop dialling anything for a few minutes and then start again a much slower pace (hopper would remain above 2000 during this time); thus agents would wait for 10-15 minutes between calls..

mtop on 5 second inteval on the db would not really show anything useful or any "stuck" queries, however the slow-query-log would capture a query (as I understand this is AST_VDhopper.pl)

Code: Select all

# Time: 131105 16:43:30
# User@Host: cron[cron] @ localhost []
# Query_time: 19  Lock_time: 0  Rows_sent: 2000  Rows_examined: 6401430
SELECT lead_id,list_id,gmt_offset_now,phone_number,state,status,modify_date,user,vendor_lead_code FROM vicidial_list where called_since_last_reset='N' and status IN('AA','NEW') and list_id IN('500','202') and lead_id NOT IN('5832389','5832129','5832131','5832881','5832157','5832110','5832101','5832461','5832691','5832852','5832999','5832274','5832608','5832885','5832091','5833016','5832600','5832246','5832207','5832730','5832563','5832831','5832718','5832968','5832212','5832405','5832867','5832752','5832315','5832277','5832455','5832801','5832725','5832306','5832180','5832942','5832784','5832999','5832983','5832971','5832963','5832955','5832947','5832939','5832104','5832931','5832923','5832915','5832907','5832102','5832899','5832514','5832316','5832891','5832704','5832740','5832201','5832883','5832975','5832875','5832867','5832925','5832813','5832954','5832859','5832415','5832625','5832529','5832851','5832788','5832812','5832815','5832692','5832843','5832835','5832827','5832161','5832819','5832911','5832811','5832803','5832795','5832440','5832787','5832394','5832779','5832771','5832763','5832755','5832747','5832739','5832731','5832723','5832715','5832707','5832699','5832691','5832621','5832683','5832675','5832667','5832659','5832651','5832643','5832499','5832467','5832635','5832627','5832619','5832710','5832611','5832914','5832603','5832591','5832575','5832559','5832543','5832527','5832515','5832483','5832451','5832439','5832423','5832407','5832391','5832375','5832359','5832351','5832343','5832335','5832327','5832319','5832311','5832303','5833018','5832295','5832287','5832279','5832915','5832271','5832267','5832251','5832509','5832237','5832227','5832219','5832211','5832273','5832257','5832841','5832241','5832199','5832191','5832183','5832506','5832747','5832175','5832429','5832364','5832167','5832159','5832151','5832143','5832135','5832131','5832119','5832111','5832694','5832103','5832095','5832083','5832087','5832127','5832279','5832817','5832617','5832924','5832242','5832172','5832306','5832906','5832686','5832774','5832696','5832511','5832607','5832779','5832413','5832484','5832355','5832378','5832890','5832209','5833049','5832380','5832580','5832505','5832565','5832421','5833009','5832685','5832717','5832957','5832661','5832343','5832988','5832479','5832289','5832219','5832314','5832614','5832780','5832652','5832123','5832537','5832137','5832281','5832429','5832469','5832605','5832385','5832941','5832741','5832837','5832772','5832585','5832501','5832433','5832564','5832571','5832605','5832919','5832660','5832337','5832151','5832555','5832349','5832458','5832237','5832884','5832120','5832951','5832808','5832460','5833041','5833025','5832325','5832281','5832523','5832760','5832519','5832732','5832596','5832857','5832874','5832412','5832751','5832575','5832656','5832285','5832712','5832088','5832453','5832792','5832557','5832959','5832501','5832521','5832707','5832912','5832589','5832139','5832130','5832673','5832146','5832733','5832162','5832556','5832113','5832082','5832090','5832103','5832098','5832689','5832106','5832114','5832124','5832235','5832140','5832172','5832864','5832497','5832474','5832294','5832310','5832322','5832340','5832358','5832840','5832374','5832398','5832777','5832182','5832190','5832198','5832206','5832756','5832598','5832214','5832222','5833043','5832503','5832230','5832238','5832246','5832224','5832254','5832262','5832373','5832768','5832270','5832278','5832300','5832286','5832302','5832842','5832318','5832213','5832330','5832350','5832366','5832662','5832382','5832390','5832406','5832414','5832422','5832430','5832620','5832973','5832991','5832438','5832446','5832454','5832462','5832470','5832757','5832478','5832486','5832494','5832502','5832612','5832132','5832664','5832647','5832173','5832583','5832510','5832518','5832636','5832526','5832469','5832288','5832534','5832320','5832542','5832657','5832827','5832164','5832546','5832403','5832745','5832554','5832360','5832895','5832562','5832778','5832570','5832578','5832485','5832586','5832594','5832339','5832602','5832610','5832618','5832215','5832257','5832268','5832787','5832626','5832634','5832220','5832721','5832573','5832113','5832642','5832650','5832664','5832674','5832682','5832690','5832698','5832706','5832714','5832722','5832730','5832738','5832746','5832754','5832762','5832770','5833033','5832778','5832786','5832794','5832802','5832156','5832157','5832697','5832810','5832818','5832826','5832830','5833031','5832834','5832441','5832842','5832861','5832850','5832984','5832862','5832878','5832894','5833017','5832906','5832922','5832938','5832907','5832954','5832966','5832974','5832982','5832463','5832990','5832998','5832096','5832252','5833006','5832613','5832304','5833014','5833022','5832417','5833030','5832282','5833040','5832866','5832524','5832882','5832898','5832918','5832475','5832934','5832970','5832950','5833048','5833027','5832973','5832940','5833036','5832161','5832234','5832381','5832386','5832408','5832473','5832244','5832629','5832701','5832765','5832183','5833023','5832322','5832208','5832684','5832796','5832193','5832173','5832197','5832481','5832449','5832673','5832801','5832401','5832505','5832672','5832125','5833013','5833015','5832997','5832537','5833039','5833042','5833026','5833010','5832994','5832978','5832962','5832500','5832401','5832946','5832721','5832930','5832914','5832898','5832093','5832092','5832181','5832882','5832964','5832866','5832921','5832850','5832834','5832818','5832457','5832802','5832627','5832786','5832761','5832770','5832175','5832101','5832728','5832201','5832754','5832345','5832247','5832517','5832738','5832550','5832845','5832722','5832706','5832693','5832690','5832789','5832674','5832933','5832658','5832642','5832177','5832609','5833017','5832618','5832586','5832678','5832570','5832546','5832428','5832530','5832514','5832111','5832379','5832858','5832626','5832594','5832459','5832615','5832266','5833011','5832164','5832482','5832490','5832411','5832458','5832442','5832443','5832467','5832426','5832410','5832479','5832394','5832378','5832511','5832362','5832547','5832346','5832330','5832579','5832314','5832298','5832328','5832987','5832282','5832258','5832734','5832235','5832230','5832927','5832849','5832214','5832264','5832202','5832365','5832186','5832162','5832146','5832122','5832130','5832099','5832392','5832856','5832857','5832098','5832082','5832977','5832152','5832248','5832727','5832445','5832628','5832200','5832805','5832731','5832962','5832717','5832535','5832699','5832334','5832453','5832233','5832875','5832956','5832566','5832905','5832114','5832606','5832969','5832492','5832170','5833036','5832978','5832777','5832333','5832181','5832565','5832986','5832233','5832388','5832587','5832338','5832089','5832669','5832871','5832138','5832828','5833024','5832437','5833049','5832307','5832771','5832545','5832723','5832985','5832939','5832291','5832155','5832419','5832387','5832811','5832650','5832758','5832313','5832097','5832560','5832127','5832929','5832905','5832341','5832489','5832089','5832209','5832250','5832559','5832270','5832097','5832217','5832797','5832402','5832204','5832967','5832685','5832137','5832285','5832922','5832352','5832798','5832817','5832700','5833010','5832681','5832426','5832218','5832979','5832846','5832571','5832539','5832503','5832193','5832267','5832471','5832435','5832403','5832371','5832416','5833051','5832367','5832577','5832106','5832941','5832543','5832963','5832573','5832368','5832109','5832588','5832168','5832466','5833051','5832434','5832793','5833035','5832186','5832347','5833011','5833009','5832800','5832274','5832993','5832179','5832497','5832196','5832538','5832421','5832657','5832926','5832601','5832269','5832821','5833048','5832455','5832488','5832547','5832531','5832890','5833001','5832245','5833019','5832835','5832357','5832423','5832892','5832393','5832958','5832809','5832979','5832323','5832689','5832681','5832156','5832961','5832231','5832804','5833045','5832195','5832863','5832676','5832599','5832312','5832321','5832517','5832241','5832338','5832791','5832649','5832766','5832986','5832933','5832854','5832465','5832189','5832983','5832308','5832287','5832419','5832987','5832645','5832865','5832841','5832277','5832203','5833012','5832610','5832643','5832536','5832437','5832568','5832522','5832433','5832086','5832225','5832110','5832624','5832507','5832102','5832150','5832301','5832945','5832190','5832272','5832218','5832603','5832236','5832266','5832775','5832286','5832790','5832302','5832870','5832318','5832595','5832229','5832334','5832350','5832263','5832366','5832638','5832555','5832513','5832382','5832391','5832398','5833027','5833034','5832414','5832194','5832456','5832430','5832371','5832446','5832633','5832675','5832169','5832470','5832462','5832960','5832494','5832606','5832502','5832518','5832534','5832554','5832869','5832757','5832661','5832621','5832744','5832457','5832313','5832169','5832574','5832324','5832112','5832598','5832820','5832436','5832630','5832646','5832662','5832678','5832981','5832992','5832694','5832710','5832726','5832558','5832742','5832758','5832094','5832853','5832774','5832790','5832822','5832533','5832806','5832321','5832525','5832822','5832561','5832838','5832109','5832854','5832870','5832184','5832886','5832902','5832353','5832918','5832118','5832158','5832893','5832989','5832773','5832198','5832934','5832776','5832950','5832105','5832896','5832966','5832296','5832982','5832998','5833014','5833030','5832086','5832785','5833046','5832538','5832167','5832794','5832671','5832641','5832290','5832305','5832825','5832724','5833003','5832991','5832975','5832967','5832959','5832951','5832943','5832935','5832927','5832919','5832911','5832903','5832532','5832895','5832292','5832129','5832309','5832601','5832873','5832737','5832897','5832887','5832879','5832871','5832863','5832855','5832847','5832839','5832831','5832668','5832823','5832171','5832592','5832815','5832807','5832729','5832799','5832791','5832783','5832345','5832775','5832767','5832759','5832189','5832736','5832751','5832743','5832735','5832727','5832354','5832719','5832945','5832711','5832293','5832703','5832695','5832687','5832530','5832384','5832679','5832663','5832671','5832663','5832655','5832647','5832507','5832284','5832909','5832765','5832475','5832639','5832631','5832623','5832615','5832607','5832197','5832687','5832239','5832485','5832883','5832832','5832336','5832599','5832583','5832567','5832178','5832793','5832551','5832535','5832886','5832519','5832471','5832491','5832459','5832447','5832226','5832548','5832735','5832431','5832932','5832397','5833050','5832415','5832399','5832383','5832367','5832355','5832826','5832347','5832339','5832160','5832508','5832331','5832323','5832315','5832307','5832742','5833034','5832299','5832291','5832122','5832617','5832283','5832275','5832259','5832243','5832477','5832231','5832715','5832223','5832215','5832353','5832207','5832265','5832249','5832960','5832203','5832195','5832944','5832187','5832179','5832432','5832899','5832143','5832171','5832436','5832163','5832155','5832713','5832640','5832147','5832139','5832928','5832079','5832123','5832228','5832115','5832107','5832912','5832099','5832091','5832892','5832259','5832876','5832107','5832972','5832860','5832705','5833038','5832649','5832405','5832499','5833013','5832716','5832714','5832688','5833028','5832138','5833020','5833012','5833004','5832996','5832623','5832988','5832980','5832258','5833002','5832528','5832719','5832972','5832964','5832816','5832948','5832932','5832916','5832904','5832383','5832221','5832900','5832888','5832901','5832872','5832753','5832293','5832133','5832516','5832562','5832856','5832848','5832840','5832513','5832541','5832191','5832165','5832232','5832584','5832340','5832450','5832848','5832832','5832874','5832824','5832816','5832808','5832115','5832800','5832792','5832788','5832776','5832369','5832768','5832753','5832748','5832760','5832147','5832752','5832740','5832724','5832117','5832708','5832736','5832829','5832720','5832572','5832704','5832442','5832696','5832688','5832970','5832680','5832672','5833053','5832660','5832450','5832648','5832640','5832567','5832632','5832145','5832476','5832624','5832616','5833015','5832608','5832600','5832395','5832592','5832782','5832584','5832576','5832568','5832560','5832552','5832404','5832149','5832357','5832785','5832176','5832482','5832225','5832152','5832937','5832769','5832311','5832540','5832949','5832528','5832512','5832708','5832319','5832500','5833008','5832078','5832484','5832178','5832468','5832081','5832847','5832400','5832402','5832646','5832452','5832420','5832404','5832582','5832632','5832799','5832396','5832388','5832658','5832364','5832352','5832953','5832923','5832332','5832083','5832320','5832763','5832304','5832665','5832288','5832938','5832276','5832569','5832809','5832253','5832876','5832260','5832487','5832252','5832244','5832217','5832997','5832236','5832228','5832597','5833052','5832220','5832212','5832204','5832873','5833002','5832196','5833040','5832303','5832188','5832425','5832180','5832888','5832295','5832160','5832659','5832136','5832729','5832120','5832427','5832709','5832853','5832238','5832859','5832079','5832136','5832410','5833043','5832585','5832619','5832112','5832317','5832104','5832569','5832096','5832088','5832080','5832805','5832709','5832549','5832365','5832255','5832356','5832124','5832158','5832670','5832142','5832126','5832885','5832829','5832337','5832271','5832125','5832093','5832409','5832995','5832354','5832679','5832656','5832961','5832298','5832496','5832931','5832407','5832844','5833000','5832726','5832921','5832269','5832372','5832910','5832985','5833005','5833007','5833031','5832622','5832434','5833047','5832506','5832222','5832764','5833028','5832452','5832363','5832395','5832427','5832463','5832495','5832495','5832531','5832855','5833035','5832563','5832595','5833021','5832327','5832154','5833037','5833053','5832448','5832444','5833001','5832965','5832205','5832085','5832145','5832185','5832261','5832229','5832289','5832305','5832634','5832381','5832413','5832445','5832533','5832597','5832227','5832613','5832387','5832629','5832837','5832645','5832897','5832861','5832893','5832677','5832925','5832653','5832669','5832080','5832128','5832733','5832749','5832377','5832574','5832781','5832425','5832797','5832813','5832746','5832461','5832838','5832493','5832917','5832949','5832529','5832833','5832561','5832593','5832239','5832202','5832946','5832370','5832418','5832490','5832980','5832148','5832242','5832276','5832552','5832836','5832245','5832081','5832117','5832153','5832213','5832297','5832329','5832397','5832489','5832581','5832637','5832877','5832909','5832677','5832725','5832773','5832821','5832901','5832957','5832141','5832221','5832698','5832590','5832865','5832326','5832532','5832516','5833029','5832496','5832283','5832431','5832789','5832133','5832480','5832078','5832094','5832887','5832399','5832126','5832464','5832134','5832587','5832170','5832651','5832240','5832210','5832759','5832226','5832823','5832100','5832903','5832176','5832448','5832250','5832278','5832432','5832294','5832310','5832416','5832326','5832342','5832400','5832261','5832358','5832384','5832845','5832362','5832374','5832376','5833018','5832390','5832406','5832360','5832422','5832720','5832336','5832438','5832454','5832342','5832486','5832478','5832308','5832590','5832622','5832292','5832510','5832526','5832272','5832542','5832566','5832582','5832614','5832153','5832638','5832654','5832670','5832686','5832375','5832702','5832718','5832635','5832174','5832734','5832807','5832750','5832766','5832782','5832798','5832223','5832814','5832830','5832872','5832846','5832862','5832309','5832878','5832894','5832910','5832520','5833004','5832926','5832142','5832182','5832256','5832260','5832942','5832958','5832637','5832974','5832084','5832251','5832417','5832990','5832868','5833041','5832576','5833006','5832977','5833022','5832913','5833038','5832118','5832825','5832121','5832194','5832468','5832993','5833044','5832498','5832908','5832880','5832553','5832666','5832996','5832141','5832349','5832589','5832625','5832665','5832761','5832881','5833042','5832465','5833052','5832952','5832444','5832936','5832920','5832900','5832944','5832884','5832483','5832868','5832132','5833044','5833032','5833024','5833016','5833008','5833000','5832992','5833003','5832443','5832984','5832512','5832976','5832639','5832199','5832968','5832956','5832940','5832924','5832908','5832896','5832880','5832864','5832234','5832852','5832844','5832836','5832828','5832820','5832812','5832804','5832255','5832796','5832703','5832902','5832784','5832493','5832780','5832772','5832819','5832764','5832920','5832418','5832756','5832748','5832732','5832716','5832744','5832728','5832712','5832700','5832692','5832936','5832684','5832676','5832889','5832668','5832205','5832652','5832644','5832630','5832636','5833023','5832843','5832579','5832385','5832628','5832620','5832612','5832604','5832596','5832588','5832580','5832917','5832572','5832564','5832556','5832548','5832168','5832916','5832544','5832536','5832144','5832520','5832508','5832346','5832492','5832476','5832460','5832253','5832428','5832412','5832396','5833046','5832380','5833021','5832781','5832593','5832379','5832372','5832929','5832356','5832344','5832324','5832312','5832296','5832280','5832268','5832256','5832248','5832240','5832232','5832224','5832216','5832208','5832200','5832192','5832184','5832174','5832144','5832969','5832128','5832087','5833026','5832544','5832116','5832108','5832100','5832092','5832084','5832449','5832166','5832150','5832134','5832930','5832420','5832317','5832889','5832745','5832913','5832539','5832243','5832609','5833025','5832810','5832851','5832192','5832359','5833007','5832578','5832955','5832491','5832591','5832348','5832877','5832447','5832474','5832333','5832755','5832860','5832481','5832290','5832480','5833029','5833045','5832767','5832121','5832273','5832361','5832701','5832393','5832216','5832441','5832477','5832116','5832737','5832509','5832644','5832971','5832545','5832577','5832602','5832361','5832869','5832211','5832487','5832981','5832373','5832163','5832149','5832299','5832683','5832928','5832952','5832344','5832090','5833033','5832525','5832166','5832247','5832389','5832541','5832325','5832633','5832937','5832705','5832769','5832833','5832953','5832762','5832741','5832341','5833037','5832523','5832743','5832935','5832154','5832424','5832994','5833047','5832524','5832504','5832488','5832377','5832472','5832456','5832254','5832440','5832439','5832424','5832549','5832408','5832392','5832943','5832206','5832783','5832504','5832666','5832368','5832655','5832348','5832328','5832316','5832693','5832300','5832284','5832264','5832581','5832814','5832148','5832904','5832891','5832711','5832527','5832409','5832667','5833019','5832540','5832948','5833039','5832749','5832473','5832803','5832265','5832210','5832165','5832331','5832369','5832551','5832739','5832947','5832976','5832616','5832464','5832188','5832702','5832522','5832521','5832095','5832653','5832472','5832879','5832451','5832611','5832654','5832187','5832553','5832604','5832085','5832363','5832249','5832697','5832989','5832140','5832329','5832386','5832262','5832370','5833032','5832280','5832806','5832515','5832435','5832695','5832995','5832631','5832301','5832680','5832839','5832275','5832411','5833005','5833020','5832550','5832351','5832648','5832177','5832682','5832858','5832466','5832335','5832332','5832376','5832965','5832824','5832108','5832498','5832849','5832713','5832641','5832557','5832263','5832105','5832750','5833050','5832297','5832558','5832795','5832159','5832135','5832185','5832119','0')  and ((gmt_offset_now IN('13.00','12.75','12.50','12.25','12.00','11.75','11.50','11.25','11.00','10.75','10.50','10.25','10.00','9.75','9.50','9.25','9.00','8.75','8.50','8.25','8.00','7.75','7.50','7.25','7.00','6.75','6.50','6.25','6.00','5.75','5.50','5.25','5.00','4.75','4.50','4.25','4.00','3.75','3.50','3.25','3.00','2.75','2.50','2.25','2.00','1.75','1.50','1.25','1.00','0.75','0.50','0.25','0.00','-0.25','-0.50','-0.75','-1.00','-1.25','-1.50','-1.75','-2.00','-2.25','-2.50','-2.75','-3.00','-3.25','-3.50','-3.75','-4.00','-4.25','-4.50','-4.75','-5.00','-5.25','-5.50','-5.75','-6.00','-6.25','-6.50','-6.75','-7.00','-7.25','-7.50','-7.75','-8.00','-8.25','-8.50','-8.75','-9.00','-9.25','-9.50','-9.75','-10.00','-10.25','-10.50','-10.75','-11.00','-11.25','-11.50','-11.75','-12.00','-12.25','-12.50','-12.75','99') ) )    order by lead_id asc limit 2000;


I will try the settings from Vicibox tomorrow and let you now if it helps

Code: Select all

max_connections=16000
max_heap_table_size=4096M
query_cache_size = 0
skip-external-locking                                                                                                           
key_buffer_size = 640M                                                                                                         
max_allowed_packet = 2M                                                                                                         
table_open_cache = 8192                                                                                                         
sort_buffer_size = 4M                                                                                                           
net_buffer_length = 8K                                                                                                         
read_buffer_size = 4M                                                                                                           
read_rnd_buffer_size = 16M                                                                                                     
myisam_sort_buffer_size = 64M                                                                                                   
thread_cache_size = 50                                                                                                                                                                                                               
thread_concurrency=8                                                                                                           
skip-name-resolve                                                                                                               
connect_timeout=60                                                                                                             
long_query_time=3                                                                                                               
log_slow_queries                                                                                       
open_files_limit=24576                                                                                             
expire_logs_days=3                                                     
default-storage-engine=MyISAM                                           
table_definition_cache=8192                                             
table_cache=8192                                                       
myisam_recover                                                         
myisam_repair_threads=1



Is there anything else would you suggest to try?


Thanks.

[gardo]

You should be seeing different values when you ran that query. Based from your description, it looks like you have congested/blocked channels in the vicidial_auto_calls tables.

This should clear them up:

Code: Select all

mysql> DELETE from vicidial_auto_calls where channel is null and (now() - last_update_time)>=100;

Let us know if this helps.

And try running mtop between 1 to 3 second intervals so you can see what's happening with your DB.

[dbjsolutions]

Hello Guys, have you figured out the problem on this matter ? I am having the _EXACT_ same issue with my vicidial cluster. I am also quite sure the problem comes from my dedicated db server. I have used vicibox installation all across the cluster and used mybigvici.cnf and followed recommendations given by mysql-tuning.

Asterisk servers will stop dialing all the sudden after about 1 hour. restarting mysql process will fix it, but all agents have to re-login. I also found that rebooting any one of the asterisk servers fixes the issue temporairely until next crash.

Please help! Really appreciated.

[williamconley]

You need to find out why it stops dialing, now that you know it can be fixed by restarting a process ...

Stage 1: Are there leads available to be dialed on the campaign?
* Look in the campaign
Stage 2: Are there leads in the hopper?
* Look in the campaign and/or in the hopper itself
* Verify that the leads are real 10 digit numbers
Stage 3: Is the dialer Attempting To Dial?
* Check the screens in screen -list and find what's going on in the decisionmaking process.

All sorts of problems can result in "not dialing", you need to troubleshoot to find which direction this one's coming from so you can fix it.