DEAD CALL PROBLEMS

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

DEAD CALL PROBLEMS

Postby Stevo_Alex19 » Mon Aug 10, 2015 5:17 am

Hi all,
i need some help. I installed goautodial + vicidial and everythign was working fine. 6 months everything was working, 32 agents was callign without any problems, sometimes call was dropped ( DEAD ) but at now there are a lot of DEAD CALLS. And its a big problem now.

Agent is calling like 2 minutes and BAM, call drop. Sometimes agent call even 30 minutes and end call without problem, sometimes there is like 10 call dropped after 30 seconds...

What i need help with is: HOW CAN I DEBUG THIS PROBLEM ? How can i find whats happening? I would like to watch whats happening... So i would like to watch ONE agent and debug: agent call number, agent get to INCALL ... and then when DEAD happend i would like to find WHY.

Is there any way how to find this? I want to see if tehre is no problem with GSM GATE, like short lost signal or mobile operator problem... How can i find where is a problem?

Thank you very much all.
Stevo_Alex19
 
Posts: 1
Joined: Mon Aug 10, 2015 5:03 am

Re: DEAD CALL PROBLEMS

Postby williamconley » Thu Aug 27, 2015 10:40 pm

1) Welcome to the Party! 8-)

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

3) You can start with the asterisk CLI output ("asterisk -R" at the linux command line ... "exit" to get out when you're done). You can also look at logs in /var/log/asterisk and /var/log/astguiclient

4) Troubleshooting the GSM Gate, however, is something you would need to get from a site dealing with that subject. You have not even listed your hardware, though so there is not a lot of likelihood that someone could even accidentally help you (we have no idea what you have to know if there are normal problems with that setup).

5) htop can often show if you are overloading your server.

6) Reinstall with Vicibox.com's .iso image ... perhaps it's related to goautodial :) (Not likely, but never fear the reinstall!) Or build a new one and pop the GSM gate into / onto it and see if that server works better "fresh" with either goauto or vicibox.com's .iso freshly installed.

Happy Hunting! 8-)
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20258
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: DEAD CALL PROBLEMS

Postby pintudas9051 » Thu Feb 06, 2020 6:18 am

i am facing the same issue can you help
pintudas9051
 
Posts: 15
Joined: Thu Jan 30, 2020 6:47 am

Re: DEAD CALL PROBLEMS

Postby VFRDavid » Thu Feb 13, 2020 6:33 pm

I believe I am having a similar/related problem. We get many agents on the read time screen showing as in DEAD calls when they're not, or on very long calls when they're not, and other incorrect statuses and inconsistencies. There is a lot more to it than that, and I believe that the DEAD calls are the result of other issues, and not the cause - but - I could be wrong, which is why I am posting this to this thread.

Here is my config:
3 server cluster, SVN 2973, db Schema 1542, VERSION: 2.14-670a, BUILD: 180424-1521. DB server is running asterisk 13.29.2-vici and the two telephony servers are running 11.22.0-vici. Originally, all 3 servers were installed from the v7 ISO, but, as we grew (we currently have about 15 locations, with a total of 80-100 agents logging in daily), we replaced the DB server with what should be a better system (128Gb RAM, 40 core processor, 12 TB RAID - previous server was 64Gb RAM, 24 core processor, 500GB RAID)...we did this Sunday, 4 days ago, but - we installed via the v8 ISO (SuSE version on the v7 ISO didn't support the new server's RAID controller, but v8's slightly newer SuSE version did). So - the "only" software difference that I can think of between the old / new box is the version of SuSE and Asterisk they are running. I couldn't find anything online that said that was a problem - so - we went forward with that and maintained the same SVN number from the existing servers - they have been pretty solid for about a year, I didn't want to introduce too much new all at once, and would upgrade the SVN's once we had a solid week under out belts...but, we had issues immediately, and continue to do so - so much, that I am considering reverting to the original server tonight (moving the db back over, etc).

To bring the new server online, we transferred the asterisk database as well as our edits to crontab, my.cnf, server-tuning.conf, and others, rebooted the systems, made some test calls and brought it into production - but since about an hour after the first agents logged in Monday morning, we have been having problems. MySQL queries that took a few seconds on the previous server are taking 45 seconds or more. There are the many DEAD calls on the real time screen, which seem to be the manifestation of the issues - I don't think they are the cause - but - it is rampant. When we get a lot of DEAD calls, Asterisk almost always randomly shuts down on the new server. I have recently enabled the Auto-restart option, which helps - but - the agents all lose their calls, at least those connected via the new server, when asterisk is restarted).

Since the first problem noticed was the query speed, we ran a mysqlcheck - but that appeared to be clean. I made some additional changes to the my.cnf and limits.conf, but, haven't noticed any improvement from the MySQL server. The last change I am planning on making is setting the query_cache_size to 0 - seems that smaller is better for this one, at least for more powerful systems, if I am to believe what I read online... Haven't tested that change out yet - won't be able to restart MySQL for another few hours...

Some of the messages I am getting are:
Occasional message on any of the VICI Admin web pages sites:
->"MySQL ERROR: Can't create a new thread (errno 11 "Resource temporarily unavailable"); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug".
In the new servers Asterisk CLI:
->"ERROR[4093]: tcptls.c:897 ast_tcptls_server_root: TCP/TLS unable to launch helper thread for peer '127.0.0.1:21116': Resource temporarily unavailable"
-> "WARNING[9598][C-0000fe50]: func_hangupcause.c:140 hangupcause_read: Unable to find information for channel" (when we get a lot of these - contact is horrible).

I have checked my.cnf and limits.conf, as suggested by the search hits I found...but, I can't seem to find anything that will correct this.

I know that the connection to the "DEAD CALL PROBLEMS" is maybe thin - but - in case I have this backwards, and the DEAD calls are the beginning and not the result - I figured it couldn't hurt to post my misery here...

Anyone with any ideas, I sure would appreciate them!!! Any screen shots or file contents or anything that you need to help me figure this out, let me know, and I will give provide whatever I can...

Thanks...David
David
VFRDavid
 
Posts: 69
Joined: Wed Dec 24, 2014 10:48 am
Location: Deerfield Beach, FL

Re: DEAD CALL PROBLEMS

Postby williamconley » Tue Feb 18, 2020 9:30 am

1) Speed of the 40 cores? Note that more core is cool, but slower cpu speed could result in individual processes taking longer than they did on your older faster system with less cores. So it's possible your system is experiencing some choking due to large tables and smaller tables (read: Prune/Archive your logs and vicidial_list tables!).

2) What do you get for this (during an issue, not a fresh boot):

Code: Select all
mysql -u cron -p\$VARDB_pass -e "show status like '%Aborted%'";


If there are thousands of aborts the Vicibox / OpenSuSE solution would be:

Code: Select all
nano /etc/systemd/system.conf


Change

Code: Select all
#DefaultTasksMax=512


To

Code: Select all
DefaultTasksMax=infinity


Happy Hunting! 8-)
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20258
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: DEAD CALL PROBLEMS

Postby VFRDavid » Tue Feb 18, 2020 9:21 pm

Thank you for the reply - I think that I did the "infinity" modification - but - I'll have to double check on that. Also - I never ran the "Aborted" count check - so I'll have to do that...however, we had to shelf the new system and go back to the old one - so - I will do my best to get it going again this weekend and test it out.

As far as the Core speeds go, here is what hwinfo shows:
YaST2 - hwinfo @ NxNEW161

Hardware Information
┌All Entries──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│───Architecture: x86_64 ┬
│─+─BIOS │
│─+─Block Devices │
│───Boot Architecture: grub │
│─+─CD-ROM │
│─┬─CPU │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz │
│ └+─Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz


and for the current / original server, hwinfo shows:

YaST2 - hwinfo @ NxDB161

Hardware Information
┌All Entries──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│───Architecture: x86_64 │
│─+─BIOS │
│─+─Block Devices │
│───Boot Architecture: grub │
│─+─CD-ROM │
│─┬─CPU │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ ├+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz │
│ └+─Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz


So - Seems like the new one should be a bit better, no?

Thanks again for your reply...will let you know if I get anywhere with this.
David
VFRDavid
 
Posts: 69
Joined: Wed Dec 24, 2014 10:48 am
Location: Deerfield Beach, FL

Re: DEAD CALL PROBLEMS

Postby VFRDavid » Tue Feb 18, 2020 9:43 pm

One more thing that might be important to mention...

When I moved to the new server, I had to remove the "COLLATE utf8_unicode_ci" from all of the WHERE statements making use of VARCHAR IN parameter values (and add them all back when I reverted back to the original server). For example:

Original Server
Code: Select all
UPDATE vicidial_campaigns cm
      SET cm.dial_method=peDialMethod,
          cm.adaptive_dropped_percentage=pcAdaptiveDroppedPercentage,
          cm.adaptive_maximum_level=pcAdaptiveMaximumLevel,
          cm.drop_action=peDropAction
      WHERE cm.campaign_id=pcCampaignID COLLATE utf8_unicode_ci;


New Server
Code: Select all
UPDATE vicidial_campaigns cm
      SET cm.dial_method=peDialMethod,
          cm.adaptive_dropped_percentage=pcAdaptiveDroppedPercentage,
          cm.adaptive_maximum_level=pcAdaptiveMaximumLevel,
          cm.drop_action=peDropAction
      WHERE cm.campaign_id=pcCampaignID;


In both cases, the stored procedure had the pcCampaignID parameter as VARCHAR(8):

Code: Select all
CREATE DEFINER=`dblum`@`%` PROCEDURE `nx_init_camp`(
   IN `pcCampaignID` VARCHAR(8),
   IN `peDialMethod` ENUM('MANUAL','RATIO','ADAPT_HARD_LIMIT','ADAPT_TAPERED','ADAPT_AVERAGE','INBOUND_MAN'),
   IN `pcAdaptiveDroppedPercentage` VARCHAR(4),
   IN `pcAdaptiveMaximumLevel` VARCHAR(6),
   IN `peDropAction` ENUM('HANGUP','MESSAGE','VOICEMAIL','IN_GROUP','AUDIO','CALLMENU','VMAIL_NO_INST')
)


and I would only get errors in WHEREs - not assigning values (SET, UPDATE), and I don't believe I had problems using them in any IF / CASE type of logic either (if, for example, I SELECTed a value INTO a local variable, then had IF @theINTOvariable >= theVARCHARparametervariable THEN; would work fine).

Anyway - not sure if this might be part of the performance issues - if all of the WHERE clauses in all of the VICIdial code were either FAILING or having to internally do some sort of conversion on the fly every time - that could introduce a lot of overhead.

Total guess on my part there, if you couldn't tell - probably didn't use much of the correct terminology either - but - I hope you get what I am trying to explain...

So - again - thanks for your reply...please let me know if any of this could be part of my issues - and if so - is there something different I should have done when installing?
David
VFRDavid
 
Posts: 69
Joined: Wed Dec 24, 2014 10:48 am
Location: Deerfield Beach, FL

Re: DEAD CALL PROBLEMS

Postby williamconley » Tue Feb 18, 2020 9:49 pm

check your MySQL version and your MySQL defaults for charset and such.

Also, in looking back I note you only posted the Major Revision Level of the installer. Always include the full version of the installer used (8.0.1? 8.1.2?). We've not had any issues with DB char set mismatches at all, so be sure you did not change anything "because you're smart" in the db configuration files before you noticed that problem. If you did, revert ... reimport .. and test again.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20258
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)


Return to Support

Who is online

Users browsing this forum: Google [Bot] and 59 guests