vicidial.org

by **dgroth02** » Tue Aug 07, 2018 10:25 am

Ok this is now happening to me on multiple systems (all hosted in datacenters), but I'll give the details of one of the systems here

System Configuration:
Multi-Server System, 1 DB/Web Server, 7 dialing servers
ViciBox v.7.0.4-170113
VERSION: 2.14-667a
BUILD: 180331-1715
Asterisk 11.25.1-Vici on all servers

Dual Quad core Xeon in each server Minimum (Slightly varying speeds). DB Server with 128GB of RAM, each dialing server has 16GB RAM. SSD drives in all servers

This particular system has 50 agents spread across the servers, dialing at approximately 7 to 1.

The problem is essentially that occasionally, at various dialing levels, a server will go "Red" in the summary screen. When that happens, all the agents with their phones on that server will get the dreaded Timesync Error. and will be unable to log back in. The asterisk command line shows hundreds of " channel.c:1310 __ast_queue_frame: Exceptionally long voice queue length queuing" warnings.

After a reboot, the "Red" is cleared and phones can re-register, agents can log in, and things will continue as normal.

Here is what I have noticed that does/does not affect the problem and what I have done to date

Seems to happen most often when a remote client has a particularly bad Internet connection, but not always - Connection can be great and it will still happen
A servers are in time synchronization with each other
All servers have remote client IP's http, sip, and rtp white listed only
System uses SIP trunks and have their IP's white listed.
All other access to the system on all ports is closed
Trunks are balanced across all dialing servers

Yes I have read the manual and searched the forums, and while I see others mentioning this issue, no real solution was presented (apart from an NTPdate sync in the crontab - which doesn't work) and that was 2 years ago.

For my clients that I recommend Vicidial to as a dialing server, this is pretty much the ONLY issue they ever run into, but it is quickly becoming the issue that causes them to leave Vicidial.

I would really appreciate ANY help or direction.

Thank you.

by **uncapped_shady** » Tue Aug 07, 2018 12:47 pm

Hi what timing devices are you using in your servers? Vicidial recommends the Amfeltec PCI Express cards.

by **dgroth02** » Tue Aug 07, 2018 2:17 pm

Just the standard dahdi_dummy internal timing. I've been told by a friend that this may have to do with Asterisk 11.25.1 and that I should upgrade to 11.25.3 - Anyone else have this issue with this resolution?

by **dgroth02** » Thu Aug 09, 2018 6:10 pm

Have upgraded asterisk and still having this issue...still suspect Internet related timeouts. Any suggestions on making vic close connections faster or make it less sensitive?

by **williamconley** » Thu Aug 09, 2018 6:26 pm

Time sync error is tossed during two basic categories of fault:

1) Time is out of sync between servers. Solution: Sync all the servers to ONE of the servers in the cluster (with iBurst) so they are always in sync. Use NTP, don't "set the time" periodically. Note that this is not something that would affect individual agents, it would down an entire server, and it would be off by about 6 seconds to cause the issue. 5% of the time this is the problem.

2) The time field of the agent's session is not updated at all (instead of being 'out of sync', it's just 'not updating', but tosses the same error!) This can be caused by a bad connection between the agent's web screen and the agent's web server, although it can also happen if the agent's dialer (the server to which the agent has registered his phone) processes fail. There are screens (screen -list) running which must update various fields continually. If one of those terminates OR Just Stops working even though it's still listed, the results can be deadly to the cluster. The one that ordinarily causes "red server" in the "Reports" page is the update script. Often just killing that script (and allowing it to regenerate on it's own, at the 1 minute mark from the keepalive script) will remove the redness and bring the server back online.

Sounds like you have more than one of the 2) problems going if you have bad internet for some agent and red servers.

by **dgroth02** » Fri Aug 10, 2018 9:32 am

I understand timesync, but this still doesn't explain LongQueue length errors on Asterisk console. Would the update script cause this to happen?

If the time field is sensitive to updates - is there a way to make it less sensitive?

by **williamconley** » Fri Aug 10, 2018 2:35 pm

It's not "sensitive", it's a purposebuilt "symptom" indicating "you have a problem, you must fix this". Connectivity or time, one of them is broken. If you make it "not sensitive", then the system will stop working and you'll never know. Data will be incorrect. Calls won't transfer. All sorts of bad things will happen. But you will receive NO warnings.

Long data queues have their own cause, unrelated to Vicidial code. Likely also due to connectivity issues. Another symptom. Treat the cause.

by **dgroth02** » Tue Oct 30, 2018 9:41 am

Still having major issues with this and the timesync was a red herring. Of course its throwing a timesync error when Asterisk is having issues. Makes perfect sense. Still doesn't explain why asterisk is going nuts.

Basically the servers in the cluster will operate fine for hours to days. Then one (specifically with agents on it) will start this issue.

Anyone have any ideas?

by **thephaseusa** » Tue Oct 30, 2018 10:34 am

I’m glad you posted again, Mr. Roth. I wanted to tell you that using the kernel parameters Kumba gave us:

nopti nospectre_v2 nospec

It reduced the load on my db server by half.

As for this current problem, i remember you saying before you had vicidial 7.04 on your cluster, and everything was working perfectly. You upgraded all boxes to 8.1.2, you weren’t happy, so you went back to 7.0.4. Were you able to restore all your boxes to 7.0.4 from backups?

John

by **dgroth02** » Tue Oct 30, 2018 12:11 pm

Nope - that was a COMPLETELY different issue - this is the same issue on v7.0.4 - and I have it with several different clients running 7.0.4

BUT - I will try that kernel parameter and see how it looks.

by **williamconley** » Wed Oct 31, 2018 11:04 pm

1) Do you reboot these servers nightly?

2) Does this happen on all servers only certain servers?

3) Do the servers upon which this happens have inbound calls on them?

4) Time Sync isn't a red herring: It's a symptom, as stated earlier. A warning sign that "now is a good time to look at this system to see what's wrong". Whatever is causing your time sync errors may be causing all your other problems, too. Or at least some of them.

So check logs when you get time sync. Check for dropped packets. Check load. Use the time sync as a trigger and see if you can clear the time sync. If that fixes your problem you can do a little dance. If fixing the time sync doesn't fix your other problem(s), at least you don't get the time sync problem any more.

And to be clear: Time sync isn't generated by asterisk. It's perl script based unless time is actually out of sync (which is worth checking: it happens! and it can cause all sorts of issues!)

by **dgroth02** » Thu Nov 01, 2018 3:17 pm

1) Tried both rebooting and not rebooting, it didn't make a difference
2) Mostly those with agents on them, the other "dialing only" servers (they aren't using cross server extensions) don't seem to have issues, but to be sure, they do occasionally have this problem, just much less often.
3) No, except for one server.
4) I'm sorry about that - what I mean to say is that I understand Time Sync is an issue with communication and the system and Asterisk isn't involved in generating that error per se. All the servers are synced to the Web/DB server, which is synced to the ntp.org servers - that's what I meant by red herring - Its totally a warning sign that something isn't right, but so is the big red bar on the "Reports" page

HOWEVER - I'm pleased to report that as of two days ago, I made the kernel boot parameter changes suggested (nopti nospectre_v2 nospec) on both the dialing/agent servers and the DB server - and as of two days of operation, not a single issue. Noticed also that the server load has drastically reduced.

Will continue to monitor and keep everyone up to date.

by **williamconley** » Thu Nov 01, 2018 4:21 pm

Excellent Postback!!

by **dgroth02** » Mon Nov 19, 2018 11:44 am

update:

The saga continues.

While the servers are a BIT more stable, they still, approximately once every 3 days, have the "Exceptionally Long Queue Length" error and need to be rebooted.

Before anyone asks, yes I have tried turning on auto reboot at night - no difference.

I'm tearing my hair out here. I can't believe that an acceptable level of performance is "oh well reboot it when it happens and it will be fine".

I have noticed ONE trend - that 95% of the time - the servers I have to reboot are the servers with agent phones on them. Remember I have agents spread across servers, and only 4 of the 9 servers have agents on them.

by **williamconley** » Mon Nov 19, 2018 8:25 pm

williamconley wrote:3) Do the servers upon which this happens have inbound calls on them?

4) Time Sync isn't a red herring: It's a symptom, as stated earlier. A warning sign that "now is a good time to look at this system to see what's wrong". Whatever is causing your time sync errors may be causing all your other problems, too. Or at least some of them.

So check logs when you get time sync. Check for dropped packets. Check load. Use the time sync as a trigger and see if you can clear the time sync. If that fixes your problem you can do a little dance. If fixing the time sync doesn't fix your other problem(s), at least you don't get the time sync problem any more.

And to be clear: Time sync isn't generated by asterisk. It's perl script based unless time is actually out of sync (which is worth checking: it happens! and it can cause all sorts of issues!)

So ... inbound? Has time sync been resolved? Do you have anything else in the logs from the moment of an occurrence? For instance do they turn "red" on the Reports page? vicidial/admin.php?ADD=999999

Does it happen on ALL servers at the same time? On all servers but one at a time? On some servers? Are you logging these occurrences to get a handle on it?

by **Op3r** » Tue Nov 20, 2018 10:08 am

timesync means your servers are not in sync with ntp.

time sync gets fixed if you have ntpd running.

Check the load of the servers. everything goes to shit when its overloaded.

Further more long queue length and timesync error may also be connected with your db. Check the health of your db. Separate the webserver from the DB and gain a little more time watching youtube videos during shift.

by **dgroth02** » Tue Nov 20, 2018 10:39 am

Ok - maybe I wasn't completely clear.

With respect to the inbound calls, no they do not (there is ONE server on the cluster that handles inbound calls, but thats it)

NTPD IS RUNNING AND IN SYNC, even when this is happening.

The "load" of any the servers never goes over 40%

The Database server load is consistently at 10% or less.

The servers "go red" in the reports page when this is happening most of the time, but often I will hear of problems from the agent (can't log in - no "you are the only one in this conference) and dialing issues long before that happens.

As far as logging goes, yes I have logging enabled across all the servers. However, with a thousand calls simmultaneously and a hundred agents, where would you suggest I begin looking for this issue. I've got through the asterisk log - the Exceptionally long queue length errors just appear - no error precedes them.

by **Op3r** » Tue Nov 20, 2018 1:21 pm

dgroth02 wrote:Ok - maybe I wasn't completely clear.

With respect to the inbound calls, no they do not (there is ONE server on the cluster that handles inbound calls, but thats it)

NTPD IS RUNNING AND IN SYNC, even when this is happening.

--- Dont think so. Time Sync comes when ntpd is not running on the all the servers. Double check it. The resolution of that issue lies there.

The "load" of any the servers never goes over 40%

The Database server load is consistently at 10% or less.

-- Check for locks and waiting.

The servers "go red" in the reports page when this is happening most of the time, but often I will hear of problems from the agent (can't log in - no "you are the only one in this conference) and dialing issues long before that happens.

-- Dmesg. something's wrong with your server.

As far as logging goes, yes I have logging enabled across all the servers. However, with a thousand calls simmultaneously and a hundred agents, where would you suggest I begin looking for this issue. I've got through the asterisk log - the Exceptionally long queue length errors just appear - no error precedes them.[/quote]
-- Just told you to look at the locks on the db.

remove the webserver with the db. remove the crontab too.

oh btw check at your crontabs too. People are so used at using the automated scripts now allowing people to install "cluester" install without checking out if it is installed properly or not.

by **williamconley** » Tue Nov 20, 2018 3:25 pm

Op3r wrote:timesync means your servers are not in sync with ntp.

time sync gets fixed if you have ntpd running.

You havin' a bad day? Neither of those statements is technically true, and I'm certain you are aware of it.

Time Sync errors can also be caused by anything that blocks the agent session's "every second" packet from updating a specific value in the DB. This can be caused by overloaded workstations, bad networking, even a crashed "screen" on the dialer.

Time sync can still be OFF even with ntp running if ntp is not configured properly OR (in some wacky circumstances) has decided that it doesn't want to sync properly during startup. We've had several machines (some still in use) that must have ntp restarted about a minute after boot to "reset" time sync so it will actually sync instead of "syncing with itself" which doesn't keep it in sync with the DB server for the cluster. Odd, but this is linux. Strange shit happens and a blanket "NTPD is running so time sync must be fixed" is not a good approach.

Just like politics: Trust but verify.

Code: Select all: ntpq -p

This should show sync symbols in the far left column indicating sync with the master (or internet masters) and an offset in single digits. Anything else could be out of sync.

by **Op3r** » Tue Nov 20, 2018 4:56 pm

Hahaha, Yep i know that ntpd is not always the case however the most common error that throws out timesync is actually due to ntpd not running or not syncing properly even when you thought its running. restarting ntpd service does the trick. If it doesnt then something more drastic is about to be done cos production time comes first than dicking around with finding out why shit is broken.

Considering he is on a hosted dedicated server environment, I would really advise him to check out his ntp connectivity. Most likely its not connecting to the pool.ntp.org or whatnot.

If its not the ntpd causing the problem then time to freaking reinstall VICIDIAL and on another set of boxes. That cluster got issues that are not worth the trouble troubleshooting and much faster resolution comes from reinstalling everything (ask the ops a single day of downtime or hindered output cost more than the freaking cost of reinstall.)

by **frequency** » Tue Nov 20, 2018 10:45 pm

Normally it takes place if one of the applications to take load and it crashes. i have seen multiple instances with 11.25.1 where asterisk takes the load, the load avg's haywire like 100 and time sync error and server needs to be rebooted. 11.25.3 is a bit stable for what i have seen.

also, are you taking inbound calls? webrtc?

by **dgroth02** » Wed Nov 21, 2018 11:25 am

I have had this problem on both Asterisk 11.25.1 and 11.25.3

As I stated before, there is only 1 server in this cluster taking inbound calls - and it just so happens it is not the servers having these problems. It seems to be the servers with agents on them that are having this issues.

Additionally, I checked ntpq -p

during production and received the following:

Code: Select all: remote refid st t when poll reach delay offset jitter ============================================================================== +pyrrha.fi.muni. 147.231.2.6 2 u 849 1024 377 167.201 1.929 1.261 -ns3.switch.ca 206.108.0.134 3 u 695 1024 377 73.601 9.841 1.123 *t2.time.gq1.yah 10.211.121.37 2 u 908 1024 377 33.333 -0.320 1.277 +t1.time.bf1.yah 98.139.133.62 2 u 940 1024 377 79.310 -4.701 1.287

However:

I did manage to capture a dmesg and am seeing a ton of

Code: Select all: [710303.849121] net_ratelimit: 96 callbacks suppressed [710758.909989] net_ratelimit: 143 callbacks suppressed [710989.582516] net_ratelimit: 241 callbacks suppressed [710994.622822] net_ratelimit: 242 callbacks suppressed [711234.727354] net_ratelimit: 51 callbacks suppressed [711524.170307] net_ratelimit: 58 callbacks suppressed [714007.497591] net_ratelimit: 167 callbacks suppressed [770058.038706] net_ratelimit: 64 callbacks suppressed [770617.699543] net_ratelimit: 115 callbacks suppressed [771170.784027] net_ratelimit: 121 callbacks suppressed [771678.128427] net_ratelimit: 81 callbacks suppressed [772783.721849] net_ratelimit: 57 callbacks suppressed

I know this has to do with number of logged network errors, but is it possible that a network issue is causing this?

These are all Dell servers - identical.

by **dgroth02** » Thu Nov 29, 2018 2:56 pm

Anyone????

by **williamconley** » Thu Nov 29, 2018 3:16 pm

dgroth02 wrote:but is it possible that a network issue is causing this?

Yes. As stated earlier:

Time Sync errors can also be caused by anything that blocks the agent session's "every second" packet from updating a specific value in the DB. This can be caused by overloaded workstations, bad networking, even a crashed "screen" on the dialer.

The Vicidial Agent Screen uses AJAX to send/receive a packet every second. That packet doesn't just collect information, it also causes a value to be updated in the DB. Without that value update: "Time Sync Error" is tossed. Thus networking or any other crash during that process from agent web page to db update can cause time sync error.

This could be DB socket shortfall, code errors in any of the scripts, literally anything in that path.

This error exists to make it a requirement that communication from the agent to the DB is impeccable, as anything short of perfect WILL lead to incorrect data storage and missing data for the agent. If your networking packets are dropping, you'll have to fix it to get rid of that error. Even "slow" is acceptable: just NOT dropped packets.

by **dgroth02** » Thu Nov 29, 2018 4:46 pm

I get that about the Time Sync error - but that doesn't explain the large string "Exceptionally Long Queue Length" errors that get thrown and keep going and cause the server not to properly work (i.e. agents can't login, calls can't be made on that server) until I reboot it. That causes a server to "go red", then throw timesync errors for agents on that server.

by **williamconley** » Thu Nov 29, 2018 6:09 pm

dgroth02 wrote:I get that about the Time Sync error - but that doesn't explain the large string "Exceptionally Long Queue Length" errors that get thrown and keep going and cause the server not to properly work (i.e. agents can't login, calls can't be made on that server) until I reboot it. That causes a server to "go red", then throw timesync errors for agents on that server.

So fix your networking problem causing the time sync error (if there is one) and hope that the network error is also causing the queue issue.

Also:

http://forums.asterisk.org/viewtopic.php?f=1&t=91780

by **dgroth02** » Fri Feb 22, 2019 12:57 pm

Update on this error

If you review the above thread, you'll see that:
-Timesync is set to DB server, DB server set to sync to pool.ntp.org (and all servers are in sync - verified while working, and at time of "crash")
-servers meet specs
-It would appear that a possible deadlock is occurring
-I've tried changing kernel parameters, max open files, patching Asterisk from 11.25.1-vici to 11.25.3, all to no avail (although changing kernel parameters did improve things slightly)

I have been tracking the reboots and trying to capture the status of Asterisk threads when it happens. I've noticed a curious thing once a server is reported as non-responsive:
The number of IAX2 threads active is high (approximately 60% of them are active and not progressing), unlike on the "functioning" servers where the IAX2 threads move quickly from Active to Idle (most I can't catch as active).

Has anyone seen this or is this just a symptom of a deadlock?

Since the vicibox install does not include the dont_optimize flags by default, i really hate to compile asterisk and get into a "Schrodinger's cat" scenario - but if at the end of the day that is the recourse, then I guess that's the route I will take.

Has ANYONE else experienced this? Feedback would be appreciated.

by **dgroth02** » Fri Feb 22, 2019 4:21 pm

A sample of what I'm referring to:

Code: Select all: asterisk -rx 'iax2 show threads' IAX2 Thread Information Idle Threads: Thread 44: state=0, update=4, actions=757903, func='' Thread 6: state=0, update=4, actions=757869, func='' Thread 2: state=0, update=4, actions=757793, func='' Thread 8: state=0, update=4, actions=757777, func='' Thread 49: state=0, update=4, actions=757831, func='' Thread 38: state=0, update=4, actions=757739, func='' Thread 14: state=0, update=4, actions=757889, func='' Thread 41: state=0, update=4, actions=757868, func='' Thread 35: state=0, update=4, actions=757868, func='' Thread 3: state=0, update=4, actions=757814, func='' Thread 27: state=0, update=4, actions=757881, func='' Thread 1: state=0, update=4, actions=757874, func='' Thread 4: state=0, update=4, actions=757961, func='' Thread 18: state=0, update=4, actions=757896, func='' Thread 20: state=0, update=3, actions=757836, func='' Thread 48: state=0, update=3, actions=757845, func='' Thread 29: state=0, update=3, actions=757839, func='' Thread 33: state=0, update=3, actions=757903, func='' Thread 36: state=0, update=3, actions=757846, func='' Thread 47: state=0, update=3, actions=757859, func='' Thread 13: state=0, update=2, actions=757907, func='' Thread 39: state=0, update=2, actions=757859, func='' Thread 34: state=0, update=2, actions=757834, func='' Thread 25: state=0, update=2, actions=757818, func='' Thread 19: state=0, update=2, actions=757800, func='' Thread 10: state=0, update=2, actions=757860, func='' Thread 5: state=0, update=2, actions=757801, func='' Thread 24: state=0, update=2, actions=757852, func='' Thread 16: state=0, update=2, actions=757838, func='' Thread 32: state=0, update=1, actions=757810, func='' Thread 15: state=0, update=1, actions=757881, func='' Thread 45: state=0, update=1, actions=757808, func='' Thread 11: state=0, update=1, actions=757837, func='' Thread 46: state=0, update=1, actions=757839, func='' Thread 43: state=0, update=1, actions=757843, func='' Thread 22: state=0, update=1, actions=757866, func='' Thread 21: state=0, update=0, actions=757909, func='' Thread 9: state=0, update=0, actions=757894, func='' Thread 12: state=0, update=0, actions=757808, func='' Active Threads: Thread P30: state=1, update=75, actions=757793, func='socket_process' Thread P28: state=1, update=90, actions=757722, func='socket_process' Thread P37: state=1, update=98, actions=757757, func='socket_process' Thread P26: state=1, update=103, actions=757771, func='socket_process' Thread P42: state=1, update=106, actions=757779, func='socket_process' Thread P50: state=1, update=114, actions=757732, func='socket_process' Thread P40: state=1, update=134, actions=757609, func='socket_process' Thread P31: state=1, update=137, actions=757607, func='socket_process' Thread P7: state=1, update=139, actions=757504, func='socket_process' Thread P23: state=1, update=140, actions=757553, func='socket_process' Thread P17: state=1, update=141, actions=757522, func='socket_process' Dynamic Threads:

Code: Select all: asterisk -rx 'core show threads' Setting max files open to 65000 0x7fb2a7013700 5050 netconsole started at [ 1491] asterisk.c listener() 0x7fb2a708f700 4994 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a710b700 4989 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7187700 4984 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7203700 4978 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a727f700 4964 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a72fb700 4955 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7377700 4950 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a73f3700 4946 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a746f700 4941 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a74eb700 4935 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7567700 4931 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a75e3700 4926 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a765f700 4922 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a76db700 4917 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7757700 4913 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a77d3700 4799 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a784f700 4633 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a78cb700 4574 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7947700 4551 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a79c3700 4482 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7a3f700 4452 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7abb700 4423 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7b37700 4412 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7bb3700 4392 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7c2f700 4390 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7cab700 4387 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7d27700 4386 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7da3700 4378 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7e1f700 4374 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7e9b700 4369 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7f17700 4364 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a7f93700 4358 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a800f700 4354 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a808b700 4348 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8107700 4344 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8183700 4341 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a81ff700 4340 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a827b700 4334 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a82f7700 4329 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8373700 4325 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a83ef700 4323 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a846b700 4322 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a84e7700 4314 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8563700 4309 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a85df700 4303 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a865b700 4293 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a86d7700 4282 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8753700 4262 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a87cf700 4248 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a884b700 4245 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a88c7700 4244 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8943700 4236 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a89bf700 4204 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8a3b700 4178 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8ab7700 4177 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8b33700 4171 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8baf700 4168 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8c2b700 4167 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8ca7700 4162 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8d23700 4159 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8d9f700 4155 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8e1b700 4132 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8e97700 4129 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8f13700 4125 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a8f8f700 4121 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a900b700 4117 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9087700 4113 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9103700 4111 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a917f700 4107 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a91fb700 4106 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9277700 4099 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a92f3700 4096 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a936f700 4094 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a93eb700 4089 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9467700 4085 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a94e3700 4082 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a955f700 4079 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a95db700 4076 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9657700 4072 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a96d3700 4071 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a974f700 4068 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a97cb700 4064 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9847700 4061 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a98c3700 4060 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a993f700 4054 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a99bb700 4053 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9a37700 4048 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9ab3700 4042 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9b2f700 4040 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9bab700 4037 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9c27700 4034 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9ca3700 4030 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9d1f700 4025 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9d9b700 4023 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9e17700 4017 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9e93700 4014 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9f0f700 4013 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2a9f8b700 4012 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa007700 4007 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa083700 4003 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa0ff700 4000 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa17b700 3997 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa1f7700 3993 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa273700 3992 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa2ef700 3991 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa36b700 3984 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa3e7700 3981 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa463700 3978 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa4df700 3974 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa55b700 3971 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa5d7700 3968 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa653700 3964 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa6cf700 3961 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa74b700 3958 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa843700 3955 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aaa33700 3951 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aaaaf700 3950 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aaf0b700 3947 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab003700 3944 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab1f3700 3940 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab2eb700 3936 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab367700 3935 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab3e3700 3929 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab45f700 3928 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab4db700 3924 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab7c3700 3921 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3881bc700 3918 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388238700 3915 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3882b4700 3911 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3883ac700 3910 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388694700 3903 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388ce0700 3899 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388d5c700 3898 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389044700 3895 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3891b8700 3891 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389614700 3888 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389690700 3883 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389cdc700 3881 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a0bc700 3877 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a138700 3876 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a2ac700 3869 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a3a4700 3864 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38acd8700 3861 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a8f8700 3857 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab747700 3856 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a974700 3854 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38abe0700 3851 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38ad54700 3850 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38878c700 3846 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38aec8700 3842 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b1b0700 3841 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38ba8f700 3840 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389424700 3835 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abc1f700 3834 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388fc8700 3830 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aac9f700 3827 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38ab64700 3824 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388428700 3821 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388710700 3817 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388618700 3814 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389f48700 3813 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a1b4700 3810 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abfff700 3806 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a68c700 3803 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abba3700 3800 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a87c700 3797 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388808700 3796 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389be4700 3795 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3889f8700 3787 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3892b0700 3786 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aac23700 3781 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38ac5c700 3778 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b0b8700 3775 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3899f4700 3771 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a230700 3770 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b22c700 3767 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389dd4700 3763 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb3893a8700 3761 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b704700 3752 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389234700 3751 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3894a0700 3745 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab937700 3742 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38913c700 3741 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aae13700 3736 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38bb87700 3729 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38932c700 3684 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38af44700 3683 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388ed0700 3678 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aae8f700 3677 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38970c700 3668 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388884700 3664 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aaf87700 3663 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b590700 3644 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aba2f700 3642 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b498700 3638 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a420700 3637 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389978700 3633 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abe0f700 3629 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b2a8700 3628 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388a74700 3620 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b41c700 3619 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a594700 3615 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb3880c4700 3614 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a9f0700 3611 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388be8700 3610 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab64f700 3591 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aa93b700 3590 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab0fb700 3579 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388b6c700 3578 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38951c700 3569 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab83f700 3568 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab5d3700 3558 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389598700 3557 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388330700 3553 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abe8b700 3548 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abd17700 3528 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a518700 3527 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389880700 3524 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aaba7700 3523 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38859c700 3521 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a328700 3520 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389ecc700 3515 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb3898fc700 3511 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aad97700 3510 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38bb0b700 3499 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b03c700 3498 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abb27700 3495 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab07f700 3494 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abc9b700 3489 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aab2b700 3488 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388c64700 3485 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a040700 3481 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab9b3700 3480 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389a70700 3471 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388af0700 3457 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aa7c7700 3456 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38bf07700 3447 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b134700 3442 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389fc4700 3441 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389c60700 3421 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb3884a4700 3420 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab8bb700 3406 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388520700 3405 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38afc0700 3384 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388e54700 3381 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aa8bf700 3380 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388140700 3366 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38897c700 3359 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38aae8700 3358 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389d58700 3344 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab557700 3343 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2aa9b7700 3331 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a708700 3330 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388900700 3299 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b780700 3298 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389788700 3285 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38ae4c700 3284 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2ab177700 3281 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab6cb700 3280 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a49c700 3276 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b324700 3275 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb2abf07700 3267 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389804700 3262 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2aad1b700 3251 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38add0700 2497 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389e50700 2496 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abd93700 2369 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38aa6c700 888 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2ab26f700 32378 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb388dd8700 2352 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b60c700 19005 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abaab700 16992 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb2abf83700 5525 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a800700 3843 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38a784700 7372 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38a610700 8926 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb389aec700 6301 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b688700 4499 pbx_thread started at [ 7085] pbx.c ast_pbx_start() 0x7fb38b7fc700 13876 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb389b68700 8882 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b514700 3252 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb3890c0700 24780 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb388f4c700 23723 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38b3a0700 1683 autoservice_run started at [ 228] autoservice.c ast_autoservice_start() 0x7fb38bc03700 1608 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38be8b700 1606 handle_tcptls_connection started at [ 745] tcptls.c ast_tcptls_server_root() 0x7fb38bf83700 1439 monitor_sig_flags started at [ 4383] asterisk.c asterisk_daemon() 0x7fb38bfff700 1438 lock_broker started at [ 524] func_lock.c load_module() 0x7fb4780d5700 1437 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 0x7fb478151700 1436 scan_thread started at [ 921] pbx_spool.c load_module() 0x7fb4781cd700 1435 network_thread started at [12663] chan_iax2.c start_network_thread() 0x7fb478249700 1434 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4782c5700 1433 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478341700 1432 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4783bd700 1431 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478439700 1430 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4784b5700 1429 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478531700 1428 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4785ad700 1427 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478629700 1426 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4786a5700 1425 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478721700 1424 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47879d700 1423 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478819700 1422 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478895700 1421 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478911700 1420 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47898d700 1419 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478a09700 1418 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478a85700 1417 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478b01700 1416 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478b7d700 1415 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478bf9700 1414 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478c75700 1413 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478cf1700 1412 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478d6d700 1411 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478de9700 1410 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478e65700 1409 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478ee1700 1408 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478f5d700 1407 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb478fd9700 1406 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479055700 1405 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4790d1700 1404 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47914d700 1403 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4791c9700 1402 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479245700 1401 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4792c1700 1400 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47933d700 1399 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4793b9700 1398 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479435700 1397 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4794b1700 1396 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47952d700 1395 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4795a9700 1394 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479625700 1393 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb4796a1700 1392 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47971d700 1391 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479799700 1390 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479815700 1389 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479891700 1388 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb47990d700 1387 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479989700 1386 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479a05700 1385 iax2_process_thread started at [12641] chan_iax2.c start_network_thread() 0x7fb479a81700 1384 sched_run started at [ 203] sched.c ast_sched_start_thread() 0x7fb479afd700 1383 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 0x7fb479c7b700 1382 do_monitor started at [29497] chan_sip.c restart_monitor() 0x7fb479cf7700 1381 do_monitor started at [12303] chan_dahdi.c restart_monitor() 0x7fb47a574700 1379 do_timing started at [ 490] res_timing_pthread.c init_timing_thread() 0x7fb4c86a9700 1378 sched_run started at [ 203] sched.c ast_sched_start_thread() 0x7fb4c8725700 1377 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 0x7fb4c87a1700 1376 do_parking_thread started at [ 9093] features.c ast_features_init() 0x7fb4c881d700 1375 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 0x7fb4c8899700 1374 do_presence_changes started at [ 318] presencestate.c ast_presence_state_engine_init() 0x7fb4c8915700 1373 do_devstate_changes started at [ 772] devicestate.c ast_device_state_engine_init() 0x7fb4c8991700 1372 desc->accept_fn started at [ 1051] tcptls.c ast_tcptls_server_start() 0x7fb4c8a42700 1371 do_refresh started at [ 490] dnsmgr.c do_reload() 0x7fb4c8abe700 1370 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 0x7fb4c8b3a700 1369 db_sync_thread started at [ 1018] db.c astdb_init() 0x7fb4cae69700 1368 logger_thread started at [ 1312] logger.c init_logger() 0x7fb4caee5700 1367 listener started at [ 1551] asterisk.c ast_makesocket() 0x7fb4caf61700 1366 tps_processing_function started at [ 472] taskprocessor.c ast_taskprocessor_get() 360 threads listed.

by **jshasteen** » Sat Feb 23, 2019 7:22 pm

I'm experiencing the same thing with 3 of my telephony servers. 2 servers running 11.25.1-vici, and 1 running 11.25.3-vici. Asterisk starts throwing the error you mentioned then crashes. Happens multiple times per day and it just started about a week ago. Servers are not overloaded and I only keep around 12 - 15 agents on each telephony server. I would love to figure out what is causing this and how to fix it.

by **williamconley** » Sun Feb 24, 2019 12:03 am

have you turned off any unneeded asterisk modules with noload (in modules.conf) ?

Code: Select all: noload => res_config_sqlite.so noload => res_config_sqlite3.so noload => cdr_sqlite.so noload => cel_sqlite3_custom.so noload => cdr_sqlite3_custom.so noload => chan_ooh323.so

by **dgroth02** » Sun Feb 24, 2019 11:18 pm

Yes,

Verified those were already turned off.

FYI - I upgraded 3 of the 10 dialers to version 8.1.2 of Vicibox tonight (same SVN 3051 of database/scripts) - one agent server, two dialing servers - will see how this progresses.

I also managed to get a PCAP of the same test phone call to two separate carriers. I'm also traveling down the rabbit hole of malformed SIP packets (even though only the carrier and remote agent IP's are whitelisted, all others blocked). In two different multi-server installations, one has far fewer instances of this happening, so I'm going to examine that as well.

I will keep posting back. I've used this forum for a long time, my turn to start contributing

by **dgroth02** » Tue Feb 26, 2019 1:38 pm

SUCCESS!

Yesterday there were none of the issues described on the 3 servers I upgraded to 8.1.2. So last night I upgraded to version 8.1.2 on all dialers (except the web/database server - still a bit gunshy on that one), and so far, not a single reboot or error as described. As luck would have it - the carrier went down today and the owner freaked, but I was able to easily trace it, and it automatically came back up once the carrier did and the servers kept right on plugging along.

Everyone keep their fingers crossed.

by **williamconley** » Tue Feb 26, 2019 11:33 pm

Excellent postback. Especially if there are No More Posts. 8-)

by **dgroth02** » Wed Feb 27, 2019 4:21 pm

Well - shit

The upgrade to Asterisk 13 stopped the Exceptionally Long Queue length and the Asterisk deadlocks.

However the Vicibox 8.1.2 upgrade has caused additional issues.

At this point - about 1/3rd of the calls come through with no audio - its frustrating since we have to run with a higher dial rate in order to get throughput. Answering machine detection? Transferring dead calls? RTP is working correctly.

Also - agents are trying to leave 3-way calls and they intermittently have issues doing so.

So - one problem solved - two more in its place.

by **williamconley** » Wed Feb 27, 2019 4:30 pm

1/3rd of the calls

would that be all the calls on one dialer?

by **rfugate** » Wed Feb 27, 2019 4:38 pm

Has ANYONE else experienced this? Feedback would be appreciated.

I've been dealing with this exact issue myself and have been anxiously following this thread. I've been lurking the forums for years and this is my first comment.

I'm running VERSION: 2.14-585a BUILD: 170114-1356 installed from a 7.0.3 iso on a clustered install (6 dialers (all agent), separate web and db). I have another cluster with this version, only one cluster seems problematic. The hardware specifications for some of the dialers on the 'problem cluster' are not as robust as my other less-utilized cluster. My problem dialers utilize either Digium T1 hardware or Amfeltec timing cards.

I've been upgrading servers, upgrading network gear, rewiring my network, and despite minimal dialer load (10 or so active agents per dialer with low cpu %), I'm going to lighten the dialer's burdens by shifting agents onto less-used dialers.

My ntp is configured and all servers stay well synced. When the problem occurs (seems to be much more frequent lately, at least once daily on at least one dialer whereas late last year it would be once weekly tops), I've been resolving it with the sequence of "killall screen, killall perl, /etc/init.d/vicidial restart" to quickly fix the problem, as that set of commands is much faster than a server reboot. Like you mentioned, there's no errors preceding the asterisk errors of Long Queue Length and Timesync in the asterisk logs. I don't see any missing screens, the ast-update one is present.

I'm VERY curious to see if you continue to be problem-free with the upgrade to 8.1.2.

by **dgroth02** » Wed Feb 27, 2019 5:57 pm

Nope - the calls are spread across all dialers. Tested myself - one call from a particular server would fail, another from the same server would be just fine.

At lunch I rebooted the main DB/web server and re-installed all the scripts/rebuilt the config files and re-checked NTP (of course was fine). No change in the issue.

I ran across another thread that mentioned AMD...it is a finicky beast. My best guest is the calls are being answered by someone who goes "shit" then hangs up the call mid-transfer. When I reduced the "drop call seconds" from 5 to 3 - that seemed to help greatly, but then when the agents complained they couldn't make a 3 way transfer, so I put it back again. Strangely enough, when I put it back, the issue is now gone (at least for the moment) - will be tracking it.

by **williamconley** » Wed Feb 27, 2019 6:21 pm

firewall. overloaded server. overloaded carrier. do these servers SHARE a firewall?

by **dgroth02** » Wed Feb 27, 2019 11:39 pm

Yes, an enterprise class firewall - have also put the servers OUTSIDE the firewall - makes no difference.

As far as an overloaded server - all the servers have < 10% load

Tonight I separated the web and DB server - the web server is a dual hex core Xeon @ 3.3ghz with 32GB Ram and SSD drives. - Will see if that takes the load off

by **williamconley** » Thu Feb 28, 2019 12:03 am

dgroth02 wrote:Yes, an enterprise class firewall - have also put the servers OUTSIDE the firewall - makes no difference.

As far as an overloaded server - all the servers have < 10% load

Tonight I separated the web and DB server - the web server is a dual hex core Xeon @ 3.3ghz with 32GB Ram and SSD drives. - Will see if that takes the load off

i'm gonna suggest you try it again and leave them outside the firewall while you work out why individual calls don't have audio. Don't try to fix all of them. Fix ONE. You may have multiple problems and if the firewall is only one of the problems ... well, you see where I'm goin with this, right? Next up we have anything else "shared". overloaded switches, routers, anything else. But try to track down and fix one call at a time. You'll likely find each one you fix represents and resolves a Group and you may well have several problems.

When I reduced the "drop call seconds" from 5 to 3 - that seemed to help greatly, but then when the agents complained they couldn't make a 3 way transfer

Those two items are unrelated. It sounds like you were overloading, though and it wore off later. Try Zero seconds and drop to an ingroup with someone whose job it is to say "Hi, wanna buy a widget?" and hang up when they say no. At least it wasn't a drop, perfectly legal (freaky if you make a sale doing that, but 99% of the calls will end almost immediately after the "wanna buy")

Zero seconds changes the dynamics of the call generation system. Can spread out the load rather nicely.

Happy Hunting 8-)

vicidial.org

Who is online