Hotkeys & Hangup

[mflorell]

listen running twice should not cause that behaviour, although it would not hurt to kill one of the instances. I have fixed that bug in the 2.0 codetree.

Watch the AST_update.pl script to see if it does die today at all when you are having problems with these hangups.

[enjay]

Ok I'll keep an eye on it if it dies within the first couple hundred errors I will let you know.

[enjay]

Several Errors have happened AST_update is not dying however I am seeing this.

Code: Select all

SIP Clients:  |admin|admin@admin|200|200@200|209|209@209|210|210@210|211|211@211|212|212@212|213|213@213|214|214@214|215|215@215|216|216@216|217|217@217|218|
-2006-08-14 10:03:48   101.23    82    38:39      41:42    0|0:0    0|0:0    100.00|32:32    100.00|10:10
-2006-08-14 10:03:53   101.23    82    38:39      41:42    0|0:0    0|0:0    100.00|32:32    100.00|10:10
-2006-08-14 10:03:58   101.27    80    36:37      41:42    0|0:0    0|0:0    100.00|32:32    100.00|10:10
-2006-08-14 10:04:03    98.70    76    36:37      39:40    0|0:0    0|0:0     96.67|29:30    100.00|10:10
-2006-08-14 10:04:08   100.00    76    34:35      40:41    0|0:0    0|0:0     96.77|30:31    100.00|10:10
------ UPDATER BAD GRAB!!!    UBGcount: 1
            3.39    2    28:29      29:30    0|0:0    0|0:0      5.00|1:20    0|0:10

-2006-08-14 10:04:13    96.61    57    28:29      29:30    0|0:0    0|0:0     90.00|18:20    100.00|10:10
-2006-08-14 10:04:18   101.64    62    28:29      31:32    0|0:0    0|0:0    100.00|22:22    100.00|10:10
-2006-08-14 10:04:23   101.61    63    28:29      32:33    0|0:0    0|0:0    100.00|22:22    100.00|11:11
-2006-08-14 10:04:27    98.11    52    25:26      26:27    0|0:0    0|0:0     93.75|15:16    100.00|11:11



SIP Clients:  |admin|admin@admin|200|200@200|209|209@209|210|210@210|211|211@211|212|212@212|213|213@213|214|214@214|215|215@215|216|216@216|217|217@217|218|
-2006-08-14 10:04:37    98.18    54    26:27      27:28    0|0:0    0|0:0     94.44|17:18    100.00|10:10
-2006-08-14 10:04:42   101.92    53    25:26      25:26    0|0:0    0|0:0    100.00|16:16    100.00|10:10
-2006-08-14 10:04:46   101.92    53    27:28      23:24    0|0:0    0|0:0    100.00|14:14    100.00|10:10
-2006-08-14 10:04:51   101.64    62    30:31      29:30    0|0:0    0|0:0    100.00|20:20    100.00|10:10
-2006-08-14 10:04:56    98.31    58    30:31      27:28    0|0:0    0|0:0     88.89|16:18    100.00|10:10
-2006-08-14 10:05:01    98.15    53    28:29      24:25    0|0:0    0|0:0     85.71|12:14    100.00|11:11
-2006-08-14 10:05:05   101.85    55    28:29      24:25    0|0:0    0|0:0    100.00|14:14    100.00|11:11
-2006-08-14 10:05:10   103.85    54    27:28      23:24    0|0:0    0|0:0    100.00|13:13    109.09|12:11
------ UPDATER BAD GRAB!!!    UBGcount: 1
            6.00    3    26:27      22:23    0|0:0    0|0:0      9.09|1:11      8.33|1:12


[mflorell]

UPDATER BAD GRAB is one indication of a stressed Asterisk server. It is not horrible if it only happens once per minute, but may be a reason for some errors that are occuring. UPDATER BAD GRAB only occurs on very busy systems.

[enjay]

The system is far from stressed in my opinion.. It is under-utilized..

I am seeing 1-4 just about every instance (runthrough) of AST_update.pl

the load is less than 1.0 its at ~0.45

-enjay

[mflorell]

I have a system that is only IAX and Zap channels with the loadavg at 5.0(250%) and I am only seeing the UPDATER BAD GRAB about once every 3 minutes. This server does run Asteirsk 1.0.9 though. Another server that is at laodavg 1.7 (85%) with 1.2.10 doesn't show any UPDATER BAD GRAB.

[enjay]

sentyou a pm

[enjay]

Im up to 2.0.1 and it appears to still be happening

Code: Select all

2006-08-15 8:14:50|0|732309
|
Action: Hangup
Channel:


2006-08-15 8:14:50|1|732309
|Response: Error
Message: No channel specified


+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| man_id | uniqueid  | entry_date          | status | response | server_ip   | channel | action | callerid           | cmd_line_b | cmd_line_c | cmd_line_d | cmd_line_e | cmd_line_f | cmd_line_g | cmd_line_h | cmd_line_i | cmd_line_j | cmd_line_k |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 732309 | 0.0000000 | 2006-08-15 08:14:50 | SENT   | N        | 192.168.0.6 |         | Hangup | RH1234591155654890 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+


[mflorell]

Do the callerIDs of all of those Hangup requests begin with RH12345?

If so they are attempts to hangup recordings, not channels that customers would be on.

I'm not sure why this error would be happening anyway, but we should be able to rule that out as a cause of problems with customers not being hungup.

[enjay]

They Vary

Code: Select all

+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| man_id | uniqueid  | entry_date          | status | response | server_ip   | channel             | action | callerid           | cmd_line_b | cmd_line_c | cmd_line_d | cmd_line_e | cmd_line_f | cmd_line_g | cmd_line_h | cmd_line_i | cmd_line_j | cmd_line_k |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733377 | 0.0000000 | 2006-08-15 10:03:32 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | RH1234591155661412 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733465 | 0.0000000 | 2006-08-15 10:05:14 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | HLvdcW11556615143213 | Channel: Zap/7-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+----------------------+------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733470 | 0.0000000 | 2006-08-15 10:05:17 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | RH1234511556615170 | Channel: Local/78600061@default-e0d2,2 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+----------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733471 | 0.0000000 | 2006-08-15 10:05:17 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | RH1234511556615171 | Channel: Local/78600061@default-87da,2 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+----------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733493 | 0.0000000 | 2006-08-15 10:05:39 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | HLvdcW11556615383213 | Channel: Zap/4-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+----------------------+------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733539 | 0.0000000 | 2006-08-15 10:06:16 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | RH1234511556615760 | Channel: Local/78600061@default-ea15,2 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+----------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733533 | 0.0000000 | 2006-08-15 10:06:13 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | HLvdcW11556615733213 | Channel: Zap/1-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+----------------------+------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733549 | 0.0000000 | 2006-08-15 10:06:27 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | RH1234591155661587 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 733624 | 0.0000000 | 2006-08-15 10:07:34 | DEAD   | N        | 192.168.0.5 | Privilege: call,all | Hangup | HLvdcW11556616505215 | Channel: Zap/52-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------------------+--------+----------------------+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+






[mflorell]

OK, it's the HLvdcW Actions that are causing the problems. And from what I saw from your logs the error sent back on those is that the channel does not exist?

[enjay]

Correct on all of those the HLand the RH are all channel does not exist.

[mflorell]

I would first suggest trying to run for at least an hour with recordings turned off and see if that helps at all.

Also, can you post a screenshot of the server_performance.php page for both of your servers?

[enjay]

turned off recording we'll see how this goes..

If the problem does become the recording what kind of solution is there? throwing more hardware at it?

[mflorell]

The load spikes in the chart are very high and there are a lot of them. I think yor major problem here is load. If you can reduce load, your problems should lessen or go away.

[enjay]

The only thing going on with this system is asterisk/VICIDIAL they are the only things causing load yet they are both imperative.. I've been running norecording for an hour and the system is kicking ass..

I must have recordings so I cant really think of an alternative other than throwing more hardware at this which in turn is another problem

since throwing more servers at this will create issues with "MAX VICIDIAL TRUNKS" as each server must have the capability to dial a pool of ~96 lines.

If I set each server to have a pool of 96 lines and the admin sets a dial level higher than x then each server will try to dial more lines then are actually available at that time and I will start having "NEW CALLED" know what Im saying?

-enjay

[mflorell]

I'm not sure how much it will help, but since yesterday I have added the option on a per-server basis of shutting off the vicidial scripts logging for the majority of the server-side applications and the heavy logging AGI scripts(call_log and VD_hangup).

Check out SVN 2-X from today and replace your scripts with the new ones(make sure you make the database changes in the /extras/upgrade_2.0.1.sql file[there are 3 new fields since 1.1.12-3 release])

Then in the servers admin section of astguiclient/admin.php at the bottom you can shut off server logs and set AGI output to NONE and there will be a lot less writing of logs to those harddrives that are writing your recordings hopefully lowering your load.

Let me know if that helps you at all.

If that doesn't work you may want to consider attempting to do your recordings on a different server from the server your agents are on.

Another option may be to add a delay to the starting of recordings on calls so that calls under 10 seconds for instance would not be recorded. Not sure how that would be done exactly but I think it should be possible.

[enjay]

It is possible to record calls on a different system? I have a system way underutilized that I could do that on. Do you have any details or a link to something on this?

-enjay

[mflorell]

I have only tested it, never in production. One person that is doing that simply changed the 8309 and 8310 extensions to go over IAX to the 8309 and 8310 extensions on the recording system. This way the bottleneck of the drives and the bandwidth of recording are not present on the dialing server, they just see the recording connection as another IAX channel in the conference.

In vicidial the starting of the recording using this method should work fine, but I do not know if vicidial.php would be able to stop the recording automatically as it does now since this currently looks for Local/ channels to Hangup the recording. This would probably require editing of a few lines within vicidial.php and manager_send.php

[enjay]

Ok I will look into this, something tells me a delay before recording might be a very good solution, if all calls are being recorded regardless of (answered or not) then if it recorded only 15 second long calls that would decrease load significantly..

[enjay]

Matt,

Could something as simple as adding a Wait in the 8309 extension help?

original:

Code: Select all

exten => 8309,1,Answer
exten => 8309,2,Monitor(wav,${CALLERIDNAME})
exten => 8309,3,Wait,3600
exten => 8309,4,Hangup


modified:

Code: Select all

exten => 8309,1,Wait(15)
exten => 8309,2,Answer
exten => 8309,3,Monitor(wav,${CALLERIDNAME})
exten => 8309,4,Wait,3600
exten => 8309,5,Hangup


or would this hork if the channel is hangup prior?

-enjay


Additionally: I still receive those "No channel specified" errors however no overlapping calls.. kinda wierd, perhaps that had nothing to do with it..

[mflorell]

Adding a wait might work, but would not be an optimal solution. I have added a feature request to the TODO that I hope to work on this week that would allow you to add a delay to recordings when in ALLCALLS mode.

As for your no-channel-specified errors, what are the vicidial_manager callerIDs for those?

[enjay]

Code: Select all

+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| man_id | uniqueid  | entry_date          | status | response | server_ip   | channel | action | callerid           | cmd_line_b | cmd_line_c | cmd_line_d | cmd_line_e | cmd_line_f | cmd_line_g | cmd_line_h | cmd_line_i | cmd_line_j | cmd_line_k |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 747855 | 0.0000000 | 2006-08-15 14:00:03 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | RH1234591155675603 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 747872 | 0.0000000 | 2006-08-15 14:00:19 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | RH1234591155675619 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 749282 | 0.0000000 | 2006-08-15 15:01:26 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | RH1234591155679286 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 749285 | 0.0000000 | 2006-08-15 15:01:27 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | HLvdcW11556792873213 | Channel: Zap/59-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+----------------------+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 750431 | 0.0000000 | 2006-08-15 15:22:51 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | HLvdcW11556805649209 | Channel: Zap/1-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+----------------------+------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 750674 | 0.0000000 | 2006-08-15 15:27:58 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | HLvdcW11556808614224 | Channel: Zap/60-1 |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+----------------------+-------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
| 751433 | 0.0000000 | 2006-08-15 15:46:43 | DEAD   | N        | 192.168.0.5 | Priv    | Hangup | RH1234591155682003 | Channel:   |            |            |            |            |            |            |            |            |            |
+--------+-----------+---------------------+--------+----------+-------------+---------+--------+--------------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+


[enjay]

and yes that wait didnt make any real difference it works how its supposed to but I still have trouble hanging up calls when its recording..

-enjay

[mflorell]

I'll post when I finish the ALLCALLS recording delay feature.

As for your no-channels, if the callerID of the vicidial_manager record begins with RH1234 then it is an attempted recording hangup.

[enjay]

Gonna try to put a recording host in place tomarrow...

[mflorell]

I looked at it some more last night and you will need to make a couple small changes to your vicidial.php and manager_send.php code for the ending of recordings to work properly. These changes are dependant on the server you will be sending the recording channels to. I will try to look more into it today.

[enjay]

I went ahead and changed it anyways it appears to be working so far, If you do find out what needs to be changed in those files please let me know. I will monitor this for a few hours and let you know what comes of it.

-enjay

[enjay]

Yea still didnt resolve the problem, with recording turned off hangups happen pronto with them turned on they take 5-10 seconds to hangup still..

[enjay]

At this point Im at a loss for what to do. I debate moving to slackware and trying that but I dont know if thats even worth the time as I doubt it would resolve my problem completely..

I debate throwing more hardware at it but then I'll just run into configuration issues (pool of phonelines). Any recommendations from anyone would be appreciated..

-enjay

[mflorell]

Would it be an acceptable solution to have recordings start 5 or 10 seconds after the call is sent to an agent?

If so I am working on that and should have some code ready for testing in the 2-X SVN tree today or tomorrow.

[enjay]

sure, but when I added that wait in the extensions.conf it worked fine but the load would still double and hangups would not happen so Im not sure that it would help much.

feel the negativity

[mflorell]

Changing it in the vicidial.php code would be different from changing it in the extensions.conf. For instance if it's an answering machine and you are using hotkeys in ALLCALLS mode you would send the signal to start the recording and create a new channel the way it is now.

Adding a delay in the vicidial.php code would not send the signal until after a timer so less traffic on the server.

[enjay]

Gotcha well Im definately without a fraction of a doubt willing to give it a shot

Also another thing I was thinking how possible would it be to not place the agent back into READY state until the channel HAS been hungup completely?

-enjay

[mflorell]

I have added the allcalls_delay feature to the SVN 2-X code tree. just update your DB(upgrade_2.0.1.sql) your vicidial/admin.php page and your vicidial.php page and it should work fine. I have tested the code and it will not issue the recording command for X seconds.

Let me know how it works for you.

As for the wait-for-hangup, the agent can use the "Hangup again" link on the disposition page if it is a big problem. Other than that, it is really a system load issue that will cause other issues even if this one is fixed.

[enjay]

updated, I will let you know how it goes.

[enjay]

It appears to have definately helped.. I will continue to keep an eye on it into tomarrow morning and let you know whats up..

BTW Thankyou so much Matt for getting that implemented so quickly I know it was on your list of todo's but Im sure my whining had something to do with it getting done so quick..

Again thanks for all that you do for us here at VICIcommunity

-enjay

[mflorell]

I'm glad we could figure something out for your problem. What do you have the delay set to?

It was really pretty easy to do, it actually took longer to add the admin.php portion to be able to edit the new database field than it was to edit the vicidial.php code to actually implement the delay.

[enjay]

I set the delay to 25 I will play with a few options this one seems the most effective at this time.

A. Agent is just about finished with their initial 'hello this is rah from rah Im calling in regards to yadda yadda'

B. When sending DTMF the recording kicks on just before I get a live caller or a answering machine.

-enjay