Agent dead after hangup with customer BOUNTY

[Cyberwolf2]

I'm currently chasing my tail on this one. I migrated to an IONOS dedicated server due to this. Any direction appreciated, willing to pay to get this running. My agent dies after hangup with customer. Hotkeys start working fine then they start lagging, and then it's a full stop on the dialer. The agent screen is ready, however in the report page the agent goes into a dead status and never recovers until the agent logs out and back in. I've tried previously directly routing to the server when it was hosted in the same office, now that it's in IONOS.

Here is my asterisk outout:

Code: Select all

Asterisk Ready.
*CLI> [Dec 28 18:06:43] NOTICE[176395]: chan_sip.c:25047 handle_response_peerpoke: Peer 'SkyetelCA' is now Reachable. (52ms / 2000ms)
[Dec 28 18:06:43] NOTICE[176395]: chan_sip.c:25047 handle_response_peerpoke: Peer 'SkyetelOUT' is now Reachable. (37ms / 2000ms)
[Dec 28 18:06:43] NOTICE[176395]: chan_sip.c:25047 handle_response_peerpoke: Peer 'SkyetelTERM' is now Reachable. (51ms / 2000ms)
[Dec 28 18:06:43] NOTICE[176395]: chan_sip.c:25047 handle_response_peerpoke: Peer 'SkyetelOR' is now Reachable. (66ms / 2000ms)
[Dec 28 18:06:47] NOTICE[176395]: chan_sip.c:30633 sip_poke_noanswer: Peer '1001' is now UNREACHABLE!  Last qualify: 0
[Dec 28 18:06:48] NOTICE[176365]: chan_iax2.c:11308 socket_process_helper: Peer 'ASTplay' is now TOO LAGGED (5500 ms)!
[Dec 28 18:06:52] NOTICE[176434]: chan_sip.c:25047 handle_response_peerpoke: Peer '1001' is now Reachable. (48ms / 2000ms)
[Dec 28 18:06:58] NOTICE[176367]: chan_iax2.c:11287 socket_process_helper: Peer 'ASTplay' is now REACHABLE! Time: 3
[Dec 28 18:07:03] WARNING[176571][C-00000002]: func_hangupcause.c:138 hangupcause_read: Unable to find information for channel
[Dec 28 18:07:03] WARNING[176449][C-00000001]: func_hangupcause.c:138 hangupcause_read: Unable to find information for channel
[Dec 28 18:07:12] WARNING[176612]: res_rtp_asterisk.c:3234 __rtp_recvfrom: 1735427232.7: DTLS packet from 102.129.235.219:65008 dropped. ICE not completed yet.
1735427271.157079
1|cid_channels_recent_104254254095|SHOW TABLES LIKE "cid_channels_recent_104254254095";
SELECT dest_channel FROM cid_channels_recent_104254254095 where (caller_id_name = 'V2281807280000476843' or connected_line_name = 'V2281807280000476843' or caller_id_name = 'V2281807280000476843' or connected_line_name = 'V2281807280000476843') and ( linkedid = '1735427248.16' or dest_uniqueid = '1735427248.16' or uniqueid = '1735427248.16') and call_date > '2024-12-28 18:05:51'
55|SIP/SkyetelOUT-00000004
INSERT INTO vicidial_manager values('','','2024-12-28 18:07:51','SENT','N','IP_REDACTED','','Redirect','V2281807280000476843','Channel: SIP/SkyetelOUT-00000004','Context: default','Exten: 8368','Priority: 1','','','','','','')
/usr/share/astguiclient/AST_send_action_child.pl --SYSLOG --PATHlogs=/var/log/astguiclient --telnet_host=localhost --telnet_port=5038 --ASTmgrUSERNAME=cron --ASTmgrSECRET=1234 --ASTmgrUSERNAMEsend=sendcron --man_id=1184 --action=Redirect --cmd_line_b='Channel: SIP/SkyetelOUT-00000004' --cmd_line_c='Context: default' --cmd_line_d='Exten: 8368' --cmd_line_e='Priority: 1' >> /var/log/astguiclient/action_send.2024-12-28
[Dec 28 18:07:51] WARNING[176659][C-00000007]: func_hangupcause.c:138 hangupcause_read: Unable to find information for channel
1735427277.172076


[Cyberwolf2]

Update

Looks like my DB is screwed, in nginx logs It's complaining regarding missing variables:

Code: Select all

Script   Line   Variable
/var/www/html/agc/conf_exten_check.php   198   $NOW_TIME
/var/www/html/agc/conf_exten_check.php   217   $NOW_TIME
/var/www/html/agc/conf_exten_check.php   230   $exten
/var/www/html/agc/functions.php   74   $DB
/var/www/html/agc/functions.php   120   $DB
/var/www/html/agc/functions.php   123   $mel
/var/www/html/agc/functions.php   183   $DB
/var/www/html/agc/functions.php   203   $DB
/var/www/html/agc/functions.php   213   $DB
/var/www/html/agc/conf_exten_check.php   933   $lead_id
/var/www/html/agc/conf_exten_check.php   1554   $live_agents_comments
/var/www/html/agc/conf_exten_check.php   1749   $lead_id

$lead_id in /var/www/html/agc/conf_exten_check.php, line 933:

Code: Select all

else
                                                        {
                                                        $tempACTION = 'dead_log';
                                                        $TEMPstage = "DEAD call first detect A|$dead_count|";
                                                        vicidial_ajax_log($NOW_TIME,$startMS,$link,$tempACTION,$php_script,$user,$TEMPstage,$lead_id,$session_name,$stmt);          <<< line 933
                                                        }


I'm going to refresh this database and/or downgrading from PHP 8.2 to 7.4

Any suggestions on how you would do it? Am I in the right path here?

[williamconley]

"bounty" is actually a different board here. If you would like I can move this thread to "jobs - request for consultation". I believe there are those who pay close attention to that board for that specific reason. They may not bump into "bounty" in the title, though.

Also since this is a scratch install and appears to possibly be an install-based issue, having the installation instructions in your signature rather than a note that it was "scratch" might be ... useful?

Given the odds, however: You're likely looking at a version issue. Guessing: You performed an install. You copied the DB onto the newly installed server. And you missed a setting required to make the DB compatible with the environment.

We like to fully test fresh installs before overwriting the DB. And once it's certified we back up the system settings and server settings tables before copying the db on top, then we restore the system and server tables. Generally that's where the problem lies.

[njr]

So this happened first pre-migration? Same setup? Did you just move a full image or was it a reinstall? Was this a working system at some point that stopped working? If so, when, and did any changes happen around that time?

What is in vicidial_ajax_log? Does it look relatively appropriate? (generally should alternate between SELECT and UPDATE statements every line, and nothing should look...missing, from the command logged. like maybe $lead_id)

I'm not sure if this is the case, but I'm pretty sure PHP 8.2 throws an error when a variable is uninitialized (and it is for a lot of those in conf_exten_check) where 7.4 would treat as null, so, could be a problem? Was this ever working with PHP 8.2?

[alo]

Did you figure this one out? we just started seeing this after upgrading to php 8.2.
ours is affecting agents hanging up and being set to Dead and we believe its because the start epoch param causing th evdc_db_query.php script to fail.

[carpenox]

i wouldnt doubt its a bug in php 8.2 - whats in the php-fpm error logs?

[mflorell]

I tested this with several manual dial calls on a brand-new VICIbox 12.0.2 install last night(with PHP 8.2.26) and I couldn't replicate the issue. Could you share more details on what kind of calls these were and any other settings that you think may trigger it?

[alo]

These were normal outbound calls from a campaign using ratio dial method and with routing extension 8369.
Any time an agent pressed hangup it was trying to post vdc_db_query.php

We think the issue is with php not setting NULL value for start _epoch

As a duck tape fix, I believe we modified line 9677 on vicidial.php

"&exten=" + recording_exten + "&channel=" + lastcustchannel + "&start_epoch=" + MDlogEPOCH + "&auto_dial_level=" + auto_dial_level
to
"&exten=" + recording_exten + "&channel=" + lastcustchannel + "&start_epoch=0" + "&auto_dial_level=" + auto_dial_level
remove + MDlogEPOCH and set &start_epoch=0

We switched and its working perfectly, but we aren't sure what logging or other features we might have broken doing this as a work around.
As a note, we re-inported the DB from vicibox11 so not sure if there was some setting that needs to be changed for the new PHP version. I didn't see anything in system settings.

[mflorell]

This has been fixed in svn/trunk revision 3900. The fix is in the vdc_db_query.php script, and it will overwrite the start_epoch only if it hasn't been populated or it has non-digits in it. Your posted workaround can result in inaccurate logging.

Please upgrade to the latest code and test.

Thank you very much for reporting!