press 1 campaign not playing message

[perlmutr]

after working for an extended time, 2 of my dialers suddenly will not play a message on a press 1 type campaign when someone answers. my database and 1 dial function properly. There have been no changes to any software.

when i run a test call on oe of the failing servers the log shows:

[Apr 8 09:39:44] VERBOSE[27411][C-00000000] pbx.c: Executing [18136822440@default:1] AGI("Local/18136822440@default-00000000;2", "agi://127.0.0.1:4577/call_log") in new stack
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] res_agi.c: AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=Health))
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] res_agi.c: <Local/18136822440@default-00000000;2>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] pbx.c: Executing [18136822440@default:2] NoOp("Local/18136822440@default-00000000;2", "leadgen1 Dialing thru aka Customer: 18136822440") in new stack
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] pbx.c: Executing [18136822440@default:3] Dial("Local/18136822440@default-00000000;2", "SIP/18136822440@aka,,To") in new stack
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] netsock2.c: Using SIP RTP CoS mark 5
[Apr 8 09:39:44] VERBOSE[27411][C-00000000] app_dial.c: Called SIP/18136822440@aka
[Apr 8 09:39:45] VERBOSE[2452][C-00000000] res_rtp_asterisk.c: 0x7fb64c006510 -- Strict RTP learning after remote address set to: 40.141.203.52:23088
[Apr 8 09:39:45] VERBOSE[27411][C-00000000] app_dial.c: SIP/aka-00000000 is making progress passing it to Local/18136822440@default-00000000;2
[Apr 8 09:39:45] VERBOSE[27410] dial.c: Local/18136822440@default-00000000;1 is making progress
[Apr 8 09:39:45] VERBOSE[27411][C-00000000] res_rtp_asterisk.c: 0x7fb64c006510 -- Strict RTP switching to RTP target address 40.141.203.52:23088 as source
[Apr 8 09:39:47] VERBOSE[27411][C-00000000] app_dial.c: SIP/aka-00000000 answered Local/18136822440@default-00000000;2
[Apr 8 09:39:47] VERBOSE[27410] dial.c: Local/18136822440@default-00000000;1 answered
[Apr 8 09:39:47] VERBOSE[27410][C-00000001] pbx.c: Executing [138366@default:1] AGI("Local/18136822440@default-00000000;1", "agi-VDAD_local_optimize.agi,V4080936310026055683") in new stack
[Apr 8 09:39:47] VERBOSE[27419][C-00000000] bridge_channel.c: Channel SIP/aka-00000000 joined 'simple_bridge' basic-bridge <f85e9baa-6672-4368-b5fd-98e94257501c>
[Apr 8 09:39:47] VERBOSE[27411][C-00000000] bridge_channel.c: Channel Local/18136822440@default-00000000;2 joined 'simple_bridge' basic-bridge <f85e9baa-6672-4368-b5fd-98e94257501c>
[Apr 8 09:39:47] VERBOSE[27410][C-00000001] res_agi.c: Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_local_optimize.agi
[Apr 8 09:39:47] VERBOSE[27410][C-00000001] res_agi.c: <Local/18136822440@default-00000000;1>AGI Script agi-VDAD_local_optimize.agi completed, returning 0
[Apr 8 09:39:47] VERBOSE[27410][C-00000001] pbx.c: Executing [138366@default:2] Wait("Local/18136822440@default-00000000;1", "2") in new stack
[Apr 8 09:39:48] VERBOSE[27409] manager.c: Manager 'sendcron' logged off from 127.0.0.1
[Apr 8 09:39:49] VERBOSE[27410][C-00000001] pbx.c: Executing [138366@default:3] Hangup("Local/18136822440@default-00000000;1", "") in new stack
[Apr 8 09:39:49] VERBOSE[27410][C-00000001] pbx.c: Spawn extension (default, 138366, 3) exited non-zero on 'Local/18136822440@default-00000000;1'
[Apr 8 09:39:49] WARNING[27410][C-00000001] func_hangupcause.c: Unable to find information for channel
[Apr 8 09:39:49] VERBOSE[27410][C-00000001] pbx.c: Executing [h@default:1] AGI("Local/18136822440@default-00000000;1", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16--------------------)") in new stack
[Apr 8 09:39:50] VERBOSE[27410][C-00000001] res_agi.c: <Local/18136822440@default-00000000;1>AGI Script agi://127.0.0.1:4577/call_log--HVcauses ... ----------) completed, returning 0
[Apr 8 09:39:50] VERBOSE[27411][C-00000000] bridge_channel.c: Channel Local/18136822440@default-00000000;2 left 'simple_bridge' basic-bridge <f85e9baa-6672-4368-b5fd-98e94257501c>
[Apr 8 09:39:50] VERBOSE[27419][C-00000000] bridge_channel.c: Channel SIP/aka-00000000 left 'simple_bridge' basic-bridge <f85e9baa-6672-4368-b5fd-98e94257501c>
[Apr 8 09:39:50] VERBOSE[27411][C-00000000] pbx.c: Spawn extension (default, 18136822440, 3) exited non-zero on 'Local/18136822440@default-00000000;2'


-------------------------------------------------------------------------------------------------------------

on a working dialer the log shows
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] pbx.c: Executing [18136822440@default:1] AGI("Local/18136822440@default-00000000;2", "agi://127.0.0.1:4577/call_log") in new stack
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] res_agi.c: AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=Health))
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] res_agi.c: <Local/18136822440@default-00000000;2>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] pbx.c: Executing [18136822440@default:2] NoOp("Local/18136822440@default-00000000;2", "leadgen1 Dialing thru aka Customer: 18136822440") in new stack
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] pbx.c: Executing [18136822440@default:3] Dial("Local/18136822440@default-00000000;2", "SIP/18136822440@aka,,To") in new stack
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] netsock2.c: Using SIP RTP CoS mark 5
[Apr 8 06:57:56] VERBOSE[9226][C-00000000] app_dial.c: Called SIP/18136822440@aka
[Apr 8 06:57:58] VERBOSE[2196][C-00000000] res_rtp_asterisk.c: 0x7f4cd400e460 -- Strict RTP learning after remote address set to: 40.141.203.69:1640
[Apr 8 06:57:58] VERBOSE[9226][C-00000000] app_dial.c: SIP/aka-00000000 is making progress passing it to Local/18136822440@default-00000000;2
[Apr 8 06:57:58] VERBOSE[9225] dial.c: Local/18136822440@default-00000000;1 is making progress
[Apr 8 06:57:58] VERBOSE[9226][C-00000000] res_rtp_asterisk.c: 0x7f4cd400e460 -- Strict RTP switching to RTP target address 40.141.203.69:1640 as source
[Apr 8 06:58:00] VERBOSE[9226][C-00000000] app_dial.c: SIP/aka-00000000 answered Local/18136822440@default-00000000;2
[Apr 8 06:58:00] VERBOSE[9225] dial.c: Local/18136822440@default-00000000;1 answered
[Apr 8 06:58:00] VERBOSE[9225][C-00000001] pbx.c: Executing [138366@default:1] AGI("Local/18136822440@default-00000000;1", "agi-VDAD_local_optimize.agi,V4081053470026055686") in new stack
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] bridge_channel.c: Channel SIP/aka-00000000 joined 'simple_bridge' basic-bridge <81950569-4d24-4905-9d73-70c153307efb>
[Apr 8 06:58:00] VERBOSE[9226][C-00000000] bridge_channel.c: Channel Local/18136822440@default-00000000;2 joined 'simple_bridge' basic-bridge <81950569-4d24-4905-9d73-70c153307efb>
[Apr 8 06:58:00] VERBOSE[9225][C-00000001] res_agi.c: Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_local_optimize.agi
[Apr 8 06:58:00] VERBOSE[9225][C-00000001] res_agi.c: <Local/18136822440@default-00000000;1>AGI Script agi-VDAD_local_optimize.agi completed, returning 0
[Apr 8 06:58:00] VERBOSE[9225][C-00000001] pbx.c: Executing [138366@default:2] Wait("Local/18136822440@default-00000000;1", "2") in new stack
[Apr 8 06:58:00] VERBOSE[9236] manager.c: Manager 'sendcron' logged on from 127.0.0.1
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] bridge_channel.c: Channel SIP/aka-00000000 left 'simple_bridge' basic-bridge <81950569-4d24-4905-9d73-70c153307efb>
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] pbx.c: Executing [8366@default:1] AGI("SIP/aka-00000000", "agi://127.0.0.1:4577/call_log") in new stack
[Apr 8 06:58:00] VERBOSE[9226][C-00000000] bridge_channel.c: Channel Local/18136822440@default-00000000;2 left 'simple_bridge' basic-bridge <81950569-4d24-4905-9d73-70c153307efb>
[Apr 8 06:58:00] VERBOSE[9226][C-00000000] pbx.c: Spawn extension (default, 18136822440, 3) exited non-zero on 'Local/18136822440@default-00000000;2'
[Apr 8 06:58:00] VERBOSE[9226][C-00000000] pbx.c: Executing [h@default:1] AGI("Local/18136822440@default-00000000;2", "agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----16-----ANSWER-----3-----0-----SIP 200 OK)") in new stack
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] res_agi.c: AGI Script Executing Application: (EXEC) Options: (Set(_CAMPCUST=Health))
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] res_agi.c: <SIP/aka-00000000>AGI Script agi://127.0.0.1:4577/call_log completed, returning 0
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] pbx.c: Executing [8366@default:2] AGI("SIP/aka-00000000", "agi-VDAD_ALL_outbound.agi,SURVEYCAMP-----LB-----V4081053470026055686") in new stack
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] res_agi.c: Launched AGI Script /usr/share/asterisk/agi-bin/agi-VDAD_ALL_outbound.agi
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] res_agi.c: <SIP/aka-00000000> Playing 'sip-silence.gsm' (escape_digits=) (sample_offset 0) (language 'en')
[Apr 8 06:58:00] DEBUG[9232][C-00000000] format_wav.c: Skipping unknown block 'fact'
[Apr 8 06:58:00] VERBOSE[9232][C-00000000] res_agi.c: <SIP/aka-00000000> Playing 'healthinsnew.slin' (escape_digits=09) (sample_offset 0) (language 'en')
[Apr 8 06:58:01] VERBOSE[9224] manager.c: Manager 'sendcron' logged off from 127.0.0.1
[Apr 8 06:58:01] VERBOSE[9236] manager.c: Manager 'sendcron' logged off from 127.0.0.1

\
Vicidial Version 2.14-730A SVN 3180 DB schema version 1582 asterisk version 13.21.1-vici

I have a 4 machine cluster
Database/ WEB / Dialer machine dual: quad core Xeon 2.67 GHz 48GB mem / Dialer dual: dual core Xeon 2.80 GHz 8GB mem / dialer core i5 3.0 GHz 4GB mem / dialer core-2 duo 2.93GHz 3GB mem / soon archive I3 3.1 GHz 10GB mem
.. all machines except archive have 2 nic's.. one internal and 1 external for traffic.