Hacking via trunkinbound

[inthegenes]

Hi,

I got hacked on the 30th and the 31st of August. This is the second time I have witnessed a hack via this method where someone makes an inbound call to a SIP trunk, keeps the line open and then makes a call through the dialplan. The VoIP provider logs calls going out to an international destination and bills us accordingly. In this case the destination was Haiti. The SIP trunk is connected via a VLAN, which is provided by the provider. The network configuration is as such;
ISP box -> VICIbox.

How do they do this and how does one combat against it? My box is currently heavily restricted via IPtables (apf). Only allowed IP addresses can access the box. That has kept hackers out via other methods, but clearly there is a hole in my system configuration.

Here is an example of the call record/log in the CDR;

clid src dst dcontext channel lastapp lastdata
12 12 s trunkinbound SIP/provider_sip_trunk-00000008 Wait 600
12 12 s trunkinbound SIP/provider_sip_trunk-00000007 Wait 600
12 12 s trunkinbound SIP/provider_sip_trunk-00000002 Wait 600
12 12 s trunkinbound SIP/provider_sip_trunk-00000003 Wait 600
12 12 s trunkinbound SIP/provider_sip_trunk-00000004 Wait 600

I do not have a destination s in my trunkinbound context.

Install via ISO -
ViciBox Redux v.6.0.3-141118
Updates to SVN 2192 v.2.10-452a build 141111-0554
Asterisk 1.8.29.0-vici
3 boxes: DB - Telephony - Archive
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz | Dual hexacore

[williamconley]

1) Good job posting your specs
2) You posted a good description of your problem except: The CDR has no information that's useful, you want to look at the asterisk CLI log which is "/var/log/asterisk/messages" or "/var/log/astguiclient/screenlog.0". This will show a record of the dialplan method used to route the calls.
3) IAX2 could be the entry point, but this would still show in the CLI log when the calls are generated
4) In a properly configured system an inbound call can NOT become an outbound call because inbound calls are in the context [trunkinbound] and outbound calls are in the context [default]. Since it's impossible to jump between contexts without a dialplan or agi assist, there would have to be a connection. If YOU created a connection or allowed inbound calls into [default], then fix that obvious security violation error immediately. The method used to jump from inbound to outbound will be visible in the aforementioned cli logs. If you received "advice" from someone that resulted in inbound calls in [default], that was likely also your intruder. If they are in the same country you are, it may be time for a visit with some badges.
5) Firewall: "Heavily restricted" means nothing and is in fact pointless unless you are using whitelisting (ie: authorized IPs only).
6) DGG from viciwiki.com contains whitelist lockdown procedures (in addition to actually installing Dynamic Good Guys firewall which allows easy addition of "good" IPs).

[inthegenes]

Thanks Will,

2. I'll look into the CLI logs.
3. I have since white listed IAX2
4. No 'advice' was given regarding the inbound context. I too questioned if an agi was used given there isn't anyway to access the default context from, trunkinbound.
5. And yes the firewall was had a white list configuration for port 5060 when it was hacked.
6. Will research. I use apf for easy addition.

[williamconley]

5. And yes the firewall was had a white list configuration for port 5060 when it was hacked.

Whitelist is for the entire system, NOT just port 5060. You don't just lock the driver's door on your car do you?

[inthegenes]

4. See below for the trunkinbound context;

[trunkinbound]
; DID call routing process
exten => _XXXXXXXXXX,1,AGI(agi-DID_route.agi) ; use this one instead of the one below if you are having delay issues, and match to number of received digits
;exten => _X.,1,AGI(agi-DID_route.agi)
;exten => _X.,n,Hangup()
; If you have DIDs that arrive with a plus sign at the beginning then uncomment
;exten => _+X.,1,AGI(agi-DID_route.agi)
;exten => _+X.,n,Hangup()

; FastAGI for VICIDIAL/astGUIclient call logging
exten => h,1,AGI(agi://127.0.0.1:4577/call_log--HVcauses ... EBUG-----${HANGUPCAUSE}-----${DIALSTATUS}-----${DIALEDTIME}-----${ANSWEREDTIME})

5. Will have to undertake that task over the weekend. Granted this is a production server.

[inthegenes]

Ok,

I took a look at the messages file for the dates where we had recorded issues and found entries similar to that which is in the CDR log.

On the 28th unsuccessful attempts where made by the hacker:
[Aug 28 00:53:14] NOTICE[5573] pbx_spool.c: Call failed to go through, reason (0) Call Failure (not BUSY, and not NO_ANSWER, maybe Circuit busy or down?)
[Aug 28 00:53:14] NOTICE[5573] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932782901 expired without completion after 0 attempts
[Aug 28 00:53:15] NOTICE[6061] channel.c: Unable to request channel SIP/provider_sip_trunk/01150932787775
[Aug 28 00:53:15] NOTICE[6061] pbx_spool.c: Call failed to go through, reason (0) Call Failure (not BUSY, and not NO_ANSWER, maybe Circuit busy or down?)
[Aug 28 00:53:15] NOTICE[6061] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932787775 expired without completion after 0 attempts
[Aug 28 00:53:16] NOTICE[6549] channel.c: Unable to request channel SIP/provider_sip_trunk/01150932781309
[Aug 28 00:53:16] NOTICE[6549] pbx_spool.c: Call failed to go through, reason (0) Call Failure (not BUSY, and not NO_ANSWER, maybe Circuit busy or down?)
[Aug 28 00:53:16] NOTICE[6549] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932781309 expired without completion after 0 attempts
[Aug 28 00:53:18] NOTICE[7039] channel.c: Unable to request channel SIP/provider_sip_trunk/01150932785827


Failed attempts were also recorded on the 30th:
[Aug 30 07:03:03] VERBOSE[4636] pbx_spool.c: [Aug 30 07:03:03] -- Attempting call on SIP/provider_sip_trunk/01150932785827 for application Wait(600) (Retry 1)
[Aug 30 07:03:03] VERBOSE[4636] netsock2.c: [Aug 30 07:03:03] == Using SIP RTP CoS mark 5
[Aug 30 07:03:04] VERBOSE[4126] manager.c: [Aug 30 07:03:04] == Manager 'sendcron' logged off from 127.0.0.1
[Aug 30 07:03:04] WARNING[5508] chan_sip.c: Received response: "Forbidden" from '"12" <sip:XXXXXXXXXX@provider>;tag=as0b22c15f'
[Aug 30 07:03:04] NOTICE[4636] pbx_spool.c: Call failed to go through, reason (1) Hangup
[Aug 30 07:03:04] NOTICE[4636] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932785827 expired without completion after 0 attempts


Until somehow the calls started going through. I do not understand nor see how.
[Aug 30 07:03:55] VERBOSE[10669] pbx_spool.c: [Aug 30 07:03:55] -- Attempting call on SIP/provider_sip_trunk/01150932785827 for application Wait(600) (Retry 1)
[Aug 30 07:03:55] VERBOSE[10669] netsock2.c: [Aug 30 07:03:55] == Using SIP RTP CoS mark 5
[Aug 30 07:03:56] WARNING[5508] chan_sip.c: Received response: "Forbidden" from '"12" <sip:XXXXXXXXX@provider>;tag=as04ed75ea'
[Aug 30 07:03:56] NOTICE[10669] pbx_spool.c: Call failed to go through, reason (1) Hangup
[Aug 30 07:03:56] NOTICE[10669] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932785827 expired without completion after 0 attempts
[Aug 30 07:03:59] VERBOSE[5326] asterisk.c: [Aug 30 07:03:59] -- Remote UNIX connection
[Aug 30 07:03:59] VERBOSE[11159] asterisk.c: [Aug 30 07:03:59] -- Remote UNIX connection disconnected
[Aug 30 07:03:59] VERBOSE[11163] pbx_spool.c: [Aug 30 07:03:59] -- Attempting call on SIP/provider_sip_trunk/01150932782901 for application Wait(600) (Retry 1)
[Aug 30 07:03:59] VERBOSE[11163] netsock2.c: [Aug 30 07:03:59] == Using SIP RTP CoS mark 5
[Aug 30 07:04:00] WARNING[5508] chan_sip.c: Received response: "Forbidden" from '"12" <sip:XXXXXXXXX@provider>;tag=as2eb8f148'
[Aug 30 07:04:00] NOTICE[11163] pbx_spool.c: Call failed to go through, reason (1) Hangup
[Aug 30 07:04:00] NOTICE[11163] pbx_spool.c: Queued call to SIP/provider_sip_trunk/01150932782901 expired without completion after 0 attempts
[Aug 30 07:04:00] NOTICE[25567] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932785827
[Aug 30 07:04:01] VERBOSE[11200] manager.c: [Aug 30 07:04:01] == Manager 'sendcron' logged on from 127.0.0.1
[Aug 30 07:04:01] VERBOSE[11201] manager.c: [Aug 30 07:04:01] == Manager 'sendcron' logged on from 127.0.0.1
[Aug 30 07:04:01] VERBOSE[11201] manager.c: [Aug 30 07:04:01] == Manager 'sendcron' logged off from 127.0.0.1
[Aug 30 07:04:02] NOTICE[26058] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932782901
[Aug 30 07:04:03] NOTICE[26587] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932787775
[Aug 30 07:04:03] VERBOSE[5326] asterisk.c: [Aug 30 07:04:03] -- Remote UNIX connection
[Aug 30 07:04:03] VERBOSE[11689] asterisk.c: [Aug 30 07:04:03] -- Remote UNIX connection disconnected
[Aug 30 07:04:03] VERBOSE[11693] pbx_spool.c: [Aug 30 07:04:03] -- Attempting call on SIP/provider_sip_trunk/01150932787775 for application Wait(600) (Retry 1)
[Aug 30 07:04:03] VERBOSE[11693] netsock2.c: [Aug 30 07:04:03] == Using SIP RTP CoS mark 5
[Aug 30 07:04:04] DEBUG[5556] pbx_spool.c: Delaying retry since we're currently running '/var/spool/asterisk/outgoing/check.call'
[Aug 30 07:04:04] VERBOSE[11200] manager.c: [Aug 30 07:04:04] == Manager 'sendcron' logged off from 127.0.0.1
[Aug 30 07:04:05] NOTICE[27084] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932781309
[Aug 30 07:04:06] VERBOSE[11701] manager.c: [Aug 30 07:04:06] == Manager 'sendcron' logged on from 127.0.0.1
[Aug 30 07:04:06] VERBOSE[11701] manager.c: [Aug 30 07:04:06] == Manager 'sendcron' logged off from 127.0.0.1
[Aug 30 07:04:08] VERBOSE[5326] asterisk.c: [Aug 30 07:04:08] -- Remote UNIX connection
[Aug 30 07:04:08] VERBOSE[12186] asterisk.c: [Aug 30 07:04:08] -- Remote UNIX connection disconnected
[Aug 30 07:04:08] VERBOSE[12190] pbx_spool.c: [Aug 30 07:04:08] -- Attempting call on SIP/provider_sip_trunk/01150932781309 for application Wait(600) (Retry 1)
[Aug 30 07:04:08] VERBOSE[12190] netsock2.c: [Aug 30 07:04:08] == Using SIP RTP CoS mark 5
[Aug 30 07:04:09] DEBUG[5556] pbx_spool.c: Delaying retry since we're currently running '/var/spool/asterisk/outgoing/check.call'
[Aug 30 07:04:10] VERBOSE[12190] pbx.c: [Aug 30 07:04:10] > Channel SIP/provider_sip_trunk-000000a9 was answered.
[Aug 30 07:04:10] VERBOSE[12190] pbx.c: [Aug 30 07:04:10] > Launching Wait(600) on SIP/provider_sip_trunk-000000a9
[Aug 30 07:04:10] NOTICE[29051] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932781309
[Aug 30 07:04:11] NOTICE[29541] pbx_spool.c: Call completed to SIP/provider_sip_trunk/01150932785827


Any assistance in understanding what happened here?

[mattyou1985]

whats your Dialplan Entry: ??

to me it looks like thir calling your your server using your inbound root then rerooting out agen

i head about this attak some wair and thay sead that this attak is possible if admins set thir Dialplan Entry with a Tt funsion

witch alows remote users to call in using your number then thanks to the Tt funsion thay can call strate out agen using your carrier and server

example
exten => _44.,2,Dial(sip/44${EXTEN:2}@mycarriername,Tt,) <<<< ive been trying to get more info on the Tt as thir are mutaple varations of this i want to no what thay all do ....tT,Tor,r,o,roTt,ect

what elce i can tell you is thay allso affect how the Ringing tone sounds like try some tests and place out bound calls youll see what i mean (ime gessing its to do with signling .....but could be rong)

[inthegenes]

Thanks, I have a few dial plan entries with Tt

If anything I will go through and remove them and see if that helps.

[inthegenes]

Can someone verify that what mattyou1985 posted is the reason and solution for this break-in.

I just received the bill for the calls that were logged from this hack and I need to know how to approach this situation.

After reviewing my dialplan I did in fact see tTo entries. However, the provider that had the tTo entries was not hacked. A separate provider was hacked. Is this possible?

I was hoping for some advice and or feedback from the vici moderators given my dire plight...

[williamconley]

5. And yes the firewall was had a white list configuration for port 5060 when it was hacked.

Whitelist is for the entire system, NOT just port 5060. You don't just lock the driver's door on your car do you?

I haven't had the opportunity to read through all of this (no time tonight), but I don't think my glance received an answer to this query/solution.

Have you locked the server down on all ports?

This could very well be someone placing text files on your server in the asterisk folder that causes calls to generate. Thus, access via a port other than IAX or SIP resulting in your server initiating a call to both sides of a conversation.

Next up: "pbx_spool.c"? Where did this come from? Do you have customizations on your system aside from the standard Vicidial extensions.conf and extensions-vicidial.conf? Have you edited any of the .conf files for any of the modules so they are non-standard?

[inthegenes]

No this is a standard install. No customization, standard vicidial extensions.conf and extensions-vicidial.conf all the modules are also standard.

[williamconley]

And you still did not answer the question of "have you locked down all ports and whitelisted your system". So I'll take that to be a "no" and assume that this is your problem. No whitelist = hacked, as a rule.

[inthegenes]

The box is whitelisted now. Needs further testing to ensure that the configuration is bulletproof.

Based on my research the recommendation is to format the box (fresh install) in cases like this were there has been a hack that cannot be diagnosed properly. Lest there is a back door left open.

[williamconley]

To review ... no further problems since the lockdown?

I do, of course, agree that if you never found the exploit, a box that has been compromised should be rebuild to avoid a backdoor.

Simple concept being that Vicibox does not build a Hardened server. Whitelist is still not perfect, but it does require that you invite your exploiter to the party which reduces the odds enough for most scenarios.