No audio ... sometimes.

[tarikzz]

Hello,

Short version : since the asterisk CLI output is exactly the same for an OK call and a problematic one (no sound heard by the agent) , is there any way to dig into this and find out what might cause these no-sound calls ?

Details :

That problem that seems to be appearing quite randomly (but everyday), on our server that handles a dozen of agents answering to inbound calls and transferring some of them to an external carrier (let's say 8 to 10 simultaneous incoming calls + a dozen of simultaneous blind transferred calls)

The problem is that some of our inbound calls (few percent) seems to have no sound (nothing heard by the agent) and are getting cut after 5 to 30 seconds ;
I guess customers hang up because they don't have any sound either.

For such problematic calls, the voice of the agent is well recorded on file, but nothing from the client.

This seems to appear randomly during the day, but sometimes there are specific peaks with several no-sound calls, one after another. This could mean that this is related to some overload of the server ; but the cpu load is most of the time around 1 ; the 8 cores and the 32GO RAM are never fully used. Hard drive is SSD. We use G729 for bandwidth saving, for the agents, that might cause more cpu load but a load around 1, for 8 cores, that seems OK to me.

On another post, for a similar problem, @williamconley talked about the audio data that could sometimes come from a non authorized IP ; that is not the case here, as every calls show the right remote IP for the RTP stream in asterisk CLI : same -authorized- remote IP for both signal and audio streams.

Regarding the load, I think that our 20 simultaneous calls + recording of everything is not too much for a single non-clustered server with our specifications ?
And I noted that the problem can appear also with low traffic (2-3 incoming + 4-5 outgoing calls).

So the question is : since the asterisk CLI output is exactly the same for an OK call and a problematic one (no sound heard by the agent) , is there any way to dig into this and find out what might cause these no-sound calls ?

Any help you could provide would be terribly appreciated !

[williamconley]

Not all inbound calls arrive from the same sending IP address. Telcos use muliple inbound pathways for audio, but they often use a single signalling server. So port 5060 (signalling/control) always works and that initiates the calls. But the audio may arrive from a completely different server and will be using a random port for audio, entirely unrelated in any observable way from the port 5060 server. Your firewall considers this to be a bad thing, it expects that port to be used by the same IP as the 5060 server to be "related". Your Asterisk server expects the call, knows the port upon which it will arrive (which will have been negotiated on port 5060 with the siginalling server), and doesn't care what the sending IP for the audio is. But since the firewall blocks those packets, asterisk never sees them, you never get audio, packets to that port from that unkown IP are dropped.

SIP was originally designed from Server to Server both on public IPs and both knowing the authorized IPs to communicate via SIP (ignoring all unknown IPs). So any associated firewall would Always have All the "good" IPs. Times have changed.

There are many different designs for firewalls and routers for call centers, you'll have to work out how to resolve this by getting the IPs from your Carrier OR dropping your firewall and getting the IPs from your logs OR modifying your firewall to Log Dropped Packets so you can check the firewall logs during an "event" to get the IP. If you're lucky, you can just get a list of ALL IPs your carrier will send calls on, open those IPs through your firewall, and you're done.

Hint: If they refuse to give you the IPs, they are afraid you will bypass them and become a client for their "upline" (removing them from the food chain). Thus, if they refuse to provide them, get one or two yourself (from the logs) and ... cause their fears to come true. IPs are often trackable to corporations ...

[tarikzz]

Hey William,

Thank you for the detailed answer.

Here is what made me believe that the RTP stream is coming from the authorized IP (grep RTP) :

Code: Select all

[Dec  5 18:02:06] VERBOSE[2595][C-00000f7b] netsock2.c: [Dec  5 18:02:06]   == Using SIP RTP CoS mark 5
[Dec  5 18:02:06] VERBOSE[2595][C-00000f7b] res_rtp_asterisk.c: [Dec  5 18:02:06]        > 0x7f458340faa0 -- Strict RTP learning after remote address set to: X.X.X.X:19166
[Dec  5 18:02:07] VERBOSE[9780][C-00000f7b] res_rtp_asterisk.c: [Dec  5 18:02:07]        > 0x7f458340faa0 -- Strict RTP switching to RTP remote address X.X.X.X:19166 as source
[Dec  5 18:02:08] VERBOSE[9780][C-00000f7b] res_rtp_asterisk.c: [Dec  5 18:02:08]        > 0x7f458340faa0 -- Strict RTP learning complete - Locking on source address X.X.X.X:19166


Where X.X.X.X is the only one IP provided by my client (who is himself getting these calls from its carrier)
This X.X.X.X IP is defined in my configuration as a carrier and i usually have no problem receiving calls (except these silent calls), and also, i'm using this same carrier for outbound calls.
These logs are the same for an OK call and a no-sound call.

Are you saying that even though the RTP IP seems to be right on asterisk CLI (same as signaling IP), that RTP stream could actually come from an other IP, and be rejected ?
But the firewall is currently not enabled on the server ; and we receive calls on a public IP.

I will debug rtp paquets to verify their IP source.

Thanks

[tarikzz]

With RTP debug enabled, i can see that -almost- all RTP packets goes through known and authorized IPs of my clients.

However, sometimes the remote IP for RTP is set to 192.168.0.x which is the private IP range of the agents and should not appear on the server (?)

Code: Select all

[Dec  6 10:58:11] VERBOSE[2593][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11]        > 0x7fa81801ccb0 -- Strict RTP learning after remote address set to: 192.168.0.114:8000
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005824, ts 000160, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005825, ts 000320, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005826, ts 000480, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005827, ts 000640, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005828, ts 000800, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005829, ts 000960, len 000020)
[Dec  6 10:58:11] VERBOSE[30477][C-00001a70] res_rtp_asterisk.c: [Dec  6 10:58:11] Sent RTP packet to      192.168.0.114:8000 (type 18, seq 005830, ts 001120, len 000020)


I made sure that agent weren't calling each other and not doing any kind of listening

So these RTP packets are sent to private IPs (of course only "to", never "from"), but it doesn't seem to be related to the no-sound calls anyway ; it seems to be related to agent login.

As the RTP debug does not show any unknown public IP, is there any other lead you think i could investigate ?

[frequency]

Private IPs with dead air's and false answers are usually carrier issues. You can consult your carrier for routing change with examples.

[williamconley]

With RTP debug enabled

Asterisk RTP debug will only show packets that arrive. When the firewall blocks them, they never arrive, never show in the log. The problem is not fixable in Asterisk since Asterisk is not receiving the packets.

Either your router or your firewall is likely dropping the packets. That's where dropped packet logs MAY be available.

[tarikzz]

There is no firewall,
And SIP ALG has been disabled on the router.

Problematic no-audio calls do actually have RTP stream in both ways (showed by sngrep)

Also, there is no particular difference between the INVITE of a no-sound all and INVITE of a normal call.
Same thing with SDP part: nothing particularly different.

Sometimes - very rare - the agent hear his own voice just like an echo. I didn’t mention it because it happens only on 1/500 calls, but it might be a clue ...

I didn’t mention either that agents are always connected : they are called by the server when they login, then the server send them calls by making the clients join the agent conference.
So when there is a no audio call, either the agent or the client hangs up, and the agent is then atill connected to the server, he gets the next call.

The agents being always connected, and waiting for client in their conference, makes me think that this should not be a firewall problem...

[williamconley]

There is no firewall,

So you're on a public IP, this traffic does not pass through a router, and iptables is off?

And SIP ALG has been disabled on the router.

oops: Turns out there IS a router! Routers that provide both public and private IP ARE the firewall. SIP Alg being turned off may be your problem. Note that every router manufacturer has their own terminology and methodology, there is no standard. Even if you "turn off" filtering and put the server in the DMZ, the router is still making firewall decisions. Thus we prefer a public IP and NO router for Asterisk PBX servers using SIP. SIP is notorious for complicating NAT traversal and always has been.

The only way you can claim NO firewall is if there is no NAT, only public IPs, and the router is configured to pass ALL traffic for a given IP to the receiving server without any filtering whatsoever. The only exception allowed while still allowing the phrase "no firewall" is if there is a function for "this port is broken, cut it off" which is common now even in dumb switches and routers. Even if all you have is DDoS protection in the router, it functions as a firewall for that purpose and even DDoS protection HAS been known to block SIP traffic because it mistakes the SIP traffic (coming from Multiple IPs toward a single IP) as DDoS in poorly designed router software.

Problematic no-audio calls do actually have RTP stream in both ways (showed by sngrep)

Have you tried a different carrier? How's your server load during this? Do you have any networking errors on the server during this? (Not firewall, networking)

Also, there is no particular difference between the INVITE of a no-sound all and INVITE of a normal call.

Not expected to be. That's unlikely to be where the problem arises.

Same thing with SDP part: nothing particularly different.

Interesting. This combined with SDP traffic should mean sound is in fact passing, but not getting to the agent. An overloaded server or one running out of ports or sockets OR audio drop between servers are my only guesses if this is all true.

Sometimes - very rare - the agent hear his own voice just like an echo. I didn’t mention it because it happens only on 1/500 calls, but it might be a clue ...

Interesting.

I didn’t mention either that agents are always connected : they are called by the server when they login, then the server send them calls by making the clients join the agent conference. So when there is a no audio call, either the agent or the client hangs up, and the agent is then atill connected to the server, he gets the next call.

Standard and assumed. The theoretical breakdown is during the bridge of the call to the meetme room.

The agents being always connected, and waiting for client in their conference, makes me think that this should not be a firewall problem...

Entirely unrelated. The agent being previously locally connected to the meetme room has no relation to the later attempt to connect another call to that server. What reason is given for the disconnected call if you run a test and don't hang up? RTP timeout?

[tarikzz]

Thank you so much to help digging into this.

So you're on a public IP, this traffic does not pass through a router, and iptables is off?

Very sorry, i was not clear. The Vicidial server has a public IP (no firewall, just an iptables allowing everything at this moment). And most of the agents are behind a firewall where SIP ALG and SPI Firewall were enabled ; I disabled SIP ALG, the problem was still happening. Then I also disabled SPI Firewall, the problem was still happening.
I have some other agents in a second location, also behind a router - but this one does not allow to disable SIP ALG. Its a Tenda 11N router that seems to be known to have a SIP ALG feature that cannot be disabled. However, disabling it on the router of the first location didn't change anything to the problem.
The problem (short no-sound calls) keep happenning on both callcenters.

Have you tried a different carrier?

We do have different carriers sending us incoming calls. The Agents say that the problem only occurs on carrier1.
But carrier1 is sending much more calls than carrier2, which might be the reason why the agents are telling me that the problem only occurs on carrier1.
I'm not sure i can believe them. The statistics about short calls (5-to-15 seconds) show me that this seems to happens also with carrier2. But i cannot be sure, as false-positive are quite common : a client can -intentionally or not- mute his phone, or hear the agent voice and just say nothing. It happens. Silent suppression also can be tricky as we cannot refer to the background-noise to know for sure if the client is not talking, or if there is an actual problem. Also, recorded files are mixed. Cannot easily say if the background noise is the callcenter's one or the client's one.

What is making this problem harder to fix, is that there is no efficient way to find and identify precisely the no-audio calls. Everything seems exactly the same on asterisk CLI and sngrepped SIP&RTP debug.

The weird thing is that these problematic calls do have RTP packets on both ways, between the carrier and the vicidial server. Despite That, the agent cannot hear a thing, and me either while listening to the recordings.

How's your server load during this?

Problematic calls do not seem to be related to CPU load.
The server load is most of the time slightly above 1/1/1 and never reaches 2.

Do you have any networking errors on the server during this? (Not firewall, networking)

On the network interface (ifconfig), no error shows up.
And here is the netstat UDP report (other protocols show no error) :

Code: Select all

Udp:
    7390407 packets received
    7282 packets to unknown port received.
    499 packet receive errors
    7418231 packets sent
    RcvbufErrors: 499
    SndbufErrors: 0
    InCsumErrors: 0
    IgnoredMulti: 0

But I monitored "Packet receive errors" and "packets to unknown port" , and didn't saw them increase whereas we were having a couple of short no-audio calls.

if this is all true

For having being working with overseas agents for years, i know for a fact that the information coming from them need to be taken with extreme caution.
And right now, i'm starting thinking the same thing about myself, as having the nose stuck in this same matter for days can eventually make me blind.

An overloaded server or one running out of ports or sockets OR audio drop between servers are my only guesses if this is all true.

This 8 cores server really never goes above 2.0 load

Code: Select all

vicibox8:~ # uptime
 11:32am  up   3:16,  5 users,  load average: 0.55, 0.71, 0.95


200-300 tcp connections at the same time only (mostly tcp vicidial web traffic)

Code: Select all

vicibox8:~ # netstat -natp | wc -l
231


socket limit doesn't seem to be the issue as there are very few tcp connections opened - and total opened files are still far from the max ;

vicibox8:~ # lsof | wc -l
112919
vicibox8:~ # cat /proc/sys/fs/file-max
3261783

What reason is given for the disconnected call if you run a test and don't hang up? RTP timeout?

I didn't succeed duplicating the problem myself as it happens only a few dozen of times a day - out of 1000 to 1500 calls.
But usually the agent stay on line at least 30 seconds before hanging up - if the client did not hung up himself in the meantime.

The theoretical breakdown is during the bridge of the call to the meetme room

OK ; so that should be the actual moment where the problem is supposed to occur : is there any way to follow the logic of that bridge, to check out if it was successfully done and not - and if not, to check out why ? Maybe some specific logs somewhere ... ?

[williamconley]

Wireshark or similar. Packet sniff and see which calls have missing audio. Then you can work out why. (or if!: note that sometimes this is merely an overloaded server or "out of network sockets")

You can also attempt to try to get half your inbound calls routed to each of two servers instead of only a single server. Some clients rotate among all servers to avoid overload on a single server for inbound. Other clients have a single NON-Vicidial server to pass inbound calls through and use that server to route calls in rotation to all Vicidial servers. Pure asterisk doesn't overload when all it needs to do is route. There are also gateway SIP servers (such as Kamailio) that only handle signalling, not audio, and can thus route thousands of calls while managing the routing. This also allows failover as the gateway server (with or without audio-passthrough) can be configured to auto-shunt if a server goes down, and even pass to a completely different cluster at a different physical location (ie: the gateway server need not necessarily be in the same facility as the Vicidial cluster). Essentially the gateway server make you your own Carrier.

[tarikzz]

I keep digging. Will update this thread if i found out something