SERVER CRASHING INTERNET CIRCUITS

[isakovk]

Hello Vicidial enthusiasts,
This is my first post here.
I know i have to list my dialer credentials, but due to my issue i have used every single Vici version out there.
My office is a Vici box and internet circuit crusher.
What happens is : whenever my employees come to work and log in into the vici, it knock s down the internet circuit at the location of the server.
Creepy right!!!!!!! It happens on random occasions, there is no particular time.
It doesn't mater whether i have 4 employees calling or 20. My line dialing never goes above 180.
In the beginning i thought that i might be my isp, so i changed them, then changed it again, which led to me trying the following carriers:
Optimum online in NYC
Verizon DSL in NYC
Time Warner Cable NYC
Verizon Fios NYC
Cox California
AND 4 DIFFERENT VICI HOSTING FACILITY.
If anyone can help me with this issue, i will be very thank full.

P.S. Some one told me that it might be DOS attack

[DomeDan]

Hello!

When you say it knocks down the connections to internet, do you mean like pulling the network-cable? resulting in timeouts or is it different?
If you have a modem, router or switch, what does the logs say?
also the vicibox logs like /var/log/messages and /var/log/asterisk/messages

put up a bridge between the internet and your network, run tcpdump and check the packages collected when the connection gets knocked down

[williamconley]

No. He means (at least in one instance) that the fiber goes dark.

I also observed that there were a WHOLE LOT of IP addresses hitting his server at one point (one screen load in iftop), but that could also have been his agents (then the line went dark and there was No Net ... so no way to tell).

But in at least one instance, the fiber provider said the ONT logs show dark fiber. When moved to another circuit, he took that circuit down, too (not dark, but offline for all users of the circuit). Then for a 3rd. Then moved to a different physical facility and killed a 4th circuit. In each instance a reset of the network hardware resolved the issue in a few minutes (ie: Unplug from power and backup to allow full power-down).

I can't say anything about the ones I did not personally observe, but I would LOVE to find out how that happened.

I've observed DDOS and brute force many times, but nothing this bad and this fast. I don't know if he pissed off someone in a bad country or if there's a coincidental string that happens to give the network indigestion (I saw a case-study once of a special string that could be passed through certain NICs to turn them off ... until the firmware was updated! That's the only thing other than DDOS I can think of to explain it!).

Hasn't happened in a few days, though. And I've been watching. Only seen one hour of usage on it during that time, but it has not "downed" in several days.

[DomeDan]

So you say it does not only kill his connection, it kills it for the other users on the same line? ..if so, Cool!

What hardware/software do they use?
Have they tried changing the hardware and the server NIC?

I've also heard about a special string that can take down a NIC,
Put up a bride and check the last packages before the connection goes down http://tomsalmon.eu/2011/02/linux-networking-bridge/
Test it out, its awesome!

I've had a problem where my firewall had a buggy NIC, had to manually set WAN NIC to 100baseTX instead of auto-detection (used pfSense)

[williamconley]

New NIC was tried (twice). I'm not sure why a bridge would be needed to use tcpdump to monitor all network traffic. Instead of tcpdump on the bridge, tcpdump works just as well on the original network. however, I do not think this client wants to pay for hundreds of thousands of packets to be checked to find out what happened. LOL

If it were still in use, however, it would be worthy to see if there were any unusual connection types and/or what that proverbial last packet was (in case there actually is a smoking gun). But it is not in use at present.

We always steer clients away from pfSense for Vicidial servers. they either end up paying us to configure their firewall or just switch to a generic router to resolve the issue. pfSense is powerful, but not as simple as it pretends to be. LOL

[isakovk]

Bill,
How come you never offered me this solution, maybe i would be willing to pay and find out who is doing it to me.
Also you haven't seen the server go down, because i haven't used it, because obviously all i get in response from you: lets restart the network, lets move your server, but not actual evidence of something happening towards fixing the problem.One time you even told me that what ever is causing it crashed DYNAMIC GOOD GUYS,which i understood from your posts in this forum is impossible to be hacked . So i stopped using it for this matter.

[williamconley]

I'm not sure how you can say I haven't seen the server go down, I watched it go down at least 6 times the week of our move at the prior location and twice at the new location. And then we were done moving and you stopped using the server. Oddly enough when you stopped using it (except for that one hour, during which it did not die) it did not die again. I told you then (like I'm tellin' you now) that when we are outside the "COLOCATION FACILITY IS IN MOTION!" window, we would be willing to troubleshoot. But NOT during the moving week.

And since we have not had the opportunity to troubleshoot it, there is no way to tell how, what, or why it crashed.

As I mentioned earlier: If you have a "bad guy" or active virus in your office, no firewall will help you. Since the firewall is lifted for your office ... you are exposed to those scenarios. And we have not had a moment (when not moving servers) to troubleshoot.

Perhaps you were involved with your problems and not listening.

[DomeDan]

So you changed the NIC twice, that leads me to the question I asked before, what hardware do you use? and did you change to a different type of NIC?

I'm talking about a bridge because I imagine the network looks a bit like this:

Code: Select all

                /------ Vicibox
Provider --- Switch --- Agents
                \------ more stuff

Then its useful to put a bridge between the Provider and the Switch.

And again. what's the network hardware (make and model) that you need to reset when the connection dies?

[williamconley]

Um ... no. We changed the NIC and the ONT (each of which is connected to a different switch) and then we move the server to a new state with a new ONT (again, so also a new switch). In the end this means three switches, three NICs (one built-in and two cards), and three ONTs in two physical locations, but all on Verizon fiber (but in two different states, each of which uses different upline equipment based on the contracts with providers for the hardware in each state). The brand of the NIC was different on the two externals and quite obviously different on the built-in (Intel motherboard vs cheap NICs, not the same drivers or hardware).

None of this involved agents, the only equipment in our facility is the Vicidial server. The three circuits involved each had a different mix of "other servers".

The ONT is what must be reset to "awaken the circuit" after darkness occurs. The ONT is the Fiber connection box from Verizon. The makes and models aren't entirely relevant (as they are not consumer-level, fiber connections provided by Verizon to connect to their fiber! and different between the two states as their contracts always differ by state).

[DomeDan]

Ok, weird.
Try to change the ip-address and the domain name to make sure its not a denial of service,
check the server to make sure there's no backdoor somewhere.
And then, package inspection! you don't need to look at every package, use filters to remove packages you know what they are and where they come from

[williamconley]

IP and domain were both changed as well. server was swapped out.

all new server was stock vicibox.

the ONLY thing these servers had in common ... was the client accessing them!

[DomeDan]

Thats good more and more things to exclude from the list of possible reasons!

What lights are on/off/flashing on the ONT when the connection dies? and what make and model is it? (yes it does matter because I cant just look in any manual or datasheet)
And yes I know you wrote:

the fiber provider said the ONT logs show dark fiber.

But that's not enough, and btw do you mean Dark Fiber or you you mean that the fiber was down?
If you mean that the provider switched to use dark fiber (to handle a greater load maybe?) then the dark fiber connection maybe is broken and they automatically switched over to a connection that are down..?

[williamconley]

by dark fiber (which is not a 'faster method of fiber', it's fiber that is not "live" because fiber requires Light to work) I mean tech support said that the logs for the ONT showed that it had gone dark. once rebooted it came back online (at some point in history). and as I said, there were at least two (possibly three) models of ONT involved as well. so once again, not a factor any more than the model of switch or NIC which also changed.

at verizon (and most other providers), dark fiber is usually the fiber reserved for later use (dark as in not being used). i used the phrase to describe fiber that way because the technician on the phone said it to me. given the odds it was not truly "dark" (as there would have been no way to get it lit again were that the case ... if the verizon side was no longer connected it would not have known we reset our side), but that is a semantic issue. mostly at issue is the fact that it no longer worked and that it only happens for this client.

i would have to focus on what this client does. maybe he has a federal 'bad guy' phone number in his list and there's a security system in place to kill network connections that call that number for tracing purposes. i dunno. it's far-fetched (as are all the other ideas), but once we got down to having changed everything but the client three times ... i was beginning to wonder. LOL

[DomeDan]

The make and model of the ONT does matter! because then I can look up a manual and see if there is any way to get error codes from the ONT,
and maybe it does try to tell the user something with flashing lights, there might be combinations with the lights that can tell you more about the issue.

Other then that I only see that the packages needs to be analyzed, have fun!
or maybe just look at the call-log and see what call caused the blackout

and yes dark fiber refers to cables not in use "but now also refers to the increasingly common practice of leasing fiber optic cables from a network service provider, or, generally, to the fiber installations not owned or controlled by traditional carriers." (wikipedia) but never mind

[williamconley]

we have NO access to the ONT. No, there is no way to get error codes from it on our side. That would require at least an hour on the phone with Verizon and a ticket upgrade to get to at least the second tier (the last two times they could only IM the second tier to ask them to check). At any given moment in time there is only (usually) one light. It's on. Unless the battery dies in which case the battery light comes on. This product is NOT designed to interact with the consumer in any way. The last thing verizon wants is someone trying to interact with the fiber. Whoa.

Poring through all the logs on all three servers would be fun, for someone in a jail cell, I guess. But for the rest of us that is thousands of lines of code that will be very unlikely to show a direct cause regardless of how many errors may occur.

If it happens again, we can clean out the logs and see if we can find a (much smaller) set of logs to peruse. If the client were to "stick with it" and wait for it to come up and kill it again a few times, we may be able to find a pattern. But he has moved on to a new server and this one is idle. Oddly enough, without so much as a hiccup in its network connection (he did use it for an hour a few days ago ... and it did not die ... so we have no idea at this point). But I'm not going to authorize a technician to spend 18-20 hours poring over logs trying to "get lucky" any more than I'd go looking for a diamond ring dropped from an airplane over the ocean. In this case, we need a few more shots to be fired so we can watch for the smoking gun.