When Networks Go Bad
Tags: irc, servers
I run some services here for lots of projects. Let me rephrase that, I run a LOT of services here for various projects, development and customers for the two different companies that I own. At any one time there are at least 7 servers up and running here doing various things; generating content, serving webpages, building software, whatever.
This includes personal mail for users (with imap and webmail access), about 2-dozen mailing lists, web hosting for about 70 different domains and projects, bug tracking, blogs, portals, revision control for Open Source projects, irc for developers, torrent tracker for releases, compile farm, and quite a few other things I can’t think of at the moment. All of this requires solid, reliable, 24×7 bandwidth to sustain… and clean power.
All of this comes out of my own pocket: bandwidth, power, servers, hardware, time. Its not cheap.
About a month or so ago, I decided to double the speed of the incoming bandwidth on the server’s dedicated lines (hosting the services above) and my inbound DSL connection. The DSL is my own “personal” Internet line, and the other lines are dedicated to the servers themselves. This also doubled the price I pay for the bandwidth, but the Google Ads seem to be sustaining most of that now.
I graph and monitor all the inbound and outbound traffic with quite a few different tools, so I can track and notice trends, attacks, statistics for customers, and many other things. Things were good for about two weeks… but then it started sliding downhill.
Here is an example of a recent Slashdotting that we cleanly survived:
Over the last two weeks, I’ve noticed the VPN to IBM that I keep open has been dropping out many times per-day. I’d try to restart the VPN and get errors. I went into the server room (where the DSL and other networking lines come in) and noticed that the DSL modem didn’t even have a line to the CO. A quick power-cycle of the DSL modem would cure it for a few hours. It started happening late at night and early in the morning, during lower traffic times for me.
Fishy. I checked to see if I was being “packeted” by some script kiddies or attacked, and nothing obvious showed up in the graphs. A call to my provider after 2 weeks of dozens of dropouts per-day seemed to provide some action. They believe the problem is with the port I’m using at the DSLAM, specifically that it is “over-provisioned”. They tried capping my line down a few Mbit, which helped for a day or three, but then the dropouts started again.
Its gotten significantly worse now, and my speed on DSL is slightly slower than a 28k dialup modem. I can barely use the web now because of it. Its painful to watch servers and DNS queries time out, because I’m browsing at less than 5k/sec. Yowch!
If my provider can’t fix this (and credit me for the horrible speed and downtime), I’m going to explore moving to cable modem service again, like I had in Westerly.
Is providing broadband REALLY this difficult? I pay $180/month for 1.5Mb-6.0Mb/384-608Kb here and I barely reach the low-end of that scale. I’m 8k feet from the CO, so I should have a nice solid signal. Other countries have 10-times the bandwidth and pay pennies for it.
To their credit, my provider has been very patient and helpful during these stressful times, and we’re working through the issues to try to solve it, but… its been two weeks now. Let’s hope they solve it tomorrow when the landline provider shows up to test the lines and figure out the problem.
I use the Internet every day for research, for my job, and for other development purposes. I can’t have it go down like this, at these speeds now.
This is ridiculous.