random neuron misfires

I’ve been running an Apache server for over a decade, serving up hundreds of websites over the years, and one thing remains constant: abusers attacking Apache, looking for a way in, or a way to DDoS attack your server so others can’t get to the content you’re providing.

We don’t call these people ‘hackers‘, ‘crackers’ nor do we even call them ‘criminals’. They’re just idiots, and they’re easily stopped.

The rest of this post will show quite a few ways to slow or stop these attackers from taking down your Apache web server or abusing it in any way.

After testing several different methods, I finally managed to get around the 5-client restriction on the Novatel MiFi 2372 I recently purchased from Amazon.

First, some background information:

The Novatel 2372 I purchased had “some minor issues“, and I was looking for a way to get at the firmware, or updated versions of the firmware, so I could address and hopefully fix these. I was more than happy to start rolling my own firmware, to put onto the device if necessary.

If you search the Interweb, there are hundreds of posts, blogs and webpages out there on how to tweak the MiFi device by making changes to the exported config.xml file and re-import it. Some of these work, some (even after making the changes), do not. One of them is the hard-coded limit on incoming device (client) connections on the MiFi itself… this is hard-locked at 5 connections, no matter how you modify the config.xml to support more.

You can however, update the number of DHCP addresses the DHCP server on the MiFi will give out, just not the number of incoming connections to the MiFi.

But I figured out a better way to solve this in a very clean and elegant way. Unfortunately, it involves a second router… but one with a LOT more functionality. This can probably be reproduced by a smaller router, but I used what I already had in my personal lab to create this working proof of concept.

I woke up this morning to very slow response time on my server, and decided to check the statistics. I graph these things with a great deal of detail so I can see precisely when it happened and begin narrowing down where I need to go to fix it or report it upstream.

In this case, my incoming connections went from under 500/second to well over 3,000/second. Owch!

You can see the “wall” of traffic growing from our normal traffic rate to this enormously-increased rate:

I checked all of the services, logs and protocols and didn’t see anything out of the ordinary. I started shutting down services one at a time and regenerated the graphs, to see if I could see any change.

One thing I noticed was that I had poppassd open on the public port. Not a huge problem, but it was something that was unnecessary on the public interface. I locked that down with iptables:

$IPT -A INPUT -s ! 127.0.0.1 -d ! 127.0.0.1 -p tcp -m tcp --dport 106 -j DROP

But as I looked further, I noticed even more:

netstat -tulpn | grep LISTEN

This showed that I had Squid listening on the public interface as well (0.0.0.0:3128). I jumped to the squid logs and was shocked to see that they were scrolling so fast that I couldn’t even read them. Ut oh!

Apparently some enterprising young spammer found my squid instance and decided to try to hijack it for his own needs. It was already locked down internally in my squid.conf to restrict use from only my block of IPs, but he was hammering it with 8,466 separate IPs trying to use it to send spam on port 25.

# cat access.log* | cut -b20-300 | grep ':25' | perl -lne 'print /((?:\d+\.){3}\d+)/' | sort | uniq  | wc -l
8464

Damn! There goes a few gigabytes of bandwidth that were eaten in the last 11 hours while I was sleeping.

I locked that down in a similar fashion:

$IPT -A INPUT -s ! 127.0.0.1 -d ! 127.0.0.1 -p tcp -m tcp --dport 3128 -j DROP

A bit more poking around with nmap, netstat, Webmin, HotSanIC and other tools allowed me to lock down some other services that incorrectly bind to the public interface and not the internal interface.

The result is that we’re back to normal:

One last piece needed my attention. Because this was an active spam attack, propagated using the IP of my server as a vector, I had to make sure to check my mail logs and delist myself from the various RBLs who had listed me as a spammer for sending out 43,745 separate spam attempts through my IP in a matter of hours.

SpamCop originally listed me, but I corrected that, and a few others. I also reported it to my provider so they can be sure to keep a closer eye on it.

You can see the drop-off on the far right of the last two graphs above and in the traffic graph below.

Problem solved.