Counter-attacking the Botnet Counter-Attack On My Servers
Tags: servers
As expected, once I started proactively blocking the botnet from my production servers, they decided to launch a counter-attack against me…
But first, let me rewind a bit.
About 2 years ago, I started aggressively looking at my incoming traffic to determine who was hitting me, how frequently, and when. I needed to increase the performance, and reduce the number of misconfigured spiders and rss readers.
This analysis revealed that there were hundreds of thousands of requests happening on a weekly basis for identical content, every single day from these spiders and misconfigured rss aggregators (which happens to be most of them).
Every day, all day.
24×7, 7 days a week. Every week.
Repeatedly.
Ironically, the bulk of these requests was coming from the .cr domain… Costa Rica. Those requests alone, were more than 50% of my total outgoing bandwidth. They were requesting valid resources, valid files, valid data repeatedly, over and over and over.
So I blocked the entire country using iptables on port 80.
$IPTABLES -A INPUT -p tcp -m iprange --src-range X.x.0.0-X.x.255.255 -m tcp --dport 80 -j DROP
But there were a lot more coming from China, Russia, Korea, Israel, and other places.
Then I noticed an almost-immediate change in the activity. Now I was being hit with multiple thousands of requests for non-existant content, all trying to hit my Mediawiki pages, Drupal pages, WordPress pages (all written in PHP, if you notice).
What they were trying to do, is generate bogus HTTP_REFERERs for my logs, which would point back to a malicious script that would hijack the machine of the person who clicked on the link from the web-facing statistics. They were also trying to hijack the wiki pages to include these links masqueraded as “valid” links.
I’ve examined quite a few of the malicious scripts (all written in PHP also), and over time, the scripts have changed. They were originally 100% readable, but are now obfuscated and encoded, so they prevent casual dissection. Apparently they looked at THEIR logs, and noticed people were looking at the code, and not just visiting the malicious URL that included that link.
The other thing these forged HTTP_REFERER requests do is cause any log analysis package written in PHP, to parse the code, thus hijacking the server itself. Lovely.
So I started blocking the IPs originating those requests too.
When I did that, I noticed another interesting trend. If I blocked 5 unique, malicious IPs in the first hour, 10 uniques would hit me in the next hour. If I blocked those 10 unique IPs, 50 would hit me in the hour after that. The more I blocked, the faster they started coming in. If I left those 50 unique IPs alone for 24 hours, it would remain constant… 50 unique, malicious IPs/hour, never changing.
If I blocked those 50, then 200 more would come in the next hour. The faster I blocked, the more “the botnet” would send my way.
And then they counter-attacked
Over the last 2 days, after finally blocking over 1,000 unique IPs, they decided to counter-attack, and hit my webservers with http requests which were constructed to intentionally drop the TCP connection, leaving Apache in a CLOSE_WAIT state. I think their attempt was to try to tie up Apache’s listeners so other “valid” users wouldn’t be able to get in.
But I’ve already worked around that with some sysctl and kernel tweaks.
And on top of that, I’ve now automated the blocking, and now instead of blocking them on port 80, I block them on all ports, all protocols, automatically.
# iptables-save | wc -l 1392
That’s 1,392 separate, UNIQUE IPs being blocked now on all ports. That number may continue to grow, but it won’t shrink. The more machines they hijack to try to reach my servers, the more I’ll continue to block.
These amateurs really need to find another hobby, this one is just getting old.