random neuron misfires

I have over 14 years of email stored in Google Gmail now, all sorted, organized and tagged. It’s a huge archive going back through several jobs, plenty of experiences personal and professional. Some day it might make a good bit of data mining for an autobiography, but I digress..

One thing about Google Gmail has been bugging me for years! Google “folds” mail away in your Inbox, out of view, hidden, but not in any folder (what Google calls “Labels”, but they’re really individual IMAP folders).

There doesn’t seem to be any sort of reason for this, no algorithm and no obvious method to why they decide to take email out of your Inbox and hide it away from you. A bug? A feature? Who knows, but it’s annoying.

Here’s an example of what this looks like:

Notice that my search for “pinbot@pinterest.com” (Pinterest‘s mail robot) returns several hits, but only one of which is in my “Inbox”. Those others appear nowhere. They’re not in any folder anywhere in my entire IMAP or Gmail heirarchy.

They’re completely hidden, invisible, and only show up when you do a specific search for those terms. In other words, you can’t clean that junk out, delete it, unless you search for it first. Chicken-and-Egg problem, because you can’t search for what you don’t know exists.

There have been hundreds and hundreds of posts trying to come up with solutions to this problem, including using the “-label:” syntax to exclude labels from the search, leaving only “unlabeled” email.

That works great, if you have a handful of labels, maybe a dozen or two, but I have hundreds of IMAP folders (erm, “labels”), and they’re nested pretty deep in some cases. Trying to append all of my labels into one big long search string, does not work, because of a string limit in the search field. Fail.

So then I tried the somewhat magical “-label:*” search, but it returns mail with labels too, for some randomly odd reason. Another fail.

Then I tripped on someone who has a more elegant solution to the problem, rolled up into a Greasemonkey extension. Enter “gmailUnlabelled“!

Once you install the extension, you’ll find a new “Unlabelled” link on the left side, in your labels group. Clicking that will reveal email with no labels, the “hidden” email that Gmail ferrets away from you, away from your searches, away from your folders.

Find it, label it, or as in my case, kill it off. I have 10,543 emails in my “All Mail” folder, and I’m sure a few hundred to a few thousand are going to fall into the Unlabeled category.

Now I can begin the process of pruning that out and cleaning out my mail even further. I hope this helps others who may be facing the same problems.

I woke up this morning to very slow response time on my server, and decided to check the statistics. I graph these things with a great deal of detail so I can see precisely when it happened and begin narrowing down where I need to go to fix it or report it upstream.

In this case, my incoming connections went from under 500/second to well over 3,000/second. Owch!

You can see the “wall” of traffic growing from our normal traffic rate to this enormously-increased rate:

I checked all of the services, logs and protocols and didn’t see anything out of the ordinary. I started shutting down services one at a time and regenerated the graphs, to see if I could see any change.

One thing I noticed was that I had poppassd open on the public port. Not a huge problem, but it was something that was unnecessary on the public interface. I locked that down with iptables:

$IPT -A INPUT -s ! 127.0.0.1 -d ! 127.0.0.1 -p tcp -m tcp --dport 106 -j DROP

But as I looked further, I noticed even more:

netstat -tulpn | grep LISTEN

This showed that I had Squid listening on the public interface as well (0.0.0.0:3128). I jumped to the squid logs and was shocked to see that they were scrolling so fast that I couldn’t even read them. Ut oh!

Apparently some enterprising young spammer found my squid instance and decided to try to hijack it for his own needs. It was already locked down internally in my squid.conf to restrict use from only my block of IPs, but he was hammering it with 8,466 separate IPs trying to use it to send spam on port 25.

# cat access.log* | cut -b20-300 | grep ':25' | perl -lne 'print /((?:\d+\.){3}\d+)/' | sort | uniq  | wc -l
8464

Damn! There goes a few gigabytes of bandwidth that were eaten in the last 11 hours while I was sleeping.

I locked that down in a similar fashion:

$IPT -A INPUT -s ! 127.0.0.1 -d ! 127.0.0.1 -p tcp -m tcp --dport 3128 -j DROP

A bit more poking around with nmap, netstat, Webmin, HotSanIC and other tools allowed me to lock down some other services that incorrectly bind to the public interface and not the internal interface.

The result is that we’re back to normal:

One last piece needed my attention. Because this was an active spam attack, propagated using the IP of my server as a vector, I had to make sure to check my mail logs and delist myself from the various RBLs who had listed me as a spammer for sending out 43,745 separate spam attempts through my IP in a matter of hours.

SpamCop originally listed me, but I corrected that, and a few others. I also reported it to my provider so they can be sure to keep a closer eye on it.

You can see the drop-off on the far right of the last two graphs above and in the traffic graph below.

Problem solved.