random neuron misfires

I was pointed to this interesting writeup describing how spammers are now using a sophisticated “host cloaking” technique to hijack valid IP addresses to send their spam through web tunnels to the outside world, thwarting detection and having their accounts deleted/disabled.

It goes like this:

1. The spammer obtains a dedicated server at the victim service provider. The server shares a subnet with other customers.
2. The spammer runs a special daemon program on the dedicated server. The daemon places the network interface into “promiscuous mode” so that it will snoop on all network packets, spying on the local subnet.
3. The daemon determines which IP addresses on the local subnet are not in use. It also determines the addresses of the network routers. One or more unused IP addresses are commandeered for use by the spammer.
4. ARP (Address Resolution Protocol) responses are sent from the daemon to the routers, binding the unused IP addresses to the server. This allows the spammer’s server to “steal” those IP addresses. The daemon does not answer ARP requests from any other source, so the stolen IP addresses remain invisible to all other systems and diagnostic equipment.
5. Finally, GRE and IPIP tunneling (a method used to connect two private networks together) is used to connect the stolen IP addresses to the spammer’s real servers hosted elsewhere.

“The end result is that the spammer has created a server at an IP address which not even the owners of the network are aware of.”

I’ve been using dspam for a few years now, with great success, but over the last month I’ve noticed a trend… After 3-4 years of not a single spam slipping through to a mailbox, I am now seeing 10-15 of them per-day getting through. The filters are still reporting a solid accuracy rate (MUCH higher than SpamAssassin), but spam is slipping through.

Overall accuracy (since last reset)     99.590%
Spam identification (since last reset)  99.395%
Spam ratio (of total processed)         48.265%

I’m also actively blocking IPs of known spammers through the firewall rules:

# iptables-save | grep -c "dport 25"
13165

That’s over 13k unique IPs that have reached our server trying to send spam, malware and other garbage to our users. Normal mail from valid hosts is still flowing in and out, as it should…

I’ll have to see what else I can do to slow it down or stop it again.

My second HDD died today, with the standard error:

[4294730.480000] hdc: dma_intr: status=0x51 { IfYouSeeMessagesLikeThis }
[4294895.037000] hdc: dma_intr: status=0x51 { YourDriveIsDying } { DriveReady SeekComplete Error }
[4294895.037000] hdc: dma_intr: error=0x01 { BackupNowAndPray, } { AddrMarkNotFound }, LBAsect=126657484, high=7, low=9216972, sector=126657431

I went to Hitachi’s website to check the warrantee, and found that I am still within the warrantee period.

I proceeded through the form to fill out an RMA, and saw this interesting disclaimer I had to agree to, in order to proceed:

Export Compliance Certification

By clicking the “I agree” button, I agree that Hitachi GST products will not be used for the design, development, manufacturing, testing, stockpiling, or use of biological, nuclear, missile or chemical weapons.

Interesting times we live in, isn’t it?

These drives seem to only last me 6-8 months. I can’t figure out why they just up and die so frequently. Bad quality control over at Hitachi?

Now I’m crippled for a little bit.

It seems that quite a few people (including myself) ran into trouble trying to get SuSE 10.1 GM to install from DVD in VMware.

The exact error you’ll get will be:

No catalogue found at dvd:///?devices%3d%2fdev%2fhdc

After a bit of tinkering with the DVD image, I realized that it was a problem with the .iso file itself. The md5sums matched perfectly, and the DVD was “intact”, it was just incorrect because the wrong mkisofs params were used when it was created. You’ll need to mount, copy the contents out, and rebuild the ISO with the proper options to make it work as a bootable ISO for VMware.

The following steps will fix it:

# Mount the original ISO as a loopback image
mount -oloop SUSE-Linux-10.1-GM-DVD-i386.iso /mnt/SuSE

# Copy all of the files out of the image to a directory on disk
rsync -avSP /mnt/SuSE/. /tmp/SuSE

# Rebuild the ISO image with the proper parameters
mkisofs -v -V SuSE10.1GM -r -J -l -L -P "OpenSuSE Linux 10.1 GM" \
-b "boot/i386/loader/isolinux.bin" -c "boot/i386/loader/boot.cat" \
-no-emul-boot -boot-load-size 4 -boot-info-table -graft-points \
-o /tmp/SuSE-Linux-10.1-DVD.iso /tmp/SuSE

Voila! Problem solved!

------=_NextPart_000_0001_01C68BAC.D6AFF790
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8bit

Hi,

VALlU jc M from o rl nly $ ok 1,2 ok 1
Ambi an en
ClAL ar lS from on ln ly $ eg 3,7 jv 5
Proza uv c
Som ri a
Levit jw ra
Merid gr ia
VlA ua GRA from o sq nly $ tz 3,3 ul 3
Xan he ax

all 50 fd % of jz f
--------

What exactly are you trying to sell me here? A dictionary?

If they think these horrible misspellings are getting their junk through the spam filters, they’re wrong. dspam is catching every single one of them, and marking them as spam, as it should be.

Subject: test pofi
Date: Fri, 9 Jun 2006 10:09:47 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0001_01C68BAC.D6AFF790"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
X-gnu-designs.com-MailScanner: Found to be clean
X-MailScanner-From: florinaprasad@cps.org
X-DSPAM-Result: Spam
X-DSPAM-Processed: Fri Jun  9 13:14:06 2006
X-DSPAM-Confidence: 0.7923
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 1,4489ac5e295161867218249
X-DSPAM-User: hacker

Why would I ever buy something from a company, or individual, who can’t even properly spell the names of his products in his email.

Are these Internet Spam Kids 6 years old? Cat walk on the keyboard again while you were typing your advertisement?

Well, at least MailScanner didn’t find any hidden malware or trojans buried in this one.

What a bunch of sad, sad individuals.

I was just diagnosing some weird kernel crashes on the BSD server, and decided to pop open the case to take a look and add some more cooling. The inside of the case was pretty hot, so I added two new Antec Pro Cooling fans, mounted inside with some stiff clipped coathanger over the drives and the CPU cooler (which has its own fan on it also). The case is roomy enough to handle it, so no biggie.

As I was closing up the case, the sliding lid didn’t seem to fit right. Its slotted tabs kept running into something. I looked closer, and it looked like a clump of dust or fuzz or something was wrapped around one of the wires.

I reached in and grabbed it to remove it and it was… how shall I say… stiff and crunchy, not like the dustball I expected to grab. I put it in the light so I could see it and… EEEECH!

Wrapped around one of the case’s signal cables, was the empty shell-of-a-carcass, of a little grey field mouse. There was only the top of a skull, the fur coat around the back, and a little segmented tail.. and two eyes. No organs, no skeleton, nothing at all that would make up muscles, bones or body.

Inside the bottom of the case, were thousands of bits of what looked like more dust, but apparently was the remnants of what used to be its coat, fur, skeleton and organs.

YUCK!

I cleaned it out, and all should be well again…. I hope, but where there’s one mouse, there’s bound to be more. I’ll have to start opening up the other servers and see what might in their cases. Maybe a few mouse traps behind the rack would be useful.

My wife and I are finding less and less time to suit up and get outside to train on our bikes, with work, travel, house renovations and parenting duties stacking up for both of us.

I needed an easier way to get on the bike and put on some miles, get the heart pumping and the blood flowing to keep my health and metabolism up.

I went down to the local bike shop here in New London called Wayfarer Bicycle 3 blocks down the road from my house, and started asking questions about the trainers, the technology, and the pitfalls.

• Can I use these with my road AND my mountain bike? (some can only be used with one or the other)
• Is it fairly stable under heavy hammering? Will I tip over or bend the arms?
• Does it fold up small enough to store in a closet or hang on the rafters?
• Do I need to service these parts? Disassemble and clean/oil them?
• What is the difference between a Magnetic and Wind trainer?
• Do I need the handlebar attachment to change the resistance?
• Is there anything else I’ll need to combine with this before I can use it?

Bob from Wayfarer was very helpful and gave me everything I needed to make the right decision.

After going through the catalog with him and looking at their in-store inventory, I went with a CycleOps Magneto mag trainer. No handlebar control to worry about, extra oversize tubing for strength and stability, sealed rear bearing, and folds pretty small and compact. If necessary, I could even travel with it.

I ordered it on Wednesday afternoon, and they had it in the store on Thursday morning. Now THAT’s what I call fast service!

The staff at Wayfarer are amazingly helpful in every way. If you’re ever in the Southeastern CT area, I highly recommend stopping in. Its right on the same road you’ll be taking to get to Ocean Beach Park… which you should be going to anyway, to enjoy the coastal views.

My wife and I both tried the trainer last night, and she absolutely loves it. It was simple to get her bike (a 2003 Fuji Team racing bike) and my mountain bike (a Trek Y22 front/rear suspended bike) into the trainer. Just pop out your standard QR skewer and slide in the CycleOps skewer, tighten up and you’re done.

The reason for the separate skewer, is because the clamping mounts on the CycleOps accept a rounded ‘nub’ skewer, and most road/mountain frames have oddly shaped, elliptical, square or oval skewers. It only took 2 minutes to switch it out and clamp the bike in. Problem solved. I can probably buy another skewer and keep it handy so I don’t have to swap it between my wife’s bike and mine.

She took to it right away, and did 15-20 minutes of moderate riding, and loved every minute of it. Her only complaint was that she thought she was “angled down”, or riding downhill. The top tube was level, so I couldn’t understand why she’d feel that way.

When she was done, I swapped the skewer and did about 20 minutes of hard riding in the trainer with my Ekho heartrate monitor. I was wearing the monitor because I wanted to make sure I was within range, and wasn’t going to blow up. I’ve been having some chest pains and breathing difficulties lately, so I didn’t want to push it too far.

At around 85rpm cadence in a pretty fat gear, my heart was at 179bpm and stable. I was just beginning to break a sweat, when I decided to stop before I did any real damage..

Overall, I’m completely impressed with the trainer, and with the stellar service Wayfarer Bicycle provided for me. They even assembled the trainer for me, so I didn’t have to.

After riding, and trying to pedal sitting up, I realized what she was talking about when she said she was “angled down”. The bike does seem like its pointed downward, even though the rear tire is only slightly elevated from the ground. I couldn’t pedal while sitting straight up (arms off of the bars), without feeling like I was sliding off the front of the saddle.

CycleOps does make something called a climbing block, which is supposed to elevate the bike’s front wheel slightly to provide the feeling of riding on differing terrain such as hills or steep climbs. I’ll see what Wayfarer thinks and ask if they have a solution.

Another worthy and worthwhile purchase.

I’ve been configuring one of my machines as a FreeBSD 6.1 server in my spare picoseconds, to serve as a fileserver, rsync server, public mirror, backup server, transparent squid proxy, coffee maker and whatever else I can make it do.

In order to do this, I needed to add a bunch of packages and tweak quite a few hundred things in various places on the system (sysctl, make.conf, loader.conf, add missing tweaks and options to the kernel config) and so on.

I rebuild kernels nightly and keep ports up to date with cvsup. I rarely find time or a need to run portupdate or buildworld, but I know I should, so I found this useful article which describes how to stay current with all of these pieces (and there are a LOT of them).

Buried in the comments, was a pointer to ‘sysutils/fastest_cvsup‘, which is probably the BSD ports equivalent of Debian’s netselect-apt tool. fastest_cvsup is described as follows:

Perl script to find fastest CVSup server:

* uses socket connections not just 'pings'
* takes notice of server responses
* connects to servers in countries specified on the command line 
   - or - 
  connects to the 'local' servers defined in the script
   - or -
  connects to ALL the servers in ALL the countries
* returns either fastest server or top 3 (useful for scripts)
* returns exit codes (useful for scripts)
* can re-write itself to update the CVSup server list, obtained
  from the online FreeBSD Handbook
* can easily add other CVSup servers (NetBSD/OpenBSD...etc)

WWW: http://fastest-cvsup.sourceforge.net/

Running it was as simple as executing it with the right country in mind (or you can pass it ‘-c all’ and test all of them):

# fastest_cvsup -c us
>>  Querying servers in countries: us
--> Connecting to cvsup.us.freebsd.org [198.104.69.57]...
    - server replied: ! Access limit exceeded; try again later
    - time taken: 146.89 ms
--> Connecting to cvsup2.us.freebsd.org [130.94.149.166]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 42.03 ms
--> Connecting to cvsup3.us.freebsd.org [128.31.0.28]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 35.09 ms
--> Connecting to cvsup4.us.freebsd.org [204.152.184.73]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 114.04 ms
--> Connecting to cvsup5.us.freebsd.org [64.157.15.40]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 74.04 ms
--> Connecting to cvsup6.us.freebsd.org [69.31.98.210]...
    * error: connect: Invalid argument
--> Connecting to cvsup7.us.freebsd.org [129.250.31.140]...
    - server replied: OK 17 0 SNAP_16_1g CVSup server ready
    - time taken: 104.05 ms
--> Connecting to cvsup8.us.freebsd.org [216.165.129.134]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 57.24 ms
--> Connecting to cvsup9.us.freebsd.org [128.205.32.21]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 53.00 ms
--> Connecting to cvsup10.us.freebsd.org [128.205.32.10]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 52.02 ms
--> Connecting to cvsup11.us.freebsd.org [63.87.62.77]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 33.05 ms
--> Connecting to cvsup12.us.freebsd.org [128.46.156.46]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 49.02 ms
--> Connecting to cvsup13.us.freebsd.org [216.144.193.227]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 50.02 ms
--> Connecting to cvsup14.us.freebsd.org [64.78.150.180]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 156.16 ms
--> Connecting to cvsup15.us.freebsd.org [131.193.178.106]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 50.05 ms
--> Connecting to cvsup16.us.freebsd.org [128.143.108.35]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 65.03 ms
--> Connecting to cvsup17.us.freebsd.org [65.212.71.21]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 43.05 ms
--> Connecting to cvsup18.us.freebsd.org [128.205.32.37]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 56.02 ms

>>  Speed Daemons:
    - 1st: cvsup11.us.freebsd.org   33.05 ms
    - 2st: cvsup3.us.freebsd.org    35.09 ms
    - 3st: cvsup2.us.freebsd.org    42.03 ms

So now I can stick cvsup11.us.freebsd.org in my .sup files and hopefully gain the fastest connection to those servers for updates.

I’ve been looking for a good snapshot tool for FreeBSD, and I think I’ve finally found it.

Today, I stumbled across a slick little tool called rsnapshot, written by Nathan Rosenquist. It was featured in the BSD Hacks book by O’Reilly Media.

rsnapshot is based on rsync, and allows quick snapshots of a filesystem in-time, using hardlinks to preserve space. Basically you configure it using the default config file, tweak a few options, and check the syntax.

rsnapshot, like the rsync tool it is built upon, has a wealth of options (including inheriting those from rsync), including ‘configtest’ to check the syntax of the config file, and ‘du’ to check the size of the snapshot tree.

Here’s an example from my own backups:

# rsnapshot du
771.7G    /usr/local/array/.snapshots/hourly.0/
771.7G    total

You can set up rsnapshot to run from cron (and you should), like this:

0 */4 * * *     /usr/local/bin/rsnapshot hourly
30 23 * * *     /usr/local/bin/rsnapshot daily

From here, you’ll see it create a tree like:

[22:28:20 Thu Jun 01]
[117] flea:/usr/local/array/.snapshots
# ls -l
drwxr-xr-x    7 root     wheel         4096 May 28 00:00 daily.0
drwxr-xr-x    7 root     wheel         4096 May 27 00:00 daily.1
drwxr-xr-x    7 root     wheel         4096 May 26 00:00 daily.2
drwxr-xr-x    7 root     wheel         4096 May 25 00:00 daily.3
drwxr-xr-x    7 root     wheel         4096 May 24 00:00 daily.4
drwxr-xr-x    7 root     wheel         4096 May 23 00:00 daily.5
drwxr-xr-x    7 root     wheel         4096 May 22 00:00 daily.6
drwxr-xr-x    7 root     wheel          512 May 29 00:00 hourly.0
drwxr-xr-x    7 root     wheel          512 May 28 20:00 hourly.1
drwxr-xr-x    7 root     wheel          512 May 28 16:00 hourly.2
drwxr-xr-x    7 root     wheel          512 May 28 12:00 hourly.3
drwxr-xr-x    7 root     wheel          512 May 28 08:00 hourly.4
drwxr-xr-x    7 root     wheel          512 May 28 04:00 hourly.5

Another great feature is how it intelligently rotates the hourlys into daily and folds them together.

When ‘rsnapshot daily’ is run, it will rotate all the daily.X directories and then copy the contents of hourly.5 into daily.0. hourly.0 will always contain the most recent snapshot, and daily.6 will always contain a snapshot from a week ago. Unless the files change between snapshots, the “full” backups are really just multiple hard links to the same files.

If a file changes at any point, the next backup will unlink the hard link in hourly.0, and replace it with a brand new file. This will now take double the disk space it did before, but it is still considerably less than it would be to have full unique copies of this file 13 times over.

Once I manage to get this all working, I’m going to be migrating the Windows login and profiles over to the Samba server (now acting as a PDC) and start doing backups of that data into the snapshots as well.

The volume that this data is stored on and backed up on is also protected by geli, which wraps around the cryptographic GEOM class available in the BSD kernel. I started with the GEOM encryption, but it was a bit slower than I needed.

‘geli’ improves upon GEOM by a little bit, in the following ways:

• Utilizes the crypto(9) framework — when cryptographic hardware is available, geli will use it automatically.
• Supports multiple cryptographic algorithms (currently AES, Blowfish, and 3DES).
• Allows the root partition to be encrypted. The passphrase used to access the encrypted root partition will be requested during the system boot.
• Allows the use of two independent keys (e.g. a “key” and a “company key”).
• geli is fast – performs simple sector-to-sector encryption.
• Allows backup and restore of Master Keys. When a user has to destroy his keys, it will be possible to get access to the data again by restoring keys from the backup.
• Allows to attach a disk with a random, one-time key — useful for swap partitions and temporary file systems.

And here’s the best part..

“Unlike cumbersome encryption methods that encrypt only individual files, gbde and geli transparently encrypt entire file systems. No cleartext ever touches the hard drive’s platter.”

The goal for this server, besides its many workhorse duties, is to be the backup and mirror server for several terabytes of projects, such as Project Gutenberg, CPAN, LDP, Wikipedia, and dozens of other projects.

Its also moving into its role as PDC with Samba using LDAP authentication over SSL/TLS for all clients, who will have their profiles and home directories transparently mapped and mounted on the server, backed up regularly with rsnapshot. It should all work out great when I’m done with it.