random neuron misfires

I run some services here for lots of projects. Let me rephrase that, I run a LOT of services here for various projects, development and customers for the two different companies that I own. At any one time there are at least 7 servers up and running here doing various things; generating content, serving webpages, building software, whatever.

This includes personal mail for users (with imap and webmail access), about 2-dozen mailing lists, web hosting for about 70 different domains and projects, bug tracking, blogs, portals, revision control for Open Source projects, irc for developers, torrent tracker for releases, compile farm, and quite a few other things I can’t think of at the moment. All of this requires solid, reliable, 24×7 bandwidth to sustain… and clean power.

All of this comes out of my own pocket: bandwidth, power, servers, hardware, time. Its not cheap.

About a month or so ago, I decided to double the speed of the incoming bandwidth on the server’s dedicated lines (hosting the services above) and my inbound DSL connection. The DSL is my own “personal” Internet line, and the other lines are dedicated to the servers themselves. This also doubled the price I pay for the bandwidth, but the Google Ads seem to be sustaining most of that now.

I graph and monitor all the inbound and outbound traffic with quite a few different tools, so I can track and notice trends, attacks, statistics for customers, and many other things. Things were good for about two weeks… but then it started sliding downhill.

Here is an example of a recent Slashdotting that we cleanly survived:

Over the last two weeks, I’ve noticed the VPN to IBM that I keep open has been dropping out many times per-day. I’d try to restart the VPN and get errors. I went into the server room (where the DSL and other networking lines come in) and noticed that the DSL modem didn’t even have a line to the CO. A quick power-cycle of the DSL modem would cure it for a few hours. It started happening late at night and early in the morning, during lower traffic times for me.

Fishy. I checked to see if I was being “packeted” by some script kiddies or attacked, and nothing obvious showed up in the graphs. A call to my provider after 2 weeks of dozens of dropouts per-day seemed to provide some action. They believe the problem is with the port I’m using at the DSLAM, specifically that it is “over-provisioned”. They tried capping my line down a few Mbit, which helped for a day or three, but then the dropouts started again.

Its gotten significantly worse now, and my speed on DSL is slightly slower than a 28k dialup modem. I can barely use the web now because of it. Its painful to watch servers and DNS queries time out, because I’m browsing at less than 5k/sec. Yowch!

If my provider can’t fix this (and credit me for the horrible speed and downtime), I’m going to explore moving to cable modem service again, like I had in Westerly.

Is providing broadband REALLY this difficult? I pay $180/month for 1.5Mb-6.0Mb/384-608Kb here and I barely reach the low-end of that scale. I’m 8k feet from the CO, so I should have a nice solid signal. Other countries have 10-times the bandwidth and pay pennies for it.

To their credit, my provider has been very patient and helpful during these stressful times, and we’re working through the issues to try to solve it, but… its been two weeks now. Let’s hope they solve it tomorrow when the landline provider shows up to test the lines and figure out the problem.

I use the Internet every day for research, for my job, and for other development purposes. I can’t have it go down like this, at these speeds now.

This is ridiculous.

A couple of years ago, I purchased a Western Digital external combo drive to back up my laptops and a couple of the critical servers here. It was also partitioned for holding the digital images we take with our Minolta DiMAGE 7Hi. It was only a mere 120gb of capacity, but it lasted for quite a long time… but it was time to upgrade it.

The enclosure has two interfaces: usb2.0 and Firewire 400 (1394a). It works great, and has served me well for the couple of years I’ve had it. No complaints at all with it.

I recently went out and bought two Maxtor MaxLine Plus II 250gb drives; one for the main server, and one to replace the 120gb drive in the WD enclosure.

The upgrade of the external enclosure’s drive went pretty smoothly (full details of the disassembly), and recognizing the new drive went smoothly. I proceeded to back up 3 of the servers here to the drive, including making a duplicate copy of what was on the 120gb WD onto this new 250gb drive. I made sure to verify the backups to be sure things were intact. I’ve had a LOT of bad luck with storage and computer peripherals in general, so I was taking no chances.

The other drive went into the main server here, and that wasn’t so easy. I did an rsync of the existing running data to the Maxtor while installed in the primary slave location. So far, so good. I wanted to chroot to that drive’s mountpoint and just re-run lilo to create a working mbr on the slave, but that didn’t work so well.

Ok, second plan: switch the drives, boot the server to KNOPPIX and chroot from there, and run lilo. Nope, of course not. My KNOPPIX disks, which I use almost weekly were all no longer recognized in the CDROM drive in the server. In fact NO cdrom was recognized in that drive. Arg!

So I had to put the original drive back in as slave, switch the bios to allow me to boot to that second drive, and then re-ran lilo from there, which put the right mbr on the master. Whew. A few hiccups with some startup scripts, and I was back in business. The drive is pushing about 1gb/sec. over cache, and 49mb/sec. over disk reads. Not bad at all.

Once I wiped the servers after doing the backup, I stupidly decided to try to defrag the ext partition. It was ext3, so e2defrag barfed on it. I used tune2fs to take off the has_journal and dir_index bits from the drive metadata, and tried again.

This time it got as far as calculating the inode indices, then crashed. Ut oh. I ran e2fsck on the drive, and it segfaulted about 70% into the process. Double-ut-oh! I ran it several times, all segfaulting in the same place. Running it under gdb produced the following barf:

     0xb7fcf45b in ext2fs_unmark_generic_bitmap () from /lib/libext2fs.so.2

Rut-roh! So I decided to yank all of the data off of the backup drive onto other systems with enough free space to hold it, and reformatted it to XFS instead. After restoring the data across, all seems well.

Whew!

I briefly corresponded with a user who was asking for access to CVS for pilot-link, to try to solve a problem he was having with photos on his Palm.

I mentioned that CVS was not public, and he responded that he googled around and found a message from me on a mailing list I run, that helped him out.

“Wait, how did google spider a list that I know I restrict them from being able to index…”

So I started googling, and found this little site. It is a site in .ph (the Phillapines).

The problem with this, isn’t really that they provide an offsite archive of lists, but that they remove all email obfuscation from the posts. This means anyone posting to my lists, under the knowledge that their email address will be protected (by my site configuration and Mailman itself), will no longer have that address protected when it gets indexed by this site in .ph.

I also noticed a few moderated lists there, which I know have member-only viewable archives. This means you can’t google around and find posts made in those archives… except that google spiders THIS site, and picks them up, including the user’s email addresses.

I sent the webmaster a VERY harsh email about the situation, giving him a deadline of 5 days to remove any and all references to our lists from his/their servers. I also blocked their entire netblock on port 25 and 80, so he can’t even fetch the mbox version of the archives, and I unsubscribed the user “lurker” from all of the lists I run here.

We’ll see what happens. Probably nothing, but at least I can stop rogue users from subscribing to the list, purely for the purpose of putting list archives somewhere else on the Interweb.

I started going through my weblogs for all the domains I host, looking for 404’s, and correcting them. Many of the domains we host have updated their pages, moved files around, etc. and other sites and servers and users still point to the old files and content. Those were easy to fix with a bit of mod_rewrite and mod_redir hackery, and it keeps the users happy and logs nice and clean.

But as I was parsing out the logs, I noticed quite a few other curious things, which led me to poke through the referer logs and start tracing some interesting hits.

..which led me to these two sites:

http://www.actionweb.com/hosting/clients/
http://www.firstwebserver.com/hosting/clients/index.html

Both of these domains are registered in completely different states, by two completely different people, and yet… other than page color, they are identical, even down to the “testimonials” page. Whomever ripped this off from whom, can’t possibly be that stupid… or can they?

I’ve been taking a stern look at the various websites out there, especially those hosted and created by people local to me, in my community. Disgusting. There is one “designer” (and I use that term very loosely), who is trying to snatch up all of the local businesses here with his “Word-to-HTML” template sites. He charges these sites $250.00 “setup fee” and $50-$500/month for hosting and updates to these sites.

He puts them all on dynamic yahoo-based “free” storage, and rapes the customer for these prices. No quality at all behind his work, and in fact, he takes the website content from other sites directly. I found a complete rip of some CSS in one of his sites from a site in .nl, and he didn’t even edit it out. In fact, the page’s title tags still referenced the .nl site. Here are two more examples:

http://www.captainfish.com/home.htm

http://www.brotherstoofishing.com/home.htm

Both sites, competitors of each other in the same town, created by the same person, using the same design (and ugly buttons, stuck in the 80’s of web-design), and hosted on the same servers. I’m sure they’d find it interesting to know that little tidbit.

So the end-result is that I’m taking this work, all of it, and am not going to give it back to these people, until they get some sort of clue about usability, design, and proper web techniques. I’ve emailed the person who did the two sites above with a 4-page message detailing all of his mistakes on all of his sites, pointing to the proper tools he should be using, etc. and he never replied or even said thanks. Shrug.

We’re going to make a killing in this town, once these businesses see what real quality can look like, at much less cost to them in the long run, for much greater speed, usability, and prompt attention to updates.

“..there are some 2.5 million servers running Linux and that SCO has ‘identified by name’ those companies running many of them.”

“We are in the process of contacting them about coming into compliance and taking a UnixWare license from us. If they refuse to do so, we will sue them directly and see them in court,” he said.

“In a nutshell, this litigation is essentially about the GNU General Public License and all it stands for. That license has not yet been challenged or tested in court, but it is now going to be. We are also firmly and aggressively challenging the notion that Linux is a free operating system,” McBride said.

http://www.eweek.com/article2/0,3959,1224000,00.asp

Does anybody else think that this charade has gone on long enough?

Where is the proof that any of this IP is actually in the code they claim that these 2.5 million servers are running? Are there really 2.5 million Linux servers running SMP/NUMA/RCU/JFS in the public internet? I’d be very surprised if that figure was true.

Has anyone (or their company) actually received a letter from SCO requesting (demanding?) a license to UnixWare, or face litigation from them? Wouldn’t this constitute mail fraud? If a company sends you a bill, through the United States Postal Service, and that bill cannot be proven to be valid, isn’t that considered fraud? Time to talk to the postmaster and see.

This game has gone on long enough. First it was a contract dispute case against IBM. Then it was an IP case against Linux (the kernel). Then it was an IP case against ANYONE using Linux (the operating system as a whole). Now it’s all about the GPL? There is only one company that spins FUD like this.. and it seems as though they are doing the actual speech-writing for SCO these days.

linux != Linux, and I think SCO and the media need to get straight on those facts.

Prove that my linux kernel is running your intellectual property, and I will remove the infringing code myself, and run it sans your IP. Period. This is how it works. If you can’t prove it, legally or morally, then I’m sorry, I (and everybody else) don’t owe you a damn thing.

I see lilo is up to his “banging-the-tin-cup” again in his latest antics on OPN. Let’s review some history of when I caught him doing this well over a year ago:

**** BEGIN LOGGING AT Sun Apr 15 16:02:17 2001
16:04:30 <lilo> listen, I am not paid to talk to perennially angry 
   people
16:04:39 <lilo> such I judge you to be at present 
16:04:55 <setuid> Are you saying you draw a salary based on 
   donations from the community?
[...]

dopey…and let’s not forget that OPN is a private network, not a public one.

**** BEGIN LOGGING AT Sun Apr 15 16:02:17 2001
16:07:10 <lilo> please see paragraph 2 of the motd 
16:07:19 <setuid> I've seen it, which you are legally violating.
16:07:23 <setuid> Which has also been noted. 
16:07:28 <setuid> Next? 
16:07:28 <lilo> this is a private network
16:07:37 <lilo> your access may be revoked at any time
16:07:40 <lilo> that part
16:07:44 <setuid> That's fine with me.
[...]
16:08:33 -lilo- lilo is ignoring you

What’s funny about this particular interchange (one of dozens lilo and I have had) is that I was g-lined from OPN for putting lilo on /ignore awhile ago, but he sees fit to put me on ignore.

Sorry, OPN is not “open”, nor friendly, nor does it in any way contribute to the furthering of any community spirit. When it ceased to be Linpeople, it become another entity entirely. Looking at all the suspicious things going on behind the scenes at OPN, I’m definitely directing people away from it. There are dozens of other freely available irc networks that cater to specific tastes, including my own server, that don’t force this level of “management” down on it’s userbase.

lilo, really. I’ve personally been out of work for over six months, and I’m not begging for money from anyone. I could always break down and work at McDonalds or as a school janitor, or mowing lawns. Drop the ego, and do what you must to support your family. This is getting ridiculous.

You don’t “deserve” a salary for setting up OPN, just as I don’t “deserve” one for all the unselfish giving I do for the community, in mailing lists, CVS hosting, gratis web development, IRC servers, and so on. I do it because it needs to get done, and it benefits the community as a whole. You also don’t have to personally micro-manage the network. An irc network, properly configured, runs itself. Delegate out the responsibilities, if you must. Let it be what it needs to be.

Enough already.

Dear Diary:

It’s been awhile since my last entry, and a lot has happened.

An acquaintence and friend of mine was murdered yesterday. Rex, you will be missed. I was just in CT last week, and he was asking about me through another friend. I should have stopped by his place and talked to him. Maybe that would have been even harder to swallow if I had.

Talking to a friend of yours, and then hearing that he’s been slain two days later.