random neuron misfires

This weekend was just as busy as the week at work. It’s Sunday afternoon, and I’m still going…

Reconstructing Maildir from Backups

Moments ago, I found that my archive of the Coldsync mailing list in Maildir format somehow became corrupt, so attempts to copy those messages to Gmail failed using my Thunderbird trick.

I found an older copy that was in mbox format, and used the “Perfect” mbox to Maildir converter script to convert it to Maildir format.

Now I’m back to populating Gmail with my email once again (8,427 in Gmail now, with about 112,000 left to go).

Calendaring Conundrum

Also this past week, I realized that my calendar in Outlook had somehow duplicated over 1,700 of my events. I’m sure it was the result of using things like PocketMirror and other sync tools for Palm with it. I’m going to be cleaning that up next. That requires manual, visual inspection of each event, to make sure I’m deleting the dupe and not the original (thus leaving the dupe copy on the calendar). Very odd.

Once that is done, I have to reinstall all of my Palm conduits on the Thinkpad X61s and get that all sync’d to my Treo. With everything on my Treo, I can then begin consolidating my various calendars and task lists into one clean interface.

Back to the Blog; 8 years of postings

I also cleaned up 8 years of blog postings, reformatted them all and cleaned up the broken HTML that was the result of importing the diary entries from Advogato. That was 353 separate posts to go through by hand and clean everything up. Now it looks as it should.

Going back and reading through those old diary posts was… interesting. I didn’t realize how much I’d done in those 8 years, all of the people I’d met, projects I’d completed, places I’d been. I might turn the whole blog into a set of memoirs for my daughter Seryn for when she’s old enough to understand all of the things her daddy did in his life.

Movies, movies, movies!

I managed to pack in watching 4 movies while I worked this weekend, but the two best ones were “Maxed Out” and “The Man from Earth“. Both of them were equally good, and worth watching. I highly recommend both of them.

“Maxed Out” was eye-opening, and depressing at points, because of the situation our country is in right now. People are literally killing themselves (3 cases are described in the movie) because of their debt. The industry specifically caters to those who can NOT pay their debts down, because those people are the cash-cow for them. These people pay their minimum payments for life and pass on their debts to their children. They don’t want people who pay their credit cards in-full every month to be their customers, there’s no profit in that. Watch the movie for the rest of the details.

“The Man from Earth” brings new ideas to our concepts of religion, biology, archeology and many other fields of traditional study. It reminded me somewhat of the information that was in the first 1/3 of “Zeitgeist: The Movie” (freely downloadable, or viewable online). The Man from Earth is a low-budget movie, but packs a punch in the back story. I won’t spoil it here, but definitely go rent it if you can.

AT&T WWAN in VMware

I managed to get my physical Windows machine (an HP machine I purchased at BestBuy a few years ago), virtualized and configured in VMware using VMware Converter. I had to hack into it to get the vm to recognize my legitimate Microsoft Product Key, but after that, it was a snap.



Then I installed and configured the AT&T Communication Manager software to talk to the physical SIM card inside my laptop, so I can go online with the laptop wherever there is valid GSM signal, at 3G speeds.

I didn’t think the vm would recognize the physical card in the machine, if Linux didn’t see it natively… but it does. It’s a bit slow, but at least I can function with one laptop connected to the WWAN on the train with a larger screen at 1920×1200 resolution, instead of the smaller laptop with the 12″ screen at 1024×768 resolution.

The next step is to get the second laptop networking across the connection that the first laptop provides. That should be interesting to solve, since one of these is Windows, and the other one is Linux.

One possible solution would be to take the SIM card that is physically inside the laptop, put it into an external 3G PC-Express card (as in the image here), and then put that into a WWAN router, and carry THAT with me on the train. It has to be portable, of course…

But if I use that approach, not only can I share the connection with both of my laptops, I can also provide “free” wireless to anyone on the train who wants to get online. Maybe I can solicit free beer or donations in the cafe car to help offset those costs.

New Web SEO

An old acquaintance from a Business Development Group in New London has recently contacted me asking me for help with his website. He pointed out that his site is losing customers to two other sites in his very narrow niche here in Connecticut.

I looked at the competition, and noted that they’re not doing anything special that would merit that increase in customers for them. But I also noted about a dozen problems with this person’s company website in question, that needs immediate attention. His site is ranking at PR0, and the other two sites are pushing PR1/PR2, with no real traffic.

So I think I might pick up a little side project to help him out, to bring his site up to where it should be, and up to the level of standards that I consider acceptable. It looks like it’ll be fun, and it should bring in more income to help fund some projects I’m working on in the background (mostly Seryn’s secret project :)

I make a decent amount of money with my websites now, and I think with the proper care and attention, he can too.

Wrestling MySQL

Speaking of websites, a few months back, I started making some MFA 2.0 websites of my own, based on WordPress that are populated with hundreds of public-domain articles on niche content.

To do that, I wrote some tools (in Perl, of course) to spider the remote sites, pull the articles, and stick them into my WordPress databases, complete with proper author in the Author field, original posting date and so on.

Here’s one example that took me less than an hour to create and populate with articles, using these tools.

This particular site has 220 articles in it, but if you look at them articles, they’re all linked to external quality citations and resources. This site, with zero marketing, pulls in an average of about 6,000 hits a month. I have a handful of others, with between 200 and 1,000 articles in each, all of similar high-quality and care behind their creation.

The hard part is that in order to have an article attributed to an author in WordPress, that person has to be in the User table. That gets complicated to get right, but I managed to figure it out.

But I needed a way to ensure that the site’s content didn’t remain “stale”, so I came up with another trick to do that:

UPDATE wp_posts SET post_date='1999-01-01' + interval rand() * 3391 day;
UPDATE wp_posts SET post_modified = post_date;

I put this in /etc/cron.weekly/, and now my articles are randomized on a weekly basis, so visitors or search engines coming to the site will constantly see “new” content.

So far, they’re doing well, and bringing in more than enough to cover the monthly costs of running the servers, bandwidth and power.

The Day Job

The day job continues to go well, and I’m picking up significant speed on those projects.

The more I work within the system, the more I see where some “optimization” can be put into place, including some automation to make things much easier for myself and others in my group. I need to carve out some time to do exactly that, within the limits of my time allotted for the critical tasks that have established deadlines.

The commute has worked itself out and my routine is fairly static now, so that is no longer an unknown. Now I just need to get my foot in the door and keep things moving forward at a cheetah’s pace.

Fun times, definitely fun times. The positive energy has come back in rushing waves, and the good luck has overwhelmed me.

I have over 10 years of email on my machine, which I refer to from time to time for various projects and historical reasons. Many of these emails are from very active mailing lists I’m still subscribed to. The total space consumed by all of these messages is currently 2.3 gigabytes, and it is stored in Maildir format.

I’ve been spending the last 2-3 years pushing myself to become more and more productive using a collection of various systems mostly based around David Allen’s GTD system. The whole premise behind David’s system is to “dump your head” into a trusted system, always filter every input through that system. There’s quite a lot more to it, but once you get the methodology down, it really, REALLY does improve how much you can do. Not only can you do more with less time, but you can do what you’re already doing now, and get a LOT more free time back in your day. No, seriously.

Click the image below for a full-size version:

My own “hybrid” system encompasses analogue and digital formats, because of my specific and unique needs for the kind of work I do. At the core of the hybrid system is my PDA; a Treo 680 smartphone. If you want to see why I use a Treo instead of an iPhone, read my previous post on the matter.

Still with me so far?

As expected, once I started proactively blocking the botnet from my production servers, they decided to launch a counter-attack against me…

But first, let me rewind a bit.

About 2 years ago, I started aggressively looking at my incoming traffic to determine who was hitting me, how frequently, and when. I needed to increase the performance, and reduce the number of misconfigured spiders and rss readers.

This analysis revealed that there were hundreds of thousands of requests happening on a weekly basis for identical content, every single day from these spiders and misconfigured rss aggregators (which happens to be most of them).

Every day, all day.

24×7, 7 days a week. Every week.

Repeatedly.

Ironically, the bulk of these requests was coming from the .cr domain… Costa Rica. Those requests alone, were more than 50% of my total outgoing bandwidth. They were requesting valid resources, valid files, valid data repeatedly, over and over and over.

So I blocked the entire country using iptables on port 80.

$IPTABLES  -A INPUT -p tcp -m iprange --src-range X.x.0.0-X.x.255.255 -m tcp --dport 80 -j DROP

But there were a lot more coming from China, Russia, Korea, Israel, and other places.

Then I noticed an almost-immediate change in the activity. Now I was being hit with multiple thousands of requests for non-existant content, all trying to hit my Mediawiki pages, Drupal pages, WordPress pages (all written in PHP, if you notice).

What they were trying to do, is generate bogus HTTP_REFERERs for my logs, which would point back to a malicious script that would hijack the machine of the person who clicked on the link from the web-facing statistics. They were also trying to hijack the wiki pages to include these links masqueraded as “valid” links.

I’ve examined quite a few of the malicious scripts (all written in PHP also), and over time, the scripts have changed. They were originally 100% readable, but are now obfuscated and encoded, so they prevent casual dissection. Apparently they looked at THEIR logs, and noticed people were looking at the code, and not just visiting the malicious URL that included that link.

The other thing these forged HTTP_REFERER requests do is cause any log analysis package written in PHP, to parse the code, thus hijacking the server itself. Lovely.

So I started blocking the IPs originating those requests too.

When I did that, I noticed another interesting trend. If I blocked 5 unique, malicious IPs in the first hour, 10 uniques would hit me in the next hour. If I blocked those 10 unique IPs, 50 would hit me in the hour after that. The more I blocked, the faster they started coming in. If I left those 50 unique IPs alone for 24 hours, it would remain constant… 50 unique, malicious IPs/hour, never changing.

If I blocked those 50, then 200 more would come in the next hour. The faster I blocked, the more “the botnet” would send my way.

And then they counter-attacked

Over the last 2 days, after finally blocking over 1,000 unique IPs, they decided to counter-attack, and hit my webservers with http requests which were constructed to intentionally drop the TCP connection, leaving Apache in a CLOSE_WAIT state. I think their attempt was to try to tie up Apache’s listeners so other “valid” users wouldn’t be able to get in.

But I’ve already worked around that with some sysctl and kernel tweaks.

And on top of that, I’ve now automated the blocking, and now instead of blocking them on port 80, I block them on all ports, all protocols, automatically.

# iptables-save | wc -l
1392

That’s 1,392 separate, UNIQUE IPs being blocked now on all ports. That number may continue to grow, but it won’t shrink. The more machines they hijack to try to reach my servers, the more I’ll continue to block.

These amateurs really need to find another hobby, this one is just getting old.

If you’re like me, moving to the latest 64-bit hardware has made an ENORMOUS difference in my productivity. I’ve moved all of my personal machines and servers to AMD64/4600+ machines with a minimum of 4gb of RAM.

The problem is that not everything moves over so seamlessly; case in point: prc-tools. The problem with prc-tools not functioning on AMD64 has nothing to do with John Marshall, the maintainer of prc-tools… John is a great person, once you get used to how he works and how he expects bug reports and submissions <ducking from johnm’s swing>

I use prc-tools in several-dozen hourly cron jobs to build Palm software for projects like Plucker and several others, and I wanted to decommission the existing AMD32 machine that was doing those builds up to this point to migrate everything to the faster, less power-hungry AMD64 machines.

I didn’t really want to have to move everything to the new AMD64 machines EXCEPT this one AMD32 machine powered up @400W 24×7 just to build Palm software. That would cost me too much money each month in power costs for a server which isn’t really being used for anything other than cranking out hourly builds of Palm software.

prc-tools is a series of patches to the mainline gcc compiler that we’re all used to using on Linux and other POSIX systems like BSD and Solaris. The problem is that the prc-tools patches are mated to gcc-2.95, which was released back in July 31 of 1999. 64-bitness didn’t even exist back in 1999, 8 years ago.

I’m also not the only one with this problem.

I started patching up gcc’s configuration files to detect 64-bit procs a bit better, but it dead-ended quite early. The autoconfiscation process doesn’t even detect the architecture via config.guess. Dropping in a more-recent config.guess and rewrapping configure.ac helped a little bit, but it died further on in the process. Iterative fixes got it quite far, but eventually I had to dive into gcc itself to patch it, and that’s an area I leave to more-seasoned experts than myself.

I’ve been thinking a lot about power and power-consumption over the last six months to a year or so. I have already replaced every single bulb in the house and office with CFLs. We went from 60W and 70W bulbs in the house to 11W, 13W and 23W bulbs inside and outside (our exterior spotlights were 100W+, and are now brighter, whiter, 23W bulbs).

I’ve replaced my ageing servers with AMD64 dual-core machines and their PSUs with Antec NeoHE 550W power supplies, and I’m very conscious about turning off bulbs in rooms I’m not in when I’m not there.

My last post touched on some power consumption issues, and I just found this Slashdot article, which talks about making the s3 suspend/wakeup work better on on all machines… and then I stumbled upon this email from Bill Gates (marked as Plaintiff’s Exhibit 3020 in Comes v. Microsoft). It states:

One thing I find myself wondering about is whether we shouldn’t try and make the “ACPI” extensions somehow Windows specific.

It seems unfortunate if we do this work and get our partners to do the work and the result is that Linux works great without having to do the work.

Maybe there is no way to avoid this problem but it does bother me.

Maybe we could define the APIs so that they work well with NT and not the others even though they are open.

Or maybe we could patent something related to this.

This was penned on Sunday, January 24, 1999 at 8:41am. I can’t believe the gall of Bill Gates, to even suggest such a thing, given our already-stressed power grid.

I’m going to get a Watt’s Up power meter to measure the consumption of some key devices in the house and office, or one of the Kill-a-Watt products that do the same thing.

I’m so happy I continue to run Linux, for so many reasons other than the Freedom it affords me (liberty as well as financial). Windows just can’t even come close anymore, even with Bill Gates actively trying to stifle our ability to evolve.

I’ve been seeing a sharp rise in the amount of “Gutenspam” sent to my mail servers lately. It started as innocent garbage paragraphs buried in HTML comment structures, so if you viewed it in a normal mail client that “rendered” HTML (dangerous), it wouldn’t be visible. If you viewed the source of the email, you’d see the random snippets of public poetry, books and other materials.

Here’s an example:

perturb this carport this open phony plant crawl I eyeful
these when beograd ex place ivy them befit good income
muncie differ sheep prophet eyeful match made bull one merritt
stonecrop downpour degrade four sheep adolescent does spoon place proust
match head by about electrolyte inconsolable turn z's great pastor
baylor off sheep downpour cabinetmake ywca boy income change bull
furl calendar wreak when sis throaty change deliquesce word berserk
downpour build tribesman cubicsis inexperience late persia line muncie
z's many mud answer chef sony city ridgway now brownell

This does nothing at all, except to possibly try to confuse the filters which are searching for ‘spammy’ words, by lowering the threshold of the Bayesian filtering schemes. In the case of dspam, it does absolutely nothing, and dspam happily catches all of them and marks them as spam, quarantines them and users never see them.

But I see 100-200 per-day now, coming in. 99.514% of them are caught, but some are slipping through.

There was a story on NPR the other morning about how these spammers are just taking public works from projects like Project Gutenberg and including them in their email body.

But they’re not selling anything.

No links to some Viagra site in Singapore.

Nothing useful, except garbage random snippets from etexts and other literary works on the web.

Gutenspam.

The other funny trend I’ve been seeing, as more and more people are exposed to spam, is the emails which come across looking like this:

Date: Wed, 9 Aug 2006 10:50:11 -0400
From: "Al B. Sure" 
To: David A. Desrosiers 
Subject: RE:%%SUBJECT%%

%%BODY%%

The funny part about this, is that it too has no useful content at all, just some unpopulated macros. Its like there are more people trying to use some of the newer spam applications, and these people don’t have a clue what they’re doing with them.

They’ve probably forgotten to add their text to the e-mails to fill in these macros, or are just not reading the documentation on how to successfully send their spam to people.

What a waste of perfectly good oxygen recycling organisms.

I was pointed to this interesting writeup describing how spammers are now using a sophisticated “host cloaking” technique to hijack valid IP addresses to send their spam through web tunnels to the outside world, thwarting detection and having their accounts deleted/disabled.

It goes like this:

1. The spammer obtains a dedicated server at the victim service provider. The server shares a subnet with other customers.
2. The spammer runs a special daemon program on the dedicated server. The daemon places the network interface into “promiscuous mode” so that it will snoop on all network packets, spying on the local subnet.
3. The daemon determines which IP addresses on the local subnet are not in use. It also determines the addresses of the network routers. One or more unused IP addresses are commandeered for use by the spammer.
4. ARP (Address Resolution Protocol) responses are sent from the daemon to the routers, binding the unused IP addresses to the server. This allows the spammer’s server to “steal” those IP addresses. The daemon does not answer ARP requests from any other source, so the stolen IP addresses remain invisible to all other systems and diagnostic equipment.
5. Finally, GRE and IPIP tunneling (a method used to connect two private networks together) is used to connect the stolen IP addresses to the spammer’s real servers hosted elsewhere.

“The end result is that the spammer has created a server at an IP address which not even the owners of the network are aware of.”

I’ve been using dspam for a few years now, with great success, but over the last month I’ve noticed a trend… After 3-4 years of not a single spam slipping through to a mailbox, I am now seeing 10-15 of them per-day getting through. The filters are still reporting a solid accuracy rate (MUCH higher than SpamAssassin), but spam is slipping through.

Overall accuracy (since last reset)     99.590%
Spam identification (since last reset)  99.395%
Spam ratio (of total processed)         48.265%

I’m also actively blocking IPs of known spammers through the firewall rules:

# iptables-save | grep -c "dport 25"
13165

That’s over 13k unique IPs that have reached our server trying to send spam, malware and other garbage to our users. Normal mail from valid hosts is still flowing in and out, as it should…

I’ll have to see what else I can do to slow it down or stop it again.

I was just diagnosing some weird kernel crashes on the BSD server, and decided to pop open the case to take a look and add some more cooling. The inside of the case was pretty hot, so I added two new Antec Pro Cooling fans, mounted inside with some stiff clipped coathanger over the drives and the CPU cooler (which has its own fan on it also). The case is roomy enough to handle it, so no biggie.

As I was closing up the case, the sliding lid didn’t seem to fit right. Its slotted tabs kept running into something. I looked closer, and it looked like a clump of dust or fuzz or something was wrapped around one of the wires.

I reached in and grabbed it to remove it and it was… how shall I say… stiff and crunchy, not like the dustball I expected to grab. I put it in the light so I could see it and… EEEECH!

Wrapped around one of the case’s signal cables, was the empty shell-of-a-carcass, of a little grey field mouse. There was only the top of a skull, the fur coat around the back, and a little segmented tail.. and two eyes. No organs, no skeleton, nothing at all that would make up muscles, bones or body.

Inside the bottom of the case, were thousands of bits of what looked like more dust, but apparently was the remnants of what used to be its coat, fur, skeleton and organs.

YUCK!

I cleaned it out, and all should be well again…. I hope, but where there’s one mouse, there’s bound to be more. I’ll have to start opening up the other servers and see what might in their cases. Maybe a few mouse traps behind the rack would be useful.

I’ve been configuring one of my machines as a FreeBSD 6.1 server in my spare picoseconds, to serve as a fileserver, rsync server, public mirror, backup server, transparent squid proxy, coffee maker and whatever else I can make it do.

In order to do this, I needed to add a bunch of packages and tweak quite a few hundred things in various places on the system (sysctl, make.conf, loader.conf, add missing tweaks and options to the kernel config) and so on.

I rebuild kernels nightly and keep ports up to date with cvsup. I rarely find time or a need to run portupdate or buildworld, but I know I should, so I found this useful article which describes how to stay current with all of these pieces (and there are a LOT of them).

Buried in the comments, was a pointer to ‘sysutils/fastest_cvsup‘, which is probably the BSD ports equivalent of Debian’s netselect-apt tool. fastest_cvsup is described as follows:

Perl script to find fastest CVSup server:

* uses socket connections not just 'pings'
* takes notice of server responses
* connects to servers in countries specified on the command line 
   - or - 
  connects to the 'local' servers defined in the script
   - or -
  connects to ALL the servers in ALL the countries
* returns either fastest server or top 3 (useful for scripts)
* returns exit codes (useful for scripts)
* can re-write itself to update the CVSup server list, obtained
  from the online FreeBSD Handbook
* can easily add other CVSup servers (NetBSD/OpenBSD...etc)

WWW: http://fastest-cvsup.sourceforge.net/

Running it was as simple as executing it with the right country in mind (or you can pass it ‘-c all’ and test all of them):

# fastest_cvsup -c us
>>  Querying servers in countries: us
--> Connecting to cvsup.us.freebsd.org [198.104.69.57]...
    - server replied: ! Access limit exceeded; try again later
    - time taken: 146.89 ms
--> Connecting to cvsup2.us.freebsd.org [130.94.149.166]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 42.03 ms
--> Connecting to cvsup3.us.freebsd.org [128.31.0.28]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 35.09 ms
--> Connecting to cvsup4.us.freebsd.org [204.152.184.73]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 114.04 ms
--> Connecting to cvsup5.us.freebsd.org [64.157.15.40]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 74.04 ms
--> Connecting to cvsup6.us.freebsd.org [69.31.98.210]...
    * error: connect: Invalid argument
--> Connecting to cvsup7.us.freebsd.org [129.250.31.140]...
    - server replied: OK 17 0 SNAP_16_1g CVSup server ready
    - time taken: 104.05 ms
--> Connecting to cvsup8.us.freebsd.org [216.165.129.134]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 57.24 ms
--> Connecting to cvsup9.us.freebsd.org [128.205.32.21]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 53.00 ms
--> Connecting to cvsup10.us.freebsd.org [128.205.32.10]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 52.02 ms
--> Connecting to cvsup11.us.freebsd.org [63.87.62.77]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 33.05 ms
--> Connecting to cvsup12.us.freebsd.org [128.46.156.46]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 49.02 ms
--> Connecting to cvsup13.us.freebsd.org [216.144.193.227]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 50.02 ms
--> Connecting to cvsup14.us.freebsd.org [64.78.150.180]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 156.16 ms
--> Connecting to cvsup15.us.freebsd.org [131.193.178.106]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 50.05 ms
--> Connecting to cvsup16.us.freebsd.org [128.143.108.35]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 65.03 ms
--> Connecting to cvsup17.us.freebsd.org [65.212.71.21]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 43.05 ms
--> Connecting to cvsup18.us.freebsd.org [128.205.32.37]...
    - server replied: OK 17 0 SNAP_16_1h CVSup server ready
    - time taken: 56.02 ms

>>  Speed Daemons:
    - 1st: cvsup11.us.freebsd.org   33.05 ms
    - 2st: cvsup3.us.freebsd.org    35.09 ms
    - 3st: cvsup2.us.freebsd.org    42.03 ms

So now I can stick cvsup11.us.freebsd.org in my .sup files and hopefully gain the fastest connection to those servers for updates.

I’ve been a long-time Mozilla user. I love the interface, I love the fonting, I love the extensions, and in general, it loaded and ran faster than Firefox.

But I’ve now switched over to Firefox for all of my browsing. I have a ton of extensions loaded in it to make it useful for my day’s work (which is to say, my most-used tool, next to email and gcc).

Here’s a list of the extensions I’m currently using in my Firefox build (you can see how I have it tricked out with all of my theming and extensions over here):

• Sage, a really slick and fast rss aggregator/reader for Firefox. It docks on the sidebar and is visible with a simple Alt-S keystroke. Very nice, and easy for me to catch up on some quick headlines when I need to.
• AdBlock Plus with the AdBlock Filterset G Updater to stop the flood of useless ads from coming at me. I did have to add one small rule for Google’s ads, because I do actually like the recommendations they provide from time to time, and it helps out sites I visit with a little revenue. That regex looks like this:

  @@*.googlesyndication.com/*

• Web Developer, a very useful and slick toolbar/menu driven suite that allows me to do all kinds of things to websites I’m viewing, including validation, showing where their css classes are, manipulating forms, cookies, images, and dozens of other features. Hands-down, the most-useful extension I have as a developer/tweaker of web content.
• PrefBar, another powerful extension I use every single day. This one allows me to change the capabilities of my browser with a simple click of a checkbox. Want Java enabled? Click. Sick of popups? Click. I have Colors, Images, Javascript, Java, Flash, Popups, Proxies, Pipelining, Referers, Cache on my bar. Its completely customizable, and very well-done.
• SwitchProxy lets me manage and switch between multiple proxy configurations quickly and easily. I can also use it as an anonymizer to protect my system from prying eyes. I have Squid, Squid + Privoxy, Privoxy + Tor and i2p enabled in my configuration at the moment. Quick and easy, and one status-bar dropdown lets me change from one to another.
• FasterFox gives me a little boost by auto-configuring some parameters for faster browsing, such as link prefetching, pipelining, DNS cache, paint delay, and others.
• ForecastFox, weather.. in my status bar. I’ve changed the icons a bit with a separate icon pack called Lansing, which is nice adn small and out of the way. Minimal is the way to go on my toolbars and status bars.
• Linky lets me open or download all or selected links in a page, image links and even web addresses found in the text in separate or different tabs or windows. A simple right-click on any link or web address, and away I go.
• Google PageRank Status gives me a quick overview of the PR of a site in the current view. This is useful as I do a lot of web work, and knowing what kind of sites get a decent or poor PR is useful information.
• SearchStatus is another SEO toolbar for Firefox, which I use quite a bit. With this extension, I can see a site’s Google PageRank, Google Category, Alexa popularity ranking, Alexa incoming links, Alexa related links and backward links from Google, Yahoo! and MSN, and others. Beautiful and easy. It sits quietly in the status bar and out of the way until I need it.
• FireBug is another great web developer extension, which shows me exactly how pages are failing when they error out. I can step into the code via the DOM, and see exactly what went wrong. OTHER pages of course, my pages never have troubles…
• Google Advanced Operations Toolbar uncovers the often-cryptic syntax that Google uses to search in more detail to find information. Need to know how to use the ‘site:’ syntax? Just use this toolbar and it’ll do it all for you. Quick, easy, simple.
• DownThemAll! is a downloader for Firefox. With this, I can right-click a page of links, and specify by a wildcard or any of the preloaded templates, which links on the page I want to download. Want to download all of the Linux 2.6 kernels matching a specific version? This can do it in one click. Very well-done, slick, and useful when you want to download a lot of links from one particular page.
• TamperData gives me the power to monkey with the form data being received or sent to servers. Want to malform that POST request? TamperData can do it. Need to send more parameters in with that form submission? This extension can do it. You can trace and time http responses and requests, validate your web applications against security issues by stuffing garbage into POSTs, and more.
• RankQuest SEO Toolbar, yet another SEO tool I use quite a bit. This one gives me access to over 25 different SEO tools to check, test, and qualify websites against their SEO health.
• HyperWords is probably the second-slickest extension I have. I can highlight any word or series of words on a page, and a menu will pop up allowing me to search major search engines for those words, or look them up on dictionary and reference sites, Wikipedia, stock exchange, IMDB and dozens of others. I can blog about the highlighted words, map them, translate them, and a truckload of other options. AMAZING extension.
• Free eBook Search lets me search the highlighted text for ebooks on Free eBook Zone. I can search using the Book Title, ISBN (10 Digit), Description, Book Author and even the ebook backward link.
• CacheView gives me the power to see a site’s cached copy through Google’s(tm) Cache, Coral Distribution Network‘s Cache, Wayback Machine‘s Cache, Dot Cache, and Tech Guru’s Cache of the current tab open via right-click.
• Live HTTPHeaders shows me the actual headers being passed in every request of a page or content. Want to make sure those headers in your web application are showing accurate data? This will do that for you.

These are the extensions I use every day, in my browser. Without them, I’d be spending a LOT more time hunting down links, sites, creating Javascriptlets, and lots of other tools. You can see what the whole extension list looks like in this screenshot.

As I mentioned, I tend to use Firefox with a lot of proxies (Squid, Tor, Privoxy, i2p and others). This generally means poking at non-standard ports. Until recently, this hasn’t been a problem for me..

But today, I noticed I can’t get to “non-standard” ports under 1024 anymore, with the current 1.5.0.3 version of Firefox.

To see this in action, point Firefox to http://www.example.com:72 and see what you get. In my case, I see:

This address is restricted
This address uses a network port which is normally used for purposes other than Web browsing. Firefox has canceled the request for your protection.

But there’s a way around it! Mozilla has Port Blocking enabled by default.

To enable some ports in Firefox, simply do the following in your user.js file

user_pref(“network.security.ports.banned.override”, “72,73,74”);

To disable ports, use this construct:

user_pref(“network.security.ports.banned”, “81,90”);

For an easier way, type ‘about:config‘ in your browser’s URL field, and you will see all of the tunable settings that Firefox has to offer.

Within these settings, you can create value that will allow or disallow these ports. Follow these steps:

1. In the list of values provided, right-click any line and choose “New -> String”
2. A dialog box will pop up asking for the name. Type ‘network.security.ports.banned.override’ into this box and hit enter to save the value. Do not include the single-quotes when you add this name.
3. A second box will pop up. Type each port number you need to use, separated by commas, into this box, for example ‘72,73’ (again, do not include the single quotes)
4. Click on “Ok” to confirm and save these values.

Now you should be able to access these ports on the servers that require them.

Perhaps this little “feature” is there to protect Windows users from being exploited by malware or phishing attacks, but it certainly got in the way of my daily use of Firefox when I realized it.