random neuron misfires

The corporate VPN software I use to get onto the work LAN uses a Java-based “viewer” applet in a browser to get to my desktop machine (think “remote desktop”, but using a browser + Java applet), is an HTML page which uses Flash to deliver a Java applet, which then is used to do the “remote desktop” functionality. Yes, you read that right… HTML delivers the Flash which delivers the Java.

But I’ve recently rebuilt one of my Windows machines here at the office to include a 64-bit version of Windows Server 2003, and because of that, it’s been a bit of a struggle to find and reload all 174 applications I was previously using on the previous machine, in 64-bit format, as well as the 64-bit drivers necessary to support the peripherals (network, disk, scanner, printer, etc.). One of those was making sure my browsers were functioning correctly for everything I use it for, including accessing the VPN when I work remotely.

To that end, I installed the 32-bit version of Firefox for Windows, then the 32-bit Java SE Runtime 6u13, but that didn’t let Java applets to function in Firefox at all. about:plugins showed that Java u13 was seen and enabled as a valid plugin but applets would not work in Firefox. I tried using the Java Applets Test page, and it would just show a blank region where the applet should have been.

So I tried downloading the 64-bit Firefox for Windows and the matching 64-bit Java SE Runtime 6u13, but that too failed.

After I installed both versions of Java onto the machine, I decided to try the VPN connection using Internet Explorer, which worked, so I knew Java and Java applets were functioning correctly.

After some poking around, I found that there was one minor tweak that was required to get the Java applets to function in Firefox (either bitness).

While MSIE is loaded, go to the Java Test page and while there, you’ll see a Java applet icon in your Taskbar. Right-click on that, and you’ll see something like the following:

In the Java Control Panel, there is an option under (Advanced | Java Plug-in) labeled “Enable the next-generation Java Plug-in (requires browser restart)”. Disable (uncheck) that checkbox to enable Java to function correctly with Mozilla, as shown below:

Also, verify that Mozilla is enabled under “Default Java for browsers”, as shown below:

Click Apply, click Ok, then launch Firefox and try your applet again. You should see something like the following:

That’s it, you’re done! Now Java u13 should work fine in Firefox on 64-bit, without any issues.

Whew!

This week has been absolutely amazing for me, lots of postive successes on all fronts; work and personal. So much energy and glow all around, it’s so very motivating and invigorating.

With all of this activity going on, I’ve been focusing a lot on managing my calendars a bit more efficiently than I probably was before. I have a LOT of calendaring to manage. Here’s a small example of my calendar for this
week (this is a screenshot of my Microsoft Outlook instance below):

To that end, I’ve (re)discovered the Chandler Project after putting it down a few years ago and also something new called “Funambol” (formerly the sync4j project, based on SyncML).

Chandler unfortunately doesn’t build or run on 64-bit Linux systems, so I can’t do any real testing or integration with that at this point. This is 2009, right? 64-bit systems have been around for several years now; at least
long enough to be able to execute a port of Chandler to make it work there. But now it seems that Chandler may be dead now, with Mitch Kapor leaving the project
and taking his funding with him.

Lovely.

Conversely, Funambol installed very easily, runs smoothly, and their administrative GUI is very comprehensive (though, being written in Java, the controls themselves are quite clunky and non-intuitive). There’s a lot of
complicated bits in Funambol, so their documentation portal is a necessary stop if you’re going to install and configure it on your server.

Now the bad news: Neither of them solves the actual problem: universal, location-free calendaring.

What do I mean by that? What I mean is… treating separate calendars as separate in their own realm, but aggregating the relevant pieces where appropriate in the overlap realms.

This means that my personal Microsoft Outlook session running inside VMware, configured with at least 5 separate calendars (Personal, Work, Events, Holidays, Merged) should be treated as separate, and not glommed
together into one sloppy calendar when events are pulled from the individual source calendars. Almost every single sync tool and plugin that claims to support multiple calendars does exactly this. I’ve tried dozens
of them. This is wrong!

Funambol looked promising… except they got it wrong too. For example, the Funambol Outlook plugin only supports one calendar at a time (and sub-folders). You can’t point it to multiple calendars and have it treat them as
separate or even as one merged calendar. Useless.

The Funambol Blackberry app only supports 1 month of prior events and and 6 months of future events when doing an OTA (“Over the Air”) sync to the Funambol server. Also useless.

The Blackberry software, through Blackberry Desktop Manager (based on Intellisync) handles this somewhat cleanly, by allowing me to define as many folders (Calendar folders, Contact folders, etc.) as I want, and then
designating one of those folders as the “default” folder in the sync pairs. Any item I create on the Blackberry goes into that “default” folder, which makes its way to Outlook in the manner expected. If I need it to exist in
another folder, I’ll move it to the other folder when it gets to the Outlook side.

Unfortunately, Blackberry devices are still years behind Palm OS devices with regard to PIM functionality. They’re barely out of diapers. Blackberry devices don’t support Categories for calendar items, for example.
Contacts can have categories, but Calendar items can’t? Huh? HUGE oversight. This is almost as braindead as Apple omitting the very basic Cut and Paste functionality from the iPhone.

Is this really that complicated to solve? Really? In 2009?

So I’ve engineered a process that allows me to keep a clean copy of events on my Blackberry (sans categories, grr!), which is then synchronized over local USB with Outlook running inside a VM (so I can get my work calendar
on my handheld and push my on-handheld events back to Outlook). Outlook is connected to several of my Gmail accounts and Cozi for some additional calendars.

From there, I use a product inside Windows called SyncMyCal to push my calendars upstream to Google Calendar, where they are then marked as either private or shared, depending
on who I need to have access to them.

On the Linux side, Evolution then subscribes to those calendars in Google Calendar, so I can see them from Evolution.

Confused yet? I am too.

My goal is to try to get one central “repository” where all of my PIM data (Calendar, Contacts, Tasks, Memos) can reside. That repository would then have connectors to the other applications and services to manage that PIM
data. Each application would either push, pull or sync their respective bits of data into the repository, where it would then be immediately available/replicated to the other components in the system (based on permissions
and rules, of course).

I thought Chandler and/or Funambol would be this repository. I was wrong, and we’re back to the beginning again.

But at least I have a process where I’m not losing data, I’m keeping everything centralized in one place on my handheld (heavily locked down and very strongly secured; thank you RIM), and I can push it to wherever I
need it from there.

Not only am I trying to augment the lack of core function in the Blackberry to bring it up where my Palm OS Treo 680 was, but I’m trying to merge, maintain and manage 1/2 dozen calendars in at least 3 different systems
(Linux, Windows and Web).

The search continues…

I stumbled across a site called “FLOSS Manuals” recently, and thought that it would be a great place to create some new Plucker documents for our users, and distribute them. I’ve create hundreds of other Plucker documents for users in years past, so this was a natural progression of that.

You can (and should!) read all about their mission and see what they’re up to. Credit goes to Adam Hyde (the founder), Lotte Meijer (for the design), Aleksandar Erkalovic (developer) and “Mr Snow” for keeping the servers running, among many other contributors.

The FLOSS Manuals Foundation (Stichting FLOSS Manuals) creates free, open source documentation for free, open source software. FLOSS Manuals is a community of free documentation writers that publish free manuals about free software across multiple languages.

Free software can be freely run, studied, redistributed and improved without the restrictive and often expensive licensing systems attached to commercial proprietary software. Developers can adapt free software to their own needs, and can contribute to its ongoing communal development. FLOSS Manuals specifically document software that is free in this development sense and also in price. Free software projects are developed using established methodologies and tools, and sites like Savannah and Sourceforge support established social production models for developing free software. FLOSS Manuals provides the methodologies, tools, and social production models for developing documentation of free software.

By supporting quality, user-friendly documentation of Free, Libre, Open Source Software, FLOSS Manuals aims to encourage the use of this software, to support the technical and social revolution it enables.

If you want to support their cause (and I strongly recommend you do), you can visit their bookstore directly. (Note: I get absolutely nothing for hosting a link to their bookstore here; no affiliate links or commisions whatsoever).

When I quickly Googled around, I found someone was already doing exactly that, albeit in a one-off shell script.

I decided to take his work and build upon it, making it self-healing, and created what I call the “Plucker FLOSS Manuals Autobuilder v1.0” :)

This is all in Perl, clean, and is self-healing. When FLOSS Manuals updates their site with more content, this script will continue to be able to download, convert and build that new content for you… no twiddling necessary.

The code is well-commented, and should be clear and concise enough for you to be able to use it straight away.

Have fun!

#!/usr/bin/perl -w

use strict;
use warnings;
use diagnostics;
use LWP::UserAgent;
use HTML::SimpleLinkExtor;

my $flossurl    = 'http://en.flossmanuals.net';
my $ua          = 'Plucker FLOSS Manuals Autobuilder v1.0 [desrod@gnu-designs.com]';
my $top_extor   = HTML::SimpleLinkExtor->new();

# fetch the top-level page and extract the child pages
$top_extor->parse_url($flossurl, $ua);
my @links       = grep(m:^/:, $top_extor->a);
pop @links;     # get rid of '/news' item from @links; fragile

# Get the print-only page of each child page
get_printpages($flossurl . $_) for @links;

############################################################################# 
#
# Get the pages themselves, and return their content to the caller
#
#############################################################################
sub get_content {
        my $url = shift;

        my $ua          = 'Mozilla/5.0 (en-US; rv:1.4b) Gecko/20030514';
        my $browser     = LWP::UserAgent->new();
        $browser->agent($ua);
        my $response    = $browser->get($url);
        my $decoded     = $response->decoded_content;
 
        # This was necessary, because of a bug in ::SimpleLinkExtor,
        # otherwise this code would be 10 lines shorter. Sigh.
        if ($response->is_success) {
                return $decoded;
        }
}


############################################################################# 
#
# Fetch the print links from the child pages snarfed from the top-level page
#
#############################################################################
sub get_printpages {
        my $page = shift;

        my $sub_extor   = HTML::SimpleLinkExtor->new();
        $sub_extor->parse(get_content($page));

        # Single out only the /print links on each sub-page
        my @printlinks  = grep(m:^/.*/print$:, $sub_extor->a);

        my $url         = $flossurl . $printlinks[0];
        (my $title      = $printlinks[0]) =~ s,\/(\w+)\/print,$1,;

        # Build it with Plucker
        print "Building $title from $url\n";
        plucker_build($url, $title);
}


############################################################################# 
#
# Build the content with Plucker, using a "safe" system() call in list-mode
#
#############################################################################
sub plucker_build {
        my ($url, $title) = @_;

        my $workpath            = "/tmp/";
        my $pl_url              = $url;
        my $pl_bpp              = "8";   
        my $pl_compression      = "zlib";
        my $pl_title            = $title;
        my $pl_copyprevention   = "0";
        my $pl_no_url_info      = "0";
        my $pdb                 = $title;

        my $systemcmd   = "/usr/bin/plucker-build";

        my @systemargs  = (
                        '-p', $workpath, 
                        '-P', $workpath,
                        '-H', $pl_url,
                        $pl_bpp ? "--bpp=$pl_bpp" : (),
                        ($pl_compression ? "--${pl_compression}-compression" : ''),
                        '-N', $pl_title,
                        $pl_copyprevention ? $pl_copyprevention : (),
                        $pl_no_url_info ? $pl_no_url_info : (),
                        '-V1',
                        "--staybelow=$flossurl/floss/pub/$title/",
                        '--stayonhost',
                           '-f', "$pdb");

        system($systemcmd, @systemargs);
}

A top Republican Internet strategist who was set to testify in a case alleging election tampering in 2004 in Ohio has died in a plane crash. Michael Connell was the chief IT consultant to Karl Rove and created websites for the Bush and McCain electoral campaigns.

When rescue attempted to gain information about the accident, so they could determine what equipment to bring, they were told that information was “locked-out“:

Captain Geisner expressed considerable frustration during several Raw Story interviews over what he alleges was the withholding of critical details by authorities.

“While en route to the fire, I asked dispatch to learn the size of the plane and the number of souls on board,” Geisner explained. “This was not provided us.”

Such details allow fire department officials to determine whether additional equipment is needed and if a wider search and rescue is required. Within fifteen minutes of the crash, after officials from Akron-Canton Airport had arrived on the scene, Geisner again sought to confirm the number of passengers.

“After calls were made I was told that the ATC [Air Traffic Control] was ‘all in lockdown,’ and that they said ‘we can’t release that information,'” Geisner said.

Todd Laps, Fire Chief of the Akron-Canton airport fire department and a liaison to the Transportation Security Administration and the Air Traffic Control, echoed Geisner’s account.

“I had some phone calls placed to see if I could get that number [of people on board]. It didn’t come in a timely enough fashion,” Laps said.

But Laps says that the words “lock down” were not used. When asked to clarify his earlier comments, Gaisner insisted that the words “lock down” had been used in reference to information.

“That info was not available,” Gaisner said. “It was secured — not given out — locked down.”

Michael also set up the official Ohio State Election website reporting the 2004 Presidential Election returns. Connell was reportedly an experiences pilot. He died instantly on December 19, 2008 when his private plane crashed in a residential neighborhood near Akron, Ohio.

Connell’s track record makes it clear why he was viewed as a liability in some circles.

Ohio Republican Secretary of State J. Kenneth Blackwell hired Connell in 2004 to create a real-time computer data compilation for counting Ohio’s votes. Under Connell’s supervision, Ohio’s presidential vote count was transmitted to private, partisan computer servers owned by SmartTech housed in the basement of the Old Pioneer Bank building in Chattanooga, Tennessee.

Connell’s company, New Media Communications worked closely with SmartTech in building Republican and right-wing websites that were hosted on SmartTech servers. Among Connell’s clients were the Republican National Committee, Swift Boat Veterans for Truth and gwb43.com. The SmartTech servers at one point housed Karl Rove’s emails. Some of Rove’s email files have since mysteriously disappeared despite repeated court-sanctioned attempts to review them.

At 12:20 am on the night of the 2004 election exit polls and initial vote counts showed John Kerry the clear winner of Ohio’s presidential campaign. The Buckeye State’s 20 electoral votes would have given Kerry the presidency.

But from then until around 2am, the flow of information mysteriously ceased. After that, the vote count shifted dramatically to George W. Bush, ultimately giving him a second term. In the end there was a 6.7 percent diversion—in Bush’s favor—between highly professional, nationally funded exit polls and the final official vote count as tabulated by Blackwell and Connell. [source]

Watch this video, filled with pieces from Stephen Spoonamore, Cliff Arnebeck and others describing exactly what they believe happened here:

Michael Connell was deposed 1 day before the election this year by attorneys Cliff Arnebeck and Bob Fitrakis about his actions during the 2004 vote count and his access to Karl Rove’s email files and how they “went missing”.

Bob Fitrakis, co-author of “Did George W. Bush Steal America’s 2004 Election?” at the Election Protection Forum;Ohio Green Party candidate

Velvet Revolution reported that Michael Connell was afraid that Bush and rove would “throw him under the bus” if he was to expose what he knew or covered up.

Some pretty shocking news is coming out about the crash, much of it seems… eerily familiar.

“Under different political circumstances it might be possible to dismiss the Eveleth crash as a tragic accident whose causes, even if they cannot be precisely determined, lie in the sphere of aircraft engineering and weather phenomena. But the death of Paul Wellstone takes place under conditions in which far too many strange things are happening in America.

Wellstone’s death comes almost two years to the day after a similar plane crash killed another Democratic Senate hopeful locked in a tight election contest, Missouri Governor Mel Carnahan, on October 16, 2000. The American media duly noted the “eerie coincidence,” as though it was a statistical oddity, rather than suggesting a pattern.” [source]

An interesting anecdote about the Wellstone plane crash was that FBI agents arrived on the scene within an hour of first-responders reaching the site. This would be impossible, given the remote location of the crash, unless they left to head to the crash site, just as Wellstone’s plane was leaving the runway to take off for the flight.

We apparently have a fascination in this country by engineering cover-ups around aircraft, it seems, by this “crime family masquerading as a political party”.

This follows shortly before members of the White House were ordered to search employee computers for missing e-mails.

“A federal court ordered on Wednesday all employees working in the Bush White House to surrender media that might contain e-mails sent or received during a two and a half year period in hope of locating missing messages before President-elect Barack Obama takes over next week.”

Not only were emails missing, but 700 days of emails went missing. That’s not just a mistake, that is intentional. That is almost TWO FULL YEARS of emails and backups that just up and vanished?

If your primary job is a System Administrator or Mail Administrator, missing a tape or two happens, forgetting to schedule a weekend backup happens… but forgetting to back up mail for two full years? No. That’s not a mistake, that’s a cover-up.

The Presidential Records Act of 1978 specifically holds the President responsible for this.

“Places the responsibility for the custody and management of incumbent Presidential records with the President.”

So when are we going to start holding these United States Citizens personally responsible for the crimes they’ve committed against the best interests of this country?

2008 has been one of the best years ever… in so many ways. So much has changed and changed me, to push me further and faster forward. I don’t think I’ve felt this good, this motivated or this happy about where I am in at least 20 years.

• I’ve finally released myself from the shackles of my marriage after spending 3 years and ~$40k slogging it through the courts to keep my daughter in my life. I was with my girlfriend-then-wife for 8 years, and realized over time that she wasn’t the person I married. I also refuse to tolerate adultery in my marriage, so I ended it. Being single and alone for over 3 years is tough, but not a permanent problem.
• I’ve created a bond with my daughter that transcends words and sign language. She’s coming into her own and is learning some new abilities to “talk” and “listen” without using words or traditional language as we know it. She’s a beautiful, amazing little girl with an eye-opening list of skills and abilities, even at her tender age. Sure, life as a single father is difficult and challenging, but I’ll sacrifice whatever I need to, for her to grow up happy, healthy and reaching any goal she wants for herself.
• I’ve met some interesting and amazing people this year, some of which who have shown me facets of myself that long lay dormant, atrophied or discarded as unreachable anymore. The good, bad and ugly were all on display this year. Michele, Marci, Jennifer, Valerie, Lilibeth, Amy, Allena, Rachel, Pat & Lisa, Casey and the rest of the Edwards family and many others; you all know who you are, and in some small and large ways, you’ve made a permanent impact on my life. The genie is out of the bottle, and isn’t going back in.
• I work with a great group of people in Manhattan, doing some really cool things with technology and environments that I’ve worked with and supported for 15+ years. I wouldn’t give this up for the world at any price. I truly love what I do. I do what I do for free, I just collect a salary to pay for all of the things I don’t necessarily like doing. Ed, Phil, Dennis and others; let’s hope the fun can continue through 2009 as it did in 2008.
• After 16 years of searching inside my head for the lost memories from the first 20 years of my life, I’ve recovered what I believe to be more than half of them, all in the last 10 months or less. Remembering “other people’s” memories is a bit disconcerting, but it’s all part of the process. There’s plenty more work to be done, but after believing that bad car accident in 1992 permanently robbed me of those memories, I now know I can get back those years and begin recovering who I was, and what made me who I am today. Not only have those memories come flooding back, but I’m also able to retain thousands of bits of information at what seems to be a geometric rate, bordering on eidetic. It’s like my brain went from knowing nothing, to limitlessly absorbing everything.

There’s a lot more work to do in 2009 and years beyond, but I have no doubt I’ll be able to step up to any challenge that life throws at me. I’ve survived horrible accidents, sadness, strife and plight in all manner of ways, and I’ve always come through on top, always looking ahead, never behind.

I feel as if a huge weight was lifted from my shoulders and the metaphorical “smog” I was breathing, impeding my oxygen flow is now gone. I can stand tall, inhale clean, fresh air and move forward again.

I’m coming out of a 16-year coma and I’m just waking up.

This particular bit of yak-shaving all started because one of the Amtrak LSA staff asked me if I could write a tool to print out his MP3 collection by Artist, Album and/or Year. This LSA (Lead Service Attendant; they manage the café car) works as a DJ in his off-hours, doing various gigs for weddings and other parties.

So I took 15 minutes while traveling to the office to whip up something in Perl that did just that, and dumped it to a plain text file which I could then reformat in OpenOffice.org and then export as a pretty PDF he could print and hole-punch into his DJ binder. Problem solved, and he was impressed that it only took 15 minutes to cook that up.

And that’s when it started. The yak-shaving.

“yak shaving is what you are doing when you’re doing some stupid, fiddly little task that bears no obvious relationship to what you’re supposed to be working on, but yet a chain of twelve causal relations links what you’re doing to the original meta-task.”

Here’s how it began:

While building that list of Artist/Album/Title/Year, I realized that some of my mp3 files were missing some pieces of information. Some had the years missing, some had the genre mixed up, some were missing the data altogether.

So I went in and started fixing that.

Then I realized that the album art I was storing as “folder.jpg” was missing in some directories, and each time I rebuild my music library via amaoK or iTunes or anything else, I have to go re-fetch those missing album covers from Amazon or other online places.

So I went in and started fixing that too.

To do that, I had to use a Windows tool called Tag Tuner. I’m not a Windows person by any means, but there really is nothing as slick as TagTuner in Linux (yet?). There is kid3, but it lacks some pretty powerful features (but adds its own, like the ability to remove headers from the mp3 files).

I started adding in all of the missing cover art, storing the album art as an actual image file within the APIC field of the ID3v2 MP3 header. Some of the album art required that I scan in the actual covers from the CDs I have that aren’t available anymore, or aren’t online. Some of it was Google’d up, and others were found in other places on the ‘net.

It was (and still is) an enormous task to make sure every piece of the mp3 metadata is correct, album art is intact (including bootlegs, bonus albums, NFR [not for resale] albums and others).

Then I decided to try to “enhance” the Perl script I wrote, by slapping a web front-end on it, so I could sort by Artist, Album, Year, Genre and so on, and export that to a nicely-formatted PDF file for “Shaggy” (the Amtrak LSA/DJ) or myself.

I started down the path of looking into the Apache::MP3 Perl module on CPAN, which looked promising. When I Google’d up some example code, I saw a reference in an obscure Ubuntu forum post that mentioned using an Apache2 module called mod_musicindex, which supersedes Apache::MP3.

I installed and configured that, and found that there were some discrepancies in the configuration, and that some of the values in the default stanzas indicated in several web references on setting up mod_musicindex all pointed to. They were all incorrect. Here’s what was suggested:

Alias /music/ "/Media/Music/mp3/"
<Directory "/Media/Music/mp3/">
    AuthType Basic
    AuthName "music"
    Require group music

    Options Indexes MultiViews FollowSymlinks
    AllowOverride Indexes
    MusicIndex On +Stream +Download +Search -Rss -Tarball
    MusicFields title artist length bitrate
    MusicPageTitle Media Library
    MusicDefaultCss musicindex.css
    MusicIndexCache file://tmp/music
    MusicDirPerLine 4
    MusicIceServer [ice.gnu-designs.com]:8000
    MusicCookieLife 300
</Directory>

The problem was that restarting Apache resulted in errors with some of those options. I found a small clue buried in the README for musicindex:

“The MusicIndex Option replaces altogether MusicLister, MusicAllowDownload, MusicAllowStream, MusicAllowSearch, and MusicAllowRss.”

Removing those options and replacing them with their new equivalents solved that problem.

Alias /music/ "/Media/Music/mp3/"
<Directory "/Media/Music/mp3/">
    AuthType Basic
    AuthName "music"
    Require group music

    Options Indexes MultiViews FollowSymlinks
    AllowOverride Indexes
    MusicIndex On +Stream +Download +Search -Rss -Tarball
    MusicSortOrder album disc track artist title length bitrate freq filetype filename uri
    MusicFields title artist album length bitrate
    MusicPageTitle Media Library
    MusicDefaultCss musicindex.css
    MusicIndexCache file://tmp/music
    MusicDirPerLine 4
    MusicIceServer [ice.gnu-designs.com]:8000
    MusicCookieLife 300
</Directory>

And that worked. But it was deathly slow to render a single directory of only a handful of music files. I tried to eek out more performance, but it was just too slow to be useful.

Then I found a reference in another forum thread of a replacement for mod_musicindex called “edna“, so I decided to download that and give it a try.

edna is a standalone Python script which listens on a port and can present your music collection in a very similar way to mod_musicindex, but it is VERY fast, and has quite a few additional features that mod_musicindex does not provide.

But… it’s Python, and I have a genetic distaste for anything written in that language. I played with it for quite awhile and walked around my music collection with it. One of the limitations of edna that I found (besides being written in Python) was that it required that album art be in a single, separate file stored in the same directory as the mp3 files. Since I painstakingly took the time to store each and every album cover in the mp3 files themselves, this was a no-go for me.

So I went back to mod_musicindex while I kept looking for alternatives. One of the quirks with mod_musicindex that I found, was its rendering of proper unicode characters. I jumped into the #apache IRC channel on Freenode to ask for some guidance with respect to “tricking” the right charset to be used (for example, Björk was showing up as B?ork) and one of the lurkers in #apache asked if I’d ever heard of “Ampache” before. I hadn’t, so I trundled over and installed a copy.

The installation was really clunky and challenging, and I had to go into the code at one point and gut out a check which was throwing an error, because it made assumptions about my Apache setup that were just not valid.

I installed that, configured it, added a “catalog” (what ampache calls a collection of your music) to begin navigating through the interface.

In doing so, I realized that there were still quite a few mp3 flies with the wrong ID3v2 metadata or missing/incorrect album covers.

So that’s where I am now. I’m using a combination of Ampache + TagTuner to go through my entire MP3 collection and “normalizing” all of the data in each file. It’s long and drawn out work, but ultimately beneficial, since I only have to do it once.

And when I get back on the train to NY again and “Shaggy” is working, I can show him this system and see if it would be useful for his own DJ rig or parties.

THAT, is yak shaving in the true sense and spirit of the term.

Funny quote of the day:

>>>> Information wants to be free!

>>> Information wants to be a ballerina!

>> Then information needs to get her fat ass on a diet or she's 
>> never going to fit into that tutu and make Mommy proud!

> That kind of parenting made information a heroine addicted 
> stripper, now come over here and rub your data against me 
> for a dollar.

Source: Slashdot Managing Last.FM’s “Mountain of Data”

In my last post I wrote that I was comparing two third-party tools that are used to search and index Outlook content; Lookeen and Xobni.

DO NOT INSTALL OR USE THESE TOOLS!!

After I was done testing them and uninstalled them cleanly, they each corrupted my Outlook.pst file in two different ways… and the SCANPST.EXE tool (shipped with Office 12) does not fix the corruption. I even tried Advanced Outlook Recovery and it couldn’t fix the damage either.

I replayed the damage from a clean, previously scanned version of a .pst file before and after installing, running and uninstalling these tools, and the corruption is reproducable (and produces an unusable, unrecoverable file).

Now I have to blow away the damaged .pst file and start from clean backups, restoring from my data exports. The data in my main “Personal Store” (.pst) file goes back 10 years, and has over 11,000 entries in it. Recovery is not a simple process.

I repeat: DO NOT INSTALL OR USE THESE TOOLS!!

I’ve always been Linux and Open Source developer, supporter and user for as many years as I can remember. I’ve always sought to unify my personal space and environment starting with the tools available in the OSS community because they tend to fit my needs a LOT more than most of the proprietary vendor tools. The other benefit of using the OSS solution is
that if I run into a bug, I can either fix it myself or report it and have it fixed upstream very rapidly so everyone can take advantage of the fix.

But there really is no better calendaring solution that I’ve personally found in any flavor or OS than Microsoft Outlook 2007. Sure, it has its faults, as does any product of this level of complexity, but it works well and seems to suit my needs… and also allows me to quickly pull that information around to my other calendars (Work, Personal, Seryn, Holidays, Friends) and I can easily sync my Treo 680 to it to keep my various calendars fresh. I can also sync it to Google Calendar and export it to a CSV or XLS file for import into other incompatible systems.

However, the single biggest flaw that I’ve run into with Microsoft Outlook 2007 is the “Search” functionality. It is downright useless for anything other than taking up toolbar real-estate.

Granted, I’m spoiled by the power and simple flexibility of Google’s search. I’ve never, not even once, needed to go beyond Page 1 of any Google Search Results (SERPs). When I get results, I look at the quality, then tweak my search accordingly and re-submit. I don’t page through multiple pages until I see what I want.

Outlook doesn’t even correctly search all of the words you put into the search box. For example, if you received a message that included a subject of: “::trap id::820 (Server online) ...” and searched for “Server online” (without quotes), you get results that include the word “Server” and results that include the word “online”. If you use quotes (a Google thing), you get 0 results. If you search for “trap id”, you get 0 results. Well that’s just helpful.

I won’t ramble on about the multiple hundreds of other searches that I can clearly see results for in my Inbox, which Outlook quietly ignores… there are plenty. The Outlook search not only sucks, it’s flat-out broken. I don’t see Microsoft spending any time fixing this any time soon.

Enter third-party search tools to the rescue!

I found this thread on a quick Google search for better search tools for Outlook. The results of that thread boiled down to two leading tools:

1. Xobni (“Inbox” spelled backwards)
2. Lookeen

I decided to give them both a thorough test drive and see which one would suit my needs. I pointed both of these plugins to three email accounts I have configured in Outlook 2007 on my laptop, including one that is very-much littered with attachments of every kind and variety, thousands of calendar items, meetings, and two Gmail accounts… one of them with over 300,000 separate emails in it stretching back 10 years. Believe me, these plugins are going to get a serious workout!

One important thing to note is that my work Outlook instance inside the corporate firewall (also Outlook 2007) is so locked down that I can’t install any add-ons into it. In fact, the whole machine is so locked-down that I can’t even double-click the clock in the corner to see the mini-calendar or add a printer or change the desktop wallpaper or screensaver! At work, I’m stuck with the default, broken-by-design Outlook 2007 search. Since that one uses Exchange, it’s even slower than my local Outlook 2007 instance running inside VMware on my laptop.

Xobni

http://www.xobni.com/

First, the bugs and negative points:

• Slow, slow slow!
• Requires installing itself as a Windows service which runs all the time, not just when Outlook is running. The problem here is that I keep all of my Outlook data files password-protected (for obvious reasons), and when Outlook isn’t running, the service fires up to index the Outlook data and prompts me for a password. Very inefficient, and doing so loads Outlook, which I do not always want.
• No advanced search (this thing -“not this one” +that)
• Broken search when using punctuation (search for “DBD::Sybase”, for example)
• Seems to incessantly try to get to the net to look up photos, Facebook profiles, LinkIn profiles and so on with no way to disable it. Inside the corporate LAN, none of these sites are accessible, and it drags the search to a crawl, as it attempts to contact those sites and times out.
• Related to the last item, disabling those “Third-party Extensions” (as Xobni calls them), does not remove the buttons for those extensions from the UI, so they are still accessible to be clicked, they still attempt to query the net, and they are still visible on the GUI.
• Nasty bug when backspacing to the beginning of the search box and then typing again. 100% reproducible and annoying. I basically have to type at 1/3 the speed to stop it from triggering the bug. Unacceptable.
• Minimizing the Xobni sidebar uses a minimize widget that sits on the bottom of the bar, completely out of place from the » chevrons for minimizing other widgets which are at the top of each of their respective sidebars.
• Minor nit: The Xobni skin/theme doesn’t match the Outlook skin/theme, so it stands out like a sore thumb and looks like a clunky “bolt-on” rather than an enhancement. If you have the folder tree, Inbox, message preview, Xobni and Task bar all displayed in 4 vertical columns.. the Xobni one (3rd column) really sticks out. In fact, when you minimize the sidebar, it goes from a black/dark skin when exposed, to the proper, Outlook-compatible color theme/skin when minimized. It’s not consistent. Even the size of the chevrons differs in the minimized and maximized versions of that sidebar.

The good points:

• Free as in beer
• Lots of great features that no other search or plugin I’ve found has (“network”, statistics, attachments widget)
• Relatively fast searching and display of results, including a hover preview mode

Lookeen

http://www.lookeen.net/

The bugs and negative points:

• Everything is in a popup window, nothing docks with Outlook itself (like Xobni does, for example). Popups are sometimes ok because they can be minimized and moved out of the way, but in this case, having the search results side-by-side with the actual content would be better.
• Can’t remove searches to re-edit them (changing “Search This” to “Search this” (letter case changes) are ignored when you type them, but are Case Sensitive in the actual search results). Ideally I should be able to expand the search history and hit Delete on each one like I can with the search history in a web browser. I can clear the whole search history, but that’s an all or nothing operation. I can’t delete bogus searches, leaving the others there (which might be frequently-used searches)
• $39.80/USD price tag. While I have no problem paying this for a functional plugin, there are quite a few niggling things that don’t justify that cost at this point.

The good points:

• A LOT faster than Xobni. I mean a LOT. Searching is faster, indexing is faster, everything is faster.
• Between the two, Lookeen handles all sorts of attachments much better and visually represents them in a way that is very quick and easy to visually filter in or out what you’re looking for, IMO. It also has a “Conversations” button in several places that makes zooming into a relevant conversation from a highlighted message or user very quick. Additionally, you can go from 1 day, 1 week, month of conversations to further narrow your search results and scope. Very well thought-out.
• Lookeen docks in the toolbar, which has a few advantages, while Xobni docks in the sidebar (and based on its featureset, leverages slightly different advantages there)

The Conclusion

Ultimately, I’ll probably continue to use Lookeen, because it does what it advertises to do much better… searching and finding. Yes, their implementation of searching is a bit “clunky” with the popup windows instead of docking against Outlook, but I can deal with that for now. The graphical slowness and constant “Internet poking” that Xobni does completely negates any benefits it provides with the statistics and grouping of messages and search results.

I’m still looking for other options that Lookeen and Xobni provide. If anyone has any ideas or suggestions, I’m all ears.

I’ve recently installed Tracks on my server to help me manage my myriad of tasks, projects and timelines utilizing the GTD methodology created by David Allen.

After installing it on my server I almost immediately noticed that Ruby and Rails was going to have a problem coexisting with my very tweaked Apache instances (running roughly 300 separate websites).

I found a project called Passenger, and immediately got that working with Tracks running on top of it. No fuss, no muss.

But one problem with Tracks that I’ve found (well, one of several, none of them showstoppers), is that you can’t interact with Tracks any other way other than the web interface. The interface is nice and clean and slick, but I don’t always have access to the web where I am.. and that limits the functionality of Tracks for me. I do, however… have access to some sort of email account everywhere I go.

So I whipped up a quick little script to allow me to send email to Tracks and have it post Tasks in the right contexts and on the right due dates for me.

Here’s the code for that:

#!/usr/bin/perl
#        _._   
#       /_ _`.      (c) 2008, David A. Desrosiers
#       (.(.)|      setuid at gmail.com
#       |\_/'|   
#       )____`\     Email to Tracks interface
#      //_V _\ \ 
#     ((  |  `(_)   If you find this useful, please drop me 
#    / \> '   / \   an email, or send me bug reports if you find
#    \  \.__./  /   problems with it. I accept PayPal too! =-)
#     `-'    `-' 
#
##############################################################################
# 
# License
#
##############################################################################
#
# This script is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
# 
# This script is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
# 
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc., 59
# Temple Place
#
# - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
# 
# Email format can be any of the following:
#
#       Subject: The main description of the Task
#
#       @Context
#       01/02/2003
#       This is my note text
#
# or:
# 
#       Subject: Default Task description
#
#       c: @context
#       d: 2008/12/31
#       n: The note text to insert
#
# Most valid date formats are accepted, and this will do its best to
# "correct" and normalize them. You can also prefix your lines with the
# modifiers above, or read the regexes below for more. For example: 
# c:, context:, cxt:, con:, ct: are all valid prefixes for "context"
# n: and note: are all valid prefixes for "notes", and so on.
#
##############################################################################
use strict;
use XML::Simple;
use LWP::UserAgent;
use URI::Escape;
use Email::Abstract;
use Date::Manip;
use Date::Parse;
use Data::Dumper; 

my $url                 = "http://your.tracks.site/todos.xml";
my $contextUrl          = "http://your.tracks.site/contexts.xml";

# The default contextid where you want the todo added
# SELECT id,name FROM contexts;
my $contextid           = "8";

my $user                = "yourname";
my $password            = "yourpass";

# Leave these tokens alone.  They are valid as of Tracks 1.5 RESTful API.
my %todo = map { +($_ => "todo[$_]") } qw(notes context_id description due);

# Get the context legend in order to match by name
my $ua                  = new LWP::UserAgent;
my $req                 = new HTTP::Request 'GET',$contextUrl;
$req->authorization_basic($user,$password);
my $res                 = $ua->request($req);
my $contexts            = XMLin($res->content);

# Split apart the email into Subject and Body
my $message             = do { local $/; <STDIN> };
my $email               = Email::Abstract->new($message);
my $subject             = $email->get_header("Subject");
my $body                = $email->get_body;

# These can probably be cleaned up a bit
my ($context_line)      = $body =~ /^(?:c:|ct:|cxt:|con:|context:|@)\s*(.+)$/mi;
my ($date_line)         = $body =~ /^(?:d:|date:)\s*(\d.*)$/m;
$date_line              = UnixDate(ParseDate("today"), "%g") if (length($date_line) == 0);
my $time                = str2time($date_line);
my $due_date            = UnixDate(scalar gmtime($time), "%m/%d/%Y");
my ($note_line)         = $body =~ /^(?:n:|note:)\s*(.*?)$/mi; 

# Concatenate the data here before we send POST to the Tracks server
my $post_data = 
        $todo{'context_id'} . "=" . $contextid . "&" . 
        $todo{'description'}. "=" . uri_escape($subject) . "&" . 
        $todo{'notes'} . "=" . uri_escape($note_line) . "&" . 
        $todo{'due'} . "=" . uri_escape($due_date);

# Use LWP to do the posting ($ua was created earlier)
$req = new HTTP::Request 'POST',$url;
$req->content_type('application/x-www-form-urlencoded');
$req->content($post_data);
$req->authorization_basic($user,$password);
$res = $ua->request($req);

You can send emails in the format in the comments above to your Tracks install and it will create Tasks for you. I could refactor those regexes a bit, and that’s a task on my list, but so far, this works…

Emails look like this:

Subject: Update regexes in Tracks perl email interface
From: "David A. Desrosiers" <desrod at gnu-designs.com>
To: 2e59561d76 at gnu-designs.com
Content-Type: text/plain
Message-Id: <1228182961.14783.123.camel at gnu-designs.com>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.3.1 
Content-Transfer-Encoding: 7bit
Date: Mon, 01 Dec 2008 20:56:02 -0500
X-Evolution-Format: text/plain

c: @Internet
d: 12/15/2008
n: Clean up the regexes that processes email-based Tracks tasks

Set it up in your MTA as an alias, as follows:

2e59561d76:                         "|/path/to/tracks-mail.pl"

Make sure you re-run newaliases(1) after adding this entry:

$ sudo /usr/sbin/newaliases
/etc/mail/aliases: 248 aliases, longest 82 bytes, 15141 bytes total

I tried to make it smart enough to figure out most common (valid please) date formats and if you omit that, it just sets it to today’s date so you can change it later.

The next step is to figure out how to make this script multi-user aware, without forcing users to expose their username or password in email directly. I have some ideas for that and will test that in v1.1 of this script and release it later on.

The Tracks Forums are pretty busy and lots of people are beating up the code, testing it in many different ways. I’m just trying to contribute back in whatever way I can.. in the hopes that the project continues to thrive and grow.

If you have any suggestions or ideas, let me know!