random neuron misfires

I’ve been using TweetDeck for quite some time on Linux, after I managed to get getting Twhirl working on Linux under 64-bit Adobe Air.

TweetDeck is a lovely app, graceful and very useful. It has its minor visual and UI bugs, but it’s the best I’ve seen out of the other hundreds of Twitter apps out there… and it’s 100% free.

I have 3 laptops I use on a regular basis running both Windows and Linux. I’m not always on the Linux laptop, but I wanted to make sure that my TweetDeck settings on my Windows laptops were identical to the ones on my Linux laptop, including all of my searches, columns and other settings. I plug one of my Windows laptops into my television, so I can use the larger screen as my monitor (see below).

There’s been a lot of talk lately about “the cloud”. We’ve done the cloud before. First we called it “clusters”. Then we called it “grid”. Now we call it “the cloud”. I’m not sure what term marketing will need to call it in a year or two, but one thing is for sure: “cloud sync” is doomed to fail, before it even gets started.

“Cloud sync” is the term used to describe sending your data from various end-user devices (“clients”, usually handheld devices or desktop PIM apps) to the cloud; usually a central server somewhere. Some popular examples of this are:

• ScheduleWorld
• Funambol
• cloudsync
• Apple’s “MobileMe“
• Google Sync
• Nokia’s “Ovi“
• Palm’s “Synergy“
• Vodafone “Zyb“
• Cozi Calendar

…and literally dozens of others.

Of these, Funambol, ScheduleWorld and Google Sync are among the most mature, and also among the most problematic (they are based on SyncML).

And every single one of them is failing, because they’re all based on some very broken logic and monolithic designs:

1. Not a single one of them supports synchronizing multiple calendars or other data sources while keeping them separate in the cloud’s datastore itself. Without the ability to sync multiple data sources under the same user, the cloud becomes completely useless.
2. Not a single one of them understands how to get sync right, without corrupting, duplicating or deleting data. Every single one of them has an issue here.
3. None of them are prepared for handling a device they’ve never seen before, or a device which has records that include a field or format they don’t already know about. Everything has to be pre-determined, pre-defined, and that will never scale.

I’ll give you one example, one I’m fighting with every single day. I’ve run this same (very structured) scenario through about two-dozen vendor, free and commercial projects and products, and every single one of them breaks down and fails in one way or another.

1. Back up your device, making sure you have a complete copy of the data, in case anything goes wrong (and it will).
2. Sync your client device to its native software. In the case of Palm, you’d sync that to Synergy. In the case of Blackberry, you’d sync that to BlackBerry Desktop Manager, and so on.
3. Sync that same device to your favorite PIM package. If you’re using an iPhone, sync that to iCal or Microsoft Outlook. If you’re using Linux, sync your device to Kontact or Evolution, if you’re using a BlackBerry you’d sync that to Lotus Notes or Microsoft Outlook.
4. Sync this same device to “the cloud” using whatever software is provided by the cloud project you’re synchronizing with. If you’re using ScheduleWorld, use their sync software. If you’re using another, use whatever software they provide.
5. Now install the sync tool for your target “cloud” service in your PIM application. If you’re using Microsoft Outlook for example and Funambol, you’d install the “Funambol Outlook Plugin”. Likewise for whatever other PIM and service you choose.

At this point, your client device (your iPhone, BlackBerry, Palm or other handheld) and your PIM device (Microsoft Outlook, Lotus Notes, Evolution, Kontact, etc.) should all contain exactly the same data, and that same data should now be available “in the cloud”.

Now sync your handheld device to the cloud again, and watch what happens. In every single case, without fail, you will get a.) corruption, b.) lost data, or c.) duplicated data.

There have been numerous discussions on dozens of mailing lists about how to get sync to work correctly, and I can say from over a decade of in-depth personal experience with synchronization, that it is downright impossible to have this work correctly, without very aggressive, deep inspection of every single field of every single record, at each sync, once you add more than a 1:1 relationship. The “cloud” concept already implies a many:1 or many:many relationship.

Let me replay it to show what I mean:

1. Sync your handheld device to your PIM (usually using a USB connection). Right now, your handheld and your PIM should contain exactly the same data.
2. Sync handheld device to the cloud. This creates new records in the cloud which should match your PIM exactly. Now PIM, device and cloud should contain the same data.
3. Sync your PIM to the cloud. Depending on the software used, you’ll trash data here. In the case of something like Funambol, you’ll take multiple data sources and merge them into one source in the PIM, creating a mess and lots of duplicates. Other software packages do similar things to the data.
4. Sync your handheld device to the cloud again, which will now duplicate and transport that mangled data to the handheld.

There is absolutely no way for the cloud to know that your handheld doesn’t have identical data that the cloud or PIM has, so it must inspect every record at sync time. None of them do this.

You’ll probably hear the terms “slow sync” and “fast sync” in reference to these issues, and I can say with certainty that nobody is doing it 100% correctly.

“Fast sync” is a term used to describe synchronizing one device with one server. Pointers to the last changes on either end are captured, so when you sync again, it only has to send the updates and changes across. Palm handheld devices use a “LastSyncID” on the device to identify the client to the server-side. If the server sees a different incoming client connection, it knows that it isn’t the device it last talked to, so it initiates a “slow sync”.

“Slow sync” is a term used to describe a sync that happens when the server and the client device can’t determine if they’ve “spoken” before, so they send the entire contents of each datastore (calendar, contacts, tasks, memos, etc.) and then compare on a record-by-record basis, to see if there are any changes.

Once you sync more than one device to a server, there is no way you can maintain a “fast sync” relationship with each client.

A == handheld (iPhone, BlackBerry or other)
B == PIM application (Microsoft Outlook, Lotus Notes, Sunbird)
C == "cloud" server

A -> C
B -> C

There is no way for C to know that ‘B’ contains the same data as ‘A’, so it must inspect every record as it arrives, to consolidate the changes, if any are found.

The closest I’ve come to a “perfect solution” is using Intellisync’s backend behind BlackBerry Desktop Manager, but it requires a local (USB or Bluetooth) connection, and requires Microsoft Windows. This is an unacceptable solution to me.

When you sync your BlackBerry to BDM, it does a minimum of 4 separate passes across the data, to compare, inspect and merge any and all changes that may be necessary. If anything has changed or is in conflict, a dialog will be displayed that give you the ability to re-sync, cancel, accept, etc. the changes, as well as inspect them on a change-by-change, field-by-field basis. It does this every single time. There is no “fast sync” capability with BlackBerry Desktop Manager. It does not trust the incoming data by default, nor should it.

But using BDM, there is no way to accept some changes, reject others, merge yet others. It’s an all or nothing solution, and when you have 4,700+ Calendar events as I do for example and a sync process that takes over 20 minutes for each sync pass, it can be quite painful.

Until we develop a way to accept anything the client sends, store it on the server in a unique, per-device datastore, and then aggregate that data in an intelligent way back down to the other requesting clients, “cloud sync” is doomed to fail, before it even gets off the ground.

The current thinking is that we just push the data up to the server(s) (“the cloud”), and the pull it back down to other client or end-devices. That logic is broken, because it isn’t that simple.

If I have a device which supports photos in a Contact record, and I sync to my PIM which does not support photos in the Contact records, and then sync my device to the cloud, and my PIM to the cloud… what happens to the photos? I’ll tell you… they’re lost. Once I sync that PIM to the cloud, the photos are gone, and then when I sync my handheld to the cloud, those (photo-less) records in the cloud are sent to the handheld, trashing all of my local contact photos.

This is just one example, but it proves the point that we can’t continue storing data “in the cloud” in a single, large datastore. Each device is unique, and each device’s way of representing the data is unique. Without keeping that data separate, you’re going to trash, duplicate or lose data.

There are ways to fix this and plenty of bright people working on these problems, but most of the projects I’ve encountered are all so stuck in the dark ages, they refuse to think about ways to solve this that are 100% future-proof, and allow any device with any data representation to be able to cohabitate within the cloud with these other devices and services.

I’m still on a quest, a long and arduous quest to find the right series of tools and techniques that will let me get my Work (gnu-designs, inc. and $EMPL) and Personal calendars into a format that I can use, without losing appointments, data or context and manage them on my handheld device (currently a Blackberry Bold 9000).

I tried using Google Calendar Sync, and found the following outstanding issues with it:

1. No support for multiple calendar files. This is a HUGE oversight, and unfortunately, most vendor products lack this capability, but I don’t think I know anyone who doesn’t have a separate Work and Personal calendar of some sort.
2. Duplicating events. That’s just bad programming logic.
3. Ignoring/missing events. See above, just poor logic flow in the sync design.

After getting frustrated with that, and trying various methods to try to work around these deficiencies, I was led over to “SyncMyCal“, which is a pretty slick solution overall. Except there’s no way to keep the data separate once it gets to Google Calendar. If you want to sync the data back from Google Calendar to Microsoft Outlook, it just gets glommed into the main (default, as defined by your configuration) Calendar file on your desktop.

Yecch.

Ironically, the company with the absolute worst track record for managing PIM data (RIM, aka makers of Blackberry handheld devices), has a solution based upon Intellisync (now owned by Nokia).

Their sync solution supports multiple source calendars (Work + Personal), synchronizes them to a single handheld device, and keeps them separate, without intermingling them on the way back to the desktop when you sync again. This is the closest thing to calendaring nirvana that I’ve ever used. This isn’t without its own issues, however:

The major problems I’ve found so far are:

1. Major, major data corruption about 75% of the time, when a sync hangs or fails outright. In order to get around this, you have to delete the Intellisync directory (C:\Documents and Settings\$USER\Application Data\Research In Motion\BlackBerry\Intellisync), and re-create your sync rules from scratch.
2. The catch (and the second major issue)? You need to be connected to the live Internet in order to configure the sync of your Blackberry to Outlook, using Intellisync through Blackberry Desktop Manager. Why? WHY do I need to be connected to the Internet to create a local sync rule from Outlook <-> Blackberry and back? Nothing at all in that process touches the LAN network or the Internet. Something fishy about that.

I found an interesting thread that suggested using Mozilla Sunbird as a “trampoline”, and going from Outlook -> Sunbird -> Google Calendar -> Google Sync for Blackberry.

In that thread, user “ssitter” mentions FreeMiCal, which is a project aimed at creating proper, standards-compliant iCalendar files from Outlook as a data source, and using those .ics files as the import source into Sunbird. From there, you can then use the Sunbird Provider for Google Calendar mentioned in this other thread to get the calendar data into Google Calendar, in a two-way fashion.

Or so I thought… Here’s a quick snip of what Outlook sees in my calendars (4.074 events total):

And here’s what FreeMiCal sees when I launch it (402 events out of 4,074 total):

In other words, it too doesn’t see my other calendar (the one with 3,672 events in it). Sigh.

So I’m back to square-one again, trying to find a solution that allows me to keep my calendars separate in Outlook (or Sunbird or Evolution or whatever desktop-side PIM client I choose), and aggregate them on my handheld device, but then keep them separate when they get synchronized back.

Here’s what I’ve used and excluded thus far, because nothing fits what I’m looking for:

1. Blackberry Desktop Manager
2. SyncMyCal
3. CompanionLink for Google
4. FreeMiCal
5. Google Sync for Outlook
6. Google Sync for Blackberry
7. Funambol

Still nothing, still searching, and any suggestions or ideas anyone has… I’m all ears and eyes.

Let me just start by saying that I have a lot of data. In multiple places. Some on laptops, some on servers, some on removable drives and mirrored hard disks sitting in a bank vault (yes, really). Lots of data on lots of systems in different states and locations: client data, personal data, work data, community data and lots more.

Over the years, I’ve tried my best to unify where that data is sourced from, for example by relocating the standard “My Documents” location on all of my Windows machines (physical and virtual), to point to a Samba share that is served up by a GELI-encrypted volume on my FreeBSD or Linux servers. That part works well, so far, but that’s only a small piece of the larger puzzle.

Over the last decade, the amount of data I’m holding and responsible for managing has grown significantly, and I needed a better way to manage it all.

There are plenty of backup solutions for Linux including the popular Amanda and Bacula, but I needed something more portable, leaner and much more efficient. That quest led me to look to find Unison mostly due to it’s cross-platform support, but it was still a bit more complicated than I needed.

So I kept looking and eventually found rsnapshot, a Perl-based tool wrapped around the standard rsync utility written by Andrew Tridgell.

Since I’d already been using rsync quite a bit over the last 10 years or so to copy data around as I needed it and to perform nightly full backups of my remote servers, I decided to look into using rsync to manage a new backup solution based around incremental backups as well as full backups.

I’m already using rsync to pull a couple of terabytes of mirrored data to my servers on a nightly basis. I’m mirroring CPAN, FreeBSD, Project Gutenberg, Cygwin, Wikipedia and several other key projects, so this was a natural graft onto my existing environment.

Whew!

This week has been absolutely amazing for me, lots of postive successes on all fronts; work and personal. So much energy and glow all around, it’s so very motivating and invigorating.

With all of this activity going on, I’ve been focusing a lot on managing my calendars a bit more efficiently than I probably was before. I have a LOT of calendaring to manage. Here’s a small example of my calendar for this
week (this is a screenshot of my Microsoft Outlook instance below):

To that end, I’ve (re)discovered the Chandler Project after putting it down a few years ago and also something new called “Funambol” (formerly the sync4j project, based on SyncML).

Chandler unfortunately doesn’t build or run on 64-bit Linux systems, so I can’t do any real testing or integration with that at this point. This is 2009, right? 64-bit systems have been around for several years now; at least
long enough to be able to execute a port of Chandler to make it work there. But now it seems that Chandler may be dead now, with Mitch Kapor leaving the project
and taking his funding with him.

Lovely.

Conversely, Funambol installed very easily, runs smoothly, and their administrative GUI is very comprehensive (though, being written in Java, the controls themselves are quite clunky and non-intuitive). There’s a lot of
complicated bits in Funambol, so their documentation portal is a necessary stop if you’re going to install and configure it on your server.

Now the bad news: Neither of them solves the actual problem: universal, location-free calendaring.

What do I mean by that? What I mean is… treating separate calendars as separate in their own realm, but aggregating the relevant pieces where appropriate in the overlap realms.

This means that my personal Microsoft Outlook session running inside VMware, configured with at least 5 separate calendars (Personal, Work, Events, Holidays, Merged) should be treated as separate, and not glommed
together into one sloppy calendar when events are pulled from the individual source calendars. Almost every single sync tool and plugin that claims to support multiple calendars does exactly this. I’ve tried dozens
of them. This is wrong!

Funambol looked promising… except they got it wrong too. For example, the Funambol Outlook plugin only supports one calendar at a time (and sub-folders). You can’t point it to multiple calendars and have it treat them as
separate or even as one merged calendar. Useless.

The Funambol Blackberry app only supports 1 month of prior events and and 6 months of future events when doing an OTA (“Over the Air”) sync to the Funambol server. Also useless.

The Blackberry software, through Blackberry Desktop Manager (based on Intellisync) handles this somewhat cleanly, by allowing me to define as many folders (Calendar folders, Contact folders, etc.) as I want, and then
designating one of those folders as the “default” folder in the sync pairs. Any item I create on the Blackberry goes into that “default” folder, which makes its way to Outlook in the manner expected. If I need it to exist in
another folder, I’ll move it to the other folder when it gets to the Outlook side.

Unfortunately, Blackberry devices are still years behind Palm OS devices with regard to PIM functionality. They’re barely out of diapers. Blackberry devices don’t support Categories for calendar items, for example.
Contacts can have categories, but Calendar items can’t? Huh? HUGE oversight. This is almost as braindead as Apple omitting the very basic Cut and Paste functionality from the iPhone.

Is this really that complicated to solve? Really? In 2009?

So I’ve engineered a process that allows me to keep a clean copy of events on my Blackberry (sans categories, grr!), which is then synchronized over local USB with Outlook running inside a VM (so I can get my work calendar
on my handheld and push my on-handheld events back to Outlook). Outlook is connected to several of my Gmail accounts and Cozi for some additional calendars.

From there, I use a product inside Windows called SyncMyCal to push my calendars upstream to Google Calendar, where they are then marked as either private or shared, depending
on who I need to have access to them.

On the Linux side, Evolution then subscribes to those calendars in Google Calendar, so I can see them from Evolution.

Confused yet? I am too.

My goal is to try to get one central “repository” where all of my PIM data (Calendar, Contacts, Tasks, Memos) can reside. That repository would then have connectors to the other applications and services to manage that PIM
data. Each application would either push, pull or sync their respective bits of data into the repository, where it would then be immediately available/replicated to the other components in the system (based on permissions
and rules, of course).

I thought Chandler and/or Funambol would be this repository. I was wrong, and we’re back to the beginning again.

But at least I have a process where I’m not losing data, I’m keeping everything centralized in one place on my handheld (heavily locked down and very strongly secured; thank you RIM), and I can push it to wherever I
need it from there.

Not only am I trying to augment the lack of core function in the Blackberry to bring it up where my Palm OS Treo 680 was, but I’m trying to merge, maintain and manage 1/2 dozen calendars in at least 3 different systems
(Linux, Windows and Web).

The search continues…

I’ve always been Linux and Open Source developer, supporter and user for as many years as I can remember. I’ve always sought to unify my personal space and environment starting with the tools available in the OSS community because they tend to fit my needs a LOT more than most of the proprietary vendor tools. The other benefit of using the OSS solution is
that if I run into a bug, I can either fix it myself or report it and have it fixed upstream very rapidly so everyone can take advantage of the fix.

But there really is no better calendaring solution that I’ve personally found in any flavor or OS than Microsoft Outlook 2007. Sure, it has its faults, as does any product of this level of complexity, but it works well and seems to suit my needs… and also allows me to quickly pull that information around to my other calendars (Work, Personal, Seryn, Holidays, Friends) and I can easily sync my Treo 680 to it to keep my various calendars fresh. I can also sync it to Google Calendar and export it to a CSV or XLS file for import into other incompatible systems.

However, the single biggest flaw that I’ve run into with Microsoft Outlook 2007 is the “Search” functionality. It is downright useless for anything other than taking up toolbar real-estate.

Granted, I’m spoiled by the power and simple flexibility of Google’s search. I’ve never, not even once, needed to go beyond Page 1 of any Google Search Results (SERPs). When I get results, I look at the quality, then tweak my search accordingly and re-submit. I don’t page through multiple pages until I see what I want.

Outlook doesn’t even correctly search all of the words you put into the search box. For example, if you received a message that included a subject of: “::trap id::820 (Server online) ...” and searched for “Server online” (without quotes), you get results that include the word “Server” and results that include the word “online”. If you use quotes (a Google thing), you get 0 results. If you search for “trap id”, you get 0 results. Well that’s just helpful.

I won’t ramble on about the multiple hundreds of other searches that I can clearly see results for in my Inbox, which Outlook quietly ignores… there are plenty. The Outlook search not only sucks, it’s flat-out broken. I don’t see Microsoft spending any time fixing this any time soon.

Enter third-party search tools to the rescue!

I found this thread on a quick Google search for better search tools for Outlook. The results of that thread boiled down to two leading tools:

1. Xobni (“Inbox” spelled backwards)
2. Lookeen

I decided to give them both a thorough test drive and see which one would suit my needs. I pointed both of these plugins to three email accounts I have configured in Outlook 2007 on my laptop, including one that is very-much littered with attachments of every kind and variety, thousands of calendar items, meetings, and two Gmail accounts… one of them with over 300,000 separate emails in it stretching back 10 years. Believe me, these plugins are going to get a serious workout!

One important thing to note is that my work Outlook instance inside the corporate firewall (also Outlook 2007) is so locked down that I can’t install any add-ons into it. In fact, the whole machine is so locked-down that I can’t even double-click the clock in the corner to see the mini-calendar or add a printer or change the desktop wallpaper or screensaver! At work, I’m stuck with the default, broken-by-design Outlook 2007 search. Since that one uses Exchange, it’s even slower than my local Outlook 2007 instance running inside VMware on my laptop.

Xobni

http://www.xobni.com/

First, the bugs and negative points:

• Slow, slow slow!
• Requires installing itself as a Windows service which runs all the time, not just when Outlook is running. The problem here is that I keep all of my Outlook data files password-protected (for obvious reasons), and when Outlook isn’t running, the service fires up to index the Outlook data and prompts me for a password. Very inefficient, and doing so loads Outlook, which I do not always want.
• No advanced search (this thing -“not this one” +that)
• Broken search when using punctuation (search for “DBD::Sybase”, for example)
• Seems to incessantly try to get to the net to look up photos, Facebook profiles, LinkIn profiles and so on with no way to disable it. Inside the corporate LAN, none of these sites are accessible, and it drags the search to a crawl, as it attempts to contact those sites and times out.
• Related to the last item, disabling those “Third-party Extensions” (as Xobni calls them), does not remove the buttons for those extensions from the UI, so they are still accessible to be clicked, they still attempt to query the net, and they are still visible on the GUI.
• Nasty bug when backspacing to the beginning of the search box and then typing again. 100% reproducible and annoying. I basically have to type at 1/3 the speed to stop it from triggering the bug. Unacceptable.
• Minimizing the Xobni sidebar uses a minimize widget that sits on the bottom of the bar, completely out of place from the » chevrons for minimizing other widgets which are at the top of each of their respective sidebars.
• Minor nit: The Xobni skin/theme doesn’t match the Outlook skin/theme, so it stands out like a sore thumb and looks like a clunky “bolt-on” rather than an enhancement. If you have the folder tree, Inbox, message preview, Xobni and Task bar all displayed in 4 vertical columns.. the Xobni one (3rd column) really sticks out. In fact, when you minimize the sidebar, it goes from a black/dark skin when exposed, to the proper, Outlook-compatible color theme/skin when minimized. It’s not consistent. Even the size of the chevrons differs in the minimized and maximized versions of that sidebar.

The good points:

• Free as in beer
• Lots of great features that no other search or plugin I’ve found has (“network”, statistics, attachments widget)
• Relatively fast searching and display of results, including a hover preview mode

Lookeen

http://www.lookeen.net/

The bugs and negative points:

• Everything is in a popup window, nothing docks with Outlook itself (like Xobni does, for example). Popups are sometimes ok because they can be minimized and moved out of the way, but in this case, having the search results side-by-side with the actual content would be better.
• Can’t remove searches to re-edit them (changing “Search This” to “Search this” (letter case changes) are ignored when you type them, but are Case Sensitive in the actual search results). Ideally I should be able to expand the search history and hit Delete on each one like I can with the search history in a web browser. I can clear the whole search history, but that’s an all or nothing operation. I can’t delete bogus searches, leaving the others there (which might be frequently-used searches)
• $39.80/USD price tag. While I have no problem paying this for a functional plugin, there are quite a few niggling things that don’t justify that cost at this point.

The good points:

• A LOT faster than Xobni. I mean a LOT. Searching is faster, indexing is faster, everything is faster.
• Between the two, Lookeen handles all sorts of attachments much better and visually represents them in a way that is very quick and easy to visually filter in or out what you’re looking for, IMO. It also has a “Conversations” button in several places that makes zooming into a relevant conversation from a highlighted message or user very quick. Additionally, you can go from 1 day, 1 week, month of conversations to further narrow your search results and scope. Very well thought-out.
• Lookeen docks in the toolbar, which has a few advantages, while Xobni docks in the sidebar (and based on its featureset, leverages slightly different advantages there)

The Conclusion

Ultimately, I’ll probably continue to use Lookeen, because it does what it advertises to do much better… searching and finding. Yes, their implementation of searching is a bit “clunky” with the popup windows instead of docking against Outlook, but I can deal with that for now. The graphical slowness and constant “Internet poking” that Xobni does completely negates any benefits it provides with the statistics and grouping of messages and search results.

I’m still looking for other options that Lookeen and Xobni provide. If anyone has any ideas or suggestions, I’m all ears.

First, I know you’re going to ask why you’d want to go from a high-quality VBR format like Ogg Vorbis to the paltry, low-quality mp3 format… and to that I have one word: Apple.

Apple saw fit in their infinite wisdom to NOT support the freely-available and license/patent free Ogg Vorbis audio format in their iPod and iPhone devices. Instead, they support the proprietary, licensed, restrictive mp3 formats instead.

The near-sightedness of commercial companies never ceases to amaze me.

To solve that, you can convert your oggs to mp3 (obviously, keeping the original .ogg files for your personal library), using the following one-liner (separated into multiple lines for ease of explanation):

for i in *.ogg; do 
[[ ! -a "${i%ogg}mp3" ]] && 
oggdec "$i" -o - | lame --preset standard - "${i%ogg}mp3" ; 
done

Now you can drag those mp3 files onto iTunes and sync them to your iPod or iPhone device. I much prefer SongBird on Windows and amaroK on Linux over iTunes, but… some may not have that option.

If you’re like me, you use VMware Workstation heavily. I use it for testing, development, cloning and all manner of other things. One thing that has nagged me since upgrading to the 6.x series of Workstation is that native USB Palm synchronization stopped working.

I can sync over Bluetooth to Windows XP running inside the VM, but I can’t sync natively using the USB cable itself. Windows sees the device as a “Palm Handheld”, VMware connects it, all looks good… but it never actually wakes up the Palm HotSync applet in the System Tray.

Apparently VMware added “port reset forwarding” to the host for Workstation 6.5 but Palm devices (most, if not all) don’t work with this change. To offset that, they’ve added a way to control this, including a new way to control the “skipsetconfig” parameters on a per-device basis.

First, you’ll need to find the vendor id (vid) and product id (pid) of the Palm device you’re trying to connect to. You can usually snarf this out of the vmware.log for that VM session. Mine reported something like this:

Nov 23 13:56:52.371: vmx| USB: Found device [name:Palm\ Handheld vid:0830 pid:0061 path:5/1 speed:full family:vendor]

This means the Treo 680 was: vid:0830 pid:0061

You’ll also note that these are identical to the vendor_id and product_id that we use in pilot-link and other projects on the Linux side.

Then, you add a usb.quirks.deviceX line with the vid:pid pair, followed by the quirk. The two known “quirks” that help USB Palm devices are skip-setconfig and skip-reset. Here is the quirk entry to add to your VM’s vmx config file for the above example:

usb.quirks.device0 = "0x0830:0x0061 skip-setconfig, skip-reset"

Note that you need to use a 0x to denote that the number is hex, which is different from the vid:(hex number) used in log line and autoconnect lines. If you don’t put a 0x, it assumes a decimal number.

From the original thread in the VMware forums:

“This also acts differently than the usb.generic.skipsetconfig setting in that the skip-setconfig is only activated for a specifc device using a quirk versus having it turned on for all devices with the global usb.generic.skipsetconfig. Most devices can tolerate receiving multiple setconfigs and most devices don’t use anything but the first config. There are some devices that may need to set a second or third config, so using a skip-setconfig quirk for only those devices that need it and not using the global usb.generic.skipsetconfig is preferred.”

After setting this, uninstalling Palm Desktop, shutting down VMware entirely, rebooting and reinstalling Palm Desktop + HotSync, everything started working again.

I am a customer of a tool called CompanionLink for Google which allows me to sync my Outlook/Palm calendar to Google Calendar, so my friends/family can see where I am at any given time.

The cost for their software was relatively cheap, and it appeared to work well. There are some inherent issues and bugs in the tool, which still aren’t resolved, but it does what it claims to do in a very simple and fairly painless way.

Recently I noticed that Google has released their own free tool called “Google Calendar Sync” to do essentially the same thing.

The Google tool doesn’t support as many platforms and PIMs as CompanionLink, but it is free, and it is written/maintained by the people who interface directy with Gmail, so I figured it should work “better”.

I was wrong.

For some reason, both of these tools seem to “ignore” random events on my calendar, while including other random events around the same days and weeks. There is no obvious pattern to why these events are skipped, or why I can’t get them into my Google Calendar.

I’ve exported all of my calendar items to a local file, removed the categories, deleted my local calendar entirely, re-imported all of the events clean and told both tools to do a “Purge and Reload”, pushing everything from my local calendar into Google Calendar.

Nada.

Both tools fail in almost exactly the same way. This leads me to believe something in the way Gmail is accepting the calendar data is causing it to “ignore” some of my events (a few hundred).

So now I’m back to not using either of them, because they can’t do what I need. I’ll probably talk to CompanionLink and see if they have a fix, or can issue me a refund for my purchase price.

Frustrating, since I don’t yet have one unified place for all of my calendaring to go. Grr!

UPDATE: I have a fix. The fix is to export my entire calendar into CSV format, then log into Google Calendar, delete all of the calendar items, then import the CSV fresh every time.

So much for 2-way syncronization. Sigh.

This weekend was just as busy as the week at work. It’s Sunday afternoon, and I’m still going…

Reconstructing Maildir from Backups

Moments ago, I found that my archive of the Coldsync mailing list in Maildir format somehow became corrupt, so attempts to copy those messages to Gmail failed using my Thunderbird trick.

I found an older copy that was in mbox format, and used the “Perfect” mbox to Maildir converter script to convert it to Maildir format.

Now I’m back to populating Gmail with my email once again (8,427 in Gmail now, with about 112,000 left to go).

Calendaring Conundrum

Also this past week, I realized that my calendar in Outlook had somehow duplicated over 1,700 of my events. I’m sure it was the result of using things like PocketMirror and other sync tools for Palm with it. I’m going to be cleaning that up next. That requires manual, visual inspection of each event, to make sure I’m deleting the dupe and not the original (thus leaving the dupe copy on the calendar). Very odd.

Once that is done, I have to reinstall all of my Palm conduits on the Thinkpad X61s and get that all sync’d to my Treo. With everything on my Treo, I can then begin consolidating my various calendars and task lists into one clean interface.

Back to the Blog; 8 years of postings

I also cleaned up 8 years of blog postings, reformatted them all and cleaned up the broken HTML that was the result of importing the diary entries from Advogato. That was 353 separate posts to go through by hand and clean everything up. Now it looks as it should.

Going back and reading through those old diary posts was… interesting. I didn’t realize how much I’d done in those 8 years, all of the people I’d met, projects I’d completed, places I’d been. I might turn the whole blog into a set of memoirs for my daughter Seryn for when she’s old enough to understand all of the things her daddy did in his life.

Movies, movies, movies!

I managed to pack in watching 4 movies while I worked this weekend, but the two best ones were “Maxed Out” and “The Man from Earth“. Both of them were equally good, and worth watching. I highly recommend both of them.

“Maxed Out” was eye-opening, and depressing at points, because of the situation our country is in right now. People are literally killing themselves (3 cases are described in the movie) because of their debt. The industry specifically caters to those who can NOT pay their debts down, because those people are the cash-cow for them. These people pay their minimum payments for life and pass on their debts to their children. They don’t want people who pay their credit cards in-full every month to be their customers, there’s no profit in that. Watch the movie for the rest of the details.

“The Man from Earth” brings new ideas to our concepts of religion, biology, archeology and many other fields of traditional study. It reminded me somewhat of the information that was in the first 1/3 of “Zeitgeist: The Movie” (freely downloadable, or viewable online). The Man from Earth is a low-budget movie, but packs a punch in the back story. I won’t spoil it here, but definitely go rent it if you can.

AT&T WWAN in VMware

I managed to get my physical Windows machine (an HP machine I purchased at BestBuy a few years ago), virtualized and configured in VMware using VMware Converter. I had to hack into it to get the vm to recognize my legitimate Microsoft Product Key, but after that, it was a snap.



Then I installed and configured the AT&T Communication Manager software to talk to the physical SIM card inside my laptop, so I can go online with the laptop wherever there is valid GSM signal, at 3G speeds.

I didn’t think the vm would recognize the physical card in the machine, if Linux didn’t see it natively… but it does. It’s a bit slow, but at least I can function with one laptop connected to the WWAN on the train with a larger screen at 1920×1200 resolution, instead of the smaller laptop with the 12″ screen at 1024×768 resolution.

The next step is to get the second laptop networking across the connection that the first laptop provides. That should be interesting to solve, since one of these is Windows, and the other one is Linux.

One possible solution would be to take the SIM card that is physically inside the laptop, put it into an external 3G PC-Express card (as in the image here), and then put that into a WWAN router, and carry THAT with me on the train. It has to be portable, of course…

But if I use that approach, not only can I share the connection with both of my laptops, I can also provide “free” wireless to anyone on the train who wants to get online. Maybe I can solicit free beer or donations in the cafe car to help offset those costs.

New Web SEO

An old acquaintance from a Business Development Group in New London has recently contacted me asking me for help with his website. He pointed out that his site is losing customers to two other sites in his very narrow niche here in Connecticut.

I looked at the competition, and noted that they’re not doing anything special that would merit that increase in customers for them. But I also noted about a dozen problems with this person’s company website in question, that needs immediate attention. His site is ranking at PR0, and the other two sites are pushing PR1/PR2, with no real traffic.

So I think I might pick up a little side project to help him out, to bring his site up to where it should be, and up to the level of standards that I consider acceptable. It looks like it’ll be fun, and it should bring in more income to help fund some projects I’m working on in the background (mostly Seryn’s secret project :)

I make a decent amount of money with my websites now, and I think with the proper care and attention, he can too.

Wrestling MySQL

Speaking of websites, a few months back, I started making some MFA 2.0 websites of my own, based on WordPress that are populated with hundreds of public-domain articles on niche content.

To do that, I wrote some tools (in Perl, of course) to spider the remote sites, pull the articles, and stick them into my WordPress databases, complete with proper author in the Author field, original posting date and so on.

Here’s one example that took me less than an hour to create and populate with articles, using these tools.

This particular site has 220 articles in it, but if you look at them articles, they’re all linked to external quality citations and resources. This site, with zero marketing, pulls in an average of about 6,000 hits a month. I have a handful of others, with between 200 and 1,000 articles in each, all of similar high-quality and care behind their creation.

The hard part is that in order to have an article attributed to an author in WordPress, that person has to be in the User table. That gets complicated to get right, but I managed to figure it out.

But I needed a way to ensure that the site’s content didn’t remain “stale”, so I came up with another trick to do that:

UPDATE wp_posts SET post_date='1999-01-01' + interval rand() * 3391 day;
UPDATE wp_posts SET post_modified = post_date;

I put this in /etc/cron.weekly/, and now my articles are randomized on a weekly basis, so visitors or search engines coming to the site will constantly see “new” content.

So far, they’re doing well, and bringing in more than enough to cover the monthly costs of running the servers, bandwidth and power.

The Day Job

The day job continues to go well, and I’m picking up significant speed on those projects.

The more I work within the system, the more I see where some “optimization” can be put into place, including some automation to make things much easier for myself and others in my group. I need to carve out some time to do exactly that, within the limits of my time allotted for the critical tasks that have established deadlines.

The commute has worked itself out and my routine is fairly static now, so that is no longer an unknown. Now I just need to get my foot in the door and keep things moving forward at a cheetah’s pace.

Fun times, definitely fun times. The positive energy has come back in rushing waves, and the good luck has overwhelmed me.