HOWTO: Back up your Android device with native rsync

Android
Recently, one of my Android devices stopped reading the memory card. Opening the device, the microSD card was so hot I couldn’t hold it in my hand. The battery on that corner of the device had started to swell slightly. I’ve used this device every day for 3+ years without any issues. Until this week.

I also use TitaniumBackup to back up my Android to this external memory card, but since the device can’t read the card, I can’t back it up to the card.

The card is fine, and works in my other devices, as well as being seen from the desktop. Other, blank microSD card can’t be read in the device and similarly overheat within seconds. It’s bad.

Enter rsync, the Swiss-Army Knife of power, to back up my Android device!

Here’s how:

  1. Download the Android SDK and unpack or install it on your machine.
  2. Plug in your Android via USB cable to your machine (Mac, Windows or Linux).
  3. Go into Settings → About, and tap on the Build Number section until you’ve enabled “Developer Mode”.
  4. Exit out of the “About” section, and now you’ll see a new menu option: “Developer Options”.
  5. Select “Developer Options” and go to “USB Debugging”, and enable it. This will enable you to manage your Android device over USB.
  6. cd into the ‘platform-tools’ directory and run “adb”, so we can find your device:
    $ ./adb devices
    List of devices attached
    d682520f	device
  7. Let’s use “adb” again and open up a shell on your Android device as root. We’re going to run the rsync process as root in a moment. We could run it as a normal user, but would be restricted in the files we are permitted to read from the device. To make things easier, we’ll just do this all as root:
    $ ./adb -s d682520f shell
    shell@hlteatt:/ $ su -
    root@hlteatt:/ # id
    uid=0(root) gid=0(root) context=u:r:init:s0
    root@hlteatt:/ #
  8. Now we need to create a configuration file for the rsyncd server process to use in a moment. Using the ‘vi’ editor, create the following file, call it “rsyncd.conf” and put it wherever you think it appropriate. In my example, I put mine in /data/local/tmp/rsyncd.conf. The contents looks like this:
    address = 127.0.0.1
    port = 1873
    
    [root]
    path = /
    use chroot = false
    read only = false
    uid = root
    gid = root
  9. We need to download the ARM version of the rsync binary and push it to the device:
    wget -O rsync.bin http://github.com/pts/rsyncbin/raw/master/rsync.rsync4android
    $ file rsync.bin
    rsync.bin: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, stripped
    
  10. Let’s push that binary to the Android, using “adb” again:
    $ ./adb -s d682520f push rsync.bin /data/local/tmp/rsync
    2357 KB/s (793148 bytes in 0.328s)
  11. To execute rsync as root, we have to adjust the permissions of that binary:
    $ ./adb -s d682520f shell su - -c "chmod 755 /data/local/tmp/rsync"
  12. Run the rsync server process, listening on the standard port, by chaining the command through “adb”. You can also do this by logging into the Android via “adb shell” and running the same command directly. I’ve added the end-of-line continuation markers here so it doesn’t wrap wrong in this post:
    $ ./adb -s d682520f shell su - -c 
    "/data/local/tmp/rsync                 \
    --daemon --no-detach                   \
    --config=/data/local/tmp/rsyncd.conf   \
    --log-file=/proc/self/fd/2"
    
    2016/09/25 23:05:17 [21300] rsyncd version 3.1.1 starting, listening on port 1873

    All in one line here:

    $ ./adb -s d682520f shell su - -c "/data/local/tmp/rsync --daemon --no-detach --config=/data/local/tmp/rsyncd.conf --log-file=/proc/self/fd/2"
  13. In another terminal/shell on your host machine, forward the rsyncd port from inside your Android device to a listening port on your host machine. This command should produce no output:
    ./adb forward tcp:6010 tcp:1873
  14. Now we can run rsync on the host to pull the data from your Android over to your local disk. Create a directory to hold your backup and cd into it, then run the following:
    sudo rsync -avP --delete --inplace \
    --stats --exclude=/proc --exclude=/sys  \
    rsync://localhost:6010/root .
  15. It should then show your device starting to back up to your local disk. When it completes, you’ll see a summary, similar to the following:
    Number of files: 85920
    Number of files transferred: 3040
    Total file size: 39514100245 bytes
    Total transferred file size: 770291255 bytes
    Literal data: 215397682 bytes
    Matched data: 554901587 bytes
    File list size: 2311538
    File list generation time: 10.609 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 1529010
    Total bytes received: 218618474
    
    sent 1529010 bytes  received 218618474 bytes  1842238.36 bytes/sec
    total size is 39514100245  speedup is 179.49

Why would I want to do this instead of using “adb backup”? That style backup packs files into a single, binary container. Using rsync, you have direct access to individual files on your device, without having to restore them first.

That’s it! You now have a full (and I mean FULL) backup of every bit of data on your device. You can now push that to an SD card, thumb drive, storage media or another Android device.

Have fun!

HOWTO: Fix missing mouse clicks in VMware with Linux guests

This was a bugger to find out, and required installing and reinstalling Linux a dozen times in different ways, to narrow down on the actual cause.

If you run VMware Workstation on your Linux host, and are also trying to run Linux guests, you may run into a situation where your mouse cursor in the guest moves and tracks as you would expect, but any attempt to right-click or left-click is ignored entirely.

During my tests, I noticed that this happens 100% of the time with Ubuntu 14.04.4, does not happen with Ubuntu 14.04.2, and does not happen with Ubuntu 16.04.

But it is reproducible.

Originally, I thought this was due to having the wrong “mouse” vs. “vmmouse” driver in my Xorg configuration. That didn’t prove to be true.

Next, I tried doing some VMX file hacks, which also failed. I also tried disabling the “Drag and Drop” and “Cut and Paste” options in the VMware options. That also, did not solve the problem.

To fix the problem, go into your Guest settings and under the Processor sub-section, you want to uncheck the “Virtualize CPU performance counters” option. You can leave the rest as-is.

The option should look like this:

VMware disable CPU counters

That’s it! Hopefully that helps someone else out there running into the same problem!

HOWTO: Purge Amazon Echo History with iMacros

Amazon Echo IoT Companion

This one is quick and easy… Have you ever wanted to go back into your Amazon Echo device and delete the history of all commands you asked Alexa to do for you? All the searches? All the weather requests?

Well, you can… manually from the mobile app, or from the Amazon Alexa Configuration page, but that can take hours, because each card you wan to remove is a minimum of two taps or clicks.

But there’s an even easier way… iMacros!

Load up the iMacros browser extension (Chrome version) (Firefox version) and create a new macro. You can edit it ‘raw’, if you wish, but you want only these lines in your macro:

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
URL GOTO=http://alexa.amazon.com/spa/index.html#cards
TAG POS=1 TYPE=BUTTON ATTR=TXT:More
TAG POS=1 TYPE=SPAN ATTR=TXT:Removecard

Now when you load up the Amazon Alexa Configuration page, you can just launch your macro from iMacros and play it in a loop to progressively delete each and every one of those cards in seconds.

I personally wiped out over 5,000 cards in under 2 minutes with this approach. It works great!

Comment below if you have any luck with it, or modify it in a way that becomes more useful to others.

No more criminals for president. Not this time. Never again.

No more Hillary Clinton - no more fraudI rarely speak out on specific presidential candidates, but I’ve pretty much had enough of this comedy unfolding that we call our political process. With Trump making a complete arse of himself, Ted Cruz saying “we need another war president” and other candidates flat-out lying, making up facts and paying third-party companies to create false campaigns and disinformation on their competition, I’ve had it.

The worst offender? That’s right… Hillary Rodham Clinton. A felon, admittedly breaking dozens of federal laws, and trying to use her fame to erase the legality of those laws. Her campaign continues to build on its own corruption, greed and malice.

Read the rest of this entry »

HOWTO: Run multiple Zwift sessions on the same PC (Windows only)

Zwift LogoMany people have asked me to write this up and I’m happy to be the first person to push Zwift this far with multiple, simultaneous sessions.

I can say with confidence that up to this point, I’m actually the only person who has this working correctly without overwriting or clobbering critical logs and data files. Others have tried some hacky methods, but they all result in instability and data loss (see “What does NOT work, and why” below).

I started this quest because I am working on a product design (“Secret Sauce” to be withheld in this HOWTO) that involves running multiple Zwift sessions on a single, 100% wireless PC, with the only wire being the single power cable to the wall. No USB cables, no video cables, no HDMI cables, no network cables.

Let’s get some general housekeeping out of the way first…

Read the rest of this entry »

HOWTO: Correct and avoid clock skew on Windows and OS X platforms

ntpd-server-statsThis has come up a lot recently in the context of Zwift rides and races, becasue many rider’s PC and Mac gaming rigs are suffering clock skew. In short, it can be defined as:

“Clock skew is when the clock arrives at different points of the circuit at different times due to the distance, capacitance etc which may cause it to malfunction.”

If you want more detailed set of diagrams and explanations describing clock skew, there’s a great discussion on the StackExchange electronics forum about it.

When your machine is suffering from clock skew, you can inadvertently enter a ride or race earlier or later than you meant to do, and your overall finish time can be later than you expect. For races where the winner is separated from the other podium places by mere seconds or less, having an accurate clock matters!

Here’s how to fix it for both Microsoft Windows and Mac OS X platforms!
Read the rest of this entry »

HOWTO: Fully automated Zwift login on Mac OS X

Zwift LogoQuite a few riders on the Facebook Zwift Riders group have expressed an interest in this, so I decided to take a couple of hours, learn AppleScript and knock this out. Done! (if you’re on Windows, you want this other HOWTO instead)

What this code does, is allows you to create a single icon that will log you into Zwift, with no human interaction needed. It will put in your email, password, click the “Start Ride” button and away you go!

This also leverages the OS X Keychain to store your Zwift email address and password, so it’s secure, not leaked into the filesystem and is able to be called on by any other apps that might need it (ahem, like… Zwift itself!) :D

So here’s how to get it working…

First, we need to create a separate keychain to store the Zwift credentials. You could store them in the main keychain, but I’m a fan of credential separation, so let’s use that.

Launch Keychain Access on your Mac (cmd + spacebar, type in “Keychain”).
Mac OS X Keychain Access

You’ll see a number of keychains listed there in the upper-left of Keychain Access. We’re going to create a new one, so go to File -> New Keychain and call it whatever you want.

I called mine “Zwift” so I can remember it when I see it on the filesystem or in the app later. It should default to save in ~/Library/Keychains/. Don’t change this path for now.

When you click “Create”, you’ll be prompted for a password to secure that keychain. Make it something relatively strong if you want to protect your credentials. If you don’t care, make it weak. Click on “Ok” and it will be created and saved.

Creating the Zwift keychain

Now right-click on the new keychain you just created in the list and select “Change Settings for Keychain Zwift”. We’re going to adjust the timeout when you have to re-enter your password to unlock this keychain.

Changing Zwift keychain settings

If you want a fully automated login, where you never have to enter a password or interact with this at all, uncheck both boxes, so it doesn’t lock after inactivity or when your computer goes to sleep.

If you prefer a bit more control/security, change the settings as you see fit.

Mine looks like this:

Zwift keychain timeout settings

Click on “Save” to save those settings.

Next, we need to add an account to the keychain. This will be your Zwift account, the same one you use to log into Zwift itself in the app and on the website. Click the little [+] at the bottom of the Keychain Access window to create a new entry. Here’s what it should look like when you’ve got it filled out correctly:

Creating a new Zwift keychain entry

Click on “Add” to add this entry to the keychain. Now you’ll see one entry in your list.

Zwift keychain user login entry

If you right-click on the entry, you can add some more details to it, but you don’t need to. I left it at the defaults.

Zwift keychain account additional details

Now let’s test that it locks and unlocks properly. Right-click on the keychain in the list on the left side and choose “Lock Keychain Zwift” (do not accidentally choose “Make Keychain Zwift Default”, or you’ll have a bad time)

Locking the Zwift keychain

Once locked, you’ll see the litte padlock icon next to it show “closed”. Right-click again and select “Unlock Keychain Zwift”, put in your password and see that it cleanly unlocks and that the padlock shows “open” next to the name:

Unlocking the Zwift keychain

That’s just about the hardest part of this process. Now on to the code!

I’ve never written a single byte of AppleScript until today, so I decided to give it a shot, learn the language, tried a few early attempts at this storing passwords in the code, then in files I’d read from disk, then encrypted files I’d decrypt, but that was messy. Why reinvent the wheel when OS X already has an encrypted keystore I can use? So I did.

Launch “Automator” (cmd + spacebar) and when prompted, select “Application” and click “Choose” to create one.
Mac OS X Automator
Mac OS X Automator Start screen

You’ll see a blank screen on the right and some macros and variables on the left. Don’t be scared, this is going to be EASY!

In the search dialog in the upper-left area, start typing “AppleScript”. You should see the list of items shorten to only one, as shown here:

Automator with AppleScript

Click that one entry and drag it to the empty canvas area on the right side of the Automator screen. When you let go, you’ll see something that looks like this on the right:

AppleScript starting point

Put your cursor in that window, select all of that default boilerplate and delete it, we’re going to start with a blank script here.

Blank AppleScript script

I’ve already written the code for you, so all you need to do here is cut and paste it into this window. Here’s the code (also available as a downloadable file by clicking this link)

on run {input, parameters}
  set userName to long user name of (system info)                 # User's full name
  set userHome to (system attribute "HOME")                       # User's home directory
  set secBin to "/usr/bin/security"                               # Full path to 'security' binary
  set kcName to "\"Zwift Login\""                                 # Keychain Name
  set kcPath to userHome & "/Library/Keychains/Zwift.keychain"    # Path to where the Zwift keychain lives
  set mySedMess to "sed 's/.*\"acct\"<blob>=\"\\(.*\\)\"/\\1/'"   # A horrible mess of sed. Nuff sed.
  
  # This is ugly, but it's the only way I could find to pull the account name from the Keychain.
  # Don't forget all of those escaping backslashes! (LTS - Leaning Toothpick Syndrome)  
  set zUser to do shell script (secBin & " " & "find-generic-password 2>&1 /dev/null -gs " & kcName & " " & kcPath & " | grep acct | " & mySedMess)
 
  set zPass to do shell script (secBin & " " & "find-generic-password -wa " & zUser & " -gs " & kcName & " " & kcPath)
  
  activate application "Zwift"

  tell application "System Events"
    delay 3                                              # Wait for the login dialog to show up
    set frontmost of process "Zwift" to true             # Force Zwift process to the front
    keystroke tab                                        # Put the cursor into the Email field
    keystroke zUser                                      # Send the username (from above)
    keystroke tab                                        # Jump to the Password field
    keystroke zPass                                      # Send the password
    keystroke return                                     # Press Enter to start the fun!
    
  end tell
  return input
end run

Cut and paste that into the script window (or use the direct link to the file).

At the very top of the file are a couple of minor tunables. Make sure those match what your system and environment are set up with. If you chose a different name for your keychain file for example, you’ll need to change that here. Likewise with the name of the account’s title within that keychain; change that here as well. If you called it “Zwift” and used “Zwift Login” as I did, you don’t need to change anything.

Also, there are some delays built into the script (search for the word ‘delay’). If your system is a bit slower, you may need to increase that delay by a few seconds.

cmd + S to save the script, which should prompt you for a name. I called this one “AutoZwift”, but you can call it whatever you like. This will become its own standalone .app file you can launch from anywhere by double-clicking on it, so feel free to put it wherever you want.

We’re not quite done! Before you close Automator, let’s make sure it works as expected. Click the little “Run” button on the far, upper-right corner of the Automator GUI to test the script. If you got everything correct, you should get no warnings, errors or popup dialog boxes.

Automator Run button

One last thing: Because you’re asking Automator to read events and pass keyboard events into windows owned by other processes, you need to grant Automator the permission and access to do so. To do that, go into your System Preferences -> Security & Privacy and make sure you enable (check the box) Automator to do so:

OS X Security & Privacy

OS X Automator Permissions

Now you should have a fully-automated Zwift login icon with credentials secured by your OS’ built-in encrypted keystore.

Good luck and #RideOn!

(p.s. For those run Zwift on Microsoft Windows, I’ve written a detailed HOWTO for you too! Stay tuned for more great HOWTOs for Zwift!)

HOWTO: Configure Windows to remember Zwift application passwords

Zwift Virtual Gaming Environnent
This will be a quick-n-dirty post to solve a very tactical problem. (if you’re a Mac user, use this other HOWTO I wrote for OS X instead)

This topic has been talked about dozens to hundreds of times in the Zwift Riders Facebook group and on the Zwift Support area through their Feature Request tracker. It’s become enough of an itch, that I decided to write up this post to detail how to “fix” the “problem” (not a Zwift bug, however).

First, make sure you’re using the latest available launcher from the Zwift website. Even if you have Zwift installed and it’s updating itself regularly, the launcher itself does not yet auto-update, so you need to go back to the website from time to time to pull a new version and update that.

Once installed, launch Zwift, and you’ll see a familiar dialog:
Zwift Application Login Dialog

Also, if you right-click on the Zwift icon in the System Tray, you should see the versions of both the Launcher and the Game itself:

Zwift System Tray applet details

If you double-click in the “Email” field of the Zwift login dialog, you should see nothing at all happen. This is what we expect, if your machine is not configured to remember stored passwords. We’re going to fix this in a moment…

Now enter your Zwift login email address and your password, and click “Log In”. When you do, you should see something like this pop up:

Zwift Internet Explorer Remember Passwords Dialog

If you don’t see this pop up at all, it means your machine (through a browser control) is not configured to remember passwords. We’re going to fix this.

If you do see this, you’re in good shape! Just click “Yes” on the dialog and your password will be remembered. You can head straight to “Go” and collect your 200W credit. If you log out and launch Zwift again, double-clicking on the “Email” field should now show you a drop-down with your saved username and password in it.

Back to those who aren’t seeing this dialog, let’s walk through the steps to fix it.

First, close the Zwift login window and then right-click the Zwift applet in the System Tray and select “Exit”.

Now launch Microsoft Internet Explorer. If you don’t use this browser, that’s ok, you may have never launched it before. Is it within this browser that we’re going to tell your OS to start saving your passwords.

Go to Start → Run and type in inetcpl.cpl to launch the Internet Settings Control Panel:

Windows - Start - Run dialog

Internet Settings Control Panel

Once here, click the “Content” tab, and then click “Settings” next to the Auto-complete section:

MSIE Content Settings

Now you’ll see a dialog that looks like the below. Don’t change any checkboxes there, but make sure the two boxes listed “User names and passwords on forms” and “Ask me before saving passwords” is checked, then click on “Delete Autocomplete Content”

MSIE Delete AutoComplete History...

Here, you’ll want to uncheck everything except “Passwords”, as shown here. Note, by changing this, you will be enabling Zwift to prompt you with the dialog to save passwords, but you will also be deleting any saved passwords you may have saved while using MSIE as an actual browser.

The next time you go to websites that you had saved passwords for, you just have to enter them again one time, and it will save them again. Unfortunately, there’s no way for MSIE to delete individual saved passwords, it’s an all-or-nothing ordeal.

MSIE Delete Browsing History Passwords

Now click “Delete” at the bottom to purge that data:

MSIE Delete Saved Passwords

Now if you click “Ok” on your way back to the main settings dialog for MSIE, and close the browser, your work there is done.

Launch Zwift again, put in your username and password, then click “Log In”, and it should prompt you to save your password. If it does, you’ve fixed the issue.

Close Zwift, re-launch it and now double-click the “Email” field in the login dialog, and your previously saved username should now show up there.

Zwift with Saved Passwords

Selecting it from that dropdown should also populate the password field, so you don’t have to type it in. Now you click “Log In”, and you’re done!

Not so painful, right? #RideOn!

2015 Tour de France Stages Mapped and Ready


The 2015 Tour de France is almost here! I did some searching and noticed there aren’t a lot of detailed maps and routes for this year’s Tour yet, as it’s still pretty early. Sooo I decided to start putting the routes together, road by road, turn by turn, lane change by lane change, as close to accurate as I could find them.

I’ll continue to update this blog post with detailed route maps, commentary, links to stage videos and more as more details come in. They’re still early and not 100% locked-down yet, so it’s unclear exactly which roads might be used for some of the later stages (as of 2015-07-01).

Here are the stages for the 2015 Tour de France, the 102nd tour:

July 4, Stage 1: Utrecht, individual time trial, 13.7km

July 5, Stage 2: Utrecht – Zélande, 166km

July 6, Stage 3: Anvers – Huy, 154km

July 7, Stage 4: Seriang – Cambrai, 221km

July 8, Stage 5: Arras – Amiens Métropole, 189km

July 9, Stage 6: Abbeville – Le Havre, 191km

July 10, Stage 7: Livarot – Fougères, 190km

July 11, Stage 8: Rennes – Mûr-de-Bretagne, 719km

July 12, Stage 9: Vannes – Plumelec, team time trial, 28km

July 13, rest day: Pau

July 14, Stage 10: Tarbes – La Pierre Saint-Martin, 167km

July 15, Stage 11: Pau – Cauterets-Vallée de Saint-Savin, 188km

July 16, Stage 12: Lannemazen – Plateau de Beille, 195km

July 17, Stage 13: Muret – Rodez, 200km

July 18, Stage 14: Rodez – Mende – Montée Laurent Jalabert, 178km

July 19, Stage 15: Mende – Valence, 182km

July 20, Stage 16: Bourg-de-Péage – Gap, 201km

July 21, Rest day: Gap/Digne-les-Bains

July 22, Stage 17: Digne-les-Bains – Pra-Loup, 161km

July 23, Stage 18: Gap – Saint-Jean-de-Maurienne, 185km

July 24, Stage 19: Saint-Jean-de-Maurienne – La Toussuire – Les Sybelles, 138km

July 25, Stage 20: Modane Valfréjus – Alpe d’Huez, 110km

July 26, Stage 21: Sèvres – Grand Paris Seine Ouest – Paris Champs-Élysées, 107km

HOWTO: Enable Docker API through firewalld on CentOS 7.x (el7)

centos-dockerPlaying more and more with Docker across multiple Linux distributions has taught me that not all Linux distributions are treated the same.

There’s a discord right now in the Linux community about systemd vs. SysV init. In our example, CentOS 7.x uses systemd, where all system services are spawned and started.

I am using this version of Linux to set up my own Docker lab host for tire-kicking, but it needs some tweaks.

I also wanted to see if I could use the Docker API from my Android phone, using DockerDroid, which (after configuring this) works famously!

Here’s what you need to do:

  1. Log into your CentOS machine and update to the most-current Docker version. The version shipped with CentOS 7 in the repo as I write this post, is “docker-1.3.2-4.el7.centos.x86_64”. You want to be using something more current, and 1.4 is the latest. To fetch that (and preserve your existing version), run the following:
    $ su -
    # cd /bin && mv /bin/docker /bin/docker.el7
    # wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker
    # systemctl restart docker
    # exit
    $ 
    

    Now you should have a working Docker with the right version (current). You can verify that:

    $ sudo docker version
    Client version: 1.4.1
    Client API version: 1.16
    Go version (client): go1.3.3
    Git commit (client): 5bc2ff8
    OS/Arch (client): linux/amd64
    Server version: 1.4.1
    Server API version: 1.16
    Go version (server): go1.3.3
    Git commit (server): 5bc2ff8
  2. So far, so good! Now we need to make sure firewalld has a rule to permit this port to be exposed for external connections:
    $ sudo firewall-cmd --zone=public --add-port=4243/tcp --permanent
    $ sudo firewall-cmd --reload
    success
    

    You can verify that this new rule was added, by looking at /etc/firewalld/zones/public.xml, which should now have a line that looks like this:

    <port protocol="tcp" port="4243"/>
  3. Now let’s reconfigure Docker to expose the API to external client connections, by making sure the OPTIONS line in /etc/sysconfig/docker looks like this (note the portion in bold):
    OPTIONS=--selinux-enabled -H fd:// -H tcp://0.0.0.0:4243
    
  4. Restart the Docker service to enact the API on that port (if successful, you will not see any output):
    sudo systemctl restart docker
  5. To test the port locally, install telnet and then try telnet’ing to the port on localhost:
    $ sudo telnet localhost 4243
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    
    HTTP/1.1 400 Bad Request
    
    Connection closed by foreign host.

    All looks good so far!

  6. Lastly, install DockerDroid and configure it to talk to your server on this port:

    DockerDroid connecting to CentOS via API

  7. Now you should be able to use DockerDroid to navigate your Images, Containers and API.

    Good luck!

Bad Behavior has blocked 698 access attempts in the last 7 days.