Edward Thomson

Managing Dotfiles with Git

March 28, 2017  •  7:15 PM

Professional cooks have a term: "mise en place", which translates literally to "everything in its place". In a kitchen, it means that your station is prepared and well-organized; all the ingredients for every dish that you cook are prepared and set out in front of you so that they're ready to use. But perhaps most importantly: your station is clean, because professional cooks have another phrase:

Messy station equals messy mind. Clean station equals clean mind.

This idea extends far beyond the kitchen and - yes - even into software engineering and IT. Don't believe me? Take a look at your computer… Unless you're excruciatingly well-organized, you've probably got a junk folder somewhere. I call mine "Temp" on the very optimistic idea that I'm just throwing things in there for a little while until I find the right place for them.

But the reality is that most things never leave; they just accumulate.

Just like in a kitchen, all this crud and all these little annoyances take a mental tax and make you less productive. That's why one of my guilty pleasures is to "repave" my machine, reinstalling everything onto a freshly formatted drive with no trace of the old detritus, then carefully reinstalling only the apps that I'm interested in and copying over only the data that's worth keeping.

I try to do this every year or two, both to my main personal machine and my main work machine. And it turns out that I just finished this up, since last week I had a little free time on my hands:

At a high level, this process is pretty straightforward: I back up my home directory (twice), format the drive, reinstall the operating system, and carefully copy the contents of my home directory back, tidying as I go. On the whole, this is pretty tedious, except for one set of files: my dotfiles.

Years of working with large networks of Unix machines has taught me to version control my dotfiles so that I can get up and running on any new machine quickly. I keep my dotfiles checked in to a git repository, except for the truly important ones - the ones that I need to keep secure, like my SSH keys - which I keep with me.

SSH Keys

When I'm setting up a new workstation, I start by copying over my SSH keys since I'll actually need them in order to clone the rest of my dotfiles. (I skip this step if I'm setting up a new account on a remote machine, since I'll just use SSH Agent Forwarding to keep my keys on my local machine.)

I keep my most secure bits on a USB key that I keep with me. Most USB keys have some sort of attachment for a keyring - hence the name. Maybe it's a little lanyard, or a metal ring, or even a plastic tab of some sort.

USB keys that will inevitably fall off your keychain
These little keychain hooks are all going to break and your SSH keys will be left behind somewhere.

But just because you can attach these to a keyring doesn't mean that you should attach these to a keyring. That little hook is going to fall apart over time as you pull your keys out of your pocket or they jostle around in your purse, and one day your USB key isn't going to be attached to your keyring anymore. And then you've lost a copy of your SSH private key.

What I use is a USB key that is made from a hunk of aluminum:

A USB Key that might not fall apart
My keychain will fall apart before this USB key does. This one was made by Lacie, but they discontinued it a few years ago. Now Samsung produces one and there's a generic one as well.

This is much, much less likely to fall off my keychain unexpectedly. Even with that in mind, of course, I still turn on full-disk encryption to protect me in case I accidentally leave my keys laying around after a night out at the pub.

This particular key is an encrypted HFS+ partition for Mac OS, but I have a separate key for setting up new Windows machines, which is encrypted with Bitlocker To Go. (But more on that later.)

The Rest of the Dotfiles

After copying over my SSH public and private key into my home directory, I run a script that clones my dotfiles repository from my personal Visual Studio Team Services account. (I use VSTS for private repositories and GitHub for all my open source repositories.) I keep this on my USB key so that it's handy:

git clone --separate-git-dir=.dotfiles.git ethomson.visualstudio.com:DefaultCollection/personal/_git/dotfiles .
rm .git
echo '*' > .dotfiles.git/info/exclude

This script is pretty simple, but there are a few odd things going on:

  1. It uses the --separate-git-dir option to git clone. It does this so that my git repository in my home directory is called .dotfiles.git, instead of the usual name, .git. This is important so that when I run git somewhere in my home directory it doesn't accidentally do work inside my dotfiles repository.

    In particular, I might be somewhere beneath my home directory - say in ~/src/my_new_project - and run git status. If I haven't yet inited a repository for my new project, then I'll actually see the status for the git repository in my home directory that controls my dotfiles. This is confusing at best; it's best to make this a separate git directory instead.

  2. This removes the .git file that git clone creates. When I clone with a separate-git-dir, git will helpfully set up a .git file that points to that separate git dir so that future invocations of git can find it. This would be useful if my goal was to put the git directory on another device but still have it work transparently as if it were a normal git repository.

    That transparency is exactly the behavior I don't want, though. By removing this file, I'll get the isolation that I wanted in step 1, but I'll have to specify the git-dir every time I run git to work on my dotfiles. (More on that below.)

  3. This sets up an info/exclude in the dotfiles git repository with a * wildcard. This means that all untracked files in my home directory will be ignored. As a result, I will have to explicitly add new files to my dotfiles repository. Without this, git status would show all the files underneath my home directory which is very noisy and - worse - makes it very easy to accidentally add things like my SSH keys.

    I could also accomplish this with a .gitignore, but then I've got a .gitignore hanging out in my home directory and the whole point of repaving my machine was to clean up the unnecessary cruft in my home directory.

Managing Dotfiles

As a consequence of using a separate git directory for my dotfiles, I can't simply run git commands - like git status - in my home directory. If I do, git will tell me that it can't find a repository:

fatal: Not a git repository (or any of the parent directories): .git

Instead, I need to explicitly pass the --git-dir option to git so that it can find my repository. The easiest way to do that is to set up an alias in my shell's startup configuration.

I have this in my .zshrc:

alias dotfiles="git --git-dir=$HOME/.dotfiles.git"

Now instead of running git status, I run dotfiles status:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   .zshrc

no changes added to commit (use "git add" and/or "git commit -a")

And it's straightforward to dotfiles add .zshrc, dotfiles commit and finally dotfiles push to get my changes back up to my hosted git repository. I simply replace git with dotfiles in any command I want to run when I want to work on my dotfiles repository.

The only thing to remember is that - by virtue of my info/excludes above - I'm ignoring all the files in my home directory that aren't already part of the repository. This means that when I create a new file in my home directory, git ignores its presence:

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean

This is generally what I want; most of my home directory probably shouldn't be under version control or pushed to every computer that I use. But when I do create a file that I actually want to push to my dotfiles repository, it's easy for me to add:

dotfiles add -f .dotfile

The -f flag is necessary to override the info/excludes file. If I forget it, git will remind me. Once the file is staged, now git will begin tracking it, and dotfiles status will show:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   .dotfile

This straightforward setup lets me get my home directory configured on a new machine quickly and easily, and if I make any changes on that machine, I can share them with my home directory on every computer just by pushing and pulling.

The Git Contributor Summit

February 5, 2017  •  5:20 PM

The annual Git Merge conference just wrapped up, and it was another exciting year. As always, the speakers were excellent, the training was informative, and the after-party was a blast. But my favorite part was a part of Git Merge that most people don't see: the Git Contributor Summit.

The Git Contributor Summit is held in conjunction with Git Merge, and it allows the people who work on the git project, and related projects like libgit2 and jgit, to get together and talk shop.

The most important part of the Git Contributor Summit, I think, is the face-to-face time that it enables. This is a rare pleasure for most of us; the git contributors tend to chat on the mailing list, the libgit2 contributors talk in GitHub issues or in our Slack room, and the jgit contributors… well, I don't contribute to jgit, but I assume they have all the same problems that the rest of us do where we start to lack the personal touch.

And the personal touch is critical. When I first started working on the libgit2 project, I suggested bringing in some dependency on another project… and the maintainer wasn't enthused by this idea. In fact, he hated it. He hated the idea so much that he threatened to fly to the United States and murder my family if I actually merged that change.

Needless to say: we didn't really get off on the right foot.

But not too long after that, we met at a Contributor Summit and I discovered that he's actually a lovely, friendly person… he just happens to have a bit of an odd sense of humor. And so I knew that the next time he threatened to murder my family that he was just kidding.

Of course, it's not all just social time. Jeff King is part of the team of three maintainers who see to the actual business of the git project and he kicks off the contributor summit with that business. He catches us up on the health of the project as an entity and its budgetary and legal concerns.

But primarily we talk about the technical issues around git: the problems we have, and how we intend to fix them. Often these are discussions about scaling git in various directions, but we also talk about making git easier to use and, perhaps more importantly, easier to contribute to.

The amazing part about this is that we do this working together, despite having different interests and different needs. There are employees from git hosting companies, hackers who work on git in their spare time, and people who maintain git in their organization, whether that's a school, a small website or one of the world's largest news organizations.

And the hosting companies like GitHub, GitLab, Atlassian, and Microsoft all sit down together to talk about the problems they have in common and the ways they've each solved them, and they do it politely and without a hint of competition. There's no bragging about success or laughing at failures, just working together for the benefit of all our users.

It's an honor to be able to sit down with all the Git Contributors and work together on these solutions, and something I look forward to every year. Thanks so much to the organizers of Git Merge, and to the organizers and attendees of the Git Contributor Summit, for making it happen.

Goodbye, Microsoft

December 3, 2015  •  10:33 AM

Six years ago, Microsoft bought the tiny company that I worked for and moved five of us out of a cornfield in Central Illinois and into the largest software company in the world. In that time, I've done some unlikely things like hack on obscure Unix systems, write a bunch of GPL-licensed code and helped bring distributed version control into Microsoft.

But after six years, I've decided that it's time for a change. Yesterday was my last day at Microsoft, and I'll be starting a new position at GitHub in the new year.

Microsoft was great fun and I learned so much. I worked with so many great people and I'm sad to be leaving. But I'm also very happy that my friends and family will stop asking me why they can't run their old games on Windows 10 and what kind of monsters are we? And, of course, I'm also nervously excited for the next chapter in my career.

Authenticating Git clients to TFS using Kerberos

November 14, 2015  •  10:56 AM

You can authenticate to correctly configured instances of Visual Studio Team Foundation Server by using Kerberos over the Negotiate (SPNEGO) protocol. By using authentication with a Kerberos ticket, you can more securely authenticate from supported clients to your server without providing your password. After you obtain a Kerberos ticket, you can configure your git client to use Kerberos.

Configure Team Foundation Server to accept Kerberos Tickets

Team Foundation Server accepts only NTLM credentials by default, however it may be configured to accept Kerberos authentication. Access the Team Foundation Server Configuration Center by launching Team Foundation Server Administration Console.

Required Permissions
You must be a member of the Administrators security group on the server on which you will configure TFS.

To configure Team Foundation Server to accept Kerberos Tickets:

  1. In the Team Foundation Server Administration Console, select the Application Tier settings.
  2. On the Application Tier settings, select Authentication Settings.

    Application Tier Settings Dialog

  3. In the Authentication Settings dialog box, select Kerberos. This will enable both Kerberos and NTLM authentication.

    Authentication Settings Dialog

Configure the correct service principal name (SPN) for your server

A service principal name links the Active Directory service account running your Team Foundation Server instance to the service itself, and is required for Kerberos authentication to your Team Foundation Server.

Required Permissions
You must be a member of the Domain Administrators or Enterprise Administrators security group, or have been delegated the Validated Write to Service Principal Name permission, on the server on which you will configure an SPN.

You should configure an SPN for the fully-qualified domain name as well as the short name of the machine. If your server name is servername.contoso.com, you should configure an SPN for HTTP/servername and HTTP/servername.contoso.com.

If your Team Foundation Server Application Tiers are configured to use a network service account then you must configure an SPN for the computer account. For example:

setspn -c -S HTTP/servername SERVERNAME$
setspn -c -S HTTP/servername.contoso.com SERVERNAME$

If your Team Foundation Server Application Tiers are configured to use a domain account then you must configure an SPN for the domain account instead. If the application tiers are configured to run as CONTOSO\tfsuser for example:

setspn -c -S HTTP/servername CONTOSO\tfsuser
setspn -c -S HTTP/servername.mycompany.com CONTOSO\tfsuser

For more information, see the https://technet.microsoft.com/en-us/library/cc731241(WS.10).aspx on Microsoft TechNet.

Authenticate to Team Foundation Server Using Git

Ensure that you have a Kerberos Ticket Granting Ticket (TGT) for your user principal. If you log in to the computer using your Kerberos credentials, this may be done for you. If not, you may perform this manually using the kinit command and providing your Active Directory domain credentials, specifying the Active Directory domain as its DNS name. For example, the user CONTOSO\testuser should be specified as testuser@CONTOSO.COM:

kinit testuser@CONTOSO.COM

You can verify that you have a TGT using the klist command. You should see a “default principal” specified as testuser@CONTOSO.COM.

Finally, you may use the git tool with a Team Foundation Server remote repository. To indicate to git that it should authenticate with your Kerberos credentials, enter an empty username and password. For example, to clone a repository named “Test”:

git clone http://servername.contoso.com:8080/tfs/DefaultCollection/Test

And simply press Enter when prompted for your username and password.


  1. Ensure that your Team Foundation Server accepts Kerberos credentials. Use a network monitoring tool such as Wireshark or Fiddler to examine a successful connection from Visual Studio to Team Foundation Server. If the connection uses NTLM authentication instead of Kerberos, examine your server configuration.

  2. Ensure that your git client was compiled with the Kerberos system libraries. Use a network monitoring tool to examine an unsuccessful connection from git to TFS. If the connection does not attempt to use Kerberos, contact your operating system vendor for assistance.

Setting Files Executable in Git on Windows

June 9, 2015  •  11:36 AM

Sometimes you need to check an executable into your Git repository and - even though Windows doesn't really have a concept of an "executable bit" - you might need to set it executable for the other platforms.

For example, a handful of my native code projects use the very clever clar project for unit testing. Clar uses a Python script to introspect your unit tests for the actual test functions and generate the harness. It's handy if this generator is executable on Unix boxes and - thankfully - Git will let you set it executable even on a Windows box.

Once you've staged your file, you can set it executable with the git-update-index --chmod=+x command. For example:

C:\Repo> git add generate.py
C:\Repo> git update-index --chmod=+x

And now you can ensure that the file is executable by inspecting the index:

C:\Repo> git ls-files --stage
100755 587efb519c0ab60692e8d4c19f3471e0dbbecd3d 0       generate.py

If you're familiar with Unix permissions, then you'll note that first column looks like a Unix permission. Indeed, coming from a nice Unix heritage, Git uses the mode 100644 to represent a non-executable file and 100755 to represent an executable file.1

Now when you commit this change and check it out on a Unix system, generate.py will be set executable!

  1. Note, however, that this only looks a lot like a Unix permission. Regardless of the file's actual permission, Git only stores 100644 for regular files and 100755 for executable files. Effectively, this only stores whether a file is executable or not.