As one of the primary developers touching certificates in the Mercurial world, I am often asked how they work, what people need to do to use them and how they can work well with Mercurial.

A principal thing to remember here is that Mercurial will not work as a complete server out of the box, requesting authentication information, in the form of basic, digest, or certificates, at all. This means that in order to use X.509 certificates with Mercurial, one needs to place a web server that knows of these authentication mechanisms in front of it.

This guide is written at the existence of Mercurial version 1.5.1 and touches on many newer features, so if your distribution is old or lacks these features, perhaps it's time to upgrade to the latest and greatest.

X.509 certificates and SSL/TLS

The most commonly known aspect of X.509 are the certificates. At some level, one can think of a X.509 certificate as having a public and a private key. The public key is the one that secure websites present to you when you, for instance, visit Gmail. You can encrypt data to the intended recipient using this key and only with the corresponding private key can this information be decrypted. In reality, the actual public/private key pair is only used to compute a shared secret, and this is the secret used to communicate securely for the remainder of the conversation.

A X.509 certificate is issued by a certificate authority—some of the commonly known certificate authorities, i.e. that are trusted by a lot of people, can be seen in your browser's certificate settings—however, a certificate authority can be anybody you trust, e.g. yourself. This certificate authority can then issue server certificates (that represent a server's identity), and client certificates (that represent a client's identity). As long as you control the ecosystem of your environment, e.g. you know who your clients are, they know who you are, there is no reason to give out money to some third-party certificate authority, you can accomplish everything you need to on your own.

In order to become your own certificate authority and certificate issuer, you need one of two tools: OpenSSL or makecert.exe, the latter only being available on Windows (there are probably countless other tools as well, to be fair, but I do not cover them here). Below I will cover how to use each of these to set up everything, but before we delve into the details, we will look very briefly at what a certificate authority certificate is, and what ‘normal’ certificates are.

Certificate authority: A certificate authority is basically just a certificate that has signed itself. If you trust this certificate's public key, you will also trust any certificate that has been issued by this certificate authority.

Normal certificates: A normal certificate is just a certificate that has been signed by a certificate authority (or in reality a certificate authority or an intermediate certificate—you can have a full chain of certificates signing each other if needed).

Now, there are a lot of weird details in these certificates. You may hear terms thrown around such as OU (Organisational Units) and whatnot. The only important part for our purposes is the CN (Common Name). This is the name of the certificate holder, e.g. the server's name, or the user's name. So if you access your server as ‘hg.my.cool.server’, your server certificate's CN must be ‘hg.my.cool.server’.

OpenSSL

Like everything else in the X.509 certificate landscape, OpenSSL has a lot of options, switches and strange voodoo attached to it. Lest this post be turned into a really boring tutorial on certificates, I will just give a few very cursory examples of how you can generate the different certificates. There are plenty of different tutorials on this topic already. Note that Microsoft's IIS is a tad weird in how exactly it requires the certificate to be constructed, so we will not cover this for OpenSSL, but only for makecert.exe (inquiring minds can figure out all the details of generating certificate requests and processing these in OpenSSL if they prefer).

  • Certificate authority: openssl req -x509 -nodes -days 365 -subj '/CN=MyCA' -newkey rsa:1024 -keyout ca.key -out ca.pem

    The ‘MyCA’ part specifies the name of your certificate authority, so you will probably want to name it something a bit more descriptive.

    This gives us a certificate authority certificate with a private key stored in ca.key and the public certificate in ca.pem. You will need both of these to issue other certificates. Also, if you do not want to get scary pop-ups with invalid certificate trust models in your browser when you browse the hg repositories through hgweb, you will need to install ca.pem into the trusted root certificate authorities in your browsers.

  • Normal certificates:
    • Generate request for signing: openssl req -new -nodes -out name-req.pem -keyout name.key -subj '/CN=name'

      You should, of course, replace name with something more sensible, e.g. your server's fully qualified domain name (the one people will be entering in their browser to visit it).

    • Signing the request: openssl x509 -req -in name-req.pem -CA ca.pem -CAkey ca.key -out name.pem -CAcreateserial

    Do note that this is the quick-and-dirty way of generating certificates for use in testing/simpler controlled production scenarios. There are a lot more advanced features (like openssl ca) that will allow you to act more like a real certificate authority. If you need these things, go look up some OpenSSL tutorials, then come back here to see how things work with Mercurial.

    The final issue is getting these certificates into a form that your browser will like. They require the certificate to be in a special format, e.g. PKCS#12. To get that we use the openssl tool again like this: openssl pkcs12 -export -in name.pem -inkey name.key -out name.pfx. This file can be imported into your browser in a fitting place.

makecert.exe

makecert.exe is a bit of a different beast from openssl as it interfaces directly with the machine's or user's certificate store (the special place where certificates live a happy life in Windows).

  • Certificate authority: makecert.exe -pe -r -ss My -m 12 -n "CN=MyCA"

    This will install the root certificate into the current user's personal certificate store and mark the private key exportable. To see the current user's certificate store, run certmgr.msc.

    The certificate's public key will need to be exported and subsequently imported into the ‘Trusted Root Certification Authorities’ on each machine that trusts this certificate to issue other certificates.

  • Server certificate: makecert.exe -pe -is My -in "MyCA" -n "CN=name" -sky exchange -ss My -sr LocalMachine -m 12 -eku 1.3.6.1.5.5.7.3.1 -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12

    Do note that this certificate is imported into the local machine's personal certificate store—this is where IIS expects to find server certificates. To see these certificates, start mmc manually and add the certificate snap-in.

  • Client certificate: makecert.exe -pe -is My -in "MyCA" -n "CN=name" -sky exchange -ss My -m 12 -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12

If the client and server certificates aren't generated on the machines you need them on, you need to select the certificate in the management snap-in and export it, including the private key. Exporting as PKCS#12 is usually the safest bet.

This should basically cover how we get the actual certificates. Now, to use them.

Configuring a web server to use certificates

There are countless web servers and operating systems to run them on. I will give a very brief guide to running it on two of the most archetypical web server/operating system combinations: Apache 2.2 on Linux (Ubuntu), and IIS7 on Windows Server 2008. In particular, I will assume that you have already got Mercurial running and working just fine on both web servers.

Apache 2.2 on Linux (Ubuntu)

Setting up apache with mod_ssl can be done in a multitude of ways, I'll just presume you have enabled mod_ssl already and show how to create a really, really skeleton version of a SSL site that serves Mercurial repositories. We will set up a server that serves a few Mercurial repositories using hgwebdir, from the root of the website, and we'll only define a virtual host for listening on port 443.

Server certificates

The main configuration is as follows:

SSLEngine on SSLCertificateFile /path/to/myserver.pem SSLCertificateKeyFile /path/to/myserver.key SSLCACertificateFile /path/to/ca.pem SSLCACertificatePath /path/to ScriptAliasMatch ^(.*) /path/to/hgwebdir.cgi$1

Do note that if you have cloned the Mercurial repository, you should copy hgwebdir.cgi to a different location and make it executable. Furthermore, see HgWebDirStepByStep for further instructions.

This simple solution serves our self-issued server certificate when we access our website using TLS/SSL.

Client certificates

Client certificates do not require much more than that:

SSLEngine on SSLCertificateFile /path/to/myserver.pem SSLCertificateKeyFile /path/to/myserver.key SSLCACertificateFile /path/to/ca.pem SSLCACertificatePath /path/to SSLVerifyClient require ScriptAliasMatch ^(.*) /path/to/hgwebdir.cgi$1

This allows anyone with a validating certificate to connect, but no other checks are made. In order to restrict which certificates can be used to connect to the server, we can use the built-in fake authentication mechanism of mod_ssl's.

SSLEngine on SSLCertificateFile /path/to/myserver.pem SSLCertificateKeyFile /path/to/myserver.key SSLCACertificateFile /path/to/ca.pem SSLCACertificatePath /path/to SSLVerifyClient require SSLRequireSSL SSLOptions +FakeBasicAuth AuthName "FakeBasicAuth" AuthType Basic AuthUserFile /path/to/httpd.passwd require valid-user ScriptAliasMatch ^(.*) /path/to/hgwebdir.cgi$1

The fake basic auth basically means that we will create a file, httpd.passwd that contains the certificate subjects as usernames and password in its hashed form as password. Then mod_ssl takes care of verifying that the certificate is in this file before allowing it to access the remote location. To get the certificate subject to use as username, you can use openssl to get it: openssl x509 -subject -in name.pem. So for our simple client certificate example above, we would have a httpd.passwd file with the following contents:

/CN=name:xxj31ZMTZzkVA

The xxj31ZMTZzkVA part is password that has been hashed. Usually, if you've bought your certificate from a real certificate authority, your certificate will also contain country, organisation, and a host of other fancy things. You can also get these by embellishing the subject when you generate the certificate request, thus a real ‘username’ for a certificate might be /C=US/O=My fancy company/OU=West coast offices/CN=John Doe, but for our simple explanation purposes they merely detract from the message.

This is all it takes to set up client certificate authentication and authorization on Apache.

Internet Information Server 7 (Windows Server 2008)

Server certificates

The server certificate part is easy. Make sure the certificate is installed in the local machine's personal certificate store, then add a HTTPS-binding to the site you want to host and select the relevant certificate in the drop-down list. That's it.

Client certificates

Under SSL settings, set that the site requires SSL and that client certificates are required (few systems really work well with the optional part). This is the easy part. The hard part is that after IIS6, Microsoft has decided to drop the interface management UI for mapping certificates to Windows accounts completely, so we will have to do all this in an XML configuration file, and change some system XML-files and whatnot to get everything working.

The first file we need to edit is %SystemRoot%\system32\inetsrv\config\applicationHost.config where we need to find a line like this: <section name="iisClientCertificateMappingAuthentication" overrideModeDefault="Deny"> and change the Deny to Allow. This will allow us to configure this setting in the root of each website that your server is hosting. (If you have users who have write access to the file system, then they will also be able to override the setting for their virtual directories; it is up to you to figure out whether this poses a security risk for you).

In the root directory of your website, edit (or create) the file web.config that we can use to configure IIS7 (and ASP.NET for that matter, but that's probably not terribly relevant here). For each user we want to grant access we want the certificate's public key in PEM-encoded format (that is Base-64 in Microsoft parlance), formatted as a single string (i.e., you remove the newlines from the text-file and copy-paste the contents of the certificate, excluding the first and last line with the dashes). Below is a brief snippet that illustrates this:

Now we have told IIS that the user with the certificate CERTIFICATE_STRING logs in as the user testuser1 using the password test. However, even though we've set the website to require the use of client certificates, anyone presenting an arbitrary certificate can log in, since anonymous access is enabled. To turn this off, and only allow access for the specified users in the authentication section, we will need to modify the authorization rules as well as the authentication rules to disallow anonymous users. Below the /authentication line in the above configuration we add the following:

Unlike the certificate mapping, this can also be handled from inside the IIS configuration manager under authorization. Now, only testuser1 can log on to the site, using his client certificate. In general, you probably want some more users to be able to access your site, so you can go right ahead and add those as well.

Mercurial and X.509 certificates

All this, finally, brings us to how Mercurial interacts with certificates. There are, indeed, several things we need to cover here: how to use server certificates with ‘hg serve’ (the built-in web server), accessing a web server that requires client certificates from the Mercurial client, and how we can use the usual access control methods in Mercurial together with client certificates.

The built-in server, hg serve

Note: For hg serve to use SSL, you must have installed pyopenssl.

hg serve supports a --certificate option where you can specify a certificate file for a certificate that verifies. This detail is important, as right now, with the commands we issued above, we have a certificate file, we have a certificate authority file, and we have a private key file. We need to combine all these into a single file with this command: cat name.key name.pem ca.pem > hgserver.pem. If you have a certificate from a known certificate authority, you do not need to combine the certificate authority certificate into your certificate file.

To use this, you now simply just write hg serve --certificate hgserver.pem and you can access your stand-alone server using HTTPS.

Accessing a web server that requires client certificates

This is, of course, the really interesting part where everything comes together. In order to use client certificates you need them split up into two different files: name.pem and name.key (if you have followed the guide above for OpenSSL, you already have these two files, otherwise you need to generate them). Now we need to set up an auth section in our local hgrc file:

[auth] something.prefix = my.server.tld something.cert = name.pem something.key = name.key

When you try to clone, push, pull, incoming, outgoing to my.server.tld (or any path below that), we will automatically send along the specified certificate file and decrypt stuff using the key file. Also note that the something can really be anything, as long as they are connected. That is, if you have two different client certificates for two different servers you could do something like this:

[auth] something.prefix = my.server.tld something.cert = name.pem something.key = name.key somethingelse.prefix = my.other.server.tld somethingelse.cert = othername.pem somethingelse.key = othername.key

Be aware that the paths to the certificate and keyfile should be absolute (otherwise they're read relative to your current working directory).

There is currently no way to discern that certificates are included in the request or not (other than the request possibly failing). Also, the error messages that are given back if there are problems with certificates are less than stellar, but that is, unfortunately, the par for OpenSSL (which Mercurial uses by way of Python).

Access control using certificate users

Mercurial contains a number of mechanisms for controlling who can read/push from/to a specific repository. Furthermore, there are extensions such as the acl extension that allows you more fine-grained access control on who may push what files. The usernames that you can specify are delimited with either a space or a comma. See the problem?

A very common certificate username for Apache might be ‘/C=US/S=CA/O=ACME, Inc./OU=Coyote Solutions/CN=John Doe’. This will be parsed as allowing/denying access to the users ‘/C=US/S=CA/O=ACME’, ‘Inc./OU=Coyote’, ‘Solutions/CN=John’, and ‘Doe’. In short, you cannot use these mechanisms with Apache and client certificates right now (this should, of course, be rectified in the Mercurial source code somehow).

With IIS we have no such problem as we have mapped the client certificates to actual Windows account names, and we must just limit based on the Windows account names.

Verifying the server identity

The default for Mercurial (or any Python application for that matter), is to connect to a SSL/TLS server and not check that we actually trust the certificate being served. In fact, it is only possible to make Python do this from Python 2.6 and onward, and only if you specify special flags and special files to some of the SSL/TLS calls. Some kind developer who shall remain unnamed has added the necessary support to Mercurial so that we from Python 2.6 can actually verify the remote party. This is done by specifying a certificate trust file in your hgrc file in the web section:

[web] cacerts = /path/to/cacerts.pem

This file should contain a list of the certificate authority certificates (PEM encoded), one after the other. If you're already on a system that has OpenSSL installed, e.g. most GNU/Linux systems, this can be found in the file /etc/ssl/certs/ca-certificates.crt (or wherever your distribution places it). If not, then you will have to construct this file manually.

If verification fails, you will, again, get some cryptic OpenSSL error message.

Conclusion/Summary

These are all the many fancy ways that Mercurial can work together with certificates currently. Some of the ways are cumbersome, but they are hopefully not cumbersome due to Mercurial. We have tried to make them as easy to use as possible (if it can be said that certificates are easy to work with). It is also worth noting that prior to Python 2.6.4 (I believe), it is not possible to use client certificates with Mercurial, if you are also using a HTTP proxy server.

I will try to update this post if/when changes occur to the client and server certificate support in Mercurial. If you have suggestions or issues with using these, please file them on the Mercurial issue tracker, thanks!

Update: Thanks to Claudio for some useful comments and corrections.