Update Nov 2024
Whilst the content of this post is as valid today as it was originally, we became frustrated with being limited to signing on one machine. That meant our build agents were doing a lot of copying of files to and from the server with the token.
Our solution was to build a Code Signing Server - Signotaur - keep reading and then take a look at how Signotaur solves the problems we talk about in this post.
Big changes are coming for code signing certificates in 2023. New and reissued publicly trusted organisation validation (OV) and individual validation (IV) code signing certificates will have to be issued or stored on preconfigured secure hardware by the issuing Certificate Authority (CA) and the device must meet FIPS 140 Level 2, Common Criteria EAL 4+ or equivalent security standards.
This is already the case for EV (Extended Validation) certificates, and it presents some problems in an automated build environment. In this post we'll take a look at the issues with hardware-based certificates and how to work around them.
Why is this change necessary?
If you work in IT, you will have heard or read about the SolarWinds supply chain hack. It was a big deal. It's more common than we might think - in February 2022 NVIDIA had their code signing certificates stolen and they were used to sign malware (those certificates have since expired).
These (and other) episodes made many in the industry (Microsoft in particular) very nervous. Trust is a big deal when it comes to certificates, and that is certainly the case when it comes to certificate issuance, but there is not a lot of trust in how those certificates are secured by the developers using them. Ask anyone who has done the merry validation dance with a CA, it's not that easy to get a code signing certificate these days. With that in mind, the CA/Browser forum adopted a proposal to change the requirements for how issued certificates are stored.
The change makes a lot of sense - it's much harder to steal hardware than it is to steal files.
What does this mean
From 1 June 2023, all new and reissued publicly trusted OV and IV code signing certificates will have to be issued or stored on a pre-configured secure hardware device by the issuing certificate authority (CA) and the device must meet FIPS 140 Level 2, Common Criteria EAL 4+ or equivalent security standards.
Existing OV/IV certificates will continue to work, but if you need your certificate to be reissued you may encounter issues due to key size requirements changing (so don't lose your certificate). The reality is that most certificate providers have already switched to issuing certificates on tokens (or discontinued selling OV/IV certificates). This is the end of simply downloading a pfx.
What are these hardware devices
These devices fall broadly into 3 categories:
Network-attached Hardware Security Modules (HSM)
HSM's in this class bring a lot of benefits and functionality (like key usage audit logs) - code signing is just part of that. These devices are not cheap - usually in the "if you have to ask you probably can't afford" price range! There's a reason for that steep price though. They are designed to be ultra-secure and tamper proof - open the lid and you will likely lock it up or brick it.
CA's will charge you a premium if you BYOD (bring your own device) - expect an audit fee of around $500 - or you can employ your own suitably qualified auditor (no idea what the criteria is for that but it sounds expensive). You will also have to deal with creating a Certificate Signing Request (CSR) to send to the CA. The process varies depending on the device, and the CA websites don't offer much guidance there.
Cloud HSM's
Azure, AWS, Google and others provide cloud HSM's, a service layer in front of network-attached HSM's - you basically rent time/space on them. They typically charge a monthly fee and then a fee per cryptographic operation. CA's charge a hefty fee to create certificates for use on these services - up to $1200 - and on top of that some CA's charge per number of signings per year (no idea how they police that). You also need to do some work to configure secure access to the HSM. These services make sense if you are already running on the cloud. Like other HSM's you will need to create a CSR to send to the CA during the order/validation process
SSL.com offer an eSigner cloud-based service (they are also a CA), but the prices will make you think twice about code signing: USD $100 per month for 10 signings, plus $10 for each additional signing. We sign every build that might escape the building and each build has several files that need signing!
USB tokens
In my research so far, the most common USB token is the Gemalto/Thales SafeNet token. The only other ones I have encountered are Yubikey (only SSL.com seems to be using those) and Certum (for which I could find very little info on the client software). The token tends to be included in the price of the certificate (I did see one case where it was not). You do not need to create a CSR (unless you are using your own existing token) as the CA loads the certificate on the device before posting it to you. The one I have (from Digicert) is a Safenet token. It's already obsolete and cannot be used for new certificates as it doesn't support the larger key size required now (newer model required).
Locked In
One thing to note about all the possible hardware devices, whether it's yours or one you rent, is that once a certificate is installed on that device, it's private key cannot be exported. So if you decide the cloud service you are using is too expensive and want to move, well it's time for a new certificate.
Some CA's say they cannot reissue EV's on USB tokens, whilst others provide a procedure - it's likely you will be up for a new token cost and more verification hoops to jump through. So don't lose or damage it!
In the rest of this post, I'm only going to cover USB tokens. If you have access to network or cloud HSM's then you are probably well past this point.
So, what's the problem then?
Different USB tokens might use different client software/drivers - but they all have one thing in common - the USB token needs to be present (i.e. plugged in to the machine) when code signing. This seemingly innocuous little USB token (which looks just like a memory stick) needs to be physically secure. If someone walks past your machine and takes it (likely thinking it's a memory stick), well you are up a creek without a paddle. My SafeNet token has a bright blue LED on the end that just screams "Take me!". Our build servers are colocated at a data centre - so leaving things like USB devices plugged in is asking for trouble. It's not like I can walk over and plug it in when needed (every day!). The data center is 300km from where I live/work.
Add to this that build machines are typically virtual, so you are into the realm of USB passthrough. If you use Hyper-V Server (as we do), well you are bang out of luck.. not supported. I have heard that VMWare ESXI supports it just fine but have never used it. I tested with XCP-ng and did get it working, but it was a major hassle to configure (reams of commands and copying of guids).
USB - Remotely
Fortunately, there are alternatives to USB passthrough. I looked at a bunch of USB remoting products (USB over IP/UDP), and after poor results with most, I found one that works. In fact it works incredibly well, with much better pricing than the others.
That product is VirtualHere (VH). Of all the vendors I contacted, they were the only one who actually responded and answered the question I asked - "Does it support code signing tokens?". The author responded, "Actually I use my (Digicert) JC Token via VirtualHere to sign VirtualHere when I build VirtualHere inside a VM." Good enough for me to give it a try!
The VH Server runs on Windows, Linux, MacOS, a bunch of NAS servers, even a Raspberry Pi! The server licence is locked to the host machine, so if you decide to move the USB token to another host you will need to purchase another license - but at USD$49 that probably won't break the bank!
I installed the VH server software on my XCP-ng host - installation was trivial and took all of 2 minutes (I'm no Linux expert). I then plugged the USB token in (the server is in my mini rack at home) and installed the VirtualHere client software on my Windows machine.
The VH client can auto detect servers, however in my case the two machines were on different subnets, so I had to manually specify it. With the trial version, a message box shows up when it first connects. The client immediately showed a tree of the USB devices plugged into the server. The SafeNet token shows up as Token JC, right-click on it and select "Auto use this device" so it connects automatically. When I did this, the familiar Windows sound indicated a device had plugged in. I already had the SafeNet software installed so it didn't prompt me for drivers etc.
The last step in this USB remoting journey was to install the client as a service (right click on the USB Hubs node). This can only connect to licensed servers, so leave this step until you have purchased.
NOTE : I didn't make it clear before, but the VH client and token client software need to be installed on the machine where the signing takes place, ie your build machine or build agent machine.
Prompting for Passwords
Another issue with the USB tokens being present during code signing, is they also expect a human to be present - a password prompt is shown. That flies in the face of conventional wisdom - automate all the things!
Fortunately, for the SafeNet token at least, there is a work around for this.
Open the Safenet Authentication Client Tools, click on the Advanced View button (the gear). You may be prompted for the token password if you haven't already entered it. In the tree on the left, right-click on the certificate (under User certificates) and select Export certificate. Just to be clear here, this is the certificate without the private key (which stays on the token) - you can't use this exported certificate on a machine that does not have access to the USB token.
In the certificate view, under Private Key, take note of the Cryptographic Provider value (likely "eToken Base Cryptographic Provider" and the Container Name (p11#xxxxxxxxxxx). You will have to manually type them out somewhere as it doesn't support the clipboard. Save those values somewhere - as you will need them in your build process for code signing.
Whilst still in the Client tools, select Client Settings and go to the Advance tab, check the "Enable single Login" and "Enable single Logon for PKCS#11." options, and set Automatic Logoff to Never - then hit Save. You can close the client now.
Code Signing
With all that done, we can use Signtool action in FinalBuilder. The Signing option tab is where those values we saved earlier come into play.
The only difficult one is the Private Key container. Fortunately, some clever person on StackOverflow figure out the required format:
[{{%CSPWD%}}]=p11#xxxxxxxxxx
I have used a variable CSPWD for the token password.
That's it. In my tests I have run FinalBuilder from the Windows task scheduler while logged out, and from a Continua CI build agent service (again while logged out) and it worked in both instances. I did a bunch of login/out/reboot testing and it continued to work. VirtualHere has been flawless. The next step is to configure our CI agents to access the USB token over VPN. Sadly our EV token is about to expire (and we never used it once in production, our OV cert still has another year left) - so I first have to jump through the validation hoops to get a new one.
When using this technique on a CI server, you will need to take care that only one build at a time is using the token. In Continua CI, that is trivial to achieve using a shared resource lock.
I would love to test out some of the cloud HSM services too, but purchasing a certificate for each one is too rich for me. If you are using any of those with a cloud-based HSM, jump on our forums and let us know your experiences. If you experiment with or get up and running using VH let us know how it went - I might do a follow up post as we all gain more experience with usb tokens and code signing.
Warning (added 19 Oct 2022)
I probably should have pointed out that most tokens are configured to lock you out after too many authentication failures. So if you are getting auth failures when setting this up, stop, and manually login to the token to reset the failed count.
Rant
CA's (and their resellers) have some of the worst websites I have ever had the displeasure of reading. Pages and pages of useless or contradictory information with links promising more information that take you around in circles. Grrrrr.