Inventory of certificates, private keys and nShield HSM kmdata files

2018-05-22 18:54:00

Building on my previous Thales nShield HSM blog post, here's a nice improvement.

If you make an array with (FQDN) hostnames of HSM-clients you can run the following Powershell script on your RFS-box to traverse all HSM-systems so you can cross-reference their certs to the kmdata files in your nShield RFS.

$Hosts="host1","host2","host3"

ForEach ($TargetHost) in $Hosts)

{
               Invoke-Command -ComputerName $TargetHost -ScriptBlock {
                              $Thumbs=Get-ChildItem cert:LocalMachineMy
                             ForEach ($TP in $Thumbs.thumbprint) {
                                             $BLOB=(certutil -store My $TP);
                                             $HOSTNAME=(hostname);
                                             $SUBJ=($BLOB | Select-String "Subject:").ToString().Replace("Subject: ","");
                                             $CONT=($BLOB | Select-String "Key Container =").ToString().Replace("Key Container = ","").Replace(" ","");
                                             Write-Output "$HOSTNAME $TP ""$SUBJ"" ""$CONT""";
                             }
              }
} 

 
$KeyFiles = Get-ChildItem 'C:ProgramData CipherKey Management DataLocalkey_caping*'
ForEach ($KMData in $KeyFiles) {
               $CONT=(kmfile-dump -p $KMData | Select -First 7 | Select -Last 1)
               Write-Output "$KMData $CONT";
}

For example, output for the previous example would be:

TESTBOX F34F7A37C39255FA7E007AE68C1FE3BD92603A0D "CN=testbox, C=thomas, C=NL" "ThomasTest"

C:ProgramData CipherKey Management DataLocalkey_caping_machine--a45b47a3cee75df2fe462521313eebe9ef5ab4                    ThomasTest

The first line is for host TESTBOX and it shows the certificate for the testbox certificate, with a link to the ThomasTest container. The second line shows the specific kmdata file that is tied to the ThomasTest container. Nice :)

kilala.nl tags: security, work, sysadmin,

View or add comments (curr. 0)

Matching Windows certificates to nShield protected keys (kmdata)

2018-05-22 18:39:00

Over the past few weeks I've had a nagging question: Windows certutil / certlm.msc has an overview of the active certificates and key pairs for a computer system, but when your keys are protected by an Thales nShield HSM you can't get to the private keys. Fair enough. But then there's the %NFAST_KMDATA% directory on the nShield RFS-server, whose local subdirectory contains all of the private keys that are protected by the HSM. And I do mean all the key materials. And those files are not marked in easy to identify ways. 

So my question? Which of the files on the %NFAST_KMDATA%/local ties to which certificate on which HSM-client?

I've finally figured it all out :) Let's go to Powershell!

PS C:Windowssystem32> cd cert:LocalMachineMy

PS Cert:LocalMachineMy> dir
   Directory: Microsoft.PowerShell.SecurityCertificate::LocalMachineMy

Thumbprint                                Subject
----------                                -------
F34F7A37C39255FA7E007AE68C1FE3BD92603A0D  CN=testbox, C=thomas, C=NL
...

So! After moving into the "Personal" keystore for the local system you can see all certs by simply running dir. This will show you both the thumbprint and the Subject of the cert in question. Using the Powershell Format-List command will show you the interesting meta-info (the example below has many lines remove).

PS Cert:LocalMachineMy> dir F34F7A37C39255FA7E007AE68C1FE3BD92603A0D | fl *
...
DnsNameList              : {testbox}
...
HasPrivateKey            : True
PrivateKey               :
PublicKey                : System.Security.Cryptography.X509Certificates.PublicKey
SerialNumber             : 6FE2C038ED73E7A0469E5E3641BD3690
Subject                  : CN=testbox, C=thomas, C=NL

 

Cool! Now, the two bold-printed, underlined lines are interesting, because the system tells you that it does have access to the relevant private key, but it does not have clear informatin as to where this key lives. We can turn to the certutil tool to find the important piece to the puzzle: the key container name. 

PS Cert:LocalMachineMy> certutil -store My F34F7A37C39255FA7E007AE68C1FE3BD92603A0D
...
Serial Number: 6fe2c038ed73e7a0469e5e3641bd3690
Subject: CN=testbox, C=thomas, C=NL
 Key Container = ThomasTest
 Provider = nCipher Security World Key Storage Provider
Private key is NOT exportable
... 

Again, the interesting stuff is bold and underlined. This shows that the private key is accessible through the Key Storage Provider (KSP) "nCipher Security World KSP" and that the relevant container is named "ThomasTest". This name is confirmed by the nShield command to list your keys:

PS Cert:LocalMachineMy> cnglist --list-keys
ThomasTest: RSA machine
...

Now comes the tricky part: the key management data files (kmdata) don't have a filename tying them to the container names:

PS Cert:LocalMachineMy> cd 'C:programdata CipherKey Management DataLocal'

PS C:programdata CipherKey Management DataLocal> dir
...
-a---        27-12-2017     14:03       5336 key_caping_machine--...
-a---        27-12-2017     14:03       5336 key_caping_machine--...
-a---        27-12-2017     11:46       5336 key_caping_machine--...
-a---         15-5-2018     13:37       5188 key_caping_machine--a45b47a3cee75df2fe462521313eebb1e9ef5ab4...

So, let's try an old-fashioned grep shall we? :)

PS C:programdata CipherKey Management DataLocal> Select-String thomastest *caping_*
key_caping_machine--a45b47a3cee75df2fe462521313eebb1e9ef5ab4:2:   ThomasTest  ?   ∂   Vu ?{?%f?&??)?U;?m???   ??  ??  ??  1???B'?????'@??I?MK?+9$KdMt??})???7?em??pm?? ?

This suggests that we could inspect the kmdata files and find out their key container name. 

PS C:programdata CipherKey Management DataLocal> kmfile-dump -p key_caping_machine--a45b47a3cee75df2fe462521313eebe9ef5ab4
key_caping_machine--a45b47a3cee75df2fe462521313eebb1e9ef5ab4
 AppName
       caping
 Ident
       machine--a45b47a3cee75df2fe462521313eebb1e9ef5ab4
 Name
       ThomasTest
...

SHAZAM! 

Of course we can also inspect all the key management data files in one go:

PS: C:> $Files = Get-ChildItem 'C:ProgramData CipherKey Management DataLocalkey_caping*'

PS: C:> ForEach ($KMData in $Files) {kmfile-dump -p $KMData | Select -First 7)
C:ProgramData CipherKey Management DataLocalkey_caping_machine--a45b47a3cee75df2fe462521313eebe9ef5ab4
 AppName
       caping
 Ident
       machine--a45b47a3cee75df2fe462521313eebb1e9ef5ab4
 Name
       ThomasTest

 

kilala.nl tags: work, sysadmin,

View or add comments (curr. 0)

Microsoft OCSP Responders, nShield HSMs and vagueries

2018-05-17 20:18:00

Over the past few months I've built a few PKI environments, all based on Microsoft's ADCS. One of the services I've rolled out is the Microsoft OCSP Responder Array: a group of servers working together to provide OCSP responses across your network. 

I've run into some weirdness with the OCSP Responders, when working with the Thales / nCipher nShield HSMs. For example, the array would consist of a handful of slaves and one master server. Everything'd be running just fine for a week or so, until it's time to refresh the OCSP signing certificates. Then, one out of the array starts misbehaving! All the other nodes are fine, but one of'm just stops serving responses. 

The Windows Event Log contains error codes involving “CRYPT_E_NO_PROVIDER”, “NCCNG_NCryptCreatePersistedKey existing lock file” and "The Online Responder Service could not locatie a signing certificate for configuration XXXX. (Cannot find the original signer)". Now that second one is a big hint!

I haven't found out why yet, but the problem lies in lock files with the HSM's security world. If you check %NFAST_KMDATA%local you'll find a file with "lock" at the end of its name. Normally when requesting a keypair from the HSM, a temporary lock is created which gets removed once the keypair is provided. But for some reason the transaction doesn't finish and the lock file stays in place.

For now, the temporary solution is to:

1. Stop the Online Responder Service.
2. Remove the lock file from %NFAST_KMDATA%local.
3. Restart the Oniine Responder Service. 

With that out of the way, here's two other random tidbits :)

In some cases the service may throw out errors like "Online Responder failed to create an enrollment request" in close proximity to "This operation requires an interactive window station". This happens when you did not setup the keys to be module-protected. The service is asking your HSM for its keys and the HSM is in turn asking you to provide a quorum of OCS (operator cards). If you want the Windows services to auto-start at boot time, always set their keys up as "module protected". And don't forget to run both capingwizard64.exe and domesticwizard64.exe to set this as the default as well!

Finally, from this awesome presentation which explains common mistakes when building an AD PKI: using certutil -getreg provides boatloads of useful information! For example, in order for OCSP responses to be properly signed after rolling over your keypairs, you'll need to certutil -setreg caUseDefinedCACertInRequest 1.

(Seriously, Mark Cooper is a PKI wizard!)

kilala.nl tags: work, sysadmin,

View or add comments (curr. 0)