What are CSP and PKCS#11

CSP (Cryptographic Service Provider) and PKCS#11 (Cryptoki) are programming interfaces that allow applications to perform cryptographic operations using keys stored in external hardware (HSMs, smart cards, USB tokens, TPMs) without the application ever accessing the raw key material. CSP/CNG is the Windows-native interface; PKCS#11 is the cross-platform standard (Linux, macOS, Windows). When you sign code with a key stored in an HSM, the signing tool communicates through one of these interfaces — it sends data to the hardware, the hardware signs it internally, and returns the signature.

Why it matters

• Hardware key access — without CSP/PKCS#11, applications can’t use HSM-stored keys. These interfaces are the bridge between signing tools and secure key storage.
• Key never extracted — the application sends “sign this hash” through the interface. The hardware performs the operation and returns the signature. The private key never exists in application memory.
• CA/Browser Forum requirement — since June 2023, all code signing certificates must have private keys stored in hardware (HSM, smart card, or cloud HSM). CSP/PKCS#11 is how signing tools access these hardware-stored keys.
• Multi-vendor support — PKCS#11 is standardized. A signing tool that supports PKCS#11 works with any vendor’s HSM (Thales, Entrust, YubiKey, SoftHSM) without code changes.
• Audit compliance — using CSP/PKCS#11 with hardware keys provides evidence that signing keys are hardware-protected (FIPS 140-2 Level 2+), satisfying audit requirements.

How it works

PKCS#11 (cross-platform):

1. Application loads the PKCS#11 shared library (.so on Linux, .dll on Windows) provided by the HSM vendor
2. Application calls C_Initialize() to start a session with the token (HSM/smart card)
3. Application authenticates with a PIN: C_Login(session, CKU_USER, pin)
4. Application finds the signing key: C_FindObjects() with key label/ID
5. Application calls C_SignInit() + C_Sign() with the data hash
6. Hardware performs the signing operation internally
7. Signature bytes returned to the application

CSP/CNG (Windows):

1. CSP is registered in the Windows registry (installed by HSM vendor driver)
2. Application calls Windows Crypto API (CryptoAPI or CNG)
3. Windows routes the request to the appropriate CSP based on the certificate’s provider info
4. CSP communicates with the hardware (USB, network HSM, TPM)
5. Hardware signs, result returned through the CSP to the application

In real systems

signtool with CSP (Windows Authenticode):

# Sign using certificate in Windows certificate store (backed by HSM CSP)
signtool sign /n "My Company" /fd SHA256 /tr http://timestamp.digicert.com /td SHA256 MyApp.exe

# The certificate's private key is in an HSM accessed via CSP
# signtool never sees the private key — Windows routes through the CSP automatically
# Certificate store shows: "Private key is stored on a hardware device"

OpenSSL with PKCS#11 engine:

# Sign using HSM key via PKCS#11
openssl dgst -engine pkcs11 \
  -keyform engine \
  -sign "pkcs11:token=CodeSigning;object=release-key;type=private;pin-value=1234" \
  -sha256 -out signature.bin artifact.bin

# Or configure in openssl.cnf:
[engine_section]
pkcs11 = pkcs11_section
[pkcs11_section]
engine_id = pkcs11
MODULE_PATH = /usr/lib/libeToken.so

jarsigner with PKCS#11 (Java):

# Create PKCS#11 config file (pkcs11.cfg):
name = HSM
library = /usr/lib/libCryptoki2_64.so
slot = 0

# Sign JAR using HSM key
jarsigner -keystore NONE -storetype PKCS11 \
  -providerClass sun.security.pkcs11.SunPKCS11 \
  -providerArg pkcs11.cfg \
  -tsa http://timestamp.digicert.com \
  myapp.jar "release-key"

YubiKey as code signing token (PKCS#11):

# YubiKey provides PKCS#11 via ykcs11 library
# Generate key on YubiKey
yubico-piv-tool -s 9c -a generate -k --algorithm=ECCP256 -o public.pem

# Sign with OpenSSL using YubiKey PKCS#11
openssl dgst -engine pkcs11 \
  -keyform engine \
  -sign "pkcs11:token=YubiKey;object=Private key for Digital Signature;type=private" \
  -sha256 artifact.bin

Where it breaks

PKCS#11 library version mismatch — the signing tool expects PKCS#11 v2.40 functions, but the HSM vendor’s library implements v2.20. Certain operations fail with cryptic error codes (CKR_FUNCTION_NOT_SUPPORTED). Or the library works on one OS version but not another. Always test the exact library version with your exact signing tool version before production use.

CSP not registered after HSM driver install — the HSM vendor’s Windows driver is installed, but the CSP isn’t properly registered in the registry. signtool can’t find the private key even though the certificate is in the store. The certificate shows “You have a private key that corresponds to this certificate” but signing fails with “No private key.” Reinstall the vendor’s CSP/KSP middleware and verify registration with certutil -csp.

PIN caching and timeout — PKCS#11 sessions require PIN authentication. In CI/CD, the PIN must be provided programmatically. Some tokens lock after failed attempts. Some sessions timeout after inactivity, requiring re-authentication mid-signing batch. For automated signing, configure the token for long session timeouts and handle re-authentication in your signing scripts.

Operational insight

The CA/Browser Forum’s 2023 requirement that all code signing keys must be hardware-stored means every organization doing code signing now needs CSP or PKCS#11 integration. The simplest path for small teams: a YubiKey (FIPS 140-2 Level 2) with PKCS#11 — costs $50-100 and works with signtool, jarsigner, and OpenSSL. For CI/CD automation: cloud KMS (AWS CloudHSM, Azure Key Vault, Google Cloud KMS) with their native APIs or PKCS#11 interfaces. The days of code signing with a PFX file on a developer’s laptop are over — hardware key storage is now mandatory, and CSP/PKCS#11 is how you access those keys.

Related topics

• What is Code Signing
• Code Signing in CI/CD Pipelines
• What are Hardware Security Modules
• What is PKCS
• What is FIPS 140-2/140-3