The Certificate Nobody Knew Existed (Until It Broke Production)
It was 11:47 PM on a Tuesday when the first alert fired.
API gateway returning 502 errors. Intermittent. Some requests succeeding, some failing. The on-call engineer checked the application servers — all healthy. Restarted the service — no change. Checked DNS, checked firewall rules, checked load balancer health probes.
By 2 AM, three separate services were affected. Internal billing system. Partner API integration. Customer-facing dashboard. All routed through the same F5 load balancer.
By 4 AM, someone finally checked the certificate on the F5.
Expired. Two hours ago.
The Certificate Nobody Tracked
This certificate wasn’t in any spreadsheet. It wasn’t in any monitoring tool. It wasn’t issued by the internal CA. It had no owner, no renewal process, and no documentation.
Two years earlier, a contractor had imported a wildcard certificate to the F5 to terminate SSL for a new API gateway project. The project launched. The contractor’s engagement ended. The certificate stayed.
Nobody documented it. Nobody added it to any tracking. Nobody assigned ownership. The certificate sat there, silently serving traffic, counting down to expiry.
Why This Happens More Than You Think
This isn’t an edge case. In every certificate discovery we run, we find certificates that:
- Were imported manually and never tracked
- Were configured by someone who no longer works there
- Exist on network appliances outside AD CS scope
- Have no documented owner or renewal process
- Were “temporary” and became permanent
| Category | Typical Count Found | Tracked? |
|---|---|---|
| Manually imported to load balancers | 15–40 per environment | Rarely |
| Installed by former employees/contractors | 20–60 | Never |
| On network appliances (F5, WAF, proxy) | 10–30 | Sometimes |
| ”Temporary” certs in production | 5–15 | No |
| Self-signed certs on internal services | 50–200 | No |
The Real Cost of This Incident
| Cost Component | This Incident |
|---|---|
| Downtime duration | 6 hours (partial) |
| Services affected | 3 |
| Engineer hours (incident response) | 4 engineers × 6 hours = 24 hours |
| Customer-visible impact | Dashboard errors for 4 hours |
| Partner SLA violation | Billing API down 6 hours |
| Root cause discovery time | 4 hours of the 6 |
| Post-incident review | 8 person-hours |
| Total estimated cost | ~$45,000 |
The fix took 15 minutes. Re-importing a renewed certificate. The other 5 hours and 45 minutes were spent finding the problem.
The Root Cause Isn’t Technical
The certificate expired because nobody knew it existed. But why didn’t anyone know?
No ownership model. When the contractor left, the certificate had no transfer of ownership. It belonged to nobody.
No inventory coverage. The F5 isn’t domain-joined. AD CS doesn’t see it. Auto-enrollment doesn’t reach it. It exists in a blind spot.
No monitoring scope. Monitoring was configured for certificates on web servers — not on load balancers. The F5 certificate was outside the monitoring boundary.
No documentation. The contractor completed the project, tested it, and moved on. No runbook, no cert tracking entry, no expiry reminder.
What Should Have Existed
For every certificate in production — regardless of how it got there:
- Documented existence — what is it, where is it, what does it do
- Assigned owner — who is responsible for its lifecycle
- Expiry visibility — when does it expire, who gets alerted
- Renewal process — how is it renewed, is it automated or manual
- Dependencies mapped — what breaks if this certificate expires
If any one of these had been in place, this outage would have been prevented.
The Pattern
Every “forgotten certificate” outage follows the same pattern:
Certificate installed → Person leaves → No documentation exists →
Time passes → Certificate expires → Outage →
Root cause hunt → "Who put this here?" → Nobody knows →
Manual fix → Promise to "track these better" →
Repeat in 6 months with a different certBreaking this cycle requires a system — not a promise.
How to Find Your Forgotten Certificates
Step 1: Audit all network appliances for SSL/TLS certificates (F5, HAProxy, nginx, WAF, reverse proxies)
Step 2: Check for certificates imported from external CAs (these won’t appear in your AD CS database)
Step 3: Interview teams who’ve onboarded infrastructure in the past 2–3 years — what did they install?
Step 4: Scan internal networks for services presenting certificates you don’t recognize
Step 5: Cross-reference findings against your current tracking
Anything that exists in Step 1–4 but not in Step 5 is a forgotten certificate waiting to cause an outage.
Next Step
We run a certificate discovery assessment that specifically looks for these shadow certificates — the ones outside AD CS, on appliances, imported by former staff, with no owner and no monitoring.
If your last outage took hours to diagnose — and the fix was a 15-minute cert replacement — the real problem is visibility.
Tags: Certificate Ownership, Shadow Certificates, Certificate Outage, Load Balancer, PKI, Certificate Visibility, Certificate Discovery, CLM