The Certificate That Expired on a Load Balancer (And Nobody Noticed for 3 Days)

The Incident That Nobody Detected

A certificate expired on a Friday evening. The F5 load balancer kept running. Some traffic continued flowing. Nobody got paged. No alerts fired.

For three days — Friday, Saturday, and Sunday — a production service was partially broken, silently dropping connections from a subset of clients while passing health checks perfectly.

The operations team only found out on Monday morning when a customer escalation landed: “Your API has been returning TLS errors since Friday.”

This is the anatomy of a “silent certificate failure” — and it’s far more common than you’d think.

What Actually Happened

The Architecture

                    ┌─────────────────────────┐
Internet Traffic────▶│   F5 BIG-IP (Active)    │────▶ Backend Pool
                    │   VIP: 203.0.113.10     │     (8 app servers)
                    │   Cert: api.company.com  │
                    └─────────────────────────┘
                                │
                    ┌─────────────────────────┐
                    │   F5 BIG-IP (Standby)   │
                    │   VIP: (failover)       │
                    │   Cert: api.company.com  │  ← SAME expired cert
                    └─────────────────────────┘

The Timeline

Day 0 (Friday) - 18:00:
  Certificate "api.company.com" expires
  F5 continues serving the expired certificate
  
  Why no immediate failure?
  - F5 doesn't stop serving when cert expires
  - It continues presenting the expired cert to clients
  - Some clients reject it, some don't

Day 0 (Friday) - 18:00 to 23:59:
  ❌ Modern browsers: Show warning, users leave
  ❌ Strict API clients: TLS handshake fails immediately
  ⚠️  Older clients: Accept expired cert (insecure settings)
  ✅ Internal health checks: Don't validate cert dates
  
Day 1 (Saturday):
  Traffic drops 40% — interpreted as "weekend pattern"
  No alerts (monitoring checks HTTP 200 on backend)
  3 customer emails arrive in support queue (unread until Monday)

Day 2 (Sunday):
  Traffic remains low — still "weekend"
  2 more customer complaints
  One partner's automated system fails, sends error email

Day 3 (Monday) - 08:15:
  Support team reads customer escalations
  Incident declared at 08:30
  Root cause identified at 08:45 (certificate expiry)
  New certificate deployed at 09:30
  Full service restored at 09:45

Total silent failure window: ~63 hours

Certificate Outage Cost Breakdown

Per-incident cost across three severity scenarios

$8,095

Conservative

$22,260

Moderate (typical)

$44,220

Severe

Why Detection Failed

Health Check Configuration

# F5 health monitor configuration (actual config)
ltm monitor https /Common/api-health-check {
    defaults-from /Common/https
    interval 30
    timeout 91
    send "GET /health HTTP/1.1\r\nHost: api.company.com\r\n\r\n"
    recv "OK"
    # NOTE: No certificate validation settings
    # The F5 talks to backend over HTTP internally
}

The health check was monitoring the backend servers, not the client-facing TLS certificate on the F5 itself.

Monitoring Stack Gaps

Monitoring Layer	What It Checked	Certificate Status
F5 health monitor	Backend HTTP response	Not checked
Nagios	F5 management interface ping	Not relevant
Uptime Robot	HTTP 200 from public endpoint	Returns 200 (with expired cert)
Datadog APM	Application response times	Backend is fine
PagerDuty	Alert routing	No alert to route

None of these tools validated the client-facing certificate. The system was designed to detect backend failures, not infrastructure certificate problems.

The Partial Failure Problem

Not all clients handle expired certificates the same way:

Client Behavior with Expired Certificate:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Chrome/Firefox (recent):     ❌ Shows warning page, blocks
Safari (macOS):              ❌ Shows warning, allows bypass
curl (default):              ❌ Rejects with SSL error
curl (with -k flag):         ✅ Connects (ignores expiry)
Python requests:             ❌ Rejects (ssl.SSLCertVerificationError)
Python urllib3 (no verify):  ✅ Connects
Java (default):              ❌ Rejects
Java (trust all):            ✅ Connects
Mobile apps (pinning):       ❌ Rejects immediately
Mobile apps (system trust):  ❌ Rejects
Older IoT devices:           ✅ May accept (no validation)
Internal health checks:      ✅ Don't check cert validity

Because some traffic continued flowing, aggregate metrics (total requests, average latency) showed a decline but not a cliff — easily mistaken for normal weekend variation.

The Business Impact

Impact Assessment:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Duration:                    63 hours
Requests affected:           ~147,000 (failed TLS)
Requests succeeded:          ~89,000 (weak clients)
Revenue impact:              $234,000 (estimated)
Customer SLA violations:     3 enterprise accounts
Support ticket volume:       47 tickets
Engineering hours (incident): 12 hours
Post-mortem action items:    8 items
Trust impact:                Significant (3 escalations)

How Load Balancers Handle Expired Certificates

F5 BIG-IP Behavior

Certificate State → F5 Action:
  Valid            → Present to client (normal)
  Expired         → Still present to client (no change)
  Revoked         → Still present to client (F5 doesn't check OCSP on its own certs)
  Missing         → VIP stops accepting connections (clear failure)

F5 does NOT automatically stop serving when a certificate expires. It continues presenting the expired certificate, leaving it to clients to accept or reject.

Other Load Balancer Behaviors

Platform	Expired Cert Behavior	Built-in Alert
F5 BIG-IP	Continues serving	Optional (must configure)
Citrix ADC (NetScaler)	Continues serving	Optional
HAProxy	Continues serving	None built-in
nginx	Continues serving	None built-in
AWS ALB (ACM-managed)	Auto-renews (no expiry)	N/A
Azure Application Gateway	Continues serving	Optional metric
Cloudflare	Auto-renews	N/A

F5 Certificate Expiry Alerting (Often Not Configured)

# F5 iRule for certificate expiry monitoring (custom)
# This is what SHOULD be configured but often isn't

# tmsh command to check certificate expiry
tmsh list sys crypto cert /Common/api.company.com | grep expiration

# F5 BIG-IP built-in SNMP trap for cert expiry
# Must be explicitly enabled:
tmsh modify sys snmp traps {
    certificate-expiry {
        community public
        host 10.0.0.50
        port 162
    }
}

Prevention: Multi-Layer Detection

Layer 1: Certificate-Aware Health Checks

#!/bin/bash
# external-cert-check.sh - Run from OUTSIDE the load balancer
# Tests the actual client experience

ENDPOINTS=(
  "api.company.com:443"
  "www.company.com:443"
  "portal.company.com:443"
)

for endpoint in "${ENDPOINTS[@]}"; do
  host="${endpoint%%:*}"
  port="${endpoint##*:}"
  
  # Get certificate expiry
  expiry=$(echo | openssl s_client -connect "$endpoint" \
    -servername "$host" 2>/dev/null | \
    openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  
  if [ -z "$expiry" ]; then
    echo "CRITICAL: Cannot retrieve certificate from $endpoint"
    # Send alert
    continue
  fi
  
  expiry_epoch=$(date -d "$expiry" +%s)
  now_epoch=$(date +%s)
  days_left=$(( (expiry_epoch - now_epoch) / 86400 ))
  
  if [ $days_left -le 0 ]; then
    echo "CRITICAL: Certificate EXPIRED on $endpoint ($days_left days ago)"
  elif [ $days_left -le 7 ]; then
    echo "WARNING: Certificate expiring in $days_left days on $endpoint"
  elif [ $days_left -le 30 ]; then
    echo "INFO: Certificate expiring in $days_left days on $endpoint"
  fi
done

Layer 2: Synthetic TLS Monitoring

# Prometheus blackbox exporter configuration
# This performs REAL TLS validation like a client would
modules:
  tls_verify:
    prober: tcp
    timeout: 5s
    tcp:
      tls: true
      tls_config:
        insecure_skip_verify: false  # Actually validate the cert!
      
# Prometheus alert rules
groups:
  - name: certificate_monitoring
    rules:
      - alert: CertificateExpired
        expr: probe_ssl_earliest_cert_expiry - time() < 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "EXPIRED certificate on {{ $labels.instance }}"
          
      - alert: CertificateExpiring7Days
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate expiring within 7 days on {{ $labels.instance }}"
          
      - alert: CertificateExpiring30Days
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
        for: 4h
        labels:
          severity: info
        annotations:
          summary: "Certificate expiring within 30 days on {{ $labels.instance }}"

Layer 3: Load Balancer-Native Monitoring

# F5 BIG-IP - Enable certificate monitoring
# Create a script that runs daily on the F5

#!/bin/bash
# /config/scripts/cert-check.sh
THRESHOLD=30
NOW=$(date +%s)

for cert in $(tmsh list sys crypto cert one-line | awk '{print $4}'); do
  expiry=$(tmsh list sys crypto cert $cert | grep "expiration" | awk '{print $2, $3, $4, $5}')
  expiry_epoch=$(date -d "$expiry" +%s 2>/dev/null)
  
  if [ -n "$expiry_epoch" ]; then
    days_left=$(( (expiry_epoch - NOW) / 86400 ))
    if [ $days_left -lt $THRESHOLD ]; then
      logger -p local0.warning "CERT-EXPIRY: $cert expires in $days_left days"
    fi
  fi
done

Layer 4: End-to-End Mail Flow Test (for the actual incident)

# Simulate real client connection with STRICT TLS validation
# Run every 5 minutes from an external monitoring point

result=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time 10 \
  --cacert /etc/ssl/certs/ca-certificates.crt \
  "https://api.company.com/health")

if [ "$result" != "200" ]; then
  # Alert! Either the cert is bad or the service is down
  # Either way, customers are affected
  echo "ALERT: api.company.com returned $result (cert or service failure)"
fi

The Post-Mortem Action Items

After this incident, the team implemented these changes:

┌─────────────────────────────────────────────────────────┐
│ POST-MORTEM ACTION ITEMS                                 │
├────┬─────────────────────────────────────┬──────────────┤
│ #  │ Action                              │ Status       │
├────┼─────────────────────────────────────┼──────────────┤
│ 1  │ Deploy cert-aware external monitor  │ ✅ Done      │
│ 2  │ Add F5 cert expiry SNMP traps       │ ✅ Done      │
│ 3  │ Create cert inventory for all LBs   │ ✅ Done      │
│ 4  │ Implement 60/30/7-day alert ladder  │ ✅ Done      │
│ 5  │ Add TLS validation to health checks │ ✅ Done      │
│ 6  │ Document cert ownership for LBs     │ ✅ Done      │
│ 7  │ Evaluate CLM platform               │ In Progress  │
│ 8  │ Quarterly cert review for infra     │ Scheduled    │
└────┴─────────────────────────────────────┴──────────────┘

Load Balancer Certificate Management Checklist

Item	Frequency	Owner
Inventory all LB certificates	Monthly	Network team
External certificate validation scan	Every 5 minutes	Monitoring
Expiry alert at 90 days	Automated	CLM/Monitoring
Expiry alert at 60 days	Automated	CLM/Monitoring
Expiry alert at 30 days	Automated + escalation	Security
Certificate renewal	30 days before expiry	Certificate owner
Post-renewal validation	Same day as renewal	Network team
Cert sync between HA pairs	Every renewal	Network team

Key Lesson

The certificate didn’t fail catastrophically — it failed partially. And partial failures are harder to detect than complete outages. Your monitoring needs to validate what clients see, not just what servers report.

If your health checks don’t validate TLS certificates, you’re monitoring availability without monitoring usability. Your service could be “up” for days while customers can’t connect.

---

About QCecuring

QCecuring monitors certificates across all infrastructure layers, including load balancers. Our platform provides external validation that catches expired certificates the same way your customers would — before they file a support ticket, not after.

Tags: load balancer, F5 BIG-IP, certificate expiry, silent failure, partial outage, monitoring gaps, TLS validation, certificate monitoring, infrastructure certificates, production outage