TLS certificates protect browser and API connections, but certificate work is rarely limited to clicking renew. This runbook shows how to choose an appropriate certificate, automate ACME renewal, verify the served chain, monitor expiry, and prepare CDNs, load balancers, and origin servers for a change without causing an outage.
Overview
A TLS certificate binds a hostname to a public key and gives clients information they can use to establish an encrypted HTTPS connection. In production, certificate failures usually occur at the edges of an architecture: a CDN still serves an old certificate, a load balancer has not imported the renewed files, an API hostname was omitted from the certificate, or the server sends an incomplete chain.
Start with an inventory rather than a renewal command. Record each public hostname, certificate type, issuing account, validation method, termination point, renewal owner, and expiry date. Include non-browser endpoints such as API subdomains, webhook receivers, admin portals, mail-related web services, and regional or staging environments that clients actually reach.
Certificate management has two separate concerns:
- Issuance and renewal: proving control of a hostname and obtaining a new certificate before the current one expires.
- Deployment and verification: placing the certificate and private key at every TLS termination point, then confirming that external clients receive the intended certificate and chain.
Keeping these concerns separate makes troubleshooting more precise. A successful ACME order does not prove that a CDN, reverse proxy, or origin server is serving the new certificate.
What to track
Choose the certificate shape deliberately
| Type | Useful when | Operational considerations |
|---|---|---|
| Single-domain | One stable hostname needs coverage. | Simple scope; adding another hostname requires a new certificate. |
| Multi-domain (SAN) | Several related hostnames must be covered together. | Maintain an explicit hostname list and remove names that should no longer be included. |
| Wildcard | Many subdomains at one DNS level share a certificate and trust boundary. | Usually requires DNS validation; protect the private key carefully because it can cover multiple services. |
| Private or internal CA | Internal services use managed client trust rather than the public web PKI. | Every client must trust the private root or issuing chain, and revocation and distribution need an internal process. |
Use the narrowest practical scope. A wildcard can simplify operations, but it also increases the impact of a private-key compromise. For public websites and APIs, confirm that every required hostname appears in the certificate's Subject Alternative Name list. The common name alone is not a reliable inventory of modern hostname coverage.
Track ownership, validation, and deployment
Your certificate register should contain at least:
- Hostname and environment, including production, staging, and regional variants.
- Certificate type, serial number, issuer, not-before date, and not-after date.
- ACME account or certificate-management owner, renewal method, and validation method.
- DNS provider or HTTP-serving path used for validation.
- Every TLS termination point: CDN, WAF, load balancer, ingress controller, reverse proxy, and origin.
- Alert destinations, last successful renewal, last successful deployment, and a documented rollback path.
For API certificate monitoring, test the hostname clients use from outside your network. An internal check against the origin can remain green while the public CDN continues to serve an expired or mismatched certificate.
Use ACME validation that matches the architecture
HTTP-01 validation requires the CA to retrieve a token from a predictable HTTP path, commonly under /.well-known/acme-challenge/. The path must reach the ACME client or challenge responder without authentication, an incorrect redirect, or a proxy rule that returns an application page. A simple check is:
curl -i http://example.com/.well-known/acme-challenge/test-token
That command is a routing check, not a valid certificate request. In a real order, the ACME client creates the token and serves the expected content. Confirm that port 80 reaches the correct edge and that deployment automation does not delete the challenge before validation completes.
DNS-01 validation places a TXT record at a name such as _acme-challenge.example.com. It supports certificates that HTTP-01 cannot validate, including many wildcard use cases, but it requires controlled DNS automation. Check the authoritative DNS response rather than relying only on a local resolver:
dig TXT _acme-challenge.example.com
Use narrowly scoped DNS API credentials where possible. A delayed or conflicting TXT record can make an otherwise correct ACME workflow fail. If several orders share a name, make sure cleanup does not remove a still-active validation record.
Automate renewal, then verify the result
ACME clients differ in configuration, so treat commands as patterns and consult the installed client's documentation. A typical Certbot-style dry run looks like:
sudo certbot renew --dry-run
The dry run tests the renewal path but does not replace your production certificate. A scheduled renewal should also trigger a controlled deployment or reload. For a service using PEM files, the sequence is commonly: obtain the new files, validate permissions, install them atomically, reload the service, and run an external HTTPS check.
Do not assume every service reloads certificate files automatically. Some cloud load balancers and CDNs require importing a certificate into a managed certificate store or attaching a new certificate version to a listener. Document that provider-specific step and test it in a non-production environment when the platform permits.
Inspect the certificate and chain
Use OpenSSL to inspect what a public endpoint presents:
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts
The -servername option matters when several HTTPS sites share an address. Review the negotiated endpoint, the hostname coverage, validity dates, issuer, and whether the server sends the expected intermediate certificates. To inspect a local certificate file:
openssl x509 -in fullchain.pem -noout -subject -issuer -dates -ext subjectAltName
Chain validation depends on the client's trust store, so test with more than one relevant client class when possible. A browser, a container image, an older operating-system image, and an API runtime may not have identical trust stores or TLS defaults. Avoid treating a successful browser test as complete evidence for every API consumer.
Cadence and checkpoints
A recurring review is more dependable than an emergency reminder. Use a monthly or quarterly cadence based on the size and criticality of your inventory.
Monthly checks
- Confirm every production hostname is present in the inventory.
- Review certificates approaching their internal renewal threshold, not only those near expiry.
- Check the last successful ACME order, deployment, and external certificate observation.
- Review failed validation, DNS API, reload, and notification jobs.
- Test that expiry alerts reach an on-call destination and have an owner.
Quarterly checks
- Run a controlled renewal or renewal dry run for each distinct workflow.
- Confirm CDN, load-balancer, ingress, and origin coverage after infrastructure changes.
- Review private-key storage, access permissions, backup handling, and secret rotation procedures.
- Remove retired hostnames and stale DNS validation records from the register.
- Test the documented rollback procedure on a representative service.
After a domain migration, DNS-provider change, CDN migration, load-balancer replacement, or hosting move, perform the checks immediately rather than waiting for the next scheduled review. Certificate failures often follow an infrastructure change because validation and deployment dependencies move with the service.
How to interpret changes
Expiry is approaching but renewal has not started: verify the scheduler, ACME account, credentials, and clock on the renewal host. Confirm that the job can write to its working directory and that notifications are functioning.
HTTP-01 validation fails: inspect DNS resolution, port 80 routing, redirects, CDN caching, authentication middleware, and IPv4 versus IPv6 behavior. The challenge must reach the responder consistently from the public internet.
DNS-01 validation fails: query the authoritative nameservers, check the exact TXT label, and inspect propagation and automation permissions. A successful local lookup is not enough if authoritative servers do not yet publish the record.
The new certificate is valid locally but old externally: identify the actual TLS termination point. The certificate may have been renewed on the origin while the CDN or load balancer still has the previous version attached.
Clients report an untrusted or incomplete chain: compare the served chain with the provider's recommended full-chain bundle. Check whether the server is sending intermediate certificates and test with the client environments used by your API consumers.
Only some clients fail: compare hostname, IP family, SNI, TLS policy, trust store, and network path. Multiple addresses or regions may not be serving the same certificate. Capture the endpoint and client details before changing configuration.
When to revisit
Revisit this runbook monthly or quarterly as part of certificate inventory and monitoring reviews. Update it whenever a CA changes validation behavior, a browser or client platform changes certificate requirements, an ACME client changes its commands, or your hosting provider changes certificate import and attachment procedures. The exact thresholds and compatibility rules should be verified against the current documentation for the CA, client, CDN, load balancer, and operating systems you use.
Use this pre-renewal checklist before any planned change:
- List every hostname and confirm the intended certificate scope.
- Identify the validation method and test DNS or HTTP routing.
- Confirm ACME credentials, DNS permissions, scheduler status, and alert delivery.
- Record every certificate termination point and its deployment procedure.
- Run a renewal dry run or staging order where supported.
- Back up configuration without broadly duplicating private keys.
- Deploy the certificate, reload or update each edge, and verify externally with SNI.
- Inspect the served chain and test representative browsers, API clients, and regions.
- Record the result, next review date, and rollback decision in the inventory.
For related operational work, connect certificate checks to your wider cloud workflow: browser tests can verify HTTPS navigation, an OpenAPI process can document API hostnames and environments, and a security-header review can confirm that HTTPS behavior remains consistent with HSTS and other protections. The goal is not merely to renew on time; it is to make certificate state observable, repeatable, and safe to change.