30 October 2026, 08:05Виталий Лещенко

Kubernetes for beginners. Part 19: How does cert-manager issue certificates?

All parts of the series “Kubernetes for beginners”

In the previous part Traefik became the single entry point at 192.168.10.200, and the apps gateway got a *.apps.example.com certificate — self-signed, issued by hand with openssl for thirty days. In a month it expires, browsers do not trust it, and the key sat on a laptop. Today we hand certificates over to the cluster: cert-manager 1.21 obtains them from Let's Encrypt over the ACME protocol or from our own certificate authority, puts them into a Secret and renews them ahead of time. How certificates, chains of trust and ACME work is covered in detail in part 23 of the networking series; here we look at what Kubernetes does with it and which objects appear in the API.

The lab is as it was at the end of the previous part: the demo namespace with the hello Deployment and service from part 11 and part 15, MetalLB 0.16.1 in L2 mode from part 17, the Gateway API 1.6.3 CRDs, Traefik 3.7.16 with the apps gateway, and two Ingresses for the name hello-ingress.apps.example.com: hello on the websecure entry point with TLS from the hello-ingress-tls Secret, and hello-redirect, which redirects HTTP to HTTPS. In the previous part we created the hello-ingress-tls Secret by hand; now it does not exist, and cert-manager will issue it — a real one, from Let's Encrypt. For that the router forwards ports 80 and 443 from its public address 203.0.113.10 to 192.168.10.200, and the public DNS zone has the record hello-ingress.apps.example.com A 203.0.113.10.

What cert-manager is

cert-manager is a CNCF project (Cloud Native Computing Foundation, the foundation where Kubernetes and related projects are developed), a set of controllers and CRDs that turn a certificate into an API object. We write "I want a certificate for these names from this issuer, stored in this Secret", and the controller makes it so and keeps it so. It is the same reconciliation loop as a Deployment's, only the desired state is a valid certificate.

The installation consists of three Deployments in the cert-manager namespace:

cert-manager (the controller) — the main controller: it watches Certificates, Issuers and the other resources, talks to ACME servers and creates Secrets;
cert-manager-webhook — an admission webhook: the API server sends it every cert-manager object before writing it to etcd, the webhook fills in defaults and validates the object, and a mistake in a manifest shows up right at kubectl apply instead of later in the controller log;
cert-manager-cainjector — writes CA certificates into the caBundle field of objects the API server has to trust when it calls webhooks (ValidatingWebhookConfiguration, MutatingWebhookConfiguration, CRDs with conversion). First of all it serves cert-manager itself: its webhooks carry the annotation cert-manager.io/inject-ca-from-secret: cert-manager/cert-manager-webhook-ca.

Installation

The official manifest installs everything at once, CRDs included. We install the current release, 1.21.3:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.3/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
...
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

The manifest puts the leader election roles into kube-system: that is where the controller and cainjector keep their Leases cert-manager-controller and cert-manager-cainjector-leader-election. Worth remembering when uninstalling — the Leases stay behind if you delete only the namespace.

Gateway API integration is off by default in cert-manager — the log of the freshly started controller has the line "skipping disabled controller" ... controller="gateway-shim". It is turned on with the controller flag --enable-gateway-api. Let's add it to the container arguments with a JSON patch:

$ kubectl -n cert-manager patch deployment cert-manager --type=json \
    -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enable-gateway-api"}]'
deployment.apps/cert-manager patched
$ kubectl -n cert-manager logs deploy/cert-manager | grep -i gateway
I1029 13:15:44.246719       1 options.go:296] "enabling the sig-network Gateway API certificate-shim and HTTP-01 solver" logger="cert-manager"

Two things got enabled: the gateway-shim, which issues certificates from annotations on a Gateway, and the HTTP-01 solver through an HTTPRoute — more on both below. The Gateway API CRDs are already installed by now (we installed them in the previous part), and that matters: the controller checks for them only at startup. If you install them later, cert-manager has to be restarted — kubectl -n cert-manager rollout restart deployment cert-manager.

One Helm command gives the same result: helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=true --set config.gatewayAPI.enabled=true; the chart puts gatewayAPI.enabled: true into the controller configuration file (the cert-manager ConfigMap, a ControllerConfiguration resource) instead of the flag. What Helm and values are is covered in detail in part 27. Besides the cluster itself we need the cmctl utility — the cert-manager command-line client for checking status and renewing by hand. It lives in a separate repository, cert-manager/cmctl, and is installed as a binary from its releases page (2.6.2 here, with a checksums.txt next to it for verification).

API readiness is checked like this:

$ cmctl check api --wait=2m
The cert-manager API is ready

This command tries to create a test object in dry-run mode: if the webhook is not up yet or cainjector has not yet written its CA, it waits. A typical mistake is creating an Issuer right after kubectl apply and getting a "failed calling webhook" rejection: the webhook is not ready yet.

The chain of objects

cert-manager has six CRDs, and when issuing over ACME their objects appear in a chain:

 Issuer / ClusterIssuer        who issues: ACME, own CA, Vault, self-signed
        ▲
        │ issuerRef
 Certificate                   what is needed: names, lifetime, key, Secret
        │ creates
        ▼
 CertificateRequest            one request: CSR + issuer reference; Approved
        │ creates (ACME only)
        ▼
 Order                         an order at the ACME server for a set of names
        │ creates one per name
        ▼
 Challenge                     prove control of a name: HTTP-01 or DNS-01
        │ creates
        ▼
 solver pod, Service, HTTPRoute    or    TXT record in DNS
        │
        ▼ (after issuance)
 Secret kubernetes.io/tls      tls.crt, tls.key  ── read by Traefik, apps

An Issuer and a ClusterIssuer describe who signs certificates and how. An Issuer lives in a namespace and serves only Certificates from that namespace; a ClusterIssuer is cluster-wide, and Certificates from any namespace can use it. Issuer types: acme (Let's Encrypt and any other ACME server), ca (sign with your own CA whose key is in a Secret), selfSigned (every certificate signs itself), vault (HashiCorp Vault, a secrets store with a built-in certificate authority), venafi (a commercial certificate management platform), and external issuers installed as separate controllers.

A Certificate is the desired state: names (dnsNames, ipAddresses), the issuer, secretName, lifetime, key parameters. Its controller compares the desired state with what is in the Secret: no certificate, it expires soon, the names are wrong, the issuer is different — so a new one is needed.

A CertificateRequest is one specific signing request: a CSR (Certificate Signing Request, a request with the public key and names, signed with the private key) and a reference to the issuer. Before signing, the request must be approved (the Approved condition); cert-manager's built-in approver approves requests to all Issuers and ClusterIssuers, but it can be replaced with a policy that, for example, does not let the demo team issue a certificate for someone else's name. Every renewal creates a new CertificateRequest with the revision number in its name, and old ones are deleted according to revisionHistoryLimit. It defaults to 1: only the latest request stays in the history, and on the lab request -1 disappeared together with its Order after a renewal.

Order and Challenge appear only with ACME issuers and mirror the protocol's entities one to one.

ACME, HTTP-01 and DNS-01

ACME (Automatic Certificate Management Environment, RFC 8555) is the protocol a client uses to obtain a certificate from a certificate authority automatically. The client registers an account (a key pair), creates an order for a set of names, gets a challenge for each name and proves it controls the name; then it sends the CSR and downloads the certificate. Details are in part 23 of the networking series.

There are two kinds of challenges, and the choice determines what has to be set up in the network.

HTTP-01. The certificate authority requests http://<name>/.well-known/acme-challenge/<token> on port 80 and expects the string <token>.<account key thumbprint> in the response. For this cert-manager starts a small solver pod (cm-acme-http-solver-…, the cert-manager-acmesolver image), a NodePort Service for it and a route — an HTTPRoute with the exact path /.well-known/acme-challenge/<token>, or an Ingress. Let's Encrypt does not check the name from a single point: the primary validation server comes first, followed by several more from other networks and regions, and the response has to match for most of them. So hijacking the route to one server is not enough to fake the answer. Limitations:

the name must resolve in public DNS to an address whose port 80 Let's Encrypt can reach from the internet. Our 192.168.10.200 is a private address, so the router needs port forwarding from a public address (203.0.113.10 for us) to 192.168.10.200 and a public record hello-ingress.apps.example.com A 203.0.113.10 (or *.apps.example.com right away);
a wildcard certificate cannot be obtained with HTTP-01.

DNS-01. The certificate authority queries the TXT record _acme-challenge.<name> and expects a hash of the same string in it. cert-manager creates the record through the DNS provider's API, waits until it is visible and deletes it after validation. Built-in support covers Cloudflare, Route53, Google Cloud DNS, Azure DNS, DigitalOcean, Akamai, ACME-DNS and RFC2136 — the standard DNS dynamic update protocol signed with a TSIG key, understood by BIND and Windows DNS; other providers are plugged in through external webhooks. DNS-01 is needed when:

you need a wildcard — *.apps.example.com;
the service is not reachable from the internet: internal dashboards, a cluster without port forwarding. It is enough for the DNS zone to be public; the server itself can be anywhere.

DNS-01 has its price: the provider's API key sits in the cluster, and it can change any record in the zone. So the token is issued with minimal rights — editing records of one zone — and, if the provider allows it, _acme-challenge is moved to a separate zone through delegation or a CNAME (cert-manager follows such a CNAME when the solver has cnameStrategy: Follow).

One common consequence: all certificates from public certificate authorities end up in Certificate Transparency logs — public logs of all issued certificates, where entries are only appended and never removed; browsers require a certificate to be recorded in them. The name of an internal service in a Let's Encrypt certificate becomes known to the whole world, and this is not theory: on the lab, two minutes after issuance the Traefik log showed requests to hello-ingress.apps.example.com from unknown addresses — scanners read the CT logs and probe new names right away. For internal names this is an argument for your own CA, covered below.

Let's Encrypt: staging, prod and limits

Let's Encrypt has two environments:

	Directory URL	How it differs
staging	`https://acme-staging-v02.api.letsencrypt.org/directory`	certificates from a test hierarchy with names like `(STAGING) Pretend Pear X1` that nobody trusts; much higher limits
prod	`https://acme-v02.api.letsencrypt.org/directory`	real certificates, strict limits

The production limits most often hit while debugging:

Limit	Value
new certificates per registered domain (`example.com`)	50 per 7 days
certificates for the exact same set of names	5 per 7 days
failed validations per name and account	5 per hour
new orders per account	300 per 3 hours

Hence the rule: staging first. We point the issuer at staging, get Ready: True, and only then switch the issuer to prod. Five failed validations an hour are used up in a couple of minutes if HTTP-01 fails because of a routing mistake and cert-manager keeps retrying.

Let's Encrypt certificate lifetimes are getting shorter. Today the default classic profile issues 90-day certificates; from February 10, 2027 it will issue 64-day ones, from February 16, 2028 — 45-day ones. The tlsserver profile already gives 45 days, shortlived — 160 hours, about six and a half days. Staging gets such changes before production: on the lab staging issued a 64-day certificate and production a 90-day one. The profile is chosen with the profile field of the ACME issuer. The takeaway for configuration: do not set renewal as a fixed number of days, more on that below.

Issuers for the lab

The DNS provider's API key goes into a Secret. For a ClusterIssuer, secrets are looked up in cert-manager's own namespace (called the cluster resource namespace; in the manifest it is the flag --cluster-resource-namespace=$(POD_NAMESPACE)), so the Secret is created there. Say the example.com zone is served by Cloudflare:

$ kubectl -n cert-manager create secret generic cloudflare-api-token \
    --from-literal=api-token='<token with Zone:DNS:Edit for example.com only>'
secret/cloudflare-api-token created

The staging issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-account
    solvers:
      - selector:
          dnsNames:
            - "*.apps.example.com"
        dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: apps
                namespace: traefik
                kind: Gateway
                sectionName: http

privateKeySecretRef is the Secret where cert-manager puts the ACME account key on first contact (RSA 2048 by default). There is no email field here, on purpose. The address is an optional account contact; Let's Encrypt stopped sending certificate expiration emails on June 4, 2025, so watching expiry dates is the job of cert-manager and your monitoring anyway. If you do set an address, it has to be real: with email: admin@example.com the account does not get registered:

$ kubectl describe clusterissuer letsencrypt-staging | sed -n '/^Status:/,$p'
Status:
  Acme:
  Conditions:
    Last Transition Time:  2026-10-29T13:17:07Z
    Message:               Failed to register ACME account: 400 urn:ietf:params:acme:error:invalidContact: Error validating contact(s) :: contact email has forbidden domain "example.com"
    Observed Generation:   1
    Reason:                ErrRegisterACMEAccount
    Status:                False
    Type:                  Ready
Events:                    <none>

Without the email field the issuer became ready within seconds — Reason: ACMEAccountRegistered, Message: The ACME account was registered with the ACME server, and status.acme.uri got the account URL on the server.

Solvers are chosen by selectors: dnsNames matches only exactly those names, dnsZones matches the zone and all its subdomains, matchLabels matches the Certificate's labels; a solver without a selector matches everything, and when several match, the more specific one wins. Here the wildcard goes through DNS-01 and all other names through HTTP-01, with a route on the http listener of our gateway.

The prod issuer differs only in its name, the server URL and the account Secret (letsencrypt-prod-account). Staging and prod have different accounts, and that is normal.

An internal certificate authority

For names that should not show up in public logs (hello.demo.svc.cluster.local, internal dashboards, TLS between services) we set up our own CA. That takes three objects: a self-signed issuer, a root certificate issued by it, and a ca issuer that signs everything else with this root:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: example.com Internal Root CA
  secretName: internal-root-ca
  duration: 87600h          # 10 years
  privateKey:
    algorithm: ECDSA
    size: 384
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-root-ca

Everything is ready in a couple of seconds:

$ kubectl get clusterissuer
NAME                  READY   AGE
internal-ca           True    3s
letsencrypt-prod      True    2m15s
letsencrypt-staging   True    5m45s
selfsigned            True    3s
$ kubectl -n cert-manager get certificate
NAME               READY   SECRET             AGE
internal-root-ca   True    internal-root-ca   4s

The selfsigned issuer is needed exactly once — to issue the root. The root Secret lives in cert-manager (with the keys ca.crt, tls.crt, tls.key): the internal-ca ClusterIssuer reads it from there. The root key is the most valuable piece of this setup: whoever reads it can issue a certificate for any internal name. Only administrators should have access to Secrets in cert-manager (RBAC is in part 29), and Secrets in etcd should be encrypted, as in part 12. More serious organizations keep the root outside the cluster and give the cluster only an intermediate CA.

Clients need to trust this root. trust-manager, a separate controller from the same project, helps distribute it across namespaces. It reads a Bundle resource (cluster-scoped, trust.cert-manager.io/v1alpha1), assembles a set of trusted certificates from sources and writes it into a ConfigMap in selected namespaces; applications mount this ConfigMap as a file. The project ships it as a Helm chart, installed with one command: helm install trust-manager oci://quay.io/jetstack/charts/trust-manager -n cert-manager --wait. The 0.26.0 chart itself uses cert-manager: the certificate for the trust-manager webhook is issued by the trust-manager Issuer in the cert-manager namespace.

The trust-manager documentation warns: do not use the root's Secret itself as a source. When the root is reissued, trust-manager immediately distributes the new one, the old one drops out of the trusted set, and everything signed by the old one stops validating. The right way is to copy the root certificate (without the key!) into a separate ConfigMap in the cert-manager namespace and change it when you decide to:

$ kubectl -n cert-manager get secret internal-root-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > internal-root-ca.crt
$ kubectl -n cert-manager create configmap internal-root-ca-public --from-file=ca.crt=internal-root-ca.crt
configmap/internal-root-ca-public created

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: example-internal-ca
spec:
  sources:
    - useDefaultCAs: true        # public roots from the ca-certificates package
    - configMap:
        name: internal-root-ca-public
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.pem
    namespaceSelector:
      matchLabels:
        trust.example.com/internal-ca: "true"

Let's label demo and look at the result:

$ kubectl label namespace demo trust.example.com/internal-ca=true
namespace/demo labeled
$ kubectl get bundle
NAME                  CONFIGMAP TARGET   SECRET TARGET   SYNCED   REASON   AGE
example-internal-ca   ca-bundle.pem                      True     Synced   4s
$ kubectl describe bundle example-internal-ca | sed -n '/^Status:/,$p'
Status:
  Conditions:
    Last Transition Time:  2026-10-29T13:27:25Z
    Message:               Successfully synced Bundle to namespaces that match this label selector: trust.example.com/internal-ca=true
    Observed Generation:   1
    Reason:                Synced
    Status:                True
    Type:                  Synced
  Default CAVersion:       cert-manager-debian-trixie-20250419+deb13u1.1-e9c7b63ee2623386
Events:
  Type    Reason  Age   From     Message
  ----    ------  ----  ----     -------
  Normal  Synced  4s    bundles  Successfully synced Bundle to namespaces that match this label selector: trust.example.com/internal-ca=true
  Normal  Synced  3s    bundles  Successfully synced Bundle to namespaces that match this label selector: trust.example.com/internal-ca=true
$ kubectl -n demo get configmap example-internal-ca -o jsonpath='{.data.ca-bundle\.pem}' | grep -c 'BEGIN CERTIFICATE'
149

The demo namespace got the example-internal-ca ConfigMap with the ca-bundle.pem key: 148 public roots from the Debian package (the chart installs its trust-pkg-debian-trixie image next to trust-manager; the version shows in Default CAVersion) plus our root. The ConfigMap is owned by the Bundle, so editing it by hand is pointless. A pod mounts it into /etc/ssl/certs/ or passes the path to the application in an environment variable, and trusts both Let's Encrypt and the internal CA alike. The selector is set explicitly: an empty selector currently means "all namespaces", but the project warns that this behaviour will change in future versions.

Annotation or an explicit Certificate

There are two ways to get a certificate.

An annotation on a Gateway or Ingress. cert-manager watches Gateways and Ingresses (this part is called ingress-shim, for Gateways — gateway-shim) and creates a Certificate from their description by itself. Let's put an annotation back on the apps gateway from the previous part:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apps
  namespace: traefik
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  # listeners as in the previous part

For every listener that has a hostname, tls.mode: Terminate and certificateRefs to a Secret in the same namespace, cert-manager creates a Certificate: the name and secretName come from certificateRefs, dnsNames from hostname. Listeners with the same Secret produce one Certificate with a merged list of names. The self-signed apps-example-com-tls created by hand will be overwritten by a Let's Encrypt certificate for *.apps.example.com, and by the issuer's selector it will go through DNS-01.

On the lab the mechanism was checked without contacting Let's Encrypt — with the internal-ca issuer:

$ kubectl -n traefik annotate gateway apps cert-manager.io/cluster-issuer=internal-ca
gateway.gateway.networking.k8s.io/apps annotated
$ kubectl -n traefik get certificate
NAME                   READY   SECRET                 AGE
apps-example-com-tls   True    apps-example-com-tls   4s
$ kubectl -n traefik get certificate apps-example-com-tls -o yaml
...
  annotations:
    acme.cert-manager.io/http01-parentrefkind: Gateway
    acme.cert-manager.io/http01-parentrefname: apps
...
  ownerReferences:
  - apiVersion: gateway.networking.k8s.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Gateway
    name: apps
...
spec:
  dnsNames:
  - '*.apps.example.com'
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: internal-ca
  secretName: apps-example-com-tls
  usages:
  - digital signature
  - key encipherment
$ kubectl -n traefik describe certificate apps-example-com-tls | sed -n '/^Events:/,$p'
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    3s    cert-manager-certificates-trigger          Issuing certificate as Secret was previously issued by "Issuer.cert-manager.io/"
  Normal  Generated  3s    cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "apps-example-com-tls-zn9pj"
  Normal  Requested  3s    cert-manager-certificates-request-manager  Created new CertificateRequest resource "apps-example-com-tls-1"
  Normal  Issuing    2s    cert-manager-certificates-issuing          The certificate has been successfully issued

The Certificate's owner is the Gateway: remove the gateway and the Certificate goes too. The trigger honestly says why it reissues: the Secret was issued by an "issuer" without a name, that is, not by cert-manager. The http01-parentref… annotations tell the HTTP-01 solver to attach its route directly to this gateway.

With an Ingress it works the same way, with the cert-manager.io/cluster-issuer (or cert-manager.io/issuer) annotation: a Certificate is created for every spec.tls item, names are taken from hosts, the Secret from secretName.

An explicit Certificate. We write the object ourselves:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hello-ingress-tls
  namespace: demo
spec:
  secretName: hello-ingress-tls
  dnsNames:
    - hello-ingress.apps.example.com
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  secretTemplate:
    labels:
      app.kubernetes.io/part-of: hello

When to choose which. An annotation is convenient when the certificate is needed exactly for this Gateway or Ingress and there is nothing special about it: no need to repeat the names twice. An explicit Certificate is for when the certificate is used by an application rather than a proxy (the pod mounts the Secret itself), when you need fields the annotations lack, when several consumers need one Secret, and when you want to see the certificate in Git as a separate object. Annotations can do a lot too (cert-manager.io/duration, cert-manager.io/private-key-algorithm and others), but all parameters in annotations are the same schema-less strings as with Ingress.

The key. If no algorithm is set, cert-manager creates an RSA 2048 key; ECDSA P-256 is shorter, faster in the handshake and supported by all modern clients. rotationPolicy: Always means a new key on every renewal; since version 1.18 it is the default, and switching it to Never makes sense only when the key is pinned somewhere and changing it would break something.

Renewal and where the key lives

cert-manager computes the renewal time itself and writes it into status.renewalTime. If nothing is set, renewal starts at two thirds of the issued certificate's lifetime: for a 90-day one — 30 days before expiry, for a 64-day one — about 21 days before, for the 160-hour shortlived — about two days before. It can be set with two fields:

renewBefore — a fixed margin ("360 hours before expiry");
renewBeforePercentage — a share of the actual lifetime ("33 % before expiry").

For ACME, duration is only a wish: Let's Encrypt ignores the requested lifetime and issues a certificate for its profile's lifetime, and cert-manager computes renewal from the actual lifetime of the issued certificate. So with public ACME it is better not to touch duration and renewBefore at all, and if you need a margin, set it as a percentage. A fixed renewBefore: 720h is still sensible for a 64-day certificate, for a 45-day one it means renewing just 15 days after issuance, and if the margin is not shorter than the whole lifetime (a 6-day certificate), cert-manager silently ignores it and falls back to two thirds — the setting stops doing what it says.

1.21 added one more way, still alpha behind the ACMEUseARI feature gate: ARI (ACME Renewal Information, RFC 9773), an ACME extension through which the server itself suggests a renewal window to the client, for example when it is about to revoke certificates en masse. With it, cert-manager picks a random time inside the suggested window; Let's Encrypt supports ARI and recommends it.

The certificate and key are stored in an ordinary Secret of type kubernetes.io/tls:

Key	What
`tls.crt`	the certificate followed by the intermediate certificates of the chain
`tls.key`	the private key in PEM
`ca.crt`	the CA certificate, if the issuer knows it (`ca` and `selfSigned` — yes, ACME — no)

cert-manager puts the annotations cert-manager.io/certificate-name, cert-manager.io/issuer-name, cert-manager.io/issuer-kind, cert-manager.io/alt-names, cert-manager.io/common-name and others on the Secret — they show where it came from — and the label controller.cert-manager.io/fao: "true", by which the controller watches only its own Secrets. Before the certificate is issued, the new key is kept in a temporary Secret with a random suffix, and only when the certificate is ready do both land in the target Secret in a single write — a consumer never sees the new key with the old certificate.

Who reads the Secret. Traefik watches Secrets and swaps the certificate on the fly, without a restart: on the lab the new certificate was served twenty seconds after issuance. An application that mounts the Secret as a volume also gets the new files — kubelet updates such volumes — but whether the process rereads them depends on the application; if not, a restart on change helps, for example the Reloader controller from part 12, which restarts pods when a Secret changes.

The key lives in etcd. Without etcd encryption at rest it is stored in base64, that is, in plain text, and anyone with get on Secrets in that namespace can read it. One more argument for keeping gateway certificates in the traefik namespace, which application teams cannot access.

HTTPS redirects and HTTP-01

A common question: does an HTTP → HTTPS redirect break HTTP-01 validation? By itself — no. Let's Encrypt follows redirects (up to ten, only to the http and https schemes and only to ports 80 and 443) and does not validate the certificate when switching to HTTPS: getting the first real certificate is the whole point of the validation. cert-manager's self-check, covered below, also follows the redirect. What breaks is something else: after the redirect the request has to reach the solver's route, and that route exists only on HTTP.

In Traefik on the lab it looks like this:

Redirect with a route — the hello-redirect Ingress with the redirectScheme middleware, or an HTTPRoute with the RequestRedirect filter from the previous part. No problem: the solver's route matches the exact path /.well-known/acme-challenge/<token> and in Traefik gets a higher priority than a rule with the / prefix. On the lab the hello-redirect Ingress for the same name stayed in place, and as soon as Traefik applied the solver's HTTPRoute, the validation requests went to the solver while everything else still went to the redirect (301 for GET).
Redirect on the entry point (ports.web.http.redirections.entryPoint in the values) — the redirect router gets priority MaxInt-1 and takes all requests on web, validation included. The request then arrives at websecure, where there is no solver route (it hangs on the http listener), and validation gets a 404. On the lab, with such a redirect, a request to a test HTTPRoute with the exact path /.well-known/acme-challenge/test-token got a 301 to HTTPS. For this case the entry point has the allowACMEByPass: true option (ports.web.allowACMEByPass in the chart values): the redirect router stops touching the /.well-known/acme-challenge/ path, and the solver route works over HTTP — with it the same request reached the HTTPRoute, while a request to / still got a 301.
Solver through an Ingress (http01.ingress) — its Ingress without Traefik annotations lands on all entry points, websecure included, so validation may succeed even after a redirect. It works, but you should not rely on it; an explicit allowACMEByPass is more reliable.

Hands-on

Let's create an explicit Certificate with the staging issuer and watch the chain unfold. Six seconds after apply:

$ kubectl apply -f hello-ingress-tls.yaml
certificate.cert-manager.io/hello-ingress-tls created
$ kubectl -n demo get certificate,certificaterequest,order,challenge
NAME                                            READY   SECRET              AGE
certificate.cert-manager.io/hello-ingress-tls   False   hello-ingress-tls   6s

NAME                                                     APPROVED   DENIED   READY   ISSUER                REQUESTER                                         AGE
certificaterequest.cert-manager.io/hello-ingress-tls-1   True                False   letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   6s

NAME                                                        STATE     AGE
order.acme.cert-manager.io/hello-ingress-tls-1-3799463221   pending   5s

NAME                                                                       STATE     DOMAIN                           AGE
challenge.acme.cert-manager.io/hello-ingress-tls-1-3799463221-2553886130   pending   hello-ingress.apps.example.com   4s

The names tell where things come from: request -1 is the first revision of the certificate, the order and challenge carry the request's name and a hash. The solver has appeared too:

$ kubectl -n demo get pods,svc,httproute -l acme.cert-manager.io/http01-solver=true
NAME                            READY   STATUS              RESTARTS   AGE
pod/cm-acme-http-solver-frw5d   0/1     ContainerCreating   0          1s

NAME                                TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/cm-acme-http-solver-4ntl5   NodePort   10.99.81.36   <none>        8089:31457/TCP   1s

NAME                                                            HOSTNAMES                            AGE
httproute.gateway.networking.k8s.io/cm-acme-http-solver-2gg8b   ["hello-ingress.apps.example.com"]   1s

The solver's HTTPRoute is an ordinary Gateway API route: parentRefs from the issuer, the name from the challenge, one type: Exact rule with the token path and backendRefs to the solver service, owned by the Challenge. Traefik accepted it (Accepted: True, ResolvedRefs: True) because the demo namespace was labelled for the gateway's allowedRoutes back in the previous part. The challenge in detail:

$ kubectl -n demo describe challenge hello-ingress-tls-1-3799463221-2553886130
...
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz/123456789/6267561333
  Dns Name:           hello-ingress.apps.example.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-staging
  Key:     FXgzGQZcP6xg….EL36boxmInjy…
  Solver:
    http01:
      Gateway HTTPRoute:
        Parent Refs:
          Group:         gateway.networking.k8s.io
          Kind:          Gateway
          Name:          apps
          Namespace:     traefik
          Section Name:  http
  Token:                 FXgzGQZcP6xg…
  Type:                  HTTP-01
  URL:                   https://acme-staging-v02.api.letsencrypt.org/acme/chall/123456789/6267561333/041Rwg
  Wildcard:              false
Status:
  Presented:     true
  Presented At:  2026-10-29T13:18:14Z
  Processing:    true
  Reason:        Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:         pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    2s    cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  0s    cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

Key is exactly the <token>.<account key thumbprint> string the solver will return (87 bytes). Reason is the self-check: before asking Let's Encrypt to validate the name, cert-manager itself requests http://hello-ingress.apps.example.com/.well-known/acme-challenge/<token> and waits for the right answer — that way it does not waste attempts from the failure limit. Ten seconds later the reason changed to wrong status code '503', and ten seconds after that the challenge passed. What was going on shows in the logs of both Traefik pods:

$ kubectl -n traefik logs traefik-6b7fb4b765-qb2wb | grep acme-challenge
172.20.211.24 - - [29/Oct/2026:13:18:14 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 301 17 "-" "-" 1 "demo-hello-redirect-hello-ingress-apps-example-com@kubernetes" "-" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:14 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 301 17 "-" "-" 2 "demo-hello-redirect-hello-ingress-apps-example-com@kubernetes" "-" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:14 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 404 19 "-" "-" 3 "demo-hello-hello-ingress-apps-example-com@kubernetes" "http://172.20.211.17:8080" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:24 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 503 20 "-" "-" 4 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "-" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:34 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 5 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:36 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 6 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
172.20.211.24 - - [29/Oct/2026:13:18:40 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 7 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
$ kubectl -n traefik logs traefik-6b7fb4b765-f972j | grep acme-challenge
...
172.20.211.24 - - [29/Oct/2026:13:18:42 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 9 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
66.133.109.36 - - [29/Oct/2026:13:18:42 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 10 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
13.60.200.158 - - [29/Oct/2026:13:18:43 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 11 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
44.252.78.39 - - [29/Oct/2026:13:18:43 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 12 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms
...
18.141.229.111 - - [29/Oct/2026:13:18:44 +0000] "GET /.well-known/acme-challenge/FXgzGQZcP6xg… HTTP/1.1" 200 87 "-" "-" 16 "httproute-demo-cm-acme-http-solver-2gg8b-gw-traefik-apps-ep-web-0-e3f853cb74c1116d3e60@kubernetesgateway" "http://172.20.211.29:8089" 0ms

Reading in order. 172.20.211.24 is the cert-manager pod, so this is the self-check from inside the cluster. At 13:18:14 Traefik had not yet applied the solver's HTTPRoute, the request hit the hello-redirect Ingress, got a 301, the self-check followed the redirect to HTTPS and got a 404 from hello itself, which has no such path. At 13:18:24 the route exists, but the solver pod is still pulling its image — a 503 with an empty backend. From 13:18:34 the solver answers 200 with 87 bytes; the self-check passed five times in a row two seconds apart, and only after that did cert-manager tell Let's Encrypt "go ahead and validate". Then the Let's Encrypt validation servers arrived: the first, 66.133.109.36, and six more from other addresses within two seconds — the multi-point validation in action. Their addresses in the log are real because the Traefik service has externalTrafficPolicy: Local; there is no User-Agent header because the Traefik chart does not write headers to the access log by default.

For the self-check to arrive at all, the cert-manager pod has to reach the gateway by the same name. On the lab the network's internal DNS servers, to which CoreDNS forwards external names (part 16), answer hello-ingress.apps.example.com with the private address 192.168.10.200; without that the pod would get the public 203.0.113.10 and go to the router from the outside, which works only if the router can "turn" such connections back into the local network (hairpin NAT). Our router cannot.

Forty-four seconds after creation the certificate is ready:

$ kubectl -n demo get certificate,certificaterequest,order,challenge
NAME                                            READY   SECRET              AGE
certificate.cert-manager.io/hello-ingress-tls   True    hello-ingress-tls   45s

NAME                                                     APPROVED   DENIED   READY   ISSUER                REQUESTER                                         AGE
certificaterequest.cert-manager.io/hello-ingress-tls-1   True                True    letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   45s

NAME                                                        STATE   AGE
order.acme.cert-manager.io/hello-ingress-tls-1-3799463221   valid   44s
$ kubectl -n demo get pods,svc,httproute -l acme.cert-manager.io/http01-solver=true
No resources found in demo namespace.

The challenge, the solver pod, the Service and the HTTPRoute are deleted after validation; the Order stays in the valid state. Certificate details:

$ kubectl -n demo describe certificate hello-ingress-tls | sed -n '/^Status:/,$p'
Status:
  Conditions:
    Last Transition Time:  2026-10-29T13:18:52Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2027-01-01T12:20:17Z
  Not Before:              2026-10-29T12:20:18Z
  Renewal Time:            2026-12-11T04:20:17Z
  Revision:                1
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    58s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  58s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "hello-ingress-tls-nnz59"
  Normal  Requested  58s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "hello-ingress-tls-1"
  Normal  Issuing    14s   cert-manager-certificates-issuing          The certificate has been successfully issued

A 64-day lifetime — staging already issues certificates on the 2027 schedule — and Renewal Time exactly at two thirds: 42 days and 16 hours after Not Before. Not Before is an hour before issuance — Let's Encrypt backdates the start to allow for clock skew. The events show the same chain: the trigger "Secret does not exist", a temporary Secret with the key, the request, issuance.

cmctl status gathers everything related in one output:

$ cmctl status certificate hello-ingress-tls -n demo
Name: hello-ingress-tls
Namespace: demo
Created at: 2026-10-29T16:18:08+03:00
Conditions:
  Ready: True, Reason: Ready, Message: Certificate is up to date and has not expired
DNS Names:
- hello-ingress.apps.example.com
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    58s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  58s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "hello-ingress-tls-nnz59"
  Normal  Requested  58s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "hello-ingress-tls-1"
  Normal  Issuing    14s   cert-manager-certificates-issuing          The certificate has been successfully issued
Issuer:
  Name: letsencrypt-staging
  Kind: ClusterIssuer
  Conditions:
    Ready: True, Reason: ACMEAccountRegistered, Message: The ACME account was registered with the ACME server
  Events:  <none>
Secret:
  Name: hello-ingress-tls
  Issuer Country: US
  Issuer Organisation: Let's Encrypt
  Issuer Common Name: (STAGING) Baloney Bulgur YE2
  Key Usage: Digital Signature
  Extended Key Usages: Server Authentication
  Public Key Algorithm: ECDSA
  Signature Algorithm: ECDSA-SHA384
  Subject Key ID: …
  Authority Key ID: deda4225f234c0279b6616cf656a90ac21d53087
  Serial Number: …
  Events:  <none>
Not Before: 2026-10-29T15:20:18+03:00
Not After: 2027-01-01T15:20:17+03:00
Renewal Time: 2026-12-11T07:20:17+03:00
No CertificateRequest found for this Certificate

cmctl shows times in the time zone of the machine it runs on. The last line is not an error: the request is already done, and cmctl shows only unfinished ones. While a certificate is being issued, the CertificateRequest, Order and Challenge with their states show up right here — the fastest way to see which link is stuck. The certificate was issued by the staging intermediate (STAGING) Baloney Bulgur YE2, and tls.crt holds four certificates: ours and the chain of the test hierarchy up to (STAGING) Bogus Broccoli X2, signed by (STAGING) Pretend Pear X1.

A forced renewal checks the whole path without waiting for renewalTime — and staging is the right place to do it:

$ cmctl renew hello-ingress-tls -n demo
Manually triggered issuance of Certificate demo/hello-ingress-tls
$ cmctl status certificate hello-ingress-tls -n demo
...
CertificateRequest:
  Name: hello-ingress-tls-2
  Namespace: demo
  Conditions:
    Approved: True, Reason: cert-manager.io, Message: Certificate request has been approved by cert-manager.io
  Ready: False, Reason: Pending, Message: Waiting on certificate issuance from order demo/hello-ingress-tls-2-3799463221: "ready"
  Events:
    Type    Reason              Age   From                                                Message
    ----    ------              ----  ----                                                -------
    Normal  WaitingForApproval  3s    cert-manager-certificaterequests-issuer-acme        Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  3s    cert-manager-certificaterequests-issuer-venafi      Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  3s    cert-manager-certificaterequests-issuer-ca          Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  3s    cert-manager-certificaterequests-issuer-selfsigned  Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  3s    cert-manager-certificaterequests-issuer-vault       Not signing CertificateRequest until it is Approved
    Normal  cert-manager.io     3s    cert-manager-certificaterequests-approver           Certificate request has been approved by cert-manager.io
    Normal  OrderCreated        3s    cert-manager-certificaterequests-issuer-acme        Created Order resource demo/hello-ingress-tls-2-3799463221
Order:
  Name: hello-ingress-tls-2-3799463221
  State: ready, Reason: 
  Authorizations:
    URL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/123456789/6267561333, Identifier: hello-ingress.apps.example.com, Initial State: valid, Wildcard: false
No Challenges found for this Certificate
$ kubectl -n demo get certificaterequest,order
NAME                                                     APPROVED   DENIED   READY   ISSUER                REQUESTER                                         AGE
certificaterequest.cert-manager.io/hello-ingress-tls-2   True                True    letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   5s

NAME                                                        STATE   AGE
order.acme.cert-manager.io/hello-ingress-tls-2-3799463221   valid   5s

Several things show here at once. Approval is a separate step: every issuer controller waits for Approved, and only after the built-in approver does the ACME controller create the Order. There were no challenges at all this time — Initial State: valid: Let's Encrypt remembers that this account recently proved control of the name and reuses the authorization (up to 30 days for the classic profile, 7 hours for the short ones), so the new certificate was issued in five seconds. Request -1 disappeared together with its Order — revisionHistoryLimit: 1 at work. And the public key in the Secret changed: rotationPolicy: Always.

Staging works — let's change issuerRef.name to letsencrypt-prod. The issuer is different, so the certificate no longer matches the desired state, and cert-manager issues a new one right away without waiting for the renewal time:

$ kubectl -n demo patch certificate hello-ingress-tls --type=merge -p '{"spec":{"issuerRef":{"name":"letsencrypt-prod"}}}'
certificate.cert-manager.io/hello-ingress-tls patched
$ kubectl -n demo get certificate,certificaterequest,order,challenge
NAME                                            READY   SECRET              AGE
certificate.cert-manager.io/hello-ingress-tls   False   hello-ingress-tls   2m35s

NAME                                                     APPROVED   DENIED   READY   ISSUER                REQUESTER                                         AGE
certificaterequest.cert-manager.io/hello-ingress-tls-2   True                True    letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   82s
certificaterequest.cert-manager.io/hello-ingress-tls-3   True                False   letsencrypt-prod      system:serviceaccount:cert-manager:cert-manager   4s

NAME                                                        STATE     AGE
order.acme.cert-manager.io/hello-ingress-tls-2-3799463221   valid     82s
order.acme.cert-manager.io/hello-ingress-tls-3-2743847525   pending   4s

NAME                                                                       STATE     DOMAIN                           AGE
challenge.acme.cert-manager.io/hello-ingress-tls-3-2743847525-2581890648   pending   hello-ingress.apps.example.com   2s

The prod account has no authorization for this name yet, so the challenge runs again, the same way. A kubectl get -w started in parallel showed the state change:

$ kubectl -n demo get certificate hello-ingress-tls -w
NAME                READY   SECRET              AGE
hello-ingress-tls   True    hello-ingress-tls   2m28s
hello-ingress-tls   True    hello-ingress-tls   2m30s
hello-ingress-tls   True    hello-ingress-tls   2m30s
hello-ingress-tls   False   hello-ingress-tls   2m31s
hello-ingress-tls   False   hello-ingress-tls   2m31s
hello-ingress-tls   False   hello-ingress-tls   3m
hello-ingress-tls   False   hello-ingress-tls   3m
hello-ingress-tls   True    hello-ingress-tls   3m
hello-ingress-tls   True    hello-ingress-tls   3m

Lines repeat because watch prints every change of the object, including changes to its status only. In the Certificate events the trigger explains the reason: Issuing certificate as Secret was previously issued by "ClusterIssuer.cert-manager.io/letsencrypt-staging". What is in the Secret:

$ kubectl -n demo get secret hello-ingress-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
    | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
subject=CN=hello-ingress.apps.example.com
issuer=C=US, O=Let's Encrypt, CN=YE1
notBefore=Oct 29 12:22:36 2026 GMT
notAfter=Jan 27 12:22:35 2027 GMT
X509v3 Subject Alternative Name: 
    DNS:hello-ingress.apps.example.com
$ kubectl -n demo get secret hello-ingress-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
    | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout
subject=CN=hello-ingress.apps.example.com
issuer=C=US, O=Let's Encrypt, CN=YE1

subject=C=US, O=Let's Encrypt, CN=YE1
issuer=C=US, O=ISRG, CN=Root YE

subject=C=US, O=ISRG, CN=Root YE
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X2

subject=C=US, O=Internet Security Research Group, CN=ISRG Root X2
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X1

Production issued 90 days — the classic profile; cert-manager scheduled renewal for December 28, 60 days later. It was signed by the YE1 intermediate of the new Let's Encrypt hierarchy generation (ECDSA, the Root YE root). Root YE is not yet in browser and OS trust stores, so the default chain is long: Root YE is cross-signed by ISRG Root X2, and X2 by ISRG Root X1, which is trusted everywhere.

Let's check from the laptop that Traefik serves exactly this certificate:

$ curl -v --resolve hello-ingress.apps.example.com:443:192.168.10.200 https://hello-ingress.apps.example.com/
...
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=hello-ingress.apps.example.com
*  start date: Oct 29 12:22:36 2026 GMT
*  expire date: Jan 27 12:22:35 2027 GMT
*  subjectAltName: host "hello-ingress.apps.example.com" matched cert's "hello-ingress.apps.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=YE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 3: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
...
< HTTP/2 200 
< content-type: text/plain; charset=utf-8
...
< x-app-version: 1.0
...
hello from hello-6d5c6db6b4-w98sn

SSL certificate verify ok — curl validated the chain against the system roots without any -k. The key is the ECDSA P-256 we ordered in privateKey. The same chain through the eyes of openssl s_client:

$ openssl s_client -connect 192.168.10.200:443 -servername hello-ingress.apps.example.com -verify_return_error </dev/null
depth=3 C = US, O = Internet Security Research Group, CN = ISRG Root X2
verify return:1
depth=2 C = US, O = ISRG, CN = Root YE
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = YE1
verify return:1
depth=0 CN = hello-ingress.apps.example.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = hello-ingress.apps.example.com
   i:C = US, O = Let's Encrypt, CN = YE1
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Oct 29 12:22:36 2026 GMT; NotAfter: Jan 27 12:22:35 2027 GMT
 1 s:C = US, O = Let's Encrypt, CN = YE1
   i:C = US, O = ISRG, CN = Root YE
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Sep  3 00:00:00 2025 GMT; NotAfter: Sep  2 23:59:59 2028 GMT
 2 s:C = US, O = ISRG, CN = Root YE
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X2
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
 3 s:C = US, O = Internet Security Research Group, CN = ISRG Root X2
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 13 00:00:00 2026 GMT; NotAfter: Sep  2 23:59:59 2032 GMT
---
...

The server sent all four certificates from tls.crt, and validation on the laptop stopped at depth=3: ISRG Root X2 is already in its trust store, so there was no need to go up to X1.

An internal certificate — for example, so that the hello pod itself accepts TLS from other services in the cluster — is issued the same way, only instantly, without an Order and a Challenge:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hello-internal-tls
  namespace: demo
spec:
  secretName: hello-internal-tls
  dnsNames:
    - hello.demo.svc
    - hello.demo.svc.cluster.local
  duration: 720h
  renewBeforePercentage: 33
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: internal-ca

Here we set the lifetime ourselves: our own CA issues as much as was asked for, and a short lifetime with automatic renewal beats a long one.

$ kubectl -n demo get certificaterequest hello-internal-tls-1
NAME                   APPROVED   DENIED   READY   ISSUER        REQUESTER                                         AGE
hello-internal-tls-1   True                True    internal-ca   system:serviceaccount:cert-manager:cert-manager   2s
$ kubectl -n demo get secret hello-internal-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject
subject=CN=example.com Internal Root CA
$ kubectl -n demo get secret hello-internal-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
    | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
subject=
issuer=CN=example.com Internal Root CA
notBefore=Oct 29 13:22:51 2026 GMT
notAfter=Nov 28 13:22:51 2026 GMT
X509v3 Subject Alternative Name: critical
    DNS:hello.demo.svc, DNS:hello.demo.svc.cluster.local

The subject field is empty: we did not set commonName, and cert-manager does not make one up — the names live only in the Subject Alternative Name extension, which is therefore marked critical, as the standard requires for a certificate without a name in the subject. Modern clients look only at the SAN. Renewal is renewalTime: 2026-11-18T15:46:51Z, at 67 % of the 30 days.

Common mistakes

Symptom	Cause	What to do
Challenge `pending`, `wrong status code '404'`	the solver route did not attach: namespace without the `allowedRoutes` label, wrong `sectionName`, a redirect on the entry point; in the first seconds — Traefik has not applied the route yet	`kubectl get httproute -n <ns> -o yaml` — check `Accepted`; namespace label; `allowACMEByPass`
`wrong status code '503'`	the route exists, but the solver pod is not ready yet (pulling its image)	wait; if it does not pass, check the `cm-acme-http-solver-…` pod
the same, but for a Certificate from a gateway annotation	the Certificate is created in the gateway's namespace (`traefik`), the solver HTTPRoute is created there too, and the `traefik` namespace is not in `allowedRoutes`	label it too, or issue the gateway certificate through DNS-01
self-check passes, Let's Encrypt does not; Order `invalid`, Challenge says `Timeout during connect` or `Connection refused`	port 80 is not reachable from the internet: no forwarding on the router, a firewall	check from an outside address; for internal services use DNS-01
DNS-01 Challenge `pending`, `DNS record ... not yet propagated`	cert-manager checks the TXT record through cluster DNS (the startup log has `configured acme dns01 nameservers ... ["10.96.0.10:53"]`), and the internal DNS serves its own copy of the `example.com` zone (split-horizon) and does not see the public record	controller flags `--dns01-recursive-nameservers-only` and `--dns01-recursive-nameservers=1.1.1.1:53,9.9.9.9:53`
`NXDOMAIN` or a wrong address in the Order	there is no public record, or it points elsewhere	check the public answer: `dig +short hello-ingress.apps.example.com @1.1.1.1`
Order `invalid`: `CAA record ... prevents issuance`	DNS has a CAA record (the list of certificate authorities allowed to issue certificates for the domain), and Let's Encrypt is not in it	add `CAA 0 issue "letsencrypt.org"`
ClusterIssuer `Ready: False`, `invalidContact ... forbidden domain`	`email` has an address in a non-existent or reserved domain (`example.com`)	set a real address or remove the field
`too many failed authorizations` / `too many certificates`	the prod limit is exhausted after a series of errors	debug on staging, wait for the limit window
a DNS-01 Challenge does not create the record and reports that the `cloudflare-api-token` Secret was not found	for a ClusterIssuer the Secret was created outside the `cert-manager` namespace; issuer readiness does not catch this — it depends only on account registration	move the Secret
certificates for a Gateway are not created	Gateway API is not enabled (the log says `skipping disabled controller ... gateway-shim`) or the CRDs were installed after cert-manager started	`--enable-gateway-api`, restart cert-manager

Sources

cert-manager: Installation with kubectl and with Helm — the release manifest, the chart, crds.enabled; Supported Releases — releases and support periods.
cert-manager: cmctl — check api, status certificate, renew; cert-manager/cmctl — the utility's releases.
cert-manager: Issuer, CertificateRequest and ACME Orders and Challenges — the chain of objects, approval, revisions.
cert-manager: Certificate resource — renewBefore, renewBeforePercentage, rotationPolicy, revisionHistoryLimit, secretTemplate, the temporary Secret.
cert-manager: ACME — privateKeySecretRef, email, solvers and selectors, profile; HTTP01 — the ingress and gatewayHTTPRoute solvers; DNS01 — providers, cnameStrategy, self-check nameservers.
cert-manager: SelfSigned and CA — the internal certificate authority.
cert-manager: Annotated Gateway resource and Annotated Ingress resource — gateway-shim and ingress-shim, listener rules, enabling Gateway API.
cert-manager: trust-manager — Bundle, the public roots package, the warning about the root Secret.
cert-manager: Troubleshooting Problems with ACME — diagnosing Orders and Challenges.
Let's Encrypt: Staging Environment and Rate Limits — environments and limits.
Let's Encrypt: Challenge Types — HTTP-01, DNS-01, redirects; Certificate Authority Authorization (CAA); Certificate Transparency (CT) Logs.
Let's Encrypt: Profiles and Decreasing Certificate Lifetimes to 45 Days — profiles, lifetimes and the reduction schedule.
Let's Encrypt: Chains of Trust — roots and intermediates, Root YE and YE1, cross-signatures.
Let's Encrypt: Expiration Notification Service Has Ended — the end of expiration emails.
RFC 8555: Automatic Certificate Management Environment (ACME) — the protocol, the HTTP-01 and DNS-01 challenges; RFC 9773: ACME Renewal Information (ARI) Extension; RFC 2136: Dynamic Updates in the Domain Name System.
Kubernetes: Secrets — TLS Secrets, Ingress — TLS, Gateway API, Dynamic Admission Control and Encrypting Confidential Data at Rest.
Gateway API: HTTPRoute — path match types and their precedence.
Traefik: EntryPoints — entry point redirects and allowACMEByPass.

Now every site on *.apps.example.com can have a real certificate that renews itself, and internal services have their own certificate authority and a way to trust it. Traffic from outside to the pods is protected, but inside the cluster any pod can still talk to any other.

Next — NetworkPolicy: how to restrict who can talk to whom inside the cluster, what breaks after a default deny, and how Calico turns policies into rules on the nodes.