28 October 2026, 13:53Виталий Лещенко

Kubernetes for beginners. Part 9: How do Job and CronJob work?

All parts of the series “Kubernetes for beginners”

The Deployment from part 7 and the StatefulSet and DaemonSet from the previous part solve one problem: keeping processes running all the time. If a container exits, it is restarted — even with code 0, because for these controllers the pod's restartPolicy can only be Always. But many tasks must run once and finish: a database schema migration before a new release, a nightly dump, recalculating a report, processing a batch of files. They need a controller that understands what "completed successfully" means, retries on failure but not forever, and can split the work across several pods. That is a Job. And a CronJob creates such Jobs on a schedule — like cron on a server, only with retries, history and no ties to a particular machine.

Job: a pod that must finish

A minimal Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: once
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: once
        image: busybox:1.37
        command: ["sh", "-c", "echo start; sleep 3; echo done"]

Inside is an ordinary pod template from part 6, with one restriction: restartPolicy is only Never or OnFailure. The API rejects Always (spec.template.spec.restartPolicy: Required value: valid values: "OnFailure", "Never"): a pod that is restarted after a successful exit never finishes, and the Job could never record a success.

What happens after kubectl apply. A Job is an object of the batch/v1 group, served by the Job controller in kube-controller-manager. The controller sees a Job without pods and creates a pod from the template, adding to it:

the labels batch.kubernetes.io/job-name: once and batch.kubernetes.io/controller-uid: <Job uid> — the Job finds its pods by them. The Job's selector is generated automatically from controller-uid, so that two Jobs with the same labels in their templates do not mix up each other's pods; you can set it by hand only with manualSelector: true, and you almost never need to;
an owner reference (ownerReferences) — when the Job is deleted, the garbage collector deletes the pods too;
the finalizer batch.kubernetes.io/job-tracking. A finalizer is an entry in metadata.finalizers that keeps an object from disappearing from the API until whoever set it removes it. The controller removes it only once it has counted the pod in the Job status as succeeded or failed. Without it, a pod deleted by someone before the controller saw it would drop out of the count.

Then the kubelet runs the pod as usual, the container exits, and the pod moves to the Succeeded or Failed phase. The controller increments status.succeeded or status.failed and decides what to do next: create another pod, retry or finish. The state of the Job is recorded in conditions in status.conditions:

Condition	When it appears
`SuccessCriteriaMet`	the success criteria are met, the remaining pods are being stopped
`FailureTarget`	the Job has been judged failed, the remaining pods are being stopped
`Complete`	final success: all pods are already stopped
`Failed`	final failure: all pods are already stopped
`Suspended`	the Job is suspended (`suspend: true`)

The intermediate conditions exist because time passes between "decided to finish" and "all pods have really stopped" — a pod's terminationGracePeriodSeconds may well be a minute. Since 1.31 Complete and Failed are set only after the last pod has stopped, and whoever waits for the Job (a release script, for example) can rely on nothing else running any more. The STATUS column of kubectl get job shows the latest of the conditions, so the intermediate SuccessCriteriaMet and FailureTarget are visible there too.

After it finishes, the Job and its pods stay in the API: you can look at the status and read the logs. You remove them either by hand or with the ttlSecondsAfterFinished field (more on it below). A Job is not a replacement for a Deployment: if a process must run permanently and you wrap it in a Job with OnFailure, it will be restarted until it exhausts the limit, and in the end the Job fails.

completions and parallelism

Two fields define how much work there is and how much of it runs in parallel:

completions — how many successful completions are needed for the whole Job to succeed;
parallelism — how many pods may run at the same time (1 by default; 0 effectively pauses the Job).

Their combinations give three scenarios:

 1. A single task             completions: 1 (or unset), parallelism: 1
    [pod] ──success──► Complete

 2. A fixed count             completions: 6, parallelism: 2
    [pod][pod] → [pod][pod] → [pod][pod]       6 successes → Complete

 3. A work queue              completions unset, parallelism: 3
    [pod][pod][pod]  each takes tasks from an external queue itself;
                     first success → no new pods, wait for the rest → Complete

The third scenario fits when the tasks sit in an external queue (a message broker, a table in a database): the pods decide by themselves that the queue is empty and exit with code 0. After the first successful exit the controller creates no new pods and considers the Job done when all the others have stopped. This requires care from the application: a successful exit means "there is no more work", and the other pods must come to the same conclusion.

Indexed Job

In the second scenario all six pods are the same and do not know which part of the work to do. The completionMode: Indexed mode (stable since 1.24) gives each pod a number from 0 to completions - 1:

in the JOB_COMPLETION_INDEX environment variable;
in the batch.kubernetes.io/job-completion-index annotation and label;
in the pod name (shards-2-pk6ns) and in the pod's hostname: <Job name>-<number>.

The Job is considered done when there is one successful pod for every number. If the pod with number 3 failed, its replacement gets the same number 3. This is how static partitioning of work is done without an external queue: the pod with number i processes the i-th part of a file, the i-th key range, the i-th customer. With a headless Service from the previous part the pods of an indexed Job can also find each other by name — that is how distributed computations where each participant has its own number are run.

Indexed Jobs also have conveniences ordinary ones lack:

backoffLimitPerIndex and maxFailedIndexes (stable since 1.33) — a retry limit for each number separately. Without them a single persistently failing number exhausts the shared retry limit and fails the whole Job, even though the other 99 parts would have worked. With them the failed number is recorded in status.failedIndexes, the others continue, and in the end the Job is marked failed (Failed with the reason FailedIndexes), but with a clear list of what exactly to redo: on the stand a Job of four numbers with backoffLimitPerIndex: 1, where number 2 always fails, finished with completedIndexes: 0,1,3 and failedIndexes: "2" after two attempts at that number. maxFailedIndexes limits how many numbers may fail before the Job is aborted as a whole.
Elasticity (stable since 1.31): for a running indexed Job you can change completions together with parallelism, as long as they are equal. The API will not let you change completions alone (spec.completions: Invalid value: 2: can only be modified in tandem with spec.parallelism). When scaling down, the pods with the highest numbers are deleted.

Failures: backoffLimit and restartPolicy

backoffLimit (6 by default) — how many failures are tolerated before the Job is marked Failed with the reason BackoffLimitExceeded. Retries come with an exponential delay: 10 seconds, 20, 40 and so on, but no more than ten minutes. The delay keeps a job that fails because the database is unavailable from hammering it every second.

How failures are counted depends on the restartPolicy in the template, and the difference is significant:

restartPolicy: Never. The failed pod stays in the Failed phase, and the Job controller creates a new pod. A failure = a failed pod. Every attempt is a separate pod with its own logs, and after the Job fails they are all there for analysis: kubectl logs fail-rs22z.

restartPolicy: OnFailure. The pod stays the same, and the kubelet restarts the container inside it — with the same exponential delay you saw as CrashLoopBackOff. Each container restart counts as a failure. When the limit is exhausted, the controller deletes the pod (the event SuccessfulDelete … Deleted pod), and the logs of the last attempt go with it — the documentation warns explicitly that this gets in the way of analysis. The upside of OnFailure: a restart in the same pod is faster, there is no need to schedule and pull the image again, and the contents of emptyDir are kept.

Recommendation: Never by default. The logs of all attempts are kept, failures are counted clearly, and only with Never does the podFailurePolicy from the next section work. OnFailure is for jobs where a fast restart matters more than analysis and the logs go to a log collection system anyway (part 33).

One more detail: at what moment to create the replacement. By default (podReplacementPolicy: TerminatingOrFailed) the controller creates a new pod as soon as the old one starts stopping — so for a while both run. A deleted pod counts as a failure here: on the stand, after kubectl delete pod such a Job immediately had active: 1, failed: 1, terminating: 1 in its status. For jobs that must not run twice at once (they hold a lock, write to one file) there is podReplacementPolicy: Failed (stable since 1.34): the replacement appears only after the old pod has finally moved to Failed. The number of stopping pods is visible in status.terminating.

podFailurePolicy: not all failures are equal

backoffLimit counts all failures the same way. But the causes differ:

a bug in the code or the data — retrying is pointless, however many times you try the result is the same; it is better to fail the Job at once and not spend six attempts and tens of minutes of delays between them;
an infrastructure failure — a node is taken down for maintenance (kubectl drain), the pod was preempted by a higher-priority one, a node disappeared. The job is not to blame, and such failures are better not counted against the limit.

That is what podFailurePolicy (stable since 1.31) is for — an ordered list of rules checked for every failed pod:

spec:
  backoffLimit: 3
  podFailurePolicy:
    rules:
    - action: FailJob            # code 42 = "the data is broken", retrying is pointless
      onExitCodes:
        containerName: worker
        operator: In
        values: [42]
    - action: Ignore             # the pod was evicted — do not count it as an attempt
      onPodConditions:
      - type: DisruptionTarget
  template:
    spec:
      restartPolicy: Never       # required for podFailurePolicy
      containers:
      - name: worker
        # ...

Rule actions:

Action	What it does
`FailJob`	fail the whole Job at once and stop the other pods
`Ignore`	do not increment the failure counter, just create a replacement
`Count`	normal handling: count it against `backoffLimit`
`FailIndex`	for indexed Jobs with `backoffLimitPerIndex`: fail only this number, without retries

The onExitCodes requirement looks at container exit codes (In or NotIn), onPodConditions at pod conditions. The kubelet and controllers set DisruptionTarget on a pod that is deleted for an external reason: preemption, API-initiated eviction (that is how drain works; the condition's reason is EvictionByEvictionAPI), removal from a node with a NoExecute taint, garbage collection on a vanished node. The rules are checked in order and the first matching one applies; if none matches, the failure is counted as usual. On the stand a Job with backoffLimit: 0 and an Ignore rule survived the eviction of its pod: the failed counter did not grow, and the controller created the replacement once the evicted pod was gone.

Requirements: restartPolicy: Never in the template (otherwise only "Never" is supported when podFailurePolicy is specified), and with a failure policy podReplacementPolicy can only be Failed — the controller must see the pod's final phase and its exit codes before deciding. If the field is not set, the API fills in Failed itself, and rejects TerminatingOrFailed with the error Unsupported value: "TerminatingOrFailed": supported values: "Failed".

For the application this leads to a simple rule: agree on exit codes. Code 0 is success, a temporary error (the database is unavailable) is an ordinary non-zero code worth retrying, a permanent error (wrong parameters, broken input) is a separate code that FailJob is set up for.

successPolicy: when success comes earlier

Normally an indexed Job succeeds when all numbers have run. Sometimes part of them is enough: out of five calculation variants you need one that converged; in a "leader and workers" scheme only the leader with number 0 determines the result. That is what successPolicy (stable since 1.33, only for completionMode: Indexed) is for:

spec:
  completionMode: Indexed
  completions: 5
  parallelism: 5
  successPolicy:
    rules:
    - succeededIndexes: "0"      # success as soon as number 0 has succeeded

A rule has two fields that can be combined: succeededIndexes — a list of numbers and ranges ("0,2-3"), succeededCount — how many of them must succeed. A rule with only succeededCount: 3 means "any three numbers". When the rule is satisfied, the Job gets the SuccessCriteriaMet condition with the reason SuccessPolicy, the controller stops the remaining pods and, once they have stopped, sets Complete. If a failure criterion (backoffLimit, podFailurePolicy) is met at the same time, the failure wins.

Time and cleanup: activeDeadlineSeconds, ttlSecondsAfterFinished, suspend

activeDeadlineSeconds in the Job spec is the maximum duration of the whole Job, with all retries. When it runs out, the controller stops the running pods and marks the Job Failed with the reason DeadlineExceeded (Job was active longer than specified deadline), even if backoffLimit is not yet exhausted. Do not confuse it with the field of the same name in the pod spec: that one limits the life of a single pod, and the Job does not take it into account. For any scheduled job a Job-level limit is always worth setting: a hung dump must not run until the next start.

ttlSecondsAfterFinished — how many seconds after it finishes (successfully or not) the Job is deleted together with its pods. A separate TTL controller in kube-controller-manager takes care of it. The documentation recommends setting this field on all Jobs created directly (not through a CronJob): finished objects pile up in etcd and load the API server. Besides, such Jobs have orphanDependents as their default deletion policy: if you delete a Job through the API without asking for cascading deletion, its pods stay (kubectl delete asks for them to be deleted by itself). A value of 0 means delete right after finishing; a value long enough to read the logs is more sensible: an hour or a day.

suspend: true suspends the Job: running pods are deleted, new ones are not created, and the Job gets the Suspended condition with the reason JobSuspended. To resume it, set it back to false: a Resumed event appears, and status.startTime gets a new value, so the activeDeadlineSeconds countdown (and the DURATION column) starts over. This is how external queueing systems (for example, Kueue — a batch job dispatcher for Kubernetes that distributes quotas between teams) create Jobs suspended and start them when resources have been found for them. In a suspended Job you can change the placement constraints in the template, and since 1.36, in beta, the container resources too.

And about the sidecar containers from part 6. They used to be a problem for Jobs: a proxy or a log shipper in the pod never exits on its own, and a pod with one never reached Succeeded. Native sidecar containers (an init container with restartPolicy: Always, stable since 1.33) solve this: the kubelet stops them when the main containers have finished, and the Job gets a successful pod.

CronJob: Jobs on a schedule

A CronJob is a Job template plus a schedule:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: tick
spec:
  schedule: "*/1 * * * *"
  timeZone: Etc/UTC
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: tick
            image: busybox:1.37
            command: ["sh", "-c", "date -u; echo tick from $(hostname)"]

It is served by the CronJob controller in kube-controller-manager. It does not keep a timer per schedule: for each CronJob it computes the next start time, puts the object in a queue for that time, and when the time comes, creates a Job from jobTemplate and records the time in status.lastScheduleTime, and after the Job finishes successfully, in status.lastSuccessfulTime. From then on the Job lives its own life; the CronJob only keeps track of the history.

Schedule format

Five fields, as in classic cron:

 ┌───────────── minute (0–59)
 │ ┌─────────── hour (0–23)
 │ │ ┌───────── day of month (1–31)
 │ │ │ ┌─────── month (1–12)
 │ │ │ │ ┌───── day of week (0–6, 0 = Sunday; or sun–sat)
 │ │ │ │ │
 30 1 * * *      every day at 01:30
 0 */4 * * *     every four hours, exactly on the hour
 15 3 * * 1-5    at 03:15 on weekdays
 0 0 1 * *       at midnight on the first of the month

Ranges (1-5), lists (1,15), steps (*/4, 0-23/2), ? as a synonym for * and the macros @hourly, @daily (also @midnight), @weekly, @monthly, @yearly (also @annually) all work. There is no seconds field: the minimum interval is a minute, and the API rejects a six-field schedule (expected exactly 5 fields, found 6).

Time zone. Without the timeZone field the schedule is interpreted in the time zone of the kube-controller-manager process. With kubeadm it runs in a static pod container where the node's /etc/localtime is not mounted, so in practice that is UTC — a surprise for whoever wrote 0 2 * * * meaning two in the morning local time. On the stand the nodes live in the +03 zone, and a CronJob without timeZone and with the schedule 38 7 * * * started at 07:38 UTC. The timeZone field (stable since 1.27) takes a name from the IANA time zone database: Etc/UTC, Europe/Berlin, Asia/Tokyo; the API does not let a typo through (unknown time zone Europe/Minskk). The database is built into the Kubernetes binaries, so it does not depend on the node. You cannot put the time zone into the schedule (CRON_TZ=Europe/Berlin 0 2 * * *): the API rejects such a CronJob with the error cannot use TZ or CRON_TZ in schedule, use timeZone field instead.

A local time zone has one trap: on the days of the switch to and from daylight saving time one hour of the day does not exist or repeats. A job scheduled for that hour may not start or may shift. Important jobs are either scheduled in UTC or put at a time the switches do not touch.

Names, history and manual runs

A Job created on schedule is named <CronJob name>-<start time in minutes since the Unix epoch>: tick-29886232. The CronJob name is limited to 52 characters (must be no more than 52 characters): today the suffix takes nine characters — a hyphen and eight digits — but the documentation reserves 11 for it, and a Job name cannot be longer than 63. The scheduled time is written to the batch.kubernetes.io/cronjob-scheduled-timestamp annotation in the RFC 3339 date-time format (since 1.32).

The CronJob deletes finished Jobs itself, keeping successfulJobsHistoryLimit successful ones (3 by default) and failedJobsHistoryLimit failed ones (1 by default). Failed ones are worth keeping longer — at least 3 — so that there is something to investigate in the morning.

To run a job outside the schedule — for example, to check a new CronJob without waiting for the night:

$ kubectl create job --from=cronjob/pg-backup pg-backup-manual-1
job.batch/pg-backup-manual-1 created

kubectl takes jobTemplate from the CronJob and creates a Job with the annotation cronjob.kubernetes.io/instantiate: manual and a reference to the CronJob in ownerReferences. This does not shift the schedule. Because of the owner reference the manual Job becomes part of the CronJob's history: the controller writes an UnexpectedJob warning about it ("Saw a job that the controller did not create or forgot"), counts it in successfulJobsHistoryLimit and deletes it together with the CronJob.

concurrencyPolicy: what if the previous run is still going

Value	Behaviour
`Allow` (default)	a new Job is created even if the previous one is running; they run in parallel
`Forbid`	the new run is postponed until the previous one finishes
`Replace`	the previous Job is deleted and a new one is started in its place

The policy applies only to the Jobs of one CronJob. Backups, migrations and anything that holds locks need Forbid: two dumps of the same database at once at best double its load and at worst corrupt the file. Replace suits jobs where only the freshest result matters and an old run is no longer relevant.

Forbid has a subtlety: a run that falls on a running Job is not lost right away. The controller writes the JobAlreadyActive event ("Not starting job because prior execution is running and concurrency policy is Forbid"), and when the previous Job finishes, it starts the postponed one late — if the delay fits into startingDeadlineSeconds. On the stand a CronJob running every minute with a 90-second task started the Job for 07:38 at 07:38:49, right after the previous one finished, and skipped the 07:39 run. Without startingDeadlineSeconds the delay is unbounded, and after a long dump the next one may start right away, in the middle of the day.

startingDeadlineSeconds and missed runs

The controller may fail to create a Job on time: kube-controller-manager was restarting, the control plane node was under maintenance, with Forbid the previous run was still going. startingDeadlineSeconds says how late a run still makes sense: with a value of 600, a Job that could not be created within ten minutes of the scheduled time is skipped and counted as failed. Without this field the delay is unbounded.

However many runs were missed, the controller does not catch up on them one by one: it creates at most one Job — for the latest missed time. Along the way it counts how many runs were missed since the last scheduled time (or, if the field is set, only within the last startingDeadlineSeconds seconds), and if there are more than 100, it writes the TooManyMissedTimes event: too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew. For a CronJob running every minute this happens after 100 minutes of downtime. The documentation says that in this case the Job is not started, but in 1.36 it is only a warning: on the stand a CronJob running every minute whose last run was three hours ago got this event and still created a Job for the current minute. With startingDeadlineSeconds the counting window is short, and there is no warning. A value under 10 seconds is not a good idea: the controller checks schedules every 10 seconds and may miss the window entirely.

suspend

suspend: true on a CronJob stops the creation of new Jobs; those already started keep running. It is handy during database maintenance: kubectl patch cronjob pg-backup -p '{"spec":{"suspend":true}}'. While the CronJob is suspended, runs pile up as missed, and after it is resumed the controller immediately creates a Job for the latest of them, if startingDeadlineSeconds allows it.

Jobs must be idempotent

The documentation says it plainly: a CronJob creates a Job approximately once per scheduled time. In rare cases — a controller failure at the moment of creation, a retry — a Job may be created twice or not at all. Add to that the Job's own retries under backoffLimit: a dump that failed halfway will be started again. So the job must be idempotent: running it again with the same input gives the same result and breaks nothing.

Practical techniques:

name the result after the run, not after the current time. A repeated run of the same schedule overwrites the same file instead of creating a second one;
write to a temporary file and rename atomically. A process that fails halfway leaves a .tmp behind, not a truncated file with the right name that will later be taken for a whole one;
check "already done" at the start: if the result for this run exists and is whole, exit with code 0;
a lock in an external system, if the job changes shared data: an advisory lock in the database or a Lease object in the API — the same thing Kubernetes components use to agree which of them is currently the leader;
for migrations, a tool that keeps its own log of applied steps and does not apply them again.

Example: a scheduled PostgreSQL dump

Let us put it all together. The cluster runs a training PostgreSQL — a relational DBMS — with the Service pg on port 5432; the password is in the Secret pg-credentials (how to store and pass secrets properly is in part 12, how to run databases in the cluster is in parts 25 and 28). The dumps go to a volume from the claim pg-backups: in part 23 it will be provisioned on an NFS server, and until then the pod will wait for it in Pending.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-backup
spec:
  schedule: "30 1 * * *"              # 01:30 — DST switches do not touch it
  timeZone: Europe/Berlin
  concurrencyPolicy: Forbid           # no need for two dumps at once
  startingDeadlineSeconds: 1800       # more than 30 minutes late — skip
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600     # no more than an hour with all retries
      podFailurePolicy:
        rules:
        - action: Ignore              # eviction from a node does not count as an attempt
          onPodConditions:
          - type: DisruptionTarget
      template:
        spec:
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 70             # the postgres user in the alpine image
            runAsGroup: 70
            fsGroup: 70
          containers:
          - name: dump
            image: postgres:18.6-alpine
            env:
            - name: PGHOST
              value: pg
            - name: PGUSER
              value: app
            - name: PGDATABASE
              value: app
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg-credentials
                  key: password
            - name: JOB_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['batch.kubernetes.io/job-name']
            command:
            - sh
            - -ec
            - |
              out="/backups/${PGDATABASE}-${JOB_NAME}.dump"
              if [ -s "$out" ]; then echo "already done: $out"; exit 0; fi
              pg_dump --format=custom --file="$out.tmp"
              pg_restore --list "$out.tmp" > /dev/null    # check that the archive is readable
              mv "$out.tmp" "$out"
              find /backups -name '*.dump' -mtime +14 -delete
              echo "ok: $out $(du -h "$out" | cut -f1)"
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
              limits:
                memory: 512Mi
            volumeMounts:
            - name: backups
              mountPath: /backups
          volumes:
          - name: backups
            persistentVolumeClaim:
              claimName: pg-backups

What is done here for reliability:

the file name comes from the Job name, which contains the scheduled time: the run on 29 October 2026 at 01:30 Berlin time writes app-pg-backup-29887230.dump. The name gets into the container through the Downward API — the mechanism by which a pod receives information about itself (labels, name, resources) in variables or files. A repeat of the same run finds the finished file and exits, a manual run pg-backup-manual-1 creates a separate file;
the dump is written to .tmp, checked with pg_restore --list and only then renamed; mv within one directory is atomic;
sh -e aborts the script on the first error, and the pod exits with a non-zero code instead of reporting success after a failed pg_dump;
Forbid, activeDeadlineSeconds, startingDeadlineSeconds — a dump will not overlap a dump, will not hang for days and will not start in the middle of the day after a night of downtime;
the password comes from a Secret through secretKeyRef, not from the manifest text; PGPASSWORD is a variable all PostgreSQL utilities read;
the process is not root and has a memory limit; fsGroup: 70 gives the volume directory to group 70, and the postgres user can write to it;
three failed Jobs are kept, so there is something to read in the morning.

With a temporary database and a temporary volume the example works like this: the first manual run prints ok: /backups/app-pg-backup-manual-1.dump 4.0K, and a repeated run of a Job with the same name prints already done: /backups/app-pg-backup-manual-1.dump.

What this example does not do and what real operation needs on top: a copy of the dumps outside the cluster and outside the same storage (a backup on the same NFS as the database will not survive the loss of the NFS) and regular restore tests — a dump nobody has ever restored from is not yet a backup. And an alert that the last successful dump is older than a day: a CronJob that quietly stopped working is a common story. The kube_cronjob_status_last_successful_time metric from kube-state-metrics (an exporter of metrics about the state of Kubernetes objects) and an alerting rule in the Prometheus monitoring system cover this; more on them in part 32.

Hands-on

Let us start with an indexed Job. Four pieces of work, two at a time:

apiVersion: batch/v1
kind: Job
metadata:
  name: shards
spec:
  completionMode: Indexed
  completions: 4
  parallelism: 2
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: busybox:1.37
        command: ["sh", "-c", "echo shard $JOB_COMPLETION_INDEX of 4 on $(hostname); sleep 5"]

$ kubectl apply -f shards.yaml
job.batch/shards created

$ kubectl get pods -l batch.kubernetes.io/job-name=shards -w
NAME             READY   STATUS    RESTARTS   AGE
shards-0-plmbf   0/1     Pending   0          0s
shards-1-6589g   0/1     Pending   0          0s
shards-0-plmbf   0/1     Pending   0          0s
shards-1-6589g   0/1     Pending   0          0s
shards-0-plmbf   0/1     ContainerCreating   0          0s
shards-1-6589g   0/1     ContainerCreating   0          0s
shards-0-plmbf   0/1     ContainerCreating   0          1s
shards-1-6589g   0/1     ContainerCreating   0          2s
shards-0-plmbf   0/1     ContainerCreating   0          3s
shards-1-6589g   0/1     ContainerCreating   0          4s
shards-0-plmbf   1/1     Running             0          4s
shards-1-6589g   1/1     Running             0          5s
shards-0-plmbf   0/1     Completed           0          10s
shards-1-6589g   0/1     Completed           0          11s
shards-0-plmbf   0/1     Completed           0          12s
shards-1-6589g   0/1     Completed           0          12s
shards-0-plmbf   0/1     Completed           0          13s
shards-1-6589g   0/1     Completed           0          13s
shards-2-pk6ns   0/1     Pending             0          0s
shards-2-pk6ns   0/1     Pending             0          0s
shards-3-hwx4w   0/1     Pending             0          0s
shards-3-hwx4w   0/1     Pending             0          0s
shards-2-pk6ns   0/1     ContainerCreating   0          0s
shards-0-plmbf   0/1     Completed           0          15s
shards-1-6589g   0/1     Completed           0          15s
shards-3-hwx4w   0/1     ContainerCreating   0          1s
shards-2-pk6ns   0/1     ContainerCreating   0          2s
shards-3-hwx4w   0/1     ContainerCreating   0          2s
shards-2-pk6ns   0/1     ContainerCreating   0          3s
shards-3-hwx4w   0/1     ContainerCreating   0          4s
shards-2-pk6ns   1/1     Running             0          5s
shards-3-hwx4w   1/1     Running             0          6s
shards-2-pk6ns   0/1     Completed           0          11s
shards-3-hwx4w   0/1     Completed           0          12s
shards-3-hwx4w   0/1     Completed           0          13s
shards-2-pk6ns   0/1     Completed           0          13s
shards-3-hwx4w   0/1     Completed           0          14s
shards-3-hwx4w   0/1     Completed           0          14s
shards-2-pk6ns   0/1     Completed           0          14s

$ kubectl logs shards-2-pk6ns
shard 2 of 4 on shards-2

kubectl get -w prints a line for every change of the object, so identical lines come in a row: after Completed the pod's status and finalizers change, and each such change is one more line. The number is in the pod name and in the hostname. While the pod is running, the finalizer and the Job labels are visible on it:

$ kubectl get pod shards-1-6589g -o jsonpath='{.metadata.finalizers}{"\n"}{.metadata.labels}{"\n"}'
["batch.kubernetes.io/job-tracking"]
{"batch.kubernetes.io/controller-uid":"24fe1b9f-84b8-4ba6-907b-100e61078139","batch.kubernetes.io/job-completion-index":"1","batch.kubernetes.io/job-name":"shards","controller-uid":"24fe1b9f-84b8-4ba6-907b-100e61078139","job-name":"shards"}

The labels without the batch.kubernetes.io/ prefix are old names the controller sets for compatibility. The Job status after it finishes:

$ kubectl get job shards
NAME     STATUS     COMPLETIONS   DURATION   AGE
shards   Complete   4/4           29s        32s

$ kubectl get job shards -o yaml | sed -n '/^status:/,$p'
status:
  completedIndexes: 0-3
  completionTime: "2026-10-28T07:30:31Z"
  conditions:
  - lastProbeTime: "2026-10-28T07:30:31Z"
    lastTransitionTime: "2026-10-28T07:30:31Z"
    message: Reached expected number of succeeded pods
    reason: CompletionsReached
    status: "True"
    type: SuccessCriteriaMet
  - lastProbeTime: "2026-10-28T07:30:31Z"
    lastTransitionTime: "2026-10-28T07:30:31Z"
    message: Reached expected number of succeeded pods
    reason: CompletionsReached
    status: "True"
    type: Complete
  ready: 0
  startTime: "2026-10-28T07:30:02Z"
  succeeded: 4
  terminating: 0
  uncountedTerminatedPods: {}

uncountedTerminatedPods are pods that have already finished but are not yet counted: those are the ones the controller removes the finalizer from after counting.

Now a failure. A Job that always fails, with backoffLimit: 2:

apiVersion: batch/v1
kind: Job
metadata:
  name: fail
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: busybox:1.37
        command: ["sh", "-c", "echo trying; exit 1"]

$ kubectl apply -f fail.yaml
job.batch/fail created

$ kubectl get pods -l batch.kubernetes.io/job-name=fail
NAME         READY   STATUS   RESTARTS   AGE
fail-6xrgr   0/1     Error    0          39s
fail-hlzgt   0/1     Error    0          61s
fail-rs22z   0/1     Error    0          75s

$ kubectl describe job fail | sed -n '/^Events:/,$p'
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Normal   SuccessfulCreate      75s   job-controller  Created pod: fail-rs22z
  Normal   SuccessfulCreate      61s   job-controller  Created pod: fail-hlzgt
  Normal   SuccessfulCreate      39s   job-controller  Created pod: fail-6xrgr
  Warning  BackoffLimitExceeded  32s   job-controller  Job has reached the specified backoff limit

With restartPolicy: Never every attempt is a separate pod: the first one and two retries. The pauses are counted from the failure of the previous pod: the container runs for about three seconds, then there is a 10-second wait before the second attempt and 20 before the third. All three pods are still there, and you can read the logs of each. The Job status, as with success, has two conditions: FailureTarget and Failed, both with the reason BackoffLimitExceeded.

The failure policy. The Job datacheck is the same manifest, but the container exits with code 42, and the spec has the podFailurePolicy from the section above:

$ kubectl get job datacheck -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
FailureTarget	PodFailurePolicy	Container worker for pod default/datacheck-h7pnp failed with exit code 42 matching FailJob rule at index 0
Failed	PodFailurePolicy	Container worker for pod default/datacheck-h7pnp failed with exit code 42 matching FailJob rule at index 0

$ kubectl get pods -l batch.kubernetes.io/job-name=datacheck
NAME              READY   STATUS   RESTARTS   AGE
datacheck-h7pnp   0/1     Error    0          20s

One attempt instead of four: the rule fired on the very first failure.

The intermediate conditions are best seen with successPolicy. The Job leader: five numbers, number 0 works for five seconds, the others for ten minutes:

apiVersion: batch/v1
kind: Job
metadata:
  name: leader
spec:
  completionMode: Indexed
  completions: 5
  parallelism: 5
  successPolicy:
    rules:
    - succeededIndexes: "0"
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: busybox:1.37
        command: ["sh", "-c", "if [ $JOB_COMPLETION_INDEX -eq 0 ]; then sleep 5; else sleep 600; fi"]

$ kubectl get job leader
NAME     STATUS               COMPLETIONS   DURATION   AGE
leader   SuccessCriteriaMet   1/5           46s        46s

$ kubectl get pods -l batch.kubernetes.io/job-name=leader
NAME             READY   STATUS        RESTARTS   AGE
leader-0-92xjf   0/1     Completed     0          46s
leader-1-76jp9   1/1     Terminating   0          46s
leader-2-l9r54   1/1     Terminating   0          46s
leader-3-8jrs7   1/1     Terminating   0          46s
leader-4-268qv   1/1     Terminating   0          46s

$ kubectl get job leader -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.reason}{"\t"}{.lastTransitionTime}{"\n"}{end}'
SuccessCriteriaMet	SuccessPolicy	2026-10-28T07:34:41Z
Complete	SuccessPolicy	2026-10-28T07:35:18Z

Number 0 finished, the rule is satisfied, and the controller stops the other pods. sh as PID 1 does not handle SIGTERM, so the pods live until the end of terminationGracePeriodSeconds (30 seconds by default), and Complete appears 37 seconds after SuccessCriteriaMet, when the last one has stopped. We will not need this Job any more:

$ kubectl delete job leader
job.batch "leader" deleted from default namespace

Now the CronJob tick from the CronJob section, once a minute:

$ kubectl apply -f tick.yaml
cronjob.batch/tick created

$ kubectl get cronjob tick
NAME   SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
tick   */1 * * * *   Etc/UTC    False     0        55s             3m5s

$ kubectl get jobs
NAME            STATUS     COMPLETIONS   DURATION   AGE
datacheck       Failed     0/1           3m20s      3m20s
fail            Failed     0/1           4m30s      4m30s
shards          Complete   4/4           30s        16m
tick-29886241   Complete   1/1           7s         2m55s
tick-29886242   Complete   1/1           7s         115s
tick-29886243   Complete   1/1           7s         55s

$ kubectl get job tick-29886243 -o jsonpath='{.metadata.annotations}{"\n"}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
{"batch.kubernetes.io/cronjob-scheduled-timestamp":"2026-10-28T08:03:00Z"}
CronJob/tick

$ kubectl logs job/tick-29886243
Wed Oct 28 08:03:03 UTC 2026
tick from tick-29886243-vscfd

29886243 minutes after 1 January 1970 is 28 October 2026, 08:03 UTC, which is what the annotation says. For the failed fail and datacheck the DURATION column grows along with AGE: completionTime is set only on a successful Job, and kubectl counts the duration up to now. Five minutes later the list will still have three Jobs from tick: old ones are deleted under successfulJobsHistoryLimit. A manual run and suspension:

$ kubectl create job --from=cronjob/tick tick-manual
job.batch/tick-manual created

$ kubectl get job tick-manual -o jsonpath='{.metadata.annotations}{"\n"}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
{"cronjob.kubernetes.io/instantiate":"manual"}
CronJob/tick

$ kubectl patch cronjob tick -p '{"spec":{"suspend":true}}'
cronjob.batch/tick patched

$ kubectl get cronjob tick
NAME   SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
tick   */1 * * * *   Etc/UTC    True      0        81s             3m31s

$ kubectl get jobs
NAME            STATUS     COMPLETIONS   DURATION   AGE
datacheck       Failed     0/1           3m46s      3m46s
fail            Failed     0/1           4m56s      4m56s
shards          Complete   4/4           30s        16m
tick-29886242   Complete   1/1           7s         2m21s
tick-29886243   Complete   1/1           7s         81s
tick-manual     Complete   1/1           7s         25s

$ kubectl describe cronjob tick | sed -n '/^Events:/,$p'
Events:
  Type     Reason            Age                From                Message
  ----     ------            ----               ----                -------
  Normal   SuccessfulCreate  3m21s              cronjob-controller  Created job tick-29886241
  Normal   SawCompletedJob   3m14s              cronjob-controller  Saw completed job: tick-29886241, condition: Complete
  Normal   SuccessfulCreate  2m21s              cronjob-controller  Created job tick-29886242
  Normal   SawCompletedJob   2m14s              cronjob-controller  Saw completed job: tick-29886242, condition: Complete
  Normal   SuccessfulCreate  81s                cronjob-controller  Created job tick-29886243
  Normal   SawCompletedJob   74s                cronjob-controller  Saw completed job: tick-29886243, condition: Complete
  Warning  UnexpectedJob     18s (x5 over 25s)  cronjob-controller  Saw a job that the controller did not create or forgot: tick-manual
  Normal   SuccessfulDelete  18s                cronjob-controller  Deleted job tick-29886241

The manual tick-manual belongs to the CronJob, so it got into the history: there are three successful Jobs again, and the controller deleted the oldest one, tick-29886241, although after the suspension there were no new scheduled runs.

And finally, where all this lives. Jobs and CronJobs are stored in etcd — the store where the API server keeps all cluster objects — under the keys /registry/jobs/<namespace>/<name> and /registry/cronjobs/<namespace>/<name>, like any other objects (how to read etcd directly is in part 34). These controllers have no state "outside the API": restarting kube-controller-manager loses nothing, the new instance reads the Jobs, the pods and their finalizers and carries on from the same place. Let us delete the training objects:

$ kubectl delete cronjob tick
cronjob.batch "tick" deleted from default namespace
$ kubectl delete job fail datacheck
job.batch "fail" deleted from default namespace
job.batch "datacheck" deleted from default namespace

The garbage collector deletes the Jobs created by the CronJob, including the manual tick-manual, together with it: their owner is the CronJob. The TTL controller will delete the Job shards an hour after it finished.

The examples in this and the previous part had containers with resources holding requests and limits, but what those numbers mean has not been explained yet.

Sources

Kubernetes: Job (completions and parallelism modes, Indexed Job, backoffLimit and delays, backoffLimitPerIndex, podFailurePolicy, successPolicy, Job conditions, suspend, elastic indexed Jobs, podReplacementPolicy, the job-tracking finalizer), CronJob (schedule format, timeZone, concurrencyPolicy, startingDeadlineSeconds, history, limitations), automatic cleanup for finished Jobs, handling retriable and non-retriable pod failures with a pod failure policy, an indexed Job for parallel processing with static work assignment, running automated tasks with a CronJob.
API reference: Job, CronJob; well-known labels and annotations (batch.kubernetes.io/job-name, job-completion-index, cronjob-scheduled-timestamp), kubectl create job, feature gates (MutablePodResourcesForSuspendedJobs).
Objects and pods: finalizers, owners and dependents, garbage collection, pod disruption conditions (DisruptionTarget), sidecar containers, Downward API, Leases.
Controller source code in the 1.36 branch: the Job controller (the delay cap MaxJobPodFailureBackOff is ten minutes), counting missed CronJob runs.
Kueue, the IANA time zone database, CronJob metrics in kube-state-metrics.
PostgreSQL: pg_dump, pg_restore, libpq environment variables (PGHOST, PGPASSWORD), the postgres image.

Next — requests and limits: how the scheduler uses requests to decide whether a pod fits on a node, what limits turn into in cgroup v2 on the node, what CPU throttling and OOMKilled are, how QoS classes decide who gets evicted first when memory runs short, and what LimitRange and ResourceQuota do.