The discount is real; the engineering is the price

Every major cloud sells the same GPUs twice. Once at the on-demand rate, with an implicit promise that the machine stays yours until you give it back. And once at a large discount under a name like spot, preemptible, interruptible or low-priority, with the explicit condition that the provider can take it back whenever it wants the capacity for someone paying full price. The silicon is identical. The interconnect is identical. The only difference is who holds the option to end the lease.

As of August 2026, the discount you will actually observe on GPU-class instances usually lands in a 40 to 70 per cent band against the equivalent on-demand rate, and providers advertise headline savings that reach higher on particular shapes. That is a band rather than a quote, and it is worth being pedantic about why: the number moves with the provider, the region, the GPU class and the hour of the day. A discount you measured in AWS Mumbai last quarter tells you very little about the same accelerator in London this quarter.

The catch is a single sentence, and everything else in this article follows from it: your job can be killed with very little notice, at any point, with no appeal. Not "might be slower". Not "could be rescheduled". Terminated, with the local disk destroyed alongside it. If a job cannot survive that, the discount is not a discount — it is a way to lose a week of GPU-hours and still get the invoice.

Which is the honest framing. Spot capacity is not a billing setting you flip on; it is a design constraint you build around, and the build costs real engineering time. The consolation is that the harness is generic: once one job resumes cleanly from an arbitrary kill, every subsequent job on your platform inherits the capability. The rest of this guide is that harness — what to put on spot, what your eviction notice really buys you, how often to checkpoint and why, what "resumable" has to contain, and how to keep a run moving when your preferred capacity evaporates.

Which workloads belong on interruptible capacity

The single most useful question to ask about any job is: if this were killed at a random moment, how much work would be lost and how hard is it to pick up again? Jobs where the answer is "a few minutes, automatically" belong on spot. Jobs where the answer is "a person has to look at it" do not, no matter how attractive the rate looks on the pricing page.

WorkloadInterruption toleranceVerdictWhy
LoRA / QLoRA fine-tuning High Strong fit Adapter state is tens to hundreds of MB, so checkpoints are cheap and cadence can be aggressive
Full fine-tune with frequent checkpoints Moderate Good, with care Works, but multi-hundred-GB optimiser state makes checkpoint cost the binding constraint
Hyperparameter sweeps Very high Excellent fit Each trial is independent; losing one trial costs one trial, provided trial state is persisted
Batch inference and embedding generation Very high Excellent fit Item-level idempotency means resume is just "skip what is already written"
Evaluation and benchmark runs High Good fit Per-example results are independently persistable; no gradient state to reconstruct
Data preprocessing and tokenisation Very high Excellent fit Shard-level checkpointing is trivial and the work is usually CPU-bound anyway
Interactive or serving traffic None Do not A request in flight when the node dies is a failed request your user sees
Anything under a hard SLA None Do not You cannot commit to availability on capacity someone else can reclaim
Serving fleet with a spot overflow tier Partial, by design Nuanced — workable On-demand baseline carries the SLA; spot absorbs peaks and is allowed to vanish

That last row is where most of the disagreement lives. A serving fleet can use interruptible capacity, but only as a tier that is architecturally permitted to disappear. Size your on-demand or reserved baseline to the traffic you have committed to serving, then let an autoscaler add spot nodes above it; when evictions arrive, the fleet degrades to the baseline rather than to nothing. This only works if your health checks, connection draining and queue-depth behaviour are honest about eviction, and if the plan has been tested against the overflow tier vanishing during a peak rather than a quiet Tuesday — a scenario that belongs in your load-testing and capacity-planning suite rather than in a design document.

Avoid

Putting a latency-SLA serving path on pure spot capacity. Evictions are not independent — the provider reclaims capacity because demand rose, which means several of your nodes can go at once, precisely when traffic is highest. A fleet that is 100 per cent interruptible has no floor, and the day it discovers that will be the day it matters most.

Batch inference deserves a mention because it is the most under-used win on the table. If you are generating embeddings for a corpus or running a large offline classification pass, the job is a stream of independent items with natural idempotency: write results keyed by input hash, skip anything already written on resume, and eviction costs you the handful of items in flight. If your workload is asynchronous by nature, also check whether a provider's own asynchronous batch API gets you a comparable discount with none of the harness — sometimes the cheapest interruptible GPU is the one you never have to operate.

Know your eviction contract

Before designing anything, get precise about what your provider actually promises. The notice period and its delivery mechanism are the two facts that shape every downstream decision, and they differ enough between providers that a harness built for one will silently fail on another.

Provider classDocumented noticeHow it reaches your processDesign implication
AWS EC2 Spot Two-minute interruption notice Instance metadata (spot/instance-action) and an EventBridge interruption warning; you poll or subscribe No signal arrives by default — you must poll IMDS or wire the event yourself
GCP Spot VMs / preemptible VMs Roughly 30 seconds' shutdown notice ACPI G2 soft-off to the guest, which triggers your shutdown script and SIGTERM via init You get a signal for free, but only about half a minute of it
Azure Spot VMs Short eviction notice (documented as 30 seconds) Scheduled Events on the instance metadata endpoint, event type Preempt Poll Scheduled Events; also choose deallocate versus delete deliberately
Kubernetes on any of the above Inherits the underlying notice A node-termination handler cordons and drains, sending SIGTERM to pods Your pod grace period must be shorter than the provider's notice, not longer
Serverless / managed GPU platforms Varies; often none exposed Platform-specific, sometimes nothing at all Design as if there is zero notice until the vendor documents otherwise

These are the documented notice windows as of August 2026, and they are worth verifying against the primary sources rather than a blog: AWS spot instance interruption notices, Google Cloud Spot VMs and Azure Spot Virtual Machines. One legacy detail is still worth knowing: GCP's older preemptible VMs carry a 24-hour maximum runtime, whereas Spot VMs do not, so a long training run on the legacy shape gets evicted on a timer regardless of demand.

Now the sentence that reorganises most people's mental model. Thirty seconds is not enough time to write a multi-hundred-gigabyte checkpoint. Neither, for most realistic state sizes and storage throughput, is two minutes. This means the eviction handler is not your safety mechanism. Your safety mechanism is the checkpoint you already wrote, at a cadence you chose, before anyone told you anything. The notice handler is a bonus that occasionally saves you a few extra minutes of progress on small state; treating it as the plan is the most common structural mistake in this whole area.

The other operational fact worth internalising is that interruption rates vary enormously — by GPU class, by region, by instance family and over time. A shape that is stable for days in one region can be reclaimed within the hour in another, and the popular accelerators are reclaimed more aggressively than the unfashionable ones precisely because everyone wants them. Providers publish availability advisories and interruption-frequency data for exactly this reason, and checking them before committing a training plan regularly changes the plan. I am deliberately not quoting a percentage: any figure printed here would be stale before you read it, and the useful skill is re-measuring for your own account and region.

Watch out

Do not infer your interruption rate from one weekend of observation. Reclamation is driven by other people's demand, which is seasonal, regional and correlated with events you cannot see. A region that looked stable during Diwali week may behave very differently in the middle of a European quarter-end. Instrument the rate continuously and treat it as a moving input, not a constant.

Checkpoint cadence: the actual maths

Most guidance on this topic converges on "save every ten to thirty minutes" without explaining where the number comes from, which is unhelpful the moment your checkpoint is unusually large or your interruption rate is unusually high. The derivation is short enough to do here, and once you have it you can re-derive the interval for any job you own.

There are exactly two costs pulling in opposite directions. Checkpointing itself costs time: if a checkpoint takes T minutes to write and you write one every I minutes, you spend (60/I) × T minutes per hour of wall-clock on writing rather than on gradients. Eviction costs time too: if you are evicted, you lose on average half an interval of work, plus a restart cost R covering acquiring a replacement node, pulling the image and reloading state. With p evictions per hour, the expected overhead per hour of training is:

overhead_minutes_per_hour  =  (60 / I) * T  +  p * (I / 2 + R)

  I  = checkpoint interval, minutes
  T  = time to write one checkpoint, minutes
  p  = observed evictions per hour (e.g. 0.15 = 15% chance per hour)
  R  = restart cost: re-provision + image pull + state reload, minutes

# Differentiating with respect to I and setting to zero gives the optimum.
# R drops out — it is a fixed cost per eviction, not a function of cadence.

  I_optimal  =  sqrt(120 * T / p)

Work an example. Take a full fine-tune whose sharded state — weights, optimiser moments and master copies — comes to roughly 90 GB, written to object storage at an aggregate 1 GB/s across ranks, so T is about 1.5 minutes. Assume a restart cost R of six minutes, which is realistic once you count capacity acquisition. The table below runs the numbers at a quiet 2 per cent per hour and a busy 15 per cent per hour.

IntervalCheckpoint cost / hrLost work at 2%/hrTotal at 2%/hrLost work at 15%/hrTotal at 15%/hr
5 min18.0 min0.2 min18.2 min1.3 min19.3 min
15 min6.0 min0.3 min6.3 min2.0 min8.0 min
30 min3.0 min0.4 min3.4 min3.2 min6.2 min
60 min1.5 min0.7 min2.2 min5.4 min6.9 min
120 min0.75 min1.3 min2.1 min9.9 min10.7 min

Two things fall out. At a busy 15 per cent per hour the total is minimised around the 30-minute mark, and the closed form agrees: the square root of (120 × 1.5 / 0.15) is roughly 35 minutes. At a quiet 2 per cent per hour the same formula gives roughly 95 minutes, and the table shows the curve is nearly flat from an hour onward — checkpointing every five minutes on a stable region is pure waste, costing 18 minutes an hour to insure against a risk of well under one minute an hour.

Now run it for a LoRA adapter. Adapter weights plus optimiser state plus the loader position might be 400 MB, writing in about ten seconds, so T is 0.17 minutes. At 15 per cent per hour the optimum is the square root of (120 × 0.17 / 0.15), which is about 12 minutes; at 2 per cent per hour it is about 32 minutes. That is where the familiar ten-to-thirty-minute rule comes from — it is the right answer for small checkpoints, and quoting it at a job with 90 GB of state is how teams end up spending a third of their GPU time on serialisation.

The practical corollary is genuinely important for cost work: cheap checkpoints are the lever, not aggressive cadence. Parameter-efficient fine-tuning makes checkpoints two to three orders of magnitude smaller, which collapses the overhead curve almost to the floor and makes an eviction a shrug rather than an incident. If you are choosing between full fine-tuning and PEFT on cost grounds, interruption resilience belongs in the comparison alongside memory and quality — the eval-driven LoRA and QLoRA recipe covers the quality side, and our walkthrough of fine-tuning on a budget covers the arithmetic. For full fine-tunes, attack T directly: shard the checkpoint across ranks so each writes its own slice in parallel, and overlap the upload with the next training step so the GPU is not idle while bytes move.

Pro tip

Set your interval by wall-clock time, not by step count. Step duration drifts with sequence length, gradient accumulation and node performance, so "every 500 steps" means twenty minutes on one run and ninety on another. A timer gives you a bounded worst-case loss that you can actually reason about, which is the whole point of the exercise.

What "resumable" actually means

This is where most homegrown harnesses fail, and they fail quietly. A file containing model weights is not a resumable checkpoint. Reload weights alone and your optimiser restarts from zero momentum, your schedule restarts from the warm-up, your data loader restarts from the top of the dataset, and your run silently becomes a different experiment — one whose loss curve has a discontinuity someone will spend two days investigating.

A checkpoint that genuinely resumes contains, written together as one atomic unit:

  • Model weights — the obvious part, and the only part most people save.
  • Optimiser state — momentum and variance buffers. For Adam-family optimisers this is typically larger than the weights themselves.
  • Learning-rate scheduler state — so warm-up and decay continue rather than restart.
  • RNG states — Python's random, NumPy, the framework's CPU generator and every CUDA device generator. Without these, dropout masks and augmentation diverge from the original trajectory.
  • Data loader position — which shard, which sample within it, which epoch, and the shuffle seed. This is the single most commonly omitted field and the most damaging, because re-reading the same examples quietly changes your effective epoch count.
  • Global step and token count — the authority for schedules, logging and stopping conditions.
  • Metric history — so early stopping, best-checkpoint tracking and your dashboards survive the restart rather than starting a new line.
  • Provenance — code revision, config hash, base model identifier. Cheap to write, and the difference between debugging in ten minutes and archaeology.

The other half of correctness is atomicity. A checkpoint write is not instantaneous, and an eviction lands wherever it lands — including halfway through your largest file. If your resume logic picks "the newest file in the directory", the newest file after a badly timed eviction is a truncated one. Write to a temporary path, flush, fsync so the bytes are actually on the device rather than in the page cache, then rename into place: rename within a filesystem is atomic, so the checkpoint either exists completely or not at all. Do the equivalent for object storage by uploading to a temporary key and only writing the "latest" pointer once the upload has completed.

And keep the last N checkpoints, not just one. Three is a reasonable default. Rotation costs a little storage and buys you a fallback when the newest checkpoint is bad for a reason unrelated to truncation — a NaN loss, a corrupted shard, a bug you shipped an hour ago.

Watch out

The failure that costs the most is "we resumed from a half-written checkpoint". The job restarts, the file loads without an obvious error because the truncation landed inside a tensor blob rather than the header, training continues from subtly corrupted state, and nobody notices until the eval run days later. Atomic temp-then-rename plus a checksum written alongside the file eliminates an entire class of very expensive mystery.

A resumable training loop

Here is the shape of a checkpoint save and load pair that satisfies the list above. It is deliberately plain PyTorch rather than a framework abstraction, because the point is to make the required fields visible; every serious training framework has an equivalent, and the value of reading this version is knowing what to check for in theirs.

import glob, os, random
import numpy as np
import torch

CKPT_DIR = "/mnt/scratch/checkpoints"   # local NVMe: fast, and dies with the node
KEEP_LAST = 3


def save_checkpoint(step, model, optimizer, scheduler, sampler_state, metrics):
    state = {
        "global_step": step,
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),      # usually the largest piece
        "scheduler": scheduler.state_dict(),
        "rng": {
            "python": random.getstate(),
            "numpy": np.random.get_state(),
            "torch": torch.get_rng_state(),
            "cuda": torch.cuda.get_rng_state_all()
                    if torch.cuda.is_available() else None,
        },
        "data": sampler_state,      # {"shard": 12, "offset": 4096, "epoch": 1, "seed": 7}
        "metrics": metrics,
        "code_rev": os.environ.get("GIT_SHA", "unknown"),
    }

    os.makedirs(CKPT_DIR, exist_ok=True)
    final = os.path.join(CKPT_DIR, f"step-{step:09d}.pt")
    tmp = final + ".tmp"

    with open(tmp, "wb") as f:
        torch.save(state, f)
        f.flush()
        os.fsync(f.fileno())     # bytes on the device, not just in the page cache
    os.replace(tmp, final)       # atomic within one filesystem

    # Durable copy. Local disk is a write buffer; object storage is the truth.
    upload_to_object_store(final, key=f"runs/{RUN_ID}/{os.path.basename(final)}")
    write_latest_pointer(RUN_ID, os.path.basename(final))   # only after upload succeeds

    for stale in sorted(glob.glob(os.path.join(CKPT_DIR, "step-*.pt")))[:-KEEP_LAST]:
        os.remove(stale)


def load_checkpoint(path, model, optimizer, scheduler):
    # weights_only=False is required: RNG states are Python tuples and NumPy
    # objects, not tensors. Only ever load checkpoints you produced yourself.
    state = torch.load(path, map_location="cpu", weights_only=False)

    model.load_state_dict(state["model"])
    optimizer.load_state_dict(state["optimizer"])
    scheduler.load_state_dict(state["scheduler"])

    random.setstate(state["rng"]["python"])
    np.random.set_state(state["rng"]["numpy"])
    torch.set_rng_state(state["rng"]["torch"])
    if state["rng"]["cuda"] is not None and torch.cuda.is_available():
        torch.cuda.set_rng_state_all(state["rng"]["cuda"])

    return state["global_step"], state["data"], state["metrics"]

Note the ordering in save_checkpoint: the "latest" pointer is written last, after the upload has returned successfully. That ordering is what makes resume safe — a job that dies mid-upload leaves an orphaned object and a pointer still aimed at the previous good checkpoint. Resume should read the pointer, fetch that object and verify its checksum before touching it, never list the bucket and take whichever key sorts highest.

The second snippet is the eviction handler. It is worth building, and it is worth being clear-eyed about what it buys you.

import signal, threading

_evicting = threading.Event()
_stop = threading.Event()


def _on_sigterm(signum, frame):
    # Keep the handler trivial: set a flag. Never do I/O inside a signal handler.
    _evicting.set()


signal.signal(signal.SIGTERM, _on_sigterm)   # GCP and Kubernetes deliver this


def poll_ec2_interruption(interval=5):
    """AWS does NOT send a signal. The two-minute notice lives in IMDS."""
    import urllib.request

    def _open(url, headers, method="GET"):
        req = urllib.request.Request(url, headers=headers, method=method)
        return urllib.request.urlopen(req, timeout=2)

    while not _stop.is_set():
        try:
            token = _open(
                "http://169.254.169.254/latest/api/token",
                {"X-aws-ec2-metadata-token-ttl-seconds": "60"},
                method="PUT",
            ).read()
            resp = _open(
                "http://169.254.169.254/latest/meta-data/spot/instance-action",
                {"X-aws-ec2-metadata-token": token},
            )
            if resp.getcode() == 200:      # 404 is the normal, healthy case
                _evicting.set()
                return
        except Exception:
            pass
        _stop.wait(interval)


threading.Thread(target=poll_ec2_interruption, daemon=True).start()

# --- inside the training loop -------------------------------------------------
for batch in loader:
    train_step(batch)
    step += 1

    if _evicting.is_set():
        # Best effort ONLY. On a ~30s notice this completes for an adapter-sized
        # checkpoint. It will not complete for 90 GB of sharded optimizer state.
        # The checkpoint that saves you is the one written on the timer, earlier.
        save_checkpoint(step, model, optimizer, scheduler, loader.state(), metrics)
        raise SystemExit(75)         # distinct code (EX_TEMPFAIL): evicted, not crashed,
                                     # and not finished - the orchestrator reschedules

    if minutes_since_last_checkpoint() >= CHECKPOINT_INTERVAL_MIN:
        save_checkpoint(step, model, optimizer, scheduler, loader.state(), metrics)

Two details that matter in production. Exit cleanly with a distinguishable status rather than letting the process be killed, so your orchestrator can tell an eviction apart from a crash and apply a different retry policy to each. And if you run under Kubernetes, set terminationGracePeriodSeconds shorter than the provider's notice window — a 120-second grace period on a 30-second notice means the platform kills your pod before your handler has finished, which is worse than having no handler at all.

Every article here is written by a Verified Builder. Want your name on the next one?

AI Tech Connect lists AI engineers, founders and researchers across India and the UK — and the people hiring browse it to find them. Adding your profile is free.

Become a Verified Builder →

Orchestration: the fallback ladder

A resumable job still needs somewhere to resume. The pattern that works is a ladder of capacity preferences, walked in order, with a bounded amount of waiting at each rung and an explicit backstop at the bottom.

# capacity-ladder.yaml — walked top to bottom, per attempt
run: sft-support-assistant-v4
budget:
  max_on_demand_hours: 6        # hard ceiling on the expensive backstop
  max_total_cost_usd: 900       # kill switch, not a suggestion

ladder:
  - tier: preferred-spot
    gpu_class: A                # your preferred accelerator
    region: ap-south-1          # AWS Mumbai
    market: spot
    wait_for_capacity: 10m

  - tier: alt-region-spot
    gpu_class: A
    region: eu-west-2           # AWS London
    market: spot
    wait_for_capacity: 10m
    requires: [checkpoints_replicated, dataset_readable]

  - tier: alt-class-spot
    gpu_class: B                # slower, but available; expect longer wall-clock
    region: ap-south-1
    market: spot
    wait_for_capacity: 20m

  - tier: on-demand-backstop
    gpu_class: A
    region: ap-south-1
    market: on-demand
    guard: budget.max_on_demand_hours

The rungs are, in order: preferred GPU class in your preferred region on spot; the same class in an alternate region on spot; a different but acceptable class; and finally on-demand as a backstop with a hard cost ceiling. AWS Mumbai (ap-south-1) and London (eu-west-2) are the natural pair for a team operating across India and the UK, and a useful one because their demand peaks are offset by several hours — capacity pressure in one often coincides with quiet in the other.

The constraint that catches people is data gravity. Cross-region failover only works if your checkpoints and your dataset are readable from both regions, which means replication configured in advance rather than a frantic copy while a job sits idle. It also means egress, which is not free. A team in Manchester failing a run over to Mumbai pays for the transfer, and if the training data contains personal data there may be a residency obligation attached to that movement that has nothing to do with cost — a decision that belongs in your DPDP and GDPR routing design before it becomes an incident. Treat any run whose data cannot legally leave a region as pinned to that region's ladder.

Recommended

Diversify the instance shapes you will accept within each rung rather than pinning to exactly one. Requesting several compatible shapes across multiple availability zones materially improves the odds of being placed and reduces the chance that a single capacity crunch takes out your whole ladder. The cost is that your job must tolerate a slightly different node than it started on, which a properly resumable job already does.

Every scheduler expresses this differently. A Kubernetes-based stack encodes it as node pools, taints, tolerations and priority classes; a managed training service exposes it as a capacity or instance-type preference list; a serverless GPU platform mostly hides it and gives you a queue instead. The ladder is the same in all three — what changes is where you write it down. If you are choosing a platform layer rather than building one, the trade-offs are covered in the guides to serverless GPU platforms and autoscaling LLM inference on Kubernetes rather than re-derived here.

A worked savings model

Numbers make the decision concrete. The model below is illustrative and uses a round on-demand rate of $3.00 per GPU-hour with a spot rate of $1.20, a 60 per cent discount sitting mid-band. Per-hour rates move constantly and vary by provider, region and GPU class — substitute your own quoted rates before making a decision. Assume a programme running 3,000 GPU-hours a month across sweeps, fine-tunes and evaluation runs, at an observed 10 per cent eviction rate per hour with a 30-minute checkpoint cadence.

Line itemOn-demandSpot with harnessNote
3,000 GPU-hours of compute$9,000$3,600$3.00 vs $1.20 per GPU-hour, illustrative
Re-run waste from evictions$0+$220~6% wall-clock overhead at the modelled cadence and rate
Checkpoint storage~$10+$80Rotation of 3 checkpoints across concurrent runs
Cross-region egress$0+$150Occasional failover between Mumbai and London
Idle attached storage$0+$60Volumes still billed while jobs wait for capacity
Harness maintenance$0+$240~4 engineer-hours a month once built
Monthly total (steady state)$9,010$4,350Net saving ~$4,660 a month, ~52%
One-off build cost~$2,400~40 engineer-hours to build the harness once

At this volume the harness pays for itself inside the first month and then saves roughly half the compute line indefinitely. Notice how much of the headline 60 per cent discount is eaten on the way down: the sticker saving is $5,400 and the realised saving about $4,660, so roughly 14 per cent of the discount goes to making interruption survivable. A good trade, and not a free one.

Now run the same model at 100 GPU-hours a month — a team doing one fine-tune a quarter. On-demand is $300. Spot compute is $120, plus perhaps $8 of re-run waste, $20 of storage and the same $240 of monthly maintenance, giving $388. The harness makes the small programme more expensive, before you have even paid the $2,400 to build it. The break-even in this model sits somewhere around 150 GPU-hours a month, and below that the correct engineering decision is to pay the on-demand rate and spend the time elsewhere.

Pro tip

Tag every spot job with the run identifier, the team and the experiment, and reconcile realised spend against the model monthly. The gap between modelled and realised cost is where capacity chasing, forgotten volumes and thrashing runs hide. The same tagging discipline that powers per-feature cost attribution and showback works unchanged for training spend.

Pitfalls that cost real money

These are the recurring ways teams lose the discount they came for. Each one is a design flaw rather than a knowledge gap — every team that hit them knew, in the abstract, that spot instances get reclaimed.

PitfallHow it shows upThe fix
Checkpointing to local disk only Eviction destroys the instance store; the checkpoint dies with it Local write as a buffer, immediate replication to object storage, resume reads only from object storage
No checkpoint rotation The only checkpoint is bad, and there is nothing to fall back to Keep the last three, plus a checksum, plus a pointer written after upload
Capacity chasing across regions Egress and duplicate storage quietly exceed the discount Bound failover to a small number of pre-replicated regions; make egress a line item
Idle attached storage Volumes billed for days while a job waits in the ladder Ephemeral volumes for scratch; automatic reaping of orphaned volumes
Sweep trials without persisted state Eviction restarts the sweep and re-runs completed trials Persist trial results and status to a durable store; resume skips completed trials
Quoting the spot rate as the budget Wall-clock inflates, deadlines slip, the finance conversation goes badly Budget at spot rate plus modelled overhead; report realised rate, not quoted rate
Grace period longer than the notice The platform kills the pod before the emergency checkpoint finishes Set the pod grace period below the provider's documented notice window

The sweep pitfall deserves emphasis because it is so easy to miss. A sweep looks like the perfect spot workload — independent trials, natural parallelism — right up until the coordinator itself is evicted and restarts with no memory of which trials finished. You then pay to re-run work you already completed, and the sweep appears to take twice as long for no visible reason. Persist trial state to the same durable store as your checkpoints, keyed by configuration hash, and make the coordinator's first action on start-up a reconciliation pass.

Pro tip

Instrument eviction rate as a first-class metric, broken down by region, GPU class and instance shape, and chart it next to job wall-clock. Without it you cannot tell a bad region from a bad job — a run that keeps dying might be hitting a capacity-starved zone, or it might be running out of memory and blaming the cloud. One dashboard settles that argument permanently, and it is the input to the cadence formula above.

Where to start

If you are starting from nothing, do these in order. First, pick one non-urgent job — an evaluation run or an embedding pass is ideal — and make it resumable end to end, including the data loader position. Second, add atomic temp-then-rename writes with a checksum and three-deep rotation. Third, replicate checkpoints to object storage and make resume read exclusively from there. Fourth, add the SIGTERM handler and, on AWS, the IMDS poller, knowing they are a bonus rather than the plan. Fifth, measure your actual eviction rate and checkpoint write time for a fortnight, then set the interval from the formula rather than from a blog. Sixth, write the capacity ladder down as configuration, with an on-demand backstop and a hard budget ceiling. Only then start moving real training volume across.

None of this is exotic, and that is rather the point. Interruption resilience is ordinary, careful platform engineering, and it is disproportionately valuable because so few teams have actually built it. A working checkpoint-resume harness sits alongside decisions like whether to self-host or use an API and how to squeeze throughput out of your own serving stack — unglamorous work that quietly determines whether an AI programme is affordable.

It is also excellent proof-of-work. "Cut training spend by half" is a claim anyone can make; a public repository containing a resumable training harness, a cadence model with the maths shown, and an eviction-rate dashboard is a claim that verifies itself. If you have built one, put it on a Builder profile where the people hiring for platform and infrastructure roles across Bengaluru, Chennai, London and Manchester can actually find it.