Serving a Trillion-Parameter MoE on Your Own Hardware

What you need to know

• Three questions decide everything. Can you hold the weights? Route to experts fast enough? Keep the GPUs busy? Fail one and the deployment is not slow, it is impossible.
• Total parameters set memory; active parameters set compute. It is why a model activating 41 billion parameters still demands terabytes.
• Quantisation is the admission ticket. Kimi K3's 2.8 trillion parameters are roughly 1.4 TB at MXFP4 against roughly 5.6 TB at 16-bit, both before KV cache.
• A QAT release is a different risk from one you compress yourself. K3 is quantisation-aware trained from the SFT stage; a bf16 model you quantise post hoc leaves the quality cost for you to find.
• MoE pushes you towards expert parallelism, and interconnect is the hidden constraint. All-to-all on every layer means the network between your GPUs matters as much as the GPUs.
• Compute break-even, do not assert it — and check it against a capacity ceiling. Most teams reading this should not self-host a 1T MoE.

The three questions

A dense model is easy to reason about: one parameter count drives memory and compute together. Mixture-of-experts breaks that link deliberately. Kimi K3, released by Moonshot AI at 00:00 UTC on 27 July 2026 on Hugging Face under moonshotai, holds 2.8 trillion total parameters and fires roughly 16 of 896 experts per token — about 50 billion parameters of live compute per step. Kimi K2 held 1 trillion total with 32 billion activated; Moonshot reports roughly a 2.5× improvement in overall scaling efficiency for K3 over K2, a first-party claim. Inkling, from Thinking Machines Lab on 15 July 2026 under Apache 2.0, holds 975 billion total with 41 billion active.

Compute scales with the active count; memory scales with the total count, because every expert must be reachable within microseconds even if it fires for one token in fifty. Hence the three gates, in the order they will kill you:

1. Can you hold the weights? Total parameters times bytes per parameter, plus KV cache and activation memory, must fit in aggregate GPU memory — or in a tiered arrangement keeping the hot fraction resident.
2. Can you route to experts fast enough? Every MoE layer performs an all-to-all — tokens go to whichever GPU owns their chosen experts, results come back — on every layer, every token, prefill and decode.
3. Can you keep the GPUs busy? Sparse activation means a GPU idles on any token that does not route to its experts. Without careful batching and placement you can buy a terabyte of accelerator memory and run it far below capability.

This is what practitioners mean by hardware-software co-design across levels of parallelism, scheduling and kernels: no single knob saves you.

Prerequisites

Assemble six things first. Teams that skip this list find the gap mid-migration.

• A GPU inventory with honest usable-memory figures — what remains after driver, context and framework overhead.
• Your interconnect topology, written down — which GPUs share a fast intra-node fabric, which sit on PCIe, what the inter-node link is.
• CPU RAM and pinned-memory headroom, if offloading is even possible. Pinned memory is neither free nor swappable.
• Fast local NVMe for the weight set, or every restart and autoscale event becomes an outage.
• A throughput target and a p95 latency budget, in writing. Half the decisions below trade one against the other.
• A quality evaluation you already trust, or you cannot tell a regression from noise. Our vLLM production playbook covers the baseline this guide assumes you run at smaller scale.

Sizing the weight set: the arithmetic

Start with the number that decides most of these projects. The same Kimi K3 checkpoint is roughly 1.4 TB at MXFP4 and roughly 5.6 TB at 16-bit — both before any KV cache or context loads. That fourfold swing on identical weights separates a model a well-funded team can rent hardware for from one that needs a research cluster.

The calculation is deliberately boring: bytes = total_parameters × (bits_per_parameter / 8) — 2 bytes per parameter at bf16, 1 at fp8. MXFP4 is a block format: four-bit elements share an 8-bit scale across each block of 32, so the effective cost is 4 + (8/32) = 4.25 bits, or 0.53125 bytes.

BYTES_PER_PARAM = {
    "bf16": 2.0,          # 16-bit
    "fp8": 1.0,           # 8-bit
    "int4": 0.5,          # 4-bit, no block scale accounted
    "mxfp4": 4.25 / 8,    # 4-bit elements + one 8-bit scale per 32-element block
}

def weight_set_bytes(total_params: int, dtype: str) -> float:
    """Weights only. Excludes KV cache, activations and framework overhead."""
    return total_params * BYTES_PER_PARAM[dtype]

def min_gpus(total_params: int, dtype: str, usable_gb_per_gpu: float,
             overhead_factor: float = 1.20) -> int:
    """overhead_factor is a rough allowance for KV cache, activations and
    framework reserve. Replace it with a measured number as soon as you have one."""
    needed = weight_set_bytes(total_params, dtype) * overhead_factor
    return -(-int(needed) // int(usable_gb_per_gpu * 1e9))   # ceiling division

MODELS = {"Kimi K3": 2_800_000_000_000,
          "Kimi K2": 1_000_000_000_000,
          "Inkling":   975_000_000_000}

for name, params in MODELS.items():
    for dtype in ("bf16", "fp8", "mxfp4"):
        tb = weight_set_bytes(params, dtype) / 1e12    # decimal TB
        print(f"{name:9} {dtype:6} {tb:6.2f} TB")

Model	Total params	Active per token	bf16 (2 B/param)	fp8 (1 B/param)	MXFP4 (~0.53 B/param)
Kimi K3	2.8T	~16 of 896 (~50B)	5.60 TB	2.80 TB	~1.49 TB (shipped set ~1.4 TB)
Kimi K2	1T	32B	2.00 TB	1.00 TB	~0.53 TB
Inkling	975B	41B	1.95 TB	0.98 TB	~0.52 TB

These are weights only, and two sanity checks confirm the method. Moonshot's published ~1.4 TB and ~5.6 TB sit alongside both calculated columns. The four-bit figures reconcile once you watch the units: 1.4875 × 10¹² bytes is 1.35 TiB, so a published "1.4 TB" is the same quantity rounded, not a different one. Always check whether a stated weight size is decimal TB or binary TiB before you size a fleet against it. And Thinking Machines states Inkling needs more than two terabytes at native 16-bit against a calculated 1.95 TB — consistent with the overhead this table excludes. That source cites roughly eight NVIDIA B300 or sixteen NVIDIA H200 accelerators as practical setups. On the sixteen-accelerator branch that implies near 130 to 140 GB usable each; on the eight-accelerator branch it implies roughly double that. Both are inferences from the published configuration, not vendor capacity specifications.

Quantisation choices

At this scale quantisation is entry, not optimisation, and both major open stacks have first-class MXFP4 paths. In vLLM the GptOssMxfp4MoEMethod path handles MXFP4 weights and selects a backend such as FlashInfer or Triton; FlashInfer ships a variant optimised for MXFP4 in its MoE runners, aimed at memory-efficient low-precision inference. Kernel support matters: a format with no matching kernel gives you the memory saving and none of the speed. SGLang takes a different route on AMD hardware — on GPUs with hardware FP4 support it accepts --quantization quark_mxfp4, quantising BF16 weights to MXFP4 at load time, so you keep one high-precision master artefact.

Post-training quantisation is not the same risk as quantisation-aware training

Builders routinely conflate two things with very different risk profiles. Post-training quantisation is what you do to a model released at bf16: you compress afterwards, it never saw four-bit numerics in training, and any quality cost is yours to discover. Quantisation-aware training puts those numerics in the training loop, so the model learns weights that survive the format. Kimi K3 applies QAT from the supervised fine-tuning stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility. Four-bit is K3's native format rather than a lossy afterthought, and that training choice is intended to absorb the cost upstream — which changes your question from "how much did I break" to "does this build behave on my workload".

Note too that weight and activation precision are separate decisions with separate hardware requirements. A stack supporting one but not the other will fall back to a slower path or refuse to load, and the error rarely names the real cause.

What to test before you trust a low-precision build

Measure — heavily for a post-hoc build, still meaningfully for a QAT one. Two places deserve your budget first, framed as things to test rather than findings.

• Routing decisions. Noise in the small layer that decides which experts fire can shift routing, and that changes which experts contribute at all — a different failure mode from the gentle degradation dense models show. Diff expert-selection distributions between builds.
• Long-context recall. These models advertise 1M-token windows, and accumulated error over long sequences is a plausible place for divergence that short prompts never surface. Test at the lengths you intend to use.

Parallelism layout

No accelerator holds a terabyte-plus weight set, so the model is split three ways — and MoE strongly favours one. SGLang documents the expert-parallel path in detail. Tensor parallelism shards each weight matrix across GPUs and pays an all-reduce after each sharded operation, multiplied enormously across 896 experts. Expert parallelism places whole experts on different GPUs and moves the tokens to them, mapping far better onto how MoE computes; SGLang documents it in docs/advanced_features/expert_parallelism.md. Its cost is an all-to-all dispatch and combine per MoE layer — harder on topology than an all-reduce, because every GPU talks to every other. Pipeline parallelism passes activations between stages point-to-point: cheap communication and the pragmatic way to cross a slow node boundary, at the cost of bubbles that hurt TTFT.

Strategy	What it splits	Communication	Memory relief	Best for	Main cost
Tensor parallel	Every weight matrix	All-reduce per layer, per token	Linear in TP degree	Attention and dense blocks, one node	Chatty; needs fast intra-node links
Expert parallel	Whole experts across GPUs	All-to-all dispatch and combine	Linear in EP degree	MoE blocks — the bulk of the weights	Topology-sensitive; load imbalance
Pipeline parallel	Layers into sequential stages	Point-to-point activations	Linear in stage count	Crossing a slower node boundary	Bubbles; degrades TTFT
Expert offload	Cold experts moved off GPU	Host-to-device transfer on cache miss	Bounded by host RAM, not GPU memory	Last resort when weights will not fit	Per-miss latency on the critical path

In practice you combine them: tensor parallelism within a node for attention, expert parallelism across the fleet for the MoE blocks, pipeline parallelism only to span a slow boundary. The winning layout keeps all-to-all traffic on your fastest links, which is why two fleets with identical aggregate memory can differ by a large factor in throughput. It is also why the stack matters: both vLLM and TensorRT-LLM added MoE-specific optimisations through 2025 because general-purpose serving does not handle expert routing efficiently, and SGLang sometimes outperforms vLLM for MoE on throughput and token generation speed. Benchmark both — the ranking moves between releases.

# vLLM — sketch. Verify flag names against your installed version: vllm serve --help
# The MXFP4 path (GptOssMxfp4MoEMethod) is selected from the checkpoint's
# quantization metadata, then picks a backend such as FlashInfer or Triton.

vllm serve moonshotai/Kimi-K3 \
  --served-model-name kimi-k3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --download-dir /mnt/nvme/weights \
  --port 8000

#   --tensor-parallel-size    keep within a single high-bandwidth node
#   --pipeline-parallel-size  use only to cross a slower node boundary
#   --max-model-len           what you will actually serve; a 1M window costs
#                             KV cache you may not be able to afford
#   --gpu-memory-utilization  leave real headroom; 0.95+ is where OOMs live

# SGLang — load-time quantization from BF16 weights to MXFP4.
# Documented for AMD GPUs with hardware FP4 support.
# Use this on a model RELEASED at BF16 that you compress yourself — such as
# Inkling. Do NOT apply it to a QAT-native checkpoint like Kimi K3, which
# already ships MXFP4: there is no BF16 master there to convert.
# Expert parallelism: docs/advanced_features/expert_parallelism.md

python -m sglang.launch_server \
  --model-path <org>/Inkling \
  --quantization quark_mxfp4 \
  --tp-size 8 \
  --context-length 262144 \
  --host 0.0.0.0 --port 30000

# quark_mxfp4 converts BF16 weights to MXFP4 at load time, so you keep one
# high-precision master artifact. Expect a longer cold start in exchange.
# Other flag names (expert-parallel size, memory fraction, scheduler policy)
# vary between releases — check --help for your version.

Expert offloading, when it genuinely will not fit

Sometimes no amount of quantisation closes the gap, and sparse activation offers an escape dense models cannot use: most expert weights are idle most of the time. vLLM RFC issue #38256, "Incremental MoE Expert Offloading — GPU Cache + Async Pipeline", describes the design. Expert weights live in CPU pinned memory; a fixed-size GPU cache holds the hottest experts; an LFRU eviction policy — how often and how recently an expert was used — decides what stays resident; and cross-layer prediction anticipates which experts an upcoming layer needs, so transfers start before they are required.

The honest accounting is that every cache miss puts a host-to-device transfer on a token's critical path. Prediction hides some of that latency, never all of it, because routing is input-dependent by design. The result is a heavier latency tail — and tails are what a p95 budget measures. Offloading is right when throughput beats latency: batch processing, overnight evaluations, offline dataset generation. It is wrong for interactive chat with a tight p95, and wrong as a way to avoid confronting under-provisioning. If it is load-bearing, the real decision is a smaller model or more hardware.

Keeping the GPU busy

Clearing the memory gate does not earn you good economics. Continuous batching is the baseline — the scheduler admits new requests as slots free and evicts finished ones immediately, so the GPU never waits on the slowest sequence in a static batch. If you have tuned that for a dense model the mechanics carry over, but the cost model does not.

In a dense model a batch of 32 tokens costs roughly 32 times one token, because all 32 traverse the same weights. In an MoE model cost depends on which experts the batch touches: tokens routing to a small overlapping set are cheap, while the same batch scattered across many distinct experts costs far more, with GPUs owning popular experts bottlenecking while others idle. That is expert load imbalance — a property of your traffic, not just your configuration — so measure with your traffic mix. A benchmark built from one prompt template flatters an MoE deployment badly.

The usual levers still stack on top: prefix caching where requests share long prefixes, and speculative decoding where the acceptance rate justifies the draft cost — though the draft model's own footprint is a budget line.

A benchmark protocol before you commit

Commit hardware only after five numbers, measured on your model, layout and traffic: sustained output tokens per second at target concurrency; TTFT; p95 end-to-end, because the median is meaningless when routing variance and offload misses live in the tail; GPU memory headroom at peak, with the KV cache full at maximum context and concurrency; and cost per million output tokens:

cost_per_MTok = (fleet_hourly_cost / (output_tokens_per_second × 3600)) × 1,000,000

import time, statistics
from concurrent.futures import ThreadPoolExecutor

def benchmark(prompts, concurrency, fleet_hourly_cost_usd):
    """Feed it YOUR traffic mix, not one repeated template: MoE routing
    concentration depends on prompt diversity and a single template flatters."""
    ttfts, e2es, output_tokens = [], [], 0

    def one(prompt):
        t0 = time.perf_counter()
        first_token_at, n_out = None, 0
        for _chunk in send_request(prompt, stream=True):      # your client here
            if first_token_at is None:
                first_token_at = time.perf_counter()
            n_out += 1
        return (first_token_at - t0), (time.perf_counter() - t0), n_out

    wall_start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        for ttft, e2e, n_out in pool.map(one, prompts):
            ttfts.append(ttft * 1000.0)
            e2es.append(e2e * 1000.0)
            output_tokens += n_out
    wall = time.perf_counter() - wall_start

    tok_per_s = output_tokens / wall
    return {
        "output_tok_per_s": round(tok_per_s, 1),
        "ttft_p50_ms": round(statistics.median(ttfts), 1),
        "e2e_p95_ms": round(sorted(e2es)[int(0.95 * len(e2es)) - 1], 1),
        "cost_per_mtok_usd": round(
            (fleet_hourly_cost_usd / (tok_per_s * 3600)) * 1_000_000, 2),
    }

# Sweep concurrency to find where p95 breaks your budget:
#   for c in (1, 4, 16, 32, 64, 128):
#       print(c, benchmark(real_traffic_sample, c, fleet_hourly_cost_usd=32.16))

Sweep concurrency rather than testing one point: you want the highest concurrency at which p95 still clears your budget, because that is your operating point. Our guide to load-testing and capacity planning for LLM apps covers building a representative traffic sample, which matters more for MoE than for dense serving.

Cost snapshot: July 2026

The figures below are a snapshot dated July 2026. GPU pricing moves quickly enough that anything older than about six months should be re-checked; the framework has the long shelf life.

Item	Figure (July 2026)	What it tells you
A100 80GB rental	$1.09 – $5.07 per hour	Nearly a 5× spread — provider, region and commitment matter more than the silicon.
H100 80GB rental	$2.01 – $11.06 per hour	A 5.5× spread. Quoting "the H100 price" without saying where is meaningless.
SXM vs PCIe form factor	$2.69/hr vs $1.99/hr	26% saved on PCIe if you do not need NVLink — but MoE all-to-all often does.
Used H100 cards	~$40,000 (late 2023) → as low as $6,000 (mid-2026)	An ~85% fall. Owning is far cheaper than it was, if you can operate it.
AWS EC2 Capacity Blocks	~+20% effective 1 July 2026, after ~15% in January 2026	Reserved capacity trends up even as spot and used hardware fall.

The form-factor point cuts both ways: 26% is real money on sixteen accelerators, but expert parallelism leans hardest on exactly those high-bandwidth intra-node links. Measure both before assuming the cheaper form factor is a saving rather than a false economy. Break-even is a formula you run with your own inputs, not a number to inherit:

monthly_self_host = fleet_hourly_cost × hours_per_month + ops_cost
break_even_MTok = monthly_self_host / api_price_per_MTok
fleet_capacity_MTok = (output_tokens_per_second × 3600 × hours_per_month) / 1e6

Worked: sixteen H100-class accelerators at the low end of the July 2026 range, $2.01/hr each, is $32.16/hr — $23,155 a month always-on at 720 hours, before operations cost, which is not zero. Suppose your benchmark measures 1,000 output tokens per second (a placeholder — substitute what you measure): capacity is 2,592 MTok a month and cost is $32.16 / 3.6 = $8.93 per million output tokens at full utilisation. The P values below are placeholders showing the curve's shape, not quoted prices.

Illustrative price P ($/MTok)	Break-even volume (MTok/month)	vs capacity of 2,592 MTok/month	Verdict at this throughput
$2	11,578	4.5× the fleet's output	Self-hosting cannot pay
$5	4,631	1.8× the fleet's output	Self-hosting cannot pay
$10	2,316	89% of capacity — barely reachable	Only at near-full utilisation
$20	1,158	45% of capacity	Plausible, with real headroom

The capacity-ceiling column is the check most break-even analyses omit. A break-even volume above what your fleet can physically generate is not a target to grow into; it is proof that self-hosting cannot pay at that price and throughput. Lift throughput to 3,000 tokens per second and capacity rises to 7,776 MTok, putting the $5 row within reach at around 60% utilisation — which is why throughput work returns more than hardware shopping. Our self-host or API decision guide works the same trade at friendlier scale.

Region choice is a second-order cost with first-order compliance consequences. AWS Mumbai (ap-south-1) optimises for Indian user latency and keeps personal data onshore under India's data-protection regime; AWS London (eu-west-2) weighs UK GDPR obligations and EU adequacy for anything crossing. Serving both markets from one fleet also pays inter-region egress on every cross-border call. The pragmatic pattern is one fleet per market — worse on utilisation, better on latency and residency — and two half-loaded fleets are much harder to justify than one full one.

Common pitfalls

1. Sizing on active parameters. "It only activates 41 billion" is true and irrelevant to memory. Size on total; use active only for compute.
2. Forgetting the KV cache at 1M context. At high concurrency it can rival the weight set. Provision for maximum context times maximum concurrency, then verify by measurement.
3. Choosing a parallelism layout from the memory spreadsheet. Memory that fits will start the server; only topology decides whether the per-layer all-to-all is a rounding error or the dominant cost.
4. Benchmarking with a single prompt template. Homogeneous prompts route to a concentrated expert set and produce throughput real traffic will never reproduce.
5. Treating offloading as capacity rather than margin. It reshapes the latency tail; it does not create memory you did not buy.
6. Conflating post-training quantisation with QAT. Compressing a bf16 release yourself leaves the quality cost undiscovered; a QAT release has absorbed it. Check hardware support for activation precision as well as weight precision.
7. Comparing rental prices without form factor, region and commitment. A 5× spread on the same nominal card makes any single quoted price close to meaningless.
8. Ignoring cold-start on a 1.4 TB weight set. Stage weights on local NVMe; our KServe and KEDA autoscaling guide covers why load time dominates scaling policy here.

When not to do this

Most teams reading this should not self-host a trillion-parameter MoE, and saying so plainly is more useful than another optimisation tip. Do not if sustained volume sits below the break-even your own inputs produce — arithmetic, not opinion. Do not if you lack the operational capacity for a multi-node GPU fleet: on-call, capacity planning and hardware-failure procedures, not someone maintaining it alongside other work. Do not if a smaller model clears your quality bar — Inkling-Small at 276 billion total and 12 billion active is right far more often than the flagship. And do not if the requirement is really data residency alone; a regional API deployment satisfies that for a fraction of the effort.

Do consider it if volume is genuinely sustained, if per-token pricing has become your largest infrastructure line, if you need weight-level control no API exposes, or if a regulatory position requires the weights on infrastructure you control. As of July 2026, Kimi K3 is the largest open-weight release, and its significance is optionality: the frontier is now something you can run, which is different from something you should. We covered it in our news piece on Kimi K3 topping the frontend code arena.

Next steps

Work the gates in order and stop at the first failure. Run the weight-set calculation against your real usable GPU memory — most projects end here, correctly and cheaply. If it clears, choose a layout that keeps all-to-all traffic on your fastest links. Benchmark vLLM and SGLang on a representative traffic sample, measure the five numbers, sweep concurrency to find where p95 breaks, then run the break-even formula with your own price and measured throughput, including the capacity-ceiling check.

The models here are worked examples and they will date. The framework will not: total parameters set memory, active parameters set compute, interconnect decides whether expert parallelism flies, and break-even is a formula you run rather than a number you inherit. Swap the model and pricing tables in six months and the rest still holds.