FreeToken: Edge-Native MoE Serving
with Bandwidth-Adaptive Execution

Overview: the claim and the key numbers

MoE models are a natural fit for the edge: each token passes through only a few of hundreds of experts. But saving computation doesn't save weights — the full expert pool can exceed VRAM by orders of magnitude. FreeToken's core claim: don't treat edge hardware as a "mini GPU"; treat the whole machine as one elastic inference platform, continuously remapping GPU, CPU, memory and PCIe interconnect according to the bandwidth and capacity actually available at runtime.

• Mechanism ① · Bandwidth-adaptive execution:prefill hides expert transfers behind compute with full-layer double buffering; decode splits cache misses between PCIe fills and in-place CPU execution according to the measured bandwidths.
• Mechanism ② · Semantics-aware caching:recurrent-state checkpoints are anchored at the semantic boundaries agent frameworks actually edit (think / tool call / turn); the expert cache is one shared all-layer LRU that follows routing locality.
• Mechanism ③ · Elastic edge resource management:the GPU expert cache is rebuilt at scheduling safe points without restarting; weights load directly into their final host layout, so startup needs no GPU warm-up.

Background: the usability gap

Open models are rapidly closing the capability gap with closed ones (Kimi-K3, GLM-5.2, DeepSeek-V4-Flash-0731…); the access gap is not closing — frontier models still live on million-dollar data-center GPUs, and continuous API spending weighs on individuals and small teams. MoE opens a door: DeepSeek-V4-Flash has 284B parameters across 43 MoE layers, each with 256 routed experts of which only 6 activate per token — 13B active parameters, which fits an RTX 5090's 32GB at deployment precision. But sparsity saves per-token compute, not the memory of the full expert pool: the complete weights still far exceed VRAM, so inactive experts must live in host memory (or disk) and stream in on demand. MoE thus hands us both the opportunity (computation is feasible) and the systems challenge (serving is hard).

The infrastructure reality backing the claim: Steam has 200M+ monthly active users, ~72% of systems with a discrete NVIDIA GPU — hundreds of millions of "capable but idle" machines. What's scarce isn't hardware, but a serving system that treats heterogeneous consumer machines as one platform and automatically maps GPU/CPU/memory/interconnect onto the strongest runnable configuration.

Existing edge engines (llama.cpp, KTransformers, Ollama) fall short of the theoretical capability on three axes: prefill destroys sparsity (the union of routes over thousands of prompt tokens touches nearly every expert per layer — the working set turns dense); decode is the opposite trap (few experts per token, but misses cause repeated transfer/eviction/in-memory execution with no principled policy); and edge resources are diverse and dynamic (VRAM budgets shift under the browser and the game). The cost–capability picture, and where FreeToken lands:

💡 Click any image to view the original 300-DPI version; click again or press Esc to close.

Three challenges of agentic MoE on the edge

Challenge 1: prefill = expert transfer + recomputation, a double cost

Every prefill adds seconds of expert streaming. Decode touches k experts per token, but prefill is thousands of tokens × every layer — the union of routes activates essentially the whole expert pool, so one prefill streams the entire pool from host memory across the CPU–GPU link. For FP4-deployed DeepSeek-V4-Flash: ~140GB of expert weights needs ~2 s over PCIe 5.0 ×16 (~60GB/s, RTX 5090), 5 s over PCIe 4.0 ×16 (~25GB/s, RTX 4090/3090), 10+ s over the ×8 links common in laptops. In engines that fetch experts on demand, those seconds are pure GPU idle.

And agent tool calls trigger re-prefill constantly. Hybrid-attention architectures (full attention + sliding window, as in DSV4-Flash / GPT-OSS; or recurrent layers like Qwen3.6's gated DeltaNet, Kimi-K3's Delta Attention) compress past context into a single state or a recent window; each state costs as much KV memory as hundreds of tokens, so engines keep only a few checkpoints. But agents edit the context almost every turn — dropping old tool outputs, trimming thinking segments — invalidating every checkpoint after the edit point and forcing a re-prefill of thousands of tokens back to the nearest survivor. Consumer GPUs can't afford the repetition: an RTX 5090's dense BF16 compute is ~1/5 of an H100's and ~1/10 of a B200's.

Challenge 2: how to serve decode misses is decided by two measured bandwidths

Three root causes behind slow baselines: static placement misses routing traffic — llama.cpp splits MoE tensors across devices at load time, KTransformers pins "hot" experts at load time, but routing changes token by token, so placement frozen at prefill catches only a fraction of traffic and leaves GPU and PCIe idle. Consumer CPUs alone can't carry decode — at small batch, expert execution is memory-bandwidth-bound; consumer CPUs with dual-channel memory deliver ~50GB/s (DDR4) or 80–90GB/s (DDR5) against 1–1.8TB/s from GPU HBM. The right split is hardware-dependent — a miss can be transferred over PCIe and run on the GPU, or executed in place on the CPU, and neither is universally optimal; an RTX 4060 laptop (LPDDR5) and an RTX 5090 desktop (DDR5) sit at opposite ends of the "memory bandwidth vs. PCIe bandwidth" scale, and the optimal mix can't be read off a spec sheet — it must be measured on the real machine.

Challenge 3: no dedicated resources on the edge — even startup is slow

VRAM budgets fluctuate: the GPU shares with the compositor, the browser, the game — hundreds of MB to several GB get snatched at any moment; agent sessions accumulate context across turns while the expert working set stays fixed, so the KV/expert split chosen in round one is wrong many rounds later. The split must be adjustable at runtime without restarting the engine. Startup is slow and frequent: reading DSV4-Flash's ~140GB FP4 pool from 7GB/s NVMe takes ~20 s before any warm-up; edge users open and close engines and switch models constantly.

Design: two-tier expert memory + three mechanisms

The system is organized around a two-tier expert memory hierarchy: the expert pool in CPU memory (the complete routed-expert weights, always the source of truth) + a single elastic all-layer shared expert cache on the GPU (each slot holds all tensors for one "layer–expert" pair; residency, lookup and execution all use the logical (layer, expert) identity, not tensor shards). Non-expert weights stay resident on the GPU.

Mechanism ① · Prefill: full-layer double buffering + semantics-aware state cache

Double buffering hides transfer behind compute.Because prefill activates nearly every expert per layer, FreeToken doesn't fetch on demand: it borrows two "full-layer buffers" from the global slot pool — while the GPU computes layer l's routed experts from buffer A, a dedicated transfer stream fills buffer B with layer l+1's complete expert set. Whole-layer transfer needs no routing results, so weight streaming proceeds continuously in the background. The buffers share one slot pool with the decode cache — no separate prefill cache, no phase handoff; entries surviving prefill serve latency-sensitive decode directly. When the pool can't free two whole layers, it falls back to on-demand loading and never oversubscribes VRAM.

Semantic anchors let recurrent states survive edits.Hybrid-attention models carry a second prefix resource besides the KV cache: the recurrent layers' "evolving state". Full-attention KV is managed by a radix prefix tree (as in SGLang); recurrent states can't be partially reused, so they live on checkpoints taken during prefill/decode. FreeToken keeps a small semantics-aware state cache: checkpoints hang on prefix-tree nodes, and a new request resumes from the nearest checkpoint that survives the edit. The checkpoint budget goes to special-token boundaries — thinking segments, tool calls, tool outputs, turn boundaries — precisely where agent frameworks edit: OpenClaw strips thinking blocks from all but the latest assistant turn, OpenCode replaces tool outputs beyond a protection window with placeholders, SWE-agent keeps only the last n observations. Frameworks preserve the exact prefix up to the edited block, so checkpoints anchored there are the most likely to survive; full-attention layers reuse KV up to the edit point, recurrent layers resume from the anchor, and only the genuinely new suffix is re-prefilled.

Mechanism ② · Decode: semantics-aware expert cache + the q* bandwidth-adaptive policy

At decode, the GPU router plus a cache lookup identify the set H of active experts already cached (executed on the GPU directly); the remaining m unique missing experts M are the hard part. Routing has strong temporal locality across steps (the same layer's consecutive tokens repeatedly route to overlapping/recent experts — measured across model families, Liang et al. 2025), so instead of a load-time placement, FreeToken keeps one all-layer shared LRU residency space: hits refresh recency, fills absorb newly selected experts, evictions drop the least recently demanded — scarce VRAM continuously tracks the current working set. The cache can't eliminate all misses (cold start, working-set shifts, capacity limits); the residual misses go to bandwidth-adaptive execution.

Split the m misses into a cache-fill set F and a CPU-execution set C (M = F ∪ C, q = |F|): F's experts transfer into cache slots, execute on the GPU, and stay for reuse; C's experts execute directly from the resident host pool without touching residency state. The two branches run concurrently: fills run at full PCIe speed while the CPU consumes only the host bandwidth left after the link saturates — turning "residual bandwidth" into progress on the current token without stalling cache updates.

The optimal split falls out of a residual-bandwidth argument. With S bytes per expert, and expert DMA sharing the host memory subsystem with CPU execution, the bandwidth left after PCIe saturates is:

The two branches take:

Balancing the two concurrent branches (the layer's exposed latency is the slower of the two):

This single formula covers the whole hardware spectrum: as $B_H$ approaches $B_P$, $q^*$ approaches m and the system degenerates to pure on-demand cache filling (no separate execution branch needed). In practice q* is rounded; which experts enter F is the replacement policy's business; and at least one fill is always kept, so the cache keeps warming even when the CPU carries most of the load. Both bandwidths are profiled on the target machine at deployment.

Mechanism ③ · Elastic memory: VRAM affects performance, never correctness

Because the CPU-resident pool is the source of truth, any change in GPU memory moves performance only. Runtime cache reconfiguration:after non-expert weights and runtime state are allocated, the remaining budget splits between KV pages and whole expert slots — and the split isn't pinned at startup: at any scheduling safe point the GPU expert cache can be rebuilt with a revised budget, without restarting the engine or reloading the host pool, re-capturing the execution path dynamically. Fast startup:weights read directly into their final host layout and pin memory only after filling (pinning empty buffers first faults in and zeroes several GB of pages for nothing); no warm-up is needed — the first request starts cold, misses flow through the ordinary decode path of §3.2, and the cache warms up by serving.

Implementation: putting "dynamic" inside a "static" CUDA Graph

FreeToken keeps the SGLang/vLLM GPU-centric architecture (paged KV + radix prefix reuse) and plugs into community kernel libraries (FlashInfer, Flash Linear Attention). On top sit two layers: a graph-compatible expert cache and storage/platform plumbing.

A CUDA-Graph-compatible LRU cache

The expert cache is inherently dynamic — which experts are missing, how many to fetch, which slots to evict change every step; host-driven control flow would pay an expensive device sync per MoE layer. FreeToken keeps all routing-related control on the GPU, expressing dynamic behavior with data captured inside a static graph: fixed-shape work buffers + a device-side valid count. A single GPU kernel per MoE layer deduplicates routed experts, classifies them against the residency table, derives q from the bandwidths, picks eviction victims, and rewrites logical routing IDs into physical slot IDs (or flags "CPU-execute"). Victim selection avoids the classic LRU trap of scanning the whole cache per eviction: one kernel pass finds the K least-recently-used candidate slots up front, and the miss path consumes the first q ≤ K on demand — victim discovery always costs one pass, independent of miss count. The resulting copy work-list drives a single fused transfer; expert banks share one logical-ID→slot mapping, so one device-side source/destination index list, launched once in fixed shape, applies to every bank; valid counts mask unused work. Result: fewer kernel launches, high PCIe utilization, and routing decisions moved off the host.

CPU execution resident in the graph

The CPU branch is captured into the same graph: for each supported decode batch size, stable pinned I/O buffers and persistent task descriptors are prepared, and the device→host copy, host-function submission node, concurrent GPU path, synchronization node and host→device result copy are all captured together. A replay is the complete heterogeneous step — no per-token Python scheduling. Workers are a persistent C++ pool pinned to physical cores, consuming expert weights with architecture-specific SIMD + in-kernel dequantization to stay bandwidth-bound, and returning gate-weighted per-token partial outputs.

Expert storage and platform plumbing

Expert banks + the FTW format:model-specific checkpoint layouts are normalized into a few expert banks, each with the flattened "layer–expert" ID (lE+e) as the leading dimension; rows with the same ID across banks compose one complete expert, and the GPU kernels and CPU executor share the same logical identity. FTW (FreeToken Weight) pre-merges expert weights into the runtime bank layout, so startup skips tensor discovery and repacking — parallel direct I/O reads aligned blocks straight into exactly-sized host banks, pinning only after the fill. Platform adaptation:GPU kernels are selected at load by expert representation/GPU architecture/CUDA environment; the CPU executor dispatches to available SIMD implementations and core topologies. When the full pool can't be pinned/registered for DMA (OS and driver limits), a pure-CPU MoE backend kicks in: weights stay in pageable host memory, all routed experts execute on the CPU, non-expert layers stay on the GPU, and only activation-sized inputs, routing metadata and aggregated outputs cross the CPU–GPU boundary — trading peak transfer bandwidth for "deployable even when the fast path can't come up".

Evaluation: 4 real agent workloads × 6 machines × 4 baselines

Setup

Hardware:six discrete-GPU systems — five consumer machines + one workstation (RTX PRO 6000 Blackwell, 96GB). The 3090/4090/5090 are rented dual-socket servers whose CPUs far outclass edge hosts, so all serving and bandwidth measurements cap them at 6 CPU threads pinned to the GPU's NUMA node; capped this way they deliver 56.7–77.3GB/s host bandwidth, the same magnitude as the two real edge machines at full threads (desktop 16-core: 53.8; laptop 14-core: 47.5). All bandwidths are measured on deployed tensor shapes, not taken from spec sheets.

Models:DeepSeek-V4-Flash (284B/13B active, natively MXFP4-quantized experts) and Qwen3.6-35B-A3B (BF16; the 8GB laptop uses the official NVFP4 build); the cross-hardware study adds GLM-5.2 (753B/40B active, NVFP4, 433GB checkpoint). Workloads:W1 math reasoning (AIME — long CoT, no tools, single-turn, decode-dominated); W2 coding agent (SWE-bench tasks through the OpenCode framework with real tool execution); W3 coding agent on its native protocol (the same tasks through Claude Code's Anthropic-compatible endpoint, spawning concurrent sub-agents, sessions reaching 56–65k tokens); W4 email/calendar agent (OpenClaw default config for thirteen rounds, its 120s idle watchdog disabled so slower engines remain testable, ~24.5k-token system-context base). Coding tasks must produce the reference gold patch; W4 must complete all thirteen rounds. Baselines:llama.cpp / Ollama / KTransformers / MoE-Infinity (where supported), weight formats bit-aligned (MXFP4 blocks bit-exact). Metrics:mean per-request decode throughput and mean TTFT; agent trajectories differ per engine, so total wall-clock is not compared.

End-to-end results (RTX 5090)

Attribution: where the gains come from

Pipelined prefill (Figure 4a).Double buffering makes prefill transfer-bound: with overlap on, each 8,192-token chunk completes in 1.19–1.22 s — exactly the time to stream the 64.4GB expert pool at 52.7GB/s, i.e. the practical limit of PCIe 5.0 ×16, with expert compute fully hidden; throughput reaches 6.7k tok/s at 16k tokens. Disabling the second buffer serializes transfer and compute, losing 19% / 25% / 26% at 4k / 8k / 16k — the penalty grows with prompt length.

Expert locality (Figure 4b).Decode routing has strong short-range locality — per-miss LRU beats any placement chosen at prefill time. Replaying identical routing traces from all four workloads over three placement policies at equal cache capacity: at RTX 5090 capacity (37% of Qwen3.6's expert pool, 11% of DSV4-Flash's), FreeToken's global LRU achieves decode expert-read miss rates of 16% / 39%, KTransformers' prefill-updated placement 41% / 59%, and llama.cpp's routing-blind static split 62% / 89% — the ordering holds at every capacity short of the full pool.

Cross-hardware serving (Figure 5).W2 repeats on five consumer machines: FreeToken leads the best baseline by 1.3× (3090/4090), 1.9× (5090 server), 2.1× (5090 desktop), 1.8× (4060 laptop); on the 8GB ×8-lane laptop the NVFP4 build sustains 39.3 tok/s — 92% of a 4090. The two 5090 columns share silicon and differ only in the host: swapping from the multi-channel server to the dual-channel consumer desktop costs FreeToken 4% of decode speed, while llama.cpp — whose CPU-resident experts starve on dual-channel DDR5 — drops to 80%: dynamic split following measured bandwidths at work. Frontier tier: on a single RTX PRO 6000, FreeToken serves GLM-5.2 at 14.9 tok/s vs. llama.cpp's 7.3 (2.0×), with bit-identical expert weights and comparable mean TTFT (7.5 vs. 7.8 s); KTransformers has no servable path on this machine at all — its GLM-5.2 approach needs 753GB–1.5TB of host memory (the machine has 512GiB) and its CPU kernels can't read GLM-5.2's NVFP4 layout.

Commentary: strengths and open questions

Highlights

• Turning the bottleneck into a signal.Where others route around bandwidth, FreeToken puts the two measured bandwidths straight into the scheduling formula. q* = m·B_P/B_H is simple enough to live on-device inside a CUDA Graph, yet covers the entire hardware spectrum from "all fill" to "all CPU".
• No correctness sacrificed.Unlike HOBBIT/SiDA/SMoE-style lower-precision replicas or expert dropping, the output merge is exact and the model is untouched — a much wider compatibility surface for a general-purpose engine.
• System-level completeness.From the FTW weight format and startup layout, through CUDA-Graph-resident dynamic caching and runtime elastic rebuilds, to agent semantics (where anchors go, how to recover after edits) — a whole-stack design rather than an assembled demo; 20+ MoE models supported.
• Honest evaluation.Four real agent harnesses with real tool execution, gold-patch acceptance, bit-exact weight alignment — and candid disclosure that the rented "consumer" servers are 6-thread-capped simulations, validated against the real machines.

Open questions

• The table is the data.Table 1 admits that three of the "consumer machines" are capped servers; there is exactly one real 5090 desktop and one 4060 laptop — cross-machine generality rests on "capped simulation + deployment-time bandwidth profiling".
• Scope.The main comparison is thorough on two models (GLM-5.2 is a single-machine demo); "20+ MoE models" is a support statement, not all tested. The CPU branch depends on host memory bandwidth — if a browser or game runs alongside, the residual-bandwidth assumption shifts; the paper gives resource dynamics to runtime rebuilds, but q*'s B_H is profiled once at deployment, and online re-profiling isn't discussed.
• What's not covered.Multi-request concurrency and continuous batching (all metrics are per-request means); batch-size scaling; CUDA-Graph capture cost given "branches and syncs live in the graph"; and the relation to WiSP (same topic, bandwidth-aware expert paging) stays a textual comparison.

Glossary

Hover any dotted-underlined abbreviation in the text, or come back here any time.