Beyond the Pod: Disaggregation-Aware Scheduling for Heterogeneous AI Accelerators

Srihari Maddula • Founder & Technical Lead, Eurth Techtronics Pvt Ltd 

Category: AI Innovations 

Estimated Reading Time: 7 min

A cluster reports 40% average GPU utilization. Finance sees the invoice and asks why. The platform team points at the scheduler and says it did its job — every pod that asked for a GPU got one. Both are right, and both are describing a system that is architecturally blind to the thing that actually matters: what a workload does with the GPU once it has it.

This is the gap between resource allocation and resource placement. Kubernetes-native schedulers were built for a world where a “resource” was a roughly fungible slice of CPU and memory. A GPU is not fungible in that sense. Two workloads that each request “one GPU” can have arithmetic intensities that differ by two orders of magnitude, memory-bandwidth requirements that saturate an interconnect differently, and latency sensitivities that make co-location either free or catastrophic. Naive pooling — bin-packing pods onto whichever node has a free accelerator — throws all of that away.

The Roofline Problem, Restated for Schedulers

The roofline model gives every kernel a single number that matters for placement: arithmetic intensity — FLOPs performed per byte moved. A kernel with low arithmetic intensity is memory-bound; its ceiling is set by HBM bandwidth, not compute throughput. A kernel with high arithmetic intensity is compute-bound; the ceiling is FLOPs. A scheduler that allocates purely on GPU count or GPU memory requested cannot distinguish these, and the two failure modes it produces are opposite.

•  Co-locating two memory-bound kernels on accelerators sharing an interconnect (NVLink/NVSwitch, or the ICI mesh on TPU pods) causes bandwidth contention neither workload's SLA accounted for.

•  Fragmenting a single compute-bound workload across accelerators separated by a slow fabric turns a problem that should stay on one die into a distributed-systems problem with synchronization overhead the model author never budgeted for.

The case that makes this concrete is prefill/decode disaggregation in LLM serving. Prefill — processing the prompt — is compute-bound and highly parallel; it wants raw FLOPs and barely cares about interconnect latency. Decode — generating tokens one at a time — is memory-bound and latency-sensitive; it wants fast HBM access and gains little from additional FLOPs. Running both phases on identically-provisioned accelerators, as most clusters do by default, means over-provisioning decode with unused compute or starving prefill of bandwidth it doesn't need but still gets billed the same node cost for. Splitting the two phases onto differently-shaped hardware pools, and routing requests between them, is disaggregation-aware serving. It isn't exotic — it's the roofline model applied to a scheduling layer that has mostly ignored it.

THE RULE:  A scheduler that cannot see arithmetic intensity is not scheduling — it is dealing cards.

Why the CNCF Stack Is Converging Here, Not Diverging

The last eighteen months of the CNCF AI infrastructure stack read like a slow admission that Kubernetes' original resource model was insufficient for accelerators. Three projects matter, each attacking a different layer of the same problem.

Kueue — admission and queueing

Kueue decides whether a workload is allowed to start, based on quota, priority, and fair-share policy across Resource Flavors. It is workload-aware in that it understands job semantics — a training job is not a pod — but by default it doesn't understand what the workload will do with the accelerator once admitted.

KAI Scheduler — placement and binding

KAI Scheduler goes further into the placement decision itself: gang scheduling, bin-packing with fairness guarantees, and GPU-sharing primitives. This is the layer closest to “placement,” but its scoring functions remain largely resource-quantity based rather than workload-shape based.

DRA — describing the hardware graph accurately

Dynamic Resource Allocation is the structural piece that makes the other two viable long-term. Where the old device-plugin model exposed a GPU as an opaque, indivisible unit, DRA lets a cluster express structured claims — “give me a GPU with at least 40GB HBM and NVLink connectivity to any other claim in this Resource Claim Template,” for example. This is what makes it possible to build a scheduler extension that reasons about interconnect topology instead of node labels.

None of these three, alone, is a disaggregation-aware placement engine. Kueue admits, KAI Scheduler scores and binds, DRA describes the hardware graph accurately enough to make good scoring possible. The placement intelligence itself — the piece that reads a workload's roofline profile against a cluster's topology graph and produces a genuinely good match — sits in the gap between what these three provide and what a production AI platform actually needs. An Admission Check-based workload compiler lives in that gap: intercepting the Kueue admission decision, running the workload through a profiling pass, and emitting a placement recommendation KAI Scheduler or a custom scorer can act on.

What a Workload Compiler Actually Does

Calling it a compiler rather than a scheduler plugin is deliberate. A traditional scheduler makes a decision at admission time using whatever the pod spec declares. A compilation pass does three things a spec-reading scheduler cannot.

•  Static analysis of the workload artifact — for a training job, inspecting the model definition or a captured trace to estimate arithmetic intensity, memory footprint, and communication pattern (data-parallel, tensor-parallel, pipeline-parallel) before a single accelerator is claimed.

•  Topology-aware scoring — given a DRA-described hardware graph (which GPUs share an NVSwitch domain, which TPU chips share an ICI mesh segment, what cross-rack fabric bandwidth is available), score candidate placements against the workload's actual communication pattern instead of raw availability.

•  Compliance and disaggregation constraints as first-class inputs — data residency and tenant isolation rules aren't bolted on as admission webhooks after the fact; they're inputs to the same scoring pass, because in practice they interact directly with the performance placement decision.

THE RULE:  Placement quality is a function of what the scheduler is allowed to know about the workload, not how sophisticated its bin-packing algorithm is.

The Sovereign Compute Wrinkle

One requirement surfaces repeatedly once compliance becomes a first-class scoring input: sovereign compute — the rule that a workload's data, and sometimes its accelerator cycles, never cross a jurisdictional boundary. For CPU workloads this is mostly a pure policy filter. For accelerator workloads it interacts with placement in an uncomfortable way: sovereign-compliant hardware pools are usually smaller and less topologically ideal than the full fleet, so the compliant placement is frequently not the performance-optimal one.

A workload compiler that treats compliance as a hard pre-filter, and performance as a secondary optimization within the filtered set, handles this correctly. A scheduler that treats them as independent scoring dimensions to be summed can produce placements that are compliant on paper but violate the spirit of isolation — for example, splitting a tensor-parallel job's shards across sovereign and non-sovereign nodes because the summed score still came out favorable. That's a real failure mode, and the kind of bug that surfaces in an audit, not a load test.

Where This Leaves Cluster Operators Today

None of this exists as a single off-the-shelf product. What exists is a set of composable primitives — Kueue for admission, DRA for structured hardware description, KAI Scheduler or a custom scorer for placement — and a real, well-documented gap in the middle where workload-shape intelligence is supposed to live. Teams solving this today mostly do it with hand-rolled Admission Check webhooks and node-affinity heuristics derived from tribal knowledge about which jobs run well next to which other jobs. That's a stopgap, not an architecture, and it doesn't survive a fleet refresh, a new accelerator generation, or a new engineer joining the platform team.

The architectural bet worth making is that placement intelligence belongs as a distinct compilation pass between admission and binding — not folded into either. It needs its own introspection stage, its own topology model consuming what DRA exposes, and its own scoring function that can evolve independently of both admission policy and low-level binding mechanics. Get that separation right and the 40% utilization number stops being an invoice mystery and starts being a dashboard you can actually act on.

EurthTech delivers AI-powered embedded systems, IoT product engineering, and smart infrastructure solutions — Hyderabad, India. www.eurthtech.com