GPU Utilisation Explained: Why Clusters Sit at 5 Percent and How MIG and Time-Slicing Help

Photo: Laura Ockel / Unsplash

Five percent.

That is the average GPU compute utilisation reported by Cast AI's 2026 State of Kubernetes Optimization report, drawn from tens of thousands of production clusters between January 2025 and April 2026. CPU averaged 8 percent and memory 20 percent in the same data. The best case the report observed was a 136-node H200 inference fleet sustaining 49 percent.

Read the number carefully before you build a budget case on it. It measures compute activity, and the report's own commentary warns against assuming that 95 percent of GPU capacity is free to reclaim. A GPU can show 5 percent compute use and still be fully allocated to a job, with no room for another. That distinction is the framework of this post: measure two numbers, not one, and fix whichever of them is wrong.

Two Numbers, Not One: Allocation and Compute Use

The first number is allocation. Of the GPUs you own or rent, what share is currently assigned to a workload, so that nothing else can use them? The scheduler knows this exactly. The second number is compute use: of the time a GPU is assigned, how much of its arithmetic capacity is the workload using? This comes from device telemetry.

The two tell different stories. A cluster with high allocation and low compute use has a sharing problem: GPUs are held by jobs that use a small part of them. A cluster with low allocation and low compute use has an idle capacity problem: you bought more than you need, or demand arrives in bursts. A cluster with low allocation and high compute use among the allocated devices is healthy and may simply need to grow. A cluster with high allocation and high compute use is busy and efficient, and its only problem is queueing.

Each pattern has a different remedy, and the wrong remedy wastes effort. Sharing technologies such as MIG and time-slicing help the first pattern. They do nothing for the second, where the fix is to release or consolidate hardware, or to schedule bursty work onto spare capacity. Autoscaling that adds more GPUs makes the first pattern worse.

Before changing anything, write down two numbers for your cluster: the share of GPUs allocated, and the average compute use of the allocated ones. The pair tells you which problem you have.

What the Five Percent Figure Does and Does Not Say

The Cast AI report is a large and useful dataset, and precision about it makes your own analysis stronger.

It covers production Kubernetes clusters, which skews toward cloud-native teams and away from traditional high-performance computing sites. It measures compute activity as reported by device telemetry, and a figure of 5 percent says that on average the arithmetic units were busy a small fraction of the time. It does not say how much capacity sat unassigned, and the report's commentary explicitly cautions against reading it as 95 percent immediately recoverable.

The report adds two related findings. Fewer than 2 percent of GPUs in the sampled enterprise clusters ran on spot or preemptible capacity, despite substantial discounts for spot instances. And AWS raised H200 Capacity Block pricing by 15 percent in January 2026, which the report describes as the first GPU price increase in roughly two decades.

What the number does support is a narrower and still important statement: in typical clusters, GPUs are acquired in units of whole devices and used at a small fraction of their capacity, and the gap between a well-tuned fleet at 49 percent and the average is large. Your own cluster could be at 3 percent or at 40 percent. Measure it.

Why a Whole GPU Goes to a Small Job

Several causes combine, and most are not technical errors.

Kubernetes treats a GPU as an indivisible integer. The default device plugin advertises each physical device as one unit of the resource named nvidia.com/gpu, and a pod that requests one gets exclusive ownership of the whole device for its lifetime. A pod that needs 8 gigabytes of an 80-gigabyte H100 holds all 80. Other workloads that need a slice find none available, and wait, even when the device's compute use is under 10 percent.

Habits add to it. Notebooks and interactive environments are started and left running overnight. A training job that finishes at midnight leaves its reservation in place until someone releases it. Teams reserve for peak demand, the way they reserve conference rooms, and the reservation sits unused between peaks. Inference endpoints sized for a traffic spike run mostly idle, and a model that needs 7 gigabytes of memory is given a whole device because that is the unit on offer.

Procurement adds a structural cause. Capacity is bought in large blocks, sometimes months ahead, and committed spending creates an incentive to use or hold the capacity once it is acquired. A team that releases a GPU may not get one back when it needs it.

None of these causes is solved by buying faster GPUs. Each is a matter of how the devices are carved up, requested, scheduled and released.

What 5 Percent Does to the Bill

If you pay for a GPU by the hour and use 5 percent of its compute, each hour of useful compute costs twenty hours of rent. The arithmetic is trivial and sobering, and it is a useful number to bring to a budget meeting.

def cost_per_useful_hour(price_per_hr, util):
    return price_per_hr / util

PRICE = 250.0       # illustrative rupees per GPU-hour, replace with your own rate
for u in (0.05, 0.20, 0.49):
    print(u, round(1 / u, 1), "x", round(cost_per_useful_hour(PRICE, u)))
# 0.05 -> 20.0 x, 5000 rupees per useful hour
# 0.2  ->  5.0 x, 1250
# 0.49 ->  2.0 x,  510

Average compute use	Cost multiple on useful work	Rupees per useful GPU-hour at an illustrative 250 per hour
5 percent	20 times	5,000
20 percent	5 times	1,250
49 percent	2 times	510

The hourly price in the example is a placeholder, so substitute your own, but the multiples hold for any price. Moving from 5 percent to 20 percent cuts the cost of a unit of useful work by 75 percent, with no change in hardware or price. Compute use cannot reach 100 percent in practice, since workloads have idle gaps, data loading and communication, and 40 to 60 percent is excellent for most fleets.

The point is the shape of the curve. The first improvements are cheap and large. The last are expensive and small.

Measure First: What to Collect and Where

You cannot improve what you only guess. The measurements needed are modest, and the tooling is standard.

NVIDIA's Data Center GPU Manager, with its Prometheus exporter, publishes per-device metrics: compute utilisation, memory used and total, streaming-multiprocessor activity, power draw, temperature, and clock speeds. The GPU Operator for Kubernetes installs the exporter along with the drivers and device plugin, and labels nodes with GPU features.

Collect allocation separately, from the scheduler: which pods hold which GPUs, for how long, owned by which team. Join the two by node and device.

One known gap deserves planning. Device-level telemetry gives the whole device's utilisation, not the share attributable to each workload when several share one. For per-workload attribution on shared GPUs you need extra instrumentation, such as framework-level metrics from the application, or the partition-level metrics that MIG exposes.

Alerts can start simple. The rule below flags a GPU that has been allocated to a pod for six hours while averaging under 15 percent compute use, which catches forgotten notebooks and over-provisioned endpoints.

groups:
  - name: gpu-waste
    rules:
      - alert: GpuAllocatedButIdle
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[6h]) < 15 and on(gpu, instance) kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0
        for: 6h
        labels: {severity: info}

Treat the alert as an invitation to ask the owner a question, not as a trigger for automatic termination. A GPU that sits idle between periodic jobs may be doing exactly what its owner intends.

The Four Fixes, in the Order to Try Them

Improvement comes from four levers, and the cheapest should be tried first.

Lever	What it does	Effort	Typical effect
Right-size and release	Stop holding devices that nothing uses, set time limits on notebooks, shut down idle endpoints	Low	Large on allocation
Pack	Change scheduler scoring from spreading to packing, so partly used nodes fill before new ones are added	Low to medium	Fewer nodes needed
Share	Run several workloads on one GPU with MIG, time-slicing or MPS	Medium	Large for small workloads
Schedule	Queue batch work, use preemptible capacity, scale idle pools down	Medium to high	Large for bursty demand

The first lever needs no new technology. It is a policy: leases on interactive GPUs, an owner on every reservation, and a weekly report. The second is a configuration change. In Kubernetes, switching the scheduler's resource scoring from the default that spreads pods across nodes to the strategy that prefers the most-allocated node concentrates workloads, which lets idle nodes be removed.

Sharing is where most of the technical interest lies, and the next sections cover it. Scheduling is where larger organisations eventually end up, with queues, priorities and fair-share rules across teams.

MIG: Hardware Slices With Real Isolation

Multi-Instance GPU partitions one physical GPU into several smaller, isolated GPUs at the hardware level. Each instance has dedicated compute units, its own memory and its own memory bandwidth and cache, so a noisy neighbour cannot starve another instance. It is available on A100, A30, H100 and H200 parts, and on some newer and workstation-class GPUs, with up to seven instances per device.

On an 80-gigabyte H100, the commonly listed profiles run from seven slices of about 10 gigabytes to a single full-device instance.

Profile	Instances per GPU	Memory per instance	Suits
1g.10gb	7	10 GB	Embedding models, small classifiers, light inference
2g.20gb	3	20 GB	Mid-size models, notebooks
3g.40gb	2	40 GB	Larger models, fine-tuning of small models
7g.80gb	1	80 GB	Full device

A short selector shows how the choice follows from model memory. It adds 20 percent headroom for the cache and runtime, then picks the smallest profile that fits.

H100_PROFILES = [("1g.10gb", 10, 7), ("2g.20gb", 20, 3), ("3g.40gb", 40, 2), ("7g.80gb", 80, 1)]

def pick_profile(model_gb, headroom=1.2):
    need = model_gb * headroom
    for name, gb, count in H100_PROFILES:
        if gb >= need:
            return name, count
    return None

for m in (3, 7, 16, 30, 60):
    print(m, pick_profile(m))   # 3 and 7 GB -> 1g.10gb x7; 16 -> 2g.20gb x3; 30 -> 3g.40gb x2; 60 -> 7g.80gb x1

A pod then requests a slice with a resource such as nvidia.com/mig-1g.10gb, and the node that served one inference endpoint now serves seven. At an illustrative 250 rupees for the full device, that is about 36 rupees per slice-hour.

MIG has costs. The partition layout is set per GPU and changing it generally requires draining workloads from that device. A model that does not fit a slice cannot use it, and the slices are fixed sizes, so a 12-gigabyte model occupies a 20-gigabyte slice and wastes the difference. Peer-to-peer communication between slices is limited, so multi-GPU training does not suit it. And the hardware list is restricted to the supported generations.

Time-Slicing: Software Sharing With No Isolation

Time-slicing is the simplest method, and it works on any NVIDIA GPU. The device plugin is configured to advertise several virtual replicas of each physical device, and the GPU's driver switches between the processes that use them. One guide reports configurations of up to 48 replicas, though practical numbers are far smaller.

version: v1
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4

With that configuration, a node with one physical GPU advertises four, and four pods can each request one. They take turns on the device's compute.

The costs are the mirror image of MIG. There is no memory isolation: all the pods share the device's memory, so one pod's allocation can push another into an out-of-memory failure. There is no performance isolation: a busy neighbour stretches everyone's latency, and tail latency becomes unpredictable. And the replica count is a promise to the scheduler, not a resource guarantee, so four replicas does not mean four equal shares.

For workloads that tolerate that, such as development notebooks, small batch jobs and low-traffic internal endpoints, it is a cheap and effective way to raise utilisation. For latency-sensitive production inference, it needs care or a different tool.

MPS and Combining the Methods

NVIDIA's Multi-Process Service is a third approach, which lets several processes submit work to the GPU concurrently, so kernels from different processes can run at the same time instead of taking turns. It can raise throughput for many small workloads, with weaker isolation than MIG, and its behaviour depends on the driver and the framework. Evaluate it where several small inference processes share a device and you can afford the shared fate of a fault.

The methods combine. A MIG partition can itself be time-sliced, so a GPU split into seven slices with four replicas each presents 28 schedulable units. That density suits large numbers of small workloads, such as per-user notebooks, at the price of both methods' costs: fixed slice sizes and no isolation within a slice.

Match the sharing method to the damage a neighbour can do. Use hardware slices where a noisy neighbour is unacceptable, and software sharing where it is merely annoying.

When Shared GPUs Misbehave: Symptoms and Likely Causes

Sharing introduces failures that whole-device allocation never showed. A short table of symptoms saves hours of diagnosis.

Symptom	Likely cause	First thing to try
Pods on one device crash with out-of-memory errors	Time-slicing has no memory isolation, so one pod's allocation starves another	Set explicit memory limits in the framework, or move to MIG slices
Latency spikes at irregular intervals	A neighbouring pod is running a heavy job on a time-sliced device	Separate latency-critical pods onto their own slice or device
A MIG layout change does not apply	Workloads are still running on the device	Drain the node, then reconfigure
A pod requests a slice that does not exist	The node's MIG profile does not match the resource name	Check the node label and the MIG manager's state
Utilisation graphs look wrong on shared devices	Device-level telemetry cannot attribute usage to individual pods	Add framework-level or partition-level metrics
A model loads on one slice and not another	Slice memory differs, and the model plus cache exceeds it	Use the selector from this post, with headroom

Two operational habits help. Test every change to the sharing configuration on one node before rolling it out, because a mistaken device-plugin configuration can make a whole node's GPUs unschedulable. And keep driver, device plugin and operator versions pinned and updated together, since mismatches are a common cause of devices that vanish from the scheduler after an upgrade.

Sharing converts a hard failure, no device available, into soft ones, such as a slow neighbour or a memory squeeze. Plan the monitoring for the soft failures before you turn it on.

Which Method for Which Workload

Workload	Suggested approach	Why
Interactive notebooks	Time-slicing, or small MIG slices, with idle timeouts	Bursty, tolerant of delay
Small-model inference with latency targets	MIG slices	Isolation keeps tail latency stable
Large model serving	Whole GPUs or several GPUs, with an inference server that batches	Memory and bandwidth dominate
Distributed training	Whole GPUs, gang-scheduled	Communication needs full devices and fast links
Batch scoring and embedding jobs	Time-slicing or queued full GPUs on spare capacity	Throughput matters more than latency
Mixed team environment	A shared pool with quotas, plus MIG for the smallest tenants	Fairness and density together

The decision depends on what a neighbour can do to you. Isolation and predictability push toward MIG, density and flexibility toward time-slicing, and scale and performance toward whole devices.

Inference Servers Already Pack the Work

Before slicing a GPU for a language model service, check what the inference server does. Modern servers batch requests continuously, adding new requests to a running batch and managing the key-value cache in blocks, so a single process serves many users and keeps the device busy. The single-process utilisation of a well-configured server under real load can be far higher than that of many small separate processes.

That means sharing is not always the answer for large models. The better lever is to give the server enough concurrent requests to batch, to size its cache and to route traffic to the busiest instances. Our hardware sizing guide shows the memory arithmetic that decides how many conversations fit.

Sharing earns its place for the long tail of small models and light workloads, each too small to fill a device alone.

On-Premise Clusters: The Same Problem With Fewer Escape Hatches

An owned cluster cannot scale to zero overnight, and the hardware is a sunk cost, so the economic argument shifts from paying less to doing more with what you own. Utilisation is then the measure of return on the purchase.

Three practices carry most of the weight. Run a shared pool with quotas and fair-share scheduling, so that teams borrow idle capacity and return it when the owner needs it. Report usage back to the teams that consume it, showing each team its allocation, its compute use and its idle time, which changes behaviour faster than any technical measure. And put batch and evaluation work in queues that fill the gaps left by interactive and production load.

An on-premise cluster for sensitive data also has fewer ways to burst. If peak demand exceeds the pool, work waits, so the scheduling policy decides who waits. Write that policy down and agree it before the first contention.

A Composite: Forty GPUs, Six Teams

This is a representative composite, genericised, from several enterprise clusters. The numbers are illustrative.

A company owned forty data-centre GPUs, split evenly by team across six groups. Each group reserved its share. The finance team saw the invoices for the hardware and asked why the research group queued for a GPU when the dashboard showed the cluster mostly idle.

The two numbers explained it. Allocation was 92 percent, since every device was reserved, and compute use among allocated devices averaged about 11 percent. Reservations were the problem. A GPU held by a group was unavailable to others even when it sat idle at night.

The fix proceeded in stages. First, interactive notebooks got time limits and a daily release, which freed about a quarter of the devices in a week at no cost. Second, the six fixed shares became one pool with guaranteed minimums and borrowing, scheduled with a packing strategy. Third, eight GPUs were partitioned into seven slices each for the notebook and small-model workload, giving fifty-six small units where eight whole devices had been. Fourth, a nightly queue absorbed evaluation and embedding jobs.

Two things went wrong. A model served on a time-sliced device suffered occasional latency spikes from a neighbouring batch job, and the team moved it to a dedicated MIG slice. And a researcher's training run was preempted by the pool's fairness rules and lost hours of work, which led to mandatory checkpointing for any job that could be preempted.

Compute use among allocated GPUs rose from about 11 percent to the mid-30s over two months, and the research queue disappeared without any purchase. The finance team's question became how many GPUs the company needed next year, which is a better question than why they were idle.

Showback: Making Usage Visible to the Teams That Cause It

Nothing changes utilisation faster than showing each team its own numbers. The technique is called showback when it only reports, and chargeback when it bills, and for a first year, showback is enough.

A good weekly report is one page per team. It lists the GPUs the team held, the hours they held them, the average compute use and memory use while held, and the number of hours idle for more than a threshold. It names the owner of every reservation and flags those with the lowest use. It shows the team's cost at an agreed internal rate, even if no money moves.

Report field	Why it matters	Who acts
GPU-hours held	The size of the reservation	Team lead
Average compute use while held	How hard the hardware was worked	Workload owner
Hours idle beyond a threshold	Forgotten notebooks and idle endpoints	Workload owner
Peak memory use	Whether a smaller slice would fit	Platform engineer
Internal cost	Makes the waste concrete	Finance and team lead

Two cautions. Do not publish a league table that shames teams, because the reaction will be to hold on to capacity defensively. And do not set a single target for everyone, since a training team and an inference team have different healthy ranges.

What a team sees about its own usage changes behaviour within weeks. The report is the cheapest optimisation tool in this post.

Preemptible Capacity for Batch GPU Work

Batch jobs that can restart, such as evaluation runs, embedding generation and some training, can use cheaper preemptible or spot capacity, where the provider can reclaim the machine at short notice. The Cast AI report found that fewer than 2 percent of GPUs in its sampled enterprise clusters ran on spot, even though discounts of well over half are reported by the major providers.

The reason is risk. A job that loses hours of work when its machine disappears costs more than it saves. The remedy is engineering discipline: write checkpoints at a regular interval, make the job resumable from the last checkpoint, handle the termination notice by saving state, and let a scheduler restart it elsewhere.

Choose workloads for preemptible capacity by how cheaply they can resume. A job that checkpoints every ten minutes loses at most ten minutes per interruption. A job that cannot checkpoint belongs on stable capacity. Whatever you choose, record the interruption rate for a month before relying on the savings in a budget.

When Sharing Is the Wrong Fix

Sharing does not help in several cases.

A latency-critical production service with a strict tail target should not share a device without isolation, and even with MIG should be tested under realistic neighbour load.

A training job that scales across many GPUs needs whole devices with fast interconnect, and slicing would break its performance.

A workload limited by memory bandwidth, such as large-model decoding, gains little from sharing compute and may lose from contention.

A multi-tenant environment with strict security separation may need physical isolation beyond what slicing offers, and should be reviewed with security staff.

And if the underlying problem is overbuying, sharing only hides it. If sustained compute use among allocated devices stays below your target after consolidation, the answer is to hold fewer GPUs.

What to Tell Finance: Turning Utilisation Into a Budget Line

Utilisation improvements matter to a company only when they reach the budget, so the engineering case needs a translation into the numbers finance uses.

Start with the spend: monthly GPU cost, whether rented or depreciated, split by team or workload type. Add the two measured numbers for each, allocation and compute use. The product of cost and the share of time the hardware was doing useful work gives the useful spend, and the rest is overhead you can attack.

Then show three scenarios in plain terms. The first holds hardware constant and raises compute use through sharing and scheduling, which shows more work done for the same spend. The second holds work constant and returns hardware, by cancelling reservations or not renewing a purchase, which shows spend reduced. The third shows avoided purchases: the next batch of GPUs deferred by a quarter because the existing pool now serves the queue.

Be honest about the uncertainty. State the measurement period, the assumptions behind the estimate and the risks, such as latency effects from sharing, the engineering time to change configurations and the time to rebuild confidence in the numbers. A modest, defended estimate is more credible than a large one, and finance will test the largest claims first.

Finally, attach a date. A promised saving with no review date vanishes into the budget. Agree a ninety-day review where the two numbers are re-measured and compared with the baseline.

Engineers speak in utilisation and finance speaks in rupees. Translate once, with the arithmetic shown, and the discussion changes from permission to follow-up.

A Four-Week Plan

1. Week one: measure. Install the exporter, collect allocation from the scheduler, and compute the two numbers per team and per workload type. Find the owners of the top ten idle reservations.

2. Week two: policy. Add leases and idle timeouts to interactive GPUs, name an owner on every reservation, and publish a weekly report.

3. Week three: pack and share. Change scheduler scoring to packing, pilot MIG on a few devices for small inference and notebooks, and pilot time-slicing where neighbours are tolerant.

4. Week four: schedule. Set up a queue for batch and evaluation work, define preemption rules with mandatory checkpoints, and set targets for both numbers.

Review the numbers monthly. Utilisation drifts back down when nobody watches it.

Back to Five Percent

The figure that opened this post is a prompt, not a verdict. Your cluster may be better or worse, and it will be improved by a sequence of unglamorous steps: measuring allocation and use separately, releasing what nobody uses, packing what remains, sharing the small workloads and queueing the rest.

Hardware is the expensive part of an AI platform, and the arithmetic of utilisation decides whether that expense is spent on computation or on waiting. Write down your two numbers this week.

Related service: GPU Optimisation and Audits

EurthTech is an engineering-first product design and embedded systems company with a development center in Hyderabad. To discuss a project, write to connect@eurthtech.com or see https://www.eurthtech.com/case-studies.