Local LLM Hardware Requirements: How Much VRAM and Memory Bandwidth You Need
How much memory does a 70-billion-parameter model need?
Ask three engineers and you will hear 40 gigabytes, 52 gigabytes and 140 gigabytes, and all three can be right. The first is thinking of the weights compressed to about four and a half bits each. The second has added the working memory for one long conversation. The third is quoting the original sixteen-bit release. The difference between the numbers is the difference between a card you can buy for a workstation and a pair of data-centre accelerators.
This is a field guide to getting that number right for your own case, and then to the second number that matters more than most buyers expect: how fast the model will talk once it fits. The arithmetic is short, the code is below, and everything is checked against hand calculations so you can replicate it with a calculator and a model card.
The Question Has Four Answers, and Buyers Usually Hear One
Four separate things occupy accelerator memory when a language model is running.
The weights come first. These are the model's learned parameters, and their size is the parameter count multiplied by the bits stored per parameter. Quantisation, which stores each weight in fewer bits, shrinks this and is the reason large models run on modest hardware at all.
The key-value cache comes second. During generation the model keeps the attention keys and values for every token it has seen, for every layer. This grows with the length of the conversation and with the number of conversations running at once, and it is the part buyers forget.
Third is activation and framework overhead: buffers, kernels, the runtime itself. A safe planning figure is a couple of gigabytes plus a margin, because the last few percent of memory is rarely usable.
Fourth is concurrency. One user at a time is a very different problem from twenty, because the cache is per conversation while the weights are shared.
Every quote you receive that says a card "runs a 70B model" has quietly chosen values for these four. Your job is to choose them yourself first.
Weights: Parameters Times Bits, Plus the Quantisation You Choose
The formula fits on one line: weight memory in bytes equals parameters times bits per weight divided by eight. A model with 8 billion parameters at 16 bits needs 16 gigabytes. At 8 bits it needs 8. At the roughly 4.5 effective bits of a typical four-bit quantisation scheme, which stores a little extra for scaling factors, it needs about 4.5.
Model class | 16-bit (FP16) | 8-bit (FP8 or INT8) | About 4.5-bit quantised |
8 billion parameters, dense | 16 GB | 8 GB | 4.5 GB |
32 billion parameters, dense | 64 GB | 32 GB | 18 GB |
70 billion parameters, dense | 141 GB | 71 GB | 40 GB |
235 billion parameters, mixture of experts | 470 GB | 235 GB | 132 GB |
Quantisation costs some quality. For many tasks a well-made four-bit model is hard to tell apart from its sixteen-bit parent, and for others, such as careful numeric reasoning or tight code generation, the gap is visible. The only reliable way to know is to run your own evaluation set at both settings before you buy hardware around the smaller one.
Decide the quantisation level with your own test set before you choose the hardware. The hardware follows the model, and the model follows the quality bar.
The KV Cache Is What Breaks the Budget
The key-value cache holds two tensors, keys and values, for every layer and every token. Its size per token is twice the layer count, times the number of key-value heads, times the head dimension, times the bytes per element. Modern models use grouped-query attention, which keeps the number of key-value heads small, and that is the only reason this is manageable.
For a typical 70-billion-parameter model with 80 layers, 8 key-value heads and a head dimension of 128 at 16-bit precision, each token costs 327,680 bytes, which is 320 kibibytes. A single user with a 32,000-token context therefore holds about 10.7 gigabytes of cache. Eight users holding such contexts together hold 86 gigabytes, more than twice the size of the quantised weights themselves.
A smaller model behaves the same way at a smaller scale. An 8-billion-parameter model with 32 layers, 8 key-value heads and head dimension 128 costs 128 kibibytes per token, so an 8,000-token context is about 1.07 gigabytes.
def weights_gb(params_b, bits_per_weight):
return params_b * 1e9 * bits_per_weight / 8 / 1e9
def kv_cache_gb(layers, kv_heads, head_dim, context_tokens, bytes_per_elem=2, batch=1):
per_token = 2 * layers * kv_heads * head_dim * bytes_per_elem
return per_token * context_tokens * batch / 1e9, per_token
def decode_ceiling_tps(bandwidth_gbs, active_weights_gb):
# each generated token reads every active weight once
return bandwidth_gbs / active_weights_gb
w70 = weights_gb(70.6, 4.5) # 39.7 GB
kv70, per_tok = kv_cache_gb(80, 8, 128, 32768) # 10.7 GB at 32k, 320 KiB per token
print(round(w70, 1), round(kv70, 1), per_tok // 1024)
print(round(w70 + kv70 + 2.0, 1)) # 52.4 GB for one user, with 2 GB overhead
Run that and you have the three numbers from the opening: 39.7 gigabytes for the weights, 52.4 gigabytes for the weights plus one long conversation plus overhead, and 141 gigabytes if you insisted on the sixteen-bit release. Different questions, different answers.
Two practical consequences follow. Cache can itself be quantised, to 8 bits for roughly half the memory at a small quality cost, and runtimes increasingly offer that. And context length is a product decision with a hardware price. Letting every user attach a 100-page document is a decision to buy more memory.
Bandwidth Sets Speed, Capacity Sets Whether It Runs at All
Generation happens one token at a time, and producing each token requires reading every active weight from memory once. That makes the speed of generation, for a single stream, bounded above by memory bandwidth divided by the bytes read per token. This is a ceiling, not a prediction. Real systems reach a fraction of it.
Hardware class | Memory | Published bandwidth | 8B Q4 ceiling | 70B Q4 ceiling | Fits 70B Q4 with 32k context? |
Workstation GPU, 32 GB class | 32 GB GDDR7 | 1,792 GB/s | about 398 tokens/s | about 45 tokens/s | No, weights alone are 40 GB |
Workstation GPU, 96 GB class | 96 GB GDDR7 | 1,792 GB/s | about 398 tokens/s | about 45 tokens/s | Yes |
Desktop AI box, 128 GB unified | 128 GB LPDDR5X | 273 GB/s | about 61 tokens/s | about 7 tokens/s | Yes |
High-end desktop, unified memory | Up to 512 GB | 819 GB/s | about 182 tokens/s | about 21 tokens/s | Yes |
Laptop-class unified memory | Up to 128 GB | 546 GB/s | about 121 tokens/s | about 14 tokens/s | Yes |
The bandwidth figures are the manufacturers' published numbers. The ceilings come from the formula above using 4.5 gigabytes for the 8-billion-parameter model and 39.7 for the 70-billion one.
Read the table in two passes. Capacity first: which rows can hold the model at all. Then bandwidth: among those that can, how fast will it speak. The desktop AI box and the 96-gigabyte workstation card both hold a 70-billion-parameter model. The card generates at several times the speed because its memory is faster by a factor of about six.
Third-party benchmarks published in 2026 show how far real systems sit below the ceiling. One comparison from October 2026 reported roughly 150 tokens per second for an 8-billion-parameter four-bit model on the 32-gigabyte card and roughly 38 on the desktop AI box. Against the ceilings of 398 and 61, those are about 38 percent and 62 percent. Another review reported single-digit tokens per second for a 70-billion-parameter model on the unified-memory box, against a ceiling of about 7. Treat these as indicative, since software versions, context length and batch settings change them, and run your own test on the machine you intend to buy.
Capacity decides whether the model loads. Bandwidth decides whether anyone wants to use it. A box can win one and lose the other.
Mixture-of-Experts Models Change the Arithmetic
Mixture-of-experts models store many specialist sub-networks but activate only a few for each token. A model described as 235 billion parameters with 22 billion active holds all 235 billion in memory, 132 gigabytes at the four-and-a-half-bit level, but reads only about 22 billion parameters' worth per token, about 12.4 gigabytes.
The consequence is a mismatch that suits unified-memory machines. Capacity must hold the full 132 gigabytes, which rules out any single 96-gigabyte card. Speed behaves like a 22-billion-parameter dense model, since that is what each token reads, so a high-capacity, moderate-bandwidth machine can run a very large mixture-of-experts model at a usable speed where a dense model of the same stored size would crawl.
The caution runs the other way. Expert routing adds overhead, and not every runtime handles it well. Memory for a mixture-of-experts model is a function of total parameters, not active ones, so buying hardware by the active figure is a reliable way to be disappointed.
Prefill and Decode Are Different Jobs
When a prompt arrives, the system first processes all of its tokens in parallel. That phase is called prefill, and it is limited by arithmetic throughput, not memory bandwidth. After that comes decode, the token-by-token generation described above, which is limited by bandwidth.
The distinction explains why two machines can rank in opposite orders depending on the workload. A box with strong arithmetic throughput and weak bandwidth reads a long document quickly and then types slowly. A box with strong bandwidth and modest arithmetic types fast and reads long prompts more slowly.
Which one hurts depends on the application. A retrieval system that stuffs 6,000 tokens of retrieved passages into every question is prefill-heavy, and the time to first token governs the experience. A drafting assistant that takes a short instruction and writes two pages is decode-heavy. Measure both numbers, time to first token and tokens per second after that, on a prompt shaped like yours.
Six Machines, One Consistent Entry Each
Each entry below follows the same pattern: what the machine is, where its strength lies, where it struggles, and when to reach for it. Prices have moved by thousands of dollars during 2026 because of memory supply, so none are quoted here. Ask for current rupee quotes that include import duty and GST.
Workstation with a 32 GB consumer-class GPU
A desktop tower with one top-end consumer graphics card. Memory is 32 gigabytes at 1,792 gigabytes per second. It is the fastest option for models that fit: 7 to 32 billion parameters at four-bit quantisation, with a few concurrent users. It cannot hold a 70-billion-parameter model without offloading to system memory, which cuts speed sharply. The card draws up to about 575 watts, so plan the power supply and cooling. Reach for it when the task is small to mid-sized models for a small team, retrieval pipelines with embedding models alongside, or fine-tuning of small models.
Workstation with a 96 GB professional GPU
The same bandwidth, three times the memory, error-correcting memory, and a board power of roughly 600 watts. It holds a four-bit 70-billion-parameter model with room for long contexts, or a mid-sized model serving many users. It also costs several times as much as the 32-gigabyte card. Reach for it when one machine must serve a team with a large model and long documents, or when error-correcting memory is a requirement.
Desktop AI box with 128 GB unified memory
A compact machine where the processor and accelerator share one pool of 128 gigabytes at 273 gigabytes per second. It holds large models and mixture-of-experts models that no consumer card can load. It generates slowly, with the ceiling for a dense 70-billion-parameter model near 7 tokens per second. It is strong at prefill. Reach for it when capacity matters more than speed, for one or two users running large models, evaluation work, or development before moving to production hardware.
High-end desktop with Apple-class unified memory
Unified memory up to 512 gigabytes with bandwidth around 819 gigabytes per second on the largest desktop configuration. The balance of capacity and bandwidth is better than the cheaper unified-memory boxes, and it is quiet and efficient. The software ecosystem is different from the CUDA world, so serving frameworks and tooling need checking. Reach for it for single-user or small-team use of very large models where noise, power and size matter.
Mini PC with x86 unified memory
A small x86 machine with up to 128 gigabytes of LPDDR5X at about 256 gigabytes per second, with a large share assignable as graphics memory. It is the cheapest route to a big pool of memory. Real-world bandwidth is lower than the headline and software support is less mature. Reach for it for experiments and low-concurrency internal tools where budget dominates.
Data-centre GPU server
Accelerators with 48 to 80 gigabytes of fast memory per card, in servers with several cards, built for continuous multi-user serving with proper cooling and management. They are the right answer for dozens of concurrent users and for models too large for any single workstation card. They are overkill for a team of five.
Four Numbers Before You Talk to a Vendor
Write these down first. They turn a vendor conversation from a pitch into a specification.
Model: which model, at which quantisation, with the weight memory from the table.
Context: the longest conversation or document you will allow, which sets the per-user cache.
Concurrency: how many users need an answer in the same minute, not how many staff exist.
Latency: time to first token and tokens per second you need for the experience to feel acceptable.
A worked example. A 40-person team wants a document assistant. The model is a 32-billion-parameter dense model at four-bit quantisation, 18.3 gigabytes of weights. Context is capped at 16,000 tokens. For this model, with 64 layers, 8 key-value heads and head dimension 128, each token costs 256 kibibytes, so each user's cache is 4.3 gigabytes. Peak concurrency is estimated at ten users.
The memory budget is 18.3 gigabytes of weights, 43 gigabytes of cache for ten users, and 2 gigabytes of overhead, a total of about 63 gigabytes. With roughly 92 percent of memory usable, a 32-gigabyte card supports only two users at that context. A 48-gigabyte card supports five. A 96-gigabyte card supports fifteen, which covers the ten with room to spare. The single-stream ceiling for this model is about 98 tokens per second on 1,792 gigabytes per second of bandwidth, and about 15 on the 273-gigabyte-per-second box.
The answer to "which machine" falls out of the arithmetic: the 96-gigabyte workstation card, or two 48-gigabyte cards, and not the machine with the most headline memory.
A Composite: The Pilot That Fit on Paper
This is a representative composite, genericised, from several customer conversations. The numbers are illustrative.
A manufacturer planned an internal engineering assistant over its design documents and chose a 70-billion-parameter model because a vendor demo had impressed the director. The plan specified a single 32-gigabyte workstation card, justified by the line "supports 70B models". It did, with two compromises the demo had not mentioned: a very aggressive two-bit quantisation that damaged accuracy on technical questions, and offloading to system memory that dropped generation to a few tokens per second.
Week three of the pilot, engineers stopped using it. The answers were wrong on part numbers and slow enough to feel broken. The team measured. The weights at the quality they needed would not fit, and no tuning would change that.
The recovery was arithmetic, not heroics. They moved to a 32-billion-parameter model at four-bit quantisation, which fit on the existing card with room for two concurrent users, then added a second card in the second quarter when usage grew. Quality on their evaluation set was within a few points of the larger model because the questions were narrow and the retrieval was good.
The lesson the team wrote into their purchasing checklist was short: size the memory for the model at the quality you tested, plus the cache for the context and users you promised, and discard any quote that does not state them.
Serving Software Changes the Answer as Much as the Card Does
Two people with the same card can report very different speeds, and the reason is usually the serving software. The categories are worth knowing before you read any benchmark.
Single-user runtimes are built for one person on one machine. They load a quantised file, keep one conversation's cache and generate as fast as bandwidth allows. They are simple, excellent for development and personal use, and wasteful when several people share the machine, because they handle requests one after another or with crude batching.
Server-class runtimes are built for many users at once. Their main tricks are continuous batching, which lets new requests join a running batch instead of waiting for it to finish, and paged cache management, which stores each conversation's keys and values in small blocks so memory is not reserved for context that is never used. Together they raise the number of concurrent users a card can hold and the total tokens per second it produces, often by a large factor over naive serving.
Compiled engines go further for specific hardware, fusing operations and choosing kernels tuned to the card. They deliver more speed and cost more effort: a build step, version pinning, and less flexibility when you change models.
Two techniques shift the arithmetic itself. Quantising the cache to 8 bits roughly halves the per-user memory in the earlier tables, at a small quality cost that you should test. Speculative decoding uses a small fast model to propose several tokens, which the large model then verifies in one pass. When the proposals are good, you get more than one token for each read of the large model's weights, which pushes real speed above the single-stream ceiling that bandwidth alone suggests.
The bandwidth ceiling applies to one token per weight-read. Batching and speculation are the two honest ways to beat it, and both depend on the software you run.
The practical rule is to benchmark the combination you intend to deploy. A card bought on the strength of a single-user number and then run under a server-class runtime with ten users will behave differently from either figure.
Reading a Vendor Benchmark Without Being Misled
Benchmarks are the most persuasive and least comparable numbers in the whole buying process. Six checks catch most of the problems.
Claim in the benchmark | What it can hide | Question to ask |
"120 tokens per second" | Whether that is one user or the aggregate of sixteen | Per-stream or total, at what concurrency? |
"Runs a 70B model" | The quantisation level and whether weights spill to system memory | At which bits per weight, and fully in GPU memory? |
"Fast response" | Time to first token on a three-line prompt, not a real one | What prompt length, and what is the time to first token at that length? |
"128K context supported" | Support in the model, not speed or memory at that length | How many users fit at that context, and at what speed? |
"Same quality as the large model" | A narrow evaluation set | Which tasks, and can I run my own? |
"Measured on our reference system" | A different card, drivers or cooling | Can I test on the exact configuration I will buy? |
The commonest trap is the first row. Aggregate throughput, summed over many simultaneous users, can be many times the single-user figure, and it is the right number for a shared server. But a person typing a question feels per-stream speed. A system that produces 600 tokens per second in total across forty users gives each of them 15 tokens per second, which is readable but not snappy.
The second trap is quality hidden behind quantisation. A vendor who advertises a 70-billion-parameter model on a small card may be running it at two or three bits, which saves memory and can badly damage accuracy on exactly the technical questions you care about. Insist on knowing the bits, and on running your own questions.
The third is the prompt. Benchmarks that use short prompts flatter machines with weak arithmetic throughput. Real retrieval-augmented use sends thousands of tokens of context with every question. Time to first token on a six-thousand-token prompt is the number that predicts whether your users wait.
A fair test is cheap: rent a machine of the same class for a day, load your model, replay a hundred real prompts from your own documents at your expected concurrency, and record time to first token, tokens per second per user and total throughput. Quote those numbers in the purchase order.
Power, Cooling and a Mains Supply That Blinks
The specification sheets say 575 or 600 watts for the card. The whole machine under load draws more, perhaps 850 watts with the processor, memory, fans and power-supply losses. Idle may be around 150. Those are planning figures for a high-end single-card workstation, and your own measurement on your own machine beats them.
The energy bill is smaller than most people expect, because the machine spends much of its time waiting. The code below computes monthly energy from idle and load power and a duty cycle, then sizes an uninterruptible supply for a clean shutdown.
def monthly_kwh(idle_w, load_w, duty, days=30):
avg_w = idle_w + (load_w - idle_w) * duty
return avg_w * 24 * days / 1000, avg_w
def ups_va(load_w, power_factor=0.8):
return load_w / power_factor
def battery_wh(load_w, minutes, usable=0.7):
return load_w * minutes / 60 / usable
kwh, avg = monthly_kwh(150, 850, 0.30)
print(avg, round(kwh), round(kwh * 9)) # 360.0 W, 259 kWh, 2333 rupees at 9 per kWh
print(round(ups_va(850)), round(battery_wh(850, 10))) # 1062 VA, 202 Wh for a 10-minute shutdown
With a 30 percent duty cycle the machine averages 360 watts, uses about 259 kilowatt-hours a month, and costs roughly 2,300 rupees at an assumed tariff of 9 rupees per kilowatt-hour. That is modest next to the cost of the hardware. The UPS figures are more important: 850 watts at a power factor of 0.8 calls for a supply of about 1.1 kilovolt-amperes at minimum, and in practice a 1.5 kVA unit leaves headroom, with about 200 watt-hours of usable battery to carry ten minutes while the system shuts down cleanly.
Heat is the harder problem. A kilowatt of electrical load becomes a kilowatt of heat in the room. In a small office with a window air conditioner on a warm afternoon, with ambient temperatures near 40 degrees outside, a machine in a closed cupboard will throttle or shut down. Give it ventilated space, keep its intake filters clean, and check the temperature log in the first summer.
The mains supply is the third item. Brownouts, surges and brief interruptions are routine in many places, and a power cut during a model write or an index rebuild can corrupt files. A UPS with automatic shutdown, a surge protector, and a habit of storing the model files and index on storage that tolerates sudden loss are cheap protections.
Plan the electrical and cooling arrangements before you order the machine. A card that throttles in April is a slow card, whatever its specification says.
When a Local Box Is the Wrong Answer
A local machine loses to rented capacity in several common situations.
If usage is occasional, the card sits idle. A rented GPU at an hourly rate for a few hours a week costs less than the depreciation of a workstation.
If you need the strongest model available and the data is not sensitive, a hosted model beats anything you can run locally today, and the gap is real.
If your premises cannot support it, a 575 to 600 watt card in a hot room is a cooling and power problem. In an Indian summer, ambient temperatures of 40 degrees change the cooling design, and a machine on an unreliable mains supply needs a proper uninterruptible supply and a clean shutdown path. Model weights and indices are not forgiving of sudden power loss.
If nobody will keep it patched, it becomes a liability. Drivers, firmware and serving software need updates, and a machine nobody owns drifts out of date.
And if the real constraint is data control, hardware is only part of the answer. The network and pipeline around the machine decide whether the data stays where you think it does.
Monday Morning: A Sizing Worksheet
Write the four numbers: model and quantisation, maximum context, peak concurrency, latency target.
Compute weights with the formula, then the per-user cache from the model's published configuration of layers, key-value heads and head dimension.
Add 2 gigabytes of overhead and divide by 0.92 to find the memory class you need.
Compute the single-stream speed ceiling for each candidate machine from its published bandwidth.
Test on a rented machine of the same class with your own prompts, and record time to first token and tokens per second.
Check power, cooling and software support at your site.
Request quotes in rupees, with import duty and GST, and ask the vendor to state the four numbers they assumed.
The Number on the Box Is Not the Number You Need
Back to the question that opened this guide. A 70-billion-parameter model needs 40 gigabytes, or 52, or 141. The right figure depends on the bits you store, the conversations you hold and the people who use it at once.
Do the sums before you shop. They take ten minutes and they decide whether the hardware you buy runs the model you wanted, at a speed people will tolerate.
Related service: On-Premise LLM and AI Servers
EurthTech is an engineering-first product design and embedded systems company with a development center in Hyderabad. To discuss a project, write to connect@eurthtech.com or see https://www.eurthtech.com/case-studies.
Comments