Model Quantisation in Practice: Eight Objections to INT8 and INT4, Tested
Every team that moves a neural network onto small hardware hears the same warnings. Quantisation destroys accuracy. INT4 is unusable. Per-channel scaling is overkill. Calibration data does not matter. Clip the outliers and everything will be fine.
Some of these are true, some are half true and some are the opposite of the truth, and the only way to find out which is to measure. This post does that. It sets out eight objections, each of which I have heard from capable engineers, and tests them with a small, runnable harness. For each, you get the measurement, a verdict and what to do about it.
Two honesty notes first. The harness uses a small classifier trained on a standard dataset of handwritten digits, not a vision model for cameras, because it runs in seconds on any laptop and every number in this post can be reproduced from the code. A small network is better behaved than a large one, so the results show the mechanisms and understate the pain you may meet in a real model with convolutions, normalisation layers and attention. Where a result depends on that, I say so. Second, the accuracy figures come from a test set of 629 samples, so one sample is 0.16 percentage points. Differences of one or two samples are noise, and I treat them that way.
The Harness
The harness trains a network with two hidden layers on the digits dataset, then simulates quantisation in software. Weights and activations are rounded to a chosen number of bits and converted back to floating point, an approach called fake quantisation. It reproduces the rounding and clipping error of integer arithmetic and not the speed.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True); X = X / 16.0
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35, random_state=0, stratify=y)
clf = MLPClassifier(hidden_layer_sizes=(64, 32), activation="relu", max_iter=400, random_state=0).fit(Xtr, ytr)
W = [w.copy() for w in clf.coefs_]; B = [b.copy() for b in clf.intercepts_]
def qsym(t, bits, axis=None, clip=None):
qmax = 2 ** (bits - 1) - 1
amax = np.max(np.abs(t), axis=axis, keepdims=axis is not None) if clip is None else np.percentile(np.abs(t), clip)
scale = np.maximum(amax, 1e-12) / qmax
return np.clip(np.round(t / scale), -qmax - 1, qmax) * scale
def forward(Xin, wbits=None, abits=None, per_channel=False, calib=None, act_clip=None):
Wq = [qsym(w, wbits, axis=0 if per_channel else None) for w in W] if wbits else W
scales = None
if abits and calib is not None: # activation ranges come from the calibration set
scales, hc = [], calib
for i, (w, b) in enumerate(zip(Wq, B)):
hc = hc @ w + b
if i < len(Wq) - 1: hc = np.maximum(hc, 0)
scales.append(np.percentile(np.abs(hc), act_clip) if act_clip else np.max(np.abs(hc)))
h = Xin
for i, (w, b) in enumerate(zip(Wq, B)):
h = h @ w + b
if i < len(Wq) - 1: h = np.maximum(h, 0)
if scales is not None:
qmax = 2 ** (abits - 1) - 1; s = scales[i] / qmax
h = np.clip(np.round(h / s), -qmax - 1, qmax) * s
return h
def acc(Xin, yin, **kw):
return round(float((forward(Xin, **kw).argmax(1) == yin).mean()) * 100, 2)
print(acc(Xte, yte)) # 98.09, the floating-point baseline
The network has 6,570 parameters. In floating point they occupy 26,280 bytes, in 8-bit integers 6,570 bytes and in 4-bit integers 3,285 bytes, the memory saving of four and eight times that makes the exercise worthwhile. The baseline accuracy is 98.09 percent.
Reading a Result: Noise, Samples and Honest Error Bars
Every accuracy figure in this post carries uncertainty, and quantisation studies are especially prone to over-reading small differences. A test set of 629 samples can distinguish large effects and cannot distinguish small ones.
The standard error of an accuracy measured on a finite test set follows from the binomial distribution. The 95 percent margin around an accuracy p measured on n samples is about 1.96 times the square root of p times one minus p, divided by n.
import math
def margin95(p, n):
return round(1.96 * math.sqrt(p * (1 - p) / n) * 100, 2)
def n_needed(p, half_width_pts):
return math.ceil(1.96 ** 2 * p * (1 - p) / (half_width_pts / 100) ** 2)
print(margin95(0.9809, 629)) # 1.07 points either way
print(n_needed(0.98, 0.5)) # 3012 samples for plus or minus 0.5 points
print(n_needed(0.98, 0.2)) # 18824 samples for plus or minus 0.2 points
Test samples | 95 percent margin at 98 percent accuracy |
629 | About 1.1 points |
2,000 | About 0.6 points |
10,000 | About 0.3 points |
The consequence is uncomfortable. On 629 samples, the true accuracy of a model that scored 98.09 could lie anywhere from about 97.0 to 99.2, and a quantised model scoring 97.93 is not distinguishable from it. Resolving a difference of 0.2 points needs nearly nineteen thousand samples.
Two practices help. Compare models on the same test samples and count the cases where they disagree, since a paired comparison is far more sensitive than comparing two separate accuracies: if the two models disagree on three samples and the quantised one is right in one of them, you have no evidence of a difference. And scale the test set to the decision. If a half-point loss would change your choice, you need a few thousand samples, and if only a two-point loss would, a few hundred suffice.
A quantisation result without a sample size is an anecdote. State the test set, count the disagreements and say what difference it could and could not detect.
Objection 1: "INT8 Always Costs Accuracy"
The measurement. Quantise to 8 bits, and compare.
print(acc(Xte, yte, wbits=8), acc(Xte, yte, wbits=8, per_channel=True)) # 97.93 98.09
rng = np.random.RandomState(1)
cal = Xtr[rng.choice(len(Xtr), 128, replace=False)]
print(acc(Xte, yte, wbits=8, abits=8, per_channel=True, calib=cal)) # 98.09
print(acc(Xte, yte, wbits=8, abits=8, per_channel=False, calib=cal)) # 97.93
Setting | Accuracy | Change from 98.09 |
8-bit weights, one scale per tensor | 97.93 | One sample worse |
8-bit weights, one scale per channel | 98.09 | None |
8-bit weights and activations, per channel | 98.09 | None |
8-bit weights and activations, per tensor | 97.93 | One sample worse |
Verdict. False for a well-behaved model. The 8-bit versions are within a single test sample of floating point, and the per-channel version matches it exactly. That is consistent with widespread experience that 8-bit post-training quantisation is usually close to lossless for networks that are not unusual.
What to do. Treat INT8 as the default, and measure. Be careful with the word usually. Models with wide-ranging channels, which the fifth objection exposes, can behave very differently.
INT8 is the default for a reason. The mistake is assuming the result without measuring it on your own model and your own data.
Objection 2: "Per-Channel Scaling Is Overkill"
The measurement. Compare one scale for a whole weight tensor with one scale per output channel, at 4 bits, and then repeat on a model in which two hidden units have a wide range of weights.
The second model is made by a function-preserving transformation: multiply the incoming weights and bias of two hidden units by 12 and divide their outgoing weights by 12. Because the network uses a rectifier, the output of the network is unchanged, so its floating-point accuracy is the same 98.09 percent. Only the way the numbers are spread has changed. This mimics what happens in real networks, where some channels carry much larger values than others, as in depthwise convolutions and layers that follow normalisation.
print(acc(Xte, yte, wbits=4), acc(Xte, yte, wbits=4, per_channel=True)) # 96.98 97.93
W0, W1, B0 = W[0].copy(), W[1].copy(), B[0].copy()
for j in (3, 17):
W0[:, j] *= 12; B0[j] *= 12; W1[j, :] /= 12
W_orig, B_orig = [w.copy() for w in W], [b.copy() for b in B]
W[0], W[1], B[0] = W0, W1, B0
print(acc(Xte, yte)) # 98.09, the function is preserved
for bits in (8, 4):
print(bits, acc(Xte, yte, wbits=bits), acc(Xte, yte, wbits=bits, per_channel=True))
# 8: 97.77 98.09 4: 29.41 97.77
W[:], B[:] = W_orig, B_orig
Model | Bits | One scale per tensor | One scale per channel |
Original | 4 | 96.98 | 97.93 |
Two wide-range channels | 8 | 97.77 | 98.09 |
Two wide-range channels | 4 | 29.41 | 97.77 |
Verdict. Fine until it is catastrophic. On the original model, per-channel scaling gains about a point at 4 bits. With two wide-range channels, per-tensor quantisation at 4 bits collapses to 29 percent, while per-channel holds at 97.8. The reason is simple: one scale must cover the widest channel, so the narrow channels are rounded to a handful of levels, or to zero.
What to do. Use per-channel scales for weights wherever your hardware and toolchain support them, which is the norm for convolution and linear layers, since the extra storage is one scale per output channel. If a runtime forces per-tensor scales, check the layers for wide-range channels and consider equalising them before quantisation.
Objection 3: "INT4 Is Unusable"
The measurement. Quantise weights to 4 bits per channel, and then add 8-bit activations.
Setting | Accuracy | Change from 98.09 |
4-bit weights, per channel, floating-point activations | 97.93 | One sample worse |
4-bit weights, per channel, 8-bit activations | 98.09 | None |
Verdict. False for weights on this model. Four-bit weights with per-channel scales cost at most one sample, and with 8-bit activations the result matches floating point. That halves the model size relative to INT8.
What to do. Consider 4-bit weights where memory or memory bandwidth limits you, which is common for language models and for large vision models on small devices, and verify per layer. Real networks vary: some layers, often the first and last, are more sensitive and are commonly kept at higher precision. Check the cost on your task and not on this one.
Objection 4: "Activations Are as Easy to Quantise as Weights"
The measurement. Hold the weights at 8 bits and reduce the activations to 4 bits, then compare with reducing the weights to 4 bits.
print(acc(Xte, yte, wbits=4, per_channel=True)) # 97.93
print(acc(Xte, yte, wbits=8, abits=4, per_channel=True, calib=cal)) # 96.34
print(acc(Xte, yte, wbits=4, abits=4, per_channel=True, calib=cal)) # 96.18
Change | Accuracy | Loss from floating point |
Weights to 4 bits | 97.93 | 0.16 points |
Activations to 4 bits | 96.34 | 1.75 points |
Both to 4 bits | 96.18 | 1.91 points |
Verdict. False. Activations are the harder tensors. Their ranges depend on the input and so vary from one sample to the next, they have no per-channel structure fixed in advance, and a few large values stretch the scale. Weights are known and fixed when you quantise them.
What to do. Treat weight and activation precision as separate decisions. A common and sound pattern is low-bit weights with 8-bit activations. When you reduce activation precision, test it explicitly, since it is where accuracy first starts to move.
Objection 5: "Clipping Outliers Always Helps"
The measurement. Set the activation range at the maximum seen in calibration, or at the 99.5th percentile, at 4 bits.
print(acc(Xte, yte, wbits=8, abits=4, per_channel=True, calib=cal)) # 96.34 with the maximum
print(acc(Xte, yte, wbits=8, abits=4, per_channel=True, calib=cal, act_clip=99.5)) # 95.55 with the 99.5th percentile
Verdict. Not always. Percentile clipping, a standard recommendation, gave 95.55 percent against 96.34 for the plain maximum, a loss of five samples. The idea is sound: clipping a few extreme values makes the scale finer for everyone else. Here, though, the extreme values carried information, and clipping them cost more than the finer scale gained.
What to do. Treat the range method as a setting to test. Try the maximum, a percentile or two, and a method that minimises the error directly, and choose by measured accuracy on your own held-out data. A rule of thumb is a hypothesis, and a model is a fact.
Objection 6: "Calibration Set Size Doesn't Matter" (and Its Opposite)
Some say a handful of images is enough. Others insist on thousands. The measurement draws calibration sets of different sizes at random, twenty times each, quantises activations to 6 bits and records the spread in accuracy.
Calibration samples | Mean accuracy | Worst of 20 draws | Spread (standard deviation) |
1 | 97.76 | 96.82 | 0.37 |
4 | 97.99 | 97.62 | 0.20 |
16 | 98.01 | 97.62 | 0.15 |
64 | 97.92 | 97.62 | 0.14 |
512 | 97.80 | 97.46 | 0.13 |
Verdict. Both camps are partly right. The mean barely moves after about sixteen samples, so a large set is not needed for the average. But a single sample gives a worst case of 96.82 against 97.62 and a spread nearly three times larger, so a tiny set exposes you to an unlucky draw. Past a few dozen, extra samples bought nothing here, and the 512-sample mean was slightly lower, a difference within the noise.
What to do. Use a few dozen to a few hundred calibration samples, drawn to represent your real inputs, and run the calibration more than once with different draws to see the spread. If the spread is large, your model is sensitive to the range and needs a better method.
Objection 7: "Calibrate on Any Data"
Two experiments test whether the calibration data needs to resemble the deployment data.
First, class coverage. Calibrate using only digits 0 to 4, and test on digits 5 to 9, which the calibration set never contained.
Activation bits | Test on classes 5 to 9, calibrated on all classes | Calibrated on classes 0 to 4 only |
8 | 97.45 | 97.45 |
6 | 97.13 | 97.13 |
4 | 92.68 | 92.68 |
The results are identical. Here, missing classes made no difference, because the largest activations occurred in the classes the calibration set did contain. That will not hold in every model.
Second, a lighting shift. Add a bright offset to every image, as glare would, calibrate on normal images, on glare images or on a mix, and test on glare.
Xglare = np.clip(Xte + 0.35, 0, 1)
print(acc(Xglare, yte)) # 70.27 in floating point
Activation bits | Calibrated on normal | Calibrated on glare | Calibrated on a mix |
8 | 69.95 | 69.16 | 69.63 |
6 | 68.52 | 68.84 | 68.20 |
4 | 65.34 | 67.25 | 66.45 |
Verdict. Mixed. Matching the calibration data to the deployment helps at 4 bits, by almost two points, and does little or nothing at 6 and 8 bits, where the 8-bit result was slightly better with normal data. The effect is real and small compared with the next objection.
What to do. Calibrate on data that includes the conditions you will meet, especially at low bit widths. And note that this test could not show what many engineers fear most, a night-versus-day failure, because this model happens to be insensitive to a brightness scaling. A real vision network with normalisation and biases may behave differently, so repeat the test on yours.
Objection 8: "Test Accuracy Tells Me What Will Happen in the Field"
The measurement. The glare experiment above already contains it. In floating point, the model's accuracy on normal test images is 98.09 percent and on glare images 70.27, a fall of 27.8 points. Quantisation, at its worst in that table, lowers the glare result from 70.27 to 65.34, a further 4.9 points.
Verdict. False, and this is the most important result in the post. The change in conditions cost the model about six times more accuracy than the most aggressive quantisation did. Teams spend weeks recovering a few tenths of a point from quantisation while their deployment data differs from their test data by an amount that dwarfs it.
What to do. Test the model in floating point on data from the deployment, not only the benchmark, before you quantise. Find out how it fails under the lighting, the angle, the dirt and the camera noise you will meet. A quantisation study on a model that fails in the field is a polishing exercise.
Quantisation error is usually smaller than distribution-shift error. Fix the larger one first.
Finding the Layers That Hurt: Per-Layer Sensitivity
A single number for the whole network hides where the damage happens. Quantising one layer at a time, and leaving the rest in floating point, shows which layers tolerate fewer bits and which do not.
def acc_layer(layer, bits, per_channel=False):
Wq = [w.copy() for w in W]
Wq[layer] = qsym(W[layer], bits, axis=0 if per_channel else None)
h = Xte
for i, (w, b) in enumerate(zip(Wq, B)):
h = h @ w + b
if i < len(Wq) - 1:
h = np.maximum(h, 0)
return round(float((h.argmax(1) == yte).mean()) * 100, 2)
for bits in (4, 3, 2):
print(bits, [acc_layer(i, bits) for i in range(3)])
# 4: [97.62, 97.46, 97.77] 3: [96.18, 97.3, 96.5] 2: [33.86, 56.12, 69.48]
Bits | Layer 1 only | Layer 2 only | Layer 3 only | All layers, per channel |
4 | 97.62 | 97.46 | 97.77 | 97.93 |
3 | 96.18 | 97.30 | 96.50 | 92.85 |
2 | 33.86 | 56.12 | 69.48 | 50.72 |
At 4 bits, each layer alone costs under a point, and the differences among them are within the noise. At 3 bits the first layer, which sees the raw input, is the most fragile at 96.18, and the last layer follows. At 2 bits every layer collapses, the first worst of all. The pattern, with the first and last layers most sensitive, matches common practice of keeping those two at higher precision while pushing the middle layers lower.
The last column makes a second point. At 3 bits, quantising all layers costs more than the sum of the single-layer effects suggest, because errors compound through the network. The per-layer table is a guide to where to spend bits, and the whole-network measurement is still the one that matters.
A practical sensitivity study takes minutes: for each layer, quantise it alone at the target precision, measure, and rank. Keep the most sensitive layers at higher precision, and give the rest the lowest precision that holds the accuracy you need.
What the Harness Cannot Tell You
A fair reading of the results needs the limits stated.
The model is tiny and shallow. Errors accumulate through depth, so deep networks are more sensitive than the two-hidden-layer network here. The network has no convolutions, normalisation layers or attention. Depthwise convolutions and transformer layers are known to produce channels and tokens with very large ranges, which is exactly the situation of the second objection and is why they are often the hard cases in practice.
The harness simulates rounding and clipping, not the integer hardware. Real accelerators add their own details: how they accumulate, how they requantise between layers, which operations run in which precision and which fall back to higher precision. A model that is accurate in simulation can lose accuracy on a device that does something slightly different.
Accuracy is a single number. A change that costs 0.2 points overall may concentrate the loss in one class that matters to you. Look at per-class results, and at the failures.
When Post-Training Quantisation Is Not Enough
Everything above quantises a finished network, which is called post-training quantisation. It is quick, needs only a small calibration set and works well down to 8 bits and often to 4 bits for weights. Below that, or for sensitive models, it runs out.
Quantisation-aware training goes further. It simulates the rounding during training, so that the weights learn to sit where rounding hurts least. The network sees the quantised values in its forward pass and passes gradients through the rounding as if it were the identity, a trick called the straight-through estimator. The result is a network adapted to its target precision, usually recovering most of the accuracy that post-training quantisation loses.
Approach | Data needed | Effort | Typical use |
Post-training, weights only | None or a few samples | Minutes | First attempt, memory savings |
Post-training with calibrated activations | Dozens to hundreds of samples | Minutes to hours | The default for 8-bit deployment |
Quantisation-aware training | The training set, or a large part | Hours to days | Low-bit activations, sensitive models, accuracy-critical products |
Use training-aware methods when post-training results are not good enough and the accuracy matters, and when you have the training data and pipeline to run it. They need the original training code, which is not always available for a purchased or third-party model, and they add a step to every retraining cycle.
Try post-training quantisation first, because it is cheap. Reach for quantisation-aware training when the numbers say you must, and plan for it in the training pipeline from the start.
What Fewer Bits Buy: Memory, Bandwidth and Speed
The case for quantisation rests on three savings, and they do not always arrive together.
Memory is the sure one. A model in 8 bits occupies a quarter of its floating-point size, and in 4 bits an eighth, as the 6,570-parameter network showed. For large models, that decides whether they fit on a device at all.
Memory bandwidth is the second. Inference on small devices is often limited by how fast weights can be read, not by arithmetic, so fewer bytes per weight can speed up a model even if the arithmetic is no faster. This effect is large for language models, where each generated token reads every weight.
Compute speed is the third, and the least reliable. It depends on the hardware having integer units for the chosen precision. Many accelerators run 8-bit arithmetic natively and at a fraction of the energy of floating point. Support for 4-bit arithmetic is patchier, and where it is missing, the weights are stored in 4 bits and expanded to 8 bits before use, saving memory without saving arithmetic.
Saving | 8-bit | 4-bit | Depends on |
Model size | One quarter of floating point | One eighth | Nothing, it is arithmetic |
Memory bandwidth | Up to four times fewer bytes | Up to eight times | Whether inference is bandwidth bound |
Arithmetic speed | Often large on integer hardware | Hardware dependent, sometimes none | The accelerator's supported precisions |
Energy | Usually lower | Usually lower again, if natively supported | Native support |
Check what your target hardware actually does before choosing a precision. Run the quantised model on the device, measure speed, memory and energy, and compare with the simulation. A precision the hardware does not support natively may give a small file and no speed-up.
Operators That Resist Quantisation
Some operations are harder to quantise than the matrix multiplications that dominate a network, and toolchains often keep them in higher precision.
Operator | Why it is hard | Common treatment |
Softmax | Exponentials produce a huge dynamic range, and small errors in large values change the result | Compute in higher precision, or use a lookup table |
Layer and batch normalisation | Divide by a statistic that can be near zero, amplifying error | Fold into neighbouring layers when possible, else keep higher precision |
Attention scores | Large outlier values in a few channels or tokens | Per-channel or per-token scales, clipping, or higher precision |
Activation functions such as GELU | Smooth non-linear curves poorly approximated by coarse steps | Lookup tables, or higher precision |
Residual additions | Two tensors with different ranges are added | Align the scales, or keep the sum wider |
First and last layers | Raw inputs and final scores | Often left at 8 bits or higher |
Real models are mixtures. A well-quantised network rarely uses one precision throughout: it runs the heavy matrix products in the lowest precision that holds accuracy and keeps the delicate operations wider. When a toolchain reports that some layers fell back to higher precision, that is usually the right outcome and not a failure.
Quantise the heavy arithmetic aggressively and the delicate operations gently. A model with a few wide layers is a good model.
Questions to Ask a Toolchain or Accelerator Vendor
The quantisation scheme is only as good as the toolchain that applies it and the hardware that runs it. Before committing to a target, ask the questions that decide whether your results will carry over.
Question | Why it matters | A good answer |
Which precisions run natively, and which are emulated | Emulated precision saves memory and not time | A table of supported operations by precision |
Are weights quantised per channel or per tensor | Decides the second objection in this post | Per channel for convolutions and linear layers |
How are activation ranges set, and can I override them | The range method changes accuracy | Several methods, with user control |
Which operators fall back to higher precision | Determines speed and accuracy | A report listing every fallback layer |
How closely does the simulator match the device | A model can pass in simulation and fail on hardware | A stated tolerance, and a way to compare outputs layer by layer |
What happens to unsupported operators | A single unsupported layer can force a split to the host | A clear error or an automatic fallback with a cost estimate |
Is the quantised model reproducible across toolchain versions | Updates can shift accuracy | Version pinning and a regression test |
Ask for a trial with your own model before you decide. A vendor's benchmark on a standard network says little about how your network's unusual layers will be handled.
Deploying a Quantised Model: Versions, Regression Tests and Rollback
A quantised model is a different artefact from the floating-point model it came from, and it deserves its own release process.
Keep the whole recipe with the model: the floating-point checkpoint, the calibration set, the range method, the per-layer precision choices, the toolchain version and the target hardware revision. Without it you cannot reproduce the result, and the next toolchain update may change the accuracy without anyone noticing.
Build a regression test around a fixed set of inputs, including your hardest cases, and require that the quantised model's accuracy and its per-class results stay within stated limits on every release. Compare the device's outputs with the simulation's on the same inputs, so that a change in the toolchain or the hardware shows up as a difference in numbers and not as a complaint from a customer.
Plan the rollback. A new quantised model deployed to a fleet should be reversible, with the previous version kept on the device or available to download, and it should be rolled out in stages, with the field accuracy monitored against the test result.
The quantised model is the product. Version its recipe, test it against a fixed set on every release, and keep the previous one ready to return to.
A Practical Quantisation Routine
Measure the floating-point model on deployment-like data, including the bad conditions.
Quantise weights to 8 bits per channel, activations to 8 bits, with calibration from a few hundred representative samples, and measure.
If memory forces it, reduce the weights to 4 bits per channel, layer by layer, keeping sensitive layers at higher precision.
Test the activation range method, maximum and a percentile or two, and keep the best by measurement.
Run the calibration several times with different draws and record the spread.
Look for wide-range channels, and use per-channel scales or equalise them where the runtime requires per-tensor.
Run the quantised model on the target hardware, compare with the simulation and check per-class results.
Keep the calibration set, the settings and the results with the model version, since the next update repeats the exercise.
Verdicts at a Glance
Objection | Verdict | One-line advice |
INT8 always costs accuracy | False for this model | Default to INT8 and measure |
Per-channel is overkill | Fine until catastrophic | Use per-channel weights |
INT4 is unusable | False for weights | Try 4-bit weights, per layer |
Activations are as easy as weights | False | Reduce activation precision last |
Clipping outliers always helps | Not always | Test the range method |
Calibration size does not matter | Half true | A few dozen samples, several draws |
Calibrate on any data | Mixed | Match deployment, especially at low bits |
Test accuracy predicts field accuracy | False | Test on field conditions first |
Measure Before You Believe
Every objection in this post is a rule of thumb someone once found true on some model. Some survive the test and some do not, and none of them tells you what will happen on yours.
The harness takes seconds to run and can be pointed at your own network. Run the eight tests, keep the results beside the model, and spend the saved argument on the number that mattered most: how the model behaves on the data it will actually see.
Related service: Industrial Edge Vision Systems
Related reading: Jetson Orin vs Hailo-8: Choosing an Edge AI Accelerator for Vision Products, Local LLM Hardware Requirements: How Much VRAM and Memory Bandwidth You Need and GPU Utilisation Explained: Why Clusters Sit at 5 Percent and How MIG and Time-Slicing Help.
EurthTech is an engineering-first product design and embedded systems company with a development center in Hyderabad. To discuss a project, write to connect@eurthtech.com or see https://www.eurthtech.com/case-studies.
Comments