Air-Gapped AI: How to Run a Private LLM With No Internet Connection
A network cable lying unplugged on a desk is the most reassuring object in information security, and the least informative. It tells you one thing: this machine cannot talk to the internet through that port. It says nothing about the second port, the wireless card nobody disabled, the USB stick that arrived on Tuesday, or the software that was installed last month from a package index nobody checked.
An air-gapped AI deployment is a private language model that runs entirely inside a network with no path to the outside world, so that the model, the questions people ask it and the documents it reads never cross a security boundary. It is the strictest form of private AI, and it is demanded by a particular set of buyers: defence and aerospace programmes, intelligence and law-enforcement work, critical infrastructure, some banks and exchanges, and design offices holding unreleased product drawings.
It is also easy to do badly. The failure is rarely the model. It is a dependency that quietly phones home, a model file nobody verified, an update process that nobody designed, or a team that believed the gap was a wall when it was a process.
This guide is a field guide to doing it properly. It defines what an air gap is, lays out the four zones every deployment needs, then walks through six working entries in the same format: acquire, verify, transfer, serve, update and audit. Each entry says what it is, how to do it, how it fails and what to check before moving on. The verification code runs, and the places where I rely on public sources or on memory are marked so you can check them.
What an Air Gap Is, and Three Levels People Confuse
The phrase is used loosely, and the looseness is where deployments go wrong. At least three different arrangements are called air-gapped.
Level | What it means | What can still go wrong |
Network-isolated | The AI system sits on a network with no route to the internet, but is reachable from other internal networks | Any compromised internal machine becomes a bridge to the AI system and its data |
One-way link | Data can flow in through a controlled one-way device, such as a data diode, and cannot flow out | The one-way device is a single point of trust, and what enters can still be malicious |
True air gap | No electrical or radio path of any kind between the AI network and any other network, and data moves only on inspected physical media | The media, the people and the supply chain remain the ways in |
Most commercial offerings described as air-gapped are the first level. That is often adequate, and sometimes exactly right, because the cost of the third level is high. The honest first step is to decide which level your data actually requires, write it down, and have your security authority sign it.
Three tests help. The route test asks whether any machine in the AI network can reach any address outside it, by any path including a cellular modem, a wireless card or a management interface on the server. The media test asks whether anything enters the network without being inspected, hashed and logged. And the people test asks who can move media, and whether two people are needed to do it.
An air gap is a procedure with a cable removed, not a cable removed. The cable stops the network. Only procedure stops the media, the people and the supply chain.
The Four Zones: Low Side, Gold Environment, Transfer and High Side
Every working air-gapped AI deployment, however it is labelled, has four zones. Naming them is the easiest way to see where responsibility lies.
The low side is an ordinary internet-connected environment where artefacts are downloaded. It is untrusted by definition. Nothing on it is allowed to touch the protected network directly.
The gold environment is a controlled staging area, often a small isolated network or a set of dedicated machines, where downloaded artefacts are verified, scanned, assembled into bundles and signed. It is where trust is created. A deployment without one is a deployment that trusts whatever the low side fetched.
The transfer zone is the physical or one-way path between the gold environment and the protected network: inspected removable media, a data diode, or a guarded transfer station. It is the most audited step in the process.
The high side is the protected network itself, where the model runs, where users work and where the sensitive documents live.
Zone | What lives there | Who has access | What must never happen |
Low side | Internet downloads, public model hubs, vendor sites | Engineers with internet access | A low-side machine connected to the gold or high side |
Gold environment | Verified artefacts, bundles, signing keys, scan results | A small, named group | Unsigned or unscanned items passed onward |
Transfer | Inspected media, one-way link, custody log | Two authorised people | Media used in both directions without wiping |
High side | Inference servers, document stores, user workstations | Cleared users | Any route outward, or any import not signed in the gold zone |
The signing keys deserve their own mention. The private key that signs bundles should live in the gold environment, preferably in hardware, with a documented procedure for who may use it. The public key lives on the high side and is the only thing that decides whether an incoming bundle is accepted.
Entry 1: Acquire
What it is. Everything the system needs is collected on the low side before the gap closes: the model weights, the tokeniser files, any embedding and re-ranking models, the inference server and its dependencies, the operating-system packages, drivers, container images and the web interface, if you use one.
How to do it. Make a list first, called a bill of materials, before downloading anything. For each item record its name, version, source, licence and, once downloaded, its SHA-256 hash. Models need extra care. Prefer weights in the safetensors format, which cannot execute code on load, over older pickle-based checkpoints, which can. Record the licence text alongside the weights, since the terms of use travel with the model.
Pin everything. A model referred to by a name such as latest is a different object next month. Container images should be referred to by digest, never by tag. Packages should be locked to exact versions with hashes.
The accepted name for a bill of materials covering models and datasets is an ML bill of materials, and CycloneDX publishes a format for it. It does not need to be elaborate. A table in a controlled repository is better than a perfect format nobody maintains.
How it fails. The commonest failure is an implicit download. A package installs a second package, the second one fetches a model at first run, and the model fetches a tokeniser from a public hub. None of it appears in your list, and none of it is verified. OWASP's 2025 list of large-language-model risks names supply chain as a primary risk class for exactly this reason. Another failure is trusting a model hub's conversion or mirroring service, which has been shown in research to be an attack surface.
Check before moving on. Run the whole system on the low side with the network restricted to your approved sources, and confirm that it works and that nothing was fetched that is not on the list.
Entry 2: Verify
What it is. Verification turns a pile of downloaded files into a bundle whose contents you can prove. It has two parts: a hash for every file, and a signature over the list of hashes.
How to do it. Compute SHA-256 for each file, write the results into a manifest, and sign the manifest with a private key held in the gold environment. On the high side, the public key checks the signature first and the hashes second. A bundle that fails either check is rejected whole.
The code below does this with the Python standard library for hashing and the widely used cryptography package for Ed25519 signatures. It also runs three attacks against its own bundle: a changed byte, an extra file, and a forged manifest.
import hashlib, json
from pathlib import Path
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.exceptions import InvalidSignature
def sha256_file(p):
h = hashlib.sha256()
with open(p, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
def build_manifest(root):
root = Path(root)
files = {str(p.relative_to(root)): {"sha256": sha256_file(p), "bytes": p.stat().st_size}
for p in sorted(root.rglob("*")) if p.is_file() and p.name != "MANIFEST.json"}
return {"version": 1, "files": files}
def canon(m):
return json.dumps(m, sort_keys=True, separators=(",", ":")).encode()
def verify_bundle(root, manifest, sig_hex, pub):
try:
pub.verify(bytes.fromhex(sig_hex), canon(manifest))
except InvalidSignature:
return ["signature invalid: do not import"]
root, problems = Path(root), []
for name, meta in manifest["files"].items():
p = root / name
if not p.exists():
problems.append("missing " + name)
elif sha256_file(p) != meta["sha256"]:
problems.append("hash mismatch " + name)
on_disk = {str(p.relative_to(root)) for p in root.rglob("*") if p.is_file() and p.name != "MANIFEST.json"}
problems += ["unexpected file " + e for e in sorted(on_disk - set(manifest["files"]))]
return problems
# gold side: priv = Ed25519PrivateKey.generate(); sig = priv.sign(canon(manifest)).hex()
# high side holds only priv.public_key(); three tests on fresh bundles gave:
# clean bundle -> []
# one byte changed -> ['hash mismatch models/weights.safetensors']
# extra file added -> ['unexpected file run.sh']
# forged manifest entry -> ['signature invalid: do not import']
How it fails. The commonest failure is verifying on the same machine that downloaded the files, so that a compromised download is faithfully hashed and signed. Verification only has value if the hashes are made from a trusted copy, and the key stays out of the low side. Another failure is signing the bundle but not recording who approved it, which leaves the signature proving nothing about review.
A further gap is worth stating plainly. A hash proves a file is the same file you hashed. It does not prove the file was safe when you hashed it. Cryptographic signing of model weights at the publisher level is not yet common practice, so your own scanning and review in the gold environment are the real check.
Check before moving on. Tamper with a copy of the bundle in each of the three ways above and confirm that your process rejects all three, in the same way that it will on the day.
Entry 3: Transfer
What it is. The movement of a signed bundle from the gold environment to the high side. It is the most audited step in the whole process, because it is the only door.
How to do it. Use dedicated removable media for the purpose, labelled, serial-numbered and never used for anything else. Write the bundle on the gold side, on a machine that is not connected to anything else. Take the media to the transfer station, a standalone machine with no network, mounted read-only. Let it check the signature and hashes automatically, scan for malware, and write a transfer receipt to the custody log with the time, the bundle identifier, the hash of the manifest and the names of the two people present. Only then does the content move onward to the high side.
Two-person control matters more than it sounds. Most incidents with media involve a single trusted person making a well-meant exception.
When the media comes back, wipe it, or destroy it. Do not reuse media in both directions, since returning media can carry something out as readily as in.
How it fails. Convenience wears procedures down. A system that takes forty minutes per transfer will acquire a shortcut within a year, such as a personal laptop used to copy a file. The fix is to make the procedure fast enough to follow, with a prepared station, a checklist and a rehearsed routine, and to log exceptions as incidents.
Check before moving on. Time a full transfer from signed bundle to running service, and then do it again with a second operator following only the written procedure. If the second operator needs help, the procedure is not yet written.
Entry 4: Serve
What it is. The running system on the high side: the inference server, the retrieval components, the web interface and the supporting services, all configured so that nothing inside can reach outside and nothing inside needs to.
How to do it. Serve the model with an inference server that can run without any outbound access, and set its configuration to forbid it. Disable every update checker, usage reporter and crash uploader. Configure the package manager so that the internal package index is the only resolvable index, so that an install that tries to fetch from anywhere else fails and the build stops. Pull container images only by digest from an internal registry.
A small script keeps the habit honest by checking that nothing in a deployment file refers to an image without a digest.
def unpinned(images):
return [i for i in images if "@sha256:" not in i]
deployed = [
"registry.internal/llm-server@sha256:9f2c0b7e1d4a",
"registry.internal/embedder:latest",
"registry.internal/ui@sha256:51ab77c0e3f2",
]
print(unpinned(deployed)) # ['registry.internal/embedder:latest']
On the network, deny all egress from the AI namespace by default, as the network policy in our earlier post on sovereign AI shows, and allow only the specific internal services each component needs. Run the policy test regularly: from inside the namespace, try to reach an outside address and confirm that the attempt fails.
How it fails. The failures are almost always dependencies that assume the internet. A web interface loads fonts or scripts from a public content network and renders badly or not at all. A vector database sends anonymous usage statistics. A licence check phones a vendor at start-up. A tokeniser library downloads a file on first use. Find them by running the system with outbound traffic blocked and logged, and read the log of every blocked attempt.
Check before moving on. Start the whole system from cold with all outbound traffic blocked, and ask a set of ten real questions. Everything should work, and the blocked-traffic log should show no attempts. Each attempt is a dependency to remove.
Entry 5: Update
What it is. The process by which the system changes after the day it was installed: new model versions, security patches for the operating system, drivers and libraries, and updates to the documents and indexes it searches.
How to do it. Treat every update as a small release. It starts on the low side with a downloaded change. It is tested in the gold environment on a copy of the production system. It is bundled, signed, transferred and installed during a planned window, with a rollback ready. And it ends with a log entry and a short note on what changed.
Decide the cadence before you need it. Security patches for the operating system and drivers may be monthly or quarterly. Model upgrades might be twice a year. Document indexes may update weekly. Each has its own bundle type.
Update type | Typical cadence | Risk if skipped | Extra check |
Operating-system and driver security patches | Monthly or quarterly | Known vulnerabilities stay open | Test the GPU stack after every change |
Inference server and libraries | Quarterly | Bugs and security fixes missed | Re-run the quality tests |
Model weights | Twice a year, or when a clear gain appears | Quality falls behind | Full evaluation on your own questions |
Document and index content | Weekly or on demand | Answers go stale | Spot-check retrieval after each load |
Certificates and keys | Before expiry | Services stop at a set date | Calendar the dates |
How it fails. The pattern is stale and brittle. Without a cadence, the system is never updated, and a year later it carries every unpatched vulnerability and an obsolete model. Or updates happen informally and nobody can say what is running. Pin and record versions so that the question what is running today can be answered in one minute.
Remember what the gap costs you. A cloud service improves silently. An air-gapped one stays where you left it. A model's knowledge and its quality freeze at the last transfer, and you have no fallback to a hosted service when something breaks, so the redundancy has to be built inside.
Check before moving on. Perform one complete dry-run update, from low-side download to production install and back to rollback, and time it. If it takes three weeks, your quarterly cadence is the best you can offer, and the plan should say so.
Entry 6: Audit
What it is. The record that lets you, an auditor or a regulator show what the system did and who did what to it.
How to do it. Log the administrative side and the usage side differently. Administrative logs record every transfer, every update, every access to the signing keys and every change of configuration, with the person and the time. Usage logs record who asked the system what, when, and which documents it consulted, within whatever privacy rules apply to the people using it. Keep the logs inside the boundary, on write-once or append-only storage, and with time kept by an internal clock source.
In India, check the CERT-In directions of April 2022, which require reporting of specified cyber incidents within six hours and retention of logs for 180 days within the country. The details are for your counsel to confirm, since they have been amended and interpreted since.
Keep an evidence folder with the artefacts an auditor will want: the bill of materials, the signed manifests, the custody log, the network policy and its test results, the update log, the access review and the record of your most recent kill-switch test.
How it fails. Logs fill the disk and stop. Clocks drift and make the records unreliable. Or logs record every prompt in full, which turns the audit trail into a store of exactly the sensitive text you were protecting. Decide what to record, how long to keep it and who may read it.
Check before moving on. Pick one question an auditor might ask, such as which model version answered a particular query last month and who approved its installation. Time how long it takes to answer from the logs alone.
Embeddings and Retrieval Inside the Boundary
A document assistant is usually two models, not one: a language model that writes the answers and an embedding model that turns documents and questions into vectors so that relevant passages can be found. Both must live inside the gap.
The embedding model is the one that causes the quiet leaks. In an earlier post on sovereign AI, a hosted embeddings service was found sending every uploaded page out of a supposedly private system. In an air-gapped design it is not a risk to manage but a defect to prevent. Choose an open embedding model, bring it in through the same signed process as the language model, and run it locally.
Everything that follows from retrieval is also local: the document ingestion pipeline that reads files, splits them into passages and stores vectors, the vector database itself, and the re-ranking model if you use one. Each is another thing to pin, hash and update.
Two design points matter. First, the quality of answers depends on retrieval more than on the size of the language model, so test retrieval on its own, with a set of questions whose correct source passages you know. Second, index versions need the same care as model versions. If you change the embedding model, every document must be re-embedded, and the old and new vectors cannot be mixed.
The language model gets the attention, and the retrieval pipeline decides the quality. Test them separately, and pin both.
Sizing Without Internet Benchmarks and Without a Fallback
Capacity planning changes when there is no cloud to burst to. A cloud deployment can rent more capacity for a spike. An air-gapped deployment has what you bought.
Size memory for the model at the quality level you tested, plus the cache for the conversation length and the number of simultaneous users, using the arithmetic in our hardware sizing guide. Then add margin, because you cannot add capacity next week.
Plan for failure inside the boundary. With no hosted fallback, you need a second inference server, or at least spare components and a rehearsed repair procedure. A single GPU server is a single point of failure for a service that people may come to depend on. Hold spares for the parts that fail: power supplies, fans, drives and, if you can justify it, a spare accelerator.
Vendor support also changes. A support engineer cannot connect to your system. Agree in advance how logs and diagnostics will be exported for them to read, through the same controlled transfer process, and how fixes will return.
Time, Certificates and Phone-Home Habits
Four small services decide whether an isolated system keeps working: time, certificates, names and licences.
Time comes first. Tokens, certificates and logs all depend on a correct clock, and a system with no internet has no public time source. Run an internal time server, preferably synchronised to a trusted reference you control, and monitor drift.
Certificates come next. Without public certificate authorities and automated renewal, you need an internal authority, a documented process for issuing and renewing certificates, and a calendar for expiry dates. A service that stops on the first of the month because nobody renewed a certificate is a very common outage.
Names matter too: internal DNS must resolve everything the system needs, and must not forward outside queries.
Licences are the fourth. Some software and some drivers expect to contact a vendor server to validate. Find out before purchase whether each component can run on an offline licence.
Phone-home suspect | How it shows up | What to do |
Package managers | Installs fail or hang | Point at the internal mirror, fail closed |
Model hubs and tokeniser downloads | First-run fetch errors | Pre-load files, set offline environment flags |
Web interface assets | Page renders poorly | Bundle fonts and scripts locally |
Usage statistics and crash reports | Blocked-traffic log entries | Disable in configuration, verify in the log |
Licence and update checks | Start-up delay or refusal | Obtain offline licences, disable update checks |
Time and certificate services | Cert errors, clock drift | Run internal time and certificate authority |
Threats That Survive the Gap
An air gap removes network attacks and leaves everything else. Four classes remain, and each has a practical response.
Malicious media is the oldest. Attackers who cannot reach a network try to reach the person who carries the media. Defences are the ones already described: dedicated media, scanning, one-way transfer stations, two-person control and wiping after use.
Poisoned artefacts are the newer. A model file or a library that is subtly altered, or a dataset that carries hidden instructions, enters through the same door as every legitimate update. Defences are the gold environment's scanning and review, pinned versions and a habit of reading the release notes of every update before it is approved.
Insiders are the hardest. People with access can copy data to media of their own. Defences are access control, logging, and a culture in which reporting a lost stick is rewarded.
Prompt injection through documents is specific to AI. A document placed in the retrieval index can contain text that tries to steer the model, such as an instruction to reveal other documents. Inside an air gap it cannot send data out, but it can still make the assistant behave badly for users. Defences include treating retrieved text as untrusted, limiting what the model's tools can do, and testing with adversarial documents.
The gap stops the network from being the way in. It turns attention to the media, the artefacts and the people, and that is where the defence has to be built.
A Composite: A Design Office and a Locked Room
This is a representative composite, genericised, from several secure-environment engagements. The numbers are illustrative.
A defence design office wanted an assistant that could answer questions about decades of internal technical reports and drawings, held in a locked room on a network with no outside connection. The security authority approved a private language model on the condition of a true air gap, with a documented transfer process.
The first deployment passed its tests on the low side, was bundled and signed, and went through the transfer. On the high side it failed three times in the first week.
The web interface refused to render properly. It loaded its fonts and scripts from a public content network, and the page hung for thirty seconds before falling back to defaults, which made it look broken. The fix was to bundle the assets locally and to add a test that loads the interface with all outbound traffic blocked.
The vector database logged connection errors every few minutes. It was trying to send anonymous usage statistics. That was a configuration switch, found only because the team read the blocked-traffic log.
Then the clocks drifted. Within two weeks, the high-side servers disagreed with each other by more than the tolerance of the authentication tokens, and users were logged out at random. The team added an internal time server and a drift alert.
The people side went better than expected, once the procedure was rehearsed. The first full transfer took about two hours. The fifth took forty minutes. The custody log became a point of pride, and the security authority asked for the same format on another system.
Quality was the larger concern in the end. After three months, the model's knowledge of new components was frozen. The team agreed a twice-yearly model update with a full evaluation on a set of two hundred questions drawn from the office's own work, and planned the schedule around the security authority's review calendar.
When an Air Gap Is the Wrong Choice
An air gap costs money, time and convenience, and it is the wrong answer in several situations.
If the data is not sensitive enough to justify it, a private deployment inside an ordinary protected network, with strict egress control, is cheaper and easier to keep current. The levels table above is the place to decide.
If you need the best current model for a task and the data can leave, a hosted service will outperform anything you can freeze inside a gap.
If you cannot staff it, an air-gapped system decays. Without a person who owns the transfer process, the update cadence and the evidence folder, it will become an unpatched, unreviewed island.
And if the real risk is insiders or the supply chain, spending the budget on the cable instead of on access control and artefact review misses the point.
Log every blocked outbound attempt and treat each one as a defect to remove. A quiet blocked-traffic log is the sign of a system that no longer needs the internet.
A Sixty-Day Plan
Week one: decide the isolation level for each data class, in writing, with your security authority.
Week two: name the owners of the gold environment, the signing keys, the transfer station and the evidence folder.
Weeks three and four: build the bill of materials and run the whole system on the low side with outbound traffic restricted. Fix every unexpected dependency.
Week five: build the gold environment, generate the signing keys and sign a first bundle. Test the three tamper cases.
Week six: write and rehearse the transfer procedure with two operators, and time it.
Week seven: install on the high side, run the cold-start test with all outbound traffic blocked, and read the blocked-traffic log.
Week eight: run a dry-run update with rollback, write the cadence into the plan and run the first kill-switch test.
Back to the Cable
The unplugged cable on the desk is where this guide started. It tells you one thing, and the rest is in the plan: the four zones, the signed bundles, the verified transfers, the dependencies found and removed, the updates rehearsed and the logs that answer an auditor's question.
Draw the zones. Sign the bundles. Block the traffic and read the log. Then decide, in writing, how much of the gap your data really needs.
Related service: Air-Gapped Edge AI Boxes
Related reading: Sovereign AI Infrastructure Explained for Indian Enterprises: What You Can Control, Secure Boot and Firmware Signing for Embedded Products: From Key Generation to Field Revocation and SBOM for Firmware: A Working Pipeline for the EU Cyber Resilience Act.
EurthTech is an engineering-first product design and embedded systems company with a development center in Hyderabad. To discuss a project, write to connect@eurthtech.com or see https://www.eurthtech.com/case-studies.
Comments