The OOM Killer Is Not Random: Diagnosing Memory Death on Edge Gateways

Srihari Maddula • Founder & Technical Lead, Eurth Techtronics Pvt Ltd 

Category: Firmware Architecture & RTOS

Estimated Reading Time: 5 min

An edge gateway running Linux — the tier between microcontroller sensor nodes and the cloud, typically the box actually doing local inference, protocol translation, and store-and-forward buffering — reboots overnight with no crash log, no panic trace, nothing in the application logs indicating what happened. The service just stops appearing, and a new process with a new PID shows up after supervisor restart. Nine times out of ten on field-deployed gateways, this is the OOM killer, and the reason it looks random is that the process it chose to kill is rarely the process that actually caused the memory pressure.

What the OOM Killer Actually Does

When the Linux kernel's memory allocator cannot satisfy a request and has exhausted its ability to reclaim memory through normal means — dropping caches, swapping, compacting — it invokes the out-of-memory killer as a last resort to prevent total system deadlock. The OOM killer's job is narrow and brutal: pick one process to terminate immediately, freeing its memory, and let the system continue. It does not ask which process is misbehaving. It computes a badness score — historically called oom_score, influenced by memory footprint, process priority (niceness), and a per-process adjustment value (oom_score_adj) — and kills whichever process scores highest. A process that simply uses a lot of memory legitimately, like a vision inference runtime with a loaded model, frequently has the highest badness score by default, regardless of whether it caused the pressure or was simply the largest innocent bystander when a different process leaked.

THE RULE:  The OOM killer doesn't kill the guilty process. It kills the process with the highest badness score, and those are frequently different processes.

Why This Looks Random on a Field Gateway Specifically

On a development machine, memory pressure is usually caused by something the developer is actively doing, and the causal chain is short and visible. On a field-deployed gateway running unattended for months, the causal chain is almost always longer and slower: a slow memory leak in a long-running MQTT client library that only manifests after tens of thousands of reconnect cycles, a log file growing without rotation until page cache pressure interacts badly with an application's working set, a buffer that grows unbounded during a network outage because store-and-forward logic queues messages faster than a recovered connection can drain them. None of these announce themselves. They accumulate quietly for days or weeks, then cross the OOM threshold at a moment that has nothing to do with when the actual leak started, and the kernel kills whatever happens to have the highest badness score at that moment — which is very often the largest, most important, most recently-restarted process on the box, not the small background service that's been silently leaking since boot.

Reading the Evidence After the Fact

The single most useful diagnostic step, and the one most frequently skipped because the failure looks like it left no trace, is checking dmesg or the kernel ring buffer for OOM killer invocation records, which persist there even when the application's own logs show nothing. An OOM kill event logs the triggering process, the victim process, and — critically — a full memory breakdown across all running processes at the moment of the kill. That breakdown is usually enough to identify the actual leak, because the leaking process shows up with an anomalously large resident set size relative to its normal footprint, even if it wasn't the process that got killed.

A gateway that reboots without an accessible dmesg history because logs aren't persisted across reboots is diagnosing memory failures blind, permanently. Persisting kernel logs to non-volatile storage, or shipping them off-device before rotation, is not optional hardening for a field-deployed system — it's the only way the eventual OOM event becomes diagnosable rather than just another unexplained reboot added to a growing pile.

Getting the Victim to Be the Culprit, on Purpose

Once the actual leaking process is identified through repeated dmesg analysis across multiple incidents, oom_score_adj becomes a genuinely useful tool rather than a workaround. Setting a strongly negative adjustment on a critical process that legitimately needs a large, stable memory footprint — the inference runtime, say — makes the kernel far less likely to select it as a victim even though its absolute memory use is high. Setting a positive adjustment on a process known to leak, until the leak itself is fixed, at least ensures the kernel kills the actual problem rather than an innocent bystander, which turns an unpredictable full-gateway failure into a predictable, supervised restart of one known-leaky component — a meaningfully different failure mode operationally, even before the underlying leak is patched.

This is a mitigation, explicitly not a fix. Using oom_score_adj to steer kills toward a known leaky process buys operational stability while the leak gets root-caused and patched; treating it as a permanent solution just means the leaking process gets killed and restarted indefinitely, which works until the leak rate outpaces the restart cycle or interacts badly with whatever state the process needed to persist across restarts.

THE RULE:  oom_score_adj should redirect a known problem toward a survivable failure, not replace fixing the problem.

Prevention, Not Just Diagnosis

A gateway that has never triggered the OOM killer isn't necessarily well-architected — it may just not have run long enough yet, or hit the right combination of conditions. Cgroup memory limits per service, set deliberately rather than left unbounded, turn an unpredictable system-wide OOM event into a predictable, contained failure of a single cgroup — the kernel can reclaim within that boundary rather than reaching for the global OOM killer at all in many cases. Log rotation with hard size caps, not just time-based rotation, prevents the specific and common failure mode where an unexpectedly verbose error condition fills a log partition faster than a daily rotation cycle can keep up. And treating available memory headroom as a monitored metric — alerting well before exhaustion, not just logging after the fact — turns a class of failure that currently announces itself as a silent overnight reboot into one that shows up on a dashboard hours or days in advance, which is the entire difference between a maintenance ticket and a field incident.

EurthTech delivers AI-powered embedded systems, IoT product engineering, and smart infrastructure solutions — Hyderabad, India. www.eurthtech.com