ISR Boundaries and the Illusion of Real-Time

Srihari Maddula • Founder & Technical Lead, Eurth Techtronics Pvt Ltd 

Category: Firmware Architecture & RTOS 

Estimated Reading Time: 6 min

“Real-time” gets used as a synonym for “fast,” and that confusion is the root of most scheduling bugs that show up months after a firmware ships to the field. Real-time means deterministic — a guarantee that a given task completes within a bounded time, every time, not most of the time. A system can be blazing fast on average and still fail a real-time requirement if its worst case is unbounded, and the place that unboundedness usually creeps in is the interrupt service routine, because an ISR is the one piece of code in an embedded system that runs outside the scheduler's normal accounting entirely.

What an ISR Is Actually For

An interrupt service routine exists to do exactly one thing: acknowledge that a hardware event happened and capture whatever data would be lost if it weren't captured immediately, as fast as physically possible, and then get out. That's it. It is not the place to parse a packet, update application state, run a filter, or make a decision about what to do with the data it just captured. Every one of those belongs in a deferred context — a task woken by the ISR, running at normal scheduler priority, with normal preemption rules applying.

The reason this discipline matters is that ISRs run above the scheduler, not within it. While an ISR executes, the RTOS's normal priority-based preemption is suspended for at least the interrupt's own priority level, and depending on the architecture, potentially longer. A long-running ISR — even one doing useful, correct work — blocks every task at or below its priority level for its entire duration, and it does this invisibly, because from the scheduler's perspective, nothing is running; the CPU is just gone for a while.

THE RULE:  An ISR that does real work is not fast code running in a privileged context. It's a scheduler-invisible priority ceiling you didn't mean to set.

The Interrupt Latency Versus Jitter Distinction

Two numbers matter for any interrupt-driven system, and they get conflated constantly. Interrupt latency is the time from the physical event to the first instruction of the ISR executing — dominated by hardware (interrupt controller response time) and by whether a higher-priority ISR or a critical section with interrupts disabled is already in progress. Jitter is the variation in that latency across many occurrences of the same event. A system can have high but perfectly consistent latency and still be usable for many applications, because the downstream logic can be designed around a known, fixed delay. A system with low average latency but high jitter is often worse in practice, because nothing downstream can be designed around an unpredictable delay — every consumer of that timing has to budget for the worst case, which erases most of the benefit of the good average.

Jitter's usual source is exactly what the previous section warned about: ISRs, or critical sections, whose duration depends on data content or system state rather than being constant. A UART receive ISR that does fixed-size register reads has effectively zero jitter contribution. The same ISR modified to also parse a variable-length frame and update a state machine now has a duration that depends on frame content, and that variability propagates directly into system-wide jitter for every other interrupt sharing or below that priority level.

Priority Inversion, Concretely

Priority inversion is the textbook case where a low-priority task holds a resource (commonly a mutex) that a high-priority task needs, and a medium-priority task — with no interest in that resource at all — preempts the low-priority task and runs indefinitely, indirectly blocking the high-priority task for far longer than the resource contention alone would justify. This is not a rare edge case; it's a predictable consequence of mixing priority-based preemption with any shared resource, and it has caused real, documented mission failures, most famously on Mars Pathfinder, where a priority inversion between a low-priority meteorological data task and a high-priority bus management task, mediated by a medium-priority communications task, caused watchdog-triggered system resets on the surface.

The standard fix is priority inheritance: when a high-priority task blocks on a mutex held by a lower-priority task, the RTOS temporarily boosts the holder's priority to match the blocked task's priority, for the duration it holds the resource. Most production RTOSes (FreeRTOS with the right mutex type, Zephyr, ThreadX) implement this, but it is not automatic in every configuration — using a plain semaphore in place of a mutex for mutual exclusion, for instance, silently loses priority inheritance because a semaphore has no concept of ownership for the inheritance mechanism to act on. This is a subtle and common mistake: semaphores and mutexes look interchangeable for basic mutual exclusion, and only diverge under contention, which is exactly the condition under which the difference matters.

THE RULE:  A semaphore used for mutual exclusion works fine right up until two tasks of different priority actually contend for it — which is the one scenario a mutex with priority inheritance was built to handle correctly.

Architecture Smells That Predict Future ISR Problems

A few patterns reliably predict that a firmware's interrupt handling will become a maintenance liability as the system grows, well before any specific bug appears.

•  ISRs that call into application-layer functions rather than posting to a queue or setting a flag — every call adds unbounded, content-dependent execution time to a context that's supposed to be bounded and minimal.

•  Shared global state written from both ISR and task context without a documented synchronization strategy — this compiles cleanly and runs correctly under light load, then produces the classic “works in the lab, fails in the field” bug once event rates rise enough to actually trigger the race.

•  Nested interrupt priorities configured once at project start and never revisited as new peripherals and their ISRs are added — priority levels accumulate ad hoc, and nobody has a current picture of the true worst-case interrupt latency for any given source.

•  No instrumentation for ISR duration in production or pre-production testing — teams frequently discover their true worst-case ISR duration for the first time during a field failure investigation, using a logic analyzer, months after the code shipped.

The Practical Discipline

Bare-metal or lightly-structured firmware can get away with informal interrupt handling because the whole system is small enough to hold in one engineer's head. That stops being true well before most teams admit it does — usually around the point a second peripheral's interrupt handler needs to interact with the first one's state, which is precisely where undocumented assumptions about execution order and duration start compounding into intermittent bugs. The fix isn't more clever ISR code; it's less ISR code; push everything that isn't strictly time-critical hardware acknowledgment into deferred task context, measure actual worst-case ISR duration rather than assuming it, and treat priority assignment as a system-wide design decision revisited whenever a new interrupt source is added, not a per-peripheral afterthought.

EurthTech delivers AI-powered embedded systems, IoT product engineering, and smart infrastructure solutions — Hyderabad, India. www.eurthtech.com