Firmware Architecture & RTOS

The Scheduler Is Not Your Problem — Your Architecture Is

Author: Srihari Maddula  •  Founder & Technical Lead, Eurth Techtronics Pvt Ltd

Category: Firmware Architecture & RTOS  •  Estimated Reading Time: 18–20 minutes

Published: July 2026

The Blame Always Goes to the Wrong Place

When an RTOS-based embedded system starts behaving unpredictably — tasks missing deadlines, watchdog resets appearing in production, intermittent hard faults that the development team cannot reproduce on the bench — the first instinct is always the same. The scheduler. Someone adjusts task priorities. Someone lowers a tick rate. Someone adds a delay. The symptom disappears for a few days. Then it comes back.

This is one of the most expensive diagnostic loops in embedded firmware development. Hours, sometimes weeks, spent tuning scheduling parameters to treat symptoms that are caused by something the scheduler has nothing to do with. The real problem, in the vast majority of cases, is architectural. The firmware was not designed to grow into the system it became, and the structural decisions made at the beginning — about task boundaries, shared state, interrupt handling, and memory ownership — are now expressing themselves as scheduling pathology.

This blog is about those structural decisions. What they are, why they matter, how they fail, and what good firmware architecture looks like for systems that need to be maintainable, testable, and debuggable at the complexity level real products eventually reach.

Failure Pattern 1: ISR Coupling — Interrupts That Know Too Much

An interrupt service routine should do one thing: capture the event, record what is necessary, and return control to the scheduler as fast as possible. The ISR is not the place to make decisions. It is not the place to run algorithms. It is not the place to call functions that might block, allocate memory, or take longer than a few microseconds.

In practice, this principle gets violated constantly — usually not through ignorance but through the incremental accretion of features. The UART receive ISR starts by reading a byte and posting it to a queue. Then someone adds a byte counter for statistics. Then someone adds a check for a specific byte that triggers an immediate response. Then someone adds a CRC update. Six months later, the ISR is doing 40 microseconds of work in a system where the highest-priority task needs a 50-microsecond response time. The jitter this introduces is invisible in testing and catastrophic in production.

The discipline required here is strict: ISRs are data capture only. They read the hardware register, write to a queue or ring buffer, set a flag, and exit. Every decision, every processing logic, every state machine transition happens in a task context — where the scheduler can preempt it, where you can set its priority, where you can measure its execution time, and where a watchdog can catch it if it runs too long.

Enforcing this discipline requires that your ISR-to-task communication be zero-copy where possible, and lock-free where practical. A well-designed ring buffer with producer in ISR context and consumer in task context, using only atomic index operations, is the correct pattern. Mutex-protected shared structures in ISR context are an antipattern that will eventually produce a priority inversion.

Failure Pattern 2: Shared State Without Discipline — The Mutex Minefield

Every real embedded system has shared data. Sensor readings that multiple tasks need to access. Configuration parameters written by one task and read by several others. Status flags that cross task boundaries. The question is not whether you have shared state — you do. The question is how rigorously you control access to it.

The failure mode here is not usually a race condition on an obvious shared variable. Experienced engineers know to protect those. The failure mode is the subtle case: the flag that is written atomically on a 32-bit ARM (so 'it doesn't need a mutex') but is read in combination with another flag, and that combination is only valid when both are in a consistent state. The temperature value that is read between two writes of a multi-field sensor struct. The state machine enum that is updated by one task while another task is making a transition decision based on its current value.

The architectural answer is to be explicit about ownership. Every piece of shared data should have a single owning task — the task responsible for writing it. Other tasks read it through a defined interface. If the interface is a direct variable read, it must be protected. If the interface is a message queue, the data travels as an immutable copy. If the interface is a callback or event, the callback runs in the context of the receiving task, not the posting task.

Priority inversion — where a high-priority task is blocked waiting for a mutex held by a low-priority task that has been preempted by a medium-priority task — is the classic scheduler pathology that emerges from shared state mismanagement. Most RTOS implementations offer priority inheritance mutexes that mitigate this. But the better solution is an architecture where high-priority tasks almost never need to wait on a mutex at all, because the shared state interfaces are designed to avoid it.

Failure Pattern 3: The Monolithic Task — Everything in One Place

Embedded firmware often starts as a single loop. Main loop calls sensor read, calls processing, calls output. It works for simple systems. When an RTOS is added, the temptation is to keep this structure and simply move it into one large task. The 'application task' that does everything.

This is the architecture that breaks at complexity. The monolithic task has no internal priority differentiation. If the sensor read takes longer than usual, the output is delayed. If the processing algorithm is in a particularly long computation path, everything else waits. There is no way to selectively prioritise part of the task's work without splitting it into separate tasks.

Good task decomposition follows a principle: tasks should be separated by rate, priority, and dependency. A 100 Hz control loop is a different task from a 1 Hz telemetry sender. A real-time motor controller is a different task from a background FLASH write. A network receive handler is a different task from a data processing pipeline. Each separation gives the scheduler the information it needs to make correct preemption decisions.

The correct task count for a given system is not one and not fifty. It is the minimum number of independent execution contexts required to correctly express the system's timing and priority requirements. For most embedded products, this is between four and twelve tasks. More than that usually indicates that the task boundaries are drawn too fine, and you are creating coordination overhead that exceeds the benefit of separation.

Failure Pattern 4: Stack Sizing by Guess — The Silent Overflow

Stack overflow is one of the most treacherous failure modes in RTOS firmware. Unlike heap overflow, which often produces an immediate and obvious fault, stack overflow can corrupt adjacent memory in ways that produce symptoms that look completely unrelated to the overflow itself. A global variable that suddenly has the wrong value. A function pointer that redirects execution to an invalid address. A task that appears to be running but is processing garbage data.

Stack sizes in most embedded projects are set by experience, intuition, or copying from a previous project. The 512-byte stack that was fine in the previous system is applied to the new system, which has deeper call chains, larger local variables, and printf-style logging added after the original sizing was done. The overflow does not happen immediately — it happens when a particular code path is taken under a particular set of conditions that only occur in the field, three months after deployment.

Every RTOS worth using has a high-watermark stack measurement mechanism. Use it. During development, measure actual peak stack usage for every task under realistic load conditions. Add a 30% safety margin to the measured peak. Review the measurements again whenever you add significant functionality to a task. This is not optional engineering — it is the difference between a firmware that works in the lab and one that works in the field.

What Good Architecture Looks Like

A well-architected RTOS firmware has clear task boundaries that reflect the system's actual timing and priority structure. ISRs are minimal — capture and notify, nothing more. Shared state has explicit ownership and defined access interfaces. Stack sizes are measured, not guessed. The scheduler is given correct information, and it makes correct decisions.

When a firmware with this structure has a bug, the bug is findable. It is localised to a task boundary. It is reproducible because the inter-task interfaces are explicit. It is debuggable because each task's behaviour is independently observable. This is the real return on investment in firmware architecture — not the initial build, but the ten subsequent debug sessions that take hours instead of weeks.

The scheduler is not your problem. But if your architecture is wrong, the scheduler will faithfully execute that wrong architecture and hand you the consequences. Design the architecture correctly, and the scheduler becomes what it was meant to be: a reliable, transparent mechanism that you never need to think about again.

EurthTech builds firmware architecture for products that need to run reliably in the field — not just in the lab.

If you are starting a new embedded product or inheriting one that has grown beyond its original architecture, we are happy to review it.

© 2026 Eurth Techtronics Pvt Ltd  |  eurthtech.com  |  All rights reserved.