David F. Carlson
31 OCT 2005

Design of a Framework for Scalable Real-Time

OR

The 10 Simple Rules for Better Real-Time Applications

Design Rule #1:   The real-time framework should allow the application to be designed for latency.

Design Rule #2:   Use Rate Monotonic compliant scheduler to provide prediction of system schedulability.

Design Rule #3:   Work-units that run to completion are dispatched to work-queues which process their work sequentially.

Design Rule #4:   Work-units are bound to a work-queue at run-time based purely on contracted latency.

Design Rule #5:   Work-units are a binding of an object, a function (most likely a member function of the object), a work-queue and an optional late-binding of an additional dispatcher argument.

Design Rule #6:   The framework owns the mechanisms to construct, clone and dispatch work-units.

Design Rule #7:   All non-system-lifetime work-unit bound object this and dispatcher arguments should be reference counted smart pointers.

Design Rule #8:   Use processor atomic ops to implement fast, scalable, and SMP-safe lock-less patterns.

Design Rule #9:   Latency-contract-based non-reentrant lockless design should be used wherever efficient and possible.

Design Rule #10:   The framework should supply design patterns such as the Observer, State, etc. as the foundation of an application toolkit.

This paper presents 10 design rules and the engineering thought process for arriving at these design goals. The intent is that by taking the reader through the rules, the reader will come to understand that there exists a better way to built real-time systems than many commercial software production facilities use.

Real-time software is software that makes latencies guarantees for its designers. If there is only one activity, this is often trivial. However, when there are hundreds or thousands or tens of thousands of activities that must take place each within their respective designed latencies, a scalable design is required. Scalable designs should be able to use a faster CPU and/or additional CPUs to do proportionally more activities without degradation of the as-designed latency guarantees. The assumption of this work is that symmetric multi-processing (SMP) is the operating system model for multiple CPUs including multi-core CPUs.

What is desired from a real-time framework is the ability to express latency contracts locally in a design. If global knowledge of the system is required to design for latency, the system will not scale. Concurrency in real-time systems is often required by the model of a physical entity. As the physical entity is modeled in software, this model is called an "object". Languages such as C++, Ada and Java have constructs to support objects, and are thus "object oriented". Concurrency is also used to allow I/O requests to be processed independent from the caller. This is often called a "device driver".

A common method for making explicit the concurrency in the system is to map a thread to the object or device driver. To protect access to the object/device drivers internals, one or more mutual exclusion data structures are used. This application model is known somewhat derisively as the "thread&mutex" model. The problem with model for the application is that the control parameter for thread scheduling is "priority". In a FIFO scheduler (or round-robin for that matter), priority determines latency for only the highest priority thread. All the lower priority threads have the possibly unbounded latency of all the threads at their own priority or greater. Priority therefore is unable to express latency in the design. For each additional element of concurrency, the designer must be able to "ascertain" the latencies or all the threads at equal or greater priority. This requires global knowledge of the application and its inter-thread interactions. "Thread&mutex" does not scale.

The body of this paper has been omitted as it is company proprietary. We would be glad to sent you the body of this paper which details the design rationale for our Chronolytics real-time framework. Just drop an email to the author.