CASE STUDY IN RT-LINUX

Ajith Kumar P. Shetty,

School of Electronics Engineering, VIT University, Vellore, Tamilnadu 63201, INDIA
{ajithkumarpshetty.2013, }@vit.ac.in

Abstract Embedded application is a hotspot at present and
Linux gradually becomes the most important operating system
for embedded applications. Aiming at the real-time problems of
Linux and from five performance parameters of real time
operating system, this paper analyzes and concludes that
scheduling latency and interrupt latency are the fundamental
constraints for improving real-time performance of Linux 2.6
kernel, then designs and implements a new task model and
new interrupt operations to solve the above problem. Hard real-
time task scheduling algorithm which is named as Priority
Bitmap Algorithm, new interrupt response and new interrupt
operations are emphasized and main codes are given out.
Through testing, response time of real-time task is indicated to
be shortened largely and meets the initial expectation.
Keywords: Real-time, Interrupt Latency, Scheduling Strategy

I. INTRODUCTION.

RT Linux is a hard real time RTOS microkernel that runs
the entire Linux operating system as fully preemptive process.
It is one of the hard real-time variants of Linux, among
several, that makes it possible to control robots, data
acquisition systems, manufacturing plants, and other time-
sensitive instruments and machines.
It was developed by Victor Yodaiken, Michael Barabanov,
Cort Dougan and others at the New Mexico Institute of
Mining and Technology and then as a commercial product
at FSMLabs. Wind River Systems acquired FSMLabs
embedded technology in February 2007 and made a version
available as Wind River Real-Time Core for Wind River
Linux. As of August 2011, Wind River has discontinued the
Wind River Real-Time Core product line, effectively ending
commercial support for the RTLinux product.
The key RTLinux design objective was to add hard real-
time capabilities to a commodity operating system to facilitate
the development of complex control programs with both
capabilities. For example, one might want to develop a real-
time motor controller that used a commodity database and
exported a web operator interface. Instead of attempting to
build a single operating system that could support real-time
and non-real-time capabilities, RTLinux was designed to share
a computing device between a real-time and non-real-time
operating system so that (1) the real-time operating system
could never be blocked from execution by the non-real-time
operating system and (2) components running in the two
different environments could easily share data. As the name
implies RTLinux was the first computer designed to use Linux
as the non-real-time system but it eventually evolved so that
the RTCore real-time kernel could run with either Linux
or BSD UNIX.
Multi-Environment Real-Time (MERT) was the first
example of a real-time operating system coexisting with a
UNIX system. MERT relied on traditional virtualization
techniques: the real-time kernel was the host operating system
(or hypervisor) and Bell Systems UNIX was the guest.
RTLinux was an attempt to update the MERT concept to the
PC era and commodity hardware. It was also an attempt to
also overcome the performance limits of MERT, particularly
the overhead introduced by virtualization.
The technique used was to only virtualize the guest
interrupt control. This method allowed the real-time kernel to
convert the guest operating system into a system that was
completely preemptible but that could still directly control, for
example, storage devices. In particular, standard drivers for
the guest worked without source modification although they
needed to be recompiled to use the virtualization "hooks". See
also paravirtualization. The UNIX "pipe" was adapted to
permit real-time and non-real-time programs to communicate
although other methods such as shared memory were also
added.
From the programmer's point of view, RTLinux originally
looked like a small threaded environment for real-time tasks
plus the standard Linux environment for everything else. The
real-time operating system was implemented as a loadable
kernel module which began by virtualizing guest interrupt
control and then started a real-time scheduler. Tasks were
assigned static priorities and scheduling was originally purely
priority driven. The guest operating system was incorporated
as the lowest priority task and essentially acted as the idle task
for the real-time system. Real-time tasks ran in kernel mode.
Later development of RTLinux adopted the POSIX
threads application programming interface (API) and then
permitted creation of threads in user mode with real-time
threads running inside guest processes. In multiprocessor
environments threads were locked to processor cores and it
was possible to prevent the guest thread from running on
designated core (effectively reserving cores for only real-time
processing).
RTLinux provides the capability of running special real-
time tasks and interrupt handlers on the same machine as
standard Linux. These tasks and handlers execute when they
need to execute no matter what Linux is doing. The worst case
time between the moment a hardware interrupt is detected by
the processor and the moment an interrupt handler starts to
execute is under 15 microseconds on RTLinux running on a
generic x86 (circa 2000). A RTLinux periodic task runs within
25 microseconds of its scheduled time on the same hardware.
These times are hardware limited, and as hardware improves
RTLinux will also improve. Standard Linux has excellent
average performance and can even provide millisecond level
scheduling precision for tasks using the POSIX soft real-time
capabilities. Standard Linux is not, however, designed to
provide sub-millisecond precision and reliable timing
guarantees. RTLinux was based on a lightweight virtual
machine where the Linux "guest" was given a virtualized
interrupt controller and timer, and all other hardware access
was direct. From the point of view of the real-time "host", the
Linux kernel is a thread. Interrupts needed for deterministic
processing are processed by the real-time core, while other
interrupts are forwarded to Linux, which runs at a lower
priority than real-time threads. Linux drivers handle almost
all I/O. First-In-First-Out pipes (FIFOs) or shared memory can
be used to share data between the operating system and
RTLinux.
RTLinux is structured as a small core component and a set
of optional components. The core component permits
installation of very low latency interrupt handlers that cannot
be delayed or preempted by Linux itself and some low level
synchronization and interrupt control routines. This core
component has been extended to support SMP and at the same
time it has been simplified by removing some functionality
that can be provided outside the core.
1. Functionality:
The majority of RTLinux functionality is in a collection of
loadable kernel modules that provide optional services and
levels of abstraction. These modules include:
1) rtl sched a priority scheduler that supports both a "lite
POSIX" interface described below and the original
V1 RTLinux API.
2) rtl time which controls the processor clocks and
exports an abstract interface for connecting handlers
to clocks.
3) rtl posixio supports POSIX style read/write/open
interface to device drivers.
4) rtl fifo connects RT tasks and interrupt handlers to
Linux processes through a device layer so that Linux
processes can read/write to RT components.
5) semaphore is a contributed package by Jerry Epplin
which gives RT tasks blocking semaphores.
6) POSIX mutex support is planned to be available in
the next minor version update of RTLinux.
7) mbuff is a contributed package written by Tomasz
Motylewski for providing shared memory between
RT components and Linux processes.

2. Kernel module:

RTLinux is in fact a Linux kernel module. It is the same
type of module that Linux uses for drivers, file systems, and
so on. The main difference between RTLinux module and an
ordinary Linux module is that RTLinux module calls
functions, which are offered by RTLinux kernel whereas
ordinary module uses only Linux kernel functions. The source
code of a simple Linux module is given below. It contains two
functions: init_module, which is called when the module is
inserted to the kernel by insmod or modprobe command, and
cleanup module, which is called before module is removed by
rmmod.

3. Architecture:

Fig 1: Architecture of RTLinux.
Standard time sharing OS and hard real time executive
running on same machine Kernel provided with the emulation
of Interrupt control H/W. RT tasks run at kernel privilege
level to provide direct H/W access. RTLinux runs Linux
kernel for booting, device drivers, networking, FS, process
control and loadable kernel modules. Linux kernel runs as a
low priority task on the RT kernel hence making sure it cannot
preempt any RT task.
RT task allocated fixed memory for data and code.RT
tasks cannot use Linux system calls , directly call routines or
access ordinary data structures in Linux Kernel. RTProcesses
without memory protection features (RTLinux Pro has some
PSDD now). The RT kernel is actually patched over the Linux
kernel and then recompiled to work as a RTLinux system.
Completely configurable RTLinux kernel.
VM layer only emulates the Interrupt Control. The 0 level
OS does not provide any basic services that can be provided
by Linux - Only RT services - No primitives for process
creation, switching or MM. Uses software stack for switching
and not the expensive H/W switching. RTKernel is not
preemptable.






II. OBJECTIVES

The key RTLinux design objective is that the system
should be transparent, modular, and extensible. Transparency
means that there are no unopenable black boxes and the cost
of any operation should be determinable. Modularity means
that it is possible to omit functionality and the expense of that
functionality if it is not needed. The base RTLinux system
supports high speed interrupt handling and no more. And
extensibility means that programmers should be able to add
modules and tailor the system to their requirements. It has
simple priority scheduler that can be easily replaced by
schedulers more suited to the needs of some specific
application. When developing RTLinux, it was designed to
maximize the advantage we get from having Linux and its
powerful capabilities available.

III. DETAILED PROBLEM DEFINITION

There are numerous real-time operating systems available
today. What was missing, however, is an open, standard,
supported, efficient, and inexpensive multitasking system with
hard real-time capabilities. Many UNIX systems meet the first
three requirements. Linux
1
, a relatively recent free UNIX-like
OS, originated by Linus Torvalds, features excellent stability,
efficiency, source code availability, not restrictive license, and
substantial user base. Source code availability is essential for
verification of the system correctness, adaptation to specific
problems, and mere bug fixing. Linux can run on the most
widely available computers: IBM PCs and compatible ma-
chines (most PC hardware is supported). Since many
computer and engineering labs have computers of this type, it
is possible for them to use Linux without any further
investment.
Linux has all features of a modern UNIX system: several
X Window System implementations, graphical user interface
toolkits, networking, databases, programming languages,
debuggers, and a variety of applications. It is embeddable and
is able to perform efficiently using relatively small amounts of
RAM and other computer resources. In short, Linux has the
potential to make an excellent development platform for a
wide variety of applications, including those involving real-
time processing. However, Linux has several problems
preventing it from being used as a hard real-time OS, most
notably the fact that interrupts are often disabled during the
course of execution of the kernel. Other problems include
time-sharing scheduling, virtual memory system timing
unpredictability, and lack of high-granularity timers. It turns
out that using software interrupts, together with several other
techniques, it is nevertheless possible to modify Linux so as to
overcome these problems. The idea to use software interrupts
so that a general-purpose operating system could coexist with
a hard real-time system is due to Victor Yodaiken (personal
communications). This thesis details how this idea and several
others were applied to build a hard real-time version of Linux.
IV. SOLUTION METHODOLOGY.

RTLinux is very much module oriented. To use RTLinux,
you load modules that implement whatever RT capabilities
you need. Two of the core modules are the scheduler and the
module that implements RT-fos. If the services provided by
these modules dont meet the requirements of the application,
they can be replaced by other modules. For example, there are
two alternative scheduling modules a earliest deadline
rst scheduler implemented by Ismael Rippol and a rate-
monotonic scheduler.
Fig.2: Flow of data and control

The basic scheduler simply runs the highest priority ready
RT task until that task suspends itself or until a higher priority
task becomes ready.
The original RTLinux scheduler used the timer in one
shot mode so that it could easily handle a collection of tasks
with periods that had a small common divisor. For example, if
one task must run every331time units and the other runs
every1027time units, there is no good choice for a timer
period. In one-shot mode, the clock would be set rst to
generate an interrupt after 331 time units and then
reprogrammed after the interrupt to generate a second
interrupt in another691time units (minus the time needed to
reprogram the clock). The price we pay is that we reprogram
the clock on every interrupt. For x86 generic motherboards,
reprogramming the clock is relatively slow. It turns out, how-
ever, that many applications dont need the generality of a
one-shot timer and can avoid the expense of reprogramming.
Professor Paolo Mantegazza of the Aerospace Engineering
Department in Politecnico di Milano wrote a scheduler that
demonstrated the utility of periodic mode and encouraged us
to put it into the standard scheduler6. The current RTLinux
scheduler offers both periodic and one shot modes. On SMP
systems the problem gets simpler because there is an on-
processor high frequency timer chip that is available to the
RTL system.



V. DETAILS OF EXPERIMENTATION, ANALYSIS, MODELLING.

In order to measure the performance of Real-Time Linux,
I have conducted several experiments. The experiments were
performed on two IBM PC compatible computers running
Linux version 2.0.29 and Real-Time Linux version 0.5a. The
results are summarized in Table. To measure the maximum
interrupt latency, an additional machine running Real-Time
Linux (Machine 1) was used to send interrupt requests to the
machine being tested (Machine 2) and to measure the
response time of the latter (Figure 1.2).
The D0 output of the parallel port of Machine 1 is
connected to the ACK parallel port input of Machine 2. When
the ACK input goes from a logic one to logic zero, an
interrupt request is sent to the processor. To provide feedback,
the D0 output of Machine 2 is connected to a parallel port
input (PE) on Machine 1.

Fig.2: Measuring interrupt latency

Table 1: interrupt latency and scheduling precision for
different machines.

Machine A: Intel 80486 33 MHz, ISA bus, 16 MB of
RAM, Western Digital Caviar 2340 IDE hard drive, 3C509
Ethernet card.
Machine B: Intel Pentium 120 MHz, PCI bus, 32 MB of
RAM, EATA-DMA DPT SCSI adapter, Conner CP30200
SCSI hard drive, NE2000 Ethernet card (in an ISA slot).
The measurement is performed as follows. A real-time
process on Machine 1 records the current time, sends a pulse
to the D0 output, and enters a busy loop waiting for the PE
input to change. An interrupt request is sent to the CPU of
Machine 2. From an interrupt handler (both real-time and
ordinary Linux handlers were used) the output D0 is toggled.
The real-time process on Machine 1 exits the busy loop,
obtains the current time, and computes how long it took
Machine 2 to respond. This sequence is performed
periodically over a substantial interval of time. The maximum
response time encountered is taken to be an estimate of the
worst-case interrupt latency.
As seen from Table, the interrupt handling latency in plain
Linux is substantially higher than in Real-Time Linux.
Moreover, it is quite possible that some device drivers that I
have not used disable interrupts for longer periods, further
increasing Linux interrupt processing latency.
To measure scheduling precision a periodic real-time task
was run. On each wake-up the current time was obtained and
compared to the estimate. Maximum deviations were recorded.
I found it impossible to reliably run periodic tasks as standard
Linux processes, partly because Linux does not provide
periodic timers facility, and also because of other problems .
During all tests the system was heavily loaded with disk
and network I/O operations. Although device drivers in Real-
Time Linux do not disable hardware interrupts, heavy I/O
does increase interrupt latency. This fact can be attributed to
the DMA cycle stealing.
Several stress tests have also been performed. In one of
them the system has successfully performed a backup of a
local system over the local network, scheduling two periodic
real-time tasks each with 1 millisecond period at the same
time. This experiment demonstrates that the presence of a
real-time system has no adverse effect on the functioning of
the Linux kernel.
Overall, the results show that Real-Time Linux is a viable
platform for hard real-time processing.

VI. APPLICATIONS

The first one is that of Harald Stauss
2
from the department
of physiology at the Hum-boldt University in Berlin,
Germany. He has implemented a system for the recording and
display of hemodynamic measurements in rats. It uses an
analog-to-digital converter card to acquire the signals from
sensors. The card based on the MAX192BCPP chip has eight
12-bit channels that are multiplexed to one serial line
connected to the serial port of the computer. The system runs
under Real-Time Linux and consists of a real-time task and a
user process. The real-time task polls the card and passes the
received data through an RT-FIFO to the user process. The
process records the data to a _le and at the same time displays
it graphically in a Motif drawing widget. Mr. Stauss reports
that the system is able to reliably acquire data with the total
sampling rate of 3000 Hz on a i486/33MHz-based machine.
On the same computer, Labtech Notebook for DOS, a
commercial data acquisition program, could only provide
sampling rates of less than 400 Hz.
Bill Crum at New Mexico Tech has developed an embedded
control and monitoring software for the Tech Sunrayce car
[3]. The system specification required data collection from 70
sensors with total sampling rate of 80 Hz. The needed
response times ranged from 200 to 300 milliseconds. These
requirements were satisfied by using one real-time task
executing with 0.0125 seconds period on a 20 MHz
i386{based computer running Real-Time Linux. To facilitate
the debugging of the system, Bill Crum has written a set of
programs that he describes as an \engineering workbench".
The programs simulate tools commonly found in electrical
engineering laboratories: an oscilloscope, a logic analyzer,
and a signal generator. The complex uses a data acquisition
card that provides analog and digital I/O. Real-time tasks are
used to communicate with the card, and Linux processes
implement graphical interfaces using the Qt widget set.
There are several other applications. A task running under
Real-Time Linux is used for real-time communication with
the Phantom, a force feedback device. This is a part of a
system that creates virtual worlds. The system allows users to
navigate through these worlds, to feel and manipulate objects.
Dan Samber
4
from Mount Sinai Medical School in New York
City uses Real-Time Linux to reliably communicate with a
patient monitor over a serial line for recording and display of
physiological parameters.

VII. CONCLUSIONS
By observing various features of the RTLinux and outcomes
of the various analysis and experiments yields a conclusion
that the Real-Time Linux is a viable platform for hard real-
time processing which involves low interrupt latency and
requires more scheduling precision.