KURT KENNETT

MICROSOFT CORPORATION
Real-Time Overview
Basic Terminology
Practical Usage Example
Windows CE Interrupt Model
Threads, Processes, IST, ISR, Priorities
Real-Time Overview
Basic Terminology
Practical Usage Example
Windows CE Interrupt Model
Threads, Processes, IST, ISR, Priorities
Real-Time specific features
Flexible Thread Quantum
Priority Inversion
Variable System Tick
Real-Time Overview
Basic Terminology
Practical Usage Example
Windows CE Interrupt Model
Threads, Processes, IST, ISR, Priorities
Real-Time specific features
Flexible Thread Quantum
Priority Inversion
Variable System Tick
Measurement Tools
Real-Time Overview
Basic Terminology
Practical Usage Example
Windows CE Interrupt Model
Threads, Processes, IST, ISR, Priorities
Real-Time specific features
Flexible Thread Quantum
Priority Inversion
Variable System Tick
Measurement Tools
Canned Demo
Interrupt
Hardware signal indicating a real-world event has happened
The corresponding hardware device needs to be
serviced in some way by the computer system
Interrupt
Hardware signal indicating a real-world event has happened
The corresponding hardware device needs to be
serviced in some way by the computer system
Latency
The time from when the interrupt occurred to when
the hardware begins to be serviced
Interrupt
Hardware signal indicating a real-world event has happened
The corresponding hardware device needs to be
serviced in some way by the computer system
Latency
The time from when the interrupt occurred to when
the hardware begins to be serviced
Jitter
Range of allowable variation in service time
Usually defined by the “tolerance” of a mechanical system
for the variability in response
Interrupt
Hardware signal indicating a real-world event has happened
The corresponding hardware device needs to be
serviced in some way by the computer system
Latency
The time from when the interrupt occurred to when
the hardware begins to be serviced
Jitter
Range of allowable variation in service time
Usually defined by the “tolerance” of a mechanical system
for the variability in response
Bounded
Extreme limits are known precisely
Interrupt
Hardware signal indicating a real-world event has happened
The corresponding hardware device needs to be
serviced in some way by the computer system
Latency
The time from when the interrupt occurred to when
the hardware begins to be serviced
Jitter
Range of allowable variation in service time
Usually defined by the “tolerance” of a mechanical system
for the variability in response
Bounded
Extreme limits are known precisely
Bounded low latency and jitter = “hard” real time
Consumers want to know if CE is hard real-time
Want to know if CE is capable of simultaneously
running radio and UI
Some system developers were concerned that CE was not
hard real-time enough to meet the requirements
Consumers want to know if CE is hard real-time
Want to know if CE is capable of simultaneously
running radio and UI
Some system developers were concerned that CE was not
hard real-time enough to meet the requirements
Requirements
Run cellular radio DSP
Meet “tight” timing requirements
ARM <= 250Mhz CPU
Full performance Windows CE UI
And play simultaneous video and audio
 OMAC represents Industrial Automation Community

100 ms
Soft Real-Time
20 ms
Cycle Time

10 ms
Hard
5 ms Real
Time

1 ms

500 us

0 100 µs 1,000 µs 5,000 µs 10,000 µs

Cycle Variation or Jitter (µs)

 OMAC represents Industrial Automation Community

100 ms
Soft Real-Time
20 ms
Cycle Time

10 ms Windows
Hard NT
5 ms Real
Time

1 ms

500 us

0 100 µs 1,000 µs 5,000 µs 10,000 µs

Cycle Variation or Jitter (µs)

 OMAC represents Industrial Automation Community

100 ms
Soft Real-Time
20 ms
Cycle Time

10 ms Windows Windows
Hard CE 2.X NT
5 ms Real
Time

1 ms

500 us

0 100 µs 1,000 µs 5,000 µs 10,000 µs

Cycle Variation or Jitter (µs)

 OMAC represents Industrial Automation Community

100 ms
Soft Real-Time
20 ms
Cycle Time

10 ms Windows Windows
Hard CE 2.X NT
5 ms Real
Time

1 ms Windows
CE .NET
500 us

0 100 µs 1,000 µs 5,000 µs 10,000 µs

Cycle Variation or Jitter (µs)

 OMAC represents Industrial Automation Community

100 ms
Soft Real-Time
20 ms
Cycle Time

10 ms Windows Windows
Hard CE 2.X NT
5 ms Real 90%
Time
of
1 ms
Apps
Windows
CE .NET
500 us

0 100 µs 1,000 µs 5,000 µs 10,000 µs

Cycle Variation or Jitter (µs)

So what were some actual requirements for a real project?
200Mhz ARM
Windows CE with Full UI
Running WMV Video playback
Interrupt as frequent as every 4.6 ms
Allowable jitter < 0.5ms

Actual Application Requirements 0.5 ms Jitter

Interrupt every 4.6 ms

 Windows CE Real-Time Test Results
ISR starts IST starts
Minimum 1.2 µs 31.7 µs
Average 3.3 µs 67.2 µs
Maximum 13.3 µs 103.0 µs
Time in microseconds (µs)

In terms of the 0.5 ms jitter alone

CE’s longest ISR response time was 13.3 µs (2.6% of max allowed)
CE’s longest IST response time was 103 µs (20.6% of max allowed)
 Windows CE Real-Time Test Results
ISR starts IST starts
Minimum 1.2 µs 31.7 µs
Average 3.3 µs 67.2 µs
Maximum 13.3 µs 103.0 µs
Time in microseconds (µs)

In terms of the 0.5 ms jitter alone

CE’s longest ISR response time was 13.3 µs (2.6% of max allowed)
CE’s longest IST response time was 103 µs (20.6% of max allowed)
Conclusion
CE’s response time was well within the requirements
Project in case study went ahead and did well
Thread
A piece of code that can be scheduled to run by the kernel
A “unit” of execution
May be launched by a process or a driver
Thread
A piece of code that can be scheduled to run by the kernel
A “unit” of execution
May be launched by a process or a driver
Process
Launched from an executable file
A collection of threads (at least one) with a
common execution environment
Can create threads to handle interrupts
Thread
A piece of code that can be scheduled to run by the kernel
A “unit” of execution
May be launched by a process or a driver
Process
Launched from an executable file
A collection of threads (at least one) with a
common execution environment
Can create threads to handle interrupts
Driver
A DLL, (dynamically loaded library) loaded into the kernel
or into a user-mode driver host process
Can create threads to handle interrupts
Interrupt Service Routine (ISR)
A piece of code built into or loaded into the kernel
Logically assigned to a particular hardware interrupt
Called within O(10) CPU instructions to service an interrupt
Should be written to run quickly with few or no outside dependencies
and no synchronization with or unbounded calls to other code
Can be chained together if multiple devices might use
the same hardware interrupt line (IRQ)
Logically notifies the kernel which IST should run
Interrupt Service Routine (ISR)
A piece of code built into or loaded into the kernel
Logically assigned to a particular hardware interrupt
Called within O(10) CPU instructions to service an interrupt
Should be written to run quickly with few or no outside dependencies
and no synchronization with or unbounded calls to other code
Can be chained together if multiple devices might use
the same hardware interrupt line (IRQ)
Logically notifies the kernel which IST should run
Interrupt Service Thread (IST)
A thread registered specifically to handle an interrupt
Can be created by either a process or a driver
Scheduled like any other thread on the system
Should be written to do the bulk of the interrupt handling work
ISRs and ISTs usually work as pairs
ISR handles the critical time-sensitive work
IST typically handles the rest
ISRs and ISTs usually work as pairs
ISR handles the critical time-sensitive work
IST typically handles the rest
They synchronize by using an OS EVENT Object
The IST creates an EVENT Object
The IST associates the EVENT with a logical interrupt
It uses the WaitForSingleObject() to wait for
the EVENT object to be signaled
The ISR tells the kernel which EVENT object to signal
When the EVENT is signaled, it unblocks the IST and
makes it able to be scheduled by the kernel to run
ISRs and ISTs usually work as pairs
ISR handles the critical time-sensitive work
IST typically handles the rest
They synchronize by using an OS EVENT Object
The IST creates an EVENT Object
The IST associates the EVENT with a logical interrupt
It uses the WaitForSingleObject() to wait for
the EVENT object to be signaled
The ISR tells the kernel which EVENT object to signal
When the EVENT is signaled, it unblocks the IST and
makes it able to be scheduled by the kernel to run
If the IST is the highest priority run-able thread,
it will get scheduled to run immediately
Windows CE 6.0 has 256 levels of priority

Levels Description
0 through 96 Real-time above drivers
97 through 152 Default used by CE device drivers
153 through 247 Real-time below drivers
248 through 255 Non-real-time priorities
Windows CE 6.0 has 256 levels of priority
Level 0 is the highest and 255 is the lowest

Levels Description
0 through 96 Real-time above drivers
97 through 152 Default used by CE device drivers
153 through 247 Real-time below drivers
248 through 255 Non-real-time priorities
Windows CE 6.0 has 256 levels of priority
Level 0 is the highest and 255 is the lowest
The default level for a thread is 252

Levels Description
0 through 96 Real-time above drivers
97 through 152 Default used by CE device drivers
153 through 247 Real-time below drivers
248 through 255 Non-real-time priorities
Windows CE 6.0 has 256 levels of priority
Level 0 is the highest and 255 is the lowest
The default level for a thread is 252
Levels 0 through 248 can be reserved by OEM

Levels Description
0 through 96 Real-time above drivers
97 through 152 Default used by CE device drivers
153 through 247 Real-time below drivers
248 through 255 Non-real-time priorities
Is responsible for determining which thread will run
Has a queue for threads for each priority level
Always schedules first thread at the highest priority level
Is responsible for determining which thread will run
Has a queue for threads for each priority level
Always schedules first thread at the highest priority level
A thread gets to run for set length of time,
called a “quantum”
Typically 100 milliseconds
A quantum of 0 means the quantum never runs out
The thread can run until blocked or interrupted
Is responsible for determining which thread will run
Has a queue for threads for each priority level
Always schedules first thread at the highest priority level
A thread gets to run for set length of time,
called a “quantum”
Typically 100 milliseconds
A quantum of 0 means the quantum never runs out
The thread can run until blocked or interrupted
A thread runs until—
Its quantum runs out
It is interrupted by a higher priority thread
It is blocked by a resource contention such as
access to a critical section or a mutex
 Interrupt
ISR runs, tells kernel
Interrupt Occurs Handler calls
which event to signal
registered ISR

IST runs and resets Scheduler sets up Kernel signals event,

the interrupt the IST to run IST becomes runable
The maximum amount of time between a hardware
interrupt being signaled and when the code in an IST
begins to run is:
The maximum amount of time between a hardware
interrupt being signaled and when the code in an IST
begins to run is:

Maximum time before ISR can run

The maximum amount of time between a hardware
interrupt being signaled and when the code in an IST
begins to run is:

Maximum time before ISR can run

+
Maximum time taken by ISR
The maximum amount of time between a hardware
interrupt being signaled and when the code in an IST
begins to run is:

Maximum time before ISR can run

+
Maximum time taken by ISR
+
Maximum time taken by kernel to be able to get to
a state where it can schedule another thread
The maximum amount of time between a hardware
interrupt being signaled and when the code in an IST
begins to run is:

Maximum time before ISR can run

+
Maximum time taken by ISR
+
Maximum time taken by kernel to be able to get to
a state where it can schedule another thread
+
Maximum time taken by kernel to switch from
whatever thread was running to the IST thread
 INTERRUPT! MAXIMUM ISR LATENCY

Normal
Int Off Normal Thread
Thread

IST

ISR
OAL

Scheduler Scheduler

ISH ISH
KERNEL

Interrupts Disabled
ISR Latency IST Latency
Preemption Disabled
 INTERRUPT! MAXIMUM ISR LATENCY

Normal
Int Off Normal Thread
Thread

IST

ISR
OAL

Scheduler Scheduler

ISH ISH
KERNEL

Interrupts Disabled
ISR Latency IST Latency
Preemption Disabled
 INTERRUPT! MAXIMUM IST LATENCY

Normal
Normal Thread
Thread

IST

ISR
OAL

KCall KCall Scheduler Scheduler

ISH ISH
KERNEL

Interrupts Disabled
ISR Latency IST Latency
Preemption Disabled
 INTERRUPT! MAXIMUM IST LATENCY

Normal
Normal Thread
Thread

IST

ISR
OAL

KCall KCall Scheduler Scheduler

ISH ISH
KERNEL

Interrupts Disabled
ISR Latency IST Latency
Preemption Disabled
Higher priority ISRs can preempt lower ISRs
Higher priority ISRs can preempt lower ISRs
Based on support by the CPU, additional hardware,
and/or OEM code
Higher priority ISRs can preempt lower ISRs
Based on support by the CPU, additional hardware,
and/or OEM code
ARM
Uses a vectored interrupt table
Only two CPU interrupt levels, each with its own hardware pin
IRQ – Normally only one used.
FIQ – Supports ISR-only scenarios for critical time-sensitive tasks
Interrupts are not turned on before entering an ISR
OEM can re-enable CPU interrupt within ISR when/if they want
OEMs can prioritize the interrupts with an external
Interrupt Controller and bit masks to turn on and off
the different interrupts
Hardware designers often attach several
devices to the same interrupt line (IRQ)
Hardware designers often attach several
devices to the same interrupt line (IRQ)
Multiple ISRs can be chained together to
handle shared interrupts
Hardware designers often attach several
devices to the same interrupt line (IRQ)
Multiple ISRs can be chained together to
handle shared interrupts
Each ISR in turn determines if it can handle the interrupt
If it can, it does its work and either completes the interrupt or
returns the logical SYSINTR value representing which IST is to
run
If not, it returns SYSINTR_CHAIN indicating the kernel should try
the next ISR in the chain.
The default OEMInterruptHandler on a system typically will
disable an interrupt that occurs that does not have a relative IST
assigned to it (it is not claimed by an ISR)
A high priority thread can get stuck waiting for a
lower priority thread to release a resource
Such as a critical section, semaphore, or mutex
This causes priority inversion
A high priority thread can get stuck waiting for a
lower priority thread to release a resource
Such as a critical section, semaphore, or mutex
This causes priority inversion
The Kernel detects priority inversion and handles it with
priority inheritance, or “boosting”
The lower priority thread temporarily inherits the
higher priority thread’s priority
The lower priority thread’s quantum is set to 0,
which lets it run until it releases the resource
A high priority thread can get stuck waiting for a
lower priority thread to release a resource
Such as a critical section, semaphore, or mutex
This causes priority inversion
The Kernel detects priority inversion and handles it with
priority inheritance, or “boosting”
The lower priority thread temporarily inherits the
higher priority thread’s priority
The lower priority thread’s quantum is set to 0,
which lets it run until it releases the resource
Supports only one level of inheritance
Kernel will only boost one thread
If the boosted thread is also in turn blocked by
a third thread, the third thread is not boosted
Care must be taken when acquiring locks on shared resources
There is a per-thread quantum
There is a per-thread quantum
The default is set by the OEM in the OAL
dwDefaultThreadQuantum
There is a per-thread quantum
The default is set by the OEM in the OAL
dwDefaultThreadQuantum
APIs to set Quantum
Ce(Set/Get)ThreadQuantum
There is a per-thread quantum
The default is set by the OEM in the OAL
dwDefaultThreadQuantum
APIs to set Quantum
Ce(Set/Get)ThreadQuantum
Quantum of 0 sets a thread to “run-to-completion”
This can occur at any priority
Preempted only by higher priority threads
Normally there is a 1 ms timer tick
Normally there is a 1 ms timer tick
A tick interrupt can cause a reschedule
When a system tick occurs the kernel checks for a reschedule or No-Op
The kernel will make sure one of the highest priority
run-able threads is executing
If there is no higher priority thread and the current thread has not
exhausted its quantum, the current thread is allowed to continue to run
Normally there is a 1 ms timer tick
A tick interrupt can cause a reschedule
When a system tick occurs the kernel checks for a reschedule or No-Op
The kernel will make sure one of the highest priority
run-able threads is executing
If there is no higher priority thread and the current thread has not
exhausted its quantum, the current thread is allowed to continue to run
Sleep(N) will generally wake up in N to N + 1 ms
Normally there is a 1 ms timer tick
A tick interrupt can cause a reschedule
When a system tick occurs the kernel checks for a reschedule or No-Op
The kernel will make sure one of the highest priority
run-able threads is executing
If there is no higher priority thread and the current thread has not
exhausted its quantum, the current thread is allowed to continue to run
Sleep(N) will generally wake up in N to N + 1 ms
When the CPU has nothing to do (no threads can run) the
system tick is reset to when the next scheduled event is to occur
An interrupt can re-set the system tick back to 1 ms
Windows CE 6.0 memory model was re-architected
GWES, Networking, File System, and Driver Manager were all
moved into the kernel’s address space
All threads from a kernel mode driver run in
the kernel process context
They have full access to the kernel space
Security checks are not required to run or call APIs
Windows CE 6.0 memory model was re-architected
GWES, Networking, File System, and Driver Manager were all
moved into the kernel’s address space
All threads from a kernel mode driver run in
the kernel process context
They have full access to the kernel space
Security checks are not required to run or call APIs
Kernel mode APIs
The exact same API interface is used, but without the overhead
of a system privilege level change or argument marshalling
These resolve to quick calls through a vector table
Windows CE 6.0 memory model was re-architected
GWES, Networking, File System, and Driver Manager were all
moved into the kernel’s address space
All threads from a kernel mode driver run in
the kernel process context
They have full access to the kernel space
Security checks are not required to run or call APIs
Kernel mode APIs
The exact same API interface is used, but without the overhead
of a system privilege level change or argument marshalling
These resolve to quick calls through a vector table
Kernel mode drivers are ideal for real-time work
Software-based real-time measurement tool
Code in kernel is minimally instrumented to
provide the data
Software-based real-time measurement tool
Code in kernel is minimally instrumented to
provide the data
Measures both ISR and IST latencies
ISR latency
From IRQ to ISR
IST latency
From the end of the ISR to the start of the IST
Software-based real-time measurement tool
Code in kernel is minimally instrumented to
provide the data
Measures both ISR and IST latencies
ISR latency
From IRQ to ISR
IST latency
From the end of the ISR to the start of the IST
Enabled for all sample platforms
Software-based real-time measurement tool
Code in kernel is minimally instrumented to
provide the data
Measures both ISR and IST latencies
ISR latency
From IRQ to ISR
IST latency
From the end of the ISR to the start of the IST
Enabled for all sample platforms
Can be used to test response under varying
system load
This does scheduler performance timing tests
This does scheduler performance timing tests
Tool enables you to determine how long it takes
to perform basic kernel tasks such as—
Acquire or release a critical section
Wait or signal an event
Create a semaphore or mutex
Yield a thread
Call system APIs
This does scheduler performance timing tests
Tool enables you to determine how long it takes
to perform basic kernel tasks such as—
Acquire or release a critical section
Wait or signal an event
Create a semaphore or mutex
Yield a thread
Call system APIs
These things can be good to know in scenarios
when more than one time-critical situation can
be occurring
Shows interaction between processes, threads, and interrupts
Shows interaction between processes, threads, and interrupts
Shows events that affect scheduling and system performance
Shows interaction between processes, threads, and interrupts
Shows events that affect scheduling and system performance
Shows thread state and migration
We want to:
Take an existing BSP
Track the occurrence of an external event via a hardware
interrupt.
Show when CE initially responds to the external event
through an ISR
Show when CE responds to that even in an IST.
We want to:
Take an existing BSP
Track the occurrence of an external event via a hardware
interrupt.
Show when CE initially responds to the external event
through an ISR
Show when CE responds to that even in an IST.
To do this, we will:
Watch the MMC card-detect signal.
Have an interrupt triggered on card insertion.
Turn on an LED when the ISR starts
Turn off the same LED when the IST starts.
Watch the LED signal relative to the MMC card-detect
signal to see how long it takes for CE to respond.
We want to:
Take an existing BSP
Track the occurrence of an external event via a hardware
interrupt.
Show when CE initially responds to the external event
through an ISR
Show when CE responds to that even in an IST.
To do this, we will:
Watch the MMC card-detect signal.
Have an interrupt triggered on card insertion.
Turn on an LED when the ISR starts
Turn off the same LED when the IST starts.
Watch the LED signal relative to the MMC card-detect
signal to see how long it takes for CE to respond.
To watch the signals we’ll use an oscilliscope.
IRQ

CPU

SOFTWARE
IRQ

CPU

IST

SOFTWARE
IRQ

CPU

IST

SOFTWARE
IRQ

CPU

IST

SOFTWARE
IRQ

CPU

IST

SOFTWARE
IRQ

CPU

ISR IST

SOFTWARE
IRQ
IRQ

CPU

ISR IST

SOFTWARE
IRQ

CPU

ISR IST

SOFTWARE
IRQ

CPU

ISR IST

SOFTWARE
CPU

IST

SOFTWARE
 Scheduler

CPU

IST

SOFTWARE
 Scheduler

CPU

IST

SOFTWARE
CPU

IST

SOFTWARE
CPU

IST

SOFTWARE
CPU

IST

SOFTWARE
Events we will see on the scope:
Card Insertion
ISR Response (turn on LED)
IST Response (turn off LED)
Events we will see on the scope:
Card Insertion
ISR Response (turn on LED)
IST Response (turn off LED)
By using a timing scale we can see how
responsive our system is.
Events we will see on the scope:
Card Insertion
ISR Response (turn on LED)
IST Response (turn off LED)
By using a timing scale we can see how
responsive our system is.
System used for test:
PXA27X (Marvell/Intel) on PhyCore270 Board
100Mhz
MMC CD and LED via GPIO
Start with the desired deadline and acceptable jitter
Start with the desired deadline and acceptable jitter
Understand the hardware
What is slow?
What can be blocked by external events?
Start with the desired deadline and acceptable jitter
Understand the hardware
What is slow?
What can be blocked by external events?
Understand the OS
Know how critical sections and other
synchronization methods work
Know what system calls can be blocked on shared resources
Start with the desired deadline and acceptable jitter
Understand the hardware
What is slow?
What can be blocked by external events?
Understand the OS
Know how critical sections and other
synchronization methods work
Know what system calls can be blocked on shared resources
Separate Real-Time threads from UI threads
Start with the desired deadline and acceptable jitter
Understand the hardware
What is slow?
What can be blocked by external events?
Understand the OS
Know how critical sections and other
synchronization methods work
Know what system calls can be blocked on shared resources
Separate Real-Time threads from UI threads
Pre-allocate critical resources
Start with the desired deadline and acceptable jitter
Understand the hardware
What is slow?
What can be blocked by external events?
Understand the OS
Know how critical sections and other
synchronization methods work
Know what system calls can be blocked on shared resources
Separate Real-Time threads from UI threads
Pre-allocate critical resources
Use buffers to communicate between time-critical
system components and applications/UI
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.