CS530: Graduate Operating Systems (Spring 2006)

Scheduler Activations
(Thomas Anderson et al., TOCS ’92)
 Kernel-Level Threads (1)
ƒ Implementation

TCB

PCB

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 2

 Kernel-Level Threads (2)
ƒ Pros
• Cheaper than processes.
• Scheduling/management done by the kernel.
– Possible to overlap I/O with the computation.
– Multiple CPUs can be exploited.

ƒ Cons
• Still too expensive.
• Thread state in the kernel.
• Need to be general to support the needs of all
programmers, languages, runtimes, etc

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 3

 User-Level Threads (1)
ƒ Implementation
• Views each process as a “virtual processor”.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 4

 User-Level Threads (2)
ƒ Pros
• Fast
• Portable
• Flexible

ƒ Cons
• Invisible to OS; OS can make poor decisions.
• Cannot exploit multiple CPUs.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 5

 Scheduler Activations (1)
ƒ Goals
• The performance and flexibility of user-level threads.
– The performance of user-level thread systems in the common
case.
– Simplify application-specific customization.
» Scheduling policy, different concurrency models, etc.
• The functionality of kernel threads.
– No processor idles in the presence of ready threads.
– No high-priority thread waits for a processor while a low-
priority thread runs.
– When a thread traps to the kernel to block, the processor can
be used to run another thread from the same or from a
different address space.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 6

 Scheduler Activations (2)
ƒ Simple solution:
• How about to provide multiple kernel threads to a user-
level thread system?

• Problems:
– Preempting lock holder?
– Scheduling an idle thread?
– Preempting high-priority thread?
– Running out of kernel threads?

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 7

 Scheduler Activations (3)
ƒ Observations
• Kernel threads are the wrong abstraction for supporting
user-level thread management.

• The kernel needs access to user-level scheduling

information.

• The user-level thread system needs to be aware of

kernel events.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 8

 Scheduler Activations (4)
ƒ Basic idea

User-level Thread System

Control the number of threads to run
Notifies the user-level on its allocated processors.
thread system whenever
system call
the kernel changes the VP VP VP
number of processors upcall Notifies the kernel when
assigned to it. the application needs
more or fewer processors.
Kernel
Control the number of processors
given to each address space.

CPU CPU CPU CPU

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 9

 Scheduler Activations (5)
ƒ Scheduler Activation:
• Serves as a vessel, or execution context, for running
user-level threads (similar to a kernel thread).
• Notifies the user-level thread system of a kernel event
via upcalls.
• Requires two stacks:
– A kernel-level stack: used during system calls.
– A user-level stack: used during upcalls.
– Note: Each user-level thread has its own stack.
• Activation control block
– For saving the processor context of the activation’s current
user-level thread, when the thread is stopped by the kernel.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 10

 Example (1)
ƒ T1: The kernel allocates two processors.
User
Program

User-Level (1) (2)

(3) (4)
Runtime
System

(A) (B)
Operating
System Add Add
Kernel Processor Processor

Processors

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 11

 Example (2)
ƒ T2: Thread 1 blocks in the kernel for I/O.
User
Program

User-Level (1) (2) (3)

(3) (4)
Runtime
System B
(A) (B) (C)
Operating
System A’s thread has blocked
Kernel

Processors

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 12

 Example (3)
ƒ T3: Thread 1 completes the I/O.
User
Program

User-Level (1) (2) (3)

(4)
Runtime
System

(A) (B) (C) (D)

Operating A’s thread and
System B’s thread can
Kernel continue

Processors

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 13

 Example (4)
ƒ T4: Thread 1 resumes.
User
Program

User-Level (3) (1) (2) (4)

Runtime
System

(C) (D)
Operating
System
Kernel

Processors

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 14

 Upcall Points
ƒ Kernel Æ User-level Threads System
• Add this processor (processor #)
– Execute a runnable user-level thread.
• Processor has been preempted
(preempted SA # and its machine state)
– Return to the ready list the user-level thread that was executing
in the context of the preempted SA.
• Scheduler activation has blocked (blocked SA #)
– The blocked SA is no longer using its processor.
• Scheduler activation has unblocked
(unblocked SA # and its machine state)
– Return to the ready list the user-level thread that was executing
in the context of the blocked SA.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 15

 System Call Points
ƒ User-level Threads System Æ Kernel
• Add more processors (additional # of processors)
– Allocate more processors to this address space and start them
running SAs.
• This processor is idle ()
– Preempt this processor if another address space needs it.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 16

 Issues (1)
ƒ Reallocating a processor
• The kernel stops the activation
A2.
• The kernel upcalls to SB using
P2 with a fresh activation A4.
User-level User-level
• The kernel takes off another Scheduler SA Scheduler SB

processor P1 in the old address

A1 A5 A2 A3 A4
space.
• The kernel upcalls to SA with P1 P1 P2 P3 P2
A1 notifying that two
activations A1 and A2 have
been preempted.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 17

 Issues (2)
ƒ Processor allocation/release
• An address space gives hints.
– It has more runnable threads than processors, or
– It has more processors than runnable threads.
– Only hints: processor allocation is not guaranteed.

• Idle processors may be left in the address space to

avoid the overhead of processor reallocation.

• Dishonest or misbehaved programs?

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 18

 Issues (3)
ƒ Priority scheduling among threads
• The user-level thread system asks the kernel.
– How?
• The kernel preempt the thread’s processor.
• The kernel does an upcall.
• The user-level thread system schedules another
• can ask the kernel to preempt another high-priority
thread on that processor.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 19

 Issues (4)
ƒ Critical sections
• What if the preempted thread is in the critical section?
– Poor performance or deadlock
• Solution based on “recovery”: the preempted thread is
continued temporarily via user-level context switch.
• Performance enhancements
– Make a copy of each critical section.
– The copy yields the processor back to the scheduler at the end
of the critical section.
– Runtime checks using the section begin/end addresses.
– Normal execution uses the original copy.
» Imposes no overhead in the common case!

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 20

 Issues (5)
ƒ Application transparency
• The application is free to build any other concurrency
model on top of scheduler activations.
• The kernel needs no knowledge of the data structures
used to represent parallelism at the user level.

• Scheduler activations provide a “mechanism”, not a

“policy”.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 21

 Solaris 2 Threads

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 22

 Solaris 9 Threads
ƒ Things change.
• Back to one-to-one model
• M:N model is too complex
– Signal handling
– Automatic concurrency management
– Poor scalability due to an internal lock in user-level thread scheduler
– Advances in kernel thread scalability
• The quality of an implementation is often more important.
– Code paths were generally more efficient than those of the old
implementation.
– More robust and intuitive.
– Simpler to develop and easier to maintain.
• Binary compatibility is preserved.
(cf.) Linux threads: NPTL (1:1) vs. NGPT (M:N)

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 23

 Conclusions
ƒ Implementations
• Topaz (original implementation)
• Taos, Mach 3.0, BSD/OS, NetBSD (Usenix ’02)
• Digital Unix (Compaq Tru64 Unix), Solaris 2.6

ƒ Lessons
• Make the common case fast.
• Separating policy from mechanisms.
• Export your functionality out of the kernel for improved
performance and flexibility.

CS530 (Spring 2006) -- Jin-Soo Kim (jinsoo@cs.kaist.ac.kr) 24