Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

developerWorks Premium An all-access pass to building your next great app!

Sign up

developerWorks Technical topics AIX and UNIX Technical library

Process priority and control on AIX

Managing processes on AIX Systems
Managing processes is quite straightforward with tools like kill and nice, but what happens when you want to provide
even finer management control over your processes? You can assign processes and threads to specific processors in
a multi-processor system using AIX®, but how do you chose the right applications and organize a larger system in
order to optimize the applications appropriately? In this article, discover the tools available to you for organizing your
processes, and take a look at the theory behind organizing and choosing processes and how to prioritize effectively.

Share:
Ken Milberg is a Technology Writer and Site Expert for techtarget.com and provides Linux technical information and support at
searchopensource.com. He is also a writer and technical editor for IBM Systems Magazine, Open Edition. Ken holds a bachelor's degree in
computer and information science and a master's degree in technology management from the University of Maryland. He is the founder and
group leader of the NY Metro POWER-AIX/Linux Users Group. Through the years, he has worked for both large and small organizations and
has held diverse positions from CIO to Senior AIX Engineer. Today, he works for Future Tech, a Long Island-based IBM business partner.
Ken is a PMI certified Project Management Professional (PMP), an IBM Certified Advanced Technical Expert (CATE, IBM System p5 2006),
and a Solaris Certified Network Administrator (SCNA). You can contact him at kmilberg@gmail.com.

20 March 2007
Also available in Chinese Russian

Introduction
As an AIX® administrator, you should already know the basics of how to
work with processes, including researching, prioritizing, and killing them. You
should also know how to tune your processes and optimize them
accordingly, using the various tools at your disposal. These tools include
some of the more recent tools available to you in AIX 5.3. To provide
effective process control on your system, it is imperative that you understand
the definition of processes and threads and the difference between them. This article also covers the ps,
nice, and schedtune commands, as well as the Process Monitor Console (procmon), AIX Workload
Manager (WLM), and other tools available to you. Let's start with definitions of processes and threads:

Processes -- A process is an activity within the system that is started with a command, shell script, or
another process.
Threads -- A thread is an independent flow of control that operates within the same address space as
other independent flows of controls within a process. A kernel thread is a single sequential flow of
control.

Another way of looking at this is that the process is the entity that the operating system uses to control
the use of system resources, while the threads control actual processor-time consumption. Most system
management tools still require you to refer to the process rather then the thread. The process itself

1 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

actually owns the kernel threads and each process can have one or more kernel threads (for example,
multi-threaded applications). With threads, you can have multiple threads running on different CPUs on a
system, which really takes advantage of computers with more then one processor (Symmetric
Multiprocessing or SMP boxes). Applications can be designed to have user-level threads that are
scheduled to work by the application or by the pthreads scheduler in libpthreads. Multiple threads of
control allow an application to service requests from multiple users at the same time. With the libpthreads
implementation, user threads sit on top of virtual processors, which are themselves on top of kernel
threads. During this article, delve into more detail on the kernel aspects of a process and tools available
to help you more effectively manage your overall system. To help you manage your environment, go
through time-tested UNIX® commands and many of the new tools available to you as an AIX
administrator.

Threads and SMT

Allowing threads to run on different CPUs also allows for effective utilization of simultaneous multi-
threading (SMT). With the system in SMT mode, the processor fetches instructions from more than one
thread. Exclusive to the POWER5 architecture, the concept of SMT is that no single process uses all
processor execution units at the same time. The POWER5 design implements two-way SMT on each of
the chip's cores. The end result is that each physical processor core is represented by two virtual
processors. SMT is primarily beneficial in commercial environments where the speed of an individual
transaction is not as important as the total number of transactions that are performed. SMT is expected to
increase the throughput of workloads with large or frequently changing working sets, such as database
servers and Web servers. Workloads that are floating-point intensive are likely to gain little from SMT and
are the ones most likely to lose performance. These workloads heavily use either the floating-point units
or the memory bandwidth. Workloads with low cycles per instruction (CPI) and low cache miss rates
might see some small benefit. Generally, you should expect to see approximately a 30 percent increase
in system performance due to SMT. You must determine whether the critical processes running on your
system benefit from SMT. Critical processes typically benefit from SMT; however, if you determine
otherwise, you need to shut it down -- it comes enabled by default.

Scheduling concepts
I'll try not to spend too much time on the kernel intricacies of the AIX scheduler, but you need to get to a
better level of understanding before going into administering processes or tuning the scheduler.

Each CPU on a system has its own dedicated run queue, which is a list of runnable threads sorted by
thread priority value. There is also another run queue called the global run queue. All new threads are
placed in the global run queue. Every time the CPU is ready to dispatch a thread, this global run queue is
checked before any other run queues. When a thread finishes its time slice on the CPU, it goes back on
the run queue of the CPU it was running on. This helps AIX to maintain its processor affinity. (I'll discuss
processor affinity in more detail later.)

There are some environmental variables you can tune to increase performance on the scheduler, which
are out of scope on this article. The CPUs on the system are shared among all of the threads by giving
each thread a certain slice of time to run. The default time slice is 10 ms (for one clock tick). It can be
changed with the schedo command. Increasing the time slice can improve system throughput due to

2 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

reduced context switching. One can look at the context switching using either the vmstat or sar
commands. If the value of context switching is very high, increasing the time slice can improve
performance, but this should only be done after extensive analysis.

Regarding system modes, there are two modes that a CPU operates in: kernel mode and user mode. In
user mode, programs have read and write access to the user data in the process private region. This is
the mode that a process should accumulate its majority of CPU time on. The other mode is kernel mode.
Some of the programs that operate in kernel mode include interrupt handlers and kernel processes. Code
operating in this mode has read and write access to the global kernel address space and kernel data in
the process region when executing within the context of a process. User data within the process address
space must be accessed using kernel services. When a user program accesses system calls, it does so
in kernel mode, not user mode. You need to understand this concept when trying to interpret the output of
commands, such as vmstat and sar.

Processor affinity and binding processors

Processor affinity is a facility provided by operating systems that is used on SMP hardware. Essentially,
all the threads within the process can be bound to run on the specified processor. AIX automatically tries
to encourage processor affinity by having one run queue per CPU, which was discussed earlier. Using
process affinity settings to bind or unbind threads can help you find the root cause of hangs or deadlocks
that are difficult to debug. Some applications might also run faster if their threads are always bound to run
on one particular CPU.

In a typical SMP system, all of the processors are identical and can run any thread on its system.
Essentially, any process or thread can be dispatched to run on any processor, except for processors or
threads that are bound to run on a specific processor. This can be accomplished using the
bindprocessor command. Let's look at an example (see Listing 1).
Listing 1. Using the bindprocessor command
# bindprocessor -q

The available processors are: 0 1 2 3

From here, you can see there are four processors to call upon. The available processors are: 0 1 2 3.

This command shows which thread is bound to CPU 3 (see Listing 2).
Listing 2. Discovering which thread is bound to CPU 3
# ps -emo THREAD | grep p3
root 401544 389152 - A 0 60 1 f10001001ece2fb8 200001 pts/0
- grep p3

You can also use the SMIT fast path, smit bindproc, to help bind processes. Another way to bind a
process is within a program using the bindprocessor API available on AIX. You should understand that
these are powerful commands. When binding a process to a CPU, you can actually reduce performance
for that process if the CPU to which the process will be bound is busy while others are idle.

PS -- Nice to meet you

Let's talk about the commands that you'll generally use to identify and work with processes.

To get a long listing of files, use the following in Listing 3.

Listing 3. Getting a long listing of files

3 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jan 08 - 0:05 /etc/init
root 82126 204974 0 Jan 08 - 0:00 /usr/sbin/snmpmibd
root 86210 106640 0 Jan 08 - 0:00 /usr/dt/bin/dtcm
root 90172 123038 0 Jan 08 - 0:35 /usr/lpp/X11/bin/X -D /usr/lib/X11//rgb
-T -force :0 -auth /var/dt/A:0-DjUjUa
root 98390 1 0 Jan 08 - 8:36 /usr/sbin/syncd 60
root 106640 131160 0 Jan 08 - 0:25 /usr/dt/bin/dtsession

To further identify processes broken down by CPU hogs, see Listing 4.

Listing 4. Identify processes
# ps aux | more
USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 8196 12.9 0.0 384 384 - A Jan 08 14695:30 wait
root 57372 12.8 0.0 384 384 - A Jan 08 14542:51 wait
root 61470 12.2 0.0 384 384 - A Jan 08 13884:38 wait
root 53274 12.0 0.0 384 384 - A Jan 08 13711:38 wait
root 245938 0.0 0.0 828 856 - A Jan 08 20:17 /usr/bin/xmwlm -
root 98390 0.0 0.0 508 516 - A Jan 08 8:36 /usr/sbin/syncd
root 69666 0.0 0.0 960 960 - A Jan 08 3:46 gil
root 0 0.0 0.0 384 384 - A Jan 08 2:49 swapper
root 49176 0.0 0.0 448 448 - A Jan 08 1:13 xmgc
root 241842 0.0 0.0 23

If you want some more information on the nice value of the process, you need to use the -l flag. The NI
column is the nice value (see Listing 5).

Listing 5. Using the -l flag to get the nice value

# ps -elf
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
200003 A root 1 0 0 60 20 14001400 660 Jan 08 - 0:05 /etc/init
240001 A root 82126 204974 0 60 20 3c22b510 1264 Jan 08 - 0:00 /usr/sbin/snmpmibd
240801 A root 86210 106640 0 60 20 584d2400 2156 Jan 08 - 0:00 /usr/dt/bin/dtcm
240001 A root 90172 123038 0 60 20 5136 f1000100224650e0 5136 Jan 08 - 0:35
/usr/lpp/X11/bin/X
-D /usr/lib/X11//rgb -T -force :0 -auth /var/dt/A:0-DjUjUa
240001 A root 98390 1 0 60 20 41a5400 508 * Jan 08 - 8:36 /usr/sbin/syncd 60
240001 A root 106640 131160 0 60 20 3816a400 1880 Jan 08 - 0:25 /usr/dt/bin/dtsession
40001 A root 123038 1 0 60 20 5c153400 380 Jan 08- 0:00 /usr/dt/bin/dtlogin
-daemon

The command in Listing 6 gives you the top three performing processes, including their nice values.
Listing 6. Getting the top 3 performing processes
# ps -elf | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 15
40401 A nobody 323762 127128 0 60 20 602dc400 660 f1000600002daa08 Jan 08 - 0:00
/usr/HTTPServer/bin/httpd -d /usr/HTTPServer -k restart
40001 A nobody 319662 127128 0 60 20 6c35f400 1336 * Jan 08 - 0:00
/usr/HTTPServer/bin/httpd -d /usr/HTTPServer -k restart
40001 A nobody 307358 127128 0 60 20 3834a400 1340 * Jan 08 - 0:00
/usr/HTTPServer/bin/httpd -d /usr/HTTPServer -k restart
240001 A daemon 254084 204974 0 60 20 58272400 1364 Jan 08 - 0:00
/usr/sbin/rpc.statd -d 0 -t 50

Now that you know which processes are killing the system (you could also use topas or nmon), what can
you do about it? Wouldn't it be nice if there were a command to let you prioritize how the kernel
schedules its processing? Of course, there is a command and also another command that allows you to
prioritize a process again that is already running. The commands are nice and renice, respectively. A
user's job in AIX carries a base priority level of 40 and a default nice value of 20. Together, these two
numbers form the default priority level of 60. The vast majority of jobs have this value. The higher the
default priority level number, the lower the priority of the job. If you want to start a job with a lower priority,
you can try the command in Listing 7.
Listing 7. Starting a job with a lower priority
# nice -n 10 thisjob

The command in Listing 7 adds 10 to the default of 20 and creates the new nice value of 30, with the
priority of 70.

4 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

Running the command in Listing 8 caused process 1683 to have a nice value of 30.
Listing 8. Causing process 1683 to have a nice value of 30
# renice -n 10 -p 1683

The procmon utility

While there are many performance tools that come with the base operating system of AIX, perhaps the
best performance monitoring tool introduced recently (in AIX 5.3) is procmon. This utility displays a
dynamic, sorted list of processes, with all sorts of information about them. It allows the execution of basic
administration commands, such as nice, renice, and kill. The procmon tool runs on the Performance
Workbench platform, which is an Eclipse-based tool and also has a nice little graphical user interface to
system activity. To start up procmon, start perfwb, which launches Eclipse with the procmon plug-in (see
Listing 9). You need the bos.perf.gtools.perfwb fileset.
Listing 9. Starting perfwb
# /usr/bin/perfwb

The procmon tool displays the following, by default:

How long a process has been running

How much CPU resource the processes are using
Whether processes are being penalized by the system
How much memory the processes are using
How much I/O a process is performing
The priority and nice values of a process
Who has created a particular process

It also has the following options to perform:

procfiles
proctree
procsig
procstack
procrun
procmap
procflags
proccred
procldd

The process table, the main component of procmon, displays the various processes that are running on
the system, and they can be ordered and filtered based on the user configuration. Although, by default,
the number of processes listed in the table is 20, the processes can easily be changed using the Table
Properties panel from the main menu. See Resources for more information about this important tool.

5 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

WLM
WLM is a complex tool you can use for performance monitoring, gathering accounting data, and also for
managing the load on a standalone system. You can also use it (with DLPAR) as a resource provisioning
tool in a partitioned environment. It is an efficient way for system administrators to monitor and control
resource usage by processes.

The WLM feature on AIX provides a set of tools that assist in gathering performance statistics and
providing you with a mechanism to control allocation of resources to processes. It is intended for use with
large systems running multiple applications, databases, and transaction processing systems, where
workloads are combined into a single large system ("vertical" server consolidation). It provides the
flexibility for dividing system resources between jobs without having to partition a system. You can use
WLM to prevent different classes of jobs from interfering with each other and to allocate resources based
on the requirements of different groups of users. Many people get this confused with Partition Load
Manager (PLM), which is a resource manager that assigns and moves resources based on defined
policies and utilization of the resources in an IBM System p™ environment, which contains Advanced
Power Virtualization. PLM is able to manage memory, dedicated processor partitions, and shared
processor partitions, using micro-partitioning technology to readjust the resources. This adds additional
flexibility to the micro-partition flexibility offered by the POWER Hypervisor. Unfortunately, PLM has no
knowledge of the importance of any workload running in the partitions and, therefore, cannot readjust
priority based on the changes of workload types.

Conclusion
Process management is far from the most exciting aspect of being a UNIX systems administrator. While it
can be tedious, it is a necessary evil of systems administration. You always have to field questions on
how to speed up a process and why it is taking it so long to finish. You need to identify problematic
processes and do everything you can to make them run more efficiently. You also have to identify the
best tool to do your job, whether that is simply running a ps command and using renice, working with a
new performance utility such as procmon, or introducing an enterprise-wide process scheduling utility like
WLM to help manage all your system processes more efficiently. Try to do some additional research on
the kernel concepts of processes and scheduling prior to introducing any new elements. Before you do
the work, it will be much more helpful in the long run if you really understand what it is you are doing.

Resources Dig deeper into AIX and Unix on

Learn developerWorks
Overview
"Processor Affinity on AIX" (developerWorks, November 2006): Learn how to
New to AIX and Unix
use processor affinity to restrict a process and run it only on a specified
Technical library (tutorials and more)
central processing unit (CPU).
Forums
AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of
Community
information relating to all aspects of AIX systems administration and
Downloads and products
expanding your UNIX skills.
Open source projects
Safari bookstore: Visit this e-reference library to find specific technical Events
resources.

developerWorks technical events and webcasts: Stay current with developerWorks Premium

6 of 7 03/07/2016 10:52 PM
Process priority and control on AIX http://www.ibm.com/developerworks/aix/library/au...

developerWorks technical events and webcasts. Exclusive tools to build your next
great app. Learn more.
Podcasts: Tune in and catch up with IBM technical experts.
developerWorks Labs
Future Tech: Visit Future Tech's site to learn more about their latest offerings. Technical resources for innovators
and early adopters to experiment
Get products and technologies with.

IBM trial software: Build your next development project with software for IBM evaluation software
Evaluate IBM software and
download directly from developerWorks. solutions, and transform
challenges into opportunities.
Discuss
Participate in the developerWorks blogs and get involved in the
developerWorks community.

AIX Power Users Group: Participate and get involved in the AIX community.

Participate in the AIX and UNIX forums

7 of 7 03/07/2016 10:52 PM