2009 - The Waf Book (v1.5.4)

10/10/2010 The Waf Book (v1.5.
4)
The Waf Book (v1.5.4)
Thomas Nagy
C opy righ t © 2008-2009 Thomas Nagy
C opies of this book may be redistributed, v erbatim, and for non-c ommercial purposes. The
lic ense for this book is by -nc-nd lic ense
Table of C ontents
Prefac e
Abo ut build sy stems in general

The W af approach
Objectiv es of this book
1. Gettin g Started
Obtaining W af
Inst alling
R unning W af
2. Introduc tion to the W af sc ripting sy stem
Empty projec t ex ample

W o rkflow illustration on a more c omplic ated ex ample
Building a projec t - Intro duction to Tasks
C onfiguring a project
Add ing c ustom co mmand-line options
Inst alling targets
3. The c onfiguration sy stem
Ov erv iew of the c onfiguration files

The c onfig uration env ironments
C onfigurat ion helper ex ec ution
Add ing new configuration helpers
4. Pac kaging and redistributing the sourc e c ode
Quic k tarball creation

C ustom arc hiv es
C leaning the projec t tree
5. The Task sy stem
Task c reation and ex ec ution

Task ex ec ution
Task ex ec ution in parallel
Task ex ec ution order
Ex ec uting tasks only wh en something changes
6. Task encapsu lation using t ask generators
Ex p licit rules for simple transformations (Make-like)

Imp licit proc essing based on file name or ex tension
C ustom processing by task generator method s
Task generator method ex ecution
7. The sc heduler for ex ec uting the t asks
The task ex ec ution model

Job contro l
W eak task order c onstraints
8. C and C ++ p rojec ts
C ommon sc ript for C /C ++ applic ations

Library interac tion (uselib)
C ustomizing objec t files
Using c onfiguratio n headers
The inc lude sy stem
C onfigurat ion helpers
Pkg-c onfig
9. Adv anc ed sc enarios
Simple file transfo rmations

Same targets, different c onfigurations (v arian ts)
Building th e c ompiler first
W riting the output of a program into a header
A c ompiler produc ing sourc e files with names unknown in ad v anc e
A task with out any file dependency
10. C ustomizing the sc ripting sy stem
Pro v iding c ustom c ommands

C haining c ommands
Pro v iding a c ustom c ommand c ontex t
C reating aliases / Injectin g new commands
freehackers.org/~tnagy/…/single.html 1/43
10/10/2010 The Waf Book (v1.5.4)
11. Using the d ev elopment v ersion
Trac ing the ex ecu tion

Benc hmarking
Pro filing
Trac ing parallel task ex ec ution
Obtaining the latest source cod e
Pro gramming c onstraints
12. Ov erv iew of the W af arc hitecture
The c ore library

Build c ontex t instanc es
Ov erv iew of the W af ex ec ution
Performanc e and build acc uracy
13. C onc lusion

Glossary
List of T ables
11.1. D eb ugging zones

12.1. The core library
12.2. Build c ontex t persistenc e
12.3. Build c ontex t ac c ess
13.1. Rec ommen ded links
Preface
Table of C ontents
About build sy stems in general

The W af approac h
Objec tiv es of this book
About build systems in general

As software is bec oming inc reasingly complex , the process of c reating software is bec oming
more co mplex t oo. Today 's software uses v arious languages, requires v arious c ompilers, and
the inpu t data is spread into many files.
Software is now used to ex press the process of building software, it can take the form of
simple sc ripts (shell scripts, makefiles), of co mpilers (C Make, Qmake), or c omp lete
applications (SC ons, W af). The term " build sy stem" is used to design th e tools used to build
applications.
The Waf approach

Build sy stems make assumptions on software it is try ing to build, and are ty pic ally limited
where it comes to proc essing other languages or d ifferen t projec ts. For ex ample, Ant is
better su ited than Make for managing Jav a project s, but is more limited than Make for
managing simple c pro jec ts.
There are so many different languages that c reatin g a build sy st em to support all the
possible kind of projec ts out-of-the-box is likely to be difficult. It looks much more
interesting to c reate one whic h c an be ex tended easily .
The W af approac h is to prov ide a minimalistic and v ery flex ible framework, whic h is p ac kaged
it with a signific ant amount of default build rules to be usable for most p rojec ts, and many
recipes for the most co mplic ated sc enarios.
Objectives of this book

The objec tiv es of this book is to ex p ose the use of the W af build sy stem though the use of
W af in practic e, the desc ription of the W af ex tension sy stem, and detailed rev iew of the W af
internals. Those who are impatient to see W af in ac tion should open the folder demo s/ in
the W af distribution, and start looking at th e ex amples.
The first c hapters cont ain a d etailed descrip tion of the basic use of W af. The c hapters on
C onfiguration, Tasks, and Task generators desc ribe the internals. These c hapt ers should be
read c arefully before try ing to ex tend W af. The nex t c hapters c ontain ex amples for the
languages such as C ++, and for special sc enarios whic h may occ ur in the most c omplic ated
builds. The last c hapters are prov ided as a referenc e for t hose who may need to mod ify W af
itself.
Chapter 1. Getting Started

Table of C ontents
Obtainin g W af
In stalling
Running W af
Obtaining Waf
The W af projec t is located on Google C ode The c urrent W af v ersion req uires P y thon ≥ 2.3
c ompiled with support for the bzip2 c ompression algorith m. Installing W af is op tional, if y ou
want to install W af on the sy stem, ju mp to the nex t sect ion.
Using the Waf binary

10/10/2010 The Waf Book (v1.5.4)
The W af binary is ready to use and c an be ex ec uted direc tly . Just rename it into "waf":
Generating Waf from source

The W af binary is usually generated from the sourc e distribution:
Installing
W af inst allation is optional an d there are many adv antages of no t installing W af, amo ng
them:
1. Inst allation: installation is c umbersome, and requires administrativ e priv ileges

2. W af v ersions: users rarely install the appropriate software v ersion (too old, too new)
3. Bug s: newer W af v ersions may result in broken build s
4. Size: W af is ex tremely small (about 70kB)
5. Operating sy stems: W af c annot be installed o n W ind ows (y et)
If, despite the adv antages enumerated prev iously , y ou really want to in stall th e W af
application on the sy stem, y ou may use the following c ommand on Unix -like sy stems such as
Linux :
Root priv ileges may be required dep ending on the final loc ation.
W af c an be uninstalled using:
Running Waf
The W af sc ript can be used direc tly :
Or, if the ex ec utable p ermissions are not set properly :
By default, the W af sc ript unc ompresses its library in the c urren t direc tory , in a hidden
fo lder. This means the W af sc ript must resid e in a writable folder. The naming is allow
different W af v ersions to run from the same folder:
Chapter 2. Introduction to the Waf scripting system

Table of C ontents
Empty project ex ample

W orkflow illustration on a more c omplicated ex amp le
Building a projec t - Introduc tion to Tasks
C onfiguring a p rojec t
Adding c ustom c ommand-line option s
In stalling targets
Empty project example

The W af sc ripts are based on the following conc epts:
Sy stem files: files and folders whic h do not belong to the project
Sou rce directory : direc tory con taining the sourc e files for an applic ation, it is meant to
be ultimately pac kaged, and redistributed to other dev elopers or to end u sers.
10/10/2010 The Waf Book (v1.5.4)
Build direc tory : all files produc ed by W af will be outp ut in that fold er
The W af philosophy is to av oid the pollution of the sourc e direct ory by letting all files into
the build direc tory . Th e build directo ry c an be loc ated on the sy stem, out of the sourc e
directory (like in /tmp for ex ample).
W hen W af is lau nched, it looks for the user-defined W af sc ripts whic h are files written in the
Py thon languag e. The most important one is the top-lev el W af sc ript file in wh ich sev eral
fu nction s and attributes must be prov ided for defining a v alid W af project:
srcd ir: string representin g the sourc e d irec tory

bldd ir: string representin g the build directory
set_ options: func tion used for adding custom c ommand-lin e options
c onfigure: func tion c alled for c onfiguring the projec t
build: func tion used for building the projec t
The top-lev el W af sc ript file name is "wsc ript"
A simple empty W af project c an be declared using a wsc ript file c ontaining the following
Py thon c ode:
The minimum workflow for an y projec t c onsists of the following steps:
C onfiguring the project: searc hing for sy stem parameters and c ompilers
Building th e project: building the software
To do th is, the followin g shell c ommands will be:
Workflow illustration on a more complicated example

Additional func tions are usually prov ided in the top-lev el wsc ript file:
APP NAME: projec t name

VER SION: projec t v ersion
init: func tion called before any t hing is ex ec uted
shutdown: func tion c alled after a succ essful build
dist: func tion c alled for p ac kaging the application
W e will now pro v ide a more c omplicated wsc ript file, ex ec ute the ty pical c ommands, and
look at the outp ut:
The outp ut will be:
10/10/2010 The Waf Book (v1.5.4)
3 4
A few files will b e produc ed by W af, let us look at them now:
A build d irec tory was c reated with the name giv en in the wsc ript file. It c ontain s a
c onfiguration log, and v arious c ache files. The last command produc ed a tarball of th e
projec t (an arc hiv e) c ontaining the sourc e files. For now it only c ontains the wscript file.
The ex ec ution c an be summed up in the following diagram:
10/10/2010 The Waf Book (v1.5.4)
Building a project - Introduction to Tasks

The func tion build indicates the start of the ac tual c onstruction of the software, it is used
fo r two main purposes:
D ec laring the location of sc ripts that p rov ide other build fun ctions
D ec laring the ac tual targ ets to build
Subdirectory recursio n
The func tion build may reference build func tions in wsc ript present in sub fold ers, for
ex ample:
The top-lev el wsc ript may ind icate th ere is a build func tion to ex ec ute in src /wsc ript using:
The wsc ript files that are not on the top-lev el often c ontain a large build funct ion, with no
additional c onfiguration or c ommand-line op tions. In this c ase th e file only c ontain on e
method and the indentation is an annoy anc e. For this reason, the wscript_build files
10/10/2010 The Waf Book (v1.5.4)
c ontain the body of a build func tion defined in a wsc ript file. A folder may only contain either
a wsc ript or a wsc ript_build but not both.
Waf task declaration

The proc ess of building a piec e of software require the transformation of sourc e c ode (input )
into the result (output ). Sinc e the sourc e c ode is usually split in v arious files, the
c onstruc tion of the result req uires th e proc essing of eac h of the files. Sev eral new c onc epts
deriv e from this:
Intermediate resu lts (objec t files) may be produc ed to speed up th e build (no need to
rebuild ev ery thing when small c hanges occ ur)
Sou rce files may be proc essed in parallel to sp eed up the build
Sou rce files may be tran sformed into o ther so urce files (seq uenc es of transformations)
In a more general sc ope, the input n ec essary for a build may co me from arbitrary data
instead o f files (database, c ommands to ex ec ute). The co ncepts of input, output and
transformatio n and the transformation co nstraints (in parallel, in sequenc e) remain
identical.
In W af, the operation of tran sforming data is performed by tasks , whic h are instances of
the c lass TaskBase (in the W af library ). The tasks in the following ex ample illustrate the
declaration of Tasks from the base c lass. The task instanc es must be loc ated in a build
c ontex t (the fu nction build) to be taken int o ac count in the build phase, and t o obtain the
referenc e on the sourc e and build folders:
The ex ec ution t rac e will be th e following:
3 4
Task generator encapsulation

In prac tic e, when lots of tasks are declared one by one, scripts t end to become c omp licated,
and c ontain a lot of redundant c ode. Special objec ts named task generators are prov ided
to fac ilitate the creation of sev eral tasks at onc e. In the following ex ample, a task generator
is used to create the c ompilation and link tasks needed by a c ++ projec t.
3 4
This ex ample prov ides a mod ified co nfigure func tion, whic h is the topic of the nex t sec tion.
The task generator sy stem an d the default generators will be rev iewed in detail in C hapter 6,
Task enc apsulation using task generators .
Configuring a project
The func tion co nfigure is used for four main purposes:
Indic ating new sc ripts that prov ide c onfiguration fun ctions to c all
Pro v iding c onfiguration c ontex t s to ho ld persistent d ata
Pro v iding c onfiguration helpers
10/10/2010 The Waf Book (v1.5.4)
Loading v arious W af tools
Indicating configuration sub folders

The func tion co nfigure may referenc e c onfigure func tions in wsc ript present in sub folders,
fo r ex ample:
The top-lev el wsc ript may ind icate th ere is a c onfigure func tion to ex ec ute in src/wsc ript
using:
As a mat ter of fac t, any wscript file may ind icate a sub-c onfiguration fu nction to ex ec ute.
Storing and loading configuratio n parameters
The ex ec ution will result in:
3 4
This c onfiguration modifications (on c onf.env ) are persistent, they are reused during the
build (using bld.env ). Let us look at the files produc ed:
The c ontents of the file default.cache.py is repro duced here:
Using configuration helpers

Setting u p man y parameters manually is a tedious proc ess, and W af prov ide c onfiguration
helpers t o automate the proc ess of finding the settings. For ex ample:
W ill produc e the following results:
10/10/2010 The Waf Book (v1.5.4)
3 4
Loading Waf tools

The W af tools are ex tensions that prov ide t ask ty pes (for languages su ch as c , jav a, ...),
more co nfiguration tests, and command-line options. To make c ertain that the W af tools
used during the c onfiguration are also av ailable during th e build, they are stored during the
c onfiguration and load ed automatic ally afterwards. Here is an ex ample:
By default, tools are loaded from the W af library , but it is possible to load tools from the
sourc e d irec tory using the fo llowing:
Adding custom command-line options

The func tion set_o ptions is used for three main purposes:
D ec laring c ustom command-line options

Indic ating that sc ripts prov ide c omman d-line options
Add ing c ommand-line op tions from W af tools
Adding options directly

W af uses the optparse module from Py thon for adding co mmand-line options, for ex ample:
The c ommand-line opt ions can be display ed afterwards:
3 4
and used like th is:
Adding options from wscript files located in sub folders
10/10/2010 The Waf Book (v1.5.4)
The func tion set_options may referenc e bu ild func tions in wsc ript present in sub folders, for
ex ample:
The top-lev el wsc ript may ind icate th ere is a build func tion to ex ec ute in src /wsc ript using:
Adding options from Waf tools

W af tool options are n ot added by d efault b ec ause the ex c ess of c ommand-line optio ns is
annoy ing. Here is an ex ample for adding the command-line options defined from a W af tools
named c ompiler_cx x
Installing targets
Targets are installed when W af is run with t he following c ommand-line waf install. If the
targets are not built, they will be installed as soon as they are built. W e will now desc ribe the
three main way s of installing the targ ets.
Installing files directly

Sourc e files may be installed directly by usin g the install methods of the build c ontex t bld:
The v ariables in curly brac kets are substitut ed by v alues found in bld.env , for ex ample
${PREFIX } is substituted by the v alue of bld.env['PREFIX '] . An env ironmen t parameter
may be prov ided if nec essary :
Though the substitutio n is on ly performed on the first parameter, it is possible to re-use the
method bld.get_install_path or the func tion Utils.subst_vars
Installing targets (most task generators)

By prov iding the attribute install_path , tasks generators will install the target (in this
ex ample, a prog ram) in the giv en path.
The v ariables in curly brac kets are substitut ed as desc ribed in the prev ious paragraph, using
main.env instead of bld .env . The chmod attribute is optional, and is used to c hange the
permissions of t he inst alled file (by default it is set to 064 4).
To prev ent the installation of a target (by d efault, programs and shared libraries are
10/10/2010 The Waf Book (v1.5.4)
installed), set install_path to None
Custom tasks
Task installation is disabled by default for most task ty pes. If a method named install is
prov ided, it will be ex ecuted immediately after the task is suc c essfully ex ec uted. Unlike the
c ompilation, the file installation is not run in parallel (there is little gain in c opy ing files in
parallel).
Chapter 3. The configuration system

Table of C ontents
Ov erv iew of the configuration files

The c onfiguration env ironments
C onfiguration helper ex ecutio n
Adding new c onfigurat ion helpers
Overview of the configuration files

C onfiguring a p rojec t c onsists in finding the parameters that will be used durin g the build
step
Sev eral c onfigu ration files are produc ed and the c onfiguration parameters are stored in the
fo lder c4che loc ated under the build direc tory
Contains a reference to the build directory and contains the command-line

configuration flags. It is located at the root of the source directory.
Contains serialized data corresponding to the build information (cache,
filesystem representation, etc)
Contains the Waf version and the list of Waf m odules to open automatically
Represents a configuration environment (parameters such as configuration
flags used during the build)
Contains the configuration test execution details
The file default.cache.py is called a c onfiguration env iron ment, and it may be edited by
hand. W e will giv e the details on the following sect ion. The other files are not meant t o be
modified by hand, ev en if they are editable.
The configuration environments

The build phase uses c onfiguration env ironments c reated during the c onfiguration ph ase.
The c onfiguration env ironments are Pytho n Dictio naries whic h map key s to v alues. The
c onfiguration env ironment files created take the fo llowing form:
10/10/2010 The Waf Book (v1.5.4)
By default, the c onf object c omes with a default built-in c onfiguration env iron ment:
To replac e the default c onfigu ration env ironment, a new one mu st be c reated and assigned:
Duplicates the default configuration environment (shallow copy)

Binds the configuration environment to the name 'debug'
Replaces the default configuration environment by the 'debug' one
Changes the CXXFLA GS entry in the 'debug' configuration environment. Since
we are using a shallow copy, we have to use the methods append_value ,
prepend_value or append_unique instead of '+=' or '[].append'
In the prev ious ex ample, the file con tents will be:
In the build sec tion, the v alues are retriev ed using bld.env_of_name , for ex ample
Configuration helper execution

C onfiguration helpers are methods prov ided by the c onf objec t to help finding parameters,
fo r ex ample the metho d conf.find_program
W hen a test fails, the ex c eption C o nfigure.C o nfiguratio nE xception is raised.
W hen c alling the c onfiguration from a tool, for ex ample:
sev eral different c onfig uration helpers may be c alled: the first one for finding t he c ompiler
and the others for finding c ompiler flags. In some c ases, it is interesting to disable some tests
(unnecessary , time c onsuming, etc ), to insert new tests, or to modify the ex isting tests. The
check_tool met hod has a parameter for passing the list o f c hec ks to perform:
3 4
If no chec k is p erformed, the c onfiguration helpers are still attac hed to the co nf object, and
may be used later:
10/10/2010 The Waf Book (v1.5.4)
To ease the c reation o f projects split into modules, conf.c hec k_tool will not load the tools
twice for the same env ironment and the same parameters.
An error handler attac hed to the con f objec t is used for c atc hing the C onfiguration
ex ceptions and proc essing the errors. Here is how to replac e the default configuration error
handler by a c ustom method which may mo dify the list of tests, stop th e ev aluation, or re-
raise the ex ception:
3 4
The following diagram illustrates the test ex ecution loop performed from conf.c hec k_tool
Adding new configuration helpers

The c onfiguration tests spec ific for t he different lan guages belong to the different W af tools
represen ting th e languages supported. For ex ample, The configuration tests for the C
language are loc ated in the to ol config_c.py . To enable t his, dec orators are used to attac h
new c onfiguration test s dy namic ally , for ex ample:
Then the new c onfiguration t est may be used easily in wsc ript:
Chapter 4. Packaging and redistributing the source

code
Table of C ontents
Quick tarball c reation

C ustom archiv es
C leaning the project tree
Quick tarball creation

C reating a tarball of the projec t is as simple as run ning
10/10/2010 The Waf Book (v1.5.4)
By default, the archiv e name uses the v ersion and the projec t n ame found in the top-lev el
wsc ript file
The default c ompression format is bzip2. It may be changed to g zip by u sing the following
c ode:
Custom archives
C ustom archiv es c an b e c reated by prov iding a func tion named " dist" in the t op-lev el sc ript
file. The following ex ample prints a c hec ksum immediately after the arc hiv e is c reated:
A func tion may be defined for ex clud ing or adding more files to the arc hiv e. The hook is
ex ec uted from the temporary direct ory after the dist func tion h as finish ed to run:
W hen c opy ing the files, the d ist func tion automatic ally ex c ludes temporary files. The
fu nction dont_d ist in t he mod ule Sc ripting does the filtering, the c ode is reproduc ed below:
3 4
To ex c lude new files, it is possible to modify the ex tensions, or t o prov ide a new method
do nt_dist , for ex ample, to disable most of the filtering:
3 4
10/10/2010 The Waf Book (v1.5.4)
Cleaning the project tree
By default, all the files produc ed by W af are produc ed into the b uild direc tory . An additional
file named .lock-wscript is c reated to store information on the build direc tory :
The c ommand waf distclean remov es these two elements from the filesy stem. Afterwards, it
is necessary to rec ompile the projec t again.
To remov e the build files without remov ing the con figuration, use the c omman d waf clean
instead. The files c reat ed in the build direc tory may be remov ed manually , but the
modific ations will not be detec ted by W af.
Chapter 5. The Task system

Table of C ontents
Task c reation and ex ec ution

Task ex ec ution
Task ex ec ution in parallel
Task ex ec ution order
Ex ecutin g tasks only when something c han ges
Task creation and execution

W hen W af tasks are created, they are not ex ec uted immediately , for ex ample it is not
necessary to rebuild an applic ation if the sourc e files hav e not ch anged. Some tasks may
also dep end on other t asks whic h may not be c reated y et. For t his reasons W af delay s the
ex ec ution of tasks to the point when all build func tions hav e been ex ec uted.
C reating all tasks by hand is a tedious proc ess that the task gen erators (C hapt er 6, Task
encapsulation using task generators ) may automate. Before starting the build, W af asks eac h
task generator to prod uce the c orrespondin g tasks. If W af is launc hed from a sub folder
inside the sourc e direc tory , it will try to av oid the c reatio n of the tasks that are not relev ant
fo r that particu lar sub folder (optimization).
Once the tasks are c reated, W af will rev iew eac h o f them one by one to dec ide whether to
ex ec ute them o r not. A summary c an be found on the following diagram:
Task execution
Ex ecutin g a task c onsists in c alling the method run on that task, and setting the task
ex ec ution state. The fo llowing diagram is a summary of the proc ess:
10/10/2010 The Waf Book (v1.5.4)
The met hod post_run can be used to c hec k if the files hav e been produc ed, it must throw
an OSError if th e task has no t c omp leted properly .
Task execution in parallel

Tasks may be ex ecuted in parallel to take adv antage of the hardware (multi-co re) or the
env ironment (distribut ed builds). By default W af does not ex ec ute immediately the tasks
that are ready . Instead, tasks are ad ded to a queue whic h is c onsumed by threads. W af
detec ts the number of installed proc essors. For uni-proc essor only one task is ex ecut ed at a
time, for dual-proc essors two tasks are ex ecuted at a time, and so on. To disable task
parallelization, use the option -j1 . To enhanc e parallelizatio n, use the opt ion -j with the
amount of c onsumers:
By default, W af does n ot allow consumer threads t o ac cess the tasks direc tly :
There is little need for parallelizing the c omput ation of the nex t task to ex ecute,
c hoosing t he nex t task is fast enough
The thread issues are limited to a v ery small sec tion of the c ode
The produc er-con sumer sc heme prev ents busy waiting for the nex t task
A simple global error han dler c an be used for proc essing the errors and to dec ide to
stop the build
The following illustrates the relationship pro ducer-c onsumer performed for the builds:
10/10/2010 The Waf Book (v1.5.4)
Task execution order
Running tasks in parallel is a simple problem, but in prac tic e it is more c omplic ated:
D ependenc ies c an be disc ov ered durin g the build (dy namic task c reation)
New ordering con straints c an be disc ov ered after files are c ompiled
The amount of tasks and ordering con straints (graph size) c an be huge and
performanc e may be a problem
To make the problem more simple, it is div id ed by the different c onc ern s, and the ord ering
c onstraints c an be giv en on three different lev els:
1. gro ups of tasks may run only after ano ther group of tasks has finished to run, t his
represents a stric t sequential order bet ween g roups of tasks, for ex ample a c ompiler is
pro duced and used to c ompile the tasks in the nex t group
2. task ty pes to indic ate the instan ce will run aft er other task ty pe instanc es, for ex ample
linking object files may o nly oc c ur after compiling the sourc e files
3. specific c onstraint s for task instanc es t hat c an only run after a few other task
instances
Task groups
The following is used to dec lare grou ps of tasks to run one after the other:
The effec t of task groups when runn ing tasks in parallel is illustrated by the following
diagram. Three groups of tasks hav e been added, and th e ex ec ution of the nex t group only
starts when the ex ec ution of the tasks in the prev ious group is c omplet e.
Precedence constraints
The attributes before and after are used t o dec lare ordering co nstraints between tasks:
Another way to dec lare prec edenc e constraints is to dec lare a file ex ten sion produc tion, for
ex ample:
The ex tensions hav e to matc h to add a v alid prec edence c onstraint, bu t they are only
annotations, th ey do not mean the tasks ac tually hav e to produc e files of that ty pe.
Precedence constraints on task instances

The met hod set_run_after is used to declare ordering c onstrain ts between tasks:
unlike the prev ious con straints, it is used on the instanc es of c lass Task which is a subc lass
of c lass T askBase
Executing tasks only when something changes

The direc t instanc es of TaskBase are quite limited and do not trac k the c hanges to the
sourc e files. The c lass T ask prov ides the necessary features for the most c ommo n bu ilds in
which so urce files are u sed to produ ce targ et files. The idea is to c reate a unique signature
fo r tasks, and t o represent the dependencies on files or o ther tasks by inc luding them in the
signature. A hashing func tion is used for c omputin g the signature, by default it is md 5.
10/10/2010 The Waf Book (v1.5.4)
The following diagram illustrates the task proc essing inc luding the signature, it is only v alid
fo r Task instanc e (not TaskBase instanc es):
The signature c omputation uses the followin g data:
1. ex plic it dep endenc ies: input files and dependenc ies set ex plicitly using task.deps_man
or bld.depends_on
2. implic it dependenc ies: dependenc ies searc hed by the task it self (like sourc e files
inc luded from other sourc e files).
3. parameters: c ompilation flags and c ommand-line parameters.
Here is an ex ample illustrating the different kinds of dependencies:
3 4
Environment variable dependencies (compilation flags)

Implicit dependencies: a method returns a list containing the list of
additional nodes to take into account, and the list of the files that could not
be found (cache)
Explicit dependencies as input files (nodes)
Explicit dependencies as manual dependencies
Manual dependencies on source files, the second parameter can be a string, a
node object or a function returning a string
Manual dependencies with nodes, the first node represents a target (which
may or may not exist in the build), and the second parameter represents a
file in the source directory.
Chapter 6. Task encapsulation using task generators

Table of C ontents
Ex plic it rules for simple transformatio ns (Make-like)

Implic it p rocessing based on file name or ex tension
C ustom proc essing by task g enerator methods
Task generator method ex ec ution
Although task in stances may be created direc tly , t hey ac t like global v ariables in the sc ripts,
and they may lead to the c reation o f spaghetti cod e. The c ontainers named task generators
are used for c reating t he tasks and keeping trac k o f them.
10/10/2010 The Waf Book (v1.5.4)
Explicit rules for simple transformations (Make-like)
For simple transformations, a make-like sc heme en ables the creation of tasks by means of
task generators. The task is c reated lazily , for ex ample only if waf is c alled from a
subdirec tory of the projec t c ontainin g the declaration.
Simple rules
The following ex amples illustrate how to perform simple rule-based transformations, t he
attribute rule is mandatory , but the input or output files are optional:
A basic example with an input and an output. T he task is run whenever the
input or the rule change
T he task will run whenever the input or the rule change, but there is no
output
Creates a task with no input, the output is regenerated whenever the PREFIX
variable changes
Creates a task without inputs our outputs, it is run whenever the rule text
changes
Advanced features
The nex t ex amples illustrate v arious attribu tes whic h may be used to alter task ex ec ution.
3 4
10/10/2010 The Waf Book (v1.5.4)
3 4
Make a task which is always run (attribute always )

Demonstrates how to add an ordering constraint (attribute before ) to
execute the rule before the one that produces test.k1. It also demonstrates
how to set the directory from which the command is run (attribute cwd )
Rules may be given as functions. In this case the task has no input or output
files
the dependant task copy_svnversion is executed when the actual output of
svnversion changes (attribute on_results )
By binding the rule to the act ual dec laration, this system limits code reuse . For this
reason, it is rec ommended fo r solv ing v ery spec ific problems in whic h a library would not
make sense. The c reat ion of lots of small rules may also impact performanc e negativ ely .
Implicit processing based on file name or extension

The ex c lic it rules desc ribed in the prev ious sec tion becomes limited for proc essing sev eral
files of the same kind. Instead of adding loops, we will now desc ribe how to proc ess files
using the name or the ex tension, for files dec lared in the source attribu te. This sc heme is
the most c ommon tec hnique used in the W af library .
Task chain declaration

The following ex ample illustrates how to add a rule for transforming .lua files into .luac files:
In prac tic e, the c haining dec laration is located in a separate py thon file whic h may be re-
used for other projec ts.
Using the extension decorator

A special method named apply_core prov ides file ex tension proc essing using the attribute
source . On the user side, this takes the following form:
The met hod to proc ess .c files (main.c ) is c alled automatic ally by apply_core , and is
declared in the followin g man ner:
W hen th e method apply_co re is c alled, the following operations are performed :
1. If a method is mapped ( @extension ) to a name list ed in sourc es, it is c alled with that
name as a parameter
2. The names that d o not hav e a mapping are su pposed to be files and are transformed
into Node instanc es
3. For eac h node (temporary v ariable self.allno des ), a method c orresponding to the file
ex tension is applied
In this sy stem, the list of sou rces for an applic ation does not hav e to ex actly represent files.
Also, methods may re-injec t new nodes to proc ess by ex tension (in the temporary v ariable
self.allnodes ), without keeping an ex act referenc e on the method that will be c alled.
10/10/2010 The Waf Book (v1.5.4)
Custom processing by task generator methods
In some cases the info rmation to pro cess is v ery spec ific and does not really fit in the list of
sourc es. The data may be att ac hed to v arious attribute of the task generator, and a distinct
task generator method may p rocess the attributes dy namic ally .
In the fo llowing ex ample, a new attribute 'tst' is added to task generato rs instanc es, and is
used to call a method to print the c ontents of that attribute at the start of the build:
To attac h a new metho d to the task_gen c lass (the ty pe of foo ), the dec orator @taskgen
is used in the following manner:
At this point, th e funct ion print_tst is attac hed to the c lass as a new method, but to proc ess
the attribute foo .tst , the method has to be c alled ex plicitly :
To sc hed ule that method for automatic ex ec ution, it is nec essary to bind it somehow. The
aliases are giv en in the task generato r attribute features :
For map ping th e method to all featu res, the wildc ard * may be used, a c omplete ex ample is
reproduc ed below:
Upon ex ecution , this produc es the following output:
Task generator method execution

Methods may b e added to task generators, and th ose methods are bound to features. At th e
beginning of the build, the task generator in stances are ex ec uted one by one, and in turn
10/10/2010
beginning of the The
build, the task generator in stances are ex ec Waf
uted oneBook (v1.5.4)
by one, and in turn
their methods are ex ec uted to generate the tasks or to modify addition al data. W e will now
desc ribe what make task generator methods with an ex ample:
Task generator methods take a single parameter self

They are bound to partic ular names representing features (or to '*')
They hav e order restric tions before and after
Here is an ex ample of a task generator method from the W af sourc e c ode:
3 4
In general, the methods c ann ot be c alled sev eral times or in random order. For ex ample, in
c ++ programming c ontex ts, the c ompilatio n tasks must be c reated before link tasks bec ause
link task will referenc e the c ompilation tasks. This leads to a part icular workflow, reproduc ed
in the following illustration:
The list o f methods to ex ecut e is kep t in the attrib ute meths of the task generator instanc e.
The list o f methods to ex ecut e may be giv en direc tly .
It is howev er more c onv enient to use the attribute feature to add the method s
automatic ally . For ex ample, for c reating a c application, the met hods to ex ec ute depend on
the operating sy stem, so the feature sy stem ac tually simplifies the c ode reuse.
The order c onstraints on the methods (after/before), are used to sort the list of meth ods in
the attribute meths . Th e sorting is p erformed onc e, and t he list is c onsu med as meth ods are
ex ec uted. Thou gh no new feature may be added onc e the first method is ex ec uted, new
methods may be added dy namic ally in self.meths. Here is how to make an infin ite loop:
Chapter 7. The scheduler for executing the tasks

Table of C ontents
The task ex ec ution mo del

Job c ontrol
W eak task order c onstraints
The task execution model

Task dependenc ies and task o rdering spec ify the ex act order in whic h tasks must be
ex ec uted. W hen tasks are ex ecuted in parallel, different algorithms may be used to improv e
the c ompilation times. For ex ample, tasks that are known to last longer may be launc hed
first. Linking tasks that use a lot of ram (in the c ontex t o f c ++ applications) may be
launched alone to av oid disk thrashing by sav ing R AM.
To make this possible, the task ex ec ution is organized in the follo wing manner:
10/10/2010 The Waf Book (v1.5.4)
Job control
Job c ontrol is related to the parallelization algorithms used for launc hing the tasks. W hile the
aim of parallelization is to max imize the amo unt of tasks in ex ec ution, d ifferent algorithms
may be used in practic e
In the NORMAL orderin g, task groups are c reated, and a topologic al sort is performed in
adv anc e. The ov erall performanc e penalty for c omplete b uilds is usually small, like a few
sec onds on builds during min utes.
In the JOBC ONTROL ordering, groups are c reated in adv anc e, and a flag indic ates the
max imum amount of jobs to be used when the c onsumer threads ex ec ute the tasks. This
prev ents parallelization of tasks whic h use a lot of resourc es. For ex ample, linking c ++ objec t
files uses a lot of R AM.
In the MAXPARALLEL ordering, Eac h task ho lds a list of tasks it must run after (there is only
one list o f tasks waiting to be ex ec uted). Though this v ersion parallelizes tasks v ery well, it
c onsumes more memory and proc essing. In prac tic e, W af may last 20% more on builds
when all tasks are up-to-date.
Warning
Because most task classes use ordering constraints, the maximum
parallelization can only be achieved if the constraints between
task classes are relaxed, and if all task instances know their
predecessors. In the example graph, this was achieved by
removing the ordering constraints between the compilation tasks
classes and the link tasks classes
10/10/2010 The Waf Book (v1.5.4)
From this, we c an immediately notic e the fo llowing :
An assumption is made that all tasks hav e input and output nodes, and that ord ering
c onstraints can be deduc ed from them
D educ ing t he c onstraints from the input and output nodes ex hibits a n^2 behav iour
Note
In practice, the NO RMA L algorithm should be used whenever
possible, and the MA XPA RA LLEL should be used if substantial
gains are expected and if the ordering is specified between all
tasks. T he JO BCONT ROL system may be useful for tasks that
consume a vast amount of resources.
Weak task order constraints

Tasks that are known to take a lot of time may be launc hed first to improv e th e build times.
The general pro blem o f finding an optimal order fo r launc hing tasks in p arallel and with
c onstraints is c alled Job Shop . In prac tic e this prob lem c an often be reduc ed to a critic al
path pro blem (approx imation).
The following pic tures illustrate the differenc e in sc heduling a build with different independent
tasks, in whic h a slow task is clearly identified, and launch ed first:
W af prov ides a func tion for reordering the tasks b efore t hey are launc hed in the module
Runner, the default reordering may be c han ged by dy namic method replac ement in P y thon:
If the reordering is not to be performed eac h time a task is retriev ed, th e list of task may be
reordered when the nex t gro up is retriev ed:
It is possible to measure the task ex ec ution times by intercepting the func tion c alls. The task
ex ec ution times may be re-used for optimizing the schedule for subsequent bu ilds:
10/10/2010 The Waf Book (v1.5.4)
Chapter 8. C and C++ projects

Table of C ontents
C ommon sc ript for C /C ++ applic ations

Library interact ion (uselib)
C ustomizing ob jec t files
Using c onfiguration headers
The inc lude sy stem
C onfiguration helpers
Pkg-con fig
Common script for C/C++ applications

The c /c ++ builds c onsist in transforming sourc e files into objec t files, and to assemble the
objec t files at the end. In theory a single programming language should be sufficient for
writing any applic ation , but in prac tic e slight v ariations ex ist:
App lications may be div ided in dy namic or static libraries

Add itional files may enter in the link step (libraries, object files)
Sou rce files may be generated b y other c ompilers
The c onstructio n of c/c++ applic ations c an be quite c omplicated, and sev eral measures must
be taken to ensure coh erent interac tion with new c ompilation rules. The c anon ical co de for a
task generator building a c /c++ app lication is the following:
T ask generator declaration; each element in the list represent a feature; it is

possible to add several languages at once ( ocaml and c++ for example),
but the one of objects, staticlib, shlib or program must be chosen.
List of source, it may be either a python list, or a string containing the file
nam es separated with spaces. T his list may contain file names of different
extensions to make hybrid applications.
T arget name, it is concerted to the name of the binary name.so or
name.exe depending on the platform and the features.
List of include paths, it may be either a python list, or a string containing the
paths separated by spaces. T he paths are used for both the command-line
and for finding the implicit dependencies (headers). In general, include paths
must be relative to the wscript file and given explicitly. See the section called
“T he include system”.
Installation directory, this is where to install the library or program
produced. T he ${} expression is a reference to a variable to be extracted
from tgen.env . By default it is set to ${PREFIX}/bin for program s and
${PREFIX}/lib for libraries. T o disable the installation, set it to None .
Command-line defines: list of defines to add to the command-line with the -
D prefix. T o reduce the size of the comm and-line, it is possible to use a
configuration header, see the following section for more details.
Command-line compilation flags, for the c++ language the attribute is called
cxxflags
Shared libraries may be given directly (use staticlib and staticlibpath for
static libraries)
A dditional pararameters may be added from a task generator reference. T he
next section describes a technique to gather the conditions into the
configuration section.
Library interaction (uselib)

To link a library against another one c reated in the same W af project, the attribute
uselib_local may be used. The includ e paths, the link path and the library name are
10/10/2010 The Waf Book (v1.5.4)
automatic ally ex ported, and the dependent binary is rec ompiled when the library c hanges:
A static library
Include paths to export for use with uselib_local (include paths are not
added automatically). T hese folders are taken relatively to the current target.
A program using the static library declared previously
A list of references to existing libraries declared in the project (either a
python list or a string containing the names space-separated)
To link an applic ation against v arious system libraries , sev eral c ompilation flags and link
flags must be g iv en at once. To reduc e the maintenance, a sy stem called uselib can be used
to giv e all the flags at the same time:
Declare a few variables during the configuration, the variables follow the
convention VA R_NA ME
A dd all the VA R_NA ME corresponding to the uselib NA ME, which is 'T EST ' in
this example
The v ariables used for c /c ++ are the following: STATIC LIB, LIB, LIBPATH, LINKFLAGS,
RPATH, C XXFLAGS, C C FLAGS, C PPPATH, C PPFLAGS, C XXD EFINES, FRAMEW OR K,
FRAMEW ORKPATH, C XXDEPS . The u selib is similar to the c asc ading sty le sheet (C SS)
princ iple.
Customizing object files

In some cases, it is nec essary to re-use objec t files generated by another task generator to
av oid rec ompilations. This is similar to copy -pasting code, so it is disc ouraged in general.
Another use for this is to enable some c omp ilation flags fo r spec ific files. The attribute
"add_objects" c an be used, like in the following ex ample:
Files will be com piled in c mode, but no program or library will be produced
Different compilation flags m ay be used
T he objects will be added automatically in the link stage
Using configuration headers

Adding lots of c ommand-line define v alues inc reases the size of the c ommand-line and
c onceals the useful information (differenc es). Some projec ts use headers whic h are
generated during the c onfiguration, they are not modified during the build and they are not
installed or redistributed. This sy stem is useful for huge projec ts, and h as been made popular
10/10/2010
installed or redistributed. Thets,Waf
This sy stem is useful for huge projec and Book (v1.5.4)
h as been made popular
by autoc onf-based projec ts.
W riting c onfigu ration headers can be performed using the following methods:
The c ode snipped will produc e the following co nfig.h in th e build direct ory :
The c ontents of the c onfig.h for this ex ample are
The include system

The Waf preprocessor
In clude paths are used by the c /c ++ c ompilers for finding the headers. W hen one header
c hanges, the files are rec ompiled aut omatic ally . Inc ludes must be giv en in the following form:
Local folders are giv en relativ e to the folder of the c urren t script . The equiv alent build
directory folder is then added automatically for headers or sourc e files produc ed in the build
directory .
W af uses a prep rocessor writ ten in Py thon for adding the dependenc ies on th e headers. A
simple parser looking at #include statements would miss c onstruc ts suc h as:
Using th e c ompiler for finding the dependenc ies would not work for app lications requiring file
preprocessing suc h as Qt. Fo r Qt, special inc lude files hav ing the .mo c ex tension must be
detec ted by the build sy stem and produc ed ahead of time. The c compiler c ould not parse
suc h files.
Using th e W af preproc essor also makes the scripts strongly c ompiler-independent.
Portability recommandations
By default, the preproc essing does not climb to sy stem headers. This may lead to missed
depdend enc ies in the c ases similar to the following:
10/10/2010 The Waf Book (v1.5.4)
To write portable c ode and to ease debugging, it is strongly reco mmended to put all the
c onditions used in the projec t into a co nfig.h file.
Although it is disc ouraged, it is possible to enable the preprocessor to lo ok into sy stem

headers by usin g the followin g c ode:
Debugging dependencies
The W af prepro cessor contains a spec ific debugging zone:
To display the dependencies obtained or missed, use the following:
The dependenc y c omp utation is performed only when the files are not up-to-d ate, so these
c ommands will display someth ing on ly when there is a file to c ompile.
Configuration helpers
The met hods check_cc and check_cxx are used to detec t parameters using a small build
projec t. The main parameters are the following
msg: title o f the t est to ex ecute

okmsg: message to display when the t est suc ceeds
errmsg: message to display when the t est fails
man datory : when true, raise a c onfigu ration ex c eption if the test fails
env : env ironment to use for the build (conf.env is u sed by default )
define_name: add a define for the c onfiguration header when the t est suc ceeds (in
most c ases it is c alc ulated automatic ally )
Here is a c oncrete ex ample:
3 4
T ry to compile a program using the configuration header time.h, if present

on the system, if the test is successful, the define HA VE_T IME_H will be
added
T ry to compile a program with the function printf, adding the header stdio.h
(the header_name may be a list of additional headers). T he param eter
mandatory will m ake the test raise an exception if it fails.
T ry to compile a piece of code, and if the test is successful, define the name
boobah
Modifications made to the task generator environment are not stored. When
the test is successful and when the attribute uselib_store is provided, the
nam es lib, cflags and defines will be converted into uselib variables LIB_M,
10/10/2010 The Waf Book (v1.5.4)
CCFLA GS_M and DEFINE_M and the flag values are added to the
configuration environm ent.
T ry to compile a simple c program against a library called 'linux', and reuse
the previous parameters for libm (uselib)
Execute a simple program, collect the output, and put it in a define when
successful
A fter all the tests are executed, write a configuration header in the build
directory (optional). T he configuration header is used to limit the size of the
command-line.
Here is an ex ample of a config.h pro duced with th e prev ious test c ode:
The file default.cache.py will c ontain the following v ariables:
3 4
Pkg-config
In stead of duplic ating the con figuration detection in all dependent projec ts, co nfiguration
files may be written when libraries are installed. To ease the interac tion with build sy stems
based on Make (c anno t query datab ases or apis), small applic atio ns hav e been c reated for
reading the cac he files and to interpret the parameters (with names traditionally ending in -
config ): pkg-co nfig, wx -config, sdl-c onfig, etc .
W af prov ides the method check_cfg for qu ery ing c onfig parameters:
3 4
Check for the pkg-config version

Check for a module version
Obtain the flags for a package and assign them to the uselib PA NGO
(calculated automatically from the package name, can be overridden with the
attribute "uselib_store='MY PA NGO '")
Retrieve the module version for a package. T he returned object is a string
containing the version number or an empty string in case of any errors. If
there were no errors, 'PA NGO _VERSION' is defined, can be overridden with
the attribute "uselib_store='MY PA NGO'".
Obtain the flags for a different configuration program (sdl-config). The
example is applicable for other configuration program s such as wx-config,
pcre-config, etc
D ue to the amo unt of flags, the lac k of standards between c onfig applic ations, and to the
output c hangin g for different operating sy stems (-I for gc c , /I for msv c ), the output of pkg-
c onfig is parsed, and the v ariables for the c orresponding uselib are set in a go . The func tion
parse_flags(line, uselib, env) in the W af module c onfig_c .py performs the output
analy sis.
Chapter 9. Advanced scenarios
10/10/2010 The Waf Book (v1.5.4)
Table of C ontents
Simple file transformations

Same targets, different configurations (v ariants)
Building the c ompiler first
W riting the out put of a program into a header
A compiler prod ucing source files with names unknown in adv anc e
A task without any file dependenc y
This c hapter prov ides ex amples for adv anc ed scenarios to demonstrate the prac tic al use of
the W af library .
Simple file transformations

The W af tool misc con tains v arious routines to ease file manipulations such as substituting
paramet ers or ex ecutin g c ust om compilers. The objectiv e of this sectio n is to illustrat e the
princ iples of the current apis.
File substitutions
The following ex ample illustrates how to produc e a pkg-c onfig file from a temp late:
The v ariables must be declared in th e template file by enc losing the names bet ween @
characters (m4 sy ntax ).
The default substitutio n func tion may be replaced by changing the attribute func of the task
generator.
Compiler output
The command-o utput sy stem enab les ex ec ution o f sy stem commands without requiring a
shell. Th e following ex ample illustrates how to ex ec ute a sc ript p resent in the sourc e
directory each time it is ex ec uted.
misc is the Waf module providing the command-output features

T he standard output will be redirected to the file test1
T he file 'wscript_build' will be passed to the standard input
T he program to execute is assumed to be present in the directory of the
wscript file, unless command_is_external is set to True
A dditional parameters are provided in the argv attribute, as a list.
Parameters representing files must be added through the special classes
input_file, output_file, input_dir and output_dir to resolve the paths properly
A n additional dependency is added on the time, the task will be executed
whenever the time changes
Same targets, different configurations (variants)

In many builds, it is nec essary to bu ild the same targets for different purposes: debugging,
profiling, optimizations. A sy stem is p rov ided for duplic ating the targets easily .
10/10/2010 The Waf Book (v1.5.4)
Variants and environments
The following co de demonstrates ho w to create a c onfigu ration for a new v ariant named
'debug'
After running waf configure , the bu ild folders c orresponding to the default and to the
debug v ariants are produc ed:
To use t he debug v ariant, th e env ironment must be giv en direc tly , by default, a cop y of
the env ironmen t named default is set:
Cloning targets
Task generators may be c opied for different v arian ts easily using the clo ne method. The
paramet er is the name of the env ironment to use (not the v ariant). The named
env ironments are c reated during the configuration, as shown in the prev ious subsec tion.
It is also possible to c lone all task generators from a projec t using a c ode similar to the
fo llowing :
Cross-variant dependencies are disco uraged

The v ariant sy stem is meant for auto mating the duplic ation of targets, whic h are sup posed
to run in in isolation. As a c onsequenc e, it is not spec ified how lin king a program against a
library c reated in a different v ariant should be proc essed. The general rec ommendatio n is to
av oid op timizations and to duplic ate the targets as necessary . The follo wing bug trac ker
entry prov ides more d etails http://c ode.google.c om/p/waf/issues/detail? id=171
Building the compiler first

The ex ample below demonstrates ho w to build a c ompiler which is used for building the
remaining targets. The requirements are the following:
C ompile the c omp iler with all its intermediate t asks

R e-use the c ompiler in a sec ond build step
This sc enario demonstrates the bld.add_gro up to segment the build in to stric tly seq uential
parts:
10/10/2010 The Waf Book (v1.5.4)
3 4
Create the compiler.

Create a new group
T he rest of the build follows
Writing the output of a program into a header

The ex ample below demonstrates ho w to write the output of sv n v ersion into a header. The
c ommand is alway s run , and the c /c ++ compilation will be ex ec uted if the header c hanges
(implic it depend enc ies obtained by the W af preproc essor):
10/10/2010 The Waf Book (v1.5.4)
3 4
Exam ple in a user script, the attribute svn_header is used to indicate a

header to create as a build file
Process the attribute svn_header when present, create a task instance
A function for obtaining the svn version of the project
Obtain the svn version and write it to the header
Declaration of the task which will execute the svn processing
T he task must be run each time - there is no signature involved
Mark the outputs as changed if necessary by changing the signature
Task c reation is performed through a new task generator method (proc ess_sv n) whic h is
ex ec uted only when the task generator c reates all its tasks. This organization makes it
possible to c ompile only the c ode nec essary for a spec ific target (tasks are c reated lazily ,
and are grouped by task gen erators). To ad dress the scenario desc ribed, a new task ty pe is
c reated, and methods are replac ed dy namic ally : runnable_status (ex ec ute eac h time), and
post_run (c hange the signature of the outputs acc ording to the contents of the file
c reated).
A compiler producing source files with names

unknown in advance
The ex ample below demonstrates ho w to tackle th e following requirements:
A c ompiler produces source files (.c files) for whic h tasks must be c reated
The sourc e file names are no t kno wn in advance
The task must be run o nly if necessary
Other tasks may depend on the tasks proc essing the sou rce produc ed (c omp ile and
link the .c files)
The main difficu lty in this scenario is to store the information on the sourc e file produ ced
and to c reate the c orrespond ing tasks each time.
10/10/2010 The Waf Book (v1.5.4)
10/10/2010 The Waf Book (v1.5.4)
3 4
A n example. T he source line contains a directive foo.shpip which triggers

the creation of a shpip task (it does not represent a real file)
T his method is used to create the shpip task when a file ending in .shpip is
found
Create the new task type
Disable the warnings raised because the task has no input and outputs
Execute the task
Retrieve the information on the source files created
Create the c++ tasks used for processing the source files found
If the tasks are created during a task execution (in an execution thread), the
tasks must be re-injected by adding them to the attribute more_tasks
If the tasks are created during the task examination (runnable_status), the
tasks can be injected directly in the build by using the attribute outstanding
of the scheduler
A task without any file dependency

Giv en the following req uirements
A task x produces a header used in c/c ++ c ompilations

The c/c++ task depends o n the pro duction o f that header
That c task must be ex ecuted whenever the header actually changes
The header may or may not be updated, but the header pro duction task must run
each time
The following ex ample demon strates how to c reate a task ty pe fulfilling the requirements,
and how to use it.
10/10/2010 The Waf Book (v1.5.4)
3 4
Create a new task type, in this example, we pretend it runs a command to

retrieve the svn version and stores it in a header.
Replace runnable_status by a new method which indicates the task must
be run each time
Indicate that the task must always be run before c++ ones
By default, the task signature is assigned to the node information. In our
case the task signature is always the same, and we need a way to indicate to
dependent tasks that something has changed. T he solution is to compute a
hash of the file produced, and to assign it to the node information.
Demonstrate the manual creation of the task
T he task outputs are Node instances, but there are no inputs.
Chapter 10. Customizing the scripting system

Table of C ontents
Prov iding c ustom c ommands

C haining c ommands
Prov iding a c ustom co mmand c ontex t
C reating aliases / Injec ting new c ommands
Providing custom commands

W af c ommands are fun ctions defined on th e top-lev el wscript file, for ex ample, with the
fo llowing wsc ript file:
ex ec uting the sc ript with "foo" as argument will produce the following output
The parameter ctx in the func tion foo abov e is prov ided to enable rec ursion and sharing
data with the sc ripts. In the following ex ample, the script subdir/wscript_foo will be looked
up, and ex ec uted if found. Else, the sc ript subdir/wscript will be looked up and loaded as a
module. If suc c essfully loaded, the func tion foo will be ex ecuted, else an ex c eption is raised.
For bac kward c ompatibility reasons, the c ommand s init, shutdown, dist and distcheck do
not ac c ept a co ntex t as parameter y et.
10/10/2010 The Waf Book (v1.5.4)
Chaining commands
Sinc e W af 1.5.4, it is possible to c hain sev eral c ommands, for ex ample:
The ex ec ution will stop as soo n as an error is enc ountered, for ex ample if distc lean fails, the
c onfiguration and the build will not b e ex ec uted, so the c omman d is equiv alent to:
A frequent usec ase is to c hain the ex ec ution of bu ild and c leaning steps:
As a special ex c eption, the func tion init is alway s prepended to the commands to ex ecute,
and the functio n shutdown is alway s appen ded.
Providing a custom command context

The c ontex t for a command is c reated auto matically , and is deriv ed fro m the c lass
Utils.C ontext . C ustom c ontex t instance may be prov ided in the following manner:
C ustom c ontex ts may be prov ided for the function s co nfigure and build .
Creating aliases / Injecting new commands

New c ommands may be injec ted in the following manner:
In jec ting new c ommands is u seful for writin g testc ases. By ex ec uting waf test , the following
sc ript will c onfigure a projec t, create source files in the sourc e d irec tory , build a program,
modify the sourc es, and rebu ild the program. In this c ase, the program must be rebuilt
because a header (implic it dependenc y ) has c hanged.
10/10/2010 The Waf Book (v1.5.4)
3 4
Chapter 11. Using the development version

Table of C ontents
Tracing the ex ec ution

Benchmarking
Profiling
Tracing parallel task ex ec ution
Obtainin g the latest sourc e c ode
Programming c onstrain ts
Tracing the execution

The generic flags to ad d more information to the stack traces or to the messages is -v
(v erbosity ), it is used t o display the c ommand-lines ex ecu ted during a b uild:
To display all the trac es (useful for bug reports), use the following flag:
D ebugging information c an b e filtered easily with the flag zones :
Tracing zones must be comma-separated, for ex ample:
The W af module Logs replac es the P y thon module logging. In the sourc e c ode, traces are
prov ided by using the debug func tion, they must obey t he format "zon e: message" like in
the following:
3 4
The following zones are used in W af:
Table 11.1. Debugging zones
Zone Meaning
command-lines executed (enabled when -v is provided without
runner
debugging zones)
task_gen task creation (from task generators)
action functions to execute for building the targets
env environment contents
envhash hashes of the environm ent objects - helps seeing what changes
build build context operations such as filesystem access
deps implicit dependencies found (task scanners)
Benchmarking
The scrip t utils/genbench.py generates a simple benc hmark for W af. The habitual use is the
fo llowing :
The projec t c reated c ontains 50 libraries with 100 c lasses for eac h, eac h sourc e file hav ing
15 includ e head ers pointing t o the same library and 5 headers pointing to the headers of
other lib raries in the projec t.
The time taken to c reate the tasks and to resolv e the dep endenc ies can be ob tained by
10/10/2010 The Waf Book (v1.5.4)
injec ting c ode to disable the ac tual c ompilation, fo r ex ample:
Profiling
Profiling information is obtain ed by c alling the module cProfile an d by injecting spec ific c ode.
The two most in teresting met hods to profile are flush and compile . The most important
number from the profiling information is the amount of func tion calls, and reduc ing it results
in noticeable speedups. Here is an ex ample on the metho d c omp ile:
3 4
Here the output obtained on a benc hmark build created as ex plained in the prev ious sec tion:
3 4
From the profile information, it appears that the ho t spots are, in order:
The persist enc e implemented by the cP ickle module (the c ache file to serialize takes
about 3Mb)
Ac c essing c onfiguration data fro m the Env iron ment instanc es
C omputing implicit dependencies (the test projec t co ntains lots of interleav ed
dependenc ies)
In prac tic e, the time taken by these methods is not signific ant enough to justify code
c hanges. Also, profiling must be carried out on builds which last at least sev eral sec onds.
Tracing parallel task execution

A special W af tool named ParallelDebug is used to inject c ode in W af modules and o btain
information on the ex ecution . This module is prov ided in the folder playground and must be
imported in one's project before use:
After ex ecuting a full build, a trac e of the ex ecutio n is ou tput in /tmp/test.dat ; it may be
proc essed by o ther applic atio ns suc h as Gnuplot:
10/10/2010 The Waf Book (v1.5.4)
Obtaining the latest source code

W af is hosted on Google c ode, and u ses Subv ersion for sourc e c ontrol. To obt ain the
dev elopment c opy , use:
To av oid regenerating W af eac h time, the env ironment v ariable W AFD IR c an be used to
point to the direc tory containing wafadmin:
Although the W af binary depends on Py thon 2.3, the W af sourc e code depen ds on P y thon
2.4. W hen generating W af, a parser modifies the sourc e c ode and perfo rms the following
operatio ns:
Mov e the dec orat ors into simple function c alls

Add the imports for the module sets
Eliminate redundant spac es
Strip c omments to reduc e the size
Programming constraints
Though W af is written in Py th on, additional restric tions apply to the sourc e c ode:
Identation is tab-o nly , and the max imum line length should be about 200 charac ters
The dev elo pment code is kept c ompatible with Py tho n 2.3, to the ex ception of
dec orators in the Tools d irec tory . In partic ular, the W af binary c an be generated using
Py t hon 2.3
The wafadmin modules must be insulated from the Tools modules to keep the W af c ore
small and language independent
Api c ompatibility is maint ained in the c y c le of a mino r v ersio n (from 1.5.0 to 1.5.9)
More c ode alway s means more bugs. W hen ev er possible, unnecessary c ode must be
remov ed, and the ex isting c ode base should be simplified .
Chapter 12. Overview of the Waf architecture

Table of C ontents
The c ore library

Build con tex t instanc es
Ov erv iew of the W af ex ec ution
Performance an d build ac curac y
The core library

W af is based on 12 modules which c onstitu te the c ore library . They are loc ated in the
directory wafadmin/ . The modules loc ated under wafadmin/Tools add su pport for
programming languages and more to ols, bu t are n ot essential for the W af c ore.
Table 12.1. The co re library
Module Role
Defines the build context class, which holds the data for one build
Build
(paths, configuration data)
Contains the configuration context class, which is used for
Configure
launching configuration tests, and the extension system
Constants Provides the constants used in the project
Contains a dictionary class which supports a lightweight copy
Environment
scheme and provides persistence services
Logs Provide a logging system
Node Contains the file system representation class
Provides a custom command-line option processing system based
O ptions
on optparse
Runner Contains the task execution system (threaded producer-consumer)
Constitutes the entry point of the Waf application, use the
Scripting
command-line for launching the configuration, the build, etc
Provide the task generator system, and its extension system based
T askGen
on method addition
T ask Contains the task classes, and the task containers.
10/10/2010 The Waf Book (v1.5.4)
Contains the support functions and classes re-used in other Waf
Utils
modules
The essential c lasses and methods from the core library are represented on the following
diagram:
Build context instances

Ex ecutin g tasks, acc essing th e file sy stem and c onsulting the results of a prev ious bu ild are
v ery different c onc ern s whic h hav e to be enc apsulated p roperly . The c ore class representing
a build is a build contex t.
Build context and persistence

The build c ontex t holds all th e information necessary for a build. To ac c elerate the st art-up,
a part of the information is stored an d loaded between th e runs. The persistent attrib utes
are the following:
Table 12.2. B uild context persistence
A ttribute Information
root Node representing the root of the file system
srcnode Node representing the source directory
bldnode Node representing the build directory
node_sigs File hashes (dict mapping Node ids to hashes)
node_deps Implicit dependencies (dict mapping Node ids)
Implicit file dependencies which could not be resolved (dict mapping
raw_deps
Node ids to lists of strings)
Signature of the tasks previously run (dict mapping a T ask id to a
task_sigs
hash)
Sequence for generating unique node instance ids (id of the last Node
id_nodes
created)
10/10/2010 The Waf Book (v1.5.4)
Build context access
In prev ious W af releases, the build c ontex t was su pposed to be a uniqu e object (one build
activ e at a time). To enable the use of W af as a lib rary , the dependency on the singleton
Build.bld was remov ed. This implies that eac h objec t should be able to obtain its build
c ontex t from its attributes. Here are a few ex amples:
Table 12.3. B uild context access
Object type Build context access

Node self.__class__.bld
task_gen self.bld
T ask self.generator.bld
Parallelization co ncerns
Build con tex ts p erform an os.chdir c all before starting to ex ec ute the tasks. W hen running
build con tex ts within build c ontex ts (tasks), the c urrent working directo ry may c ause
v arious problems. To work around them, it may be nec essary to c hang e the c ompilat ion
rules (co mpile from the file sy stem root) an d to inject c ode (replac e bld.c ompile).
D irec t Node instances are not used any where in the W af c ode. Instead, eac h b uild c ontex t
c reates a new Node subc lass (bld.node_class), on whic h t he build c ontex t instance is
attac hed as a c lass att ribute.
Threading concerns
Nearly all the c ode is ex ec uted in the main thread. The ot her threads are merely waiting for
new tasks, and ex ecut ing the methods run from the task instanc es. Suc h meth ods should
c ontain as little code as possible, and ac cess the BuildC ontex t in a read-only manner.
If the ru n meth ods hav e to modify the build c ontex t, it is rec ommended to ov erload the
method get_out of the sc heduler and to ex ec ute methods in an ev ent-like manner (data is
attac hed to the task, and the metho d get_out ex ecutes the cod e). Adding more tasks from
a runnin g task is demonstrated the section c alled “A c omp iler produc ing sourc e files with
names unknown in adv ance”.
Overview of the Waf execution

File system access
File sy stem ac c ess is performed through an abstrac tion lay er formed by the build c ontex t
and No de instanc es. Th e data structu re was carefu lly designed to max imize performanc e, so
it is unlikely that it will c hange again in the future. The idea is to represent one file or one
directory by a single Node instance. Dependent data suc h as file hashes are stored on the
build con tex t object and allowed to be persisted. Three kinds of nodes are dec lared: files,
build files and fo lders. The nodes in a partic ular direc tory are unique, but build files used in
sev eral v ariant add duplic ate entry on the build c ontex t c ac he.
To ac c ess a file, the methods No de::find_resource , No de::find_build (find an ex isting

resourc e or dec lare a build node) an d No de::find_dir must be used. W hile searc hing for a
particular node, the fo lders are auto matically searc hed onc e for the files. Old nodes (whic h
do not h av e a c orresponding file) are automatically remov ed, ex c ept for the b uild no des. In
some c ases (lots of files added and remov ed), it may be n ec essary to perform a Waf clean
to eliminate the information o n build files wh ich do not ex ist any more.
Task classes
The whole proc ess of generat ing tasks thro ugh W af is performed by methods added on the
c lass task_gen by c ode inject ion. This often puzzles the programmers u sed to static
languages where new func tions or c lasses c annot b e defin ed at runtime.
The task generators automat ically inherit th e build c ontex t attribute bld when c reated from
bld.new_task_gen. Likewise, tasks created from a task generator (create_task) automatic ally
inherit th eir generator, and th eir build c ontex t. D irect instantiation may result in problems
when running W af as a library .
The tasks c reat ed by task generator methods are automatic ally stored on the build c ontex t
task man ager, whic h stores the task into a task group. The task groups are later used by th e
sc heduler to obtain the task which may run (state machin e). Target (un)installation is
performed right after a task has run, using the method install.
Performance and build accuracy

From the ex perience with too ls suc h as SC ons, users may be c onc erned about performance
and thin k that all build sy stems based on in terpreted lang uages such as Py thon would not
sc ale. W e will now desc ribe why this is not t he c ase for W af and why W af should be c hosen
fo r build ing v ery large projec ts.
Comparing Waf against other build systems

Sinc e W af c onsiders th e file c ontents in the build proc ess, it is often thought that W af would
be much slower than make. For a test projec t hav ing 5000 files (generated from the sc ript
loc ated in tools/genbench.py ), on a 1.5Ghz c omputer, th e W af runtime is ac tually slightly
faster than the Gnu/Make one (less than one sec ond). The reason is the time t o launc h a
new proc ess - make is usually c alled rec ursiv ely , on ce by direc tory .
10/10/2010 The Waf Book (v1.5.4)
For huge projec ts, c alling make rec ursiv ely is nec essary for flex ibility , b ut it hurts
performance (launch many proc esses), and C PU utilizatio n (runn ing tasks in parallel). Make-
based bu ild sy stems suc h as C Make or Autotools in herit the limitations of Make.
Though W af uses a similar design as SC ons, W af is about 15 times faster for similar features
and without sac rificing build ac curac y . The main reasons for this are the following:
The W af data struc tures (file sy stem representation, tasks) hav e been c arefully c hosen
to minimize memo ry usage and data duplic atio n
For a projec t of the same size, SC ons requires at least 10 times as many func tion c alls
A few benc hmarks are maintained at this loc ation
Waf hashing schemes and build accuracy

To rebuild targets when sourc e file c hange, the file contents are hashed and c ompared. The
hashes are used to identify the tasks, and to retriev e the files fro m a c ache (fo lder defined
by the env ironment v ariable WA FC A C HE ). Besides c ommand-lines, this sc heme also takes
file dependenc ies into acc ount: it is more ac c urate than c ac hing sy stems suc h as ccache .
The W af hashing sc heme uses the md5 algorithm p rov ided by the Py thon distribution . It is
fast enough for up to about 100Mb of data and ab out 10 000 files and v ery safe (v irt ually no
risk of c ollision).
If more than 10 0Mb of data is present in th e project, it may be necessary to use a faster
hashing algorithm. An implementation of the fnv algorithm is present in the W af distribution,
and c an replac e md5 without really d egrading acc urac y .
If more than 10 000 files are present, it may be nec essary to replace the hashing sy stem by
a file name+size+timestamp hash scheme . An ex ample is pro v ided in the c omment
sec tion o f the module Utils.py . That scheme is more effic ient b ut less ac c urat e: the W af
c ac he should not be used with this sc heme.
Chapter 13. Conclusion

The feat ures prov ided by W af aim at building software in a maintainable mann er, in whic h
the dependencies for other p iec e of software is limited to the c ompilers and a Py thon
installation. D ue to the amount of features prov ided by W af, the book is forc ibly inco mplete.
For greater und erstand ing and pract ice the following links are recommended t o the reader:
Table 13.1. Recommended links
Link Description
http://code.google.com/p/waf The Waf project page
The Waf wiki, including the frequently
http://code.google.com/p/waf/w/list
asked questions (FA Q )
http://groups.google.com/group/waf-
The Waf mailing-list
users
Glossary
W af essential terms
Task Gen erator
A task gen erator is an ob jec t in stance of the c lass Task.task_gen. The task generators
enc apsulate the c reation of v arious task instanc es at a time, and simplify the creation
of ordering constraints between them (for ex ample, c ompilation tasks are ex ecu ted
before link tasks).
Task
A W af task is an o bjec t instance of the c lass Task.TaskBase. W af tasks may be simple

(Task.TaskBase) o r related to the filesy stem (Task.Task). Tasks rep resent the
pro duction of something during the build (files in general), and may be ex ec uted in
seq uenc e (with ordering c onstraints) or in parallel.
Tool
A W af tool is a py thon module c ontain ing W af-spec ific ex tensions. The W af tools are
loc ated in the fold er wafadmin/Tools/ and usually c ontain a global v ariable detect
whic h may referen ce fun ctions to ex ec ute in the c onfiguration.

2009 - The Waf Book (v1.5.4)

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

2009 - The Waf Book (v1.5.4)

Hochgeladen von

Copyright:

Verfügbare Formate

10/10/2010 The Waf Book (v1.5.

Abo ut build sy stems in general

2. Introduc tion to the W af sc ripting sy stem

Empty projec t ex ample

3. The c onfiguration sy stem

Ov erv iew of the c onfiguration files

4. Pac kaging and redistributing the sourc e c ode

Quic k tarball creation

5. The Task sy stem

Task c reation and ex ec ution

6. Task encapsu lation using t ask generators

Ex p licit rules for simple transformations (Make-like)

7. The sc heduler for ex ec uting the t asks

The task ex ec ution model

C ommon sc ript for C /C ++ applic ations

9. Adv anc ed sc enarios

Simple file transfo rmations

10. C ustomizing the sc ripting sy stem

Pro v iding c ustom c ommands

Trac ing the ex ecu tion

12. Ov erv iew of the W af arc hitecture

The c ore library

13. C onc lusion

11.1. D eb ugging zones

About build sy stems in general

About build systems in general

The Waf approach

Objectives of this book

Chapter 1. Getting Started

Using the Waf binary

Generating Waf from source

1. Inst allation: installation is c umbersome, and requires administrativ e priv ileges

Or, if the ex ec utable p ermissions are not set properly :

Chapter 2. Introduction to the Waf scripting system

Empty project ex ample

Empty project example

srcd ir: string representin g the sourc e d irec tory

The top-lev el W af sc ript file name is "wsc ript"

The minimum workflow for an y projec t c onsists of the following steps:

To do th is, the followin g shell c ommands will be:

Workflow illustration on a more complicated example

APP NAME: projec t name

The outp ut will be:

A few files will b e produc ed by W af, let us look at them now:

The ex ec ution c an be summed up in the following diagram:

Building a project - Introduction to Tasks

Waf task declaration

The ex ec ution t rac e will be th e following:

Task generator encapsulation

Indicating configuration sub folders

Storing and loading configuratio n parameters

The ex ec ution will result in:

The c ontents of the file default.cache.py is repro duced here:

Using configuration helpers

W ill produc e the following results:

Loading Waf tools

Adding custom command-line options

D ec laring c ustom command-line options

Adding options directly

The c ommand-line opt ions can be display ed afterwards:

and used like th is:

Adding options from wscript files located in sub folders

Adding options from Waf tools

Installing files directly