The Linux Command Line

William E. Shotts, Jr.

A LinuxCommand.org Book

Copyright 2008-2009, William E. Shotts, Jr. This work is lice se! " !er the Creati#e Commo s $ttri%"tio -&o commercial-&o 'eri#ati#e Works (.0 ) ite! States *ice se. To #iew a copy o+ this lice se, #isit http,--creati#ecommo s.org-lice ses-%y- c- !-(.0-"s- or se ! a letter to Creati#e Commo s, ./. Seco ! Street, S"ite (00, Sa 0ra cisco, Cali+or ia, 91.02, )S$. *i "3 is the registere! tra!emark o+ *i "s Tor#al!s. $ll other tra!emarks %elo g to their respecti#e ow ers. This %ook is part o+ the *i "3Comma !.org pro4ect, a site +or *i "3 e!"catio a ! a!#ocacy !e#ote! to helpi g "sers o+ legacy operati g systems migrate i to the +"t"re. 5o" may co tact the *i "3Comma !.org pro4ect at http,--li "3comma !.org. 6ri te! copies o+ this %ook, i large, easy-to-rea!-+ormat, are a#aila%le +or p"rchase +rom l"l".com. 7r!ers may %e place! at the +ollowi g, http,--www.l"l".com-co te t-paper%ack-%ook-the-li "3-comma !-li e-/291.81 Release History Version 09..2 09... 09..0 Date 'ecem%er .1, 2009 &o#em%er .9, 2009 7cto%er (, 2009 Description 0irst E!itio . 0o"rth !ra+t with almost all re#iewer +ee!%ack i corporate! a ! e!ite! thro"gh chapter (/. Thir! !ra+t with re#ise! ta%le +ormatti g, partial applicatio o+ re#iewers +ee!%ack a ! e!ite! thro"gh chapter .8. Seco ! !ra+t i corporati g the +irst e!iti g pass. Complete! +irst !ra+t.

09.08 09.0/

$"g"st .2, 2009 J"ly .8, 2009

Table of Contents

Part 1 Introduction.........................................................................1
1 Introduction................................................................................................2
Why Use The Command Line?.......................................................................................2 What This Book Is About.................................................................................................3 Who Shou d !ead This Book..........................................................................................3 What"s In This Book.........................................................................................................# $o% To !ead This Book..................................................................................................& 'rere(uisites...............................................................................................................& Why I )on"t Ca It *+,U-Linux..............................................................................& A/kno% edgments...........................................................................................................0 1our 2eedba/k Is ,eeded3..............................................................................................4 2urther !eading...............................................................................................................4 Co o5hon.........................................................................................................................4

Part 2 Learning The Shell..............................................................

2 What Is The Shell?...................................................................................10
Termina 6mu ators........................................................................................................78 1our 2irst 9eystrokes....................................................................................................78 Command $istory.....................................................................................................77 Cursor :o;ement.....................................................................................................77 A 2e% Words About :i/e And 2o/us...................................................................77 Try Some Sim5 e Commands........................................................................................72 6nding A Termina Session............................................................................................73 The Conso e Behind The Curtain........................................................................73 2urther !eading.............................................................................................................73

3 Navigation.................................................................................................1
Understanding The 2i e System Tree............................................................................7# The Current Working )ire/tory......................................................................................7# Listing The Contents <= A )ire/tory..............................................................................7& Changing The Current Working )ire/tory.....................................................................70 Abso ute 'athnames................................................................................................70 !e ati;e 'athnames.................................................................................................70 Some $e 5=u Short/uts............................................................................................7>

Im5ortant 2a/ts About 2i enames........................................................................7>

!"#loring The S$ste%.............................................................................20

:ore 2un With s...........................................................................................................28 <5tions And Arguments............................................................................................27 A Longer Look At Long 2ormat.................................................................................22 )etermining A 2i e"s Ty5e With =i e................................................................................23 ?ie%ing 2i e Contents With ess....................................................................................2# What Is *Text.?.....................................................................................................2# Less Is :ore........................................................................................................20 A +uided Tour................................................................................................................20 Symbo i/ Links...............................................................................................................2@ $ard Links.....................................................................................................................38 2urther !eading.............................................................................................................38

& 'ani#ulating (iles )nd *irectories........................................................31

Wi d/ards.......................................................................................................................37 Chara/ter !anges................................................................................................33 Wi d/ards Work In The +UI Too..........................................................................33 mkdir A Create )ire/tories.............................................................................................3# /5 A Co5y 2i es And )ire/tories....................................................................................3# Use=u <5tions And 6xam5 es..................................................................................3& m; A :o;e And !ename 2i es......................................................................................30 Use=u <5tions And 6xam5 es..................................................................................30 rm A !emo;e 2i es And )ire/tories...............................................................................34 Use=u <5tions And 6xam5 es..................................................................................34 Be Care=u With rm3.............................................................................................3> n A Create Links............................................................................................................3@ $ard Links................................................................................................................3@ Symbo i/ Links..........................................................................................................#8 Let"s Bui d A ' ayground................................................................................................#8 Creating )ire/tories..................................................................................................#8 Co5ying 2i es............................................................................................................#7 :o;ing And !enaming 2i es.....................................................................................#2 Creating $ard Links..................................................................................................#3 Creating Symbo i/ Links...........................................................................................## !emo;ing 2i es And )ire/tories...............................................................................#& Creating Sym inks With The +UI.........................................................................#4 Summing U5.............................................................................................................#4

+ Wor,ing With Co%%ands........................................................................ What 6xa/t y Are Commands?......................................................................................#> Identi=ying Commands...................................................................................................#@ ty5e A )is5 ay A Command"s Ty5e...........................................................................#@ %hi/h A )is5 ay An 6xe/utab e"s Lo/ation...............................................................#@ +etting A Command"s )o/umentation..........................................................................&8 he 5 A +et $e 5 2or She Bui tins............................................................................&8 BBhe 5 A )is5 ay Usage In=ormation..........................................................................&7 man A )is5 ay A 'rogram"s :anua 'age................................................................&7 a5ro5os A )is5 ay A55ro5riate Commands..............................................................&3

ii

%hatis A )is5 ay A ?ery Brie= )es/ri5tion <= A Command.......................................&3 The :ost Bruta :an 'age <= Them A ..............................................................&3 in=o A )is5 ay A 'rogram"s In=o 6ntry.......................................................................&# !6A):6 And <ther 'rogram )o/umentation 2i es................................................&& Creating 1our <%n Commands With a ias....................................................................&0 !e;isiting < d 2riends...................................................................................................&> 2urther !eading.............................................................................................................&>

. /edirection................................................................................................&0
Standard In5utC <ut5utC And 6rror.................................................................................&@ !edire/ting Standard <ut5ut.........................................................................................08 !edire/ting Standard 6rror............................................................................................07 !edire/ting Standard <ut5ut And Standard 6rror To <ne 2i e................................02 )is5osing <= Un%anted <ut5ut................................................................................03 -de;-nu In Unix Cu ture......................................................................................03 !edire/ting Standard In5ut............................................................................................03 /at A Con/atenate 2i es............................................................................................03 'i5e ines........................................................................................................................0& 2i ters........................................................................................................................00 uni( B !e5ort <r <mit !e5eated Lines....................................................................00 %/ A 'rint LineC WordC And Byte Counts..................................................................04 gre5 A 'rint Lines :at/hing A 'attern......................................................................04 head - tai A 'rint 2irst - Last 'art <= 2i es................................................................0> tee A !ead 2rom Stdin And <ut5ut To Stdout And 2i es..........................................0@ Summing U5..................................................................................................................48 Linux Is About Imagination..................................................................................48

- Seeing The World )s The Shell Sees It...................................................2

6x5ansion......................................................................................................................42 'athname 6x5ansion...............................................................................................43 'athname 6x5ansion <= $idden 2i es.................................................................4# Ti de 6x5ansion........................................................................................................4# Arithmeti/ 6x5ansion................................................................................................4& Bra/e 6x5ansion......................................................................................................40 'arameter 6x5ansion...............................................................................................44 Command Substitution.............................................................................................4> Duoting..........................................................................................................................4@ )oub e Duotes..........................................................................................................4@ Sing e Duotes...........................................................................................................>7 6s/a5ing Chara/ters................................................................................................>2 Ba/ks ash 6s/a5e Se(uen/es............................................................................>2 Summing U5..................................................................................................................>3 2urther !eading.............................................................................................................>3

0 )dvanced 1e$board Tric,s.....................................................................Command Line 6diting..................................................................................................># Cursor :o;ement.....................................................................................................># :odi=ying Text...........................................................................................................>& Cutting And 'asting E9i ing And 1ankingF Text........................................................>& The :eta 9ey......................................................................................................>0

iii

Com5 etion....................................................................................................................>0 'rogrammab e Com5 etion..................................................................................>> Using $istory.................................................................................................................>> Sear/hing $istory.....................................................................................................>@ $istory 6x5ansion.....................................................................................................@7 s/ri5t....................................................................................................................@7 Summing U5..................................................................................................................@2 2urther !eading.............................................................................................................@2

10 2er%issions............................................................................................03
<%nersC +rou5 :embersC And 6;erybody 6 se............................................................@# !eadingC WritingC And 6xe/uting...................................................................................@& /hmod A Change =i e mode.......................................................................................@4 What The $e/k Is </ta ?.....................................................................................@> Setting 2i e :ode With The +UI.............................................................................788 umask A Set )e=au t 'ermissions..........................................................................787 Some S5e/ia 'ermissions................................................................................783 Changing Identities......................................................................................................78# su A !un A She With Substitute User And +rou5 I)s..........................................78& sudo A 6xe/ute A Command As Another User.......................................................780 Ubuntu And sudo...............................................................................................784 /ho%n A Change 2i e <%ner And +rou5................................................................784 /hgr5 A Change +rou5 <%nershi5.........................................................................78@ 6xer/ising <ur 'ri;i eges............................................................................................78@ Changing 1our 'ass%ord.............................................................................................777 2urther !eading...........................................................................................................772

11 2rocesses..............................................................................................113
$o% A 'ro/ess Works.................................................................................................773 ?ie%ing 'ro/esses......................................................................................................77# ?ie%ing 'ro/esses )ynami/a y With to5...............................................................770 Contro ing 'ro/esses..................................................................................................77> Interru5ting A 'ro/ess.............................................................................................77@ 'utting A 'ro/ess In The Ba/kground....................................................................77@ !eturning A 'ro/ess To The 2oreground...............................................................728 Sto55ing E'ausingF A 'ro/ess................................................................................727 Signa s.........................................................................................................................722 Sending Signa s To 'ro/esses With ki .................................................................722 Sending Signa s To :u ti5 e 'ro/esses With ki a ................................................72& :ore 'ro/ess !e ated Commands.............................................................................72&

Part ! Con"iguration #nd The $n%ironment.............................12&

12 The !nviron%ent..................................................................................12What Is Stored In The 6n;ironment?..........................................................................72> 6xamining The 6n;ironment..................................................................................72> Some Interesting ?ariab es.....................................................................................738 $o% Is The 6n;ironment 6stab ished?.......................................................................737 What"s In A Startu5 2i e?........................................................................................732

i#

:odi=ying The 6n;ironment.........................................................................................73# Whi/h 2i es Shou d We :odi=y?.............................................................................73# Text 6ditors.............................................................................................................73# Using A Text 6ditor.................................................................................................73& Why Comments Are Im5ortant..........................................................................73> A/ti;ating <ur Changes..........................................................................................73@ Summing U5................................................................................................................73@ 2urther !eading..........................................................................................................73@

13 ) 3entle Introduction To vi.................................................................1 0

Why We Shou d Learn ;i.............................................................................................7#8 A Litt e Ba/kground ....................................................................................................7#7 Starting And Sto55ing ;i..............................................................................................7#7 Com5atibi ity :ode............................................................................................7#2 6diting :odes..............................................................................................................7#3 6ntering Insert :ode..............................................................................................7## Sa;ing <ur Work....................................................................................................7## :o;ing The Cursor Around.........................................................................................7#& Basi/ 6diting................................................................................................................7#0 A55ending Text.......................................................................................................7#0 <5ening A Line.......................................................................................................7#4 )e eting Text...........................................................................................................7#> CuttingC Co5ying And 'asting Text.........................................................................7#@ Goining Lines...........................................................................................................7&7 Sear/h And !e5 a/e....................................................................................................7&7 Sear/hing Within A Line..........................................................................................7&7 Sear/hing The 6ntire 2i e.......................................................................................7&7 + oba Sear/h And !e5 a/e...................................................................................7&2 6diting :u ti5 e 2i es...................................................................................................7&# S%it/hing Bet%een 2i es.........................................................................................7&& <5ening Additiona 2i es 2or 6diting......................................................................7&& Co5ying Content 2rom <ne 2i e Into Another........................................................7&0 Inserting An 6ntire 2i e Into Another.......................................................................7&4 Sa;ing <ur Work.........................................................................................................7&> 2urther !eading..........................................................................................................7&@

1 Custo%i4ing The 2ro%#t.....................................................................1+0

Anatomy <= A 'rom5t..................................................................................................708 Trying Some A ternate 'rom5t )esigns......................................................................702 Adding Co or................................................................................................................703 Termina Con=usion............................................................................................703 :o;ing The Cursor......................................................................................................700 Sa;ing The 'rom5t......................................................................................................704 Summing U5................................................................................................................704 2urther !eading..........................................................................................................70>

Part ' Common Tas(s #nd $ssential Tools.............................1)

1& 2ac,age 'anage%ent..........................................................................1.0 #

'a/kaging Systems.....................................................................................................748 $o% A 'a/kage System Works...................................................................................747 'a/kage 2i es.........................................................................................................747 !e5ositories............................................................................................................747 )e5enden/ies.........................................................................................................742 $igh And Lo%B e;e 'a/kage Too s........................................................................742 Common 'a/kage :anagement Tasks.......................................................................743 2inding A 'a/kage In A !e5ository........................................................................743 Insta ing A 'a/kage 2rom A !e5ository.................................................................743 Insta ing A 'a/kage 2rom A 'a/kage 2i e.............................................................74# !emo;ing A 'a/kage.............................................................................................74# U5dating 'a/kages 2rom A !e5ository..................................................................74& U5grading A 'a/kage 2rom A 'a/kage 2i e...........................................................74& Listing Insta ed 'a/kages......................................................................................740 )etermining I= A 'a/kage Is Insta ed.....................................................................740 )is5 aying In=o About An Insta ed 'a/kage...........................................................744 2inding Whi/h 'a/kage Insta ed A 2i e.................................................................744 Summing U5................................................................................................................744 The Linux So=t%are Insta ation :yth.................................................................74> 2urther !eading..........................................................................................................74@

1+ Storage 'edia.......................................................................................1-0
:ounting And Unmounting Storage )e;i/es..............................................................7>8 ?ie%ing A List <= :ounted 2i e Systems................................................................7>2 Why Unmounting Is Im5ortant...........................................................................7>& )etermining )e;i/e ,ames....................................................................................7>0 Creating ,e% 2i e Systems.........................................................................................7>@ :ani5u ating 'artitions With =disk..........................................................................7>@ Creating A ,e% 2i e System With mk=s..................................................................7@2 Testing And !e5airing 2i e Systems............................................................................7@3 What The =s/k?..................................................................................................7@3 2ormatting 2 o55y )isks..............................................................................................7@3 :o;ing )ata )ire/t y To-2rom )e;i/es.......................................................................7@# Creating C)B!<: Images..........................................................................................7@& Creating An Image Co5y <= A C)B!<:.................................................................7@& Creating An Image 2rom A Co e/tion <= 2i es.......................................................7@& A 'rogram By Any <ther ,ame.........................................................................7@0 Writing C)B!<: Images.............................................................................................7@0 :ounting An IS< Image )ire/t y............................................................................7@0 B anking A !eBWritab e C)B!<:...........................................................................7@4 Writing An Image....................................................................................................7@4 2urther !eading..........................................................................................................7@4 6xtra Credit..................................................................................................................7@4

1. Net5or,ing............................................................................................100
6xamining And :onitoring A ,et%ork.........................................................................288 5ing.........................................................................................................................288 tra/eroute...............................................................................................................287 netstat.....................................................................................................................282 Trans5orting 2i es <;er A ,et%ork..............................................................................283

#i

=t5............................................................................................................................28# =t5 A A Better =t5.....................................................................................................280 %get........................................................................................................................280 Se/ure Communi/ation With !emote $osts...............................................................280 ssh..........................................................................................................................284 Tunne ing With SS$..........................................................................................278 s/5 And s=t5............................................................................................................277 An SS$ C ient 2or Windo%s?............................................................................272 2urther !eading..........................................................................................................272

1- Searching (or (iles..............................................................................213

o/ate A 2ind 2i es The 6asy Way...............................................................................273 Where )oes The o/ate )atabase Come 2rom?..............................................27& =ind A 2ind 2i es The $ard Way...................................................................................27& Tests.......................................................................................................................270 <5erators................................................................................................................27> 'rede=ined A/tions..................................................................................................227 User )e=ined A/tions..............................................................................................223 Im5ro;ing 6==i/ien/y...............................................................................................22# xargs.......................................................................................................................22# )ea ing With 2unny 2i enames..........................................................................22& A !eturn To The ' ayground..................................................................................220 <5tions...................................................................................................................22> 2urther !eading..........................................................................................................22@

10 )rchiving )nd 6ac,u#.........................................................................230

Com5ressing 2i es.......................................................................................................238 gHi5.........................................................................................................................237 bHi52.......................................................................................................................233 )onIt Be Com5ressi;e Com5u si;e...................................................................23# Ar/hi;ing 2i es.............................................................................................................23# tar............................................................................................................................23# Hi5...........................................................................................................................2#8 Syn/hroniHing 2i es And )ire/tories............................................................................2#2 Using rsyn/ <;er A ,et%ork...................................................................................2#& 2urther !eading..........................................................................................................2#0

20 /egular !"#ressions...........................................................................2 .
What Are !egu ar 6x5ressions?............................................................................2#4 gre5.............................................................................................................................2#4 :eta/hara/ters And Litera s........................................................................................2#@ The Any Chara/ter......................................................................................................2&8 An/hors.......................................................................................................................2&7 A Cross%ord 'uHH e $e 5er...............................................................................2&7 Bra/ket 6x5ressions And Chara/ter C asses..............................................................2&2 ,egation.................................................................................................................2&2 Traditiona Chara/ter !anges.................................................................................2&3 '<SIJ Chara/ter C asses......................................................................................2&# !e;erting To Traditiona Co ation <rder............................................................2&4 '<SIJ Basi/ ?s. 6xtended !egu ar 6x5ressions.......................................................2&>

#ii

'<SIJ................................................................................................................2&> A ternation...................................................................................................................2&@ Duanti=iers...................................................................................................................208 ? B :at/h An 6 ement Kero <r <ne Time...............................................................208 L B :at/h An 6 ement Kero <r :ore Times............................................................207 M B :at/h An 6 ement <ne <r :ore Times............................................................202 N O B :at/h An 6 ement A S5e/i=i/ ,umber <= Times..............................................202 'utting !egu ar 6x5ressions To Work.........................................................................203 ?a idating A 'hone List With gre5...........................................................................203 2inding Ug y 2i enames With =ind...........................................................................20# Sear/hing 2or 2i es With o/ate.............................................................................20& Sear/hing 2or Text In ess And ;im........................................................................20& Summing U5................................................................................................................204 2urther !eading..........................................................................................................204

21 Te"t 2rocessing....................................................................................2+A55 i/ations <= Text.....................................................................................................20> )o/uments.............................................................................................................20@ Web 'ages.............................................................................................................20@ 6mai .......................................................................................................................20@ 'rinter <ut5ut.........................................................................................................20@ 'rogram Sour/e Code............................................................................................20@ !e;isiting Some < d 2riends.......................................................................................20@ /at...........................................................................................................................248 :SB)<S Text ?s. Unix Text...............................................................................247 sort..........................................................................................................................247 uni(.........................................................................................................................24@ S i/ing And )i/ing........................................................................................................2>8 /ut...........................................................................................................................2>8 6x5anding Tabs.................................................................................................2>3 5aste.......................................................................................................................2># Poin..........................................................................................................................2>& Com5aring Text...........................................................................................................2>4 /omm......................................................................................................................2>> di==...........................................................................................................................2>> 5at/h.......................................................................................................................2@7 6diting <n The 2 y.......................................................................................................2@2 tr..............................................................................................................................2@2 !<T73Q The ,otBSoBSe/ret )e/oder !ing........................................................2@# sed..........................................................................................................................2@& 'eo5 e Who Like sed A so Like.........................................................................383 as5e ......................................................................................................................383 Summing U5................................................................................................................384 2urther !eading..........................................................................................................384 6xtra Credit..................................................................................................................38>

22 (or%atting 7ut#ut................................................................................300
Sim5 e 2ormatting Too s..............................................................................................38@ n A ,umber Lines..................................................................................................38@ =o d A Wra5 6a/h Line To A S5e/i=ied Length........................................................373

#iii

=mt A A Sim5 e Text 2ormatter................................................................................373 5r A 2ormat Text 2or 'rinting..................................................................................374 5rint= A 2ormat And 'rint )ata................................................................................37> )o/ument 2ormatting Systems...................................................................................327 gro==.........................................................................................................................322 Summing U5................................................................................................................32> 2urther !eading..........................................................................................................32>

23 2rinting..................................................................................................330
A Brie= $istory <= 'rinting............................................................................................338 'rinting In The )im Times......................................................................................338 Chara/terBbased 'rinters.......................................................................................337 +ra5hi/a 'rinters...................................................................................................332 'rinting With Linux......................................................................................................333 're5aring 2i es 2or 'rinting.........................................................................................333 5r A Con;ert Text 2i es 2or 'rinting........................................................................333 Sending A 'rint Gob To A 'rinter..................................................................................33& 5r A 'rint 2i es EBerke ey Sty eF.............................................................................33& 5 A 'rint 2i es ESystem ? Sty eF.............................................................................330 Another <5tionQ a25s..............................................................................................334 :onitoring And Contro ing 'rint Gobs.........................................................................3#8 5stat A )is5 ay 'rint System Status......................................................................3#8 5( A )is5 ay 'rinter Dueue Status........................................................................3#7 5rm - /an/e A Can/e 'rint Gobs..........................................................................3#2 Summing U5................................................................................................................3#2 2urther !eading..........................................................................................................3#2

2 Co%#iling 2rogra%s............................................................................3
What Is Com5i ing?.....................................................................................................3## Are A 'rograms Com5i ed?..................................................................................3#& Com5i ing A C 'rogram...............................................................................................3#0 <btaining The Sour/e Code...................................................................................3#0 6xamining The Sour/e Tree...................................................................................3#> Bui ding The 'rogram.............................................................................................3&8 Insta ing The 'rogram...........................................................................................3&# Summing U5................................................................................................................3&# 2urther !eading..........................................................................................................3&&

Part * +riting Shell Scripts.......................................................!*&

2& Writing 8our (irst Scri#t......................................................................3&What Are She S/ri5ts?...............................................................................................3&> $o% To Write A She S/ri5t.........................................................................................3&> S/ri5t 2i e 2ormat........................................................................................................3&@ 6xe/utab e 'ermissions..............................................................................................308 S/ri5t 2i e Lo/ation......................................................................................................308 +ood Lo/ations 2or S/ri5ts....................................................................................302 :ore 2ormatting Tri/ks................................................................................................302 Long <5tion ,ames................................................................................................302

i3

Indentation And Line Continuation.........................................................................302 Con=iguring ;im 2or S/ri5t Writing.....................................................................303 Summing U5................................................................................................................30# 2urther !eading..........................................................................................................30#

2+ Starting ) 2ro9ect.................................................................................3+&
2irst StageQ :inima )o/ument...................................................................................30& Se/ond StageQ Adding A Litt e )ata............................................................................304 ?ariab es And Constants.............................................................................................30> Assigning ?a ues To ?ariab es And Constants.......................................................347 $ere )o/uments.........................................................................................................342 Summing U5................................................................................................................34& 2urther !eading..........................................................................................................34&

2. To#:*o5n *esign.................................................................................3.+
She 2un/tions............................................................................................................344 Lo/a ?ariab es............................................................................................................3>8 9ee5 S/ri5ts !unning..................................................................................................3>7 She 2un/tions In 1our .bashr/ 2i e..................................................................3># Summing U5................................................................................................................3># 2urther !eading..........................................................................................................3>#

2- (lo5 Control; 6ranching With if.........................................................3-&

i=...................................................................................................................................3>& 6xit Status...................................................................................................................3>0 test...............................................................................................................................3>> 2i e 6x5ressions.....................................................................................................3>> String 6x5ressions..................................................................................................3@7 Integer 6x5ressions................................................................................................3@2 A :ore :odern ?ersion <= test...................................................................................3@3 EE FF B )esigned 2or Integers........................................................................................3@& Combining 6x5ressions...............................................................................................3@0 'ortabi ity Is The $obgob in <= Litt e :inds.......................................................3@> Contro <5eratorsQ Another Way To Bran/h................................................................3@@ Summing U5................................................................................................................3@@ 2urther !eading..........................................................................................................#88

20 /eading 1e$board In#ut...................................................................... 01

read A !ead ?a ues 2rom Standard In5ut...................................................................#82 <5tions...................................................................................................................#8# I2S..........................................................................................................................#80 1ou CanIt 'i5e read...........................................................................................#84 ?a idating In5ut............................................................................................................#8> :enus..........................................................................................................................#8@ Summing U5................................................................................................................#77 6xtra Credit.............................................................................................................#77 2urther !eading...........................................................................................................#77

30 (lo5 Control; <oo#ing With 5hile = until........................................... 12 3

Loo5ing........................................................................................................................#72 %hi e........................................................................................................................#72 Breaking <ut <= A Loo5...............................................................................................#7& unti .........................................................................................................................#70 !eading 2i es With Loo5s...........................................................................................#74 Summing U5................................................................................................................#7> 2urther !eading..........................................................................................................#7>

31 Troubleshooting................................................................................... 10
Synta/ti/ 6rrors...........................................................................................................#7@ :issing Duotes.......................................................................................................#28 :issing <r Unex5e/ted Tokens..............................................................................#28 Unanti/i5ated 6x5ansions......................................................................................#27 Logi/a 6rrors .............................................................................................................#23 )e=ensi;e 'rogramming.........................................................................................#23 ?eri=ying In5ut.........................................................................................................#2& )esign Is A 2un/tion <= Time............................................................................#2& Testing.........................................................................................................................#2& Test Cases..............................................................................................................#20 )ebugging...................................................................................................................#24 2inding The 'rob em Area......................................................................................#24 Tra/ing....................................................................................................................#24 6xamining ?a ues )uring 6xe/ution......................................................................#38 Summing U5................................................................................................................#38 2urther !eading..........................................................................................................#37

32 (lo5 Control; 6ranching With case................................................... 32

/ase.............................................................................................................................#32 'atterns..................................................................................................................#3# Summing U5................................................................................................................#30 2urther !eading..........................................................................................................#30

33 2ositional 2ara%eters......................................................................... 3.
A//essing The Command Line...................................................................................#34 )etermining The ,umber o= Arguments.................................................................#3> shi=t A +etting A//ess To :any Arguments............................................................#3@ Sim5 e A55 i/ations................................................................................................##8 Using 'ositiona 'arameters With She 2un/tions................................................##7 $and ing 'ositiona 'arameters 6n :asse.................................................................##2 A :ore Com5 ete A55 i/ation......................................................................................### Summing U5................................................................................................................##4 2urther !eading..........................................................................................................#&8

3 (lo5 Control; <oo#ing With for.......................................................... &1

=orQ Traditiona She 2orm...........................................................................................#&7 Why i?................................................................................................................#&3 =orQ C Language 2orm.................................................................................................#&# Summing U5................................................................................................................#&& 2urther !eading..........................................................................................................#&0

3i

3& Strings )nd Nu%bers.......................................................................... &.

'arameter 6x5ansion..................................................................................................#&4 Basi/ 'arameters...................................................................................................#&4 6x5ansions To :anage 6m5ty ?ariab es...............................................................#&> 6x5ansions That !eturn ?ariab e ,ames..............................................................#08 String <5erations....................................................................................................#08 Arithmeti/ 6;a uation And 6x5ansion.........................................................................#03 ,umber Bases........................................................................................................#0# Unary <5erators.....................................................................................................#0# Sim5 e Arithmeti/....................................................................................................#0# Assignment.............................................................................................................#00 Bit <5erations.........................................................................................................#0> Logi/.......................................................................................................................#0@ b/ A An Arbitrary 're/ision Ca /u ator Language........................................................#42 Using b/..................................................................................................................#43 An 6xam5 e S/ri5t..................................................................................................#43 Summing U5................................................................................................................#4& 6xtra Credit..................................................................................................................#4& 2urther !eading..........................................................................................................#4&

3+ )rra$s.................................................................................................... .+
What Are Arrays?........................................................................................................#40 Creating An Array........................................................................................................#40 Assigning ?a ues To An Array......................................................................................#44 A//essing Array 6 ements...........................................................................................#4> Array <5erations.........................................................................................................#>8 <ut5utting The 6ntire Contents <= An Array..........................................................#>8 )etermining The ,umber <= Array 6 ements.........................................................#>8 2inding The Subs/ri5ts Used By An Array.............................................................#>7 Adding 6 ements To The 6nd <= An Array.............................................................#>7 Sorting An Array......................................................................................................#>2 )e eting An Array....................................................................................................#>2 Summing U5................................................................................................................#>3 2urther !eading..........................................................................................................#>#

3. !"otica................................................................................................... -&
+rou5 Commands And Subshe s...............................................................................#>& 'ro/ess Substitution...............................................................................................#>0 Tra5s............................................................................................................................#>> Tem5orary 2i es.................................................................................................#@7 Asyn/hronous 6xe/ution.............................................................................................#@2 %ait.........................................................................................................................#@2 ,amed 'i5es...............................................................................................................#@3 Setting U5 A ,amed 'i5e.......................................................................................#@# Using ,amed 'i5es................................................................................................#@# Summing U5................................................................................................................#@& 2urther !eading..........................................................................................................#@&

Inde".............................................................................................................. 0+ 3ii

6art . 8 9 tro!"ctio

'art 7 A Introdu/tion

. 8 9 tro!"ctio

1 Introduction
9 wa t to tell yo" a story. &o, ot the story o+ how, i .99., *i "s Tor#al!s wrote the +irst #ersio o+ the *i "3 ker el. 5o" ca rea! that story i lots o+ *i "3 %ooks. &or am 9 goi g to tell yo" the story o+ how, some years earlier, :ichar! Stallma %ega the ;&) 6ro4ect to create a +ree ) i3-like operati g system. That<s a importa t story too, %"t most other *i "3 %ooks ha#e that o e, as well. &o, 9 wa t to tell yo" the story o+ how yo" ca take %ack co trol o+ yo"r comp"ter. Whe 9 %ega worki g with comp"ters as a college st"!e t i the late .9/0s, there was a re#ol"tio goi g o . The i #e tio o+ the microprocessor ha! ma!e it possi%le +or or!i ary people like yo" a ! me to act"ally ow a comp"ter. 9t<s har! +or ma y people to!ay to imagi e what the worl! was like whe o ly %ig %"si ess a ! %ig go#er me t ra all the comp"ters. *et<s 4"st say, yo" co"l! <t get m"ch !o e. To!ay, the worl! is #ery !i++ere t. Comp"ters are e#erywhere, +rom ti y wristwatches to gia t !ata ce ters to e#erythi g i %etwee . 9 a!!itio to "%i="ito"s comp"ters, we also ha#e a "%i="ito"s etwork co ecti g them together. This has create! a wo !ro"s ew age o+ perso al empowerme t a ! creati#e +ree!om, %"t o#er the last co"ple o+ !eca!es somethi g else has %ee happe i g. $ si gle gia t corporatio has %ee imposi g its co trol o#er most o+ the worl!<s comp"ters a ! !eci!i g what yo" ca a ! ca ot !o with them. 0ort" ately, people +rom all o#er the worl! are !oi g somethi g a%o"t it. They are +ighti g to mai tai co trol o+ their comp"ters %y writi g their ow so+tware. They are %"il!i g *i "3. >a y people speak o+ ?+ree!om@ with regar! to *i "3, %"t 9 !o <t thi k most people k ow what this +ree!om really mea s. 0ree!om is the power to !eci!e what yo"r comp"ter !oes, a ! the o ly way to ha#e this +ree!om is to k ow what yo"r comp"ter is !oi g. 0ree!om is a comp"ter that is witho"t secrets, o e where e#erythi g ca %e k ow i+ yo" care e o"gh to +i ! o"t.

+h, -se The Command Line.

Aa#e yo" e#er otice! i the mo#ies whe the ?s"per hacker,@B yo" k ow, the g"y who ca %reak i to the "ltra-sec"re military comp"ter i " !er thirty seco !sBsits !ow at 2

Why )se The Comma ! *i eC the comp"ter, he e#er to"ches a mo"seC 9t<s %eca"se mo#ie makers realiDe that we, as h"ma %ei gs, i sti cti#ely k ow the o ly way to really get a ythi g !o e o a comp"ter is %y typi g o a key%oar!. >ost comp"ter "sers to!ay are o ly +amiliar with the graphical user interface E;)9F a ! ha#e %ee ta"ght %y #e !ors a ! p" !its that the command line interface EC*9F is a terri+yi g thi g o+ the past. This is " +ort" ate, %eca"se a goo! comma ! li e i ter+ace is a mar#elo"sly e3pressi#e way o+ comm" icati g with a comp"ter i m"ch the same way the writte wor! is +or h"ma %ei gs. 9t<s %ee sai! that ?graphical "ser i ter+aces make easy tasks easy, while comma ! li e i ter+aces make !i++ic"lt tasks possi%le@ a ! this is still #ery tr"e to!ay. Si ce *i "3 is mo!ele! a+ter the ) i3 +amily o+ operati g systems, it shares the same rich heritage o+ comma ! li e tools as ) i3. ) i3 came i to promi e ce !"ri g the early .980s Ealtho"gh it was +irst !e#elope! a !eca!e earlierF, %e+ore the wi!esprea! a!optio o+ the graphical "ser i ter+ace a !, as a res"lt, !e#elope! a e3te si#e comma ! li e i ter+ace i stea!. 9 +act, o e o+ the stro gest reaso s early a!opters o+ *i "3 chose it o#er, say, Wi !ows &T was the power+"l comma ! li e i ter+ace which ma!e the ?!i++ic"lt tasks possi%le.@

+hat This /oo( Is #0out

This %ook is a %roa! o#er#iew o+ ?li#i g@ o the *i "3 comma ! li e. ) like some %ooks that co ce trate o 4"st a si gle program, s"ch as the shell program, bash, this %ook will try to co #ey how to get alo g with the comma ! li e i ter+ace i a larger se se. Aow !oes it all workC What ca it !oC What<s the %est way to "se itC This is not a book about Linux system administration. While a y serio"s !isc"ssio o+ the comma ! li e will i #aria%ly lea! to system a!mi istratio topics, this %ook o ly to"ches o a +ew a!mi istratio iss"es. 9t will, howe#er, prepare the rea!er +or a!!itio al st"!y %y pro#i!i g a soli! +o" !atio i the "se o+ the comma ! li e, a esse tial tool +or a y serio"s system a!mi istratio task. This book is very Linux-centric. >a y other %ooks try to %roa!e their appeal %y i cl"!i g other plat+orms s"ch as ge eric ) i3 a ! >ac7S G. 9 !oi g so, they ?water !ow @ their co te t to +eat"re o ly ge eral topics. This %ook, o the other ha !, o ly co#ers co temporary *i "3 !istri%"tio s. &i ety-+i#e perce t o+ the co te t is "se+"l +or "sers o+ other ) i3-like systems, %"t this %ook is highly targete! at the mo!er *i "3 comma ! li e "ser.

+ho Should 1ead This /oo(

This %ook is +or ew *i "3 "sers who ha#e migrate! +rom other plat+orms. >ost likely yo" are a ?power "ser@ o+ some #ersio o+ >icroso+t Wi !ows. 6erhaps yo"r %oss has tol! yo" to a!mi ister a *i "3 ser#er, or may%e yo"<re 4"st a !esktop "ser who is tire! o+ (

. 8 9 tro!"ctio all the sec"rity pro%lems a ! wa t to gi#e *i "3 a try. That<s +i e. here. $ll are welcome

That %ei g sai!, there is o shortc"t to *i "3 e lighte me t. *ear i g the comma ! li e is challe gi g a ! takes real e++ort. 9t<s ot that it<s so har!, %"t rather it<s so vast. The a#erage *i "3 system has literally thousands o+ programs yo" ca employ o the comma ! li e. Co si!er yo"rsel+ war e!H lear i g the comma ! li e is ot a cas"al e !ea#or. 7 the other ha !, lear i g the *i "3 comma ! li e is e3tremely rewar!i g. 9+ yo" thi k yo"<re a ?power "ser@ ow, 4"st wait. 5o" !o <t k ow what real power isByet. $ !, " like ma y other comp"ter skills, k owle!ge o+ the comma ! li e is lo g lasti g. The skills lear e! to!ay will still %e "se+"l te years +rom ow. The comma ! li e has s"r#i#e! the test o+ time. 9t is also ass"me! that yo" ha#e o programmi g e3perie ce, %"t ot to worry, we<ll start yo" !ow that path as well.

+hat2s In This /oo(

This material is prese te! i a care+"lly chose se="e ce, m"ch like a t"tor sitti g e3t to yo" g"i!i g yo" alo g. >a y a"thors treat this material i a ?systematic@ +ashio , which makes se se +rom a writerIs perspecti#e, %"t ca %e #ery co +"si g to ew "sers. $ other goal is to ac="ai t yo" with the ) i3 way o+ thi ki g, which is !i++ere t +rom the Wi !ows way o+ thi ki g. $lo g the way, we<ll go o a +ew si!e trips to help yo" " !ersta ! why certai thi gs work the way they !o a ! how they got that way. *i "3 is ot 4"st a piece o+ so+tware, it<s also a small part o+ the larger ) i3 c"lt"re, which has its ow la g"age a ! history. 9 might throw i a ra t or two, as well. This %ook is !i#i!e! i to +i#e parts, each co#eri g some aspect o+ the comma ! li e e3perie ce. Jesi!es the +irst part, which yo" are rea!i g ow, this %ook co tai s,

Part 2 Learning The Shell starts o"r e3ploratio o+ the %asic la g"age o+ the comma ! li e i cl"!i g s"ch thi gs as the str"ct"re o+ comma !s, +ile system a#igatio , comma ! li e e!iti g, a ! +i !i g help a ! !oc"me tatio +or comma !s. Part !on"iguration #nd The $nvironment co#ers e!iti g co +ig"ratio +iles that co trol the comp"ter<s operatio +rom the comma ! li e. Part % !ommon Tasks #nd $ssential Tools e3plores ma y o+ the or!i ary tasks that are commo ly per+orme! +rom the comma ! li e. ) i3-like operati g systems, s"ch as *i "3, co tai ma y ?classic@ comma ! li e programs that are "se! to per+orm power+"l operatio s o !ata. Part & 'riting Shell Scri(ts i tro!"ces shell programmi g, a a!mitte!ly

What<s 9 This Jook r"!ime tary, %"t easy to lear , tech i="e +or a"tomati g ma y commo comp"ti g tasks. Jy lear i g shell programmi g, yo" will %ecome +amiliar with co cepts that ca %e applie! to ma y other programmi g la g"ages.

3o4 To 1ead This /oo(

Start at the %egi i g o+ the %ook a ! +ollow it to the e !. 9t is It writte as a re+ere ce work, it<s really more like a story with a %egi i g, mi!!le, a ! a e !.

'rere(uisites
To "se this %ook, all yo" will ee! is a worki g *i "3 i stallatio . 5o" ca get this i o e o+ two ways, .. )nstall Linux on a *not so ne+, com(uter. 9t !oes <t matter which !istri%"tio yo" choose, tho"gh most people to!ay start o"t with either )%" t", 0e!ora, or 7pe S)SE. 9+ i !o"%t, try )%" t" +irst. 9 stalli g a mo!er *i "3 !istri%"tio ca %e ri!ic"lo"sly easy or ri!ic"lo"sly !i++ic"lt !epe !i g o yo"r har!ware. 9 s"ggest a !esktop comp"ter that is a co"ple o+ years ol! a ! has at least 22K mega%ytes o+ :$> a ! K giga%ytes o+ +ree har! !isk space. $#oi! laptops a ! wireless etworks i+ at all possi%le, as these are o+te more !i++ic"lt to get worki g. 2. -se a .Live !/.0 7 e o+ the cool thi gs yo" ca !o with ma y *i "3 !istri%"tio s is r" them !irectly +rom a C':7> witho"t i stalli g them at all. J"st go i to yo"r J97S set"p a ! set yo"r comp"ter to ?Joot +rom C':7>,@ i sert the li#e C', a ! re%oot. )si g a li#e C' is a great way to test a comp"ter +or *i "3 compati%ility prior to i stallatio . The !isa!#a tage o+ "si g a li#e C' is that it may %e #ery slow compare! to ha#i g *i "3 i stalle! o yo"r har! !ri#e. Joth )%" t" a ! 0e!ora Eamo g othersF ha#e li#e C' #ersio s. :egar!less o+ how yo" i stall *i "3, yo" will ee! to ha#e occasio al s"per"ser Ei.e., a!mi istrati#eF pri#ileges to carry o"t the lesso s i this %ook. $+ter yo" ha#e a worki g i stallatio , start rea!i g a ! +ollow alo g with yo"r ow comp"ter. >ost o+ the material i this %ook is ?ha !s o ,@ so sit !ow a ! get typi gL

+h, I Don2t Call It 567-8Linux9

9 some ="arters, it<s politically correct to call the *i "3 operati g system the ?;&)-*i "3 operati g system.@ The pro%lem with ?*i "3@ is that there is o completely correct way to ame it %eca"se it was writte %y ma y !i++ere t

. 8 9 tro!"ctio

people i a #ast, !istri%"te! !e#elopme t e++ort. Tech ically speaki g, *i "3 is the ame o+ the operati g system<s ker el, othi g more. The ker el is #ery importa t o+ co"rse, si ce it makes the operati g system go, %"t it<s ot e o"gh to +orm a complete operati g system. E ter :ichar! Stallma , the ge i"s-philosopher who +o" !e! the 0ree So+tware mo#eme t, starte! the 0ree So+tware 0o" !atio , +orme! the ;&) 6ro4ect, wrote the +irst #ersio o+ the ;&) C Compiler EgccF, create! the ;&) ;e eral 6"%lic *ice se Ethe ;6*F, etc., etc., etc. Ae insists that yo" call it ?;&)-*i "3@ to properly re+lect the co tri%"tio s o+ the ;&) 6ro4ect. While the ;&) 6ro4ect pre!ates the *i "3 ker el, a ! the pro4ect<s co tri%"tio s are e3tremely !eser#i g o+ recog itio , placi g them i the ame is " +air to e#eryo e else who ma!e sig i+ica t co tri%"tio s. Jesi!es, 9 thi k ?*i "3-;&)@ wo"l! %e more tech ically acc"rate si ce the ker el %oots +irst a ! e#erythi g else r" s o top o+ it. 9 pop"lar "sage, ?*i "3@ re+ers to the ker el a ! all the other +ree a ! ope so"rce so+tware +o" ! i the typical *i "3 !istri%"tio H that is, the e tire *i "3 ecosystem, ot 4"st the ;&) compo e ts. The operati g system marketplace seems to pre+er o e-wor! ames s"ch as '7S, Wi !ows, >ac7S, Solaris, 9ri3, $9G. 9 ha#e chose to "se the pop"lar +ormat. 9+, howe#er, yo" pre+er to "se ?;&)-*i "3@ i stea!, please per+orm a me tal search a ! replace while rea!i g this %ook. 9 wo <t mi !.

#c(no4ledgments
9 wa t to tha k the +ollowi g people, who helpe! make this %ook possi%le, Je y Watso , $c="isitio s E!itor at Wiley 6"%lishi g who origi ally s"ggeste! that 9 write a shell scripti g %ook. Joh C. '#orak, ote! col"m ist a ! p" !it. 9 a episo!e o+ his #i!eo po!cast, ?Cra ky ;eeks,@ >r. '#orak !escri%e! the process o+ writi g, ?Aell. Write 200 wor!s a !ay a ! i a year, yo" ha#e a o#el.@ This a!#ice le! me to write a page a !ay " til 9 ha! a %ook. 'mitri 6opo# wrote a article i 0ree So+tware >agaDi e title!, ?Creati g a %ook template with Writer,@ which i spire! me to "se 7pe 7++ice.org Writer +or composi g the te3t. $s it t"r e! o"t, it worke! wo !er+"lly. >ark 6olesky per+orme! a e3traor!i ary re#iew a ! test o+ the te3t. Jesse Jecker, TomasD ChrDcDo owicD, >ichael *e#i , Spe ce >i er also teste! a ! re#iewe! portio s o+ the te3t.

$ck owle!gme ts Mare >. Shotts co tri%"te! a lot o+ ho"rs, polishi g my so-calle! E glish %y e!iti g the te3t. $ ! lastly, the rea!ers o+ *i "3Comma !.org, who ha#e se t me so ma y ki ! emails. Their e co"rageme t ga#e me the i!ea that 9 was really o to somethi gL

:our ;eed0ac( Is 7eeded<

This %ook is a o goi g pro4ect, like ma y ope so"rce so+tware pro4ects. 9+ yo" +i ! a tech ical error, !rop me a li e at, %shottsN"sers.so"rce+orge. et 5o"r cha ges a ! s"ggestio s may get i to +"t"re releases.

;urther 1eading

Aere are some Wikipe!ia articles o the +amo"s people me tio e! i this chapter, http,--e .wikipe!ia.org-wiki-*i "3OTor#al!s http,--e .wikipe!ia.org-wiki-:ichar!OStallma The 0ree So+tware 0o" !atio a ! the ;&) 6ro4ect, http,--e .wikipe!ia.org-wiki-0reeOSo+twareO0o" !atio http,--www.+s+.org http,--www.g ".org :ichar! Stallma has writte e3te si#ely o the ?;&)-*i "3@ ami g iss"e, http,--www.g ".org-g "-why-g "-li "3.html http,--www.g ".org-g "-g "-li "3-+a=.htmlPtools

Colophon
This %ook was writte "si g 7pe 7++ice.org Writer i *i%eratio Seri+ a ! Sa s +o ts o a 'ell 9 spiro 2(0&, +actory co +ig"re! with )%" t" 8.01. The 6'0 #ersio o+ the te3t was ge erate! !irectly %y 7pe 7++ice.org Writer. The co#er was pro!"ce! "si g 9 kscape. Co#er !esig %y William E. Shotts, Jr. $"thorIs co#er photograph %y Mare >. Shotts.

6art 2 8 *ear i g The Shell

'art 2 A Learning The She

2 8 What 9s The ShellC

2 What Is The Shell?

Whe we speak o+ the comma ! li e, we are really re+erri g to the shell. The shell is a program that takes key%oar! comma !s a ! passes them to the operati g system to carry o"t. $lmost all *i "3 !istri%"tio s s"pply a shell program +rom the ;&) 6ro4ect calle! bash. The ame ?%ash@ is a acro ym +or ?Jo"r e $gai SAell@, a re+ere ce to the +act bash is a e ha ce! replaceme t +or sh, the origi al ) i3 shell program writte %y Ste#e Jo"r e.

Terminal $mulators
Whe "si g a graphical "ser i ter+ace, we ee! a other program calle! a terminal emulator to i teract with the shell. 9+ we look thro"gh o"r !esktop me "s, we will pro%a%ly +i ! o e. M'E "ses konsole a ! ;&7>E "ses gnome-terminal, tho"gh it<s likely calle! simply ?termi al@ o o"r me ". There are a "m%er o+ other termi al em"lators a#aila%le +or *i "3, %"t they all %asically !o the same thi gH gi#e "s access to the shell. 5o" will pro%a%ly !e#elop a pre+ere ce +or o e or a other %ase! o the "m%er o+ %ells a ! whistles it has.

:our ;irst =e,stro(es

So let<s get starte!. *a" ch the termi al em"latorL 7 ce it comes "p, we sho"l! see somethi g like this,
[me@linuxbox ~]$

This is calle! a shell prompt a ! it will appear whe e#er the shell is rea!y to accept i p"t. While it may #ary i appeara ce somewhat !epe !i g o the !istri%"tio , it will "s"ally i cl"!e yo"r username@machinename, +ollowe! %y the c"rre t worki g !irectory Emore a%o"t that i a little %itF a ! a !ollar sig . 9+ the last character o+ the prompt is a po" ! sig E?P@F rather tha a !ollar sig , the termi al sessio has superuser pri#ileges. This mea s either we are logge! i as the root "ser or we selecte! a termi al em"lator that pro#i!es s"per"ser Ea!mi istrati#eF .0

5o"r 0irst Meystrokes pri#ileges. $ss"mi g that thi gs are goo! so +ar, let<s try some typi g. Type some gi%%erish at the prompt like so,
[me@linuxbox ~]$ kaekfjaeifj

Si ce this comma ! makes o se se, the shell will tell "s so a ! gi#e "s a other cha ce,
bash: kaekfjaeifj: command not found [me@linuxbox ~]$

Command $istory
9+ we press the "p-arrow key, we will see that the pre#io"s comma ! ?kaek+4aei+4@ reappears a+ter the prompt. This is calle! command history. >ost *i "3 !istri%"tio s remem%er the last +i#e h" !re! comma !s %y !e+a"lt. 6ress the !ow -arrow key a ! the pre#io"s comma ! !isappears.

Cursor :o;ement
:ecall the pre#io"s comma ! with the "p-arrow key agai . &ow try the le+t a ! rightarrow keys. See how we ca positio the c"rsor a ywhere o the comma ! li eC This makes e!iti g comma !s easy.

# ;e4 +ords #0out >ice #nd ;ocus

While the shell is all a%o"t the key%oar!, yo" ca also "se a mo"se with yo"r termi al em"lator. There is a mecha ism %"ilt i to the G Wi !ow System Ethe " !erlyi g e gi e that makes the ;)9 goF that s"pports a ="ick copy a ! paste tech i="e. 9+ yo" highlight some te3t %y hol!i g !ow the le+t mo"se %"tto a ! !raggi g the mo"se o#er it Eor !o"%le clicki g o a wor!F, it is copie! i to a %"++er mai tai e! %y G. 6ressi g the mi!!le mo"se %"tto will ca"se the te3t to %e paste! at the c"rsor locatio . Try it. 1ote, 'o <t %e tempte! to "se Ctrl-c a ! Ctrl-v to per+orm copy a ! paste i si!e a termi al wi !ow. They !o <t work. These co trol co!es ha#e !i++ere t mea i gs to the shell a ! were assig e! ma y years %e+ore >icroso+t Wi !ows.

..

2 8 What 9s The ShellC

5o"r graphical !esktop e #iro me t Emost likely M'E or ;&7>EF, i a e++ort to %eha#e like Wi !ows, pro%a%ly has its focus policy set to ?click to +oc"s.@ This mea s +or a wi !ow to get +oc"s E%ecome acti#eF yo" ee! to click o it. This is co trary to the tra!itio al G %eha#ior o+ ?+oc"s +ollows mo"se@ which mea s that a wi !ow gets +oc"s %y 4"st passi g the mo"se o#er it. The wi !ow will ot come to the +oregro" ! " til yo" click o it %"t it will %e a%le to recei#e i p"t. Setti g the +oc"s policy to ?+oc"s +ollows mo"se@ will make the copy a ! paste tech i="e e#e more "se+"l. ;i#e it a try. 9 thi k i+ yo" gi#e it a cha ce yo" will pre+er it. 5o" will +i ! this setti g i the co +ig"ratio program +or yo"r wi !ow ma ager.

Tr, Some Simple Commands

&ow that we ha#e lear e! to type, let<s try a +ew simple comma !s. The +irst o e is date. This comma ! !isplays the c"rre t time a ! !ate.
[me@linuxbox ~]$ date Thu ct !" #$:"#:"% &'T !(()

$ relate! comma ! is cal which, %y !e+a"lt, !isplays a cale !ar o+ the c"rre t mo th.
[me@linuxbox ~]$ cal ctober !(() *u +o Tu ,e Th -r *a # ! $ % " . ) / 0 #( ## #! #$ #% #" #. #) #/ #0 !( !# !! !$ !% !" !. !) !/ !0 $( $#

To see the c"rre t amo" t o+ +ree space o yo"r !isk !ri#es, type df,
[me@linuxbox ~]$ df -iles1stem 6dev6sda! 6dev6sda" 6dev6sda# tm7fs

#2-blocks #"##"%"! "0.$#0(/ #%)).% !"./".

3sed 4vailable 3se5 +ounted on "(#!$0! 00%0)#. $%5 6 !."%"%!% $(((/%$! %)5 6home #)$)( #!!)." #$5 6boot ( !"./". (5 6dev6shm

.2

Try Some Simple Comma !s *ikewise, to !isplay the amo" t o+ +ree memory, type the free comma !.
[me@linuxbox ~]$ free total used +em: "#$)#! "($0). -68 buffers6cache: $)")%/ *9a7: #("!!%/ #(%)#!

free 0)$. #$)0.% 0%)"$.

shared (

buffers "$#!

cached #!!0#.

$nding # Terminal Session

We ca e ! a termi al sessio %y either closi g the termi al em"lator wi !ow, or %y e teri g the exit comma ! at the shell prompt,
[me@linuxbox ~]$ exit

The Console /ehind The Curtain

E#e i+ we ha#e o termi al em"lator r" i g, se#eral termi al sessio s co ti "e to r" %ehi ! the graphical !esktop. Calle! virtual terminals or virtual consoles, these sessio s ca %e accesse! o most *i "3 !istri%"tio s %y pressi g Ctrl4lt--# thro"gh Ctrl-4lt--. o most systems. Whe a sessio is accesse!, it prese ts a logi prompt i to which we ca e ter o"r "ser ame a ! passwor!. To switch +rom o e #irt"al co sole to a other, press 4lt a ! -#--.. To ret"r to the graphical !esktop, press 4lt--).

;urther 1eading

To lear more a%o"t Ste#e Jo"r e, +ather o+ the Jo"r e Shell, see this Wikipe!ia article, http,--e .wikipe!ia.org-wiki-Ste#eOJo"r e Aere is a article a%o"t the co cept o+ shells i comp"ti g, http,--e .wikipe!ia.org-wiki-ShellOEcomp"ti gF

.(

( 8 &a#igatio

3 Navigation
The +irst thi g we ee! to lear to !o E%esi!es 4"st typi gF is how to a#igate the +ile system o o"r *i "3 system. 9 this chapter we will i tro!"ce the +ollowi g comma !s,

79d - 6ri t ame o+ c"rre t worki g !irectory cd - Cha ge !irectory ls - *ist !irectory co te ts

-nderstanding The ;ile S,stem Tree

*ike Wi !ows, a ) i3-like operati g system s"ch as *i "3 orga iDes its +iles i what is calle! a hierarchical directory structure. This mea s that they are orga iDe! i a tree-like patter o+ !irectories Esometimes calle! +ol!ers i other systemsF, which may co tai +iles a ! other !irectories. The +irst !irectory i the +ile system is calle! the root directory. The root !irectory co tai s +iles a ! s"%!irectories, which co tai more +iles a ! s"%!irectories a ! so o a ! so o . &ote that " like Wi !ows, which has a separate +ile system tree +or each storage !e#ice, ) i3-like systems s"ch as *i "3 always ha#e a si gle +ile system tree, regar!less o+ how ma y !ri#es or storage !e#ices are attache! to the comp"ter. Storage !e#ices are attache! Eor more correctly, mountedF at #ario"s poi ts o the tree accor!i g to the whims o+ the system administrator, the perso Eor perso sF respo si%le +or the mai te a ce o+ the system.

The Current +or(ing Director,

>ost o+ "s are pro%a%ly +amiliar with a graphical +ile ma ager which represe ts the +ile system tree as i 0ig"re .. &otice that the tree is "s"ally show "pe !e!, that is, with the root at the top a ! the #ario"s %ra ches !esce !i g %elow. Aowe#er, the comma ! li e has o pict"res, so to a#igate the +ile system tree we ee! to thi k o+ it i a !i++ere t way.

.1

The C"rre t Worki g 'irectory 9magi e that the +ile system is a maDe shape! like a "psi!e-!ow tree a ! we are a%le to

Figure 1: File system tree as shown by a graphical file manager sta ! i the mi!!le o+ it. $t a y gi#e time, we are i si!e a si gle !irectory a ! we ca see the +iles co tai e! i the !irectory a ! the pathway to the !irectory a%o#e "s Ecalle! the parent directoryF a ! a y s"%!irectories %elow "s. The !irectory we are sta !i g i is calle! the current working directory. To !isplay the c"rre t worki g !irectory, we "se the 79d Epri t worki g !irectoryF comma !.
[me@linuxbox ~]$ pwd 6home6me

Whe we +irst log i to o"r system Eor start a termi al em"lator sessio F o"r c"rre t worki g !irectory is set to o"r home directory. Each "ser acco" t is gi#e its ow home !irectory a ! whe operati g as a reg"lar "ser, the home !irectory is the o ly place the "ser is allowe! to write +iles.

Listing The Contents ?" # Director,

To list the +iles a ! !irectories i the c"rre t worki g !irectory, we "se the ls comma !.
[me@linuxbox ~]$ ls 'eskto7 'ocuments +usic

:ictures

:ublic

Tem7lates

;ideos

.2

( 8 &a#igatio $ct"ally, we ca "se the ls comma ! to list the co te ts o+ a y !irectory, ot 4"st the c"rre t worki g !irectory, a ! there are ma y other +" thi gs it ca !o as well. We<ll spe ! more time with ls i the e3t chapter.

Changing The Current +or(ing Director,

To cha ge yo"r worki g !irectory Ewhere we are sta !i g i o"r tree-shape! maDeF we "se the cd comma !. To !o this, type cd +ollowe! %y the pathname o+ the !esire! worki g !irectory. $ path ame is the ro"te we take alo g the %ra ches o+ the tree to get to the !irectory we wa t. 6ath ames ca %e speci+ie! i o e o+ two !i++ere t waysH as absolute pathnames or as relative pathnames. *et<s !eal with a%sol"te path ames +irst.

Abso ute 'athnames

$ a%sol"te path ame %egi s with the root !irectory a ! +ollows the tree %ra ch %y %ra ch " til the path to the !esire! !irectory or +ile is complete!. 0or e3ample, there is a !irectory o yo"r system i which most o+ yo"r system<s programs are i stalle!. The path ame o+ the !irectory is 6usr6bin. This mea s +rom the root !irectory Ereprese te! %y the lea!i g slash i the path ameF there is a !irectory calle! Q"srQ which co tai s a !irectory calle! Q%i Q.
[me@linuxbox ~]$ cd /usr/bin [me@linuxbox bin]$ pwd 6usr6bin [me@linuxbox bin]$ ls

<<<=isting of man1> man1 files <<<

&ow we ca see that we ha#e cha ge! the c"rre t worki g !irectory to 6usr6bin a ! that it is +"ll o+ +iles. &otice how the shell prompt has cha ge!C $s a co #e ie ce, it is "s"ally set "p to a"tomatically !isplay the ame o+ the worki g !irectory.

!e ati;e 'athnames
Where a a%sol"te path ame starts +rom the root !irectory a ! lea!s to its !esti atio , a relati#e path ame starts +rom the worki g !irectory. To !o this, it "ses a co"ple o+ special sym%ols to represe t relati#e positio s i the +ile system tree. These special sym%ols are Q.Q E!otF a ! Q..Q E!ot !otF. The Q.Q sym%ol re+ers to the worki g !irectory a ! the Q..Q sym%ol re+ers to the worki g !irectory<s pare t !irectory. Aere is how it works. *et<s cha ge the worki g !irectory to 6 .K

Cha gi g The C"rre t Worki g 'irectory usr6bin agai ,

[me@linuxbox ~]$ cd /usr/bin [me@linuxbox bin]$ pwd 6usr6bin

7kay, ow let<s say that we wa te! to cha ge the worki g !irectory to the pare t o+ 6usr6bin which is 6usr. We co"l! !o that two !i++ere t ways. Either with a a%sol"te path ame,
[me@linuxbox bin]$ cd /usr [me@linuxbox usr]$ pwd 6usr

7r, with a relati#e path ame,

[me@linuxbox bin]$ cd .. [me@linuxbox usr]$ pwd 6usr

Two !i++ere t metho!s with i!e tical res"lts. Which o e sho"l! we "seC The o e that re="ires the least typi gL *ikewise, we ca cha ge the worki g !irectory +rom 6usr to 6usr6bin i two !i++ere t ways. Either "si g a a%sol"te path ame,
[me@linuxbox usr]$ cd /usr/bin [me@linuxbox bin]$ pwd 6usr6bin

7r, with a relati#e path ame,

[me@linuxbox usr]$ cd ./bin [me@linuxbox bin]$ pwd 6usr6bin

&ow, there is somethi g importa t that 9 m"st poi t o"t here. 9 almost all cases, yo" ca ./

( 8 &a#igatio omit the Q.-Q. 9t is implie!. Typi g,

[me@linuxbox usr]$ cd bin

!oes the same thi g. 9 ge eral, i+ yo" !o ot speci+y a path ame to somethi g, the worki g !irectory will %e ass"me!.

Some $e 5=u Short/uts

9 ta%le (-. we see some "se+"l ways the c"rre t worki g !irectory ca %e ="ickly cha ge!. Table 3- 1: cd Shortcuts Shortcut cd cd cd ~user_name 1esult Cha ges the worki g !irectory to yo"r home !irectory. Cha ges the worki g !irectory to the pre#io"s worki g !irectory. Cha ges the worki g !irectory to the home !irectory o+ user name. 0or e3ample, cd ~bob will cha ge the !irectory to the home !irectory o+ "ser ?%o%.@

Important ;acts #0out ;ilenames

.. 0ile ames that %egi with a perio! character are hi!!e . This o ly mea s that ls will ot list them " less yo" say ls -a. Whe yo"r acco" t was create!, se#eral hi!!e +iles were place! i yo"r home !irectory to co +ig"re thi gs +or yo"r acco" t. *ater o we will take a closer look at some o+ these +iles to see how yo" ca c"stomiDe yo"r e #iro me t. 9 a!!itio , some applicatio s place their co +ig"ratio a ! setti gs +iles i yo"r home !irectory as hi!!e +iles. 2. 0ile ames a ! comma !s i *i "3, like ) i3, are case se siti#e. The +ile ames ?0ile.@ a ! ?+ile.@ re+er to !i++ere t +iles. (. *i "3 has o co cept o+ a ?+ile e3te sio @ like some other operati g systems. 5o" may ame +iles a y way yo" like. The co te ts a !-or p"rpose o+ a +ile is

.8

Cha gi g The C"rre t Worki g 'irectory

!etermi e! %y other mea s. $ltho"gh ) i3-like operati g system !o It "se +ile e3te sio s to !etermi e the co te ts-p"rpose o+ +iles, some applicatio programs !o. 1. Tho"gh *i "3 s"pports lo g +ile ames which may co tai em%e!!e! spaces a ! p" ct"atio characters, limit the p" ct"atio characters i the ames o+ +iles yo" create to perio!, !ash, a ! " !erscore. !ost importantly" do not embed spaces in filenames# 9+ yo" wa t to represe t spaces %etwee wor!s i a +ile ame, "se " !erscore characters. 5o" will tha k yo"rsel+ later.

.9

1 8 E3plori g The System

!"#loring The S$ste%

&ow that we k ow how to mo#e aro" ! the +ile system, it<s time +or a g"i!e! to"r o+ o"r *i "3 system. Je+ore we start howe#er, weIre goi g to lear some more comma !s that will %e "se+"l alo g the way,

ls 8 *ist !irectory co te ts file 8 'etermi e +ile type less 8 Riew +ile co te ts

>ore ;un +ith ls

The ls comma ! is pro%a%ly the most "se! comma !, a ! +or goo! reaso . With it, we ca see !irectory co te ts a ! !etermi e a #ariety o+ importa t +ile a ! !irectory attri%"tes. $s we ha#e see , we ca simply type ls to see a list o+ +iles a ! s"%!irectories co tai e! i the c"rre t worki g !irectory,
[me@linuxbox ~]$ ls 'eskto7 'ocuments +usic

:ictures

:ublic

Tem7lates

;ideos

Jesi!es the c"rre t worki g !irectory, we ca speci+y the !irectory to list, like so,
me@linuxbox ~]$ ls /usr bin games kerberos libexec etc include lib local

sbin share

src tm7

7r e#e speci+y m"ltiple !irectories. 9 this e3ample we will list %oth the "ser<s home !irectory Esym%oliDe! %y the ?S@ characterF a ! the 6usr !irectory,
[me@linuxbox ~]$ ls ~ /usr 6home6me:

20

>ore 0" With ls

'eskto7 'ocuments +usic :ictures :ublic Tem7lates ;ideos

6usr: bin games etc include

kerberos lib

libexec local

sbin share

src tm7

We ca also cha ge the +ormat o+ the o"tp"t to re#eal more !etail,

[me@linuxbox total ". dr9xr9xr-x ! dr9xr9xr-x ! dr9xr9xr-x ! dr9xr9xr-x ! dr9xr9xr-x ! dr9xr9xr-x ! dr9xr9xr-x ! ~]$ ls -l me me me me me me me me me me me me me me %(0. %(0. %(0. %(0. %(0. %(0. %(0. !(()-#(-!. !(()-#(-!. !(()-#(-!. !(()-#(-!. !(()-#(-!. !(()-#(-!. !(()-#(-!. #):!( #):!( #):!( #):!( #):!( #):!( #):!( 'eskto7 'ocuments +usic :ictures :ublic Tem7lates ;ideos

Jy a!!i g ?-l@ to the comma !, we cha ge! the o"tp"t to the lo g +ormat.

<5tions And Arguments

This %ri gs "s to a #ery importa t poi t a%o"t how most comma !s work. Comma !s are o+te +ollowe! %y o e or more options that mo!i+y their %eha#ior, a ! +"rther, %y o e or more arguments, the items "po which the comma ! acts. So most comma !s look ki ! o+ like this,
command -options arguments

>ost comma !s "se optio s co sisti g o+ a si gle character prece!e! %y a !ash, +or e3ample, ?-l@, %"t ma y comma !s, i cl"!i g those +rom the ;&) 6ro4ect, also s"pport long options, co sisti g o+ a wor! prece!e! %y two !ashes. $lso, ma y comma !s allow m"ltiple short optio s to %e str" g together. 9 this e3ample, the ls comma ! is gi#e two optio s, the ?l@ optio to pro!"ce lo g +ormat o"tp"t, a ! the ?t@ optio to sort the res"lt %y the +ile<s mo!i+icatio time.
[me@linuxbox ~]$ ls -lt

2.

1 8 E3plori g The System We<ll a!! the lo g optio ?--re#erse@ to re#erse the or!er o+ the sort,
[me@linuxbox ~]$ ls -lt --reverse

The ls comma ! has a large "m%er o+ possi%le optio s. The most commo are liste! i the Ta%le 1-.. Table $- 1: %ommon ls &ptions ?ption -a Long ?ption --all Description *ist all +iles, e#e those with ames that %egi with a perio!, which are ormally ot liste! Ei.e., hi!!e F. 7r!i arily, i+ a !irectory is speci+ie!, ls will list the co te ts o+ the !irectory, ot the !irectory itsel+. )se this optio i co 4" ctio with the -l optio to see !etails a%o"t the !irectory rather tha its co te ts. This optio will appe ! a i !icator character to the e ! o+ each liste! ame. 0or e3ample, a ?-@ i+ the ame is a !irectory. 9 lo g +ormat listi gs, !isplay +ile siDes i h"ma rea!a%le +ormat rather tha i %ytes. 'isplay res"lts i lo g +ormat. --reverse 'isplay the res"lts i re#erse or!er. &ormally, ls !isplays its res"lts i asce !i g alpha%etical or!er. Sort res"lts %y +ile siDe. Sort %y mo!i+icatio time.

-d

--director1

--

--classif1

-h -l -r

--human-readable

-* -t

A Longer Look At Long 2ormat

$s we saw %e+ore, the ?-l@ optio ca"ses ls to !isplay its res"lts i lo g +ormat. This +ormat co tai s a great !eal o+ "se+"l i +ormatio . Aere is the &xam7les !irectory +rom a )%" t" system,

22

>ore 0" With ls

-r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r--r9-r--r-# # # # # # # # # # # root root root root root root root root root root root root $").!0. !(()-(%-($ ##:(" &x7erience ubuntu<ogg root ##/.!#0 !(()-(%-($ ##:(" kubuntu-leaflet<7ng root %)"/% !(()-(%-($ ##:(" logo-&dubuntu<7ng root %%$"" !(()-(%-($ ##:(" logo-2ubuntu<7ng root $%$0# !(()-(%-($ ##:(" logo-3buntu<7ng root $!("0 !(()-(%-($ ##:(" oo-cd-cover<odf root #"0)%% !(()-(%-($ ##:(" oo-derivatives<doc root !)/$) !(()-(%-($ ##:(" oo-max9ell<odt root 0//#. !(()-(%-($ ##:(" oo-trig<xls root %"$).% !(()-(%-($ ##:(" oo-9elcome<odt root $"/$)% !(()-(%-($ ##:(" ubuntu *ax<ogg

*et<s look at the !i++ere t +iel!s +rom o e o+ the +iles a ! e3ami e their mea i gs, Table $-': ls (ong (isting Fields ;ield -r9-r--r->eaning $ccess rights to the +ile. The +irst character i !icates the type o+ +ile. $mo g the !i++ere t types, a lea!i g !ash mea s a reg"lar +ile, while a ?!@ i !icates a !irectory. The e3t three characters are the access rights +or the +ile<s ow er, the e3t three are +or mem%ers o+ the +ile<s gro"p, a ! the +i al three are +or e#eryo e else. The +"ll mea i g o+ this is !isc"sse! i Chapter .0 8 6ermissio s. 0ile<s "m%er o+ har! li ks. See the !isc"ssio o+ li ks later i this chapter. The "ser ame o+ the +ile<s ow er. The ame o+ the gro"p which ow s the +ile. SiDe o+ the +ile i %ytes. 'ate a ! time o+ the +ile<s last mo!i+icatio . &ame o+ the +ile.

# root root $!("0 !(()-(%-($ ##:(" oo-cd-cover<odf

Determining # ;ile2s T,pe +ith file

$s we e3plore the system it will %e "se+"l to k ow what +iles co tai . To !o this we will "se the file comma ! to !etermi e a +ile<s type. $s we !isc"sse! earlier, +ile ames i *i "3 are ot re="ire! to re+lect a +ile<s co te ts. While a +ile ame like ?pict"re.4pg@ wo"l! ormally %e e3pecte! to co tai a J6E; compresse! image, it is ot re="ire! to i *i "3. We ca i #oke the file comma ! this way, 2(

1 8 E3plori g The System

file filename

Whe i #oke!, the file comma ! will pri t a %rie+ !escriptio o+ the +ile<s co te ts. 0or e3ample,
[me@linuxbox ~]$ file picture.jpg 7icture<j7g: ?:&@ image data> ?-A- standard #<(#

There are ma y ki !s o+ +iles. 9 +act, o e o+ the commo i!eas i ) i3-like operati g systems s"ch as *i "3 is that ?e#erythi g is a +ile.@ $s we procee! with o"r lesso s, we will see 4"st how tr"e that stateme t is. While ma y o+ the +iles o yo"r system are +amiliar, +or e3ample >6( a ! J6E;, there are ma y ki !s that are a little less o%#io"s a ! a +ew that are ="ite stra ge.

Vie4ing ;ile Contents +ith less

The less comma ! is a program to #iew te3t +iles. Thro"gho"t o"r *i "3 system, there are ma y +iles that co tai h"ma -rea!a%le te3t. The less program pro#i!es a co #e ie t way to e3ami e them.

+hat Is 5Text9.
There are ma y ways to represe t i +ormatio o a comp"ter. $ll metho!s i #ol#e !e+i i g a relatio ship %etwee the i +ormatio a ! some "m%ers that will %e "se! to represe t it. Comp"ters, a+ter all, o ly " !ersta ! "m%ers a ! all !ata is co #erte! to "meric represe tatio . Some o+ these represe tatio systems are #ery comple3 Es"ch as compresse! #i!eo +ilesF, while others are rather simple. 7 e o+ the earliest a ! simplest is calle! )S%** te+t. $SC99 Epro o" ce! Q$s-MeyQF is short +or $merica Sta !ar! Co!e +or 9 +ormatio 9 tercha ge. This is a simple e co!i g scheme that was +irst "se! o Teletype machi es to map key%oar! characters to "m%ers. Te3t is a simple o e-to-o e mappi g o+ characters to "m%ers. 9t is #ery compact. 0i+ty characters o+ te3t tra slates to +i+ty %ytes o+ !ata. 9t is importa t to " !ersta ! that te3t o ly co tai s a simple mappi g o+ characters to "m%ers. 9t is ot the same as a wor! processor !oc"me t s"ch as o e create! %y >icroso+t Wor! or 7pe 7++ice.org Writer. Those +iles, i co trast to simple $SC99 te3t,

21

Riewi g 0ile Co te ts With less

co tai ma y o -te3t eleme ts that are "se! to !escri%e its str"ct"re a ! +ormatti g. 6lai $SC99 te3t +iles co tai o ly the characters themsel#es a ! a +ew r"!ime tary co trol co!es like ta%s, carriage ret"r s a ! li e +ee!s. Thro"gho"t a *i "3 system, ma y +iles are store! i te3t +ormat a ! there are ma y *i "3 tools that work with te3t +iles. E#e Wi !ows recog iDes the importa ce o+ this +ormat. The well-k ow &7TE6$'.EGE program is a e!itor +or plai $SC99 te3t +iles. Why wo"l! we wa t to e3ami e te3t +ilesC Jeca"se ma y o+ the +iles that co tai system setti gs Ecalle! configuration filesF are store! i this +ormat, a ! %ei g a%le to rea! them gi#es "s i sight a%o"t how the system works. 9 a!!itio , ma y o+ the act"al programs that the system "ses Ecalle! scriptsF are store! i this +ormat. 9 later chapters, we will lear how to e!it te3t +iles i or!er to mo!i+y systems setti gs a ! write o"r ow scripts, %"t +or ow we will 4"st look at their co te ts. The less comma ! is "se! like this,
less filename

7 ce starte!, the less program allows yo" to scroll +orwar! a ! %ackwar! thro"gh a te3t +ile. 0or e3ample, to e3ami e the +ile that !e+i es all the system<s "ser acco" ts, e ter the +ollowi g comma !,
[me@linuxbox ~]$ less /etc/passwd

7 ce the less program starts, we may #iew the co te ts o+ the +ile. 9+ the +ile is lo ger tha o e page, we ca scroll "p a ! !ow . To e3it less, press the ?=@ key. The ta%le %elow lists the most commo key%oar! comma !s "se! %y less. Table $-3: less %ommands Command 6age )p or b 6age 'ow or space )p $rrow #ction Scroll %ack o e page Scroll +orwar! o e page Scroll "p o e li e

22

1 8 E3plori g The System 'ow $rrow @ #@ or g 6characters n h B Scroll !ow o e li e >o#e to the e ! o+ the te3t +ile >o#e to the %egi i g o+ the te3t +ile Search +orwar! to the e3t occ"rre ce o+ characters Search +or the e3t occ"rre ce o+ the pre#io"s search 'isplay help scree T"it less

Less Is >ore
The less program was !esig e! as a impro#e! replaceme t o+ a earlier ) i3 program calle! more. The ame ?less@ is a play o the phrase ?less is more@Ba motto o+ mo!er ist architects a ! !esig ers. less +alls i to the class o+ programs calle! ?pagers,@ programs that allow the easy #iewi g o+ lo g te3t !oc"me ts i a page %y page ma er. Whereas the more program co"l! o ly page +orwar!, the less program allows pagi g %oth +orwar! a ! %ackwar! a ! has ma y other +eat"res as well.

# 6uided Tour
The +ile system layo"t o yo"r *i "3 system is m"ch like that +o" ! o other ) i3-like systems. The !esig is act"ally speci+ie! i a p"%lishe! sta !ar! calle! the (inu+ Filesystem ,ierarchy Standard. &ot all *i "3 !istri%"tio s co +orm to the sta !ar! e3actly %"t most come pretty close. &e3t, we are goi g to wa !er aro" ! the +ile system o"rsel#es to see what makes o"r *i "3 system tick. This will gi#e yo" a cha ce to practice yo"r a#igatio skills. 7 e o+ the thi gs we will !isco#er is that ma y o+ the i teresti g +iles are i plai h"ma rea!a%le te3t. $s we go a%o"t o"r to"r, try the +ollowi g, .. cd i to a gi#e !irectory 2. *ist the !irectory co te ts with ls -l (. 9+ yo" see a i teresti g +ile, !etermi e its co te ts with file 1. 9+ it looks like it might %e te3t, try #iewi g it with less 2K

$ ;"i!e! To"r Remember the co(y and (aste trick2 9+ yo" are "si g a mo"se, yo" ca !o"%le click o a +ile ame to copy it a ! mi!!le click to paste it i to comma !s. $s we wa !er aro" !, !o <t %e a+rai! to look at st"++. :eg"lar "sers are largely prohi%ite! +rom messi g thi gs "p. That<s the system a!mi istrators 4o%L 9+ a comma ! complai s a%o"t somethi g, 4"st mo#e o to somethi g else. Spe ! some time looki g aro" !. The system is o"rs to e3plore. :emem%er, i *i "3, there are o secretsL Ta%le 1-1 lists 4"st a +ew o+ the !irectories we ca e3plore. 0eel +ree to try moreL Table $-$: -irectories Found &n (inu+ Systems Director, 6 6bin 6boot Comments The root !irectory. Where e#erythi g %egi s. Co tai s %i aries EprogramsF that m"st %e prese t +or the system to %oot a ! r" . Co tai s the *i "3 ker el, i itial :$> !isk image E+or !ri#ers ee!e! at %oot timeF, a ! the %oot loa!er. 9 teresti g +iles, 6boot6grub6grub<conf or menu<lst, which are "se! to co +ig"re the %oot loa!er. 6boot6vmlinuC, the li "3 ker el 6dev This is a special !irectory which co tai s device nodes. ?E#erythi g is a +ile@ also applies to !e#ices. Aere is where the ker el mai tai s a list o+ all the !e#ices it " !ersta !s. The 6etc !irectory co tai s all o+ the system-wi!e co +ig"ratio +iles. 9t also co tai s a collectio o+ shell scripts which start each o+ the system ser#ices at %oot time. E#erythi g i this !irectory sho"l! %e rea!a%le te3t. 9 teresti g +iles, While e#erythi g i 6etc is i teresti g, here are some o+ my all-time +a#orites, 6etc6crontab, a +ile that !e+i es whe a"tomate! 4o%s will r" . 6etc6fstab, a ta%le o+ storage !e#ices a ! their associate! mo" t poi ts. 6etc67ass9d, a list o+ the "ser acco" ts.

6etc

2/

1 8 E3plori g The System Director, 6home Comments 9 ormal co +ig"ratio s, each "ser is gi#e a !irectory i 6home. 7r!i ary "sers ca o ly write +iles i their home !irectories. This limitatio protects the system +rom erra t "ser acti#ity. Co tai s share! li%rary +iles "se! %y the core system programs. These are similar to '**s i Wi !ows. Each +ormatte! partitio or !e#ice "si g a *i "3 +ile system, s"ch as e3t(, will ha#e this !irectory. 9t is "se! i the case o+ a partial reco#ery +rom a +ile system corr"ptio e#e t. ) less somethi g really %a! has happe e! to yo"r system, this !irectory will remai empty. 7 mo!er *i "3 systems the 6media !irectory will co tai the mo" t poi ts +or remo#a%le me!ia s"ch )SJ !ri#es, C'-:7>s, etc. that are mo" te! a"tomatically at i sertio . 7 ol!er *i "3 systems, the 6mnt !irectory co tai s mo" t poi ts +or remo#a%le !e#ices that ha#e %ee mo" te! ma "ally. The 6o7t !irectory is "se! to i stall ?optio al@ so+tware. This is mai ly "se! to hol! commercial so+tware pro!"cts that may %e i stalle! o yo"r system. The 67roc !irectory is special. 9t<s ot a real +ile system i the se se o+ +iles store! o yo"r har! !ri#e. :ather, it is a #irt"al +ile system mai tai e! %y the *i "3 ker el. The ?+iles@ it co tai s are peepholes i to the ker el itsel+. The +iles are rea!a%le a ! will gi#e yo" a pict"re o+ how the ker el sees yo"r comp"ter. This is the home !irectory +or the root acco" t. This !irectory co tai s ?system@ %i aries. These are programs that per+orm #ital system tasks that are ge erally reser#e! +or the s"per"ser. The 6tm7 !irectory is i te !e! +or storage o+ temporary, tra sie t +iles create! %y #ario"s programs. Some co +ig"ratio s ca"se this !irectory to %e emptie! each time the system is re%oote!.

6lib 6lost8found

6media

6mnt

6o7t

67roc

6root 6sbin

6tm7

28

$ ;"i!e! To"r Director, 6usr Comments The 6usr !irectory tree is likely the largest o e o a *i "3 system. 9t co tai s all the programs a ! s"pport +iles "se! %y reg"lar "sers. 6usr6bin co tai s the e3ec"ta%le programs i stalle! %y yo"r *i "3 !istri%"tio . 9t is ot " commo +or this !irectory to hol! tho"sa !s o+ programs. The share! li%raries +or the programs i 6usr6bin. The 6usr6local tree is where programs that are ot i cl"!e! with yo"r !istri%"tio %"t are i te !e! +or systemwi!e "se are i stalle!. 6rograms compile! +rom so"rce co!e are ormally i stalle! i 6usr6local6bin. 7 a ewly i stalle! *i "3 system, this tree e3ists, %"t it will %e empty " til the system a!mi istrator p"ts somethi g i it. Co tai s more system a!mi istratio programs. 6usr6share co tai s all the share! !ata "se! %y programs i 6usr6bin. This i cl"!es thi gs like !e+a"lt co +ig"ratio +iles, ico s, scree %ackgro" !s, so" ! +iles, etc. >ost packages i stalle! o the system will i cl"!e some ki ! o+ !oc"me tatio . 9 6usr6share6doc, we will +i ! !oc"me tatio +iles orga iDe! %y package. With the e3ceptio o+ 6tm7 a ! 6home, the !irectories we ha#e looke! at so +ar remai relati#ely static, that is, their co te ts !o <t cha ge. The 6var !irectory tree is where !ata that is likely to cha ge is store!. Rario"s !ata%ases, spool +iles, "ser mail, etc. are locate! here. 6var6log co tai s log files, recor!s o+ #ario"s system acti#ity. These are #ery importa t a ! sho"l! %e mo itore! +rom time to time. The most "se+"l o e is 6var6log6messages. &ote that +or sec"rity reaso s o some systems, yo" m"st %e the s"per"ser to #iew log +iles .

6usr6bin

6usr6lib 6usr6local

6usr6sbin 6usr6share

6usr6share6doc

6var

6var6log

S,m0olic Lin(s
$s we look aro" !, we are likely to see a !irectory listi g with a e try like this,

29

1 8 E3plori g The System

lr9xr9xr9x # root root ## !(()-(/-## ():$% libc<so<. -D libc-!<.<so

&otice how the +irst letter o+ the listi g is ?l@ a ! the e try seems to ha#e two +ile amesC This is a special ki ! o+ a +ile calle! a symbolic link Ealso k ow as a soft link or symlink.F 9 most ) i3-like systems it is possi%le to ha#e a +ile re+ere ce! %y m"ltiple ames. While the #al"e o+ this may ot %e o%#io"s, it is really a "se+"l +eat"re. 6ict"re this sce ario, a program re="ires the "se o+ a share! reso"rce o+ some ki ! co tai e! i a +ile ame! ?+oo,@ %"t ?+oo@ has +re="e t #ersio cha ges. 9t wo"l! %e goo! to i cl"!e the #ersio "m%er i the +ile ame so the a!mi istrator or other i tereste! party co"l! see what #ersio o+ ?+oo@ is i stalle!. This prese ts a pro%lem. 9+ we cha ge the ame o+ the share! reso"rce, we ha#e to track !ow e#ery program that might "se it a ! cha ge it to look +or a ew reso"rce ame e#ery time a ew #ersio o+ the reso"rce is i stalle!. That !oes <t so" ! like +" at all. Aere is where sym%olic li ks sa#e the !ay. *et<s say we i stall #ersio 2.K o+ ?+oo,@ which has the +ile ame ?+oo-2.K@ a ! the create a sym%olic li k simply calle! ?+oo@ that poi ts to ?+oo-2.K.@ This mea s that whe a program ope s the +ile ?+oo@, it is act"ally ope i g the +ile ?+oo-2.K@. &ow e#ery%o!y is happy. The programs that rely o ?+oo@ ca +i ! it a ! we ca still see what act"al #ersio is i stalle!. Whe it is time to "pgra!e to ?+oo-2./,@ we 4"st a!! the +ile to o"r system, !elete the sym%olic li k ?+oo@ a ! create a ew o e that poi ts to the ew #ersio . &ot o ly !oes this sol#e the pro%lem o+ the #ersio "pgra!e, %"t it also allows "s to keep %oth #ersio s o o"r machi e. 9magi e that ?+oo-2./@ has a %"g E!am those !e#elopersLF a ! we ee! to re#ert to the ol! #ersio . $gai , we 4"st !elete the sym%olic li k poi ti g to the ew #ersio a ! create a ew sym%olic li k poi ti g to the ol! #ersio . The !irectory listi g a%o#e E+rom the 6lib !irectory o+ a 0e!ora systemF shows a sym%olic li k calle! ?li%c.so.K@ that poi ts to a share! li%rary +ile calle! ?li%c-2.K.so.@ This mea s that programs looki g +or ?li%c.so.K@ will act"ally get the +ile ?li%c-2.K.so.@ We will lear how to create sym%olic li ks i the e3t chapter.

3ard Lin(s
While we are o the s"%4ect o+ li ks, we ee! to me tio that there is a seco ! type o+ li k calle! a hard link. Aar! li ks also allow +iles to ha#e m"ltiple ames, %"t they !o it i a !i++ere t way. WeIll talk more a%o"t the !i++ere ces %etwee sym%olic a ! har! li ks i the e3t chapter.

;urther 1eading

The +"ll #ersio o+ the (inu+ Filesystem ,ierarchy Standard ca %e +o" ! here, http,--www.path ame.com-+hs-

(0

2 8 >a ip"lati g 0iles $ ! 'irectories

& 'ani#ulating (iles )nd *irectories

$t this poi t, we are rea!y +or some real workL This chapter will i tro!"ce the +ollowi g comma !s,

c7 8 Copy +iles a ! !irectories mv 8 >o#e-re ame +iles a ! !irectories mkdir 8 Create !irectories rm 8 :emo#e +iles a ! !irectories ln 8 Create har! a ! sym%olic li ks

These +i#e comma !s are amo g the most +re="e tly "se! *i "3 comma !s. They are "se! +or ma ip"lati g %oth +iles a ! !irectories. &ow, to %e +ra k, some o+ the tasks per+orme! %y these comma !s are more easily !o e with a graphical +ile ma ager. With a +ile ma ager, we ca !rag a ! !rop a +ile +rom o e !irectory to a other, c"t a ! paste +iles, !elete +iles, etc. So why "se these ol! comma ! li e programsC The a swer is power a ! +le3i%ility. While it is easy to per+orm simple +ile ma ip"latio s with a graphical +ile ma ager, complicate! tasks ca %e easier with the comma ! li e programs. 0or e3ample, how co"l! we copy all the AT>* +iles +rom o e !irectory to a other, %"t o ly copy +iles that !o ot e3ist i the !esti atio !irectory or are ewer tha the #ersio s i the !esti atio !irectoryC 6retty har! with with a +ile ma ager. 6retty easy with the comma ! li e,
cp -u *.html destination

+ildcards
Je+ore we %egi "si g o"r comma !s, we ee! to talk a%o"t a shell +eat"re that makes these comma !s so power+"l. Si ce the shell "ses +ile ames so m"ch, it pro#i!es special characters to help yo" rapi!ly speci+y gro"ps o+ +ile ames. These special characters are (.

2 8 >a ip"lati g 0iles $ ! 'irectories calle! wildcards. )si g wil!car!s Ewhich is also k ow as globbingF allow yo" to select +ile ames %ase! o patter s o+ characters. The ta%le %elow lists the wil!car!s a ! what they select, Table .-1: /ildcards +ildcard E F [characters] [Gcharacters] [[:class:]] >eaning >atches a y characters >atches a y si gle character >atches a y character that is a mem%er o+ the set characters >atches a y character that is ot a mem%er o+ the set characters >atches a y character that is a mem%er o+ the speci+ie! class

Ta%le 2-2 lists the most commo ly "se! character classes, Table .-': %ommonly 0sed %haracter %lasses Character Class [:alnum:] [:al7ha:] [:digit:] [:lo9er:] [:u77er:] >eaning >atches a y alpha "meric character >atches a y alpha%etic character >atches a y "meral >atches a y lowercase letter >atches a y "ppercase letter

)si g wil!car!s makes it possi%le to co str"ct #ery sophisticate! selectio criteria +or +ile ames. Aere are some e3amples o+ patter s a ! what they match, Table .-3: /ildcard 1+amples Pattern E gE bE<txt >atches $ll +iles $ y +ile %egi i g with ?g@ $ y +ile %egi i g with ?%@ +ollowe! %y a y characters a ! e !i g with ?.t3t@

(2

Wil!car!s 'ataFFF [abc]E H4C23:<[(-0][(-0][(-0] [[:u77er:]]E [G[:digit:]]E E[[:lo9er:]#!$] $ y +ile %egi i g with ?'ata@ +ollowe! %y e3actly three characters $ y +ile %egi ?%@, or a ?c@ i g with either a ?a@, a

$ y +ile %egi i g with ?J$CM)6.@ +ollowe! %y e3actly three "merals $ y +ile %egi i g with a "ppercase letter $ y +ile ot %egi i g with a "meral $ y +ile e !i g with a lowercase letter or the "merals ?.@, ?2@, or ?(@

Wil!car!s ca %e "se! with a y comma ! that accepts +ile ames as arg"me ts, %"t weIll talk more a%o"t that i Chapter 8.

Character 1anges
9+ yo" are comi g +rom a other ) i3-like e #iro me t or ha#e %ee rea!i g some other %ooks o this s"%4ect, yo" may ha#e e co" tere! the [4-I] or the [a-C] character ra ge otatio s. These are tra!itio al ) i3 otatio s a ! worke! i ol!er #ersio s o+ *i "3 as well. They ca still work, %"t yo" ha#e to %e #ery care+"l with them %eca"se they will ot pro!"ce the e3pecte! res"lts " less properly co +ig"re!. 0or ow, yo" sho"l! a#oi! "si g them a ! "se character classes i stea!.

+ildcards +or( In The 6-I Too

Wil!car!s are especially #al"a%le ot o ly %eca"se they are "se! so +re="e tly o the comma ! li e, %"t are also s"pporte! %y some graphical +ile ma agers.

9 1autilus Ethe +ile ma ager +or ;&7>EF, yo" ca select +iles "si g the E!it-Select 6atter me " item. J"st e ter a +ile selectio patter with wil!car!s a ! the +iles i the c"rre tly #iewe! !irectory will %e highlighte! +or selectio . 9 /ol(hin a ! 3on4ueror Ethe +ile ma agers +or M'EF, yo" ca e ter wil!car!s !irectly o the locatio %ar. 0or e3ample, i+ yo" wa t to see all the +iles starti g with a lowercase ?"@ i the -"sr-%i !irectory, type ?-"sr-%i -"U@ i to the locatio %ar a ! it will !isplay the res"lt.

((

2 8 >a ip"lati g 0iles $ ! 'irectories

>a y i!eas origi ally +o" ! i the comma ! li e i ter+ace make their way i to the graphical i ter+ace, too. 9t is o e o+ the ma y thi gs that make the *i "3 !esktop so power+"l.

mkdir Create Directories

The mkdir comma ! is "se! to create !irectories. 9t works like this,
mkdir directory...

# note on notation5 Whe three perio!s +ollow a arg"me t i the !escriptio o+ a comma ! Eas a%o#eF, it mea s that the arg"me t ca %e repeate!, th"s,
mkdir dir1

wo"l! create a si gle !irectory ame! ?!ir.@, while

mkdir dir1 dir2 dir

wo"l! create three !irectories ame! ?!ir.@, ?!ir2@, a ! ?!ir(@.

cp Cop, ;iles #nd Directories

The c7 comma ! copies +iles or !irectories. 9t ca %e "se! two !i++ere t ways,
cp item1 item2

to copy the si gle +ile or !irectory ?item.@ to +ile or !irectory ?item2@ a !,

cp item... directory

to copy m"ltiple items Eeither +iles or !irectoriesF i to a !irectory. (1

cp 8 Copy 0iles $ ! 'irectories

Use=u <5tions And 6xam5 es

Aere are some o+ the commo ly "se! optio s Ethe short optio a ! the e="i#ale t lo g optio F +or c7, Table .-$: cp &ptions ?ption -a> --archive >eaning Copy the +iles a ! !irectories a ! all o+ their attri%"tes, i cl"!i g ow erships a ! permissio s. &ormally, copies take o the !e+a"lt attri%"tes o+ the "ser per+ormi g the copy. Je+ore o#erwriti g a e3isti g +ile, prompt the "ser +or co +irmatio . )" this o(tion is not s(eci"ied6 cp +ill silently over+rite "iles. :ec"rsi#ely copy !irectories a ! their co te ts. This optio Eor the -a optio F is re="ire! whe copyi g !irectories. Whe copyi g +iles +rom o e !irectory to a other, o ly copy +iles that either !o <t e3ist, or are ewer tha the e3isti g correspo !i g +iles, i the !esti atio !irectory. 'isplay i +ormati#e messages as the copy is per+orme!.

-i> --interactive

-r> --recursive

-u> --u7date

-v> --verbose

Table .-.: cp 1+amples Command c7 file1 file2 1esults Copy file1 to file'. )" file2 exists6 it is over+ritten +ith the contents o" file1. 9+ file' !oes ot e3ist, it is create!. Same as a%o#e, e3cept that i+ file' e3ists, the "ser is prompte! %e+ore it is o#erwritte . Copy file1 a ! file' i to !irectory dir1. dir1 m"st alrea!y e3ist. )si g a wil!car!, all the +iles i dir1 are copie! i to dir'. dir' m"st alrea!y e3ist.

c7 -i file1 file2 c7 file1 file2 dir1 c7 dir#6E dir!

(2

2 8 >a ip"lati g 0iles $ ! 'irectories c7 -r dir1 dir2 Copy the co te ts o+ !irectory dir1 to !irectory dir'. 9+ !irectory dir' !oes ot e3ist, it is create! a !, a+ter the copy, will co tai the same co te ts as !irectory dir1. 9+ !irectory dir' !oes e3ist, the !irectory dir1 Ea ! its co te tsF will %e copie! i to dir'.

mv >o%e #nd 1ename ;iles

The mv comma ! per+orms %oth +ile mo#i g a ! +ile re ami g, !epe !i g o how it is "se!. 9 either case, the origi al +ile ame o lo ger e3ists a+ter the operatio . mv is "se! i m"ch the same way as c7,
mv item1 item2

to mo#e or re ame +ile or !irectory ?item.@ to ?item2@ or,

mv item... directory

to mo#e o e or more items +rom o e !irectory to a other.

Use=u <5tions And 6xam5 es

mv shares ma y o+ the same optio s as c7, Table .-2: mv &ptions ?ption -i> --interactive >eaning Je+ore o#erwriti g a e3isti g +ile, prompt the "ser +or co +irmatio . )" this o(tion is not s(eci"ied6 mv +ill silently over+rite "iles. Whe mo#i g +iles +rom o e !irectory to a other, o ly mo#e +iles that either !o <t e3ist, or are ewer tha the e3isti g correspo !i g +iles i the !esti atio !irectory. 'isplay i +ormati#e messages as the mo#e is

-u> --u7date

-v> --verbose

(K

m# 8 >o#e $ ! :e ame 0iles per+orme!.

Table .-3: mv 1+amples Command mv file1 file2 1esults >o#e file1 to file'. )" file2 exists6 it is over+ritten +ith the contents o" file1. 9+ file' !oes ot e3ist, it is create!. )n either case6 file1 ceases to exist. Same as a%o#e, e3cept that i+ file' e3ists, the "ser is prompte! %e+ore it is o#erwritte . >o#e file1 a ! file' i to !irectory dir1. dir1 m"st alrea!y e3ist. 9+ !irectory dir' !oes ot e3ist, create !irectory dir' a ! mo#e the co te ts o+ !irectory dir1 i to dir' a ! !elete !irectory dir1. 9+ !irectory dir' !oes e3ist, mo#e !irectory dir1 Ea ! its co te tsF i to !irectory dir'.

mv -i file1 file2 mv file1 file2 dir1 mv dir1 dir2

rm 1emo%e ;iles #nd Directories

The rm comma ! is "se! to remo#e E!eleteF +iles a ! !irectories,
rm item...

where ?item@ is o e or more +iles or !irectories.

Use=u <5tions And 6xam5 es

Aere are some o+ the commo optio s +or rm, Table .-4: rm &ptions ?ption -i> --interactive >eaning Je+ore !eleti g a e3isti g +ile, prompt the "ser +or co +irmatio . )" this o(tion is not s(eci"ied6 rm +ill silently delete "iles. (/

2 8 >a ip"lati g 0iles $ ! 'irectories -r> --recursive :ec"rsi#ely !elete !irectories. This mea s that i+ a !irectory %ei g !elete! has s"%!irectories, !elete them too. To !elete a !irectory, this optio m"st %e speci+ie!. 9g ore o e3iste t +iles a ! !o ot prompt. This o#erri!es the --interactive optio . 'isplay i +ormati#e messages as the !eletio is per+orme!.

-f> --force -v> --verbose

Table .-5: rm 1+amples Command rm file1 rm -i file1 rm -r file1 dir1 rm -rf file1 dir1 1esults 'elete file1 sile tly. Same as a%o#e, e3cept that the "ser is prompte! +or co +irmatio %e+ore the !eletio is per+orme!. 'elete file1 a ! dir1 a ! its co te ts. Same as a%o#e, e3cept that i+ either file1 or dir1 !o ot e3ist, rm will co ti "e sile tly.

/e Care"ul +ith rm<

) i3-like operati g systems s"ch as *i "3 !o ot ha#e a " !elete comma !. 7 ce yo" !elete somethi g with rm, it<s go e. *i "3 ass"mes yo"<re smart a ! yo" k ow what yo"<re !oi g. Je partic"larly care+"l with wil!car!s. Co si!er this classic e3ample. *et<s say yo" wa t to !elete 4"st the AT>* +iles i a !irectory. To !o this, yo" type, rm E<html which is correct, %"t i+ yo" acci!e tally place a space %etwee the ?U@ a ! the ?.html@ like so, rm E <html the rm comma ! will !elete all the +iles i the !irectory a ! the complai that there is o +ile calle! ?.html@.

(8

rm 8 :emo#e 0iles $ ! 'irectories

Here is a use"ul ti(. Whe e#er yo" "se wil!car!s with rm E%esi!es care+"lly checki g yo"r typi gLF, test the wil!car! +irst with ls. This will let yo" see the +iles that will %e !elete!. The press the "p arrow key to recall the comma ! a ! replace the ls with rm.

ln Create Lin(s
The ln comma ! is "se! to create either har! or sym%olic li ks. 9t is "se! i o e o+ two ways,
ln file link

to create a har! li k, a !,
ln -s item link

to create a sym%olic li k where ?item@ is either a +ile or a !irectory.

$ard Links
Aar! li ks are the origi al ) i3 way o+ creati g li ks, compare! to sym%olic li ks, which are more mo!er . Jy !e+a"lt, e#ery +ile has a si gle har! li k that gi#es the +ile its ame. Whe we create a har! li k, we create a a!!itio al !irectory e try +or a +ile. Aar! li ks ha#e two importa t limitatio s, .. $ har! li k ca ot re+ere ce a +ile o"tsi!e its ow +ile system. This mea s a li k may ot re+ere ce a +ile that is ot o the same !isk partitio as the li k itsel+. 2. $ har! li k may ot re+ere ce a !irectory. $ har! li k is i !isti g"isha%le +rom the +ile itsel+. ) like a sym%olic li k, whe yo" list a !irectory co tai i g a har! li k yo" will see o special i !icatio o+ the li k. Whe a har! li k is !elete!, the li k is remo#e! %"t the co te ts o+ the +ile itsel+ co ti "e to e3ist Ethat is, its space is ot !eallocate!F " til all li ks to the +ile are !elete!. 9t is importa t to %e aware o+ har! li ks %eca"se yo" might e co" ter them +rom time to time, %"t mo!er practice pre+ers sym%olic li ks, which we will co#er e3t.

(9

2 8 >a ip"lati g 0iles $ ! 'irectories

Symbo i/ Links
Sym%olic li ks were create! to o#ercome the limitatio s o+ har! li ks. Sym%olic li ks work %y creati g a special type o+ +ile that co tai s a te3t poi ter to the re+ere ce! +ile or !irectory. 9 this regar!, they operate i m"ch the same way as a Wi !ows shortc"t tho"gh o+ co"rse, they pre!ate the Wi !ows +eat"re %y ma y years H-F $ +ile poi te! to %y a sym%olic li k, a ! the sym%olic li k itsel+ are largely i !isti g"isha%le +rom o e a other. 0or e3ample, i+ yo" write some somethi g to the sym%olic li k, the re+ere ce! +ile is also writte to. Aowe#er whe yo" !elete a sym%olic li k, o ly the li k is !elete!, ot the +ile itsel+. 9+ the +ile is !elete! %e+ore the sym%olic li k, the li k will co ti "e to e3ist, %"t will poi t to othi g. 9 this case, the li k is sai! to %e broken. 9 ma y impleme tatio s, the ls comma ! will !isplay %roke li ks i a !isti g"ishi g color, s"ch as re!, to re#eal their prese ce. The co cept o+ li ks ca seem #ery co +"si g, %"t ha g i there. We<re goi g to try all this st"++ a ! it will, hope+"lly, %ecome clear.

Let2s /uild # Pla,ground

Si ce we are goi g to !o some real +ile ma ip"latio , let<s %"il! a sa+e place to ?play@ with o"r +ile ma ip"latio comma !s. 0irst we ee! a !irectory to work i . We<ll create o e i o"r home !irectory a ! call it ?playgro" !.@

Creating )ire/tories
The mkdir comma ! is "se! to create a !irectory. To create o"r playgro" ! !irectory we will +irst make s"re we are i o"r home !irectory a ! will the create the ew !irectory,
[me@linuxbox ~]$ cd [me@linuxbox ~]$ mkdir pla!ground

To make o"r playgro" ! a little more i teresti g, let<s create a co"ple o+ !irectories i si!e it calle! ?!ir.@ a ! ?!ir2@. To !o this, we will cha ge o"r c"rre t worki g !irectory to 7la1ground a ! e3ec"te a other mkdir,
[me@linuxbox ~]$ cd pla!ground [me@linuxbox 7la1ground]$ mkdir dir1 dir2

&otice that the mkdir comma ! will accept m"ltiple arg"me ts allowi g "s to create 10

*et<s J"il! $ 6laygro" ! %oth !irectories with a si gle comma !.

Co5ying 2i es
&e3t, let<s get some !ata i to o"r playgro" !. We<ll !o this %y copyi g a +ile. )si g the c7 comma !, we<ll copy the 7ass9d +ile +rom the 6etc !irectory to the c"rre t worki g !irectory,
[me@linuxbox 7la1ground]$ cp /etc/passwd .

&otice how we "se! the shortha ! +or the c"rre t worki g !irectory, the si gle traili g perio!. So ow i+ we per+orm a ls, we will see o"r +ile,
[me@linuxbox total #! dr9xr9xr-x ! dr9xr9xr-x ! -r9-r--r-- # 7la1ground]$ ls -l me me me me %(0. !((/-(#-#( #.:%( dir# me %(0. !((/-(#-#( #.:%( dir! me #."( !((/-(#-#( #.:() 7ass9d

&ow, 4"st +or +" , let<s repeat the copy "si g the ?-#@ optio E#er%oseF to see what it !oes,
[me@linuxbox 7la1ground]$ cp -v /etc/passwd . J6etc67ass9dK -D J<67ass9dK

The c7 comma ! per+orme! the copy agai , %"t this time !isplaye! a co cise message i !icati g what operatio it was per+ormi g. &otice that c7 o#erwrote the +irst copy witho"t a y war i g. $gai this is a case o+ c7 ass"mi g that yo" k ow what yo"Ire are !oi g. To get a war i g, we<ll i cl"!e the ?-i@ Ei teracti#eF optio ,
[me@linuxbox 7la1ground]$ cp -i /etc/passwd . c7: over9rite J<67ass9dKF

:espo !i g to the prompt %y e teri g a ?y@ will ca"se the +ile to %e o#erwritte , a y other character E+or e3ample, ? @F will ca"se c7 to lea#e the +ile alo e.

1.

2 8 >a ip"lati g 0iles $ ! 'irectories

:o;ing And !enaming 2i es

&ow, the ame ?passw!@ !oes <t seem #ery play+"l a ! this is a playgro" !, so let<s cha ge it to somethi g else,
[me@linuxbox 7la1ground]$ mv passwd fun

*et<s pass the +" aro" ! a little %y mo#i g o"r re ame! +ile to each o+ the !irectories a ! %ack agai ,
[me@linuxbox 7la1ground]$ mv fun dir1

to mo#e it +irst to !irectory dir#, the ,

[me@linuxbox 7la1ground]$ mv dir1/fun dir2

to mo#e it +rom dir# to dir!, the ,

[me@linuxbox 7la1ground]$ mv dir2/fun .

to +i ally %ri gi g it %ack to the c"rre t worki g !irectory. &e3t, let<s see the e++ect o+ mv o !irectories. 0irst we will mo#e o"r !ata +ile i to dir# agai ,
[me@linuxbox 7la1ground]$ mv fun dir1

the mo#e dir# i to dir! a ! co +irm it with ls,

[me@linuxbox [me@linuxbox total % dr9xr9xr-x ! [me@linuxbox total % -r9-r--r-- # 7la1ground]$ mv dir1 dir2 7la1ground]$ ls -l dir2 me me %(0. !((/-(#-## (.:(. dir# 7la1ground]$ ls -l dir2/dir1 me me #."( !((/-(#-#( #.:$$ fun

12

*et<s J"il! $ 6laygro" ! &ote that si ce dir! alrea!y e3iste!, mv mo#e! dir# i to dir!. 9+ dir! ha! ot e3iste!, mv wo"l! ha#e re ame! dir# to dir!. *astly, let<s p"t e#erythi g %ack,
[me@linuxbox 7la1ground]$ mv dir2/dir1 . [me@linuxbox 7la1ground]$ mv dir1/fun .

Creating $ard Links

&ow we<ll try some li ks. 0irst the har! li ks. WeIll create some li ks to o"r !ata +ile like so,
[me@linuxbox 7la1ground]$ ln fun fun-hard [me@linuxbox 7la1ground]$ ln fun dir1/fun-hard [me@linuxbox 7la1ground]$ ln fun dir2/fun-hard

So ow we ha#e +o"r i sta ces o+ the +ile ?+" @. *et<s take a look o"r playgro" ! !irectory,
[me@linuxbox total #. dr9xr9xr-x ! dr9xr9xr-x ! -r9-r--r-- % -r9-r--r-- % 7la1ground]$ ls -l me me me me me me me me %(0. %(0. #."( #."( !((/-(#-#% !((/-(#-#% !((/-(#-#( !((/-(#-#( #.:#) #.:#) #.:$$ #.:$$ dir# dir! fun fun-hard

7 e thi g yo" otice is that the seco ! +iel! i the listi g +or fun a ! fun-hard %oth co tai a ?1@ which is the "m%er o+ har! li ks that ow e3ist +or the +ile. 5o"<ll remem%er that a +ile will aways ha#e at least o e %eca"se the +ile<s ame is create! %y a li k. So, how !o we k ow that fun a ! fun-hard are, i +act, the same +ileC 9 this case, ls is ot #ery help+"l. While we ca see that fun a ! fun-hard are %oth the same siDe E+iel! 2F, o"r listi g pro#i!es o way to %e s"re. To sol#e this pro%lem, we<re goi g to ha#e to !ig a little !eeper. Whe thi ki g a%o"t har! li ks, it is help+"l to imagi e that +iles are ma!e "p o+ two parts, the !ata part co tai i g the +ile<s co te ts a ! the ame part which hol!s the +ile<s ame. Whe we create har! li ks, we are act"ally creati g a!!itio al ame parts that all re+er to the same !ata part. The system assig s a chai o+ !isk %locks to what is calle! a inode, which is the associate! with the ame part. Each har! li k there+ore re+ers to a speci+ic i o!e co tai i g the +ile<s co te ts. 1(

2 8 >a ip"lati g 0iles $ ! 'irectories The ls comma ! has a way to re#eal this i +ormatio . 9t is i #oke! with the ?-i@ optio ,
[me@linuxbox 7la1ground]$ ls -li total #. #!$"$"$0 dr9xr9xr-x ! me me %(0. #!$"$"%( dr9xr9xr-x ! me me %(0. #!$"$"$/ -r9-r--r-- % me me #."( #!$"$"$/ -r9-r--r-- % me me #."(

!((/-(#-#% !((/-(#-#% !((/-(#-#( !((/-(#-#(

#.:#) #.:#) #.:$$ #.:$$

dir# dir! fun fun-hard

9 this #ersio o+ the listi g, the +irst +iel! is the i o!e "m%er a !, as we ca see, %oth fun a ! fun-hard share the same i o!e "m%er, which co +irms they are the same +ile.

Creating Symbo i/ Links

Sym%olic li ks were create! to o#ercome the two !isa!#a tages o+ har! li ks, har! li ks ca ot spa physical !e#ices a ! har! li ks ca ot re+ere ce !irectories, o ly +iles. Sym%olic li ks are a special type o+ +ile that co tai s a te3t poi ter to the target +ile or !irectory. Creati g sym%olic li ks is similar to creati g har! li ks,
[me@linuxbox 7la1ground]$ ln -s fun fun-s!m [me@linuxbox 7la1ground]$ ln -s ../fun dir1/fun-s!m [me@linuxbox 7la1ground]$ ln -s ../fun dir2/fun-s!m

The +irst e3ample is pretty straight+orwar!, we simply a!! the ?-s@ optio to create a sym%olic li k rather tha a har! li k. J"t what a%o"t the e3t twoC :emem%er, whe we create a sym%olic li k, we are creati g a te3t !escriptio o+ where the target +ile is relati#e to the sym%olic li k. 9t<s easier to see i+ we look at the ls o"tp"t,
[me@linuxbox 7la1ground]$ ls -l dir1 total % -r9-r--r-- % me me #."( !((/-(#-#( #.:$$ fun-hard lr9xr9xr9x # me me . !((/-(#-#" #":#) fun-s1m -D <<6fun

The listi g +or fun-s1m i dir# shows that is it a sym%olic li k %y the lea!i g ?l@ i the +irst +iel! a ! that it poi ts to ?..-+" @, which is correct. :elati#e to the locatio o+ fun-s1m, fun is i the !irectory a%o#e it. &otice too, that the le gth o+ the sym%olic li k +ile is K, the "m%er o+ characters i the stri g ?..-+" @ rather tha the le gth o+ the 11

*et<s J"il! $ 6laygro" ! +ile to which it is poi ti g. Whe creati g sym%olic li ks, yo" ca either "se a%sol"te path ames,
ln -s /home/me/pla!ground/fun dir1/fun-s!m

or relati#e path ames, as we !i! i o"r earlier e3ample. )si g relati#e path ames is more !esira%le %eca"se it allows a !irectory co tai i g sym%olic li ks to %e re ame! a !-or mo#e! witho"t %reaki g the li ks. 9 a!!itio to reg"lar +iles, sym%olic li ks ca also re+ere ce !irectories,
[me@linuxbox [me@linuxbox total #. dr9xr9xr-x ! lr9xr9xr9x # dr9xr9xr-x ! -r9-r--r-- % -r9-r--r-- % lr9xr9xr9x # 7la1ground]$ ln -s dir1 dir1-s!m 7la1ground]$ ls -l me me me me me me me me me me me me %(0. % %(0. #."( #."( $ !((/-(#-#" !((/-(#-#. !((/-(#-#" !((/-(#-#( !((/-(#-#( !((/-(#-#" #":#) #%:%" #":#) #.:$$ #.:$$ #":#" dir# dir#-s1m -D dir# dir! fun fun-hard fun-s1m -D fun

!emo;ing 2i es And )ire/tories

$s we co#ere! earlier, the rm comma ! is "se! to !elete +iles a ! !irectories. We are goi g to "se it to clea "p o"r playgro" ! a little %it. 0irst, let<s !elete o e o+ o"r har! li ks,
[me@linuxbox [me@linuxbox total #! dr9xr9xr-x ! lr9xr9xr9x # dr9xr9xr-x ! -r9-r--r-- $ lr9xr9xr9x # 7la1ground]$ rm fun-hard 7la1ground]$ ls -l me me me me me me me me me me %(0. % %(0. #."( $ !((/-(#-#" !((/-(#-#. !((/-(#-#" !((/-(#-#( !((/-(#-#" #":#) #%:%" #":#) #.:$$ #":#" dir# dir#-s1m -D dir# dir! fun fun-s1m -D fun

That worke! as e3pecte!. The +ile fun-hard is go e a ! the li k co" t show +or fun is re!"ce! +rom +o"r to three, as i !icate! i the seco ! +iel! o+ the !irectory listi g. &e3t, we<ll !elete the +ile fun, a ! 4"st +or e 4oyme t, we<ll i cl"!e the ?-i@ optio to show what that !oes, 12

2 8 >a ip"lati g 0iles $ ! 'irectories

[me@linuxbox 7la1ground]$ rm -i fun rm: remove regular file JfunKF

E ter ?y@ at the prompt a ! the +ile is !elete!. J"t let<s look at the o"tp"t o+ ls ow. &otice! what happe e! to fun-s1mC Si ce it<s a sym%olic li k poi ti g to a owo e3iste t +ile, the li k is broken,
[me@linuxbox total / dr9xr9xr-x ! lr9xr9xr9x # dr9xr9xr-x ! lr9xr9xr9x # 7la1ground]$ ls -l me me me me me me me me %(0. % %(0. $ !((/-(#-#" !((/-(#-#. !((/-(#-#" !((/-(#-#" #":#) #%:%" #":#) #":#" dir# dir#-s1m -D dir# dir! fun-s1m -D fun

>ost *i "3 !istri%"tio s co +ig"re ls to !isplay %roke li ks. 7 a 0e!ora %o3, %roke li ks are !isplaye! i %li ki g re! te3tL The prese ce o+ a %roke li k is ot, i a ! o+ itsel+ !a gero"s %"t it is rather messy. 9+ we try to "se a %roke li k we will see this,
[me@linuxbox 7la1ground]$ less fun-s!m fun-s1m: Lo such file or director1

*et<s clea "p a little. We<ll !elete the sym%olic li ks,

[me@linuxbox [me@linuxbox total / dr9xr9xr-x ! dr9xr9xr-x ! 7la1ground]$ rm fun-s!m dir1-s!m 7la1ground]$ ls -l me me me me %(0. !((/-(#-#" #":#) dir# %(0. !((/-(#-#" #":#) dir!

7 e thi g to remem%er a%o"t sym%olic li ks is that most +ile operatio s are carrie! o"t o the li k<s target, ot the li k itsel+. rm is a e3ceptio . Whe yo" !elete a li k, it is the li k that is !elete!, ot the target. 0i ally, we will remo#e o"r playgro" !. To !o this, we will ret"r to o"r home !irectory a ! "se rm with the rec"rsi#e optio E-rF to !elete playgro" ! a ! all o+ its co te ts, i cl"!i g its s"%!irectories,
[me@linuxbox 7la1ground]$ cd

1K

*et<s J"il! $ 6laygro" !

[me@linuxbox ~]$ rm -r pla!ground

Creating S,mlin(s +ith The 6-I

The +ile ma agers i %oth ;&7>E a ! M'E pro#i!e a easy a ! a"tomatic metho! o+ creati g sym%olic li ks. With ;&7>E, hol!i g the CtrlVShi+t keys while !raggi g a +ile will create a li k rather tha copyi g Eor mo#i gF the +ile. 9 M'E, a small me " appears whe e#er a +ile is !roppe!, o++eri g a choice o+ copyi g, mo#i g, or li ki g the +ile.

Summing U5
We<#e co#ere! a lot o+ gro" ! here a ! it will take a while to +"lly si k i . 6er+orm the playgro" ! e3ercise o#er a ! o#er " til it makes se se. 9t is importa t to get a goo! " !ersta !i g o+ %asic +ile ma ip"latio comma !s a ! wil!car!s. 0eel +ree to e3pa ! o the playgro" ! e3ercise %y a!!i g more +iles a ! !irectories, "si g wil!car!s to speci+y +iles +or #ario"s operatio s. The co cept o+ li ks is a little co +"si g at +irst, %"t take the time to lear how they work. They ca %e a real li+esa#er.

1/

K 8 Worki g With Comma !s

+ Wor,ing With Co%%ands

)p to this poi t, we ha#e see a series o+ mysterio"s comma !s, each with its ow mysterio"s optio s a ! arg"me ts. 9 this chapter, we will attempt to remo#e some o+ that mystery a ! e#e create some o+ o"r ow comma !s. The comma !s i tro!"ce! i this chapter are,

t17e 8 9 !icate how a comma ! ame is i terprete! 9hich 8 'isplay which e3ec"ta%le program will %e e3ec"te! man 8 'isplay a comma !<s ma "al page a7ro7os 8 'isplay a list o+ appropriate comma !s info 8 'isplay a comma !<s i +o e try 9hatis 8 'isplay a #ery %rie+ !escriptio o+ a comma ! alias 8 Create a alias +or a comma !

+hat $xactl, #re Commands.

$ comma ! ca %e o e o+ +o"r !i++ere t thi gs, .. #n executable (rogram like all those +iles we saw i 6usr6bin. Withi this category, programs ca %e compiled binaries s"ch as programs writte i C a ! CVV, or programs writte i scripting languages s"ch as the shell, perl, pytho , r"%y, etc. 2. # command built into the shell itsel". bash s"pports a "m%er o+ comma !s i ter ally calle! shell builtins. The cd comma !, +or e3ample, is a shell %"ilti . (. # shell "unction. These are mi iat"re shell scripts i corporate! i to the environment. We will co#er co +ig"ri g the e #iro me t a ! writi g shell +" ctio s i later chapters, %"t +or ow, 4"st %e aware that they e3ist. 1. #n alias. Comma !s that we ca !e+i e o"rsel#es, %"ilt +rom other comma !s.

18

9!e ti+yi g Comma !s

Identi",ing Commands
9t is o+te "se+"l to k ow e3actly which o+ the +o"r ki !s o+ comma !s is %ei g "se! a ! *i "3 pro#i!es a co"ple o+ ways to +i ! o"t.

t17e A )is5 ay A Command"s Ty5e

The t17e comma ! is a shell %"ilti that !isplays the ki ! o+ comma ! the shell will e3ec"te, gi#e a partic"lar comma ! ame. 9t works like this,
t!pe command

where ?comma !@ is the ame o+ the comma ! yo" wa t to e3ami e. Aere are some e3amples,
[me@linuxbox ~]$ t!pe t!pe t17e is a shell builtin [me@linuxbox ~]$ t!pe ls ls is aliased to Jls --colorMtt1K [me@linuxbox ~]$ t!pe cp c7 is 6bin6c7

Aere we see the res"lts +or three !i++ere t comma !s. &otice that the o e +or ls Etake +rom a 0e!ora systemF a ! how the ls comma ! is act"ally a alias +or the ls comma ! with the ?-- colorWtty@ optio a!!e!. &ow we k ow why the o"tp"t +rom ls is !isplaye! i colorL

9hich A )is5 ay An 6xe/utab e"s Lo/ation

Sometimes there is more tha o e #ersio o+ a e3ec"ta%le program i stalle! o a system. While this is ot #ery commo o !esktop systems, it<s ot " "s"al o large ser#ers. To !etermi e the e3act locatio o+ a gi#e e3ec"ta%le, the 9hich comma ! is "se!,
[me@linuxbox ~]$ which ls 6bin6ls

9hich o ly works +or e3ec"ta%le programs, ot %"ilti s or aliases that are s"%stit"tes +or act"al e3ec"ta%le programs. Whe we try to "se 9hich o a shell %"ilti , +or 19

K 8 Worki g With Comma !s e3ample, cd, we either get o respo se or a error message,
[me@linuxbox ~]$ which cd 6usr6bin69hich: no cd in N6o7t6jre#<.<(O($6bin:6usr6lib6Bt-$<$6bin:6usr6kerberos6bin:6o7t6jre# <.<(O($6bin:6usr6lib6ccache:6usr6local6bin:6usr6bin:6bin:6home6me6bin P

which is a +a cy way o+ sayi g ?comma ! ot +o" !.@

6etting # Command2s Documentation

With this k owle!ge o+ what a comma ! is, we ca a#aila%le +or each ki ! o+ comma !. ow search +or the !oc"me tatio

hel7 A +et $e 5 2or She Bui tins

bash has a %"ilt-i help +acility a#aila%le +or each o+ the shell %"ilti s. To "se it, type ?help@ +ollowe! %y the ame o+ the shell %"ilti . 0or e3ample,
[me@linuxbox ~]$ help cd cd: cd [-=Q-:] [dir] Change the current director1 to 'AR< The variable $S +& is the default 'AR< The variable C':4TS defines the search 7ath for the director1 containing 'AR< 4lternative director1 names in C':4TS are se7arated b1 a colon N:P< 4 null director1 name is the same as the current director1> i<e< J<K< Af 'AR begins 9ith a slash N6P> then C':4TS is not used< Af the director1 is not found> and the shell o7tion JcdableOvarsK is set> then tr1 the 9ord as a variable name< Af that variable has a value> then cd to the value of that variable< The -: o7tion sa1s to use the 7h1sical director1 structure instead of follo9ing s1mbolic linksT the -= o7tion forces s1mbolic links to be follo9ed<

# note on notation5 Whe s="are %rackets appear i the !escriptio o+ a comma !<s sy ta3, they i !icate optio al items. $ #ertical %ar character i !icates m"t"ally e3cl"si#e items. 9 the case o+ the cd comma ! a%o#e, cd [-=Q-:] [dir] This otatio says that the comma ! cd may %e +ollowe! optio ally %y either a ?-*@ or a ?-6@ a ! +"rther, optio ally +ollowe! %y the arg"me t ?!ir@. While the o"tp"t o+ hel7 +or the cd comma !s is co cise a ! acc"rate, it is %y o 20

;etti g $ Comma !<s 'oc"me tatio mea s t"torial a ! as we ca see, it also seems to me tio a lot o+ thi gs we ha#e <t talke! a%o"t yetL 'o <t worry. We<ll get there.

--hel7 A )is5 ay Usage In=ormation

>a y e3ec"ta%le programs s"pport a ?--help@ optio that !isplays a !escriptio o+ the comma !<s s"pporte! sy ta3 a ! optio s. 0or e3ample,
[me@linuxbox ~]$ mkdir --help 3sage: mkdir [ :TA L] 'AR&CT RU<<< Create the 'AR&CT RUNiesP> if the1 do not alread1 exist< -I> --contextMC LT&VT N*&=inuxP set securit1 context to C LT&VT +andator1 arguments to long o7tions are mandator1 for short o7tions too< -m> --modeM+ '& set file mode Nas in chmodP> not aMr9x W umask -7> --7arents no error if existing> make 7arent directories as needed -v> --verbose 7rint a message for each created director1 --hel7 dis7la1 this hel7 and exit --version out7ut version information and exit

Re7ort bugs to Xbug-coreutils@gnu<orgD<

Some programs !o <t s"pport the ?--help@ optio , %"t try it a yway. 7+te it res"lts i a error message that will re#eal the same "sage i +ormatio .

man A )is5 ay A 'rogram"s :anua 'age

>ost e3ec"ta%le programs i te !e! +or comma ! li e "se pro#i!e a +ormal piece o+ !oc"me tatio calle! a manual or man page. $ special pagi g program calle! man is "se! to #iew them. 9t is "se! like this,
man program

where ?program@ is the ame o+ the comma ! to #iew. >a pages #ary somewhat i +ormat %"t ge erally co tai a title, a sy opsis o+ the comma !<s sy ta3, a !escriptio o+ the comma !<s p"rpose, a ! a listi g a ! !escriptio o+ each o+ the comma !<s optio s. >a pages, howe#er, !o ot "s"ally i cl"!e e3amples, a ! are i te !e! as a re+ere ce, ot a t"torial. $s a e3ample, let<s try #iewi g 2.

K 8 Worki g With Comma !s the ma page +or the ls comma !,

[me@linuxbox ~]$ man ls

7 most *i "3 systems, man "ses less to !isplay the ma "al page, so all o+ the +amiliar less comma !s work while !isplayi g the page. The ?ma "al@ that man !isplays is %roke i to sectio s a ! ot o ly co#ers "ser comma !s %"t also system a!mi istratio comma !s, programmi g i ter+aces, +ile +ormats a ! more. The ta%le %elow !escri%es the layo"t o+ the ma "al, Table 2-1: !an 6age &rgani7ation Section . 2 ( 1 2 K / 8 Contents )ser comma !s 6rogrammi g i ter+aces ker el system calls 6rogrammi g i ter+aces to the C li%rary Special +iles s"ch as !e#ice o!es a ! !ri#ers 0ile +ormats ;ames a ! am"seme ts s"ch as scree sa#ers >iscella eo"s System a!mi istratio comma !s

Sometimes we ee! to look i a speci+ic sectio o+ the ma "al to +i ! what we are looki g +or. This is partic"larly tr"e i+ we are looki g +or a +ile +ormat that is also the ame o+ a comma !. Witho"t speci+yi g a sectio "m%er, we will always get the +irst i sta ce o+ a match, pro%a%ly i sectio .. To speci+y a sectio "m%er, we "se man like this,
man section search_term

0or e3ample,
[me@linuxbox ~]$ man " passwd

22

;etti g $ Comma !<s 'oc"me tatio This will !isplay the ma page !escri%i g the +ile +ormat o+ the 6etc67ass9d +ile.

a7ro7os A )is5 ay A55ro5riate Commands

9t is also possi%le to search the list o+ ma pages +or possi%le matches %ase! o a search term. 9t<s #ery cr"!e %"t sometimes help+"l. Aere is a e3ample o+ a search +or ma pages "si g the search term ?+loppy@,
[me@linuxbox ~]$ apropos flopp! createOflo771Odevices N/P - udev callout to create all 7ossible flo771 device based on the C+ * t17e fdformat N/P - =o9-level formats a flo771 disk flo771 N/P - format flo771 disks gflo771 N#P - a sim7le flo771 formatter for the @L +& mbadblocks N#P - tests a flo771 disk> and marks the bad blocks in the -4T mformat N#P - add an +*' * files1stem to a lo9-level formatted flo771 disk

The +irst +iel! i each li e o+ o"tp"t is the ame o+ the ma page, the seco ! +iel! shows the sectio . &ote that the man comma ! with the ?-k@ optio per+orms the e3act same +" ctio as a7ro7os.

9hatis A )is5 ay A ?ery Brie= )es/ri5tion <= A Command

The 9hatis program !isplays the ame a ! a o e li e !escriptio o+ a ma page matchi g a speci+ie! keywor!,
[me@linuxbox ~]$ whatis ls ls N#P - list director1 contents

The >ost /rutal >an Page ?" Them #ll

$s we ha#e see , the ma "al pages s"pplie! with *i "3 a ! other ) i3-like systems are i te !e! as re+ere ce !oc"me tatio a ! ot as t"torials. >a y ma pages are har! to rea!, %"t 9 thi k that the gra ! priDe +or !i++ic"lty has got to go to the ma page +or bash. $s 9 was !oi g my research +or this %ook, 9 ga#e it care+"l re#iew to e s"re that 9 was co#eri g most o+ its topics. Whe pri te!, it<s

2(

K 8 Worki g With Comma !s

o#er eighty pages lo g a ! e3tremely !e se, a ! its str"ct"re makes a%sol"tely o se se to a ew "ser. 7 the other ha !, it is #ery acc"rate a ! co cise, as well as %ei g e3tremely complete. So check it o"t i+ yo" !are a ! look +orwar! to the !ay whe yo" ca rea! it a ! it all makes se se.

info A )is5 ay A 'rogram"s In=o 6ntry

The ;&) 6ro4ect pro#i!es a alter ati#e to ma pages +or their programs, calle! ?i +o.@ 9 +o pages are !isplaye! with a rea!er program ame!, appropriately e o"gh, info. 9 +o pages are hyperlinked m"ch like we% pages. Aere is a sample,
-ile: coreutils<info> 37: 'irector1 listing Lode: ls invocation> Lext: dir invocation>

#(<# JlsK: =ist director1 contents MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM

The JlsK 7rogram lists information about files Nof an1 t17e> including directoriesP< 7tions and file arguments can be intermixed arbitraril1> as usual<

-or non-o7tion command-line arguments that are directories> b1 default JlsK lists the contents of directories> not recursivel1> and omitting files 9ith names beginning 9ith J<K< -or other non-o7tion arguments> b1 default JlsK lists just the filename< Af no non-o7tion argument is s7ecified> JlsK o7erates on the current director1> acting as if it had been invoked 9ith a single argument of J<K<

H1 default> the out7ut is sorted al7habeticall1> according to the --CC-Anfo: Ncoreutils<info<gCPls invocation> .$ lines --To7----------

The info program rea!s info files, which are tree str"ct"re! i to i !i#i!"al nodes, each co tai i g a si gle topic. 9 +o +iles co tai hyperli ks that ca mo#e yo" +rom o!e to o!e. $ hyperli k ca %e i!e ti+ie! %y its lea!i g asterisk, a ! is acti#ate! %y placi g the c"rsor "po it a ! pressi g the e ter key. 21

;etti g $ Comma !<s 'oc"me tatio To i #oke info, type ?i +o@ +ollowe! optio ally %y the ame o+ a program. Jelow is a ta%le o+ comma !s "se! to co trol the rea!er while !isplayi g a i +o page, Table 2-': info %ommands Command F :g37 or Hacks7ace :g'n or *7ace n 7 u &nter B #ction 'isplay comma ! help 'isplay pre#io"s page 'isplay e3t page &e3t - 'isplay the e3t o!e 6re#io"s - 'isplay the pre#io"s o!e )p - 'isplay the pare t o!e o+ the c"rre tly !isplaye! o!e, "s"ally a me ". 0ollow the hyperli k at the c"rsor locatio T"it

>ost o+ the comma ! li e programs we ha#e !isc"sse! so +ar are part o+ the ;&) 6ro4ect<s ?core"tils@ package, so typi g,
[me@linuxbox ~]$ info coreutils

will !isplay a me " page with hyperli ks to each program co tai e! i the core"tils package.

!6A):6 And <ther 'rogram )o/umentation 2i es

>a y so+tware packages i stalle! o yo"r system ha#e !oc"me tatio +iles resi!i g i the 6usr6share6doc !irectory. >ost o+ these are store! i plai te3t +ormat a ! ca %e #iewe! with less. Some o+ the +iles are i AT>* +ormat a ! ca %e #iewe! with a we% %rowser. We may e co" ter some +iles e !i g with a ?.gD@ e3te sio . This i !icates that they ha#e %ee compresse! with the gCi7 compressio program. The gDip package i cl"!es a special #ersio o+ less calle! Cless that will !isplay the co te ts o+ gDip-compresse! te3t +iles.

22

K 8 Worki g With Comma !s

Creating :our ?4n Commands +ith alias

&ow +or o"r #ery +irst e3perie ce with programmi gL We will create a comma ! o+ o"r ow "si g the alias comma !. J"t %e+ore we start, we ee! to re#eal a small comma ! li e trick. 9t<s possi%le to p"t more tha o e comma ! o a li e %y separati g each comma ! with a semicolo character. 9t works like this,
command1T command2T command3<<<

Aere<s the e3ample we will "se,

[me@linuxbox ~]$ cd /usr# ls# cd bin games kerberos lib.% local etc include lib libexec sbin 6home6me [me@linuxbox ~]$

share src

tm7

$s we ca see, we ha#e com%i e! three comma !s o o e li e. 0irst we cha ge !irectory to 6usr the list the !irectory a ! +i ally ret"r to the origi al !irectory E%y "si g Kcd -KF so we e ! "p where we starte!. &ow let<s t"r this se="e ce i to a ew comma ! "si g alias. The +irst thi g we ha#e to !o is !ream "p a ame +or o"r ew comma !. *et<s try ?test@. Je+ore we !o that, it wo"l! %e a goo! i!ea to +i ! o"t i+ the ame ?test@ is alrea!y %ei g "se!. To +i ! o"t, we ca "se the t17e comma ! agai ,
[me@linuxbox ~]$ t!pe test test is a shell builtin

7opsL The ame ?test@ is alrea!y take . *et<s try ?+oo@,

[me@linuxbox ~]$ t!pe foo bash: t17e: foo: not found

;reatL ?+oo@ is ot take . So let<s create o"r alias,

[me@linuxbox ~]$ alias foo$Kcd /usr# ls# cd -%

2K

Creati g 5o"r 7w Comma !s With alias &otice the str"ct"re o+ this comma !,
alias name$%string%

$+ter the comma ! ?alias@ we gi#e alias a ame +ollowe! imme!iately E o whitespace allowe!F %y a e="als sig , +ollowe! imme!iately %y a ="ote! stri g co tai i g the mea i g to %e assig e! to the ame. $+ter we !e+i e o"r alias, it ca %e "se! a ywhere the shell wo"l! e3pect a comma !. *et<s try it,
[me@linuxbox ~]$ foo bin games kerberos etc include lib 6home6me [me@linuxbox ~]$

lib.% libexec

local sbin

share src

tm7

We ca also "se the t17e comma ! agai to see o"r alias,

[me@linuxbox ~]$ t!pe foo foo is aliased to Jcd 6usrT ls T cd -K

To remo#e a alias, the unalias comma ! is "se!, like so,

[me@linuxbox ~]$ unalias foo [me@linuxbox ~]$ t!pe foo bash: t17e: foo: not found

While we p"rpose+"lly a#oi!e! ami g o"r alias with a e3isti g comma ! ame, it is ot " commo to !o so. This is o+te !o e to apply a commo ly !esire! optio to each i #ocatio o+ a commo comma !. 0or i sta ce, we saw earlier how the ls comma ! is o+te aliase! to a!! color s"pport,
[me@linuxbox ~]$ t!pe ls ls is aliased to Jls --colorMtt1K

To see all the aliases !e+i e! i the e #iro me t, "se the alias comma ! witho"t arg"me ts. Aere are some o+ the aliases !e+i e! %y !e+a"lt o a 0e!ora system. Try a ! 2/

K 8 Worki g With Comma !s +ig"re o"t what they all !o,

[me@linuxbox alias l<MKls alias llMKls alias lsMKls ~]$ alias -d <E --colorMtt1K -l --colorMtt1K --colorMtt1K

There is o e ti y pro%lem with !e+i i g aliases o the comma ! li e. They #a ish whe yo"r shell sessio e !s. 9 a later chapter, we will see how to a!! o"r ow aliases to the +iles that esta%lish the e #iro me t each time we log o , %"t +or ow, e 4oy the +act that we ha#e take o"r +irst, al%eit ti y, step i to the worl! o+ shell programmi gL

1e%isiting ?ld ;riends

&ow that we ha#e lear e! how to +i ! the !oc"me tatio +or comma !s, go a ! look "p the !oc"me tatio +or all the comma !s we ha#e e co" tere! so +ar. St"!y what a!!itio al optio s are a#aila%le a ! try them o"tL

;urther 1eading
There are ma y o li e so"rces o+ !oc"me tatio +or *i "3 a ! the comma ! li e. Aere are some o+ the %est,

The 8ash 9eference !anual is a re+ere ce g"i!e to the bash shell. 9tIs still a re+ere ce work %"t co tai s e3amples a ! is easier to rea! tha the bash ma page. http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.html The 8ash F): co tai s a swers to +re="e tly aske! ="estio s regar!i g bash. This list is aime! at i terme!iate to a!#a ce! "sers, %"t co tai s a lot o+ goo! i +ormatio . http,--mywiki.woole!ge.org-Jash0$T The ;&) 6ro4ect pro#i!es e3te si#e !oc"me tatio +or its programs, which +orm the core o+ the *i "3 comma ! li e e3perie ce. 5o" ca see a complete list here, http,--www.g ".org-ma "al-ma "al.html Wikipe!ia has a i teresti g article o ma pages, http,--e .wikipe!ia.org-wiki->a Opage

28

/ 8 :e!irectio

. /edirection
9 this lesso we are goi g to " leash what may %e the coolest +eat"re o+ the comma ! li e. 9t<s calle! *;& redirection. The ?9-7@ sta !s +or input;output a ! with this +acility yo" ca re!irect the i p"t a ! o"tp"t o+ comma !s to a ! +rom +iles, as well as co ect m"ltiple comma !s together i to power+"l comma ! pipelines. To show o++ this +acility, we will i tro!"ce the +ollowi g comma !s,

cat - Co cate ate +iles sort - Sort li es o+ te3t uniB - :eport or omit repeate! li es gre7 - 6ri t li es matchi g a patter 9c - 6ri t ewli e, wor!, a ! %yte co" ts +or each +ile head - 7"tp"t the +irst part o+ a +ile tail - 7"tp"t the last part o+ a +ile tee - :ea! +rom sta !ar! i p"t a ! write to sta !ar! o"tp"t a ! +iles

Standard Input@ ?utput@ #nd $rror

>a y o+ the programs that we ha#e "se! so +ar pro!"ce o"tp"t o+ some ki !. This o"tp"t o+te co sists o+ two types. 0irst, we ha#e the program<s res"ltsH that is, the !ata the program is !esig e! to pro!"ce, a ! seco !, we ha#e stat"s a ! error messages that tell "s how the program is getti g alo g. 9+ we look at a comma ! like ls, we ca see that it !isplays its res"lts a ! its error messages o the scree . Meepi g with the ) i3 theme o+ ?e#erythi g is a +ile,@ programs s"ch as ls act"ally se ! their res"lts to a special +ile calle! standard output Eo+te e3presse! as stdoutF a ! their stat"s messages to a other +ile calle! standard error EstderrF. Jy !e+a"lt, %oth sta !ar! o"tp"t a ! sta !ar! error are li ke! to the scree a ! ot sa#e! i to a !isk +ile. 9 a!!itio , ma y programs take i p"t +rom a +acility calle! standard input EstdinF which is, %y !e+a"lt, attache! to the key%oar!.

29

/ 8 :e!irectio 9-7 re!irectio allows "s to cha ge where o"tp"t goes a ! where i p"t comes +rom. &ormally, o"tp"t goes to the scree a ! i p"t comes +rom the key%oar!, %"t with 9-7 re!irectio , we ca cha ge that.

1edirecting Standard ?utput

9-7 re!irectio allows "s to re!e+i e where sta !ar! o"tp"t goes. To re!irect sta !ar! o"tp"t to a other +ile %esi!es the scree , we "se the ?X@ re!irectio operator +ollowe! %y the ame o+ the +ile. Why wo"l! we wa t to !o thisC 9t<s o+te "se+"l to store the o"tp"t o+ a comma ! i a +ile. 0or e3ample, we co"l! tell the shell to se ! the o"tp"t o+ the ls comma ! to the +ile ls-out7ut<txt i stea! o+ the scree ,
[me@linuxbox ~]$ ls -l /usr/bin & ls-output.txt

Aere, we create! a lo g listi g o+ the 6usr6bin !irectory a ! se t the res"lts to the +ile ls-out7ut<txt. *et<s e3ami e the re!irecte! o"tp"t o+ the comma !,
[me@linuxbox ~]$ ls -l ls-output.txt -r9-r9-r-- # me me #.)/)/ !((/-(!-(# #":() ls-out7ut<txt

;oo!H a ice, large, te3t +ile. 9+ we look at the +ile with less, we will see that the +ile ls-out7ut<txt !oes i !ee! co tai the res"lts +rom o"r ls comma !,
[me@linuxbox ~]$ less ls-output.txt

&ow, let<s repeat o"r re!irectio test, %"t this time with a twist. We<ll cha ge the ame o+ the !irectory to o e that !oes ot e3ist,
[me@linuxbox ~]$ ls -l /bin/usr & ls-output.txt ls: cannot access 6bin6usr: Lo such file or director1

We recei#e! a error message. This makes se se si ce we speci+ie! the o -e3iste t !irectory 6bin6usr, %"t why was the error message !isplaye! o the scree rather tha %ei g re!irecte! to the +ile ls-out7ut<txtC The a swer is that the ls program !oes ot se ! its error messages to sta !ar! o"tp"t. 9 stea!, like most well-writte ) i3 programs, it se !s its error messages to sta !ar! error. Si ce we o ly re!irecte! sta !ar! o"tp"t a ! ot sta !ar! error, the error message was still se t to the scree . We<ll see K0

:e!irecti g Sta !ar! 7"tp"t how to re!irect sta !ar! error i 4"st a mi "te, %"t +irst, let<s look at what happe e! to o"r o"tp"t +ile,
[me@linuxbox ~]$ ls -l ls-output.txt -r9-r9-r-- # me me ( !((/-(!-(# #":(/ ls-out7ut<txt

The +ile ow has Dero le gthL This is %eca"se, whe we re!irect o"tp"t with the ?X@ re!irectio operator, the !esti atio +ile is always rewritte +rom the %egi i g. Si ce o"r ls comma ! ge erate! o res"lts a ! o ly a error message, the re!irectio operatio starte! to rewrite the +ile a ! the stoppe! %eca"se o+ the error, res"lti g i its tr" catio . 9 +act, i+ we e#er ee! to act"ally tr" cate a +ile Eor create a ew, empty +ileF we ca "se a trick like this,
[me@linuxbox ~]$ & ls-output.txt

Simply "si g the re!irectio operator with o comma ! prece!i g it will tr" cate a e3isti g +ile or create a ew, empty +ile. So, how ca we appe ! re!irecte! o"tp"t to a +ile i stea! o+ o#erwriti g the +ile +rom the %egi i gC 0or that, we "se the ?XX@ re!irectio operator, like so,
[me@linuxbox ~]$ ls -l /usr/bin && ls-output.txt

)si g the ?XX@ operator will res"lt i the o"tp"t %ei g appe !e! to the +ile. 9+ the +ile !oes ot alrea!y e3ist, it is create! 4"st as tho"gh the ?X@ operator ha! %ee "se!. *et<s p"t it to the test,
[me@linuxbox [me@linuxbox [me@linuxbox [me@linuxbox -r9-r9-r-- # ~]$ ~]$ ~]$ ~]$ me ls -l ls -l ls -l ls -l me /usr/bin && ls-output.txt /usr/bin && ls-output.txt /usr/bin && ls-output.txt ls-output.txt "($.$% !((/-(!-(# #":%" ls-out7ut<txt

We repeate! the comma ! three times res"lti g i a o"tp"t +ile three times as large.

1edirecting Standard $rror

:e!irecti g sta !ar! error lacks the ease o+ a !e!icate! re!irectio operator. To re!irect K.

/ 8 :e!irectio sta !ar! error we m"st re+er to its file descriptor. $ program ca o+ se#eral "m%ere! +ile streams. While we ha#e re+erre! to the streams as sta !ar! i p"t, o"tp"t a ! error, the shell re+ere ces !escriptors Dero, o e a ! two, respecti#ely. The shell pro#i!es a +iles "si g the +ile !escriptor "m%er. Si ce sta !ar! error is the "m%er two, we ca re!irect sta !ar! error with this otatio ,
[me@linuxbox ~]$ ls -l /bin/usr 2& ls-error.txt

pro!"ce o"tp"t o a y +irst three o+ these +ile them i ter ally as +ile otatio +or re!irecti g same as +ile !escriptor

The +ile !escriptor ?2@ is place! imme!iately %e+ore the re!irectio operator to per+orm the re!irectio o+ sta !ar! error to the +ile ls-error<txt.

!edire/ting Standard <ut5ut And Standard 6rror To <ne 2i e

There are cases i which we may wish to capt"re all o+ the o"tp"t o+ a comma ! to a si gle +ile. To !o this, we m"st re!irect %oth sta !ar! o"tp"t a ! sta !ar! error at the same time. There are two ways to !o this. 0irst, the tra!itio al way, which works with ol! #ersio s o+ the shell,
[me@linuxbox ~]$ ls -l /bin/usr & ls-output.txt 2&'1

)si g this metho!, we per+orm two re!irectio s. 0irst we re!irect sta !ar! o"tp"t to the +ile ls-out7ut<txt a ! the we re!irect +ile !escriptor two Esta !ar! errorF to +ile !escriptor o e Esta !ar! o"tp"tF "si g the otatio !DY#. 1otice that the order o" the redirections is signi"icant. The re!irectio o+ sta !ar! error m"st always occ"r after re!irecti g sta !ar! o"tp"t or it !oes <t work. 9 the e3ample a%o#e, Dls-out7ut<txt !DY# re!irects sta !ar! error to the +ile ls-out7ut<txt, %"t i+ the or!er is cha ge! to !DY# Dls-out7ut<txt sta !ar! error is !irecte! to the scree . :ece t #ersio s o+ bash pro#i!e a seco !, more streamli e! metho! +or per+ormi g this K2

:e!irecti g Sta !ar! Error com%i e! re!irectio ,

[me@linuxbox ~]$ ls -l /bin/usr '& ls-output.txt

9 this e3ample, we "se the si gle otatio YD to re!irect %oth sta !ar! o"tp"t a ! sta !ar! error to the +ile ls-out7ut<txt.

)is5osing <= Un%anted <ut5ut

Sometimes ?sile ce is gol!e ,@ a ! we !o <t wa t o"tp"t +rom a comma !, we 4"st wa t to throw it away. This applies partic"larly to error a ! stat"s messages. The system pro#i!es a way to !o this %y re!irecti g o"tp"t to a special +ile calle! ?-!e#- "ll@. This +ile is a system !e#ice calle! a bit bucket which accepts i p"t a ! !oes othi g with it. To s"ppress error messages +rom a comma !, we !o this,
[me@linuxbox ~]$ ls -l /bin/usr 2& /dev/null

/dev/null In -nix Culture

The %it %"cket is a a cie t ) i3 co cept a ! !"e to its " i#ersality, has appeare! i ma y parts o+ ) i3 c"lt"re. Whe someo e says he-she is se !i g yo"r comme ts to 6dev6null, ow yo" k ow what it mea s. 0or more e3amples, see the Wikipe!ia article o ?-!e#- "ll@.

1edirecting Standard Input

)p to ow, we ha#e <t e co" tere! a y comma !s that make "se o+ sta !ar! i p"t Eact"ally we ha#e, %"t weIll re#eal that s"rprise a little %it laterF, so we ee! to i tro!"ce o e.

cat A Con/atenate 2i es
The cat comma ! rea!s o e or more +iles a ! copies them to sta !ar! o"tp"t like so,
cat (file...)

K(

/ 8 :e!irectio 9 most cases, yo" ca thi k o+ cat as %ei g a alogo"s to the TU:& comma ! i '7S. 5o" ca "se it to !isplay +iles witho"t pagi g, +or e3ample,
[me@linuxbox ~]$ cat ls-output.txt

will !isplay the co te ts o+ the +ile ls-out7ut<txt. cat is o+te "se! to !isplay short te3t +iles. Si ce cat ca accept more tha o e +ile as a arg"me t, it ca also %e "se! to 4oi +iles together. Say we ha#e !ow loa!e! a large +ile that has %ee split i to m"ltiple parts Em"ltime!ia +iles are o+te split this way o )SE&ETF, a ! we wa t to 4oi them %ack together. 9+ the +iles were ame!, movie<m7eg<((# movie<m7eg<((! <<< movie<m7eg<(00 we co"l! 4oi them %ack together with this comma !,
cat movie.mpeg.** & movie.mpeg

Si ce wil!car!s always e3pa ! i sorte! or!er, the arg"me ts will %e arra ge! i the correct or!er. This is all well a ! goo!, %"t what !oes this ha#e to !o with sta !ar! i p"tC &othi g yet, %"t let<s try somethi g else. What happe s i+ we type ?cat@ with o arg"me ts,
[me@linuxbox ~]$ cat

&othi g happe s, it 4"st sits there like it<s h" g. 9t may seem that way, %"t it<s really !oi g e3actly what it<s s"ppose! to. 9+ cat is ot gi#e a y arg"me ts, it rea!s +rom sta !ar! i p"t a ! si ce sta !ar! i p"t is, %y !e+a"lt, attache! to the key%oar!, it<s waiti g +or "s to type somethi gL Try this,
[me@linuxbox ~]$ cat +he ,uick brown fox jumped over the la-! dog.

&e3t, type a Ctrl-d Ei.e., hol! !ow the Ctrl key a ! press ?!@F to tell cat that it has reache! end of file EE70F o sta !ar! i p"t,

K1

:e!irecti g Sta !ar! 9 p"t

[me@linuxbox ~]$ cat +he ,uick brown fox jumped over the la-! dog. The Buick bro9n fox jum7ed over the laC1 dog<

9 the a%se ce o+ +ile ame arg"me ts, cat copies sta !ar! i p"t to sta !ar! o"tp"t, so we see o"r li e o+ te3t repeate!. We ca "se this %eha#ior to create short te3t +iles. *et<s say that we wa te! to create a +ile calle! ?laDyO!og.t3t@ co tai i g the te3t i o"r e3ample. We wo"l! !o this,
[me@linuxbox ~]$ cat & la-!.dog.txt +he ,uick brown fox jumped over the la-! dog.

Type the comma ! +ollowe! %y the te3t we wa t i to place i the +ile. :emem%er to type Ctrl-d at the e !. )si g the comma ! li e, we ha#e impleme te! the worl!<s !"m%est wor! processorL To see o"r res"lts, we ca "se cat to copy the +ile to st!o"t agai ,
[me@linuxbox ~]$ cat la-!.dog.txt The Buick bro9n fox jum7ed over the laC1 dog<

&ow that we k ow how cat accepts sta !ar! i p"t, i a!!itio to +ile ame arg"me ts, let<s try re!irecti g sta !ar! i p"t,
[me@linuxbox ~]$ cat / la-!.dog.txt The Buick bro9n fox jum7ed over the laC1 dog<

)si g the ?Y@ re!irectio operator, we cha ge the so"rce o+ sta !ar! i p"t +rom the key%oar! to the +ile laC1Odog<txt. We see that the res"lt is the same as passi g a si gle +ile ame arg"me t. This is ot partic"larly "se+"l compare! to passi g a +ile ame arg"me t, %"t it ser#es to !emo strate "si g a +ile as a so"rce o+ sta !ar! i p"t. 7ther comma !s make %etter "se o+ sta !ar! i p"t, as we shall soo see. Je+ore we mo#e o , check o"t the ma page +or cat, as it has se#eral i teresti g optio s.

Pipelines
The a%ility o+ comma !s to rea! !ata +rom sta !ar! i p"t a ! se ! to sta !ar! o"tp"t is "tiliDe! %y a shell +eat"re calle! pipelines. )si g the pipe operator ?Z@ E#ertical %arF, the K2

/ 8 :e!irectio sta !ar! o"tp"t o+ o e comma ! ca %e piped i to the sta !ar! i p"t o+ a other,
command1 Q command2

To +"lly !emo strate this, we are goi g to ee! some comma !s. :emem%er how we sai! there was o e we alrea!y k ew that accepts sta !ar! i p"tC 9t<s less. We ca "se less to !isplay, page-%y-page, the o"tp"t o+ a y comma ! that se !s its res"lts to sta !ar! o"tp"t,
[me@linuxbox ~]$ ls -l /usr/bin 0 less

This is e3tremely ha !yL )si g this tech i="e, we ca co #e ie tly e3ami e the o"tp"t o+ a y comma ! that pro!"ces sta !ar! o"tp"t.

2i ters
6ipeli es are o+te "se! to per+orm comple3 operatio s o !ata. 9t is possi%le to p"t se#eral comma !s together i to a pipeli e. 0re="e tly, the comma !s "se! this way are re+erre! to as filters. 0ilters take i p"t, cha ge it somehow a ! the o"tp"t it. The +irst o e we will try is sort. 9magi e we wa te! to make a com%i e! list o+ all o+ the e3ec"ta%le programs i 6bin a ! 6usr6bin, p"t them i sorte! or!er a ! #iew it,
[me@linuxbox ~]$ ls /bin /usr/bin 0 sort 0 less

Si ce we speci+ie! two !irectories E6bin a ! 6usr6binF, the o"tp"t o+ ls wo"l! ha#e co siste! o+ two sorte! lists, o e +or each !irectory. Jy i cl"!i g sort i o"r pipeli e, we cha ge! the !ata to pro!"ce a si gle, sorte! list.

uni( B !e5ort <r <mit !e5eated Lines

The uniB comma ! is o+te "se! i co 4" ctio with sort. uniB accepts a sorte! list o+ !ata +rom either sta !ar! i p"t or a si gle +ile ame arg"me t Esee the uniB ma page +or !etailsF a !, %y !e+a"lt, remo#es a y !"plicates +rom the list. So, to make s"re o"r list has o !"plicates Ethat is, a y programs o+ the same ame that appear i %oth the 6bin a ! 6usr6bin !irectoriesF we will a!! uniB to o"r pipeli e,

KK

6ipeli es
[me@linuxbox ~]$ ls /bin /usr/bin 0 sort 0 uni, 0 less

9 this e3ample, we "se uniB to remo#e a y !"plicates +rom the o"tp"t o+ the sort comma !. 9+ we wa t to see the list o+ !"plicates i stea!, we a!! the ?-!@ optio to uniB like so,
[me@linuxbox ~]$ ls /bin /usr/bin 0 sort 0 uni, -d 0 less

9c A 'rint LineC WordC And Byte Counts

The 9c Ewor! co" tF comma ! is "se! to !isplay the "m%er o+ li es, wor!s, a ! %ytes co tai e! i +iles. 0or e3ample,
[me@linuxbox ~]$ wc ls-output.txt )0(! .%".. "($.$% ls-out7ut<txt

9 this case it pri ts o"t three "m%ers, li es, wor!s, a ! %ytes co tai e! i lsout7ut<txt. *ike o"r pre#io"s comma !s, i+ e3ec"te! witho"t comma ! li e arg"me ts, 9c accepts sta !ar! i p"t. The ?-l@ optio limits its o"tp"t to o ly report li es. $!!i g it to a pipeli e is a ha !y way to co" t thi gs. To see the "m%er o+ programs we ha#e i o"r sorte! list, we ca !o this,
[me@linuxbox ~]$ ls /bin /usr/bin 0 sort 0 uni, 0 wc -l !)!/

gre7 A 'rint Lines :at/hing A 'attern

gre7 is a power+"l program "se! to +i ! te3t patter s withi +iles. 9t<s "se! like this,
gre7 pattern [file...]

Whe gre7 e co" ters a ?patter @ i the +ile, it pri ts o"t the li es co tai i g it. The patter s that gre7 ca match ca %e #ery comple3, %"t +or ow we will co ce trate o simple te3t matches. We<ll co#er the a!#a ce! patter s, calle! regular e+pressions i a K/

/ 8 :e!irectio later chapter. *et<s say we wa t to +i ! all the +iles i o"r list o+ programs that ha! the wor! ?Dip@ em%e!!e! i the ame. S"ch a search might gi#e "s a i!ea o+ some o+ the programs o o"r system that ha! somethi g to !o with +ile compressio . We wo"l! !o this,
[me@linuxbox ~]$ ls /bin /usr/bin 0 sort 0 uni, 0 grep -ip bunCi7! bCi7! gunCi7 gCi7 unCi7 Ci7 Ci7cloak Ci7gre7 Ci7info Ci7note Ci7s7lit

There are a co"ple o+ ha !y optio s +or gre7, ?-i@ which ca"ses gre7 to ig ore case whe per+ormi g the search E ormally searches are case se siti#eF a ! ?-#@ which tells gre7 to o ly pri t li es that !o ot match the patter .

head - tail A 'rint 2irst - Last 'art <= 2i es

Sometimes yo" !o <t wa t all o+ the o"tp"t +rom a comma !. 5o" may o ly wa t the +irst +ew li es or the last +ew li es. The head comma ! pri ts the +irst te li es o+ a +ile a ! the tail comma ! pri ts the last te li es. Jy !e+a"lt, %oth comma !s pri t te li es o+ te3t, %"t this ca %e a!4"ste! with the ?- @ optio ,
[me@linuxbox total $%$%0. -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # [me@linuxbox -r9xr-xr-x # -r9xr-xr-x # -r9-r--r-- # -r9-r--r-- # lr9xr9xr9x # ~]$ head -n " ls-output.txt root root $#$#. !(()-#!-(" root root /!%( !(()-#!-(0 root root ###!). !(()-##-!. root root !"$./ !((.-#(-(. ~]$ tail -n " ls-output.txt root root "!$% !(()-(.-!) root root .0# !(("-(0-#( root root 0$( !(()-##-(# root root 0$( !(()-##-(# root root . !((/-(#-$# (/:"/ #$:$0 #%:!) !(:#. #(:". (%:!# #!:!$ #!:!$ (":!! [ %##to77m a!7 a"!dec Cne9 Conetab!7ot<71 Conetab!7ot<71c Conetab!7ot<71o Csoelim -D soelim

These ca %e "se! i pipeli es as well, K8

6ipeli es
[me@linuxbox ~]$ ls /usr/bin 0 tail -n " Cne9 Conetab!7ot<71 Conetab!7ot<71c Conetab!7ot<71o Csoelim

tail has a optio which allows yo" to #iew +iles i real-time. This is "se+"l +or watchi g the progress o+ log +iles as they are %ei g writte . 9 the +ollowi g e3ample, we will look at the messages +ile i 6var6log. S"per"ser pri#ileges are re="ire! to !o this o some *i "3 !istri%"tio s, si ce the 6var6log6messages +ile may co tai sec"rity i +ormatio ,
[me@linuxbox ~]$ tail -f /var/log/messages -eb / #$:%(:(" t9in% dhclient: 'SC:4C2 from #0!<#./<#<# -eb / #$:%(:(" t9in% dhclient: bound to #0!<#./<#<% -- rene9al in #."! seconds< -eb / #$:"":$! t9in% mountd[$0"$]: 6var6L-*v%6musicbox ex7orted to both #0!<#./<#<(6!% and t9in)<localdomain in #0!<#./<#<(6!%>t9in)<localdomain -eb / #%:():$) t9in% dhclient: 'SC:R&Z3&*T on eth( to #0!<#./<#<# 7ort .) -eb / #%:():$) t9in% dhclient: 'SC:4C2 from #0!<#./<#<# -eb / #%:():$) t9in% dhclient: bound to #0!<#./<#<% -- rene9al in #))# seconds< -eb / #%:(0:". t9in% smartd[$%./]: 'evice: 6dev6hda> *+4RT :refailure 4ttribute: / *eekOTimeO:erformance changed from !$) to !$. -eb / #%:#(:$) t9in% mountd[$0"$]: 6var6L-*v%6musicbox ex7orted to both #0!<#./<#<(6!% and t9in)<localdomain in #0!<#./<#<(6!%>t9in)<localdomain -eb / #%:!":() t9in% sshdN7amOunixP[!0!$%]: session o7ened for user me b1 NuidM(P -eb / #%:!":$. t9in% suN7amOunixP[!0!)0]: session o7ened for user root b1 meNuidM"((P

)si g the ?-+@ optio , tail co ti "es to mo itor the +ile a ! whe ew li es are appe !e!, they imme!iately appear o the !isplay. This co ti "es " til yo" type Ctrlc.

tee A !ead 2rom Stdin And <ut5ut To Stdout And 2i es

9 keepi g with o"r pl"m%i g metaphor, *i "3 pro#i!es a comma ! calle! tee which creates a ?tee@ +itti g o o"r pipe. The tee program rea!s sta !ar! i p"t a ! copies it to %oth sta !ar! o"tp"t Eallowi g the !ata to co ti "e !ow the pipeli eF a ! to o e or more K9

/ 8 :e!irectio +iles. This is "se+"l +or capt"ri g a pipeli e<s co te ts at a i terme!iate stage o+ processi g. Aere we repeat o e o+ o"r earlier e3amples, this time i cl"!i g tee to capt"re the e tire !irectory listi g to the +ile ls<txt %e+ore gre7 +ilters the pipeli e<s co te ts,
[me@linuxbox ~]$ ls /usr/bin 0 tee ls.txt 0 grep -ip bunCi7! bCi7! gunCi7 gCi7 unCi7 Ci7 Ci7cloak Ci7gre7 Ci7info Ci7note Ci7s7lit

Summing -p
$s always, check o"t the !oc"me tatio o+ each o+ the comma !s we ha#e co#ere! i this chapter. We ha#e o ly see their most %asic "sage. They all ha#e a "m%er o+ i teresti g optio s. $s we gai *i "3 e3perie ce, we will see that the re!irectio +eat"re o+ the comma ! li e is e3tremely "se+"l +or sol#i g specialiDe! pro%lems. There are ma y comma !s that make "se o+ sta !ar! i p"t a ! o"tp"t, a ! almost all comma ! li e programs "se sta !ar! error to !isplay their i +ormati#e messages.

Linux Is #0out Imagination

Whe 9 am aske! to e3plai the !i++ere ce %etwee Wi !ows a ! *i "3, 9 o+te "se a toy a alogy. Wi !ows is like a ;ame Joy. 5o" go to the store a ! %"y o e all shi y ew i the %o3. 5o" take it home, t"r it o a ! play with it. 6retty graphics, c"te so" !s. $+ter a while tho"gh, yo" get tire! o+ the game that came with it so yo" go %ack to the store a ! %"y a other o e. This cycle repeats o#er a ! o#er. 0i ally, yo" go %ack to the store a ! say to the perso %ehi ! the co" ter, ?9 wa t a game that !oes thisL@ o ly to %e tol! that o s"ch game e3ists %eca"se there is o ?market !ema !@ +or it. The yo" say, ?J"t 9 o ly ee! to cha ge this o e thi gL@ The perso %ehi ! the co" ter says yo" ca <t cha ge it. The games are

/0

S"mmi g )p

all seale! "p i their cartri!ges. 5o" !isco#er that yo"r toy is limite! to the games that others ha#e !eci!e! that yo" ee! a ! o more. *i "3, o the other ha !, is like the worl!<s largest Erector Set. 5o" ope it "p a ! it<s 4"st a h"ge collectio o+ parts. $ lot o+ steel str"ts, screws, "ts, gears, p"lleys, motors, a ! a +ew s"ggestio s o what to %"il!. So yo" start to play with it. 5o" %"il! o e o+ the s"ggestio s a ! the a other. $+ter a while yo" !isco#er that yo" ha#e yo"r ow i!eas o+ what to make. 5o" !o <t e#er ha#e to go %ack to the store, as yo" alrea!y ha#e e#erythi g yo" ee!. The Erector Set takes o the shape o+ yo"r imagi atio . 9t !oes what yo" wa t. 5o"r choice o+ toys is, o+ co"rse, a perso al thi g, so which toy wo"l! yo" +i ! more satis+yi gC

/.

8 8 Seei g The Worl! $s The Shell Sees 9t

- Seeing The World )s The Shell Sees It

9 this chapter we are goi g to look at some o+ the ?magic@ that occ"rs o the comma ! li e whe yo" press the e ter key. While we will e3ami e se#eral i teresti g a ! comple3 +eat"res o+ the shell, we will !o it with 4"st o e ew comma !,

echo 8 'isplay a li e o+ te3t

$xpansion
Each time yo" type a comma ! li e a ! press the e ter key, bash per+orms se#eral processes "po the te3t %e+ore it carries o"t yo"r comma !. We ha#e see a co"ple o+ cases o+ how a simple character se="e ce, +or e3ample ?U@, ca ha#e a lot o+ mea i g to the shell. The process that makes this happe is calle! e+pansion. With e3pa sio , yo" type somethi g a ! it is e3pa !e! i to somethi g else %e+ore the shell acts "po it. To !emo strate what we mea %y this, let<s take a look at the echo comma !. echo is a shell %"ilti that per+orms a #ery simple task. 9t pri ts o"t its te3t arg"me ts o sta !ar! o"tp"t,
[me@linuxbox ~]$ echo this is a test this is a test

That<s pretty straight+orwar!. $ y arg"me t passe! to echo gets !isplaye!. *et<s try a other e3ample,
[me@linuxbox ~]$ echo * 'eskto7 'ocuments ls-out7ut<txt +usic :ictures :ublic Tem7lates ;ideos

So what 4"st happe e!C Why !i! <t echo pri t ?U@C $s yo" recall +rom o"r work with wil!car!s, the ?U@ character mea s match a y characters i a +ile ame, %"t what we !i! <t see i o"r origi al !isc"ssio was how the shell !oes that. The simple a swer is that the shell e3pa !s the ?U@ i to somethi g else Ei this i sta ce, the ames o+ the +iles i the /2

E3pa sio c"rre t worki g !irectoryF %e+ore the echo comma ! is e3ec"te!. Whe the e ter key is presse!, the shell a"tomatically e3pa !s a y ="ali+yi g characters o the comma ! li e %e+ore the comma ! is carrie! o"t, so the echo comma ! e#er saw the ?U@, o ly its e3pa !e! res"lt. M owi g this, we ca see that echo %eha#e! as e3pecte!.

'athname 6x5ansion
The mecha ism %y which wil!car!s work is calle! pathname e+pansion. 9+ we try some o+ the tech i="es that we employe! i o"r earlier chapters, we will see that they are really e3pa sio s. ;i#e a home !irectory that looks like this,
[me@linuxbox ~]$ ls 'eskto7 ls-out7ut<txt 'ocuments +usic

:ictures :ublic

Tem7lates ;ideos

we co"l! carry o"t the +ollowi g e3pa sio s,

[me@linuxbox ~]$ echo 1* 'eskto7 'ocuments

a !,
[me@linuxbox ~]$ echo *s 'ocuments :ictures Tem7lates ;ideos

or e#e ,
[me@linuxbox ~]$ echo ((2upper2))* 'eskto7 'ocuments +usic :ictures :ublic Tem7lates ;ideos

a ! looki g %eyo ! o"r home !irectory,

[me@linuxbox ~]$ echo /usr/*/share 6usr6kerberos6share 6usr6local6share

/(

8 8 Seei g The Worl! $s The Shell Sees 9t

Pathname $xpansion ?" 3idden ;iles

$s we k ow, +ile ames that %egi with a perio! character are hi!!e . 6ath ame e3pa sio also respects this %eha#ior. $ e3pa sio s"ch as, echo E !oes ot re#eal hi!!e +iles. 9t might appear at +irst gla ce that we co"l! i cl"!e hi!!e +iles i a e3pa sio %y starti g the patter with a lea!i g perio!, like this, echo <E 9t almost works. Aowe#er, i+ we e3ami e the res"lts closely, we will see that the ames ?.@ a ! ?..@ will also appear i the res"lts. Si ce these ames re+er to the c"rre t worki g !irectory a ! its pare t !irectory, "si g this patter will likely pro!"ce a i correct res"lt. We ca see this i+ we try the comma !, ls -d <E Q less To correctly per+orm path ame e3pa sio i this sit"atio , we ha#e to employ a more speci+ic patter . This will work correctly, ls -d <[G<]FE This patter e3pa !s i to e#ery +ile ame that %egi s with a perio!, !oes ot i cl"!e a seco ! perio!, co tai s at least o e a!!itio al character a ! ca %e +ollowe! %y a y other characters.

Ti de 6x5ansion
$s yo" may recall +rom o"r i tro!"ctio to the cd comma !, the til!e character E?S@F has a special mea i g. Whe "se! at the %egi i g o+ a wor!, it e3pa !s i to the ame o+ the home !irectory o+ the ame! "ser, or i+ o "ser is ame!, the home !irectory o+ the c"rre t "ser,
[me@linuxbox ~]$ echo ~ 6home6me

9+ "ser ?+oo@ has a acco" t, the ,

[me@linuxbox ~]$ echo ~foo

/1

E3pa sio
6home6foo

Arithmeti/ 6x5ansion
The shell allows arithmetic to %e per+orme! %y e3pa sio . This allow "s to "se the shell prompt as a calc"lator,
[me@linuxbox ~]$ echo 3442 5 266 %

$rithmetic e3pa sio "ses the +orm, $NNexpressionPP where e+pression is a operators. arithmetic e3pressio co sisti g o+ #al"es a ! arithmetic

$rithmetic e3pa sio o ly s"pports i tegers Ewhole "m%ers, o !ecimalsF, %"t ca per+orm ="ite a "m%er o+ !i++ere t operatio s. Aere are a +ew o+ the s"pporte! operators, Table 4-1: )rithmetic &perators ?perator 8 E 6 5 EE Description $!!itio S"%tractio >"ltiplicatio 'i#isio E%"t remem%er, si ce e3pa sio o ly s"pports i teger arithmetic, res"lts are i tegers.F >o!"lo, which simply mea s, ? remai !er.@ E3po e tiatio

Spaces are ot sig i+ica t i arithmetic e3pressio s a ! e3pressio s may %e este!. 0or e3ample, to m"ltiply +i#e s="are! %y three,
[me@linuxbox ~]$ echo 344344"**266 * )" 66

/2

8 8 Seei g The Worl! $s The Shell Sees 9t Si gle pare theses may %e "se! to gro"p m"ltiple s"%e3pressio s. With this tech i="e, we ca rewrite the e3ample a%o#e a ! get the same res"lt "si g a si gle e3pa sio i stea! o+ two,
[me@linuxbox ~]$ echo 3444"**26 * )" 66

Aere is a e3ample "si g the !i#isio a ! remai !er operators. &otice the e++ect o+ i teger !i#isio ,
[me@linuxbox ~]$ echo 7ive divided b! two e,uals 344"/266 -ive divided b1 t9o eBuals ! [me@linuxbox ~]$ echo with 344"8266 left over. 9ith # left over<

$rithmetic e3pa sio is co#ere! i greater !etail i Chapter (2.

Bra/e 6x5ansion
6erhaps the stra gest e3pa sio is calle! brace e+pansion. With it, yo" ca create m"ltiple te3t stri gs +rom a patter co tai i g %races. Aere<s a e3ample,
[me@linuxbox ~]$ echo 7ront-9:;<;=>-<ack -ront-4-Hack -ront-H-Hack -ront-C-Hack

6atter s to %e %race e3pa !e! may co tai a lea!i g portio calle! a preamble a ! a traili g portio calle! a postscript. The %race e3pressio itsel+ may co tai either a comma-separate! list o+ stri gs, or a ra ge o+ i tegers or si gle characters. The patter may ot co tai em%e!!e! whitespace. Aere is a e3ample "si g a ra ge o+ i tegers,
[me@linuxbox ~]$ echo ?umber.91.."> LumberO# LumberO! LumberO$ LumberO% LumberO"

$ ra ge o+ letters i re#erse or!er,

[me@linuxbox ~]$ echo 9@..:> I U V , ; 3 T * R Z : L + = 2 ? A S @ - & ' C H 4

/K

E3pa sio Jrace e3pa sio s may %e este!,

[me@linuxbox ~]$ echo a9:91;2>;<9 ;A>>b a4#b a4!b aH$b aH%b

So what is this goo! +orC The most commo applicatio is to make lists o+ +iles or !irectories to %e create!. 0or e3ample, i+ we were photographers a ! ha! a large collectio o+ images that we wa te! to orga iDe i to years a ! mo ths, the +irst thi g we might !o is create a series o+ !irectories ame! i "meric ?5ear->o th@ +ormat. This way, the !irectory ames will sort i chro ological or!er. We co"l! type o"t a complete list o+ !irectories, %"t that<s a lot o+ work a ! it<s error-pro e too. 9 stea!, we co"l! !o this,
[me@linuxbox ~]$ mkdir Bics [me@linuxbox ~]$ cd Bics [me@linuxbox :ics]$ mkdir 92**C..2**D>-*91..D> 92**C..2**D>-91*..12> [me@linuxbox :ics]$ ls !(()-(# !(()-() !((/-(# !((/-() !((0-(# !((0-() !(()-(! !(()-(/ !((/-(! !((/-(/ !((0-(! !((0-(/ !(()-($ !(()-(0 !((/-($ !((/-(0 !((0-($ !((0-(0 !(()-(% !(()-#( !((/-(% !((/-#( !((0-(% !((0-#( !(()-(" !(()-## !((/-(" !((/-## !((0-(" !((0-## !(()-(. !(()-#! !((/-(. !((/-#! !((0-(. !((0-#!

6retty slickL

'arameter 6x5ansion
We<re o ly goi g to to"ch %rie+ly o parameter e3pa sio i this chapter, %"t we<ll %e co#eri g it e3te si#ely later. 9t<s a +eat"re that is more "se+"l i shell scripts tha !irectly o the comma ! li e. >a y o+ its capa%ilities ha#e to !o with the system<s a%ility to store small ch" ks o+ !ata a ! to gi#e each ch" k a ame. >a y s"ch ch" ks, more properly calle! variables, are a#aila%le +or yo"r e3ami atio . 0or e3ample, the #aria%le ame! ?)SE:@ co tai s yo"r "ser ame. To i #oke parameter e3pa sio a ! re#eal the co te ts o+ 3*&R yo" wo"l! !o this,
[me@linuxbox ~]$ echo 3EFGH me

To see a list o+ a#aila%le #aria%les, try this, //

8 8 Seei g The Worl! $s The Shell Sees 9t

[me@linuxbox ~]$ printenv 0 less

5o" may ha#e otice! that with other types o+ e3pa sio , i+ yo" mistype a patter , the e3pa sio will ot take place a ! the echo comma ! will simply !isplay the mistype! patter . With parameter e3pa sio , i+ yo" misspell the ame o+ a #aria%le, the e3pa sio will still take place, %"t will res"lt i a empty stri g,
[me@linuxbox ~]$ echo 3FEGH [me@linuxbox ~]$

Command Substitution
Comma ! s"%stit"tio allows "s to "se the o"tp"t o+ a comma ! as a e3pa sio ,
[me@linuxbox ~]$ echo 34ls6 'eskto7 'ocuments ls-out7ut<txt +usic :ictures :ublic Tem7lates ;ideos

7 e o+ my +a#orites goes somethi g like this,

[me@linuxbox ~]$ ls -l 34which cp6 -r9xr-xr-x # root root )#"#. !(()-#!-(" (/:"/ 6bin6c7

Aere we passe! the res"lts o+ 9hich c7 as a arg"me t to the ls comma !, there%y getti g the listi g o+ o+ the c7 program witho"t ha#i g to k ow its +"ll path ame. We are ot limite! to 4"st simple comma !s. E tire pipeli es ca %e "se! Eo ly partial o"tp"t show F,
[me@linuxbox ~]$ file 34ls /usr/bin/* 0 grep -ip6 6usr6bin6bunCi7!: s1mbolic link to JbCi7!K 6usr6bin6bCi7!: &=- $!-bit =*H executable> version # N*U*;P> d1namicall1 linked Nuses shared @L36=inux !<.<0> stri77ed 6usr6bin6bCi7!recover: &=- $!-bit =*H executable> version # N*U*;P> d1namicall1 linked Nuses shared @L36=inux !<.<0> stri77ed

Antel /($/.> libsP> for Antel /($/.> libsP> for

/8

E3pa sio
6usr6bin6funCi7: &=- $!-bit =*H executable> Antel /($/.> version # N*U*;P> d1namicall1 linked Nuses shared libsP> for @L36=inux !<.<0> stri77ed 6usr6bin6g7g-Ci7: Hourne shell scri7t text executable 6usr6bin6gunCi7: s1mbolic link to J<<6<<6bin6gunCi7K 6usr6bin6gCi7: s1mbolic link to J<<6<<6bin6gCi7K 6usr6bin6mCi7: s1mbolic link to JmtoolsK

9 this e3ample, the res"lts o+ the pipeli e %ecame the arg"me t list o+ the file comma !. There is a alter ate sy ta3 +or comma ! s"%stit"tio i ol!er shell programs which is also s"pporte! i bash. 9t "ses back-<uotes i stea! o+ the !ollar sig a ! pare theses,
[me@linuxbox ~]$ ls -l Iwhich cpI -r9xr-xr-x # root root )#"#. !(()-#!-(" (/:"/ 6bin6c7

Auoting
&ow that we<#e see how ma y ways the shell ca per+orm e3pa sio s, it<s time to lear how we ca co trol it. Take +or e3ample,
[me@linuxbox ~]$ echo this is a this is a test test

or,
[me@linuxbox ~]$ echo +he total is 31**.** The total is ((<((

9 the +irst e3ample, word-splitting %y the shell remo#e! e3tra whitespace +rom the echo comma !<s list o+ arg"me ts. 9 the seco ! e3ample, parameter e3pa sio s"%stit"te! a empty stri g +or the #al"e o+ ?[.@ %eca"se it was a " !e+i e! #aria%le. The shell pro#i!es a mecha ism calle! <uoting to selecti#ely s"ppress " wa te! e3pa sio s.

)oub e Duotes
The +irst type o+ ="oti g we will look at is double <uotes. 9+ yo" place te3t i si!e !o"%le ="otes, all the special characters "se! %y the shell lose their special mea i g a ! are /9

8 8 Seei g The Worl! $s The Shell Sees 9t treate! as or!i ary characters. The e3ceptio s are ?[@, ?\@ E%ackslashF, a ! ?]@ E%ack="oteF. This mea s that wor!-splitti g, path ame e3pa sio , til!e e3pa sio , a ! %race e3pa sio are s"ppresse!, %"t parameter e3pa sio , arithmetic e3pa sio , a ! comma ! s"%stit"tio are still carrie! o"t. )si g !o"%le ="otes, we ca cope with +ile ames co tai i g em%e!!e! spaces. Say we were the " +ort" ate #ictim o+ a +ile calle! t9o 9ords<txt. 9+ we trie! to "se this o the comma ! li e, wor!-splitti g wo"l! ca"se this to %e treate! as two separate arg"me ts rather tha the !esire! si gle arg"me t,
[me@linuxbox ~]$ ls -l two words.txt ls: cannot access t9o: Lo such file or director1 ls: cannot access 9ords<txt: Lo such file or director1

Jy "si g !o"%le ="otes, we stop the wor!-splitti g a ! get the !esire! res"ltH +"rther, we ca e#e repair the !amage,
[me@linuxbox ~]$ ls -l Jtwo words.txtJ -r9-r9-r-- # me me #/ !((/-(!-!( #$:($ t9o 9ords<txt [me@linuxbox ~]$ mv Jtwo words.txtJ two.words.txt

ThereL &ow we !o <t ha#e to keep typi g those pesky !o"%le ="otes. :emem%er, parameter e3pa sio , arithmetic e3pa sio , a ! comma ! s"%stit"tio still take place withi !o"%le ="otes,
[me@linuxbox ~]$ echo J3EFGH 34425266 34cal6J me % -ebruar1 !((/ *u +o Tu ,e Th -r *a # ! $ % " . ) / 0 #( ## #! #$ #% #" #. #) #/ #0 !( !# !! !$ !% !" !. !) !/ !0

We sho"l! take a mome t to look at the e++ect o+ !o"%le ="otes o comma ! s"%stit"tio . 0irst let<s look a little !eeper at how wor! splitti g works. 9 o"r earlier e3ample, we saw how wor!-splitti g appears to remo#e e3tra spaces i o"r te3t,
[me@linuxbox ~]$ echo this is a this is a test test

80

T"oti g Jy !e+a"lt, wor!-splitti g looks +or the prese ce o+ spaces, ta%s, a ! ewli es Eli e+ee! charactersF a ! treats them as delimiters %etwee wor!s. This mea s that " ="ote! spaces, ta%s, a ! ewli es are ot co si!ere! to %e part o+ the te3t. They o ly ser#e as separators. Si ce they separate the wor!s i to !i++ere t arg"me ts, o"r e3ample comma ! li e co tai s a comma ! +ollowe! %y +o"r !isti ct arg"me ts. 9+ we a!! !o"%le ="otes,
[me@linuxbox ~]$ echo Jthis is a this is a test testJ

wor!-splitti g is s"ppresse! a ! the em%e!!e! spaces are ot treate! as !elimiters, rather they %ecome part o+ the arg"me t. 7 ce the !o"%le ="otes are a!!e!, o"r comma ! li e co tai s a comma ! +ollowe! %y a si gle arg"me t. The +act that ewli es are co si!ere! !elimiters %y the wor!-splitti g mecha ism ca"ses a i teresti g, al%eit s"%tle, e++ect o comma ! s"%stit"tio . Co si!er the +ollowi g,
[me@linuxbox ~]$ echo 34cal6 -ebruar1 !((/ *u +o Tu ,e Th -r *a # ! $ % " . ) / 0 #( ## #! #$ #% #" #. #) #/ #0 !( !# !! !$ !% !" !. !) !/ !0 [me@linuxbox ~]$ echo J34cal6J -ebruar1 !((/ *u +o Tu ,e Th -r *a # ! $ % " . ) / 0 #( ## #! #$ #% #" #. #) #/ #0 !( !# !! !$ !% !" !. !) !/ !0

9 the +irst i sta ce, the " ="ote! comma ! s"%stit"tio res"lte! i a comma ! li e co tai i g thirty-eight arg"me ts. 9 the seco !, a comma ! li e with o e arg"me t that i cl"!es the em%e!!e! spaces a ! ewli es.

Sing e Duotes
9+ we ee! to s"ppress all e3pa sio s, we "se single <uotes. Aere is a compariso o+ " ="ote!, !o"%le ="otes, a ! si gle ="otes,
[me@linuxbox ~]$ echo text ~/*.txt 9a;b> 34echo foo6 34425266 3EFGH text 6home6me6ls-out7ut<txt a b foo % me

8.

8 8 Seei g The Worl! $s The Shell Sees 9t

[me@linuxbox text ~6E<txt [me@linuxbox text ~6E<txt

~]$ echo Jtext ~/*.txt 9a;b> 34echo foo6 34425266 3EFGHJ [a>b\ foo % me ~]$ echo %text ~/*.txt 9a;b> 34echo foo6 34425266 3EFGH% [a>b\ $Necho fooP $NN!8!PP $3*&R

$s we ca see, with each s"ccee!i g le#el o+ ="oti g, more a ! more o+ the e3pa sio s are s"ppresse!.

6s/a5ing Chara/ters
Sometimes we o ly wa t to ="ote a si gle character. To !o this, we ca prece!e a character with a %ackslash, which i this co te3t is calle! the escape character. 7+te this is !o e i si!e !o"%le ="otes to selecti#ely pre#e t a e3pa sio ,
[me@linuxbox ~]$ echo J+he balance for user 3EFGH is2 K3".**J The balance for user me is: $"<((

9t is also commo to "se escapi g to elimi ate the special mea i g o+ a character i a +ile ame. 0or e3ample, it is possi%le to "se characters i +ile ames that ormally ha#e special mea i g to the shell. These wo"l! i cl"!e ?[@, ?L@, ?^@, ? ?, a ! others. To i cl"!e a special character i a +ile ame yo" ca to this,
[me@linuxbox ~]$ mv badK'filename good.filename

To allow a %ackslash character to appear, escape it %y typi g ?\\@. &ote that withi si gle ="otes, the %ackslash loses its special mea i g a ! is treate! as a or!i ary character.

/ac(slash $scape SeBuences

9 a!!itio to its role as the escape character, the %ackslash is also "se! as part o+ a otatio to represe t certai special characters calle! control codes. The +irst thirty-two characters i the $SC99 co!i g scheme are "se! to tra smit comma !s to teletype-like !e#ices. Some o+ these co!es are +amiliar Eta%, %ackspace, li e+ee!, a ! carriage ret"r F, while others are ot E "ll, e !-o+-tra smissio , a ! ack owle!geF.

82

T"oti g

$scape SeBuence ]a ]b ]n ]r ]t

>eaning Jell E?$lert@ - ca"ses the comp"ter to %eepF Jackspace &ewli e. 7 ) i3-like systems, this pro!"ces a li e+ee!. Carriage ret"r Ta%

The ta%le a%o#e lists some o+ the commo %ackslash escape se="e ces. The i!ea %ehi ! this represe tatio "si g the %ackslash origi ate! i the C programmi g la g"age a ! has %ee a!opte! %y ma y others, i cl"!i g the shell. $!!i g the ?-e@ optio to echo will e a%le i terpretatio o+ escape se="e ces. 5o" may also place them i si!e $K K. Aere, "si g the slee7 comma !, a simple program that 4"st waits +or the speci+ie! "m%er o+ seco !s a ! the e3its, we ca create a primiti#e co" t!ow timer,
sleep 1*# echo -e J+ime%s upKaJ

We co"l! also !o this,

sleep 1*# echo J+ime%s upJ 3%Ka%

Summing -p
$s we mo#e +orwar! with "si g the shell, we will +i ! that e3pa sio s a ! ="oti g will %e "se! with i creasi g +re="e cy, so it makes se se to get a goo! " !ersta !i g o+ the way they works. 9 +act, it co"l! %e arg"e! that they are the most importa t s"%4ects to lear a%o"t the shell. Witho"t a proper " !ersta !i g o+ e3pa sio , the shell will always %e a so"rce o+ mystery a ! co +"sio , a ! m"ch o+ it pote tial power waste!.

;urther 1eading

The bash ma page has ma4or sectio s o %oth e3pa sio a ! ="oti g which co#er these topics i a more +ormal ma er. The 8ash 9eference !anual also co tai s chapters o e3pa sio a ! ="oti g, http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.html

8(

9 8 $!#a ce! Mey%oar! Tricks

0 )dvanced 1e$board Tric,s

9 o+te ki!!i gly !escri%e ) i3 as ?the operati g system +or people who like to type.@ 7+ co"rse, the +act that it e#e has a comma ! li e is a testame t to that. J"t comma ! li e "sers !o <t like to type that m"ch. Why else wo"l! so ma y comma !s ha#e s"ch short ames like c7, ls, mv, a ! rmC 9 +act, o e o+ the most cherishe! goals o+ the comma ! li e is laDi essH !oi g the most work with the +ewest "m%er o+ keystrokes. $ other goal is e#er ha#i g to li+t yo"r +i gers +rom the key%oar!, e#er reachi g +or the mo"se. 9 this chapter, we will look at bash +eat"res that make key%oar! "se +aster a ! more e++icie t. The +ollowi g comma !s will make a appeara ce,

clear 8 Clear the scree histor1 8 'isplay the co te ts o+ the history list

Command Line $diting

bash "ses a li%rary Ea share! collectio o+ ro"ti es that !i++ere t programs ca "seF calle! 9eadline to impleme t comma ! li e e!iti g. We ha#e alrea!y see some o+ this. We k ow, +or e3ample, that the arrow keys mo#e the c"rsor %"t there are ma y more +eat"res. Thi k o+ these as a!!itio al tools that we ca employ i o"r work. 9tIs ot importa t to lear all o+ them, %"t ma y o+ them are #ery "se+"l. 6ick a ! choose as !esire!. 1ote5 Some o+ the key se="e ces %elow Epartic"larly those which "se the 4lt keyF may %e i tercepte! %y the ;)9 +or other +" ctio s. $ll o+ the key se="e ces sho"l! work properly whe "si g a #irt"al co sole.

Cursor :o;ement
The +ollowi g ta%le lists the keys "se! to mo#e the c"rsor,

81

Comma ! *i e E!iti g Table 5-1: %ursor !ovement %ommands =e, Ctrl-a Ctrl-e Ctrl-f Ctrl-b 4lt-f 4lt-b Ctrl-l #ction >o#e c"rsor to the %egi i g o+ the li e. >o#e c"rsor to the e ! o+ the li e. >o#e c"rsor +orwar! o e characterH same as the right arrow key. >o#e c"rsor %ackwar! o e characterH same as the le+t arrow key. >o#e c"rsor +orwar! o e wor!. >o#e c"rsor %ackwar! o e wor!. Clear the scree a ! mo#e the c"rsor to the top le+t cor er. The clear comma ! !oes the same thi g.

:odi=ying Text
Ta%le 9-2 lists key%oar! comma !s that are "se! to e!it characters o the comma ! li e. Table 5-': Te+t 1diting %ommands =e, Ctrl-d Ctrl-t 4lt-t 4lt-l 4lt-u #ction 'elete the character at the c"rsor locatio Tra spose Ee3cha geF the character at the c"rsor locatio with the o e prece!i g it. Tra spose the wor! at the c"rsor locatio with the o e prece!i g it. Co #ert the characters +rom the c"rsor locatio to the e ! o+ the wor! to lowercase. Co #ert the characters +rom the c"rsor locatio to the e ! o+ the wor! to "ppercase.

Cutting And 'asting E9i ing And 1ankingF Text

The :ea!li e !oc"me tatio "ses the terms killing a ! yanking to re+er to what we wo"l! commo ly call c"tti g a ! pasti g. 9tems that are c"t are store! i a %"++er calle! the kill-ring.

82

9 8 $!#a ce! Mey%oar! Tricks Table 5-3: %ut )nd 6aste %ommands =e, Ctrl-k Ctrl-u 4lt-d 4ltHacks7ace Ctrl-1 #ction Mill te3t +rom the c"rsor locatio to the e ! o+ li e. Mill te3t +rom the c"rsor locatio to the %egi i g o+ the li e. Mill te3t +rom the c"rsor locatio to the e ! o+ the c"rre t wor!. Mill te3t +rom the c"rsor locatio to the %egi i g o+ the c"rre t wor!. 9+ the c"rsor is at the %egi i g o+ a wor!, kill the pre#io"s wor!. 5a k te3t +rom the kill-ri g a ! i sert it at the c"rsor locatio .

The >eta =e,

9+ yo" #e t"re i to the :ea!li e !oc"me tatio , which ca %e +o" ! i the :E$'*9&E sectio o+ the bash ma page, yo" will e co" ter the term ?meta key.@ 7 mo!er key%oar!s this maps to the 4lt key %"t it was <t always so. Jack i the !im times E%e+ore 6Cs %"t a+ter ) i3F ot e#ery%o!y ha! their ow comp"ter. What they might ha#e ha! was a !e#ice calle! a terminal. $ termi al was a comm" icatio !e#ice that +eat"re! a te3t !isplay scree a ! a key%oar! a ! 4"st e o"gh electro ics i si!e to !isplay te3t characters a ! mo#e the c"rsor aro" !. 9t was attache! E"s"ally %y serial ca%leF to a larger comp"ter or the comm" icatio etwork o+ a larger comp"ter. There were ma y !i++ere t %ra !s o+ termi als a ! they all ha! !i++ere t key%oar!s a ! !isplay +eat"re sets. Si ce they all te !e! to at least " !ersta ! $SC99, so+tware !e#elopers wa ti g porta%le applicatio s wrote to the lowest commo !e omi ator. ) i3 systems ha#e a #ery ela%orate way o+ !eali g with termi als a ! their !i++ere t !isplay +eat"res. Si ce the !e#elopers o+ :ea!li e co"l! ot %e s"re o+ the prese ce o+ a !e!icate! e3tra co trol key, they i #e te! o e a ! calle! it ?meta.@ While the 4lt key ser#es as the meta key o mo!er key%oar!s, yo" ca also press a ! release the &sc key to get the same e++ect as hol!i g !ow the 4lt key i+ yo"<re still "si g a termi al Ewhich yo" ca still !o i *i "3LF.

Completion
$ other way that the shell ca help yo" is thro"gh a mecha ism calle! completion. Completio occ"rs whe yo" press the ta% key while typi g a comma !. *et<s see how 8K

Completio this works. ;i#e a home !irectory that looks like this,
[me@linuxbox ~]$ ls 'eskto7 ls-out7ut<txt 'ocuments +usic

:ictures :ublic

Tem7lates

;ideos

Try typi g the +ollowi g %"t don7t (ress the Gnter key,
[me@linuxbox ~]$ ls l

&ow press the ta% key,

[me@linuxbox ~]$ ls ls-output.txt

See how the shell complete! the li e +or yo"C *et<s try a other o e. $gai , !o <t press &nter,
[me@linuxbox ~]$ ls 1

6ress ta%,
[me@linuxbox ~]$ ls 1

&o completio , 4"st a %eep. This happe e! %eca"se ?'@ matches more tha o e e try i the !irectory. 0or completio to %e s"ccess+"l, the ?cl"e@ yo" gi#e it has to %e " am%ig"o"s. 9+ we go +"rther,
[me@linuxbox ~]$ ls 1o

The press ta%,

[me@linuxbox ~]$ ls 1ocuments

8/

9 8 $!#a ce! Mey%oar! Tricks The completio is s"ccess+"l. While this e3ample shows completio o+ path ames, which is its most commo "se, completio will also work o #aria%les Ei+ the %egi i g o+ the wor! is a ?[@F, "ser ames Ei+ the wor! %egi s with ?S@F, comma !s Ei+ the wor! is the +irst wor! o the li e.F a ! host ames Ei+ the %egi i g o+ the wor! is ?N@F. Aost ame completio o ly works +or host ames liste! i 6etc6hosts. There are a completio , "m%er o+ co trol a ! meta key se="e ces that are associate! with

Table 5-$: %ompletion %ommands =e, 4lt-F 4lt-E #ction 'isplay list o+ possi%le completio s. &n most systems you can also do this by pressing the tab key a second time" which is much easier# 9 sert all possi%le completio s. This is "se+"l whe yo" wa t to "se more tha o e possi%le match.

There ="ite a +ew more that 9 +i ! rather o%sc"re. 5o" ca see a list i the bash ma page " !er ?:E$'*9&E@.

Programma0le Completion
:ece t #ersio s o+ bash ha#e a +acility calle! programmable completion. 6rogramma%le completio allows yo" Eor more likely, yo"r !istri%"tio pro#i!erF to a!! a!!itio al completio r"les. )s"ally this is !o e to a!! s"pport +or speci+ic applicatio s. 0or e3ample it is possi%le to a!! completio s +or the optio list o+ a comma ! or match partic"lar +ile types that a applicatio s"pports. )%" t" has a +airly large set !e+i e! %y !e+a"lt. 6rogramma%le completio is impleme te! %y shell +" ctio s, a ki ! o+ mi i shell script that we will co#er i later chapters. 9+ yo" are c"rio"s, try,
set 0 less

a ! see i+ yo" ca +i ! them. &ot all !istri%"tio s i cl"!e them %y !e+a"lt.

-sing 3istor,
$s we !isco#ere! i Chapter 2, bash mai tai s a history o+ comma !s that ha#e %ee e tere!. This list o+ comma !s is kept i yo"r home !irectory i a +ile calle! 88

)si g Aistory <bashOhistor1. The history +acility is a "se+"l reso"rce +or re!"ci g the amo" t o+ typi g yo" ha#e to !o, especially whe com%i e! with comma ! li e e!iti g.

Sear/hing $istory
$t a y time, we ca #iew the co te ts o+ the history list %y,
[me@linuxbox ~]$ histor! 0 less

Jy !e+a"lt, %ash stores the last +i#e h" !re! comma !s yo" ha#e e tere!. We will see how to a!4"st this #al"e i a later chapter. *et<s say we wa t to +i ! the comma !s we "se! to list 6usr6bin. 7 e way we co"l! !o this,
[me@linuxbox ~]$ histor! 0 grep /usr/bin

$ ! let<s say that amo g o"r res"lts we got a li e co tai i g a i teresti g comma ! like this, // ls -l 6usr6bin D ls-out7ut<txt The "m%er ?88@ is the li e "m%er o+ the comma ! i the history list. We co"l! "se this imme!iately "si g a other type o+ e3pa sio calle! history e+pansion. To "se o"r !isco#ere! li e we co"l! !o this,
[me@linuxbox ~]$ LMM

bash will e3pa ! ?L88@ i to the co te ts o+ the eighty-eighth li e i the history list. There are other +orms o+ history e3pa sio that we will co#er a little later. bash also pro#i!es the a%ility to search the history list i creme tally. This mea s that we ca tell bash to search the history list as we e ter characters, with each a!!itio al character +"rther re+i i g o"r search. To start i creme tal search type Ctrl-r +ollowe! %y the te3t yo" are looki g +or. Whe yo" +i ! it, yo" ca either type &nter to e3ec"te the comma ! or type Ctrl-j to copy the li e +rom the history list to the c"rre t comma ! li e. To +i ! the e3t occ"rre ce o+ the te3t Emo#i g ?"p@ the history listF, type Ctrl-r agai . To ="it searchi g, type either Ctrl-g or Ctrl-c. Aere we see it i actio ,

89

9 8 $!#a ce! Mey%oar! Tricks

[me@linuxbox ~]$

0irst type Ctrl-r,

Nreverse-i-searchPJK:

The prompt cha ges to i !icate that we are per+ormi g a re#erse i creme tal search. 9t is ?re#erse@ %eca"se we are searchi g +rom ? ow@ to some time i the past. &e3t, we start typi g o"r search te3t. 9 this e3ample ?-"sr-%i @,
Nreverse-i-searchPJ/usr/binK: ls -l 6usr6bin D ls-out7ut<txt

9mme!iately, the search ret"r s o"r res"lt. With o"r res"lt, we ca e3ec"te the comma ! %y pressi g &nter, or we ca copy the comma ! to o"r c"rre t comma ! li e +or +"rther e!iti g %y typi g Ctrl-j. *et<s copy it. Type Ctrl-j,
[me@linuxbox ~]$ ls -l /usr/bin & ls-output.txt

7"r shell prompt ret"r s a ! o"r comma ! li e is loa!e! a ! rea!y +or actio L The ta%le %elow lists some o+ the keystrokes "se! to ma ip"late the history list, Table 5-.: ,istory %ommands =e, Ctrl-7 Ctrl-n 4lt-X 4lt-D Ctrl-r 4lt-7 #ction >o#e to the pre#io"s history e try. Same actio as the "p arrow. >o#e to the e3t history e try. Same actio as the !ow arrow. >o#e to the %egi i g EtopF o+ the history list. >o#e to the e ! E%ottomF o+ the history list, i.e., the c"rre t comma ! li e. :e#erse i creme tal search. Searches i creme tally +rom the c"rre t comma ! li e "p the history list. :e#erse search, o -i creme tal. With this key, type i the search stri g a ! press e ter %e+ore the search is per+orme!.

90

)si g Aistory 4lt-n Ctrl-o 0orwar! search, o -i creme tal. E3ec"te the c"rre t item i the history list a ! a!#a ce to the e3t o e. This is ha !y i+ yo" are tryi g to re-e3ec"te a se="e ce o+ comma !s i the history list.

$istory 6x5ansion
The shell o++ers a specialiDe! type o+ e3pa sio +or items i the history list %y "si g the ?L@ character. We ha#e alrea!y see how the e3clamatio poi t ca %e +ollowe! %y a "m%er to i sert a e try +rom the history list. There are a "m%er o+ other e3pa sio +eat"res, Table 5-2: ,istory 1+pansion %ommands SeBuence GG Gnumber Gstring GFstring #ction :epeat the last comma !. 9t is pro%a%ly easier to press "p arrow a ! e ter. :epeat history list item number. :epeat last history list item starti g with stri g. :epeat last history list item co tai i g stri g.

9 wo"l! ca"tio agai st "si g the ?Lstri g@ a ! ?LCstri g@ +orms " less yo" are a%sol"tely s"re o+ the co te ts o+ the history list items. There are ma y more eleme ts a#aila%le i the history e3pa sio mecha ism, %"t this s"%4ect is alrea!y too arca e a ! o"r hea!s may e3plo!e i+ we co ti "e. The A9ST7:5 EG6$&S97& sectio o+ the bash ma page goes i to all the gory !etails. 0eel +ree to e3ploreL

script
9 a!!itio to the comma ! history +eat"re i bash, most *i "3 !istri%"tio s i cl"!e a program calle! scri7t that ca %e "se! to recor! a e tire shell sessio a ! store it i a +ile. The %asic sy ta3 o+ the comma ! is, scri7t [file]

9.

9 8 $!#a ce! Mey%oar! Tricks

where file is the ame o+ the +ile "se! +or stori g the recor!i g. 9+ o +ile is speci+ie!, the +ile t17escri7t is "se!. See the scri7t ma page +or a complete list o+ the programIs optio s a ! +eat"res.

Summing -p
9 this chapter we ha#e co#ere! some o+ the key%oar! tricks that the shell pro#i!es to help har!core typists re!"ce their workloa!s. 9 s"spect that as time goes %y a ! yo" %ecome more i #ol#e! with the comma ! li e, yo" will re+er %ack to this chapter to pick "p more o+ these tricks. 0or ow, co si!er them optio al a ! pote tially help+"l.

;urther 1eading

The Wikipe!ia has a goo! article o comp"ter termi als, http,--e .wikipe!ia.org-wiki-Comp"terOtermi al

92

.0 8 6ermissio s

10 2er%issions
7perati g systems i the ) i3 tra!itio !i++er +rom those i the >S-'7S tra!itio that they are ot o ly multitasking systems, %"t also multi-user systems, as well. i

What e3actly !oes this mea C 9t mea s that more tha o e perso ca %e "si g the comp"ter at the same time. While a typical comp"ter will likely ha#e o ly o e key%oar! a ! mo itor, it ca still %e "se! %y more tha o e "ser. 0or e3ample, i+ a comp"ter is attache! to a etwork or the 9 ter et, remote "sers ca log i #ia ssh Esec"re shellF a ! operate the comp"ter. 9 +act, remote "sers ca e3ec"te graphical applicatio s a ! ha#e the graphical o"tp"t appear o a remote !isplay. The G Wi !ow System s"pports this as part o+ its %asic !esig . The m"lti-"ser capa%ility o+ *i "3 is ot a rece t Qi o#atio ,Q %"t rather a +eat"re that is !eeply em%e!!e! i to the !esig o+ the operati g system. Co si!eri g the e #iro me t i which ) i3 was create!, this makes per+ect se se. 5ears ago, %e+ore comp"ters were Qperso al,Q they were large, e3pe si#e, a ! ce traliDe!. $ typical " i#ersity comp"ter system, +or e3ample, co siste! o+ a large ce tral comp"ter locate! i o e %"il!i g a ! termi als which were locate! thro"gho"t the camp"s, each co ecte! to the large ce tral comp"ter. The comp"ter wo"l! s"pport ma y "sers at the same time. 9 or!er to make this practical, a metho! ha! to %e !e#ise! to protect the "sers +rom each other. $+ter all, the actio s o+ o e "ser co"l! ot %e allowe! to crash the comp"ter, or co"l! o e "ser i ter+ere with the +iles %elo gi g to a other "ser. 9 this chapter we are goi g to look at this esse tial part o+ system sec"rity a ! i tro!"ce the +ollowi g comma !s,

id 8 'isplay "ser i!e tity chmod 8 Cha ge a +ile<s mo!e umask 8 Set the !e+a"lt +ile permissio s su 8 :" a shell as a other "ser sudo 8 E3ec"te a comma ! as a other "ser cho9n 8 Cha ge a +ile<s ow er

9(

.0 8 6ermissio s

chgr7 8 Cha ge a +ile<s gro"p ow ership 7ass9d 8 Cha ge a "ser<s passwor!

?4ners@ 6roup >em0ers@ #nd $%er,0od, $lse

Whe we were e3plori g the system %ack i Chapter 1, we may ha#e e co" tere! a pro%lem whe tryi g to e3ami e a +ile s"ch as 6etc6shado9,
[me@linuxbox 6etc6shado9: [me@linuxbox 6etc6shado9: ~]$ file /etc/shadow regular file> no read 7ermission ~]$ less /etc/shadow :ermission denied

The reaso +or this error message is that, as reg"lar "sers, we !o ot ha#e permissio to rea! this +ile. 9 the ) i3 sec"rity mo!el, a "ser may own +iles a ! !irectories. Whe a "ser ow s a +ile or !irectory, the "ser has co trol o#er its access. )sers ca , i t"r , %elo g to a group co sisti g o+ o e or more "sers who are gi#e access to +iles a ! !irectories %y their ow ers. 9 a!!itio to gra ti g access to a gro"p, a ow er may also gra t some set o+ access rights to e#ery%o!y, which i ) i3 terms is re+erre! to as the world. To +i ! o"t i +ormatio a%o"t yo"r i!e tity, "se the id comma !,
[me@linuxbox ~]$ id uidM"((NmeP gidM"((NmeP grou7sM"((NmeP

*et<s look at the o"tp"t. Whe "ser acco" ts are create!, "sers are assig e! a "m%er calle! a user *- or uid which is the , +or the sake o+ the h"ma s, mappe! to a "ser ame. The "ser is assig e! a primary group *- or gid a ! may %elo g to a!!itio al gro"ps. The a%o#e e3ample is +rom a 0e!ora system. 7 other systems, s"ch as )%" t", the o"tp"t may look a little !i++ere t,
[me@linuxbox ~]$ id uidM#(((NmeP gidM#(((NmeP grou7sM%NadmP>!(NdialoutP>!%NcdromP>!"Nflo771P>!0NaudioP>$(Ndi7P>%%Nv ideoP>%.N7lugdevP>#(/Nl7adminP>##%NadminP>#(((NmeP

$s we ca see, the "i! a ! gi! "m%ers are !i++ere t. This is simply %eca"se 0e!ora starts its "m%eri g o+ reg"lar "ser acco" ts at 200, while )%" t" starts at .000. We ca 91

7w ers, ;ro"p >em%ers, $ ! E#ery%o!y Else also see that the )%" t" "ser %elo gs to a lot more gro"ps. This has to !o with the way )%" t" ma ages pri#ileges +or system !e#ices a ! ser#ices. So where !oes this i +ormatio come +romC *ike so ma y thi gs i *i "3, +rom a co"ple o+ te3t +iles. )ser acco" ts are !e+i e! i the 6etc67ass9d +ile a ! gro"ps are !e+i e! i the 6etc6grou7 +ile. Whe "ser acco" ts a ! gro"ps are create!, these +iles are mo!i+ie! alo g with 6etc6shado9 which hol!s i +ormatio a%o"t the "ser<s passwor!. 0or each "ser acco" t, the 6etc67ass9d +ile !e+i es the "ser Elogi F ame, "i!, gi!, the acco" t<s real ame, home !irectory, a ! logi shell. 9+ yo" e3ami e the co te ts o+ 6etc67ass9d a ! 6etc6grou7, yo" will otice that %esi!es the reg"lar "ser acco" ts, there are acco" ts +or the s"per"ser E"i! 0F a ! #ario"s other system "sers. 9 the e3t chapter, whe we co#er processes, yo" will see that some o+ these other ?"sers@ are, i +act, ="ite %"sy. While ma y ) i3-like systems assig reg"lar "sers to a commo gro"p s"ch as ?"sers@, mo!er *i "3 practice is to create a " i="e, si gle-mem%er gro"p with the same ame as the "ser. This makes certai types o+ permissio assig me t easier.

1eading@ +riting@ #nd $xecuting

$ccess rights to +iles a ! !irectories are !e+i e! i terms o+ rea! access, write access, a ! e3ec"tio access. 9+ we look at the o"tp"t o+ the ls comma !, we ca get some cl"e as to how this is impleme te!,
[me@linuxbox ~]$ & foo.txt [me@linuxbox ~]$ ls -l foo.txt -r9-r9-r-- # me me ( !((/-($-(. #%:"! foo<txt

The +irst te characters o+ the listi g are the file attributes. The +irst o+ these characters is the file type. Aere are the +ile types yo" are most likely to see Ethere are other, less commo types tooF, Table 1=-1: File Types #ttri0ute d l ;ile T,pe $ reg"lar +ile. $ !irectory. $ sym%olic li k. &otice that with sym%olic li ks, the remai i g +ile attri%"tes are always ?rw3rw3rw3@ a ! are !"mmy #al"es. The real +ile attri%"tes are those o+ the +ile the sym%olic li k poi ts to.

92

.0 8 6ermissio s c b $ character special file. This +ile type re+ers to a !e#ice that ha !les !ata as a stream o+ %ytes, s"ch as a termi al or mo!em. $ block special file. This +ile type re+ers to a !e#ice that ha !les !ata i %locks, s"ch as a har! !ri#e or C'-:7> !ri#e.

The remai i g i e characters o+ the +ile attri%"tes, calle! the file mode" represe t the rea!, write, a ! e3ec"te permissio s +or the +ile<s ow er, the +ile<s gro"p ow er, a ! e#ery%o!y else, ?4ner 6roup +orld

r9x

r9x

r9x
+iles a !

Whe set, the r, 9, a ! x mo!e attri%"tes ha#e the +ollowi g e++ect o !irectories, Table 1=-': 6ermission )ttributes #ttri0ute r ;iles $llows a +ile to %e ope e! a ! rea!. $llows a +ile to %e writte to or tr" cate!, howe#er this attri%"te !oes ot allow +iles to %e re ame! or !elete!. The a%ility to !elete or re ame +iles is !etermi e! %y !irectory attri%"tes. $llows a +ile to %e treate! as a program a ! e3ec"te!. 6rogram +iles writte i scripti g la g"ages m"st also %e set as rea!a%le to %e e3ec"te!. Directories

$llows a !irectory<s co te ts to %e liste! i+ the e3ec"te attri%"te is also set. $llows +iles withi a !irectory to %e create!, !elete!, a ! re ame! i+ the e3ec"te attri%"te is also set.

$llows a !irectory to %e e tere!, e.g., c! directory.

Aere are some e3amples o+ +ile attri%"te setti gs,

9K

:ea!i g, Writi g, $ ! E3ec"ti g Table 1=-3: 6ermission )ttribute 1+amples ;ile #ttri0utes -r9x------r9-------r9-r--r->eaning $ reg"lar +ile that is rea!a%le, writa%le, a ! e3ec"ta%le %y the +ile<s ow er. &o o e else has a y access. $ reg"lar +ile that is rea!a%le a ! writa%le %y the +ile<s ow er. &o o e else has a y access. $ reg"lar +ile that is rea!a%le a ! writa%le %y the +ile<s ow er. >em%ers o+ the ow er gro"p may rea! the +ile. The +ile is worl!-rea!a%le. $ reg"lar +ile that is rea!a%le, writa%le, a ! e3ec"ta%le %y the +ile<s ow er. The +ile may %e rea! a ! e3ec"te! %y e#ery%o!y else. $ reg"lar +ile that is rea!a%le a ! writa%le %y the +ile<s ow er a ! mem%ers o+ the +ile<s gro"p ow er o ly. $ sym%olic li k. $ll sym%olic li ks ha#e ?!"mmy@ permissio s. The real permissio s are kept with the act"al +ile poi te! to %y the sym%olic li k. $ !irectory. The ow er a ! the mem%ers o+ the ow er gro"p may e ter the !irectory, create, re ame a ! remo#e +iles withi the !irectory. $ !irectory. The ow er may e ter the !irectory a ! create, re ame a ! !elete +iles withi the !irectory. >em%ers o+ the ow er gro"p may e ter the !irectory %"t ca ot create, !elete or re ame +iles.

-r9xr-xr-x

-r9-r9---lr9xr9xr9x

dr9xr9x---

dr9xr-x---

chmod A Change =i e mode

To cha ge the mo!e Epermissio sF o+ a +ile or !irectory, the chmod comma ! is "se!. Je aware that o ly the +ileIs ow er or the s"per"ser ca cha ge the mo!e o+ a +ile or !irectory. chmod s"pports two !isti ct ways o+ speci+yi g mo!e cha ges, octal "m%er represe tatio , or sym%olic represe tatio . We will co#er octal "m%er represe tatio +irst.

9/

.0 8 6ermissio s

+hat The 3ec( Is ?ctal.

&ctal E%ase 8F, a ! its co"si , he+adecimal E%ase .KF are "m%er systems o+te "se! to e3press "m%ers o comp"ters. We h"ma s, owi g to the +act that we Eor at least most o+ "sF were %or with te +i gers, co" t "si g a %ase .0 "m%er system. Comp"ters, o the other the other ha !, were %or with o ly o e +i ger a ! th"s !o all all their co" ti g i binary E%ase 2F. Their "m%er system o ly has two "merals, Dero a ! o e. So i %i ary, co" ti g looks like this, 0, ., .0, .., .00, .0., ..0, ..., .000, .00., .0.0, .0..... 9 octal, co" ti g is !o e with the "merals Dero thro"gh se#e , like so, 0, ., 2, (, 1, 2, K, /, .0, .., .2, .(, .1, .2, .K, ./, 20, 2.... Ae3a!ecimal co" ti g "ses the "merals Dero thro"gh i e pl"s the letters ?$@ thro"gh ?0@, 0, ., 2, (, 1, 2, K, /, 8, 9, $, J, C, ', E, 0, .0, .., .2, .(... While we ca see the se se i %i ary Esi ce comp"ters o ly ha#e o e +i gerF, what are octal a ! he3a!ecimal goo! +orC The a swer has to !o with h"ma co #e ie ce. >a y times, small portio s o+ !ata are represe te! o comp"ters as bit patterns. Take +or e3ample a :;J color. 7 most comp"ter !isplays, each pi3el is compose! o+ three color compo e ts, eight %its o+ re!, eight %its o+ gree , a ! eight %its o+ %l"e. $ lo#ely me!i"m %l"e wo"l! %e a twe ty-+o"r !igit "m%er, 0.0000..0..0......00..0. Aow wo"l! yo" like to rea! a ! write those ki !s o+ "m%ers all !ayC 9 !i! <t thi k so. Aere<s where a other "m%er system wo"l! help. Each !igit i a he3a!ecimal "m%er represe ts +o"r !igits i %i ary. 9 octal, each !igit represe ts three %i ary !igits. So o"r twe ty-+o"r !igit me!i"m %l"e co"l! %e co !e se! !ow to a si3 !igit he3a!ecimal "m%er, 1(K0C' Si ce the !igits i the he3a!ecimal "m%er ?li e "p@ with the %its i the %i ary "m%er we ca see that the re! compo e t o+ o"r color is ?1(@, the gree ?K0@, a ! the %l"e ?C'@. These !ays, he3a!ecimal otatio Eo+te spoke as ?he3@F is more commo tha octal, %"t as we shall soo see, octal<s a%ility to e3press three %its o+ %i ary will %e #ery "se+"l... With octal otatio we "se octal "m%ers to set the patter o+ !esire! permissio s. Si ce 98

:ea!i g, Writi g, $ ! E3ec"ti g each !igit i a octal "m%er represe ts three %i ary !igits, this maps icely to the scheme "se! to store the +ile mo!e. This ta%le shows what we mea , ?ctal ( # ! $ % " . ) /inar, ((( ((# (#( (## #(( #(# ##( ### ;ile >ode ----x -9-9x r-r-x r9r9x

Jy "si g three octal !igits, we ca set the +ile mo!e +or the ow er, gro"p ow er, a ! worl!,
[me@linuxbox [me@linuxbox -r9-r9-r-- # [me@linuxbox [me@linuxbox -r9------- # ~]$ ~]$ me ~]$ ~]$ me & foo.txt ls -l foo.txt me ( !((/-($-(. #%:"! foo<txt chmod N** foo.txt ls -l foo.txt me ( !((/-($-(. #%:"! foo<txt

Jy passi g the arg"me t ?K00@, we were a%le to set the permissio s o+ the ow er to rea! a ! write while remo#i g all permissio s +rom the gro"p ow er a ! worl!. Tho"gh remem%eri g the octal to %i ary mappi g may seem i co #e ie t, yo" will "s"ally o ly ha#e to "se a +ew commo o es, / Er9xF, K Er9-F, 2 Er-xF, 1 Er--F, a ! 0 E---F. chmod also s"pports a sym%olic otatio +or speci+yi g +ile mo!es. Sym%olic otatio is !i#i!e! i to three parts, who the cha ge will a++ect, which operatio will %e per+orme!, a ! what permissio will %e set. To speci+y who is a++ecte!, a com%i atio o+ the characters ?"@, ?g@, ?o@, a ! ?a@ is "se! as +ollows, Table 1=-$: chmod Symbolic >otation S,m0ol u g >eaning Short +or ?"se,r@ %"t mea s the +ile or !irectory ow er. ;ro"p ow er.

99

.0 8 6ermissio s o a Short +or ?others,@ %"t mea s worl!. Short +or ?all.@ The com%i atio o+ ?"@, ?g@, a ! ?o@.

9+ o character is speci+ie!, ?all@ will %e ass"me!. The operatio may %e a ?V@ i !icati g that a permissio is to %e a!!e!, a ?-@ i !icati g that a permissio is to %e take away, or a ?W@ i !icati g that o ly the speci+ie! permissio s are to %e applie! a ! that all others are to %e remo#e!. 6ermissio s are speci+ie! with the ?r@, ?w@, a ! ?3@ characters. Aere are some e3amples o+ sym%olic otatio , Table 1=-.: chmod Symbolic >otation 1+amples 7otation u8x u-x 8x o-r9 goMr9 >eaning $!! e3ec"te permissio +or the ow er. :emo#e e3ec"te permissio +rom the ow er. $!! e3ec"te permissio +or the ow er, gro"p, a ! worl!. E="i#ale t to a8x. :emo#e the rea! a ! write permissio +rom a yo e %esi!es the ow er a ! gro"p ow er. Set the gro"p ow er a ! a yo e %esi!es the ow er to ha#e rea! a ! write permissio . 9+ either the gro"p ow er or worl! pre#io"sly ha! e3ec"te permissio s, they are remo#e!. $!! e3ec"te permissio +or the ow er a ! set the permissio s +or the gro"p a ! others to rea! a ! e3ec"te. >"ltiple speci+icatio s may %e separate! %y commas.

u8x>goMrx

Some people pre+er to "se octal otatio , some +olks really like the sym%olic. Sym%olic otatio !oes o++er the a!#a tage o+ allowi g yo" to set a si gle attri%"te witho"t !ist"r%i g a y o+ the others. Take a look at the chmod ma page +or more !etails a ! a list o+ optio s. $ wor! o+ ca"tio regar!i g the ?--rec"rsi#e@ optio , it acts o %oth +iles a ! !irectories, so it<s ot as "se+"l as o e wo"l! hope si ce, we rarely wa t +iles a ! !irectories to ha#e the same permissio s.

Setting 2i e :ode With The +UI

&ow that we ha#e see how the permissio s o +iles a ! !irectories are set, we ca %etter .00

:ea!i g, Writi g, $ ! E3ec"ti g " !ersta ! the permissio !ialogs i the ;)9. 9 %oth &a"til"s E;&7>EF a ! Mo ="eror EM'EF, right-clicki g a +ile or !irectory ico will e3pose a properties !ialog. Aere is a e3ample +rom M'E (.2,

Figure ': ?-1 3#. File 6roperties -ialog Aere we ca see the setti gs +or the ow er, gro"p, a ! worl!. 9 M'E, clicki g o the ?$!#a ce! 6ermissio s@ %"tto %ri gs "p a other !ialog that allows yo" to set each o+ the mo!e attri%"tes i !i#i!"ally. $ other #ictory +or " !ersta !i g %ro"ght to "s %y the comma ! li eL

umask A Set )e=au t 'ermissions

The umask comma ! co trols the !e+a"lt permissio s gi#e to a +ile whe it is create!. 9t "ses octal otatio to e3press a mask o+ %its to %e remo#e! +rom a +ile<s mo!e attri%"tes. *et<s take a look,
[me@linuxbox ~]$ rm -f foo.txt [me@linuxbox ~]$ umask (((! [me@linuxbox ~]$ & foo.txt

.0.

.0 8 6ermissio s
[me@linuxbox ~]$ ls -l foo.txt -r9-r9-r-- # me me ( !((/-($-(. #%:"$ foo<txt

We +irst remo#e! a y ol! copy o+ foo<txt to make s"re we were starti g +resh. &e3t, we ra the umask comma ! witho"t a arg"me t to see the c"rre t #al"e. 9t respo !e! with the #al"e (((! Ethe #al"e ((!! is a other commo !e+a"lt #al"eF, which is the octal represe tatio o+ o"r mask. We e3t create a ew i sta ce o+ the +ile foo<txt a ! o%ser#e its permissio s. We ca see that %oth the ow er a ! gro"p %oth get rea! a ! write permissio , while e#eryo e else o ly gets rea! permissio . The reaso that worl! !oes ot ha#e write permissio is %eca"se o+ the #al"e o+ the mask. *et<s repeat o"r e3ample, this time setti g the mask o"rsel#es,
[me@linuxbox [me@linuxbox [me@linuxbox [me@linuxbox -r9-r9-r9- # ~]$ ~]$ ~]$ ~]$ me rm foo.txt umask **** & foo.txt ls -l foo.txt me ( !((/-($-(. #%:"/ foo<txt

Whe we set the mask to (((( Ee++ecti#ely t"r i g it o++F, we see that the +ile is ow worl! writa%le. To " !ersta ! how this works, we ha#e to look at octal "m%ers agai . 9+ we take the mask a ! e3pa ! it i to %i ary, the compare it to the attri%"tes we ca see what happe s, ?riginal "ile mode >as( 1esult --- r9- r9- r9((( ((( ((( (#( --- r9- r9- r--

9g ore +or the mome t the lea!i g Deros Ewe<ll get to those i a mi "teF a ! o%ser#e that where the . appears i o"r mask, a attri%"te was remo#e!Bi this case, the the worl! write permissio . That<s what the mask !oes. E#erywhere a . appears i the %i ary #al"e o+ the mask, a attri%"te is " set. 9+ we look at a mask #al"e o+ ((!!, we ca see what it !oes, ?riginal "ile mode >as( 1esult .02 --- r9- r9- r9((( ((( (#( (#( --- r9- r-- r--

:ea!i g, Writi g, $ ! E3ec"ti g

$gai , where a . appears i the %i ary #al"e, the correspo !i g attri%"te is " set. 6lay with some #al"es Etry some se#e sF to get "se! to how this works. Whe yo"<re !o e, remem%er to clea "p,
[me@linuxbox ~]$ rm foo.txt# umask ***2

>ost o+ the time yo" wo <t ha#e to cha ge the maskH the !e+a"lt pro#i!e! %y yo"r !istri%"tio will %e +i e. 9 some high-sec"rity sit"atio s, howe#er, yo" will wa t to co trol it.

Some Special Permissions

Tho"gh we "s"ally see a octal permissio mask e3presse! as a three !igit "m%er, it is more tech ically correct to e3press it i +o"r !igits. WhyC Jeca"se, i a!!itio to rea!, write, a ! e3ec"te permissio , there are some other, less "se!, permissio setti gs. The +irst o+ these is the setuid bit Eoctal 1000F. Whe applie! to a e3ec"ta%le +ile, it sets the effective user *- +rom that o+ real "ser Ethe "ser act"ally r" i g the programF to that o+ the program<s ow er. >ost o+te this is gi#e to a +ew programs ow e! %y the s"per"ser. Whe a or!i ary "ser r" s a program that is ?setuid root@ , the program r" s with the e++ecti#e pri#ileges o+ the s"per"ser. This allows the program to access +iles a ! !irectories that a or!i ary "ser wo"l! ormally %e prohi%ite! +rom accessi g. Clearly, %eca"se this raises sec"rity co cer s, "m%er o+ set"i! programs m"st %e hel! to a a%sol"te mi im"m. The seco ! is the setgid bit Eoctal 2000F which, like the set"i! %it, cha ges the effective group *- +rom the real group *- o+ the "ser to that o+ the +ile ow er. 9+ the setgi! %it is set o a !irectory, ewly create! +iles i the !irectory will %e gi#e the gro"p ow ership o+ the !irectory rather the gro"p ow ership o+ the +ile<s creator. This is "se+"l i a share! !irectory whe mem%ers o+ a commo gro"p ee! access to all the +iles i the !irectory, regar!less o+ the +ile ow er<s primary gro"p. The thir! is calle! the sticky bit Eoctal .000F. This is a hol!o#er +rom a cie t ) i3, where it was possi%le to mark a e3ec"ta%le +ile as ? ot swappa%le.@ 7 +iles, *i "3 ig ores the sticky %it, %"t i+ applie! to a !irectory, it pre#e ts "sers +rom !eleti g or re ami g +iles " less the "ser is either the ow er o+ the !irectory,

.0(

.0 8 6ermissio s

the ow er o+ the +ile, or the s"per"ser. This is o+te "se! to co trol access to a share! !irectory, s"ch as 6tm7. Aere are some e3amples o+ "si g chmod with sym%olic otatio to set these special permissio s. 0irst assig i g set"i! to a program, chmod u8s program &e3t, assig i g setgi! to a !irectory, chmod g8s dir 0i ally, assig i g the sticky %it to a !irectory, chmod 8t dir Whe #iewi g the o"tp"t +rom ls, yo" ca !etermi e the special permissio s. Aere are some e3amples. 0irst, a program that is set"i!, -r9sr-xr-x $ !irectory that has the setgi! attri%"te, dr9xr9sr-x $ !irectory with the sticky %it set, dr9xr9xr9t

Changing Identities
$t #ario"s times, we may +i ! it ecessary to take o the i!e tity o+ a other "ser. 7+te we wa t to gai s"per"ser pri#ileges to carry o"t some a!mi istrati#e task, %"t it is also possi%le to ?%ecome@ a other reg"lar "ser +or s"ch thi gs as testi g a acco" t. There are three ways to take o a alter ate i!e tity, .. *og o"t a ! log %ack i as the alter ate "ser. 2. )se the su comma !. (. )se the sudo comma !. We will skip the +irst tech i="e si ce we k ow how to !o it a ! it lacks the co #e ie ce o+ the other two. 0rom withi o"r ow shell sessio , the su comma ! allows yo" to ass"me the i!e tity o+ a other "ser, a ! either start a ew shell sessio with that "ser<s 9's, or to iss"e a si gle comma ! as that "ser. The sudo comma ! allows a a!mi istrator to set "p a co +ig"ratio +ile calle! 6etc6sudoers, a ! !e+i e speci+ic comma !s that partic"lar "sers are permitte! to e3ec"te " !er a ass"me! i!e tity. The .01

Cha gi g 9!e tities choice o+ which comma ! to "se is largely !etermi e! %y which *i "3 !istri%"tio yo" "se. 5o"r !istri%"tio pro%a%ly i cl"!es %oth comma !s, %"t its co +ig"ratio will +a#or either o e or the other. We<ll start with su.

su A !un A She With Substitute User And +rou5 I)s

The su comma ! is "se! to start a shell as a other "ser. The comma ! sy ta3 looks like this,
su (-(l)) (user)

9+ the ?-l@ optio is i cl"!e!, the res"lti g shell sessio is a login shell +or the speci+ie! "ser. This mea s that the "ser<s e #iro me t is loa!e! a ! the worki g !irectory is cha ge! to the "ser<s home !irectory. This is "s"ally what we wa t. 9+ the "ser is ot speci+ie!, the s"per"ser is ass"me!. &otice that Estra gelyF the ?-l@ may %e a%%re#iate! ?-@, which is how it is most o+te "se!. To start a shell +or the s"per"ser, we wo"l! !o this,
[me@linuxbox ~]$ su :ass9ord: [root@linuxbox ~]^

$+ter e teri g the comma !, we are prompte! +or the s"per"ser<s passwor!. 9+ it is s"ccess+"lly e tere!, a ew shell prompt appears i !icati g that this shell has s"per"ser pri#ileges Ethe traili g ?P@ rather tha a ?[@F a ! the c"rre t worki g !irectory is ow the home !irectory +or the s"per"ser E ormally 6root.F 7 ce i the ew shell, we ca carry o"t comma !s as the s"per"ser. Whe +i ishe!, type ?e3it@ to ret"r to the pre#io"s shell,
[root@linuxbox ~]^ exit [me@linuxbox ~]$

9t is also possi%le to e3ec"te a si gle comma ! rather tha starti g a ew i teracti#e comma ! %y "si g su this way,
su -c KcommandK

.02

.0 8 6ermissio s )si g this +orm, a si gle comma ! li e is passe! to the ew shell +or e3ec"tio . 9t is importa t to e close the comma ! i ="otes, as we !o ot wa t e3pa sio to occ"r i o"r shell, %"t rather i the ew shell,
[me@linuxbox ~]$ su -c %ls -l /root/*% :ass9ord: -r9------- # root root )"% !(()-(/-## ($:#0 6root6anaconda-ks<cfg 6root6+ail: total ( [me@linuxbox ~]$

sudo A 6xe/ute A Command As Another User

The sudo comma ! is like su i ma y ways, %"t has some importa t a!!itio al capa%ilities. The a!mi istrator ca co +ig"re sudo to allow a or!i ary "ser to e3ec"te comma !s as a !i++ere t "ser E"s"ally the s"per"serF i a #ery co trolle! way. 9 partic"lar, a "ser may %e restricte! to o e or more speci+ic comma !s a ! o others. $ other importa t !i++ere ce is that the "se o+ sudo !oes ot re="ire access to the s"per"ser<s passwor!. To a"the ticate "si g sudo, the "ser "ses his-her ow passwor!. *et<s say, +or e3ample, that sudo has %ee co +ig"re! to allow "s to r" a +ictitio"s %ack"p program calle! ?%ack"pOscript@, which re="ires s"per"ser pri#ileges. With sudo it wo"l! %e !o e like this,
[me@linuxbox ~]$ sudo backup.script :ass9ord: *1stem Hacku7 *tarting<<<

$+ter e teri g the comma !, we are prompte! +or o"r passwor! E ot the s"per"ser<sF a ! o ce the a"the ticatio is complete, the speci+ie! comma ! is carrie! o"t. 7 e importa t !i++ere ce %etwee su a ! sudo is that sudo !oes ot start a ew shell, or !oes it loa! a other "ser<s e #iro me t. This mea s that comma !s !o ot ee! to %e ="ote! a y !i++ere tly tha they wo"l! %e witho"t "si g sudo. &ote that this %eha#ior ca %e o#erri!!e %y speci+yi g #ario"s optio s. See the sudo ma page +or !etails. To see what pri#ileges are gra te! %y sudo, "se the ?-l@ optio to list them,
[me@linuxbox ~]$ sudo -l 3ser me ma1 run the follo9ing commands on this host:

.0K

Cha gi g 9!e tities

N4==P 4==

-0untu #nd sudo

7 e o+ the rec"rre t pro%lems +or reg"lar "sers is how to per+orm certai tasks that re="ire s"per"ser pri#ileges. These tasks i cl"!e i stalli g a ! "p!ati g so+tware, e!iti g system co +ig"ratio +iles, a ! accessi g !e#ices. 9 the Wi !ows worl!, this is o+te !o e %y gi#i g "sers a!mi istrati#e pri#ileges. This allows "sers to per+orm these tasks. Aowe#er, it also e a%les programs e3ec"te! %y the "ser to ha#e the same a%ilities. This is !esira%le i most cases, %"t it also permits malware Emalicio"s so+twareF s"ch as #ir"ses to ha#e +ree reig o+ the comp"ter. 9 the ) i3 worl!, there has always %ee a larger !i#isio %etwee reg"lar "sers a ! a!mi istrators, owi g to the m"lti-"ser heritage o+ ) i3. The approach take i ) i3 is to gra t s"per"ser pri#ileges o ly whe ee!e!. To !o this, the su a ! sudo comma !s are commo ly "se!. )p " til a co"ple o+ years ago, most *i "3 !istri%"tio s relie! o su +or this p"rpose. su !i! <t re="ire the co +ig"ratio that sudo re="ire!, a ! ha#i g a root acco" t is tra!itio al i ) i3. This i tro!"ce! a pro%lem. )sers were tempte! to operate as root " ecessarily. 9 +act, some "sers operate! their systems as the root "ser e3cl"si#ely, si ce it !oes away with all those a oyi g ?permissio !e ie!@ messages. This is how yo" re!"ce the sec"rity o+ a *i "3 system to that o+ a Wi !ows system. &ot a goo! i!ea. Whe )%" t" was i tro!"ce!, its creators took a !i++ere t tack. Jy !e+a"lt, )%" t" !isa%les logi s to the root acco" t E%y +aili g to set a passwor! +or the acco" tF, a ! i stea! "ses sudo to gra t s"per"ser pri#ileges. The i itial "ser acco" t is gra te! +"ll access to s"per"ser pri#ileges #ia sudo a ! may gra t similar powers to s"%se="e t "ser acco" ts.

/ho%n A Change 2i e <%ner And +rou5

The cho9n comma ! is "se! to cha ge the ow er a ! gro"p ow er o+ a +ile or !irectory. S"per"ser pri#ileges are re="ire! to "se this comma !. The sy ta3 o+ cho9n looks like this,

.0/

.0 8 6ermissio s
chown (owner)(2(group)) file...

cho9n ca cha ge the +ile ow er a !-or the +ile gro"p ow er !epe !i g o the +irst arg"me t o+ the comma !. Aere are some e3amples, Table 1=-2: chown )rgument 1+amples #rgument bob bob:users :admins bob: 1esults Cha ges the ow ership o+ the +ile +rom its c"rre t ow er to "ser bob. Cha ges the ow ership o+ the +ile +rom its c"rre t ow er to "ser bob a ! cha ges the +ile gro"p ow er to gro"p "sers. Cha ges the gro"p ow er to the gro"p admins. The +ile ow er is " cha ge!. Cha ge the +ile ow er +rom the c"rre t ow er to "ser bob a ! cha ges the gro"p ow er to the logi gro"p o+ "ser bob.

*et<s say that we ha#e two "sersH janet, who has access to s"per"ser pri#ileges a ! ton1, who !oes ot. )ser janet wa ts to copy a +ile +rom her home !irectory to the home !irectory o+ "ser ton1. Si ce "ser janet wa ts ton1 to %e a%le to e!it the +ile, janet cha ges the ow ership o+ the copie! +ile +rom janet to ton1,
[janet@linuxbox ~]$ :ass9ord: [janet@linuxbox ~]$ -r9-r--r-- # root [janet@linuxbox ~]$ [janet@linuxbox ~]$ -r9-r--r-- # ton1 sudo cp m!file.txt ~ton! sudo ls -l ~ton!/m!file.txt root /($# !((/-($-!( #%:$( 6home6ton16m1file<txt sudo chown ton!2 ~ton!/m!file.txt sudo ls -l ~ton!/m!file.txt ton1 /($# !((/-($-!( #%:$( 6home6ton16m1file<txt

Aere we see "ser janet copy the +ile +rom his !irectory to the home !irectory o+ "ser ton1. &e3t, janet cha ges the ow ership o+ the +ile +rom root Ea res"lt o+ "si g sudoF to ton1. )si g the traili g colo i the +irst arg"me t, janet also cha ge! the gro"p ow ership o+ the +ile to the logi gro"p o+ ton1, which happe s to %e gro"p ton1. &otice that a+ter the +irst "se o+ sudo, janet was ot prompte! +or her passwor!C This is %eca"se sudo, i most co +ig"ratio s, ?tr"sts@ yo" +or se#eral mi "tes " til its timer .08

Cha gi g 9!e tities r" s o"t.

/hgr5 A Change +rou5 <%nershi5

9 ol!er #ersio s o+ ) i3, the cho9n comma ! o ly cha ge! +ile ow ership, ot gro"p ow ership. 0or that p"rpose, a separate comma !, chgr7 was "se!. 9t works m"ch the same way as cho9n, e3cept +or %ei g more limite!.

$xercising ?ur Pri%ileges

&ow that we ha#e lear e! how this permissio s thi g works, it<s time to show it o++. We are goi g to !emo strate the sol"tio to a commo pro%lemBsetti g "p a share! !irectory. *et<s imagi e that we ha#e two "sers ame! ?%ill@ a ! ?kare .@ They %oth ha#e m"sic C' collectio s a ! wish to set "p a share! !irectory, where they will each store their m"sic +iles as 7gg Ror%is or >6(. )ser bill has access to s"per"ser pri#ileges #ia sudo. The +irst thi g that ee!s to happe is creati g a gro"p that will ha#e %oth bill a ! karen as mem%ers. )si g the graphical "ser ma ageme t tool, bill creates a gro"p calle! music a ! a!!s "sers bill a ! karen to it,

Figure 3: %reating ) >ew @roup /ith @>&!1

&e3t, bill creates the !irectory +or the m"sic +iles,

.09

.0 8 6ermissio s
[bill@linuxbox ~]$ sudo mkdir /usr/local/share/Ousic :ass9ord:

Si ce bill is ma ip"lati g +iles o"tsi!e his home !irectory, s"per"ser pri#ileges are re="ire!. $+ter the !irectory is create!, it has the +ollowi g ow erships a ! permissio s,
[bill@linuxbox ~]$ ls -ld /usr/local/share/Ousic dr9xr-xr-x ! root root %(0. !((/-($-!# #/:(" 6usr6local6share6+usic

$s we ca see, the !irectory is ow e! %y root a ! has /22 permissio s. To make this !irectory shara%le, bill ee!s to cha ge the gro"p ow ership a ! the gro"p permissio s to allow writi g,
[bill@linuxbox ~]$ sudo chown 2music /usr/local/share/Ousic [bill@linuxbox ~]$ sudo chmod CC" /usr/local/share/Ousic [bill@linuxbox ~]$ ls -ld /usr/local/share/Ousic dr9xr9xr-x ! root music %(0. !((/-($-!# #/:(" 6usr6local6share6+usic

So what !oes this all mea C 9t mea s that we ow ha#e a !irectory, 6usr6local6share6+usic that is ow e! %y root a ! allows rea! a ! write access to gro"p music. ;ro"p music has mem%ers bill a ! karen, th"s bill a ! karen ca create +iles i !irectory 6usr6local6share6+usic. 7ther "sers ca list the co te ts o+ the !irectory %"t ca ot create +iles there. J"t we still ha#e a pro%lem. With the c"rre t permissio s, +iles a ! !irectories create! withi the >"sic !irectory will ha#e the ormal permissio s o+ the "sers bill a ! karen,
[bill@linuxbox ~]$ & /usr/local/share/Ousic/test.file [bill@linuxbox ~]$ ls -l /usr/local/share/Ousic -r9-r--r-- # bill bill ( !((/-($-!% !(:($ testOfile

$ct"ally there are two pro%lems. 0irst, the !e+a"lt umask o this system is 0022 which pre#e ts gro"p mem%ers +rom writi g +iles %elo gi g to other mem%ers o+ the gro"p. This wo"l! ot %e a pro%lem i+ the share! !irectory o ly co tai e! +iles, %"t si ce this !irectory will store m"sic, a ! m"sic is "s"ally orga iDe! i a hierarchy o+ artists a ! al%"ms, mem%ers o+ the gro"p will ee! the a%ility to create +iles a ! !irectories i si!e !irectories create! %y other mem%ers. We ee! to cha ge the umask "se! %y bill a ! ..0

E3ercisi g 7"r 6ri#ileges karen to 0002 i stea!. Seco !, each +ile a ! !irectory create! %y o e mem%er will %e set to the primary gro"p o+ the "ser rather tha the gro"p music. This ca %e +i3e! %y setti g the setgi! %it o the !irectory,
[bill@linuxbox ~]$ sudo chmod g5s /usr/local/share/Ousic [bill@linuxbox ~]$ ls -ld /usr/local/share/Ousic dr9xr9sr-x ! root music %(0. !((/-($-!% !(:($ 6usr6local6share6+usic

&ow we test to see i+ the ew permissio s +i3 the pro%lem. bill sets his umask to 0002, remo#es the pre#io"s test +ile, creates a ew test +ile a ! !irectory,
[bill@linuxbox ~]$ [bill@linuxbox ~]$ [bill@linuxbox ~]$ [bill@linuxbox ~]$ [bill@linuxbox ~]$ dr9xr9sr-x ! bill -r9-r9-r-- # bill [bill@linuxbox ~]$ umask ***2 rm /usr/local/share/Ousic/test.file & /usr/local/share/Ousic/test.file mkdir /usr/local/share/Ousic/test.dir ls -l /usr/local/share/Ousic music %(0. !((/-($-!% !(:!% testOdir music ( !((/-($-!% !(:!! testOfile

Joth +iles a ! !irectories are ow create! with the correct permissio s to allow all mem%ers o+ the gro"p music to create +iles a ! !irectories i si!e the +usic !irectory. The o e remai i g iss"e is umask. The ecessary setti g o ly lasts " til the e ! o+ sessio a ! m"st %e reset. 9 the e3t part o+ the %ook, we<ll look at maki g the cha ge to umask perma e t.

Changing :our Pass4ord

The last topic we<ll co#er i this chapter is setti g passwor!s +or yo"rsel+ Ea ! +or other "sers i+ yo" ha#e access to s"per"ser pri#ileges.F To set or cha ge a passwor!, the 7ass9d comma ! is "se!. The comma ! sy ta3 looks like this,
passwd (user)

To cha ge yo"r passwor!, 4"st e ter the 7ass9d comma !. 5o" will %e prompte! +or yo"r ol! passwor! a ! yo"r ew passwor!,

...

.0 8 6ermissio s
[me@linuxbox ~]$ passwd NcurrentP 3LAV 7ass9ord: Le9 3LAV 7ass9ord:

The 7ass9d comma ! will try to e +orce "se o+ ?stro g@ passwor!s. This mea s the it will re+"se to accept passwor!s that are too short, too similar to pre#io"s passwor!s, are !ictio ary wor!s, or too easily g"esse!,
[me@linuxbox ~]$ passwd NcurrentP 3LAV 7ass9ord: Le9 3LAV 7ass9ord: H4' :4**, R': is too similar to the old one Le9 3LAV 7ass9ord: H4' :4**, R': it is ,4U too short Le9 3LAV 7ass9ord: H4' :4**, R': it is based on a dictionar1 9ord

9+ yo" ha#e s"per"ser pri#ileges, yo" ca speci+y a "ser ame as a arg"me t to the 7ass9d comma ! to set the passwor! +or a other "ser. There are other optio s a#aila%le to the s"per"ser to allow acco" t locki g, passwor! e3piratio , etc. See the 7ass9d ma page +or !etails.

;urther 1eading

Wikipe!ia has a goo! article o malware, http,--e .wikipe!ia.org-wiki->alware

There are "m%er o+ comma ! li e programs "se! to create a ! mai tai "sers a ! gro"ps. 0or more i +ormatio , see the ma pages +or the +ollowi g comma !s,

adduser useradd grou7add

..2

.. 8 6rocesses

11 2rocesses
>o!er operati g systems are "s"ally multitasking, mea i g that they create the ill"sio o+ !oi g more tha o e thi g at o ce %y rapi!ly switchi g +rom o e e3ec"ti g program to a other. The *i "3 ker el ma ages this thro"gh the "se o+ processes. 6rocesses are how *i "3 orga iDes the !i++ere t programs waiti g +or their t"r at the C6). Sometimes a comp"ter will %ecome sl"ggish or a applicatio will stop respo !i g. 9 this chapter, we will look at some o+ the tools a#aila%le at the comma ! li e that let "s e3ami e what programs are !oi g, a ! how to termi ate processes that are mis%eha#i g. This chapter will i tro!"ce the +ollowi g comma !s,

7s 8 :eport a s apshot o+ c"rre t processes to7 8 'isplay tasks jobs 8 *ist acti#e 4o%s bg 8 6lace a 4o% i the %ackgro" ! fg 8 6lace a 4o% i the +oregro" ! kill 8 Se ! a sig al to a process killall 8 Mill processes %y ame shutdo9n 8 Sh"t!ow or re%oot the system

3o4 # Process +or(s

Whe a system starts "p, the ker el i itiates a +ew o+ its ow acti#ities as processes a ! la" ches a program calle! init. init, i t"r , r" s a series o+ shell scripts Elocate! i 6etcF calle! init scripts, which start all the system ser#ices. >a y o+ these ser#ices are impleme te! as daemon programs, programs that 4"st sit i the %ackgro" ! a ! !o their thi g witho"t ha#i g a y "ser i ter+ace. So e#e i+ we are ot logge! i , the system is at least a little %"sy per+ormi g ro"ti e st"++. The +act that a program ca la" ch other programs is e3presse! i the process scheme as a parent process pro!"ci g a child process. ..(

.. 8 6rocesses The ker el mai tai s i +ormatio a%o"t each process to help keep thi gs orga iDe!. 0or e3ample, each process is assig e! a "m%er calle! a process *- or 6*-. 69's are assig e! i asce !i g or!er, with init always getti g 69' .. The ker el also keeps track o+ the memory assig e! to each process, as well as the processes< rea!i ess to res"me e3ec"tio . *ike +iles, processes also ha#e ow ers a ! "ser 9's, e++ecti#e "ser 9's, etc.

Vie4ing Processes
The most commo ly "se! comma ! to #iew processes Ethere are se#eralF is 7s. The 7s program has a lot o+ optio s, %"t i it simplest +orm it is "se! like this,
[me@linuxbox ~]$ ps :A' TTU TA+& C+' "#0/ 7ts6# ((:((:(( bash #(#!0 7ts6# ((:((:(( 7s

The res"lt i this e3ample lists two processes, process 2.98 a ! process .0.29, which are bash a ! 7s respecti#ely. $s we ca see, %y !e+a"lt, 7s !oes <t show "s #ery m"ch, 4"st the processes associate! with the c"rre t termi al sessio . To see more, we ee! to a!! some optio s, %"t %e+ore we !o that, let<s look at the other +iel!s pro!"ce! %y 7s. TTU is short +or ?Teletype,@ a ! re+ers to the controlling terminal +or the process. ) i3 is showi g its age here. The TA+& +iel! is the amo" t o+ C6) time co s"me! %y the process. $s we ca see, either process makes the comp"ter work #ery har!. 9+ we a!! a optio , we ca get a %igger pict"re o+ what the system is !oi g,
[me@linuxbox ~]$ ps x :A' TTU *T4T TA+& C ++4L' !)00 F *sl (:(( 6usr6libexec6bonobo-activation-server Wac !/!( F *l (:(# 6usr6libexec6evolution-data-server-#<#( -#".%) F *s (:(( 6bin6sh 6usr6bin6startkde #")"# F *s (:(( 6usr6bin6ssh-agent 6usr6bin6dbus-launch -#")"% F * (:(( 6usr6bin6dbus-launch --exit-9ith-session #")"" F *s (:(# 6bin6dbus-daemon --fork --7rint-7id % W7r #"))% F *s (:(! 6usr6bin6g7g-agent -s Wdaemon #")0$ F * (:(( startOkdeinit --ne9-startu7 8kcminitOstart #")0% F *s (:(( kdeinit Running<<< #")0) F * (:(( dco7server Wnosid

and man1 more<<<

..1

Riewi g 6rocesses $!!i g the ?3@ optio E ote that there is o lea!i g !ashF tells 7s to show all o+ o"r processes regar!less o+ what termi al Ei+ a yF they are co trolle! %y. The prese ce o+ a ?C@ i the TTU col"m i !icates o co trolli g termi al. )si g this optio , we see a list o+ e#ery process that we ow . Si ce the system is r" i g a lot o+ processes, 7s pro!"ces a lo g list. 9t is o+te help+"l to pipe the o"tp"t +rom 7s i to less +or easier #iewi g. Some optio com%i atio s also pro!"ce lo g li es o+ o"tp"t, so ma3imiDi g the termi al em"lator wi !ow may %e a goo! i!ea, too. $ ew col"m title! *T4T has %ee a!!e! to the o"tp"t. *T4T is short +or ?state@ a ! re#eals the c"rre t stat"s o+ the process, Table 11-1: 6rocess States State R * ' T I X >eaning :" i g. This mea s that the process is r" i g or rea!y to r" . Sleepi g. $ process is ot r" i gH rather, it is waiti g +or a e#e t, s"ch as a keystroke or etwork packet. ) i terr"pti%le Sleep. 6rocess is waiti g +or 9-7 s"ch as a !isk !ri#e. Stoppe!. 6rocess has %ee i str"cte! to stop. >ore o this later. $ !e+" ct or ?Dom%ie@ process. This is a chil! process that has termi ate!, %"t has ot %ee clea e! "p %y its pare t. $ high priority process. 9t<s possi%le to gra t more importa ce to a process, gi#i g it more time o the C6). This property o+ a process is calle! niceness. $ process with high priority is sai! to %e less nice %eca"se it<s taki g more o+ the C6)<s time, which lea#es less +or e#ery%o!y else. $ low priority process. $ process with low priority Ea ? ice@ processF will o ly get processor time a+ter other processes with higher priority ha#e %ee ser#ice!.

The process state may %e +ollowe! %y other characters. These i !icate #ario"s e3otic process characteristics. See the 7s ma page +or more !etail. $ other pop"lar set o+ optio s is ?a"3@ Ewitho"t a lea!i g !ashF. This gi#es "s e#e more i +ormatio ,

..2

.. 8 6rocesses
[me@linuxbox ~]$ ps aux 3*&R :A' 5C:3 5+&+ ;*I root # (<( (<( !#$. root ! (<( (<( ( root $ (<( (<( ( root % (<( (<( ( root " (<( (<( ( root . (<( (<( ( root ) (<( (<( (

R** .%% ( ( ( ( ( (

TTU F F F F F F F

*T4T *s *X *X *X *X *X *X

*T4RT +ar(" +ar(" +ar(" +ar(" +ar(" +ar(" +ar("

TA+& (:$# (:(( (:(( (:(( (:(. (:$. (:((

C ++4L' init [kt] [mi] [ks] [9a] [ev] [kh]

and man1 more<<<

This set o+ optio s !isplays the processes %elo gi g to e#ery "ser. )si g the optio s witho"t the lea!i g !ash i #okes the comma ! with ?JS' style@ %eha#ior. The *i "3 #ersio o+ 7s ca em"late the %eha#ior o+ the 7s program +o" ! i se#eral !i++ere t ) i3 impleme tatio s. With these optio s, we get these a!!itio al col"m s, Table 11-': 8S- Style ps %olumn ,eaders 3eader 3*&R 5C:3 5+&+ ;*I R** *T4RT >eaning )ser 9'. This is the ow er o+ the process. C6) "sage i perce t. >emory "sage i perce t. Rirt"al memory siDe. :esi!e t Set SiDe. The amo" t o+ physical memory E:$>F the process is "si g i kilo%ytes. Time whe the process starte!. 0or #al"es o#er twe ty +o"r ho"rs, a !ate is "se!.

?ie%ing 'ro/esses )ynami/a y With to7

While the 7s comma ! ca re#eal a lot a%o"t what the machi e is !oi g, it pro#i!es o ly a s apshot o+ the machi e<s state at the mome t the 7s comma ! is e3ec"te!. To see a more !y amic #iew o+ the machi e<s acti#ity, we "se the to7 comma !,
[me@linuxbox ~]$ top

..K

Riewi g 6rocesses The to7 program !isplays a co ti "o"sly "p!ati g E%y !e+a"lt, e#ery ( seco !sF !isplay o+ the system processes liste! i or!er o+ process acti#ity. The ame ?top@ comes +rom the +act that the to7 program is "se! to see the ?top@ processes o the system. The to7 !isplay co sists o+ two parts, a system s"mmary at the top o+ the !isplay, +ollowe! %y a ta%le o+ processes sorte! %y C6) acti#ity,
to7 - #%:"0:!( u7 .:$(> ! users> load average: (<()> (<(!> (<(( Tasks: #(0 total> # running> #(. slee7ing> ( sto77ed> ! Combie C7uNsP: (<)5us> #<(5s1> (<(5ni> 0/<$5id> (<(59a> (<(5hi> (<(5si +em: $#0%0.k total> $#%/.(k used> %.$.k free> #0$0!k buff *9a7: /)""((k total> #%0#!/k used> )!.$)!k free> ##%.).k cach :A' .!%% ##()# .#/( .$!# %0"" # ! $ % " . ) %# .) ##% ##. 3*&R me me me me root root root root root root root root root root root root :R $0 !( !( !( !( !( #" RT #" RT #" #" #" #" !( #" LA ;ART R&* *SR * 5C:3 5+&+ #0 $#)"! $#!% !#// * .<$ #<( ( !$(% #(0! /%( R #<$ (<$ ( !)(( ##(( ))! * (<) (<$ ( !(0%% )!%/ .".( * (<) !<$ ( #(%m 0../ ")). * (<$ $<( ( !0). "!/ %). * (<( (<! -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( -" ( ( ( * (<( (<( ( ( ( ( * (<( (<( -" ( ( ( * (<( (<( TA+&8 #.:!%<%! (:((<#% (:($<.. !:"#<$/ !:#0<$0 (:($<#% (:((<(( (:((<(( (:((<)! (:((<(% (:((<%! (:((<(. (:(#<(/ (:((<(( (:(#<.! (:(!<%% C ++4L' trackerd to7 dbus-dae multiloa Vorg init kthreadd migratio ksoftirB 9atchdog events6( khel7er kblockd6 kseriod 7dflush ks9a7d(

The system s"mmary co tai s a lot o+ goo! st"++. Aere<s a r" !ow , Table 11-3: top *nformation Fields 1o4 . ;ield to7 #%:"0:!( u7 .:$( >eaning &ame o+ the program. C"rre t time o+ !ay. This is calle! uptime. 9t is the amo" t o+ time si ce the machi e was last %oote!. 9 this e3ample, the system has %ee "p +or si3 a ! a hal+ ho"rs. There are two "sers logge! i . (oad average re+ers to the "m%er o+ processes ../

! users load average:

.. 8 6rocesses that are waiti g to r" , that is, the "m%er o+ processes that are i a r" a%le state a ! are shari g the C6). Three #al"es are show , each +or a !i++ere t perio! o+ time. The +irst is the a#erage +or the last K0 seco !s, the e3t the pre#io"s 2 mi "tes, a ! +i ally the pre#io"s .2 mi "tes. Ral"es " !er ..0 i !icate that the machi e is ot %"sy. 2 ( Tasks: C7uNsP: (<)5us This s"mmariDes the "m%er o+ processes a ! their #ario"s process states. This row !escri%es the character o+ the acti#ities that the C6) is per+ormi g. 0./_ o+ the C6) is %ei g "se! +or user processes. This mea s processes o"tsi!e o+ the ker el itsel+. ..0_ o+ the C6) is %ei g "se! +or system Eker elF processes. 0.0_ o+ the C6) is %ei g "se! %y ? ice@ Elow priorityF processes. 98.(_ o+ the C6) is i!le. 0.0_ o+ the C6) is waiti g +or 9-7. Shows how physical :$> is %ei g "se!. Shows how swap space E#irt"al memoryF is %ei g "se!.

#<(5s1 (<(5ni 0/<$5id (<(59a 1 2 +em: *9a7:

The top program accepts a "m%er o+ key%oar! comma !s. The two most i teresti g are h, which !isplays the program<s help scree , a ! B, which ="its to7. Joth ma4or !esktop e #iro me ts pro#i!e graphical applicatio s that !isplay i +ormatio similar to to7 Ei m"ch the same way that Task >a ager i Wi !ows worksF, %"t 9 +i ! that to7 is %etter tha the graphical #ersio s %eca"se it is +aster a ! it co s"mes +ar +ewer system reso"rces. $+ter all, o"r system mo itor program sho"l! <t %e the so"rce o+ the system slow!ow that we are tryi g to track.

Controlling Processes
&ow that we ca see a ! mo itor processes, let<s gai some co trol o#er them. 0or o"r ..8

Co trolli g 6rocesses e3perime ts, we<re goi g to "se a little program calle! xlogo as o"r g"i ea pig. The xlogo program is a sample program s"pplie! with the G Wi !ow System Ethe " !erlyi g e gi e that makes the graphics o o"r !isplay goF which simply !isplays a resiDa%le wi !ow co tai i g the G logo. 0irst, we<ll get to k ow o"r test s"%4ect,
[me@linuxbox ~]$ xlogo

$+ter e teri g the comma !, a small wi !ow co tai i g the logo sho"l! appear somewhere o the scree . 7 some systems, xlogo may pri t a war i g message, %"t it may %e sa+ely ig ore!. Ti(5 9+ yo"r system !oes ot i cl"!e the xlogo program, try "si g gedit or k9rite i stea!. We ca #eri+y that xlogo is r" i g %y resiDi g its wi !ow. 9+ the logo is re!raw i the ew siDe, the program is r" i g. &otice how o"r shell prompt has ot ret"r e!C This is %eca"se the shell is waiti g +or the program to +i ish, 4"st like all the other programs we ha#e "se! so +ar. 9+ we close the xlogo wi !ow, the prompt ret"r s.

Interru5ting A 'ro/ess
*et<s o%ser#e what happe s whe we r" comma ! a ! #eri+y that the program is r" a ! type Ctrl-c.
[me@linuxbox ~]$ xlogo [me@linuxbox ~]$

xlogo agai . 0irst, e ter the xlogo i g. &e3t, ret"r to the termi al wi !ow

9 a termi al, typi g Ctrl-c, interrupts a program. This mea s that we politely aske! the program to termi ate. $+ter typi g Ctrl-c, the xlogo wi !ow close! a ! the shell prompt ret"r e!. >a y E%"t ot allF comma ! li e programs ca %e i terr"pte! %y "si g this tech i="e.

'utting A 'ro/ess In The Ba/kground

*et<s say we wa te! to get the shell prompt %ack witho"t termi ati g the xlogo ..9

.. 8 6rocesses program. WeIll !o this %y placi g the program i the background. Thi k o+ the termi al as ha#i g a foreground Ewith st"++ #isi%le o the s"r+ace like the shell promptF a ! a %ackgro" ! Ewith hi!!e st"++ %ehi ! the s"r+ace.F To la" ch a program so that it is imme!iately place! i the %ackgro" !, we +ollow the comma ! with a - ?^@ character,
[me@linuxbox ~]$ xlogo ' [#] !/!$. [me@linuxbox ~]$

$+ter e teri g the comma !, the xlogo wi !ow appeare! a ! the shell prompt ret"r e!, %"t some +" y "m%ers were pri te! too. This message is part o+ a shell +eat"re calle! Aob control. With this message, the shell is telli g "s that we ha#e starte! 4o% "m%er . E?[#]@F a ! that it has 69' 282(K. 9+ we r" 7s, we ca see o"r process,
[me@linuxbox ~]$ ps :A' TTU TA+& #(.($ 7ts6# ((:((:(( !/!$. 7ts6# ((:((:(( !/!$0 7ts6# ((:((:((

C+' bash xlogo 7s

The shell<s 4o% co trol +acility also gi#es "s a way to list the 4o%s that are ha#e %ee la" che! +rom o"r termi al. )si g the jobs comma !, we ca see this list,
[me@linuxbox ~]$ jobs [#]8 Running

xlogo Y

The res"lts show that we ha#e o e 4o%, "m%ere! ?.@, that it is r" comma ! was xlogo Y.

i g, a ! that the

!eturning A 'ro/ess To The 2oreground

$ process i the %ackgro" ! is imm" e +rom key%oar! i p"t, i cl"!i g a y attempt i terr"pt it with a Ctrl-c. To ret"r a process to the +oregro" !, "se the fg comma !, this way,
[me@linuxbox ~]$ jobs [#]8 Running [me@linuxbox ~]$ fg 81

xlogo Y

.20

Co trolli g 6rocesses
xlogo

The comma ! fg +ollowe! %y a perce t sig a ! the 4o% "m%er Ecalle! a AobspecF !oes the trick. 9+ we o ly ha#e o e %ackgro" ! 4o%, the 4o%spec is optio al. To termi ate xlogo, type Ctrl-c.

Sto55ing E'ausingF A 'ro/ess

Sometimes we<ll wa t to stop a process witho"t termi ati g it. This is o+te !o e to allow a +oregro" ! process to %e mo#e! to the %ackgro" !. To stop a +oregro" ! process, type Ctrl-C. *et<s try it. $t the comma ! prompt, type xlogo, the &nter key, the Ctrl-C,
[me@linuxbox ~]$ xlogo [#]8 *to77ed [me@linuxbox ~]$

xlogo

$+ter stoppi g xlogo, we ca #eri+y that the program has stoppe! %y attempti g to resiDe the xlogo wi !ow. We will see that it appears ="ite !ea!. We ca either restore the program to the +oregro" !, "si g the fg comma !, or mo#e the program to the %ackgro" ! with the bg comma !,
[me@linuxbox ~]$ bg 81 [#]8 xlogo Y [me@linuxbox ~]$

$s with the fg comma !, the 4o%spec is optio al i+ there is o ly o e 4o%. >o#i g a process +rom the +oregro" ! to the %ackgro" ! is ha !y i+ we la" ch a graphical program +rom the comma !, %"t +orget to place it i the %ackgro" ! %y appe !i g the traili g ?^@. Why wo"l! yo" wa t to la" ch a graphical program +rom the comma ! li eC There are two reaso s. 0irst, the program yo" wish to r" might ot %e liste! o the wi !ow ma ager<s me "s Es"ch as xlogoF. Seco !ly, %y la" chi g a program +rom the comma ! li e, yo" might %e a%le to see error messages that wo"l! otherwise %e i #isi%le i+ the program were la" che! graphically. Sometimes, a program will +ail to start "p whe la" che! +rom the graphical me ". Jy la" chi g it +rom the comma ! li e i stea!, we may see a error message that will re#eal the pro%lem. $lso, some graphical programs ha#e ma y i teresti g a ! "se+"l comma ! li e optio s. .2.

.. 8 6rocesses

Signals
The kill comma ! is "se! to ?kill@ programs. This allows "s to termi ate programs that ee! killi g. Aere<s a e3ample,
[me@linuxbox ~]$ xlogo Y [#] !/%(# [me@linuxbox ~]$ kill !/%(# [#]8 Terminated

xlogo

We +irst la" ch xlogo i the %ackgro" !. The shell pri ts the 4o%spec a ! the 69' o+ the %ackgro" ! process. &e3t, we "se the kill comma ! a ! speci+y the 69' o+ the process we wa t to termi ate. We co"l! ha#e also speci+ie! the process "si g a 4o%spec E+or e3ample, ?_.@F i stea! o+ a 69'. While this is all #ery straight+orwar!, there is more to it tha that. The kill comma ! !oes <t e3actly ?kill@ programs, rather it se !s them signals. Sig als are o e o+ se#eral ways that the operati g system comm" icates with programs. We ha#e alrea!y see sig als i actio with the "se o+ Ctrl-c a ! Ctrl-C. Whe the termi al recei#es o e o+ these keystrokes, it se !s a sig al to the program i the +oregro" !. 9 the case o+ Ctrl-c, a sig al calle! ALT E9 terr"ptF is se tH with Ctrl-C, a sig al calle! T*T: ETermi al Stop.F 6rograms, i t"r , ?liste @ +or sig als a ! may act "po them as they are recei#e!. The +act that a program ca liste a ! act "po sig als allows a program to !o thi gs like sa#e work i progress whe it is se t a termi atio sig al.

Sending Signa s To 'ro/esses With kill

The kill comma ! is "se! to se ! sig als to programs. 9ts most commo sy ta3 looks like this,
kill (-signal) BP1...

9+ o sig al is speci+ie! o the comma ! li e, the the T&R+ ETermi ateF sig al is se t %y !e+a"lt. The kill comma ! is most o+te "se! to se ! the +ollowi g sig als, Table 11-$: %ommon Signals 7um0er # 7ame S3: >eaning Aa g"p. This is a #estige o+ the goo! ol! !ays whe termi als were attache! to remote

.22

Sig als comp"ters with pho e li es a ! mo!ems. The sig al is "se! to i !icate to programs that the co trolli g termi al has ?h" g "p.@ The e++ect o+ this sig al ca %e !emo strate! %y closi g a termi al sessio . The +oregro" ! program r" i g o the termi al will %e se t the sig al a ! will termi ate. This sig al is also "se! %y ma y !aemo programs to ca"se a rei itialiDatio . This mea s that whe a !aemo is se t this sig al, it will restart a ! re-rea! its co +ig"ratio +ile. The $pache we% ser#er is a e3ample o+ a !aemo that "ses the S3: sig al i this way. ! ALT 9 terr"pt. 6er+orms the same +" ctio as the Ctrl-c key se t +rom the termi al. 9t will "s"ally termi ate a program. Mill. This sig al is special. Whereas programs may choose to ha !le sig als se t to them i !i++ere t ways, i cl"!i g ig ori g them all together, the 2A== sig al is e#er act"ally se t to the target program. :ather, the ker el imme!iately termi ates the process. Whe a process is termi ate! i this ma er, it is gi#e o opport" ity to ?clea "p@ a+ter itsel+ or sa#e its work. 0or this reaso , the 2A== sig al sho"l! o ly %e "se! as a last resort whe other termi atio sig als +ail. Termi ate. This is the !e+a"lt sig al se t %y the kill comma !. 9+ a program is still ?ali#e@ e o"gh to recei#e sig als, it will termi ate. Co ti "e. This will restore a process a+ter a *T : sig al. Stop. This sig al ca"ses a process to pa"se witho"t termi ati g. *ike the 2A== sig al, it is ot se t to the target process, a ! th"s it ca ot %e ig ore!.

2A==

#"

T&R+

#/ #0

C LT *T :

.2(

.. 8 6rocesses *et<s try o"t the kill comma !,

[me@linuxbox ~]$ xlogo ' [#] #$"%. [me@linuxbox ~]$ kill -1 1 "AN [#]8 Sangu7 xlogo

9 this e3ample, we start the xlogo program i the %ackgro" ! a ! the se ! it a S3: sig al with kill. The xlogo program termi ates a ! the shell i !icates that the %ackgro" ! process has recei#e! a ha g"p sig al. 5o" may ee! to press the e ter key a co"ple o+ times %e+ore yo" see the message. &ote that sig als may %e speci+ie! either %y "m%er or %y ame, i cl"!i g the ame pre+i3e! with the letters ?S9;@,
[me@linuxbox ~]$ [#] #$.(# [me@linuxbox ~]$ [#]8 Anterru7t [me@linuxbox ~]$ [#] #$.(/ [me@linuxbox ~]$ [#]8 Anterru7t xlogo ' kill -P?+ 1 N*1 xlogo xlogo ' kill -FPQP?+ 1 N*M xlogo

:epeat the e3ample a%o#e a ! try o"t the other sig als. :emem%er, yo" ca also "se 4o%specs i place o+ 69's. 6rocesses, like +iles, ha#e ow ers, a ! yo" m"st %e the ow er o+ a process Eor the s"per"serF i or!er to se ! it sig als with kill. 9 a!!itio to the list o+ sig als a%o#e, which are most o+te "se! with kill, there are other sig als +re="e tly "se! %y the system. Aere is a list o+ other commo sig als, Table 11-.: &ther %ommon Signals 7um0er $ ## 7ame Z3AT *&@; >eaning T"it. Segme tatio Riolatio . This sig al is se t i+ a program makes illegal "se o+ memory, that is, it trie! to write somewhere it was ot allowe! to. Termi al Stop. This is the sig al se t %y the termi al whe the Ctrl-C key is presse!. ) like the *T : sig al, the T*T: sig al is

!(

T*T:

.21

Sig als recei#e! %y the process a ! may %e ig ore!. !/ ,ALCS Wi !ow Cha ge. This is a sig al se t %y the system whe a wi !ow cha ges siDe. Some programs , like to7 a ! less will respo ! to this sig al %y re!rawi g themsel#es to +it the ew wi !ow !ime sio s.

0or the c"rio"s, a complete list o+ sig als ca %e see with the +ollowi g comma !,
[me@linuxbox ~]$ kill -l

Sending Signa s To :u ti5 e 'ro/esses With killall

9t<s also possi%le to se ! sig als to m"ltiple processes matchi g a speci+ie! program or "ser ame %y "si g the killall comma !. Aere is the sy ta3,
killall (-u user) (-signal) name...

To !emo strate, we will start a co"ple o+ i sta ces o+ the xlogo program a ! the termi ate them,
[me@linuxbox ~]$ xlogo ' [#] #//(# [me@linuxbox ~]$ xlogo ' [!] #//(! [me@linuxbox ~]$ killall xlogo [#]- Terminated xlogo [!]8 Terminated xlogo

:emem%er, as with kill, yo" m"st ha#e s"per"ser pri#ileges to se ! sig als to processes that !o ot %elo g to yo".

>ore Process 1elated Commands

Si ce mo itori g processes is a importa t system a!mi istratio task, there are a lot o+ comma !s +or it. Aere are some to play with,

.22

.. 8 6rocesses Table 11-2: &ther 6rocess 9elated %ommands Command 7stree vmstat Description 7"tp"ts a process list arra ge! i a tree-like patter showi g the pare t-chil! relatio ships %etwee processes. 7"tp"ts a s apshot o+ system reso"rce "sage i cl"!i g, memory, swap a ! !isk 9-7. To see a co ti "o"s !isplay, +ollow the comma ! with a time !elay Ei seco !sF +or "p!ates. 0or e3ample, vmstat ". Termi ate the o"tp"t with Ctrl-c. $ graphical program that !raws a graph showi g system loa! o#er time. Similar to the xload program, %"t !raws the graph i the termi al. Termi ate the o"tp"t with Ctrl-c.

xload tload

.2K

6art ( 8 Co +ig"ratio $ ! The E #iro me t

'art 3 A Con=iguration And The 6n;ironment

.2/

.2 8 The E #iro me t

12 The !nviron%ent
$s we !isc"sse! earlier, the shell mai tai s a %o!y o+ i +ormatio !"ri g o"r shell sessio calle! the environment. 'ata store! i the e #iro me t is "se! %y programs to !etermi e +acts a%o"t o"r co +ig"ratio . While most programs "se configuration files to store program setti gs, some programs will also look +or #al"es store! i the e #iro me t to a!4"st their %eha#ior. M owi g this, we ca "se the e #iro me t to c"stomiDe o"r shell e3perie ce. 9 this chapter, we will work with the +ollowi g comma !s,

7rintenv 8 6ri t part or all o+ the e #iro me t set 8 Set shell optio s ex7ort 8 E3port e #iro me t to s"%se="e tly e3ec"te! programs alias 8 Create a alias +or a comma !

+hat Is Stored In The $n%ironment.

The shell stores two %asic types o+ !ata i the e #iro me t, tho"gh, with bash> the types are largely i !isti g"isha%le. They are environment variables a ! shell variables. Shell #aria%les are %its o+ !ata place! there %y bash, a ! e #iro me t #aria%les are %asically e#erythi g else. 9 a!!itio to #aria%les, the shell also stores some programmatic !ata, amely aliases a ! shell functions. We co#ere! aliases i Chapter K, a ! shell +" ctio s Ewhich are relate! to shell scripti gF will %e co#ere! i 6art 2.

6xamining The 6n;ironment

We ca "se either the set %"ilti i bash or the 7rintenv program to see what is store! i the e #iro me t. The set comma ! will show %oth the shell a ! e #iro me t #aria%les, while 7rintenv will o ly !isplay the latter. Si ce the list o+ e #iro me t co te ts will %e +airly lo g, it is %est to pipe the o"tp"t o+ either comma ! i to less,
[me@linuxbox ~]$ printenv 0 less

.28

What 9s Store! 9 The E #iro me tC

'oi g so, we sho"l! get somethi g that looks like this,

2'&O+3=TAS&4'Mfalse **SO4@&LTO:A'M.... S *TL4+&Mlinuxbox @:@O4@&LTOAL- M6tm76g7g-:d t)g6*<g7g-agent:../0:# *S&==M6bin6bash T&R+Mxterm V'@O+&L3O:R&-AVMkdeSA*T*AI&M#((( V'@O*&**A LOC 2A&M.d)b("c."/%.c$eaf$#(#b((%.bd!b((-#!(/"!#00(<00.)(" -##))(".#00 @T2!ORCO-A=&*M6etc6gtk-!<(6gtkrc:6home6me6<gtkrc-!<(:6home6me6<kde6sh are6config6gtkrc-!<( @T2ORCO-A=&*M6etc6gtk6gtkrc:6home6me6<gtkrc:6home6me6<kde6share6confi g6gtkrc @*O=AHM6home6me6<fonts ,AL' ,A'M!0$.(#$. ZT'ARM6usr6lib6Bt-$<$ ZTALCM6usr6lib6Bt-$<$6include 2'&O-3==O*&**A LMtrue 3*&RMme =*OC = R*MnoM((:fiM((:diM((T$%:lnM((T$.:7iM%(T$$:soM((T$":bdM%(T$$T(# :cdM%(T$$T(#:orM(#T("T$)T%#:miM(#T("T$)T%#:exM((T$!:E<cmdM((T$!:E<exe :

What we see is a list o+ e #iro me t #aria%les a ! their #al"es. 0or e3ample, we see a #aria%le calle! 3*&R, which co tai s the #al"e ?me@. The 7rintenv comma ! ca also list the #al"e o+ a speci+ic #aria%le,
[me@linuxbox ~]$ printenv EFGH me

The set comma !, whe "se! witho"t optio s or arg"me ts, will !isplay %oth the shell a ! e #iro me t #aria%les, as well as a y !e+i e! shell +" ctio s. ) like 7rintenv, its o"tp"t is co"rteo"sly sorte! i alpha%etical or!er,
[me@linuxbox ~]$ set 0 less

9t is also possi%le to #iew the co te ts o+ a #aria%le "si g the echo comma !, like this, .29

.2 8 The E #iro me t
[me@linuxbox ~]$ echo 3RSOG 6home6me

7 e eleme t o+ the e #iro me t that either set or 7rintenv !isplays is aliases. To see them, e ter the alias comma ! witho"t arg"me ts,
[me@linuxbox ~]$ alias alias l<MKls -d <E --colorMtt1K alias llMKls -l --colorMtt1K alias lsMKls --colorMtt1K alias viMKvimK alias 9hichMKalias Q 6usr6bin69hich --tt1-onl1 --read-alias --sho9dot --sho9-tildeK

Some Interesting ?ariab es

The e #iro me t co tai s ="ite a +ew #aria%les, a ! tho"gh yo"r e #iro me t may !i++er +rom the o e prese te! here, yo" will likely see the +ollowi g #aria%les i yo"r e #iro me t, Table 1'-1: 1nvironment Bariables Varia0le 'A*:=4U Contents The ame o+ yo"r !isplay i+ yo" are r" i g a graphical e #iro me t. )s"ally this is ?,0@, mea i g the +irst !isplay ge erate! %y the G ser#er. Tha ame o+ the program to %e "se! +or te3t e!iti g. The ame o+ yo"r shell program. The path ame o+ yo"r home !irectory. 'e+i es the character set a ! collatio or!er o+ yo"r la g"age. The pre#io"s worki g !irectory. The ame o+ the program to %e "se! +or pagi g o"tp"t. This is o+te set to 6usr6bin6less. $ colo -separate! list o+ !irectories that are searche! whe yo" e ter the ame o+ a e3ec"ta%le program. 6rompt Stri g .. This !e+i es the co te ts o+ yo"r shell prompt. $s we will later see, this ca %e e3te si#ely c"stomiDe!.

&'AT R *S&== S +& =4L@ ='O:,' :4@&R :4TS :*#

.(0

What 9s Store! 9 The E #iro me tC :,' T&R+ The c"rre t worki g !irectory. The ame o+ yo"r termi al type. ) i3-like systems s"pport ma y termi al protocolsH this #aria%le sets the protocol to %e "se! with yo"r termi al em"lator. Speci+ies yo"r timeDo e. >ost ) i3-like systems mai tai the comp"terIs i ter al clock i %oordinated 0niversal Time E)TCF a ! the !isplays the local time %y applyi g a o++set speci+ie! %y this #aria%le. 5o"r "ser ame.

TI

3*&R

'o <t worry i+ some o+ these #al"es are missi g. They #ary %y !istri%"tio .

3o4 Is The $n%ironment $sta0lished.

Whe we log o to the system, the bash program starts, a ! rea!s a series o+ co +ig"ratio scripts calle! startup files, which !e+i e the !e+a"lt e #iro me t share! %y all "sers. This is +ollowe! %y more start"p +iles i o"r home !irectory that !e+i e o"r perso al e #iro me t. The e3act se="e ce !epe !s o the type o+ shell sessio %ei g starte!. There are two ki !s, a logi shell sessio a ! a o -logi shell sessio . $ logi shell sessio is o e i which we are prompte! +or o"r "ser ame a ! passwor!H whe we start a #irt"al co sole sessio , +or e3ample. $ o -logi shell sessio typically occ"rs whe we la" ch a termi al sessio i the ;)9. *ogi shells rea! o e or more start"p +iles as show i Ta%le .2-2, Table 1'-': Startup Files For (ogin Shell Sessions ;ile 6etc67rofile ~6<bashO7rofile ~6<bashOlogin ~6<7rofile Contents $ glo%al co +ig"ratio script that applies to all "sers. $ "ser<s perso al start"p +ile. Ca %e "se! to e3te ! or o#erri!e setti gs i the glo%al co +ig"ratio script. 9+ ~6<bashO7rofile is ot +o" !, bash attempts to rea! this script. 9+ either ~6<bashO7rofile or ~6<bashOlogin is +o" !, bash attempts to rea! this +ile. This is the !e+a"lt i 'e%ia -%ase! !istri%"tio s, s"ch as )%" t".

&o -logi shell sessio s rea! the +ollowi g start"p +iles, .(.

.2 8 The E #iro me t Table 1'-3: Startup Files For >on-(ogin Shell Sessions ;ile 6etc6bash<bashrc ~6<bashrc Contents $ glo%al co +ig"ratio script that applies to all "sers. $ "ser<s perso al start"p +ile. Ca %e "se! to e3te ! or o#erri!e setti gs i the glo%al co +ig"ratio script. shells also i herit the

9 a!!itio to rea!i g the start"p +iles a%o#e, o -logi e #iro me t +rom their pare t process, "s"ally a logi shell.

Take a look at yo"r system a ! see which o+ these start"p +iles yo" ha#e. :emem%erB si ce most o+ the +ile ames liste! a%o#e start with a perio! Emea i g that they are hi!!e F, yo" will ee! to "se the ?-a@ optio whe "si g ls. The ~6<bashrc +ile is pro%a%ly the most importa t start"p +ile +rom the or!i ary "serIs poi t o+ #iew, si ce it is almost always rea!. &o -logi shells rea! it %y !e+a"lt a ! most start"p +iles +or logi shells are writte i s"ch a way as to rea! the ~6<bashrc +ile as well.

What"s In A Startu5 2i e?
9+ we take a look i si!e a typical <bashO7rofile Etake +rom a Ce t7S 1 systemF, it looks somethi g like this,
^ <bashO7rofile ^ @et the aliases and functions if [ -f ~6<bashrc ]T then < ~6<bashrc fi ^ 3ser s7ecific environment and startu7 7rograms :4TSM$:4TS:$S +&6bin ex7ort :4TS

*i es that %egi with a ?P@ are comments a ! are ot rea! %y the shell. These are there +or h"ma rea!a%ility. The +irst i teresti g thi g occ"rs o the +o"rth li e, with the +ollowi g co!e,
if [ -f ~6<bashrc ]T then

.(2

Aow 9s The E #iro me t Esta%lishe!C

< ~6<bashrc fi

This is calle! a if compound command, which we will co#er +"lly whe we get to shell scripti g i 6art 2, %"t +or ow we will tra slate,
Af the file _~6<bashrc_ exists> then read the _~6<bashrc_ file<

We ca see that this %it o+ co!e is how a logi shell gets the co te ts o+ <bashrc. The e3t thi g i o"r start"p +ile has to !o with the :4TS #aria%le. E#er wo !er how the shell k ows where to +i ! comma !s whe we e ter them o the comma ! li eC 0or e3ample, whe we e ter ls, the shell !oes ot search the e tire comp"ter to +i ! 6bin6ls Ethe +"ll path ame o+ the ls comma !F, rather, it searches a list o+ !irectories that are co tai e! i the :4TS #aria%le. The :4TS #aria%le is o+te E%"t ot always, !epe !i g o the !istri%"tio F set %y the 6etc67rofile start"p +ile a ! with this co!e,
:4TSM$:4TS:$S +&6bin

:4TS is mo!i+ie! to a!! the !irectory $S +&6bin to the e ! o+ the list. This is a e3ample o+ parameter e3pa sio , which we to"che! o i Chapter 8. To !emo strate how this works, try the +ollowi g,
[me@linuxbox [me@linuxbox This is some [me@linuxbox [me@linuxbox This is some ~]$ foo$J+his is some J ~]$ echo 3foo ~]$ foo$3fooJtext.J ~]$ echo 3foo text<

)si g this tech i="e, we ca appe ! te3t to the e ! o+ a #aria%le<s co te ts. Jy a!!i g the stri g $S +&6bin to the e ! o+ the :4TS #aria%le<s co te ts, the !irectory $S +&6bin is a!!e! to the list o+ !irectories searche! whe a comma ! is e tere!. This mea s that whe we wa t to create a !irectory withi o"r home !irectory +or stori g o"r ow pri#ate programs, the shell is rea!y to accommo!ate "s. $ll we ha#e .((

.2 8 The E #iro me t to !o is call it bin, a ! weIre rea!y to go. 1ote, >a y !istri%"tio s pro#i!e this :4TS setti g %y !e+a"lt. Some 'e%ia %ase! !istri%"tio s, s"ch as )%" t", test +or the e3iste ce o+ the ~6bin !irectory at logi , a ! !y amically a!! it to the :4TS #aria%le i+ the !irectory is +o" !. *astly, we ha#e,
ex7ort :4TS

The ex7ort comma ! tells the shell to make the co te ts o+ :4TS a#aila%le to chil! processes o+ this shell.

>odi",ing The $n%ironment

Si ce we k ow where the start"p +iles are a ! what they co tai , we ca mo!i+y them to c"stomiDe o"r e #iro me t.

Whi/h 2i es Shou d We :odi=y?

$s a ge eral r"le, to a!! !irectories to yo"r :4TS, or !e+i e a!!itio al e #iro me t #aria%les, place those cha ges i <bashO7rofile Eor e="i#ale t, accor!i g to yo"r !istri%"tio . 0or e3ample, )%" t" "ses <7rofile.F 0or e#erythi g else, place the cha ges i <bashrc. ) less yo" are the system a!mi istrator a ! ee! to cha ge the !e+a"lts +or all "sers o+ the system, restrict yo"r mo!i+icatio s to the +iles i yo"r home !irectory. 9t is certai ly possi%le to cha ge the +iles i 6etc s"ch as 7rofile, a ! i ma y cases it wo"l! %e se si%le to !o so, %"t +or ow, let<s play it sa+e.

Text 6ditors
To e!it Ei.e., mo!i+yF the shell<s start"p +iles, as well as most o+ the other co +ig"ratio +iles o the system, we "se a program calle! a te+t editor. $ te3t e!itor is a program that is, i some ways, like a wor! processor i that it allows yo" to e!it the wor!s o the scree with a mo#i g c"rsor. 9t !i++ers +rom a wor! processor %y o ly s"pporti g p"re te3t, a ! o+te co tai s +eat"res !esig e! +or writi g programs. Te3t e!itors are the ce tral tool "se! %y so+tware !e#elopers to write co!e, a ! %y system a!mi istrators to ma age the co +ig"ratio +iles that co trol the system. There are a lot o+ !i++ere t te3t e!itors a#aila%le +or *i "3H yo"r system pro%a%ly has se#eral i stalle!. Why so ma y !i++ere t o esC 6ro%a%ly %eca"se programmers like .(1

>o!i+yi g The E #iro me t writi g them, a ! si ce programmers "se them e3te si#ely, they write e!itors to e3press their ow !esires as to how they sho"l! work. Te3t e!itors +all i to two %asic categories, graphical a ! te3t %ase!. ;&7>E a ! M'E %oth i cl"!e some pop"lar graphical e!itors. ;&7>E ships with a e!itor calle! gedit, which is "s"ally calle! ?Te3t E!itor@ i the ;&7>E me ". M'E "s"ally ships with three which are Ei or!er o+ i creasi g comple3ityF kedit, k9rite, a ! kate. There are ma y te3t-%ase! e!itors. The pop"lar o es yo" will e co" ter are nano, vi, a ! emacs. The nano e!itor is a simple, easy-to-"se e!itor !esig e! as a replaceme t +or the 7ico e!itor s"pplie! with the 69&E email s"ite. The vi e!itor Eo most *i "3 systems replace! %y a program ame! vim, which is short +or ?Ri 9>pro#e!@F is the tra!itio al e!itor +or ) i3-like systems. 9t will %e the s"%4ect o+ o"r e3t chapter. The emacs e!itor was origi ally writte %y :ichar! Stallma . 9t is a giga tic, all-p"rpose, !oes-e#erythi g programmi g e #iro me t. While rea!ily a#aila%le, it is sel!om i stalle! o most *i "3 systems %y !e+a"lt.

Using A Text 6ditor

$ll te3t e!itors ca %e i #oke! +rom the comma ! li e %y typi g the ame o+ the e!itor +ollowe! %y the ame o+ the +ile yo" wa t to e!it. 9+ the +ile !oes ot alrea!y e3ist, the e!itor will ass"me that yo" wa t to create a ew +ile. Aere is a e3ample "si g gedit,
[me@linuxbox ~]$ gedit some.file

This comma ! will start the gedit te3t e!itor a ! loa! the +ile ame! ?someO+ile@, i+ it e3ists. $ll graphical te3t e!itors are pretty sel+-e3pla atory, so we wo <t co#er them here. 9 stea!, we will co ce trate o o"r +irst te3t-%ase! te3t e!itor, nano. *et<s +ire "p nano a ! e!it the <bashrc +ile. J"t %e+ore we !o that, let<s practice some ?sa+e comp"ti g.@ Whe e#er we e!it a importa t co +ig"ratio +ile, it is always a goo! i!ea to create a %ack"p copy o+ the +ile +irst. This protects "s i case we mess the +ile "p while e!iti g. To create a %ack"p o+ the <bashrc +ile, !o this,
[me@linuxbox ~]$ c7 <bashrc <bashrc<bak

9t !oes <t matter what yo" call the %ack"p +ile, 4"st pick a " !ersta !a%le ame. The e3te sio s ?.%ak@, ?.sa#@, ?.ol!@, a ! ?.orig@ are all pop"lar ways o+ i !icati g a %ack"p +ile. 7h, a ! remem%er that c7 will overwrite e+isting files sile tly. .(2

.2 8 The E #iro me t &ow that we ha#e a %ack"p +ile, we<ll start the e!itor,
[me@linuxbox ~]$ nano .bashrc

7 ce nano starts, weIll get a scree like this,

@L3 nano !<(<$ ^ <bashrc ^ *ource global definitions if [ -f 6etc6bashrc ]T then < 6etc6bashrc fi ^ 3ser s7ecific aliases and functions -ile: <bashrc

[ Read / lines ] `@ @et Sel7` ,rite ut`R Read -il`U :rev :ag`2 Cut Text`C Cur :os `V &xit `? ?ustif1 `, ,here As`; Lext :ag`3 3nCut Te`T To *7ell

1ote5 9+ yo"r system !oes ot ha#e nano i stalle!, yo" may "se a graphical e!itor i stea!. The scree co sists o+ a hea!er at the top, the te3t o+ the +ile %ei g e!ite! i the mi!!le a ! a me " o+ comma !s at the %ottom. Si ce nano was !esig e! to replace the te3t e!itor s"pplie! with a email clie t, it is rather short o e!iti g +eat"res. The +irst comma ! yo" sho"l! lear i a y te3t e!itor is how to e3it the program. 9 the case o+ nano, yo" type Ctrl-x to e3it. This is i !icate! i the me " at the %ottom o+ the scree . The otatio ?`G@ mea s Ctrl-x. This is a commo otatio +or co trol characters "se! %y ma y programs. The seco ! comma ! we ee! to k ow is how to sa#e o"r work. With nano it<s Ctrl.(K

>o!i+yi g The E #iro me t o. With this k owle!ge " !er o"r %elts, we<re rea!y to !o some e!iti g. )si g the !ow arrow key a !-or the :age'o9n key, mo#e the c"rsor to the e ! o+ the +ile, the a!! the +ollowi g li es to the <bashrc +ile,
umask ***2 export RPF+=S?+HST$ignoredups export RPF+FP@G$1*** alias l.$%ls -d .* --color$auto% alias ll$%ls -l --color$auto%

1ote5 5o"r !istri%"tio may alrea!y i cl"!e some o+ these, %"t !"plicates wo <t h"rt a ythi g. Aere is the mea i g o+ o"r a!!itio s, Line umask (((! >eaning Sets the umask to sol#e the pro%lem with share! !irectories we !isc"sse! i Chapter .0. Ca"ses the shell<s history recor!i g +eat"re to ig ore a comma ! i+ the same comma ! was 4"st recor!e!. 9 creases the siDe o+ the comma ! history +rom the !e+a"lt o+ 200 li es to .000 li es. Creates a ew comma ! calle! ?l<@ which !isplays all !irectory e tries that %egi with a !ot. Creates a ew comma ! calle! ?ll@ which !isplays a lo g +ormat !irectory listi g.

ex7ort SA*TC LTR =Mignoredu7s

ex7ort SA*T*AI&M#(((

alias l<MKls -d <E --colorMautoK

alias llMKls -l --colorMautoK

$s we ca see, ma y o+ o"r a!!itio s are ot i t"iti#ely o%#io"s, so it wo"l! %e a goo! i!ea to a!! some comme ts to o"r <bashrc +ile to help e3plai thi gs to the h"ma s. )si g the e!itor, cha ge o"r a!!itio s to look like this,

.(/

.2 8 The E #iro me t
U =hange umask to make director! sharing easier umask (((! U Pgnore duplicates in command histor! and increase U histor! si-e to 1*** lines ex7ort SA*TC LTR =Mignoredu7s ex7ort SA*T*AI&M#((( U :dd some helpful aliases alias l<MKls -d <E --colorMautoK alias llMKls -l --colorMautoK

$h, m"ch %etterL With o"r cha ges complete, type Ctrl-o to sa#e o"r mo!i+ie! <bashrc +ile, a ! Ctrl-x to e3it nano.

+h, Comments #re Important

Whe e#er yo" mo!i+y co +ig"ratio +iles it<s a goo! i!ea to a!! some comme ts to !oc"me t yo"r cha ges. S"re, yo" will remem%er what yo" cha ge! tomorrow, %"t what a%o"t si3 mo ths +rom owC 'o yo"rsel+ a +a#or a ! a!! some comme ts. While yo"<re at it, itIs ot a %a! i!ea to keep a log o+ what cha ges yo" make. Shell scripts a ! bash start"p +iles "se a ?P@ sym%ol to %egi a comme t. 7ther co +ig"ratio +iles may "se other sym%ols. >ost co +ig"ratio +iles will ha#e comme ts. )se them as a g"i!e. 5o" will o+te see li es i co +ig"ratio +iles that are commented out to pre#e t them +rom %ei g "se! %y the a++ecte! program. This is !o e to gi#e the rea!er s"ggestio s +or possi%le co +ig"ratio choices or e3amples o+ correct co +ig"ratio sy ta3. 0or e3ample, the <bashrc +ile o+ )%" t" 8.01 co tai s these li es, ^ some ^alias ^alias ^alias more ls aliases llMKls -lK laMKls -4K lMKls -C-K

The last three li es are #ali! alias !e+i itio s that ha#e %ee comme te! o"t. 9+ yo" remo#e the lea!i g ?P@ sym%ols +rom these three li es, a tech i="e calle! uncommenting, yo" will acti#ate the aliases. Co #ersely, i+ yo" a!! a ?P@ sym%ol to the %egi i g o+ a li e, yo" ca !eacti#ate a co +ig"ratio li e while preser#i g the i +ormatio it co tai s.

.(8

>o!i+yi g The E #iro me t

A/ti;ating <ur Changes

The cha ges we ha#e ma!e to o"r <bashrc will ot take a++ect " til we close o"r termi al sessio a ! start a ew o e, si ce the <bashrc +ile is o ly rea! at the %egi i g o+ a sessio . Aowe#er, we ca +orce bash to re-rea! the mo!i+ie! <bashrc +ile with the +ollowi g comma !,
[me@linuxbox ~]$ source .bashrc

$+ter !oi g this, we sho"l! %e a%le to see the e++ect o+ o"r cha ges. Try o"t o e o+ the ew aliases,
[me@linuxbox ~]$ ll

Summing -p
9 this chapter we lear e! a esse tial skillBe!iti g co +ig"ratio +iles with a te3t e!itor. >o#i g +orwar!, as we rea! ma pages +or comma !s, take ote o+ the e #iro me t #aria%les that comma !s s"pport. There may %e a gem or two. 9 later chapters, we will lear a%o"t shell +" ctio s, a power+"l +eat"re that yo" ca also i cl"!e i the bash start"p +iles to a!! to yo"r arse al o+ c"stom comma !s.

;urther 1eading

The 9&R7C$T97& sectio o+ the bash ma page co#ers the bash start"p +iles i gory !etail.

.(9

.( 8 $ ;e tle 9 tro!"ctio To #i

13 ) 3entle Introduction To vi
There is a ol! 4oke a%o"t a #isitor to &ew 5ork City aski g a passer%y +or !irectio s to the city<s +amo"s classical m"sic #e "e, Risitor, E3c"se me, how !o 9 get to Car egie AallC 6asser%y, 6ractice, practice, practiceL *ear i g the *i "3 comma ! li e, like %ecomi g a accomplishe! pia ist, is ot somethi g that we pick "p i a a+ter oo . 9t takes years o+ practice. 9 this chapter, we will i tro!"ce the vi Epro o" ce! ?#ee eye@F te3t e!itor, o e o+ the core programs i the ) i3 tra!itio . vi is somewhat otorio"s +or its !i++ic"lt "ser i ter+ace, %"t whe we see a master sit !ow at the key%oar! a ! %egi to ?play,@ we will i !ee! %e wit ess to some great art. We wo <t %ecome masters i this chapter, %"t whe we are !o e, we will k ow how to play ?chopsticks@ i vi.

+h, +e Should Learn %i

9 this mo!er age o+ graphical e!itors a ! easy-to-"se te3t-%ase! e!itors s"ch as nano, why sho"l! we lear viC There are three goo! reaso s,

vi is always a#aila%le. This ca %e a li+esa#er i+ we ha#e a system with o graphical i ter+ace, s"ch as a remote ser#er or a local system with a %roke G co +ig"ratio . nano, while i creasi gly pop"lar is still ot " i#ersal. 67S9G, a sta !ar! +or program compati%ility o ) i3 systems, re="ires that vi %e prese t. vi is lightweight a ! +ast. 0or ma y tasks, it<s easier to %ri g "p vi tha it is to +i ! the graphical te3t e!itor i the me "s a ! wait +or its m"ltiple mega%ytes to loa!. 9 a!!itio , vi is !esig e! +or typi g spee!. $s we shall see, a skille! vi "ser e#er has to li+t his or her +i gers +rom the key%oar! while e!iti g. We !o <t wa t other *i "3 a ! ) i3 "sers to thi k we are sissies.

7kay, may%e two goo! reaso s.

.10

$ *ittle Jackgro" !

# Little /ac(ground
The +irst #ersio o+ vi was writte i .9/K %y Jill Joy, a ) i#ersity o+ Cali+or ia at Jerkley st"!e t who later we t o to co-+o" ! S" >icrosystems. vi !eri#es its ame +rom the wor! ?#is"al,@ %eca"se it was i te !e! to allow e!iti g o a #i!eo termi al with a mo#i g c"rsor. 6re#io"s to visual editors, there were line editors which operate! o a si gle li e o+ te3t at a time. To speci+y a cha ge, we tell a li e e!itor to go to a partic"lar li e a ! !escri%e what cha ge to make, s"ch as a!!i g or !eleti g te3t. With the a!#e t o+ #i!eo termi als Erather tha pri ter-%ase! termi als like teletypesF #is"al e!iti g %ecame possi%le. vi act"ally i corporates a power+"l li e e!itor calle! ex, a ! we ca "se li e e!iti g comma !s while "si g vi. >ost *i "3 !istri%"tio s !o <t i cl"!e real viH rather, they ship with a e ha ce! replaceme t calle! vim Ewhich is short +or ?#i impro#e!@F writte %y Jram >oole aar. vim is a s"%sta tial impro#eme t o#er tra!itio al ) i3 vi a ! is "s"ally sym%olically li ke! Eor aliase!F to the ame ?#i@ o *i "3 systems. 9 the !isc"ssio s that +ollow, we will ass"me that we ha#e a program calle! ?#i@ that is really vim.

Starting #nd Stopping %i

To start vi, we simply type the +ollowi g,
[me@linuxbox ~]$ vi

$ ! a scree like this sho"l! appear,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~

;A+ - ;i Am7roved version )<#<#$/ b1 Hram +oolenaar et al< ;im is o7en source and freel1 distributable t17e t17e t17e t17e t17e *7onsor ;im develo7mentG :hel7 s7onsorX&nterD for information :BX&nterD :hel7X&nterD or X-#D :hel7 version)X&nterD to exit for on-line hel7 for version info

Running in ;i com7atible mode :set noc7X&nterD for ;im defaults

.1.

.( 8 $ ;e tle 9 tro!"ctio To #i
~ ~ ~ ~ t17e :hel7 c7-defaultX&nterD for info on this

J"st as we !i! with nano earlier, the +irst thi g to lear is how to e3it. To e3it, we e ter the +ollowi g comma ! E ote that the colo character is part o+ the comma !F,
2,

The shell prompt sho"l! ret"r . 9+, +or some reaso , vi will ot ="it E"s"ally %eca"se we ma!e a cha ge to a +ile that has ot yet %ee sa#e!F, we ca tell vi that we really mea it %y a!!i g a e3clamatio poi t to the comma !,
2,L

Ti(5 9+ yo" get ?lost@ i vi, try pressi g the &sc key twice to +i ! yo"r way agai .

Compati0ilit, >ode
9 the e3ample start"p scree a%o#e Etake +rom )%" t" 8.01F, we see the te3t ?:" i g i Ri compati%le mo!e.@ This mea s that vim will r" i a mo!e that is closer to the ormal %eha#ior o+ vi rather tha the e ha ce! %eha#ior o+ vim. 0or p"rposes o+ this chapter, we will wa t to r" vim with its e ha ce! %eha#ior. To !o this, yo" ha#e a +ew optio s, Try r" i g vim i stea! o+ vi. 9+ that works, co si!er a!!i g alias viMKvimK to yo"r <bashrc +ile. $lter ately, "se this comma ! to a!! a li e to yo"r vim co +ig"ratio +ile, echo _set noc7_ DD ~6<vimrc 'i++ere t *i "3 !istri%"tio s package vim i !i++ere t ways. Some !istri%"tio s i stall a mi imal #ersio o+ vim %y !e+a"lt that o ly s"pports a limiti g set o+

.12

Starti g $ ! Stoppi g #i

vim +eat"res. While pre+ormi g the lesso s that +ollow, yo" may e co" ter missi g +eat"res. 9+ this is the case, i stall the +"ll #ersio o+ vim.

$diting >odes
*et<s start "p vi agai , this time passi g to it the ame o+ a o e3iste t +ile. This is how we ca create a ew +ile with vi,
[me@linuxbox ~]$ rm -f foo.txt [me@linuxbox ~]$ vi foo.txt

9+ all goes well, we sho"l! get a scree like this,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ _foo<txt_ [Le9 -ile]

The lea!i g til!e characters E@S@F i !icate that o te3t e3ists o that li e. This shows that we ha#e a empty +ile. /o not ty(e anything yet2 The seco ! most importa t thi g to lear a%o"t vi Ea+ter lear i g how to e3itF is that vi .1(

.( 8 $ ;e tle 9 tro!"ctio To #i is a modal editor. Whe vi starts "p, it %egi s i command mode. 9 this mo!e, almost e#ery key is a comma !, so i+ we were to start typi g, vi wo"l! %asically go craDy a ! make a %ig mess.

6ntering Insert :ode

9 or!er to a!! some te3t to o"r +ile, we m"st +irst e ter insert mode. To !o this, we press the ?i@ key. $+terwar!s, we sho"l! see the +ollowi g at the %ottom o+ the scree i+ vim is r" i g i its "s"al e ha ce! mo!e Ethis will ot appear i vi compati%le mo!eF,
-- P?FGH+ --

&ow we ca e ter some te3t. Try this,

+he ,uick brown fox jumped over the la-! dog.

To e3it i sert mo!e a ! ret"r to comma ! mo!e, press the &sc key.

Sa;ing <ur Work

To sa#e the cha ge we 4"st ma!e to o"r +ile, we m"st e ter a e+ command while i comma ! mo!e. This is easily !o e %y pressi g the ?,@ key. $+ter !oi g this, a colo character sho"l! appear at the %ottom o+ the scree ,
:

To write o"r mo!i+ie! +ile, we +ollow the colo with a ?w@ the E ter,
:w

The +ile will %e writte to the har! !ri#e a ! we sho"l! get a co +irmatio message at the %ottom o+ the scree , like this,
_foo<txt_ [Le9] #=> %.C 9ritten

.11

E!iti g >o!es Ti(5 9+ yo" rea! the vim !oc"me tatio , yo" will otice that Eco +"si glyF comma ! mo!e is calle! normal mode a ! ex comma !s are calle! command mode. Jeware.

>o%ing The Cursor #round

While i comma ! mo!e, vi o++ers a large "m%er o+ mo#eme t comma !s, some o+ which it shares with less. Aere is a s"%set, Table 13-1: %ursor !ovement ?eys =e, l or :ight $rrow h or *e+t $rrow j or 'ow $rrow k or )p $rrow ( EDeroF ` $ 9 , b H Ctrl-f or :age 'o9n Ctrl-b or :age 37 number@ @ >o%es The Cursor :ight o e character. *e+t o e character. 'ow o e li e. )p o e li e. To the %egi i g o+ the c"rre t li e. To the +irst o -whitespace character o the c"rre t li e. To the e ! o+ the c"rre t li e. To the %egi i g o+ the e3t wor! or p" ct"atio character. To the %egi i g o+ the e3t wor!, ig ori g p" ct"atio characters. To the %egi i g o+ the pre#io"s wor! or p" ct"atio character. To the %egi i g o+ the pre#io"s wor!, ig ori g p" ct"atio characters. 'ow o e page. )p o e page. To li e number. 0or e3ample, #@ mo#es to the +irst li e o+ the +ile. To the last li e o+ the +ile. .12

.( 8 $ ;e tle 9 tro!"ctio To #i

Why are the h, j, k, a ! l keys "se! +or c"rsor mo#eme tC Jeca"se whe vi was origi ally writte , ot all #i!eo termi als ha! arrow keys, a ! skille! typists co"l! "se reg"lar key%oar! keys to mo#e the c"rsor witho"t e#er ha#i g to li+t their +i gers +rom the key%oar!. >a y comma !s i vi ca %e pre+i3e! with a "m%er, as with the ?;@ comma ! liste! a%o#e. Jy pre+i3i g a comma ! with a "m%er, we may speci+y the "m%er o+ times a comma ! is to %e carrie! o"t. 0or e3ample, the comma ! ?24@ ca"ses vi to mo#e the c"rsor !ow +i#e li es.

/asic $diting
>ost e!iti g co sists o+ a +ew %asic operatio s s"ch as i serti g te3t, !eleti g te3t a ! mo#i g te3t aro" ! %y c"tti g a ! pasti g. vi, o+ co"rse, s"pports all o+ these operatio s i its ow " i="e way. vi also pro#i!es a limite! +orm o+ " !o. 9+ we press the ?"@ key while i comma ! mo!e, vi will " !o the last cha ge that yo" ma!e. This will come i ha !y as we try o"t some o+ the %asic e!iti g comma !s.

A55ending Text
vi has se#eral !i++ere t ways o+ e teri g i sert mo!e. We ha#e alrea!y "se! the i comma ! to i sert te3t. *et<s go %ack to o"r foo<txt +ile +or a mome t,
The Buick bro9n fox jum7ed over the laC1 dog<

9+ we wa te! to a!! some te3t to the e ! o+ this se te ce, we wo"l! !isco#er that the i comma ! will ot !o it, si ce we ca <t mo#e the c"rsor %eyo ! the e ! o+ the li e. vi pro#i!es a comma ! to appe ! te3t, the se si%ly ame! ?a@ comma !. 9+ we mo#e the c"rsor to the e ! o+ the li e a ! type ?a@, the c"rsor will mo#e past the e ! o+ the li e a ! vi will e ter i sert mo!e. This will allow "s to a!! some more te3t,
The Buick bro9n fox jum7ed over the laC1 dog< Pt was cool.

:emem%er to press the &sc key to e3it i sert mo!e. Si ce we will almost always wa t to appe ! te3t to the e ! o+ a li e, vi o++ers a shortc"t to mo#e to e ! o+ the c"rre t li e a ! start appe !i g. 9t<s the ?$@ comma !. *et<s try it .1K

Jasic E!iti g a ! a!! some more li es to o"r +ile. 0irst, we<ll mo#e the c"rsor to the %egi i g o+ the li e "si g the ?0@ EDeroF comma !. &ow we type ?$@ a ! a!! the +ollowi g li es o+ te3t,
The Buick bro9n fox jum7ed over the laC1 dog< Tine 2 Tine Tine A Tine " At 9as cool<

$gai , press the &sc key to e3it i sert mo!e. $s we ca see, the ?$@ comma ! is more "se+"l as it mo#es the c"rsor to the e ! o+ the li e %e+ore starti g i sert mo!e.

<5ening A Line
$ other way we ca i sert te3t is %y ?ope i g@ a li e. This i serts a %la k li e %etwee two e3isti g li es a ! e ters i sert mo!e. This has two #aria ts, Table 13-': (ine &pening ?eys Command o ?pens The li e %elow the c"rre t li e. The li e a%o#e the c"rre t li e. We ca !emo strate this as +ollows, place the c"rsor o ?*i e (@ the press the o key.
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " At 9as cool<

$ ew li e was ope e! %elow the thir! li e a ! we e tere! i sert mo!e. E3it i sert mo!e %y pressi g the &sc key. 6ress the u key to " !o o"r cha ge. 6ress the key to ope the li e a%o#e the c"rsor,

.1/

.( 8 $ ;e tle 9 tro!"ctio To #i
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " At 9as cool<

E3it i sert mo!e %y pressi g the &sc key a ! " !o o"r cha ge %y pressi g u.

)e eting Text
$s we might e3pect, vi o++ers a #ariety o+ ways to !elete te3t, all o+ which co tai o e o+ two keystrokes. 0irst, the x key will !elete a character at the c"rsor locatio . x may %e prece!e! %y a "m%er speci+yi g how ma y characters are to %e !elete!. The d key is more ge eral p"rpose. *ike x, it may %e prece!e! %y a "m%er speci+yi g the "m%er o+ times the !eletio is to %e per+orme!. 9 a!!itio , d is always +ollowe! %y a mo#eme t comma ! that co trols the siDe o+ the !eletio . Aere are some e3amples, Table 13-3: Te+t -eletion %ommands Command x $x dd "dd d, d$ d( d` d@ d!(@ Deletes The c"rre t character. The c"rre t character a ! the e3t two characters. The c"rre t li e. The c"rre t li e a ! the e3t +o"r li es. 0rom the c"rre t c"rsor positio to the %egi the e3t wor!. i g o+

0rom the c"rre t c"rsor locatio to the e ! o+ the c"rre t li e. 0rom the c"rre t c"rsor locatio to the %egi the li e. i g o+

0rom the c"rre t c"rsor locatio to the +irst o whitespace character i the li e. 0rom the c"rre t li e to the e ! o+ the +ile. 0rom the c"rre t li e to the twe tieth li e o+ the +ile.

6lace the c"rsor o the wor! ?9t@ o the +irst li e o+ o"r te3t. 6ress the x key repeate!ly .18

Jasic E!iti g " til the rest o+ the se te ce is !elete!. &e3t, press the u key repeate!ly " til the !eletio is " !o e. 1ote5 :eal vi o ly s"pports a si gle le#el o+ " !o. vim s"pports m"ltiple le#els. *et<s try the !eletio agai , this time "si g the d comma !. $gai , mo#e the c"rsor to the wor! ?9t@ a ! press d, to !elete the wor!,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " 9as cool<

6ress d$ to !elete +rom the c"rsor positio to the e ! o+ the li e,

The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine "

6ress d@ to !elete +rom the c"rre t li e to the e ! o+ the +ile,

~ ~ ~ ~ ~

6ress u three times to " !o the !eletio .

CuttingC Co5ying And 'asting Text

The d comma ! ot o ly !eletes te3t, it also ?c"ts@ te3t. Each time we "se the d comma ! the !eletio is copie! i to a paste %"++er Ethi k clip%oar!F that we ca later recall with the 7 comma ! to paste the co te ts o+ the %"++er a+ter the c"rsor or the : comma ! to paste the co te ts %e+ore the c"rsor.

.19

.( 8 $ ;e tle 9 tro!"ctio To #i The 1 comma ! is "se! to ?ya k@ EcopyF te3t i m"ch the same way the d comma ! is "se! to c"t te3t. Aere are some e3amples com%i i g the 1 comma ! with #ario"s mo#eme t comma !s, Table13- $: Canking %ommands Command 11 "11 1, 1$ 1( 1` 1@ 1!(@ Copies The c"rre t li e. The c"rre t li e a ! the e3t +o"r li es. 0rom the c"rre t c"rsor positio to the %egi the e3t wor!. i g o+

0rom the c"rre t c"rsor locatio to the e ! o+ the c"rre t li e. 0rom the c"rre t c"rsor locatio to the %egi the li e. i g o+

0rom the c"rre t c"rsor locatio to the +irst o whitespace character i the li e. 0rom the c"rre t li e to the e ! o+ the +ile. 0rom the c"rre t li e to the twe tieth li e o+ the +ile.

*et<s try some copy a ! paste. 6lace the c"rsor o the +irst li e o+ the te3t a ! type 11 to copy the c"rre t li e. &e3t, mo#e the c"rsor to the last li e E @F a ! type 7 to paste the li e %elow the c"rre t li e,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " +he ,uick brown fox jumped over the la-! dog. At 9as cool<

Pt was cool.

J"st as %e+ore, the u comma ! will " !o o"r cha ge. With the c"rsor still positio e! o the last li e o+ the +ile, type : to paste the te3t a%o#e the c"rre t li e,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! At 9as cool<

.20

Jasic E!iti g
=ine $ =ine % +he ,uick brown fox jumped over the la-! dog. =ine "

Pt was cool.

Try o"t some o+ the other 1 comma !s i the ta%le a%o#e a ! get to k ow the %eha#ior o+ %oth the 7 a ! : comma !s. Whe yo" are !o e, ret"r the +ile to its origi al state.

Goining Lines
vi is rather strict a%o"t its i!ea o+ a li e. &ormally, it is ot possi%le to mo#e the c"rsor to the e ! o+ a li e a ! !elete the e !-o+-li e character to 4oi o e li e with the o e %elow it. Jeca"se o+ this, vi pro#i!es a speci+ic comma !, ? E ot to %e co +"se! with j, which is +or c"rsor mo#eme tF to 4oi li es together. 9+ we place the c"rsor o li e ( a ! type the ? comma !, here<s what happe s,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! Tine Tine A =ine " At 9as cool<

Search #nd 1eplace

vi has the a%ility to mo#e the c"rsor to locatio s %ase! o searches. 9t ca !o this o %oth a si gle li e or o#er a e tire +ile. 9t ca also per+orm te3t replaceme ts with or witho"t co +irmatio +rom the "ser.

Sear/hing Within A Line

The f comma ! searches a li e a ! mo#es the c"rsor to the e3t i sta ce o+ a speci+ie! character. 0or e3ample, the comma ! fa wo"l! mo#e the c"rsor to the e3t occ"rre ce o+ the character ?a@ withi the c"rre t li e. $+ter per+ormi g a character search withi a li e, the search may %e repeate! %y typi g a semicolo .

Sear/hing The 6ntire 2i e

To mo#e the c"rsor to the e3t occ"rre ce o+ a wor! or phrase, the 6 comma ! is "se!. This works the same way as we lear e! earlier i the less program. Whe yo" type the 6 comma ! a ?-@ will appear at the %ottom o+ the scree . &e3t, type the wor! or phrase

.2.

.( 8 $ ;e tle 9 tro!"ctio To #i to %e searche! +or, +ollowe! %y the &nter key. The c"rsor will mo#e to the e3t locatio co tai i g the search stri g. $ search may %e repeate! "si g the pre#io"s search stri g with the n comma !. Aere<s a e3ample,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " At 9as cool<

6lace the c"rsor o the +irst li e o+ the +ile. Type,

/Tine

+ollowe! %y the &nter key. The c"rsor will mo#e to li e 2. &e3t, type n a ! the c"rsor will mo#e to li e (. :epeati g the n comma ! will mo#e the c"rsor !ow the +ile " til it r" s o"t o+ matches. While we ha#e so +ar o ly "se! wor!s a ! phrases +or o"r search patter s, vi allows the "se o+ regular e+pressions, a power+"l metho! o+ e3pressi g comple3 te3t patter s. We will co#er reg"lar e3pressio s i some !etail i a later chapter.

+ oba Sear/h And !e5 a/e

vi "ses a e3 comma ! to per+orm search a ! replace operatio s Ecalle! ?s"%stit"tio @ i viF o#er a ra ge o+ li es or the e tire +ile. To cha ge the wor! ?*i e@ to ?li e@ +or the e tire +ile, we wo"l! e ter the +ollowi g comma !,
28s/Tine/line/g

*et<s %reak this comma ! !ow i to separate items a ! see what each o e !oes, Item : 5 >eaning The colo character starts a e3 comma !. Speci+ies the ra ge o+ li es +or the operatio . 5 is a shortc"t mea i g +rom the +irst li e to the last li e. $lter ately, the ra ge co"l! ha#e %ee speci+ie! #>" Esi ce o"r +ile is +i#e li es lo gF, or #>$ which mea s ?+rom li e . to the last li e i the +ile.@ 9+ the ra ge o+ li es is omitte!, the operatio is o ly

.22

Search $ ! :eplace per+orme! o the c"rre t li e. s 6=ine6line6 g Speci+ies the operatio . 9 this case, s"%stit"tio Esearch a ! replaceF. The search patter a ! the replaceme t te3t. This mea s ?glo%al@ i the se se that the search a ! replace is per+orme! o e#ery i sta ce o+ the search stri g i the li e. 9+ omitte!, o ly the +irst i sta ce o+ the search stri g o each li e is replace!.

$+ter e3ec"ti g o"r search a ! replace comma ! o"r +ile looks like this,
The Buick bro9n fox jum7ed over the laC1 dog< line ! line $ line % line " At 9as cool<

We ca also speci+y a s"%stit"tio comma ! with "ser co +irmatio . This is !o e %y a!!i g a ?c@ to the e ! o+ the comma !. 0or e3ample,
28s/line/Tine/gc

This comma ! will cha ge o"r +ile %ack to its pre#io"s +ormH howe#er, %e+ore each s"%stit"tio , vi stops a ! asks "s to co +irm the s"%stit"tio with this message,
re7lace 9ith =ine N16n6a6B6l6`&6`UPF

Each o+ the characters withi the pare theses is a possi%le choice as +ollows, Table 13-.: 9eplace %onfirmation ?eys =e, 1 n a #ction 6er+orm the s"%stit"tio . Skip this i sta ce o+ the patter . 6er+orm the s"%stit"tio o this a ! all s"%se="e t i sta ces .2(

.( 8 $ ;e tle 9 tro!"ctio To #i o+ the patter . B or &sc l Ctrl-e> Ctrl-1 T"it s"%stit"ti g. 6er+orm this s"%stit"tio a ! the ="it. Short +or ?last.@ Scroll !ow a ! scroll "p, respecti#ely. )se+"l +or #iewi g the co te3t o+ the propose! s"%stit"tio .

9+ yo" type 1, the s"%stit"tio will %e per+orme!, n will ca"se vi to skip this i sta ce a ! mo#e o to the e3t o e.

$diting >ultiple ;iles

9t<s o+te "se+"l to e!it more tha o e +ile at a time. 5o" might ee! to make cha ges to m"ltiple +iles or yo" may ee! to copy co te t +rom o e +ile i to a other. With vi we ca ope m"ltiple +iles +or e!iti g %y speci+yi g them o the comma ! li e,
vi file1 file2 file3...

*et<s e3it o"r e3isti g #i sessio a ! create a ew +ile +or e!iti g. Type :9B to e3it vi sa#i g o"r mo!i+ie! te3t. &e3t, we<ll create a a!!itio al +ile i o"r home !irectory that we ca play with. We<ll create the +ile %y capt"ri g some o"tp"t +rom the ls comma !,
[me@linuxbox ~]$ ls -l /usr/bin & ls-output.txt

*et<s e!it o"r ol! +ile a ! o"r ew o e with vi,

[me@linuxbox ~]$ vi foo.txt ls-output.txt

vi will start "p a ! we will see the +irst +ile o the scree ,
The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " At 9as cool<

.21

E!iti g >"ltiple 0iles

S%it/hing Bet%een 2i es
To switch +rom o e +ile to the e3t, "se this e3 comma !,
2n

To mo#e %ack to the pre#io"s +ile "se,

2?

While we ca mo#e +rom o e +ile to a other, vi e +orces a policy that pre#e ts "s +rom switchi g +iles i+ the c"rre t +ile has " sa#e! cha ges. To +orce vi to switch +iles a ! a%a !o yo"r cha ges, a!! a e3clamatio poi t ELF to the comma !. 9 a!!itio to the switchi g metho! !escri%e! a%o#e, vim Ea ! some #ersio s o+ viF also pro#i!e some e3 comma !s that make m"ltiple +iles easier to ma age. We ca #iew a list o+ +iles %ei g e!ite! with the :buffers comma !. 'oi g so will !isplay a list o+ the +iles at the %ottom o+ the !isplay,
2buffers # 5a _foo<txt_ line # ! _ls-out7ut<txt_ line ( :ress &LT&R or t17e command to continue

To switch to a other %"++er E+ileF, type :buffer +ollowe! %y the "m%er o+ the %"++er yo" wish to e!it. 0or e3ample, to switch +rom %"++er . which co tai s the +ile foo<txt to %"++er two co tai i g the +ile ls-out7ut<txt we wo"l! type this,
2buffer 2

a ! o"r scree

ow !isplays the seco ! +ile.

<5ening Additiona 2i es 2or 6diting

9t<s also possi%le to a!! +iles to o"r c"rre t e!iti g sessio . The e3 comma ! :e Eshort +or ?e!it@F +ollowe! %y a +ile ame will ope a a!!itio al +ile. *et<s e ! o"r c"rre t e!iti g sessio a ! ret"r to the comma ! li e.

.22

.( 8 $ ;e tle 9 tro!"ctio To #i Start vi agai with 4"st o e +ile,

[me@linuxbox ~]$ vi foo.txt

To a!! o"r seco ! +ile, e ter,

2e ls-output.txt

$ ! it sho"l! appear o the scree . The +irst +ile is still prese t as we ca #eri+y,
2buffers # ^ _foo<txt_ line # ! 5a _ls-out7ut<txt_ line ( :ress &LT&R or t17e command to continue

1ote5 5o" ca ot switch to +iles loa!e! with the :e comma ! "si g either the :n or :L comma !. To switch +iles, "se the :buffer comma ! +ollowe! %y the %"++er "m%er.

Co5ying Content 2rom <ne 2i e Into Another

7+te while e!iti g m"ltiple +iles, we will wa t to copy a portio o+ o e +ile i to a other +ile that we are e!iti g. This is easily !o e "si g the "s"al ya k a ! paste comma !s we "se! earlier. We ca !emo strate as +ollows. 0irst, "si g o"r two +iles, switch to %"++er . Efoo<txtP %y e teri g,
2buffer 1

which sho"l! gi#e "s this,

.2K

E!iti g >"ltiple 0iles

The Buick bro9n fox jum7ed over the laC1 dog< =ine ! =ine $ =ine % =ine " At 9as cool<

&e3t, mo#e the c"rsor to the +irst li e, a ! type 11 to ya k EcopyF the li e. Switch to the seco ! %"++er %y e teri g,
2buffer 2

The scree will ow co tai some +ile listi gs like this Eo ly a portio is show hereF,
total $%$)(( -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x #

root root root root root root

root root root root root root

$#$#. /!%( ###!). !"$./ ##"$! )!0!

!(()-#!-(" !(()-#!-(0 !((/-(#-$# !((.-#(-(. !(()-("-(% !(()-("-(%

(/:"/ #$:$0 #$:$. !(:#. #):%$ #):%$

[ %##to77m a!7 a"!dec aafire aainfo

>o#e the c"rsor to the +irst li e a ! paste the li e we copie! +rom the prece!i g +ile %y typi g the 7 comma !,
total $%$)(( +he ,uick brown fox jumped over the la-! dog. -r9xr-xr-x # root root $#$#. !(()-#!-(" -r9xr-xr-x # root root /!%( !(()-#!-(0 -r9xr-xr-x # root root ###!). !((/-(#-$# -r9xr-xr-x # root root !"$./ !((.-#(-(. -r9xr-xr-x # root root ##"$! !(()-("-(% -r9xr-xr-x # root root )!0! !(()-("-(%

Pt was cool. (/:"/ [ #$:$0 %##to77m #$:$. a!7 !(:#. a"!dec #):%$ aafire #):%$ aainfo

Inserting An 6ntire 2i e Into Another

9t<s also possi%le to i sert a e tire +ile i to o e that we are e!iti g. To see this i actio , let<s e ! o"r vi sessio a ! start a ew o e with 4"st a si gle +ile,

.2/

.( 8 $ ;e tle 9 tro!"ctio To #i
[me@linuxbox ~]$ vi ls-output.txt

We will see o"r +ile listi g agai ,

total $%$)(( -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x #

root root root root root root

root root root root root root

$#$#. /!%( ###!). !"$./ ##"$! )!0!

!(()-#!-(" !(()-#!-(0 !((/-(#-$# !((.-#(-(. !(()-("-(% !(()-("-(%

(/:"/ #$:$0 #$:$. !(:#. #):%$ #):%$

[ %##to77m a!7 a"!dec aafire aainfo

>o#e the c"rsor to the thir! li e, the e ter the +ollowi g e3 comma !,
2r foo.txt

The :r comma ! Eshort +or ?rea!@F i serts the speci+ie! +ile %e+ore the c"rsor positio . 7"r scree sho"l! ow look like this,
total $%$)(( -r9xr-xr-x # root root $#$#. !(()-#!-(" -r9xr-xr-x # root root /!%( !(()-#!-(0 +he ,uick brown fox jumped over the la-! dog. Tine 2 Tine Tine A Tine " -r9xr-xr-x # root root ###!). !((/-(#-$# -r9xr-xr-x # root root !"$./ !((.-#(-(. -r9xr-xr-x # root root ##"$! !(()-("-(% -r9xr-xr-x # root root )!0! !(()-("-(%

(/:"/ [ #$:$0 %##to77m Pt was cool.

#$:$. !(:#. #):%$ #):%$

a!7 a"!dec aafire aainfo

Sa%ing ?ur +or(

*ike e#erythi g else i vi, there are se#eral !i++ere t ways to sa#e o"r e!ite! +iles. We ha#e alrea!y co#ere! the e3 comma ! :9, %"t there are some others we may also +i ! help+"l. 9 comma ! mo!e, typi g II will sa#e the c"rre t +ile a ! e3it vi. *ikewise, the e3 comma ! :9B will com%i e the :9 a ! :B comma !s i to o e that will %oth sa#e the .28

Sa#i g 7"r Work +ile a ! e3it. The :9 comma ! may also speci+y a optio al +ile ame. This acts like ?Sa#e $s...@ 0or e3ample, i+ we were e!iti g foo<txt a ! wa te! to sa#e a alter ate #ersio calle! foo#<txt, we wo"l! e ter the +ollowi g,
2w foo1.txt

1ote5 While the comma ! a%o#e sa#es the +ile " !er a ew ame, it !oes ot cha ge the ame o+ the +ile yo" are e!iti g. $s yo" co ti "e to e!it, yo" will still %e e!iti g foo<txt, ot foo#<txt.

;urther 1eading
E#e with all that we ha#e co#ere! i this chapter, we ha#e %arely scratche! the s"r+ace o+ what vi a ! vim ca !o. Aere are a co"ple o+ o -li e reso"rces yo" ca "se to co ti "e yo"r 4o"r ey towar!s vi mastery,

(earning The vi 1ditor 8 $ Wiki%ook +rom Wikipe!ia that o++ers a co cise g"i!e to vi a ! se#eral o+ its work-a-likes i cl"!i g vim. 9t<s a#aila%le at, http,--e .wiki%ooks.org-wiki-Ri The Bim 8ook - The vim pro4ect has a 2/0-page %ook that co#ers EalmostF all o+ the +eat"res i vim. 5o" ca +i ! it at, +tp,--+tp.#im.org-p"%-#im-!oc-%ook-#im%ook-76*.p!+. $ Wikipe!ia article o Jill Joy, the creator o+ vi., http,--e .wikipe!ia.org-wiki-JillOJoy $ Wikipe!ia article o Jram >oole aar, the a"thor o+ vim, http,--e .wikipe!ia.org-wiki-JramO>oole aar

.29

.1 8 C"stomiDi g The 6rompt

1 Custo%i4ing The 2ro%#t

9 this chapter we will look at a seemi gly tri#ial !etailBo"r shell prompt. This e3ami atio will re#eal some o+ the i er worki gs o+ the shell a ! the termi al em"lator program itsel+. *ike so ma y thi gs i *i "3, the shell prompt is highly co +ig"ra%le, a ! while we ha#e pretty m"ch take it +or gra te!, the prompt is a really "se+"l !e#ice o ce we lear how to co trol it.

#natom, ?" # Prompt

7"r !e+a"lt prompt looks somethi g like this,
[me@linuxbox ~]$

&otice that it co tai s o"r "ser ame, o"r host ame a ! o"r c"rre t worki g !irectory, %"t how !i! it get that wayC Rery simply, it t"r s o"t. The prompt is !e+i e! %y a e #iro me t #aria%le ame! :*# Eshort +or ?prompt stri g o e@F. We ca #iew the co te ts o+ :*# with the echo comma !,
[me@linuxbox ~]$ echo 3BF1 []u@]h ],]]$

1ote5 'o <t worry i+ yo"r res"lts are ot e3actly the same as the e3ample a%o#e. E#ery *i "3 !istri%"tio !e+i es the prompt stri g a little !i++ere tly, some ="ite e3otically. 0rom the res"lts, we ca see that :*# co tai s a +ew o+ the characters we see i o"r prompt s"ch as the %rackets, the at-sig , a ! the !ollar sig , %"t the rest are a mystery. The ast"te amo g "s will recog iDe these as backslash-escaped special characters like .K0

$ atomy 7+ $ 6rompt those we saw i Chapter 8. Aere is a partial list o+ the characters that the shell treats specially i the prompt stri g, Table 1$-1: 1scape %odes 0sed *n Shell 6rompts SeBuence ]a ]d ]h ]S ]j ]l ]n ]r ]s ]t ]T ]@ ]4 ]u ]v ]; ]9 ], ]G ]^ ]$ ][ Value Displa,ed $SC99 %ell. This makes the comp"ter %eep whe it is e co" tere!. C"rre t !ate i !ay, mo th, !ate +ormat. 0or e3ample, ?>o >ay 2K.@ Aost ame o+ the local machi e mi "s the traili g !omai 0"ll host ame. &"m%er o+ 4o%s r" $ ewli e character. $ carriage ret"r . &ame o+ the shell program. C"rre t time i 21 ho"r ho"rs,mi "tes,seco !s +ormat. C"rre t time i .2 ho"r +ormat. C"rre t time i .2 ho"r $>-6> +ormat. C"rre t time i 21 ho"r ho"rs,mi "tes +ormat. )ser ame o+ the c"rre t "ser. Rersio "m%er o+ the shell. Rersio a ! release "m%ers o+ the shell. &ame o+ the c"rre t worki g !irectory. *ast part o+ the c"rre t worki g !irectory ame. Aistory "m%er o+ the c"rre t comma !. &"m%er o+ comma !s e tere! i to this shell sessio . This !isplays a ?[@ character " less yo" ha#e s"per"ser pri#ileges. 9 that case, it !isplays a ?P@ i stea!. Sig als the start o+ a series o+ o e or more o -pri ti g characters. This is "se! to em%e! o -pri ti g co trol characters which ma ip"late the termi al em"lator i some way, s"ch as mo#i g the .K. i g i the c"rre t shell sessio . &ame o+ the c"rre t termi al !e#ice. ame.

.1 8 C"stomiDi g The 6rompt c"rsor or cha gi g te3t colors. ]] Sig als the e ! o+ a o -pri ti g character se="e ce.

Tr,ing Some #lternate Prompt Designs

With this list o+ special characters, we ca cha ge the prompt to see the e++ect. 0irst, we<ll %ack "p the e3isti g stri g so we ca restore it later. To !o this, we will copy the e3isti g stri g i to a other shell #aria%le that we create o"rsel#es,
[me@linuxbox ~]$ ps1.old$J3BF1J

We create a ew #aria%le calle! 7s#Oold a ! assig the #al"e o+ :*# to it. We ca #eri+y that the stri g has %ee copie! with the echo comma !,
[me@linuxbox ~]$ echo 3ps1.old []u@]h ],]]$

We ca restore the origi al prompt at a y time !"ri g o"r termi al sessio %y simply re#ersi g the process,
[me@linuxbox ~]$ BF1$J3ps1.oldJ

&ow that we are rea!y to procee!, let<s see what happe s i+ we ha#e a empty prompt stri g,
[me@linuxbox ~]$ BF1$

9+ we assig othi g to the prompt stri g, we get othi g. &o prompt stri g at allL The prompt is still there, %"t !isplays othi g, 4"st as we aske! it to. Si ce this is ki ! o+ !isco certi g to look at, we<ll replace it with a mi imal prompt,
BF1$JK3 J

That<s %etter. $t least ow we ca see what we are !oi g. &otice the traili g space withi the !o"%le ="otes. This pro#i!es the space %etwee the !ollar sig a ! the c"rsor .K2

Tryi g Some $lter ate 6rompt 'esig s whe the prompt is !isplaye!. *et<s a!! a %ell to o"r prompt,
$ BF1$JKaK3 J

&ow we sho"l! hear a %eep each time the prompt is !isplaye!. This co"l! get a oyi g, %"t it might %e "se+"l i+ we ee!e! oti+icatio whe a especially lo g-r" i g comma ! has %ee e3ec"te!. &e3t, let<s try to make a i +ormati#e prompt with some host ame a ! time-o+-!ay i +ormatio ,
$ BF1$JK: Kh K3 J #):$$ linuxbox $

$!!i g time-o+-!ay to o"r prompt will %e "se+"l i+ we ee! to keep track o+ whe we per+orm certai tasks. 0i ally, we<ll make a ew prompt that is similar to o"r origi al,
#):$) linuxbox $ BF1$J/KuVKh KW&K3 J Xme@linuxbox ~D$

Try o"t the other se="e ces liste! i the ta%le a%o#e a ! see i+ yo" ca come "p with a %rillia t ew prompt.

#dding Color
>ost termi al em"lator programs respo ! to certai o -pri ti g character se="e ces to co trol s"ch thi gs as character attri%"tes Elike color, %ol! te3t a ! the !rea!e! %li ki g te3tF a ! c"rsor positio . We<ll co#er c"rsor positio i a little %it, %"t +irst we<ll look at color.

Terminal Con"usion
Jack i a cie t times, whe termi als were hooke! to remote comp"ters, there were ma y competi g %ra !s o+ termi als a ! they all worke! !i++ere tly. They ha! !i++ere t key%oar!s a ! they all ha! !i++ere t ways o+ i terpreti g co trol

.K(

.1 8 C"stomiDi g The 6rompt

i +ormatio . ) i3 a ! ) i3-like systems ha#e two rather comple3 s"%systems to !eal with the %a%el o+ termi al co trol Ecalle! termca7 a ! terminfoF. 9+ yo" look i the !eepest recesses o+ yo"r termi al em"lator setti gs yo" may +i ! a setti g +or the type o+ termi al em"latio . 9 a e++ort to make termi als speak some sort o+ commo la g"age, the $merica &atio al Sta !ar!s 9 stit"te E$&S9F !e#elope! a sta !ar! set o+ character se="e ces to co trol #i!eo termi als. 7l! time '7S "sers will remem%er the 4L*A<*U* +ile that was "se! to e a%le i terpretatio o+ these co!es. Character color is co trolle! %y se !i g the termi al em"lator a )>S* escape code em%e!!e! i the stream o+ characters to %e !isplaye!. The co trol co!e !oes ot ?pri t o"t@ o the !isplay, rather it is i terprete! %y the termi al as a i str"ctio . $s we saw i the ta%le a%o#e, the ][ a ! ]] se="e ces are "se! to e caps"late o -pri ti g characters. $ $&S9 escape co!e %egi s with a octal 0(( Ethe co!e ge erate! %y the escape keyF +ollowe! %y a optio al character attri%"te +ollowe! %y a i str"ctio . 0or e3ample, the co!e to set the te3t color to ormal Eattri%"te W 0F, %lack te3t is, ]($$[(T$(m Aere is a ta%le o+ a#aila%le te3t colors. &otice that the colors are !i#i!e! i to two gro"ps, !i++ere tiate! %y the applicatio o+ the %ol! character attri%"te E.F which creates the appeara ce o+ ?light@ colors, Table1$- ': 1scape Se<uences 0sed To Set Te+t %olors SeBuence ]($$[(T$(m ]($$[(T$#m ]($$[(T$!m ]($$[(T$$m ]($$[(T$%m ]($$[(T$"m ]($$[(T$.m ]($$[(T$)m Text Color Jlack :e! ;ree Jrow Jl"e 6"rple Cya *ight ;rey SeBuence ]($$[#T$(m ]($$[#T$#m ]($$[#T$!m ]($$[#T$$m ]($$[#T$%m ]($$[#T$"m ]($$[#T$.m ]($$[#T$)m Text Color 'ark ;ray *ight :e! *ight ;ree 5ellow *ight Jl"e *ight 6"rple *ight Cya White

*et<s try to make a re! prompt. We<ll i sert the escape co!e at the %egi i g, .K1

$!!i g Color
Xme@linuxbox ~D$ BF1$JK(K* Xme@linuxbox ~D$ (*# 1mK)/KuVKh KW&K3 J

That works, %"t otice that all the te3t that we type a+ter the prompt is also re!. To +i3 this, we will a!! a other escape co!e to the e ! o+ the prompt that tells the termi al em"lator to ret"r to the pre#io"s color,
Xme@linuxbox ~D$ BF1$JK(K* Xme@linuxbox ~D$ (*# 1mK)/KuVKh KW&K3K(K* (*mK) J

That<s %etterL 9t<s also possi%le to set the te3t %ackgro" ! color "si g the co!es liste! %elow. The %ackgro" ! colors !o ot s"pport the %ol! attri%"te. Table 1$-3: 1scape Se<uences 0sed To Set 8ackground %olor SeBuence ]($$[(T%(m ]($$[(T%#m ]($$[(T%!m ]($$[(T%$m /ac(ground Color Jlack :e! ;ree Jrow SeBuence ]($$[(T%%m ]($$[(T%"m ]($$[(T%.m ]($$[(T%)m /ac(ground Color Jl"e 6"rple Cya *ight ;rey

We ca create a prompt with a re! %ackgro" ! %y applyi g a simple cha ge to the +irst escape co!e,
Xme@linuxbox ~D$ BF1$JK(K* Xme@linuxbox ~D$ (*#A1mK)/KuVKh KW&K3K(K* (*mK) J

Try o"t the color co!es a ! see what yo" ca createL 1ote5 Jesi!es the ormal E0F a ! %ol! E.F character attri%"tes, te3t may also %e gi#e " !erscore E1F, %li ki g E2F, a ! i #erse E/F attri%"tes as well. 9 the i terests o+ goo! taste, ma y termi al em"lators re+"se to ho or the %li ki g attri%"te, howe#er.

.K2

.1 8 C"stomiDi g The 6rompt

>o%ing The Cursor

Escape co!es ca %e "se! to positio the c"rsor. This is commo ly "se! to pro#i!e a clock or some other ki ! o+ i +ormatio at a !i++ere t locatio o the scree s"ch as a "pper cor er each time the prompt is !raw . Aere is a list o+ the escape co!es that positio the c"rsor, Table 1$-$: %ursor !ovement 1scape Se<uences $scape Code ]($$[lTcS ]($$[n4 ]($$[nH ]($$[nC ]($$[n' ]($$[!? ]($$[2 ]($$[s ]($$[u #ction >o#e the c"rsor to li e l a ! col"m c. >o#e the c"rsor "p n li es. >o#e the c"rsor !ow n li es. >o#e the c"rsor +orwar! n characters. >o#e the c"rsor %ackwar! n characters. Clear the scree a ! mo#e the c"rsor to the "pper le+t cor er Eli e 0, col"m 0F. Clear +rom the c"rsor positio to the e ! o+ the c"rre t li e. Store the c"rre t c"rsor positio . :ecall the store! c"rsor positio .

)si g the co!es a%o#e, we<ll co str"ct a prompt that !raws a re! %ar at the top o+ the scree co tai i g a clock Ere !ere! i yellow te3tF each time the prompt is !isplaye!. The co!e +or the prompt is this +ormi!a%le looki g stri g,
BF1$JK(K* (sK* /KuVKh KW&K3 J (*#*RK* (*#A1mK* (XK* (1# mKtK* (*mK* (uK)

*et<s take a look at each part o+ the stri g to see what it !oes, SeBuence ][ #ction Jegi s a o -pri ti g character se="e ce. The real p"rpose o+ this is to allow bash to correctly calc"late the siDe o+ the #isi%le prompt. Witho"t this, comma ! li e e!iti g +eat"res will improperly positio the c"rsor. Store the c"rsor positio . This is ee!e! to ret"r to the prompt

]($$[s .KK

>o#i g The C"rsor locatio a+ter the %ar a ! clock ha#e %ee !raw at the top o+ the scree . 8e aware that some terminal emulators do not honor this code# ]($$[(T(S ]($$[(T%#m ]($$[2 >o#e the c"rsor to the "pper le+t cor er, which is li e Dero, col"m Dero. Set the %ackgro" ! color to re!. Clear +rom the c"rre t c"rsor locatio Ethe top le+t cor erF to the e ! o+ the li e. Si ce the %ackgro" ! color is ow re!, the li e is cleare! to that color creati g o"r %ar. &ote that cleari g to the e ! o+ the li e !oes ot cha ge the c"rsor positio , which remai s at the "pper le+t cor er. Set the te3t color to yellow. 'isplay the c"rre t time. While this is a ?pri ti g@ eleme t, we still i cl"!e it i the o -pri ti g portio o+ the prompt, si ce we !o <t wa t bash to i cl"!e the clock whe calc"lati g the tr"e siDe o+ the !isplaye! prompt. T"r o++ color. This a++ects %oth the te3t a ! %ackgro" !. :estore the c"rsor positio sa#e! earlier. E ! o -pri ti g characters se="e ce. 6rompt stri g.

]($$[#T$$m ]t

]($$[(m ]($$[u ]] X]u@]h ],D]$

Sa%ing The Prompt

7%#io"sly, we !o <t wa t to %e typi g that mo ster all the time, so we<ll wa t to store o"r prompt someplace. We ca make the prompt perma e t %y a!!i g it to o"r <bashrc +ile. To !o so, a!! these two li es to the +ile,
BF1$JK(K* (sK* /KuVKh KW&K3 J export BF1 (*#*RK* (*#A1mK* (XK* (1# mKtK* (*mK* (uK)

Summing -p
Jelie#e it or ot, there is m"ch more that ca %e !o e with prompts i #ol#i g shell +" ctio s a ! scripts that we ha#e <t co#ere! here, %"t this is a goo! start. &ot e#eryo e .K/

.1 8 C"stomiDi g The 6rompt will care e o"gh to cha ge the prompt, si ce the !e+a"lt prompt is "s"ally satis+actory. J"t +or those o+ "s who like to ti ker, the shell pro#i!es the opport" ity +or ma y ho"rs o+ tri#ial +" .

;urther 1eading

The 8ash 6rompt ,&/T& +rom the *i "3 'oc"me tatio 6ro4ect pro#i!es a pretty complete !isc"ssio o+ what the shell prompt ca %e ma!e to !o. 9t is a#aila%le at, http,--tl!p.org-A7WT7-Jash-6rompt-A7WT7Wikipe!ia has a goo! article o the $&S9 Escape Co!es, http,--e .wikipe!ia.org-wiki-$&S9OescapeOco!e

.K8

6art 1 8 Commo Tasks $ ! Esse tial Tools

'art # A Common Tasks And 6ssentia Too s

.K9

.2 8 6ackage >a ageme t

1& 2ac,age 'anage%ent

9+ we spe ! a y time i the *i "3 comm" ity, we hear ma y opi io s as to which o+ the ma y *i "3 !istri%"tio s is ?%est.@ 7+te , these !isc"ssio s get really silly, +oc"si g o s"ch thi gs as the pretti ess o+ the !esktop %ackgro" ! Esome people wo <t "se )%" t" %eca"se its !e+a"lt color scheme is %row LF a ! other tri#ial matters. The most importa t !etermi a t o+ !istri%"tio ="ality is the packaging system a ! the #itality o+ the !istri%"tio <s s"pport comm" ity. $s we spe ! more time with *i "3, we see that its so+tware la !scape is e3tremely !y amic. Thi gs are co sta tly cha gi g. >ost o+ the top-tier *i "3 !istri%"tio s release ew #ersio s e#ery si3 mo ths a ! ma y i !i#i!"al program "p!ates e#ery !ay. To keep "p with this %liDDar! o+ so+tware, we ee! goo! tools +or package management. 6ackage ma ageme t is a metho! o+ i stalli g a ! mai tai i g so+tware o the system. To!ay, most people ca satis+y all o+ their so+tware ee!s %y i stalli g packages +rom their *i "3 !istri%"tor. This co trasts with the early !ays o+ *i "3, whe o e ha! to !ow loa! a ! compile source code i or!er to i stall so+tware. &ot that there is a ythi g wro g with compili g so"rce co!eH i +act, ha#i g access to so"rce co!e is the great wo !er o+ *i "3. 9t gi#es "s Ea ! e#ery%o!y elseF the a%ility to e3ami e a ! impro#e the system. 9t<s 4"st that ha#i g a pre-compile! package is +aster a ! easier to !eal with. 9 this chapter, we will look at some o+ the comma ! li e tools "se! +or package ma ageme t. While all o+ the ma4or !istri%"tio s pro#i!e power+"l a ! sophisticate! graphical programs +or mai tai i g the system, it is importa t to lear a%o"t the comma ! li e programs, too. They ca per+orm ma y tasks that are !i++ic"lt Eor impossi%leF to !o with their graphical co" terparts.

Pac(aging S,stems
'i++ere t !istri%"tio s "se !i++ere t packagi g systems a ! as a ge eral r"le, a package i te !e! +or o e !istri%"tio is ot compati%le with a other !istri%"tio . >ost !istri%"tio s +all i to o e o+ two camps o+ packagi g tech ologies, the 'e%ia ?.!e%@ camp a ! the :e! Aat ?.rpm@ camp. There are some importa t e3ceptio s s"ch as ;e too, Slackware, a ! 0oresight, %"t most others "se o e o+ these two %asic systems.

./0

6ackagi g Systems Table 1.-1: !aAor 6ackaging System Families Pac(aging S,stem 'e%ia Style E.!e%F :e! Aat Style E.rpmF Distri0utions CPartial ListingD 'e%ia , )%" t", Ga !ros, *i spire 0e!ora, Ce t7S, :e! Aat E terprise *i "3, 7pe S)SE, >a !ri#a, 6C*i "37S

3o4 # Pac(age S,stem +or(s

The metho! o+ so+tware !istri%"tio +o" ! i the proprietary so+tware i !"stry "s"ally e tails %"yi g a piece o+ i stallatio me!ia s"ch as a ?i stall !isk@ a ! the r" i g a ?i stallatio wiDar!@ to i stall a ew applicatio o the system. *i "3 !oes <t work that way. Rirt"ally all so+tware +or a *i "3 system will %e +o" ! o the 9 ter et. >ost o+ it will %e pro#i!e! %y the !istri%"tio #e !or i the +orm o+ package files a ! the rest will %e a#aila%le i so"rce co!e +orm that ca %e i stalle! ma "ally. We<ll talk a little a%o"t how to i stall so+tware %y compili g so"rce co!e i a later chapter.

'a/kage 2i es
The %asic " it o+ so+tware i a packagi g system is the package file. $ package +ile is a compresse! collectio o+ +iles that comprise the so+tware package. $ package may co sist o+ "mero"s programs a ! !ata +iles that s"pport the programs. 9 a!!itio to the +iles to %e i stalle!, the package +ile also i cl"!es meta!ata a%o"t the package, s"ch as a te3t !escriptio o+ the package a ! its co te ts. $!!itio ally, ma y packages co tai pre- a ! post-i stallatio scripts that per+orm co +ig"ratio tasks %e+ore a ! a+ter the package i stallatio . 6ackage +iles are create! %y a perso k ow as a package maintainer, o+te E%"t ot alwaysF a employee o+ the !istri%"tio #e !or. The package mai tai er gets the so+tware i so"rce co!e +orm +rom the upstream provider Ethe a"thor o+ the programF, compiles it, a ! creates the package meta!ata a ! a y ecessary i stallatio scripts. 7+te , the package mai tai er will apply mo!i+icatio s to the origi al so"rce co!e to impro#e the program<s i tegratio with the other parts o+ the *i "3 !istri%"tio .

!e5ositories
While some so+tware pro4ects choose to per+orm their ow packagi g a ! !istri%"tio , most packages to!ay are create! %y the !istri%"tio #e !ors a ! i tereste! thir! parties. 6ackages are ma!e a#aila%le to the "sers o+ a !istri%"tio i ce tral repositories that may co tai ma y tho"sa !s o+ packages, each specially %"ilt a ! mai tai e! +or the !istri%"tio . ./.

.2 8 6ackage >a ageme t $ !istri%"tio may mai tai se#eral !i++ere t repositories +or !i++ere t stages o+ the so+tware !e#elopme t li+e cycle. 0or e3ample, there will "s"ally %e a ?testi g@ repository that co tai s packages that ha#e 4"st %ee %"ilt a ! are i te !e! +or "se %y %ra#e so"ls who are looki g +or %"gs %e+ore they are release! +or ge eral !istri%"tio . $ !istri%"tio will o+te ha#e a ?!e#elopme t@ repository where work-i -progress packages !esti e! +or i cl"sio i the !istri%"tio <s e3t ma4or release are kept. $ !istri%"tio may also ha#e relate! thir!-party repositories. These are o+te ee!e! to s"pply so+tware that, +or legal reaso s s"ch as pate ts or ':> a ti-circ"m#e tio iss"es, ca ot %e i cl"!e! with the !istri%"tio . 6erhaps the %est k ow case is that o+ e crypte! 'R' s"pport, which is ot legal i the ) ite! States. The thir!-party repositories operate i co" tries where so+tware pate ts a ! a ti-circ"m#e tio laws !o ot apply. These repositories are "s"ally wholly i !epe !e t o+ the !istri%"tio they s"pport a ! to "se them, o e m"st k ow a%o"t them a ! ma "ally i cl"!e them i the co +ig"ratio +iles +or the package ma ageme t system.

)e5enden/ies
6rograms sel!om ?sta !alo eH@ rather they rely o the prese ce o+ other so+tware compo e ts to get their work !o e. Commo acti#ities, s"ch as i p"t-o"tp"t +or e3ample, are ha !le! %y ro"ti es share! %y ma y programs. These ro"ti es are store! i what are calle! shared libraries, which pro#i!e esse tial ser#ices to more tha o e program. 9+ a package re="ires a share! reso"rce s"ch as a share! li%rary, it is sai! to ha#e a dependency. >o!er package ma ageme t systems all pro#i!e some metho! o+ dependency resolution to e s"re that whe a package is i stalle!, all o+ its !epe !e cies are i stalle!, too.

$igh And Lo%B e;e 'a/kage Too s

6ackage ma ageme t systems "s"ally co sist o+ two types o+ tools, low-le#el tools which ha !le tasks s"ch as i stalli g a ! remo#i g package +iles, a ! high-le#el tools that per+orm meta!ata searchi g a ! !epe !e cy resol"tio . 9 this chapter, we will look at the tools s"pplie! with 'e%ia -style systems Es"ch as )%" t" a ! ma y othersF a ! those "se! %y rece t :e! Aat pro!"cts. While all :e! Aat-style !istri%"tio s rely o the same low-le#el program Er7mF, they "se !i++ere t high-le#el tools. 0or o"r !isc"ssio , we will co#er the high-le#el program 1um, "se! %y 0e!ora, :e! Aat E terprise *i "3, a ! Ce t7S. 7ther :e! Aat-style !istri%"tio s pro#i!e high-le#el tools with compara%le +eat"res. Table1.- ': 6ackaging System Tools Distri0utions 'e%ia -Style ./2 Lo4ELe%el Tools d7kg 3ighELe%el Tools a7t-get> a7titude

Aow $ 6ackage System Works 0e!ora, :e! Aat E terprise *i "3, Ce t7S r7m 1um

Common Pac(age >anagement Tas(s

There are ma y operatio s that ca %e per+orme! with the comma ! li e package ma ageme t tools. We will look at the most commo . Je aware that the low-le#el tools also s"pport creatio o+ package +iles, a acti#ity o"tsi!e the scope o+ this %ook. 9 the !isc"ssio %elow, the term ?package_name@ re+ers to the act"al ame o+ a package rather tha the term ?package_file,@ which is the ame o+ the +ile that co tai s the package.

2inding A 'a/kage In A !e5ository

)si g the high-le#el tools to search repository meta!ata, a package ca %e locate! %ase! o its ame or !escriptio . Table 1.-3: 6ackage Search %ommands St,le 'e%ia :e! Aat CommandCsD a7t-get u7date a7t-cache search search_string 1um search search_string

E3ample, To search a 1um repository +or the emacs te3t e!itor, this comma ! co"l! %e "se!,
1um search emacs

Insta ing A 'a/kage 2rom A !e5ository

Aigh-le#el tools permit a package to %e !ow loa!e! +rom a repository a ! i stalle! with +"ll !epe !e cy resol"tio . Table 1.-$: 6ackage *nstallation %ommands St,le 'e%ia CommandCsD a7t-get u7date

./(

.2 8 6ackage >a ageme t a7t-get install package_name :e! Aat 1um install package_name

E3ample, To i stall the emacs te3t e!itor +rom a a7t repository,

a7t-get u7dateT a7t-get install emacs

Insta ing A 'a/kage 2rom A 'a/kage 2i e

9+ a package +ile has %ee !ow loa!e! +rom a so"rce other tha a repository, it ca %e i stalle! !irectly Etho"gh witho"t !epe !e cy resol"tio F "si g a low-le#el tool. Table 1.-.: (ow-(evel 6ackage *nstallation %ommands St,le 'e%ia :e! Aat CommandCsD d7kg --install package_file r7m -i package_file

E3ample, 9+ the emacs-!!<#-)<fc)-i$/.<r7m package +ile ha! %ee !ow loa!e! +rom a o -repository site, it wo"l! %e i stalle! this way,
r7m -i emacs-!!<#-)<fc)-i$/.<r7m

1ote5 Si ce this tech i="e "ses the low-le#el r7m program to per+orm the i stallatio , o !epe !e cy resol"tio is per+orme!. 9+ r7m !isco#ers a missi g !epe !e cy, r7m will e3it with a error.

!emo;ing A 'a/kage
6ackages ca %e " i stalle! "si g either the high-le#el or low-tools. The high-le#el tools are show %elow.

./1

Commo 6ackage >a ageme t Tasks Table1.- 2: 6ackage 9emoval %ommands St,le 'e%ia :e! Aat CommandCsD a7t-get remove package_name 1um erase package_name

E3ample, To " i stall the emacs package +rom a 'e%ia -style system,
a7t-get remove emacs

U5dating 'a/kages 2rom A !e5ository

The most commo package ma ageme t task is keepi g the system "p-to-!ate with the latest packages. The high-le#el tools ca per+orm this #ital task i o e si gle step. Table 1.-3: 6ackage 0pdate %ommands St,le 'e%ia :e! Aat CommandCsD a7t-get u7dateT a7t-get u7grade 1um u7date

E3ample, To apply a y a#aila%le "p!ates to the i stalle! packages o a 'e%ia -style system,
a7t-get u7dateT a7t-get u7grade

U5grading A 'a/kage 2rom A 'a/kage 2i e

9+ a "p!ate! #ersio o+ a package has %ee !ow loa!e! +rom a o -repository so"rce, it ca %e i stalle!, replaci g the pre#io"s #ersio , Table 1.-4: (ow-(evel 6ackage 0pgrade %ommands St,le 'e%ia CommandCsD d7kg --install package_file

./2

.2 8 6ackage >a ageme t :e! Aat r7m -3 package_file

E3ample, )p!ati g a e3isti g i stallatio o+ emacs to the #ersio co tai e! i the package +ile emacs-!!<#-)<fc)-i$/.<r7m o a :e! Aat system,
r7m -3 emacs-!!<#-)<fc)-i$/.<r7m

1ote5 d7kg !oes ot ha#e a speci+ic optio +or "pgra!i g a package #ers"s i stalli g o e as r7m !oes.

Listing Insta ed 'a/kages

These comma !s ca %e "se! to !isplay a list o+ all the packages i stalle! o the system, Table 1.-5: 6ackage (isting %ommands St,le 'e%ia :e! Aat CommandCsD d7kg --list r7m -Ba

)etermining I= A 'a/kage Is Insta ed

These low-le#el tools ca %e "se! to !isplay whether a speci+ie! package is i stalle!, Table 1.-1=: 6ackage Status %ommands St,le 'e%ia :e! Aat CommandCsD d7kg --status package_name r7m -B package_name

E3ample, To !etermi e i+ the emacs package is i stalle! o a 'e%ia style system,

d7kg --status emacs

./K

Commo 6ackage >a ageme t Tasks

)is5 aying In=o About An Insta ed 'a/kage

9+ the ame o+ a i stalle! package is k ow , the +ollowi g comma !s ca %e "se! to !isplay a !escriptio o+ the package, Table 1.-11: 6ackage *nformation %ommands St,le 'e%ia :e! Aat CommandCsD a7t-cache sho9 package_name 1um info package_name

E3ample, To see a !escriptio o+ the emacs package o a 'e%ia -style system,

a7t-cache sho9 emacs

2inding Whi/h 'a/kage Insta ed A 2i e

To !etermi e what package is respo si%le +or the i stallatio o+ a partic"lar +ile, the +ollowi g comma !s ca %e "se!, Table 1.-1': 6ackage File *dentification %ommands St,le 'e%ia :e! Aat CommandCsD d7kg --search file_name r7m -Bf file_name

E3ample, To see what package i stalle! the 6usr6bin6vim +ile o a :e! Aat system,
r7m -Bf 6usr6bin6vim

Summing -p
9 the chapters that +ollow, we will e3plore ma y !i++ere t programs co#eri g a wi!e ra ge o+ applicatio areas. While most o+ these programs are commo ly i stalle! %y !e+a"lt, we may ee! to i stall a!!itio al packages i+ ecessary programs are ot alrea!y i stalle! o o"r system. With o"r ew+o" ! k owle!ge Ea ! appreciatio F o+ package .//

.2 8 6ackage >a ageme t ma ageme t, we sho"l! ha#e o pro%lem i stalli g a ! ma agi g the programs we ee!.

The Linux So"t4are Installation >,th

6eople migrati g +rom other plat+orms sometimes +all #ictim to the myth that so+tware is somehow !i++ic"lt to i stall " !er *i "3 a ! that the #ariety o+ packagi g schemes "se! %y !i++ere t !istri%"tio s is a hi !ra ce. Well, it is a hi !ra ce, %"t o ly to proprietary so+tware #e !ors who wish to !istri%"te %i aryo ly #ersio s o+ their secret so+tware. The *i "3 so+tware ecosystem is %ase! o the i!ea o+ ope so"rce co!e. 9+ a program !e#eloper releases so"rce co!e +or a pro!"ct, it is likely that a perso associate! with a !istri%"tio will package the pro!"ct a ! i cl"!e it i their repository. This metho! e s"res that the pro!"ct is well i tegrate! i to the !istri%"tio a ! the "ser is gi#e the co #e ie ce o+ ?o e-stop shoppi g@ +or so+tware, rather tha ha#i g to search +or each pro!"ct<s we% site. 'e#ice !ri#ers are are ha !le! i m"ch the same way, e3cept that i stea! o+ %ei g separate items i a !istri%"tio <s repository, they %ecome part o+ the *i "3 ker el itsel+. ;e erally speaki g, there is o s"ch thi g as a ?!ri#er !isk@ i *i "3. Either the ker el s"pports a !e#ice or it !oes <t, a ! the *i "3 ker el s"pports a lot o+ !e#ices. >a y more, i +act, tha Wi !ows !oes. 7+ co"rse, this is o+ o co solatio i+ the partic"lar !e#ice yo" ee! is ot s"pporte!. Whe that happe s, yo" ee! to look at the ca"se. $ lack o+ !ri#er s"pport is "s"ally ca"se! %y o e o+ three thi gs, 8. The device is too ne+. Si ce ma y har!ware #e !ors !o <t acti#ely s"pport *i "3 !e#elopme t, it +alls "po a mem%er o+ the *i "3 comm" ity to write the ker el !ri#er co!e. This takes time. 2. The device is too exotic. &ot all !istri%"tio s i cl"!e e#ery possi%le !e#ice !ri#er. Each !istri%"tio %"il!s their ow ker els, a ! si ce ker els are #ery co +ig"ra%le Ewhich is what makes it possi%le to r" *i "3 o e#erythi g +rom wristwatches to mai +ramesF they may ha#e o#erlooke! a partic"lar !e#ice. Jy locati g a ! !ow loa!i g the so"rce co!e +or the !ri#er, it is possi%le +or yo" Eyes, yo"F to compile a ! i stall the !ri#er yo"rsel+. This process is ot o#erly !i++ic"lt, %"t it is rather i #ol#e!. We<ll talk a%o"t compili g so+tware i a later chapter. . The hard+are vendor is hiding something. They ha#e either release! so"rce co!e +or a *i "3 !ri#er, or ha#e they release! the tech ical !oc"me tatio +or some%o!y to create o e +or them. This mea s that the har!ware #e !or is tryi g to keep the programmi g i ter+aces to the !e#ice a

./8

S"mmi g )p

secret. Si ce we !o <t wa t secret !e#ices i o"r comp"ters, 9 s"ggest that yo" remo#e the o++e !i g har!ware a ! pitch it i to the trash, with yo"r other "seless items.

;urther 1eading
Spe ! some time getti g to k ow the package ma ageme t system +or yo"r !istri%"tio . Each !istri%"tio pro#i!es !oc"me tatio +or its package ma ageme t tools. 9 a!!itio , here are some more ge eric so"rces,

The -ebian @>0;(inu+ F): chapter o package ma ageme t pro#i!es a o#er#iew o+ package ma ageme t o 'e%ia systems , http,--www.!e%ia .org-!oc-0$T-ch-pkgtools.e .html The home page +or the :6> pro4ect, http,--www.rpm.org The home page +or the 5)> pro4ect at '"ke ) i#ersity, http,--li "3.!"ke.e!"-pro4ects-y"m0or a little %ackgro" !, the Wikipe!ia has a article o meta!ata, http,--e .wikipe!ia.org-wiki->eta!ata

./9

.K 8 Storage >e!ia

1+ Storage 'edia
9 pre#io"s chapters weI#e looke! at ma ip"lati g !ata at the +ile le#el. 9 this chapter, we will co si!er !ata at the !e#ice le#el. *i "3 has amaDi g capa%ilities +or ha !li g storage !e#ices, whether physical storage, s"ch as har! !isks, or etwork storage, or #irt"al storage !e#ices like :$9' E:e!" !a t $rray o+ 9 !epe !e t 'isksF a ! *R> E*ogical Rol"me >a agerF. Aowe#er, si ce this is ot a %ook a%o"t system a!mi istratio , we will ot try to co#er this e tire topic i !epth. What we will try to !o is i tro!"ce some o+ the co cepts a ! key comma !s that are "se! to ma age storage !e#ices. To carry o"t the e3ercises i this chapter, we will "se a )SJ +lash !ri#e, a C'-:W !isk E+or systems e="ippe! with a C'-:7> %"r erF a ! a +loppy !isk Eagai , i+ the system is so e="ippe!.F We will look at the +ollowi g comma !s,

mount 8 >o" t a +ile system umount 8 ) mo" t a +ile system fsck 8 Check a ! repair a +ile system fdisk 8 6artitio ta%le ma ip"lator mkfs 8 Create a +ile system fdformat 8 0ormat a +loppy !isk dd 8 Write %lock orie te! !ata !irectly to a !e#ice genisoimage NmkisofsP 8 Create a 9S7 9KK0 image +ile 9odim NcdrecordP 8 Write !ata to optical storage me!ia md"sum 8 Calc"late a >'2 checks"m

>ounting #nd -nmounting Storage De%ices

:ece t a!#a ces i the *i "3 !esktop ha#e ma!e storage !e#ice ma ageme t e3tremely .80

>o" ti g $ ! ) mo" ti g Storage 'e#ices easy +or !esktop "sers. 0or the most part, we attach a !e#ice to o"r system a ! it ?4"st works.@ Jack i the ol! !ays Esay, 2001F, this st"++ ha! to %e !o e ma "ally. 7 o !esktop systems Ei.e., ser#ersF this is still a largely ma "al proce!"re si ce ser#ers o+te ha#e e3treme storage ee!s a ! comple3 co +ig"ratio re="ireme ts. The +irst step i ma agi g a storage !e#ice is attachi g the !e#ice to the +ile system tree. This process, calle! mounting, allows the !e#ice to participate with the operati g system. $s we recall +rom Chapter (, ) i3-like operati g systems, like *i "3, mai tai a si gle +ile system tree with !e#ices attache! at #ario"s poi ts. This co trasts with other operati g systems s"ch as >S-'7S a ! Wi !ows that mai tai separate trees +or each !e#ice E+or e3ample C:], ':], etc.F. There is a +ile ame! 6etc6fstab that lists the !e#ices Etypically har! !isk partitio sF that are to %e mo" te! at %oot time. Aere is a e3ample 6etc6fstab +ile +rom a 0e!ora / system,
=4H&=M6#! =4H&=M6home =4H&=M6boot tm7fs dev7ts s1sfs 7roc =4H&=M*,4:-sda$ 6 6home 6boot 6dev6shm 6dev67ts 6s1s 67roc s9a7 ext$ ext$ ext$ tm7fs dev7ts s1sfs 7roc s9a7 defaults defaults defaults defaults gidM">modeM.!( defaults defaults defaults # # # ( ( ( ( ( # ! ! ( ( ( ( (

>ost o+ the +ile systems liste! i this e3ample +ile are #irt"al a ! are ot applica%le to o"r !isc"ssio . 0or o"r p"rposes, the i teresti g o es are the +irst three,
=4H&=M6#! =4H&=M6home =4H&=M6boot 6 6home 6boot ext$ ext$ ext$ defaults defaults defaults # # # ! # !

These are the har! !isk partitio s. Each li e o+ the +ile co sists o+ si3 +iel!s, as +ollows, Table 12-1: ;etc;fstab Fields ;ield . Contents 'e#ice Description Tra!itio ally, this +iel! co tai s the act"al ame o+ a !e#ice +ile associate! with the physical !e#ice, s"ch as 6dev6hda# Ethe +irst partitio o+ the master !e#ice o the +irst 9'E cha elF. J"t with to!ay<s comp"ters, which ha#e ma y !e#ices that are hot pl"gga%le Elike .8.

.K 8 Storage >e!ia )SJ !ri#esF, ma y mo!er *i "3 !istri%"tio s associate a !e#ice with a te3t la%el i stea!. This la%el Ewhich is a!!e! to the storage me!ia whe it is +ormatte!F is rea! %y the operati g system whe the !e#ice is attache! to the system. That way, o matter which !e#ice +ile is assig e! to the act"al physical !e#ice, it ca still %e correctly i!e ti+ie!. 2 ( >o" t 6oi t 0ile System Type The !irectory where the !e#ice is attache! to the +ile system tree. *i "3 allows ma y +ile system types to %e mo" te!. >ost ati#e *i "3 +ile systems are e3t(, %"t ma y others are s"pporte!, s"ch as 0$T.K EmsdosF, 0$T(2 EvfatF, &T0S EntfsF, C'-:7> Eiso0..(F, etc. 0ile systems ca %e mo" te! with #ario"s optio s. 9t is possi%le, +or e3ample, to mo" t +ile systems as rea!-o ly, or pre#e t a y programs +rom %ei g e3ec"te! +rom them Ea "se+"l sec"rity +eat"re +or remo#a%le me!ia.F $ si gle "m%er that speci+ies i+ a ! whe a +ile system is to %e %acke! "p with the dum7 comma !. $ si gle "m%er that speci+ies i what or!er +ile systems sho"l! %e checke! with the fsck comma !.

7ptio s

2 K

0re="e cy 7r!er

?ie%ing A List <= :ounted 2i e Systems

The mount comma ! is "se! to mo" t +ile systems. E teri g the comma ! witho"t arg"me ts will !isplay a list o+ the +ile systems c"rre tly mo" te!,
[me@linuxbox ~]$ mount 6dev6sda! on 6 t17e ext$ Nr9P 7roc on 67roc t17e 7roc Nr9P s1sfs on 6s1s t17e s1sfs Nr9P dev7ts on 6dev67ts t17e dev7ts Nr9>gidM">modeM.!(P 6dev6sda" on 6home t17e ext$ Nr9P 6dev6sda# on 6boot t17e ext$ Nr9P tm7fs on 6dev6shm t17e tm7fs Nr9P none on 67roc6s1s6fs6binfmtOmisc t17e binfmtOmisc Nr9P sunr7c on 6var6lib6nfs6r7cO7i7efs t17e r7cO7i7efs Nr9P

.82

>o" ti g $ ! ) mo" ti g Storage 'e#ices

fusectl on 6s1s6fs6fuse6connections t17e fusectl Nr9P 6dev6sdd# on 6media6disk t17e vfat Nr9>nosuid>nodev>noatime> uhel7erMhal>uidM"((>utf/>shortnameMlo9erP t9in%:6musicbox on 6misc6musicbox t17e nfs% Nr9>addrM#0!<#./<#<%P

The +ormat o+ the listi g is, device o mount point type file system type EoptionsF. 0or e3ample, the +irst li e shows that !e#ice 6dev6sda! is mo" te! as the root +ile system a ! it is o+ type e3t( a ! is %oth rea!a%le a ! writa%le Ethe optio ?rw@F. This listi g also has two i teresti g e tries at the %ottom o+ the list. The e3t to last e try shows a 2 giga%yte S' memory car! i a car! rea!er mo" te! at 6media6disk, a ! the last e try is a etwork !ri#e mo" te! at 6misc6musicbox. 0or o"r +irst e3perime t, we will work with a C'-:7>. 0irst, let<s look at a system %e+ore a C'-:7> is i serte!,
[me@linuxbox ~]$ mount 6dev6ma77er6;ol@rou7((-=og;ol(( on 6 t17e ext$ Nr9P 7roc on 67roc t17e 7roc Nr9P s1sfs on 6s1s t17e s1sfs Nr9P dev7ts on 6dev67ts t17e dev7ts Nr9>gidM">modeM.!(P 6dev6hda# on 6boot t17e ext$ Nr9P tm7fs on 6dev6shm t17e tm7fs Nr9P none on 67roc6s1s6fs6binfmtOmisc t17e binfmtOmisc Nr9P sunr7c on 6var6lib6nfs6r7cO7i7efs t17e r7cO7i7efs Nr9P

This listi g is +rom a Ce t7S 2 system, which is "si g *R> E*ogical Rol"me >a agerF to create its root +ile system. *ike ma y mo!er *i "3 !istri%"tio s, this system will attempt to a"tomatically mo" t the C'-:7> a+ter i sertio . $+ter we i sert the !isk, we see the +ollowi g,
[me@linuxbox ~]$ mount 6dev6ma77er6;ol@rou7((-=og;ol(( on 6 t17e ext$ Nr9P 7roc on 67roc t17e 7roc Nr9P s1sfs on 6s1s t17e s1sfs Nr9P dev7ts on 6dev67ts t17e dev7ts Nr9>gidM">modeM.!(P 6dev6hda# on 6boot t17e ext$ Nr9P tm7fs on 6dev6shm t17e tm7fs Nr9P none on 67roc6s1s6fs6binfmtOmisc t17e binfmtOmisc Nr9P sunr7c on 6var6lib6nfs6r7cO7i7efs t17e r7cO7i7efs Nr9P 6dev6hdc on 6media6live-#<(<#(-/ t17e iso0..( Nro>noexec>nosuid> nodev>uidM"((P

$+ter we i sert the !isk, we see the same listi g as %e+ore with o e a!!itio al e try. $t .8(

.K 8 Storage >e!ia the e ! o+ the listi g we see that the C'-:7> Ewhich is !e#ice 6dev6hdc o this systemF has %ee mo" te! o 6media6live-#<(<#(-/, a ! is type iso9KK0 Ea C':7>F. 0or p"rposes o+ o"r e3perime t, we<re i tereste! i the ame o+ the !e#ice. Whe yo" co !"ct this e3perime t yo"rsel+, the !e#ice ame will most likely %e !i++ere t. 'arning5 9 the e3amples that +ollow, it is #itally importa t that yo" pay close atte tio to the act"al !e#ice ames i "se o yo"r system a ! do not use the names used in this text2 $lso ote that a"!io C's are ot the same as C'-:7>s. $"!io C's !o ot co tai +ile systems a ! th"s ca ot %e mo" te! i the "s"al se se. &ow that we ha#e the !e#ice ame o+ the C'-:7> !ri#e, let<s " mo" t the !isk a ! remo" t it a other locatio i the +ile system tree. To !o this, we %ecome the s"per"ser E"si g the comma ! appropriate +or o"r systemF a ! " mo" t the !isk with the umount E otice the spelli gF comma !,
[me@linuxbox ~]$ su :ass9ord: [root@linuxbox ~]^ umount /dev/hdc

The e3t step is to create a ew mount point +or the !isk. $ mo" t poi t is simply a !irectory somewhere o the +ile system tree. &othi g special a%o"t it. 9t !oes <t e#e ha#e to %e a empty !irectory, tho"gh i+ yo" mo" t a !e#ice o a o -empty !irectory, yo" will ot %e a%le to see the !irectory<s pre#io"s co te ts " til yo" " mo" t the !e#ice. 0or o"r p"rposes, we will create a ew !irectory,
[root@linuxbox ~]^ mkdir /mnt/cdrom

0i ally, we mo" t the C'-:7> at the ew mo" t poi t. The -t optio is "se! to speci+y the +ile system type,
[root@linuxbox ~]^ mount -t isoDNN* /dev/hdc /mnt/cdrom

$+terwar!, we ca e3ami e the co te ts o+ the C'-:7> #ia the ew mo" t poi t,

.81

>o" ti g $ ! ) mo" ti g Storage 'e#ices

[root@linuxbox ~]^ cd /mnt/cdrom [root@linuxbox cdrom]^ ls

&otice what happe s whe we try to " mo" t the C'-:7>,

[root@linuxbox cdrom]^ umount /dev/hdc umount: 6mnt6cdrom: device is bus1

Why is thisC The reaso is that we ca ot " mo" t a !e#ice i+ the !e#ice is %ei g "se! %y someo e or some process. 9 this case, we cha ge! o"r worki g !irectory to the mo" t poi t +or the C'-:7>, which ca"ses the !e#ice to %e %"sy. We ca easily reme!y the iss"e %y cha gi g the worki g !irectory to somethi g other tha the mo" t poi t,
[root@linuxbox cdrom]^ cd [root@linuxbox ~]^ umount /dev/hdc

&ow the !e#ice " mo" ts s"ccess+"lly.

+h, -nmounting Is Important

9+ yo" look at the o"tp"t o+ the free comma !, which !isplays statistics a%o"t memory "sage, yo" will see a statistic calle! ?%"++ers.@ Comp"ter systems are !esig e! to go as +ast as possi%le. 7 e o+ the impe!ime ts to system spee! is slow !e#ices. 6ri ters are a goo! e3ample. E#e the +astest pri ter is e3tremely slow %y comp"ter sta !ar!s. $ comp"ter wo"l! %e #ery slow i !ee! i+ it ha! to stop a ! wait +or a pri ter to +i ish pri ti g a page. 9 the early !ays o+ 6Cs E%e+ore m"lti-taski gF, this was a real pro%lem. 9+ yo" were worki g o a sprea!sheet or te3t !oc"me t, the comp"ter wo"l! stop a ! %ecome " a#aila%le e#ery time yo" pri te!. The comp"ter wo"l! se ! the !ata to the pri ter as +ast as the pri ter co"l! accept it, %"t it was #ery slow si ce pri ters !o <t pri t #ery +ast. This pro%lem was sol#e! %y the a!#e t o+ the printer buffer, a !e#ice co tai i g some :$> memory that wo"l! sit %etwee the comp"ter a ! the pri ter. With the pri ter %"++er i place, the comp"ter wo"l! se ! the pri ter o"tp"t to the %"++er a ! it wo"l! ="ickly %e store! i the +ast :$> so the comp"ter co"l! go %ack to work witho"t waiti g. >ea while, the pri ter %"++er wo"l! slowly spool the !ata to the pri ter +rom the %"++er<s memory at the spee! at which the pri ter co"l! accept it.

.82

.K 8 Storage >e!ia

This i!ea o+ %"++eri g is "se! e3te si#ely i comp"ters to make them +aster. 'o <t let the ee! to occasio ally rea! or write !ata to-+rom slow !e#ices impe!e the spee! o+ the system. 7perati g systems store !ata rea! +rom, a ! to %e writte to storage !e#ices i memory +or as lo g as possi%le %e+ore act"ally ha#i g to i teract with the slower !e#ice. 7 a *i "3 system +or e3ample, yo" will otice that the system seems to +ill "p memory the lo ger it is "se!. This !oes ot mea *i "3 is ?"si g? all the memory, it mea s that *i "3 is taki g a!#a tage o+ all the a#aila%le memory to !o as m"ch %"++eri g as it ca . This %"++eri g allows writi g to storage !e#ices to %e !o e #ery ="ickly, %eca"se the writi g to the physical !e#ice is %ei g !e+erre! to a +"t"re time. 9 the mea time, the !ata !esti e! +or the !e#ice is pili g "p i memory. 0rom time to time, the operati g system will write this !ata to the physical !e#ice. ) mo" ti g a !e#ice e tails writi g all the remai i g !ata to the !e#ice so that it ca %e sa+ely remo#e!. 9+ the !e#ice is remo#e! witho"t " mo" ti g it +irst, the possi%ility e3ists that ot all the !ata !esti e! +or the !e#ice has %ee tra s+erre!. 9 some cases, this !ata may i cl"!e #ital !irectory "p!ates, which will lea! to file system corruption, o e o+ the worst thi gs that ca happe o a comp"ter.

)etermining )e;i/e ,ames

9t<s sometimes !i++ic"lt to !etermi e the ame o+ a !e#ice. Jack i the ol! !ays, it was <t #ery har!. $ !e#ice was always i the same place a ! it !i! <t cha ge. ) i3-like systems like it that way. Jack whe ) i3 was !e#elope!, ?cha gi g a !isk !ri#e@ i #ol#e! "si g a +orkli+t to remo#e a washi g machi e-siDe! !e#ice +rom the comp"ter room. 9 rece t years, the typical !esktop har!ware co +ig"ratio has %ecome ="ite !y amic a ! *i "3 has e#ol#e! to %ecome more +le3i%le tha its a cestors. 9 the e3amples a%o#e we took a!#a tage o+ the mo!er *i "3 !esktop<s a%ility to ?a"tomagically@ mo" t the !e#ice a ! the !etermi e the ame a+ter the +act. J"t what i+ we are ma agi g a ser#er or some other e #iro me t where this !oes ot occ"rC Aow ca we +ig"re it o"tC 0irst, let<s look at how the system ames !e#ices. 9+ we list the co te ts o+ the 6dev !irectory Ewhere all !e#ices li#eF, we ca see that there are lots a ! lots o+ !e#ices,
[me@linuxbox ~]$ ls /dev

The co te ts o+ this listi g re#eal some patter s o+ !e#ice ami g. Aere are a +ew, .8K

>o" ti g $ ! ) mo" ti g Storage 'e#ices Table 12-': (inu+ Storage -evice >ames Pattern 6dev6fdE 6dev6hdE De%ice 0loppy !isk !ri#es. 9'E E6$T$F !isks o ol!er systems. Typical mother%oar!s co tai two 9'E co ectors or channels, each with a ca%le with two attachme t poi ts +or !ri#es. The +irst !ri#e o the ca%le is calle! the master !e#ice a ! the seco ! is calle! the slave !e#ice. The !e#ice ames are or!ere! s"ch that 6dev6hda re+ers to the master !e#ice o the +irst cha el, 6dev6hdb is the sla#e !e#ice o the +irst cha elH 6dev6hdc, the master !e#ice o the seco ! cha el, a ! so o . $ traili g !igit i !icates the partitio "m%er o the !e#ice. 0or e3ample, -dev6hda# re+ers to the +irst partitio o the +irst har! !ri#e o the system while 6 dev6hda re+ers to the e tire !ri#e. 6ri ters. SCS9 !isks. 7 rece t *i "3 systems, the ker el treats all !isklike !e#ices Ei cl"!i g 6$T$-S$T$ har! !isks, +lash !ri#es, a ! )SJ mass storage !e#ices, s"ch as porta%le m"sic players a ! !igital camerasF as SCS9 !isks. The rest o+ the ami g system is similar to the ol!er 6dev6hdE ami g scheme !escri%e! a%o#e. 7ptical !ri#es EC'-'R' rea!ers a ! %"r ersF

6dev6l7E 6dev6sdE

6dev6srE

9 a!!itio , we o+te see sym%olic li ks s"ch as 6dev6cdrom, 6dev6dvd a ! 6dev6 flo771, which poi t to the act"al !e#ice +iles, pro#i!e! as a co #e ie ce. 9+ yo" are worki g o a system that !oes ot a"tomatically mo" t remo#a%le !e#ices, yo" ca "se the +ollowi g tech i="e to !etermi e how the remo#a%le !e#ice is ame! whe it is attache!. 0irst, start a real-time #iew o+ the 6var6log6messages +ile Eyo" may re="ire s"per"ser pri#ileges +or thisF,
[me@linuxbox ~]$ sudo tail -f /var/log/messages

The last +ew li es o+ the +ile will %e !isplaye! a ! the pa"se. &e3t, pl"g i the remo#a%le !e#ice. 9 this e3ample, we will "se a .K >J +lash !ri#e. $lmost imme!iately, the ker el will otice the !e#ice a ! pro%e it,

.8/

.K 8 Storage >e!ia
?ul !$ #(:():"$ linuxbox kernel: usb $-!: ne9 full s7eed 3*H device using uhciOhcd and address ! ?ul !$ #(:():"$ linuxbox kernel: usb $-!: configuration ^# chosen from # choice ?ul !$ #(:():"$ linuxbox kernel: scsi$ : *C*A emulation for 3*H +ass *torage devices ?ul !$ #(:():"/ linuxbox kernel: scsi scan: ALZ3ARU result too short N"P> using $. ?ul !$ #(:():"/ linuxbox kernel: scsi $:(:(:(: 'irect-4ccess &as1 'isk #<(( :Z: ( 4L*A: ! ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] $#!.$ "#!-b1te hard9are sectors N#. +HP ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] ,rite :rotect is off ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] 4ssuming drive cache: 9rite through ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] $#!.$ "#!-b1te hard9are sectors N#. +HP ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] ,rite :rotect is off ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] 4ssuming drive cache: 9rite through ?ul !$ #(:():"0 linuxbox kernel: sdb: sdb# ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] 4ttached *C*A removable disk ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: 4ttached scsi generic sg$ t17e (

$+ter the !isplay pa"ses agai , type Ctrl-c to get the prompt %ack. The i teresti g parts o+ the o"tp"t are the repeate! re+ere ces to ?as!%b@ which matches o"r e3pectatio o+ a SCS9 !isk !e#ice ame. M owi g this, two li es %ecome partic"larly ill"mi ati g,
?ul !$ #(:():"0 linuxbox kernel: sdb: sdb# ?ul !$ #(:():"0 linuxbox kernel: sd $:(:(:(: [sdb] 4ttached *C*A removable disk

This tells "s the !e#ice ame is 6dev6sdb +or the e tire !e#ice a ! 6dev6sdb# +or the +irst partitio o the !e#ice. $s we ha#e see , worki g with *i "3 is +"ll o+ i teresti g !etecti#e workL Ti(5 )si g the tail -f 6var6log6messages tech i="e is a great way to watch what the system is !oi g i ear real-time. With o"r !e#ice ame i ha !, we ca .88 ow mo" t the +lash !ri#e,

>o" ti g $ ! ) mo" ti g Storage 'e#ices

[me@linuxbox ~]$ sudo mkdir /mnt/flash [me@linuxbox ~]$ sudo mount /dev/sdb1 /mnt/flash [me@linuxbox ~]$ df -iles1stem #2-blocks 3sed 4vailable 3se5 +ounted on 6dev6sda! #"##"%"! "#/.0%% 0))"#.% $"5 6 6dev6sda" "0.$#0(/ $#)))$). !%)).%/( ")5 6home 6dev6sda# #%)).% #)!)) #!!/"/ #$5 6boot tm7fs ))./(/ ( ))./(/ (5 6dev6shm 6dev6sdb# #"".( ( #"".( (5 6mnt6flash

The !e#ice ame will remai the same as lo g as it remai s physically attache! to the comp"ter a ! the comp"ter is ot re%oote!.

Creating 7e4 ;ile S,stems

*et<s say that we wa t to re+ormat the +lash !ri#e with a *i "3 ati#e +ile system, rather tha the 0$T(2 system it has ow. This i #ol#es two steps, .. Eoptio alF create a ew partitio layo"t i+ the e3isti g o e is ot to o"r liki g, a ! 2. create a ew, empty +ile system o the !ri#e. 'arning2 9 the +ollowi g e3ercise, we are goi g to +ormat a +lash !ri#e. )se a !ri#e that co tai s othi g yo" care a%o"t %eca"se it will %e erase!L $gai , make absolutely sure you are s(eci"ying the correct device name "or your system6 not the one sho+n in the text. 9ailure to heed this +arning could result in you "ormatting *i.e.6 erasing, the +rong drive2

:ani5u ating 'artitions With fdisk

The fdisk program allows "s to i teract !irectly with !isk-like !e#ices Es"ch as har! !isk !ri#es a ! +lash !ri#esF at a #ery low le#el. With this tool we ca e!it, !elete, a ! create partitio s o the !e#ice. To work with o"r +lash !ri#e, we m"st +irst " mo" t it Ei+ ee!e!F a ! the i #oke the fdisk program as +ollows,
[me@linuxbox ~]$ sudo umount /dev/sdb1 [me@linuxbox ~]$ sudo fdisk /dev/sdb

&otice that we m"st speci+y the !e#ice i terms o+ the e tire !e#ice, ot %y partitio "m%er. $+ter the program starts "p, we will see the +ollowi g prompt,

.89

.K 8 Storage >e!ia
Command Nm for hel7P:

E teri g a ?m@ will !isplay the program me ",

Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos com7atibilit1 flag d delete a 7artition l list kno9n 7artition t17es m 7rint this menu n add a ne9 7artition o create a ne9 em7t1 ' * 7artition table 7 7rint the 7artition table B Buit 9ithout saving changes s create a ne9 em7t1 *un disklabel t change a 7artitionKs s1stem id u change dis7la16entr1 units v verif1 the 7artition table 9 9rite table to disk and exit x extra functionalit1 Nex7erts onl1P

Command Nm for hel7P:

The +irst thi g we wa t to !o is e3ami e the e3isti g partitio layo"t. We !o this %y e teri g ?p@ to pri t the partitio ta%le +or the !e#ice,
Command Nm for hel7P: p 'isk 6dev6sdb: #. +H> #.((..". b1tes # heads> $# sectors6track> #((/ c1linders 3nits M c1linders of $# E "#! M #"/)! b1tes 'evice Hoot 6dev6sdb# *tart ! &nd #((/ Hlocks #".(/8 Ad b *1stem ,0" -4T$!

9 this e3ample, we see a .K >J !e#ice with a si gle partitio E.F that "ses .00K o+ the a#aila%le .008 cyli !ers o the !e#ice. The partitio is i!e ti+ie! as Wi !ows 92 0$T(2 partitio . Some programs will "se this i!e ti+ier to limit the ki !s o+ operatio that ca %e !o e to the !isk, %"t most o+ the time it is ot critical to cha ge it. Aowe#er, i the .90

Creati g &ew 0ile Systems i terest o+ !emo stratio , we will cha ge it to i !icate a *i "3 partitio . To !o this, we m"st +irst +i ! o"t what 9' is "se! to i!e ti+y a *i "3 partitio . 9 the listi g a%o#e, we see that the 9' ?%@ is "se! to speci+y the e3iti g partitio . To see a list o+ the a#aila%le partitio types, we re+er %ack to the program me ". There we ca see the +ollowi g choice,
l list kno9n 7artition t17es

9+ we e ter ?l@ at the prompt, a large list o+ possi%le types is !isplaye!. $mo g them we see ?%@ +or o"r e3isti g partitio type a ! ?8(@ +or *i "3. ;oi g %ack to the me ", we see this choice to cha ge a partitio 9',
t change a 7artitionKs s1stem id

We e ter ?t@ at the prompt e ter the ew 9',

Command Nm for hel7P: t *elected 7artition # Sex code Nt17e = to list codesP: M Changed s1stem t17e of 7artition # to /$ N=inuxP

This completes all the cha ges that we ee! to make. )p to this poi t, the !e#ice has %ee " to"che! Eall the cha ges ha#e %ee store! i memory, ot o the physical !e#iceF, so we will write the mo!i+ie! partitio ta%le to the !e#ice a ! e3it. To !o this, we e ter ?w@ at the prompt,
Command Nm for hel7P: w The 7artition table has been alteredG Calling ioctlNP to re-read 7artition table< ,4RLAL@: Af 1ou have created or modified an1 ' * .<x 7artitions> 7lease see the fdisk manual 7age for additional information< *1ncing disks< [me@linuxbox ~]$

9+ we ha! !eci!e! to lea#e the !e#ice " altere!, we co"l! ha#e e tere! ?=@ at the prompt, .9.

.K 8 Storage >e!ia which wo"l! ha#e e3ite! the program witho"t writi g the cha ges. We ca sa+ely ig ore the omi o"s so" !i g war i g message.

Creating A ,e% 2i e System With mkfs

With o"r partitio e!iti g !o e Elightweight tho"gh it might ha#e %ee F itIs time to create a ew +ile system o o"r +lash !ri#e. To !o this, we will "se mkfs Eshort +or ?make +ile system@F, which ca create +ile systems i a #ariety o+ +ormats. To create a e3t( +ile system o the !e#ice, we "se the ?-t@ optio to speci+y the ?e3t(@ system type, +ollowe! %y the ame o+ !e#ice co tai i g the partitio we wish to +ormat,
[me@linuxbox ~]$ sudo mkfs -t ext /dev/sdb1 mke!fs #<%(<! N#!-?ul-!(()P -iles1stem labelM * t17e: =inux Hlock siCeM#(!% NlogM(P -ragment siCeM#(!% NlogM(P $0(% inodes> #".(/ blocks )/( blocks N"<((5P reserved for the su7er user -irst data blockM# +aximum files1stem blocksM#"00()/% ! block grou7s /#0! blocks 7er grou7> /#0! fragments 7er grou7 #0"! inodes 7er grou7 *u7erblock backu7s stored on blocks: /#0$ ,riting inode tables: done Creating journal N#(!% blocksP: done ,riting su7erblocks and files1stem accounting information: done This files1stem 9ill be automaticall1 checked ever1 $% mounts or #/( da1s> 9hichever comes first< 3se tune!fs -c or -i to override< [me@linuxbox ~]$

The program will !isplay a lot o+ i +ormatio whe e3t( is the chose +ile system type. To re-+ormat the !e#ice to its origi al 0$T(2 +ile system, speci+y ?#+at@ as the +ile system type,
[me@linuxbox ~]$ sudo mkfs -t vfat /dev/sdb1

This process o+ partitio i g a ! +ormatti g ca %e "se! a ytime a!!itio al storage !e#ices are a!!e! to the system. While we worke! with a ti y +lash !ri#e, the same .92

Creati g &ew 0ile Systems process ca %e applie! to i ter al har! !isks a ! other remo#a%le storage !e#ices like )SJ har! !ri#es.

Testing #nd 1epairing ;ile S,stems

9 o"r earlier !isc"ssio o+ the 6etc6fstab +ile, we saw some mysterio"s !igits at the e ! o+ each li e. Each time the system %oots, it ro"ti ely checks the i tegrity o+ the +ile systems %e+ore mo" ti g them. This is !o e %y the fsck program Eshort +or ?+ile system check@F. The last "m%er i each fstab e try speci+ies the or!er the !e#ices are to %e checke!. 9 o"r e3ample a%o#e, we see that the root +ile system is checke! +irst, +ollowe! %y the home a ! boot +ile systems. 'e#ices with a Dero as the last !igit are ot ro"ti ely checke!. 9 a!!itio to checki g the i tegrity o+ +ile systems, fsck ca also repair corr"pt +ile systems with #aryi g !egrees o+ s"ccess, !epe !i g o the amo" t o+ !amage. 7 ) i3like +ile systems, reco#ere! portio s o+ +iles are place! i the lost8found !irectory, locate! i the root o+ each +ile system. To check o"r +lash !ri#e Ewhich sho"l! %e " mo" te! +irstF, we co"l! !o the +ollowi g,
[me@linuxbox ~]$ sudo fsck /dev/sdb1 fsck #<%(</ N#$-+ar-!((/P e!fsck #<%(</ N#$-+ar-!((/P 6dev6sdb#: clean> ##6$0(% files> #..#6#".(/ blocks

9 my e3perie ce, +ile system corr"ptio is ="ite rare " less there is a har!ware pro%lem, s"ch as a +aili g !isk !ri#e. 7 most systems, +ile system corr"ptio !etecte! at %oot time will ca"se the system to stop a ! !irect yo" to r" fsck %e+ore co ti "i g.

+hat The fsck.

9 ) i3 c"lt"re, the wor! ?+sck@ is o+te "se! i place o+ a pop"lar wor! with which it shares three letters. This is especially appropriate, gi#e that yo" will pro%a%ly %e "tteri g the a+oreme tio e! wor! i+ yo" +i ! yo"rsel+ i a sit"atio where yo" are +orce! to r" fsck.

;ormatting ;lopp, Dis(s

0or those o+ "s still "si g comp"ters ol! e o"gh to %e e="ippe! with +loppy !iskette .9(

.K 8 Storage >e!ia !ri#es, we ca ma age those !e#ices, too. 6repari g a %la k +loppy +or "se is a two step process. 0irst, we per+orm a low-+ormat o the !iskette, the create a +ile system. To accomplish the +ormatti g, we "se the fdformat program speci+yi g the ame o+ the +loppy !e#ice E"s"ally 6dev6fd(F,
[me@linuxbox ~]$ sudo fdformat /dev/fd* 'ouble-sided> /( tracks> #/ sec6track< Total ca7acit1 #%%( kH< -ormatting <<< done ;erif1ing <<< done

&e3t, we apply a 0$T +ile system to the !iskette with mkfs,

[me@linuxbox ~]$ sudo mkfs -t msdos /dev/fd*

&otice that we "se the ?ms!os@ +ile system type to get the ol!er Ea ! smallerF style +ile allocatio ta%les. $+ter a !iskette is prepare!, it may %e mo" te! like other !e#ices.

>o%ing Data Directl, To8;rom De%ices

While we "s"ally thi k o+ !ata o o"r comp"ters as %ei g orga iDe! i to +iles, it is also possi%le to thi k o+ the !ata i ?raw@ +orm. 9+ we look at a !isk !ri#e, +or e3ample, we see that it co sists o+ a large "m%er o+ ?%locks@ o+ !ata that the operati g system sees as !irectories a ! +iles. Aowe#er, i+ we co"l! treat a !isk !ri#e as simply a large collectio o+ !ata %locks, we co"l! per+orm "se+"l tasks, s"ch as clo i g !e#ices. The dd program per+orms this task. 9t copies %locks o+ !ata +rom o e place to a other. 9t "ses a " i="e sy ta3 E+or historical reaso sF a ! is "s"ally "se! this way,
dd if$input_file of$output_file (bs$block_size (count$blocks))

*etIs say we ha! two )SJ +lash !ri#es o+ the same siDe a ! we wa te! to e3actly copy the +irst !ri#e to the seco !. 9+ we attache! %oth !ri#es to the comp"ter a ! they are assig e! to !e#ices 6dev6sdb a ! 6dev6sdc respecti#ely, we co"l! copy e#erythi g o the +irst !ri#e to the seco ! !ri#e with the +ollowi g,
dd if$/dev/sdb of$/dev/sdc

.91

>o#i g 'ata 'irectly To-0rom 'e#ices $lter ately, i+ o ly the +irst !e#ice were attache! to the comp"ter, we co"l! copy its co te ts to a or!i ary +ile +or later restoratio or copyi g,
dd if$/dev/sdb of$flash.drive.img

'arning2 The dd comma ! is #ery power+"l. Tho"gh its ame !eri#es +rom ?!ata !e+i itio ,@ it is sometimes calle! ?!estroy !isk@ %eca"se "sers o+te mistype either the if or of speci+icatio s. #l+ays double check your in(ut and out(ut s(eci"ications be"ore (ressing enter2

Creating CDE1?> Images

Writi g a recor!a%le C'-:7> Eeither a C'-: or C'-:WF co sists o+ two stepsH +irst, co str"cti g a iso image file that is the e3act +ile system image o+ the C'-:7> a ! seco !, writi g the image +ile o to the C'-:7> me!ia.

Creating An Image Co5y <= A C)B!<:

9+ we wa t to make a iso image o+ a e3isti g C'-:7>, we ca "se !! to rea! all the !ata %locks o++ the C'-:7> a ! copy them to a local +ile. Say we ha! a )%" t" C' a ! we wa te! to make a iso +ile that we co"l! later "se to make more copies. $+ter i serti g the C' a ! !etermi i g its !e#ice ame EweIll ass"me 6dev6cdromF, we ca make the iso +ile like so,
dd if$/dev/cdrom of$ubuntu.iso

This tech i="e works +or !ata 'R's as well, %"t will ot work +or a"!io C's, as they !o ot "se a +ile system +or storage. 0or a"!io C's, look at the cdrdao comma !.

Creating An Image 2rom A Co e/tion <= 2i es

To create a iso image +ile co tai i g the co te ts o+ a !irectory, we "se the genisoimage program. To !o this, we +irst create a !irectory co tai i g all the +iles we wish to i cl"!e i the image a ! the e3ec"te the genisoimage comma ! to create the image +ile. 0or e3ample, i+ we ha! create! a !irectory calle! ~6cd-rom-files a ! +ille! it with +iles +or o"r C'-:7>, we co"l! create a image +ile ame! cdrom<iso with the +ollowi g comma !, .92

.K 8 Storage >e!ia
genisoimage -o cd-rom.iso -H -Y ~/cd-rom-files

The ?-:@ optio a!!s meta!ata +or the 9ock 9idge e+tensions, which allows the "se o+ lo g +ile ames a ! 67S9G style +ile permissio s. *ikewise, the ?-J@ optio e a%les the Doliet e+tensions, which permit lo g +ile ames +or Wi !ows.

# Program /, #n, ?ther 7ame...

9+ yo" look at o -li e t"torials +or creati g a ! %"r i g optical me!ia like C':7>s a ! 'R's, yo" will +re="e tly e co" ter two programs calle! mkisofs a ! cdrecord. These programs were part o+ a pop"lar package calle! ?c!rtools@ a"thore! %y Jorg Schilli g. 9 the s"mmer o+ 200K, >r. Schilli g ma!e a lice se cha ge to a portio o+ the c!rtools package which, i the opi io o+ ma y i the *i "3 comm" ity, create! a lice se i compati%ility with the ;&) ;6*. $s a res"lt, a fork o+ the c!rtools pro4ect was starte! that ow i cl"!es replaceme t programs +or cdrecord a ! mkisofs ame! 9odim a ! genisoimage, respecti#ely.

+riting CDE1?> Images

$+ter we ha#e a image +ile, we ca %"r it o to o"r optical me!ia. >ost o+ the comma !s we will !isc"ss %elow ca %e applie! to %oth recor!a%le C'-:7> a ! 'R' me!ia.

:ounting An IS< Image )ire/t y

There is a trick that we ca "se to mo" t a iso image while it is still o o"r har! !isk a ! treat it as tho"gh it was alrea!y o optical me!ia. Jy a!!i g the ?-o loop@ optio to mount Ealo g with the re="ire! ?-t iso9KK0@ +ile system typeF, we ca mo" t the image +ile as tho"gh it were a !e#ice a ! attach it to the +ile system tree,
mkdir /mnt/iso.image mount -t isoDNN* -o loop image.iso /mnt/iso.image

9 the e3ample a%o#e, we create! a mo" t poi t ame! 6mnt6isoOimage a ! the mo" te! the image +ile image<iso at that mo" t poi t. $+ter the image is mo" te!, it ca %e treate! 4"st as tho"gh it were a real C'-:7> or 'R'. 9emember to unmount the .9K

Writi g C'-:7> 9mages image when it is no longer needed#

B anking A !eBWritab e C)B!<:

:ewrita%le C'-:W me!ia ee!s to %e erase! or blanked %e+ore it ca %e re"se!. To !o this, we ca "se 9odim, speci+yi g the !e#ice ame +or the C' writer a ! the type o+ %la ki g to %e per+orme!. The 9odim program o++ers se#eral types. The most mi imal Ea ! +astestF is the ?+ast@ type,
wodim dev$/dev/cdrw blank$fast

Writing An Image
To write a image, we agai "se 9odim, speci+yi g the ame o+ the optical me!ia writer !e#ice a ! the ame o+ the image +ile,
wodim dev$/dev/cdrw image.iso

9 a!!itio to the !e#ice ame a ! image +ile, 9odim s"pports a #ery large set o+ optio s. Two commo o es are ?-#@ +or #er%ose o"tp"t, a ! ?-!ao@ which writes the !isk i disk-at-once mo!e. This mo!e sho"l! %e "se! i+ yo" are prepari g a !isk +or commercial repro!"ctio . The !e+a"lt mo!e +or 9odim is track-at-once, which is "se+"l +or recor!i g m"sic tracks.

;urther 1eading
We ha#e 4"st to"che! o the ma y ways that the comma ! li e ca %e "se! to ma age storage me!ia. Take a look at the ma pages o+ the comma !s we ha#e co#ere!. Some o+ them s"pport h"ge "m%ers o+ optio s a ! operatio s. $lso, look +or o -li e t"torials +or a!!i g har! !ri#es to yo"r *i "3 system Ethere are ma yF a ! worki g with optical me!ia.

$xtra Credit
9tIs o+te "se+"l to #eri+y the i tegrity o+ a iso image that we ha#e !ow loa!e!. 9 most cases, a !istri%"tor o+ a iso image will also s"pply a checksum file. $ checks"m is the res"lt o+ a e3otic mathematical calc"latio res"lti g i a "m%er that represe ts the co te t o+ the target +ile. 9+ the co te ts o+ the +ile cha ge %y e#e o e %it, the res"lti g checks"m will %e m"ch !i++ere t. The most commo metho! o+ checks"m ge eratio .9/

.K 8 Storage >e!ia "ses the md"sum program. Whe yo" "se md"sum, it pro!"ces a " i="e he3a!ecimal "m%er,
md"sum image.iso $%e$"%).(f0bb)fbf/"c0.f.a$f0%ece

image<iso

$+ter yo" !ow loa! a image, yo" sho"l! r" md"sum agai st it a ! compare the res"lts with the md"sum #al"e s"pplie! %y the p"%lisher. 9 a!!itio to checki g the i tegrity o+ a !ow loa!e! +ile, we ca "se m!2s"m to #eri+y ewly writte optical me!ia. To !o this, we +irst calc"late the checks"m o+ the image +ile a ! the calc"late a checks"m +or the me!ia. The trick to #eri+yi g the me!ia is to limit the calc"latio to o ly the portio o+ the optical me!ia that co tai s the image. We !o this %y !etermi i g the "m%er o+ 2018 %yte %locks the image co tai s Eoptical me!ia is always writte i 2018 %yte %locksF a ! rea!i g that ma y %locks +rom the me!ia. 7 some types o+ me!ia, this is ot re="ire!. $ C'-: writte i !isk-at-o ce mo!e ca %e checke! this way,
md"sum /dev/cdrom $%e$"%).(f0bb)fbf/"c0.f.a$f0%ece

6dev6cdrom

>a y types o+ me!ia, s"ch as 'R's re="ire a precise calc"latio o+ the "m%er o+ %locks. 9 the e3ample %elow, we check the i tegrity o+ the image +ile dvdimage<iso a ! the !isk i the 'R' rea!er 6dev6dvd. Ca yo" +ig"re o"t how this worksC
md"sum dvd-image.iso# dd if$/dev/dvd bs$2*AM count$344 34stat -c J8sJ dvd-image.iso6 / 2*AM 66 0 md"sum

.98

./ 8 &etworki g

1. Net5or,ing
Whe it comes to etworki g, there is pro%a%ly othi g that ca ot %e !o e with *i "3. *i "3 is "se! to %"il! all sorts o+ etworki g systems a ! applia ces, i cl"!i g +irewalls, ro"ters, ame ser#ers, &$S E&etwork $ttache! StorageF %o3es a ! o a ! o . J"st as the s"%4ect o+ etworki g is #ast, so are the "m%er o+ comma !s that ca %e "se! to co +ig"re a ! co trol it. We will +oc"s o"r atte tio o 4"st a +ew o+ the most +re="e tly "se! o es. The comma !s chose +or e3ami atio i cl"!e those "se! to mo itor etworks a ! those "se! to tra s+er +iles. 9 a!!itio , we are goi g to e3plore the ssh program that is "se! to per+orm remote logi s. This chapter will co#er,

7ing - Se ! a 9C>6 ECA7O:ET)EST to etwork hosts traceroute - 6ri t the ro"te packets trace to a etwork host netstat - 6ri t etwork co ectio s, ro"ti g ta%les, i ter+ace statistics, mas="era!e co ectio s, a ! m"lticast mem%erships ft7 - 9 ter et +ile tra s+er program 9get - &o -i teracti#e etwork !ow loa!er ssh - 7pe SSA SSA clie t Eremote logi programF

WeIre goi g to ass"me a little %ackgro" ! i etworki g. 9 this, the 9 ter et age, e#eryo e "si g a comp"ter ee!s a %asic " !ersta !i g o+ etworki g co cepts. To make +"ll "se o+ this chapter we sho"l! %e +amiliar with the +ollowi g terms,

96 E9 ter et 6rotocolF a!!ress Aost a ! !omai ame ):9 E) i+orm :eso"rce 9!e ti+ierF

6lease see the ?0"rther :ea!i g@ sectio %elow +or some "se+"l articles regar!i g these terms. 1ote5 Some o+ the comma !s we will co#er may E!epe !i g o yo"r !istri%"tio F re="ire the i stallatio o+ a!!itio al packages +rom yo"r !istri%"tio Is repositories, .99

./ 8 &etworki g a ! some may re="ire s"per"ser pri#ileges to e3ec"te.

$xamining #nd >onitoring # 7et4or(

E#e i+ yo"Ire ot the system a!mi istrator, itIs o+te help+"l to e3ami e the per+orma ce a ! operatio o+ a etwork.

7ing
The most %asic etwork comma ! is 7ing. The 7ing comma ! se !s a special etwork packet calle! a 9>C6 ECA7O:ET)EST to a speci+ie! host. >ost etwork !e#ices recei#i g this packet will reply to it, allowi g the etwork co ectio to %e #eri+ie!. 1ote5 9t is possi%le to co +ig"re most etwork !e#ices Ei cl"!i g *i "3 hostsF to ig ore these packets. This is "s"ally !o e +or sec"rity reaso s, to partially o%sc"re a host +rom a pote tial attacker. 9t is also commo +or +irewalls to %e co +ig"re! to %lock 9>C6 tra++ic. 0or e3ample, to see i+ we ca reach linuxcommand<org Eo e o+ o"r +a#orite sites H-F, we ca "se "se 7ing like this,
[me@linuxbox ~]$ ping linuxcommand.org

7 ce starte!, 7ing co ti "es to se ! packets at a speci+ie! i ter#al E!e+a"lt is o e seco !F " til it is i terr"pte!,
[me@linuxbox ~]$ ping linuxcommand.org :AL@ linuxcommand<org N..<$"<!"(<!#(P ".N/%P b1tes of data< .% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM# ttlM%$ timeM#() ms .% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM! ttlM%$ timeM#(/ ms .% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM$ ttlM%$ timeM#(. ms .% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM% ttlM%$ timeM#(. ms .% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM" ttlM%$ timeM#(" ms

200

E3ami i g $ ! >o itori g $ &etwork

.% b1tes from vhost<sourceforge<net N..<$"<!"(<!#(P: icm7OseBM. ttlM%$ timeM#() ms --- linuxcommand<org 7ing statistics --. 7ackets transmitted> . received> (5 7acket loss> time .(#(ms rtt min6avg6max6mdev M #("<.%)6#()<("!6#(/<##/6(</!% ms

$+ter it is i terr"pte! Ei this case a+ter the si3th packetF %y pressi g Ctrl-c, 7ing pri ts per+orma ce statistics. $ properly per+ormi g etwork will e3hi%it Dero perce t packet loss. $ s"ccess+"l ?pi g@ will i !icate that the eleme ts o+ the etwork Eits i ter+ace car!s, ca%li g, ro"ti g a ! gatewaysF are i ge erally goo! worki g or!er.

traceroute
The traceroute program Esome systems "se the similar trace7ath program i stea!F !isplays a listi g o+ all the ?hops@ etwork tra++ic takes to get +rom the local system to a speci+ie! host. 0or e3ample, to see the ro"te take to reach slashdot<org, we wo"l! !o this,
[me@linuxbox ~]$ traceroute slashdot.org

The o"tp"t looks like this,

traceroute to slashdot<org N!#.<$%<#/#<%"P> $( ho7s max> %( b1te 7ackets # i7co7<localdomain N#0!<#./<#<#P #<(.. ms #<$.. ms #<)!( ms ! E E E $ ge-%-#$-ur(#<rockville<md<bad<comcast<net N./</)<#$(<0P #%<.!! ms #%<//" ms #"<#.0 ms % 7o-$(-ur(!<rockville<md<bad<comcast<net N./</)<#!0<#"%P #)<.$% ms #)<.!. ms #)</00 ms " 7o-.(-ur($<rockville<md<bad<comcast<net N./</)<#!0<#"/P #"<00! ms #"<0/$ ms #.<!". ms . 7o-$(-ar(#<ho9ardcount1<md<bad<comcast<net N./</)<#$.<"P !!</$" ms #%<!$$ ms #%<%(" ms ) 7o-#(-ar(!<9hitemarsh<md<bad<comcast<net N./</)<#!0<$%P #.<#"% ms #$<.(( ms #/</.) ms / te-(-$-(-#-cr(#<7hiladel7hia<7a<ibone<comcast<net N./</.<0(<))P !#<0"# ms !#<()$ ms !#<"") ms 0 7os-(-/-(-(-cr(#<ne91ork<n1<ibone<comcast<net N./</.</"<#(P !!<0#) ms !#<//% ms !!<#!. ms #( !(%<)(<#%%<# N!(%<)(<#%%<#P %$<##( ms !#<!%/ ms !#<!.% ms ## cr#-7os-(-)-$-#<ne91ork<savvis<net N!(%<)(<#0"<0$P !#</") ms

20.

./ 8 &etworki g
cr!-7os-(-(-$-#<ne91ork<savvis<net N!(%<)(<!(%<!$/P #0<"". ms cr#7os-(-)-$-#<ne91ork<savvis<net N!(%<)(<#0"<0$P #0<.$% ms #! cr!-7os-(-)-$-(<chicago<savvis<net N!(%<)(<#0!<#(0P %#<"/. ms %!</%$ ms cr!-tengig-(-(-!-(<chicago<savvis<net N!(%<)(<#0.<!%!P %$<##" ms #$ hr!-tengigabitethernet-#!-#<elkgrovech$<savvis<net N!(%<)(<#0"<#!!P %%<!#" ms %#</$$ ms %"<."/ ms #% csr#-ve!%#<elkgrovech$<savvis<net N!#.<.%<#0%<%!P %.</%( ms %$<$)! ms %)<(%# ms #" .%<!)<#.(<#0% N.%<!)<#.(<#0%P ".<#$) ms ""<//) ms "!</#( ms #. slashdot<org N!#.<$%<#/#<%"P %!<)!) ms %!<(#. ms %#<%$) ms

9 the o"tp"t, we ca see that co ecti g +rom o"r test system to slashdot<org re="ires tra#ersi g si3tee ro"ters. 0or ro"ters that pro#i!e! i!e ti+yi g i +ormatio , we see their host ames, 96 a!!resses a ! per+orma ce !ata, which i cl"!es three samples o+ ro" !-trip time +rom the local system to the ro"ter. 0or ro"ters that !o ot pro#i!e i!e ti+yi g i +ormatio E%eca"se o+ ro"ter co +ig"ratio , etwork co gestio , +irewalls, etc.F, we see asterisks as i the li e +or hop "m%er two.

netstat
The netstat program is "se! to e3ami e #ario"s etwork setti gs a ! statistics. Thro"gh the "se o+ its ma y optio s, we ca look at a #ariety o+ +eat"res i o"r etwork set"p. )si g the ?-ie@ optio , we ca e3ami e the etwork i ter+aces i o"r system,
[me@linuxbox ~]$ netstat -ie eth( =ink enca7:&thernet S,addr ((:#d:(0:0b:00:.) inet addr:#0!<#./<#<! Hcast:#0!<#./<#<!"" +ask:!""<!""<!""<( inet. addr: fe/(::!#d:0ff:fe0b:00.)6.% *co7e:=ink 3: HR 4'C4*T R3LLAL@ +3=TAC4*T +T3:#"(( +etric:# RV 7ackets:!$/%// errors:( dro77ed:( overruns:( frame:( TV 7ackets:%($!#) errors:( dro77ed:( overruns:( carrier:( collisions:( txBueuelen:#(( RV b1tes:#"$(0/0!# N#%.<( +HP TV b1tes:!.#($"!%. N!%/<0 +HP +emor1:fdfc((((-fdfe(((( lo =ink enca7:=ocal =oo7back inet addr:#!)<(<(<# +ask:!""<(<(<( inet. addr: ::#6#!/ *co7e:Sost 3: = :H4C2 R3LLAL@ +T3:#.%$. +etric:# RV 7ackets:!!(/ errors:( dro77ed:( overruns:( frame:( TV 7ackets:!!(/ errors:( dro77ed:( overruns:( carrier:( collisions:( txBueuelen:( RV b1tes:###%0( N#(/</ 2HP TV b1tes:###%0( N#(/</ 2HP

202

E3ami i g $ ! >o itori g $ &etwork 9 the e3ample a%o#e, we see that o"r test system has two etwork i ter+aces. The +irst, calle! eth(, is the Ether et i ter+ace a ! the seco !, calle! lo, is the loopback interface, a #irt"al i ter+ace that the system "ses to ?talk to itsel+.@ Whe per+ormi g ca"sal etwork !iag ostics, the importa t thi gs to look +or are the prese ce o+ the wor! ?)6@ at the %egi i g o+ the +o"rth li e +or each i ter+ace, i !icati g that the etwork i ter+ace is e a%le!, a ! the prese ce o+ a #ali! 96 a!!ress i the inet addr +iel! o the seco ! li e. 0or systems "si g 'AC6 E'y amic Aost Co +ig"ratio 6rotocolF, a #ali! 96 a!!ress i this +iel! will #eri+y that the 'AC6 is worki g. )si g the ?-r@ optio will !isplay the ker elIs etwork ro"ti g ta%le. This shows how the etwork is co +ig"re! to se ! packets +rom etwork to etwork,
[me@linuxbox ~]$ netstat -r 2ernel A: routing table 'estination @ate9a1 @enmask #0!<#./<#<( default

-lags

+** ,indo9 ( ( ( (

irtt Aface ( eth( ( eth(

E !""<!""<!""<( 3 #0!<#./<#<# (<(<(<( 3@

9 this simple e3ample, we see a typical ro"ti g ta%le +or a clie t machi e o a *$& E*ocal $rea &etworkF %ehi ! a +irewall-ro"ter. The +irst li e o+ the listi g shows the !esti atio #0!<#./<#<(. 96 a!!resses that e ! i Dero re+er to etworks rather tha i !i#i!"al hosts, so this !esti atio mea s a y host o the *$&. The e3t +iel!, @ate9a1, is the ame or 96 a!!ress o+ the gateway Ero"terF "se! to go +rom the c"rre t host to the !esti atio etwork. $ asterisk i this +iel! i !icates that o gateway is ee!e!. The last li e co tai s the !esti atio default. This mea s a y tra++ic !esti e! +or a etwork that is ot otherwise liste! i the ta%le. 9 o"r e3ample, we see that the gateway is !e+i e! as a ro"ter with the a!!ress o+ #0!<#./<#<#, which pres"ma%ly k ows what to !o with the !esti atio tra++ic. The netstat program has ma y optio s a ! we ha#e o ly looke! at a co"ple. Check o"t the netstat ma page +or a complete list.

Transporting ;iles ?%er # 7et4or(

What goo! is a etwork " less we k ow how to mo#e +iles across itC There are ma y programs that mo#e !ata o#er etworks. We will co#er two o+ them ow a ! se#eral more i later sectio s.

20(

./ 8 &etworki g

ft7
7 e o+ the tr"e ?classic@ programs, ft7 gets it ame +rom the protocol it "ses, the File Transfer 6rotocol. 0T6 is "se! wi!ely o the 9 ter et +or +ile !ow loa!s. >ost, i+ ot all, we% %rowsers s"pport it a ! yo" o+te see ):9s starti g with the protocol ft7:66. Je+ore there were we% %rowsers, there was the ft7 program. ft7 is "se! to comm" icate with FT6 servers, machi es that co tai +iles that ca %e "ploa!e! a ! !ow loa!e! o#er a etwork. 0T6 Ei its origi al +ormF is ot sec"re, %eca"se it se !s acco" t ames a ! passwor!s i clearte+t. This mea s that they are ot e crypte! a ! a yo e sniffing the etwork ca see them. Jeca"se o+ this, almost all 0T6 !o e o#er the 9 ter et is !o e %y anonymous FT6 servers. $ a o ymo"s ser#er allows a yo e to logi "si g the logi ame ?a o ymo"s@ a ! a mea i gless passwor!. 9 the e3ample %elow, we show a typical sessio with the ft7 program !ow loa!i g a )%" t" iso image locate! i the 67ub6cdOimages63buntu-/<(% !irectory o+ the a o ymo"s 0T6 ser#er fileserver,
[me@linuxbox ~]$ ftp fileserver Connected to fileserver<localdomain< !!( Nvs-T:d !<(<#P Lame Nfileserver:meP: anon!mous $$# :lease s7ecif1 the 7ass9ord< :ass9ord: !$( =ogin successful< Remote s1stem t17e is 3LAV< 3sing binar1 mode to transfer files< ft7D cd pub/cd.images/Ebuntu-M.*A !"( 'irector1 successfull1 changed< ft7D ls !(( : RT command successful< Consider using :4*;< #"( Sere comes the director1 listing< -r9-r9-r-# "(( "(( )$$()0""! 47r !" ($:"$ ubuntu-/<(%deskto7-i$/.<iso !!. 'irector1 send 2< ft7D lcd 1esktop =ocal director1 no9 6home6me6'eskto7 ft7D get ubuntu-M.*A-desktop-i MN.iso local: ubuntu-/<(%-deskto7-i$/.<iso remote: ubuntu-/<(%-deskto7i$/.<iso !(( : RT command successful< Consider using :4*;< #"( 7ening HAL4RU mode data connection for ubuntu-/<(%-deskto7i$/.<iso N)$$()0""! b1tesP< !!. -ile send 2< )$$()0""! b1tes received in ./<". secs N#(%%#<" kH6sP

201

Tra sporti g 0iles 7#er $ &etwork

ft7D b!e

Aere is a e3pla atio o+ the comma !s e tere! !"ri g this sessio , Command ft7 fileserver >eaning 9 #oke the ft7 program a ! ha#e it co ect to the 0T6 ser#er fileserver. *ogi ame. $+ter the logi prompt, a passwor! prompt will appear. Some ser#ers will accept a %la k passwor!, others will re="ire a passwor! i the +orm o+ a email a!!ress. 9 that case, try somethi g like ?"serNe3ample.com@. Cha ge to the !irectory o the remote system co tai i g the !esire! +ile. &ote that o most a o ymo"s 0T6 ser#ers, the +iles +or p"%lic !ow loa!i g are +o" ! somewhere " !er the 7ub !irectory. *ist the !irectory o the remote system. Cha ge the !irectory o the local system to ~6'eskto7. 9 the e3ample, the ft7 program was i #oke! whe the worki g !irectory was ~. This comma ! cha ges the worki g !irectory to ~6'eskto7. Tell the remote system to tra s+er the +ile ubuntu-/<(%-deskto7i$/.<iso to the local system. Si ce the worki g !irectory o the local system was cha ge! to ~6'eskto7, the +ile will %e !ow loa!e! there. *og o++ the remote ser#er a ! e ! the ft7 program sessio . The comma !s

anon1mous

cd 7ub6cdOimages63buntu-/<(%

ls lcd 'eskto7

get ubuntu-/<(%-deskto7i$/.<iso

b1e

202

./ 8 &etworki g Buit a ! exit may also %e "se!. Typi g ?help@ at the ?+tpX@ prompt will !isplay a list o+ the s"pporte! comma !s. )si g +tp o a ser#er where s"++icie t permissio s ha#e %ee gra te!, it is possi%le to per+orm ma y or!i ary +ile ma ageme t tasks. 9tIs cl"msy, %"t it !oes work.

lft7 A A Better ft7

ft7 is ot the o ly comma ! li e 0T6 clie t. 9 +act, there are ma y. 7 e o+ %etter Ea ! more pop"larF o es is lft7 %y $le3a !er *"kya o#. 9t works m"ch like the tra!itio al ft7 program, %"t has ma y a!!itio al co #e ie ce +eat"res i cl"!i g m"ltiple protocol s"pport Ei cl"!i g ATT6F, a"tomatic re-try o +aile! !ow loa!s, %ackgro" ! processes, ta% completio o+ path ames, a ! ma y more.

9get
$ other pop"lar comma ! li e program +or +ile !ow loa!i g is 9get. 9t is "se+"l +or !ow loa!i g co te t +rom %oth we% a ! 0T6 sites. Si gle +iles, m"ltiple +iles, a ! e#e e tire sites ca %e !ow loa!e!. To !ow loa! the +irst page o+ linuxcommand<org we co"l! !o this,
[me@linuxbox ~]$ wget http2//linuxcommand.org/index.php --##:(!:"#-- htt7:66linuxcommand<org6index<7h7 MD Jindex<7h7K Resolving linuxcommand<org<<< ..<$"<!"(<!#( Connecting to linuxcommand<orgQ..<$"<!"(<!#(Q:/(<<< connected< STT: reBuest sent> a9aiting res7onse<<< !(( 2 =ength: uns7ecified [text6html] [ XMD ] $>#!( --<--26s

##:(!:"# N#.#<)" +H6sP - Jindex<7h7K saved [$#!(]

The program<s ma y optio s allow 9get to rec"rsi#ely !ow loa!, !ow loa! +iles i the %ackgro" ! Eallowi g yo" to log o++ %"t co ti "e !ow loa!i gF, a ! complete the !ow loa! o+ a partially !ow loa!e! +ile. These +eat"res are well !oc"me te! i its %etter-tha -a#erage ma page.

Secure Communication +ith 1emote 3osts

0or ma y years, ) i3-like operati g systems ha#e ha! the a%ility to %e a!mi istere! 20K

Sec"re Comm" icatio With :emote Aosts remotely #ia a etwork. 9 the early !ays, %e+ore the ge eral a!optio o+ the 9 ter et, there were a co"ple o+ pop"lar programs "se! to log i to remote hosts. These were the rlogin a ! telnet programs. These programs, howe#er, s"++er +rom the same +atal +law that the ft7 program !oesH they tra smit all their comm" icatio s Ei cl"!i g logi ames a ! passwor!sF i clearte3t. This makes them wholly i appropriate +or "se i the 9 ter et age.

ssh
To a!!ress this pro%lem, a ew protocol calle! SSA ESec"re ShellF was !e#elope!. SSA sol#es the two %asic pro%lems o+ sec"re comm" icatio with a remote host. 0irst, it a"the ticates that the remote host is who it says it is Eth"s pre#e ti g so-calle! ?ma i the mi!!le@ attacksF, a ! seco !, it e crypts all o+ the comm" icatio s %etwee the local a ! remote hosts. SSA co sists o+ two parts. $ SSA ser#er r" s o the remote host, liste i g +or i comi g co ectio s o port twe ty-two, while a SSA clie t is "se! o the local system to comm" icate with the remote ser#er. >ost *i "3 !istri%"tio s ship a impleme tatio o+ SSA calle! 7pe SSA +rom the JS' pro4ect. Some !istri%"tio s i cl"!e %oth the clie t a ! the ser#er packages %y !e+a"lt E+or e3ample, :e! AatF, while others Es"ch as )%" t"F o ly s"pply the clie t. To e a%le a system to recei#e remote co ectio s, it m"st ha#e the 77en**S-server package i stalle!, co +ig"re! a ! r" i g, a ! Ei+ the system is either r" i g or is %ehi ! a +irewallF it m"st allow i comi g etwork co ectio s o TC6 port 22. Ti(, 9+ yo" !o It ha#e a remote system to co ect to %"t wa t to try these e3amples, make s"re the 7en**S-server package is i stalle! o yo"r system a ! "se localhost as the ame o+ the remote host. That way, yo"r machi e will create etwork co ectio s with itsel+. The SSA clie t program "se! to co ect to remote SSA ser#ers is calle!, appropriately e o"gh, ssh. To co ect to a remote host ame! remote-s1s, we wo"l! "se the ssh clie t program like so,
[me@linuxbox ~]$ ssh remote-s!s The authenticit1 of host Kremote-s1s N#0!<#./<#<%PK canKt be established< R*4 ke1 finger7rint is %#:ed:)a:df:!$:#0:bf:$c:a":#):bc:.#:b$:)f:d0:bb< 4re 1ou sure 1ou 9ant to continue connecting N1es6noPF

20/

./ 8 &etworki g The +irst time the co ectio is attempte!, a message is !isplaye! i !icati g that the a"the ticity o+ the remote host ca ot %e esta%lishe!. This is %eca"se the clie t program has e#er see this remote host %e+ore. To accept the cre!e tials o+ the remote host, e ter ?yes@ whe prompte!. 7 ce the co ectio is esta%lishe!, the "ser is prompte! +or his-her passwor!,
,arning: :ermanentl1 added Kremote-s1s>#0!<#./<#<%K NR*4P to the list of kno9n hosts< me@remote-s1sKs 7ass9ord:

$+ter the passwor! is s"ccess+"lly e tere!, we recei#e the shell prompt +rom the remote system,
=ast login: *at 4ug $( #$:((:%/ !((/ [me@remote-s1s ~]$

The remote shell sessio co ti "es " til the "ser e ters the exit comma ! at the remote shell prompt, there%y closi g the remote co ectio . $t this poi t, the local shell sessio res"mes a ! the local shell prompt reappears. 9t is also possi%le to co ect to remote systems "si g a !i++ere t "ser ame. 0or e3ample, i+ the local "ser ?me@ ha! a acco" t ame! ?%o%@ o a remote system, "ser me co"l! log i to the acco" t bob o the remote system as +ollows,
[me@linuxbox ~]$ ssh bobVremote-s!s bob@remote-s1sKs 7ass9ord: =ast login: *at 4ug $( #$:($:!# !((/ [bob@remote-s1s ~]$

$s state! %e+ore, ssh #eri+ies the a"the ticity o+ the remote host. 9+ the remote host !oes ot s"ccess+"lly a"the ticate, the +ollowi g message appears,
[me@linuxbox ~]$ ssh remote-s!s @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ ,4RLAL@: R&+ T& S *T A'&LTA-AC4TA L S4* CS4L@&'G @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ AT A* : **AH=& TS4T * +& L& A* ' AL@ * +&TSAL@ L4*TUG *omeone could be eavesdro77ing on 1ou right no9 Nman-in-the-middle attackPG

208

Sec"re Comm" icatio With :emote Aosts

At is also 7ossible that the R*4 host ke1 has just been changed< The finger7rint for the R*4 ke1 sent b1 the remote host is %#:ed:)a:df:!$:#0:bf:$c:a":#):bc:.#:b$:)f:d0:bb< :lease contact 1our s1stem administrator< 4dd correct host ke1 in 6home6me6<ssh6kno9nOhosts to get rid of this message< ffending ke1 in 6home6me6<ssh6kno9nOhosts:# R*4 host ke1 for remote-s1s has changed and 1ou have reBuested strict checking< Sost ke1 verification failed<

This message is ca"se! %y o e o+ two possi%le sit"atio s. 0irst, a attacker may %e attempti g a ?ma -i -the-mi!!le@ attack. This is rare, si ce e#ery%o!y k ows that ssh alerts the "ser to this. The more likely c"lprit is that the remote system has %ee cha ge! somehowH +or e3ample, its operati g system or SSA ser#er has %ee rei stalle!. 9 the i terests o+ sec"rity a ! sa+ety howe#er, the +irst possi%ility sho"l! ot %e !ismisse! o"t o+ ha !. $lways check with the a!mi istrator o+ the remote system whe this message occ"rs. $+ter it has %ee !etermi e! that the message is !"e to a %e ig ca"se, it is sa+e to correct the pro%lem o the clie t si!e. This is !o e %y "si g a te3t e!itor Evim perhapsF to remo#e the o%solete key +rom the ~6<ssh6kno9nOhosts +ile. 9 the e3ample message a%o#e, we see this,
ffending ke1 in 6home6me6<ssh6kno9nOhosts:#

This mea s that li e o e o+ the kno9nOhosts +ile co tai s the o++e !i g key. 'elete this li e +rom the +ile, a ! the ssh program will %e a%le to accept ew a"the ticatio cre!e tials +rom the remote system. Jesi!es ope i g a shell sessio o a remote system, ssh also allows "s to e3ec"te a si gle comma ! o a remote system. 0or e3ample, to e3ec"te the free comma ! o a remote host ame! remote-s1s a ! ha#e the res"lts !isplaye! o the local system,
[me@linuxbox ~]$ ssh remote-s!s free me@t9in%Ks 7ass9ord: total used free +em: ))""$. "()#/% !%!"!( ( !./$"! "$$(#. #")!/".

shared (

buffers ##((./

cached #"%"0.

-68 buffers6cache: *9a7: #")!/".

209

./ 8 &etworki g
[me@linuxbox ~]$

9tIs possi%le to "se this tech i="e i more i teresti g ways, s"ch as this e3ample i which we per+orm a ls o the remote system a ! re!irect the o"tp"t to a +ile o the local system,
[me@linuxbox ~]$ ssh remote-s!s %ls *% & dirlist.txt me@t9in%Ks 7ass9ord: [me@linuxbox ~]$

&otice the "se o+ the si gle ="otes i the comma ! a%o#e. This is !o e %eca"se we !o ot wa t the path ame e3pa sio per+orme! o the local machi eH rather, we wa t it to %e per+orme! o the remote system. *ikewise, i+ we ha! wa te! the o"tp"t re!irecte! to a +ile o the remote machi e, we co"l! ha#e place! the re!irectio operator a ! the +ile ame withi the si gle ="otes,
[me@linuxbox ~]$ ssh remote-s!s %ls * & dirlist.txt%

Tunneling +ith SS3

6art o+ what happe s whe yo" esta%lish a co ectio with a remote host #ia SSA is that a encrypted tunnel is create! %etwee the local a ! remote systems. &ormally, this t" el is "se! to allow comma !s type! at the local system to %e tra smitte! sa+ely to the remote system, a ! +or the res"lts to %e tra smitte! sa+ely %ack. 9 a!!itio to this %asic +" ctio , the SSA protocol allows most types o+ etwork tra++ic to %e se t thro"gh the e crypte! t" el, creati g a sort o+ B6> ERirt"al 6ri#ate &etworkF %etwee the local a ! remote systems. 6erhaps the most commo "se o+ this +eat"re is to allow G Wi !ow system tra++ic to %e tra smitte!. 7 a system r" i g a G ser#er Ethat is, a machi e !isplayi g a ;)9F, it is possi%le to la" ch a ! r" a G clie t program Ea graphical applicatio F o a remote system a ! ha#e its !isplay appear o the local system. 9tIs easy to !o, hereIs a e3ample, letIs say we are sitti g at a *i "3 system calle! linuxbox which is r" i g a G ser#er, a ! we wa t to r" the xload program o a remote system ame! remote-s1s a ! see the programIs graphical o"tp"t o o"r local system. We co"l! !o this,

2.0

Sec"re Comm" icatio With :emote Aosts

[me@linuxbox ~]$ ssh -Z remote-s!s me@remote-s1sKs 7ass9ord: =ast login: +on *e7 (/ #$:!$:## !((/ [me@remote-s1s ~]$ xload $+ter the xload comma ! is e3ec"te! o the remote system, its wi !ow appears o the local system. 7 some systems, yo" may ee! to "se the ?-5@ optio rather tha the ?-G@ optio to !o this.

sc7 And sft7

The 7pe SSA package also i cl"!es two programs that ca make "se o+ a SSA e crypte! t" el to copy +iles across the etwork. The +irst, sc7 Esec"re copyF is "se! m"ch like the +amiliar c7 program to copy +iles. The most ota%le !i++ere ce is that the so"rce or !esti atio path ames may %e prece!e! with the ame o+ a remote host, +ollowe! %y a colo character. 0or e3ample, i+ we wa te! to copy a !oc"me t ame! document<txt +rom o"r home !irectory o the remote system, remote-s1s, to the c"rre t worki g !irectory o o"r local system, we co"l! !o this,
[me@linuxbox ~]$ scp remote-s!s2document.txt . me@remote-s1sKs 7ass9ord: document<txt #((5 ""/# [me@linuxbox ~]$

"<"2H6s

((:((

$s with ssh, yo" may apply a "ser ame to the %egi i g o+ the remote hostIs ame i+ the !esire! remote host acco" t ame !oes ot match that o+ the local system,
[me@linuxbox ~]$ scp bobVremote-s!s2document.txt .

The seco ! SSA +ile copyi g program is sft7 which, as its ame implies, is a sec"re replaceme t +or the ft7 program. sft7 works m"ch like the origi al ft7 program that we "se! earlierH howe#er, i stea! o+ tra smitti g e#erythi g i clearte3t, it "ses a SSA e crypte! t" el. sft7 has a importa t a!#a tage o#er co #e tio al ft7 i that it !oes ot re="ire a 0T6 ser#er to %e r" i g o the remote host. 9t o ly re="ires the SSA ser#er. This mea s that a y remote machi e that ca co ect with the SSA clie t ca also %e "se! as a 0T6-like ser#er. Aere is a sample sessio ,

2..

./ 8 &etworki g
[me@linuxbox ~]$ sftp remote-s!s Connecting to remote-s1s<<< me@remote-s1sKs 7ass9ord: sft7D ls ubuntu-/<(%-deskto7-i$/.<iso sft7D lcd 1esktop sft7D get ubuntu-M.*A-desktop-i MN.iso -etching 6home6me6ubuntu-/<(%-deskto7-i$/.<iso to ubuntu-/<(%deskto7-i$/.<iso 6home6me6ubuntu-/<(%-deskto7-i$/.<iso #((5 .00+H )<%+H6s (#:$" sft7D b!e

Ti(5 The S0T6 protocol is s"pporte! %y ma y o+ the graphical +ile ma agers +o" ! i *i "3 !istri%"tio s. )si g either &a"til"s E;&7>EF or Mo ="eror EM'EF, we ca e ter a ):9 %egi i g with sft7:66 i to the locatio %ar a ! operate o +iles store! o a remote system r" i g a SSA ser#er.

#n SS3 Client ;or +indo4s.

*etIs say yo" are sitti g at a Wi !ows machi e %"t yo" ee! to log i to yo"r *i "3 ser#er a ! get some real work !o e, what !o yo" !oC ;et a SSA clie t program +or yo"r Wi !ows %o3, o+ co"rseL There are a "m%er o+ these. The most pop"lar o e is pro%a%ly 6"TT5 %y Simo Tatham a ! his team. The 6"TT5 program !isplays a termi al wi !ow a ! allow a Wi !ows "ser to ope a SSA Eor tel etF sessio o a remote host. The program also pro#i!es a alogs +or the sc7 a ! sft7 programs. 6"TT5 is a#aila%le at http,--www.chiark.gree e !.org."k-Ssgtatham-p"tty-

;urther 1eading

0or a %roa! Eal%eit !ate!F look at etwork a!mi istratio , the *i "3 'oc"me tatio 6ro4ect pro#i!es the (inu+ >etwork )dministratorEs @uide, http,--tl!p.org-*'6- ag2-i !e3.html Wikipe!ia co tai s ma y goo! etworki g articles. Aere are some o+ the %asics, http,--e .wikipe!ia.org-wiki-9 ter etOprotocolOa!!ress http,--e .wikipe!ia.org-wiki-AostO ame http,--e .wikipe!ia.org-wiki-) i+ormO:eso"rceO9!e ti+ier

2.2

.8 8 Searchi g 0or 0iles

1- Searching (or (iles

$s we ha#e wa !ere! aro" ! o"r *i "3 system, o e thi g has %ecome a%" !a tly clear, a typical *i "3 system has a lot o+ +ilesL This %egs the ="estio , ?how !o we +i ! thi gsC@ We alrea!y k ow that the *i "3 +ile system is well orga iDe! accor!i g to co #e tio s that ha#e %ee passe! !ow +rom o e ge eratio o+ ) i3-like system to the e3t, %"t the sheer "m%er o+ +iles ca prese t a !a" ti g pro%lem. 9 this chapter, we will look at two tools that are "se! to +i ! +iles o a system. These tools are,

locate 8 0i ! +iles %y ame find 8 Search +or +iles i a !irectory hierarchy

We will also look at a comma ! that is o+te "se! with +ile search comma !s to process the res"lti g list o+ +iles,

xargs 8 J"il! a ! e3ec"te comma ! li es +rom sta !ar! i p"t touch 8 Cha ge +ile times stat 8 'isplay +ile or +ile system stat"s

9 a!!itio , we will i tro!"ce a co"ple o+ comma !s to assist "s i or e3ploratio ,

locate ;ind ;iles The $as, +a,

The locate program per+orms a rapi! !ata%ase search o+ path ames a ! o"tp"ts e#ery ame that matches a gi#e s"%stri g. Say, +or e3ample, we wa t to +i ! all the programs with ames that %egi with ?Dip.@ Si ce we are looki g +or programs, we ca ass"me that the !irectory co tai i g the programs wo"l! e ! with ?%i -@. There+ore, we co"l! try to "se locate this way to +i ! o"r +iles,
[me@linuxbox ~]$ locate bin/-ip

locate will search its !ata%ase o+ path ames a ! o"tp"t a y that co tai the stri g 2.(

.8 8 Searchi g 0or 0iles ?%i -Dip@,

6usr6bin6Ci7 6usr6bin6Ci7cloak 6usr6bin6Ci7gre7 6usr6bin6Ci7info 6usr6bin6Ci7note 6usr6bin6Ci7s7lit

9+ the search re="ireme t is ot so simple, locate ca %e com%i e! with other tools s"ch as gre7 to !esig more i teresti g searches,
[me@linuxbox ~]$ locate -ip 0 grep bin 6bin6bunCi7! 6bin6bCi7! 6bin6bCi7!recover 6bin6gunCi7 6bin6gCi7 6usr6bin6funCi7 6usr6bin6g7g-Ci7 6usr6bin67reunCi7 6usr6bin67reCi7 6usr6bin67reCi7-bin 6usr6bin6unCi7 6usr6bin6unCi7sfx 6usr6bin6Ci7 6usr6bin6Ci7cloak 6usr6bin6Ci7gre7 6usr6bin6Ci7info 6usr6bin6Ci7note 6usr6bin6Ci7s7lit

The locate program has %ee aro" ! +or a "m%er o+ years, a ! there are se#eral !i++ere t #aria ts i commo "se. The two most commo o es +o" ! i mo!er *i "3 !istri%"tio s are slocate a ! mlocate, tho"gh they are "s"ally accesse! %y a sym%olic li k ame! locate. The !i++ere t #ersio s o+ locate ha#e o#erlappi g optio s sets. Some #ersio s i cl"!e reg"lar e3pressio matchi g Ewhich weIll co#er i a "pcomi g chapterF a ! wil! car! s"pport. Check the ma page +or locate to !etermi e which #ersio o+ locate is i stalle!.

2.1

locate 8 0i ! 0iles The Easy Way

+here Does The locate Data0ase Come ;rom.

5o" may otice that, o some !istri%"tio s, locate +ails to work 4"st a+ter the system is i stalle!, %"t i+ yo" try agai the e3t !ay, it works +i e. What gi#esC The locate !ata%ase is create! %y a other program ame! u7datedb. )s"ally, it is r" perio!ically as a cron AobH that is, a task per+orme! at reg"lar i ter#als %y the cro !aemo . >ost systems e="ippe! with locate r" u7datedb o ce a !ay. Si ce the !ata%ase is ot "p!ate! co ti "o"sly, yo" will otice that #ery rece t +iles !o ot show "p whe "si g locate. To o#ercome this, itIs possi%le to r" the u7datedb program ma "ally %y %ecomi g the s"per"ser a ! r" i g u7datedb at the prompt.

find ;ind ;iles The 3ard +a,

While the locate program ca +i ! a +ile %ase! solely o its ame, the find program searches a gi#e !irectory Ea ! its s"%!irectoriesF +or +iles %ase! o a #ariety o+ attri%"tes. WeIre goi g to spe ! a lot o+ time with find %eca"se it has a lot o+ i teresti g +eat"res that we will see agai a ! agai whe we start to co#er programmi g co cepts i later chapters. 9 its simplest "se, find is gi#e o e or more ames o+ !irectories to search. 0or e3ample, to pro!"ce a list o+ o"r home !irectory,
[me@linuxbox ~]$ find ~

7 most acti#e "ser acco" ts, this will pro!"ce a large list. Si ce the list is se t to sta !ar! o"tp"t, we ca pipe the list i to other programs. *etIs "se 9c to co" t the "m%er o+ +iles,
[me@linuxbox ~]$ find ~ 0 wc -l %)(./

Wow, weI#e %ee %"syL The %ea"ty o+ find is that it ca %e "se! to i!e ti+y +iles that meet speci+ic criteria. 9t !oes this thro"gh the Eslightly stra geF applicatio o+ options, tests, a ! actions. WeIll look at the tests +irst.

2.2

.8 8 Searchi g 0or 0iles

Tests
*etIs say that we wa t a list o+ !irectories +rom o"r search. To !o this, we co"l! a!! the +ollowi g test,
[me@linuxbox ~]$ find ~ -t!pe d 0 wc -l #.0"

$!!i g the test -t17e d limite! the search to !irectories. Co #ersely, we co"l! ha#e limite! the search to reg"lar +iles with this test,
[me@linuxbox ~]$ find ~ -t!pe f 0 wc -l $/)$)

Aere are the commo +ile type tests s"pporte! %y +i !, Table 14-1: find File Types ;ile T,pe b c d f l Description Jlock special !e#ice +ile Character special !e#ice +ile 'irectory :eg"lar +ile Sym%olic li k

We ca also search %y +ile siDe a ! +ile ame %y a!!i g some a!!itio al tests, *etIs look +or all the reg"lar +iles that match the wil! car! patter ?U.J6;@ a ! are larger tha o e mega%yte,
[me@linuxbox ~]$ find ~ -t!pe f -name J*.YBQJ -si-e 51O 0 wc -l /%(

9 this e3ample, we a!! the -name test +ollowe! %y the wil! car! patter . &otice how we e close it i ="otes to pre#e t path ame e3pa sio %y the shell. &e3t, we a!! the -siCe test +ollowe! %y the stri g ?V.>@. The lea!i g pl"s sig i !icates that we are looki g +or +iles larger tha the speci+ie! "m%er. $ lea!i g mi "s sig wo"l! cha ge 2.K

+i ! 8 0i ! 0iles The Aar! Way the mea i g o+ the stri g to %e smaller tha the speci+ie! "m%er. &o sig mea s, ?match the #al"e e3actly.@ The traili g letter ?>@ i !icates that the " it o+ meas"reme t is mega%ytes. The +ollowi g characters may %e "se! to speci+y " its, Table 14-': find Si7e 0nits Character b c 9 k + @ -nit 2.2 %yte %locks. This is the !e+a"lt i+ o " it is speci+ie!. Jytes Two %yte wor!s Milo%ytes E) its o+ .021 %ytesF >ega%ytes E) its o+ .0182/K %ytesF ;iga%ytes E) its o+ .0/(/1.821 %ytesF

find s"pports a large "m%er o+ !i++ere t tests. Jelow is a r" !ow o+ the commo o es. &ote that i cases where a "meric arg"me t is re="ire!, the same ?V@ a ! ?-@ otatio !isc"sse! a%o#e ca %e applie!, Table 14-3: find Tests Test -cmin n Description >atch +iles or !irectories whose co te t or attri%"tes were last mo!i+ie! e3actly n mi "tes ago. To speci+y less tha n mi "tes ago, "se -n a ! to speci+y more tha n mi "tes ago, "se +n. >atch +iles or !irectories whose co te ts or attri%"tes were last mo!i+ie! more rece tly tha those o+ file. >atch +iles or !irectories whose co te ts or attri%"tes were last mo!i+ie! nU21 ho"rs ago. >atch empty +iles a ! !irectories. >atch +ile or !irectories %elo gi g to group. group may %e e3presse! as either a gro"p ame or as a "meric gro"p 9'. *ike the -name test %"t case i se siti#e. >atch +iles with i o!e "m%er n. This is help+"l +or +i !i g all the har! li ks to a partic"lar i o!e. 2./

-cne9er file -ctime n -em7t1 -grou7 name

-iname pattern -inum n

.8 8 Searchi g 0or 0iles -mmin n -mtime n -name pattern -ne9er file >atch +iles or !irectories whose co te ts were mo!i+ie! n mi "tes ago. >atch +iles or !irectories whose co te ts were mo!i+ie! nU21 ho"rs ago. >atch +iles a ! !irectories with the speci+ie! wil! car! pattern. >atch +iles a ! !irectories whose co te ts were mo!i+ie! more rece tly tha the speci+ie! file. This is #ery "se+"l whe writi g shell scripts that per+orm +ile %ack"ps. Each time yo" make a %ack"p, "p!ate a +ile Es"ch as a logF, the "se find to !etermi e which +iles that ha#e cha ge! si ce the last "p!ate. >atch +ile a ! !irectories that !o ot %elo g to a #ali! "ser. This ca %e "se! to +i ! +iles %elo gi g to !elete! acco" ts or to !etect acti#ity %y attackers. >atch +iles a ! !irectories that !o ot %elo g to a #ali! gro"p. >atch +iles or !irectories that ha#e permissio s set to the speci+ie! mode. mode may %e e3presse! %y either octal or sym%olic otatio . Similar to the -inum test. >atches +iles that share the same i o!e "m%er as +ile name. >atch +iles o+ siDe n. >atch +iles o+ type c. >atch +iles or !irectories %elo gi g to "ser name. The "ser may %e e3presse! %y a "ser ame or %y a "meric "ser 9'.

-nouser

-nogrou7 -7erm mode

-samefile name -siCe n -t17e c -user name

This is ot a complete list. The find ma page has all the !etails.

<5erators
E#e with all the tests that find pro#i!es, we may still ee! a %etter way to !escri%e the logical relationships %etwee the tests. 0or e3ample, what i+ we ee!e! to !etermi e i+ all the +iles a ! s"%!irectories i a !irectory ha! sec"re permissio sC We wo"l! look +or all the +iles with permissio s that are ot 0K00 a ! the !irectories with permissio s that 2.8

+i ! 8 0i ! 0iles The Aar! Way are ot 0/00. 0ort" ately, find pro#i!es a way to com%i e tests "si g logical operators to create more comple3 logical relatio ships. To e3press the a+oreme tio e! test, we co"l! !o this,
[me@linuxbox ~]$ find ~ K4 -t!pe f -not -perm *N** K6 -or K4 -t!pe d -not -perm *C** K6

5ikesL That s"re looks weir!. What is all this st"++C $ct"ally, the operators are ot that complicate! o ce yo" get to k ow them. Aere is the list, Table 14-$: find (ogical &perators ?perator -and Description >atch i+ the tests o %oth si!es o+ the operator are tr"e. >ay %e shorte e! to -a. &ote that whe o operator is prese t, -and is implie! %y !e+a"lt. >atch i+ a test o either si!e o+ the operator is tr"e. >ay %e shorte e! to -o. >atch i+ the test +ollowi g the operator is +alse. >ay %e a%%re#iate! with a e3clamatio poi t EGF. ;ro"ps tests a ! operators together to +orm larger e3pressio s. This is "se! to co trol the prece!e ce o+ the logical e#al"atio s. Jy !e+a"lt, find e#al"ates +rom le+t to right. 9t is o+te ecessary to o#erri!e the !e+a"lt e#al"atio or!er to o%tai the !esire! res"lt. E#e i+ ot ee!e!, it is help+"l sometimes to i cl"!e the gro"pi g characters to impro#e rea!a%ility o+ the comma !. &ote that si ce the pare theses characters ha#e special mea i g to the shell, they m"st %e ="ote! whe "si g them o the comma ! li e to allow them to %e passe! as arg"me ts to find. )s"ally the %ackslash character is "se! to escape them.

-or -not N P

With this list o+ operators i ha !, letIs !eco str"ct o"r find comma !. Whe #iewe! +rom the "ppermost le#el, we see that o"r tests are arra ge! as two gro"pi gs separate! %y a -or operator, N expression 1 P -or N expression 2 P

2.9

.8 8 Searchi g 0or 0iles This makes se se, si ce we are searchi g +or +iles with a certai set o+ permissio s a ! +or !irectories with a !i++ere t set. 9+ we are looki g +or %oth +iles a ! !irectories, why !o we "se -or i stea! o+ -andC Jeca"se as find sca s thro"gh the +iles a ! !irectories, each o e is e#al"ate! to see i+ it matches the speci+ie! tests. We wa t to k ow i+ it is either a +ile with %a! permissio s or a !irectory with %a! permissio s. 9t ca It %e %oth at the same time. So i+ we e3pa ! the gro"pe! e3pressio s, we ca see it this way, N file 9ith bad 7erms P -or N director1 9ith bad 7erms P 7"r e3t challe ge is how to test +or ?%a! permissio s.@ Aow !o we !o thatC $ct"ally we !o It. What we will test +or is ? ot goo! permissio s,@ si ce we k ow what ?goo! permissio s@ are. 9 the case o+ +iles, we !e+i e goo! as 0K00 a ! +or !irectories, as 0/... The e3pressio that will test +iles +or ? ot goo!@ permissio s is, -t17e f -and -not -7erms (.(( a ! +or !irectories, -t17e d -and -not -7erms ()(( $s ote! i the ta%le o+ operators a%o#e, the -and operator ca %e sa+ely remo#e!, si ce it is implie! %y !e+a"lt. So i+ we p"t this all %ack together, we get o"r +i al comma !, find ~ N -t17e f -not -7erms (.(( P -or N -t17e d -not -7erms ()(( P Aowe#er, si ce the pare theses ha#e special mea i g to the shell, we m"st escape them to pre#e t the shell +rom tryi g to i terpret them. 6rece!i g each o e with a %ackslash character !oes the trick. There is a other +eat"re o+ logical operators that is importa t to " !ersta !. *etIs say that we ha#e two e3pressio s separate! %y a logical operator, expr1 -o7erator expr2 9 all cases, expr1 will always %e per+orme!H howe#er the operator will !etermi e i+ expr2 is per+orme!. AereIs how it works, Table 14-.: find )>-;&9 (ogic 1esults o" expr1 Tr"e 0alse Tr"e 0alse ?perator -and -and -or -or expr2 is... $lways per+orme! &e#er per+orme! &e#er per+orme! $lways per+orme!

220

+i ! 8 0i ! 0iles The Aar! Way

Why !oes this happe C 9tIs !o e to impro#e per+orma ce. Take -and, +or e3ample. We k ow that the e3pressio expr1 -and expr2 ca ot %e tr"e i+ the res"lt o+ expr1 is +alse, so there is o poi t i per+ormi g expr2. *ikewise, i+ we ha#e the e3pressio expr1 -or expr2 a ! the res"lt o+ expr1 is tr"e, there is o poi t i per+ormi g expr2" as we alrea!y k ow that the e3pressio expr1 -or expr2 is tr"e. 7M, so it helps it go +aster. Why is this importa tC 9tIs importa t %eca"se we ca rely o this %eha#ior to co trol how actio s are per+orme!, as we shall soo see..

'rede=ined A/tions
*etIs get some work !o eL Aa#i g a list o+ res"lts +rom o"r find comma ! is "se+"l, %"t what we really wa t to !o is act o the items o the list. 0ort" ately, find allows actio s to %e per+orme! %ase! o the search res"lts. There are a set o+ pre!e+i e! actio s a ! se#eral ways to apply "ser-!e+i e! actio s. 0irst letIs look at a +ew o+ the pre!e+i e! actio s, Table 14-2: 6redefined find )ctions #ction -delete -ls -7rint Description 'elete the c"rre tly matchi g +ile. 6er+orm the e="i#ale t o+ ls -dils o the matchi g +ile. 7"tp"t is se t to sta !ar! o"tp"t. 7"tp"t the +"ll path ame o+ the matchi g +ile to sta !ar! o"tp"t. This is the !e+a"lt actio i+ o other actio is speci+ie!. T"it o ce a match has %ee ma!e.

-Buit

$s with the tests, there are ma y more actio s. See the find ma page +or +"ll !etails. 9 o"r #ery +irst e3ample, we !i! this,
find ~

which pro!"ce! a list o+ e#ery +ile a ! s"%!irectory co tai e! withi o"r home !irectory. 9t pro!"ce! a list %eca"se the -7rint actio is implie! i+ o other actio is speci+ie!. Th"s o"r comma ! co"l! also %e e3presse! as,

22.

.8 8 Searchi g 0or 0iles

find ~ -print

We ca "se find to !elete +iles that meet certai criteria. 0or e3ample, to !elete +iles that ha#e the +ile e3te sio ?.J$M@ Ewhich is o+te "se! to !esig ate %ack"p +ilesF, we co"l! "se this comma !,
find ~ -t!pe f -name %*.<:X% -delete

9 this e3ample, e#ery +ile i the "serIs home !irectory Ea ! its s"%!irectoriesF is searche! +or +ile ames e !i g i <H42. Whe they are +o" !, they are !elete!. 'arning5 9t sho"l! go witho"t sayi g that yo" sho"l! use extreme caution whe "si g the -delete actio . $lways test the comma ! +irst %y s"%stit"ti g the -7rint actio +or -delete to co +irm the search res"lts. Je+ore we go o , letIs take a other look at how the logical operators a++ect actio s. Co si!er the +ollowi g comma !,
find ~ -t!pe f -name %*.<:X% -print

$s we ha#e see , this comma ! will look +or e#ery reg"lar +ile E-t17e fF whose ame e !s with <H42 E-name %*<H42KF a ! will o"tp"t the relati#e path ame o+ each matchi g +ile to sta !ar! o"tp"t E-7rintF. Aowe#er, the reaso the comma ! per+orms the way it !oes is !etermi e! %y the logical relatio ships %etwee each o+ the tests a ! actio s. :emem%er, there is, %y !e+a"lt, a implie! -and relatio ship %etwee each test a ! actio . We co"l! also e3press the comma ! this way to make the logical relatio ships easier to see,
find ~ -t!pe f -and -name %*.<:X% -and -print

With o"r comma ! +"lly e3presse!, letIs look at how the logical operators a++ect its e3ec"tio , Test8#ction -7rint Is Per"ormed ?nl, I"... -t17e f a ! -name KE<H42K are tr"e

222

+i ! 8 0i ! 0iles The Aar! Way -name aE<H42b -t17e f -t17e f is tr"e 9s always per+orme!, si ce it is the +irst test-actio i a -and relatio ship.

Si ce the logical relatio ship %etwee the tests a ! actio s !etermi es which o+ them are per+orme!, we ca see that the or!er o+ the tests a ! actio s is importa t. 0or i sta ce, i+ we were to reor!er the tests a ! actio s so that the -7rint actio was the +irst o e, the comma ! wo"l! %eha#e m"ch !i++ere tly,
find ~ -print -and -t!pe f -and -name %*.<:X%

This #ersio o+ the comma ! will pri t each +ile Ethe -7rint actio always e#al"ates to tr"eF a ! the test +or +ile type a ! the speci+ie! +ile e3te sio .

User )e=ined A/tions

9 a!!itio to the pre!e+i e! actio s, we ca also i #oke ar%itrary comma !s. The tra!itio al way o+ !oi g this is with the -exec actio . This actio works like this, -exec command [\ T where command is the ame o+ a comma !, cd is a sym%olic represe tatio o+ the c"rre t path ame a ! the semicolo is a re="ire! !elimiter i !icati g the e ! o+ the comma !. AereIs a e3ample o+ "si g -exec to act like the -delete actio !isc"sse! earlier,
-exec rm K[\K KTK

$gai , si ce the %race a ! semicolo characters ha#e special mea i g to the shell, they m"st %e ="ote! or escape!. 9tIs also possi%le to e3ec"te a "ser !e+i e! actio i teracti#ely. Jy "si g the -ok actio i place o+ -exec, the "ser is prompte! %e+ore e3ec"tio o+ each speci+ie! comma !,
find ~ -t!pe f -name %foo*% -ok ls -l %9>% %#% X ls <<< 6home6me6bin6foo D F ! -r9xr-xr-x # me me !!% !(()-#(-!0 #/:%% 6home6me6bin6foo X ls <<< 6home6me6foo<txt D F ! -r9-r--r-- # me me ( !((/-(0-#0 #!:"$ 6home6me6foo<txt

22(

.8 8 Searchi g 0or 0iles 9 this e3ample, we search +or +iles with ames starti g with the stri g ?+oo@ a ! e3ec"te the comma ! ls -l each time o e is +o" !. )si g the -ok actio prompts the "ser %e+ore the ls comma ! is e3ec"te!.

Im5ro;ing 6==i/ien/y
Whe the -exec actio is "se!, it la" ches a ew i sta ce o+ the speci+ie! comma ! each time a matchi g +ile is +o" !. There are times whe we might pre+er to com%i e all o+ the search res"lts a ! la" ch a si gle i sta ce o+ the comma !. 0or e3ample, rather tha e3ec"ti g the comma !s like this, ls -l file1 ls -l file2 we may pre+er to e3ec"te it this way, ls -l file1 file2 th"s ca"si g the comma ! to %e e3ec"te! o ly o e time rather tha m"ltiple times. There are two ways we ca !o this. The tra!itio al way, "si g the e3ter al comma ! xargs a ! the alter ate way, "si g a ew +eat"re i find itsel+. WeIll talk a%o"t the alter ate way +irst. Jy cha gi g the traili g semicolo character to a pl"s sig , we acti#ate the a%ility o+ find to com%i e the res"lts o+ the search i to a arg"me t list +or a si gle e3ec"tio o+ the !esire! comma !. ;oi g %ack to o"r e3ample, this,
find ~ -t!pe f -name %foo*% -exec ls -l %9>% %#% -r9xr-xr-x # me me !!% !(()-#(-!0 #/:%% 6home6me6bin6foo -r9-r--r-- # me me ( !((/-(0-#0 #!:"$ 6home6me6foo<txt

will e3ec"te ls each time a matchi g +ile is +o" !. Jy cha gi g the comma ! to,
find ~ -t!pe f -name %foo*% -exec ls -l %9>% 5 -r9xr-xr-x # me me !!% !(()-#(-!0 #/:%% 6home6me6bin6foo -r9-r--r-- # me me ( !((/-(0-#0 #!:"$ 6home6me6foo<txt

we get the same res"lts, %"t the system o ly has to e3ec"te the ls comma ! o ce.

xargs
The xargs comma ! per+orms a i teresti g +" ctio . 9t accepts i p"t +rom sta !ar! 221

+i ! 8 0i ! 0iles The Aar! Way i p"t a ! co #erts it i to a arg"me t list +or a speci+ie! comma !. With o"r e3ample, we wo"l! "se it like this,
find ~ -t!pe f -name %foo*% -print 0 xargs ls -l -r9xr-xr-x # me me !!% !(()-#(-!0 #/:%% 6home6me6bin6foo -r9-r--r-- # me me ( !((/-(0-#0 #!:"$ 6home6me6foo<txt

Aere we see the o"tp"t o+ the find comma ! pipe! i to xargs which, i co str"cts a arg"me t list +or ls comma ! a ! the e3ec"tes it.

t"r ,

1ote5 While the "m%er o+ arg"me ts that ca %e place! i to a comma ! li e is ="ite large, itIs ot " limite!. 9t is possi%le to create comma !s that are too lo g +or the shell to accept. Whe a comma ! li e e3cee!s the ma3im"m le gth s"pporte! %y the system, xargs e3ec"tes the speci+ie! comma ! with the ma3im"m "m%er o+ arg"me ts possi%le a ! the repeats this process " til sta !ar! i p"t is e3ha"ste!. To see the ma3im"m siDe o+ the comma ! li e, e3ec"te xargs with the --sho9-limits optio .

Dealing +ith ;unn, ;ilenames

) i3-like systems allow em%e!!e! spaces Ea ! e#e ewli esLF i +ile ames. This ca"ses pro%lems +or programs like xargs that co str"ct arg"me t lists +or other programs. $ em%e!!e! space will %e treate! as a !elimiter a ! the res"lti g comma ! will i terpret each space-separate! wor! as a separate arg"me t. To o#ercome this, find a ! xarg allow the optio al "se o+ a null character as arg"me t separator. $ "ll character is !e+i e! i $SC99 as the character represe te! %y the "m%er Dero Eas oppose! to, +or e3ample, the space character, which is !e+i e! i $SC99 as the character represe te! %y the "m%er (2F. The find comma ! pro#i!es the actio -7rint(, which pro!"ces "ll separate! o"tp"t, a ! the xargs comma ! has the --null optio , which accepts "ll separate! i p"t. AereIs a e3ample, find ~ -iname KE<j7gK -7rint( Q xargs --null ls -l )si g this tech i="e, we ca e s"re that all +iles, e#e those co tai i g em%e!!e! spaces i their ames, are ha !le! correctly.

222

.8 8 Searchi g 0or 0iles

A !eturn To The ' ayground

9tIs time to p"t find to some EalmostF practical "se. WeIll create a playgro" ! a ! try o"t some o+ what we ha#e lear e!. 0irst, letIs create a playgro" ! with lots o+ s"%!irectories a ! +iles,
[me@linuxbox ~]$ mkdir -p pla!ground/dir-9**91..D>;*91*..DD>;1**> [me@linuxbox ~]$ touch pla!ground/dir-9**91..D>;*91*..DD>;1**>/file9:..@>

>ar#el i the power o+ the comma ! li eL With these two li es, we create! a playgro" ! !irectory co tai i g o e h" !re! s"%!irectories each co tai i g twe ty-si3 empty +iles. Try that with the ;)9L The metho! we employe! to accomplish this magic i #ol#e! a +amiliar comma ! EmkdirF, a e3otic shell e3pa sio E%racesF a ! a ew comma !, touch. Jy com%i i g mkdir with the -7 optio Ewhich ca"ses mkdir to create the pare t !irectories o+ the speci+ie! pathsF with %race e3pa sio , we were a%le to create o e h" !re! !irectories. The touch comma ! is "s"ally "se! to set or "p!ate the access, cha ge, a ! mo!i+y times o+ +iles. Aowe#er, i+ a +ile ame arg"me t is that o+ a o e3iste t +ile, a empty +ile is create!. 9 o"r playgro" !, we create! o e h" !re! i sta ces o+ a +ile ame! file-4. *etIs +i ! them,
[me@linuxbox ~]$ find pla!ground -t!pe f -name %file-:%

&ote that " like ls, find !oes ot pro!"ce res"lts i sorte! or!er. 9ts or!er is !etermi e! %y the layo"t o+ the storage !e#ice. To co +irm that we act"ally ha#e o e h" !re! i sta ces o+ the +ile we ca co +irm it this way,
[me@linuxbox ~]$ find pla!ground -t!pe f -name %file-:% 0 wc -l #((

&e3t, letIs look at +i !i g +iles %ase! o their mo!i+icatio times. This will %e help+"l whe creati g %ack"ps or orga iDi g +iles i chro ological or!er. To !o this, we will +irst create a re+ere ce +ile agai st which we will compare mo!i+icatio time,

22K

+i ! 8 0i ! 0iles The Aar! Way

[me@linuxbox ~]$ touch pla!ground/timestamp

This creates a empty +ile ame! timestam7 a ! sets its mo!i+icatio time to the c"rre t time. We ca #eri+y this %y "si g a other ha !y comma !, stat, which is a ki ! o+ so"pe!-"p #ersio o+ ls. The stat comma ! re#eals all that the system " !ersta !s a%o"t a +ile a ! its attri%"tes,
[me@linuxbox ~]$ stat pla!ground/timestamp -ile: J7la1ground6timestam7K *iCe: ( Hlocks: ( A Hlock: %(0. regular em7t1 file 'evice: /($h6!("#d Anode: #%!."(.# =inks: # 4ccess: N(.%%6-r9-r--r--P 3id: N #((#6 meP @id: N #((#6 meP 4ccess: !((/-#(-(/ #":#":$0<((((((((( -(%(( +odif1: !((/-#(-(/ #":#":$0<((((((((( -(%(( Change: !((/-#(-(/ #":#":$0<((((((((( -(%((

9+ we touch the +ile agai a ! the e3ami e it with stat, we will see that the +ileIs times ha#e %ee "p!ate!.,
[me@linuxbox ~]$ touch pla!ground/timestamp [me@linuxbox ~]$ stat pla!ground/timestamp -ile: J7la1ground6timestam7K *iCe: ( Hlocks: ( A Hlock: %(0. regular em7t1 file 'evice: /($h6!("#d Anode: #%!."(.# =inks: # 4ccess: N(.%%6-r9-r--r--P 3id: N #((#6 meP @id: N #((#6 meP 4ccess: !((/-#(-(/ #":!$:$$<((((((((( -(%(( +odif1: !((/-#(-(/ #":!$:$$<((((((((( -(%(( Change: !((/-#(-(/ #":!$:$$<((((((((( -(%((

&e3t, letIs "se find to "p!ate some o+ o"r playgro" ! +iles,

[me@linuxbox ~]$ find pla!ground -t!pe f -name %file-<% -exec touch %9>% %#%

This "p!ates all +iles i the playgro" ! ame! file-H. &e3t weIll "se find to i!e ti+y the "p!ate! +iles %y compari g all the +iles to the re+ere ce +ile timestam7,
[me@linuxbox ~]$ find pla!ground -t!pe f -newer pla!ground/timestamp

22/

.8 8 Searchi g 0or 0iles The res"lts co tai all o e h" !re! i sta ces o+ file-H. Si ce we per+orme! a touch o all the +iles i the playgro" ! ame! file-H a+ter we "p!ate! timestam7, they are ow ? ewer@ tha timestam7 a ! th"s ca %e i!e ti+ie! with the -ne9er test. 0i ally, letIs go %ack to the %a! permissio s test we per+orme! earlier a ! apply it to 7la1ground,
[me@linuxbox ~]$ find pla!ground K4 -t!pe f -not -perm *N** K6 -or K4 -t!pe d -not -perm *C** K6

This comma ! lists all o e h" !re! !irectories a ! twe ty-si3 h" !re! +iles i 7la1ground Eas well as timestam7 a ! 7la1ground itsel+, +or a total o+ 2/02F %eca"se o e o+ them meets o"r !e+i itio o+ ?goo! permissio s.@ With o"r k owle!ge o+ operators a ! actio s, we ca a!! actio s to this comma ! to apply ew permissio s to the +iles a ! !irectories i o"r playgro" !,
[me@linuxbox ~]$ find pla!ground K4 -t!pe f -not -perm *N** -exec chmod *N** %9>% %#% K6 -or K4 -t!pe d -not -perm *C11 -exec chmod *C** %9>% %#% K6

7 a !ay-to-!ay %asis, we might +i ! it easier to iss"e two comma !s, o e +or the !irectories a ! o e +or the +iles, rather tha this o e large compo" ! comma !, %"t itIs ice to k ow that we ca !o it this way. The importa t poi t here is to " !ersta ! how the operators a ! actio s ca %e "se! together to per+orm "se+"l tasks.

<5tions
0i ally, we ha#e the optio s. The optio s are "se! to co trol the scope o+ a find search. They may %e i cl"!e! with other tests a ! actio s whe co str"cti g find e3pressio s. Aere is a list o+ the most commo ly "se! o es, Table 14-3: find &ptions ?ption -de7th Description 'irect find to process a !irectoryIs +iles %e+ore the !irectory itsel+. This optio is a"tomatically applie! whe the -delete actio is speci+ie!. Set the ma3im"m "m%er o+ le#els that find will !esce ! i to a !irectory tree whe per+ormi g tests a !

-maxde7th levels

228

+i ! 8 0i ! 0iles The Aar! Way actio s. -minde7th levels Set the mi im"m "m%er o+ le#els that find will !esce ! i to a !irectory tree %e+ore applyi g tests a ! actio s. 'irect find ot to tra#erse !irectories that are mo" te! o other +ile systems. 'irect find ot to optimiDe its search %ase! o the ass"mptio that it is searchi g a ) i3-like +ile system. This is ee!e! whe sca i g '7S-Wi !ows +ile systems a ! C'-:7>s.

-mount -noleaf

;urther 1eading

The locate, u7datedb, find, a ! xargs programs are all part the ;&) 6ro4ectIs findutils package. The ;&) 6ro4ect pro#i!es a we%site with e3te si#e o -li e !oc"me tatio , which is ="ite goo! a ! sho"l! %e rea! i+ yo" are "si g these programs i high sec"rity e #iro me ts, http,--www.g ".org-so+tware-+i !"tils-

229

.9 8 $rchi#i g $ ! Jack"p

10 )rchiving )nd 6ac,u#

7 e o+ the primary tasks o+ a comp"ter systemIs a!mi istrator is keepi g the systemIs !ata sec"re. 7 e way this is !o e is %y per+ormi g timely %ack"ps o+ the systemIs +iles. E#e i+ yo"Ire ot system a!mi istrators, it is o+te "se+"l to make copies o+ thi gs a ! to mo#e large collectio s o+ +iles +rom place to place a ! +rom !e#ice to !e#ice. 9 this chapter, we will look at se#eral commo programs that are "se! to ma age collectio s o+ +iles. There are the +ile compressio programs,

gCi7 8 Compress or e3pa ! +iles bCi7! 8 $ %lock sorti g +ile compressor tar 8 Tape archi#i g "tility Ci7 8 6ackage a ! compress +iles rs1nc 8 :emote +ile a ! !irectory sy chro iDatio

The archi#i g programs,

$ ! the +ile sy chro iDatio program,

Compressing ;iles
Thro"gho"t the history o+ comp"ti g, there has %ee a str"ggle to get the most !ata i to the smallest a#aila%le space, whether that space %e memory, storage !e#ices or etwork %a !wi!th. >a y o+ the !ata ser#ices that we take +or gra te! to!ay, s"ch as porta%le m"sic players, high !e+i itio tele#isio , or %roa!%a ! 9 ter et, owe their e3iste ce to e++ecti#e data compression tech i="es. 'ata compressio is the process o+ remo#i g redundancy +rom !ata. *etIs co si!er a imagi ary e3ample. Say we ha! a e tirely %lack pict"re +ile with the !ime sio s o+ o e h" !re! pi3els %y o e h" !re! pi3els. 9 terms o+ !ata storage Eass"mi g twe ty-+o"r %its, or three %ytes per pi3elF, the image will occ"py thirty tho"sa ! %ytes o+ storage, .00 U .00 U ( W (0,000 $ image that is all o e color co tai s e tirely re!" !a t !ata. 9+ we were cle#er, we co"l! e co!e the !ata i s"ch a way that we simply !escri%e the +act that we ha#e a %lock 2(0

Compressi g 0iles o+ thirty tho"sa ! %lack pi3els. So, i stea! o+ stori g a %lock o+ !ata co tai i g thirty tho"sa ! Deros E%lack is "s"ally represe te! i image +iles as DeroF, we co"l! compress the !ata i to the "m%er (0,000, +ollowe! %y a Dero to represe t o"r !ata. S"ch a !ata compressio scheme is calle! run-length encoding a ! is o e o+ the most r"!ime tary compressio tech i="es. To!ayIs tech i="es are m"ch more a!#a ce! a ! comple3 %"t the %asic goal remai s the sameBget ri! o+ re!" !a t !ata. %ompression algorithms Ethe mathematical tech i="es "se! to carry o"t the compressio F +all i to two ge eral categories, lossless a ! lossy. *ossless compressio preser#es all the !ata co tai e! i the origi al. This mea s that whe a +ile is restore! +rom a compresse! #ersio , the restore! +ile is e3actly the same as the origi al, " compresse! #ersio . *ossy compressio , o the other ha !, remo#es !ata as the compressio is per+orme!, to allow more compressio to %e applie!. Whe a lossy +ile is restore!, it !oes ot match the origi al #ersio H rather, it is a close appro3imatio . E3amples o+ lossy compressio are J6E; E+or imagesF a ! >6( E+or m"sic.F 9 o"r !isc"ssio , we will look e3cl"si#ely at lossless compressio , si ce most !ata o comp"ters ca ot tolerate a y !ata loss.

gHi5
The gCi7 program is "se! to compress o e or more +iles. Whe e3ec"te!, it replaces the origi al +ile with a compresse! #ersio o+ the origi al. The correspo !i g gunCi7 program is "se! to restore compresse! +iles to their origi al, " compresse! +orm. Aere is a e3ample,
[me@linuxbox [me@linuxbox -r9-r--r-- # [me@linuxbox [me@linuxbox -r9-r--r-- # [me@linuxbox [me@linuxbox -r9-r--r-- # ~]$ ~]$ me ~]$ ~]$ me ~]$ ~]$ me ls -l /etc & foo.txt ls -l foo.* me #")$/ !((/-#(-#% ():#" foo<txt g-ip foo.txt ls -l foo.* me $!$( !((/-#(-#% ():#" foo<txt<gC gun-ip foo.txt ls -l foo.* me #")$/ !((/-#(-#% ():#" foo<txt

9 this e3ample, we create a te3t +ile ame! foo<txt +rom a !irectory listi g. &e3t, we r" gCi7, which replaces the origi al +ile with a compresse! #ersio ame! foo<txt<gC. 9 the !irectory listi g o+ +oo.U, we see that the origi al +ile has %ee replace! with the compresse! #ersio , a ! that the compresse! #ersio a%o"t o e-+i+th the siDe o+ the origi al. We ca also see that the compresse! +ile has the same permissio s a ! time stamp as the origi al. &e3t, we r" the gunCi7 program to " compress the +ile. $+terwar!, we ca see that 2(.

.9 8 $rchi#i g $ ! Jack"p the compresse! #ersio o+ the +ile has %ee replace! with the origi al, agai with the permissio s a ! time stamp preser#e!. gCi7 has ma y optio s. Aere are a +ew, Table 15-1: g7ip &ptions ?ption -c -d -f -h -l -r Description Write o"tp"t to sta !ar! o"tp"t a ! keep origi al +iles. >ay also %e speci+ie! with --stdout a ! --to-stdout. 'ecompress. This ca"ses gCi7 to act like gunCi7. >ay also %e speci+ie! with --decom7ress or --uncom7ress. 0orce compressio e#e i+ compresse! #ersio o+ the origi al +ile alrea!y e3ists. >ay also %e speci+ie! with --force. 'isplay "sage i +ormatio . >ay also %e speci+ie! with --hel7. *ist compressio statistics +or each +ile compresse!. >ay also %e speci+ie! with --list. 9+ o e or more arg"me ts o the comma ! li e are !irectories, rec"rsi#ely compress +iles co tai e! withi them. >ay also %e speci+ie! with --recursive. Test the i tegrity o+ a compresse! +ile. >ay also %e speci+ie! with --test. 'isplay #er%ose messages while compressi g. >ay also %e speci+ie! with --verbose. Set amo" t o+ compressio . number is a i teger i the ra ge o+ . E+astest, least compressio F to 9 Eslowest, most compressio F. The #al"es . a ! 9 may also %e e3presse! as --fast a ! --best, respecti#ely. The !e+a"lt #al"e is K.

-t -v -number

;oi g %ack to o"r earlier e3ample,

[me@linuxbox ~]$ g-ip foo.txt [me@linuxbox ~]$ g-ip -tv foo.txt.gfoo<txt<gC: 2 [me@linuxbox ~]$ g-ip -d foo.txt.g-

Aere, we replace! the +ile foo<txt with a compresse! #ersio , ame! foo<txt<gC. 2(2

Compressi g 0iles &e3t, we teste! the i tegrity o+ the compresse! #ersio , "si g the -t a ! -v optio s. 0i ally, we !ecompresse! the +ile %ack to its origi al +orm. gCi7 ca also %e "se! i i teresti g ways #ia sta !ar! i p"t a ! o"tp"t,
[me@linuxbox ~]$ ls -l /etc 0 g-ip & foo.txt.g-

This comma ! creates a compresse! #ersio o+ a !irectory listi g. The gunCi7 program, which " compresses gDip +iles, ass"mes that +ile ames e ! i the e3te sio <gC, so itIs ot ecessary to speci+y it, as lo g as the speci+ie! ame is ot i co +lict with a e3isti g " compresse! +ile,
[me@linuxbox ~]$ gun-ip foo.txt

9+ o"r goal were o ly to #iew the co te ts o+ a compresse! te3t +ile, we ca !o this,

[me@linuxbox ~]$ gun-ip -c foo.txt 0 less

$lter ately, there is a program s"pplie! with gCi7, calle! Ccat, that is e="i#ale t to gunCi7 with the -c optio . 9t ca %e "se! like the cat comma ! o gCi7 compresse! +iles,
[me@linuxbox ~]$ -cat foo.txt.g- 0 less

Ti(5 There is a Cless program, too. 9t per+orms the same +" ctio as the pipeli e a%o#e.

bHi52
The bCi7! program, %y J"lia Sewar!, is similar to gDip, %"t "ses a !i++ere t compressio algorithm that achie#es higher le#els o+ compressio at the cost o+ compressio spee!. 9 most regar!s, it works i the same +ashio as gCi7. $ +ile compresse! with bCi7! is !e ote! with the e3te sio <bC!,

2((

.9 8 $rchi#i g $ ! Jack"p
[me@linuxbox [me@linuxbox -r9-r--r-- # [me@linuxbox [me@linuxbox -r9-r--r-- # [me@linuxbox ~]$ ~]$ me ~]$ ~]$ me ~]$ ls -l /etc & foo.txt ls -l foo.txt me #")$/ !((/-#(-#) #$:"# foo<txt b-ip2 foo.txt ls -l foo.txt.b-2 me !)0! !((/-#(-#) #$:"# foo<txt<bC! bun-ip2 foo.txt.b-2

$s we ca see, bCi7! ca %e "se! the same way as gCi7. $ll the optio s Ee3cept +or rF that we !isc"sse! +or gCi7 are also s"pporte! i bCi7!. &ote, howe#er, that the compressio le#el optio E-numberF has a somewhat !i++ere t mea i g to bCi7!. bCi7! comes with bunCi7! a ! bCcat +or !ecompressi g +iles. bCi7! also comes with the bCi7!recover program, which will try to reco#er !amage! <bC! +iles.

DonFt /e Compressi%e Compulsi%e

9 occasio ally see people attempti g to compress a +ile, which has %ee alrea!y compresse! with a e++ecti#e compressio algorithm, %y !oi g somethi g like this, $ g-ip picture.jpg 'o It !o it. 5o"Ire pro%a%ly 4"st wasti g time a ! spaceL 9+ yo" apply compressio to a +ile that is alrea!y compresse!, yo" will act"ally e ! "p a larger +ile. This is %eca"se all compressio tech i="es i #ol#e some o#erhea! that is a!!e! to the +ile to !escri%e the compressio . 9+ yo" try to compress a +ile that alrea!y co tai s o re!" !a t i +ormatio , the compressio will ot res"lt i a y sa#i gs to o++set the a!!itio al o#erhea!.

#rchi%ing ;iles
$ commo +ile ma ageme t task "se! i co 4" ctio with compressio is archiving. $rchi#i g is the process o+ gatheri g "p ma y +iles a ! %" !li g them together i to a si gle large +ile. $rchi#i g is o+te !o e as a part o+ system %ack"ps. 9t is also "se! whe ol! !ata is mo#e! +rom a system to some type o+ lo g-term storage.

tar
9 the ) i3-like worl! o+ so+tware, the tar program is the classic tool +or archi#i g +iles. 2(1

$rchi#i g 0iles 9ts ame, short +or tape archive, re#eals its roots as a tool +or maki g %ack"p tapes. While it is still "se! +or that tra!itio al task, it is e="ally a!ept o other storage !e#ices as well. We o+te see +ile ames that e ! with the e3te sio <tar or <tgC which i !icate a ?plai @ tar archi#e a ! a gDippe! archi#e, respecti#ely. $ tar archi#e ca co sist o+ a gro"p o+ separate +iles, o e or more !irectory hierarchies, or a mi3t"re o+ %oth. The comma ! sy ta3 works like this, tar mode[options] pathname<<< where mode is o e o+ the +ollowi g operati g mo!es Eo ly a partial list is show hereH see the tar ma page +or a complete listF, Table 15-': tar !odes >ode c x r t Description Create a archi#e +rom a list o+ +iles a !-or !irectories. E3tract a archi#e. $ppe ! speci+ie! path ames to the e ! o+ a archi#e. *ist the co te ts o+ a archi#e.

tar "ses a slightly o!! way o+ e3pressi g optio s, so weIll ee! some e3amples to show how it works. 0irst, letIs re-create o"r playgro" ! +rom the pre#io"s chapter,
[me@linuxbox ~]$ mkdir -p pla!ground/dir-9**91..D>;*91*..DD>;1**> [me@linuxbox ~]$ touch pla!ground/dir-9**91..D>;*91*..DD>;1**>/file9:-@>

&e3t, letIs create a tar archi#e o+ the e tire playgro" !,

[me@linuxbox ~]$ tar cf pla!ground.tar pla!ground

This comma ! creates a tar archi#e ame! 7la1ground<tar that co tai s the e tire playgro" ! !irectory hierarchy. We ca see that the mo!e a ! the f optio , which is "se! to speci+y the ame o+ the tar archi#e, may %e 4oi e! together, a ! !o ot re="ire a lea!i g !ash. &ote, howe#er, that the mo!e m"st always %e speci+ie! +irst, %e+ore a y other optio . To list the co te ts o+ the archi#e, we ca !o this,

2(2

.9 8 $rchi#i g $ ! Jack"p
[me@linuxbox ~]$ tar tf pla!ground.tar

0or a more !etaile! listi g, we ca a!! the v E#er%oseF optio ,

[me@linuxbox ~]$ tar tvf pla!ground.tar

&ow, letIs e3tract the playgro" ! i a ew locatio . We will !o this %y creati g a ew !irectory ame! foo, a ! cha gi g the !irectory a ! e3tracti g the tar archi#e,
[me@linuxbox [me@linuxbox [me@linuxbox [me@linuxbox 7la1ground ~]$ mkdir foo ~]$ cd foo foo]$ tar xf ../pla!ground.tar foo]$ ls

9+ we e3ami e the co te ts o+ ~6foo67la1ground, we see that the archi#e was s"ccess+"lly i stalle!, creati g a precise repro!"ctio o+ the origi al +iles. There is o e ca#eat, howe#er, " less yo" are operati g as the s"per"ser, +iles a ! !irectories e3tracte! +rom archi#es take o the ow ership o+ the "ser per+ormi g the restoratio , rather tha the origi al ow er. $ other i teresti g %eha#ior o+ tar is the way it ha !les path ames i archi#es. The !e+a"lt +or path ames is relati#e, rather tha a%sol"te. tar !oes this %y simply remo#i g a y lea!i g slash +rom the path ame whe creati g the archi#e. To !emo strate, we will recreate o"r archi#e, this time speci+yi g a a%sol"te path ame,
[me@linuxbox foo]$ cd [me@linuxbox ~]$ tar cf pla!ground2.tar ~/pla!ground

:emem%er, ~67la1ground will e3pa ! i to 6home6me67la1ground whe we press the e ter key, so we will get a a%sol"te path ame +or o"r !emo stratio . &e3t, we will e3tract the archi#e as %e+ore a ! watch what happe s,
[me@linuxbox ~]$ cd foo [me@linuxbox foo]$ tar xf ../pla!ground2.tar [me@linuxbox foo]$ ls home 7la1ground [me@linuxbox foo]$ ls home

2(K

$rchi#i g 0iles
me [me@linuxbox foo]$ ls home/me 7la1ground

Aere we ca see that whe we e3tracte! o"r seco ! archi#e, it recreate! the !irectory home6me67la1ground relati#e to o"r c"rre t worki g !irectory, ~6foo, ot relati#e to the root !irectory, as wo"l! ha#e %ee the case with a a%sol"te path ame. This may seem like a o!! way +or it to work, %"t itIs act"ally more "se+"l this way, as it allows "s to e3tract archi#es to a y locatio rather tha %ei g +orce! to e3tract them to their origi al locatio s. :epeati g the e3ercise with the i cl"sio o+ the #er%ose optio EvF will gi#e a clearer pict"re o+ whatIs goi g o . *etIs co si!er a hypothetical, yet practical e3ample, o+ tar i actio . 9magi e we wa t to copy the home !irectory a ! its co te ts +rom o e system to a other a ! we ha#e a large )SJ har! !ri#e that we ca "se +or the tra s+er. 7 o"r mo!er *i "3 system, the !ri#e is ?a"tomagically@ mo" te! i the 6media !irectory. *etIs also imagi e that the !isk has a #ol"me ame o+ Hig'isk whe we attach it. To make the tar archi#e, we ca !o the +ollowi g,
[me@linuxbox ~]$ sudo tar cf /media/<ig1isk/home.tar /home

$+ter the tar +ile is writte , we " mo" t the !ri#e a ! attach it to the seco ! comp"ter. $gai , it is mo" te! at 6media6Hig'isk. To e3tract the archi#e, we !o this,
[me@linuxbox! ~]$ cd / [me@linuxbox! 6]$ sudo tar xf /media/<ig1isk/home.tar

WhatIs importa t to see here is that we m"st +irst cha ge !irectory to 6, so that the e3tractio is relati#e to the root !irectory, si ce all path ames withi the archi#e are relati#e. Whe e3tracti g a archi#e, itIs possi%le to limit what is e3tracte! +rom the archi#e. 0or e3ample, i+ we wa te! to e3tract a si gle +ile +rom a archi#e, it co"l! %e !o e like this,
tar xf archive.tar pathname

Jy a!!i g the traili g pathname to the comma !, tar will o ly restore the speci+ie! +ile. >"ltiple path ames may %e speci+ie!. &ote that the path ame m"st %e the +"ll, e3act 2(/

.9 8 $rchi#i g $ ! Jack"p relati#e path ame as store! i the archi#e. Whe speci+yi g path ames, wil!car!s are ot ormally s"pporte!H howe#er, the ;&) #ersio o+ tar Ewhich is the #ersio most o+te +o" ! i *i "3 !istri%"tio sF s"pports them with the --9ildcards optio . Aere is a e3ample "si g o"r pre#io"s 7la1ground<tar +ile,
[me@linuxbox ~]$ cd foo [me@linuxbox foo]$ tar xf ../pla!ground2.tar --wildcards %home/me/pla !ground/dir-*/file-:%

This comma ! will e3tract o ly +iles matchi g the speci+ie! path ame i cl"!i g the wil!car! dir-E. tar is o+te "se! i co 4" ctio with find to pro!"ce archi#es. 9 this e3ample, we will "se find to pro!"ce a set o+ +iles to i cl"!e i a archi#e,
[me@linuxbox ~]$ find pla!ground -name %file-:% -exec tar rf pla!ground.tar %9>% %5%

Aere we "se find to match all the +iles i 7la1ground ame! file-4 a ! the , "si g the -exec actio , we i #oke tar i the appe ! mo!e ErF to a!! the matchi g +iles to the archi#e 7la1ground<tar. )si g tar with find is a goo! way o+ creati g incremental backups o+ a !irectory tree or a e tire system. Jy "si g find to match +iles ewer tha a timestamp +ile, we co"l! create a archi#e that o ly co tai s +iles ewer tha the last archi#e, ass"mi g that the timestamp +ile is "p!ate! right a+ter each archi#e is create!. tar ca also make "se o+ %oth sta !ar! i p"t a ! o"tp"t. Aere is a comprehe si#e e3ample,
[me@linuxbox foo]$ cd [me@linuxbox ~]$ find pla!ground -name %file-:% 0 tar cf - --filesfrom$- 0 g-ip & pla!ground.tg-

9 this e3ample, we "se! the find program to pro!"ce a list o+ matchi g +iles a ! pipe! them i to tar. 9+ the +ile ame ?-@ is speci+ie!, it is take to mea sta !ar! i p"t or o"tp"t, as ee!e! E%y the way, this co #e tio o+ "si g ?-@ to represe t sta !ar! i p"t-o"tp"t is "se! %y a "m%er o+ other programs, too.F The --files-from optio Ewhich may %e also %e speci+ie! as -TF ca"ses tar to rea! its list o+ path ames +rom a 2(8

$rchi#i g 0iles +ile rather tha the comma ! li e. *astly, the archi#e pro!"ce! %y tar is pipe! i to gCi7 to create the compresse! archi#e 7la1ground<tgC. The <tgC e3te sio is the co #e tio al e3te sio gi#e to gDip-compresse! tar +iles. The e3te sio <tar<gC is also "se! sometimes. While we "se! the gCi7 program e3ter ally to pro!"ce! o"r compresse! archi#e, mo!er #ersio s o+ ;&) tar s"pport %oth gDip a ! %Dip2 compressio !irectly, with the "se o+ the C a ! j optio s, respecti#ely. )si g o"r pre#io"s e3ample as a %ase, we ca simpli+y it this way,
[me@linuxbox ~]$ find pla!ground -name %file-:% 0 tar c-f pla!ground.tg- -+ -

9+ we ha! wa te! to create a %Dip2 compresse! archi#e i stea!, we co"l! ha#e !o e this,
[me@linuxbox ~]$ find pla!ground -name %file-:% 0 tar cjf pla!ground.tb- -+ -

Jy simply cha gi g the compressio optio +rom C to j Ea ! cha gi g the o"tp"t +ileIs e3te sio to <tbC to i !icate a %Dip2 compresse! +ileF we e a%le! %Dip2 compressio . $ other i teresti g "se o+ sta !ar! i p"t a ! o"tp"t with the tar comma ! i #ol#es tra s+erri g +iles %etwee systems o#er a etwork. 9magi e that we ha! two machi es r" i g a ) i3-like system e="ippe! with tar a ! ssh. 9 s"ch a sce ario, we co"l! tra s+er a !irectory +rom a remote system E ame! remote-s1s +or this e3ampleF to o"r local system,
[me@linuxbox ~]$ mkdir remote-stuff [me@linuxbox ~]$ cd remote-stuff [me@linuxbox remote-stuff]$ ssh remote-s!s %tar cf - 1ocuments% 0 tar xf me@remote-s1sbs 7ass9ord: [me@linuxbox remote-stuff]$ ls 'ocuments

Aere we were a%le to copy a !irectory ame! 'ocuments +rom the remote system remote-s1s to a !irectory withi the !irectory ame! remote-stuff o the local system. Aow !i! we !o thisC 0irst, we la" che! the tar program o the remote system "si g ssh. 5o" will recall that ssh allows "s to e3ec"te a program remotely o a etworke! comp"ter a ! ?see@ the res"lts o the local systemBthe sta !ar! o"tp"t 2(9

.9 8 $rchi#i g $ ! Jack"p pro!"ce! o the remote system is se t to the local system +or #iewi g. We ca take a!#a tage o+ this %y ha#i g tar create a archi#e Ethe c mo!eF a ! se ! it to sta !ar! o"tp"t, rather tha a +ile Ethe f optio with the !ash arg"me tF, there%y tra sporti g the archi#e o#er the e crypte! t" el pro#i!e! %y ssh to the local system. 7 the local system, we e3ec"te tar a ! ha#e it e3pa ! a archi#e Ethe x mo!eF s"pplie! +rom sta !ar! i p"t Eagai , the f optio with the !ash arg"me tF.

Hi5
The Ci7 program is %oth a compressio tool a ! a archi#er. The +ile +ormat "se! %y the program is +amiliar to Wi !ows "sers, as it rea!s a ! writes <Ci7 +iles. 9 *i "3, howe#er, gCi7 is the pre!omi a t compressio program with bCi7! %ei g a close seco !. 9 its most %asic "sage, Ci7 is i #oke! like this, Ci7 options zipfile file... 0or e3ample, to make a Dip archi#e o+ o"r playgro" !, we wo"l! !o this,
[me@linuxbox ~]$ -ip -r pla!ground.-ip pla!ground

) less we i cl"!e the -r optio +or rec"rsio , o ly the 7la1ground !irectory E%"t o e o+ its co te tsF is store!. $ltho"gh the a!!itio o+ the e3te sio <Ci7 is a"tomatic a, we will i cl"!e the +ile e3te sio +or clarity. '"ri g the creatio o+ the Dip archi#e, Ci7 will ormally !isplay a series o+ messages like this,
adding: adding: adding: adding: adding: 7la1ground6dir-(!(6file-I Nstored 7la1ground6dir-(!(6file-U Nstored 7la1ground6dir-(!(6file-V Nstored 7la1ground6dir-(/)6 Nstored (5P 7la1ground6dir-(/)6file-* Nstored (5P (5P (5P (5P

These messages show the stat"s o+ each +ile a!!e! to the archi#e. Ci7 will a!! +iles to the archi#e "si g o e o+ two storage metho!s, either it will ?store@ a +ile witho"t compressio , as show here, or it will ?!e+late@ the +ile which per+orms compressio . The "meric #al"e !isplaye! a+ter the storage metho! i !icates the amo" t o+ compressio achie#e!. Si ce o"r playgro" ! o ly co tai s empty +iles, o compressio is per+orme! o its co te ts.

210

$rchi#i g 0iles E3tracti g the co te ts o+ a Dip +ile is straight+orwar! whe "si g the unCi7 program,
[me@linuxbox ~]$ cd foo [me@linuxbox foo]$ un-ip ../pla!ground.-ip

7 e thi g to ote a%o"t Ci7 Eas oppose! to tarF is that i+ a e3isti g archi#e is speci+ie!, it is "p!ate! rather tha replace!. This mea s that the e3isti g archi#e is preser#e!, %"t ew +iles are a!!e! a ! matchi g +iles are replace!. 0iles may %e liste! a ! e3tracte! selecti#ely +rom a Dip archi#e %y speci+yi g them to unCi7,
[me@linuxbox ~]$ un-ip -l pla!ground.-ip pla!ground/dir-MC/file-@ 4rchive: <<67la1ground<Ci7 =ength 'ate Time Lame ----------------( #(-("-(/ (0:!" 7la1ground6dir-/)6file-I -------------( # file [me@linuxbox ~]$ cd foo [me@linuxbox foo]$ un-ip ./pla!ground.-ip pla!ground/dir-MC/file-@ 4rchive: <<67la1ground<Ci7 re7lace 7la1ground6dir-/)6file-IF [1]es> [n]o> [4]ll> [L]one> [r]ename: ! extracting: 7la1ground6dir-/)6file-I

)si g the -l optio ca"ses unCi7 to merely list the co te ts o+ the archi#e witho"t e3tracti g the +ile. 9+ o +ileEsF are speci+ie!, unCi7 will list all +iles i the archi#e. The -v optio ca %e a!!e! to i crease the #er%osity o+ the listi g. &ote that whe the archi#e e3tractio co +licts with a e3isti g +ile, the "ser is prompte! %e+ore the +ile is replace!. *ike tar, Ci7 ca make "se o+ sta !ar! i p"t a ! o"tp"t, tho"gh its impleme tatio is somewhat less "se+"l. 9t is possi%le to pipe a list o+ +ile ames to Ci7 #ia the -@ optio ,
[me@linuxbox foo]$ cd [me@linuxbox ~]$ find pla!ground -name Jfile-:J 0 -ip -V file-:.-ip

Aere we "se find to ge erate a list o+ +iles matchi g the test -name _file-4_, a ! pipe the list i to Ci7, which creates the archi#e file-4<Ci7 co tai i g the selecte! +iles. 21.

.9 8 $rchi#i g $ ! Jack"p Ci7 also s"pports writi g its o"tp"t to sta !ar! o"tp"t, %"t its "se is limite! %eca"se #ery +ew programs ca make "se o+ the o"tp"t. ) +ort" ately, the unCi7 program, !oes ot accept sta !ar! i p"t. This pre#e ts Ci7 a ! unCi7 +rom %ei g "se! together to per+orm etwork +ile copyi g like tar. Ci7 ca , howe#er, accept sta !ar! i p"t, so it ca %e "se! to compress the o"tp"t o+ other programs,
[me@linuxbox ~]$ ls -l /etc/ 0 -ip ls-etc.-ip adding: - Ndeflated /(5P

9 this e3ample we pipe the o"tp"t o+ ls i to Ci7. *ike tar, Ci7 i terprets the traili g !ash as ?"se sta !ar! i p"t +or the i p"t +ile.@ The unCi7 program allows its o"tp"t to %e se t to sta !ar! o"tp"t whe the -7 E+or pipeF optio is speci+ie!,
[me@linuxbox ~]$ un-ip -p ls-etc.-ip 0 less

We to"che! o some o+ the %asic thi gs that Ci7-unCi7 ca !o. They %oth ha#e a lot o+ optio s that a!! to their +le3i%ility, tho"gh some are plat+orm speci+ic to other systems. The ma pages +or %oth Ci7 a ! unCi7 are pretty goo! a ! co tai "se+"l e3amples. Aowe#er, the mai "se o+ these programs is +or e3cha gi g +iles with Wi !ows systems, rather tha per+ormi g compressio a ! archi#i g o *i "3, where tar a ! gCi7 are greatly pre+erre!.

S,nchroniGing ;iles #nd Directories

$ commo strategy +or mai tai i g a %ack"p copy o+ a system i #ol#es keepi g o e or more !irectories sy chro iDe! with a other !irectory Eor !irectoriesF locate! o either the local system E"s"ally a remo#a%le storage !e#ice o+ some ki !F or with a remote system. We might, +or e3ample, ha#e a local copy o+ a we% site " !er !e#elopme t a ! sy chro iDe it +rom time to time with the ?li#e@ copy o a remote we% ser#er. 9 the ) i3-like worl!, the pre+erre! tool +or this task is rs1nc. This program ca sy chro iDe %oth local a ! remote !irectories %y "si g the rsync remote-update protocol, which allows rs1nc to ="ickly !etect the !i++ere ces %etwee two !irectories a ! per+orm the mi im"m amo" t o+ copyi g re="ire! to %ri g them i to sy c. This makes rs1nc #ery +ast a ! eco omical to "se, compare! to other ki !s o+ copy programs. rs1nc is i #oke! like this, 212

Sy chro iDi g 0iles $ ! 'irectories rs1nc options source destination where source a ! destination are o e o+ the +ollowi g,

$ local +ile or !irectory $ remote +ile or !irectory i the +orm o+ Fuser@Ghost:path $ remote rsy c ser#er speci+ie! with a ):9 o+ rsync:;;Fuser@GhostF:portG;path

&ote that either the so"rce or !esti atio m"st %e a local +ile. :emote to remote copyi g is ot s"pporte!. *etIs try rs1nc o"t o some local +iles. 0irst, letIs clea o"t o"r foo !irectory,
[me@linuxbox ~]$ rm -rf foo/*

&e3t, weIll sy chro iDe the 7la1ground !irectory with a correspo !i g copy i foo,
[me@linuxbox ~]$ rs!nc -av pla!ground foo

WeI#e i cl"!e! %oth the -a optio E+or archi#i gBca"ses rec"rsio a ! preser#atio o+ +ile attri%"tesF a ! the -v optio E#er%ose o"tp"tF to make a mirror o+ the 7la1ground !irectory withi foo. While the comma ! r" s, we will see a list o+ the +iles a ! !irectories %ei g copie!. $t the e !, we will see a s"mmary message like this,
sent #$")"0 b1tes received ")/)( b1tes total siCe is $!$( s7eedu7 is (<(! $/)!"/<(( b1tes6sec

i !icati g the amo" t o+ copyi g per+orme!. 9+ we r" the comma ! agai , we will see a !i++ere t res"lt,
[me@linuxbox ~]$ rs!nc -av pla!gound foo building file list <<< done sent !!.$" b1tes received !( b1tes total siCe is $!$( s7eedu7 is (<#% %"$#(<(( b1tes6sec

&otice that there was o listi g o+ +iles. This is %eca"se rs1nc !etecte! that there were o !i++ere ces %etwee ~67la1ground a ! ~6foo67la1ground, a ! there+ore it 21(

.9 8 $rchi#i g $ ! Jack"p !i! It ee! to copy a ythi g. 9+ we mo!i+y a +ile i 7la1ground a ! r" rs1nc agai ,
[me@linuxbox ~]$ touch pla!ground/dir-*DD/file-@ [me@linuxbox ~]$ rs!nc -av pla!ground foo building file list <<< done 7la1ground6dir-(006file-I sent !!./" b1tes received %! b1tes %"%"%<(( b1tes6sec total siCe is $!$( s7eedu7 is (<#%

we see that rs1nc !etecte! the cha ge a ! copie! o ly the "p!ate! +ile. $s a practical e3ample, letIs co si!er the imagi ary e3ter al har! !ri#e that we "se! earlier with tar. 9+ we attach the !ri#e to o"r system a !, o ce agai , it is mo" te! at 6 media6Hig'isk, we ca per+orm a "se+"l system %ack"p %y +irst creati g a !irectory, ame! 6backu7 o the e3ter al !ri#e a ! the "si g rs1nc to copy the most importa t st"++ +rom o"r system to the e3ter al !ri#e,
[me@linuxbox ~]$ mkdir /media/<ig1isk/backup [me@linuxbox ~]$ sudo rs!nc -av --delete /etc /home /usr/local /media/<ig1isk/backup

9 this e3ample, we copie! the 6etc, 6home, a ! 6usr6local !irectories +rom o"r system to o"r imagi ary storage !e#ice. We i cl"!e! the --delete optio to remo#e +iles that may ha#e e3iste! o the %ack"p !e#ice that o lo ger e3iste! o the so"rce !e#ice Ethis is irrele#a t the +irst time we make a %ack"p, %"t will %e "se+"l o s"%se="e t copies.F :epeati g the proce!"re o+ attachi g the e3ter al !ri#e a ! r" i g this rs1nc comma ! wo"l! %e a "se+"l Etho"gh ot i!ealF way o+ keepi g a small system %acke! "p. 7+ co"rse, a alias wo"l! %e help+"l here, too. We co"l! create a alias a ! a!! it to o"r <bashrc +ile to pro#i!e this +eat"re,
alias backu7MKsudo rs1nc -av --delete 6etc 6home 6usr6local 6media6Hig'isk6backu7K

&ow all we ha#e to !o is attach o"r e3ter al !ri#e a ! r" the backu7 comma ! to !o the 4o%.

211

Sy chro iDi g 0iles $ ! 'irectories

Using rs1nc <;er A ,et%ork

7 e o+ the real %ea"ties o+ rs1nc is that it ca %e "se! to copy +iles o#er a etwork. $+ter all, the ?r@ i rs1nc sta !s +or ?remote.@ :emote copyi g ca %e !o e i o e o+ two ways. The +irst way is with a other system that has rs1nc i stalle!, alo g with a remote shell program s"ch as ssh. *etIs say we ha! a other system o o"r local etwork with a lot o+ a#aila%le har! !ri#e space a ! we wa te! to per+orm o"r %ack"p operatio "si g the remote system i stea! o+ a e3ter al !ri#e. $ss"mi g that it alrea!y ha! a !irectory ame! 6backu7 where we co"l! !eli#er o"r +iles, we co"l! !o this,
[me@linuxbox ~]$ sudo rs!nc -av --delete --rsh$ssh /etc /home /usr/local remote-s!s2/backup

We ma!e two cha ges to o"r comma ! to +acilitate the etwork copy. 0irst, we a!!e! the --rshMssh optio , which i str"cts rs1nc to "se the ssh program as its remote shell. 9 this way, we were a%le to "se a ssh e crypte! t" el to sec"rely tra s+er the !ata +rom the local system to the remote host. Seco !, we speci+ie! the remote host %y pre+i3i g its ame Ei this case the remote host is ame! remote-s1sF to the !esti atio path ame. The seco ! way that rs1nc ca %e "se! to sy chro iDe +iles o#er a etwork is %y "si g a rysnc server. rs1nc ca %e co +ig"re! to r" as a !aemo a ! liste to i comi g re="ests +or sy chro iDatio . This is o+te !o e to allow mirrori g o+ a remote system. 0or e3ample, :e! Aat So+tware mai tai s a large repository o+ so+tware packages " !er !e#elopme t +or its 0e!ora !istri%"tio . 9t is "se+"l +or so+tware testers to mirror this collectio !"ri g the testi g phase o+ the !istri%"tio release cycle. Si ce +iles i the repository cha ge +re="e tly Eo+te more tha o ce a !ayF, it is !esira%le to mai tai a local mirror %y perio!ic sy chro iDatio , rather tha %y %"lk copyi g o+ the repository. 7 e o+ these repositories is kept at ;eorgia TechH we co"l! mirror it "si g o"r local copy o+ rs1nc a ! their rsy c ser#er like this,
[me@linuxbox ~]$ mkdir fedora-devel [me@linuxbox ~]$ rs!nc -av -delete rs!nc2//rs!nc.gtlib.gatech.edu/fed ora-linux-core/development/i MN/os fedora-devel

9 this e3ample, we "se the ):9 o+ the remote rsy c ser#er, which co sists o+ a protocol Ers1nc:66F, +ollowe! %y the remote host ame Ers1nc<gtlib<gatech<eduF, +ollowe! %y the path ame o+ the repository.

212

.9 8 $rchi#i g $ ! Jack"p

;urther 1eading

The ma pages +or all o+ the comma co tai "se+"l e3amples. 9 a!!itio +or its #ersio o+ tar. 9t ca %e +o" http,--www.g ".org-so+tware-tar-ma

!s !isc"sse! here are pretty clear a ! , the ;&) 6ro4ect has a goo! o li e ma "al ! here, "al-i !e3.html

21K

20 8 :eg"lar E3pressio s

20 /egular !"#ressions
9 the e3t +ew chapters, we are goi g to look at tools "se! to ma ip"late te3t. $s we ha#e see , te3t !ata plays a importa t role o all ) i3-like systems, s"ch as *i "3. J"t %e+ore we ca +"lly appreciate all o+ the +eat"res o++ere! %y these tools, we ha#e to +irst e3ami e a tech ology that is +re="e tly associate! with the most sophisticate! "ses o+ these toolsBregular e+pressions. $s we ha#e a#igate! the ma y +eat"res a ! +acilities o++ere! %y the comma ! li e, we ha#e e co" tere! some tr"ly arca e shell +eat"res a ! comma !s, s"ch as shell e3pa sio a ! ="oti g, key%oar! shortc"ts, a ! comma ! history, ot to me tio the #i e!itor. :eg"lar e3pressio s co ti "e this ?tra!itio @ a ! may %e Earg"a%lyF the most arca e +eat"re o+ them all. This is ot to s"ggest that the time it takes to lear a%o"t them is ot worth the e++ort. T"ite the co trary. $ goo! " !ersta !i g will e a%le "s to per+orm amaDi g +eats, tho"gh their +"ll #al"e may ot %e imme!iately appare t.

What Are !egu ar 6x5ressions?

Simply p"t, reg"lar e3pressio s are sym%olic otatio s "se! to i!e ti+y patter s i te3t. 9 some ways, they resem%le the shellIs wil!car! metho! o+ matchi g +ile a ! path ames, %"t o a m"ch gra !er scale. :eg"lar e3pressio s are s"pporte! %y ma y comma ! li e tools a ! %y most programmi g la g"ages to +acilitate the sol"tio o+ te3t ma ip"latio pro%lems. Aowe#er, to +"rther co +"se thi gs, ot all reg"lar e3pressio s are the sameH they #ary slightly +rom tool to tool a ! +rom programmi g la g"age to la g"age. 0or o"r !isc"ssio , we will limit o"rsel#es to reg"lar e3pressio s as !escri%e! i the 67S9G sta !ar! Ewhich will co#er most o+ the comma ! li e toolsF, as oppose! to ma y programmi g la g"ages Emost ota%ly 6erlF, which "se slightly larger a ! richer sets o+ otatio s.

grep
The mai program we will "se to work with reg"lar e3pressio s is o"r ol! pal, gre7. The ame ?grep@ is act"ally !eri#e! +rom the phrase ?glo%al reg"lar e3pressio pri t,@ so we ca see that gre7 has somethi g to !o with reg"lar e3pressio s. 9 esse ce, gre7 searches te3t +iles +or the occ"rre ce o+ a speci+ie! reg"lar e3pressio a ! o"tp"ts a y 21/

20 8 :eg"lar E3pressio s li e co tai i g a match to sta !ar! o"tp"t. So +ar, we ha#e "se! gre7 with +i3e! stri gs, like so,
[me@linuxbox ~]$ ls /usr/bin 0 grep -ip

This will list all the +iles i the 6usr6bin !irectory whose ames co tai the s"%stri g ?Dip@. The gre7 program accepts optio s a ! arg"me ts this way, gre7 [options] regex [file...] where rege+ is a reg"lar e3pressio . Aere is a list o+ the commo ly "se! gre7 optio s, Table'=-1: grep &ptions ?ption -i -v Description 9g ore case. 'o ot !isti g"ish %etwee "pper a ! lower case characters. >ay also %e speci+ie! --ignore-case. 9 #ert match. &ormally, gre7 pri ts li es that co tai a match. This optio ca"ses gre7 to pri t e#ery li e that !oes ot co tai a match. >ay also %e speci+ie! --invert-match. 6ri t the "m%er o+ matches Eor o -matches i+ the -v optio is also speci+ie!F i stea! o+ the li es themsel#es. >ay also %e speci+ie! --count. 6ri t the ame o+ each +ile that co tai s a match i stea! o+ the li es themsel#es. >ay also %e speci+ie! --files-9ith-matches. *ike the -l optio , %"t pri t o ly the ames o+ +iles that !o ot co tai matches. >ay also %e speci+ie! --files-9ithoutmatch. 6re+i3 each matchi g li e with the "m%er o+ the li e withi the +ile. >ay also %e speci+ie! --line-number< 0or m"lti-+ile searches, s"ppress the o"tp"t o+ +ile ames. >ay also %e speci+ie! --no-filename.

-c

-l -=

-n -h

9 or!er to more +"lly e3plore gre7, letIs create some te3t +iles to search, 218

grep
[me@linuxbox ~]$ ls /bin & dirlist-bin.txt [me@linuxbox ~]$ ls /usr/bin & dirlist-usr-bin.txt [me@linuxbox ~]$ ls /sbin & dirlist-sbin.txt [me@linuxbox ~]$ ls /usr/sbin & dirlist-usr-sbin.txt [me@linuxbox ~]$ ls dirlist*.txt dirlist-bin<txt dirlist-sbin<txt dirlist-usr-sbin<txt dirlist-usr-bin<txt

We ca per+orm a simple search o+ o"r list o+ +iles like this,

[me@linuxbox ~]$ grep b-ip dirlist*.txt dirlist-bin<txt:bCi7! dirlist-bin<txt:bCi7!recover

9 this e3ample, gre7 searches all o+ the liste! +iles +or the stri g bCi7 a ! +i !s two matches, %oth i the +ile dirlist-bin<txt. 9+ we were o ly i tereste! i the list o+ +iles that co tai e! matches rather tha the matches themsel#es, we co"l! speci+y the -l optio ,
[me@linuxbox ~]$ grep -l b-ip dirlist*.txt dirlist-bin<txt

Co #ersely, i+ we wa te! o ly to see a list o+ the +iles that !i! ot co tai a match, we co"l! !o this,
[me@linuxbox ~]$ grep -T b-ip dirlist*.txt dirlist-sbin<txt dirlist-usr-bin<txt dirlist-usr-sbin<txt

>etacharacters #nd Literals

While it may ot seem appare t, o"r gre7 searches ha#e %ee "si g reg"lar e3pressio s all alo g, al%eit #ery simple o es. The reg"lar e3pressio ?%Dip@ is take to mea that a match will occ"r o ly i+ the li e i the +ile co tai s at least +o"r characters a ! that somewhere i the li e the characters ?%@, ?D@, ?i@, a ! ?p@ are +o" ! i that or!er, with o other characters i %etwee . The characters i the stri g ?%Dip@ are all literal characters, i that they match themsel#es. 9 a!!itio to literals, reg"lar e3pressio s may also 219

20 8 :eg"lar E3pressio s i cl"!e metacharacters that are "se! to speci+y more comple3 matches. e3pressio metacharacters co sist o+ the +ollowi g, ` $ < [ ] [ \ - F E 8 N P Q ] $ll other characters are co si!ere! literals, tho"gh the %ackslash character is "se! i a +ew cases to create meta se<uences, as well as allowi g the metacharacters to %e escape! a ! treate! as literals i stea! o+ %ei g i terprete! as metacharacters. 1ote5 $s we ca see, ma y o+ the reg"lar e3pressio metacharacters are also characters that ha#e mea i g to the shell whe e3pa sio is per+orme!. Whe we pass reg"lar e3pressio s co tai i g metacharacters o the comma ! li e, it is #ital that they %e e close! i ="otes to pre#e t the shell +rom attempti g to e3pa ! them. :eg"lar

The #n, Character

The +irst metacharacter we will look at is the !ot or perio! character, which is "se! to match a y character. 9+ we i cl"!e it i a reg"lar e3pressio , it will match a y character i that character positio . AereIs a e3ample,
[me@linuxbox ~]$ grep -h %.-ip% dirlist*.txt bunCi7! bCi7! bCi7!recover gunCi7 gCi7 funCi7 g7g-Ci7 7reunCi7 7reCi7 7reCi7-bin unCi7 unCi7sfx

We searche! +or a y li e i o"r +iles that matches the reg"lar e3pressio ?.Dip@. There are a co"ple o+ i teresti g thi gs to ote a%o"t the res"lts. &otice that the Ci7 program was ot +o" !. This is %eca"se the i cl"sio o+ the !ot metacharacter i o"r reg"lar e3pressio i crease! the le gth o+ the re="ire! match to +o"r characters, a ! %eca"se the ame ?Dip@ o ly co tai s three, it !oes ot match. $lso, i+ there ha! %ee a y +iles i o"r lists that co tai e! the +ile e3te sio <Ci7, they wo"l! ha#e also %ee matche! as well, %eca"se the perio! character i the +ile e3te sio is treate! as ?a y character,@ too.

220

$ chors

#nchors
The caret E`F a ! !ollar sig E$F characters are treate! as anchors i reg"lar e3pressio s. This mea s that they ca"se the match to occ"r o ly i+ the reg"lar e3pressio is +o" ! at the %egi i g o+ the li e E`F or at the e ! o+ the li e E$F,
[me@linuxbox ~]$ grep -h %[-ip% dirlist*.txt Ci7 Ci7cloak Ci7gre7 Ci7info Ci7note Ci7s7lit [me@linuxbox ~]$ grep -h %-ip3% dirlist*.txt gunCi7 gCi7 funCi7 g7g-Ci7 7reunCi7 7reCi7 unCi7 Ci7 [me@linuxbox ~]$ grep -h %[-ip3% dirlist*.txt Ci7

Aere we searche! the list o+ +iles +or the stri g ?Dip@ locate! at the %egi i g o+ the li e, the e ! o+ the li e, a ! o a li e where it is at %oth the %egi i g a ! the e ! o+ the li e Ei.e., %y itsel+ o the li e.F &ote that the reg"lar e3pressio e`[I Ea %egi i g a ! a e ! with othi g i %etwee F will match %la k li es.

# Cross4ord PuGGle 3elper

E#e with o"r limite! k owle!ge o+ reg"lar e3pressio s at this poi t, we ca !o somethi g "se+"l. >y wi+e lo#es crosswor! p"DDles a ! she will sometimes ask me +or help with a partic"lar ="estio . Somethi g like, ?whatIs a +i#e letter wor! whose thir! letter is e4I a ! last letter is erI that mea s...C@ This ki ! o+ ="estio got me thi ki g. 'i! yo" k ow that yo"r *i "3 system co tai s a !ictio aryC 9t !oes. Take a look i the 6usr6share6dict !irectory a ! yo" might +i ! o e, or se#eral. The !ictio ary +iles locate! there are 4"st lo g lists o+ wor!s, o e per li e, arra ge! i alpha%etical or!er. 7 my system, the 9ords +ile co tai s 4"st o#er 98,200

22.

20 8 :eg"lar E3pressio s

wor!s. To +i ! possi%le a swers to the crosswor! p"DDle ="estio a%o#e, we co"l! !o this,
[me@linuxbox ~]$ grep -i %[..j.r3% /usr/share/dict/words +ajor major

)si g this reg"lar e3pressio , we ca +i ! all the wor!s i o"r !ictio ary +ile that are +i#e letters lo g a ! ha#e a ?4@ i the thir! positio a ! a ?r@ i the last positio .

/rac(et $xpressions #nd Character Classes

9 a!!itio to matchi g a y character at a gi#e positio i o"r reg"lar e3pressio , we ca also match a si gle character +rom a speci+ie! set o+ characters %y "si g bracket e+pressions. With %racket e3pressio s, we ca speci+y a set o+ characters Ei cl"!i g characters that wo"l! otherwise %e i terprete! as metacharactersF to %e matche!. 9 this e3ample, "si g a two character set,
[me@linuxbox ~]$ grep -h %(bg)-ip% dirlist*.txt bCi7! bCi7!recover gCi7

we match a y li e that co tai s the stri g ?%Dip@ or ?gDip@. $ set may co tai a y "m%er o+ characters, a ! metacharacters lose their special mea i g whe place! withi %rackets. Aowe#er, there are two cases i which metacharacters are "se! withi %racket e3pressio s, a ! ha#e !i++ere t mea i gs. The +irst is the caret E`F, which is "se! to i !icate egatio H the seco ! is the !ash E-F, which is "se! to i !icate a character ra ge.

,egation
9+ the +irst character i a %racket e3pressio is a caret E`F, the remai i g characters are take to %e a set o+ characters that m"st ot %e prese t at the gi#e character positio . We !o this %y mo!i+yi g o"r pre#io"s e3ample,
[me@linuxbox ~]$ bunCi7! grep -h %([bg)-ip% dirlist*.txt

222

Jracket E3pressio s $ ! Character Classes

gunCi7 funCi7 g7g-Ci7 7reunCi7 7reCi7 7reCi7-bin unCi7 unCi7sfx

With egatio acti#ate!, we get a list o+ +iles that co tai the stri g ?Dip@ prece!e! %y a y character e3cept ?%@ or ?g@. &otice that the +ile Ci7 was ot +o" !. $ egate! character set still re="ires a character at the gi#e positio , %"t the character m"st ot %e a mem%er o+ the egate! set. The caret character o ly i #okes egatio i+ it is the +irst character withi a %racket e3pressio H otherwise, it loses its special mea i g a ! %ecomes a or!i ary character i the set.

Traditiona Chara/ter !anges

9+ we wa te! to co str"ct a reg"lar e3pressio that wo"l! +i ! e#ery +ile i o"r lists %egi i g with a "pper case letter, we co"l! !o this,
[me@linuxbox ~]$ grep -h %[(:<=1G7QRPYXTO?SB\HF+E]WZ@^)% dirlist*.txt

9tIs 4"st a matter o+ p"tti g all twe ty-si3 "pper case letters i a %racket e3pressio . J"t the i!ea o+ all that typi g is !eeply tro"%li g, so there is a other way,
[me@linuxbox ~]$ grep -h %[(:-@)% dirlist*.txt +42&'&; Control:anel @&T S&4' : *T V V## Vorg +42&-= ::A&* Let9ork+anager Let9ork+anager'is7atcher

Jy "si g a three character ra ge, we ca a%%re#iate the twe ty-si3 letters. $ y ra ge o+ 22(

20 8 :eg"lar E3pressio s characters ca %e e3presse! this way i cl"!i g m"ltiple ra ges, s"ch as this e3pressio that matches all +ile ames starti g with letters a ! "m%ers,
[me@linuxbox ~]$ grep -h %[(:-@a--*-D)% dirlist*.txt

9 character ra ges, we see that the !ash character is treate! specially, so how !o we act"ally i cl"!e a !ash character i a %racket e3pressio C Jy maki g it the +irst character i the e3pressio . Co si!er these two e3amples,
[me@linuxbox ~]$ grep -h %(:-@)% dirlist*.txt

This will match e#ery +ile ame co tai i g a "pper case letter. While,
[me@linuxbox ~]$ grep -h %(-:@)% dirlist*.txt

will match e#ery +ile ame co tai i g a !ash, or a "pper case ?$@ or a "ppercase ?f@.

'<SIJ Chara/ter C asses

The tra!itio al character ra ges are a easily " !erstoo! a ! e++ecti#e way to ha !le the pro%lem o+ ="ickly speci+yi g sets o+ characters. ) +ort" ately, they !o It always work. While we ha#e ot e co" tere! a y pro%lems with o"r "se o+ gre7 so +ar, we might r" i to pro%lems "si g other programs. Jack i Chapter 2, we looke! at how wil!car!s are "se! to per+orm path ame e3pa sio . 9 that !isc"ssio , we sai! that character ra ges co"l! %e "se! i a ma er almost i!e tical to the way they are "se! i reg"lar e3pressio s, %"t hereIs the pro%lem,
[me@linuxbox ~]$ ls /usr/sbin/(:<=1G7QRPYXTO?SB\HF+E]WZ^@)* 6usr6sbin6+42&-= ::A&* 6usr6sbin6Let9ork+anager'is7atcher 6usr6sbin6Let9ork+anager

E'epe !i g o the *i "3 !istri%"tio , we will get a !i++ere t list o+ +iles, possi%ly a empty list. This e3ample is +rom )%" t"F This comma ! pro!"ces the e3pecte! res"lt Ba list o+ o ly the +iles whose ames %egi with a "ppercase letter, %"t,

221

Jracket E3pressio s $ ! Character Classes

[me@linuxbox ~]$ ls /usr/sbin/(:-@)* 6usr6sbin6biosdecode 6usr6sbin6chat 6usr6sbin6chg7ass9d 6usr6sbin6ch7ass9d 6usr6sbin6chroot 6usr6sbin6cleanu7-info 6usr6sbin6com7lain 6usr6sbin6console-kit-daemon

with this comma ! we get a e tirely !i++ere t res"lt Eo ly a partial listi g o+ the res"lts is show F. Why is thatC 9tIs a lo g story, %"t hereIs the short #ersio , Jack whe ) i3 was +irst !e#elope!, it o ly k ew a%o"t $SC99 characters, a ! this +eat"re re+lects that +act. 9 $SC99, the +irst thirty-two characters E "m%ers 0-(.F are co trol co!es Ethi gs like ta%s, %ackspaces, a ! carriage ret"r sF. The e3t thirty-two E(2-K(F co tai pri ta%le characters, i cl"!i g most p" ct"atio characters a ! the "merals Dero thro"gh i e. The e3t thirty-two E "m%ers K1-92F co tai the "ppercase letters a ! a +ew more p" ct"atio sym%ols. The +i al thirty-o e E "m%ers 9K-.2/F co tai the lowercase letters a ! yet more p" ct"atio sym%ols. Jase! o this arra geme t, systems "si g $SC99 "se! a collation order that looke! like this, 4HC'&-@SA?2=+L :ZR*T3;,VUIabcdefghijklmno7Brstuv9x1C This !i++ers +rom proper !ictio ary or!er, which is like this, a4bHcCd'e&f-g@hSiAj?k2l=m+nLo 7:BZrRs*tTu3v;9,xV1UCI $s the pop"larity o+ ) i3 sprea! %eyo ! the ) ite! States, there grew a ee! to s"pport characters ot +o" ! i ).S. E glish. The $SC99 ta%le was e3pa !e! to "se a +"ll eight %its, a!!i g characters "m%ers .28-222, which accommo!ate! ma y more la g"ages. To s"pport this a%ility, the 67S9G sta !ar!s i tro!"ce! a co cept calle! a locale, which co"l! %e a!4"ste! to select the character set ee!e! +or a partic"lar locatio . We ca see the la g"age setti g o+ o"r system "si g this comma !,
[me@linuxbox ~]$ echo 3T:?Q enO3*<3T--/

With this setti g, 67S9G complia t applicatio s will "se a !ictio ary collatio or!er rather tha $SC99 or!er. This e3plai s the %eha#ior o+ the comma !s a%o#e. $ character ra ge o+ [4-I] whe i terprete! i !ictio ary or!er i cl"!es all o+ the alpha%etic characters e3cept the lowercase ?a@, he ce o"r res"lts. To partially work aro" ! this pro%lem, the 67S9G sta !ar! i cl"!es a "m%er o+ character classes which pro#i!e "se+"l ra ges o+ characters. They are !escri%e! i the 222

20 8 :eg"lar E3pressio s ta%le %elow, Ta%le 20-2, 67S9G Character Classes Character Class [:alnum:] [:9ord:] [:al7ha:] [:blank:] [:cntrl:] [:digit:] [:gra7h:] [:lo9er:] [:7unct:] [:7rint:] [:s7ace:] Description The alpha "meric characters. 9 $SC99, e="i#ale t to, [4-Ia-C(-0] The same as a,al "m,b, with the a!!itio o+ the " !erscore EOF character. The alpha%etic characters. 9 $SC99, e="i#ale t to, [4-Ia-C] 9 cl"!es the space a ! ta% characters. The $SC99 co trol co!es. 9 cl"!es the $SC99 characters Dero thro"gh thirty-o e a ! .2/. The "merals Dero thro"gh i e. The #isi%le characters. 9 $SC99, it i cl"!es characters (( thro"gh .2K. The lowercase letters. The p" ct"atio characters. 9 $SC99, e="i#ale t to, [-G_^$5YKNPE8><6:TXMDF@[]]]]OJ[Q\~] The pri ta%le characters. $ll the characters i [:gra7h:] pl"s the space character. The whitespace characters i cl"!i g space, ta%, carriage ret"r , ewli e, #ertical ta%, a ! +orm +ee!. 9 $SC99, e="i#ale t to, [ ]t]r]n]v]f] The "pper case characters. Characters "se! to e3press he3a!ecimal "m%ers. 9 $SC99, e="i#ale t to, [(-04--a-f]

[:u77er:] [:xdigit:]

E#e with the character classes, there is still o co #e ie t way to e3press partial ra ges, s"ch as a$->b. )si g character classes, we ca repeat o"r !irectory listi g a ! see a impro#e! res"lt,

22K

Jracket E3pressio s $ ! Character Classes

[me@linuxbox ~]$ ls /usr/sbin/((2upper2))* 6usr6sbin6+42&-= ::A&* 6usr6sbin6Let9ork+anager'is7atcher 6usr6sbin6Let9ork+anager

:emem%er, howe#er, that this is ot a e3ample o+ a reg"lar e3pressio , rather it is the shell per+ormi g path ame e3pa sio . We show it here %eca"se 67S9G character classes ca %e "se! +or %oth.

1e%erting To Traditional Collation ?rder

5o" ca opt to ha#e yo"r system "se the tra!itio al E$SC99F collatio or!er %y cha gi g the #al"e o+ the =4L@ e #iro me t #aria%le. $s we saw a%o#e, the =4L@ #aria%le co tai s the ame o+ the la g"age a ! character set "se! i yo"r locale. This #al"e was origi ally !etermi e! whe yo" selecte! a i stallatio la g"age as yo"r *i "3 was i stalle!. To see the locale setti gs, "se the locale comma !, [me@linuxbox ~]$ locale =4L@MenO3*<3T--/ =COCTU:&M_enO3*<3T--/_ =COL3+&RACM_enO3*<3T--/_ =COTA+&M_enO3*<3T--/_ =COC ==4T&M_enO3*<3T--/_ =CO+ L&T4RUM_enO3*<3T--/_ =CO+&**4@&*M_enO3*<3T--/_ =CO:4:&RM_enO3*<3T--/_ =COL4+&M_enO3*<3T--/_ =CO4''R&**M_enO3*<3T--/_ =COT&=&:S L&M_enO3*<3T--/_ =CO+&4*3R&+&LTM_enO3*<3T--/_ =COA'&LTA-AC4TA LM_enO3*<3T--/_ =CO4==M To cha ge the locale to "se the tra!itio al ) i3 %eha#iors, set the =4L@ #aria%le to : *AV, [me@linuxbox ~]$ export T:?Q$BSFPZ &ote that this cha ge co #erts the system to "se ).S. E glish Emore speci+ically, $SC99F +or its character set, so %e s"re i+ this is really what yo" wa t.

22/

20 8 :eg"lar E3pressio s

5o" ca make this cha ge perma e t %y a!!i g this li e to yo" yo"r <bashrc +ile, ex7ort =4L@M: *AV

P?SIH /asic Vs. $xtended 1egular $xpressions

J"st whe we tho"ght this co"l! It get a y more co +"si g, we !isco#er that 67S9G also splits reg"lar e3pressio impleme tatio s i to two ki !s, basic regular e+pressions H891I a ! e+tended regular e+pressions H191I. The +eat"res we ha#e co#ere! so +ar are s"pporte! %y a y applicatio that is 67S9G-complia t a ! impleme ts J:E. 7"r gre7 program is o e s"ch program. WhatIs the !i++ere ce %etwee J:E a ! E:EC 9tIs a matter o+ metacharacters. With J:E, the +ollowi g metacharacters are recog iDe!, ` $ < [ ] E $ll other characters are co si!ere! literals. With E:E, the +ollowi g metacharacters Ea ! their associate! +" ctio sF are a!!e!, N P [ \ F 8 Q Aowe#er Ea ! this is the +" partF, the ?E@, ?F@, ?c@, a ! ?d@ characters are treate! as metacharacters i J:E if they are escape! with a %ackslash, whereas with E:E, prece!i g a y metacharacter with a %ackslash ca"ses it to %e treate! as a literal. $ y weir! ess that comes alo g will %e co#ere! i the !isc"ssio s that +ollow. Si ce the +eat"res we are goi g to !isc"ss e3t are part o+ E:E, we are goi g to ee! to "se a !i++ere t gre7. Tra!itio ally, this has %ee per+orme! %y the egre7 program, %"t the ;&) #ersio o+ gre7 also s"pports e3te !e! reg"lar e3pressio s whe the -& optio is "se!.

P?SIH
'"ri g the .980Is, ) i3 %ecame a #ery pop"lar commercial operati g system, %"t %y .988, the ) i3 worl! was i t"rmoil. >a y comp"ter ma "+act"rers ha! lice se! the ) i3 so"rce co!e +rom its creators, $T^T, a ! were s"pplyi g #ario"s #ersio s o+ the operati g system with their systems. Aowe#er, i their e++orts to create pro!"ct !i++ere tiatio , each ma "+act"rer a!!e! proprietary cha ges a ! e3te sio s. This starte! to limit the compati%ility o+ the so+tware.

228

67S9G Jasic Rs. E3te !e! :eg"lar E3pressio s

$s always with proprietary #e !ors, each was tryi g to play a wi i g game o+ ?lock-i @ with their c"stomers. This !ark time i the history o+ ) i3 is k ow to!ay as ?the 8alkani7ation.@ E ter the 9EEE E9 stit"te o+ Electrical a ! Electro ics E gi eersF. 9 the mi!-.980s, the 9EEE %ega !e#elopi g a set o+ sta !ar!s that wo"l! !e+i e how ) i3 Ea ! ) i3-likeF systems wo"l! per+orm. These sta !ar!s, +ormally k ow as 9EEE .00(, !e+i e the application programming interfaces E$69sF, shell a ! "tilities that are to %e +o" ! o a sta !ar! ) i3-like system. The ame ?67S9G,@ which sta !s +or 6ortable &perating System *nterface Ewith the ?G@ a!!e! to the e ! +or e3tra s appi essF, was s"ggeste! %y :ichar! Stallma Eyes, that :ichar! Stallma F, a ! was a!opte! %y the 9EEE.

#lternation
The +irst o+ the e3te !e! reg"lar e3pressio +eat"res we will !isc"ss is calle! alternation, which is the +acility that allows a match to occ"r +rom amo g a set o+ e3pressio s. J"st as a %racket e3pressio allows a si gle character to match +rom a set o+ speci+ie! characters, alter atio allows matches +rom a set o+ stri gs or other reg"lar e3pressio s. To !emo strate, weIll "se gre7 i co 4" ctio with echo. 0irst, letIs try a plai ol! stri g match,
[me@linuxbox ~]$ echo J:::J 0 grep ::: 444 [me@linuxbox ~]$ echo J<<<J 0 grep ::: [me@linuxbox ~]$

$ pretty straight+orwar! e3ample, i which we pipe the o"tp"t o+ echo i to gre7 a ! see the res"lts. Whe a match occ"rs, we see it pri te! o"tH whe o match occ"rs, we see o res"lts. &ow weIll a!! alter atio , sig i+ie! %y the #ertical %ar metacharacter,
[me@linuxbox 444 [me@linuxbox HHH [me@linuxbox [me@linuxbox ~]$ echo J:::J 0 grep -G %:::0<<<% ~]$ echo J<<<J 0 grep -G %:::0<<<% ~]$ echo J===J 0 grep -G %:::0<<<% ~]$

229

20 8 :eg"lar E3pressio s Aere we see the reg"lar e3pressio K444QHHHK which mea s ?match either the stri g 444 or the stri g HHH.@ &otice that si ce this is a e3te !e! +eat"re, we a!!e! the -& optio to gre7 Etho"gh we co"l! ha#e 4"st "se! the egre7 program i stea!F, a ! we e close! the reg"lar e3pressio i ="otes to pre#e t the shell +rom i terpreti g the #ertical %ar metacharacter as a pipe operator. $lter atio is ot limite! to two choices,
[me@linuxbox ~]$ echo J:::J 0 grep -G %:::0<<<0===% 444

To com%i e alter atio with other reg"lar e3pressio eleme ts, we ca "se EF to separate the alter atio ,
[me@linuxbox ~]$ grep -Gh %[4b-0g-0-ip6% dirlist*.txt

This e3pressio will match the +ile ames i o"r lists that start with either ?%D@, ?gD@, or ?Dip@. Aa! we le+t o++ the pare theses, the mea i g o+ this reg"lar e3pressio ,
[me@linuxbox ~]$ grep -Gh %[b-0g-0-ip% dirlist*.txt

cha ges to match a y +ile ame that %egi s with ?%D@ or contains ?gD@ or contains ?Dip@.

Auanti"iers
E3te !e! reg"lar e3pressio s s"pport se#eral ways to speci+y the "m%er o+ times a eleme t is matche!.

? B :at/h An 6 ement Kero <r <ne Time

This ="a ti+ier mea s, i e++ect, ?make the prece!i g eleme t optio al.@ *etIs say we wa te! to check a pho e "m%er +or #ali!ity a ! we co si!ere! a pho e "m%er to %e #ali! i+ it matche! either o+ these two +orms,
NnnnP nnn-nnnn nnn nnn-nnnn

where ?n@ is a "meral. We co"l! co str"ct a reg"lar e3pressio like this,

`]NF[(-0][(-0][(-0]]PF [(-0][(-0][(-0]-[(-0][(-0][(-0][(-0]$

9 this e3pressio , we +ollow the pare theses characters with ="estio marks to i !icate that they are to %e matche! Dero or o e time. $gai , si ce the pare theses are ormally 2K0

T"a ti+iers metacharacters Ei E:EF, we prece!e them with %ackslashes to ca"se them to %e treate! as literals i stea!. *etIs try it,
[me@linuxbox ~]$ echo J4"""6 12 -A"NCJ 0 grep -G %[K4_(*-D)(*-D)(*-D) K6_ (*-D)(*-D)(*-D)3% N"""P #!$-%".) [me@linuxbox ~]$ echo J""" 12 -A"NCJ 0 grep -G %[K4_(*-D)(*-D)(*-D)K6 _ (*-D)(*-D)(*-D)-(*-D)(*-D)(*-D)(*-D)3% """ #!$-%".) [me@linuxbox ~]$ echo J::: 12 -A"NCJ 0 grep -G %[K4_(*-D)(*-D)(*-D)K6 _ (*-D)(*-D)(*-D)-(*-D)(*-D)(*-D)(*-D)3% [me@linuxbox ~]$

Aere we see that the e3pressio matches %oth +orms o+ the pho e "m%er, %"t !oes ot match o e co tai i g o - "meric characters.

L B :at/h An 6 ement Kero <r :ore Times

*ike the F metacharacter, the E is "se! to !e ote a optio al itemH howe#er, " like the F, the item may occ"r a y "m%er o+ times, ot 4"st o ce. *etIs say we wa te! to see i+ a stri g was a se te ceH that is, it starts with a "ppercase letter, the co tai s a y "m%er o+ "pper a ! lowercase letters a ! spaces, a ! e !s with a perio!. To match this E#ery cr"!eF !e+i itio o+ a se te ce, we co"l! "se a reg"lar e3pressio like this,
[[:u77er:]][[:u77er:][:lo9er:] ]E]<

The e3pressio co sists o+ three items, a %racket e3pressio co tai i g the [:u77er:] character class, a %racket e3pressio co tai i g %oth the [:u77er:] a ! [:lo9er:] character classes a ! a space, a ! a perio! escape! with a %ackslash. The seco ! eleme t is traile! with a E metacharacter, so that a+ter the lea!i g "ppercase letter i o"r se te ce, a y "m%er o+ "pper a ! lowercase letters a ! spaces may +ollow it a ! still match,
[me@linuxbox ~]$ echo J+his works.J 0 grep -G %((2upper2))((2upper2)( 2lower2) )*K.% This 9orks< [me@linuxbox ~]$ echo J+his Works.J 0 grep -G %((2upper2))((2upper2)( 2lower2) )*K.% This ,orks< [me@linuxbox ~]$ echo Jthis does notJ 0 grep -G %((2upper2))((2upper2 )(2lower2) )*K.% [me@linuxbox ~]$

2K.

20 8 :eg"lar E3pressio s The e3pressio matches the +irst two tests, %"t ot the thir!, si ce it lacks the re="ire! lea!i g "ppercase character a ! traili g perio!.

M B :at/h An 6 ement <ne <r :ore Times

The 8 metacharacter works m"ch like the E, e3cept it re="ires at least o e i sta ce o+ the prece!i g eleme t to ca"se a match. Aere is a reg"lar e3pressio that will o ly match li es co sisti g o+ gro"ps o+ o e or more alpha%etic characters separate! %y si gle spaces,
`N[[:al7ha:]]8 FP8$

[me@linuxbox This that [me@linuxbox a b c [me@linuxbox [me@linuxbox [me@linuxbox

~]$ echo J+his thatJ 0 grep -G %[4((2alpha2))5 _653% ~]$ echo Ja b cJ 0 grep -G %[4((2alpha2))5 _653% ~]$ echo Ja b DJ 0 grep -G %[4((2alpha2))5 _653% ~]$ echo Jabc dJ 0 grep -G %[4((2alpha2))5 _653% ~]$

We see that this e3pressio !oes ot match the li e ?a % 9@ %eca"se it co tai s a o alpha%etic characterH or !oes it match ?a%c !@ %eca"se more tha o e space character separates the characters ?c@ a ! ?!@.

N O B :at/h An 6 ement A S5e/i=i/ ,umber <= Times

The [ a ! \ metacharacters are "se! to e3press mi im"m a ! ma3im"m "m%ers o+ re="ire! matches. They may %e speci+ie! i +o"r possi%le ways, Table '=-3: Specifying The >umber &f !atches Speci"ier [n\ [n>m\ [n>\ [>m\ >eaning >atch the prece!i g eleme t i+ it occ"rs e3actly n times. >atch the prece!i g eleme t i+ it occ"rs at least n times, %"t o more tha m times. >atch the prece!i g eleme t i+ it occ"rs n or more times. >atch the prece!i g eleme t i+ it occ"rs o more tha m times.

;oi g %ack to o"r earlier e3ample with the pho e "m%ers, we ca "se this metho! o+ speci+yi g repetitio s to simpli+y o"r origi al reg"lar e3pressio +rom,

2K2

T"a ti+iers
`]NF[(-0][(-0][(-0]]PF [(-0][(-0][(-0]-[(-0][(-0][(-0][(-0]$

to,
`]NF[(-0][$\]PF [(-0][$\-[(-0][%\$

*etIs try it,

[me@linuxbox ~]$ echo J4"""6 12 -A"NCJ 0 grep -G %[K4_(*-D)9 >K6_ (*D)9 >-(*-D)9A>3% N"""P #!$-%".) [me@linuxbox ~]$ echo J""" 12 -A"NCJ 0 grep -G %[K4_(*-D)9 >K6_ (*-D) 9 >-(*-D)9A>3% """ #!$-%".) [me@linuxbox ~]$ echo J"""" 12 -A"NCJ 0 grep -G %[K4_(*-D)9 >K6_ (*-D )9 >-(*-D)9A>3% [me@linuxbox ~]$

$s we ca see, o"r re#ise! e3pressio ca s"ccess+"lly #ali!ate "m%ers %oth with a ! witho"t the pare theses, while re4ecti g those "m%ers that are ot properly +ormatte!.

Putting 1egular $xpressions To +or(

*etIs look at some o+ the comma !s we alrea!y k ow a ! see how they ca %e "se! with reg"lar e3pressio s.

?a idating A 'hone List With gre7

9 o"r earlier e3ample, we looke! at si gle pho e "m%ers a ! checke! them +or proper +ormatti g. $ more realistic sce ario wo"l! %e checki g a list o+ "m%ers i stea!, so letIs make a list. WeIll !o this %y reciti g a magical i ca tatio to the comma ! li e. 9t will %e magic %eca"se we ha#e ot co#ere! most o+ the comma !s i #ol#e!, %"t worry ot. We will get there i +"t"re chapters. Aere is the i ca tatio ,
[me@linuxbox ~]$ for i in 91..1*># do echo J439H:?1SO2*2 >6 39H:?1S O2*2 >-39H:?1SO2*2A>J && phonelist.txt# done

This comma ! will pro!"ce a +ile ame! 7honelist<txt co tai i g te pho e "m%ers. Each time the comma ! is repeate!, a other te "m%ers are a!!e! to the list. We ca also cha ge the #al"e #( ear the %egi i g o+ the comma ! to pro!"ce more or +ewer pho e "m%ers. 9+ we e3ami e the co te ts o+ the +ile, howe#er, we see we ha#e a pro%lem,

2K(

20 8 :eg"lar E3pressio s
[me@linuxbox ~]$ cat phonelist.txt N!$!P !0/-!!." N.!%P $/#-#()/ N"%(P #!.-#0/( N/)%P #.$-!//" N!/.P !"%-!/.( N!0!P #(/-"#/ N#!0P %%-#$)0 N%"/P !)$-#.%! N./.P !00-/!./ N#0/P $()-!%%(

Some o+ the "m%ers are mal+orme!, which is per+ect +or o"r p"rposes, si ce we will "se gre7 to #ali!ate them. 7 e "se+"l metho! o+ #ali!atio wo"l! %e to sca the +ile +or i #ali! "m%ers a ! !isplay the res"lti g list o the !isplay,
[me@linuxbox ~]$ grep -Gv %[K4(*-D)9 >K6 (*-D)9 >-(*-D)9A>3% 7honelist<txt N!0!P #(/-"#/ N#!0P %%-#$)0 [me@linuxbox ~]$

Aere we "se the -v optio to pro!"ce a i #erse match so that we will o ly o"tp"t the li es i the list that !o ot match the speci+ie! e3pressio . The e3pressio itsel+ i cl"!es the a chor metacharacters at each e ! to e s"re that the "m%er has o e3tra characters at either e !. This e3pressio also re="ires that the pare theses %e prese t i a #ali! "m%er, " like o"r earlier pho e "m%er e3ample.

2inding Ug y 2i enames With find

The find comma ! s"pports a test %ase! o a reg"lar e3pressio . There is a importa t co si!eratio to keep i mi ! whe "si g reg"lar e3pressio s i find #ers"s gre7. Whereas gre7 will pri t a li e whe the li e contains a stri g that matches a e3pressio , find re="ires that the path ame e+actly match the reg"lar e3pressio . 9 the +ollowi g e3ample, we will "se find with a reg"lar e3pressio to +i ! e#ery path ame that co tai s a y character that is ot a mem%er o+ the +ollowi g set, [-O<6(-0a-C4-I] S"ch a sca wo"l! re#eal path ames that co tai em%e!!e! spaces a ! other pote tially o++e si#e characters,

2K1

6"tti g :eg"lar E3pressio s To Work

[me@linuxbox ~]$ find . -regex %.*([-../*-Da--:-@).*%

'"e to the re="ireme t +or a e3act match o+ the e tire path ame, we "se <E at %oth e !s o+ the e3pressio to match Dero or more i sta ces o+ a y character. 9 the mi!!le o+ the e3pressio , we "se a egate! %racket e3pressio co tai i g o"r set o+ accepta%le path ame characters.

Sear/hing 2or 2i es With locate

The locate program s"pports %oth %asic Ethe --regex7 optio F a ! e3te !e! Ethe -regex optio F reg"lar e3pressio s. With it, we ca per+orm ma y o+ the same operatio s that we per+orme! earlier with o"r dirlist +iles,
[me@linuxbox ~]$ locate --regex %bin/4b-0g-0-ip6% 6bin6bCcat 6bin6bCcm7 6bin6bCdiff 6bin6bCegre7 6bin6bCexe 6bin6bCfgre7 6bin6bCgre7 6bin6bCi7! 6bin6bCi7!recover 6bin6bCless 6bin6bCmore 6bin6gCexe 6bin6gCi7 6usr6bin6Ci7 6usr6bin6Ci7cloak 6usr6bin6Ci7gre7 6usr6bin6Ci7info 6usr6bin6Ci7note 6usr6bin6Ci7s7lit

)si g alter atio , we per+orm a search +or path ames that co tai either bin6bC, bin6 gC, or 6bin6Ci7.

Sear/hing 2or Text In less And vim

less a ! vim %oth share the same metho! o+ searchi g +or te3t. 6ressi g the 6 key +ollowe! %y a reg"lar e3pressio will per+orm a search. 9+ we "se less to #iew o"r 7honelist<txt +ile,

2K2

20 8 :eg"lar E3pressio s
[me@linuxbox ~]$ less phonelist.txt

The search +or o"r #ali!atio e3pressio ,

N!$!P !0/-!!." N.!%P $/#-#()/ N"%(P #!.-#0/( N/)%P #.$-!//" N!/.P !"%-!/.( N!0!P #(/-"#/ N#!0P %%-#$)0 N%"/P !)$-#.%! N./.P !00-/!./ N#0/P $()-!%%( ~ ~ ~ /[K4(*-D)9 >K6 (*-D)9 >-(*-D)9A>3

less will highlight the stri gs that match, lea#i g the i #ali! o es easy to spot,
N!$!P N.!%P N"%(P N/)%P N!/.P N!0!P N#!0P N%"/P N./.P N#0/P ~ ~ ~ N&L'P !0/-!!." $/#-#()/ #!.-#0/( #.$-!//" !"%-!/.( #(/-"#/ %%-#$)0 !)$-#.%! !00-/!./ $()-!%%(

vim, o the other ha !, s"pports %asic reg"lar e3pressio s, so o"r search e3pressio wo"l! look like this, 6N[(-0]][$]\P [(-0]][$]\-[(-0]][%]\ We ca see that the e3pressio is mostly the sameH howe#er, ma y o+ the characters that are co si!ere! metacharacters i e3te !e! e3pressio s are co si!ere! literals i %asic e3pressio s. They are o ly treate! as metacharacters whe escape! with a %ackslash. 2KK

6"tti g :eg"lar E3pressio s To Work 'epe !i g o the partic"lar co +ig"ratio o+ vim o o"r system, the matchi g will %e highlighte!. 9+ ot, try this comma ! mo!e comma !, :hlsearch to acti#ate search highlighti g. 1ote5 'epe !i g o yo"r !istri%"tio , vim may or may ot s"pport te3t search highlighti g. )%" t", i partic"lar, s"pplies a #ery strippe!-!ow #ersio o+ vim %y !e+a"lt. 7 s"ch systems, yo" may wa t to "se yo"r package ma ager to i stall a more complete #ersio o+ vim.

Summing -p
9 this chapter, weI#e see a +ew o+ the ma y "ses o+ reg"lar e3pressio s. We ca +i ! e#e more i+ we "se reg"lar e3pressio s to search +or a!!itio al applicatio s that "se them. We ca !o that %y searchi g the ma pages,
[me@linuxbox ~]$ cd /usr/share/man/man1 [me@linuxbox man#]$ -grep -Gl %regex0regular expression% *.g-

The Cgre7 program pro#i!es a +ro t e ! +or gre7, allowi g it to rea! compresse! +iles. 9 o"r e3ample, we search the compresse! sectio o e ma page +iles locate! i their "s"al locatio . The res"lt o+ this comma ! is a list o+ +iles co tai i g either the stri g ?rege3@ or ?reg"lar e3pressio @. $s we ca see, reg"lar e3pressio s show "p i a lot o+ programs. There is o e +eat"re +o" ! i %asic reg"lar e3pressio s that we !i! ot co#er. Calle! back references, this +eat"re will %e !isc"sse! i the e3t chapter.

;urther 1eading
There are ma y o li e reso"rces +or lear i g reg"lar e3pressio s, i cl"!i g #ario"s t"torials a ! cheat sheets. 9 a!!itio , the Wikipe!ia has goo! articles o the +ollowi g %ackgro" ! topics,

67S9G, http,--e .wikipe!ia.org-wiki-6osi3 $SC99, http,--e .wikipe!ia.org-wiki-$scii

2K/

2. 8 Te3t 6rocessi g

21 Te"t 2rocessing
$ll ) i3-like operati g systems rely hea#ily o te3t +iles +or se#eral types o+ !ata storage. So it makes se se that there are ma y tools +or ma ip"lati g te3t. 9 this chapter, we will look at programs that are "se! to ?slice a ! !ice@ te3t. 9 the e3t chapter, we will look at more te3t processi g, +oc"si g o programs that are "se! to +ormat te3t +or pri ti g a ! other ki !s o+ h"ma co s"mptio . This chapter will re#isit some ol! +rie !s a ! i tro!"ce "s to some ew o es,

cat 8 Co cate ate +iles a ! pri t o the sta !ar! o"tp"t sort 8 Sort li es o+ te3t +iles uniB 8 :eport or omit repeate! li es cut 8 :emo#e sectio s +rom each li e o+ +iles 7aste 8 >erge li es o+ +iles join 8 Joi li es o+ two +iles o a commo +iel! comm 8 Compare two sorte! +iles li e %y li e diff 8 Compare +iles li e %y li e 7atch 8 $pply a !i++ +ile to a origi al tr 8 Tra slate or !elete characters sed 8 Stream e!itor +or +ilteri g a ! tra s+ormi g te3t as7ell 8 9 teracti#e spell checker

#pplications ?" Text

So +ar, we ha#e lear e! a co"ple o+ te3t e!itors Enano a ! vimF, looke! a %" ch o+ co +ig"ratio +iles, a ! ha#e wit esse! the o"tp"t o+ !oDe s o+ comma !s, all i te3t. J"t what else is te3t "se! +orC 0or ma y thi gs, it t"r s o"t.

2K8

$pplicatio s 7+ Te3t

)o/uments
>a y people write !oc"me ts "si g plai te3t +ormats. While it is easy to see how a small te3t +ile co"l! %e "se+"l +or keepi g simple otes, it is also possi%le to write large !oc"me ts i te3t +ormat, as well. 7 e pop"lar approach is to write a large !oc"me t i a te3t +ormat a ! the "se a markup language to !escri%e the +ormatti g o+ the +i ishe! !oc"me t. >a y scie ti+ic papers are writte "si g this metho!, as ) i3-%ase! te3t processi g systems were amo g the +irst systems that s"pporte! the a!#a ce! typographical layo"t ee!e! %y writers i tech ical !iscipli es.

Web 'ages
The worl!Is most pop"lar type o+ electro ic !oc"me t is pro%a%ly the we% page. We% pages are te3t !oc"me ts that "se either ,T!( H,yperte+t !arkup (anguageI or J!( H1+tensible !arkup (anguageI as mark"p la g"ages to !escri%e the !oc"me tIs #is"al +ormat.

6mai
Email is a i tri sically te3t-%ase! me!i"m. E#e o -te3t attachme ts are co #erte! i to a te3t represe tatio +or tra smissio . We ca see this +or o"rsel#es %y !ow loa!i g a email message a ! the #iewi g it i less. We will see that the message %egi s with a header that !escri%es the so"rce o+ the message a ! the processi g it recei#e! !"ri g its 4o"r ey, +ollowe! %y the body o+ the message with its co te t.

'rinter <ut5ut
7 ) i3-like systems, o"tp"t !esti e! +or a pri ter is se t as plai te3t or, i+ the page co tai s graphics, is co #erte! i to a te3t +ormat page description language k ow as 6ostScript, which is the se t to a program that ge erates the graphic !ots to %e pri te!.

'rogram Sour/e Code

>a y o+ the comma ! li e programs +o" ! o ) i3-like systems were create! to s"pport system a!mi istratio a ! so+tware !e#elopme t, a ! te3t processi g programs are o e3ceptio . >a y o+ them are !esig e! to sol#e so+tware !e#elopme t pro%lems. The reaso te3t processi g is importa t to so+tware !e#elopers is that all so+tware starts o"t as te3t. Source code, the part o+ the program the programmer act"ally writes, is always i te3t +ormat.

1e%isiting Some ?ld ;riends

Jack i Chapter / E:e!irectio F, we lear e! a%o"t some comma !s that are a%le to 2K9

2. 8 Te3t 6rocessi g accept sta !ar! i p"t i a!!itio to comma ! li e arg"me ts. We o ly to"che! o them %rie+ly the , %"t ow we will take a closer look at how they ca %e "se! to per+orm te3t processi g.

cat
The cat program has a "m%er o+ i teresti g optio s. >a y o+ them are "se! to help %etter #is"aliDe te3t co te t. 7 e e3ample is the -4 optio , which is "se! to !isplay o pri ti g characters i the te3t. There are times whe we wa t to k ow i+ co trol characters are em%e!!e! i o"r otherwise #isi%le te3t. The most commo o+ these are ta% characters Eas oppose! to spacesF a ! carriage ret"r s, o+te prese t as e !-o+-li e characters i >S-'7S style te3t +iles. $ other commo sit"atio is a +ile co tai i g li es o+ te3t with traili g spaces. *etIs create a test +ile "si g cat as a primiti#e wor! processor. To !o this, weIll 4"st e ter the comma ! cat Ealo g with speci+yi g a +ile +or re!irecte! o"tp"tF a ! type o"r te3t, +ollowe! %y &nter to properly e ! the li e, the Ctrl-d, to i !icate to cat that we ha#e reache! e !-o+-+ile. 9 this e3ample, we e ter a lea!i g ta% character a ! +ollow the li e with some traili g spaces,
[me@linuxbox ~]$ cat & foo.txt The Buick bro9n fox jum7ed over the laC1 dog< [me@linuxbox ~]$

&e3t, we will "se cat with the -4 optio to !isplay the te3t,
[me@linuxbox ~]$ cat -: foo.txt `AThe Buick bro9n fox jum7ed over the laC1 dog< [me@linuxbox ~]$

$s we ca see i the res"lts, the ta% character i o"r te3t is represe te! %y `A. This is a commo otatio that mea s ?Co trol-9@ which, as it t"r s o"t, is the same as a ta% character. We also see that a $ appears at the tr"e e ! o+ the li e, i !icati g that o"r te3t co tai s traili g spaces.

2/0

:e#isiti g Some 7l! 0rie !s

>SED?S Text Vs. -nix Text

7 e o+ the reaso s yo" may wa t to "se cat to look +or o -pri ti g characters i te3t is to spot hi!!e carriage ret"r s. Where !o hi!!e carriage ret"r s come +romC '7S a ! Wi !owsL ) i3 a ! '7S !o It !e+i e the e ! o+ a li e the same way i te3t +iles. ) i3 e !s a li e with a li e+ee! character E$SC99 .0F while >S-'7S a ! its !eri#ati#es "se the se="e ce carriage ret"r E$SC99 .(F a ! li e+ee! to termi ate each li e o+ te3t. There are a se#eral ways to co #ert +iles +rom '7S to ) i3 +ormat. 7 ma y *i "3 systems, there are programs calle! dos!unix a ! unix!dos, which ca co #ert te3t +iles to a ! +rom '7S +ormat. Aowe#er, i+ yo" !o It ha#e dos!unix o yo"r system, !o It worry. The process o+ co #erti g te3t +rom '7S to ) i3 +ormat is #ery simpleH it simply i #ol#es the remo#al o+ the o++e !i g carriage ret"r s. That is easily accomplishe! %y a co"ple o+ the programs !isc"sse! later i this chapter. cat also has optio s that are "se! to mo!i+y te3t. The two most promi e t are -n, which "m%ers li es, a ! -s, which s"ppresses the o"tp"t o+ m"ltiple %la k li es. We ca !emo strate th"sly,
[me@linuxbox ~]$ cat & foo.txt The Buick bro9n fox jum7ed over the laC1 dog< [me@linuxbox ~]$ cat -ns foo.txt # The Buick bro9n fox ! $ jum7ed over the laC1 dog< [me@linuxbox ~]$

9 this e3ample, we create a ew #ersio o+ o"r foo<txt test +ile, which co tai s two li es o+ te3t separate! %y two %la k li es. $+ter processi g %y cat with the -ns optio s, the e3tra %la k li e is remo#e! a ! the remai i g li es are "m%ere!. While this is ot m"ch o+ a process to per+orm o te3t, it is a process.

sort
The sort program sorts the co te ts o+ sta !ar! i p"t, or o e or more +iles speci+ie! o the comma ! li e, a ! se !s the res"lts to sta !ar! o"tp"t. )si g the same tech i="e 2/.

2. 8 Te3t 6rocessi g that we "se! with cat, we ca !emo strate processi g o+ sta !ar! i p"t !irectly +rom the key%oar!,
[me@linuxbox ~]$ sort & foo.txt c b a [me@linuxbox ~]$ cat foo.txt a b c

$+ter e teri g the comma !, we type the letters ?c@, ?%@, a ! ?a@, +ollowe! o ce agai %y Ctrl-d to i !icate e !-o+-+ile. We the #iew the res"lti g +ile a ! see that the li es ow appear i sorte! or!er. Si ce sort ca accept m"ltiple +iles o the comma ! li e as arg"me ts, it is possi%le to merge m"ltiple +iles i to a si gle sorte! whole. 0or e3ample, i+ we ha! three te3t +iles a ! wa te! to com%i e them i to a si gle sorte! +ile, we co"l! !o somethi g like this,
sort file1.txt file2.txt file .txt & final.sorted.list.txt

sort has se#eral i teresti g optio s. Aere is a partial list, Table '1-1: %ommon sort &ptions ?ption -b Long ?ption --ignore-leading-blanks Description Jy !e+a"lt, sorti g is per+orme! o the e tire li e, starti g with the +irst character i the li e. This optio ca"ses sort to ig ore lea!i g spaces i li es a ! calc"lates sorti g %ase! o the +irst o -whitespace character o the li e. >akes sorti g case i se siti#e. 6er+orms sorti g %ase! o the "meric e#al"atio o+ a stri g. )si g this optio allows sorti g to %e per+orme! o "meric #al"es

-f -n

--ignore-case --numeric-sort

2/2

:e#isiti g Some 7l! 0rie !s rather tha alpha%etic #al"es. -r --reverse Sort i re#erse or!er. :es"lts are i !esce !i g rather tha asce !i g or!er. Sort %ase! o a key +iel! locate! +rom field1 to field' rather tha the e tire li e. See !isc"ssio %elow. Treat each each arg"me t as the ame o+ a presorte! +ile. >erge m"ltiple +iles i to a si gle sorte! res"lt witho"t per+ormi g a y a!!itio al sorti g. Se ! sorte! o"tp"t to file rather tha sta !ar! o"tp"t. 'e+i e the +iel! separator character. Jy !e+a"lt +iel!s are separate! %y spaces or ta%s.

-k

--ke1Mfield1[>field2]

-m

--merge

-o -t

--out7utMfile --field-se7aratorMchar

$ltho"gh most o+ the optio s a%o#e are pretty sel+-e3pla atory, some are ot. 0irst, letIs look at the -n optio , "se! +or "meric sorti g. With this optio , it is possi%le to sort #al"es %ase! o "meric #al"es. We ca !emo strate this %y sorti g the res"lts o+ the du comma ! to !etermi e the largest "sers o+ !isk space. &ormally, the du comma ! lists the res"lts o+ a s"mmary i path ame or!er,
[me@linuxbox ~]$ du -s /usr/share/* 0 head !"! 6usr6share6aclocal 0. 6usr6share6ac7i-su77ort / 6usr6share6adduser #0. 6usr6share6alacarte $%% 6usr6share6alsa / 6usr6share6alsa-base #!%// 6usr6share6anth1 / 6usr6share6a7md !#%%( 6usr6share6a77-install %/ 6usr6share6a77lication-registr1

9 this e3ample, we pipe the res"lts i to head to limit the res"lts to the +irst te li es. We ca pro!"ce a "merically sorte! list to show the te largest co s"mers o+ space this way, 2/(

2. 8 Te3t 6rocessi g
[me@linuxbox ~]$ du -s /usr/share/* 0 sort -nr 0 head "(00%( 6usr6share6locale-lang7ack !%!..( 6usr6share6doc #0)".( 6usr6share6fonts #)0#%% 6usr6share6gnome #%.).% 6usr6share6m1s7ell #%%$(% 6usr6share6gim7 #$"//( 6usr6share6dict )."(/ 6usr6share6icons ./()! 6usr6share6a77s .!/%% 6usr6share6foomatic

Jy "si g the -nr optio s, we pro!"ce a re#erse "merical sort, with the largest #al"es appeari g +irst i the res"lts. This sort works %eca"se the "merical #al"es occ"r at the %egi i g o+ each li e. J"t what i+ we wa t to sort a list %ase! o some #al"e +o" ! withi the li eC 0or e3ample, the res"lts o+ a ls -l,
[me@linuxbox total #"!0%/ -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # ~]$ ls -l /usr/bin 0 head root root root root root root root root root root root root root root root root root root $%/!% #(#"". #$($. #(""! $/(( )"$. $"). !(/(/ %/0)(% !((/-(%-(% !(()-##-!) !((/-(!-!) !(()-(/-#" !((/-(%-#% !((/-(%-#0 !((/-(%-!0 !((/-(#-($ !((/-#(-(0 (!:%! (.:(/ (/:!! #(:$% ($:"# ((:#0 ():") #/:(! #):(! [ a!7 aconnect ac7i ac7iOfakeke1 ac7iOlisten add7art addr!line ade7tObatch

9g ori g, +or the mome t, that ls ca sort its res"lts %y siDe, we co"l! "se sort to sort this list %y +ile siDe, as well,
[me@linuxbox -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # -r9xr-xr-x # ~]$ ls root root root root root root root root root root -l /usr/bin 0 sort -nr -k " 0 head root /!$%!#. !((/-(%-() #):%! inksca7e root /!!!.0! !((/-(%-() #):%! inkvie9 root $)%."(/ !((/-($-() !$:%" gim7-!<% root $."%(!( !((/-(/-!. #.:#. Buanta root !0!/).( !((/-(0-#( #%:$# gdbtui root !0!/)". !((/-(0-#( #%:$# gdb root !.(!!$. !((/-#(-#( #!:". net root !$(%./% !((/-#(-#( #!:". r7cclient root !!%#/$! !((/-(%-(% (":". a7titude root !!(!%). !((/-#(-#( #!:". smbcacls

2/1

:e#isiti g Some 7l! 0rie !s >a y "ses o+ sort i #ol#e the processi g o+ tabular data, s"ch as the res"lts o+ the ls comma ! a%o#e. 9+ we apply !ata%ase termi ology to the ta%le a%o#e, we wo"l! say that each row is a record a ! that each recor! co sists o+ m"ltiple fields, s"ch as the +ile attri%"tes, li k co" t, +ile ame, +ile siDe a ! so o . sort is a%le to process i !i#i!"al +iel!s. 9 !ata%ase terms, we are a%le to speci+y o e or more key fields to "se as sort keys. 9 the e3ample a%o#e, we speci+y the n a ! r optio s to per+orm a re#erse "merical sort a ! speci+y -k " to make sort "se the +i+th +iel! as the key +or sorti g. The k optio is #ery i teresti g a ! has ma y +eat"res, %"t +irst we ee! to talk a%o"t how sort !e+i es +iel!s. *etIs co si!er a #ery simple te3t +ile co sisti g o+ a si gle li e co tai i g the a"thorIs ame,
,illiam *hotts

Jy !e+a"lt, sort sees this li e as ha#i g two +iel!s. characters, c,illiamd a ! the seco ! +iel! co tai s the characters, c *hottsd

The +irst +iel! co tai s the

mea i g that whitespace characters Espaces a ! ta%sF are "se! as !elimiters %etwee +iel!s a ! that the !elimiters are i cl"!e! i the +iel! whe sorti g is per+orme!. *ooki g agai at a li e +rom o"r ls o"tp"t, we ca see that a li e co tai s eight +iel!s a ! that the +i+th +iel! is the +ile siDe,
-r9xr-xr-x # root root /!$%!#. !((/-(%-() #):%! inksca7e

0or o"r e3t series o+ e3perime ts, letIs co si!er the +ollowi g +ile co tai i g the history o+ three pop"lar *i "3 !istri%"tio s release! +rom 200K to 2008. Each li e i the +ile has three +iel!s, the !istri%"tio ame, #ersio "m%er, a ! !ate o+ release i >>-''-5555 +ormat,
*3*& -edora *3*& 3buntu -edora *3*& #(<! #( ##<( /<(% / #(<$ #!6()6!((. ##6!"6!((/ (.6#06!((/ (%6!%6!((/ ##6(/6!(() #(6(%6!(()

2/2

2. 8 Te3t 6rocessi g
3buntu -edora 3buntu 3buntu *3*& -edora -edora 3buntu 3buntu -edora .<#( ) )<#( )<(% #(<# . 0 .<(. /<#( " #(6!.6!((. ("6$#6!(() #(6#/6!(() (%6#06!(() ("6##6!((. #(6!%6!((. ("6#$6!((/ (.6(#6!((. #(6$(6!((/ ($6!(6!((.

)si g a te3t e!itor Eperhaps vimF, weIll e ter this !ata a ! ame the res"lti g +ile distros<txt. &e3t, weIll try sorti g the +ile a ! o%ser#e the res"lts,
[me@linuxbox ~]$ sort distros.txt -edora #( ##6!"6!((/ -edora " ($6!(6!((. -edora . #(6!%6!((. -edora ) ("6$#6!(() -edora / ##6(/6!(() -edora 0 ("6#$6!((/ *3*& #(<# ("6##6!((. *3*& #(<! #!6()6!((. *3*& #(<$ #(6(%6!(() *3*& ##<( (.6#06!((/ 3buntu .<(. (.6(#6!((. 3buntu .<#( #(6!.6!((. 3buntu )<(% (%6#06!(() 3buntu )<#( #(6#/6!(() 3buntu /<(% (%6!%6!((/ 3buntu /<#( #(6$(6!((/

Well, it mostly worke!. The pro%lem occ"rs i the sorti g o+ the 0e!ora #ersio "m%ers. Si ce a ?.@ comes %e+ore a ?2@ i the character set, #ersio ?.0@ e !s "p at the top while #ersio ?9@ +alls to the %ottom. To +i3 this pro%lem we are goi g to ha#e to sort o m"ltiple keys. We wa t to per+orm a alpha%etic sort o the +irst +iel! a ! the a "meric sort o the thir! +iel!. sort allows m"ltiple i sta ces o+ the -k optio so that m"ltiple sort keys ca %e speci+ie!. 9 +act, a key may i cl"!e a ra ge o+ +iel!s. 9+ o ra ge is speci+ie! Eas has %ee the case with o"r pre#io"s e3amplesF, sort "ses a key that %egi s with the speci+ie! +iel! a ! e3te !s to the e ! o+ the li e. Aere is the sy ta3 +or o"r m"lti-key sort,

2/K

:e#isiti g Some 7l! 0rie !s

[me@linuxbox ~]$ -edora " -edora . -edora ) -edora / -edora 0 -edora #( *3*& #(<# *3*& #(<! *3*& #(<$ *3*& ##<( 3buntu .<(. 3buntu .<#( 3buntu )<(% 3buntu )<#( 3buntu /<(% 3buntu /<#( sort --ke!$1;1 --ke!$2n distros.txt ($6!(6!((. #(6!%6!((. ("6$#6!(() ##6(/6!(() ("6#$6!((/ ##6!"6!((/ ("6##6!((. #!6()6!((. #(6(%6!(() (.6#06!((/ (.6(#6!((. #(6!.6!((. (%6#06!(() #(6#/6!(() (%6!%6!((/ #(6$(6!((/

Tho"gh we "se! the lo g +orm o+ the optio +or clarity, -k #># -k !n wo"l! %e e3actly e="i#ale t. 9 the +irst i sta ce o+ the key optio , we speci+ie! a ra ge o+ +iel!s to i cl"!e i the +irst key. Si ce we wa te! to limit the sort to 4"st the +irst +iel!, we speci+ie! #># which mea s ?start at +iel! o e a ! e ! at +iel! o e.@ 9 the seco ! i sta ce, we speci+ie! !n, which mea s that +iel! two is the sort key a ! that the sort sho"l! %e "meric. $ optio letter may %e i cl"!e! at the e ! o+ a key speci+ier to i !icate the type o+ sort to %e per+orme!. These optio letters are the same as the glo%al optio s +or the sort program, b Eig ore lea!i g %la ksF, n E "meric sortF, r Ere#erse sortF, a ! so o . The thir! +iel! i o"r list co tai s a !ate i a i co #e ie t +ormat +or sorti g. 7 comp"ters, !ates are "s"ally +ormatte! i 5555->>-'' or!er to make chro ological sorti g easy, %"t o"rs are i the $merica +ormat o+ >>-''-5555. Aow ca we sort this list i chro ological or!erC 0ort" ately, sort pro#i!es a way. The key optio allows speci+icatio o+ offsets withi +iel!s, so we ca !e+i e keys withi +iel!s,
[me@linuxbox ~]$ sort -k .Cnbr -k -edora #( ##6!"6!((/ 3buntu /<#( #(6$(6!((/ *3*& ##<( (.6#06!((/ -edora 0 ("6#$6!((/ 3buntu /<(% (%6!%6!((/ -edora / ##6(/6!(() 3buntu )<#( #(6#/6!(() *3*& #(<$ #(6(%6!(() .1nbr -k .Anbr distros.txt

2//

2. 8 Te3t 6rocessi g
-edora 3buntu *3*& 3buntu -edora 3buntu *3*& -edora ) )<(% #(<! .<#( . .<(. #(<# " ("6$#6!(() (%6#06!(() #!6()6!((. #(6!.6!((. #(6!%6!((. (.6(#6!((. ("6##6!((. ($6!(6!((.

Jy speci+yi g -k $<) we i str"ct sort to "se a sort key that %egi s at the se#e th character withi the thir! +iel!, which correspo !s to the start o+ the year. *ikewise, we speci+y -k $<# a ! -k $<% to isolate the mo th a ! !ay portio s o+ the !ate. We also a!! the n a ! r optio s to achie#e a re#erse "meric sort. The b optio is i cl"!e! to s"ppress the lea!i g spaces Ewhose "m%ers #ary +rom li e to li e, there%y a++ecti g the o"tcome o+ the sortF i the !ate +iel!. Some +iles !o It "se ta%s a ! spaces as +iel! !elimitersH +or e3ample, the 6etc67ass9d +ile,
[me@linuxbox ~]$ head /etc/passwd root:x:(:(:root:6root:6bin6bash daemon:x:#:#:daemon:6usr6sbin:6bin6sh bin:x:!:!:bin:6bin:6bin6sh s1s:x:$:$:s1s:6dev:6bin6sh s1nc:x:%:.""$%:s1nc:6bin:6bin6s1nc games:x:":.(:games:6usr6games:6bin6sh man:x:.:#!:man:6var6cache6man:6bin6sh l7:x:):):l7:6var6s7ool6l7d:6bin6sh mail:x:/:/:mail:6var6mail:6bin6sh ne9s:x:0:0:ne9s:6var6s7ool6ne9s:6bin6sh

The +iel!s i this +ile are !elimite! with colo s E:F, so how wo"l! we sort this +ile "si g a key +iel!C sort pro#i!es the -t optio to !e+i e the +iel! separator character. To sort the 7ass9d +ile o the se#e th +iel! Ethe acco" tIs !e+a"lt shellF, we co"l! !o this,
[me@linuxbox ~]$ sort -t %2% -k C /etc/passwd 0 head me:x:#((#:#((#:+1self>>>:6home6me:6bin6bash root:x:(:(:root:6root:6bin6bash dhc7:x:#(#:#(!::6nonexistent:6bin6false gdm:x:#(.:##%:@nome 'is7la1 +anager:6var6lib6gdm:6bin6false h7li7:x:#(%:):S:=A: s1stem user>>>:6var6run6h7li7:6bin6false klog:x:#($:#(%::6home6klog:6bin6false messagebus:x:#(/:##0::6var6run6dbus:6bin6false

2/8

:e#isiti g Some 7l! 0rie !s

7olkituser:x:##(:#!!::olic12it>>>:6var6run6:olic12it:6bin6false 7ulse:x:#():##.::ulse4udio daemon>>>:6var6run67ulse:6bin6false

Jy speci+yi g the colo character as the +iel! separator, we ca sort o the se#e th +iel!.

uniB
Compare! to sort, the uniB program is a lightweight. uniB per+orms a seemi gly tri#ial task. Whe gi#e a sorte! +ile Ei cl"!i g sta !ar! i p"tF, it remo#es a y !"plicate li es a ! se !s the res"lts to sta !ar! o"tp"t. 9t is o+te "se! i co 4" ctio with sort to clea the o"tp"t o+ !"plicates. Ti(5 While uniB is a tra!itio al ) i3 tool o+te "se! with sort, the ;&) #ersio o+ sort s"pports a -u optio , which remo#es !"plicates +rom the sorte! o"tp"t. *etIs make a te3t +ile to try this o"t,
[me@linuxbox ~]$ cat & foo.txt a b c a b c

:emem%er to type Ctrl-d to termi ate sta !ar! i p"t. &ow, i+ we r" uniB o o"r te3t +ile,
[me@linuxbox ~]$ uni, foo.txt a b c a b c

the res"lts are o !i++ere t +rom o"r origi al +ileH the !"plicates were ot remo#e!. 0or uniB to act"ally !o its 4o%, the i p"t m"st %e sorte! +irst,

2/9

2. 8 Te3t 6rocessi g
[me@linuxbox ~]$ sort foo.txt 0 uni, a b c

This is %eca"se uniB o ly remo#es !"plicate li es which are a!4ace t to each other. uniB has se#eral optio s. Aere are the commo o es, Table '1-': %ommon uni< &ptions ?ption -c -d -f n Description 7"tp"t a list o+ !"plicate li es prece!e! %y the "m%er o+ times the li e occ"rs. 7 ly o"tp"t repeate! li es, rather tha " i="e li es. 9g ore n lea!i g +iel!s i each li e. 0iel!s are separate! %y whitespace as they are i sortH howe#er, " like sort, uniB has o optio +or setti g a alter ate +iel! separator. 9g ore case !"ri g the li e compariso s. Skip Eig oreF the lea!i g n characters o+ each li e. 7 ly o"tp"t " i="e li es. This is the !e+a"lt.

-i -s n -u

Aere we see uniB "se! to report the "m%er o+ !"plicates +o" ! i o"r te3t +ile, "si g the -c optio ,
[me@linuxbox ~]$ sort foo.txt 0 uni, -c ! a ! b ! c

Slicing #nd Dicing

The e3t three programs we will !isc"ss are "se! to peel col"m s o+ te3t o"t o+ +iles a ! recom%i e them i "se+"l ways.

cut
The cut program is "se! to e3tract a sectio o+ te3t +rom a li e a ! o"tp"t the e3tracte! 280

Slici g $ ! 'ici g sectio to sta !ar! o"tp"t. 9t ca accept m"ltiple +ile arg"me ts or i p"t +rom sta !ar! i p"t. Speci+yi g the sectio o+ the li e to %e e3tracte! is somewhat awkwar! a ! is speci+ie! "si g the +ollowi g optio s, Table '1-3: cut Selection &ptions ?ption -c char_list Description E3tract the portio o+ the li e !e+i e! %y char list. The list may co sist o+ o e or more comma-separate! "merical ra ges. E3tract o e or more +iel!s +rom the li e as !e+i e! %y field list. The list may co tai o e or more +iel!s or +iel! ra ges separate! %y commas. Whe -f is speci+ie!, "se delim char as the +iel! !elimiti g character. Jy !e+a"lt, +iel!s m"st %e separate! %y a si gle ta% character. E3tract the e tire li e o+ te3t, e3cept +or those portio s speci+ie! %y -c a !-or -f.

-f field_list

-d delim_char

--com7lement

$s we ca see, the way cut e3tracts te3t is rather i +le3i%le. cut is %est "se! to e3tract te3t +rom +iles that are pro!"ce! %y other programs, rather tha te3t !irectly type! %y h"ma s. WeIll take a look at o"r distros<txt +ile to see i+ it is ?clea @ e o"gh to %e a goo! specime +or o"r cut e3amples. 9+ we "se cat with the -4 optio , we ca see i+ the +ile meets o"r re="ireme ts o+ ta% separate! +iel!s,
[me@linuxbox ~]$ cat -: distros.txt *3*&`A#(<!`A#!6()6!((.$ -edora`A#(`A##6!"6!((/$ *3*&`A##<(`A(.6#06!((/$ 3buntu`A/<(%`A(%6!%6!((/$ -edora`A/`A##6(/6!(()$ *3*&`A#(<$`A#(6(%6!(()$ 3buntu`A.<#(`A#(6!.6!((.$ -edora`A)`A("6$#6!(()$ 3buntu`A)<#(`A#(6#/6!(()$ 3buntu`A)<(%`A(%6#06!(()$ *3*&`A#(<#`A("6##6!((.$ -edora`A.`A#(6!%6!((.$ -edora`A0`A("6#$6!((/$

28.

2. 8 Te3t 6rocessi g
3buntu`A.<(.`A(.6(#6!((.$ 3buntu`A/<#(`A#(6$(6!((/$ -edora`A"`A($6!(6!((.$

9t looks goo!. &o em%e!!e! spaces, 4"st si gle ta% characters %etwee the +iel!s. Si ce the +ile "ses ta%s rather tha spaces, weIll "se the -f optio to e3tract a +iel!,
[me@linuxbox ~]$ #!6()6!((. ##6!"6!((/ (.6#06!((/ (%6!%6!((/ ##6(/6!(() #(6(%6!(() #(6!.6!((. ("6$#6!(() #(6#/6!(() (%6#06!(() ("6##6!((. #(6!%6!((. ("6#$6!((/ (.6(#6!((. #(6$(6!((/ ($6!(6!((. cut -f distros.txt

Jeca"se o"r distros +ile is ta%-!elimite!, it is %est to "se cut to e3tract +iel!s rather tha characters. This is %eca"se whe a +ile is ta%-!elimite!, it is " likely that each li e will co tai the same "m%er o+ characters, which makes calc"lati g character positio s withi the li e !i++ic"lt or impossi%le. 9 o"r e3ample a%o#e, howe#er, we ow ha#e e3tracte! a +iel! that l"ckily co tai s !ata o+ i!e tical le gth, so we ca show how character e3tractio works %y e3tracti g the year +rom each li e,
[me@linuxbox ~]$ !((. !((/ !((/ !((/ !(() !(() !((. !(() !(() !(() !((. cut -f distros.txt 0 cut -c C-1*

282

Slici g $ ! 'ici g
!((. !((/ !((. !((/ !((.

Jy r" i g cut a seco ! time o o"r list, we are a%le to e3tract character positio s / thro"gh .0, which correspo !s to the year i o"r !ate +iel!. The )-#( otatio is a e3ample o+ a ra ge. The cut ma page co tai s a complete !escriptio o+ how ra ges ca %e speci+ie!.

$xpanding Ta0s
7"r distros<txt +ile is i!eally +ormatte! +or e3tracti g +iel!s "si g cut. J"t what i+ we wa te! a +ile that co"l! %e +"lly ma ip"late! with cut %y characters, rather tha +iel!sC This wo"l! re="ire "s to replace the ta% characters withi the +ile with the correspo !i g "m%er o+ spaces. 0ort" ately, the ;&) Core"tils package i cl"!es a tool +or that. &ame! ex7and, this program accepts either o e or more +ile arg"me ts or sta !ar! i p"t, a ! o"tp"ts the mo!i+ie! te3t to sta !ar! o"tp"t. 9+ we process o"r distros<txt +ile with ex7and, we ca "se the cut -c to e3tract a y ra ge o+ characters +rom the +ile. 0or e3ample, we co"l! "se the +ollowi g comma ! to e3tract the year o+ release +rom o"r list, %y e3pa !i g the +ile a ! "si g cut to e3tract e#ery character +rom the twe ty-thir! positio to the e ! o+ the li e,
[me@linuxbox ~]$ expand distros.txt 0 cut -c 2 -

Core"tils also pro#i!es the unex7and program to s"%stit"te ta%s +or spaces. Whe worki g with +iel!s, it is possi%le to speci+y a !i++ere t +iel! !elimiter rather tha the ta% character. Aere we will e3tract the +irst +iel! +rom the 6etc67ass9d +ile,
[me@linuxbox ~]$ cut -d %2% -f 1 /etc/passwd 0 head root daemon bin s1s s1nc games

28(

2. 8 Te3t 6rocessi g
man l7 mail ne9s

)si g the -d optio , we are a%le to speci+y the colo character as the +iel! !elimiter.

7aste
The 7aste comma ! !oes the opposite o+ cut. :ather tha e3tracti g a col"m o+ te3t +rom a +ile, it a!!s o e or more col"m s o+ te3t to a +ile. 9t !oes this %y rea!i g m"ltiple +iles a ! com%i i g the +iel!s +o" ! i each +ile i to a si gle stream o sta !ar! o"tp"t. *ike cut, 7aste accepts m"ltiple +ile arg"me ts a !-or sta !ar! i p"t. To !emo strate how 7aste operates, we will per+orm some s"rgery o o"r distros<txt +ile to pro!"ce a chro ological list o+ releases. 0rom o"r earlier work with sort, we will +irst pro!"ce a list o+ !istros sorte! %y !ate a ! store the res"lt i a +ile calle! distros-b1-date<txt,
[me@linuxbox ~]$ sort -k tros-b!-date.txt .Cnbr -k .1nbr -k .Anbr distros.txt & dis

&e3t, we will "se cut to e3tract the +irst two +iel!s +rom the +ile Ethe !istro ame a ! #ersio F, a ! store that res"lt i a +ile ame! distro-versions<txt,
[me@linuxbox ~]$ cut -f 1;2 distros-b!-date.txt & distros-versions.t xt [me@linuxbox ~]$ head distros-versions.txt -edora #( 3buntu /<#( *3*& ##<( -edora 0 3buntu /<(% -edora / 3buntu )<#( *3*& #(<$ -edora ) 3buntu )<(%

The +i al piece o+ preparatio is to e3tract the release !ates a ! store them a +ile ame! distro-dates<txt, 281

Slici g $ ! 'ici g
[me@linuxbox ~]$ cut -f distros-b!-date.txt & distros-dates.txt [me@linuxbox ~]$ head distros-dates.txt ##6!"6!((/ #(6$(6!((/ (.6#06!((/ ("6#$6!((/ (%6!%6!((/ ##6(/6!(() #(6#/6!(() #(6(%6!(() ("6$#6!(() (%6#06!(()

We ow ha#e the parts we ee!. To complete the process, "se 7aste to p"t the col"m o+ !ates ahea! o+ the !istro ames a ! #ersio s, th"s creati g a chro ological list. This is !o e simply %y "si g 7aste a ! or!eri g its arg"me ts i the !esire! arra geme t,
[me@linuxbox ~]$ paste ##6!"6!((/ -edora #(6$(6!((/ 3buntu (.6#06!((/ *3*& ("6#$6!((/ -edora (%6!%6!((/ 3buntu ##6(/6!(() -edora #(6#/6!(() 3buntu #(6(%6!(() *3*& ("6$#6!(() -edora (%6#06!(() 3buntu #!6()6!((. *3*& #(6!.6!((. 3buntu #(6!%6!((. -edora (.6(#6!((. 3buntu ("6##6!((. *3*& ($6!(6!((. -edora distros-dates.txt distros-versions.txt #( /<#( ##<( 0 /<(% / )<#( #(<$ ) )<(% #(<! .<#( . .<(. #(<# "

join
9 some ways, join is like 7aste i that it a!!s col"m s to a +ile, %"t it "ses a " i="e way to !o it. $ Aoin is a operatio "s"ally associate! with relational databases where !ata +rom m"ltiple tables with a share! key +iel! is com%i e! to +orm a !esire! res"lt. The join program per+orms the same operatio . 9t 4oi s !ata +rom m"ltiple +iles %ase! o a share! key +iel!. To see how a 4oi operatio is "se! i a relatio al !ata%ase, letIs imagi e a #ery small !ata%ase co sisti g o+ two ta%les each co tai i g a si gle recor!. The +irst ta%le, calle! 282

2. 8 Te3t 6rocessi g C)ST7>E:S, has three +iel!s, a c"stomer "m%er EC)ST&)>F, the c"stomerIs +irst ame E0&$>EF a ! the c"stomerIs last ame E*&$>EF, C3*TL3+ MMMMMMMM %./#0$% -L4+& MMMMM ?ohn =L4+& MMMMMM *mith

The seco ! ta%le is calle! 7:'E:S a ! co tai s +o"r +iel!s, a or!er "m%er E7:'E:&)>F, the c"stomer "m%er EC)ST&)>F, the ="a tity ET)$&F, a ! the item or!ere! E9TE>F. R'&RL3+ MMMMMMMM $(#%0"$$(" C3*TL3+ MMMMMMM %./#0$% Z34L AT&+ MMMM MMMM # Hlue ,idget

&ote that %oth ta%les share the +iel! C)ST&)>. This is importa t, as it allows a relatio ship %etwee the ta%les. 6er+ormi g a 4oi operatio wo"l! allow "s to com%i e the +iel!s i the two ta%les to achie#e a "se+"l res"lt, s"ch as prepari g a i #oice. )si g the matchi g #al"es i the C)ST&)> +iel!s o+ %oth ta%les, a 4oi operatio co"l! pro!"ce the +ollowi g, -L4+& MMMMM ?ohn =L4+& MMMMM *mith Z34L AT&+ MMMM MMMM # Hlue ,idget

To !emo strate the join program, weIll ee! to make a co"ple o+ +iles with a share! key. To !o this, we will "se o"r distros-b1-date<txt +ile. 0rom this +ile, we will co str"ct two a!!itio al +iles, o e co tai i g the release !ate Ewhich will %e o"r share! key +or this !emo stratio F a ! the release ame,
[me@linuxbox ~]$ cut -f 1;1 distros-b!-date.txt & distros-names.txt [me@linuxbox ~]$ 7aste distros-dates.txt distros-names.txt & distroske!-names.txt [me@linuxbox ~]$ head distros-ke!-names.txt ##6!"6!((/ -edora #(6$(6!((/ 3buntu (.6#06!((/ *3*& ("6#$6!((/ -edora (%6!%6!((/ 3buntu ##6(/6!(() -edora #(6#/6!(() 3buntu #(6(%6!(() *3*& ("6$#6!(() -edora (%6#06!(() 3buntu

a ! the seco ! +ile, which co tai s the release !ates a ! the #ersio 28K

"m%ers,

Slici g $ ! 'ici g
[me@linuxbox ~]$ cut -f 2;2 distros-b!-date.txt & distros-vernums.txt [me@linuxbox ~]$ paste distros-dates.txt distros-vernums.txt & distro s-ke!-vernums.txt [me@linuxbox ~]$ head distros-ke!-vernums.txt ##6!"6!((/ #( #(6$(6!((/ /<#( (.6#06!((/ ##<( ("6#$6!((/ 0 (%6!%6!((/ /<(% ##6(/6!(() / #(6#/6!(() )<#( #(6(%6!(() #(<$ ("6$#6!(() ) (%6#06!(() )<(%

We ow ha#e two +iles with a share! key Ethe ?release !ate@ +iel!F. 9t is importa t to poi t o"t that the +iles m"st %e sorte! o the key +iel! +or join to work properly.
[me@linuxbox ~]$ join distros-ke!-names.txt 0 head ##6!"6!((/ -edora #( #(6$(6!((/ 3buntu /<#( (.6#06!((/ *3*& ##<( ("6#$6!((/ -edora 0 (%6!%6!((/ 3buntu /<(% ##6(/6!(() -edora / #(6#/6!(() 3buntu )<#( #(6(%6!(() *3*& #(<$ ("6$#6!(() -edora ) (%6#06!(() 3buntu )<(% distros-ke!-vernums.txt

&ote also that, %y !e+a"lt, join "ses whitespace as the i p"t +iel! !elimiter a ! a si gle space as the o"tp"t +iel! !elimiter. This %eha#ior ca %e mo!i+ie! %y speci+yi g optio s. See the join ma page +or !etails.

Comparing Text
9t is o+te "se+"l to compare #ersio s o+ te3t +iles. 0or system a!mi istrators a ! so+tware !e#elopers, this is partic"larly importa t. $ system a!mi istrator may, +or e3ample, ee! to compare a e3isti g co +ig"ratio +ile to a pre#io"s #ersio to !iag ose a system pro%lem. *ikewise, a programmer +re="e tly ee!s to see what cha ges ha#e %ee ma!e to programs o#er time.

28/

2. 8 Te3t 6rocessi g

comm
The comm program compares two te3t +iles a ! !isplays the li es that are " i="e to each o e a ! the li es they ha#e i commo . To !emo strate, we will create two early i!e tical te3t +iles "si g cat,
[me@linuxbox ~]$ cat & file1.txt a b c d [me@linuxbox ~]$ cat & file2.txt b c d e

&e3t, we will compare the two +iles "si g comm,

[me@linuxbox ~]$ comm file1.txt file2.txt a b c d e

$s we ca see, comm pro!"ces three col"m s o+ o"tp"t. The +irst col"m co tai s li es " i="e to the +irst +ile arg"me tH the seco ! col"m , the li es " i="e to the seco ! +ile arg"me tH the thir! col"m co tai s the li es share! %y %oth +iles. comm s"pports optio s i the +orm -n where n is either ., 2 or (. Whe "se!, these optio s speci+y which col"m EsF to s"ppress. 0or e3ample, i+ we o ly wa te! to o"tp"t the li es share! %y %oth +iles, we wo"l! s"ppress the o"tp"t o+ col"m s o e a ! two,
[me@linuxbox ~]$ b c d comm -12 file1.txt file2.txt

diff
*ike the comm program, diff is "se! to !etect the !i++ere ces %etwee +iles. Aowe#er, 288

Compari g Te3t diff is a m"ch more comple3 tool, s"pporti g ma y o"tp"t +ormats a ! the a%ility to process large collectio s o+ te3t +iles at o ce. diff is o+te "se! %y so+tware !e#elopers to e3ami e cha ges %etwee !i++ere t #ersio s o+ program so"rce co!e, a ! th"s has the a%ility to rec"rsi#ely e3ami e !irectories o+ so"rce co!e o+te re+erre! to as source trees. 7 e commo "se +or diff is the creatio o+ diff files or patches that are "se! %y programs s"ch as 7atch Ewhich weIll !isc"ss shortlyF to co #ert o e #ersio o+ a +ile Eor +ilesF to a other #ersio . 9+ we "se diff to look at o"r pre#io"s e3ample +iles,
[me@linuxbox ~]$ diff file1.txt file2.txt #d( X a %a% D e

we see its !e+a"lt style o+ o"tp"t, a terse !escriptio o+ the !i++ere ces %etwee the two +iles. 9 the !e+a"lt +ormat, each gro"p o+ cha ges is prece!e! %y a change command i the +orm o+ range operation range to !escri%e the positio s a ! type o+ cha ges re="ire! to co #ert the +irst +ile to the seco ! +ile, Table '1-$: diff %hange %ommands Change r1ar2 r1cr2 r1dr2 Description $ppe ! the li es at the positio r' i the seco ! +ile to the positio r1 i the +irst +ile. Cha ge EreplaceF the li es at positio r1 with the li es at the positio r' i the seco ! +ile. 'elete the li es i the +irst +ile at positio r1, which wo"l! ha#e appeare! at ra ge r' i the seco ! +ile

9 this +ormat, a ra ge is a comma separate! list o+ the starti g li e a ! the e !i g li e. While this +ormat is the !e+a"lt Emostly +or 67S9G complia ce a ! %ackwar! compati%ility with tra!itio al ) i3 #ersio s o+ diffF, it is ot as wi!ely "se! as other, optio al +ormats. Two o+ the more pop"lar +ormats are the conte+t format a ! the unified format. Whe #iewe! "si g the co te3t +ormat Ethe -c optio F, we will see this,

289

2. 8 Te3t 6rocessi g
[me@linuxbox ~]$ diff -c file1.txt file2.txt EEE file#<txt !((/-#!-!$ (.:%(:#$<((((((((( -("(( --- file!<txt !((/-#!-!$ (.:%(:$%<((((((((( -("(( EEEEEEEEEEEEEEE EEE #>% EEEE - a b c d --- #>% ---b c d 8 e

The o"tp"t %egi s with the ames o+ the two +iles a ! their timestamps. The +irst +ile is marke! with asterisks a ! the seco ! +ile is marke! with !ashes. Thro"gho"t the remai !er o+ the listi g, these markers will sig i+y their respecti#e +iles. &e3t, we see gro"ps o+ cha ges, i cl"!i g the !e+a"lt "m%er o+ s"rro" !i g co te3t li es. 9 the +irst gro"p, we see, EEE #>% EEE which i !icates li es o e thro"gh +o"r i the +irst +ile. *ater we see, --- #>% --which i !icates li es o e thro"gh +o"r i the seco ! +ile. Withi a cha ge gro"p, li es %egi with o e o+ +o"r i !icators, Table '1-.: diff %onte+t Format %hange *ndicators Indicator blank 8 G >eaning $ li e show +or co te3t. 9t !oes ot i !icate a !i++ere ce %etwee the two +iles. $ li e !elete!. This li e will appear i the +irst +ile %"t ot i the seco ! +ile. $ li e a!!e!. This li e will appear i the seco ! +ile %"t ot i the +irst +ile. $ li e cha ge!. The two #ersio s o+ the li e will %e !isplaye!, each i its respecti#e sectio o+ the cha ge gro"p.

The " i+ie! +ormat is similar to the co te3t +ormat, %"t is more co cise. 9t is speci+ie! with the -u optio , 290

Compari g Te3t
[me@linuxbox ~]$ diff -u file1.txt file2.txt --- file#<txt !((/-#!-!$ (.:%(:#$<((((((((( -("(( 888 file!<txt !((/-#!-!$ (.:%(:$%<((((((((( -("(( @@ -#>% 8#>% @@ -a b c d 8e

The most ota%le !i++ere ce %etwee the co te3t a ! " i+ie! +ormats is the elimi atio o+ the !"plicate! li es o+ co te3t, maki g the res"lts o+ the " i+ie! +ormat shorter tha the co te3t +ormat. 9 o"r e3ample a%o#e, we see +ile timestamps like those o+ the co te3t +ormat, +ollowe! %y the stri g @@ -#>% 8#>% @@. This i !icates the li es i the +irst +ile a ! the li es i the seco ! +ile !escri%e! i the cha ge gro"p. 0ollowi g this are the li es themsel#es, with the !e+a"lt three li es o+ co te3t. Each li e starts with o e o+ three possi%le characters, Table '1-2: diff 0nified Format %hange *ndicators Character blank 8 >eaning This li e is share! %y %oth +iles. This li e was remo#e! +rom the +irst +ile. This li e was a!!e! to the +irst +ile.

7atch
The 7atch program is "se! to apply cha ges to te3t +iles. 9t accepts o"tp"t +rom diff a ! is ge erally "se! to co #ert ol!er #ersio o+ +iles i to ewer #ersio s. *etIs co si!er a +amo"s e3ample. The *i "3 ker el is !e#elope! %y a large, loosely orga iDe! team o+ co tri%"tors who s"%mit a co sta t stream o+ small cha ges to the so"rce co!e. The *i "3 ker el co sists o+ se#eral millio li es o+ co!e, while the cha ges that are ma!e %y o e co tri%"tor at o e time are ="ite small. 9it makes o se se +or a co tri%"tor to se ! each !e#eloper a e tire ker el so"rce tree each time a small cha ge is ma!e. 9 stea!, a !i++ +ile is s"%mitte!. The !i++ +ile co tai s the cha ge +rom the pre#io"s #ersio o+ the ker el to the ew #ersio with the co tri%"tor<s cha ges. The recei#er the "ses the 7atch program to apply the cha ge to his ow so"rce tree. )si g diff-7atch o++ers two sig i+ica t a!#a tages, .. The !i++ +ile is #ery small, compare! to the +"ll siDe o+ the so"rce tree. 2. The !i++ +ile co cisely shows the cha ge %ei g ma!e, allowi g re#iewers o+ the 29.

2. 8 Te3t 6rocessi g patch to ="ickly e#al"ate it. 7+ co"rse, diff-7atch will work o a y te3t +ile, ot 4"st so"rce co!e. 9t wo"l! %e e="ally applica%le to co +ig"ratio +iles or a y other te3t. To prepare a !i++ +ile +or "se with 7atch, the ;&) !oc"me tatio Esee 0"rther :ea!i g %elowF s"ggests "si g diff as +ollows, diff -Laur old_file new_file D diff_file Where old file a ! new file are either si gle +iles or !irectories co tai i g +iles. The r optio s"pports rec"rsio o+ a !irectory tree. 7 ce the !i++ +ile has %ee create!, we ca apply it to patch the ol! +ile i to the ew +ile, 7atch X diff_file WeIll !emo strate with o"r test +ile,
[me@linuxbox ~]$ diff -?aur file1.txt file2.txt & patchfile.txt [me@linuxbox ~]$ patch / patchfile.txt 7atching file file#<txt [me@linuxbox ~]$ cat file1.txt b c d e

9 this e3ample, we create! a !i++ +ile ame! 7atchfile<txt a ! the "se! the 7atch program to apply the patch. &ote that we !i! ot ha#e to speci+y a target +ile to 7atch, as the !i++ +ile Ei " i+ie! +ormatF alrea!y co tai s the +ile ames i the hea!er. 7 ce the patch is applie!, we ca see that file#<txt ow matches file!<txt. 7atch has a large "m%er o+ optio s, a ! there are a!!itio al "tility programs that ca %e "se! to a alyDe a ! e!it patches.

$diting ?n The ;l,

7"r e3perie ce with te3t e!itors has %ee largely interactive, mea i g that we ma "ally mo#e a c"rsor aro" !, the type o"r cha ges. Aowe#er, there are non-interactive ways to e!it te3t as well. 9tIs possi%le, +or e3ample, to apply a set o+ cha ges to m"ltiple +iles with a si gle comma !.

tr
The tr program is "se! to transliterate characters. We ca thi k o+ this as a sort o+ 292

E!iti g 7 The 0ly character-%ase! search-a !-replace operatio . Tra sliteratio is the process o+ cha gi g characters +rom o e alpha%et to a other. 0or e3ample, co #erti g characters +rom lowercase to "ppercase is tra sliteratio . We ca per+orm s"ch a co #ersio with tr as +ollows,
[me@linuxbox ~]$ echo Jlowercase lettersJ 0 tr a-- :-@ = ,&RC4*& =&TT&R*

$s we ca see, tr operates o sta !ar! i p"t, a ! o"tp"ts its res"lts o sta !ar! o"tp"t. tr accepts two arg"me ts, a set o+ characters to co #ert +rom a ! a correspo !i g set o+ characters to co #ert to. Character sets may %e e3presse! i o e o+ three ways, .. $ e "merate! list. 0or e3ample, 4HC'&-@SA?2=+L :ZR*T3;,VUI 2. $ character ra ge. 0or e3ample, 4-I. &ote that this metho! is sometimes s"%4ect to the same iss"es as other comma !s, !"e to the locale collatio or!er, a ! th"s sho"l! %e "se! with ca"tio . (. 67S9G character classes. 0or e3ample, [:u77er:]. 9 most cases, %oth character sets sho"l! %e o+ e="al le gthH howe#er, it is possi%le +or the +irst set to %e larger tha the seco !, partic"larly i+ we wish to co #ert m"ltiple characters to a si gle character,
[me@linuxbox ~]$ echo Jlowercase lettersJ 0 tr (2lower2) : 444444444 4444444

9 a!!itio to tra sliteratio , tr allows characters to simply %e !elete! +rom the i p"t stream. Earlier i this chapter, we !isc"sse! the pro%lem o+ co #erti g >S-'7S te3t +iles to ) i3 style te3t. To per+orm this co #ersio , carriage ret"r characters ee! to %e remo#e! +rom the e ! o+ each li e. This ca %e per+orme! with tr as +ollows, tr -d K]rK X dos_file D unix_file where dos file is the +ile to %e co #erte! a ! uni+ file is the res"lt. This +orm o+ the comma ! "ses the escape se="e ce ]r to represe t the carriage ret"r character. To see a complete list o+ the se="e ces a ! character classes tr s"pports, try,
[me@linuxbox ~]$ tr --help

29(

2. 8 Te3t 6rocessi g

1?T1!I The 7otESoESecret Decoder 1ing

7 e am"si g "se o+ tr is to per+orm 9&T13 encoding o+ te3t. :7T.( is a tri#ial type o+ e cryptio %ase! o a simple s"%stit"tio cipher. Calli g :7T.( ?e cryptio @ is %ei g ge ero"sH ?te3t o%+"scatio @ is more acc"rate. 9t is "se! sometimes o te3t to o%sc"re pote tially o++e si#e co te t. The metho! simply mo#es each character thirtee places "p the alpha%et. Si ce this is hal+ way "p the possi%le twe ty-si3 characters, per+ormi g the algorithm a seco ! time o the te3t restores it to its origi al +orm. To per+orm this e co!i g with tr,
echo Jsecret textJ 0 tr a--:-@ n--a-m?-@:-O fr7erg grkg

6er+ormi g the same proce!"re a seco ! time res"lts i the tra slatio ,
echo Jfrperg grkgJ 0 tr a--:-@ n--a-m?-@:-O secret text

$ "m%er o+ email programs a ! )SE&ET ews rea!ers s"pport :7T.( e co!i g. Wikipe!ia co tai s a goo! article o the s"%4ect, http,--e .wikipe!ia.org-wiki-:7T.(

tr ca per+orm a other trick, too. )si g the -s optio , tr ca ?s="eeDe@ E!eleteF repeate! i sta ces o+ a character,
[me@linuxbox ~]$ echo JaaabbbcccJ 0 tr -s ab abccc

Aere we ha#e a stri g co tai i g repeate! characters. Jy speci+yi g the set ?a%@ to tr, we elimi ate the repeate! i sta ces o+ the letters i the set, while lea#i g the character that is missi g +rom the set E?c@F " cha ge!. &ote that the repeati g characters m"st %e a!4oi i g. 9+ they are ot,
[me@linuxbox ~]$ echo JabcabcabcJ 0 tr -s ab abcabcabc

the s="eeDi g will ha#e o e++ect.

291

E!iti g 7 The 0ly

sed
The ame sed is short +or stream editor. 9t per+orms te3t e!iti g o a stream o+ te3t, either a set o+ speci+ie! +iles or sta !ar! i p"t. sed is a power+"l a ! somewhat comple3 program Ethere are e tire %ooks a%o"t itF, so we will ot co#er it completely here. 9 ge eral, the way that sed works is that it is gi#e either a si gle e!iti g comma ! Eo the comma ! li eF or the ame o+ a script +ile co tai i g m"ltiple comma !s, a ! it the per+orms these comma !s "po each li e i the stream o+ te3t. Aere is a #ery simple e3ample o+ sed i actio ,
[me@linuxbox ~]$ echo JfrontJ 0 sed %s/front/back/% back

9 this e3ample, we pro!"ce a o e wor! stream o+ te3t "si g echo a ! pipe it i to sed. sed, i t"r , carries o"t the i str"ctio s6front6back6 "po the te3t i the stream a ! pro!"ces the o"tp"t ?%ack@ as a res"lt. We ca also recog iDe this comma ! as resem%li g the ?s"%stit"tio @ Esearch a ! replaceF comma ! i vi. Comma !s i sed %egi with a si gle letter. 9 the e3ample a%o#e, the s"%stit"tio comma ! is represe te! %y the letter s a ! is +ollowe! %y the search a ! replace stri gs, separate! %y the slash character as a !elimiter. The choice o+ the !elimiter character is ar%itrary. Jy co #e tio , the slash character is o+te "se!, %"t sed will accept a y character that imme!iately +ollows the comma ! as the !elimiter. We co"l! per+orm the same comma ! this way,
[me@linuxbox ~]$ echo JfrontJ 0 sed %s.front.back.% back

Jy "si g the " !erscore character imme!iately a+ter the comma !, it %ecomes the !elimiter. The a%ility to set the !elimiter ca %e "se! to make comma !s more rea!a%le, as we shall see. >ost comma !s i sed may %e prece!e! %y a address, which speci+ies which li eEsF o+ the i p"t stream will %e e!ite!. 9+ the a!!ress is omitte!, the the e!iti g comma ! is carrie! o"t o e#ery li e i the i p"t stream. The simplest +orm o+ a!!ress is a li e "m%er. We ca a!! o e to o"r e3ample,
[me@linuxbox ~]$ echo JfrontJ 0 sed %1s/front/back/% back

292

2. 8 Te3t 6rocessi g $!!i g the a!!ress # to o"r comma ! ca"ses o"r s"%stit"tio to %e per+orme! o the +irst li e o+ o"r o e-li e i p"t stream. 9+ we speci+y a other "m%er,
[me@linuxbox ~]$ echo JfrontJ 0 sed %2s/front/back/% front

we see that the e!iti g is ot carrie! o"t, si ce o"r i p"t stream !oes ot ha#e a li e two. $!!resses may %e e3presse! i ma y ways. Aere are the most commo , Table '1-3: sed )ddress >otation #ddress n $ 6regexp6 Description $ li e "m%er where n is a positi#e i teger. The last li e. *i es matchi g a 67S9G %asic reg"lar e3pressio . &ote that the reg"lar e3pressio is !elimite! %y slash characters. 7ptio ally, the reg"lar e3pressio may %e !elimite! %y a alter ate character, %y speci+yi g the e3pressio with ]cregexpc, where c is the alter ate character. $ ra ge o+ li es +rom addr1 to addr', i cl"si#e. $!!resses may %e a y o+ the si gle a!!ress +orms a%o#e. >atch the li e represe te! %y the "m%er first, the each s"%se="e t li e at step i ter#als. 0or e3ample .S2 re+ers to each o!! "m%ere! li e, 2S2 re+ers to the +i+th li e a ! e#ery +i+th li e therea+ter. >atch addr1 a ! the +ollowi g n li es. >atch all li es e3cept addr, which may %e a y o+ the +orms a%o#e.

addr1>addr2 first~step

addr1>8n addrL

WeIll !emo strate !i++ere t ki !s o+ a!!resses "si g the distros<txt +ile +rom earlier i this chapter. 0irst, a ra ge o+ li e "m%ers,
[me@linuxbox ~]$ sed -n %1;"p% distros.txt *3*& #(<! #!6()6!((. -edora #( ##6!"6!((/ *3*& ##<( (.6#06!((/

29K

E!iti g 7 The 0ly

3buntu -edora /<(% / (%6!%6!((/ ##6(/6!(()

9 this e3ample, we pri t a ra ge o+ li es, starti g with li e o e a ! co ti "i g to li e +i#e. To !o this, we "se the 7 comma !, which simply ca"ses a matche! li e to %e pri te!. 0or this to %e e++ecti#e howe#er, we m"st i cl"!e the optio -n Ethe o a"topri t optio F to ca"se sed ot to pri t e#ery li e %y !e+a"lt. &e3t, weIll try a reg"lar e3pressio ,
[me@linuxbox ~]$ sed -n %/FEFG/p% distros.txt *3*& #(<! #!6()6!((. *3*& ##<( (.6#06!((/ *3*& #(<$ #(6(%6!(() *3*& #(<# ("6##6!((.

Jy i cl"!i g the slash-!elimite! reg"lar e3pressio 6*3*&6, we are a%le to isolate the li es co tai i g it i m"ch the same ma er as gre7. 0i ally, weIll try egatio %y a!!i g a G to the a!!ress,
[me@linuxbox ~]$ sed -n %/FEFG/Lp% distros.txt -edora #( ##6!"6!((/ 3buntu /<(% (%6!%6!((/ -edora / ##6(/6!(() 3buntu .<#( #(6!.6!((. -edora ) ("6$#6!(() 3buntu )<#( #(6#/6!(() 3buntu )<(% (%6#06!(() -edora . #(6!%6!((. -edora 0 ("6#$6!((/ 3buntu .<(. (.6(#6!((. 3buntu /<#( #(6$(6!((/ -edora " ($6!(6!((.

Aere we see the e3pecte! res"lt, all o+ the li es i the +ile e3cept the o es matche! %y the reg"lar e3pressio . So +ar, weI#e looke! at two o+ the sed e!iti g comma !s, s a ! 7. Aere is a more complete list o+ the %asic e!iti g comma !s,

29/

2. 8 Te3t 6rocessi g Table '1-4: sed 8asic 1diting %ommands Command M a d i 7 Description 7"tp"t c"rre t li e "m%er. $ppe ! te3t a+ter the c"rre t li e. 'elete the c"rre t li e. 9 sert te3t i +ro t o+ the c"rre t li e. 6ri t the c"rre t li e. Jy !e+a"lt, sed pri ts e#ery li e a ! o ly e!its li es that match a speci+ie! a!!ress withi the +ile. The !e+a"lt %eha#ior ca %e o#erri!!e %y speci+yi g the -n optio . E3it sed witho"t processi g a y more li es. 9+ the -n optio is ot speci+ie!, o"tp"t the c"rre t li e. E3it sed witho"t processi g a y more li es. S"%stit"te the co te ts o+ replacement where#er rege+p is +o" !. replacement may i cl"!e the special character Y, which is e="i#ale t to the te3t matche! %y rege+p. 9 a!!itio , replacement may i cl"!e the se="e ces ]# thro"gh ]0, which are the co te ts o+ the correspo !i g s"%e3pressio s i rege+p. 0or more a%o"t this, see the !isc"ssio o+ back references %elow. $+ter the traili g slash +ollowi g replacement, a optio al +lag may %e speci+ie! to mo!i+y the s comma !Is %eha#ior. 6er+orm tra sliteratio %y co #erti g characters +rom set1 to the correspo !i g characters i set'. &ote that " like tr, sed re="ires that %oth sets %e o+ the same le gth.

Z s6regexp6replacement6

16set16set2

The s comma ! is %y +ar the most commo ly "se! e!iti g comma !. We will !emo strate 4"st some o+ its power %y per+ormi g a e!it o o"r distros<txt +ile. We !isc"sse! %e+ore how the !ate +iel! i distros<txt was ot i a ?comp"ter+rie !ly@ +ormat. While the !ate is +ormatte! >>-''-5555, it wo"l! %e %etter E+or ease o+ sorti gF i+ the +ormat were 5555->>-''. To per+orm this cha ge o the +ile %y ha ! wo"l! %e %oth time-co s"mi g a ! error pro e, %"t with sed, this cha ge ca 298

E!iti g 7 The 0ly %e per+orme! i o e step,

[me@linuxbox ~]$ sed %s/K4(*-D)K92K>K6K/K4(*-D)K92K>K6K/K4(*-D)K9AK>K 63/K -K1-K2/% distros.txt *3*& #(<! !((.-#!-() -edora #( !((/-##-!" *3*& ##<( !((/-(.-#0 3buntu /<(% !((/-(%-!% -edora / !(()-##-(/ *3*& #(<$ !(()-#(-(% 3buntu .<#( !((.-#(-!. -edora ) !(()-("-$# 3buntu )<#( !(()-#(-#/ 3buntu )<(% !(()-(%-#0 *3*& #(<# !((.-("-## -edora . !((.-#(-!% -edora 0 !((/-("-#$ 3buntu .<(. !((.-(.-(# 3buntu /<#( !((/-#(-$( -edora " !((.-($-!(

WowL &ow that is a "gly looki g comma !. J"t it works. 9 4"st o e step, we ha#e cha ge! the !ate +ormat i o"r +ile. 9t is also a per+ect e3ample o+ why reg"lar e3pressio s are sometimes 4oki gly re+erre! to as a ?write-o ly@ me!i"m. We ca write them, %"t we sometimes ca ot rea! them. Je+ore we are tempte! to r" away i terror +rom this comma !, letIs look at how it was co str"cte!. 0irst, we k ow that the comma ! will ha#e this %asic str"ct"re,
sed %s/regexp/replacement/% distros.txt

7"r e3t step is to +ig"re o"t a reg"lar e3pressio that will isolate the !ate. Si ce it is i >>-''-5555 +ormat a ! appears at the e ! o+ the li e, we ca "se a e3pressio like this,
(*-D)92>/(*-D)92>/(*-D)9A>3

which matches two !igits, a slash, two !igits, a slash, +o"r !igits, a ! the e ! o+ li e. So that takes care o+ rege+p, %"t what a%o"t replacementC To ha !le that, we m"st i tro!"ce a ew reg"lar e3pressio +eat"re that appears i some applicatio s which "se J:E. This +eat"re is calle! back references a ! works like this, i+ the se="e ce \n appears i replacement where n is a "m%er +rom o e to i e, the se="e ce will re+er to the 299

2. 8 Te3t 6rocessi g correspo !i g s"%e3pressio i the prece!i g reg"lar e3pressio . s"%e3pressio s, we simply e close them i pare theses like so,
4(*-D)92>6/4(*-D)92>6/4(*-D)9A>63

To create the

We ow ha#e three s"%e3pressio s. The +irst co tai s the mo th, the seco ! co tai s the !ay o+ the mo th, a ! the thir! co tai s the year. &ow we ca co str"ct replacement as +ollows,
K -K1-K2

which gi#es "s the year, a !ash, the mo th, a !ash, a ! the !ay. &ow, o"r comma ! looks like this,
sed %s/4(*-D)92>6/4(*-D)92>6/4(*-D)9A>63/K -K1-K2/% distros.txt

We ha#e two remai i g pro%lems. The +irst is that the e3tra slashes i o"r reg"lar e3pressio will co +"se sed whe it tries to i terpret the s comma !. The seco ! is that si ce sed, %y !e+a"lt, accepts o ly %asic reg"lar e3pressio s, se#eral o+ the characters i o"r reg"lar e3pressio will %e take as literals, rather tha as metacharacters. We ca sol#e %oth these pro%lems with a li%eral applicatio o+ %ackslashes to escape the o++e !i g characters,
sed %s/K4(*-D)K92K>K6K/K4(*-D)K92K>K6K/K4(*-D)K9AK>K63/K -K1-K2/% dis tros.txt

$ ! there yo" ha#e itL $ other +eat"re o+ the s comma ! is the "se o+ optio al +lags that may +ollow the replaceme t stri g. The most importa t o+ these is the g +lag, which i str"cts sed to apply the search a ! replace glo%ally to a li e, ot 4"st to the +irst i sta ce, which is the !e+a"lt. Aere is a e3ample,
[me@linuxbox ~]$ echo JaaabbbcccJ 0 sed %s/b/</% aaaHbbccc

(00

E!iti g 7 The 0ly We see that the replaceme t was per+orme!, %"t o ly to the +irst i sta ce o+ the letter ?%,@ while the remai i g i sta ces were le+t " cha ge!. Jy a!!i g the g +lag, we are a%le to cha ge all the i sta ces,
[me@linuxbox ~]$ aaaHHHccc echo JaaabbbcccJ 0 sed %s/b/</g%

So +ar, we ha#e o ly gi#e sed si gle comma !s #ia the comma ! li e. 9t is also possi%le to co str"ct more comple3 comma !s i a script +ile "si g the -f optio . To !emo strate, we will "se sed with o"r distros<txt +ile to %"il! a report. 7"r report will +eat"re a title at the top, o"r mo!i+ie! !ates, a ! all the !istri%"tio ames co #erte! to "pper case. To !o this, we will ee! to write a script, so weIll +ire "p o"r te3t e!itor a ! e ter the +ollowi g,
U sed script to produce Tinux distributions report 1 iK K Tinux 1istributions HeportK s/K4(*-D)K92K>K6K/K4(*-D)K92K>K6K/K4(*-D)K9AK>K63/K -K1-K2/ !/abcdefghijklmnop,rstuvwx!-/:<=1G7QRPYXTO?SB\HF+E]WZ^@/

We will sa#e o"r sed script as distros<sed a ! r" it like this,

[me@linuxbox ~]$ sed -f distros.sed distros.txt =inux 'istributions Re7ort *3*& -&' R4 *3*& 3H3LT3 -&' R4 *3*& 3H3LT3 -&' R4 3H3LT3 3H3LT3 *3*& -&' R4 -&' R4 #(<! #( ##<( /<(% / #(<$ .<#( ) )<#( )<(% #(<# . 0 !((.-#!-() !((/-##-!" !((/-(.-#0 !((/-(%-!% !(()-##-(/ !(()-#(-(% !((.-#(-!. !(()-("-$# !(()-#(-#/ !(()-(%-#0 !((.-("-## !((.-#(-!% !((/-("-#$

(0.

2. 8 Te3t 6rocessi g
3H3LT3 3H3LT3 -&' R4 .<(. /<#( " !((.-(.-(# !((/-#(-$( !((.-($-!(

$s we ca see, o"r script pro!"ces the !esire! res"lts, %"t how !oes is !o itC *etIs take a other look at o"r script. WeIll "se cat to "m%er the li es,
[me@linuxbox ~]$ cat -n distros.sed # ^ sed scri7t to 7roduce =inux distributions re7ort ! $ # i] % ] " =inux 'istributions Re7ort] . ) s6]N[(-0]][!]\]P]6]N[(-0]][!]\]P]6]N[(-0]][%]\]P$6]$-]#-]!6 / 16abcdefghijklmno7Brstuv9x1C64HC'&-@SA?2=+L :ZR*T3;,VUI6

*i e o e o+ o"r script is a comment. *ike ma y co +ig"ratio +iles a ! programmi g la g"ages o *i "3 systems, comme ts %egi with the ^ character a ! are +ollowe! %y h"ma -rea!a%le te3t. Comme ts ca %e place! a ywhere i the script Etho"gh ot withi comma !s themsel#esF a ! are help+"l to a y h"ma s who might ee! to i!e ti+y a !-or mai tai the script. *i e two is a %la k li e. rea!a%ility. *ike comme ts, %la k li es may %e a!!e! to impro#e

>a y sed comma !s s"pport li e a!!resses. These are "se! to speci+y which li es o+ the i p"t are to %e acte! "po . *i e a!!resses may %e e3presse! as si gle li e "m%ers, li e "m%er ra ges, a ! the special li e "m%er ?[@ which i !icates the last li e o+ i p"t. *i es three thro"gh si3 co tai te3t to %e i serte! at the a!!ress ., the +irst li e o+ the i p"t. The i comma ! is +ollowe! %y the se="e ce %ackslash-carriage ret"r to pro!"ce a escape! carriage ret"r , or what is calle! a line continuation character. This se="e ce, which ca %e "se! i ma y circ"msta ces i cl"!i g shell scripts, allows a carriage ret"r to %e em%e!!e! i a stream o+ te3t witho"t sig ali g the i terpreter Ei this case sedF that the e ! o+ the li e has %ee reache!. The i, a ! likewise, the a Ewhich appe !s te3t, rather tha i serti g itF a ! c Ewhich replaces te3tF comma !s, allow m"ltiple li es o+ te3t as lo g as each li e, e3cept the last, e !s with a li e co ti "atio character. The si3th li e o+ o"r script is act"ally the e ! o+ o"r i serte! te3t a ! e !s with a plai carriage ret"r rather tha a li e co ti "atio character, sig ali g the e ! o+ the i comma !.

(02

E!iti g 7 The 0ly 1ote5 $ li e co ti "atio character is +orme! %y a %ackslash +ollowe! immediately %y a carriage ret"r . &o i terme!iary spaces are permitte!. *i e se#e is o"r search a ! replace comma !. Si ce it is ot prece!e! %y a a!!ress, each li e i the i p"t stream is s"%4ect to its actio . *i e eight per+orms tra sliteratio o+ the lowercase letters i to "ppercase letters. &ote that " like tr, the 1 comma ! i sed !oes ot s"pport character ra ges E+or e3ample, [a-C]F, or !oes it s"pport 67S9G character classes. $gai , si ce the 1 comma ! is ot prece!e! %y a a!!ress, it applies to e#ery li e i the i p"t stream.

People +ho Li(e sed #lso Li(e...

sed is a #ery capa%le program, a%le to per+orm +airly comple3 e!iti g tasks to streams o+ te3t. 9t is most o+te "se! +or simple o e li e tasks rather tha lo g scripts. >a y "sers pre+er other tools +or larger tasks. The most pop"lar o+ these are a9k a ! 7erl. These go %eyo ! mere tools, like the programs co#ere! here, a ! e3te ! i to the realm o+ complete programmi g la g"ages. 7erl, i partic"lar, is o+te "se! i place o+ shell scripts +or ma y system ma ageme t a ! a!mi istratio tasks, as well as %ei g a #ery pop"lar me!i"m +or we% !e#elopme t. a9k is a little more specialiDe!. 9ts speci+ic stre gth is its a%ility to ma ip"late ta%"lar !ata. 9t resem%les sed i that a9k programs ormally process te3t +iles li e-%y-li e, "si g a scheme similar to the sed co cept o+ a a!!ress +ollowe! %y a actio . While %oth a9k a ! 7erl are o"tsi!e the scope o+ this %ook, they are #ery goo! skills +or the *i "3 comma ! li e "ser.

as7ell
The last tool we will look at is as7ell, a i teracti#e spelli g checker. The as7ell program is the s"ccessor to a earlier program ame! is7ell, a ! ca %e "se!, +or the most part, as a !rop-i replaceme t. While the as7ell program is mostly "se! %y other programs that re="ire spell checki g capa%ility, it ca also %e "se! #ery e++ecti#ely as a sta !-alo e tool +rom the comma ! li e. 9t has the a%ility to i tellige tly check #ario"s type o+ te3t +iles, i cl"!i g AT>* !oc"me ts, C-CVV programs, email messages a ! other ki !s o+ specialiDe! te3ts. To spell check a te3t +ile co tai i g simple prose, it co"l! %e "se! like this,

(0(

2. 8 Te3t 6rocessi g
as7ell check textfile

where te+tfile is the ame o+ the +ile to check. $s a practical e3ample, letIs create a simple te3t +ile ame! foo<txt co tai i g some !eli%erate spelli g errors,
[me@linuxbox ~]$ cat & foo.txt +he ,uick brown fox jimped over the lax! dog.

&e3t weIll check the +ile "si g as7ell,

[me@linuxbox ~]$ aspell check foo.txt

$s as7ell is i teracti#e i the check mo!e, we will see a scree like this,
The Buick bro9n fox jim7ed over the lax1 dog< #P !P $P %P "P iP rP aP bP F jum7ed gim7ed com7ed lim7ed 7im7ed Agnore Re7lace 4dd 4bort .P )P /P 0P (P AP RP lP xP 9im7ed cam7ed hum7ed im7ede um7ed Agnore all Re7lace all 4dd =o9er &xit

$t the top o+ the !isplay, we see o"r te3t with a s"spicio"sly spelle! wor! highlighte!. 9 the mi!!le, we see te spelli g s"ggestio s "m%ere! Dero thro"gh i e, +ollowe! %y a list o+ other possi%le actio s. 0i ally, at the #ery %ottom, we see a prompt rea!y to accept o"r choice. 9+ we press the # key, as7ell replaces the o++e !i g wor! with the wor! ?4"mpe!@ a ! mo#es o to the e3t misspelle! wor! which is ?la3y.@ 9+ we select the replaceme t ?laDy,@ as7ell replaces it a ! termi ates. 7 ce as7ell has +i ishe!, we ca e3ami e o"r +ile a ! see that the misspelli gs ha#e %ee correcte!,

(01

E!iti g 7 The 0ly

[me@linuxbox ~]$ cat foo.txt The Buick bro9n fox jum7ed over the laC1 dog<

) less tol! otherwise #ia the comma ! li e optio --dont-backu7, as7ell creates a %ack"p +ile co tai i g the origi al te3t %y appe !i g the e3te sio <bak to the +ile ame. Showi g o++ o"r sed e!iti g prowess, weIll p"t o"r spelli g mistakes %ack i so we ca re"se o"r +ile,
[me@linuxbox ~]$ sed -i %s/la-!/lax!/# s/jumped/jimped/% foo.txt

The sed optio -i tells sed to e!it the +ile ?i -place,@ mea i g that rather tha se !i g the e!ite! o"tp"t to sta !ar! o"tp"t, it will re-write the +ile with the cha ges applie!. We also see the a%ility to place more tha o e e!iti g comma ! o the li e %y separati g them with a semicolo . &e3t, weIll look at how as7ell ca ha !le !i++ere t ki !s o+ te3t +iles. )si g a te3t e!itor s"ch as vim Ethe a!#e t"ro"s may wa t to try sedF, we will a!! some AT>* mark"p to o"r +ile,
/html& /head& /title&Oispelled R+OT file//title& //head& /bod!& /p&The Buick bro9n fox jim7ed over the lax1 dog<//p& //bod!& //html&

&ow, i+ we try to spell check o"r mo!i+ie! +ile, we r" i to a pro%lem. 9+ we !o it this way,
[me@linuxbox ~]$ aspell check foo.txt

weIll get this,

XhtmlD

(02

2. 8 Te3t 6rocessi g
XheadD X6headD Xbod1D X6htmlD #P ST+= !P ht ml $P ht-ml iP rP aP bP F Agnore Re7lace 4dd 4bort X6bod1D XtitleD+is7elled ST+= fileX6titleD X7DThe Buick bro9n fox jim7ed over the lax1 dog<X67D

%P Samel "P Samil .P hotel AP RP lP xP Agnore all Re7lace all 4dd =o9er &xit

as7ell will see the co te ts o+ the AT>* tags as misspelle!. This pro%lem ca %e o#ercome %y i cl"!i g the -S EAT>*F checki g mo!e optio , like this,
[me@linuxbox ~]$ aspell -R check foo.txt

which will res"lt i this,

XhtmlD

XheadD X6headD Xbod1D XtitleD+is7elled ST+= fileX6titleD X7DThe Buick bro9n fox jim7ed over the lax1 dog<X67D

X6htmlD #P !P $P %P "P iP rP

X6bod1D

+i s7elled +i-s7elled +iss7elled 'is7elled *7elled Agnore Re7lace

.P )P /P 0P (P AP RP

+isa77lied +iscalled Res7elled +iss7ell +isled Agnore all Re7lace all

(0K

E!iti g 7 The 0ly

aP 4dd bP 4bort F lP 4dd =o9er xP &xit

The AT>* is ig ore! a ! o ly the o -mark"p portio s o+ the +ile are checke!. 9 this mo!e, the co te ts o+ AT>* tags are ig ore! a ! ot checke! +or spelli g. Aowe#er, the co te ts o+ 4=T tags, which %e e+it +rom checki g, are checke! i this mo!e. 1ote5 Jy !e+a"lt, as7ell will ig ore ):*s a ! email a!!resses i te3t. This %eha#ior ca %e o#erri!!e with comma ! li e optio s. 9t is also possi%le to speci+y which mark"p tags are checke! a ! skippe!. See the as7ell ma page +or !etails.

Summing -p
9 this chapter, we ha#e looke! at a +ew o+ the ma y comma ! li e tools that operate o te3t. 9 the e3t chapter, we will look at se#eral more. $!mitte!ly, it may ot seem imme!iately o%#io"s how or why yo" might "se some o+ these tools o a !ay-to-!ay %asis, tho"gh we ha#e trie! to show some semi-practical e3amples o+ their "se. We will +i ! i later chapters that these tools +orm the %asis o+ a tool set that is "se! to sol#e a host o+ practical pro%lems. This will %e partic"larly tr"e whe we get i to shell scripti g, where these tools will really show their worth.

;urther 1eading
The ;&) 6ro4ect we%site co tai s ma y o li e g"i!es to the tools !isc"sse! i this chapter.

0rom the Core"tils package, http,--www.g ".org-so+tware-core"tils-ma +iles http,--www.g ".org-so+tware-core"tils-ma sorte!-+iles http,--www.g ".org-so+tware-core"tils-ma withi -a-li e http,--www.g ".org-so+tware-core"tils-ma characters

"al-core"tils.htmlP7"tp"t-o+-e tire"al-core"tils.htmlP7perati g-o "al-core"tils.htmlP7perati g-o -+iel!s"al-core"tils.htmlP7perati g-o -

0rom the 'i++"tils package, http,--www.g ".org-so+tware-!i++"tils-ma "al-htmlOmo o-!i++.html (0/

2. 8 Te3t 6rocessi g

sed, http,--www.g ".org-so+tware-se!-ma "al-se!.html as7ell, http,--aspell. et-ma -html-i !e3.html There are ma y other o li e reso"rces +or sed, i partic"lar, http,--www.grymoire.com-) i3-Se!.html http,--se!.so"rce+orge. et-se!.li e.t3t $lso try googli g ?se! o e li ers@, ?se! cheat sheets@

$xtra Credit
There are a +ew more i teresti g te3t ma ip"latio comma !s worth i #estigati g. $mo g these are, s7lit Esplit +iles i to piecesF, cs7lit Esplit +iles i to pieces %ase! o co te3tF, a ! sdiff Esi!e-%y-si!e merge o+ +ile !i++ere ces.F

(08

22 8 0ormatti g 7"tp"t

22 (or%atting 7ut#ut
9 this chapter, we co ti "e o"r look at te3t relate! tools, +oc"si g o programs that are "se! to +ormat te3t o"tp"t, rather tha cha gi g the te3t itsel+. These tools are o+te "se! to prepare te3t +or e#e t"al pri ti g, a s"%4ect that we will co#er i the e3t chapter. The programs that we will co#er i this chapter i cl"!e,

nl 8 &"m%er li es fold 8 Wrap each li e to a speci+ie! le gth fmt 8 $ simple te3t +ormatter 7r 8 6repare te3t +or pri ti g 7rintf 8 0ormat a ! pri t !ata groff 8 $ !oc"me t +ormatti g system

Simple ;ormatting Tools

WeIll look at some o+ the simple +ormatti g tools +irst. These are mostly si gle p"rpose programs, a ! a %it " sophisticate! i what they !o, %"t they ca %e "se! +or small tasks a ! as parts o+ pipeli es a ! scripts.

nl A ,umber Lines
The nl program is a rather arca e tool "se! to per+orm a simple task. 9t "m%ers li es. 9 its simplest "se, it resem%les cat -n,
[me@linuxbox ~]$ nl distros.txt 0 head # *3*& #(<! #!6()6!((. ! -edora #( ##6!"6!((/ $ *3*& ##<( (.6#06!((/ % 3buntu /<(% (%6!%6!((/ " -edora / ##6(/6!(() . *3*& #(<$ #(6(%6!(() ) 3buntu .<#( #(6!.6!((.

(09

22 8 0ormatti g 7"tp"t
/ 0 #( -edora 3buntu 3buntu ) )<#( )<(% ("6$#6!(() #(6#/6!(() (%6#06!(()

*ike cat, nl ca accept either m"ltiple +iles as comma ! li e arg"me ts, or sta !ar! i p"t. Aowe#er, nl has a "m%er o+ optio s a ! s"pports a primiti#e +orm o+ mark"p to allow more comple3 ki !s o+ "m%eri g. nl s"pports a co cept calle! ?logical pages@ whe "m%eri g. This allows nl to reset Estart o#erF the "merical se="e ce whe "m%eri g. )si g optio s, it is possi%le to set the starti g "m%er to a speci+ic #al"e a !, to a limite! e3te t, its +ormat. $ logical page is +"rther %roke !ow i to a hea!er, %o!y, a ! +ooter. Withi each o+ these sectio s, li e "m%eri g may %e reset a !-or %e assig e! a !i++ere t style. 9+ nl is gi#e m"ltiple +iles, it treats them as a si gle stream o+ te3t. Sectio s i the te3t stream are i !icate! %y the prese ce o+ some rather o!!-looki g mark"p a!!e! to the te3t, Table ''-1: nl !arkup >ar(up ]:]:]: ]:]: ]: >eaning Start o+ logical page hea!er Start o+ logical page %o!y Start o+ logical page +ooter

Each o+ the a%o#e mark"p eleme ts m"st appear alo e o its ow li e. $+ter processi g a mark"p eleme t, nl !eletes it +rom the te3t stream. Aere are the commo optio s +or nl, Table ''-': %ommon nl &ptions ?ption -b st le >eaning Set %o!y "m%eri g to style, where style is o e o+ the +ollowi g, a W "m%er all li es t W "m%er o ly o -%la k li es. This is the !e+a"lt. nW o e 7regexp W "m%er o ly li es matchi g %asic reg"lar e3pressio rege+p. Set +ooter "m%eri g to style. 'e+a"lt is Set hea!er "m%eri g to style. 'e+a"lt is E o eF. E o eF.

-f st le -h st le

(.0

Simple 0ormatti g Tools -i number -n format Set page "m%eri g i creme t to number. 'e+a"lt is o e. Sets "m%eri g +ormat to format, where +ormat is, ln W le+t 4"sti+ie!, witho"t lea!i g Deros. rn W right 4"sti+ie!, witho"t lea!i g Deros. This is the !e+a"lt. rC W right 4"sti+ie!, with lea!i g Deros. 'o ot reset page "m%eri g at the %egi i g o+ each logical page. $!! string to the e ! o+ each li e "m%er to create a separator. 'e+a"lt is a si gle ta% character. Set +irst li e "m%er o+ each logical page to number. 'e+a"lt is o e. Set wi!th o+ the li e "m%er +iel! to width. 'e+a"lt is si3.

-7 -s string -v number -9 width

$!mitte!ly, we pro%a%ly wo It %e "m%eri g li es that o+te , %"t we ca "se nl to look at how we ca com%i e m"ltiple tools to per+orm more comple3 tasks. We will %"il! o o"r work i the pre#io"s chapter to pro!"ce a *i "3 !istri%"tio s report. Si ce we will %e "si g nl, it will %e "se+"l to i cl"!e its hea!er-%o!y-+ooter mark"p. To !o this, we will a!! it to the sed script +rom the last chapter. )si g o"r te3t e!itor, we will cha ge the script as +ollows a ! sa#e it as distros-nl<sed,
^ sed scri7t to 7roduce =inux distributions re7ort # i] KK2KK2KK2K ] =inux 'istributions Re7ort] K ?ame ]er. HeleasedK ------- --------K KK2KK2 s6]N[(-0]][!]\]P]6]N[(-0]][!]\]P]6]N[(-0]][%]\]P$6]$-]#-]!6 3 iK KK2K K Gnd Sf Heport

The script ow i serts the nl logical page mark"p a ! a!!s a +ooter at the e ! o+ the report. &ote that we ha! to !o"%le "p the %ackslashes i o"r mark"p, %eca"se they are ormally i terprete! as a escape character %y sed. &e3t, weIll pro!"ce o"r e ha ce! report %y com%i i g sort, sed, a ! nl, (..

22 8 0ormatti g 7"tp"t
[me@linuxbox ~]$ sort -k 1;1 -k 2n distros.txt 0 sed -f distros-nl.s ed 0 nl =inux 'istributions Re7ort Lame ---# ! $ % " . ) / 0 #( ## #! #$ #% #" #. -edora -edora -edora -edora -edora -edora *3*& *3*& *3*& *3*& 3buntu 3buntu 3buntu 3buntu 3buntu 3buntu ;er< ---" . ) / 0 #( #(<# #(<! #(<$ ##<( .<(. .<#( )<(% )<#( /<(% /<#( Released -------!((.-($-!( !((.-#(-!% !(()-("-$# !(()-##-(/ !((/-("-#$ !((/-##-!" !((.-("-## !((.-#!-() !(()-#(-(% !((/-(.-#0 !((.-(.-(# !((.-#(-!. !(()-(%-#0 !(()-#(-#/ !((/-(%-!% !((/-#(-$(

&nd

f Re7ort

7"r report is the res"lt o+ o"r pipeli e o+ comma !s. 0irst, we sort the list %y !istri%"tio ame a ! #ersio E+iel!s o e a ! twoF, the we process the res"lts with sed, a!!i g the report hea!er Ei cl"!i g the logical page mark"p +or nlF a ! +ooter. 0i ally, we process the res"lt with nl, which, %y !e+a"lt, o ly "m%ers the li es o+ the te3t stream that %elo g to the %o!y sectio o+ the logical page. We ca repeat the comma ! a ! e3perime t with !i++ere t optio s +or nl. i teresti g o es are,
nl -n r-

Some

a !
nl -w -s % %

(.2

Simple 0ormatti g Tools

fold A Wra5 6a/h Line To A S5e/i=ied Length

Folding is the process o+ %reaki g li es o+ te3t at a speci+ie! wi!th. *ike o"r other comma !s, fold accepts either o e or more te3t +iles or sta !ar! i p"t. 9+ we se ! fold a simple stream o+ te3t, we ca see how it works,
[me@linuxbox ~]$ echo J+he ,uick brown fox jumped over the la-! dog.J 0 fold -w 12 The Buick br o9n fox jum7 ed over the laC1 dog<

Aere we see fold i actio . The te3t se t %y the echo comma ! is %roke i to segme ts speci+ie! %y the -9 optio . 9 this e3ample, we speci+y a li e wi!th o+ twel#e characters. 9+ o wi!th is speci+ie!, the !e+a"lt is eighty characters. &otice how the li es are %roke regar!less o+ wor! %o" !aries. The a!!itio o+ the -s optio will ca"se fold to %reak the li e at the last a#aila%le space %e+ore the li e wi!th is reache!,
[me@linuxbox ~]$ echo J+he ,uick brown fox jumped over the la-! dog.J 0 fold -w 12 -s The Buick bro9n fox jum7ed over the laC1 dog<

fmt A A Sim5 e Text 2ormatter

The fmt program also +ol!s te3t, pl"s a lot more. 9t accepts either +iles or sta !ar! i p"t a ! per+orms paragraph +ormatti g o the te3t stream. Jasically, it +ills a ! 4oi s li es i te3t while preser#i g %la k li es a ! i !e tatio . To !emo strate, weIll ee! some te3t. *etIs li+t some +rom the fmt i +o page,
JfmtK reads from the s7ecified -A=& arguments Nor standard in7ut if none are givenP> and 9rites to standard out7ut< H1 default> blank lines> s7aces bet9een 9ords> and indentation are

(.(

22 8 0ormatti g 7"tp"t
7reserved in the out7utT successive in7ut lines 9ith different indentation are not joinedT tabs are ex7anded on in7ut and introduced on out7ut< JfmtK 7refers breaking lines at the end of a sentence> and tries to avoid line breaks after the first 9ord of a sentence or before the last 9ord of a sentence< 4 _sentence break_ is defined as either the end of a 7aragra7h or a 9ord ending in an1 of J<FGK> follo9ed b1 t9o s7aces or end of line> ignoring an1 intervening 7arentheses or Buotes< =ike TeV> JfmtK reads entire _7aragra7hs_ before choosing line breaksT the algorithm is a variant of that given b1 'onald &< 2nuth and +ichael -< :lass in _Hreaking :aragra7hs Anto =ines_> J*oft9are--:ractice Y &x7erienceK ##> ## NLovember #0/#P> ###0-##/%<

WeIll copy this te3t i to o"r te3t e!itor a ! sa#e the +ile as fmt-info<txt. &ow, letIs say we wa te! to re+ormat this te3t to +it a +i+ty character wi!e col"m . We co"l! !o this %y processi g the +ile with fmt a ! the -9 optio ,
[me@linuxbox ~]$ fmt -w "* fmt-info.txt 0 head JfmtK reads from the s7ecified -A=& arguments Nor standard in7ut if none are givenP> and 9rites to standard out7ut< H1 default> blank lines> s7aces bet9een 9ords> and indentation are 7reserved in the out7utT successive in7ut lines 9ith different indentation are not joinedT tabs are ex7anded on in7ut and introduced on out7ut<

Well, thatIs a awkwar! res"lt. e3plai s whatIs goi g o ,

6erhaps we sho"l! act"ally rea! this te3t, si ce it

?8y default" blank lines" spaces between words" and indentation are preserved in the outputK successive input lines with different indentation are not AoinedK tabs are e+panded on input and introduced on output#@ So, fmt is preser#i g the i !e tatio o+ the +irst li e. 0ort" ately, fmt pro#i!es a optio to correct this,
[me@linuxbox ~]$ fmt -cw "* fmt-info.txt JfmtK reads from the s7ecified -A=& arguments Nor standard in7ut if none are givenP> and 9rites to standard out7ut<

(.1

Simple 0ormatti g Tools

H1 default> blank lines> s7aces bet9een 9ords> and indentation are 7reserved in the out7utT successive in7ut lines 9ith different indentation are not joinedT tabs are ex7anded on in7ut and introduced on out7ut< JfmtK 7refers breaking lines at the end of a sentence> and tries to avoid line breaks after the first 9ord of a sentence or before the last 9ord of a sentence< 4 _sentence break_ is defined as either the end of a 7aragra7h or a 9ord ending in an1 of J<FGK> follo9ed b1 t9o s7aces or end of line> ignoring an1 intervening 7arentheses or Buotes< =ike TeV> JfmtK reads entire _7aragra7hs_ before choosing line breaksT the algorithm is a variant of that given b1 'onald &< 2nuth and +ichael -< :lass in _Hreaking :aragra7hs Anto =ines_> J*oft9are--:ractice Y &x7erienceK ##> ## NLovember #0/#P> ###0-##/%<

>"ch %etter. Jy a!!i g the -c optio , we ow ha#e the !esire! res"lt. fmt has some i teresti g optio s, Table ''-3: fmt &ptions ?ption -c Description 7perate i crown margin mo!e. This preser#es the i !e tatio o+ the +irst two li es o+ a paragraph. S"%se="e t li es are alig e! with the i !e tatio o+ the seco ! li e. 7 ly +ormat those li es %egi i g with the pre+i3 string. $+ter +ormatti g, the co te ts o+ string are pre+i3e! to each re+ormatte! li e. This optio ca %e "se! to +ormat te3t i so"rce co!e comme ts. 0or e3ample, a y programmi g la g"age or co +ig"ratio +ile that "ses a ?P@ character to !eli eate a comme t co"l! %e +ormatte! %y speci+yi g -7 K^ K so that o ly the comme ts will %e +ormatte!. See the e3ample %elow. Split-o ly mo!e. 9 this mo!e, li es will o ly %e split to +it the speci+ie! col"m wi!th. Short li es will ot %e 4oi e! to +ill li es. This mo!e is "se+"l whe +ormatti g te3t s"ch as co!e where 4oi i g is ot !esire!. 6er+orm " i+orm spaci g. This will apply tra!itio al ?typewriter

-7 string

-s

-u

(.2

22 8 0ormatti g 7"tp"t style@ +ormatti g to the te3t. This mea s a si gle space %etwee wor!s a ! two spaces %etwee se te ces. This mo!e is "se+"l +or remo#i g ?4"sti+icatio ,@ that is, te3t that has %ee pa!!e! with spaces to +orce alig me t o %oth the le+t a ! right margi s. -9 width 0ormat te3t to +it withi a col"m width characters wi!e. The !e+a"lt is /2 characters. &ote, fmt act"ally +ormats li es slightly shorter tha the speci+ie! wi!th to allow +or li e %ala ci g.

The -7 optio is partic"larly i teresti g. With it, we ca +ormat selecte! portio s o+ a +ile, pro#i!e! that the li es to %e +ormatte! all %egi with the same se="e ce o+ characters. >a y programmi g la g"ages "se the po" ! sig EPF to i !icate the %egi i g o+ a comme t a ! th"s ca %e +ormatte! "si g this optio . *etIs create a +ile that sim"lates a program that "ses comme ts,
[me@linuxbox ~]$ cat & fmt-code.txt ^ This file contains code 9ith comments< ^ This line is a comment< ^ -ollo9ed b1 another comment line< ^ 4nd another< This> on the other hand> is a line of code< 4nd another line of code< 4nd another<

7"r sample +ile co tai s comme ts which %egi the stri g ?P ? Ea P +ollowe! %y a spaceF a ! li es o+ ?co!e@ which !o ot. &ow, "si g fmt, we ca +ormat the comme ts a ! lea#e the co!e " to"che!,
[me@linuxbox ~]$ fmt -w "* -p %U % fmt-code.txt ^ This file contains code 9ith comments< ^ This line is a comment< -ollo9ed b1 another ^ comment line< 4nd another< This> on the other hand> is a line of code< 4nd another line of code< 4nd another<

&otice that the a!4oi i g comme t li es are 4oi e!, while the %la k li es a ! the li es that !o ot %egi with the speci+ie! pre+i3 are preser#e!. (.K

Simple 0ormatti g Tools

7r A 2ormat Text 2or 'rinting

The 7r program is "se to paginate te3t. Whe pri ti g te3t, it is o+te !esira%le to separate the pages o+ o"tp"t with se#eral li es o+ whitespace, to pro#i!e a top a ! %ottom margi +or each page. 0"rther, this whitespace ca %e "se! to i sert a hea!er a ! +ooter o each page. WeIll !emo strate 7r %y +ormatti g o"r distros<txt +ile i to a series o+ #ery short pages Eo ly the +irst two pages show F,
[me@linuxbox ~]$ pr -l 1" -w N" distros.txt

!((/-#!-## #/:!)

distros<txt

:age #

*3*& -edora *3*& 3buntu -edora

#(<! #( ##<( /<(% /

#!6()6!((. ##6!"6!((/ (.6#06!((/ (%6!%6!((/ ##6(/6!(()

!((/-#!-## #/:!)

distros<txt

:age !

*3*& 3buntu -edora 3buntu 3buntu

#(<$ .<#( ) )<#( )<(%

#(6(%6!(() #(6!.6!((. ("6$#6!(() #(6#/6!(() (%6#06!(()

9 this e3ample, we employ the -l optio E+or page le gthF a ! the -9 optio Epage wi!thF to !e+i e a ?page@ that is si3ty-+i#e col"m s wi!e a ! +i+tee li es lo g. 7r pagi ates the co te ts o+ the distros<txt +ile, separates each page with se#eral li es o+ whitespace a ! creates a !e+a"lt hea!er co tai i g the +ile mo!i+icatio time, +ile ame, a ! page "m%er. The 7r program pro#i!es ma y optio s to co trol page layo"t. WeIll take a look at more o+ them i the e3t chapter.

(./

22 8 0ormatti g 7"tp"t

7rintf A 2ormat And 'rint )ata

) like the other comma !s i this chapter, the 7rintf comma ! is ot "se! +or pipeli es Eit !oes ot accept sta !ar! i p"tF or !oes it +i ! +re="e t applicatio !irectly o the comma ! li e EitIs mostly "se! i scriptsF. So why is it importa tC Jeca"se it is so wi!ely "se!. 7rintf E+rom the phrase, ?pri t +ormatte!@F was origi ally !e#elope! +or the C programmi g la g"age a ! has %ee impleme te! i ma y programmi g la g"ages i cl"!i g the shell. 9 +act, i bash, 7rintf is a %"ilti . 7rintf works like this, 7rintf cformatd arguments The comma ! is gi#e a stri g co tai i g a +ormat !escriptio which is the applie! to a list o+ arg"me ts. The +ormatte! res"lt is se t to sta !ar! o"tp"t. Aere is a tri#ial e3ample,
[me@linuxbox ~]$ printf JP formatted the string2 8sKnJ foo A formatted the string: foo

The +ormat stri g may co tai literal te3t Elike ?9 +ormatte! the stri g,@F, escape se="e ces Es"ch as ]n, a ewli e characterF, a ! se="e ces %egi i g with the 5 character, which are calle! conversion specifications. 9 the e3ample a%o#e, the co #ersio speci+icatio 5s is "se! to +ormat the stri g ?+oo@ a ! place it i the comma !Is o"tp"t. Aere it is agai ,
[me@linuxbox ~]$ printf JP formatted %8s% as a string.KnJ foo A formatted KfooK as a string<

$s we ca see, the 5s co #ersio speci+icatio is replace! %y the stri g ?+oo@ i the comma !Is o"tp"t. The s co #ersio is "se! to +ormat stri g !ata. There are other speci+iers +or other ki !s o+ !ata. This ta%le lists the commo ly "se! !ata types, Table ''-$: %ommon printf -ata Type Specifiers Speci"ier d f o (.8 Description 0ormat a "m%er as a sig e! !ecimal i teger. 0ormat a ! o"tp"t a +loati g poi t "m%er. 0ormat a i teger as a octal "m%er.

Simple 0ormatti g Tools s x V 5 0ormat a stri g. 0ormat a i teger as a he3a!ecimal "m%er "si g lowercase a-+ where ee!e!. Same as x %"t "se "pper case letters. 6ri t a literal _ sym%ol Ei.e. speci+y ?__@F

WeIll !emo strate the e++ect each o+ the co #ersio speci+iers o the stri g ?(80@,
[me@linuxbox ~]$ printf J8d; 8f; 8o; 8s; 8x; 8ZKnJ M* M* $/(> $/(<((((((> ")%> $/(> #)c> #)C M* M* M* M*

Si ce we speci+ie! si3 co #ersio speci+iers, we m"st also s"pply si3 arg"me ts +or 7rintf to process. The si3 res"lts show the e++ect o+ each speci+ier. There are se#eral optio al compo e ts that may %e a!!e! to the co #ersio speci+ier to a!4"st its o"tp"t. $ complete co #ersio speci+icatio may co sist o+ the +ollowi g,
5[flags][width][.precision]conversionOs7ecification

>"ltiple optio al compo e ts, whe "se!, m"st appear i the or!er speci+ie! a%o#e to %e properly i terprete!. Aere is a !escriptio o+ each, Table ''-.: printf %onversion Specification %omponents Component flags Description There are +i#e !i++ere t +lags, ^ 8 "se the ?alter ate +ormat@ +or o"tp"t. This #aries %y !ata type. 0or o Eoctal "m%erF co #ersio , the o"tp"t is pre+i3e! with (. 0or x a ! V Ehe3a!ecimal "m%erF co #ersio s, the o"tp"t is pre+i3e! with (x or (V respecti#ely. (WEa DeroF pa! the o"tp"t with Deros. This mea s that the +iel! will %e +ille! with lea!i g Deros, as i ?000(80@. - 8 Ea !ashF le+t-alig the o"tp"t. Jy !e+a"lt, 7rintf rightalig s o"tp"t. e I 8 Ea spaceF pro!"ces a lea!i g space +or positi#e "m%ers. 8 8 Ea pl"s sig F sig s positi#e "m%ers. Jy !e+a"lt, 7rintf

(.9

22 8 0ormatti g 7"tp"t o ly sig s egati#e "m%ers. width .precision $ "m%er speci+yi g the mi im"m +iel! wi!th. 0or +loati g poi t "m%ers, speci+y the "m%er o+ !igits o+ precisio to %e o"tp"t a+ter the !ecimal poi t. 0or stri g co #ersio , precision speci+ies the "m%er o+ characters to o"tp"t.

Aere are some e3amples o+ !i++ere t +ormats i actio , Table ''-2: print %onversion Specification 1+amples #rgument $/( $/( ;ormat _5d_ _5^x_ 1esult $/( (x#)c 7otes Simple +ormatti g o+ a i teger. 9 teger +ormatte! as a he3a!ecimal "m%er "si g the ?alter ate +ormat@ +lag. 9 teger +ormatte! with lea!i g Deros Epa!!i gF a ! a mi im"m +iel! wi!th o+ +i#e characters. &"m%er +ormatte! as a +loati g poi t "m%er with pa!!i g a ! +i#e !ecimal places o+ precisio . Si ce the speci+ie! mi im"m +iel! wi!th E2F is less tha the act"al wi!th o+ the +ormatte! "m%er, the pa!!i g has o e++ect. Jy i creasi g the mi im"m +iel! wi!th to .0 the pa!!i g is ow #isi%le. The V +lag sig s a positi#e "m%er. The - +lag le+t alig s the +ormatti g.

$/(

_5("d_

(($/(

$/(

_5("<"f_

$/(<(((((

$/(

_5(#(<"f_

($/(<(((((

$/( $/(

_58d_ _5-d_

8$/( $/(

(20

Simple 0ormatti g Tools abcdefghijk abcdefghijk _"s_ _<"s_ abcedfghijk abcde $ stri g +ormatte! with a mi im"m +iel! wi!th. Jy applyi g precisio to a stri g, it is tr" cate!.

$gai , 7rintf is "se! mostly i scripts where it is employe! to +ormat ta%"lar !ata, rather tha o the comma ! li e !irectly. J"t we ca still show how it ca %e "se! to sol#e #ario"s +ormatti g pro%lems. 0irst, letIs o"tp"t some +iel!s separate! %y ta% characters,
[me@linuxbox ~]$ printf J8sKt8sKt8sKnJ str1 str2 str str# str! str$

Jy i serti g ]t Ethe escape se="e ce +or a ta%F, we achie#e the !esire! e++ect. &e3t, some "m%ers with eat +ormatti g,
[me@linuxbox ~]$ printf JTine2 8*"d 81". f Hesult2 851"dKnJ 1*C1 .1A1"N2D" 2"MD =ine: (#()# $<#%! Result: 8$!"/0

This shows the e++ect o+ mi im"m +iel! wi!th o the spaci g o+ the +iel!s. 7r how a%o"t +ormatti g a ti y we% page,
[me@linuxbox ~]$ printf J/html&KnKt/head&KnKtKt/title&8s//title&Kn Kt//head&KnKt/bod!&KnKtKt/p&8s//p&KnKt//bod!&Kn//html&KnJ JBage +it leJ JBage =ontentJ XhtmlD XheadD XtitleD:age TitleX6titleD X6headD Xbod1D X7D:age ContentX67D X6bod1D X6htmlD

Document ;ormatting S,stems

So +ar, we ha#e e3ami e! the simple te3t +ormatti g tools. These are goo! +or small, (2.

22 8 0ormatti g 7"tp"t simple tasks, %"t what a%o"t larger 4o%sC 7 e o+ the reaso s that ) i3 %ecame a pop"lar operati g system amo g tech ical a ! scie ti+ic "sers Easi!e +rom pro#i!i g a power+"l m"ltitaski g, m"lti"ser e #iro me t +or all ki !s o+ so+tware !e#elopme tF is that it o++ere! tools that co"l! %e "se! to pro!"ce ma y types o+ !oc"me ts, partic"larly scie ti+ic a ! aca!emic p"%licatio s. 9 +act, as the ;&) !oc"me tatio !escri%es, !oc"me t preparatio was i str"me tal to the !e#elopme t o+ ) i3, The first version of 0>*J was developed on a 6-6-3 which was sitting around 8ell (abs# *n 1531 the developers wanted to get a 6-6-11 for further work on the operating system# *n order to Austify the cost for this system" they proposed that they would implement a document formatting system for the )TLT patents division# This first formatting program was a reimplementation of !c*llroyMs NroffM" written by D# F# &ssanna# There are two mai +amilies o+ !oc"me t +ormatters that !omi ate the +iel!, those !esce !e! +rom the origi al roff program i cl"!i g nroff a ! troff, a ! those %ase! o 'o al! M "thIs TEG Epro o" ce! ?tek@F typesetti g system. $ ! yes, the !roppe! ?E@ i the mi!!le is part o+ its ame. The ame ?ro++@ is !eri#e! +rom the term ?r" o++@ as i , ?9Ill r" o++ a copy +or yo".@ The nroff program is "se! to +ormat !oc"me ts +or o"tp"t to !e#ices that "se mo ospace! +o ts, s"ch as character termi als a ! typewriter-style pri ters. $t the time o+ its i tro!"ctio , this i cl"!e! early all pri ti g !e#ices attache! to comp"ters. The later troff program +ormats !oc"me ts +or o"tp"t o typesetters, !e#ices "se! to pro!"ce ?camera-rea!y@ type +or commercial pri ti g. >ost comp"ter pri ters to!ay are a%le to sim"late the o"tp"t o+ typesetters. The roff +amily also i cl"!es some other programs that are "se! to prepare portio s o+ !oc"me ts. These i cl"!e eBn E+or mathematical e="atio sF a ! tbl E+or ta%lesF. The TEG system Ei sta%le +ormF +irst appeare! i .989 a ! has, to some !egree, !isplace! troff as the tool o+ choice +or typesetter o"tp"t. We wo It %e co#eri g TEG here, !"e %oth to its comple3ity Ethere are e tire %ooks a%o"t itF a ! to the +act that it is ot i stalle! %y !e+a"lt o most mo!er *i "3 systems. Ti(5 0or those i tereste! i i stalli g TEG, check o"t the texlive package which ca %e +o" ! i most !istri%"tio repositories, a ! the *yG graphical co te t e!itor.

groff
groff is s"ite o+ programs co tai i g the ;&) impleme tatio o+ troff. 9t also i cl"!es a script that is "se! to em"late nroff a ! the rest o+ the roff +amily as well. (22

'oc"me t 0ormatti g Systems While roff a ! its !esce !a ts are "se! to make +ormatte! !oc"me ts, they !o it i a way that is rather +oreig to mo!er "sers. >ost !oc"me ts to!ay are pro!"ce! "si g wor! processors that are a%le to per+orm %oth the compositio a ! layo"t o+ a !oc"me t i a si gle step. 6rior to the a!#e t o+ the graphical wor! processor, !oc"me ts were o+te pro!"ce! i a two-step process i #ol#i g the "se o+ a te3t e!itor to per+orm compositio a ! a processor, s"ch as troff to apply the +ormatti g. 9 str"ctio s +or the +ormatti g program were em%e!!e! i to the compose! te3t thro"gh the "se o+ a mark"p la g"age. The mo!er a alog +or s"ch a process is the we% page, which is compose! "si g a te3t e!itor o+ some ki ! a ! the re !ere! %y a we% %rowser "si g AT>* as the mark"p la g"age to !escri%e the +i al page layo"t. WeIre ot goi g to co#er groff i its e tirety, as ma y eleme ts o+ its mark"p la g"age !eal with rather arca e !etails o+ typography. 9 stea! we will co ce trate o o e o+ its macro packages that remai s i wi!e "se. These macro packages co !e se ma y o+ its low-le#el comma !s i to a smaller set o+ high-le#el comma !s that make "si g groff m"ch easier. 0or a mome t, letIs co si!er the h"m%le ma page. 9t li#es i the 6usr6share6man !irectory as a gCi7 compresse! te3t +ile. 9+ we were to e3ami e its " compresse! co te ts, we wo"l! see the +ollowi g Ethe ma page +or ls i sectio . is show F,
[me@linuxbox ~]$ -cat /usr/share/man/man1/ls.1.g- 0 head <]_ ' L T + 'A-U TSA* -A=&G At 9as generated b1 hel7!man #<$"< <TS =* _#_ _47ril !((/_ _@L3 coreutils .<#(_ _3ser Commands_ <*S L4+& ls ]- list director1 contents <*S *UL :*A* <H ls []fA :TA L]fR]<<< []fA-A=&]fR]<<< <*S '&*CRA:TA L <]_ 4dd an1 additional descri7tion here <::

Compare! to the ma page i its ormal prese tatio , we ca %egi to see a correlatio %etwee the mark"p la g"age a ! its res"lts,
[me@linuxbox ~]$ man ls 0 head =*N#P 3ser Commands L4+&

=*N#P

ls - list director1 contents

(2(

22 8 0ormatti g 7"tp"t
*UL :*A* ls [ :TA L]<<< [-A=&]<<<

The reaso this is o+ i terest is that ma pages are re !ere! %y groff, "si g the mandoc macro package. 9 +act, we ca sim"late the man comma ! with the +ollowi g pipeli e,
[me@linuxbox ~]$ -cat /usr/share/man/man1/ls.1.g- 0 groff -mandoc -+ ascii 0 head =*N#P 3ser Commands =*N#P

L4+& ls - list director1 contents *UL :*A* ls [ :TA L]<<< [-A=&]<<<

Aere we "se the groff program with the optio s set to speci+y the mandoc macro package a ! the o"tp"t !ri#er +or $SC99. groff ca pro!"ce o"tp"t i se#eral +ormats. 9+ o +ormat is speci+ie!, 6ostScript is o"tp"t %y !e+a"lt,
[me@linuxbox ~]$ -cat /usr/share/man/man1/ls.1.g- 0 groff -mandoc 0 head 5G:*-4dobe-$<( 55Creator: groff version #<#/<# 55Creation'ate: Thu -eb " #$:%%:$) !((0 55'ocumentLeededResources: font Times-Roman 558 font Times-Hold 558 font Times-Atalic 55'ocument*u77liedResources: 7rocset gro7s #<#/ # 55:ages: % 55:age rder: 4scend 55 rientation: :ortrait

We %rie+ly me tio e! 6ostScript i the pre#io"s chapter, a ! will agai i the e3t chapter. 6ostScript is a page !escriptio la g"age that is "se! to !escri%e the co te ts o+ a pri te! page to a typesetter-like !e#ice. 9+ we take the o"tp"t o+ o"r comma ! a ! store it to a +ile Eass"mi g that we are "si g a graphical !esktop with a 'eskto7 !irectoryF,

(21

'oc"me t 0ormatti g Systems

[me@linuxbox ~]$ ~/1esktop/foo.ps -cat /usr/share/man/man1/ls.1.g- 0 groff -mandoc &

$ ico +or the o"tp"t +ile sho"l! appear o the !esktop. Jy !o"%le-clicki g o the ico , a page #iewer sho"l! start "p a ! re#eal the +ile i its re !ere! +orm,

Figure $: Biewing 6ostScript &utput /ith ) 6age Biewer *n @>&!1 What we see is a icely typeset ma page +or lsL 9 +act, itIs possi%le to co #ert the 6ostScript +ile i to a 6'0 E6ortable -ocument FormatF +ile with this comma !,
[me@linuxbox ~]$ ps2pdf ~/1esktop/foo.ps ~/1esktop/ls.pdf

The 7s!7df program is part o+ the ghostscri7t package, which is i stalle! o most *i "3 systems that s"pport pri ti g. Ti(, *i "3 systems o+te i cl"!e ma y comma ! li e programs +or +ile +ormat

(22

22 8 0ormatti g 7"tp"t co #ersio . They are o+te ame! "si g the co #e tio o+ format2format. Try "si g the comma ! ls 6usr6bin6E[[:al7ha:]]![[:al7ha:]]E to i!e ti+y them. $lso try searchi g +or programs ame! formattoformat. 0or o"r last e3ercise with groff, we will re#isit o"r ol! +rie ! distros<txt o ce more. This time, we will "se the tbl program which is "se! to +ormat ta%les to typeset o"r list o+ *i "3 !istri%"tio s. To !o this, we are goi g to "se o"r earlier sed script to a!! mark"p to a te3t stream that we will +ee! to groff. 0irst, we ee! to mo!i+y o"r sed script to a!! the ecessary re="ests that tbl re="ires. )si g a te3t e!itor, we will cha ge distros<sed to the +ollowi g,
^ sed scri7t to 7roduce =inux distributions re7ort 1 iK .+FK center box#K cb s sK cb cb cbK l n c.K Tinux 1istributions HeportK $K ?ame ]ersion HeleasedK . s6]N[(-0]][!]\]P]6]N[(-0]][!]\]P]6]N[(-0]][%]\]P$6]$-]#-]!6 3 aK .+G

&ote that +or the script to work properly, care m"st %ee take to see that the wor!s ?&ame Rersio :elease!@ are separate! %y ta%s, ot spaces. WeIll sa#e the res"lti g +ile as distros-tbl<sed. tbl "ses the <T* a ! <T& re="ests to start a ! e ! the ta%le. The rows +ollowi g the <T* re="est !e+i e glo%al properties o+ the ta%le which, +or o"r e3ample, are ce tere! horiDo tally o the page a ! s"rro" !e! %y a %o3. The remai i g li es o+ the !e+i itio !escri%e the layo"t o+ each ta%le row. &ow, i+ we r" o"r reportge erati g pipeli e agai with the ew sed script, weIll get the +ollowi g ,
[me@linuxbox ~]$ sort -k 1;1 -k 2n distros.txt 0 sed -f distros-tbl .sed 0 groff -t -+ ascii 2&/dev/null 8------------------------------8 Q Tinux 1istributions Heport Q 8------------------------------8 Q ?ame ]ersion Heleased Q

(2K

'oc"me t 0ormatti g Systems

8------------------------------8 Q-edora " !((.-($-!( Q Q-edora . !((.-#(-!% Q Q-edora ) !(()-("-$# Q Q-edora / !(()-##-(/ Q Q-edora 0 !((/-("-#$ Q Q-edora #( !((/-##-!" Q Q*3*& #(<# !((.-("-## Q Q*3*& #(<! !((.-#!-() Q Q*3*& #(<$ !(()-#(-(% Q Q*3*& ##<( !((/-(.-#0 Q Q3buntu .<(. !((.-(.-(# Q Q3buntu .<#( !((.-#(-!. Q Q3buntu )<(% !(()-(%-#0 Q Q3buntu )<#( !(()-#(-#/ Q Q3buntu /<(% !((/-(%-!% Q Q3buntu /<#( !((/-#(-$( Q 8------------------------------8

$!!i g the -t optio to groff i str"cts it to pre-process the te3t stream with tbl. *ikewise, the -T optio is "se! to o"tp"t to $SC99 rather tha the !e+a"lt o"tp"t me!i"m, 6ostScript. The +ormat o+ the o"tp"t is the %est we ca e3pect i+ we are limite! to the capa%ilities o+ a termi al scree or typewriter-style pri ter. 9+ we speci+y 6ostScript o"tp"t a ! graphically #iew the res"lti g o"tp"t, we get a m"ch more satis+yi g res"lt,
[me@linuxbox ~]$ sort -k 1;1 -k 2n distros.txt 0 sed -f distros-tbl .sed 0 groff -t & ~/1esktop/foo.ps

(2/

22 8 0ormatti g 7"tp"t

Figure .: Biewing The Finished Table

Summing -p
;i#e that te3t is so ce tral to the character o+ ) i3-like operati g systems, it makes se se that there wo"l! %e ma y tools that are "se! to ma ip"late a ! +ormat te3t. $s we ha#e see , there areL The simple +ormatti g tools like fmt a ! 7r will +i ! ma y "ses i scripts that pro!"ce short !oc"me ts, while groff Ea ! +rie !sF ca %e "se! to write %ooks. We may e#er write a tech ical paper "si g comma ! li e tools Etho"gh there are ma y people who !oLF, %"t itIs goo! to k ow that we co"l!.

;urther 1eading

groff )serIs ;"i!e http,--www.g ".org-so+tware-gro++-ma "al/riting 6apers /ith nroff 0sing -me, http,--!ocs.+ree%s!.org-11!oc-"s!-.9.memacros-paper.p!+ -me 9eference !anual,

(28

0"rther :ea!i g http,--!ocs.+ree%s!.org-11!oc-"s!-20.mere+-paper.p!+

Tbl O ) 6rogram To Format Tables, http,--pla 9.%ell-la%s.com-.0thE!>a -t%l.p!+ $ !, o+ co"rse, try the +ollowi g articles at Wikipe!ia, http,--e .wikipe!ia.org-wiki-TeG http,--e .wikipe!ia.org-wiki-'o al!OM "th http,--e .wikipe!ia.org-wiki-Typesetti g

(29

2( 8 6ri ti g

23 2rinting
$+ter spe !i g the last co"ple o+ chapters ma ip"lati g te3t, itIs time to p"t that te3t o paper. 9 this chapter, weIll look at the comma ! li e tools that are "se! to pri t +iles a ! co trol pri ter operatio . We wo It %e looki g at how to co +ig"re pri ti g, as that #aries +rom !istri%"tio to !istri%"tio a ! is "s"ally set "p a"tomatically !"ri g i stallatio . &ote that we will ee! a worki g pri ter co +ig"ratio to per+orm the e3ercises i this chapter. We will !isc"ss the +ollowi g comma !s,

7r 8 Co #ert te3t +iles +or pri ti g l7r 8 6ri t +iles a!7s 8 0ormat +iles +or pri ti g o a 6ostScript pri ter l7stat 8 Show pri ter stat"s i +ormatio l7B 8 Show pri ter ="e"e stat"s l7rm 8 Ca cel pri t 4o%s

# /rie" 3istor, ?" Printing

To +"lly " !ersta ! the pri ti g +eat"res +o" ! i ) i3-like operati g systems, we m"st +irst lear some history. 6ri ti g o ) i3-like systems goes way %ack to the %egi i g o+ the operati g system itsel+. 9 those !ays, pri ters a ! how they were "se! was m"ch !i++ere t +rom to!ay.

'rinting In The )im Times

*ike the comp"ters themsel#es, pri ters i the pre-6C era te !e! to %e large, e3pe si#e a ! ce traliDe!. The typical comp"ter "ser o+ .980 worke! at a termi al co ecte! to a comp"ter some !ista ce away. The pri ter was locate! ear the comp"ter a ! was " !er the watch+"l eyes o+ the comp"terIs operators. Whe pri ters were e3pe si#e a ! ce traliDe!, as they o+te were i the early !ays o+ ) i3, it was commo practice +or ma y "sers to share a pri ter. To i!e ti+y pri t 4o%s ((0

$ Jrie+ Aistory 7+ 6ri ti g %elo gi g to a partic"lar "ser, a banner page !isplayi g the ame o+ the "ser was o+te pri te! at the %egi i g o+ each pri t 4o%. The comp"ter s"pport sta++ wo"l! the loa! "p a cart co tai i g the !ayIs pri t 4o%s a ! !eli#er them to the i !i#i!"al "sers.

Chara/terBbased 'rinters
The pri ter tech ology o+ the 80s was #ery !i++ere t i two respects. 0irst, pri ters o+ that perio! were almost always impact printers. 9mpact pri ters "se a mecha ical mecha ism which strikes a ri%%o agai st the paper to +orm character impressio s o the page. Two o+ the pop"lar tech ologies o+ that time were daisy wheel pri ti g a ! dotmatri+ pri ti g. The seco !, a ! more importa t characteristic o+ early pri ters was that pri ters "se! a +i3e! set o+ characters that were i tri sic to the !e#ice itsel+. 0or e3ample, a !aisy wheel pri ter co"l! o ly pri t the characters act"ally mol!e! i to the petals o+ the !aisy wheel. This ma!e the pri ters m"ch like high-spee! typewriters. $s with most typewriters, they pri te! "si g mo ospace! E+i3e! wi!thF +o ts. This mea s that each character has the same wi!th. 6ri ti g was !o e at +i3e! positio s o the page, a ! the pri ta%le area o+ a page co tai e! a +i3e! "m%er o+ characters. >ost pri ters pri te! te characters per i ch EC69F horiDo tally a ! si3 li es per i ch E*69F #ertically. )si g this scheme, a )S letter sheet o+ paper is eighty-+i#e characters wi!e a ! si3ty-si3 li es high. Taki g i to acco" t a small margi o each si!e, eighty characters was co si!ere! the ma3im"m wi!th o+ a pri t li e. This e3plai s why termi al !isplays Ea ! o"r termi al em"latorsF are ormally eighty characters wi!e. 9t pro#i!es a /CS*/C@ E/hat Cou See *s /hat Cou @etF #iew o+ pri te! o"tp"t, "si g a mo ospace! +o t. 'ata is se t to a typewriter-like pri ter i a simple stream o+ %ytes co tai i g the characters to %e pri te!. 0or e3ample, to pri t a ?a@, the $SC99 character co!e 9/ is se t. 9 a!!itio , the low- "m%ere! $SC99 co trol co!es pro#i!e! a mea s o+ mo#i g the pri terIs carriage a ! paper, "si g co!es +or carriage ret"r , li e +ee!, +orm +ee!, etc. )si g the co trol co!es, itIs possi%le to achie#e some limite! +o t e++ects, s"ch as %ol! +ace, %y ha#i g the pri ter pri t a character, %ackspace a ! pri t the character agai to get a !arker pri t impressio o the page. We ca act"ally wit ess this i+ we "se nroff to re !er a ma page a ! e3ami e the o"tp"t "si g cat -4,
[me@linuxbox ~]$ -cat /usr/share/man/man1/ls.1.g- 0 nroff -man 0 cat -: 0 head =*N#P 3ser Commands =*N#P $ $ $ L`SL4`S4+`S+&`S&$ ls - list director1 contents$

((.

2( 8 6ri ti g
$ *`S*U`SUL`SL `S :`S:*`S*A`SA*`S*$ l`Sls`Ss [O`S O`S:O`STO`SAO`S O`SL]<<< [O`S-O`SAO`S=O`S&]<<<$

The `S ECo trol-hF characters are the %ackspaces "se! to create the %ol!+ace e++ect. *ikewise, we ca also see a %ackspace-" !erscore se="e ce "se! to pro!"ce " !erli i g.

+ra5hi/a 'rinters
The !e#elopme t o+ ;)9s le! to ma4or cha ges i pri ter tech ology. $s comp"ters mo#e! to more pict"re-%ase! !isplays, pri ti g mo#e! +rom character-%ase! to graphical tech i="es. This was +acilitate! %y the a!#e t o+ the low-cost laser pri ter which, i stea! o+ pri ti g +i3e! characters, co"l! pri t ti y !ots a ywhere i the pri ta%le area o+ the page. This ma!e pri ti g proportio al +o ts Elike those "se! %y typesettersF, a ! e#e photographs a ! high ="ality !iagrams, possi%le. Aowe#er, mo#i g +rom a character-%ase! scheme to a graphical scheme prese te! a +ormi!a%le tech ical challe ge. AereIs why, the "m%er o+ %ytes ee!e! to +ill a page "si g a character-%ase! pri ter ca %e calc"late! this way Eass"mi g si3ty li es per page each co tai i g eighty charactersF, K0 G 80 W 1800 %ytes 9 compariso , a three h" !re! !ot per i ch E'69F laser pri ter Eass"mi g a eight %y te i ch pri t area per pageF re="ires, E8 G (00F G E.0 G (00F - 8 W 900000 %ytes >a y o+ the slow 6C etworks simply co"l! ot ha !le the early o e mega%yte o+ !ata re="ire! to pri t a +"ll page o a laser pri ter, so it was clear that a cle#er i #e tio was ee!e!. That i #e tio t"r e! o"t to %e the page description language E6'*F. $ page !escriptio la g"age is a programmi g la g"age that !escri%es the co te ts o+ a page. Jasically it says, ?go to this positio , !raw the character eaI i te poi t Ael#etica, go to this positio ...@ " til e#erythi g o the page is !escri%e!. The +irst ma4or 6'* was 6ostScript +rom $!o%e Systems, which is still i wi!e "se to!ay. The 6ostScript la g"age is a complete programmi g la g"age tailore! +or typography a ! other ki !s o+ graphics a ! imagi g. 9t i cl"!es %"ilt-i s"pport +or thirty-+i#e sta !ar!, high-="ality +o ts, pl"s the a%ility to accept a!!itio al +o t !e+i itio s at r" time. $t +irst, s"pport +or 6ostScript was %"ilt i to the pri ters themsel#es. This sol#e! the !ata tra smissio pro%lem. While the typical 6ostScript program was #ery #er%ose i compariso to the simple %yte stream o+ character-%ase! pri ters, it was m"ch smaller tha the "m%er o+ %ytes re="ire! to represe t the e tire pri te! page. $ 6ostScript printer accepte! a 6ostScript program as i p"t. The pri ter co tai e! its ((2

$ Jrie+ Aistory 7+ 6ri ti g ow processor a ! memory Eo+te times maki g the pri ter a more power+"l comp"ter tha the comp"ter to which it was attache!F a ! e3ec"te! a special program calle! a 6ostScript interpreter, which rea! the i comi g 6ostScript program a ! rendered the res"lts i to the pri terIs i ter al memory, th"s +ormi g the patter o+ %its E!otsF that wo"l! %e tra s+erre! to the paper. The ge eric ame +or this process o+ re !eri g somethi g i to a large %it patter Ecalle! a bitmapF is raster image processor or :96. $s the years we t %y, %oth comp"ters a ! etworks %ecame m"ch +aster. This allowe! the :96 to mo#e +rom the pri ter to the host comp"ter, which, i t"r , permitte! high ="ality pri ters to %e m"ch less e3pe si#e. >a y pri ters to!ay still accept character-%ase! streams, %"t ma y low-cost pri ters !o ot. They rely o the host comp"terIs :96 to pro#i!e a stream o+ %its to pri t as !ots. There are still some 6ostScript pri ters, too.

Printing +ith Linux

>o!er *i "3 systems employ two so+tware s"ites to per+orm a ! ma age pri ti g. The +irst, C)6S ECommo ) i3 6ri ti g SystemF pro#i!es pri t !ri#ers a ! pri t 4o% ma ageme t , a ! the seco !, ;hostscript, a 6ostScript i terpreter, acts as a :96. C)6S ma ages pri ters %y creati g a ! mai tai i g pri t ="e"es. $s we !isc"sse! i o"r history lesso a%o#e, ) i3 pri ti g was origi ally !esig e! to ma age a ce traliDe! pri ter share! %y m"ltiple "sers. Si ce pri ters are slow %y at"re, compare! to the comp"ters that are +ee!i g them, pri ti g systems ee! a way to sche!"le m"ltiple pri t 4o%s a ! keep thi gs orga iDe!. C)6S also has the a%ility to recog iDe !i++ere t types o+ !ata Ewithi reaso F a ! ca co #ert +iles to a pri ta%le +orm.

Preparing ;iles ;or Printing

$s comma ! li e "sers, we are mostly i tereste! i pri ti g te3t, tho"gh it is certai ly possi%le to pri t other !ata +ormats as well.

7r A Con;ert Text 2i es 2or 'rinting

We looke! at 7r a little i the pre#io"s chapter. &ow we will e3ami e some o+ its ma y optio s "se! i co 4" ctio with pri ti g. 9 o"r history o+ pri ti g, we saw how character-%ase! pri ters "se mo ospace! +o ts, res"lti g i +i3e! "m%ers o+ characters per li e a ! li es per page. 7r is "se! to a!4"st te3t to +it o a speci+ic page siDe, with optio al page hea!ers a ! margi s. Aere is a s"mmary o+ its most commo ly "se! optio s,

(((

2( 8 6ri ti g Table '3-1: %ommon pr &ptions ?ption 8first[:last] -columns -a -d -' cformatd Description 7"tp"t a ra ge o+ pages starti g with first a !, optio ally, e !i g with last. 7rga iDe the co te t o+ the page i to the "m%er o+ col"m s speci+ie! %y columns. Jy !e+a"lt, m"lti-col"m o"tp"t is liste! #ertically. Jy a!!i g the -a EacrossF optio , co te t is liste! horiDo tally. 'o"%le-space o"tp"t. 0ormat the !ate !isplaye! i page hea!ers "si g format. See the ma page +or the date comma ! +or a !escriptio o+ the +ormat stri g. )se +orm +ee!s rather tha carriage ret"r s to separate pages. 9 the ce ter portio o+ the page hea!er, "se header rather the ame o+ the +ile %ei g processe!. Set page le gth to length. 'e+a"lt is KK E)S letter at K li es per i chF &"m%er li es. Create a le+t margi offset characters wi!e. Set page wi!th to width. 'e+a"lt is /2.

-f -h cheaderd -l length -n -o offset -9 width

7r is o+te "se! i pipeli es as a +ilter. 9 this e3ample, we will pro!"ce a !irectory listi g o+ 6usr6bin a ! +ormat it i to pagi ate! three-col"m o"tp"t "si g 7r,
[me@linuxbox ~]$ !((0-(!-#/ #%:(( ls /usr/bin 0 pr -w N" 0 head :age #

[ %##to77m a!7 a!7s a!7s-l7r-9ra77er

a7turl ar arecord arecordmidi ark

bsd-9rite bsh btcflash bug-budd1 buildhash

((1

6repari g 0iles 0or 6ri ti g

Sending # Print Jo0 To # Printer

The C)6S pri ti g s"ite s"pports two metho!s o+ pri ti g historically "se! o ) i3-like systems. 7 e metho!, calle! Jerkeley or *6' E"se! i the Jerkeley So+tware 'istri%"tio #ersio o+ ) i3F, "ses the l7r program, while the other metho!, calle! SysR E+rom the System R #ersio o+ ) i3F, "ses the l7 program. Joth programs !o ro"ghly the same thi g. Choosi g o e o#er the other is a matter o+ perso al taste.

l7r A 'rint 2i es EBerke ey Sty eF

The l7r program ca %e "se! to se ! +iles to the pri ter. 9t may also "se! i pipeli es, as it accepts sta !ar! i p"t. 0or e3ample, to pri t the res"lts o+ o"r m"lti-col"m !irectory listi g a%o#e, we co"l! !o this,
[me@linuxbox ~]$ ls /usr/bin 0 pr 0 lpr

a ! the report wo"l! %e se t to the systemIs !e+a"lt pri ter. To se ! the +ile to a !i++ere t pri ter, the -: optio ca "se! like this,
lpr -B printer_name

where printer name is the ame o+ the !esire! pri ter. To see a list o+ pri ters k ow to the system,
[me@linuxbox ~]$ lpstat -a

Ti(5 >a y *i "3 !istri%"tio s allow yo" to !e+i e a ?pri ter@ that o"tp"ts +iles i 6'0 E6orta%le 'oc"me t 0ormatF +ormat, rather tha pri ti g o the physical pri ter. This is #ery ha !y +or e3perime ti g with pri ti g comma !s. Check yo"r pri ter co +ig"ratio program to see i+ it s"pports this co +ig"ratio . 7 some !istri%"tio s, yo" may ee! to i stall a!!itio al packages Es"ch as cu7s-7dfF to e a%le this capa%ility. Aere are some o+ the commo optio s +or l7r, ((2

2( 8 6ri ti g Table '3-': %ommon lpr &ptions ?ption -^ number -7 Description Set "m%er o+ copies to number. 6ri t each page with a sha!e! hea!er with the !ate, time, 4o% ame a ! page "m%er. This so-calle! ?pretty pri t@ optio ca %e "se! whe pri ti g te3t +iles. Speci+y the ame o+ the pri ter "se! +or o"tp"t. 9+ o pri ter is speci+ie!, the systemIs !e+a"lt pri ter is "se!. 'elete +iles a+ter pri ti g. This wo"l! %e "se+"l +or programs that pro!"ce temporary pri ter o"tp"t +iles.

-: printer -r

l7 A 'rint 2i es ESystem ? Sty eF

*ike l7r, l7 accepts either +iles or sta !ar! i p"t +or pri ti g. 9t !i++ers +rom l7r i that it s"pports a !i++ere t Ea ! slightly more sophisticate!F optio set. Aere are the commo o es, Table '3-3: %ommon lp &ptions ?ption -d printer Description Set the !esti atio Epri terF to printer. 9+ o ! optio is speci+ie!, the system !e+a"lt pri ter is "se!. Set the "m%er o+ copies to number. Set o"tp"t to la !scape orie tatio . Scale the +ile to +it the page. This is "se+"l whe pri ti g images, s"ch as J6E; +iles. Scale +ile to number. The #al"e o+ .00 +ills the page. Ral"es less tha .00 are re!"ce!, while #al"es greater tha .00 ca"se the +ile to %e pri te! across m"ltiple pages. Set the o"tp"t characters per i ch to number. 'e+a"lt is .0. Set the o"tp"t li es per i ch to number. 'e+a"lt is K.

-n number -o landsca7e -o fit7lot -o scalingMnumber

-o c7iMnumber -o l7iMnumber

((K

Se !i g $ 6ri t Jo% To $ 6ri ter -o -o -o -o 7age-bottomMpoints 7age-leftMpoints 7age-rightMpoints 7age-to7Mpoints Set the page margi s. Ral"es are e3presse! i points, a " it o+ typographic meas"reme t. There are /2 poi ts to a i ch. Speci+y the list o+ pages. pages may %e e3presse! as a comma separate! list a !-or a ra ge. 0or e3ample ?.,(,2,/-.0@

-: pages

WeIll pro!"ce o"r !irectory listi g agai , this time pri ti g twel#e C69 a ! eight *69 with a le+t margi o+ o e hal+ i ch. &ote that we ha#e to a!4"st the 7r optio s to acco" t +or the ew page siDe,
[me@linuxbox ~]$ ls /usr/bin 0 pr -A -w D* -l MM 0 lp -o page-left$ N -o cpi$12 -o lpi$M

This pipeli e pro!"ces a +o"r col"m listi g "si g smaller type tha the !e+a"lt. The i crease! "m%er o+ characters per i ch allows "s to +it more col"m s o the page.

Another <5tionQ a!7s

The a!7s program is i teresti g. $s we ca s"rmise +rom it ame, itIs a +ormat co #ersio program, %"t it also m"ch more. 9ts ame origi ally mea t ?$SC99 to 6ostScript@ a ! it was "se! to prepare te3t +iles +or pri ti g o 6ostScript pri ters. 7#er the years, howe#er, the capa%ilities o+ the program ha#e grow , a ! ow its ame mea s ?$ ythi g to 6ostScript.@ While its ame s"ggest a +ormat co #ersio program, it is act"ally a pri ti g program. 9t se !s its !e+a"lt o"tp"t to the systemIs !e+a"lt pri ter rather tha sta !ar! o"tp"t. The programIs !e+a"lt %eha#ior is that o+ a ?pretty pri ter,@ mea i g that it impro#es the appeara ce o+ o"tp"t. 9+ we "se the program to create a 6ostScript +ile o o"r !esktop,
[me@linuxbox ~]$ ls /usr/bin 0 pr - -t 0 a2ps -o ~/1esktop/ls.ps -T NN [stdin N7lainP: ## 7ages on . sheets] [Total: ## 7ages on . sheets] saved into the file J6home6me6'eskto76 ls<7sK

Aere we +ilter the stream with 7r, "si g the -t optio Eomit hea!ers a ! +ootersF a ! the with a!7s, speci+yi g a o"tp"t +ile E-o optio F a ! KK li es per page E-= optio F

((/

2( 8 6ri ti g to match the o"tp"t pagi atio o+ 7r. 9+ we #iew the res"lti g +ile with a s"ita%le +ile #iewer, we will see this,

Figure 2: Biewing a'ps &utput $s we ca see, the !e+a"lt o"tp"t layo"t is ?two "p@ +ormat. This ca"ses the co te ts o+ two pages to %e pri te! o each sheet o+ paper. a!7s applies ice page hea!ers a ! +ooters, too. a!7s has a lot o+ optio s. Aere is a s"mmary, Table '3-$: a'ps &ptions ?ption --center-title text --columns number Description Set ce ter page title to te+t. $rra ge pages i to number col"m s. 'e+a"lt is 2.

((8

Se !i g $ 6ri t Jo% To $ 6ri ter --footer text --guess Set page +ooter to te+t. :eport the types o+ +iles gi#e as arg"me ts. Si ce a!7s tries to co #ert a ! +ormat all types o+ !ata, this optio ca %e "se+"l +or pre!icti g what a!7s will !o whe gi#e a partic"lar +ile. Set le+t page +ooter to te+t. Set le+t page title to te+t. &"m%er li es o+ o"tp"t e#ery interval li es. 'isplay !e+a"lt setti gs. 'isplay setti gs +or topic, where topic is o e o+ the +ollowi g, !elegatio s Ee3ter al programs that will %e "se! to co #ert !ataF, e co!i gs, +eat"res, #aria%les, me!ia Epaper siDes a ! the likeF, pp! E6ostScript pri ter !escriptio sF, pri ters, prolog"es Eportio s o+ co!e that are pre+i3e! to ormal o"tp"tF, stylesheets, a ! "ser optio s. 6ri t pages i ra ge. Set right page +ooter to te+t. Set right page title to te+t. $rra ge pages i to number rows. 'e+a"lt is o e. &o page hea!ers. Set page hea!er to te+t. )se si7e poi t +o t. Set characters per li e to number. This a ! the -= optio E%elowF ca %e "se! to make +iles pagi ate! with other programs, s"ch as 7r, +it correctly o the page. Set li es per page to number. )se me!ia ame. 0or e3ample ?$1@. 7"tp"t number copies o+ each page.

--left-footer text --left-title text --line-numbersMinterval --listMdefaults --listMtopic

--7ages range --right-footer text --right-title text --ro9s number -H -b text -f size -l number

-= number -+ name -n number

((9

2( 8 6ri ti g -o file -: printer -R -r -T number -u text Se ! o"tp"t to file. 9+ file is speci+ie! as ?-@, "se sta !ar! o"tp"t. )se printer. 9+ a pri ter is ot speci+ie!, the system !e+a"lt pri ter is "se!. 6ortrait orie tatio . *a !scape orie tatio . Set ta% stops to e#ery number characters. ) !erlay EwatermarkF pages with te+t.

This is 4"st a s"mmary. a!7s has se#eral more optio s. 1ote5 a!7s is still i acti#e !e#elopme t. '"ri g my testi g, 9 otice! !i++ere t %eha#ior o #ario"s !istri%"tio s. 7 Ce t7S 1, o"tp"t always we t to sta !ar! o"tp"t %y !e+a"lt. 7 Ce t7S 1 a ! 0e!ora .0, o"tp"t !e+a"lte! to $1 me!ia, !espite the program %ei g co +ig"re! to "se letter siDe me!ia %y !e+a"lt. 9 co"l! o#ercome these iss"es %y e3plicitly speci+yi g the !esire! optio . 7 )%" t" 8.01, it per+orme! as !oc"me te!. $lso ote that there is a other o"tp"t +ormatter that is "se+"l +or co #erti g te3t i to 6ostScript. Calle! enscri7t, it ca per+orm ma y o+ the same ki !s o+ +ormatti g a ! pri ti g tricks, %"t " like a!7s, it o ly accepts te3t i p"t.

>onitoring #nd Controlling Print Jo0s

$s ) i3 pri ti g systems are !esig e! to ha !le m"ltiple pri t 4o%s +rom m"ltiple "sers, C)6S is !esig e! to !o the same. Each pri ter is gi#e a print <ueue, where 4o%s are parke! " til they ca %e spooled to the pri ter. C)6S s"pplies se#eral comma ! li e programs that are "se! to ma age pri ter stat"s a ! pri t ="e"es. *ike the l7r a ! l7 programs, these ma ageme t programs are mo!ele! a+ter the correspo !i g programs +rom the Jerkeley a ! System R pri ti g systems.

l7stat A )is5 ay 'rint System Status

The l7stat program is "se+"l +or !etermi i g the ames a ! a#aila%ility o+ pri ters o the system. 0or e3ample, i+ we ha! a system with %oth a physical pri ter E ame! ?pri ter@F a ! a 6'0 #irt"al pri ter E ame! ?6'0@F, we co"l! check their stat"s like this,

(10

>o itori g $ ! Co trolli g 6ri t Jo%s

[me@linuxbox ~]$ lpstat -a :'- acce7ting reBuests since +on (/ 'ec !((/ ($:(":"0 :+ &*T 7rinter acce7ting reBuests since Tue !% -eb !((0 (/:%$:!! 4+ &*T

0"rther, we co"l! !etermi e a more !etaile! !escriptio o+ the pri t system co +ig"ratio this way,
[me@linuxbox ~]$ lpstat -s s1stem default destination: 7rinter device for :'-: cu7s-7df:6 device for 7rinter: i77:667rint-server:.$#67rinters67rinter

9 this e3ample, we see that ?pri ter@ is the systemIs !e+a"lt pri ter a ! that it is a etwork pri ter "si g 9 ter et 6ri ti g 6rotocol Eipp,--F attache! to a system ame! ?pri t-ser#er@. The commo ly "se+"l optio s i cl"!e, Table '3-.: %ommon lpstat &ptions ?ption -a [printer<<<] Description 'isplay the state o+ the pri ter ="e"e +or printer. &ote that this is the stat"s o+ the pri ter ="e"eIs a%ility to accept 4o%s, ot the stat"s o+ the physical pri ters. 9+ o pri ters are speci+ie!, all pri t ="e"es are show . 'isplay the ame o+ systemIs !e+a"lt pri ter. 'isplay the stat"s o+ the speci+ie! printer. 9+ o pri ters are speci+ie!, all pri ters are show . 'isplay the stat"s o+ the pri t ser#er. 'isplay a stat"s s"mmary. 'isplay a complete stat"s report.

-d -7 [printer<<<] -r -s -t

l7B A )is5 ay 'rinter Dueue Status

To see the stat"s o+ a pri ter ="e"e, the l7B program is "se!. This allows "s to #iew the stat"s o+ the ="e"e a ! the pri t 4o%s it co tai s. Aere is a e3ample o+ a empty ="e"e +or a system !e+a"lt pri ter ame! ?pri ter@,

(1.

2( 8 6ri ti g
[me@linuxbox ~]$ lp, 7rinter is read1 no entries

9+ we !o ot speci+y a pri ter E"si g the -: optio F, the systemIs !e+a"lt pri ter is show . 9+ we se ! a 4o% to the pri ter a ! the look at the ="e"e, we will see it liste!,
[me@linuxbox ~]$ ls *.txt 0 pr - 0 lp reBuest id is 7rinter-.($ N# fileNsPP [me@linuxbox ~]$ lp, 7rinter is read1 and 7rinting Rank 9ner ?ob -ileNsP active me .($ NstdinP

Total *iCe #(!% b1tes

l7rm - cancel A Can/e 'rint Gobs

C)6S s"pplies two programs "se! to termi ate pri t 4o%s a ! remo#e them +rom the pri t ="e"e. 7 e is Jerkeley style El7rmF a ! the other is System R EcancelF. They !i++er slightly i the optio s they s"pport, %"t !o %asically the same thi g. )si g o"r pri t 4o% a%o#e as a e3ample, we co"l! stop the 4o% a ! remo#e it this way,
[me@linuxbox ~]$ cancel N* [me@linuxbox ~]$ lp, 7rinter is read1 no entries

Each comma ! has optio s +or remo#i g all the 4o%s %elo gi g to a partic"lar "ser, partic"lar pri ter, a ! m"ltiple 4o% "m%ers. Their respecti#e ma pages ha#e all the !etails.

Summing -p
9 this chapter, we ha#e see how the pri ters o+ the past i +l"e ce! the !esig o+ the pri ti g systems o ) i3-like machi es, a ! how m"ch co trol is a#aila%le o the comma ! li e to co trol ot o ly the sche!"li g a ! e3ec"tio o+ pri t 4o%s, %"t also the #ario"s o"tp"t optio s.

;urther 1eading

$ goo! article o the 6ostScript page !escriptio la g"age,

(12

0"rther :ea!i g http,--e .wikipe!ia.org-wiki-6ostScript

The Commo ) i3 6ri ti g System EC)6SF, http,--e .wikipe!ia.org-wiki-Commo O) i3O6ri ti gOSystem http,--www.c"ps.orgThe Jerkeley a ! System R 6ri ti g Systems, http,--e .wikipe!ia.org-wiki-JerkeleyOpri ti gOsystem http,--e .wikipe!ia.org-wiki-SystemOROpri ti gOsystem

(1(

21 8 Compili g 6rograms

2 Co%#iling 2rogra%s
9 this chapter, we will look at how to %"il! programs %y compili g so"rce co!e. The a#aila%ility o+ so"rce co!e is the esse tial +ree!om that makes *i "3 possi%le. The e tire ecosystem o+ *i "3 !e#elopme t relies o +ree e3cha ge %etwee !e#elopers. 0or ma y !esktop "sers, compili g is a lost art. 9t "se! to %e ="ite commo , %"t to!ay, !istri%"tio pro#i!ers mai tai h"ge repositories o+ pre-compile! %i aries, rea!y to !ow loa! a ! "se. $t the time o+ this writi g, the 'e%ia repository Eo e o+ the largest o+ a y o+ the !istri%"tio sF co tai s almost twe ty-three tho"sa ! packages. So why compile so+twareC There are two reaso s, .. #vailability. 'espite the "m%er o+ pre-compile! programs i !istri%"tio repositories, some !istri%"tio s may ot i cl"!e all the !esire! applicatio s. 9 this case, the o ly way to get the !esire! program is to compile it +rom so"rce. 2. Timeliness. While some !istri%"tio s specialiDe i c"tti g e!ge #ersio s o+ programs, ma y !o ot. This mea s that i or!er to ha#e the #ery latest #ersio o+ a program, compili g is ecessary. Compili g so+tware +rom so"rce co!e ca %ecome #ery comple3 a ! tech icalH well %eyo ! the reach o+ ma y "sers. Aowe#er, ma y compili g tasks are ="ite easy a ! i #ol#e o ly a +ew steps. 9t all !epe !s o the package. We will look at a #ery simple case i or!er to pro#i!e a o#er#iew o+ the process a ! as a starti g poi t +or those who wish to " !ertake +"rther st"!y. We will i tro!"ce o e ew comma !,

make 8 )tility To >ai tai 6rograms

+hat Is Compiling.
Simply p"t, compili g is the process o+ tra slati g source code Ethe h"ma -rea!a%le !escriptio o+ a program writte %y a programmerF i to the ati#e la g"age o+ the comp"terIs processor. The comp"terIs processor Eor %60F works at a #ery eleme tal le#el, e3ec"ti g programs i what is calle! machine language. This is a "meric co!e that !escri%es #ery small operatio s, s"ch as ?a!! this %yte,@ ?poi t to this locatio i memory,@ or ?copy this (11

What 9s Compili gC %yte.@ Each o+ these i str"ctio s is e3presse! i %i ary Eo es a ! DerosF. The earliest comp"ter programs were writte "si g this "meric co!e, which may e3plai why programmers who wrote it were sai! to smoke a lot, !ri k gallo s o+ co++ee, a ! wear thick glasses. This pro%lem was o#ercome %y the a!#e t o+ assembly language, which replace! the "meric co!es with EslightlyF easier to "se character mnemonics s"ch as C65 E+or copyF a ! >7R E+or mo#eF. 6rograms writte i assem%ly la g"age are processe! i to machi e la g"age %y a program calle! a assembler. $ssem%ly la g"age is still "se! to!ay +or certai specialiDe! programmi g tasks, s"ch as device drivers a ! embedded systems. We e3t come to what are calle!, high-level programming languages. They are calle! this %eca"se they allow the programmer to %e less co cer e! with the !etails o+ what the processor is !oi g a ! more with sol#i g the pro%lem at ha !. The early o es E!e#elope! !"ri g the .920sF i cl"!e! Fortran E!esig e! +or scie ti+ic a ! tech ical tasksF a ! %&8&( E!esig e! +or %"si ess applicatio sF. Joth are still i limite! "se to!ay. While there are ma y pop"lar programmi g la g"ages, two pre!omi ate. >ost programs writte +or mo!er systems are writte i either % or %PP. 9 the e3amples to +ollow, we will %e compili g a C program. 6rograms writte i high-le#el programmi g la g"ages are co #erte! i to machi e la g"age %y processi g them with a other program, calle! a compiler. Some compilers tra slate high-le#el i str"ctio s i to assem%ly la g"age a ! the "se a assem%ler to per+orm the +i al stage o+ tra slatio i to machi e la g"age. $ process o+te "se! i co 4" ctio with compili g is calle! linking. There are ma y commo tasks per+orme! %y programs. Take, +or i sta ce, ope i g a +ile. >a y programs per+orm this task, %"t it wo"l! %e waste+"l to ha#e each program impleme t its ow ro"ti e to ope +iles. 9t makes more se se to ha#e a si gle piece o+ programmi g that k ows how to ope +iles a ! to allow all programs that ee! it to share it. This is accomplishe! %y what are calle! libraries. They co tai m"ltiple routines, each per+ormi g some commo task that m"ltiple programs ca share. 9+ we look i the 6lib a ! 6usr6lib !irectories, we ca see where ma y o+ them li#e. $ program calle! a linker is "se! to +orm the co ectio s %etwee the o"tp"t o+ the compiler a ! the li%raries that the compile! program re="ires. The +i al res"lt o+ this process is the e+ecutable program file, rea!y +or "se.

Are A 'rograms Com5i ed?

&o. $s we ha#e see , there are programs s"ch as shell scripts that !o ot re="ire compili g. They are e3ec"te! !irectly. These are writte i what are k ow as scripting or interpreted la g"ages. These la g"ages ha#e grow i pop"larity i rece t years a ! (12

21 8 Compili g 6rograms i cl"!e 6erl, 6ython, 6,6, :uby a ! ma y others. Scripte! la g"ages are e3ec"te! %y a special program calle! a interpreter. $ i terpreter i p"ts the program +ile a ! rea!s a ! e3ec"tes each i str"ctio co tai e! withi it. 9 ge eral, i terprete! programs e3ec"te m"ch more slowly tha compile! programs. This is !"e to the +act that each so"rce co!e i str"ctio i a i terprete! program is tra slate! e#ery time it is carrie! o"t, whereas with a compile! program, a so"rce co!e i str"ctio is o ly tra slate! o ce, a ! this tra slatio is perma e tly recor!e! i the +i al e3ec"ta%le +ile. So why are i terprete! la g"ages so pop"larC 0or ma y programmi g chores, the res"lts are ?+ast e o"gh,@ %"t the real a!#a tage is that it is ge erally +aster a ! easier to !e#elop i terprete! programs tha compile! programs. 6rograms are "s"ally !e#elope! i a repeati g cycle o+ co!e, compile, test. $s a program grows i siDe, the compilatio phase o+ the cycle ca %ecome ="ite lo g. 9 terprete! la g"ages remo#e the compilatio step a ! th"s spee! "p program !e#elopme t.

Compiling # C Program
*etIs compile somethi g. Je+ore we !o that howe#er, weIre goi g to ee! some tools like the compiler, li ker, a ! make. The C compiler "se! almost " i#ersally i the *i "3 e #iro me t is calle! gcc E;&) C CompilerF, origi ally writte %y :ichar! Stallma . >ost !istri%"tio s !o ot i stall gcc %y !e+a"lt. We ca check to see i+ the compiler is prese t like this,
[me@linuxbox ~]$ which gcc 6usr6bin6gcc

The res"lts i this e3ample i !icate that the compiler is i stalle!. Ti(, 5o"r !istri%"tio may ha#e a meta-package Ea collectio o+ packagesF +or so+tware !e#elopme t. 9+ so, co si!er i stalli g it i+ yo" i te ! to compile programs o yo"r system. 9+ yo"r system !oes ot pro#i!e a meta-package, try i stalli g the gcc a ! make packages. 7 ma y !istri%"tio s, this is s"++icie t to carry o"t the e3ercise %elow.

<btaining The Sour/e Code

0or o"r compili g e3ercise, we are goi g to compile a program +rom the ;&) 6ro4ect calle! diction. This is a ha !y little program that checks te3t +iles +or writi g ="ality a ! style. $s programs go, it is +airly small a ! easy to %"il!. (1K

Compili g $ C 6rogram 0ollowi g co #e tio , weIre +irst goi g to create a !irectory +or o"r so"rce co!e ame! src a ! the !ow loa! the so"rce co!e i to it "si g ft7,
[me@linuxbox ~]$ mkdir src [me@linuxbox ~]$ cd src [me@linuxbox src]$ ftp ftp.gnu.org Connected to ft7<gnu<org< !!( @L3 -T: server read1< Lame Nft7<gnu<org:meP: anon!mous !$( =ogin successful< Remote s1stem t17e is 3LAV< 3sing binar1 mode to transfer files< ft7D cd gnu/diction !"( 'irector1 successfull1 changed< ft7D ls !(( : RT command successful< Consider using :4*;< #"( Sere comes the director1 listing< -r9-r--r-# #(($ .""$% ./0%( 4ug !/ #00/ diction-(<)<tar<gC -r9-r--r-# #(($ .""$% 0(0") +ar (% !((! diction-#<(!<tar<gC -r9-r--r-# #(($ .""$% #%#(.! *e7 #) !(() diction-#<##<tar<gC !!. 'irector1 send 2< ft7D get diction-1.11.tar.glocal: diction-#<##<tar<gC remote: diction-#<##<tar<gC !(( : RT command successful< Consider using :4*;< #"( 7ening HAL4RU mode data connection for diction-#<##<tar<gC N#%#(.! b1tesP< !!. -ile send 2< #%#(.! b1tes received in (<#. secs N/%)<% kH6sP ft7D b!e !!# @oodb1e< [me@linuxbox src]$ ls diction-#<##<tar<gC

1ote5 Si ce we are the ?mai tai er@ o+ this so"rce co!e while we compile it, we will keep it i ~6src. So"rce co!e i stalle! %y yo"r !istri%"tio will %e i stalle! i 6usr6src, while so"rce co!e i te !e! +or "se %y m"ltiple "sers is "s"ally i stalle! i 6usr6local6src. $s we ca see, so"rce co!e is "s"ally s"pplie! i the +orm o+ a compresse! tar +ile. Sometimes calle! a ?tarball,@ this +ile co tai s the source tree, or hierarchy o+ !irectories a ! +iles that comprise the so"rce co!e. $+ter arri#i g at the +tp site, we e3ami e the list o+ tar +iles a#aila%le a ! select the ewest #ersio +or !ow loa!. )si g the get comma ! withi ft7, we copy the +ile +rom the +tp ser#er to the local machi e.

(1/

21 8 Compili g 6rograms 7 ce the tar +ile is !ow loa!e!, it m"st %e " packe!. This is !o e with the tar program,
[me@linuxbox src]$ tar x-f diction-1.11.tar.g[me@linuxbox src]$ ls diction-#<## diction-#<##<tar<gC

Ti(5 The diction program, like all ;&) 6ro4ect so+tware, +ollows certai sta !ar!s +or so"rce co!e packagi g. >ost other so"rce co!e a#aila%le i the *i "3 ecosystem also +ollows this sta !ar!. 7 e eleme t o+ the sta !ar! is that %y " packi g the so"rce co!e tar +ile, a !irectory will %e create! which co tai s the so"rce tree, a ! that this !irectory will %e ame! proAect-+#++, th"s co tai i g %oth the pro4ectIs ame a ! #ersio "m%er. This scheme allows easy i stallatio o+ m"ltiple #ersio s o+ the same program. Aowe#er, it is o+te a goo! i!ea to e3ami e the layo"t o+ the tree %e+ore " packi g it. Some pro4ects will ot create the !irectory, %"t i stea! will !eli#er the +iles !irectly i to the c"rre t !irectory. This will make a mess i yo"r otherwise well-orga iDe! src !irectory. To a#oi! this, "se the +ollowi g comma ! to e3ami e the co te ts o+ the tar +ile, tar tCvf tarfile Q head

6xamining The Sour/e Tree

) packi g the tar +ile res"lts i the creatio o+ a ew !irectory, ame! diction-#<##. This !irectory co tai s the so"rce tree. *etIs look i si!e,
[me@linuxbox src]$ cd diction-1.11 [me@linuxbox diction-#<##]$ ls config<guess diction<c geto7t<c config<h<in diction<7ot geto7t<h config<sub diction<s7ec geto7tOint<h configure diction<s7ec<in AL*T4== configure<in diction<texi<in install-sh C :UAL@ en +akefile<in de enO@H misc<c de<7o enO@H<7o misc<h diction<#<in geto7t#<c L&,*

nl nl<7o R&4'+& sentence<c sentence<h st1le<#<in st1le<c test

9 it, we see a "m%er o+ +iles. 6rograms %elo gi g to the ;&) 6ro4ect, as well as ma y (18

Compili g $ C 6rogram others, will s"pply the !oc"me tatio +iles R&4'+&, AL*T4==, L&,*, a ! C :UAL@. These +iles co tai the !escriptio o+ the program, i +ormatio o how to %"il! a ! i stall it, a ! its lice si g terms. 9t is aways a goo! i!ea to care+"lly rea! the R&4'+& a ! AL*T4== +iles %e+ore attempti g to %"il! the program. The other i teresti g +iles i this !irectory are the o es e !i g with <c a ! <h,
[me@linuxbox diction-#<##]$ ls *.c diction<c geto7t#<c geto7t<c misc<c sentence<c [me@linuxbox diction-#<##]$ ls *.h geto7t<h geto7tOint<h misc<h sentence<h

st1le<c

The <c +iles co tai the two C programs s"pplie! %y the package Est1le a ! dictionF, !i#i!e! i to mo!"les. 9t is commo practice +or large programs to %e %roke i to smaller, easier to ma age pieces. The so"rce co!e +iles are or!i ary te3t a ! ca %e e3ami e! with less,
[me@linuxbox diction-#<##]$ less diction.c

The <h +iles are k ow as header files. These, too, are or!i ary te3t. Aea!er +iles co tai !escriptio s o+ the ro"ti es i cl"!e! i a so"rce co!e +ile or li%rary. 9 or!er +or the compiler to co ect the mo!"les, it m"st recei#e a !escriptio o+ all the mo!"les ee!e! to complete the e tire program. &ear the %egi i g o+ the diction<c +ile, we see this li e,
^include _geto7t<h_

This i str"cts the compiler to rea! the +ile geto7t<h as it rea!s the so"rce co!e i diction<c i or!er to ?k ow@ whatIs i geto7t<c. The geto7t<c +ile s"pplies ro"ti es that are share! %y %oth the st1le a ! diction programs. $%o#e the include stateme t +or geto7t<h, we see some other include stateme ts s"ch as these,
^include ^include ^include ^include Xregex<hD Xstdio<hD Xstdlib<hD Xstring<hD

(19

21 8 Compili g 6rograms
^include Xunistd<hD

These also re+er to hea!er +iles, %"t they re+er to hea!er +iles that li#e o"tsi!e the c"rre t so"rce tree. They are s"pplie! %y the system to s"pport the compilatio o+ e#ery program. 9+ we look i 6usr6include, we ca see them,
[me@linuxbox diction-#<##]$ ls /usr/include

The hea!er +iles i this !irectory were i stalle! whe we i stalle! the compiler.

Bui ding The 'rogram

>ost programs %"il! with a simple two comma ! se="e ce,
<6configure make

The configure program is a shell script which is s"pplie! with the so"rce tree. 9ts 4o% is to a alyDe the build environment. >ost so"rce co!e is !esig e! to %e portable. That is, it is !esig e! to %"il! o more tha o e ki ! o+ ) i3-like system. J"t i or!er to !o that, the so"rce co!e may ee! to " !ergo slight a!4"stme ts !"ri g the %"il! to accommo!ate !i++ere ces %etwee systems. configure also checks to see that ecessary e3ter al tools a ! compo e ts are i stalle!. *etIs r" configure. Si ce configure is ot locate! where the shell ormally e3pects programs to %e locate!, we m"st e3plicitly tell the shell its locatio %y pre+i3i g the comma ! with <6 to i !icate that the program is locate! i the c"rre t worki g !irectory,
[me@linuxbox diction-#<##]$ ./configure

configure will o"tp"t a lot o+ messages as it tests a ! co +ig"res the %"il!. Whe it +i ishes, it will look somethi g like this,
checking libintl<h 7resence<<< 1es checking for libintl<h<<< 1es checking for librar1 containing gettext<<< none reBuired configure: creating <6config<status

(20

Compili g $ C 6rogram
config<status: creating +akefile config<status: creating diction<# config<status: creating diction<texi config<status: creating diction<s7ec config<status: creating st1le<# config<status: creating test6rundiction config<status: creating config<h [me@linuxbox diction-#<##]$

WhatIs importa t here is that there are o error messages. 9+ there were, the co +ig"ratio +aile!, a ! the program will ot %"il! " til the errors are correcte!. We see configure create! se#eral ew +iles i o"r so"rce !irectory. The most importa t o e is +akefile. +akefile is a co +ig"ratio +ile that i str"cts the make program e3actly how to %"il! the program. Witho"t it, make will re+"se to r" . +akefile is a or!i ary te3t +ile, so we ca #iew it,
[me@linuxbox diction-#<##]$ less Oakefile

The make program takes as i p"t a makefile Ewhich is ormally ame! +akefileF, which !escri%es the relatio ships a ! !epe !e cies amo g the compo e ts that comprise the +i ishe! program. The +irst part o+ the make+ile !e+i es #aria%les that are s"%stit"te! i later sectio s o+ the make+ile. 0or e3ample we see the li e,
CCM gcc

which !e+i es the C compiler to %e gcc. *ater i the make+ile, we see o e i sta ce where it gets "se!,
diction: diction<o sentence<o misc<o geto7t<o geto7t#<o $NCCP -o $@ $N='-=4@*P diction<o sentence<o misc<o ] geto7t<o geto7t#<o $N=AH*P

$ s"%stit"tio is per+orme! here, a ! the #al"e $NCCP is replace! %y gcc at r" time. >ost o+ the make+ile co sists o+ li es, which !e+i e a target, i this case the e3ec"ta%le +ile diction, a ! the +iles o which it is !epe !e t. The remai i g li es !escri%e the comma !EsF ee!e! to create the target +rom its compo e ts. We see i this e3ample that (2.

21 8 Compili g 6rograms the e3ec"ta%le +ile diction Eo e o+ the +i al e ! pro!"ctsF !epe !s o the e3iste ce o+ diction<o, sentence<o, misc<o, geto7t<o, a ! geto7t#<o. *ater o , i the make+ile, we see !e+i itio s o+ each o+ these as targets,
diction<o: geto7t<o: geto7t#<o: misc<o: sentence<o: st1le<o: diction<c config<h geto7t<h misc<h sentence<h geto7t<c geto7t<h geto7tOint<h geto7t#<c geto7t<h geto7tOint<h misc<c config<h misc<h sentence<c config<h misc<h sentence<h st1le<c config<h geto7t<h misc<h sentence<h

Aowe#er, we !o It see a y comma ! speci+ie! +or them. This is ha !le! %y a ge eral target, earlier i the +ile, that !escri%es the comma ! "se! to compile a y <c +ile i to a <o +ile,
<c<o: $NCCP -c $NC::-=4@*P $NC-=4@*P $X

This all seems #ery complicate!. Why ot simply list all the steps to compile the parts a ! %e !o e with itC The a swer to this will %ecome clear i a mome t. 9 the mea time, letIs r" make a ! %"il! o"r programs,
[me@linuxbox diction-#<##]$ make

The make program will r" , "si g the co te ts o+ +akefile to g"i!e its actio s. 9t will pro!"ce a lot o+ messages. Whe it +i ishes, we will see that all the targets are ow prese t i o"r !irectory,
[me@linuxbox diction-#<##]$ ls config<guess de<7o config<h diction config<h<in diction<# config<log diction<#<in config<status diction<c config<sub diction<o configure diction<7ot configure<in diction<s7ec C :UAL@ diction<s7ec<in de diction<texi

en enO@H enO@H<mo enO@H<7o geto7t#<c geto7t#<o geto7t<c geto7t<h geto7tOint<h geto7t<o

install-sh +akefile +akefile<in misc<c misc<h misc<o L&,* nl nl<mo nl<7o

sentence<c sentence<h sentence<o st1le st1le<# st1le<#<in st1le<c st1le<o test

(22

Compili g $ C 6rogram
de<mo diction<texi<in AL*T4== R&4'+&

$mo g the +iles, we see diction a ! st1le, the programs that we set o"t to %"il!. Co grat"latio s are i or!erL We 4"st compile! o"r +irst programs +rom so"rce co!eL J"t 4"st o"t o+ c"riosity, letIs r" make agai ,
[me@linuxbox diction-#<##]$ make make: Lothing to be done for JallK<

9t o ly pro!"ces this stra ge message. WhatIs goi g o C Why !i! It it %"il! the program agai C $h, this is the magic o+ make. :ather tha simply %"il!i g e#erythi g agai , make o ly %"il!s what ee!s %"il!i g. With all o+ the targets prese t, make !etermi e! that there was othi g to !o. We ca !emo strate this %y !eleti g o e o+ the targets a ! r" i g make agai to see what it !oes. *etIs get ri! o+ o e o+ the i terme!iate targets,
[me@linuxbox diction-#<##]$ rm getopt.o [me@linuxbox diction-#<##]$ make

We see that make re%"il!s it a ! re-li ks the diction a ! st1le programs, si ce they !epe ! o the missi g mo!"le. This %eha#ior also poi ts o"t a other importa t +eat"re o+ make, it keeps targets "p to !ate. make i sists that targets %e ewer tha their !epe !e cies. This makes per+ect se se, as a programmer will o+te "p!ate a %it o+ so"rce co!e a ! the "se make to %"il! a ew #ersio o+ the +i ishe! pro!"ct. make e s"res that e#erythi g that ee!s %"il!i g %ase! o the "p!ate! co!e is %"ilt. 9+ we "se the touch program to ?"p!ate@ o e o+ the so"rce co!e +iles, we ca see this happe ,
[me@linuxbox -r9xr-xr-x # -r9-r--r-- # [me@linuxbox [me@linuxbox -r9xr-xr-x # -r9-r--r-- # [me@linuxbox diction-#<##]$ me me me me diction-#<##]$ diction-#<##]$ me me me me diction-#<##]$ ls -l diction getopt.c $)#.% !((0-($-(" (.:#% $$#!" !(()-($-$( #):%" touch getopt.c ls -l diction getopt.c $)#.% !((0-($-(" (.:#% $$#!" !((0-($-(" (.:!$ make

diction geto7t<c diction geto7t<c

$+ter make r" s, we see that it has restore! the target to %ei g !epe !e cy,

ewer tha

the

(2(

21 8 Compili g 6rograms
[me@linuxbox diction-#<##]$ -r9xr-xr-x # me me -r9-r--r-- # me me ls -l diction getopt.c $)#.% !((0-($-(" (.:!% diction $$#!" !((0-($-(" (.:!$ geto7t<c

The a%ility o+ make to i tellige tly %"il! o ly what ee!s %"il!i g is a great %e e+it to programmers. While the time sa#i gs may ot %e #ery appare t with o"r small pro4ect, it is #ery sig i+ica t with larger pro4ects. :emem%er, the *i "3 ker el Ea program that " !ergoes co ti "o"s mo!i+icatio a ! impro#eme tF co tai s se#eral million li es o+ co!e.

Insta ing The 'rogram

Well-package! so"rce co!e will o+te i cl"!e a special make target calle! install. This target will i stall the +i al pro!"ct i a system !irectory +or "se. )s"ally, this !irectory is 6usr6local6bin, the tra!itio al locatio +or locally %"ilt so+tware. Aowe#er, this !irectory is ot ormally writa%le %y or!i ary "sers, so we m"st %ecome the s"per"ser to per+orm the i stallatio ,
[me@linuxbox diction-#<##]$ sudo make install

$+ter we per+orm the i stallatio , we ca check that the program is rea!y to go,
[me@linuxbox diction-#<##]$ which diction 6usr6local6bin6diction [me@linuxbox diction-#<##]$ man diction

$ ! there we ha#e itL

Summing -p
9 this chapter, we ha#e see how three simple comma !s,
<6configure make make install

ca %e "se! to %"il! ma y so"rce co!e packages. We ha#e also see the importa t role that make plays i the mai te a ce o+ programs. The make program ca %e "se! +or a y task that ee!s to mai tai a target-!epe !e cy relatio ship, ot 4"st +or compili g so"rce co!e. (21

0"rther :ea!i g

;urther 1eading

The Wikipe!ia has goo! articles o compilers a ! the make program, http,--e .wikipe!ia.org-wiki-Compiler http,--e .wikipe!ia.org-wiki->akeOEso+twareF The @>0 !ake !anual, http,--www.g ".org-so+tware-make-ma "al-htmlO o!e-i !e3.html

(22

6art 2 8 Writi g Shell Scripts

'art & A Writing She S/ri5ts

(2/

22 8 Writi g 5o"r 0irst Script

2& Writing 8our (irst Scri#t

9 the prece!i g chapters, we ha#e assem%le! a arse al o+ comma ! li e tools. While these tools ca sol#e ma y ki !s o+ comp"ti g pro%lems, we are still limite! to ma "ally "si g them o e %y o e o the comma ! li e. Wo"l! It it %e great i+ we co"l! get the shell to !o more o+ the workC We ca . Jy 4oi i g o"r tools together i to programs o+ o"r ow !esig , the shell ca carry o"t comple3 se="e ces o+ tasks all %y itsel+. We ca e a%le it to !o this %y writi g shell scripts.

+hat #re Shell Scripts.

9 the simplest terms, a shell script is a +ile co tai i g a series o+ comma !s. The shell rea!s this +ile a ! carries o"t the comma !s as tho"gh they ha#e %ee e tere! !irectly o the comma ! li e. The shell is somewhat " i="e, i that it is %oth a power+"l comma ! li e i ter+ace to the system a ! a scripti g la g"age i terpreter. $s we will see, most o+ the thi gs that ca %e !o e o the comma ! li e ca %e !o e i scripts, a ! most o+ the thi gs that ca %e !o e i scripts ca %e !o e o the comma ! li e. We ha#e co#ere! ma y shell +eat"res, %"t we ha#e +oc"se! o those +eat"res most o+te "se! !irectly o the comma ! li e. The shell also pro#i!es a set o+ +eat"res "s"ally E%"t ot alwaysF "se! whe writi g programs.

3o4 To +rite # Shell Script

To s"ccess+"lly create a ! r" a shell script, we ee! to !o three thi gs, .. 'rite a scri(t. Shell scripts are or!i ary te3t +iles. So we ee! a te3t e!itor to write them. The %est te3t e!itors will pro#i!e synta+ highlighting, allowi g "s to see a color-co!e! #iew o+ the eleme ts o+ the script. Sy ta3 highlighti g will help "s spot certai ki !s o+ commo errors. vim, gedit, kate, a ! ma y other e!itors are goo! ca !i!ates +or writi g scripts. 2. :ake the scri(t executable. The system is rather +"ssy a%o"t ot letti g a y ol! te3t +ile %e treate! as a program, a ! +or goo! reaso L We ee! to set the script +ileIs permissio s to allow e3ec"tio . (28

Aow To Write $ Shell Script (. Put the scri(t some+here the shell can "ind it. The shell a"tomatically searches certai !irectories +or e3ec"ta%le +iles whe o e3plicit path ame is speci+ie!. 0or ma3im"m co #e ie ce, we will place o"r scripts i these !irectories.

Script ;ile ;ormat

9 keepi g with programmi g tra!itio , weIll create a ?hello worl!@ program to !emo strate a e3tremely simple script. So letIs +ire "p o"r te3t e!itors a ! e ter the +ollowi g script,
UL/bin/bash U +his is our first script. echo %Rello WorldL%

The last li e o+ o"r script is pretty +amiliar, 4"st a echo comma ! with a stri g arg"me t. The seco ! li e is also +amiliar. 9t looks like a comme t that we ha#e see "se! i ma y o+ the co +ig"ratio +iles we ha#e e3ami e! a ! e!ite!. 7 e thi g a%o"t comme ts i shell scripts is that they may also appear at the e ! o+ li es, like so,
echo KSello ,orldGK ^ This is a comment too

E#erythi g +rom the ^ sym%ol o war! o the li e is ig ore!. *ike ma y thi gs, this works o the comma ! li e, too,
[me@linuxbox ~]$ echo %Rello WorldL% U +his is a comment too Sello ,orldG

Tho"gh comme ts are o+ little "se o the comma ! li e, they will work. The +irst li e o+ o"r script is a little mysterio"s. 9t looks like it sho"l! %e a comme t, si ce it starts with ^, %"t it looks too p"rpose+"l to %e 4"st that. The ^G character se="e ce is, i +act, a special co str"ct calle! a shebang. The she%a g is "se! to tell the system the ame o+ the i terpreter that sho"l! %e "se! to e3ec"te the script that +ollows. E#ery shell script sho"l! i cl"!e this as its +irst li e. *etIs sa#e o"r script +ile as helloO9orld.

(29

22 8 Writi g 5o"r 0irst Script

$xecuta0le Permissions
The e3t thi g we ha#e to !o is make o"r script e3ec"ta%le. This is easily !o e "si g chmod,
[me@linuxbox -r9-r--r-- # [me@linuxbox [me@linuxbox -r9xr-xr-x # ~]$ ls -l hello.world me me .$ !((0-($-() #(:#( helloO9orld ~]$ chmod C"" hello.world ~]$ ls -l hello.world me me .$ !((0-($-() #(:#( helloO9orld

There are two commo permissio setti gs +or scriptsH /22 +or scripts that e#eryo e ca e3ec"te, a ! /00 +or scripts that o ly the ow er ca e3ec"te. &ote that scripts m"st %e rea!a%le i or!er to %e e3ec"te!.

Script ;ile Location

With the permissio s set, we ca ow e3ec"te o"r script,

[me@linuxbox ~]$ ./hello.world Sello ,orldG

9 or!er +or the script to r" , we m"st prece!e the script ame with a e3plicit path. 9+ we !o It, we get this,
[me@linuxbox ~]$ hello.world bash: helloO9orld: command not found

Why is thisC What makes o"r script !i++ere t +rom other programsC $s it t"r s o"t, othi g. 7"r script is +i e. 9ts locatio is the pro%lem. Jack i Chapter .2, we !isc"sse! the :4TS e #iro me t #aria%le a ! its e++ect o how the system searches +or e3ec"ta%le programs. To recap, the system searches a list o+ !irectories each time it ee!s to +i ! a e3ec"ta%le program, i+ o e3plicit path is speci+ie!. This is how the system k ows to e3ec"te 6bin6ls whe we type ls at the comma ! li e. The 6bin !irectory is o e o+ the !irectories that the system a"tomatically searches. The list o+ !irectories is hel! withi a e #iro me t #aria%le ame! :4TS. The :4TS #aria%le co tai s a colo separate! list o+ !irectories to %e searche!. We ca #iew the co te ts o+ :4TS,

(K0

Script 0ile *ocatio

[me@linuxbox ~]$ echo 3B:+R 6home6me6bin:6usr6local6sbin:6usr6local6bin:6usr6sbin:6usr6bin:6sbin: 6bin:6usr6games

Aere we see o"r list o+ !irectories. 9+ o"r script were locate! i a y o+ the !irectories i the list, o"r pro%lem wo"l! %e sol#e!. &otice the +irst !irectory i the list, 6home6me6bin. >ost *i "3 !istri%"tio s co +ig"re the :4TS #aria%le to co tai a bin !irectory i the "serIs home !irectory, to allow "sers to e3ec"te their ow programs. So i+ we create the bin !irectory a ! place o"r script withi it, it sho"l! start to work like other programs,
[me@linuxbox ~]$ mkdir bin [me@linuxbox ~]$ mv hello.world bin [me@linuxbox ~]$ hello.world Sello ,orldG

$ ! so it !oes. 9+ the :4TS #aria%le !oes ot co tai the !irectory, we ca easily a!! it %y i cl"!i g this li e i o"r <bashrc +ile,
export B:+R$~/bin2J3B:+RJ

$+ter this cha ge is ma!e, it will take e++ect i each ew termi al sessio . To apply the cha ge to the c"rre t termi al sessio , we m"st ha#e the shell re-rea! the <bashrc +ile. This ca %e !o e %y ?so"rci g@ it,
[me@linuxbox ~]$ . .bashrc

The !ot E<F comma ! is a sy o ym +or the source comma !, a shell %"ilti which rea!s a speci+ie! +ile o+ shell comma !s a ! treats it like i p"t +rom the key%oar!. 1ote5 )%" t" a"tomatically a!!s the ~6bin !irectory to the :4TS #aria%le i+ the ~6bin !irectory e3ists whe the "serIs <bashrc +ile is e3ec"te!. So, o )%" t" systems, i+ we create the ~6bin !irectory a ! the log o"t a ! log i agai , e#erythi g works.

(K.

22 8 Writi g 5o"r 0irst Script

+ood Lo/ations 2or S/ri5ts

The ~6bin !irectory is a goo! place to p"t scripts i te !e! +or perso al "se. 9+ we write a script that e#eryo e o a system is allowe! to "se, the tra!itio al locatio is 6usr6local6bin. Scripts i te !e! +or "se %y the system a!mi istrator are o+te locate! i 6usr6local6sbin. 9 most cases, locally s"pplie! so+tware, whether scripts or compile! programs, sho"l! %e place! i the 6usr6local hierarchy a ! ot i 6bin or 6usr6bin< These !irectories are speci+ie! %y the *i "3 0ilesystem Aierarchy Sta !ar! to co tai o ly +iles s"pplie! a ! mai tai e! %y the *i "3 !istri%"tor.

>ore ;ormatting Tric(s

7 e o+ the key goals o+ serio"s script writi g is ease o+ maintenanceH that is, the ease with which a script may %e mo!i+ie! %y its a"thor or others to a!apt it to cha gi g ee!s. >aki g a script easy to rea! a ! " !ersta ! is o e way to +acilitate easy mai te a ce.

Long <5tion ,ames

>a y o+ the comma !s we ha#e st"!ie! +eat"re %oth short a ! lo g optio ames. 0or i sta ce, the ls comma ! has ma y optio s that ca %e e3presse! i either short or lo g +orm. 0or e3ample,
[me@linuxbox ~]$ ls -ad

a !,
[me@linuxbox ~]$ ls --all --director!

are e="i#ale t comma !s. 9 the i terests o+ re!"ce! typi g, short optio s are pre+erre! whe e teri g optio s o the comma ! li e, %"t whe writi g scripts, lo g optio s ca pro#i!e impro#e! rea!a%ility.

Indentation And Line Continuation

Whe employi g lo g comma !s, rea!a%ility ca %e e ha ce! %y sprea!i g the comma ! o#er se#eral li es. 9 Chapter .8, we looke! at a partic"larly lo g e3ample o+ the +i ! comma !,

(K2

>ore 0ormatti g Tricks

[me@linuxbox ~]$ find pla!ground K4 -t!pe f -not -perm *N** -exec chmod *N** `9>a `#a K6 -or K4 -t!pe d -not -perm *C11 -exec chmod *C11 `9>a `#a K6

7%#io"sly, this comma ! is a little har! to +ig"re o"t at +irst gla ce. 9 a script, this comma ! might %e easier to " !ersta ! i+ writte this way,
find 7la1ground ] ]N ] -t17e f ] -not -7erm (.(( ] -exec chmod (.(( a[\b aTb ] ]P ] -or ] ]N ] -t17e d ] -not -7erm ()## ] -exec chmod ()## a[\b aTb ] ]P

Jy "si g li e co ti "atio s E%ackslash-li e+ee! se="e cesF a ! i !e tatio , the logic o+ this comple3 comma ! is more clearly !escri%e! to the rea!er. This tech i="e works o the comma ! li e, too, tho"gh it is sel!om "se!, as it is #ery awkwar! to type a ! e!it. 7 e !i++ere ce %etwee a script a ! the comma ! li e is that the script may employ ta% characters to achie#e i !e tatio , whereas the comma ! li e ca ot, si ce ta%s are "se! to acti#ate completio .

Con"iguring vim ;or Script +riting

The vim te3t e!itor has ma y, ma y co +ig"ratio setti gs. There are se#eral commo optio s that ca +acilitate script writi g, 2s!ntax on t"r s o sy ta3 highlighti g. With this setti g, !i++ere t eleme ts o+ shell sy ta3 will %e !isplaye! i !i++ere t colors whe #iewi g a script. This is help+"l +or i!e ti+yi g certai ki !s o+ programmi g errors. 9t looks cool, too. &ote that +or this +eat"re to work, yo" m"st ha#e a complete #ersio o+ vim i stalle!, a ! the +ile yo" are e!iti g m"st ha#e a she%a g i !icati g the +ile is a shell script. 9+ yo" ha#e !i++ic"lty with the comma ! a%o#e, try 2set s!ntax$sh i stea!.

(K(

22 8 Writi g 5o"r 0irst Script

2set hlsearch t"r s o the optio to highlight search res"lts. Say we search +or the wor! ?echo.@ With this optio o , each i sta ce o+ the wor! will %e highlighte!. 2set tabstop$A sets the "m%er o+ col"m s occ"pie! %y a ta% character. The !e+a"lt is eight col"m s. Setti g the #al"e to +o"r Ewhich is a commo practiceF allows lo g li es to +it more easily o the scree . 2set autoindent t"r s o the ?a"to i !e t@ +eat"re. This ca"ses vim to i !e t a ew li e the same amo" t as the li e 4"st type!. This spee!s "p typi g o ma y ki !s o+ programmi g co str"cts. To stop i !e tatio , type Ctrl-d< These cha ges ca %e ma!e perma e t %y a!!i g these comma !s Ewitho"t the lea!i g colo charactersF to yo"r ~6<vimrc +ile.

Summing -p
9 this +irst chapter o+ scripti g, we ha#e looke! at how scripts are writte a ! ma!e to easily e3ec"te o o"r system. We also saw how we may "se #ario"s +ormatti g tech i="es to impro#e the rea!a%ility Ea ! th"s, the mai tai a%ilityF o+ o"r scripts. 9 +"t"re chapters, ease o+ mai te a ce will come "p agai a ! agai as a ce tral pri ciple i goo! script writi g.

;urther 1eading

0or ?Aello Worl!@ programs a ! e3amples i #ario"s programmi g la g"ages, see, http,--e .wikipe!ia.org-wiki-AelloOworl! This Wikipe!ia article talks more a%o"t the she%a g mecha ism, http,--e .wikipe!ia.org-wiki-She%a gOE) i3F

(K1

2K 8 Starti g $ 6ro4ect

2+ Starting ) 2ro9ect
Starti g with this chapter, we will %egi to %"il! a program. The p"rpose o+ this pro4ect is to see how #ario"s shell +eat"res are "se! to create programs a !, more importa tly, create good programs. The program we will write is a report generator. 9t will prese t #ario"s statistics a%o"t o"r system a ! its stat"s, a ! will pro!"ce this report i AT>* +ormat, so we ca #iew it with a we% %rowser s"ch as 0ire+o3 or Mo ="eror. 6rograms are "s"ally %"ilt "p i a series o+ stages, with each stage a!!i g +eat"res a ! capa%ilities. The +irst stage o+ o"r program will pro!"ce a #ery mi imal AT>* page that co tai s o system i +ormatio . That will come later.

;irst StageI >inimal Document

The +irst thi g we ee! to k ow is the +ormat o+ a well-+orme! AT>* !oc"me t. 9t looks like this,
XST+=D XS&4'D XTAT=&D:age TitleX6TAT=&D X6S&4'D XH 'UD :age bod1< X6H 'UD X6ST+=D

9+ we e ter this i to o"r te3t e!itor a ! sa#e the +ile as foo<html, we ca "se the +ollowi g ):* i 0ire+o3 to #iew the +ile, file:666home6username6foo<html The +irst stage o+ o"r program will %e a%le to o"tp"t this AT>* +ile to sta !ar! o"tp"t. We ca write a program to !o this pretty easily. *etIs start o"r te3t e!itor a ! create a ew +ile ame! ~6bin6s1sOinfoO7age,

(K2

2K 8 Starti g $ 6ro4ect
[me@linuxbox ~]$ vim ~/bin/s!s.info.page

a ! e ter the +ollowi g program,

UL/bin/bash U Brogram to output a s!stem information page echo echo echo echo echo echo echo echo J/R+OT&J J /RG:1&J J /+P+TG&Bage +itle//+P+TG&J J //RG:1&J J /<S1^&J J Bage bod!.J J //<S1^&J J//R+OT&J

7"r +irst attempt at this pro%lem co tai s a she%a g, a comme t Ealways a goo! i!eaF a ! a se="e ce o+ echo comma !s, o e +or each li e o+ o"tp"t. $+ter sa#i g the +ile, weIll make it e3ec"ta%le a ! attempt to r" it,
[me@linuxbox ~]$ chmod C"" ~/bin/s!s.info.page [me@linuxbox ~]$ s!s.info.page

Whe the program r" s, we sho"l! see the te3t o+ the AT>* !oc"me t !isplaye! o the scree , si ce the echo comma !s i the script se ! their o"tp"t to sta !ar! o"tp"t. WeIll r" the program agai a ! re!irect the o"tp"t o+ the program to the +ile s1sOinfoO7age<html, so that we ca #iew the res"lt with a we% %rowser,
[me@linuxbox ~]$ s!s.info.page & s!s.info.page.html [me@linuxbox ~]$ firefox s!s.info.page.html

So +ar, so goo!. Whe writi g programs, itIs always a goo! i!ea to stri#e +or simplicity a ! clarity. >ai te a ce is easier whe a program is easy to rea! a ! " !ersta !, ot to me tio , it ca make the program easier to write %y re!"ci g the amo" t o+ typi g. 7"r c"rre t #ersio o+ the program works +i e, %"t it co"l! %e simpler. We co"l! act"ally com%i e all the echo comma !s i to o e, which will certai ly make it easier to a!! more li es to the programIs o"tp"t. So, letIs cha ge o"r program to this, (KK

0irst Stage, >i imal 'oc"me t

^G6bin6bash ^ :rogram to out7ut a s1stem information 7age echo J/R+OT& /RG:1& /+P+TG&Bage +itle//+P+TG& //RG:1& /<S1^& Bage bod!. //<S1^& //R+OT&J

$ ="ote! stri g may i cl"!e ewli es, a ! there+ore co tai m"ltiple li es o+ te3t. The shell will keep rea!i g the te3t " til it e co" ters the closi g ="otatio mark. 9t works this way o the comma ! li e, too,
[me@linuxbox ~]$ echo J/R+OT& D /RG:1& D /+P+TG&Bage +itle//+P+TG& D //RG:1& D /<S1^& D Bage bod!. D //<S1^& D //R+OT&J

The lea!i g ?X@ character is the shell prompt co tai e! i the :*! shell #aria%le. 9t appears whe e#er we type a m"lti-li e stateme t i to the shell. This +eat"re is a little o%sc"re right ow, %"t later, whe we co#er m"lti-li e programmi g stateme ts, it will t"r o"t to %e ="ite ha !y.

Second StageI #dding # Little Data

&ow that o"r program ca ge erate a mi imal !oc"me t, letIs p"t some !ata i the report. To !o this, we will make the +ollowi g cha ges,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age echo _XST+=D XS&4'D

XTAT=&DF!stem Pnformation HeportX6TAT=&D

(K/

2K 8 Starti g $ 6ro4ect
X6S&4'D XH 'UD X6ST+=D_ X6H 'UD

XS#DF!stem Pnformation HeportX6S#D

We a!!e! a page title a ! a hea!i g to the %o!y o+ the report.

Varia0les #nd Constants

There is a iss"e with o"r script, howe#er. &otice how the stri g ?System 9 +ormatio :eport@ is repeate!C With o"r ti y script itIs ot a pro%lem, %"t letIs imagi e that o"r script was really lo g a ! we ha! m"ltiple i sta ces o+ this stri g. 9+ we wa te! to cha ge the title to somethi g else, we wo"l! ha#e to cha ge it i m"ltiple places, which co"l! %e a lot o+ work. What i+ we co"l! arra ge the script so that the stri g o ly appeare! o ce a ! ot m"ltiple timesC That wo"l! make +"t"re mai te a ce o+ the script m"ch easier. AereIs how we co"l! !o that,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age title$JF!stem Pnformation HeportJ echo _XST+=D XS&4'D X6S&4'D XH 'UD X6ST+=D_ X6H 'UD XTAT=&D3titleX6TAT=&D XS#D3titleX6S#D

Jy creati g a variable ame! title a ! assig i g it the #al"e ?System 9 +ormatio :eport,@ we ca take a!#a tage o+ parameter e3pa sio a ! place the stri g i m"ltiple locatio s. So, how !o we create a #aria%leC Simple, we 4"st "se it. Whe the shell e co" ters a #aria%le, it a"tomatically creates it. This !i++ers +rom ma y programmi g la g"ages i which #aria%les m"st %e e3plicitly declared or !e+i e! %e+ore "se. The shell is #ery la3 a%o"t this, which ca lea! to some pro%lems. 0or e3ample, co si!er this sce ario playe! o"t o the comma ! li e,

(K8

Raria%les $ ! Co sta ts
[me@linuxbox ~]$ foo$J!esJ [me@linuxbox ~]$ echo 3foo 1es [me@linuxbox ~]$ echo 3fool [me@linuxbox ~]$

We +irst assig the #al"e ?yes@ to the #aria%le foo, the !isplay its #al"e with echo. &e3t we !isplay the #al"e o+ the #aria%le ame misspelle! as ?+ool@ a ! get a %la k res"lt. This is %eca"se the shell happily create! the #aria%le fool whe it e co" tere! it, a ! ga#e it the !e+a"lt #al"e o+ othi g, or empty. 0rom this, we lear that we m"st pay close atte tio to o"r spelli gL 9tIs also importa t to " !ersta ! what really happe e! i this e3ample. 0rom o"r pre#io"s look at how the shell per+orms e3pa sio s, we k ow that the comma !,
[me@linuxbox ~]$ echo $foo

" !ergoes parameter e3pa sio a ! res"lts i ,

[me@linuxbox ~]$ echo 1es

Whereas the comma !,

[me@linuxbox ~]$ echo $fool

e3pa !s i to,
[me@linuxbox ~]$ echo

The empty #aria%le e3pa !s i to othi gL This ca play ha#oc with comma !s that re="ire arg"me ts. AereIs a e3ample,
[me@linuxbox ~]$ foo$foo.txt [me@linuxbox ~]$ foo1$foo1.txt [me@linuxbox ~]$ cp 3foo 3fool c7: missing destination file o7erand after Jfoo<txtK

(K9

2K 8 Starti g $ 6ro4ect
Tr1 Jc7 --hel7K for more information<

We assig #al"es to two #aria%les, foo a ! foo#. We the per+orm a c7, %"t misspell the ame o+ the seco ! arg"me t. $+ter e3pa sio , the c7 comma ! is o ly se t o e arg"me t, tho"gh it re="ires two. There are some r"les a%o"t #aria%le ames, .. Raria%le ames may co sist o+ alpha "meric characters Eletters a ! "m%ersF a ! " !erscore characters. 2. The +irst character o+ a #aria%le ame m"st %e either a letter or a " !erscore. (. Spaces a ! p" ct"atio sym%ols are ot allowe!. The wor! ?#aria%le@ implies a #al"e that cha ges, a ! i ma y applicatio s, #aria%les are "se! this way. Aowe#er, the #aria%le i o"r applicatio , title, is "se! as a constant. $ co sta t is 4"st like a #aria%le i that it has a ame a ! co tai s a #al"e. The !i++ere ce is that the #al"e o+ a co sta t !oes ot cha ge. 9 a applicatio that per+orms geometric calc"latio s, we might !e+i e :A as a co sta t, a ! assig it the #al"e o+ $<#%#", i stea! o+ "si g the "m%er literally thro"gho"t o"r program. The shell makes o !isti ctio %etwee #aria%les a ! co sta tsH they are mostly +or the programmerIs co #e ie ce. $ commo co #e tio is to "se "pper case letters to !esig ate co sta ts a ! lower case letters +or tr"e #aria%les. We will mo!i+y o"r script to comply with this co #e tio ,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age +P+TGM_*1stem Anformation Re7ort 7or 3RSF+?:OG_ echo _XST+=D XS&4'D X6S&4'D XH 'UD X6ST+=D_ X6H 'UD XTAT=&D$+P+TGX6TAT=&D XS#D$+P+TGX6S#D

We also took the opport" ity to 4aDD "p o"r title %y a!!i g the #al"e o+ the shell #aria%le S *TL4+&. This is the etwork ame o+ the machi e.

(/0

Raria%les $ ! Co sta ts 1ote, The shell act"ally !oes pro#i!e a way to e +orce the imm"ta%ility o+ co sta ts, thro"gh the "se o+ the declare %"ilti comma ! with the -r Erea!o lyF optio . Aa! we assig e! TAT=& this way, declare -r TAT=&M_:age Title_ the shell wo"l! pre#e t a y s"%se="e t assig me t to TAT=&. This +eat"re is rarely "se!, %"t it e3ists +or #ery +ormal scripts.

Assigning ?a ues To ?ariab es And Constants

Aere is where o"r k owle!ge o+ e3pa sio really starts to pay o++. $s we ha#e see , #aria%les are assig e! #al"es this way, variableMvalue where variable is the ame o+ the #aria%le a ! value is a stri g. ) like some other programmi g la g"ages, the shell !oes ot care a%o"t the type o+ !ata assig e! to a #aria%leH it treats them all as stri gs. 5o" ca +orce the shell to restrict the assig me t to i tegers %y "si g the declare comma ! with the -i optio , %"t, like setti g #aria%les as rea!-o ly, this is rarely !o e. &ote that i a assig me t, there m"st %e o spaces %etwee the #aria%le ame, the e="als sig , a ! the #al"e. So what ca the #al"e co sist o+C $ ythi g that we ca e3pa ! i to a stri g,
aMC bM_a string_ cM_a string and $b_ dM$Nls -l foo<txtP eM$NN" E )PP fM_]t]ta string]n_ ^ ^ ^ ^ ^ ^ ^ 4ssign the string _C_ to variable a< &mbedded s7aces must be 9ithin Buotes< ther ex7ansions such as variables can be ex7anded into the assignment< Results of a command< 4rithmetic ex7ansion< &sca7e seBuences such as tabs and ne9lines<

>"ltiple #aria%le assig me ts may %e !o e o a si gle li e,

aM" bM_a string_

'"ri g e3pa sio , #aria%le ames may %e s"rro" !e! %y optio al c"rly %races ?cd@. This is "se+"l i cases where a #aria%le ame %ecomes am%ig"o"s !"e to its s"rro" !i g (/.

2K 8 Starti g $ 6ro4ect co te3t. Aere, we try to cha ge the ame o+ a +ile +rom m1file to m1file#, "si g a #aria%le,
[me@linuxbox ~]$ filename$Jm!fileJ [me@linuxbox ~]$ touch 3filename [me@linuxbox ~]$ mv 3filename 3filename1 mv: missing destination file o7erand after Jm1fileK Tr1 Jmv --hel7K for more information<

This attempt +ails %eca"se the shell i terprets the seco ! arg"me t o+ the mv comma ! as a ew Ea ! emptyF #aria%le. The pro%lem ca %e o#ercome this way,
[me@linuxbox ~]$ mv 3filename 39filename>1

Jy a!!i g the s"rro" !i g %races, the shell o lo ger i terprets the traili g # as part o+ the #aria%le ame. WeIll take this opport" ity to a!! some !ata to o"r report, amely the !ate a ! time the report was create! a ! the "ser ame o+ the creator,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age TAT=&M_*1stem Anformation Re7ort -or $S *TL4+&_ =EHHG?+.+POG$34date 5J8x 8r 8@J6 +POG.F+:OB$JQenerated 3=EHHG?+.+POG; b! 3EFGHJ echo _XST+=D XS&4'D X6S&4'D XH 'UD XS#D$TAT=&X6S#D /B&3+POG.F+:OB//B& X6ST+=D_ X6H 'UD

XTAT=&D$TAT=&X6TAT=&D

3ere Documents
WeI#e looke! at two !i++ere t metho!s o+ o"tp"tti g o"r te3t, %oth "si g the echo (/2

Aere 'oc"me ts comma !. There is a thir! way calle! a here document or here script. $ here !oc"me t is a a!!itio al +orm o+ 9-7 re!irectio i which we em%e! a %o!y o+ te3t i to o"r script a ! +ee! it i to the sta !ar! i p"t o+ a comma !. 9t works like this, command XX token text token where command is the ame o+ comma ! that accepts sta !ar! i p"t a ! token is a stri g "se! to i !icate the e ! o+ the em%e!!e! te3t. WeIll mo!i+y o"r script to "se a here !oc"me t,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age TAT=&M_*1stem Anformation Re7ort -or $S *TL4+&_ C3RR&LTOTA+&M$Ndate 8_5x 5r 5I_P TA+&O*T4+:M_@enerated $C3RR&LTOTA+&> b1 $3*&R_ cat // .GS7. XST+=D XS&4'D X6S&4'D XH 'UD X6H 'UD X6ST+=D .GS7. XTAT=&D$TAT=&X6TAT=&D XS#D$TAT=&X6S#D X:D$TA+&O*T4+:X6:D

9 stea! o+ "si g echo, o"r script ow "ses cat a ! a here !oc"me t. The stri g O& -O Emea i g ?E ! 7+ 0ile,@ a commo co #e tio F was selecte! as the toke , a ! marks the e ! o+ the em%e!!e! te3t. &ote that the toke m"st appear alo e a ! that there m"st ot %e traili g spaces o the li e. So whatIs the a!#a tage o+ "si g a here !oc"me tC 9tIs mostly the same as echo, e3cept that, %y !e+a"lt, si gle a ! !o"%le ="otes withi here !oc"me ts lose their special mea i g to the shell. Aere is a comma ! li e e3ample,
[me@linuxbox ~]$ foo$Jsome textJ [me@linuxbox ~]$ cat // .GS7. D 3foo

(/(

2K 8 Starti g $ 6ro4ect
D J3fooJ D %3foo% D K3foo D .GS7. some text _some text_ Ksome textK $foo

$s we ca see, the shell pays o atte tio to the ="otatio marks. 9t treats them as or!i ary characters. This allows "s to em%e! ="otes +reely withi a here !oc"me t. This co"l! t"r o"t to %e ha !y +or o"r report program. Aere !oc"me ts ca %e "se! with a y comma ! that accepts sta !ar! i p"t. 9 this e3ample, we "se a here !oc"me t to pass a series o+ comma !s to the ft7 program i or!er to retrie#e a +ile +rom a remote 0T6 ser#er,
^G6bin6bash ^ *cri7t to retrieve a file via -T: -T:O*&R;&RMft7<nl<debian<org -T:O:4TSM6debian6dists6lenn16main6installer-i$/.6current6images6cdrom R&+ T&O-A=&Mdebian-cdOinfo<tar<gC ft7 -n XX O& -O o7en $-T:O*&R;&R user anon1mous me@linuxbox cd $-T:O:4TS hash get $R&+ T&O-A=& b1e O& -O ls -l $R&+ T&O-A=&

9+ we cha ge the re!irectio operator +rom ?YY@ to ?YY-@, the shell will ig ore lea!i g ta% characters i the here !oc"me t. This allows a here !oc"me t to %e i !e te!, which ca impro#e rea!a%ility,
^G6bin6bash ^ *cri7t to retrieve a file via -T: -T:O*&R;&RMft7<nl<debian<org

(/1

Aere 'oc"me ts
-T:O:4TSM6debian6dists6lenn16main6installer-i$/.6current6images6cdrom R&+ T&O-A=&Mdebian-cdOinfo<tar<gC ft7 -n XX- O& -O o7en $-T:O*&R;&R user anon1mous me@linuxbox cd $-T:O:4TS hash get $R&+ T&O-A=& b1e O& -O ls -l $R&+ T&O-A=&

Summing -p
9 this chapter, we starte! a pro4ect that will carry "s thro"gh the process o+ %"il!i g a s"ccess+"l script. We i tro!"ce! the co cept o+ #aria%les a ! co sta ts a ! how they ca %e employe!. They are the +irst o+ ma y applicatio s we will +i ! +or parameter e3pa sio . We also looke! at how to pro!"ce o"tp"t +rom o"r script, a ! #ario"s metho!s +or em%e!!i g %locks o+ te3t.

;urther 1eading

0or more i +ormatio a%o"t AT>*, see the +ollowi g articles a ! t"torials, http,--e .wikipe!ia.org-wiki-Atml http,--e .wiki%ooks.org-wiki-AT>*O6rogrammi g http,--html. et-t"torials-htmlThe bash ma page i cl"!es a sectio e title! ?AE:E '7C)>E&TS,@ which has a +"ll !escriptio o+ this +eat"re.

(/2

2/ 8 Top-'ow 'esig

2. To#:*o5n *esign
$s programs get larger a ! more comple3, they %ecome more !i++ic"lt to !esig , co!e a ! mai tai . $s with a y large pro4ect, it is o+te a goo! i!ea to %reak large, comple3 tasks i to a series o+ small, simple tasks. *etIs imagi e that we are tryi g to !escri%e a commo , e#ery!ay task, goi g to the market to %"y +oo!, to a perso +rom >ars. We might !escri%e the o#erall process as the +ollowi g series o+ steps, .. 2. (. 1. 2. K. /. 8. 9. ;et i car. 'ri#e to market. 6ark car. E ter market. 6"rchase +oo!. :et"r to car. 'ri#e home. 6ark car. E ter ho"se.

Aowe#er, a perso +rom >ars is likely to ee! more !etail. We co"l! +"rther %reak !ow the s"%task ?6ark car@ i to this series o+ steps, .. 2. (. 1. 2. K. 0i ! parki g space. 'ri#e car i to space. T"r o++ motor. Set parki g %rake. E3it car. *ock car.

The ?T"r o++ motor@ s"%task co"l! +"rther %e %roke !ow i to steps i cl"!i g ?T"r o++ ig itio ,@ ?:emo#e ig itio key@ a ! so o , " til e#ery step o+ the e tire process o+ goi g to the market has %ee +"lly !e+i e!. This process o+ i!e ti+yi g the top-le#el steps a ! !e#elopi g i creasi gly !etaile! #iews o+ those steps is calle! top-down design. This tech i="e allows "s to %reak large comple3 tasks i to ma y small, simple tasks. Top-!ow !esig is a commo metho! o+ !esig i g (/K

2/ 8 Top-'ow 'esig programs a ! o e that is well s"ite! to shell programmi g i partic"lar. 9 this chapter, we will "se top-!ow !esig to +"rther !e#elop o"r report ge erator script.

Shell ;unctions
7"r script c"rre tly per+orms the +ollowi g steps to ge erate the AT>* !oc"me t, .. 2. (. 1. 2. K. /. 8. 9. 7pe page. 7pe page hea!er. Set page title. Close page hea!er. 7pe page %o!y. 7"tp"t page hea!i g. 7"tp"t time stamp. Close page %o!y. Close page.

0or o"r e3t stage o+ !e#elopme t, we will a!! some a!!itio al tasks %etwee steps / a ! 8. These will i cl"!e,

System "ptime a ! loa!. This is the amo" t o+ time si ce the last sh"t!ow or re%oot a ! the a#erage "m%er o+ tasks c"rre tly r" i g o the processor o#er se#eral time i ter#als. 'isk space. The o#erall "se o+ space o the systemIs storage !e#ices. Aome space. The amo" t o+ storage space %ei g "se! %y each "ser.

9+ we ha! a comma ! +or each o+ these tasks, we co"l! a!! them to o"r script simply thro"gh comma ! s"%stit"tio ,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age TAT=&M_*1stem Anformation Re7ort -or $S *TL4+&_ C3RR&LTOTA+&M$Ndate 8_5x 5r 5I_P TA+&O*T4+:M_@enerated $C3RR&LTOTA+&> b1 $3*&R_ cat XX O& -O XST+=D XS&4'D

(//

2/ 8 Top-'ow 'esig
XTAT=&D$TAT=&X6TAT=&D X6S&4'D XH 'UD XS#D$TAT=&X6S#D X:D$TA+&O*T4+:X6:D 34report.uptime6 34report.disk.space6 34report.home.space6

X6H 'UD X6ST+=D O& -O

We co"l! create these a!!itio al comma !s two ways. We co"l! write three separate scripts a ! place them i a !irectory liste! i o"r 6$TA, or we co"l! em%e! the scripts withi o"r program as shell functions. $s we ha#e me tio e! %e+ore, shell +" ctio s are ?mi i-scripts@ that are locate! i si!e other scripts a ! ca act as a"to omo"s programs. Shell +" ctio s ha#e two sy tactic +orms, function name [ commands return P a ! name NP [ commands return \ where name is the ame o+ the +" ctio a ! commands are a series o+ comma !s co tai e! withi the +" ctio . Joth +orms are e="i#ale t a ! may %e "se! i tercha gea%ly. Jelow we see a script that !emo strates the "se o+ a shell +" ctio ,
# ! $ % " . ) / 0 #( ## #! ^G6bin6bash ^ *hell function demo function funct [ echo _*te7 !_ return \ ^ +ain 7rogram starts here echo _*te7 #_

(/8

Shell 0" ctio s

#$ #% funct echo _*te7 $_

$s the shell rea!s the script, it passes o#er li es . thro"gh .., as those li es co sist o+ comme ts a ! the +" ctio !e+i itio . E3ec"tio %egi s at li e .2, with a echo comma !. *i e .( calls the shell +" ctio funct a ! the shell e3ec"tes the +" ctio 4"st as it wo"l! a y other comma !. 6rogram co trol the mo#es to li e K, a ! the seco ! echo comma ! is e3ec"te!. *i e / is e3ec"te! e3t. 9ts return comma ! termi ates the +" ctio a ! ret"r s co trol to the program at the li e +ollowi g the +" ctio call Eli e .1F, a ! the +i al echo comma ! is e3ec"te!. &ote that i or!er +or +" ctio calls to %e recog iDe! as shell +" ctio s a ! ot i terprete! as the ames o+ e3ter al programs, shell +" ctio !e+i itio s m"st appear i the script %e+ore they are calle!. WeIll a!! mi imal shell +" ctio !e+i itio s to o"r script,
^G6bin6bash ^ :rogram to out7ut a s1stem information 7age TAT=&M_*1stem Anformation Re7ort -or $S *TL4+&_ C3RR&LTOTA+&M$Ndate 8_5x 5r 5I_P TA+&O*T4+:M_@enerated $C3RR&LTOTA+&> b1 $3*&R_ report.uptime 46 9 return > report.disk.space 46 9 return > report.home.space 46 9 return > cat XX O& -O XST+=D XS&4'D XTAT=&D$TAT=&X6TAT=&D X6S&4'D XH 'UD XS#D$TAT=&X6S#D X:D$TA+&O*T4+:X6:D $Nre7ortOu7timeP $Nre7ortOdiskOs7aceP

(/9

2/ 8 Top-'ow 'esig
$Nre7ortOhomeOs7aceP X6H 'UD X6ST+=D O& -O

Shell +" ctio ames +ollow the same r"les as #aria%les. $ +" ctio m"st co tai at least o e comma !. The return comma ! Ewhich is optio alF satis+ies the re="ireme t.

Local Varia0les
9 the scripts we ha#e writte so +ar, all the #aria%les Ei cl"!i g co sta tsF ha#e %ee global variables. ;lo%al #aria%les mai tai their e3iste ce thro"gho"t the program. This is +i e +or ma y thi gs, %"t it ca sometimes complicate the "se o+ shell +" ctio s. 9 si!e shell +" ctio s, it is o+te !esira%le to ha#e local variables. *ocal #aria%les are o ly accessi%le withi the shell +" ctio i which they are !e+i e! a ! cease to e3ist o ce the shell +" ctio termi ates. Aa#i g local #aria%les allows the programmer to "se #aria%les with ames that may alrea!y e3ist, either i the script glo%ally or i other shell +" ctio s, witho"t ha#i g to worry a%o"t pote tial ame co +licts. Aere is a e3ample script that !emo strates how local #aria%les are !e+i e! a ! "se!,
^G6bin6bash ^ local-vars: scri7t to demonstrate local variables fooM( ^ global variable foo functO# NP [ local foo ^ variable foo local to functO#

fooM# echo _functO#: foo M $foo_

functO! NP [ local foo ^ variable foo local to functO!

fooM! echo _functO!: foo M $foo_ foo M $foo_

echo _global:

(80

*ocal Raria%les
functO# echo _global: functO! echo _global:

foo M $foo_ foo M $foo_

$s we ca see, local #aria%les are !e+i e! %y prece!i g the #aria%le ame with the wor! local. This creates a #aria%le that is local to the shell +" ctio i which it is !e+i e!. 7 ce o"tsi!e the shell +" ctio , the #aria%le o lo ger e3ists. Whe we r" this script, we see the res"lts,
[me@linuxbox global: foo functO#: foo global: foo functO!: foo global: foo ~]$ local-vars M ( M # M ( M ! M (

We see that the assig me t o+ #al"es to the local #aria%le foo withi %oth shell +" ctio s has o e++ect o the #al"e o+ foo !e+i e! o"tsi!e the +" ctio s. This +eat"re allows shell +" ctio s to %e writte so that they remai i !epe !e t o+ each other a ! o+ the script i which they appear. This is #ery #al"a%le, as it helps pre#e t o e part o+ a program +rom i ter+eri g with a other. 9t also allows shell +" ctio s to %e writte so that they ca %e porta%le. That is, they may %e c"t a ! paste! +rom script to script, as ee!e!.

=eep Scripts 1unning

While !e#elopi g o"r program, it is "se+"l to keep the program i a r" a%le state. Jy !oi g this, a ! testi g +re="e tly, we ca !etect errors early i the !e#elopme t process. This will make !e%"ggi g pro%lems m"ch easier. 0or e3ample, i+ we r" the program, make a small cha ge, the r" the program agai a ! +i ! a pro%lem, itIs #ery likely that the most rece t cha ge is the so"rce o+ the pro%lem. Jy a!!i g the empty +" ctio s, calle! stubs i programmer-speak, we ca #eri+y the logical +low o+ o"r program at a early stage. Whe co str"cti g a st"%, itIs a goo! i!ea to i cl"!e somethi g that pro#i!es +ee!%ack to the programmer, which shows the logical +low is %ei g carrie! o"t. 9+ we look at the o"tp"t o+ o"r script ow,
[me@linuxbox ~]$ s!s.info.page XST+=D XS&4'D

(8.

2/ 8 Top-'ow 'esig
XTAT=&D*1stem Anformation Re7ort -or t9in!X6TAT=&D X6S&4'D XH 'UD XS#D*1stem Anformation Re7ort -or linuxboxX6S#D X:D@enerated ($6#06!((0 (%:(!:#( :+ &'T> b1 meX6:D

X6H 'UD X6ST+=D

we see that there are some %la k li es i o"r o"tp"t a+ter the time stamp, %"t we ca It %e s"re o+ the ca"se. 9+ we cha ge the +" ctio s to i cl"!e some +ee!%ack,
re7ortOu7time NP [ echo J7unction report.uptime executed.J return \ re7ortOdiskOs7ace NP [ echo J7unction report.disk.space executed.J return \ re7ortOhomeOs7ace NP [ echo J7unction report.home.space executed.J return \

a ! r" the script agai ,

[me@linuxbox ~]$ s!s.info.page XST+=D XS&4'D XTAT=&D*1stem Anformation Re7ort -or linuxboxX6TAT=&D X6S&4'D XH 'UD XS#D*1stem Anformation Re7ort -or linuxboxX6S#D X:D@enerated ($6!(6!((0 (":#):!. 4+ &'T> b1 meX6:D -unction re7ortOu7time executed< -unction re7ortOdiskOs7ace executed< -unction re7ortOhomeOs7ace executed< X6H 'UD X6ST+=D

(82

Meep Scripts :" we ow see that, i +act, o"r three +" ctio s are %ei g e3ec"te!.

i g

With o"r +" ctio +ramework i place a ! worki g, itIs time to +lesh o"t some o+ the +" ctio co!e. 0irst, the re7ortOu7time +" ctio ,
re7ortOu7time NP [ cat //- .GS7. /R2&F!stem Eptime//R2& /BHG&34uptime6//BHG& .GS7. return \

9tIs pretty straight+orwar!. We "se a here !oc"me t to o"tp"t a sectio hea!er a ! the o"tp"t o+ the u7time comma !, s"rro" !e! %y X:R&D tags to preser#e the +ormatti g o+ the comma !. The re7ortOdiskOs7ace +" ctio is similar,
re7ortOdiskOs7ace NP [ cat //- .GS7. /R2&1isk Fpace Etili-ation//R2& /BHG&34df -h6//BHG& .GS7. return \

This +" ctio "ses the df -h comma ! to !etermi e the amo" t o+ !isk space. *astly, weIll %"il! the re7ortOhomeOs7ace +" ctio ,
re7ortOhomeOs7ace NP [ cat //- .GS7. /R2&Rome Fpace Etili-ation//R2& /BHG&34du -sh /home/*6//BHG& .GS7. return \

We "se the du comma ! with the -sh optio s to per+orm this task. This, howe#er, is ot a complete sol"tio to the pro%lem. While it will work o some systems E)%" t", +or e3ampleF, it will ot work o others. The reaso is that ma y systems set the permissio s o+ home !irectories to pre#e t them +rom %ei g worl!-rea!a%le, which is a reaso a%le sec"rity meas"re. 7 these systems, the re7ortOhomeOs7ace +" ctio , as writte , (8(

2/ 8 Top-'ow 'esig will o ly work i+ o"r script is r" with s"per"ser pri#ileges. $ %etter sol"tio wo"l! %e to ha#e the script co"l! a!4"st its %eha#ior accor!i g to the pri#ileges o+ the "ser. We will take this "p i the e3t chapter.

Shell ;unctions In :our .bashrc ;ile

Shell +" ctio s make e3celle t replaceme ts +or aliases, a ! are act"ally the pre+erre! metho! o+ creati g small comma !s +or perso al "se. $liases are #ery limite! i the ki ! o+ comma !s a ! shell +eat"res they s"pport, whereas shell +" ctio s allow a ythi g that ca %e scripte!. 0or e3ample, i+ we like! the re7ortOdiskOs7ace shell +" ctio that we !e#elope! +or o"r script, we co"l! create a similar +" ctio ame! ds +or o"r .bashrc +ile,
ds NP [ echo c'isk *7ace 3tiliCation -or $S *TL4+&d df -h \

Summing -p
9 this chapter, we ha#e i tro!"ce! a commo metho! o+ program !esig calle! top!ow !esig , a ! we ha#e see how shell +" ctio s are "se! to %"il! the stepwise re+i eme t that it re="ires. We ha#e also see how local #aria%les ca %e "se! to make shell +" ctio s i !epe !e t +rom o e a other a ! +rom the program i which they are place!. This makes it possi%le +or shell +" ctio s to %e writte i a porta%le ma er a ! to %e reusable %y allowi g them to %e place! i m"ltiple programsH a great time sa#er.

;urther 1eading

The Wikipe!ia has ma y articles o so+tware !esig philosophy. Aere are a co"ple o+ goo! o es, http,--e .wikipe!ia.org-wiki-Top-!ow O!esig http,--e .wikipe!ia.org-wiki-S"%ro"ti es

(81

28 8 0low Co trol, Jra chi g With i+

2- (lo5 Control; 6ranching With if

9 the last chapter, we were prese te! with a pro%lem. Aow ca we make o"r report ge erator script a!apt to the pri#ileges o+ the "ser r" i g the scriptC The sol"tio to this pro%lem will re="ire "s to +i ! a way to ?cha ge !irectio s@ withi o"r script, %ase! o a the res"lts o+ a test. 9 programmi g terms, we ee! the program to branch. *etIs co si!er a simple e3ample o+ logic e3presse! i pseudocode, a sim"latio o+ a comp"ter la g"age i te !e! +or h"ma co s"mptio , GW2 9+ G W 2, the , Say ?G e="als 2.@ 7therwise, Say ?G is ot e="al to 2.@ This is a e3ample o+ a %ra ch. Jase! o the co !itio , ?'oes G W 2C@ !o o e thi g, ?Say G e="als 2,@ otherwise !o a other thi g, ?Say G is ot e="al to 2.@

if
)si g the shell, we ca co!e the logic a%o#e as +ollows,
xM" if [ $x M " ]T then echo _x eBuals "<_ else echo _x does not eBual "<_ fi

or we ca e ter it !irectly at the comma ! li e Eslightly shorte e!F,

(82

28 8 0low Co trol, Jra chi g With i+

[me@linuxbox ~]$ [me@linuxbox ~]$ not e,ual "J# fi eBuals " [me@linuxbox ~]$ [me@linuxbox ~]$ not e,ual "J# fi does not eBual " x$" if ( 3x $ " )# then echo Je,uals "J# else echo Jdoes x$* if ( 3x $ " )# then echo Je,uals "J# else echo Jdoes

9 this e3ample, we e3ec"te the comma ! twice. 7 ce, with the #al"e o+ x set to 2, which res"lts i the stri g ?e="als 2@ %ei g o"tp"t, a ! the seco ! time with the #al"e o+ x set to 0, which res"lts i the stri g ?!oes ot e="al 2@ %ei g o"tp"t. The if stateme t has the +ollowi g sy ta3, if commandsT then commands [elif commandsT then commands<<<] [else commands] fi where commands is a list o+ comma !s. This is a little co +"si g at +irst gla ce. J"t %e+ore we ca clear this "p, we ha#e to look at how the shell e#al"ates the s"ccess or +ail"re o+ a comma !.

$xit Status
Comma !s Ei cl"!i g the scripts a ! shell +" ctio s we writeF iss"e a #al"e to the system whe they termi ate, calle! a e+it status. This #al"e, which is a i teger i the ra ge o+ 0 to 222, i !icates the s"ccess or +ail"re o+ the comma !Is e3ec"tio . Jy co #e tio , a #al"e o+ Dero i !icates s"ccess a ! a y other #al"e i !icates +ail"re. The shell pro#i!es a parameter that we ca "se to e3ami e the e3it stat"s. Aere we see it i actio ,
[me@linuxbox ~]$ ls -d /usr/bin 6usr6bin [me@linuxbox ~]$ echo 3_ ( [me@linuxbox ~]$ ls -d /bin/usr ls: cannot access 6bin6usr: Lo such file or director1 [me@linuxbox ~]$ echo 3_ !

(8K

E3it Stat"s 9 this e3ample, we e3ec"te the ls comma ! twice. The +irst time, the comma ! e3ec"tes s"ccess+"lly. 9+ we !isplay the #al"e o+ the parameter $F, we see that it is Dero. We e3ec"te the ls comma ! a seco ! time, pro!"ci g a error a ! e3ami e the parameter $F agai . This time it co tai s a 2, i !icati g that the comma ! e co" tere! a error. Some comma !s "se !i++ere t e3it stat"s #al"es to pro#i!e !iag ostics +or errors, while ma y comma !s simply e3it with a #al"e o+ o e whe they +ail. >a pages o+te i cl"!e a sectio e title! ?E3it Stat"s,@ !escri%i g what co!es are "se!. Aowe#er, a Dero always i !icates s"ccess. The shell pro#i!es two e3tremely simple %"ilti comma !s that !o othi g e3cept termi ate with either a Dero or o e e3it stat"s. The true comma ! always e3ec"tes s"ccess+"lly a ! the false comma ! always e3ec"tes " s"ccess+"lly,
[me@linuxbox [me@linuxbox ( [me@linuxbox [me@linuxbox # ~]$ true ~]$ echo 3_ ~]$ false ~]$ echo 3_

We ca "se these comma !s to see how the if stateme t works. What the if stateme t really !oes is e#al"ate the s"ccess or +ail"re o+ comma !s,
[me@linuxbox ~]$ if true# then echo JPt%s true.J# fi AtKs true< [me@linuxbox ~]$ if false# then echo JPt%s true.J# fi [me@linuxbox ~]$

The comma ! echo _AtKs true<_ is e3ec"te! whe the comma ! +ollowi g if e3ec"tes s"ccess+"lly, a ! is ot e3ec"te! whe the comma ! +ollowi g if !oes ot e3ec"te s"ccess+"lly. 9+ a list o+ comma !s +ollows if, the last comma ! i the list is e#al"ate!,
[me@linuxbox ~]$ if falseT trueT then echo _AtKs true<_T fi AtKs true< [me@linuxbox ~]$ if trueT falseT then echo _AtKs true<_T fi [me@linuxbox ~]$

(8/

28 8 0low Co trol, Jra chi g With i+

test
Jy +ar, the comma ! "se! most +re="e tly with if is test. The test comma ! per+orms a #ariety o+ checks a ! compariso s. 9t has two e="i#ale t +orms, test expression a ! the more pop"lar, [ expression ] where e+pression is a e3pressio that is e#al"ate! as either tr"e or +alse. The test comma ! ret"r s a e3it stat"s o+ Dero whe the e3pressio is tr"e a ! a stat"s o+ o e whe the e3pressio is +alse.

2i e 6x5ressions
The +ollowi g e3pressio s are "se! to e#al"ate the stat"s o+ +iles, Table '4-1: test File 1+pressions $xpression file1 -ef file2 file1 -nt file2 file1 -ot file2 -b file -c file -d file -e file -f file -g file -@ file -k file -= file file -7 file -r file (88 Is True I"I file1 a ! file' ha#e the same i o!e "m%ers Ethe two +ile ames re+er to the same +ile %y har! li ki gF. file1 is ewer tha file'. file1 is ol!er tha file'. file e3ists a ! is a %lock special E!e#iceF +ile. file e3ists a ! is a character special E!e#iceF +ile. file e3ists a ! is a !irectory. file e3ists. file e3ists a ! is a reg"lar +ile. file e3ists a ! is set-gro"p-9'. file e3ists a ! is ow e! %y the e++ecti#e gro"p 9'. file e3ists a ! has its ?sticky %it@ set. file e3ists a ! is a sym%olic li k. file e3ists a ! is ow e! %y the e++ecti#e "ser 9'. file e3ists a ! is a ame! pipe. file e3ists a ! is rea!a%le Ehas rea!a%le permissio +or

test the e++ecti#e "serF. -s file -* file -t fd file e3ists a ! has a le gth greater tha Dero. file e3ists a ! is a etwork socket. fd is a +ile !escriptor !irecte! to-+rom the termi al. This ca %e "se! to !etermi e whether sta !ar! i p"t-o"tp"terror is %ei g re!irecte!. file e3ists a ! is set"i!. file e3ists a ! is writa%le Ehas write permissio +or the e++ecti#e "serF. file e3ists a ! is e3ec"ta%le Ehas e3ec"te-search permissio +or the e++ecti#e "serF.

-u file -9 file -x file

Aere we ha#e a script that !emo strates some o+ the +ile e3pressio s,
^G6bin6bash ^ test-file: &valuate the status of a file -A=&M~6<bashrc if [ -e _$-A=&_ ]T then if [ -f _$-A=&_ ]T then echo _$-A=& is a regular file<_ fi if [ -d _$-A=&_ ]T then echo _$-A=& is a director1<_ fi if [ -r _$-A=&_ ]T then echo _$-A=& is readable<_ fi if [ -9 _$-A=&_ ]T then echo _$-A=& is 9ritable<_ fi if [ -x _$-A=&_ ]T then echo _$-A=& is executable6searchable<_ fi else echo _$-A=& does not exist_ exit # fi exit

(89

28 8 0low Co trol, Jra chi g With i+ The script e#al"ates the +ile assig e! to the co sta t -A=& a ! !isplays its res"lts as the e#al"atio is per+orme!. There are two i teresti g thi gs to ote a%o"t this script. 0irst, otice how the parameter $-A=& is ="ote! withi the e3pressio s. This is ot re="ire!, %"t is a !e+e se agai st the parameter %ei g empty. 9+ the parameter e3pa sio o+ $-A=& were to res"lt i a empty #al"e, it wo"l! ca"se a error Ethe operators wo"l! %e i terprete! as o - "ll stri gs rather tha operatorsF. )si g the ="otes aro" ! the parameter i s"res that the operator is always +ollowe! %y a stri g, e#e i+ the stri g is empty. Seco !, otice the prese ce o+ the exit comma !s ear the e ! o+ the script. The exit comma ! accepts a si gle, optio al arg"me t, which %ecomes the scriptIs e3it stat"s. Whe o arg"me t is passe!, the e3it stat"s !e+a"lts to Dero. )si g exit i this way allows the script to i !icate +ail"re i+ $-A=& e3pa !s to the ame o+ a o e3iste t +ile. The exit comma ! appeari g o the last li e o+ the script is there as a +ormality. Whe a script ?r" s o++ the e !@ Ereaches e ! o+ +ileF, it termi ates with a e3it stat"s o+ Dero %y !e+a"lt, a yway. Similarly, shell +" ctio s ca ret"r a e3it stat"s %y i cl"!i g a i teger arg"me t to the return comma !. 9+ we were to co #ert the script a%o#e to a shell +" ctio to i cl"!e it i a larger program, we co"l! replace the exit comma !s with return stateme ts a ! get the !esire! %eha#ior,
testOfile NP [ ^ test-file: &valuate the status of a file -A=&M~6<bashrc if [ -e _$-A=&_ ]T then if [ -f _$-A=&_ ]T then echo _$-A=& is a regular file<_ fi if [ -d _$-A=&_ ]T then echo _$-A=& is a director1<_ fi if [ -r _$-A=&_ ]T then echo _$-A=& is readable<_ fi if [ -9 _$-A=&_ ]T then echo _$-A=& is 9ritable<_ fi if [ -x _$-A=&_ ]T then echo _$-A=& is executable6searchable<_ fi else echo _$-A=& does not exist_ return #

(90

test
fi \

String 6x5ressions
The +ollowi g e3pressio s are "se! to e#al"ate stri gs, Table '4-': test String 1+pressions $xpression string -n string -C string string1 M string2 string1 !! string2 string1 GM string2 string1 D string2 string1 X string2 Is True I"... string is ot "ll. The le gth o+ string is greater tha Dero. The le gth o+ string is Dero. string1 a ! string' are e="al. Si gle or !o"%le e="al sig s may %e "se!, %"t the "se o+ !o"%le e="al sig s is greatly pre+erre!. string1 a ! string' are ot e="al. string1 sorts a+ter string'. string1 sorts %e+ore string'.

'arning, the D a ! X e3pressio operators m"st %e ="ote! Eor escape! with a %ackslashF whe "se! with test. 9+ they are ot, they will %e i terprete! %y the shell as re!irectio operators, with pote tially !estr"cti#e res"lts. $lso ote that while the bash !oc"me tatio states that the sorti g or!er co +orms to the collatio or!er o+ the c"rre t locale, it !oes ot. $SC99 E67S9GF or!er is "se! i #ersio s o+ bash "p to a ! i cl"!i g 1.0. Aere is a script that i corporates stri g e3pressio s,
^G6bin6bash ^ test-string: evaluate the value of a string 4L*,&RMma1be if [ -C _$4L*,&R_ ]T then

(9.

28 8 0low Co trol, Jra chi g With i+

echo _There is no ans9er<_ DY! exit #

fi

if [ _$4L*,&R_ M _1es_ ]T then echo _The ans9er is U&*<_ elif [ _$4L*,&R_ M _no_ ]T then echo _The ans9er is L <_ elif [ _$4L*,&R_ M _ma1be_ ]T then echo _The ans9er is +4UH&<_ else echo _The ans9er is 3L2L ,L<_ fi

9 this script, we e#al"ate the co sta t 4L*,&R. We +irst !etermi e i+ the stri g is empty. 9+ it is, we termi ate the script a ! set the e3it stat"s to o e. &otice the re!irectio that is applie! to the echo comma !. This re!irects the error message ?There is o a swer.@ to sta !ar! error, which is the ?proper@ thi g to !o with error messages. 9+ the stri g is ot empty, we e#al"ate the #al"e o+ the stri g to see i+ it is e="al to either ?yes,@ ? o,@ or ?may%e.@ We !o this %y "si g elif, which is short +or ?else i+.@ Jy "si g elif, we are a%le to co str"ct a more comple3 logical test.

Integer 6x5ressions
The +ollowi g e3pressio s are "se! with i tegers, Table '4-3: test *nteger 1+pressions $xpression integer1 -eB integer2 integer1 -ne integer2 integer1 -le integer2 integer1 -lt integer2 integer1 -ge integer2 integer1 -gt integer2 Is True I"... integer1 is e="al to integer'. integer1 is ot e="al to integer'. integer1 is less tha or e="al to integer'. integer1 is less tha integer'. integer1 is greater tha or e="al to integer'. integer1 is greater tha integer'.

Aere is a script that !emo strates them,

^G6bin6bash

(92

test

^ test-integer: evaluate the value of an integer< ALTM-" if [ -C _$ALT_ ]T then echo _ALT is em7t1<_ DY! exit # fi if [ $ALT -eB ( ]T then echo _ALT is Cero<_ else if [ $ALT -lt ( ]T then echo _ALT is negative<_ else echo _ALT is 7ositive<_ fi if [ $NNALT 5 !PP -eB ( ]T then echo _ALT is even<_ else echo _ALT is odd<_ fi fi

The i teresti g part o+ the script is how it !etermi es whether a i teger is e#e or o!!. Jy per+ormi g a mo!"lo 2 operatio o the "m%er, which !i#i!es the "m%er %y two a ! ret"r s the remai !er, it ca tell i+ the "m%er is o!! or e#e .

# >ore >odern Version ?" test

:ece t #ersio s o+ bash i cl"!e a compo" ! comma ! that acts as a replaceme t +or test. 9t "ses the +ollowi g sy ta3, [[ expression ]] where, like test, e+pression is a e3pressio that e#al"ates to either a tr"e or +alse res"lt. The [[ ]] comma ! is #ery similar to test Eit s"pports all o+ its e3pressio sF, %"t a!!s a importa t ew stri g e3pressio , string1 M~ regex which ret"r s tr"e i+ string1 is matche! %y the e3te !e! reg"lar e3pressio rege+. This ope s "p a lot o+ possi%ilities +or per+ormi g s"ch tasks as !ata #ali!atio . 9 o"r earlier e3ample o+ the i teger e3pressio s, the script wo"l! +ail i+ the co sta t ALT co tai e! a ythi g e3cept a i teger. The script ee!s a way to #eri+y that the co sta t co tai s a i teger. )si g [[ ]] with the M~ stri g e3pressio operator, we co"l! impro#e the (9( e ha ce!

28 8 0low Co trol, Jra chi g With i+ script this way,

^G6bin6bash ^ test-integer!: evaluate the value of an integer< ALTM-" if (( J3P?+J $~ [-_(*-D)53 ))# then if [ $ALT -eB ( ]T then echo _ALT is Cero<_ else if [ $ALT -lt ( ]T then echo _ALT is negative<_ else echo _ALT is 7ositive<_ fi if [ $NNALT 5 !PP -eB ( ]T then echo _ALT is even<_ else echo _ALT is odd<_ fi fi else echo JP?+ is not an integer.J &'2 exit 1 fi

Jy applyi g the reg"lar e3pressio , we are a%le to limit the #al"e o+ ALT to o ly stri gs that %egi with a optio al mi "s sig , +ollowe! %y o e or more "merals. This e3pressio also elimi ates the possi%ility o+ empty #al"es. $ other a!!e! +eat"re o+ [[ ]] is that the MM operator s"pports patter matchi g the same way path ame e3pa sio !oes. 0or e3ample,
[me@linuxbox ~]$ 7PTG$foo.bar [me@linuxbox ~]$ if (( 37PTG $$ foo.* ))# then D echo J37PTG matches pattern %foo.*%J D fi foo<bar matches 7attern Kfoo<EK

This makes [[ ]] "se+"l +or e#al"ati g +ile a ! path ames.

(91

EE FF - 'esig e! 0or 9 tegers

44 66 E Designed ;or Integers

9 a!!itio to the [[ ]] compo" ! comma !, bash also pro#i!es the NN PP compo" ! comma !, which is "se+"l +or operati g o i tegers. 9t s"pports a +"ll set o+ arithmetic e#al"atio s, a s"%4ect we will co#er +"lly i Chapter (2. NN PP is "se! to per+orm arithmetic truth tests. $ arithmetic tr"th test res"lts i tr"e i+ the res"lt o+ the arithmetic e#al"atio is o -Dero.
[me@linuxbox ~]$ if 44166# then echo JPt is true.J# fi At is true< [me@linuxbox ~]$ if 44*66# then echo JPt is true.J# fi [me@linuxbox ~]$

)si g NN PP, we ca slightly simpli+y the test-integer! script like this,

^G6bin6bash ^ test-integer!a: evaluate the value of an integer< ALTM-" if [[ _$ALT_ M~ `-F[(-0]8$ ]]T then if 44P?+ $$ *66T then echo _ALT is Cero<_ else if 44P?+ / *66T then echo _ALT is negative<_ else echo _ALT is 7ositive<_ fi if 44 44P?+ 8 266 $$ *66T then echo _ALT is even<_ else echo _ALT is odd<_ fi fi else echo _ALT is not an integer<_ DY! exit # fi

&otice that we "se less tha a ! greater tha sig s a ! that MM is "se! to test +or e="i#ale ce. This is a more at"ral looki g sy ta3 +or worki g with i tegers. &otice too, that %eca"se the compo" ! comma ! NN PP is part o+ the shell sy ta3 rather tha (92

28 8 0low Co trol, Jra chi g With i+ a or!i ary comma !, a ! it !eals o ly with i tegers, it is a%le to recog iDe #aria%les %y ame a ! !oes ot re="ire e3pa sio to %e per+orme!. WeIll !isc"ss NN PP a ! the relate! arithmetic e3pa sio +"rther i Chapter (2.

Com0ining $xpressions
9tIs also possi%le to com%i e e3pressio s to create more comple3 e#al"atio s. E3pressio s are com%i e! %y "si g logical operators. We saw these i Chapter .8, whe we lear e! a%o"t the find comma !. There are three logical operatio s +or test a ! [[ ]]. They are $&', 7: a ! &7T. test a ! [[ ]]"se !i++ere t operators to represe t these operatio s , Table '4-$: (ogical &perators ?peration $&' 7: &7T test -a -o G (( )) and 44 66 YY QQ G

AereIs a e3ample o+ a $&' operatio . The +ollowi g script !etermi es i+ a i teger is withi a ra ge o+ #al"es,
^G6bin6bash ^ test-integer$: determine if an integer is 9ithin a ^ s7ecified range of values< +ALO;4=M# +4VO;4=M#(( ALTM"( if [[ _$ALT_ M~ `-F[(-0]8$ ]]T then if [[ ALT -ge +ALO;4= YY ALT -le +4VO;4= ]]T then echo _$ALT is 9ithin $+ALO;4= to $+4VO;4=<_ else echo _$ALT is out of range<_ fi else echo _ALT is not an integer<_ DY! exit # fi

(9K

Com%i i g E3pressio s 9 this script, we !etermi e i+ the #al"e o+ i teger ALT lies %etwee the #al"es o+ +ALO;4= a ! +4VO;4=. This is per+orme! %y a si gle "se o+ [[ ]], which i cl"!es two e3pressio s separate! %y the YY operator. We co"l! ha#e also co!e! this "si g test,
if ( $ALT -ge $+ALO;4= -a $ALT -le $+4VO;4= )T then echo _$ALT is 9ithin $+ALO;4= to $+4VO;4=<_ else echo _$ALT is out of range<_ fi

The G egatio operator re#erses the o"tcome o+ a e3pressio . 9t ret"r s tr"e i+ a e3pressio is +alse, a ! it ret"r s +alse i+ a e3pressio is tr"e. 9 the +ollowi g script, we mo!i+y the logic o+ o"r e#al"atio to +i ! #al"es o+ ALT that are o"tsi!e the speci+ie! ra ge,
^G6bin6bash ^ test-integer%: determine if an integer is outside a ^ s7ecified range of values< +ALO;4=M# +4VO;4=M#(( ALTM"( if [[ _$ALT_ M~ `-F[(-0]8$ ]]T then if [[ G NALT -ge +ALO;4= YY ALT -le +4VO;4=P ]]T then echo _$ALT is outside $+ALO;4= to $+4VO;4=<_ else echo _$ALT is in range<_ fi else echo _ALT is not an integer<_ DY! exit # fi

We also i cl"!e pare theses aro" ! the e3pressio , +or gro"pi g. 9+ these were ot i cl"!e!, the egatio wo"l! o ly apply to the +irst e3pressio a ! ot the com%i atio o+ the two. Co!i g this with test wo"l! %e !o e this way,
if [ G ]N $ALT -ge $+ALO;4= -a $ALT -le $+4VO;4= ]P ]T then

(9/

28 8 0low Co trol, Jra chi g With i+

echo _$ALT is outside $+ALO;4= to $+4VO;4=<_ else fi echo _$ALT is in range<_

Si ce all e3pressio s a ! operators "se! %y test are treate! as comma ! arg"me ts %y the shell E" like [[ ]] a ! NN PP F, characters which ha#e special mea i g to bash, s"ch as X, D, N, a ! P, m"st %e ="ote! or escape!. Seei g that test a ! [[ ]] !o ro"ghly the same thi g, which is pre+era%leC test is tra!itio al Ea ! part o+ 67S9GF, whereas [[ ]] is speci+ic to bash. 9tIs importa t to k ow how to "se test, si ce it is #ery wi!ely "se!, %"t [[ ]] is clearly more "se+"l a ! is easier to co!e.

Porta0ilit, Is The 3o0go0lin ?" Little >inds

9+ yo" talk to ?real@ ) i3 people, yo" ="ickly !isco#er that ma y o+ them !o It like *i "3 #ery m"ch. They regar! it as imp"re a ! " clea . 7 e te et o+ ) i3 +ollowers is that e#erythi g sho"l! %e ?porta%le.@ This mea s that a y script yo" write sho"l! %e a%le to r" , " cha ge!, o a y ) i3-like system. ) i3 people ha#e goo! reaso to %elie#e this. Aa#i g see what proprietary e3te sio s to comma !s a ! shells !i! to the ) i3 worl! %e+ore 67S9G, they are at"rally wary o+ the e++ect o+ *i "3 o their %elo#e! 7S. J"t porta%ility has a serio"s !ow si!e. 9t pre#e ts progress. 9t re="ires that thi gs are always !o e "si g ?lowest commo !e omi ator@ tech i="es. 9 the case o+ shell programmi g, it mea s maki g e#erythi g compati%le with sh, the origi al Jo"r e shell. This !ow si!e is the e3c"se that proprietary #e !ors "se to 4"sti+y their proprietary e3te sio s, o ly they call them ?i o#atio s.@ J"t they are really 4"st lock-i !e#ices +or their c"stomers. The ;&) tools, s"ch as bash, ha#e o s"ch restrictio s. They e co"rage porta%ility %y s"pporti g sta !ar!s a ! %y %ei g " i#ersally a#aila%le. 5o" ca i stall bash a ! the other ;&) tools o almost a y ki ! o+ system, e#e Wi !ows, witho"t cost. So +eel +ree to "se all the +eat"res o+ bash. 9tIs really porta%le.

(98

Co trol 7perators, $ other Way To Jra ch

Control ?peratorsI #nother +a, To /ranch

bash pro#i!es two co trol operators that ca per+orm %ra chi g. The YY E$&'F a ! QQ E7:F operators work like the logical operators i the [[ ]] compo" ! comma !. This is the sy ta3, command1 YY command2 a ! command1 QQ command2 9t is importa t to " !ersta ! the %eha#ior o+ these. With the YY operator, command1 is e3ec"te! a ! command' is e3ec"te! i+, and only if, command1 is s"ccess+"l. With the QQ operator, command1 is e3ec"te! a ! command' is e3ec"te! i+, and only if, command1 is " s"ccess+"l. 9 practical terms, it mea s that we ca !o somethi g like this,
[me@linuxbox ~]$ mkdir temp '' cd temp

This will create a !irectory ame! tem7, a ! i+ it s"ccee!s, the c"rre t worki g !irectory will %e cha ge! to tem7. The seco ! comma ! is attempte! o ly i+ the mkdir comma ! is s"ccess+"l. *ikewise, a comma ! like this,
[me@linuxbox ~]$ ( -d temp ) 00 mkdir temp

will test +or the e3iste ce o+ the !irectory tem7, a ! o ly i+ the test +ails, will the !irectory %e create!. This type o+ co str"ct is #ery ha !y +or ha !li g errors i scripts, a s"%4ect we will !isc"ss more i later chapters. 0or e3ample, we co"l! !o this i a script,
[ -d tem7 ] QQ exit #

9+ the script re="ires the !irectory tem7, a ! it !oes ot e3ist, the the script will termi ate with a e3it stat"s o+ o e.

Summing -p
We starte! this chapter with a ="estio . Aow co"l! we make o"r s1sOinfoO7age script !etect i+ the "ser ha! permissio to rea! all the home !irectoriesC With o"r k owle!ge o+ if, we ca sol#e the pro%lem %y a!!i g this co!e to the (99

28 8 0low Co trol, Jra chi g With i+ re7ortOhomeOs7ace +" ctio ,

re7ortOhomeOs7ace NP [ if (( 34id -u6 -e, * ))# then cat //- .GS7. /R2&Rome Fpace Etili-ation 4:ll Esers6//R2& /BHG&34du -sh /home/*6//BHG& .GS7. else cat //- .GS7. /R2&Rome Fpace Etili-ation 43EFGH6//R2& /BHG&34du -sh 3RSOG6//BHG& .GS7. fi return \

We e#al"ate the o"tp"t o+ the id comma !. With the -u optio , id o"tp"ts the "meric "ser 9' "m%er o+ the e++ecti#e "ser. The s"per"ser is always Dero a ! e#ery other "ser is a "m%er greater tha Dero. M owi g this, we ca co str"ct two !i++ere t here !oc"me ts, o e taki g a!#a tage o+ s"per"ser pri#ileges, a ! the other, restricte! to the "serIs ow home !irectory. We are goi g to take a %reak +rom the s1sOinfoO7age program, %"t !o It worry. 9t will %e %ack. 9 the mea time, weIll co#er some topics that weIll ee! whe we res"me o"r work.

;urther 1eading
There are se#eral sectio s o+ the bash ma page that pro#i!e +"rther !etail o the topics co#ere! i this chapter,

*ists Eco#ers the co trol operators QQ a ! YYF Compo" ! Comma !s Eco#ers [[ ]], NN PP a ! ifF C7&'9T97&$* EG6:ESS97&S SAE** J)9*T9& C7>>$&'S Eco#ers testF

0"rther, the Wikipe!ia has a goo! article o the co cept o+ pse"!oco!e, http,--e .wikipe!ia.org-wiki-6se"!oco!e

100

29 8 :ea!i g Mey%oar! 9 p"t

20 /eading 1e$board In#ut

The scripts we ha#e writte so +ar lack a +eat"re commo i most comp"ter programsB interactivity. That is, the a%ility o+ the program to i teract with the "ser. While ma y programs !o It ee! to %e i teracti#e, some programs %e e+it +rom %ei g a%le to accept i p"t !irectly +rom the "ser. Take, +or e3ample, this script +rom the pre#io"s chapter,
^G6bin6bash ^ test-integer!: evaluate the value of an integer< ALTM-" if [[ _$ALT_ M~ `-F[(-0]8$ ]]T then if [ $ALT -eB ( ]T then echo _ALT is Cero<_ else if [ $ALT -lt ( ]T then echo _ALT is negative<_ else echo _ALT is 7ositive<_ fi if [ $NNALT 5 !PP -eB ( ]T then echo _ALT is even<_ else echo _ALT is odd<_ fi fi else echo _ALT is not an integer<_ DY! exit # fi

Each time we wa t to cha ge the #al"e o+ ALT, we ha#e to e!it the script. 9t wo"l! %e m"ch more "se+"l i+ the script co"l! ask the "ser +or a #al"e. 9 this chapter, we will %egi to look at how we ca a!! i teracti#ity to o"r programs.

10.

29 8 :ea!i g Mey%oar! 9 p"t

read 1ead Values ;rom Standard Input

The read %"ilti comma ! is "se! to rea! a si gle li e o+ sta !ar! i p"t. This comma ! ca %e "se! to rea! key%oar! i p"t or, whe re!irectio is employe!, a li e o+ !ata +rom a +ile. The comma ! has the +ollowi g sy ta3, read [-options] [variable<<<] where options is o e or more o+ the a#aila%le optio s liste! %elow a ! variable is the ame o+ o e or more #aria%les "se! to hol! the i p"t #al"e. 9+ o #aria%le ame is s"pplie!, the shell #aria%le R&:=U co tai s the li e o+ !ata. Jasically, read assig s +iel!s +rom sta !ar! i p"t to the speci+ie! #aria%les. 9+ we mo!i+y o"r i teger e#al"atio script to "se read, it might look like this,
^G6bin6bash ^ read-integer: evaluate the value of an integer< echo -n JBlease enter an integer -& J read int if [[ _$int_ M~ `-F[(-0]8$ ]]T then if [ $int -eB ( ]T then echo _$int is Cero<_ else if [ $int -lt ( ]T then echo _$int is negative<_ else echo _$int is 7ositive<_ fi if [ $NNint 5 !PP -eB ( ]T then echo _$int is even<_ else echo _$int is odd<_ fi fi else echo _An7ut value is not an integer<_ DY! exit # fi

We "se echo with the -n optio Ewhich s"ppresses the traili g ewli e o o"tp"tF to !isplay a prompt, the "se read to i p"t a #al"e +or the #aria%le int. :" i g this script res"lts i this,

102

rea! 8 :ea! Ral"es 0rom Sta !ar! 9 p"t

[me@linuxbox ~]$ read-integer :lease enter an integer -D " " is 7ositive< " is odd<

read ca assig i p"t to m"ltiple #aria%les, as show i this script,

^G6bin6bash ^ read-multi7le: read multi7le values from ke1board echo -n _&nter one or more values D _ read var# var! var$ var% var" echo echo echo echo echo _var# _var! _var$ _var% _var" M M M M M K$var#K_ K$var!K_ K$var$K_ K$var%K_ K$var"K_

9 this script, we assig a ! !isplay "p to +i#e #al"es. &otice how read %eha#es whe gi#e !i++ere t "m%ers o+ #al"es,
[me@linuxbox ~]$ read-multiple &nter one or more values D a b c d e var# M KaK var! M KbK var$ M KcK var% M KdK var" M KeK [me@linuxbox ~]$ read-multiple &nter one or more values D a var# M KaK var! M KK var$ M KK var% M KK var" M KK [me@linuxbox ~]$ read-multiple &nter one or more values D a b c d e f g var# M KaK var! M KbK var$ M KcK var% M KdK var" M Ke f gK

10(

29 8 :ea!i g Mey%oar! 9 p"t 9+ read recei#es +ewer tha the e3pecte! "m%er, the e3tra #aria%les are empty, while a e3cessi#e amo" t o+ i p"t res"lts i the +i al #aria%le co tai i g all o+ the e3tra i p"t. 9+ o #aria%les are liste! a+ter the read comma !, a shell #aria%le, R&:=U, will %e assig e! all the i p"t,
^G6bin6bash ^ read-single: read multi7le values into default variable echo -n _&nter one or more values D _ read echo _R&:=U M K$R&:=UK_

:"

i g this script res"lts i this,

[me@linuxbox ~]$ read-single &nter one or more values D a b c d R&:=U M Ka b c dK

<5tions
read s"pports the +ollowi g optio s, Table '5-1: read &ptions ?ption -a arra -d delimiter -e -n num -7 prompt -r Description $ssig the i p"t to array, starti g with i !e3 Dero. We will co#er arrays i Chapter (K. The +irst character i the stri g delimiter is "se! to i !icate e ! o+ i p"t, rather tha a ewli e character. )se :ea!li e to ha !le i p"t. This permits i p"t e!iti g i the same ma er as the comma ! li e. :ea! num characters o+ i p"t, rather tha a e tire li e. 'isplay a prompt +or i p"t "si g the stri g prompt. :aw mo!e. 'o ot i terpret %ackslash characters as escapes.

101

rea! 8 :ea! Ral"es 0rom Sta !ar! 9 p"t -s Sile t mo!e. 'o ot echo characters to the !isplay as they are type!. This is "se+"l whe i p"tti g passwor!s a ! other co +i!e tial i +ormatio . Timeo"t. Termi ate i p"t a+ter seconds. read ret"r s a o -Dero e3it stat"s i+ a i p"t times o"t. )se i p"t +rom +ile !escriptor fd, rather tha sta !ar! i p"t.

-t seconds -u fd

)si g the #ario"s optio s, we ca !o i teresti g thi gs with read. 0or e3ample, with the -7 optio , we ca pro#i!e a prompt stri g,
^G6bin6bash ^ read-single: read multi7le values into default variable read -7 _&nter one or more values D _ echo _R&:=U M K$R&:=UK_

With the -t a ! -s optio s we ca write a script that rea!s ?secret@ i p"t a ! times o"t i+ the i p"t is ot complete! i a speci+ie! time,
^G6bin6bash ^ read-secret: in7ut a secret 7ass 7hrase if read -t echo else echo exit fi #( -s7 _&nter secret 7ass 7hrase D _ secretO7assT then -e _]n*ecret 7ass 7hrase M K$secretO7assK_ -e _]nAn7ut timed out_ DY! #

The script prompts the "ser +or a secret pass phrase a ! waits te seco !s +or i p"t. 9+ the e try is ot complete! withi the speci+ie! time, the script e3its with a error. Si ce the -s optio is i cl"!e!, the characters o+ the pass phrase are ot echoe! to the !isplay as they are type!.

102

29 8 :ea!i g Mey%oar! 9 p"t

I2S
&ormally, the shell per+orms wor! splitti g o the i p"t pro#i!e! to read. $s we ha#e see , this mea s that m"ltiple wor!s separate! %y o e or more spaces %ecome separate items o the i p"t li e, a ! are assig e! to separate #aria%les %y read. This %eha#ior is co +ig"re! %y a shell #aria%le ame! A-* E+or 9 ter al 0iel! SeparatorF. The !e+a"lt #al"e o+ A-* co tai s a space, a ta%, a ! a ewli e character, each o+ which will separate items +rom o e a other. We ca a!4"st the #al"e o+ A-* to co trol the separatio o+ +iel!s i p"t to read. 0or e3ample, the 6etc67ass9d +ile co tai s li es o+ !ata that "se the colo character as a +iel! separator. Jy cha gi g the #al"e o+ A-* to a si gle colo , we ca "se read to i p"t the co te ts o+ 6etc67ass9d a ! s"ccess+"lly separate +iel!s i to !i++ere t #aria%les. Aere we ha#e a script that !oes 4"st that,
^G6bin6bash ^ read-ifs: read fields from a file -A=&M6etc67ass9d read -7 _&nter a user name D _ userOname fileOinfoM$Ngre7 _`$userOname:_ $-A=&P if [ -n _$fileOinfo_ ]T then A-*M_:_ read user 79 uid gid name home shell XXX _$fileOinfo_ echo _3ser M K$userK_ echo _3A' M K$uidK_ echo _@A' M K$gidK_ echo _-ull Lame M K$nameK_ echo _Some 'ir< M K$homeK_ echo _*hell M K$shellK_ else echo _Lo such user K$userOnameK_ DY! exit # fi

This script prompts the "ser to e ter the "ser ame o+ a acco" t o the system, the !isplays the !i++ere t +iel!s +o" ! i the "serIs recor! i the 6etc67ass9d +ile. The script co tai s two i teresti g li es. The +irst is,
fileOinfoM$Ngre7 _`$userOname:_ $-A=&P

This li e assig s the res"lts o+ a gre7 comma ! to the #aria%le fileOinfo. The reg"lar e3pressio "se! %y gre7 ass"res that the "ser ame will o ly match a si gle li e 10K

rea! 8 :ea! Ral"es 0rom Sta !ar! 9 p"t i the 6etc67ass9d +ile. The seco ! i teresti g li e is this o e,
A-*M_:_ read user 79 uid gid name home shell XXX _$fileOinfo_

The li e co sists o+ three parts, a #aria%le assig me t, a read comma ! with a list o+ #aria%le ames as arg"me ts, a ! a stra ge ew re!irectio operator. WeIll look at the #aria%le assig me t +irst. The shell allows o e or more #aria%le assig me ts to take place imme!iately %e+ore a comma !. These assig me ts alter the e #iro me t +or the comma ! that +ollows. The e++ect o+ the assig me t is temporaryH o ly cha gi g the e #iro me t +or the !"ratio o+ the comma !. 9 o"r case, the #al"e o+ A-* is cha ge! to a colo character. $lter ately, we co"l! ha#e co!e! it this way,
='OA-*M_$A-*_ A-*M_:_ read user 79 uid gid name home shell XXX _$fileOinfo_ A-*M_$ ='OA-*_

where we store the #al"e o+ A-*, assig a ew #al"e, per+orm the read comma !, the restore A-* to its origi al #al"e. Clearly, placi g the #aria%le assig me t i +ro t o+ the comma ! is a more co cise way o+ !oi g the same thi g. The XXX operator i !icates a here string. $ here stri g is like a here !oc"me t, o ly shorter, co sisti g o+ a si gle stri g. 9 o"r e3ample, the li e o+ !ata +rom the 6etc67ass9d +ile is +e! to the sta !ar! i p"t o+ the read comma !. We might wo !er why this rather o%li="e metho! was chose rather tha ,
echo _$fileOinfo_ Q A-*M_:_ read user 79 uid gid name home shell

Well, thereIs a reaso ...

:ou CanFt Pipe read

While the read comma ! ormally takes i p"t +rom sta !ar! i p"t, yo" ca ot !o this,
echo _foo_ Q read

We wo"l! e3pect this to work, %"t it !oes ot. The comma ! will appear to s"ccee! %"t the R&:=U #aria%le will always %e empty. Why is thisC The e3pla atio has to !o with the way the shell ha !les pipeli es. 9 bash Ea ! other shells s"ch as shF, pipeli es create subshells. These are copies o+ the shell

10/

29 8 :ea!i g Mey%oar! 9 p"t

a ! its e #iro me t which are "se! to e3ec"te the comma ! i the pipeli e. 9 o"r e3ample a%o#e, read is e3ec"te! i a s"%shell. S"%shells i ) i3-like systems create copies o+ the e #iro me t +or the processes to "se while they e3ec"te. Whe the processes +i ishes the copy o+ the e #iro me t is !estroye!. This mea s that a subshell can never alter the environment of its parent process. read assig s #aria%les, which the %ecome part o+ the e #iro me t. 9 the e3ample a%o#e, read assig s the #al"e ?+oo@ to the #aria%le R&:=U i its s"%shellIs e #iro me t, %"t whe the comma ! e3its, the s"%shell a ! its e #iro me t are !estroye!, a ! the e++ect o+ the assig me t is lost. )si g here stri gs is o e way to work aro" ! this %eha#ior. $ other metho! is !isc"sse! i Chapter (/.

Validating Input
With o"r ew a%ility to ha#e key%oar! i p"t comes a a!!itio al programmi g challe ge, #ali!ati g i p"t. Rery o+te the !i++ere ce %etwee a well-writte program a ! a poorly writte o e is i the programIs a%ility to !eal with the " e3pecte!. 0re="e tly, the " e3pecte! appears i the +orm o+ %a! i p"t. WeI#e !o e a little o+ this with o"r e#al"atio programs i the pre#io"s chapter, where we checke! the #al"e o+ i tegers a ! scree e! o"t empty #al"es a ! o - "meric characters. 9t is importa t to per+orm these ki !s o+ programmi g checks e#ery time a program recei#es i p"t, to g"ar! agai st i #ali! !ata. This is especially importa t +or programs that are share! %y m"ltiple "sers. 7mitti g these sa+eg"ar!s i the i terests o+ eco omy might %e e3c"se! i+ a program is to %e "se! o ce a ! o ly %y the a"thor to per+orm some special task. E#e the , i+ the program per+orms !a gero"s tasks s"ch as !eleti g +iles, it wo"l! %e wise to i cl"!e !ata #ali!atio , 4"st i case. Aere we ha#e a e3ample program that #ali!ates #ario"s ki !s o+ i p"t,
^G6bin6bash ^ read-validate: validate in7ut invalidOin7ut NP [ echo _Anvalid in7ut K$R&:=UK_ DY! exit # \

108

Rali!ati g 9 p"t
read -7 _&nter a single item D _ ^ in7ut is em7t1 NinvalidP [[ -C $R&:=U ]] YY invalidOin7ut ^ in7ut is multi7le items NinvalidP NN $Necho $R&:=U Q 9c -9P D # PP YY invalidOin7ut ^ is in7ut a valid filenameF if [[ $R&:=U M~ `[-[:alnum:]]<O]8$ ]]T then echo _K$R&:=UK is a valid filename<_ if [[ -e $R&:=U ]]T then echo _4nd file K$R&:=UK exists<_ else echo _So9ever> file K$R&:=UK does not exist<_ fi ^ is in7ut a floating 7oint numberF if [[ $R&:=U M~ `-F[[:digit:]]E]<[[:digit:]]8$ ]]T then echo _K$R&:=UK is a floating 7oint number<_ else echo _K$R&:=UK is not a floating 7oint number<_ fi ^ is in7ut an integerF if [[ $R&:=U M~ `-F[[:digit:]]8$ ]]T then echo _K$R&:=UK is an integer<_ else echo _K$R&:=UK is not an integer<_ fi echo _The string K$R&:=UK is not a valid filename<_

else fi

This script prompts the "ser to e ter a item. The item is s"%se="e tly a alyDe! to !etermi e its co te ts. $s we ca see, the script makes "se o+ ma y o+ the co cepts that we ha#e co#ere! th"s +ar, i cl"!i g shell +" ctio s, [[ ]], NN PP, the co trol operator YY, a ! if, as well as a healthy !ose o+ reg"lar e3pressio s.

>enus
$ commo type o+ i teracti#ity is calle! menu-driven. 9 me "-!ri#e programs, the "ser is prese te! with a list o+ choices a ! is aske! to choose o e. 0or e3ample, we co"l! imagi e a program that prese te! the +ollowi g,
:lease *elect:

109

29 8 :ea!i g Mey%oar! 9 p"t

#< !< $< (<

'is7la1 *1stem Anformation 'is7la1 'isk *7ace 'is7la1 Some *7ace 3tiliCation Zuit

&nter selection [(-$] D

)si g what we lear e! +rom writi g o"r s1sOinfoO7age program, we ca co str"ct a me "-!ri#e program to per+orm the tasks o the a%o#e me ",
^G6bin6bash ^ read-menu: a menu driven s1stem information 7rogram clear echo _ :lease *elect: #< 'is7la1 *1stem Anformation !< 'is7la1 'isk *7ace $< 'is7la1 Some *7ace 3tiliCation (< Zuit _ read -7 _&nter selection [(-$] D _ if [[ $R&:=U M~ `[(-$]$ ]]T then if [[ $R&:=U MM ( ]]T then echo _:rogram terminated<_ exit fi if [[ $R&:=U MM # ]]T then echo _Sostname: $S *TL4+&_ u7time exit fi if [[ $R&:=U MM ! ]]T then df -h exit fi if [[ $R&:=U MM $ ]]T then if [[ $Nid -uP -eB ( ]]T then echo _Some *7ace 3tiliCation N4ll 3sersP_ du -sh 6home6E else echo _Some *7ace 3tiliCation N$3*&RP_ du -sh $S +&

1.0

>e "s
fi exit

fi else fi

echo _Anvalid entr1<_ DY! exit #

This script is logically !i#i!e! i to two parts. The +irst part !isplays the me " a ! i p"ts the respo se +rom the "ser. The seco ! part i!e ti+ies the respo se a ! carries o"t the selecte! actio . &otice the "se o+ the exit comma ! i this script. 9t is "se! here to pre#e t the script +rom e3ec"ti g " ecessary co!e a+ter a actio has %ee carrie! o"t. The prese ce o+ m"ltiple e3it poi ts i a program is ge erally a %a! i!ea Eit makes program logic har!er to " !ersta !F, %"t it works i this script.

Summing -p
9 this chapter, we took o"r +irst steps towar! i teracti#ityH allowi g "sers to i p"t !ata i to o"r programs #ia the key%oar!. )si g the tech i="es prese te! th"s +ar, it is possi%le to write ma y "se+"l programs, s"ch as specialiDe! calc"latio programs a ! easy-to-"se +ro t e !s +or arca e comma ! li e tools. 9 the e3t chapter, we will %"il! o the me "-!ri#e program co cept to make it e#e %etter.

6xtra Credit
9t is importa t to st"!y the programs i this chapter care+"lly a ! ha#e a complete " !ersta !i g o+ the way they are logically str"ct"re!, as the programs to come will %e i creasi gly comple3. $s a e3ercise, rewrite the programs i this chapter "si g the test comma ! rather tha the [[ ]] compo" ! comma !. Ai t, "se gre7 to e#al"ate the reg"lar e3pressio s a ! e#al"ate its e3it stat"s. This will %e goo! practice.

;urther 1eading

The 8ash 9eference !anual co tai s a chapter o %"ilti s, which i cl"!es the read comma !, http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.htmlPJash-J"ilti s

1..

(0 8 0low Co trol, *oopi g With while - " til

30 (lo5 Control; <oo#ing With 5hile = until

9 the pre#io"s chapter, we !e#elope! a me "-!ri#e program to pro!"ce #ario"s ki !s o+ system i +ormatio . The program works, %"t it still has a sig i+ica t "sa%ility pro%lem. 9t o ly e3ec"tes a si gle choice a ! the termi ates. E#e worse, i+ a i #ali! selectio is ma!e, the program termi ates with a error, witho"t gi#i g the "ser a opport" ity to try agai . 9t wo"l! %e %etter i+ we co"l! somehow co str"ct the program so that it co"l! repeat the me " !isplay a ! selectio o#er a ! o#er, " til the "ser chooses to e3it the program. 9 this chapter, we will look at a programmi g co cept calle! looping, which ca %e "se! to make portio s o+ programs repeat. The shell pro#i!es three compo" ! comma !s +or loopi g. We will look at two o+ them i this chapter, a ! the thir! i a later o e.

Looping
'aily li+e is +"ll o+ repeate! acti#ities. ;oi g to work each !ay, walki g the !og, slici g a carrot are all tasks that i #ol#e repeati g a series o+ steps. *etIs co si!er slici g a carrot. 9+ we e3press this acti#ity i pse"!oco!e, it might look somethi g like this, .. get c"tti g %oar! 2. get k i+e (. place carrot o c"tti g %oar! 1. li+t k i+e 2. a!#a ce carrot K. slice carrot /. i+ e tire carrot slice!, the ="it, else go to step 1 Steps 1 thro"gh / +orm a loop. The actio s withi the loop are repeate! " til the co !itio , ?e tire carrot slice!,@ is reache!.

9hile
bash ca e3press a similar i!ea. *etIs say we wa te! to !isplay +i#e "m%ers i 1.2

*oopi g se="e tial or!er +rom o e to +i#e. a bash script co"l! %e co str"cte! as +ollows,
^G6bin6bash ^ 9hile-count: dis7la1 a series of numbers countM# 9hile [ $count -le " ]T do echo $count countM$NNcount 8 #PP done echo _-inished<_

Whe e3ec"te!, this script !isplays the +ollowi g,

[me@linuxbox ~]$ while-count # ! $ % " -inished<

The sy ta3 o+ the 9hile comma ! is, 9hile commandsT do commandsT done *ike if, 9hile e#al"ates the e3it stat"s o+ a list o+ comma !s. $s lo g as the e3it stat"s is Dero, it per+orms the comma !s i si!e the loop. 9 the script a%o#e, the #aria%le count is create! a ! assig e! a i itial #al"e o+ .. The 9hile comma ! e#al"ates the e3it stat"s o+ the test comma !. $s lo g as the test comma ! ret"r s a e3it stat"s o+ Dero, the comma !s withi the loop are e3ec"te!. $t the e ! o+ each cycle, the test comma ! is repeate!. $+ter si3 iteratio s o+ the loop, the #al"e o+ count has i crease! to si3, the test comma ! o lo ger ret"r s a e3it stat"s o+ Dero a ! the loop termi ates. The program co ti "es with the e3t stateme t +ollowi g the loop. We ca "se a while loop to impro#e the read-menu program +rom the pre#io"s chapter,
^G6bin6bash ^ 9hile-menu: a menu driven s1stem information 7rogram

1.(

(0 8 0low Co trol, *oopi g With while - " til

1GT:^$ ^ Lumber of seconds to dis7la1 results

while (( 3HGBT^ L$ * ))# do clear cat XX- O& -O :lease *elect: #< !< $< (< 'is7la1 *1stem Anformation 'is7la1 'isk *7ace 'is7la1 Some *7ace 3tiliCation Zuit

O& -O read -7 _&nter selection [(-$] D _ if [[ $R&:=U M~ `[(-$]$ ]]T then if [[ $R&:=U MM # ]]T then echo _Sostname: $S *TL4+&_ u7time sleep 31GT:^ fi if [[ $R&:=U MM ! ]]T then df -h sleep 31GT:^ fi if [[ $R&:=U MM $ ]]T then if [[ $Nid -uP -eB ( ]]T then echo _Some *7ace 3tiliCation N4ll 3sersP_ du -sh 6home6E else echo _Some *7ace 3tiliCation N$3*&RP_ du -sh $S +& fi sleep 31GT:^ fi else echo _Anvalid entr1<_ sleep 31GT:^ fi done echo _:rogram terminated<_

Jy e closi g the me " i a while loop, we are a%le to ha#e the program repeat the me " !isplay a+ter each selectio . The loop co ti "es as lo g as R&:=U is ot e="al to ?0@ a ! the me " is !isplaye! agai , gi#i g the "ser the opport" ity to make a other selectio . $t the e ! o+ each actio , a slee7 comma ! is e3ec"te! so the program will pa"se +or a +ew seco !s to allow the res"lts o+ the selectio to %e see %e+ore the scree is cleare! a ! the me " is re!isplaye!. 7 ce R&:=U is e="al to ?0,@ i !icati g the ?="it@ selectio , 1.1

*oopi g the loop termi ates a ! e3ec"tio co ti "es with the li e +ollowi g done.

/rea(ing ?ut ?" # Loop

bash pro#i!es two %"ilti comma !s that ca %e "se! to co trol program +low i si!e loops. The break comma ! imme!iately termi ates a loop, a ! program co trol res"mes with the e3t stateme t +ollowi g the loop. The continue comma ! ca"ses the remai !er to the loop to %e skippe!, a ! program co trol res"mes with the e3t iteratio o+ the loop. Aere we see a #ersio o+ the 9hile-menu program i corporati g %oth break a ! continue,
^G6bin6bash ^ 9hile-menu!: a menu driven s1stem information 7rogram '&=4UM$ ^ Lumber of seconds to dis7la1 results while true# do clear cat XX- O& -O :lease *elect: #< !< $< (< 'is7la1 *1stem Anformation 'is7la1 'isk *7ace 'is7la1 Some *7ace 3tiliCation Zuit

O& -O read -7 _&nter selection [(-$] D _ if [[ $R&:=U M~ `[(-$]$ ]]T then if [[ $R&:=U MM # ]]T then echo _Sostname: $S *TL4+&_ u7time slee7 $'&=4U continue fi if [[ $R&:=U MM ! ]]T then df -h slee7 $'&=4U continue fi if [[ $R&:=U MM $ ]]T then if [[ $Nid -uP -eB ( ]]T then echo _Some *7ace 3tiliCation N4ll 3sersP_ du -sh 6home6E else

1.2

(0 8 0low Co trol, *oopi g With while - " til

echo _Some *7ace 3tiliCation N$3*&RP_ du -sh $S +&

fi slee7 $'&=4U continue

fi if (( 3HGBT^ $$ * ))# then break fi else echo _Anvalid entr1<_ slee7 $'&=4U

fi done echo _:rogram terminated<_

9 this #ersio o+ the script, we set "p a endless loop Eo e that e#er termi ates o its ow F %y "si g the true comma ! to s"pply a e3it stat"s to 9hile. Si ce true will always e3it with a e3it stat"s o+ Dero, the loop will e#er e !. This is a s"rprisi gly commo scripti g tech i="e. Si ce the loop will e#er e ! o its ow , itIs "p to the programmer to pro#i!e some way to %reak o"t o+ the loop whe the time is right. 9 this script, the break comma ! is "se! to e3it the loop whe the ?0@ selectio is chose . The continue comma ! has %ee i cl"!e! at the e ! o+ the other script choices to allow +or more e++icie t e3ec"tio . Jy "si g continue, the script will skip o#er co!e that is ot ee!e! whe a selectio is i!e ti+ie!. 0or e3ample, i+ the ?.@ selectio is chose a ! i!e ti+ie!, there is o reaso to test +or the other selectio s.

until
The until comma ! is m"ch like 9hile, e3cept i stea! o+ e3iti g a loop whe a o Dero e3it stat"s is e co" tere!, it !oes the opposite. $ until loop co ti "es " til it recei#es a Dero e3it stat"s. 9 o"r 9hile-count script, we co ti "e! the loop as lo g as the #al"e o+ the count #aria%le was less tha or e="al to +i#e. We co"l! get the same res"lt %y co!i g the script with until,
^G6bin6bash ^ until-count: dis7la1 a series of numbers countM# until ( 3count -gt " )T do echo $count

1.K

Jreaki g 7"t 7+ $ *oop

countM$NNcount 8 #PP done echo _-inished<_

Jy cha gi g the test e3pressio to $count -gt ", until will termi ate the loop at the correct time. The !ecisio o+ whether to "se the 9hile or until loop is "s"ally a matter o+ choosi g the o e that allows the clearest test to %e writte .

1eading ;iles +ith Loops

9hile a ! until ca process sta !ar! i p"t. This allows +iles to %e processe! with while a ! " til loops. 9 the +ollowi g e3ample, we will !isplay the co te ts o+ the distros<txt +ile "se! i earlier chapters,
^G6bin6bash ^ 9hile-read: read lines from a file 9hile read distro version releaseT do 7rintf _'istro: 5s]t;ersion: 5s]tReleased: 5s]n_ ] $distro ] $version ] $release done / distros.txt

To re!irect a +ile to the loop, we place the re!irectio operator a+ter the done stateme t. The loop will "se read to i p"t the +iel!s +rom the re!irecte! +ile. The read comma ! will e3it a+ter each li e is rea!, with a Dero e3it stat"s " til the e !-o+-+ile is reache!. $t that poi t, it will e3it with a o -Dero e3it stat"s, there%y termi ati g the loop. 9t is also possi%le to pipe sta !ar! i p"t i to a loop,
^G6bin6bash ^ 9hile-read!: read lines from a file sort -k 1;1 -k 2n distros.txt 0 9hile read distro version releaseT do 7rintf _'istro: 5s]t;ersion: 5s]tReleased: 5s]n_ ] $distro ] $version ] $release done

1./

(0 8 0low Co trol, *oopi g With while - " til Aere we take the o"tp"t o+ the sort comma ! a ! !isplay the stream o+ te3t. Aowe#er, it is importa t to remem%er that si ce a pipe will e3ec"te the loop i a s"%shell, a y #aria%les create! or assig e! withi the loop will %e lost whe the loop termi ates.

Summing -p
With the i tro!"ctio o+ loops, a ! o"r pre#io"s e co" ters with %ra chi g, s"%ro"ti es a ! se="e ces, we ha#e co#ere! the ma4or types o+ +low co trol "se! i programs. bash has some more tricks "p its slee#e, %"t they are re+i eme ts o these %asic co cepts.

;urther 1eading

The 8ash @uide for 8eginners +rom the *i "3 'oc"me tatio 6ro4ect has some more e3amples o+ while loops, http,--tl!p.org-*'6-Jash-Jegi ers-;"i!e-html-sectO09O02.html The Wikipe!ia has a article o loops, which is part o+ a larger article o +low co trol, http,--e .wikipe!ia.org-wiki-Co trolO+lowP*oops

1.8

(. 8 Tro"%leshooti g

31 Troubleshooting
$s o"r scripts %ecome more comple3, itIs time to take a look at what happe s whe thi gs go wro g a ! they !o It !o what what we wa t. 9 this chapter, weIll look at some o+ the commo ki !s o+ errors that occ"r i scripts, a ! !escri%e a +ew "se+"l tech i="es that ca %e "se! to track !ow a ! era!icate pro%lems.

S,ntactic $rrors
7 e ge eral class o+ errors is syntactic. Sy tactic errors i #ol#e mis-typi g some eleme t o+ shell sy ta3. 9 most cases, these ki !s o+ errors will lea! to the shell re+"si g to e3ec"te the script. 9 the +ollowi g the !isc"ssio s, we will "se this script to !emo strate commo types o+ errors,
^G6bin6bash ^ trouble: scri7t to demonstrate common errors numberM# if [ $number M # ]T then echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi

$s writte , this script r" s s"ccess+"lly,

[me@linuxbox ~]$ trouble Lumber is eBual to #<

1.9

(. 8 Tro"%leshooti g

:issing Duotes
9+ we e!it o"r script a ! remo#e the traili g ="ote +rom the arg"me t +ollowi g the +irst echo comma !,
^G6bin6bash ^ trouble: scri7t to demonstrate common errors numberM# if [ $number M # ]T then echo J?umber is e,ual to 1. else echo _Lumber is not eBual to #<_ fi

watch what happe s,

[me@linuxbox ~]$ trouble 6home6me6bin6trouble: line #(: unex7ected & - 9hile looking for matching J_K 6home6me6bin6trouble: line #$: s1ntax error: unex7ected end of file

9t ge erates two errors. 9 teresti gly, the li e "m%ers reporte! are ot where the missi g ="ote was remo#e!, %"t rather m"ch later i the program. We ca see why, i+ we +ollow the program a+ter the missi g ="ote. bash will co ti "e looki g +or the closi g ="ote " til it +i !s o e, which it !oes imme!iately a+ter the seco ! echo comma !. bash %ecomes #ery co +"se! a+ter that, a ! the sy ta3 o+ the if comma ! is %roke %eca"se the fi stateme t is ow i si!e a ="ote! E%"t ope F stri g. 9 lo g scripts, this ki ! o+ error ca %e ="ite har! to +i !. )si g a e!itor with sy ta3 highlighti g will help. 9+ a complete #ersio o+ vim is i stalle!, sy ta3 highlighti g ca %e e a%le! %y e teri g the comma !,
2s!ntax on

:issing <r Unex5e/ted Tokens

$ other commo mistake is +orgetti g to complete a compo" ! comma !, s"ch as if or 120

Sy tactic Errors 9hile. *etIs look at what happe s i+ we remo#e the semicolo a+ter the test i the if comma !,
^G6bin6bash ^ trouble: scri7t to demonstrate common errors numberM# if ( 3number $ 1 ) then echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi

The res"lt is this,

[me@linuxbox ~]$ trouble 6home6me6bin6trouble: line 0: s1ntax error near unex7ected token JelseK 6home6me6bin6trouble: line 0: JelseK

$gai , the error message poi ts to a error that occ"rs later tha the act"al pro%lem. What happe s is really pretty i teresti g. $s we recall, if accepts a list o+ comma !s a ! e#al"ates the e3it co!e o+ the last comma ! i the list. 9 o"r program, we i te ! this list to co sist o+ a si gle comma !, [, a sy o ym +or test. The [ comma ! takes what +ollows it as a list o+ arg"me ts. 9 o"r case, three arg"me ts, $number, M, a ! ]. With the semicolo remo#e!, the wor! then is a!!e! to the list o+ arg"me ts, which is sy tactically legal. The +ollowi g echo comma ! is legal, too. 9tIs i terprete! as a other comma ! i the list o+ comma !s that if will e#al"ate +or a e3it co!e. The else is e co" tere! e3t, %"t itIs o"t o+ place, si ce the shell recog iDes it as a reserved word Ea wor! that has special mea i g to the shellF a ! ot the ame o+ a comma !, he ce the error message.

Unanti/i5ated 6x5ansions
9tIs possi%le to ha#e errors that o ly occ"r i termitte tly i a script. Sometimes the script will r" +i e a ! other times it will +ail %eca"se o+ res"lts o+ a e3pa sio . 9+ we ret"r o"r missi g semicolo a ! cha ge the #al"e o+ number to a empty #aria%le, we ca !emo strate,

12.

(. 8 Tro"%leshooti g
^G6bin6bash ^ trouble: scri7t to demonstrate common errors number$ if [ $number M # ]T then echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi

:"

i g the script with this cha ge res"lts i the o"tp"t,

[me@linuxbox ~]$ trouble 6home6me6bin6trouble: line ): [: M: unar1 o7erator ex7ected Lumber is not eBual to #<

We get this rather cryptic error message, +ollowe! %y the o"tp"t o+ the seco ! echo comma !. The pro%lem is the e3pa sio o+ the number #aria%le withi the test comma !. Whe the comma !,
[ $number M # ]

" !ergoes e3pa sio with number %ei g empty, the res"lt is this,
[ M # ]

which is i #ali! a ! the error is ge erate!. The M operator is a %i ary operator Eit re="ires a #al"e o each si!eF, %"t the +irst #al"e is missi g, so the test comma ! e3pects a " ary operator Es"ch as -CF i stea!. 0"rther, si ce the test +aile! E%eca"se o+ the errorF, the if comma ! recei#es a o -Dero e3it co!e a ! acts accor!i gly, a ! the seco ! echo comma ! is e3ec"te!. This pro%lem ca %e correcte! %y a!!i g ="otes aro" ! the +irst arg"me t i the test comma !,
[ _$number_ M # ]

122

Sy tactic Errors The whe e3pa sio occ"rs, the res"lt will %e this,
[ __ M # ]

which yiel!s the correct "m%er o+ arg"me ts. 9 a!!itio to empty stri gs, ="otes sho"l! %e "se! i cases where a #al"e co"l! e3pa ! i to m"lti-wor! stri gs, as with +ile ames co tai i g em%e!!e! spaces.

Logical $rrors
) like sy tactic errors, logical errors !o ot pre#e t a script +rom r" i g. The script will r" , %"t it will ot pro!"ce the !esire! res"lt, !"e to a pro%lem with its logic. There are co" tless "m%ers o+ possi%le logical errors, %"t here are a +ew o+ the most commo ki !s +o" ! i scripts, .. )ncorrect conditional ex(ressions. 9tIs easy to i correctly co!e a i+-the -else a ! ha#e the wro g logic carrie! o"t. Sometimes the logic will %e re#erse! or it will %e i complete. 2. .;"" by one0 errors. Whe co!i g loops that employ co" ters, it is possi%le to o#erlook that the loop may re="ire the co" ti g start with Dero, rather tha o e, +or the co" t to co cl"!e at the correct poi t. These ki !s o+ errors res"lt i either a loop ?goi g o++ the e !@ %y co" ti g too +ar, or else missi g the last iteratio o+ the loop %y termi ati g o e iteratio too soo . (. -nantici(ated situations. >ost logic errors res"lt +rom a program e co" teri g !ata or sit"atio s that were " +oresee %y the programmer. This ca also i cl"!e " a ticipate! e3pa sio s, s"ch as a +ile ame that co tai s em%e!!e! spaces that e3pa !s i to m"ltiple comma ! arg"me ts rather tha a si gle +ile ame.

)e=ensi;e 'rogramming
9t is importa t to #eri+y ass"mptio s whe programmi g. This mea s a care+"l e#al"atio o+ the e3it stat"s o+ programs a ! comma !s that are "se! %y a script. Aere is a e3ample, %ase! o a tr"e story. $ " +ort" ate system a!mi istrator wrote a script to per+orm a mai te a ce task o a importa t ser#er. The script co tai e! the +ollowi g two li es o+ co!e,
cd $dirOname rm E

12(

(. 8 Tro"%leshooti g There is othi g i tri sically wro g with these two li es, as lo g as the !irectory ame! i the #aria%le, dirOname, e3ists. J"t what happe s i+ it !oes otC 9 that case, the cd comma ! +ails, the script co ti "es to the e3t li e a ! !eletes the +iles i the c"rre t worki g !irectory. &ot the !esire! o"tcome at allL The hapless a!mi istrator !estroye! a importa t part o+ the ser#er %eca"se o+ this !esig !ecisio . *etIs look at some ways this !esig co"l! %e impro#e!. 0irst, it might %e wise to make the e3ec"tio o+ rm co ti ge t o the s"ccess o+ c!,
cd $dirOname YY rm E

This way, i+ the cd comma ! +ails, the rm comma ! is ot carrie! o"t. This is %etter, %"t still lea#es ope the possi%ility that the #aria%le, dirOname, is " set or empty, which wo"l! res"lt i the +iles i the "serIs home !irectory %ei g !elete!. This co"l! also %e a#oi!e! %y checki g to see that dirOname act"ally co tai s the ame o+ a e3isti g !irectory,
[[ -d $dirOname ]] YY cd $dirOname YY rm E

7+te , it is %est to termi ate the script with a error whe a sit"atio s"ch as the o e a%o#e occ"rs,
if [[ -d $dirOname ]]T then if cd $dirOnameT then rm E else echo _cannot cd to K$dirOnameK_ DY! exit # fi else echo _no such director1: K$dirOnameK_ DY! exit # fi

Aere, we check %oth the ame, to see that it is that o+ a e3isti g !irectory, a ! the s"ccess o+ the cd comma !. 9+ either +ails, a !escripti#e error message is se t to sta !ar! error a ! the script termi ates with a e3it stat"s o+ o e to i !icate a +ail"re.

121

*ogical Errors

?eri=ying In5ut
$ ge eral r"le o+ goo! programmi g is that i+ a program accepts i p"t, it m"st %e a%le to !eal with a ythi g it recei#es. This "s"ally mea s that i p"t m"st %e care+"lly scree e!, to e s"re that o ly #ali! i p"t is accepte! +or +"rther processi g. We saw a e3ample o+ this i the pre#io"s chapter whe we st"!ie! the read comma !. 7 e script co tai e! the +ollowi g test to #eri+y a me " selectio ,
[[ $R&:=U M~ `[(-$]$ ]]

This test is #ery speci+ic. 9t will o ly ret"r a Dero e3it stat"s i+ the stri g ret"r e! %y the "ser is a "meral i the ra ge o+ Dero to three. &othi g else will %e accepte!. Sometimes these sorts o+ tests ca %e #ery challe gi g to write, %"t the e++ort is ecessary to pro!"ce a high ="ality script.

Design Is # ;unction ?" Time

Whe 9 was a college st"!e t st"!yi g i !"strial !esig , a wise pro+essor state! that the !egree o+ !esig o a pro4ect was !etermi e! %y the amo" t o+ time gi#e to the !esig er. 9+ yo" were gi#e +i#e mi "tes to !esig a !e#ice ?that kills +lies,@ yo" !esig e! a +lyswatter. 9+ yo" were gi#e +i#e mo ths, yo" might come "p with a laser-g"i!e! ?a ti-+ly system@ i stea!. The same pri ciple applies to programmi g. Sometimes a ?="ick a ! !irty@ script will !o i+ itIs o ly goi g to %e "se! o ce a ! o ly "se! %y the programmer. That ki ! o+ script is commo a ! sho"l! %e !e#elope! ="ickly to make the e++ort eco omical. S"ch scripts !o It ee! a lot o+ comme ts a ! !e+e si#e checks. 7 the other ha !, i+ a script is i te !e! +or production use" that is, a script that will %e "se! o#er a ! o#er +or a importa t task or %y m"ltiple "sers, it ee!s m"ch more care+"l !e#elopme t.

Testing
Testi g is a importa t step i e#ery ki ! o+ so+tware !e#elopme t, i cl"!i g scripts. There is a sayi g i the ope so"rce worl!, ?release early, release o+te ,@ which re+lects this +act. Jy releasi g early a ! o+te , so+tware gets more e3pos"re to "se a ! testi g. E3perie ce has show that %"gs are m"ch easier to +i !, a ! m"ch less e3pe si#e to +i3, i+ they are +o" ! early i the !e#elopme t cycle. 9 a pre#io"s !isc"ssio , we saw how st"%s ca %e "se! to #eri+y program +low. 0rom 122

(. 8 Tro"%leshooti g the earliest stages o+ script !e#elopme t, they are a #al"a%le tech i="e to check the progress o+ o"r work. *etIs look at the +ile !eletio pro%lem a%o#e a ! see how this co"l! %e co!e! +or easy testi g. Testi g the origi al +ragme t o+ co!e wo"l! %e !a gero"s, si ce its p"rpose is to !elete +iles, %"t we co"l! mo!i+y the co!e to make the test sa+e,
if [[ -d $dirOname ]]T then if cd $dirOnameT then echo rm E U +GF+P?Q else echo _cannot cd to K$dirOnameK_ DY! exit # fi else echo _no such director1: K$dirOnameK_ DY! exit # fi exit U +GF+P?Q

Si ce the error co !itio s alrea!y o"tp"t "se+"l messages, we !o <t ha#e to a!! a y. The most importa t cha ge is placi g a echo comma ! 4"st %e+ore the rm comma ! to allow the comma ! a ! its e3pa !e! arg"me t list to %e !isplaye!, rather tha the comma ! act"ally %ei g e3ec"te!. This cha ge allows sa+e e3ec"tio o+ the co!e. $t the e ! o+ the co!e +ragme t, we place a exit comma ! to co cl"!e the test a ! pre#e t a y other part o+ the script +rom %ei g carrie! o"t. The ee! +or this will #ary accor!i g to the !esig o+ the script. We also i cl"!e some comme ts that act as ?markers@ +or o"r test-relate! cha ges. These ca %e "se! to help +i ! a ! remo#e the cha ges whe testi g is complete.

Test Cases
To per+orm "se+"l testi g, it<s importa t to !e#elop a ! apply goo! test cases. This is !o e %y care+"lly choosi g i p"t !ata or operati g co !itio s that re+lect edge a ! corner cases. 9 o"r co!e +ragme t Ewhich is #ery simpleF, we wa t to k ow how the co!e per+orms " !er three speci+ic co !itio s, .. dirOname co tai s the ame o+ a e3isti g !irectory 2. dirOname co tai s the ame o+ a o -e3iste t !irectory (. dirOname is empty Jy per+ormi g the test with each o+ these co !itio s, goo! test coverage is achie#e!.

12K

Testi g J"st as with !esig , testi g is a +" ctio o+ time, as well. &ot e#ery script +eat"re ee!s to %e e3te si#ely teste!. 9t<s really a matter o+ !etermi i g what is most importa t. Si ce it co"l! %e so pote tially !estr"cti#e i+ it mal+" ctio e!, o"r co!e +ragme t !eser#es care+"l co si!eratio !"ri g %oth its !esig a ! testi g.

De0ugging
9+ testi g re#eals a pro%lem with a script, the e3t step is !e%"ggi g. ?$ pro%lem@ "s"ally mea s that the script is, i some way, ot per+ormi g to the programmers e3pectatio s. 9+ this is the case, we ee! to care+"lly !etermi e e3actly what the script is act"ally !oi g a ! why. 0i !i g %"gs ca sometimes i #ol#e a lot o+ !etecti#e work. $ well !esig e! script will try to help. 9t sho"l! %e programme! !e+e si#ely, to !etect a% ormal co !itio s a ! pro#i!e "se+"l +ee!%ack to the "ser. Sometimes, howe#er, pro%lems are ="ite stra ge a ! " e3pecte! a ! more i #ol#e! tech i="es are re="ire!.

2inding The 'rob em Area

9 some scripts, partic"larly lo g o es, it is sometimes "se+"l to isolate the area o+ the script that is relate! to the pro%lem. This wo It always %e the act"al error, %"t isolatio will o+te pro#i!e i sights i to the act"al ca"se. 7 e tech i="e that ca %e "se! to isolate co!e is ?comme ti g o"t@ sectio s a script. 0or e3ample, o"r +ile !eletio +ragme t co"l! %e mo!i+ie! to !etermi e i+ the remo#e! sectio was relate! to a error,
if [[ -d $dirOname ]]T then if cd $dirOnameT then rm E else echo _cannot cd to K$dirOnameK_ DY! exit # fi U else U echo Jno such director!2 %3dir.name%J &'2 U exit 1 fi

Jy placi g comme t sym%ols at the %egi i g o+ each li e i a logical sectio o+ a script, we pre#e t that sectio +rom %ei g e3ec"te!. Testi g ca the %e per+orme! agai , to see i+ the remo#al o+ the co!e has a y impact o the %eha#ior o+ the %"g.

Tra/ing
J"gs are o+te cases o+ " e3pecte! logical +low withi a script. That is, portio s o+ the 12/

(. 8 Tro"%leshooti g script are either e#er %ei g e3ec"te!, or are %ei g e3ec"te! i the wro g or!er or at the wro g time. To #iew the act"al +low o+ the program, we "se a tech i="e calle! tracing. 7 e traci g metho! i #ol#es placi g i +ormati#e messages i a script that !isplay the locatio o+ e3ec"tio . We ca a!! messages to o"r co!e +ragme t,
echo Jpreparing to delete filesJ &'2 if [[ -d $dirOname ]]T then if cd $dirOnameT then echo Jdeleting filesJ &'2 rm E else echo _cannot cd to K$dirOnameK_ DY! exit # fi else echo _no such director1: K$dirOnameK_ DY! exit # fi echo Jfile deletion completeJ &'2

We se ! the messages to sta !ar! error to separate them +rom ormal o"tp"t. We also !o ot i !e t the li es co tai i g the messages, so it is easier to +i ! whe itIs time to remo#e them. &ow whe the script is e3ec"te!, itIs possi%le to see that the +ile !eletio has %ee per+orme!,
[me@linuxbox ~]$ deletion-scri7t 7re7aring to delete files deleting files file deletion com7lete [me@linuxbox ~]$

bash also pro#i!es a metho! o+ traci g, impleme te! %y the -x optio a ! the set comma ! with the -x optio . )si g o"r earlier trouble script, we ca acti#ate traci g +or the e tire script %y a!!i g the -x optio to the +irst li e,
UL/bin/bash -x ^ trouble: scri7t to demonstrate common errors numberM#

128

'e%"ggi g

if [ $number M # ]T then echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi

Whe e3ec"te!, the res"lts look like this,

[me@linuxbox ~]$ trouble 8 numberM# 8 K[K # M # K]K 8 echo KLumber is eBual to #<K Lumber is eBual to #<

With traci g e a%le!, we see the comma !s per+orme! with e3pa sio s applie!. The lea!i g pl"s sig s i !icate the !isplay o+ the trace to !isti g"ish them +rom li es o+ reg"lar o"tp"t. The pl"s sig is the !e+a"lt character +or trace o"tp"t. 9t is co tai e! i the :*% Eprompt stri g 1F shell #aria%le. The co te ts o+ this #aria%le ca %e a!4"ste! to make the prompt more "se+"l. Aere, we mo!i+y the co te ts o+ the #aria%le to i cl"!e the c"rre t li e "m%er i the script where the trace is per+orme!. &ote that si gle ="otes are re="ire! to pre#e t e3pa sio " til the prompt is act"ally "se!,
[me@linuxbox ~]$ export BFA$%3TP?G?S 5 % [me@linuxbox ~]$ trouble " 8 numberM# ) 8 K[K # M # K]K / 8 echo KLumber is eBual to #<K Lumber is eBual to #<

To per+orm a trace o a selecte! portio o+ a script, rather tha the e tire script, we ca "se the set comma ! with the -x optio ,
^G6bin6bash ^ trouble: scri7t to demonstrate common errors numberM# set -x U +urn on tracing if [ $number M # ]T then

129

(. 8 Tro"%leshooti g
echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi set 5x U +urn off tracing

We "se the set comma ! with the -x optio to acti#ate traci g a ! the 8x optio to !eacti#ate traci g. This tech i="e ca %e "se! to e3ami e m"ltiple portio s o+ a tro"%lesome script.

6xamining ?a ues )uring 6xe/ution

9t is o+te "se+"l, alo g with traci g, to !isplay the co te t o+ #aria%les to see the i ter al worki gs o+ a script while it is %ei g e3ec"te!. $pplyi g a!!itio al echo stateme ts will "s"ally !o the trick,
^G6bin6bash ^ trouble: scri7t to demonstrate common errors numberM# echo Jnumber$3numberJ U 1G<EQ set -x ^ Turn on tracing if [ $number M # ]T then echo _Lumber is eBual to #<_ else echo _Lumber is not eBual to #<_ fi set 8x ^ Turn off tracing

9 this tri#ial e3ample, we simply !isplay the #al"e o+ the #aria%le "m%er a ! mark the a!!e! li e with a comme t to +acilitate its later i!e ti+icatio a ! remo#al. This tech i="e is partic"larly "se+"l whe watchi g the %eha#ior o+ loops a ! arithmetic withi scripts.

Summing -p
9 this chapter, we looke! at 4"st a +ew o+ the pro%lems that ca crop "p !"ri g script !e#elopme t. 7+ co"rse, there are ma y more. The tech i="es !escri%e! here will e a%le +i !i g most commo %"gs. 'e%"ggi g is a +i e art that ca %e !e#elope! thro"gh e3perie ce, %oth i k owi g how to a#oi! %"gs Etesti g co sta tly thro"gho"t !e#elopme tF a ! i +i !i g %"gs Ee++ecti#e "se o+ traci gF. 1(0

0"rther :ea!i g

;urther 1eading

The Wikipe!ia has a co"ple o+ short articles o sy tactic a ! logical errors, http,--e .wikipe!ia.org-wiki-Sy ta3Oerror http,--e .wikipe!ia.org-wiki-*ogicOerror There are ma y o li e reso"rces +or the tech ical aspects o+ bash programmi g, http,--woole!ge.org,8000-Jash6it+alls http,--tl!p.org-*'6-a%s-html-gotchas.html http,--www.g ".org-so+tware-%ash-ma "al-htmlO o!e-:eser#e!-Wor!-9 !e3.html Eric :aymo !Is The )rt of 0ni+ 6rogramming is a great reso"rce +or lear i g the %asic co cepts +o" ! i well-writte ) i3 programs. >a y o+ these i!eas apply to shell scripts, http,--www.+a=s.org-!ocs-art"http,--www.+a=s.org-!ocs-art"-ch0.s0K.html 0or really hea#y-!"ty !e%"ggi g, there is the Jash 'e%"gger, http,--%ash!%.so"rce+orge. et-

1(.

(2 8 0low Co trol, Jra chi g With case

32 (lo5 Control; 6ranching With case

9 this chapter, we will co ti "e to look at +low co trol. 9 Chapter 29, we co str"cte! some simple me "s a ! %"ilt the logic "se! to act o a "serIs selectio . To !o this, we "se! a series o+ if comma !s to i!e ti+y which o+ the possi%le choices has %ee selecte!. This type o+ co str"ct appears +re="e tly i programs, so m"ch so that ma y programmi g la g"ages Ei cl"!i g the shellF pro#i!e a +low co trol mecha ism +or m"ltiple-choice !ecisio s.

case
The bash m"ltiple-choice compo" ! comma ! is calle! case. 9t has the +ollowi g sy ta3,
case word in [pattern [Q pattern]<<<P commands TT]<<< esac

9+ we look at the read-menu program +rom Chapter 29, we see the logic "se! to act o a "serIs selectio ,
^G6bin6bash ^ read-menu: a menu driven s1stem information 7rogram clear echo _ :lease *elect: #< 'is7la1 *1stem Anformation !< 'is7la1 'isk *7ace $< 'is7la1 Some *7ace 3tiliCation (< Zuit _ read -7 _&nter selection [(-$] D _ if (( 3HGBT^ $~ [(*- )3 ))# then if (( 3HGBT^ $$ * ))# then

1(2

case
echo JBrogram terminated.J exit

else fi

fi if (( 3HGBT^ $$ 1 ))# then echo JRostname2 3RSF+?:OGJ uptime exit fi if (( 3HGBT^ $$ 2 ))# then df -h exit fi if (( 3HGBT^ $$ ))# then if (( 34id -u6 -e, * ))# then echo JRome Fpace Etili-ation 4:ll Esers6J du -sh /home/* else echo JRome Fpace Etili-ation 43EFGH6J du -sh 3RSOG fi exit fi echo JPnvalid entr!.J &'2 exit 1

)si g case, we ca replace this logic with somethi g simpler,

^G6bin6bash ^ case-menu: a menu driven s1stem information 7rogram clear echo _ :lease *elect: #< 'is7la1 *1stem Anformation !< 'is7la1 'isk *7ace $< 'is7la1 Some *7ace 3tiliCation (< Zuit _ read -7 _&nter selection [(-$] D _ case 3HGBT^ in *6 echo JBrogram terminated.J exit ##

1((

(2 8 0low Co trol, Jra chi g With case

16 26 6 echo JRostname2 3RSF+?:OGJ uptime ## df -h ## if (( 34id -u6 -e, * ))# then echo JRome Fpace Etili-ation 4:ll Esers6J du -sh /home/* else echo JRome Fpace Etili-ation 43EFGH6J du -sh 3RSOG fi ## echo JPnvalid entr!J &'2 exit 1 ##

*6 esac

The case comma ! looks at the #al"e o+ word, i o"r e3ample, the #al"e o+ the R&:=U #aria%le, a ! the attempts to match it agai st o e o+ the speci+ie! patterns. Whe a match is +o" !, the commands associate! with the speci+ie! patter are e3ec"te!. $+ter a match is +o" !, o +"rther matches are attempte!.

'atterns
The patter s "se! %y case are the same as those "se! %y path ame e3pa sio . 6atter s are termi ate! with a ?F@ character. Aere are some #ali! patter s, Table3'- 1: case 6attern 1+amples Pattern aP [[:al7ha:]]P FFFP E<txtP EP Description >atches i+ word e="als ?a@. >atches i+ word is a si gle alpha%etic character. >atches i+ word is e3actly three characters lo g. >atches i+ word e !s with the characters ?.t3t@. >atches a y #al"e o+ word. 9t is goo! practice to i cl"!e this as the last patter i a case comma !, to catch a y #al"es o+ word that !i! ot match a pre#io"s patter H that is, to catch a y possi%le i #ali! #al"es.

Aere is a e3ample o+ patter s at work, 1(1

case
^G6bin6bash read -7 _enter 9ord D _ case $R&:=U in [[:al7ha:]]P [4HC][(-0]P FFFP E<txtP EP esac echo echo echo echo echo _is _is _is _is _is a single al7habetic character<_ TT 4> H> or C follo9ed b1 a digit<_ TT three characters long<_ TT a 9ord ending in K<txtK_ TT something else<_ TT

9t is also possi%le to com%i e m"ltiple patter s "si g the #ertical %ar character as a separator. This creates a ?or@ co !itio al patter . This is "se+"l +or s"ch thi gs as ha !li g %oth "pper a ! lower case characters. 0or e3ample,
^G6bin6bash ^ case-menu: a menu driven s1stem information 7rogram clear echo _ :lease *elect: :. 'is7la1 *1stem Anformation <. 'is7la1 'isk *7ace =. 'is7la1 Some *7ace 3tiliCation \. Zuit _ read -7 _&nter selection (:; <; = or \) D _ case $R&:=U in ,0\6 echo _:rogram terminated<_ exit TT a0:6 echo _Sostname: $S *TL4+&_ u7time TT b0<6 df -h TT c0=6 if [[ $Nid -uP -eB ( ]]T then echo _Some *7ace 3tiliCation N4ll 3sersP_ du -sh 6home6E else echo _Some *7ace 3tiliCation N$3*&RP_ du -sh $S +& fi

1(2

(2 8 0low Co trol, Jra chi g With case

TT echo _Anvalid entr1_ DY! exit # TT

EP esac

Aere, we mo!i+y the case-menu program to "se letters i stea! o+ !igits +or me " selectio . &otice how the ew patter s allow +or e try o+ %oth "pper a ! lower case letters.

Summing -p
The case comma ! is a ha !y a!!itio to o"r %ag o+ programmi g tricks. $s we will see i the e3t chapter, itIs the per+ect tool +or ha !li g certai types o+ pro%lems.

;urther 1eading

The 8ash 9eference !anual sectio o Co !itio al Co str"cts !escri%es the case comma ! i !etail, http,--tiswww.case.e!"-php-chet-%ash-%ashre+.htmlPSEC2. The )dvanced 8ash-Scripting @uide pro#i!es +"rther e3amples o+ case applicatio s, http,--tl!p.org-*'6-a%s-html-test%ra ch.html

1(K

(( 8 6ositio al 6arameters

33 2ositional 2ara%eters
7 e +eat"re that has %ee missi g +rom o"r programs is the a%ility to accept a ! process comma ! li e optio s a ! arg"me ts. 9 this chapter, we will e3ami e the shell +eat"res that allow o"r programs to get access to the co te ts o+ the comma ! li e.

#ccessing The Command Line

The shell pro#i!es a set o+ #aria%les calle! positional parameters that co tai the i !i#i!"al wor!s o the comma ! li e. The #aria%les are ame! ( thro"gh 0. They ca %e !emo strate! this way,
^G6bin6bash ^ 7osit-7aram: scri7t to vie9 command line 7arameters echo _ ]$( M $( ]$# M $# ]$! M $! ]$$ M $$ ]$% M $% ]$" M $" ]$. M $. ]$) M $) ]$/ M $/ ]$0 M $0 _

$ #ery simple script that !isplays the #al"es o+ the #aria%les $(-$0. Whe e3ec"te! with o comma ! li e arg"me ts,
[me@linuxbox ~]$ posit-param

$( M 6home6me6bin67osit-7aram

1(/

(( 8 6ositio al 6arameters
$# $! $$ $% $" $. $) $/ $0 M M M M M M M M M

E#e whe o arg"me ts are pro#i!e!, $( will always co tai the +irst item appeari g o the comma ! li e, which is the path ame o+ the program %ei g e3ec"te!. Whe arg"me ts are pro#i!e!, we see the res"lts,
[me@linuxbox ~]$ posit-param a b c d $( $# $! $$ $% $" $. $) $/ $0 M M M M M M M M M M 6home6me6bin67osit-7aram a b c d

1ote5 5o" ca act"ally access more tha i e parameters "si g parameter e3pa sio . To speci+y a "m%er greater tha i e, s"rro" ! the "m%er i %races. 0or e3ample [c.0d, [c22d, [c2..d a ! so o .

)etermining The ,umber o= Arguments

The shell also pro#i!es a #aria%le, $^, that yiel!s the "m%er o+ arg"me ts o the comma ! li e,
^G6bin6bash ^ 7osit-7aram: scri7t to vie9 command line 7arameters echo _

1(8

$ccessi g The Comma ! *i e

?umber of arguments2 3U ]$( M $( ]$# M $# ]$! M $! ]$$ M $$ ]$% M $% ]$" M $" ]$. M $. ]$) M $) ]$/ M $/ ]$0 M $0 _

The res"lt,
[me@linuxbox ~]$ posit-param a b c d Lumber of arguments: % $( M 6home6me6bin67osit-7aram $# M a $! M b $$ M c $% M d $" M $. M $) M $/ M $0 M

shift A +etting A//ess To :any Arguments

J"t what happe s whe we gi#e the program a large "m%er o+ arg"me ts s"ch as this,
[me@linuxbox ~]$ posit-param * Lumber of arguments: /! $( M 6home6me6bin67osit-7aram $# M addresses<ldif $! M bin $$ M bookmarks<html $% M debian-"((-i$/.-netinst<iso $" M debian-"((-i$/.-netinst<jigdo $. M debian-"((-i$/.-netinst<tem7late $) M debian-cdOinfo<tar<gC

1(9

(( 8 6ositio al 6arameters
$/ M 'eskto7 $0 M dirlist-bin<txt

7 this e3ample system, the wil!car! E e3pa !s i to eighty-two arg"me ts. Aow ca we process that ma yC The shell pro#i!es a metho!, al%eit cl"msy, to !o this. The shift comma ! ca"ses all o+ the parameters to ?mo#e !ow o e@ each time it is e3ec"te!. 9 +act, %y "si g shift, it is possi%le to get %y with o ly o e parameter Ei a!!itio to $(, which e#er cha gesF,
^G6bin6bash ^ 7osit-7aram!: scri7t to dis7la1 all arguments countM# 9hile [[ $^ -gt ( ]]T do echo _4rgument $count M $#_ countM$NNcount 8 #PP shift done

Each time shift is e3ec"te!, the #al"e o+ $! is mo#e! to $#, the #al"e o+ $$ is mo#e! to $! a ! so o . The #al"e o+ $^ is also re!"ce! %y o e. 9 the 7osit-7aram! program, we create a loop that e#al"ates the "m%er o+ arg"me ts remai i g a ! co ti "es as lo g as there is at least o e. We !isplay the c"rre t arg"me t, i creme t the #aria%le count with each iteratio o+ the loop to pro#i!e a r" i g co" t o+ the "m%er o+ arg"me ts processe! a !, +i ally, e3ec"te a shift to loa! $# with the e3t arg"me t. Aere is the program at work,
[me@linuxbox 4rgument # M 4rgument ! M 4rgument $ M 4rgument % M ~]$ posit-param2 a b c d a b c d

Sim5 e A55 i/ations

E#e witho"t shift, itIs possi%le to write "se+"l applicatio s "si g positio al parameters. Jy way o+ e3ample, here is a simple +ile i +ormatio program,

110

$ccessi g The Comma ! *i e

^G6bin6bash ^ fileOinfo: sim7le file information 7rogram :R @L4+&M$Nbasename $(P if [[ -e $# ]]T then echo -e _]n-ile T17e:_ file $# echo -e _]n-ile *tatus:_ stat $# else echo _$:R @L4+&: usage: $:R @L4+& file_ DY! exit # fi

This program !isplays the +ile type E!etermi e! %y the file comma ! F a ! the +ile stat"s E+rom the stat comma !F o+ a speci+ie! +ile. 7 e i teresti g +eat"re o+ this program is the :R @L4+& #aria%le. 9t is gi#e the #al"e that res"lts +rom the basename $( comma !. The basename comma ! remo#es the lea!i g portio o+ a path ame, lea#i g o ly the %ase ame o+ a +ile. 9 o"r e3ample, basename remo#es the lea!i g portio o+ the path ame co tai e! i the $( parameter, the +"ll path ame o+ o"r e3ample program. This #al"e is "se+"l whe co str"cti g messages s"ch as the "sage message at the e ! o+ the program. Jy co!i g it this way, the script ca %e re ame! a ! the message a"tomatically a!4"sts to co tai the ame o+ the program.

Using 'ositiona 'arameters With She 2un/tions

J"st as positio al parameters are "se! to pass arg"me ts to shell scripts, they ca also %e "se! to pass arg"me ts to shell +" ctio s. To !emo strate, we will co #ert the fileOinfo script i to a shell +" ctio ,
fileOinfo NP [ ^ fileOinfo: function to dis7la1 file information if [[ -e $# ]]T then echo -e _]n-ile T17e:_ file $# echo -e _]n-ile *tatus:_ stat $# else echo _$-3LCL4+&: usage: $-3LCL4+& file_ DY! return #

11.

(( 8 6ositio al 6arameters
fi \

&ow, i+ a script that i corporates the fileOinfo shell +" ctio calls the +" ctio with a +ile ame arg"me t, the arg"me t will %e passe! to the +" ctio . With this capa%ility, we ca write ma y "se+"l shell +" ctio s that ca scripts, %"t also withi the <bashrc +ile. ot o ly %e "se! i

&otice that the :R @L4+& #aria%le was cha ge! to the shell #aria%le -3LCL4+&. The shell a"tomatically "p!ates this #aria%le to keep track o+ the c"rre tly e3ec"te! shell +" ctio . &ote that $( always co tai s the +"ll path ame o+ the +irst item o the comma ! li e Ei.e., the ame o+ the programF a ! !oes ot co tai the ame o+ the shell +" ctio as we might e3pect.

3andling Positional Parameters !n 'asse

9t is sometimes "se+"l to ma age all the positio al parameters as a gro"p. 0or e3ample, we might wa t to write a ?wrapper@ aro" ! a other program. This mea s that we create a script or shell +" ctio that simpli+ies the e3ec"tio o+ a other program. The wrapper s"pplies a list o+ arca e comma ! li e optio s a ! the passes a list o+ arg"me ts to the lower le#el program. The shell pro#i!es two special parameters +or this p"rpose. They %oth e3pa ! i to the complete list o+ positio al parameters, %"t !i++er i rather s"%tle ways. They are, Table 33-1: The Q )nd R Special 6arameters Parameter $E Description E3pa !s i to the list o+ positio al parameters, starti g with .. Whe s"rro" !e! %y !o"%le ="otes, it e3pa !s i to a !o"%le ="ote! stri g co tai i g all o+ the positio al parameters, each separate! %y the +irst character o+ the 90S shell #aria%le E%y !e+a"lt a space characterF. E3pa !s i to the list o+ positio al parameters, starti g with .. Whe s"rro" !e! %y !o"%le ="otes, it e3pa !s each positio al parameter i to a separate wor! s"rro" !e! %y !o"%le ="otes.

$@

Aere is a script that shows them i actio ,

112

Aa !li g 6ositio al 6arameters E >asse

^G6bin6bash ^ 7osit-7arams$ : scri7t to demonstrate $E and $@ 7rintO7arams NP echo _]$# echo _]$! echo _]$$ echo _]$% \ [ M M M M $#_ $!_ $$_ $%_

7assO7arams NP [ echo -e _]n_ echo -e _]n_ echo -e _]n_ echo -e _]n_ \

K$E :KT K_$E_ :KT K$@ :KT K_$@_ :KT

7rintO7arams 7rintO7arams 7rintO7arams 7rintO7arams

$E _$E_ $@ _$@_

7assO7arams _9ord_ _9ords 9ith s7aces_

9 this rather co #ol"te! program, we create two arg"me ts, ?wor!@ a ! ?wor!s with spaces@ a ! pass them to the 7assO7arams +" ctio . That +" ctio , i t"r , passes them o to the 7rintO7arams +" ctio , "si g each o+ the +o"r metho!s a#aila%le with the special parameters $G a ! $@. Whe e3ec"te!, the script re#eals the !i++ere ces,
[me@linuxbox ~]$ posit-param $E : $# M 9ord $! M 9ords $$ M 9ith $% M s7aces _$E_ : $# M 9ord 9ords 9ith s7aces $! M $$ M $% M $@ : $# M 9ord $! M 9ords $$ M 9ith $% M s7aces _$@_ : $# M 9ord

11(

(( 8 6ositio al 6arameters
$! M 9ords 9ith s7aces $$ M $% M

With o"r arg"me ts, %oth $G a ! $@ pro!"ce a +o"r wor! res"lt,

9ord 9ords 9ith s7aces

_$E_ pro!"ces a o e wor! res"lt,

_9ord 9ords 9ith s7aces_

_$@_ pro!"ces a two wor! res"lt,

_9ord_ _9ords 9ith s7aces_

which matches o"r act"al i te t. The lesso to take +rom this is that e#e tho"gh the shell pro#i!es +o"r !i++ere t ways o+ getti g the list o+ positio al parameters, _$@_ is %y +ar the most "se+"l +or most sit"atio s, %eca"se it preser#es the i tegrity o+ each positio al parameter.

# >ore Complete #pplication

$+ter a lo g hiat"s, we are goi g to res"me work o o"r s1sOinfoO7age program. 7"r e3t a!!itio will a!! se#eral comma ! li e optio s to the program as +ollows,

;ut(ut "ile. We will a!! a optio to speci+y a ame +or a +ile to co tai the programIs o"tp"t. 9t will %e speci+ie! as either -f file or --file file. )nteractive mode. This optio will prompt the "ser +or a o"tp"t +ile ame a ! will !etermi e i+ the speci+ie! +ile alrea!y e3ists. 9+ it !oes, the "ser will %e prompte! %e+ore the e3isti g +ile is o#erwritte . This optio will %e speci+ie! %y either -i or --interactive. Hel(. Either -h or --hel7 may %e speci+ie! to ca"se the program to o"tp"t a i +ormati#e "sage message.

Aere is the co!e ee!e! to impleme t the comma ! li e processi g,

usage NP [ echo _$:R @L4+&: usage: $:R @L4+& [-f file Q -i]_ return \ ^ 7rocess command line o7tions interactiveM

111

$ >ore Complete $pplicatio

filenameM 9hile [[ -n $# ]]T do case $# in -f Q --fileP -i Q --interactiveP -h Q --hel7P EP esac shift done

shift filenameM$# TT interactiveM# TT usage exit TT usage DY! exit # TT

0irst, we a!! a shell +" ctio calle! usage to !isplay a message whe the help optio is i #oke! or a " k ow optio is attempte!. &e3t, we %egi the processi g loop. This loop co ti "es while the positio al parameter $# is ot empty. $t the %ottom o+ the loop, we ha#e a shift comma ! to a!#a ce the positio al parameters to e s"re that the loop will e#e t"ally termi ate. Withi the loop, we ha#e a case stateme t that e3ami es the c"rre t positio al parameter to see i+ it matches a y o+ the s"pporte! choices. 9+ a s"pporte! parameter is +o" !, it is acte! "po . 9+ ot, the "sage message is !isplaye! a ! the script termi ates with a error. The -f parameter is ha !le! i a i teresti g way. Whe !etecte!, it ca"ses a a!!itio al shift to occ"r, which a!#a ces the positio al parameter $# to the +ile ame arg"me t s"pplie! to the -f optio . We e3t a!! the co!e to impleme t the i teracti#e mo!e,
^ interactive mode if [[ -n $interactive ]]T then 9hile trueT do read -7 _&nter name of out7ut file: _ filename if [[ -e $filename ]]T then read -7 _K$filenameK exists< ver9riteF [16n6B] D _ case $R&:=U in

112

(( 8 6ositio al 6arameters
UQ1P ZQBP EP break TT echo _:rogram terminated<_ exit TT continue TT

esac elif [[ -C $filename ]]T then continue else break fi fi done

9+ the interactive #aria%le is ot empty, a e !less loop is starte!, which co tai s the +ile ame prompt a ! s"%se="e t e3isti g +ile ha !li g co!e. 9+ the !esire! o"tp"t +ile alrea!y e3ists, the "ser is prompte! to o#erwrite, choose a other +ile ame, or ="it the program. 9+ the "ser chooses to o#erwrite a e3isti g +ile, a break is e3ec"te! to termi ate the loop. &otice how the case stateme t o ly !etects i+ the "ser chooses to o#erwrite or ="it. $ y other choice ca"ses the loop to co ti "e a ! prompts the "ser agai . 9 or!er to impleme t the o"tp"t +ile ame +eat"re, we m"st +irst co #ert the e3isti g page writi g co!e i to a shell +" ctio , +or reaso s that will %ecome clear i a mome t,
9riteOhtmlO7age NP [ cat XX- O& -O XST+=D XS&4'D XTAT=&D$TAT=&X6TAT=&D X6S&4'D XH 'UD XS#D$TAT=&X6S#D X:D$TA+&O*T4+:X6:D $Nre7ortOu7timeP $Nre7ortOdiskOs7aceP $Nre7ortOhomeOs7aceP X6H 'UD X6ST+=D O& -O return \ ^ out7ut html 7age

11K

$ >ore Complete $pplicatio

if [[ -n $filename ]]T then if touch $filename YY [[ -f $filename ]]T then 9riteOhtmlO7age D $filename else echo _$:R @L4+&: Cannot 9rite file K$filenameK_ DY! exit # fi else 9riteOhtmlO7age fi

The co!e that ha !les the logic o+ the -f optio appears at the e ! o+ the listi g show a%o#e. 9 it, we test +or the e3iste ce o+ a +ile ame a !, i+ +o" !, a test is per+orme! to see i+ the +ile is i !ee! writa%le. To !o this, a touch is per+orme!, +ollowe! %y a test to !etermi e i+ the res"lti g +ile is a reg"lar +ile. These two tests take care o+ sit"atio s where a i #ali! path ame is i p"t Etouch will +ailF a ! i+ the +ile alrea!y e3ists, that itIs a reg"lar +ile. $s we ca see, the 9riteOhtmlO7age +" ctio is calle! to per+orm the act"al ge eratio o+ the page. 9ts o"tp"t is either !irecte! to sta !ar! o"tp"t Ei+ the #aria%le filename is emptyF or re!irecte! to the speci+ie! +ile.

Summing -p
With the a!!itio o+ positio al parameters, we ca ow write +airly +" ctio al scripts. 0or simple, repetiti#e tasks, positio al parameters make it possi%le to write #ery "se+"l shell +" ctio s that ca %e place! i a "serIs <bashrc +ile. 7"r s1sOinfoO7age program has grow i comple3ity a ! sophisticatio . Aere is a complete listi g, with the most rece t cha ges highlighte!,
^G6bin6bash ^ s1sOinfoO7age: 7rogram to out7ut a s1stem information 7age BHSQ?:OG$34basename 3*6 TAT=&M_*1stem Anformation Re7ort -or $S *TL4+&_ C3RR&LTOTA+&M$Ndate 8_5x 5r 5I_P TA+&O*T4+:M_@enerated $C3RR&LTOTA+&> b1 $3*&R_ re7ortOu7time NP [ cat XX- O& -O XS!D*1stem 37timeX6S!D X:R&D$Nu7timePX6:R&D

11/

(( 8 6ositio al 6arameters
O& -O return

re7ortOdiskOs7ace NP [ cat XX- O& -O XS!D'isk *7ace 3tiliCationX6S!D X:R&D$Ndf -hPX6:R&D O& -O return \ re7ortOhomeOs7ace NP [ if [[ $Nid -uP -eB ( ]]T then cat XX- O& -O XS!DSome *7ace 3tiliCation N4ll 3sersPX6S!D X:R&D$Ndu -sh 6home6EPX6:R&D O& -O else cat XX- O& -O XS!DSome *7ace 3tiliCation N$3*&RPX6S!D X:R&D$Ndu -sh $S +&PX6:R&D O& -O fi return \ usage 46 9 echo J3BHSQ?:OG2 usage2 3BHSQ?:OG (-f file 0 -i)J return > write.html.page 46 9 cat //- .GS7. /R+OT& /RG:1& /+P+TG&3+P+TG//+P+TG& //RG:1& /<S1^& /R1&3+P+TG//R1& /B&3+POG.F+:OB//B& 34report.uptime6 34report.disk.space6 34report.home.space6 //<S1^& //R+OT& .GS7. return >

118

S"mmi g )p
^ 7rocess command line o7tions interactive$ filename$ while (( -n 31 ))# do case 31 in -f 0 --file6 -i 0 --interactive6 -h 0 --help6 *6 esac shift

shift filename$31 ## interactive$1 ## usage exit ## usage &'2 exit 1 ##

done

^ interactive mode if (( -n 3interactive ))# then while true# do read -p JGnter name of output file2 J filename if (( -e 3filename ))# then read -p J%3filename% exists. Sverwrite_ (!/n/,) & J case 3HGBT^ in ^0!6 break ## \0,6 echo JBrogram terminated.J exit ## *6 continue ## esac done fi ^ out7ut html 7age if (( -n 3filename ))# then if touch 3filename '' (( -f 3filename ))# then write.html.page & 3filename else echo J3BHSQ?:OG2 =annot write file %3filename%J &'2 fi

119

(( 8 6ositio al 6arameters
exit 1 else fi fi write.html.page

WeIre ot !o e yet. There are still more thi gs we ca !o a ! impro#eme ts we ca make.

;urther 1eading

The 8ash ,ackers /iki has a goo! article o positio al parameters, http,--%ash-hackers.org-wiki-!ok".php-scripti g-posparams The 8ash 9eference !anual has a article o the special parameters, i cl"!i g $E a ! $@, http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.htmlPSpecial-6arameters 9 a!!itio to the tech i="es !isc"sse! i this chapter, bash i cl"!es a %"ilti comma ! calle! geto7ts, which ca also %e "se! +or process comma ! li e arg"me ts. 9t is !escri%e! i the SAE** J)9*T9& C7>>$&'S sectio o+ the bash ma page a ! at the 8ash ,ackers /iki, http,--%ash-hackers.org-wiki-!ok".php-howto-getoptsOt"torial

120

(1 8 0low Co trol, *oopi g With +or

3 (lo5 Control; <oo#ing With for

9 this +i al chapter o +low co trol, we will look at a other o+ the shellIs loopi g co str"cts. The for loop !i++ers +rom the while a ! " til loops i that it pro#i!es a mea s o+ processi g se="e ces !"ri g a loop. This t"r s o"t to %e #ery "se+"l whe programmi g. $ccor!i gly, the +or loop is a #ery pop"lar co str"ct i bash scripti g. $ +or loop is impleme te!, at"rally e o"gh, with the for comma !. #ersio s o+ bash, for is a#aila%le i two +orms. 9 mo!er

forI Traditional Shell ;orm

The origi al for comma !Is sy ta3 is,
for variable [in words]T do commands done

Where variable is the ame o+ a #aria%le that will i creme t !"ri g the e3ec"tio o+ the loop, words is a optio al list o+ items that will %e se="e tially assig e! to variable, a ! commands are the comma !s that are to %e e3ec"te! o each iteratio o+ the loop. The for comma ! is "se+"l o the comma ! li e. We ca easily !emo strate how it works,
[me@linuxbox ~]$ for i in : < = 1# do echo 3i# done 4 H C '

9 this e3ample, for is gi#e a list o+ +o"r wor!s, ?$,@ ?J,@ ?C,@ a ! ?'.@ With a list o+ +o"r wor!s, the loop is e3ec"te! +o"r times. Each time the loop is e3ec"te!, a wor! is assig e! to the #aria%le i. 9 si!e the loop, we ha#e a echo comma ! that !isplays the #al"e o+ i to show the assig me t. $s with the 9hile a ! until loops, the done keywor! closes the loop. 12.

(1 8 0low Co trol, *oopi g With +or The really power+"l +eat"re o+ for is the "m%er o+ i teresti g ways we ca create the list o+ wor!s. 0or e3ample, thro"gh %race e3pa sio ,
[me@linuxbox ~]$ for i in 9:..1># do echo 3i# done 4 H C '

7r path ame e3pa sio ,

[me@linuxbox ~]$ for i in distros*.txt# do echo 3i# done distros-b1-date<txt distros-dates<txt distros-ke1-names<txt distros-ke1-vernums<txt distros-names<txt distros<txt distros-vernums<txt distros-versions<txt

7r comma ! s"%stit"tio ,
^G6bin6bash ^ longest-9ord : find longest string in a file 9hile [[ -n $# ]]T do if [[ -r $# ]]T then maxO9ordM maxOlenM( for i in $Nstrings $#PT do lenM$Necho $i Q 9c -cP if NN len D maxOlen PPT then maxOlenM$len maxO9ordM$i fi done echo _$#: K$maxO9ordK N$maxOlen charactersP_ fi shift done

122

+or, Tra!itio al Shell 0orm 9 this e3ample, we look +or the lo gest stri g +o" ! withi a +ile. Whe gi#e o e or more +iles ames o the comma ! li e, this program "ses the strings program Ewhich is i cl"!e! i the ;&) %i "tils packageF to ge erate a list o+ rea!a%le te3t ?wor!s@ i each +ile. The for loop processes each wor! i t"r a ! !etermi es i+ the c"rre t wor! is the lo gest +o" ! so +ar. Whe the loop co cl"!es, the lo gest wor! is !isplaye!. 9+ the optio al in words portio o+ the for comma ! is omitte!, for !e+a"lts to processi g the positio al parameters. We will mo!i+y o"r longest-9ord script to "se this metho!,
^G6bin6bash ^ longest-9ord! : find longest string in a file for iT do if [[ -r $i ]]T then maxO9ordM maxOlenM( for j in $Nstrings $iPT do lenM$Necho $j Q 9c -cP if NN len D maxOlen PPT then maxOlenM$len maxO9ordM$j fi done echo _$i: K$maxO9ordK N$maxOlen charactersP_ fi done

$s we ca see, we ha#e cha ge! the o"termost loop to "se for i place o+ 9hile. Jy omitti g the list o+ wor!s i the for comma !, the positio al parameters are "se! i stea!. 9 si!e the loop, pre#io"s i sta ces o+ the #aria%le i ha#e %ee cha ge! to the #aria%le j. The "se o+ shift has also %ee elimi ate!.

+h, i.
5o" may ha#e otice! that the #aria%le i was chose +or each o+ the for loop e3amples a%o#e. WhyC &o speci+ic reaso act"ally, %esi!es tra!itio . The #aria%le "se! with for ca %e a y #ali! #aria%le, %"t i is the most commo , +ollowe! %y j a ! k.

12(

(1 8 0low Co trol, *oopi g With +or

The %asis o+ this tra!itio comes +rom the 0ortra programmi g la g"age. 9 0ortra , " !eclare! #aria%les starti g with the letters 9, J, M, *, a ! > are a"tomatically type! as i tegers while #aria%les %egi i g with a y other letter are type! as real E "m%ers with !ecimal +ractio sF. This %eha#ior le! programmers to "se the #aria%les 9, J, a ! M +or loop #aria%les, si ce it was less work to "se them whe a temporary #aria%le Eas loop #aria%les o+te areF was ee!e!. 9t also le! to the +ollowi g 0ortra -%ase! witticism, ?;7' is real, " less !eclare! i teger.@

forI C Language ;orm

:ece t #ersio s o+ bash ha#e a!!e! a seco ! +orm o+ for comma ! sy ta3, o e that resem%les the +orm +o" ! i the C programmi g la g"age. >a y other la g"ages s"pport this +orm, as well,
for NN expression1T expression2T expression3 PPT do commands done

where e+pression1, e+pression', a ! e+pression3 are arithmetic e3pressio s a ! commands are the comma !s to %e per+orme! !"ri g each iteratio o+ the loop. 9 terms o+ %eha#ior, this +orm is e="i#ale t to the +ollowi g co str"ct,
NN expression1 PP 9hile NN expression2 PPT do commands NN expression3 PP done

e+pression1 is "se! to i itialiDe co !itio s +or the loop, e+pression' is "se! to !etermi e whe the loop is +i ishe! a ! e+pression3 is carrie! o"t at the e ! o+ each iteratio o+ the loop. Aere is a typical applicatio ,
^G6bin6bash ^ sim7leOcounter : demo of C st1le for command for NN iM(T iX"T iMi8# PPT do echo $i done

121

+or, C *a g"age 0orm Whe e3ec"te!, it pro!"ces the +ollowi g o"tp"t,

[me@linuxbox ~]$ simple.counter ( # ! $ %

9 this e3ample, e+pression1 i itialiDes the #aria%le i with the #al"e o+ Dero, e+pression' allows the loop to co ti "e as lo g as the #al"e o+ i remai s less tha +i#e, a ! e+pression3 i creme ts the #al"e o+ i %y o e each time the loop repeats. The C la g"age +orm o+ for is "se+"l a ytime a "meric se="e ce is ee!e!. We will see se#eral applicatio s +or this i the e3t two chapters.

Summing -p
With o"r k owle!ge o+ the for comma !, we will ow apply the +i al impro#eme ts to o"r s1sOinfoO7age script. C"rre tly, the re7ortOhomeOs7ace +" ctio looks like this,
re7ortOhomeOs7ace NP [ if [[ $Nid -uP -eB ( ]]T then cat XX- O& -O XS!DSome *7ace 3tiliCation N4ll 3sersPX6S!D X:R&D$Ndu -sh 6home6EPX6:R&D O& -O else cat XX- O& -O XS!DSome *7ace 3tiliCation N$3*&RPX6S!D X:R&D$Ndu -sh $S +&PX6:R&D O& -O fi return \

&e3t, we will rewrite it to pro#i!e more !etail +or each "serIs home !irectory, a ! i cl"!e the total "m%er o+ +iles a ! s"%!irectories i each,
re7ortOhomeOs7ace NP [

122

(1 8 0low Co trol, *oopi g With +or

local formatM_5/s5#(s5#(s]n_ local i dirOlist totalOfiles totalOdirs totalOsiCe userOname if [[ $Nid -uP -eB ( ]]T then dirOlistM6home6E userOnameM_4ll 3sers_ else dirOlistM$S +& userOnameM$3*&R fi echo _XS!DSome *7ace 3tiliCation N$userOnamePX6S!D_ for i in $dirOlistT do totalOfilesM$Nfind $i -t17e f Q 9c -lP totalOdirsM$Nfind $i -t17e d Q 9c -lP totalOsiCeM$Ndu -sh $i Q cut -f #P echo _XS$D$iX6S$D_ echo _X:R&D_ 7rintf _$format_ _'irs_ _-iles_ _*iCe_ 7rintf _$format_ _----_ _-----_ _----_ 7rintf _$format_ $totalOdirs $totalOfiles $totalOsiCe echo _X6:R&D_ done return \

This rewrite applies m"ch o+ what we ha#e lear e! so +ar. We still test +or the s"per"ser, %"t i stea! o+ per+ormi g the complete set o+ actio s as part o+ the if, we set some #aria%les "se! later i a +or loop. We ha#e a!!e! se#eral local #aria%les to the +" ctio a ! ma!e "se o+ 7rintf to +ormat some o+ the o"tp"t.

;urther 1eading

The )dvanced 8ash-Scripting @uide has a chapter o loops, with a #ariety o+ e3amples "si g for, http,--tl!p.org-*'6-a%s-html-loops..html The 8ash 9eference !anual !escri%es the loopi g compo" ! comma !s, i cl"!i g for, http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.htmlP*oopi g-Co str"cts

12K

(2 8 Stri gs $ ! &"m%ers

3& Strings )nd Nu%bers

Comp"ter programs are all a%o"t worki g with !ata. 9 past chapters, we ha#e +oc"se! o processi g !ata at the +ile le#el. Aowe#er, there are ma y programmi g pro%lems that ee! to %e sol#e! "si g smaller " its o+ !ata s"ch as stri gs a ! "m%ers. 9 this chapter, we will look at se#eral shell +eat"res that are "se! to ma ip"late stri gs a ! "m%ers. The shell pro#i!es a #ariety o+ parameter e3pa sio s that per+orm stri g operatio s. 9 a!!itio to arithmetic e3pa sio Ewhich we to"che! "po i Chapter 8F, there is a commo comma ! li e program calle! bc, which per+orms higher le#el math.

Parameter $xpansion
Tho"gh parameter e3pa sio came "p i Chapter 8, we !i! ot co#er it i !etail %eca"se most parameter e3pa sio s are "se! i scripts rather tha o the comma ! li e. We ha#e alrea!y worke! with some +orms o+ parameter e3pa sio H +or e3ample, shell #aria%les. The shell pro#i!es ma y more.

Basi/ 'arameters
The simplest +orm o+ parameter e3pa sio is re+lecte! i the or!i ary "se o+ #aria%les. 0or e3ample,
$a

whe e3pa !e!, %ecomes whate#er the #aria%le a co tai s. Simple parameters may also %e s"rro" !e! %y %races,
$[a\

This has o e++ect o the e3pa sio , %"t is re="ire! i+ the #aria%le is a!4ace t to other te3t, which may co +"se the shell. 9 this e3ample, we attempt to create a +ile ame %y appe !i g the stri g ?O+ile@ to the co te ts o+ the #aria%le a.
[me@linuxbox ~]$ a$JfooJ [me@linuxbox ~]$ echo J3a.fileJ

12/

(2 8 Stri gs $ ! &"m%ers 9+ we per+orm this se="e ce, the res"lt will %e othi g, %eca"se the shell will try to e3pa ! a #aria%le ame! aOfile rather tha a. This pro%lem ca %e sol#e! %y a!!i g %races,
[me@linuxbox ~]$ echo b39a>.fileb fooOfile

We ha#e also see that positio al parameters greater tha 9 ca %e accesse! %y s"rro" !i g the "m%er i %races. 0or e3ample, to access the ele#e th positio al parameter, we ca !o this,
$[##\

6x5ansions To :anage 6m5ty ?ariab es

Se#eral parameter e3pa sio s !eal with o e3iste t a ! empty #aria%les. These e3pa sio s are ha !y +or ha !li g missi g positio al parameters a ! assig i g !e+a"lt #al"es to parameters. $[parameter:-word\ 9+ parameter is " set Ei.e., !oes ot e3istF or is empty, this e3pa sio res"lts i the #al"e o+ word. 9+ parameter is ot empty, the e3pa sio res"lts i the #al"e o+ parameter.
[me@linuxbox ~]$ [me@linuxbox ~]$ substitute value [me@linuxbox ~]$ foo$ echo 39foo2-Jsubstitute value if unsetJ> if unset echo 3foo

[me@linuxbox ~]$ foo$bar [me@linuxbox ~]$ echo 39foo2-Jsubstitute value if unsetJ> bar [me@linuxbox ~]$ echo 3foo bar

$[parameter:Mword\ 9+ parameter is " set or empty, this e3pa sio res"lts i the #al"e o+ word. 9 a!!itio , the #al"e o+ word is assig e! to parameter. 9+ parameter is ot empty, the e3pa sio res"lts i the #al"e o+ parameter.
[me@linuxbox ~]$ foo$ [me@linuxbox ~]$ echo 39foo2$Jdefault value if unsetJ>

128

6arameter E3pa sio

default value if [me@linuxbox ~]$ default value if [me@linuxbox ~]$ [me@linuxbox ~]$ bar [me@linuxbox ~]$ bar unset echo 3foo unset foo$bar echo 39foo2$Jdefault value if unsetJ> echo 3foo

1ote, 6ositio al a ! other special parameters ca ot %e assig e! this way.

$[parameter:Fword\ 9+ parameter is " set or empty, this e3pa sio ca"ses the script to e3it with a error, a ! the co te ts o+ word are se t to sta !ar! error. 9+ parameter is ot empty, the e3pa sio res"lts i the #al"e o+ parameter.
[me@linuxbox ~]$ foo$ [me@linuxbox ~]$ echo 39foo2_Jparameter is empt!J> bash: foo: 7arameter is em7t1 [me@linuxbox ~]$ echo 3_ # [me@linuxbox ~]$ foo$bar [me@linuxbox ~]$ echo 39foo2_Jparameter is empt!J> bar [me@linuxbox ~]$ echo 3_ (

$[parameter:8word\ 9+ parameter is " set or empty, the e3pa sio res"lts i othi g. 9+ parameter is ot empty, the #al"e o+ word is s"%stit"te! +or parameterH howe#er, the #al"e o+ parameter is ot cha ge!.
[me@linuxbox ~]$ foo$ [me@linuxbox ~]$ echo 39foo25Jsubstitute value if setJ> [me@linuxbox ~]$ foo$bar [me@linuxbox ~]$ echo 39foo25Jsubstitute value if setJ> substitute value if set

129

(2 8 Stri gs $ ! &"m%ers

6x5ansions That !eturn ?ariab e ,ames

The shell has the a%ility to ret"r the ames o+ #aria%les. This is "se! i some rather e3otic sit"atio s. $[GprefixE\ $[Gprefix@\ This e3pa sio ret"r s the ames o+ e3isti g #aria%les with ames %egi i g with prefi+. $ccor!i g to the bash !oc"me tatio , %oth +orms o+ the e3pa sio per+orm i!e tically. Aere, we list all the #aria%les i the e #iro me t with ames that %egi with J$SA,
[me@linuxbox ~]$ echo 39L<:FR*> H4*S H4*SO4R@C H4*SO4R@; H4*SOC ++4L' H4*SOC +:=&TA L H4*SOC +:=&TA LO'AR H4*SO=AL&L H4*SO* 3RC& H4*SO*3H*S&== H4*SO;&R*AL- H4*SO;&R*A L

String <5erations
There is a large set o+ e3pa sio s that ca %e "se! to operate o stri gs. >a y o+ these e3pa sio s are partic"larly well-s"ite! +or operatio s o path ames. $[^parameter\ E3pa !s i to the le gth o+ the stri g co tai e! %y parameter. &ormally, parameter is a stri gH howe#er, i+ parameter is either @ or E, the the e3pa sio res"lts i the "m%er o+ positio al parameters.
[me@linuxbox ~]$ foo$J+his string is long.J [me@linuxbox ~]$ echo J%3foo% is 39Ufoo> characters long.J KThis string is long<K is !( characters long<

$[parameter:offset\ $[parameter:offset:length\ This e3pa sio is "se! to e3tract a portio o+ the stri g co tai e! i parameter. The e3tractio %egi s at offset characters +rom the %egi i g o+ the stri g a ! co ti "es " til the e ! o+ the stri g " less the length is speci+ie!.
[me@linuxbox ~]$ foo$J+his string is long.J [me@linuxbox ~]$ echo 39foo2"> string is long<

1K0

6arameter E3pa sio

[me@linuxbox ~]$ echo 39foo2"2N> string

9+ the #al"e o+ offset is egati#e, it is take to mea it starts +rom the e ! o+ the stri g rather tha the %egi i g. &ote that egati#e #al"es m"st %e prece!e! %y a space to pre#e t co +"sio with the $[parameter:-word\ e3pa sio . length, i+ prese t, m"st ot %e less tha Dero. 9+ parameter is @, the res"lt o+ the e3pa sio is length positio al parameters, starti g at offset.
[me@linuxbox ~]$ foo$J+his string is long.J [me@linuxbox ~]$ echo 39foo2 -"> long< [me@linuxbox ~]$ echo 39foo2 -"22> lo

$[parameter^pattern\ $[parameter^^pattern\ This e3pa sio remo#es a lea!i g portio o+ the stri g co tai e! i parameter !e+i e! %y pattern. pattern is a wil!car! patter like those "se! i path ame e3pa sio . The !i++ere ce i the two +orms is that the ^ +orm remo#es the shortest match, while the ^^ +orm remo#es the lo gest match.
[me@linuxbox ~]$ foo$file.txt.-ip [me@linuxbox ~]$ echo 39fooU*.> txt<Ci7 [me@linuxbox ~]$ echo 39fooUU*.> Ci7

$[parameter5pattern\ $[parameter55pattern\ These e3pa sio s are the same as the ^ a ! ^^ e3pa sio s a%o#e, e3cept they remo#e te3t +rom the e ! o+ the stri g co tai e! i parameter rather tha +rom the %egi i g.
[me@linuxbox ~]$ foo$file.txt.-ip [me@linuxbox ~]$ echo 39foo8.*> file<txt [me@linuxbox ~]$ echo 39foo88.*>

1K.

(2 8 Stri gs $ ! &"m%ers
file

$[parameter6pattern6string\ $[parameter66pattern6string\ $[parameter6^pattern6string\ $[parameter65pattern6string\ This e3pa sio per+orms a search a ! replace "po the co te ts o+ parameter. 9+ te3t is +o" ! matchi g wil!car! pattern, it is replace! with the co te ts o+ string. 9 the ormal +orm, o ly the +irst occ"rre ce o+ pattern is replace!. 9 the 66 +orm, all occ"rre ces are replace!. The 6^ +orm re="ires that the match occ"r at the %egi i g o+ the stri g, a ! the 65 +orm re="ires the match to occ"r at the e ! o+ the stri g. -string may %e omitte!, which ca"ses the te3t matche! %y pattern to %e !elete!.
[me@linuxbox [me@linuxbox j7g<?:@ [me@linuxbox j7g<j7g [me@linuxbox j7g<?:@ [me@linuxbox ?:@<j7g ~]$ foo$YBQ.YBQ ~]$ echo 39foo/YBQ/jpg> ~]$ echo 39foo//YBQ/jpg> ~]$ echo 39foo/UYBQ/jpg> ~]$ echo 39foo/8YBQ/jpg>

6arameter e3pa sio is a goo! thi g to k ow. The stri g ma ip"latio e3pa sio s ca %e "se! as s"%stit"tes +or other commo comma !s s"ch as sed a ! cut. E3pa sio s impro#e the e++icie cy o+ scripts %y elimi ati g the "se o+ e3ter al programs. $s a e3ample, we will mo!i+y the longest-9ord program !isc"sse! i the pre#io"s chapter to "se the parameter e3pa sio $[^j\ i place o+ the comma ! s"%stit"tio $ Necho $j Q 9c -cP a ! its res"lti g s"%shell like so,
^G6bin6bash ^ longest-9ord$ : find longest string in a file for iT do if [[ -r $i ]]T then maxO9ordM maxOlenM for j in $Nstrings $iPT do len$39Uj> if NN len D maxOlen PPT then

1K2

6arameter E3pa sio

maxOlenM$len maxO9ordM$j

fi done echo _$i: K$maxO9ordK N$maxOlen charactersP_ fi shift done

&e3t, we will compare the e++icie cy o+ the two #ersio s %y "si g the time comma !,
[me@linuxbox ~]$ time longest-word2 dirlist-usr-bin.txt dirlist-usr-bin<txt: Kscrollkee7er-get-extended-content-listK N$/ charactersP real (m$<.#/s user (m#<"%%s s1s (m#<)./s [me@linuxbox ~]$ time longest-word dirlist-usr-bin.txt dirlist-usr-bin<txt: Kscrollkee7er-get-extended-content-listK N$/ charactersP real user s1s (m(<(.(s (m(<(".s (m(<((/s

The origi al #ersio o+ the script takes (.K.8 seco !s to sca the te3t +ile, while the ew #ersio , "si g parameter e3pa sio , takes o ly 0.0K seco !sBa #ery sig i+ica t impro#eme t.

#rithmetic $%aluation #nd $xpansion

We looke! at arithmetic e3pa sio i Chapter 8. 9t is "se! to per+orm #ario"s arithmetic operatio s o i tegers. 9ts %asic +orm is, $NNexpressionPP where e+pression is a #ali! arithmetic e3pressio . This is relate! to the compo" ! comma ! NN PP "se! +or arithmetic e#al"atio Etr"th testsF we e co" tere! i Chapter 28. 9 pre#io"s chapters, we saw some o+ the commo types o+ e3pressio s a ! operators. Aere, we will look at a more complete list.

1K(

(2 8 Stri gs $ ! &"m%ers

,umber Bases
Jack i Chapter .0, we got a look at octal E%ase 8F a ! he3a!ecimal E%ase .KF "m%ers. 9 arithmetic e3pressio s, the shell s"pports i teger co sta ts i a y %ase. Table 3.-1: Specifying -ifferent >umber 8ases 7otation number (number (xnumber base^number Some e3amples,
[me@linuxbox ~]$ echo 344*xff66 !"" [me@linuxbox ~]$ echo 3442U1111111166 !""

Description Jy !e+a"lt, "m%ers witho"t a y otatio are treate! as !ecimal E%ase .0F i tegers. 9 arithmetic e3pressio s, "m%ers with a lea!i g Dero are co si!ere! octal. Ae3a!ecimal otatio number is i base

9 the e3amples a%o#e, we pri t the #al"e o+ the he3a!ecimal "m%er ff Ethe largest two- !igit "m%erF a ! the largest eight-!igit %i ary E%ase 2F "m%er.

Unary <5erators
There are two " ary operators, the 8 a ! -, which are "se! to i !icate i+ a "m%er is positi#e or egati#e, respecti#ely. 0or e3ample, -".

Sim5 e Arithmeti/
The or!i ary arithmetic operators are liste! i the ta%le %elow, Table 3.-': )rithmetic &perators ?perator 8 1K1 Description $!!itio S"%tractio

$rithmetic E#al"atio $ ! E3pa sio E 6 EE 5 >"ltiplicatio 9 teger !i#isio E3po e tiatio >o!"lo Eremai !erF a ! mo!"lo re="ire +"rther

>ost o+ these are sel+-e3pla atory, %"t i teger !i#isio !isc"ssio .

Si ce the shellIs arithmetic o ly operates o i tegers, the res"lts o+ !i#isio are always whole "m%ers,
[me@linuxbox ~]$ echo 344 " / 2 66 !

This makes the !etermi atio o+ a remai !er i a !i#isio operatio more importa t,
[me@linuxbox ~]$ echo 344 " 8 2 66 #

Jy "si g the !i#isio a ! mo!"lo operators, we ca !etermi e that +i#e !i#i!e! %y two res"lts i two, with a remai !er o+ o e. Calc"lati g the remai !er is "se+"l i loops. 9t allows a operatio to %e per+orme! at speci+ie! i ter#als !"ri g the loop<s e3ec"tio . 9 the e3ample %elow, we !isplay a li e o+ "m%ers, highlighti g each m"ltiple o+ +i#e,
^G6bin6bash ^ modulo : demonstrate the modulo o7erator for NNi M (T i XM !(T i M i 8 #PPT do remainderM$NNi 5 "PP if NN remainder MM ( PPT then 7rintf _X5dD _ $i else 7rintf _5d _ $i fi done 7rintf _]n_

1K2

(2 8 Stri gs $ ! &"m%ers Whe e3ec"te!, the res"lts look like this,

[me@linuxbox ~]$ modulo X(D # ! $ % X"D . ) / 0 X#(D ## #! #$ #% X#"D #. #) #/ #0 X!(D

Assignment
$ltho"gh its "ses may ot %e imme!iately appare t, arithmetic e3pressio s may per+orm assig me t. We ha#e per+orme! assig me t ma y times, tho"gh i a !i++ere t co te3t. Each time we gi#e a #aria%le a #al"e, we are per+ormi g assig me t. We ca also !o it withi arithmetic e3pressio s,
[me@linuxbox ~]$ foo$ [me@linuxbox ~]$ echo 3foo [me@linuxbox ~]$ if 44 foo $ " 66#then echo JPt is true.J# fi At is true< [me@linuxbox ~]$ echo 3foo "

9 the e3ample a%o#e, we +irst assig a empty #al"e to the #aria%le foo a ! #eri+y that it is i !ee! empty. &e3t, we per+orm a if with the compo" ! comma ! NN foo M " PP. This process !oes two i teresti g thi gs, .F it assig s the #al"e o+ +i#e to the #aria%le foo a ! 2F it e#al"ates to tr"e %eca"se the assig me t was s"ccess+"l. 1ote5 9t is importa t to remem%er the e3act mea i g o+ the M i the e3pressio a%o#e. $ si gle M per+orms assig me t. foo M " says ?make foo e="al to +i#e,@ while MM e#al"ates e="i#ale ce. foo MM " says ?!oes foo e="al +i#eC@ This ca %e #ery co +"si g %eca"se the test comma ! accepts a si gle M +or stri g e="i#ale ce. This is yet a other reaso to "se the more mo!er [[ ]] a ! NN PP compo" ! comma !s i place o+ test. 9 a!!itio to the M> the shell also pro#i!es otatio s that per+orm some #ery "se+"l assig me ts, Table 3.-3: )ssignment &perators 7otation Description

1KK

$rithmetic E#al"atio $ ! E3pa sio parameter M value parameter 8M value parameter -M value parameter EM value parameter 6M value parameter 5M value parameter88 parameter"" 88parameter --parameter Simple assig me t. $ssig s value to parameter. $!!itio . E="i#ale t to parameter W parameter V value S"%tractio . E="i#ale t to parameter W parameter value >"ltiplicatio . E="i#ale t to parameter W parameter U value 9 teger !i#isio . E="i#ale t to parameter W parameter - value >o!"lo. E="i#ale t to parameter W parameter _ value Raria%le post-i creme t. E="i#ale t to parameter W parameter V . Ehowe#er, see !isc"ssio %elowF Raria%le post-!ecreme t. E="i#ale t to parameter W parameter g . Raria%le pre-i creme t. E="i#ale t to parameter W parameter V . Raria%le pre-!ecreme t. E="i#ale t to parameter W parameter g .

These assig me t operators pro#i!e a co #e ie t shortha ! +or ma y commo arithmetic tasks. 7+ special i terest are the i creme t E88F a ! !ecreme t EeeF operators, which i crease or !ecrease the #al"e o+ their parameters %y o e. This style o+ otatio is take +rom the C programmi g la g"age a ! has %ee i corporate! %y se#eral other programmi g la g"ages, i cl"!i g bash. The operators may appear either at the +ro t o+ a parameter or at the e !. While they %oth either i creme t or !ecreme t the parameter %y o e, the two placeme ts ha#e a s"%tle !i++ere ce. 9+ place! at the +ro t o+ the parameter, the parameter is i creme te! Eor !ecreme te!F %e+ore the parameter is ret"r e!. 9+ place! a+ter, the operatio is per+orme! after the parameter is ret"r e!. This is rather stra ge. %"t it is the i te !e! %eha#ior. Aere is a !emo stratio ,
[me@linuxbox ~]$ foo$1 [me@linuxbox ~]$ echo 344foo5566 # [me@linuxbox ~]$ echo 3foo

1K/

(2 8 Stri gs $ ! &"m%ers
!

9+ we assig the #al"e o+ o e to the #aria%le foo a ! the i creme t it with the 88 operator place! a+ter the parameter ame, foo is ret"r e! with the #al"e o+ o e. Aowe#er, i+ we look at the #al"e o+ the #aria%le a seco ! time, we see the i creme te! #al"e. 9+ we place the 88 operator i +ro t o+ the parameter, we get the more e3pecte! %eha#ior,
[me@linuxbox ~]$ foo$1 [me@linuxbox ~]$ echo 34455foo66 ! [me@linuxbox ~]$ echo 3foo !

0or most shell applicatio s, pre+i3i g the operator will %e the most "se+"l. The VV a ! -- operators are o+te "se! i co 4" ctio with loops. We will make some impro#eme ts to o"r mo!"lo script to tighte it "p a %it,
^G6bin6bash ^ modulo! : demonstrate the modulo o7erator for NNi M (T i XM !(T 55i PPT do if NN4i 8 "6 MM ( PPT then 7rintf _X5dD _ $i else 7rintf _5d _ $i fi done 7rintf _]n_

Bit <5erations
7 e class o+ operators ma ip"lates "m%ers i a " "s"al way. These operators work at the %it le#el. They are "se! +or certai ki !s o+ low le#el tasks, o+te i #ol#i g setti g or rea!i g %it-+lags. Table 3.-$: 8it &perators ?perator Description

1K8

$rithmetic E#al"atio $ ! E3pa sio ~ XX DD Y Q ` Jitwise egatio . &egate all the %its i a "m%er. *e+t %itwise shi+t. Shi+t all the %its i a "m%er to the le+t. :ight %itwise shi+t. Shi+t all the %its i a "m%er to the right. Jitwise $&'. 6er+orm a $&' operatio o all the %its i two "m%ers. Jitwise 7:. 6er+orm a 7: operatio o all the %its i two "m%ers. Jitwise G7:. 6er+orm a e3cl"si#e 7: operatio o all the %its i two "m%ers.

&ote that there are also correspo !i g assig me t operators E +or e3ample, YYWF +or all %"t %itwise egatio . Aere we will !emo strate pro!"ci g a list o+ powers o+ two, "si g the le+t %itwise shi+t operator,
[me@linuxbox ~]$ for 44i$*#i/M#55i66# do echo 3441//i66# done # ! % / #. $! .% #!/

Logi/
$s we !isco#ere! i Chapter 28, the NN PP compo" ! comma ! s"pports a #ariety o+ compariso operators. There are a +ew more that ca %e "se! to e#al"ate logic. Aere is the complete list, Table 3.-.: %omparison &perators ?perator XM DM X Description *ess tha or e="al to ;reater tha or e="al to *ess tha 1K9

(2 8 Stri gs $ ! &"m%ers D MM GM YY QQ expr1Fexpr2:expr3 ;reater tha E="al to &ot e="al to *ogical $&' *ogical 7: Compariso Eter aryF operator. 9+ e3pressio e+pr1 e#al"ates to %e o -Dero Earithmetic tr"eF the e+pr', else e+pr3.

Whe "se! +or logical operatio s, e3pressio s +ollow the r"les o+ arithmetic logicH that is, e3pressio s that e#al"ate as Dero are co si!ere! +alse, while o -Dero e3pressio s are co si!ere! tr"e. The NN PP compo" ! comma ! maps the res"lts i to the shellIs ormal e3it co!es,
[me@linuxbox ~]$ if 44166# then echo JtrueJ# else echo JfalseJ# fi true [me@linuxbox ~]$ if 44*66# then echo JtrueJ# else echo JfalseJ# fi false

The stra gest o+ the logical operators is the ternary operator. This operator Ewhich is mo!ele! a+ter the o e i the C programmi g la g"ageF per+orms a sta !alo e logical test. 9t ca %e "se! as a ki ! o+ i+-the -else stateme t. 9t acts o three arithmetic e3pressio s Estri gs wo It workF, a ! i+ the +irst e3pressio is tr"e Eor o -DeroF the seco ! e3pressio is per+orme!. 7therwise, the thir! e3pressio is per+orme!. We ca try this o the comma ! li e,
[me@linuxbox [me@linuxbox [me@linuxbox # [me@linuxbox [me@linuxbox ( ~]$ a$* ~]$ 44a/1_55a2--a66 ~]$ echo 3a ~]$ 44a/1_55a2--a66 ~]$ echo 3a

Aere we see a ter ary operator i actio . This e3ample impleme ts a toggle. Each time the operator is per+orme!, the #al"e o+ the #aria%le a switches +rom Dero to o e or #ice #ersa. 1/0

$rithmetic E#al"atio $ ! E3pa sio 6lease ote that per+ormi g assig me t withi the e3pressio s is ot straight+orwar!. Whe attempte!, bash will !eclare a error,
[me@linuxbox ~]$ a$* [me@linuxbox ~]$ 44a/1_a5$12a-$166 bash: NN: aX#Fa8M#:a-M#: attem7ted assignment to non-variable Nerror token is _-M#_P

This pro%lem ca pare theses,

%e mitigate! %y s"rro" !i g the assig me t e3pressio

with

[me@linuxbox ~]$ 44a/1_4a5$1624a-$1666

&e3t, we see a more complete e3ample o+ "si g arithmetic operators i a script that pro!"ces a simple ta%le o+ "m%ers,
^G6bin6bash ^ arith-loo7: scri7t to demonstrate arithmetic o7erators finishedM( aM( 7rintf _a]taEE!]taEE$]n_ 7rintf _M]tMMMM]tMMMM]n_ until NNfinishedPPT do bM$NNaEE!PP cM$NNaEE$PP 7rintf _5d]t5d]t5d]n_ $a $b $c NNaX#(F88a:NfinishedM#PPP done

9 this script, we impleme t a " til loop %ase! o the #al"e o+ the finished #aria%le. 9 itially, the #aria%le is set to Dero Earithmetic +alseF a ! we co ti "e the loop " til it %ecomes o -Dero. Withi the loop, we calc"late the s="are a ! c"%e o+ the co" ter #aria%le a. $t the e ! o+ the loop, the #al"e o+ the co" ter #aria%le is e#al"ate!. 9+ it is less tha te Ethe ma3im"m "m%er o+ iteratio sF, it is i creme te! %y o e, else the #aria%le finished is gi#e the #al"e o+ o e, maki g finished arithmetically tr"e, there%y termi ati g the loop. :" i g the script gi#es this res"lt,

1/.

(2 8 Stri gs $ ! &"m%ers
[me@linuxbox ~]$ arith-loop a aEE! aEE$ M MMMM MMMM ( ( ( # # # ! % / $ 0 !) % #. .% " !" #!" . $. !#. ) %0 $%$ / .% "#! 0 /# )!0 #( #(( #(((

bc #n #r0itrar, Precision Calculator Language

We ha#e see how the shell ca ha !le all ma er o+ i teger arithmetic, %"t what i+ we ee! to per+orm higher math or e#e 4"st "se +loati g poi t "m%ersC The a swer is, we ca It. $t least ot !irectly with the shell. To !o this, we ee! to "se a e3ter al program. There are se#eral approaches we ca take. Em%e!!i g 6erl or $WM programs is o e possi%le sol"tio , %"t " +ort" ately, o"tsi!e the scope o+ this %ook. $ other approach is to "se a specialiDe! calc"lator program. 7 e s"ch program +o" ! o most *i "3 systems is calle! bc. The bc program rea!s a +ile writte i its ow C-like la g"age a ! e3ec"tes it. $ bc script may %e a separate +ile or it may %e rea! +rom sta !ar! i p"t. The bc la g"age s"pports ="ite a +ew +eat"res i cl"!i g #aria%les, loops, a ! programmer-!e+i e! +" ctio s. We wo It co#er bc e tirely here, 4"st e o"gh to get a taste. bc is well!oc"me te! %y its ma page. *etIs start with a simple e3ample. WeIll write a bc script to a!! two pl"s two,
6E 4 ver1 sim7le bc scri7t E6 ! 8 !

The +irst li e o+ the script is a comme t. bc "ses the same sy ta3 +or comme ts as the C programmi g la g"age. Comme ts, which may spa m"ltiple li es, %egi with 6E a ! e ! with E6.

1/2

%c 8 $ $r%itrary 6recisio Calc"lator *a g"age

Using bc
9+ we sa#e the bc script a%o#e as foo<bc, we ca r" it this way,
[me@linuxbox ~]$ bc foo.bc bc #<(.<0% Co71right #00#-#00%> #00)> #00/> !(((> !((%> !((. -ree *oft9are -oundation> Anc< This is free soft9are 9ith 4H* =3T&=U L ,4RR4LTU< -or details t17e J9arrant1K< %

9+ we look care+"lly, we ca see the res"lt at the #ery %ottom, a+ter the copyright message. This message ca %e s"ppresse! with the -B E="ietF optio . bc ca also %e "se! i teracti#ely,
[me@linuxbox ~]$ bc -, 2 5 2 % ,uit

Whe "se! i teracti#ely, we simply type the calc"latio s we wish to per+orm, a ! the res"lts are imme!iately !isplaye!. The bc comma ! Buit e !s the i teracti#e sessio . 9t is also possi%le to pass a script to bc #ia sta !ar! i p"t,
[me@linuxbox ~]$ bc / foo.bc %

The a%ility to take sta !ar! i p"t mea s that we ca "se here !oc"me ts, here stri gs, a ! pipes to pass scripts. This is a here stri g e3ample,
[me@linuxbox ~]$ bc /// c252b %

An 6xam5 e S/ri5t
$s a real-worl! e3ample, we will co str"ct a script that per+orms a commo calc"latio , 1/(

(2 8 Stri gs $ ! &"m%ers mo thly loa payme ts. 9 the script %elow, we "se a here !oc"me t to pass a script to bc,
^G6bin6bash ^ loan-calc : scri7t to calculate monthl1 loan 7a1ments :R @L4+&M$Nbasename $(P usage NP [ cat XX- & 3sage: $:R @L4+& :RALCA:4= ALT&R&*T + LTS* ,here: :RALCA:4= is the amount of the loan< ALT&R&*T is the 4:R as a number N)5 M (<()P< + LTS* is the length of the loanKs term< & \ if NN$^ GM $PPT then usage exit # fi 7rinci7alM$# interestM$! monthsM$$ bc XX- & scale M #( i M $interest 6 #! 7 M $7rinci7al n M $months a M 7 E NNi E NN# 8 iP ` nPP 6 NNN# 8 iP ` nP - #PP 7rint a> _]n_ & -

Whe e3ec"te!, the res"lts look like this,

[me@linuxbox ~]$ loan-calc 1 "*** *.*CC" 1M* #!)(<)!!!%0((((

This e3ample calc"lates the mo thly payme t +or a [.(2,000 loa at /./2_ $6: +or .80 1/1

%c 8 $ $r%itrary 6recisio Calc"lator *a g"age mo ths E.2 yearsF. &otice the precisio o+ the a swer. This is !etermi e! %y the #al"e gi#e to the special scale #aria%le i the bc script. $ +"ll !escriptio o+ the bc scripti g la g"age is pro#i!e! %y the bc ma page. While its mathematical otatio is slightly !i++ere t +rom that o+ the shell Ebc more closely resem%les CF, most o+ it will %e ="ite +amiliar, %ase! o what we ha#e lear e! so +ar.

Summing -p
9 this chapter, we ha#e lear e! a%o"t ma y o+ the little thi gs that ca %e "se! to get the ?real work@ !o e i scripts. $s o"r e3perie ce with scripti g grows, the a%ility to e++ecti#ely ma ip"late stri gs a ! "m%ers will pro#e e3tremely #al"a%le. 7"r loancalc script !emo strates that e#e simple scripts ca %e create! to !o some really "se+"l thi gs.

$xtra Credit
While the %asic +" ctio ality o+ the loan-calc script is i place, the script is +ar +rom complete. 0or e3tra cre!it, try impro#i g the loan-calc script with the +ollowi g +eat"res,

0"ll #eri+icatio o+ the comma ! li e arg"me ts $ comma ! li e optio to impleme t a ?i teracti#e@ mo!e that will prompt the "ser to i p"t the pri cipal, i terest rate, a ! term o+ the loa . $ %etter +ormat +or the o"tp"t.

;urther 1eading

The 8ash ,ackers /iki has a goo! !isc"ssio o+ parameter e3pa sio , http,--%ash-hackers.org-wiki-!ok".php-sy ta3-pe The 8ash 9eference !anual co#ers this, too, http,--www.g ".org-so+tware-%ash-ma "al-%ashre+.htmlPShell-6arameterE3pa sio The /ikipedia has a goo! article !escri%i g %it operatio s, http,--e .wikipe!ia.org-wiki-JitOoperatio a ! a article o ter ary operatio s, http,--e .wikipe!ia.org-wiki-Ter aryOoperatio as well as a !escriptio o+ the +orm"la +or calc"lati g loa payme ts "se! i o"r loan-calc script, http,--e .wikipe!ia.org-wiki-$mortiDatio Ocalc"lator

1/2

(K 8 $rrays

3+ )rra$s
9 the last chapter, we looke! at how the shell ca ma ip"late stri g a ! "m%ers. The !ata types we ha#e looke! at so +ar are k ow i comp"ter scie ce circles as scalar variablesK that is, #aria%les that co tai a si gle #al"e. 9 this chapter, we will look at a other ki ! o+ !ata str"ct"re calle! a array, which hol!s m"ltiple #al"es. $rrays are a +eat"re o+ #irt"ally e#ery programmi g la g"age. The shell s"pports them, too, tho"gh i a rather limite! +ashio . E#e so, they ca %e #ery "se+"l +or sol#i g programmi g pro%lems.

+hat #re #rra,s.

$rrays are #aria%les that hol! more tha o e #al"e at a time. $rrays are orga iDe! like a ta%le. *etIs co si!er a sprea!sheet as a e3ample. $ sprea!sheet acts like a twodimensional array. 9t has %oth rows a ! col"m s, a ! a i !i#i!"al cell i the sprea!sheet ca %e locate! accor!i g to its row a ! col"m a!!ress. $ array %eha#es the same way. $ array has cells, which are calle! elements, a ! each eleme t co tai s !ata. $ i !i#i!"al array eleme t is accesse! "si g a a!!ress calle! a inde+ or subscript. >ost programmi g la g"ages s"pport multi-dimensional arrays. $ sprea!sheet is a e3ample o+ a m"lti-!ime sio al array with two !ime sio s, wi!th a ! height. >a y la g"ages s"pport arrays with a ar%itrary "m%er o+ !ime sio s, tho"gh two a ! three !ime sio al arrays are pro%a%ly the most commo ly "se!. $rrays i bash are limite! to a si gle !ime sio . We ca thi k o+ them as a sprea!sheet with a si gle col"m . E#e with this limitatio , there are ma y applicatio s +or them. $rray s"pport +irst appeare! i bash #ersio 2. The origi al ) i3 shell program, sh, !i! ot s"pport arrays at all.

Creating #n #rra,
$rray #aria%les are ame! 4"st like other bash #aria%les, a ! are create! a"tomatically whe they are accesse!. Aere is a e3ample,

1/K

Creati g $ $rray
[me@linuxbox ~]$ a(1)$foo [me@linuxbox ~]$ echo 39a(1)> foo

Aere we see a e3ample o+ %oth the assig me t a ! access o+ a array eleme t. With the +irst comma !, eleme t . o+ array a is assig e! the #al"e ?+oo@. The seco ! comma ! !isplays the store! #al"e o+ eleme t .. The "se o+ %races i the seco ! comma ! is re="ire! to pre#e t the shell +rom attempti g path ame e3pa sio o the ame o+ the array eleme t. $ array ca also %e create! with the declare comma !,
[me@linuxbox ~]$ declare -a a

)si g the -a optio , this e3ample o+ declare creates the array a.

#ssigning Values To #n #rra,

Ral"es may %e assig e! i o e o+ two ways. Si gle #al"es may %e assig e! "si g the +ollowi g sy ta3,
name[subscript]Mvalue

where name is the ame o+ the array a ! subscript is a i teger Eor arithmetic e3pressio F greater tha or e="al to Dero. &ote that the +irst eleme t o+ a array is s"%script Dero, ot o e. value is a stri g or i teger assig e! to the array eleme t. >"ltiple #al"es may %e assig e! "si g the +ollowi g sy ta3,
nameMNvalue1 value2 <<<P

where name is the ame o+ the array a ! value### are #al"es assig e! se="e tially to eleme ts o+ the array, starti g with eleme t Dero. 0or e3ample, i+ we wa te! to assig a%%re#iate! !ays o+ the week to the array da1s, we co"l! !o this,
[me@linuxbox ~]$ da!s$4Fun Oon +ue Wed +hu 7ri Fat6

9t is also possi%le to assig #al"es to a speci+ic eleme t %y speci+yi g a s"%script +or each #al"e,
[me@linuxbox ~]$ da!s$4(*)$Fun (1)$Oon (2)$+ue ( )$Wed (A)$+hu

1//

(K 8 $rrays
(")$7ri (N)$Fat6

#ccessing #rra, $lements

So what are arrays goo! +orC J"st as ma y !ata ma ageme t tasks ca %e per+orme! with a sprea!sheet program, ma y programmi g tasks ca %e per+orme! with arrays. *etIs co si!er a simple !ata gatheri g a ! prese tatio e3ample. We will co str"ct a script that e3ami es the mo!i+icatio times o+ the +iles i a speci+ie! !irectory. 0rom this !ata, o"r script will o"tp"t a ta%le showi g at what ho"r o+ the !ay the +iles were last mo!i+ie!. S"ch a script co"l! %e "se! to !etermi e whe a system is most acti#e. This script, calle! hours, pro!"ces this res"lt,
[me@linuxbox ~]$ hours . Sour -iles Sour -iles ---- ----- ---- ----(( ( #! ## (# # #$ ) (! ( #% # ($ ( #" ) (% # #. . (" # #) " (. . #/ % () $ #0 % (/ # !( # (0 #% !# ( #( ! !! ( ## " !$ ( Total files M /(

We e3ec"te the hours program, speci+yi g the c"rre t !irectory as the target. 9t pro!"ces a ta%le showi g, +or each ho"r o+ the !ay E0-2(F, how ma y +iles were last mo!i+ie!. The co!e to pro!"ce this is as +ollows,
^G6bin6bash ^ hours : scri7t to count files b1 modification time usage NP [ echo _usage: $Nbasename $(P director1_ DY! \

1/8

$ccessi g $rray Eleme ts

^ Check that argument is a director1 if [[ G -d $# ]]T then usage exit # fi ^ AnitialiCe arra1 for i in [(<<!$\T do hours[i]M(T done ^ Collect data for i in $Nstat -c 51 _$#_6E Q cut -c #!-#$PT do jM$[i6^(\ NN88hours[j]PP NN88countPP done ^ 'is7la1 data echo -e _Sour]t-iles]tSour]t-iles_ echo -e _----]t-----]t----]t-----_ for i in [(<<##\T do jM$NNi 8 #!PP 7rintf _5(!d]t5d]t5(!d]t5d]n_ $i $[hours[i]\ $j $[hours[j]\ done 7rintf _]nTotal files M 5d]n_ $count

The script co sists o+ o e +" ctio EusageF a ! a mai %o!y with +o"r sectio s. 9 the +irst sectio , we check that there is a comma ! li e arg"me t a ! that it is a !irectory. 9+ it is ot, we !isplay the "sage message a ! e3it. The seco ! sectio i itialiDes the array hours. 9t !oes this %y assig i g each eleme t a #al"e o+ Dero. There is o special re="ireme t to prepare arrays prior to "se, %"t o"r script ee!s to e s"re that o eleme t is empty. &ote the i teresti g way the loop is co str"cte!. Jy employi g %race e3pa sio E[(<<!$\F, we are a%le to easily ge erate a se="e ce o+ wor!s +or the for comma !. The e3t sectio gathers the !ata %y r" i g the stat program o each +ile i the !irectory. We "se cut to e3tract the two-!igit ho"r +rom the res"lt. 9 si!e the loop, we ee! to remo#e lea!i g Deros +rom the ho"r +iel!, si ce the shell will try Ea ! "ltimately +ailF to i terpret #al"es ?00@ thro"gh ?09@ as octal "m%ers Esee Ta%le (2-.F. &e3t, we i creme t the #al"e o+ the array eleme t correspo !i g with the ho"r o+ the !ay. 0i ally, we i creme t a co" ter EcountF to track the total "m%er o+ +iles i the !irectory. The last sectio o+ the script !isplays the co te ts o+ the array. We +irst o"tp"t a co"ple o+ hea!er li es a ! the e ter a loop that pro!"ces two col"m s o+ o"tp"t. *astly, we o"tp"t the +i al tally o+ +iles.

1/9

(K 8 $rrays

#rra, ?perations
There are ma y commo array operatio s. S"ch thi gs as !eleti g arrays, !etermi i g their siDe, sorti g, etc. ha#e ma y applicatio s i scripti g.

<ut5utting The 6ntire Contents <= An Array

The s"%scripts E a ! @ ca %e "se! to access e#ery eleme t i a array. $s with positio al parameters, the @ otatio is the more "se+"l o+ the two. Aere is a !emo stratio ,
[me@linuxbox ~]$ animals$4Ja dogJ Ja catJ Ja fishJ6 [me@linuxbox ~]$ for i in 39animals(*)># do echo 3i# done a dog a cat a fish [me@linuxbox ~]$ for i in 39animals(V)># do echo 3i# done a dog a cat a fish [me@linuxbox ~]$ for i in J39animals(*)>J# do echo 3i# done a dog a cat a fish [me@linuxbox ~]$ for i in J39animals(V)>J# do echo 3i# done a dog a cat a fish

We create the array animals a ! assig it three two-wor! stri gs. We the e3ec"te +o"r loops to see the a++ect o+ wor!-splitti g o the array co te ts. The %eha#ior o+ otatio s $[animals[E]\ a ! $[animals[@]\ are i!e tical " til they are ="ote!. The E otatio res"lts i a si gle wor! co tai i g the arrayIs co te ts, while the @ otatio res"lts i three wor!s, which matches the arrays ?real@ co te ts.

)etermining The ,umber <= Array 6 ements

)si g parameter e3pa sio , we ca !etermi e the "m%er o+ eleme ts i a array i m"ch the same way as +i !i g the le gth o+ a stri g. Aere is a e3ample,

180

$rray 7peratio s
[me@linuxbox ~]$ a(1**)$foo [me@linuxbox ~]$ echo 39Ua(V)> U number of arra! elements # [me@linuxbox ~]$ echo 39Ua(1**)> U length of element 1** $

We create array a a ! assig the stri g ?+oo@ to eleme t .00. &e3t, we "se parameter e3pa sio to e3ami e the le gth o+ the array, "si g the @ otatio . 0i ally, we look at the le gth o+ eleme t .00 which co tai s the stri g ?+oo@. 9t is i teresti g to ote that while we assig e! o"r stri g to eleme t .00, %ash o ly reports o e eleme t i the array. This !i++ers +rom the %eha#ior o+ some other la g"ages i which the " "se! eleme ts o+ the array Eeleme ts 0-99F wo"l! %e i itialiDe! with empty #al"es a ! co" te!.

2inding The Subs/ri5ts Used By An Array

$s bash allows arrays to co tai ?gaps@ i the assig me t o+ s"%scripts, it is sometimes "se+"l to !etermi e which eleme ts act"ally e3ist. This ca %e !o e with a parameter e3pa sio "si g the +ollowi g +orms, $[Garra [E]\ $[Garra [@]\ where array is the ame o+ a array #aria%le. *ike the other e3pa sio s that "se E a ! @> the @ +orm e close! i ="otes is the most "se+"l, as it e3pa !s i to separate wor!s,
[me@linuxbox ~]$ foo$4(2)$a (A)$b (N)$c6 [me@linuxbox ~]$ for i in J39foo(V)>J# do echo 3i# done a b c [me@linuxbox ~]$ for i in J39Lfoo(V)>J# do echo 3i# done ! % .

Adding 6 ements To The 6nd <= An Array

M owi g the "m%er o+ eleme ts i a array is o help i+ we ee! to appe ! #al"es to the e ! o+ a array, si ce the #al"es ret"r e! %y the E a ! @ otatio s !o ot tell "s the ma3im"m array i !e3 i "se. 0ort" ately, the shell pro#i!es "s with a sol"tio . Jy "si g the 8M assig me t operator, we ca a"tomatically appe ! #al"es to the e ! o+ a array. Aere, we assig three #al"es to the array foo a ! the appe ! three more. 18.

(K 8 $rrays
[me@linuxbox [me@linuxbox a b c [me@linuxbox [me@linuxbox a b c d e f ~]$ foo$4a b c6 ~]$ echo 39foo(V)> ~]$ foo5$4d e f6 ~]$ echo 39foo(V)>

Sorting An Array
J"st as with sprea!sheets, it is o+te ecessary to sort the #al"es i a col"m o+ !ata. The shell has o !irect way o+ !oi g this, %"t it<s ot har! to !o with a little co!i g,
^G6bin6bash ^ arra1-sort : *ort an arra1 aMNf e d c b aP echo _ riginal arra1: $[a[@]\_ aOsortedMN$Nfor i in _$[a[@]\_T do echo $iT done Q sortPP echo _*orted arra1: $[aOsorted[@]\_

Whe e3ec"te!, the script pro!"ces this,

[me@linuxbox ~]$ arra!-sort riginal arra1: f e d c b a *orted arra1: a b c d e f

The script operates %y copyi g the co te ts o+ the origi al array E aF i to a seco ! array EaOsortedF with a tricky piece o+ comma ! s"%stit"tio . This %asic tech i="e ca %e "se! to per+orm ma y ki !s o+ operatio s o the array %y cha gi g the !esig o+ the pipeli e.

)e eting An Array
To !elete a array, "se the unset comma !,
[me@linuxbox ~]$ foo$4a b c d e f6 [me@linuxbox ~]$ echo 39foo(V)> a b c d e f

182

$rray 7peratio s
[me@linuxbox ~]$ unset foo [me@linuxbox ~]$ echo 39foo(V)> [me@linuxbox ~]$

unset may also %e "se! to !elete si gle array eleme ts,

[me@linuxbox [me@linuxbox a b c d e f [me@linuxbox [me@linuxbox a b d e f ~]$ foo$4a b c d e f6 ~]$ echo 39foo(V)> ~]$ unset %foo(2)% ~]$ echo 39foo(V)>

9 this e3ample, we !elete the thir! eleme t o+ the array, s"%script 2. :emem%er, arrays start with s"%script Dero, ot o eL &otice also that the array eleme t m"st %e ="ote! to pre#e t the shell +rom per+ormi g path ame e3pa sio . 9 teresti gly, the assig me t o+ a empty #al"e to a array !oes ot empty its co te ts,
[me@linuxbox ~]$ foo$4a b c d e f6 [me@linuxbox ~]$ foo$ [me@linuxbox ~]$ echo 39foo(V)> b c d e f

$ y re+ere ce to a array #aria%le witho"t a s"%script re+ers to eleme t Dero o+ the array,
[me@linuxbox [me@linuxbox a b c d e f [me@linuxbox [me@linuxbox 4 b c d e f ~]$ foo$4a b c d e f6 ~]$ echo 39foo(V)> ~]$ foo$: ~]$ echo 39foo(V)>

Summing -p
9+ we search the bash ma page +or the wor! ?array,@ we +i ! ma y i sta ces o+ where bash makes "se o+ array #aria%les. >ost o+ these are rather o%sc"re, %"t they may pro#i!e occasio al "tility i some special circ"msta ces. 9 +act, the e tire topic o+ arrays is rather " !er-"tiliDe! i shell programmi g owi g largely to the +act that the 18(

(K 8 $rrays tra!itio al ) i3 shell programs Es"ch as shF lacke! a y s"pport +or arrays. This lack o+ pop"larity is " +ort" ate %eca"se arrays are wi!ely "se! i other programmi g la g"ages a ! pro#i!e a power+"l tool +or sol#i g ma y ki !s o+ programmi g pro%lems. $rrays a ! loops ha#e a at"ral a++i ity a ! are o+te "se! together. The for NNexprT exprT exprPP +orm o+ loop is partic"larly well-s"ite! to calc"lati g array s"%scripts.

;urther 1eading
http,--e .wikipe!ia.org-wiki-ScalarOEcomp"ti gF

181

(/ 8 E3otica

3. !"otica
9 this, the +i al chapter o+ o"r 4o"r ey, we will look at some o!!s a ! e !s. While we ha#e certai ly co#ere! a lot o+ gro" ! i the pre#io"s chapters, there are ma y bash +eat"res that we ha#e ot co#ere!. >ost are +airly o%sc"re, a ! "se+"l mai ly to those i tegrati g bash i to a *i "3 !istri%"tio . Aowe#er, there are a +ew that, while ot i commo "se, are help+"l +or certai programmi g pro%lems. We will co#er them here.

6roup Commands #nd Su0shells

bash allows comma !s to %e gro"pe! together. This ca %e !o e i o e o+ two waysH either with a group command or with a subshell. Aere are e3amples o+ the sy ta3 o+ each, ;ro"p Comma !, [ command#T command!T [command$T <<<] \ S"%shell, Ncommand#T command!T [command$T<<<]P The two +orms !i++er i that a gro"p comma ! s"rro" !s its comma !s with %races a ! a s"%shell "ses pare theses. 9t is importa t to ote that, !"e to the way bash impleme ts gro"p comma !s, the %races m"st %e separate! +rom the comma !s %y a space a ! the last comma ! m"st %e termi ate! with either a semicolo or a ewli e prior to the closi g %race. So what are gro"p comma !s a ! s"%shells goo! +orC While they ha#e a importa t !i++ere ce Ewhich we will get to i a mome tF, they are %oth "se! to ma age re!irectio . *etIs co si!er a script segme t that per+orms re!irectio s o m"ltiple comma !s,
ls -l D out7ut<txt echo _=isting of foo<txt_ DD out7ut<txt cat foo<txt DD out7ut<txt

This is pretty straight+orwar!. Three comma !s with their o"tp"t re!irecte! to a +ile 182

(/ 8 E3otica ame! out7ut<txt. )si g a gro"p comma !, we co"l! co!e this as +ollows,
[ ls -lT echo _=isting of foo<txt_T cat foo<txtT \ D out7ut<txt

)si g a s"%shell is similar,

Nls -lT echo _=isting of foo<txt_T cat foo<txtP D out7ut<txt

)si g this tech i="e, we ha#e sa#e! o"rsel#es some typi g, %"t where a gro"p comma ! or s"%shell really shi es is with pipeli es. Whe co str"cti g a pipeli e o+ comma !s, it is o+te "se+"l to com%i e the res"lts o+ se#eral comma !s i to a si gle stream. ;ro"p comma !s a ! s"%shells make this easy,
[ ls -lT echo _=isting of foo<txt_T cat foo<txtT \ Q l7r

Aere we ha#e com%i e! the o"tp"t o+ o"r three comma !s a ! pipe! them i to the i p"t o+ l7r to pro!"ce a pri te! report.

'ro/ess Substitution
While they look similar a ! ca %oth %e "se! to com%i e streams +or re!irectio , there is a importa t !i++ere ce %etwee gro"p comma !s a ! s"%shells. Whereas a gro"p comma ! e3ec"tes all o+ its comma !s i the c"rre t shell, a s"%shell Eas the ame s"ggestsF e3ec"tes its comma !s i a chil! copy o+ the c"rre t shell. This mea s that the e #iro me t is copie! a ! gi#e to a ew i sta ce o+ the shell. Whe the s"%shell e3its, the copy o+ the e #iro me t is lost, so a y cha ges ma!e to the s"%shellIs e #iro me t Ei cl"!i g #aria%le assig me tF is lost as well. There+ore, i most cases, " less a script re="ires a s"%shell, gro"p comma !s are pre+era%le to s"%shells. ;ro"p comma !s are %oth +aster a ! re="ire less memory. We saw a e3ample o+ the s"%shell e #iro me t pro%lem i Chapter 29, whe we !isco#ere! that a read comma ! i a pipeli e !oes ot work as we might i t"iti#ely e3pect. To recap, i+ we co str"ct a pipeli e like this,
echo _foo_ Q read echo $R&:=U

18K

;ro"p Comma !s $ ! S"%shells The co te t o+ the R&:=U #aria%le is always empty %eca"se the read comma ! is e3ec"te! i a s"%shell, a ! its copy o+ R&:=U is !estroye! whe the s"%shell termi ates. Jeca"se comma !s i pipeli es are always e3ec"te! i s"%shells, a y comma ! that assig s #aria%les will e co" ter this iss"e. 0ort" ately, the shell pro#i!es a e3otic +orm o+ e3pa sio calle! process substitution that ca %e "se! to work aro" ! this pro%lem. 6rocess s"%stit"tio is e3presse! i two ways, 0or processes that pro!"ce sta !ar! o"tp"t, XNlistP or, +or processes that i take sta !ar! i p"t, DNlistP where list is a list o+ comma !s. To sol#e o"r pro%lem with read, we ca employ process s"%stit"tio like this,
read X XNecho _foo_P echo $R&:=U

6rocess s"%stit"tio allows "s to treat the o"tp"t o+ a s"%shell as a or!i ary +ile +or p"rposes o+ re!irectio . 9 +act, si ce it is a +orm o+ e3pa sio , we ca e3ami e its real #al"e,
[me@linuxbox ~]$ echo XNecho _foo_P 6dev6fd6.$

Jy "si g echo to #iew the res"lt o+ the e3pa sio , we see that the o"tp"t o+ the s"%shell is %ei g pro#i!e! %y a +ile ame! 6dev6fd6.$. 6rocess s"%stit"tio is o+te "se! with loops co tai i g read. Aere is a e3ample o+ a read loop that processes the co te ts o+ a !irectory listi g create! %y a s"%shell,
^G6bin6bash ^ 7ro-sub : demo of 7rocess substitution 9hile read attr links o9ner grou7 siCe date time filenameT do cat XX- & -ilename: $filename

18/

(/ 8 E3otica
*iCe: 9ner: @rou7: +odified: =inks: 4ttributes: $siCe $o9ner $grou7 $date $time $links $attr

& done X XNls -l Q tail -n 8!P

The loop e3ec"tes read +or each li e o+ a !irectory listi g. The listi g itsel+ is pro!"ce! o the +i al li e o+ the script. This li e re!irects the o"tp"t o+ the process s"%stit"tio i to the sta !ar! i p"t o+ the loop. The tail comma ! is i cl"!e! i the process s"%stit"tio pipeli e to elimi ate the +irst li e o+ the listi g, which is ot ee!e!. Whe e3ec"te!, the script pro!"ces o"tp"t like this,
[me@linuxbox ~]$ pro.sub 0 head -n 2* -ilename: addresses<ldif *iCe: #%"%( 9ner: me @rou7: me +odified: !((0-(%-(! ##:#! =inks: # 4ttributes: -r9-r--r--ilename: *iCe: 9ner: @rou7: +odified: =inks: 4ttributes: -ilename: *iCe: 9ner: @rou7: bin %(0. me me !((0-()-#( ():$# ! dr9xr-xr-x bookmarks<html $0%!#$ me me

Traps
9 Chapter .., we saw how programs ca respo ! to sig als. We ca a!! this capa%ility to o"r scripts, too. While the scripts we ha#e writte so +ar ha#e ot ee!e! this capa%ility Eowi g to the +act that they ha#e #ery short e3ec"tio times, a ! !o ot create temporary +ilesF, larger a ! more complicate! scripts may %e e+it +rom ha#i g a sig al 188

Traps ha !li g ro"ti e. Whe we !esig a large, complicate! script, it is importa t to co si!er what happe s i+ the "ser logs o++ or sh"ts !ow the comp"ter while the script is r" i g. Whe s"ch a e#e t occ"rs, a sig al will %e se t to all a++ecte! processes a !, i t"r , the programs represe ti g those processes ca per+orm actio s to e s"re a proper a ! or!erly termi atio o+ the program. *etIs say, +or e3ample, that we wrote a script that create! a temporary +ile !"ri g its e3ec"tio . 9 the co"rse o+ goo! !esig , we wo"l! ha#e the script !elete the +ile whe the script +i ishes its work. 9t wo"l! also %e smart to ha#e the script !elete the +ile i+ a sig al is recei#e! i !icati g that the program was goi g to %e termi ate! premat"rely. bash pro#i!es a mecha ism +or this p"rpose k ow as a trap. Traps are impleme te! with the appropriately ame! %"ilti comma !, tra7. tra7 "ses the +ollowi g sy ta3, tra7 argument signal [signal<<<] where argument is a stri g which will %e rea! a ! treate! as a comma ! a ! signal is the speci+icatio o+ a sig al that will trigger the e3ec"tio o+ the i terprete! comma !. Aere is a simple e3ample,
^G6bin6bash ^ tra7-demo : sim7le signal handling demo tra7 _echo KA am ignoring 1ou<K_ *A@ALT *A@T&R+ for i in [#<<"\T do echo _Ateration $i of "_ slee7 " done

This script !e+i es a trap that will e3ec"te a echo comma ! each time either the S9;9&T or S9;TE:> sig al is recei#e! while the script is r" i g. E3ec"tio o+ the program looks like this whe the "ser attempts to stop the script %y typi g Ctrl-c,
[me@linuxbox ~]$ trap-demo Ateration # of " Ateration ! of " A am ignoring 1ou< Ateration $ of " A am ignoring 1ou< Ateration % of " Ateration " of "

189

(/ 8 E3otica $s we ca see, each time the "ser attempts to i terr"pt the program, the message is pri te! i stea!. Co str"cti g a stri g to +orm a "se+"l se="e ce o+ comma !s ca %e awkwar!, so it is commo practice to speci+y a shell +" ctio as the comma !. 9 this e3ample, a separate shell +" ctio is speci+ie! +or each sig al to %e ha !le!,
^G6bin6bash ^ tra7-demo! : sim7le signal handling demo exitOonOsignalO*A@ALT NP [ echo _*cri7t interru7ted<_ !DY# exit ( \ exitOonOsignalO*A@T&R+ NP [ echo _*cri7t terminated<_ !DY# exit ( \ tra7 exitOonOsignalO*A@ALT *A@ALT tra7 exitOonOsignalO*A@T&R+ *A@T&R+ for i in [#<<"\T do echo _Ateration $i of "_ slee7 " done

This script +eat"res two tra7 comma !s, o e +or each sig al. Each trap, i t"r , speci+ies a shell +" ctio to %e e3ec"te! whe the partic"lar sig al is recei#e!. &ote the i cl"sio o+ a exit comma ! i each o+ the sig al ha !li g +" ctio s. Witho"t a exit, the script wo"l! co ti "e a+ter completi g the +" ctio . Whe the "ser presses Ctrl-c !"ri g the e3ec"tio o+ this script, the res"lts look like this,
[me@linuxbox ~]$ trap-demo2 Ateration # of " Ateration ! of " *cri7t interru7ted<

190

Traps

Temporar, ;iles
7 e reaso sig al ha !lers are i cl"!e! i scripts is to remo#e temporary +iles that the script may create to hol! i terme!iate res"lts !"ri g e3ec"tio . There is somethi g o+ a art to ami g temporary +iles. Tra!itio ally, programs o ) i3like systems create their temporary +iles i the 6tm7 !irectory, a share! !irectory i te !e! +or s"ch +iles. Aowe#er, si ce the !irectory is share!, this poses certai sec"rity co cer s, partic"larly +or programs r" i g with s"per"ser pri#ileges. $si!e +rom the o%#io"s step o+ setti g proper permissio s +or +iles e3pose! to all "sers o+ the system, it is importa t to gi#e temporary +iles o -pre!icta%le +ile ames. This a#oi!s a e3ploit k ow as a temp race attack. 7 e way to create a o -pre!icta%le E%"t still !escripti#eF ame is to !o somethi g like this,
tem7fileM6tm76$Nbasename $(P<$$<$R4L' +

This will create a +ile ame co sisti g o+ the programIs ame, +ollowe! %y its process 9' E69'F, +ollowe! %y a ra !om i teger. &ote, howe#er, that the $R4L' + shell #aria%le o ly ret"r s a #al"e i the ra ge o+ .-(2/K/, which is ot a #ery large ra ge i comp"ter terms, so a si gle i sta ce o+ the #aria%le is ot s"++icie t to o#ercome a !etermi e! attacker. $ %etter way is to "se the mktem7 program E ot to %e co +"se! with the mktem7 sta !ar! li%rary +" ctio F to %oth ame a ! create the temporary +ile. The mktem7 program accepts a template as a arg"me t that is "se! to %"il! the +ile ame. The template sho"l! i cl"!e a series o+ ?G@ characters which are replace! %y a correspo !i g "m%er o+ ra !om letters a ! "m%ers. The lo ger the series o+ ?G@ characters, the lo ger the series o+ ra !om characters. Aere is a e3ample,
tem7fileM$Nmktem7 6tm76foobar<$$<VVVVVVVVVVP

This creates a temporary +ile a ! assig s its ame to the #aria%le tem7file. The ?G@ characters i the template are replace! with ra !om letters a ! "m%ers so that the +i al +ile ame Ewhich, i this e3ample, also i cl"!es the e3pa !e! #al"e o+ the special parameter $$ to o%tai the 69'F might %e somethi g like,
6tm76foobar<."0$<3 Iuv+.."%

While the mktem7 ma page states that mktem7 makes a temporary +ile ame, mktem7 also creates the +ile as well. 0or scripts that are e3ec"te! %y reg"lar "sers, it may %e wise to a#oi! the "se o+ the 6tm7 !irectory a ! create a !irectory +or temporary +iles withi the "serIs home !irectory, with a li e o+ co!e s"ch as this,

19.

(/ 8 E3otica

[[ -d $S +&6tm7 ]] QQ mkdir $S +&6tm7

#s,nchronous $xecution
9t is sometimes !esira%le to per+orm more tha o e task at the same time. We ha#e see how all mo!er operati g systems are at least m"ltitaski g i+ ot m"lti-"ser as well. Scripts ca %e co str"cte! to %eha#e i a m"ltitaski g +ashio . )s"ally this i #ol#es la" chi g a script that, i t"r , la" ches o e or more chil! scripts that per+orm a a!!itio al task while the pare t script co ti "es to r" . Aowe#er, whe a series o+ scripts r" s this way, there ca %e pro%lems keepi g the pare t a ! chil! coor!i ate!. That is, what i+ the pare t or chil! is !epe !e t o the other, a ! o e script m"st wait +or the other to +i ish its task %e+ore +i ishi g its ow C bash has a %"ilti comma ! to help ma age asynchronous e+ecution s"ch as this. The 9ait comma ! ca"ses a pare t script to pa"se " til a speci+ie! process Ei.e., the chil! scriptF +i ishes.

9ait
We will !emo strate the 9ait comma ! +irst. To !o this, we will ee! two scripts, a pare t script,
^G6bin6bash ^ as1nc-7arent : 4s1nchronous execution demo N7arentP echo _:arent: starting<<<_ echo _:arent: launching child scri7t<<<_ as1nc-child Y 7idM$G echo _:arent: child N:A'M $7idP launched<_ echo _:arent: continuing<<<_ slee7 ! echo _:arent: 7ausing to 9ait for child to finish<<<_ 9ait $7id echo _:arent: child is finished< Continuing<<<_

192

$sy chro o"s E3ec"tio

echo _:arent: 7arent is done< &xiting<_

a ! a chil! script,
^G6bin6bash ^ as1nc-child : 4s1nchronous execution demo NchildP echo _Child: child is running<<<_ slee7 " echo _Child: child is done< &xiting<_

9 this e3ample, we see that the chil! script is #ery simple. The real actio is %ei g per+orme! %y the pare t. 9 the pare t script, the chil! script is la" che! a ! p"t i to the %ackgro" !. The process 9' o+ the chil! script is recor!e! %y assig i g the 7id #aria%le with the #al"e o+ the $G shell parameter, which will always co tai the process 9' o+ the last 4o% p"t i to the %ackgro" !. The pare t script co ti "es a ! the e3ec"tes a 9ait comma ! with the 69' o+ the chil! process. This ca"ses the pare t script to pa"se " til the chil! script e3its, at which poi t the pare t script co cl"!es. Whe e3ec"te!, the pare t a ! chil! scripts pro!"ce the +ollowi g o"tp"t,
[me@linuxbox ~]$ as!nc-parent :arent: starting<<< :arent: launching child scri7t<<< :arent: child N:A'M .)%#P launched< :arent: continuing<<< Child: child is running<<< :arent: 7ausing to 9ait for child to finish<<< Child: child is done< &xiting< :arent: child is finished< Continuing<<< :arent: 7arent is done< &xiting<

7amed Pipes
9 most ) i3-like systems, it is possi%le to create a special type o+ +ile calle! a named pipe. &ame! pipes are "se! to create a co ectio %etwee two processes a ! ca %e "se! 4"st like other types o+ +iles. They are ot that pop"lar, %"t theyIre goo! to k ow a%o"t.

19(

(/ 8 E3otica There is a commo programmi g architect"re calle! client-server, which ca make "se o+ a comm" icatio metho! s"ch as ame! pipes, as well as other ki !s o+ interprocess communication s"ch as etwork co ectio s. The most wi!ely "se! type o+ clie t-ser#er system is, o+ co"rse, is a we% %rowser comm" icati g with a we% ser#er. The we% %rowser acts as the clie t, maki g re="ests to the ser#er a ! the ser#er respo !s to the %rowser with we% pages. &ame! pipes %eha#e like +iles, %"t act"ally +orm +irst-i +irst-o"t E0907F %"++ers. $s with or!i ary E" ame!F pipes, !ata goes i o e e ! a ! emerges o"t the other. With ame! pipes, it is possi%le to set "p somethi g like this, process1 D named_pipe a ! process2 X named_pipe a ! it will %eha#e as i+, process1 Q process2

Setting U5 A ,amed 'i5e

0irst, we m"st create a ame! pipe. This is !o e "si g the mkfifo comma !,
[me@linuxbox ~]$ mkfifo pipe1 [me@linuxbox ~]$ ls -l pipe1 7r9-r--r-- # me me ( !((0-()-#) (.:%# 7i7e#

Aere we "se mkfifo to create a ame! pipe calle! 7i7e#. )si g ls, we e3ami e the +ile a ! see that the +irst letter i the attri%"tes +iel! is ?p@, i !icati g that it is a ame! pipe.

Using ,amed 'i5es

To !emo strate how the ame! pipe works, we will ee! two termi al wi !ows Eor alter ately, two #irt"al co solesF. 9 the +irst termi al, we e ter a simple comma ! a ! re!irect its o"tp"t to the ame! pipe,
[me@linuxbox ~]$ ls -l & pipe1

$+ter pressi g the &nter key, the comma ! will appear to ha g. This is %eca"se there is 191

&ame! 6ipes othi g recei#i g !ata +rom the other e ! o+ the pipe yet. Whe this occ"rs, it is sai! that the pipe is blocked. This co !itio will clear o ce we attach a process to the other e ! a ! it %egi s to rea! i p"t +rom the pipe. )si g the seco ! termi al wi !ow, we e ter this comma !,
[me@linuxbox ~]$ cat / pipe1

a ! the !irectory listi g pro!"ce! +rom the +irst termi al wi !ow appears i the seco ! termi al as the o"tp"t +rom the cat comma !. The ls comma ! i the +irst termi al s"ccess+"lly completes o ce it is o lo ger %locke!.

Summing -p
Well, we ha#e complete! o"r 4o"r ey. The o ly thi g le+t to !o ow is practice, practice, practice. E#e tho"gh we co#ere! a lot o+ gro" ! i o"r trek, we %arely scratche! the s"r+ace as +ar as the comma ! li e goes. There are still tho"sa !s o+ comma ! li e programs le+t to %e !isco#ere! a ! e 4oye!. Start !iggi g aro" ! i 6usr6bin a ! yo"Ill seeL

;urther 1eading

The ?Compo" ! Comma !s@ sectio o+ the %ash ma page co tai s a +"ll !escriptio o+ gro"p comma ! a ! s"%shell otatio s. The EG6$&S97&S sectio o+ the %ash ma page co tai s a s"%sectio o+ process s"%stit"tio . The )dvanced 8ash-Scripting @uide also has a !isc"ssio o+ process s"%stit"tio , http,--tl!p.org-*'6-a%s-html-process-s"%.html (inu+ Dournal has two goo! articles o ame! pipes. The +irst, +rom Septem%er .99/, http,--www.li "34o"r al.com-article-2.2K a ! the seco !, +rom >arch 2009, http,--www.li "34o"r al.com-co te t-"si g- ame!-pipes-+i+os-%ash

192

9 !e3

Inde"

#
a2ps comma !...................................................((/ a%sol"te path ames...............................................K alias comma !.............................................2K, .(0 aliases.....................................................18, 2K, .28 $merica &atio al Sta !ar!s 9 stit"te Esee $&S9F .............................................................................K1 $merica Sta !ar! Co!e +or 9 +ormatio 9 tercha ge Esee $SC99F.......................................21 a chors...............................................................22. a o ymo"s 0T6 ser#ers.....................................201 $&S9...................................................................K1 $&S9 escape co!es.....................................K1, .K8 $&S9.S5S...........................................................K1 $pache we% ser#er..............................................2( apropos comma !................................................2( apt-cache comma !............................................/( apt-get comma !............................................../2p. aptit"!e comma !.............................................../2 archi#i g............................................................2(1 arithmetic e3pa sio ............../2, 80, (/., 12/, 1K( arithmetic e3pressio s................../2, 121, 1K(, 1KK arithmetic operators...................................../2, 1K1 arithmetic tr"th tests...................................(92, 1K( arrays........................................................................ appe ! #al"es to the e !..............................18. assig i g #al"es............................................1// creati g.........................................................1/K !eleti g.........................................................182 !etermi e "m%er o+ eleme ts.....................180 +i !i g "se! s"%scripts.................................18. i !e3.............................................................1/K m"lti-!ime sio al.........................................1/K rea!i g #aria%les i to...................................101 sorti g...........................................................182 s"%script.......................................................1/K two-!ime sio al...........................................1/K $SC99.............................82, 8K, 222, 222, 2K/, ((/

%ell character.................................................K. carriage ret"r ..............................................2/. collatio or!er..............................222, 22/, (9. co trol co!es..................................82, 222, ((. gro++ o"tp"t !ri#er........................................(21 li e+ee! character.........................................2/. "ll character................................................222 pri ta%le characters......................................222 te3t..................................................................21 aspell comma !.................................................(0( assem%ler............................................................(12 assem%ly la g"age.............................................(12 assig me t operators..........................................1KK asy chro o"s e3ec"tio .....................................192 a"!io C's....................................................81, .92 $WM programmi g la g"age....................(0(, 1/2

<
%ack re+ere ces........................................2K/, 298p. %ackslash escape se="e ces.................................8( %ackslash-escape! special characters..................K0 %ack"ps, i creme tal..........................................2(8 %ase ame comma !...........................................11. %ash...............................................................0, .28 %ash.......................................................................... ma page........................................................2( %asic reg"lar e3pressio s 228, 2KKp., 29K, (00, (.0 %c comma !.......................................................1/2 Jerkeley So+tware 'istri%"tio .........................((2 %g comma !........................................................2. %i ary........................................98p., .02, (12, 1K1 %it mask...............................................................0. %it operators.......................................................1K8 Jo"r e, Ste#e..................................................0, .( %race e3pa sio ....................................../K, 80, 122 %ra chi g............................................................(82 %reak comma !..........................................1.2, 11K

19K

9 !e3
%roke li ks..........................................................1K JS' style.............................................................K %"++eri g..............................................................8K %"gs............................................................122, 12/ %"il! e #iro me t..............................................(20 %Dip2 comma !..................................................2(( e3ec"ti g as a other "ser..............................01 lo g optio s....................................................2. optio s............................................................2. comme ts............................(2, .(8, (02, (29, 12/ Commo ) i3 6ri ti g System.................(((, (1( compariso operators.........................................1K9 compiler.............................................................(12 compili g...........................................................(11 completio s..........................................................8K compo" ! comma !s.............................................. case...............................................................1(2 +or.................................................................12. i+...................................................................(82 " til...............................................................1.K while.............................................................1.( EE FF................................................(92, 109, 1K( aa bb........................................................(9(, 109 compressio algorithms.....................................2(. co !itio al e3pressio s..............................100, 12( co +ig"ratio +iles..................................22, 2/, .28 co +ig"re comma !...........................................(20 co sta ts.............................................................(/0 co ti "e comma !.............................................1.2 co trol characters........................................K., 2/0 co trol co!es................................................82, 222 co trol operators...................................................... ^^........................................................(99, 109 ZZ....................................................................(99 co trolli g termi al..............................................1 C7659&;..........................................................(19 copyi g a ! pasti g................................................. i #im.............................................................19 o the comma ! li e......................................82 with G Wi !ow System.................................... core"tils package.........................2., 21p., 28(, (0/ co" ti g wor!s i a +ile........................................K/ cp comma !...................................(1, 1., .(2, 2.. C6)...........................................................(p., (11 cro 4o%..............................................................2.2 crosswor! p"DDles..............................................22. csplit comma !..................................................(08 C)6S..........................................................(((, (1( c"rre t worki g !irectory.....................................2 c"rsor mo#eme t..................................................81 c"t comma !..............................................280, 1K2

!
C programmi g la g"age...........(12, 121, 1K/, 1/0 CVV....................................................................(12 cal comma !.........................................................2 ca cel comma !.................................................(12 carriage ret"r ......22, 82p., .K., 222p., 2/0, (02p., ((1 case compo" ! comma !..................................1(2 cat comma !................................................K(, 2/0 c! comma !....................................................K, .8 C'-:7>s................................................8(p., .92 c!recor! comma !..............................................9K c!rtools................................................................9K character classes....(2p., 222, 221p., 22/, 2K., 29(, (0( character ra ges................................((, 22(p., (0( chgrp comma !...................................................09 chil! process.........................................................( chmo! comma !..................................9/, ..0, (K0 chow comma !.........................................0/, ..0 chro ological sorti g.........................................2// clearte3t......................................................201, 20/ clie t-ser#er architect"re....................................191 C7J7* programmi g la g"age........................(12 collatio or!er.....................(0, 222, 22/, 29(, (9. $SC99...................................................22/, (9. !ictio ary......................................................222 tra!itio al.....................................................22/ comm comma !.................................................288 comma ! history.............................................., 88 comma ! li e........................................................... arg"me ts.....................................................1(/ e!iti g........................................................., 81 e3pa sio ......................................................../2 history........................................................., 89 i ter+aces....................................................(, (1 comma ! optio s.................................................2. comma ! s"%stit"tio ............................/8, 80, 122 comma !s................................................................ arg"me ts...............................................2., 1(/ !etermi i g type.............................................19 !oc"me tatio ................................................20 e3ec"ta%le program +iles........................18, (12

/
!aemo programs.........................................(, .2( !ata compressio ................................................2(0 !ata re!" !a cy.................................................2(0

19/

9 !e3
!ata #ali!atio ....................................................(9( !ate comma !.......................................................2 !ate +ormats........................................................2// !! comma !........................................................91 'e%ia ................................................................./0 'e%ia Style E.!e%F............................................./. !e%"ggi g...................................................(8., 12/ !e+e si#e programmi g.............................12(, 12/ !elimiters..............................................8., 2/2, 2/8 !epe !e cies.............................................../2, (2( !esig .........................................................122, 12/ !e#ice !ri#ers............................................../8, (12 !e#ice ames.......................................................8K !e#ice o!es.........................................................2/ !+ comma !..................................................2, (8( !ictio ................................................................(1K !ictio ary collatio or!er...................................222 !i++ comma !.....................................................288 'igital :estrictio s >a ageme t E':>F.........../2 !irectories................................................................. archi#i g.......................................................2(1 cha gi g..........................................................K copyi g...........................................................(1 creati g.....................................................(1, 10 c"rre t worki g...............................................2 !eleti g.....................................................(/, 12 hierarchical......................................................1 home.................................................28, 92, (8( listi g..............................................................20 mo#i g......................................................(K, 12 a#igati g........................................................1 7*'O6W' #aria%le......................................(0 pare t...............................................................2 6$TA #aria%le...............................................(0 6W' #aria%le................................................(. remo#i g...................................................(/, 12 re ami g...................................................(K, 12 root..................................................................1 share!............................................................09 sticky %it........................................................0( sy chro iDi g...............................................212 tra s+erri g o#er a etwork..........................212 #iewi g co te ts..............................................2 !isk partitio s......................................................8. '9S6*$5 #aria%le..............................................(0 'olphi ................................................................(( !os2" i3 comma !............................................2/. !o"%le ="otes......................................................./9 !pkg comma !..................................................../2 !" comma !...............................................2/(, (8( 'y amic Aost Co +ig"ratio 6rotocol E'AC6F 20(

$
echo comma !...................................../2, .29, (KK -e optio ..........................................................8( - optio .......................................................102 e!ge a ! cor er cases.........................................12K E'9T7: #aria%le................................................(0 e++ecti#e gro"p 9'...............................................0( e++ecti#e "ser 9'..........................................0(, ..1 eli+ stateme t......................................................(92 email...................................................................2K9 em%e!!e! systems.............................................(12 empty #aria%les..................................................128 e crypte! t" els...............................................2.0 e cryptio ..........................................................291 e ! o+ +ile.....................................................K1, (/( e !less loop........................................................1.K e script comma !..............................................(10 e #iro me t........................................02, .28, 108 aliases............................................................28 esta%lishi g....................................................(. e3ami i g......................................................28 logi shell......................................................(. shell +" ctio s...............................................28 shell #aria%les................................................28 start"p +iles....................................................(. s"%shells.......................................................18K #aria%les........................................................28 e= comma !.....................................................(22 e3ec"ta%le +iles...................................................(2. e3ec"ta%le program +iles..............................18, (12 e3ec"ta%le programs................................................ !etermi i g locatio .......................................19 6$TA #aria%le...............................................(0 e3it comma !........................................(, (90, 1.. e3it stat"s...................................................(8K, (90 e3pa ! comma !...............................................28( e3pa sio s............................................................/2 arithmetic........................../2, 80, (/., 12/, 1K( %race................................................./K, 80, 122 comma ! s"%stit"tio ....................../8, 80, 122 !elimiters........................................................8. errors res"lti g +rom.....................................12. history.......................................................89, 9. parameter..........................//, 80, (K9, (/2, 12/ path ame........................................../(, 80, 122 til!e.........................................................../1, 80 wor!-splitti g............................................../9p. e3pressio s............................................................... arithmetic......................../2, 121, 1K(, 1KK, 1// co !itio al............................................100, 12(

198

9 !e3
e3t(......................................................................92 e3te !e! reg"lar e3pressio s.............................228 E3te si%le >ark"p *a g"age............................2K9 remo#i g...................................................(/, 12 re ami g...................................................(K, 12 rpm................................................................/0 share! li%rary..................................................28 start"p............................................................(. sticky %it........................................................0( sym%olic li ks..............................................2.K sy chro iDi g...............................................212 temporary.....................................................19. te3t..................................................................21 tra s+erri g o#er a etwork..........20(, 2(9, 212 tr" cati g........................................................K. type.................................................................92 #iewi g co te ts.............................................21 write access....................................................92 +ilters....................................................................KK +i ! comma !............................................2.2, 2(8 +i !"tils package................................................229 0ire+o3................................................................(K2 +irewalls..............................................................200 +irst-i +irst-o"t...................................................191 +loppy !isks.........................................80, .8/, .9( +low co trol.............................................................. %ra chi g......................................................(82 case compo" ! comma !............................1(2 eli+ stateme t................................................(92 e !less loop..................................................1.K +or compo" ! comma !...............................12. +or loop.........................................................12. +" ctio stateme t........................................(/8 i+ compo" ! comma !.................................(82 loopi g..........................................................1.2 me "-!ri#e .................................................109 m"ltiple-choice !ecisio s.............................1(2 rea!i g +iles with while a ! " til loops.......1./ termi ati g a loop.........................................1.2 traps..............................................................189 " til loop......................................................1.K while loop.....................................................1.( +mt comma !.....................................................(.( +oc"s policy...........................................................2 +ol! comma !....................................................(.( +or compo" ! comma !....................................12. +or loop...............................................................12. 0oresight............................................................./0 0ortra programmi g la g"age..................(12, 121 +ree comma !................................................(, .82 0ree So+tware 0o" !atio ...................................Kp. +sck comma !.....................................................9( +tp comma !...............................201, 2.., (1/, (/1 0T6 ser#ers.................................................201, (/1

9
+alse comma !...................................................(8/ +!+ormat comma !..............................................91 +!isk comma !....................................................89 +g comma !........................................................2. 0907...................................................................191 +ile comma !.......................................................2( +ile !escriptor.......................................................K2 +ile system corr"ptio .........................................8K 0ile Tra s+er 6rotocol E0T6F..............................201 +ile ames............................................................222 case se siti#e...................................................8 em%e!!e! spaces i .................................9, 2K1 e3te sio s........................................................9 hi!!e ..............................................................8 +iles........................................................................... access..............................................................91 archi#i g...............................................2(1, 210 attri%"tes.........................................................92 %lock special...................................................9K %lock special !e#ice.....................................2.K cha gi g +ile mo!e.........................................9/ cha gi g ow er a ! gro"p ow er.................0/ character special.............................................9K character special !e#ice................................2.K compressio ..................................................2(0 co +ig"ratio ..................................22, .28, 2K8 copyi g.....................................................(1, 1. copyi g o#er a etwork................................20( creati g empty................................................K. !e%................................................................./0 !eleti g.............................................(/, 12, 222 !etermi i g co te ts......................................2( !e#ice o!es...................................................2/ e3ec"tio access.............................................92 e3pressio s...................................................(88 +i !i g..........................................................2.( hi!!e ..............................................................8 iso image....................................................92p. listi g.........................................................2, 20 mo!e...............................................................9K mo#i g......................................................(K, 12 ow er..............................................................9/ permissio s.....................................................91 rea! access......................................................92 reg"lar...........................................................2.K

199

9 !e3
0)&C&$>E #aria%le.......................................112 +" ctio stateme t..............................................(/8 hierarchical !irectory str"ct"re.............................1 high-le#el programmi g la g"ages....................(12 history...................................................................... e3pa sio ..................................................89, 9. searchi g.........................................................89 history comma !..................................................89 home !irectories...................................................28 root acco" t....................................................28 -etc-passw!.....................................................92 home !irectory..........................2, .8, /1, .02, .(0 A7>E #aria%le...................................................(0 host ame............................................................K. AT>*........................2K9, (0(, (2(, (K2, (/2, (// Ayperte3t >ark"p *a g"age.............................2K9

=
gcc......................................................................(1K ge!it comma !.............................................9, .(2 ge isoimage comma !........................................92 ;e too................................................................./0 getopts comma !...............................................120 ;hostscript.........................................................((( gi!........................................................................91 glo%al #aria%les..................................................(80 glo%%i g...............................................................(2 ;&7>E............................0, ((, 1/, .0., .(2, 2.2 g ome-termi al.....................................................0 ;&) %i "tils package........................................12( ;&) C Compiler...............................................(1K ;&) core"tils package...............2., 21p., 28(, (0/ ;&) +i !"tils package......................................229 ;&) 6ro4ect..................Kp., 2., 229, (0/, (1K, (18 ;&) 6ro4ect............................................................. i +o comma !.................................................21 ;&)-*i "3........................................................2, / graphical "ser i ter+aces........................................( grep comma !......................................K/, 21/, 10K gro++....................................................................(22 gro"p comma !s................................................182 gro"ps...................................................................91 e++ecti#e gro"p 9'.........................................0( gi!...................................................................91 primary gro"p 9'............................................91 setgi!.............................................................0( ;)9.................................(, .., ((, 1/, 81, .00, .(. g" Dip comma !................................................2(. gDip comma !..............................................22, 2(.

)
9-7 re!irectio Esee re!irectio F...........................29 i! comma !..........................................................91 9'E......................................................................8/ i+ compo" ! comma !.......................((, 12., 1(2 90S #aria%le........................................................10K 9>C6 ECA7O:ET)EST..................................200 i creme tal %ack"ps...........................................2(8 i +o +iles...............................................................21 i it........................................................................( i it scripts.............................................................( i o!es...................................................................1( 9&ST$**...........................................................(19 i stallatio wiDar!.............................................../. i tegers..................................................................... arithmetic................................................/2, 1/2 !i#isio .................................................../K, 1K2 e3pressio s...................................................(92 i teracti#ity........................................................10. 9 ter al 0iel! Separator......................................10K i terprete! la g"ages.........................................(12 i terprete! programs..........................................(1K i terpreter...........................................................(1K iso images.........................................................92p. iso9KK0........................................................81, .9K

H
har! !isks............................................................80 har! li ks..................................................(0, (9, 1( creati g...........................................................1( listi g..............................................................11 hea! comma !.....................................................K8 hea!er +iles.........................................................(19 hello worl! program...........................................(29 help comma !......................................................20 here !oc"me ts..................................................(/( here stri gs.........................................................10/ he3a!ecimal.................................................98, 1K1 hi!!e +iles......................................................8, /1

>
4o% co trol...........................................................20 4o% "m%ers.........................................................20 4o%spec................................................................2. 4oi comma !.....................................................282 Joliet e3te sio s..................................................9K Joy, Jill...............................................................1.

200

9 !e3 3
kate comma !.....................................................(2 M'E..................................0, ((, 1/, .0., .(2, 2.2 ke!it comma !....................................................(2 ker el......2, K, 2/, 22, ..(, .2(, ./8, .8/, 29., (21 key +iel!s............................................................2/2 kill comma !......................................................22 killall comma !..................................................22 killi g te3t............................................................82 M "th, 'o al!....................................................(22 Mo ="eror....................................((, .0., 2.2, (K2 ko sole..................................................................0 kwrite comma !...........................................9, .(2 li#e C's.................................................................2 l comma !....................................................(9, 1( local #aria%les....................................................(80 locale..........................................222, 22/, 29(, (9. locale comma !.................................................22/ localhost.............................................................20/ locate comma !.........................................2.(, 2K2 logical errors......................................................12( logical operatio s...............................................(9K logical operators.................................................2.9 logical relatio ships...................................2.8, 222 logi prompt..................................................(, 202 logi shell.............................................92, .02, .(. lo g optio s..........................................................2. loop%ack i ter+ace..............................................20( loopi g...............................................................1.2 loops...................................12(, 1K2, 1K8, 181, 18/ lossless compressio ..........................................2(. lossy compressio ..............................................2(. lp comma !........................................................((K lp= comma !......................................................(1. lpr comma !......................................................((2 lprm comma !...................................................(12 lpstat comma !..................................................(10 ls comma !.....................................................2, 20 lo g +ormat.....................................................22 #iewi g +ile attri%"tes.....................................92 *"kya o#, $le3a !er.........................................20K *R> E*ogical Rol"me >a agerF................80, .8(

L
*$&; #aria%le....................................(0, 222, 22/ less comma !.................................21, KK, 212, 2K2 l+tp comma !.....................................................20K li%raries..............................................................(12 li e co ti "atio character.........................(02, (K( li e e!itors...........................................................1. li ker..................................................................(12 li ki g................................................................(12 li ks.......................................................................... %roke .............................................................1K creati g...........................................................(9 har!...........................................................(0, (9 sym%olic...................................................(0, 10 *i "3 comm" ity................................................/0 *i "3 !istri%"tio s............................................../0 Ce t7S................................................../., (10 'e%ia ................................................/0p., (11 0e!ora.........................................2, 91, ./., (10 0oresight......................................................../0 ;e too.........................................................../0 *i spire........................................................../. >a !ri#a......................................................./. 7pe S)SE...............................................2, ./. packagi g systems........................................./0 6C*i "37S.................................................../. :e! Aat E terprise *i "3............................../. Slackware....................................................../0 )%" t"...........................................2, ./0p., (10 Ga !ros........................................................./. *i "3 0ilesystem Aierarchy Sta !ar!. . .2K, (0, (K2 *i "3 ker el. .2, K, 2/, 22, ..(, .2(, ./8, .8/, 29., (21 *i "3 ker el............................................................. !e#ice !ri#ers................................................/8 literal characters.................................................219

:
machi e la g"age...............................................(11 mai te a ce...............................(K2, (KK, (K8, (/K make comma !..................................................(2. >ake+ile.............................................................(2. ma comma !......................................................2. ma pages.....................................................2., (2( mark"p la g"ages......................................2K9, (2( memory.................................................................... assig e! to each process.................................1 !isplayi g +ree.................................................( :esi!e t Set SiDe............................................K segme tatio #iolatio ..................................21 "sage...............................................................K #iewi g "sage................................................2K #irt"al..............................................................K me "-!ri#e programs.......................................109 meta key...............................................................8K meta se="e ces...................................................220 metacharacters....................................................220

20.

9 !e3
meta!ata....................................................../., ./( mk!ir comma !.............................................(1, 10 mk+i+o comma !................................................191 mk+s comma !............................................92, .91 mkiso+s comma !...............................................9K mktemp comma !..............................................19. m emo ics.........................................................(12 mo!al e!itor........................................................11 mo ospace! +o ts...............................................((( >oole aar, Jram.................................................1. more comma !.....................................................2K mo" t comma !..........................................82, .9K mo" t poi ts.........................................28, .82, .81 mo" ti g.............................................................8. >6(.....................................................................09 m"lti-"ser systems...............................................9( m"ltiple-choice !ecisio s...................................1(2 m"ltitaski g..........................................9(, ..(, 192 m# comma !..................................................(K, 12 "ll character......................................................222 "m%er %ases......................................................1K1

;
octal......................................................98, 1K1, 1/9 7gg Ror%is..........................................................09 7*'O6W' #aria%le............................................(0 7pe 7++ice.org Writer..................................Kp., 21 7pe SSA............................................................20/ operators................................................................... arithmetic................................................/2, 1K1 assig me t....................................................1KK %i ary............................................................122 compariso ...................................................1K9 ter ary...........................................................1/0 ow i g +iles..........................................................91

P
package +iles......................................................../. package mai tai ers............................................/. package ma ageme t........................................../0 !e%................................................................./0 'e%ia Style E.!e%F......................................../. +i !i g packages............................................/( high-le#el tools............................................../2 i stalli g packages......................................../( low-le#el tools.............................................../2 package repositories....................................../. :e! Aat Style E.rpmF...................................../. remo#i g packages......................................../1 :6>............................................................../0 "p!ati g packages........................................./2 packagi g systems............................................../0 page !escriptio la g"age..................2K9, (21, ((2 6$;E: #aria%le..................................................(0 pagers...................................................................2K parameter e3pa sio ..............................//, 80, 12/ pare t !irectory.....................................................2 pare t process.......................................................( passw! comma !.................................................. passwor!s.............................................................. paste comma !...................................................281 6$T$...................................................................8/ patch comma !..................................................29. patches................................................................289 6$TA #aria%le.............................(0, .((, (K0, (/8 path ame e3pa sio .............................../(, 80, 122 path ames..........................................................2K1 path ames................................................................

1
ame! pipes.......................................................19( a o comma !....................................................10 &a"til"s................................................((, .0., 2.2 etstat comma !................................................202 etworki g..........................................................99 a o ymo"s 0T6 ser#ers...............................201 !e+a"lt ro"te..................................................20( 'y amic Aost Co +ig"ratio 6rotocol E'AC6F ......................................................................20( e crypte! t" els..........................................2.0 e3ami e etwork setti gs a ! statistics.......202 0ile Tra s+er 6rotocol E0T6F........................201 +irewalls........................................................200 0T6 ser#ers...................................................201 *ocal $rea &etwork.....................................20( loop%ack i ter+ace........................................20( ma i the mi!!le attacks.............................20/ ro"ters...........................................................202 sec"re comm" icatio with remote hosts....20/ testi g i+ a host is ali#e.................................200 traci g the ro"te to a host.............................20. tra s+erri g +iles...........................................212 tra sporti g +iles...........................................20( Rirt"al 6ri#ate &etwork................................2.0 ewli e character................................................K. ewli es...............................................................8. &EWS................................................................(19 l comma !........................................................(09 ro++ comma !...................................................(22

202

9 !e3
a%sol"te............................................................K completio ......................................................8K relati#e.............................................................K 6'0............................................................(22, ((2 6erl programmi g la g"age. 18, 21/, (0(, (1K, 1/2 permissio s........................................................(28 6A6 programmi g la g"age..............................(1K pi g comma !....................................................200 pipeli es...............................................K2, 10/, 18/ pipeli es................................................................... i comma ! s"%stit"tio ................................/8 porta%ility...........................................(20, (81, (98 porta%le..............................................................(81 6orta%le 'oc"me t 0ormat........................(22, ((2 6orta%le 7perati g System 9 ter+ace.................229 positio al parameters......................1(/, 128p., 1K. 67S9G......................................9K, 222, 228p., (98 67S9G...................................................................... character classes......(2p., 221p., 22/, 2K., 29(, (0( 6ostScript...........................2K9, (21, ((2, ((/, (12 pr comma !...............................................(./, ((( primary gro"p 9'.................................................91 pri ta%le characters............................................222 pri te # comma !......................................./8, .28 pri ter %"++ers......................................................82 pri ters........................................................82, .8/ %"++eri g o"tp"t.............................................82 co trol co!es................................................((. !aisy wheel...................................................((. !e#ice ames.................................................8/ !ri#ers...........................................................((( graphical.......................................................((2 impact...........................................................((. laser..............................................................((2 pri t+ comma !..........................................(.8, 12K pri ti g..................................................................... !etermi i g system stat"s............................(10 history o+......................................................((0 9 ter et 6ri ti g 6rotocol.............................(1. mo ospace! +o ts.........................................((. prepari g te3t................................................((( pretty.............................................................((/ pri t ="e"es..................................................(10 proportio al +o ts.........................................((2 ="e"e............................................................(1. spooli g........................................................(10 termi ate pri t 4o%s.......................................(12 #iewi g 4o%s.................................................(1. process 9'.............................................................1 process s"%stit"tio ............................................18/ processes..............................................................( %ackgro" !....................................................20 chil!................................................................( co trolli g.......................................................8 +oregro" !.....................................................20 i terr"pti g.....................................................9 4o% co trol.....................................................20 killi g............................................................22 ice.................................................................2 pare t..............................................................( 69'..................................................................1 process 9'.......................................................1 S9;9&T.........................................................189 sig als............................................................22 S9;TE:>....................................................189 sleepi g...........................................................2 state.................................................................2 stoppi g.........................................................2. #iewi g...................................................1, ..K Dom%ie............................................................2 pro!"ctio "se....................................................122 programma%le completio ...................................88 ps comma !.........................................................1 6S. #aria%le................................................(0, .K0 6S2 #aria%le.......................................................(K/ ps2p!+ comma !................................................(22 6S1 #aria%le.......................................................129 pse"!oco!e.................................................(82, 1.2 pstree comma !..................................................2K 6"TT5................................................................2.2 pw! comma !.......................................................2 6W' #aria%le......................................................(. 6ytho programmi g la g"age..........................(1K

?
="oti g................................................................./9 !o"%le ="otes................................................../9 escape character..............................................82 missi g ="ote................................................120 si gle ="otes...................................................8.

R
:$9' E:e!" !a t $rray o+ 9 !epe !e t 'isksF .............................................................................80 raster image processor........................................((( rea! comma !....................102, 1.., 1./, 122, 18K :ea!li e...............................................................81 :E$'>E.....................................................22, (19 re!irectio ................................................................

20(

9 !e3
%locke! pipe.................................................192 gro"p comma !s a ! s"%shells....................182 here !oc"me ts.............................................(/( here stri gs...................................................10/ sta !ar! error..................................................K. sta !ar! i p"t.........................................K(, (/1 sta !ar! o"tp"t...............................................K0 re!irectio operators................................................ ^X...................................................................K( Y......................................................................K2 YElistF............................................................18/ YY..............................................................(/(p. YY-................................................................(/1 YYY...............................................................10/ X......................................................................K0 XElistF............................................................18/ XX...................................................................K. Z.......................................................................K2 reg"lar e3pressio s...............K/, 21/, 299, (9(, 10K reg"lar e3pressio s................................................... a chors.........................................................22. %ack re+ere ces..................................2K/, 298p. %asic...........................228, 2KKp., 29K, (00, (.0 e3te !e!.......................................................228 relatio al !ata%ases............................................282 relati#e path ames................................................K release early, release o+te .................................122 remo#i g !"plicate li es i a +ile.........................KK :E6*5 #aria%le..........................................102, 18/ report ge erator..................................................(K2 repositories........................................................../. ret"r comma !.........................................(/9, (90 re"sa%le..............................................................(81 :96.....................................................................((( rlogi comma !.................................................20/ rm comma !........................................................(/ :ock :i!ge e3te sio s........................................9K ro++......................................................................(22 :7T.( e co!i g................................................291 :6>..................................................................../0 rpm comma !...................................................../( rsy c comma !..................................................212 rsy c remote-"p!ate protocol............................212 :"%y programmi g la g"age.............................(1K scripti g la g"ages.......................................18, (12 s!i++ comma !....................................................(08 searchi g a +ile +or patter s..................................K/ searchi g history..................................................89 Sec"re Shell.......................................................20/ se! comma !.....................................292, (2K, 1K2 set comma !...............................................28, 128 setgi!...................................................................0( set"i!...........................................................0(, (89 Sewar!, J"lia ....................................................2(( s+tp comma !.....................................................2.. share! li%raries.............................................28, ./2 she%a g......................................................(29, (K1 shell %"ilti s.........................................................18 shell +" ctio s..............................18, .28, (/8, 11. shell prompts....0, .K, 90, .02, ..9, .(0, .K0, 208, (K/ shell scripts.........................................................(28 SAE** #aria%le..................................................(0 shell #aria%les......................................................28 shi+t comma !............................................110, 112 S9;9&T..............................................................189 sig als................................................................188 si gle ="otes.........................................................8. Slackware............................................................/0 sleep comma !...................................................1.1 so+t li k................................................................(0 sort comma !...............................................KK, 2/. sort keys.............................................................2/2 so"rce co!e.............................../0p., ./8, 2K9, (11 so"rce comma !.........................................(9, (K. so"rce tree..........................................................(1/ special parameters......................................112, 129 split comma !....................................................(08 SSA....................................................................20/ ssh comma !..............................................20/, 2(9 ssh program..........................................................9( Stallma , :ichar!....................2, Kp., .(2, 229, (1K sta !ar! error..............................................29p., K2 !isposi g o+....................................................K( re!irecti g to a +ile.........................................K. sta !ar! i p"t.......................................29, (/1, 102 sta !ar! i p"t........................................................... re!irecti g.......................................................K( sta !ar! o"tp"t.....................................................29 appe !i g to a +ile..........................................K. !isposi g o+....................................................K( re!irecti g sta !ar! error to...........................K2 re!irecti g to a +ile.........................................K0 start"p +iles..........................................................(. stat comma !.....................................................22/

S
scalar #aria%les...................................................1/K Schilli g, Jorg....................................................9K scp comma !......................................................2.. script comma !....................................................9.

201

9 !e3
sticky %it..............................................................0( storage !e#ices....................................................80 a"!io C's..............................................81, .92 C'-:7>s..........................................8(p., .92 creati g +ile systems......................................89 !e#ice ames.................................................8K !isk partitio s................................................8. 0$T(2............................................................89 +loppy !isks...........................................8/, .9( +ormatti g......................................................89 *R> E*ogical Rol"me >a agerF..................8( mo" t poi ts..........................................82, .81 partitio s........................................................89 rea!i g a ! writi g !irectly..........................91 repairi g +ile systems....................................9( " mo" ti g....................................................82 )SJ +lash !ri#es............................................91 stream e!itor.......................................................292 stri gs....................................................................... e3pressio s...................................................(9. e3tract a portio o+.......................................1K0 le gth o+........................................................1K0 per+orm search a ! replace "po .................1K2 remo#e lea!i g portio o+............................1K. remo#e traili g portio o+............................1K. [cparameter,o++set,le gthd...........................1K0 [cparameter,o++setd......................................1K0 stri gs comma !................................................12( st"%s...........................................................(8., 122 style....................................................................(19 s" comma !........................................................01 s"%shells.....................................................10/, 182 s"!o comma !............................................01, .0K S" >icrosystems...............................................1. s"per"ser.........................................0, 92, .02, .22 sym%olic li ks..........................................(0, 10, 11 creati g.....................................................11, 1/ listi g..............................................................11 symli k.................................................................(0 sy ta3 errors.......................................................1.9 sy ta3 highlighti g.....................................(28, (K( Task >a ager.......................................................8 Tatham, Simo ...................................................2.2 t%l comma !...............................................(22, (2K tee comma !........................................................K9 Teletype................................................................1 tel et comma !..................................................20/ TE:> #aria%le....................................................(. termi al em"lators................................................0 termi al sessio s...................................................... co trolli g termi al........................................1 e++ect o+ .%ashrc............................................(K. e #iro me t...................................................02 e3iti g..............................................................( logi shell..............................................02, .(. TE:> #aria%le..............................................(. "si g ame! pipes........................................191 #irt"al..............................................................( with remote systems.......................................9( termi als..............................8K, 92p., .K(, (22, ((. ter ary operator..................................................1/0 test cases.............................................................12K test comma !.............................(88, (9(, 1.(, 122 test co#erage.......................................................12K testi g.........................................................122, 12/ TEG....................................................................(22 te3t........................................................................21 a!4"sti g li e le gth.....................................(.( $SC99.............................................................21 carriage ret"r ..............................................2/. compari g.....................................................28/ co #erti g >S-'7S to ) i3........................29( co" ti g wor!s...............................................K/ c"tti g...........................................................280 !eleti g !"plicate li es.................................2/9 !eleti g m"ltiple %la k li es........................2/. !etecti g !i++ere ces.....................................288 !isplayi g commo li es..............................288 !isplayi g co trol characters........................2/0 '7S +ormat..................................................2/. E'9T7: #aria%le..........................................(0 e!itors....................................................(1, 2K8 e3pa !i g ta%s..............................................28( +iles.................................................................21 +ilteri g...........................................................KK +ol!i g..........................................................(.( +ormatti g.....................................................(09 +ormatti g +or typesetters.............................(22 +ormatti g ta%les...........................................(2K 4oi i g...........................................................282 li e+ee! character.........................................2/. lowercase to "ppercase co #ersio ..............29(

T
ta%les..................................................................282 ta%"lar !ata.................................................2/2, (2. tail comma !........................................................K8 tape archi#e........................................................2(2 tar comma !.......................................................2(1 tar%alls................................................................(1/ targets.................................................................(2.

202

9 !e3
"m%eri g li es....................................2/., (09 pagi ati g.....................................................(./ pasti g..........................................................281 prepari g +or pri ti g...................................((( remo#i g !"plicate li es................................KK re !eri g i 6ostScript.................................(21 :7T.( e co!e!............................................291 searchi g +or patter s.....................................K/ sorti g.....................................................KK, 2/. spell checki g...............................................(0( s"%stit"ti g...................................................298 s"%stit"ti g ta%s +or spaces...........................28( ta%-!elimite!.................................................282 tra sliterati g characters..............................292 ) i3 +ormat..................................................2/. #iewi g with less......................................21, KK te3t e!itors...........................................(1, 2K8, 292 emacs.............................................................(2 +or writi g shell scripts.................................(28 ge!it.......................................................(2, (28 i teracti#e.....................................................292 kate........................................................(2, (28 ke!it...............................................................(2 kwrite.............................................................(2 li e.................................................................1. a o.......................................................(2, .10 pico................................................................(2 stream...........................................................292 sy ta3 highlighti g...............................(28, (K( #i....................................................................(2 #im.................................................(2, (28, (K( #is"al.............................................................1. til!e e3pa sio .............................................../1, 80 tloa! comma !....................................................2K top comma !........................................................K top-!ow !esig .................................................(/K Tor#al!s, *i "s...................................................2, / to"ch comma !.......................22Kp., 211, (2(, 11/ tr comma !........................................................292 tracero"te comma !...........................................20. traci g................................................................128 tra sliterati g characters....................................292 traps....................................................................189 tro++ comma !....................................................(22 tr"e comma !.....................................................(8/ TT5......................................................................1 type comma !......................................................19 typesetters..................................................(22, ((2 Tf #aria%le..........................................................(.

)%" t"..................................91, .0/, ./0, 221, (K. "mask comma !..........................................0., ..0 "mo" t comma !................................................82 " alias comma !.................................................2/ " ary operator e3pecte!.....................................122 " ary operators...................................................1K1 " e3pa ! comma !...........................................28( " e3pecte! toke ...............................................12. " i= comma !..............................................KK, 2/9 ) i3........................................................................( ) i3 System R...................................................((2 " i32!os comma !............................................2/. " set comma !..................................................182 " til compo" ! comma !..................................1.K " til loop............................................................1.K " Dip comma !..................................................21. "p!ate!% comma !............................................2.2 "pstream pro#i!ers............................................../. "ptime................................................................(// "ptime comma !................................................(8( )SJ +lash !ri#es.........................................80, .91 )SE&ET............................................................291 )SE: #aria%le............................................29, .(. "sers......................................................................... acco" ts..........................................................91 cha gi g i!e tity...........................................01 cha gi g passwor!s......................................... e++ecti#e "ser 9'....................................0(, ..1 home !irectory................................................92 i!e tity............................................................91 passwor!.........................................................92 setti g !e+a"lt permissio s............................0. set"i!.............................................................0( s"per"ser................................92, 9/, .0(p., ..2 -etc-passw!.....................................................92 -etc-sha!ow.....................................................92

@
#ali!ati g i p"t..................................................108 #aria%les...............................................//, (K8, 12/ assig i g #al"es....................................(/., 1KK co sta ts.......................................................(/0 !eclari g...............................................(K8, (/. e #iro me t...................................................28 glo%al............................................................(80 local..............................................................(80 ames....................................................(/0, 1K0 scalar.............................................................1/K shell...............................................................28

20K

9 !e3
#+at......................................................................92 #i comma !.........................................................10 #im comma !.............................................2K/, (K( #irt"al co soles.....................................................( Rirt"al 6ri#ate &etwork.....................................2.0 #irt"al termi als....................................................( #is"al e!itors.......................................................1. #mstat comma !.................................................2K .%ashOlogi ..........................................................(. .%ashOpro+ile........................................................(. .%ashrc.................................(2, .(1, (K., (81, 112 .pro+ile.................................................................(. .ssh-k ow Ohosts...............................................209

*
EE FF compo" ! comma !...........................1K(, 1K9

'
wait comma !....................................................192 wc comma !........................................................K/ we% pages...........................................................2K9 wget comma !...................................................20K What 5o" See 9s What 5o" ;et.........................((. whatis comma !..................................................2( which comma !.............................................19, /8 while compo" ! comma !................................1.( wil!car!s..................................(2, K1, /2, 21/, 221 wo!im comma !.................................................9/ wor!-splitti g................................................../9pp. worl!....................................................................91 W5S9W5;........................................................((.

D
a comma !.........................................................12.

E
-............................................................................2/ -%i .......................................................................2/ -%oot.....................................................................2/ -%oot-gr"%-gr"%.co +............................................2/ -%oot-#mli "D.......................................................2/ -!e#.......................................................................2/ -!e#-c!rom..........................................................8/ -!e#-!#!...............................................................8/ -!e#-+loppy..........................................................8/ -!e#- "ll...............................................................K( -etc........................................................................2/ -etc-%ash.%ashrc...................................................(2 -etc-cro ta%...........................................................2/ -etc-+sta%...............................................2/, .8., .9( -etc-gro"p.............................................................92 -etc-passw!.............................2/, 92, 2/8, 28(, 10K -etc-pro+ile...................................................(., .(( -etc-sha!ow..........................................................92 -etc-s"!oers.........................................................01 -li%........................................................................28 -lostV+o" !...........................................................28 -me!ia...................................................................28 -m t......................................................................28 -opt.......................................................................28 -proc.....................................................................28 -root..............................................................28, .02 -s%i ......................................................................28 -tmp..............................................................28, 19. -"sr........................................................................29 -"sr-%i .................................................................29 -"sr-li%..................................................................29 -"sr-local...............................................................29 -"sr-local-%i ........................................29, (21, (K2 -"sr-local-s%i .....................................................(K2 -"sr-s%i ................................................................29

A
G Wi !ow System...................................., 9(, 2.0 3args comma !..................................................221 3loa! comma !...................................................2K 3logo comma !....................................................9 G>*...................................................................2K9

B
ya ki g te3t..........................................................82 y"m comma !...................................................../(

C
Dgrep comma !..................................................2K/ Dip comma !......................................................210 Dless comma !.....................................................22

--help optio .........................................................2.

.
.-co +ig"re..........................................................(20 .%ashOhistory........................................................89

20/

9 !e3
-"sr-share..............................................................29 -"sr-share-!ict.....................................................22. -"sr-share-!oc.................................................29, 22 -#ar.......................................................................29 -#ar-log.................................................................29 -#ar-log-messages...................................29, K9, .8/ [cparameter,-wor!d...........................................128 [cparameter,Cwor!d...........................................129 [cparameter,Vwor!d..........................................129 [cparameter,Wwor!d..........................................128 [cparameter--patter -stri gd..............................1K2 [cparameter-Ppatter -stri gd.............................1K2 [cparameter-_patter -stri gd............................1K2 [cparameter-patter -stri gd...............................1K2 [cparameterPPpatter d.......................................1K. [cparameterPpatter d.........................................1K. [cparameter__patter d....................................1K. [cparameter_patter d.......................................1K. [N..............................................................112, 120 [U................................................................112, 120 [P........................................................................1(8 [0........................................................................112

F
[L................................................................11(, 19( [EEe3pressio FF...................................................1K( [cLarrayaNbd......................................................18. [cLarrayaUbd........................................................18. [cLpre+i3Nd.......................................................1K0 [cLpre+i3Ud.........................................................1K0 [cPparameterd....................................................1K0

208