Complete Course in Unix

Overview File System Shell Programming Process Management Inter Process Communication System Administration Utilities

Overview

Characteristics of Unix System

Time-sharing Multi-user, multi-process system Written in high level language Hierarchical file system Consistent file formats

Architecture of Unix System

System Kernel Utilities and application software File Structure

Kernel Functions
Schedule Processes Keep track of files Control hardware devices

File System

File System
Characteristics of file system Types of files File system layout File structure related system calls

File System
Characteristics of Unix File System Hierarchical Structure Consistent treatment of file data Ability to create and delete files Dynamic growth of files Treatment of peripheral devices as files

File System
Types of Files Regular files Directories Special files Pipes Named Pipes I/O devices

File System
File System Layout Boot block contains bootstrap code to initialize the operating system Super block describes the state of a file system Inode list is a list of inodes Data blocks contain file and administrative data

10

File structure related system calls

11

OPEN
#include <fcntl.h>

fd = open ( pathname, flags, modes)

pathname is the filename

flags indicate the type of open modes give the file permission if it is being created

12

CREAT
Creates a new file with the indicated filename and access permission modes. If the file already exists, creat truncates the file.

fd = creat ( filename, mode) char *filename; int mode;

13

READ
Read returns the number of bytes it read into the buffer. It is generally count or a value less than count if the number of bytes left to be read are less than the count

number = read ( fd, buffer, count )

fd is the file descriptor buffer is the address of the data structure count is the no of bytes to be read number is the no of bytes read
14

WRITE
Write writes count bytes of data from user address buffer to the file descriptor fd number = write ( fd, buffer, count ) fd is the file descriptor buffer is the address of the data structure count is the no of bytes to be written number is the no of bytes actually written

15

LSEEK
Changes the position of read-write pointer lseek ( fd, offset, reference ) int fd is the file descriptor long offset is the byte offset int reference indicates where the offset is to be considered from reference values 0 1 2 set the pointer to offset bytes from beginning of file increment the current value of pointer by offset set the pointer to size of file plus offste bytes
16

CLOSE
Close closes a file descriptor obtained from a prior open, creat, dup, pipe or fcntl system call

close ( fd ) int fd;

17

STAT
Stat returns status information about the specified file stat ( filename, statbuffer ) char *filename is the absolute file name struct stat *statbuffer is the address of the data structure in the user process which will contain status information

18

FSTAT
Fstat returns the status information of an open file whose file descriptor is fd fstat ( fd, statbuffer ) int fd is the file descriptor struct stat *statbuffer is the address of the data structure in the user process which will contain status information

19

DUP
Dup duplicates the specified file descriptor, returning the lowest available file descriptor newfd = dup ( fd ) int fd is the file descriptor int newfd is the new file descriptor

20

LINK
Link gives another name filename2 to an existing file filename1 link ( filename1, filename2 ) char *filename1 is an existing file char *filename2 is the new name which will also point to the source file name

21

ACCESS
Checks if calling process has r,w or x permission for a file access ( filename, access_mode ) char *filename is the name of the file access_mode 04 02 01 00 read write read existence

Return value 0 -1 access allowed access not allowed

22

CHMOD
Chmod changes the access permissions of the indicated file to the specified mode

chmod(filename, mode)
char *filename Modes

4000
2000 1000

setuid bit
setgid bit sticky bit

0400
0040 0004

read for owner

read for group read for others
23

CHOWN
Chown changes the owner and the group of the indicated file to the specified owner and group IDs
chown(filename, owner, group) char *filename int owner, group name of the file

24

UMASK
Sets the file mode creation mask and return the old value. When creating a file, permissions are turned off if the corresponding bits in the mask are set
umask(mask) int mask

25

IOCTL
Ioctl does device-specific operations on the open device whose file descriptor is fd.
ioctl ( fd, cmd, arg_pointer) int fd; int cmd is the device specific command arg_pointer defines a parameter whose type depends on the command

26

SHELL PROGRAMMING

27

What is a Shell
Standard command and programming language Interprets command lines Runs other programs Reads commands either from the prompt or from a file

28

Shell Types
sh (bourne shell) csh ksh (korn shell) bash (bourne again shell)

29

Input/Output Redirection
Create Files Append to files Use files as input to the shell Truncate files

30

Input/Output Redirection
Redirector Operators < > >> 2>&1 | Open the following file as stdin Open the following file as stdout Append to the following file Merge file descriptor 2 with 1 Pipe stdout into stdin

31

File/Directory Commands
cd ls mkdir rmdir cat ln mv cp rm Change directory List directorys contents Create directory Remove directory Concatenate and write files contents Create links to existing files Move files Copy files Remove files

32

Selection Commands
awk cut diff grep sed head tail wc Pattern scanning and processing language Select columns Compare files Select lines or rows Edit streams of data Select lines from top in a file Select lines from bottom in a file Count words, characters and lines in a file

33

Shell Variables
Special Variables Environment Variables Program Variables

34

Special Variables
$# Number of positional arguments $0 Program/command name invoked $1,$2.. Positional parameters passed to the shell $* Expands to $1 $2... $@ Expands to $1 $2.. $? Return code from the last executed command $$ PID of the current shell

35

Environment Variables
HOME PATH PS1 IFS PWD Pathname of the users login directory Shells search path for commands Primary prompt Internal field separator Current working directory

36

Program Variables
Normal variables using assignment operator e.g. age=12

37

Quotes
Single Quotes Double Quotes No evaluation whatsoever No evaluation expect for $, \ and `

Backslash

Character following backslash is protected from evaluation

38

Expr
Evaluates arguments and returns 0 or 1 Returns numeric value from an expression

E.g. if expr $a = $b or if expr $str = abc a=`expr $a + 1`

39

Foreground/Background Execution
Background processing by placing an & at the end Background processes run effectively wrapped around by nohup loops run in background enclosed in parentheses fg command moves a background job into foreground

40

READ
Main input mechanism for a shell program Gets a line from the standard input and can store the value in a variable Can read a file line by line

41

Control Structures

42

If ..then ..else
If <condition>
then <command>

[elif <condition>
<command> else <command>] fi
43

Case
case word in pattern1) action1 ;;

pattern2|pattern3)

action2 ;;

....................................... .......................................

*)
esac

default action ;;

44

While .. do
while <condition> do

commands
done

45

test
Tests if an expression is true Returns a 0 if true else returns a non-zero value

46

test
Integer operators used with test eq is equal to

ne
lt le

is not equal to
is less than less than or equal to

gt
ge

greater than
greater than or equal to
47

test
Syntax test <expression>

or

[ <expression> ]

alternative syntax

48

test
File Operators -d -f file exists and is a directory file exists and is an ordinary file

-r
-s -w

file exists and is readable

file exists and has a non-zero size file exists and is writable

-x

file exists and is executable

49

Filename Wildcards
*
?

Match 0 or more characters

Match exactly one character

[1,2,3] Match any character in the square brackets

[a-z]

Match any characters between a to z

[!0-9] Match any character not in the square brackets +(abc) Match one or more instance of abc *(abc) Match zero or more instances of abc

50

Functions
function identifier {list;} or

identifier() {list;}

51

Command Execution
Grave accents or backquotes `command`

Executing in a separate environment

(command) Simple execution

{command}

52

Disk Usage Commands

df du Gives the disk usage statistics of all the filesystems Summarizes disk usage

53

Setuid/Setgid Permissions
This powerful feature enables the effective userid/groupid of the calling process to be the same as the userid/groupid of the owner of the file.

Used most frequently when a script needs to be executed as root.

54

Setuid/Setgid Permissions
The setuid bit is turned on like this chmod 4000 filename

and the permissions then look like this

-rws-----For setgid

chmod 2000 filename

----rws--55

Sticky Bit
This is set to prevent files from being removed from a directory by arbitrary users. chmod 1000 directoryname

and the permissions look like this drwxr-xr-t

In this case, only the owner of the directory, owner of the file or the superuser can remove a file in the directory

56

Find
Finds files that match a given set of parameters find -name filename Finds files with the filename

find -perm mode

find -type c find -user name

Find files with given permissions

File types Belonging to given user

find -size n
find -atime n

Of size n
Access time n days ago
57

Find
Operators used with find
-print find -name filename -print Prints the filename

-exec
find -name filename -exec rm -f {} \; Executes command

58

File
Tells about the contents of the file
file filename

59

Copy Files
cp command
example cp file1 file2

60

Copy Directories
cp -r|R command
Recursively copies the files under a directory and its subdirectories

tar command Use tar cvf to create a tar of the subdirectories and files under a given directory e.g. tar cvf tarname path of files/subdirectories

61

Copy Directories
Untar a tar file using
tar xvf tarname untar location

62

Mail
Used for sending mails
e.g. cat msg | mail -s subject recipient

Note that the mail program expects its input from the stdin

63

Cut
Selects columns or fields
-c -f for col for field

-d

for delimiter

64

Sort
Sort divides each lines into fields at whitespace (blanks or tabs) and sorts the lines by fields from left to right

sort +n -m sorts the nth field, stops sorting at the mth field

65

Process Status
Produces a report summarizing execution statistics for the current processes.
E.g. ps -ef gives general stats ps -fu username gives stats about all the processes running under the user username

66

Time
Used to time a command

e.g.

time progname

67

Uptime
Used to get a rough estimate of the system load
e.g. Uptime

The result looks like this

12:40pm up 4 day(s), 23:59, 1 user, load average: 0.00, 0.00, 0.01

68

Cron
Does periodic execution of jobs

Listing cron entries

crontab -l
Editing cron entries crontab -e e.g 20 10 * * * /$HOME/myscript

69

Process Related System Calls

70

EXEC
Execve executes the program file filename, overlaying the address space of the executing process.Argv is an array of character string parameters to the execed program and envp is an array of character strings that are the environment of the new process
execve(filename, argv, envp) char *filename char *argv[] char *envp[]

71

FORK
Fork creates a new process. The child process is a logical copy of the parent process, except that the parents return value from the fork is the process ID of the child, and the childs return value is 0

72

WAIT
Wait causes the process to sleep until it discovers a child process that had exited or a process asleep in trace mode. If wait_stat is not 0, it points to an address that contains status information on return from the call. wait(wait_stat) int *wait_stat

73

EXIT
Exit causes the calling process to terminate, reporting the 8 low-order bits of status to its waiting parent. The kernel may call exit internally, in response to certain signals exit(status) int status

74

Some other System Calls

getuid() geteuid() getgid() getegid() getpid() getppid() Returns the real user ID Returns the effective user ID Returns the real group ID Returns the effective group ID Returns the process ID Returns the parent process ID

75

SIGNAL
Software Interrupts

76

Classification of Signals
Termination of a process Process induced exceptions Unrecoverable conditions during system calls Unexpected error condition during a system call Tracing execution of a process

77

Handling Signals
Exit on receipt of a signal Ignore the signal Execute a particular function on receipt of a signal

78

Signal Syntax
#include <signal.h> signal(sig,function) int sig; void (*func)() Signal allows the calling process to control signal processing.

79

Signal Syntax
Some values of sig

SIGHUP
SIGINT SIGQUIT SIGKILL SIGFPE SIGBUS SIGPIPE SIGTERM

hangup
interrupt quit kill floating point exception bus error write on a pipe with no reader software termination
80

Signal Syntax
The function is interpreted as SIG_DFL default operation. Process terminates except for some signals ignore the signal.kernel calls a function in the process with signal number as the argument. cant be ignored

SIG_IGN

SIGKILL

81

Kill
Kill sends the signal sig to the process identified by pid

kill(pid,sig)
int pid,sig pid > 0 send signal to process whose PID is pid pid 0 send signal to processes whose process group ID is pid of sender pid -1 if effective UID of sender is of superuser send signal to all processes otherwise to processes whose real UID equals effective UID of the sender
82

Inter Process Communication

83

Types of IPC
Pipes Named Pipes Messages Shared Memory Semaphores Sockets

84

Pipes
Allow data transfer in a FIFO manner Allow synchronization of process execution Use file system for data storage Can be created using pipe() system call Pipe manipulation is done using regular system calls for files e.g. read(), write() Only related processes can access a pipe

85

Pipes
Syntax pipe (file_descriptors) where file_descriptors is a pointer to an array of 2 integers which are the file descriptors for reading and writing the pipe

86

Named Pipes
Similar to pipes Have directory entries open system call used to open named pipes mknod system call to create named pipes Unrelated processes can also communicate using named pipes

87

MESSAGES
Allow processes to send data streams to arbitrary processes Four system calls cover all the operations msgget msgsend msgrec msgctl

88

msgget

Returns the message queue identifier assocaited with the key msgget(key, flag)

key is the name of the message queue

flag is the modes which can be passed

89

msgsnd
msgsnd (id, message, size, flag) Sends message to the queue associated with the message queue identifier specified by id message is of the form struct msgbuf

{
long mtype; char mtext[]; } Flags can be IPC_NOWAIT- Return -1 if the message cant be stored MSG_NOERROR- Message bigger than size is shortened with no error returned
90

msgrcv
msgsnd (id, message, size, type, flag)

Reads message from the queue associated with the message queue identifier specified by id
message is of the form

struct msgbuf
{ long mtype; char mtext[]; }
91

msgctl
Msgctl ( id, command, buffer )

Provides a variety of message control functions as specified by command

Commands can be

IPC_STAT
IPC_SET IPC_RMID

store information about queue in the buffer

Change the information in the buffer remove the message queue identifier specified by id

92

SHARED MEMORY
Allows processes to share part of their virtual address space System calls similar to the ones in messages

93

shmget
int shmget ( key, size, flag )

key is the name of the shared memory size is the size of shared memory segment in bytes flag is a combination of IPC_CREAT IPC_EXCL mode

94

shmat
Inserts a shared memory segment into a process address space char *shmat ( id, at_address, flag ) id is the shared memory segment identifier size is the address in process address space where the shared memory segment has to be attached flag is a combination of SHM_RDONLY read only 0 read,write

95

shmdt
Removes a shared memory segment into a process address space

int shmdt ( dt_address )

dt_address is the address in the process address space of a shared memory segment to be detached

96

shmctl
shmctl ( id, cmd, buf)

Provides shared memory control operations as specified by the cmd

cmd IPC_STAT IPC_SET IPC_RMID SHM_LOCK SHM_UNLOCK

97

SEMAPHORE
Non-negative integer count used to coordinate access to the resources Initial semaphore count set to the number of free resources Count decremented or incremented by the threads or processes as and when they acquire or free resources. Threads block at count zero till it becomes positive

98

semget
semget ( key, num, flag )
Returns or creates the semaphore identifier associated with the key key is the name of the semaphore set number defines the number of semaphores in the set flag is a combination of

IPC_CREAT
IPC_EXCL
99

semop
semop ( id, operations, number )

Performs operations on the semaphore

id is the semaphore identifier operations is the array of semaphore operation structure operation consists of - semaphore number

-operation
-flags number is the number of entries in operations
100

semctl
semctl ( id, semnum, cmd, ... ) Performs operations on the semaphore as specified by the cmd id is the semaphore identifier Fourth argument is optional, depending upon the operation requested. If required, then its a buffer. cmd GETVAL, SETVAL, GETPID, GETNCNT, GETZCNT

101

SOCKET
Creates an endpoint for communication and returns a descriptor int socket ( domain, type, protocol ) domain specifies communication domain type type of communication over socket (virtual circuit or datagram) protocol protocol to control the communication

102

Other Socket System Calls

103

Close

Closes the communication endpoint close ( sd )

sd is the socket descriptor

104

Bind

Assigns a name to an unnamed socket int bind ( sd, sockname, namelength)

sd is the socket descriptor

sockname is the name of the socket namelength is the length of the name

105

Connect

Makes a connection to an existing socket int connect ( sd, sockaddr, namelength)

sd is the socket descriptor

sockaddr is the name of the target socket

106

Accept

Receives incoming requests for a connection accept ( fd, addr, addrlen )

addr user data array which kernel fills with the address of the connecting client

107

send

Transmits data over a connected socket send ( sd, msg, length, flag)

msg pointer to data being sent

length length of the data

108

recv

Receives data over a connected socket recv ( sd, buf, length, flag)

buf pointer to data being sent

length length of the data

109

shutdown

Closes a socket connection shutdown ( sd, mode )

mode specifies which side no longer permits data transmission

110

System Administration
File System Administration Machine Management System Tuning and Performance Management System Accounting User Management Backup/Restore Activities Setting up Peripheral Devices Routine Maintenance

111

File System Administration

Creation and Maintenance of File Systems File System consistency checks Monitoring of disk space Routine Maintenance such as removing inactive files

112

File System Administration Commands

TASK Check File System Display disk usage List files by size Identify FileSystem type Create File System Mount File System Unmount File System SHELL COMMAND fsck df du fstyp mkfs mount umount

113

Machine Management Commands

TASK Set boot defaults Change to firmware Power Off Reboot Display Configuration Display System SHELL COMMAND fltboot shutdown shutdown shutdown prtconf uname

114

System States
System State 0 s 2 3 5 6 q Description Powerdown State Single user Multiuser Start Remote File Share Switch to firmware Halt & Reboot OS Reexamine (/etc/inittab)

115

Changing System States

Switching from multiuser to single user

SHUTDOWN -I S
INIT S Switching to any state INIT <System State> Switching to multiuser

INIT 2
Turning system off SHUTDOWN -I 0
116

User and Group Management

Assigning and Maintaining logins and passwords Organization of system resources to suit particular needs of the users Communicating with users

117

User Management Commands

Task Add User Add Group Change password Listing users and groups Modify user attributes Modify group attributes Remove user Remove group Command useradd groupadd passwd logins, listusers usermod groupmod userdel groupdel
118

Communicating with Users

Keeping users informed about system status and servicing details Using Wall to send messages to the users Using utilities like mail Message of the day facility using /etc/motd

119

SYSTEM UTILITIES

120

MAKE

Provides a method for maintaining an up-to-date version of programs that consist of a number of files generated in a variety of ways

121

MAKE
Operations

Find the target in the description file Ensure that all the dependencies (files) of the target exist and are up to date Create the target in case any of the dependencies have been modified

122

MAKE
MakeFile

Comments

All characters after a #

Continuation lines
Macro Definition

Lines ending in \
Identifier followed by = sign

123

MAKE
Example MakeFile makedemo #makedemo prog : x.o y.o z.o cc x.o y.o z.o -o prog x.o : x.c defs.h cc x.c y.o : y.c defs.h cc y.c z.o : z.c defs.h cc z.c
124

MAKE
Macros Macro Definition name = value

Example OBJECTS = x.o y.o z.o

prog : $(OBJECTS) cc $(OBJECTS) -o prog

125

SYSTEM UTILITIES

126

Grep
Searches the named files or the standard input and prints each line that contains an instance of the pattern

127

Grep
Usage grep [options] pattern filenames

Options -n -v prints the line numbers inverts the sense of the test

-i

case insensitive search

128

Grep
Example
Locate variable in C source code grep variable *.[ch] See if abc is logged in

who | grep abc

Filenames that dont have temp in their names ls | grep -v temp List the subdirectories in the current directory ls -ltr | grep ^d

129

AWK
Scans a set of input lines, searching for lines that match any set of patterns and take a specified action for each pattern on each line that matches the pattern

130

AWK
Usage

awk pattern-action statements optional list of input files e.g. awk {print $1, $2} file1 file2 OR awk -f programfile optional list of input files where programfile is the file where the pattern-action statements are stored

131

AWK
Fields

Record is the sequence of characters separated by a newline character Fields in a record are separated by sequence of blanks or tabs

$0 refers to the entire record $1 refers to the first field $2 refers to the second field

132

AWK
Printing

{ print } prints all the records

{print $1 $3} prints the first and third fields of each of the input line

133

AWK
BuiltIn Variables

ARGC
ARGV

number of command line arguments

array of command line arguments

FILENAME name of current input file

FNR
NR NF OFS ORS

record number in current file

number of current record number of fields in the record Output field separator Output record separator
134

AWK
Examples

Print last field of each input line

{ print $NF } Print total number of input lines

END {print NR}

Print input lines with more than 4 fields NF > 4

Print the total no of lines that match string abc

/abc/ {nlines++} END {print nlines}
135

AWK
BuiltIn Arithmetic Functions

cos(x) log(x) exp(x) rand(x) sin(x)

sqrt(x)
srand(x)
136

AWK
BuiltIn String Functions

index(s,t) length(s) split(s,a) sub(s,r) substr(s,p)

returns position of t in s returns length of s splits into array a on FS substitutes s for first r returns suffix of string starting at position p

137

AWK
Control Flow Statements if (expression) stat1 else stat2 while (expression) stat for (expr1;expr2;expr3) stat

do stat while (expr)

break continue next exit(expr) return(expr)
138

SED
Stream Editor

Reads one line at a time from input files Applies the commands from a list of commands one at a time, in order, to each line

writes to standard output

139

SED
Usage

sed list of ed commands filenames... Example sed s/UNIX/UNIX(TM)/g filenames replaces UNIX by UNIX(TM) in all the file occurences and writes the result to standard output

140

SED
Commands

a\ text b label c\ text d p r w q

Appends text on the output before reading the next input line Branch to the : command bearing the label change lines to the following text delete line; read next input line print read write quit
141