CS2028 - Unix Internals Chapter 1


General Overview of the Unix History System structure User Perspective Operating System Services Assumptions about hardware

What is Unix?

What is an operating system?
The low­level (first layer loaded) software which 

handles the interface to system hardware (input/output  devices, memory, file system, etc), schedules tasks,  and provides common core services such as a basic  user interface.
all of the software that comes with a system before 

applications are installed.


 LynxOS. Others:  AOS. FreeRTOS. etc. VxWorks. eCos. etc. QNX. UnixWare. Minix. Triance. RTLinux. Chorus OS. Darwin (Mac OS X). Embedded:  BeOS.  Net. Solaris. Linux. Micro.  Windows CE.  IOS. Windows  4 . JavaOS. AIX. Hurd. LynxOS.  IRIX. Open. etc). QNX. RTAI. VSTa. MorphOS.What Is Unix? Examples of Operating Systems: Unix and Unix­like:  A/UX.  System V. Symbian.  z/OS. HP­UX. 3.1/95/98/NT/XP/2000/2003. SE­Linux. TRUSIX. *BSD (Free. Primos. VRTX. etc. Tru64. JUNOS.

5   .). etc.A portable. HP PA-RISC. SPARC. Alpha. interrupts. IBM S/390. MIPS. Preemptive multi-tasking: several programs can run at the same time (time slices. Multi-user: many users can share the computer system at the same time. Motorola 680x0. multi-tasking and multi-user operating system  What is Unix? Portable: runs on many different hardware architectures (Intel x86 and IA-64. PowerPC. and task switching).

the Unix shell. Includes support for regular expressions which are convenient for complex searching.What is Unix? Other Features     Uses a simple. which is also a convenient job programming language. uniform file model which includes devices and access to other services in a flexible. 6 . Written in a high-level language (“ C” ) making it easy to read. understand. hierarchical file system. change and port. The command prompt is a simple user process.

SGI. Linux is the fastest growing operating system in the market. it is easy to collect critical information. Source code availability provides the ability to “ get under the hood” of operating system design and function (and TCP/IP).). By understanding the Unix/Linux community and its culture. HP.    7 .Why Do I Need to Know This?  Ubiquitous: Most big computers and much of the Internet infrastructure runs on some variant of Unix (SUN. etc.


           

Multics: 1965 Bell with General Electronic Company and Massachusetts Institute of Technology.GE645. Multiuser, Large data storage & power, data sharing. UNIX: 1969 Thompson and Ritchie at AT&T Bell Labs – Unix File system – Thompson (Space travel program) – PDP 7 UNIX moved to PDP – 11 in 1971 UNIX in C (Ritchie) 1977 – UNIX ported into non PDP machine , Interdata 8/32 BSD: 1978 Berkeley Software Distribution- UNIX system III – V(1983) Commercial Vendors: Sun, HP, IBM, SGI, DEC GNU: 1984 Richard Stallman, FSF POSIX: 1986 IEEE Portable Operating System unIX Minix: 1987 Andy Tannenbaum SVR4: 1989 AT&T and Sun Linux: 1991 Linus Torvalds Intel 386 (i386) Open Source: GPL, LGPL, Cathedral and the Bazaar

Unix History

1965 joint project between AT&T Bell Labs, GE(General Electric company), and MIT(Massachusetts Institute of Technology) to develop a new OS. Goal : develop an OS that could provide computational power, data storage and the ability to share data among multiple users. Result: Multiplexed Information & Computer Service - MULTICS.


Unix History

1969 Bell Labs withdraws from group. Two Bell Lab scientists, Ken Thompson and Dennis Ritchie, continue research. They were still left without a “ Convenient interactive computing service” *.

* Ritchie, D.M. “ The Evolution of the Unix Time-sharing System” , AT&T Bell Laboratories Technical Journal, Oct. 1984, Vol 63, No.8, Part 2, pp. 1577-1594.


 11 . He was told to get the game off his work computer. The spaceship was hard to control and it was expensive to run.Unix History  At the same time Ken Thompson wrote a game “ space travel” in Fortran to run on GECOS OS (Honeywell 635).

Unics (later Unix) was born as a pun on Multics. Unix History Thompson ported the game to a little used PDP-7 computer.  12 .

Unix History PDP-7 13 .

 14 . Then wrote “ C” a compiled language.Unix History  Dennis Ritchie developed “ B” . In 1973 entire OS ported to “ C” .

 SGI. IBM. Sun. HP. Introduced many new  features. ➲ Late 70s : Thompson took a sabbatical to  teach Unix at UC Berkley –  ➲         Birth of BSD Unix. ➲ Commercialization of Unix (70s / 80s)     AT&T. 15 . DEC.Unix History Because of a 1956 Consent Decree AT&T  could not market Unix so it provided it to  academia. ➲ AT&T Bell Labs realized the commercial  potential and began distributing System V. NCR.

16 . 1994 Linux 1.Linus Torvalds   1991 Linux 0.0 is released.02 is first released to the public.

Three Definitions of Linux Kernel: The very low-level software that manages your computer hardware and provides a library (POSIX) interface for user-level software. etc. Motorola 680x0. HP PARISC. the GNU/Linux OS and lots of other software to make Linux easy to install. MIPS.). IBM S/390. The Linux kernel runs on many platforms (Intel x86 and IA-64. PowerPC. Linux GNU/Linux Linux . and use (at least for the target audience). OS: The Linux kernel plus utility software to provide a useful working environment. configure. SPARC. Alpha. Distributions: The packaging of the Linux Kernel.

the Linux Mascot .Tux.

. mail. System divided into two parts First parts-programs and services such as shell. Now Unix is maintained by Open Group.What is UNIX?       Unix is a Operating System that popular since 1969. text processing packages and source code control systems Second parts-system calls and hardware for supports to first parts Example is UNIX system V produced by AT & T.

Multi user. pascal. fortran. basic. multi process system Hides machine architecture – portable Even though Unix is in ‘ C’ we can run other languages like cobol. prolog. ada. . java with their Compiler or interpreter.Unix Features          High level language ‘ C’ Simple user interface Complex programs built from simpler programs Hierarchical file system Consistent format for files-byte streams Consistent interface to peripheral devices. lisp.

large memory. stable. applications  other features:      . large files advanced networking. supports most device hardware active development community. support. paging) shared libraries demand loading. documentation. open source GUIs. advanced filesystems efficient.Linux Features    “ UNIX-like” operating system “ aims at” standards compliance “ all the features you would expect in a modern UNIX”       preemptive multitasking virtual memory (protected memory. dynamic kernel modules shared copy-on-write executables TCP/IP networking SMP support. highly portable.

Architecture of Unix System emacs sh cpp cc as ld who date kernel ed wc nroff grep • OS interacts directly with    the hardware  • Such OS is called    system kernel hardware Other apps .

 Network • Kernel: the memory resident portion of Unix system • File system and process control system are two major   components of Unix Kernel. Memory.Unix: Structure • Operating System: a system that manages the resources    of a computer. I/O devices. . • Resources: CPUs.

Unix System Kernel • Three major tasks of kernel:  Process Management  Device Management  File Management • Three additional Services for Kernel: B Virtual Memory B Networking B Network File Systems • Experimental Kernel Features: D Multiprocessor support D Lightweight process (thread) support .

Block Diagram of System Kernel User Level Kernel   Level User Programs Libraries System Call Interface Process control Inter­process communication Scheduler Memory management File Subsystem Device drivers subsystem hardware control Hardware Level hardware  .

Process Control Subsystem • Process Synchronization • Interprocess communication • Memory management:  • Scheduler: process scheduling  (allocate CPU to Processes) .

  .File subsystem • A file system is a collection of files and directories on   a disk or tape in standard UNIX file system format.  • Kernel’s file sybsystem regulates data flow between   the kernel and secondary storage devices.

• Devices such as disks or terminals may interrupt the     CPU while a process is executing.Hardware Control • Hardware control is responsible for handling interrupts    and for communicating with the machine. . • The kernel may resume execution of the interrupted     process after servicing the interrupt.

 chown chmod. link. mount. close     inode: chdir.      exec: invoke a new program. read. unlink  .System Call • A process accesses system resources through system call. write. unmount. • System call for a Process Control:       fork: create a new process      wait: allow a parent process to synchronize its  execution with the exit of a child process.      exit: terminate process execution a File system:     File: open. lseek. stat fstat     others: pipe dup.

User Perspective-File system .

Architecture of Unix Unix file system root (/) bin dev home tmp spool (users’ accounts) mail (users’ mail) 31 usr etc var log bin X11R6 .

  ➲ home: contains the home directories of users  (/export/home on sun computers). holds the files necessary  to operate peripherals such as printers and  terminals. 32 .Typical Directories ➲ ➲  / :  Root of the tree. Where it starts. sbin. usr/bin: software for the shells and  most common Unix commands. bin.  ➲ dev: short for devices.

(Mail directories. logs.Typical Directories tmp: holds temporary files.) etc: administrative files such as lists of user names and passwords. printer spool files. 33 . var: contains files that vary in size. etc.

  34 .Typical Directories  usr: Contains application programs lib: Contains libraries for programs proc: a pseudo-filesystem used as an interface to kernel data structures.

Types      Ordinary files Directories Special files Pipes .Unix File System File system is organised as a heirarchy of directories  It starts from a single directory called root(represented by a /).

a filename(14 bytes) .an inode number (2 bytes) which acts as a pointer to where the system can find info about the file.Directory   Directory is file containing list of files and subdirectories It has fixed size records of 16 bytes each which contains . .

ex: command1 | command2 | command3. Pipes    UNIX allows us to link commands together using a pipe.  They are used to represent a real physical device such as a printer.  ./dev/null(unwanted output can be redirected).  Ex: Special device .. tape device etc..Special Files Special files are contained in the directory /dev. The pipe acts as a temporary file which only exists to hold data from one command until its read by another..

• For example: create two active processes $ emacs & $ emacs &  $ ps PID   TTY   TIME CMD 12893  pts/4 0:00 tcsh 12581  pts/4 0:01 emacs 12582  pts/4 0:01 emacs $ .Processes • A program is an executable file. • A process is an instance of the program in execution.

• For example: create two active processes $ emacs & $ emacs &  $ ps PID   TTY   TIME CMD 12893  pts/4 0:00 tcsh 12581  pts/4 0:01 emacs 12582  pts/4 0:01 emacs $ .Processes • A program is an executable file. • A process is an instance of the program in execution.

Processes • A process has   text: machine instructions   (may be shared by other processes)  data  stack • Process may execute either in user mode and in kernel    mode. • Process information are stored in two places: p Process table  p User table .

System call: fork() • fork: the only way for a user to create a process in Unix    operating system. • The process that invokes fork is called parent process    and the newly created process is called child process. newpid = 0. newpid = child process id • In child process. . the two processes have     identical copies of their user­level context except for the    return value pid.  • In parent process. • The syntax of fork system call: newpid = fork(). • On return from fork system call.

     printf("Before forking .    } $ cc forkEx1./* forkEx1.. Child Process fpid=0 After forking fpid=0 Parent Process fpid=14707 After forking fpid=14707 $ ..h> main() {    int fpid..    } else {       printf("Parent Process fpid=%d\n". fpid). fpid). fpid).    if (fpid == 0) {       printf("Child Process fpid=%d\n"..c */ #include <stdio.\n").c ­o forkEx1 $ forkEx1 Before forking .    fpid = fork().     }    printf("After forking fpid=%d\n".

 fpid). fpid).    if (fpid == 0) {       printf("Child Process fpid=%d\n".c */ #include <stdio... Child Process fpid=0 After forking fpid=0 Parent Process fpid=14707 After forking fpid=14707 $ ..\n").c ­o forkEx1 $ forkEx1 Before forking . fpid)..    } else {       printf("Parent Process fpid=%d\n".    fpid = fork().    } $ cc forkEx1.h> main() {    int fpid.     printf("Before forking ./* forkEx1.     }    printf("After forking fpid=%d\n".

\n")..    PID TTY      TIME CMD  14759 pts/9    0:00 tcsh  14778 pts/9    0:00 sh  14777 pts/9    0:00 forkEx2    PID TTY      TIME CMD  14781 pts/9    0:00 sh  14759 pts/9    0:00 tcsh  14782 pts/9    0:00 sh  14780 pts/9    0:00 forkEx2  14777 pts/9    0:00 forkEx2 After forking fpid=14780 $    PID TTY      TIME CMD  14781 pts/9    0:00 sh  14759 pts/9    0:00 tcsh  14780 pts/9    0:00 forkEx2 After forking fpid=0  .    system("ps").    system("ps").h> main() {    int fpid.     printf("Before forking ./* forkEx2..    fpid = fork(). fpid)... } $  ps    PID TTY      TIME CMD  14759 pts/9    0:00 tcsh $ $ forkEx2 Before forking .    printf("After forking  fpid=%d\n".c */ #include <stdio.

• Each process has a parent process and a corresponding     parent process ID. • Processes can obtain their parent’s PID by calling     getppid(). typically in the range 0 through 30000.System Call: getpid() getppid() • Each process has a unique process id (PID).  • PID is an integer. . • Kernel assigns the PID when a new process is created. • Processes can obtain their PID by calling getpid().

c */ #include <stdio.h> #include <sys/types. getppid())./* pid.h> #include <unistd. } $ cc pid.getpid().h> main() {    printf("pid=%d ppid=%d\n".c ­o pid $ pid pid=14935 ppid=14759 $ .

   }    printf("After forking fpid=%d pid=%d ppid=%d\n".h> #include <unistd.c */ #include <stdio.h> main() {    int fpid.         fpid.    } else {       printf("Parent Process fpid=%d pid=%d ppid=%d\n".         fpid.\n")./* forkEx3.h> #include <sys/types. getpid().. } .    if (fpid == 0) {       printf("Child Process fpid=%d pid=%d ppid=%d\n"..     printf("Before forking . getppid()).         fpid. getppid()).    fpid = fork(). getpid(). getpid(). getppid()).

..$ cc forkEx3. Parent Process fpid=14942 pid=14941 ppid=14759 After forking fpid=14942 pid=14941 ppid=14759 $ Child Process fpid=0 pid=14942 ppid=1 After forking fpid=0 pid=14942 ppid=1 $ ps     PID TTY      TIME CMD  14759 pts/9    0:00 tcsh .c ­o forkEx3 $ forkEx3 Before forking .

System Call: wait() • wait system call allows a parent process to wait    for the demise of a child process.  .

h> main() {    int fpid.\n").h> #include <sys/types.    printf("After forking fpid=%d pid=%d ppid=%d\n". getppid()).  fpid. getpid(). getpid(). getppid()). } .    fpid = fork().    }    wait(&status).h> #include <unistd. getpid(). getppid()).#include <stdio.     printf("Before forking . status.  fpid.    } else {       printf("Parent Process fpid=%d pid=%d ppid=%d\n".  fpid.    if (fpid == 0) {       printf("Child Process fpid=%d pid=%d ppid=%d\n"...

c ­o forkEx4 $ forkEx4 Before forking .. Parent Process fpid=14980 pid=14979 ppid=14759 Child Process fpid=0 pid=14980 ppid=14979 After forking fpid=0 pid=14980 ppid=14979 After forking fpid=14980 pid=14979 ppid=14759 $  ..$ cc forkEx4.

. execle. execve.    (see man page for more detail. execvp.    Thus. execv.) • exec system call allows a process to choose its successor.System Call: exec() • exec() system call invokes another program by replacing   the current process • No new process table entry is created for exec() program. • Six different exec functions:    execlp. the total number of processes in the system isn’t     changed. execl.

\n"). Sun May  9 16:39:17 CST 1999 $ .../* execEx1. 0)..h> main() {    printf("Before execing ..    printf("After exec\n"). "date".    execl("/bin/date".    } $ execEx1 Before execing .c */ #include <stdio.h> #include <unistd.

.. "date"../* execEx1. 0). Sun May  9 16:39:17 CST 1999 $ .c */ #include <stdio.    } $ execEx1 Before execing .h> main() {    printf("Before execing .    execl("/bin/date".    printf("After exec\n")..\n").h> #include <unistd.

..h> #include <unistd.    } ./* execEx2. After exec and fpid=14903 main() $ Sun May  9 16:47:08 CST 1999 { $    int fpid.    }    printf("After exec and fpid=%d\n"..fpid).\n").h>  #include <stdio. "date".c */ #include <sys/types. 0).    fpid = fork().h> $ execEx2 Before execing ..    printf("Before execing .    if (fpid == 0) {      execl("/bin/date".

 0).    }    printf("After exec and fpid=%d\n".    fpid = fork().h> $ execEx2 Before execing ..h>  #include <stdio.    printf("Before execing .\n").    if (fpid == 0) {      execl("/bin/date".fpid). "date".h> #include <unistd..c */ #include <sys/types.    } ./* execEx2.. After exec and fpid=14903 main() $ Sun May  9 16:47:08 CST 1999 { $    int fpid..

 Example: grep ssn file1. 2>(err)  (2)Pipe: mechanisms allows a stream of data to be passed between reader and writer processes.Building Block Primitives    User to create complex program effectively.c | wc -l .  User much access three files  Std input file  Std output file  Std error file  Example: ls.txt.c file3.c file2. ls > ssn.c file4. mail x < letter. Users capability to (1)redirect I/O.

>> file . FD 1 – STDOUT. << token .means open a file for writing and truncate it and associate it with STDOUT.Means use the current input stream as STDIN for the program until token is seen. FD 0 – STDIN .means open a file for reading and associate with STDIN. Eg.means open a file for writing and seek to the end and associate it with STDOUT. This is how we append to a file using a redirect. > file . FD 2 -STDERR . 2>&1 means send STDERR to the same place that STDOUT is going to.      < file . n>&m means redirect FD n to the same places as FD m.

Operating System Services    Controlling Scheduling Memory Management   Allocating to main memory Allocating to secondary memory   Process management – controlled access to devices Transparent Service – Kernel hides the type of file from user. .

ASSUMPTIONS ABOUT H/W:User mode and Kernel mode • At any given instant a computer running the Unix system    is either executing a process or the kernel itself is running • The computer is in user mode when it is executing     instructions in a user process and it is in kernel mode    when it is executing instructions in the kernel. • Executing System call ==> User mode to Kernel mode    perform I/O operations    system clock interrupt     .

 Example: virtual address space access only in kernal mode. Example: the process status register is not capable to access from the user mode. .   Kernel runs on behalf of the user processes.Difference between two modes  Process in user mode can access their own instruction not kernal instructions and data of other processes but in kernal mode can access both.  Some machine instructions are privileged. Part of each user process.

Restart the instruction after handling exception.Interrupts and Exceptions  Interrupts: for multitasking CPU is shared by many processes  Blocks low priority interrupts but services high priority interrupts.  Exceptions in middle of an instruction whereas interrupts in between 2 instructions. executing privileged instructions and dividing by zero.   Example: addressing illegal memory.  Exception refers to unexpected events caused by process. .

 Manipulating linked list pointers Machine Errors Clock Disk Network Devices Terminals Software Interrupts  High priority interrupts     Lower Priority interrupts     Masking a level will block below and allow above. .Processor Execution Levels  Kernal prevent the occurrence of interrupts during critical activity.

Memory Management    Kernal resides in main memory permanently Compiler generates virtual address space Kernal Mapping virtual address to physical address .

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.