Wipro Technologies; E-Enabling; Data Warehousing

Introduction to UNIX

History of UNIX
GE, Bell Labs, and MIT: Network OS; Multiplex Information and Computing Services
(Multics)
Dennis Ritchie, Ken Thompson, and Brian Kerningham
Unix code development language:
Assembly Code Machine dependant and not portable
B: Ken Thompson
C: Dennis Ritchie
Open source development has lead to the availability of various flavors of UNIX.

Features of UNIX
Multi-user
This feature is facilitated by approaches like Dumb Terminals, Terminal Emulation, and
Dial-up Connection
Discussion on SSH (Terminal Emulation)
>ssh –l informat app360.sea.wamu.net
>ssh informat@app360.sea.wamu.net

Shells in UNIX
Organization of the Unix System
Applications - Shell - Kernel - Hardware
Bourne Shell (sh), C Shell (csh), Korn Shell (ksh), and Bash Shell (bash)
>echo $SHELL
The default shell is set by the admin while creating the user and is given by the above
command.
Every time a user logs in, a shell process is invoked which acts as the parent process for
other processes created and serves as the command interpreter.

Login and Password resetting

/bin/passwd contains details about all the users of the system.
/etc/passwd executable that allows us to change our passwords.

Internal and External commands

The essential UNIX commands for general use are located in /usr/bin. All the commands,
which have an independent existence in the above directory, are called external
commands, while the commands that are built-in and are not stored as separate files are
called internal commands.
>type <external-command>
The above command will give you the location of an external executable command.

I/O redirection and Piping

Combine commands by using the semi colon
>cd /informat/scripts; ls –lrt
ls > dir.list puts the output of ls into a file named dir.list. ls | sort pipes the output of ls
into the sort command. The final output on the terminal will now be sorted.
UNIX I/O revolves around an integer number called a file descriptor. When a file or other
device is opened, a file descriptor number is assigned and used for all the I/O. By
convention, UNIX programs write their normal output to unit 1, and expect input on unit 0.
Error/diagnostic output goes to unit 2.

Unix commands and their help files.

Pager pg, more, less

-1-
 Wipro Technologies; E-Enabling; Data Warehousing

>export $PAGER=more
Navigation is through ‘f’ and ‘b’.
Discussion on search facility, using the forward slash (/) and how to quit (:q).
To get a one line description of the command, use
man –k, apropos, man –f, whatis

Introduction to Visual Editor (vi) (Part I)

Modes of Operation
In the Command mode, all the keys pressed by the user are interpreted as commands.
In the Insert mode, one is permitted to insert new text.
In the ex Command mode, the commands are given at the command line.

Adding, deleting and overwriting text

Hands –on
Create a new file.
Move into the inset mode
Type in text.
Save data from buffer into the file without quitting.
Navigate through the text in the file. (h,j,k,l)
Append data (within line; before current line; after current line)
Delete lines.
Overwrite data.
Join Lines.

Undoing changes.
u/U Latest change.
“[1-9]p If you want to undo changes (deletions) which were done earlier.

Quitting vi. (Temporary and Permanent)

:wq, ZZ, :x, :q! Permanent
:sh Temporary

Block commands.
All executed in the ex command mode.
Line numbers needed. (se nu, se nonu)
Deletion :m,n d
Move :m,n mo p
Copy :m,n co p
All the above commands make changes within the same file.
To copy lines to another file …
:m,n w filename
:m,n w >> filename

-2-
 Wipro Technologies; E-Enabling; Data Warehousing

Visual Editor (vi) (Part II): Advanced Features

Search
Mechanism …
Command Function
/pattern Searches forward
?pattern Searches backward
N Repeats the last search command
N Repeats the last search command in the
backward direction.

Characters to build patterns …

Character Meaning
* Zero or more characters
[] A set or range of characters
$ End of line
^ Beginning of line
\< Beginning of word
\> End of word
. Matches any single character except the
newline character

How to build patterns?

Pattern Meaning
^part All lines which begin with the word ‘part’
part$ All lines which end with the word ‘part’
\<part All strings which begin with the word part.
e.g., partner
part\> All strings which end with the word part.
e.g., depart
\<part\> The whole word ‘part’
[m-s]ore All strings which contain any character in the
range ‘m’ to‘s’ and is followed by ‘ore’.
[^p]art All strings which contain the characters ‘art’,
with ‘art’ being preceded by any character
other than ‘p’.
Wing* Wing followed any sequence of characters

Note: If any special character is being searched then they must be preceded with a
backslash (\)
Search and Replace
Mechanism …
Command Function
:s/str1/str2 Replace first occurrence of str1 with str2 in
current line
:s/str1/str2/g Replace all occurrences of str1 with str2 in
current line
:m,n s/str1/str2/g Replace all occurrences of str1 with str2
from lines m to n.
:1,$ s/str1/str2/g Replace all occurrences of str1 with str2

-3-
 Wipro Technologies; E-Enabling; Data Warehousing

from 1st line to end of file.

:1,. s/str1/str2/g Replace all occurrences of str1 with str2
from 1st line to current line.
:.,$ s/str1/str2/g Replace all occurrences of str1 with str2
from current line to end of file.

abbr
abbr command in vi stands for abbreviation. It can be used to enhance the user’s
convenience.
(Esc):abbr pr printf pr now becomes the shorthand for printf
(Esc):abbr Display the entire list of macros you have set.
(Esc):una pr The pr abbreviation is forgotten.

Set Commands and customizing the vi environment

(Esc):set all Displays the list of all options available with set.

Some commonly used options …

Command Function
:se nu Display of line numbers on
:se eb Beep the speaker when an error occurs
:se ai Auto indent on
:se ic Ignore case while searching a pattern
:se showmode Display mode in which we are working

Whenever vi is invoked, it searches for the .exrc file in the user’s default working
directory ($HOME). This file is optional, i.e., vi does not give an error if it is not able to
locate this file. However, if the file is present any set commands or abbreviations that are
stored in this file automatically become effective for all vi sessions.

Multiple file editing Command line options in vi

Comman Function Command Function
d
$vi file1 3 files are loaded into vi $vi + 10 file Loads file and places cursor
file2 file3 buffer for editing on the 10th line of the file
:n Edit next file in buffer $vi +/pattern file Loads file and places cursor
on first occurrence of
matching pattern
:n! Edit next file without $view file Opens file in read-only
saving current file mode
:rew Edit 1st file in buffer
:rew! Edit 1st file without
saving current file
:args Display names of all
files in buffer, with
current file name
enclosed in []
:f Displays name of
current file

-4-
 Wipro Technologies; E-Enabling; Data Warehousing

Shell Scripting: Introduction & Decision making statements

User defined and System defined variables

System variables are always accessible. The shell provides the values for these variables.
These variables are usually used by the system itself and govern the environment we
work under. If we so desire we can change the values of these variables as per our
preferences and customize the system environment. For e.g.,
PS1 : System Prompt 1.
PATH : The path, which the shell must search in order to execute any command or file.
HOME : The default working directory of the user.
SHELL : Default working shell.
MAIL : The file where the mail of the user is stored.
MAILCHECK : Duration after which the shell checks whether the user has received any
mail.
LOGNAME : Login name of the user
The value of the system variables can be displayed by typing the following command at
the prompt echo $<system_variable>.

User defined variables are used most extensively in shell programming. To access the
value of any kind of variable, we need to precede the variable name with a dollar ($) sign.

1. All shell variables are string variables. We cannot carry out arithmetic calculations on
them unless we use a command called expr.
2. While using the assignment operator to assign values to variables, there should be no
spaces on either side of =. During assignment, if the variable does not exist it will be
created.
3. If the variable needs to contain more than one word then the assignment should be
made using double quotes.
4. We can carry out more than one assignment in a line. So also we can echo more than
one variable’s value at a time.
5. All variables defined inside a script die the moment the execution of the script is over.
6. A constant variable can be created by saying $>readonly <variable_name>.
7. Variables can be made to cease existing by using the unset command. $>unset
<variable_name>. This command cannot be used on system variables.
8. The entire user defined variables and system defined variables can be displayed by
using the command set at the command prompt.

Positional parameters (setting and shifting)

The shell script uses positional parameters to access the command line arguments which
are passed to it. Positional parameters are 9 in number, named $1 through $9.
The set command can be used to assign values to the positional parameters.
$>set We are now trying to understand the setting of positional parameters without
calling a script
$>echo $1 $2 $3 $4 $5 $6 $7 $8 $9
$>echo Total number of arguments: $#
$>echo All positional parameters: $@
$>echo After shifting by 5 positions
$>shift 5; echo 1 $2 $3 $4 $5 $6 $7 $8 $9

Exit status of a command

-5-
 Wipro Technologies; E-Enabling; Data Warehousing

Every UNIX command/script returns an exit status of zero, if it has been successful
executed, otherwise, a value greater than zero is returned.
$>echo $?
$? stores the exit status of the last executed command.

Things to be noted while using arithmetic in shell scripts.

$>echo `expr $a \* $b`
1. The multiplication symbol should be preceded by a \. Otherwise, the shell treats it as a
wildcard character for all files in the current directory.
2. Terms of the expression provided to expr must be separated by blanks.
3. expr is only capable of carrying out integer arithmetic. To carry out arithmetic on real
numbers it is necessary to use the bc command.

Numerical Test Operators

Operator Meaning
-gt Greater than
-lt Less than
-ge Greater than or equal
to
-le Less than or equal to
-ne Not equal to
-eq Equal to

File Tests
Option TRUE if …
-s File exists and has size greater
than zero
-f File exists and is not a directory
-d File exists and is a directory
-r File exists and user has read
permission on it
-w File exists and user has write
permission on it
-x File exists and user has
execute permission on it

String Tests
Operator TRUE if …
string1 = string2 Strings are same
-n string Length of string > 0
-z string Length of string = 0
string String is not a null
string

Logical operators
Operator Meaning
-a AND
-o OR
! case valueNOT
in
choice 1)
The Case Control…..Structure
…..
;;
choice 2)
…..
…..
;;
*)
…..
;; -6-
esac
 Wipro Technologies; E-Enabling; Data Warehousing

Example Problems and Solutions.

A. Code Snippets …
• Interaction with the user
• Command line arguments.

B. Program Example …
1. The file /etc/passwd contains information about all users. However it is difficult to
decipher the information stored in it. Write a shell script which would receive the
logname during execution, obtain information about it from /etc/passwd and display
this information on the screen in easily understandable format.
2. If the cost price and selling price of an item is input through the keyboard, write a
program to find out whether the seller has made profit or incurred loss. Also
determine how much profit was made or loss incurred.

Code Snippet 1: Interaction with the

script_1.ksh
user

Code Snippet 2: Command line

arguments. script_2.ksh

Program Example 1. script_3.ksh

Program Example 2. script_4.ksh

-7-
 Wipro Technologies; E-Enabling; Data Warehousing

Shell Scripting: The Loop Control structure

Syntax of the while, until and for loop

while [ control command is until [ control command is not
true ] true]
do do
command1 command1
command2 command2
done done

for control-variable in value1 value2 value3

…
do
command1
command2
done
Nesting of loops
Nesting of loops is allowed without any upper limit but to improve readability and
maintainability, proper indentation should be followed. The do … done keyword pair are
compulsory for each looping occurrence. Care should be taken to ensure that no
mismatch of the above keyword pair is done while nesting and this is achieved if we
maintain proper indentation.

Break and Continue

A ‘break’ takes you out of the loop bypassing all commands within the loop. A ‘continue’
sends you straight to the control commands at the beginning of the loop.

Whenever the keyword ‘break’ is encountered inside any loop, control automatically
passes to the first statement after the loop in which it is placed. However, if we have
nested loops and we want to take the control outside any one of the outer loops, then we
use break <loop-number>. The same facility is also provided with the continue
statement. Both ‘break’ and ‘if’ are generally associated with an ‘if’.

Example Problems and Solutions.

A. Code Snippets …
• Input File Separator

B. Program Example …
1. You want to check whether your login is being used by anyone else. Write a shell
script which checks after every one minute whether anyone other than you has
logged in using your login name. Keep on checking until the time exceeds 5 minutes
or the number of such logins exceeds 5. Maintain a log-file for the checking.
2. You are trying to search for the word unix in your files. Your shell script should search
for the pattern in each of the files supplied at the command prompt in turn and stop
at the first file that it encounters which matches the search criteria. The filename
should be displayed on the screen. If no arguments are supplied then all the files in
the current working directory should be assumed.

-8-
 Wipro Technologies; E-Enabling; Data Warehousing

Code Snippet 1: Interaction with the

script_1.ksh
user

Program Example 1. script_2.ksh

Shell Scripting: Miscellaneous Information

Difference between executing a script normally and with a ‘.’

The preceding dot (.) while executing a script ensures that the script gets executed in the
current shell. A space between the ‘.’ And the script name is mandatory. If it is absent the
shell considers the script as a hidden file.

When do we need to export?

If we want variables to be available to all sub-shells we must export them from the
current shell.
$>export <variable>
a. A variable once exported from the parent shell becomes available to the sub shell or
any other shells launched from this shell.
b. If we are to unexport a variable, we have to first unset it and then recreate it.
c. A variable can be exported from the parent shell to its sub shell, but never the other
way round.
d. If the sub-shell changes the value of an exported variable the value of this variable in
the parent shell remains unchanged since the sub-shell always works only the copy of
the variable in the parent shell.
e. If an exported variable is modified in a sub shell then to make this modified value
available to a sub-sub-shell we must once again export the variable in the sub-shell.

Trapping signals and Terminal settings.

By trapping the program terminating signals Signal Signal Number
shown in the side table, we can use them to
Exit 0
achieve some other task totally unconnected with
Ctrl d 1
their original duties, and hence ignore the task
Del 2
they were initially instructed to do.
Kill 15

A few examples on trapping

trap “” 123 Does nothing on generation of signal numbers 1,2 or 3
trap “ls;who” 2Displays file listing and current user listing on generation of signal number
2.

Terminal Settings
Setting Action
echo Echoes back every character typed
-echo Does not echo back every character typed
sane Resets all modes to some reasonable values.

-9-
 Wipro Technologies; E-Enabling; Data Warehousing

Here document
The HERE document signifies that the data is here instead of in a separate file. Any
command using statndard input can also take input from a here document.
Example (Executing Oracle procedures from Unix scripts)

# ---------------------------------------------------------------
# Calling Database procedures from a UNIX script
# ---------------------------------------------------------------
# Script Name call_sortbydisplayorder.sh
# Usage sh call_sortbydisplayorder.sh
# ---------------------------------------------------------------
(sqlplus –s user/password@hoststring << eof
set timing on
set feedback on
exec sortbydisplayorder
set feedback off
set timing off
eof
)
# ---------------------------------------------------------------
# End of script
# ---------------------------------------------------------------

Debugging scripts
Debugging of scripts can be achieved by setting on the flags –x and –v.

Example Problems and Solutions.

A. Code Snippets
• Screen Saver in Unix

B. Program Example …
1. Accept a file name as input and delete all occurrences of the supplied word (2nd
Argument) in the file.
2. You are trying to search for the word unix in your files. Your shell script should search
for the pattern in each of the files supplied at the command prompt in turn and stop
at the first file that it encounters which matches the search criteria. The filename
should be displayed on the screen. If no arguments are supplied then all the files in
the current working directory should be assumed.

Code Snippet 1: Screen Saver in Unix script_1.ksh

Program Example 1. script_2.ksh

Program Example 2. script_3.ksh

- 10 -
 Wipro Technologies; E-Enabling; Data Warehousing

Shell Scripting: Coding Standards

Goals
1. Make the least efforts when moving scripts from one environment to another.
2. Make the least errors when testing scripts.
3. Enable running the same version of scripts simultaneously in different environments
in the same machine (in case the test/dev/prod environments reside in the same
machine).

Standards
1. Script naming & Coding style
A script name should have a suffix indicating the shell on which it has been designed
to be executed (.sh/.ksh/.csh). The script name should have a mixture of cases and
there should be an avoidance of the use of underscore.

The following sections should be a part of the script in the sequence mentioned …

a. Mandatory Shell specification

The shebang line should be the first line of the code to indicate which shell should
be used to execute the script.
#!/bin/ksh

b. Mandatory header section

This section should contain all details like the script name, created by, creation date,
updated by, description, etc.
Example:
# -------------------------------------------------------------
# Script name :
# Created by :
# Created on :
# Description :
# -------------------------------------------------------------
# Updated by :
# Updated at :
# Reason(s) for update …
# -------------------------------------------------------------

c. Mandatory environment setting

To ensure smooth transition between the various environments available to you,
initialize all commonly used variables in a single file (for ETL Services, the file is

- 11 -
 Wipro Technologies; E-Enabling; Data Warehousing

.etl.env). After this is done, put the following line of code in your script before any
processing is done …
. <path>/<envFile>
To differentiate between your various environments (especially if all of them lie on the
same machine), name the envFile in such a manner that it indicates the environment
to which it belongs and just change the above line of code while moving from one
environment to another.

Note:
For any script that will not be directly called but will be called by a parent script, and if
it references any environment variables and its parent script has already invoked the
environment variable file, then the child script does not have to re-invoke the
environment variables file.

d. Mandatory usage check section

Example:
if [ $# -ne 2 ]
then
echo "Improper Usage! Correct Usage: $0 <filename> <search_pattern>"
exit 1
fi

e. Optional variable definition section

Use this section to replace all hard coded values which are used more than once in
the script.

f. Mandatory comments section for major code blocks

Every major code block should be preceded by an explanatory comment.

g. SQL File
Scripts involving calling a SQL file should pass the parameter to the SQL file instead of
hard coding the value of the parameter in the SQL script.
Example:
Replace the statement below in the SQL file …
spool /wamu/informat/logFiles
with …
spool &1
and pass the value from the Unix script as parameter like below …
sqlplus user/passwd@connectString /wamu/informat/logFiles

2. Script log file naming

If a script generates a log file, the log file should be named according to the following
convention:
a. The log file should have the script name (without script’s suffix) as the 1st part of file
name.
b. The log file should have a timestamp (YYYYMMDD:HH:MM:SS) as the 2 nd part of file
name.
c. The log file should have .log suffix as the last part of the file name.
Example:
Script name : MyScript.sh
Server Time : 2005/04/05 10:30:56
Log File : MyScript20050405:10:30:56.log

3. Script migration
During migration of a script from development to production check the file for
execution privilege. File must have proper execution privilege.

- 12 -