DataBase Management Systems

h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
Code & Subject Name: 141405 & Database Management Systems
1
Unit Number & Name: I & Introduction Period: 1 of 45
DBMS contains information about a particular enterprise
Collection of interrelated data
Set of programs to access the data
An environment that is both convenient and efficient to use
Database Applications:
Banking: all transactions
Airlines: reservations, schedules
Universities: registration, grades
Sales: customers, products, purchases
Online retailers: order tracking, customized recommendations
Manufacturing: production, inventory, orders, supply chain
Human resources: employee records, salaries, tax deductions
Databases touch all aspects of our lives
In the early days, database applications were built directly on top of file systems
Drawbacks of using file systems to store data:
Data redundancy and inconsistency
Multiple file formats, duplication of information in different files
Difficulty in accessing data
Need to write a new program to carry out each new task
Data isolation multiple files and formats
Integrity problems
Integrity constraints (e.g. account balance > 0) become buried in
program code rather than being stated explicitly
Hard to add new constraints or change existing ones
Drawbacks of using file systems (cont.)
Atomicity of updates
Failures may leave database in an inconsistent state with partial
updates carried out
Example: Transfer of funds from one account to another should
either complete or not happen at all
Concurrent access by multiple users
Concurrent accessed needed for performance
Uncontrolled concurrent accesses can lead to inconsistencies
Example: Two people reading a balance and updating it at the
same time
Security problems
Hard to provide user access to some, but not all, data
Database systems offer solutions to all the above problems
http://csetube.co.nr/
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
2
Page: 2 of 2
Levels of Abstraction
Physical level: describes how a record (e.g., customer) is stored.
Logical level: describes data stored in database, and the relationships among the data.
type customer = record
customer_id : string;
customer_name : string;
customer_street : string;
customer_city : string;
end;
View level: application programs hide details of data types. Views can also hide
information (such as an employees salary) for security purposes.
View of Data
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
3
Page: 2 of 2
Instances and Schemas
Similar to types and variables in programming languages
Schema the logical structure of the database
Example: The database consists of information about a set of customers and
accounts and the relationship between them)
Analogous to type information of a variable in a program
Physical schema: database design at the physical level
Logical schema: database design at the logical level
Instance the actual content of the database at a particular point in time
Analogous to the value of a variable
Physical Data Independence the ability to modify the physical schema without
changing the logical schema
Applications depend on the logical schema
In general, the interfaces between the various levels and components should
be well defined so that changes in some parts do not seriously influence
others.
Data Models
A collection of tools for describing
Data
Data relationships
Data semantics
Data constraints
Relational model
Entity-Relationship data model (mainly for database design)
Object-based data models (Object-oriented and Object-relational)
Semistructured data model (XML)
Other older models:
Network model
Hierarchical model
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
4
Page: 2 of 2
Data Manipulation Language (DML)
Language for accessing and manipulating the data organized by the appropriate data
model
DML also known as query language
Two classes of languages
Procedural user specifies what data is required and how to get those data
Declarative (nonprocedural) user specifies what data is required without
specifying how to get those data
SQL is the most widely used query language
Data Definition Language (DDL)
Specification notation for defining the database schema
Example: create table account (
account_number char(10),
branch_name char(10),
balance integer)
DDL compiler generates a set of tables stored in a data dictionary
Data dictionary contains metadata (i.e., data about data)
Database schema
Data storage and definition language
Specifies the storage structure and access methods used
Integrity constraints
Domain constraints
Referential integrity (e.g. branch_name must correspond to a valid
branch in the branch table)
Authorization
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
5
Page: 2 of 2
Overall System Structure
The architecture of a database systems is greatly influenced by the underlying computer
system on which the database is running:
Centralized
Client-server
Parallel (multiple processors and disks)
Distributed
Lecture Plan R/TP/02
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
6
Page: 2 of 2
Database Users
Users are differentiated by the way they expect to interact with
the system
Application programmers interact with system through DML calls
Sophisticated users form requests in a database query language
Specialized users write specialized database applications that do not fit into the
traditional data processing framework
Nave users invoke one of the permanent application programs that have been
written previously
Examples, people accessing database over the web, bank tellers, clerical staff
Database Administrator
Coordinates all the activities of the database system
has a good understanding of the enterprises information resources and needs.
Database administrator's duties include:
Storage structure and access method definition
Schema and physical organization modification
Granting users authority to access the database
Backing up data
Monitoring performance and responding to changes
Database tuning
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
7
Page: 2 of 2
The Entity-Relationship Model
Models an enterprise as a collection of entities and relationships
Entity: a thing or object in the enterprise that is distinguishable from other
objects
Described by a set of attributes
Relationship: an association among several entities
Represented diagrammatically by an entity-relationship diagram:
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
8
Page: 2 of 2
Entity Sets customer and loan
customer_id customer_ customer_ customer_ loan_ amount
name street city number
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
9
Page: 2 of 2
Relational Model
Structure of Relational Databases
Fundamental Relational-Algebra-Operations
Additional Relational-Algebra-Operations
Extended Relational-Algebra-Operations
Null Values
Modification of the Database
Example of a Relation
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
10
Unit Number & Name: II & RELATIONAL MODEL Period: 10 of 45
Page: 2 of 2
The relational Model The catalog- Types Keys
Each attribute of a relation has a name
The set of allowed values for each attribute is called the domain of the attribute
Attribute values are (normally) required to be atomic; that is, indivisible
E.g. the value of an attribute can be an account number,
but cannot be a set of account numbers
Domain is said to be atomic if all its members are atomic
The special value null is a member of every domain
The null value causes complications in the definition of many operations
We shall ignore the effect of null values in our main presentation and consider
their effect later
Keys
Let K _ R
K is a superkey of R if values for K are sufficient to identify a unique tuple of each
possible relation r(R)
by possible r we mean a relation r that could exist in the enterprise we are
modeling.
Example: {customer_name, customer_street} and
{customer_name}
are both superkeys of Customer, if no two customers can possibly have the
same name
In real life, an attribute such as customer_id would be used instead
of customer_name to uniquely identify customers, but we omit it to
keep our examples small, and instead assume customer names are
unique.
K is a candidate key if K is minimal
Example: {customer_name} is a candidate key for Customer, since it is a superkey
and no subset of it is a superkey.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
11
Primary key: a candidate key chosen as the principal means of identifying tuples
within a relation
Should choose an attribute whose value never, or very rarely, changes.
E.g. email address is unique, but may change
Relational Algebra
Procedural language
Six basic operators
select: o
project: [
union:
set difference:
Cartesian product: x
rename:
The operators take one or two relations as inputs and produce a new relation as a
result.
Select Operation Example
Relation r
oA=B ^ D > 5 (r)
A B C D
o
o
|
|
o
|
|
|
1
5
12
23
7
7
3
10
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
12
A B C D
o
|
o
|
1
23
7
10
P Pr ro oj je ec ct t O Op pe er ra at ti io on n E Ex xa am mp pl le e
n Relation r:
A B C
o
o
|
|
10
20
30
40
1
1
1
2
A C
o
o
|
|
1
1
1
2
=
A C
o
|
|
1
1
2
[
A,C
(r)
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
13
U Un ni io on n O Op pe er ra at ti io on n E Ex xa am mp pl le e
n Relations r, s:
n r s:
A B
o
o
|
1
2
1
A B
o
|
2
3
r
s
A B
o
o
|
|
1
2
1
3
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
14
S Se et t D Di if ff fe er re en nc ce e O Op pe er ra at ti io on n E Ex xa am mp pl le e
n Relations r, s:
n r s:
A B
o
o
|
1
2
1
A B
o
|
2
3
r
s
A B
o
|
1
1
C Ca ar rt te es si ia an n- -P Pr ro od du uc ct t O Op pe er ra at ti io on n E Ex xa am mp pl le e
n Relations r, s:
n r x s:
A B
o
|
1
2
A B
o
o
o
o
|
|
|
|
1
1
1
1
2
2
2
2
C D
o
|
|
o
|
|
10
10
20
10
10
10
20
10
E
a
a
b
b
a
a
b
b
C D
o
|
|
10
10
20
10
E
a
a
b
b
r
s
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
15
C Ca ar rt te es si ia an n- -P Pr ro od du uc ct t O Op pe er ra at ti io on n E Ex xa am mp pl le e
n Relations r, s:
n r x s:
A B
o
|
1
2
A B
o
o
o
o
|
|
|
|
1
1
1
1
2
2
2
2
C D
o
|
|
o
|
|
10
10
20
10
10
10
20
10
E
a
a
b
b
a
a
b
b
C D
o
|
|
10
10
20
10
E
a
a
b
b
r
s
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
16
Page: 2 of 2
Tuple Relational Calculus
Domain Relational Calculus
Tuple Relational Calculus
A nonprocedural query language, where each query is of the form
{t | P (t ) }
It is the set of all tuples t such that predicate P is true for t
t is a tuple variable, t [A ] denotes the value of tuple t on attribute A
t e r denotes that tuple t is in relation r
P is a formula similar to that of the predicate calculus
Predicate Calculus Formula
1. Set of attributes and constants
2. Set of comparison operators: (e.g., <, s, =, =, >, >)
3. Set of connectives: and (.), or (v) not ()
4. Implication (): x y, if x if true, then y is true
x y x v y
5. Set of quantifiers:
- t e r (Q (t )) there exists a tuple in t in relation r
such that predicate Q (t ) is true
t e r (Q (t )) Q is true for all tuples t in relation r
Banking Example
branch (branch_name, branch_city, assets )
customer (customer_name, customer_street, customer_city )
account (account_number, branch_name, balance )
loan (loan_number, branch_name, amount )
depositor (customer_name, account_number )
borrower (customer_name, loan_number )
Example Queries
Find the loan_number, branch_name, and amount for loans of over $1200
{t | t e loan . t [amount ] > 1200}
n Find the loan number for each loan of an amount greater than $1200
{t | - s e loan (t [loan_number ] = s [loan_number ] . s [amount ] > 1200)}
Notice that a relation on schema [loan_number ] is implicitly defined by
the query
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
17
Domain Relational Calculus
A nonprocedural query language equivalent in power to the tuple relational calculus
Each query is an expression of the form:
{ < x1, x2, , xn > | P (x1, x2, , xn)}
x1, x2, , xn represent domain variables
P represents a formula similar to that of the predicate calculus
E Ex xa am mp pl le e Q Qu ue er ri ie es s
n Find the loan_number, branch_name, and amount for loans of over
$1200
n Find the names of all customers who have a loan from the Perryridge branch
and the loan amount:
{< c, a > | - l (< c, l > e borrower . -b (< l, b, a > e loan .
b = Perryridge))}
{< c, a > | - l (< c, l > e borrower . < l, Perryridge, a > e loan)}
{< c > | - l, b, a (< c, l > e borrower . < l, b, a > e loan . a > 1200)}
n Find the names of all customers who have a loan of over $1200
{< l, b, a > | < l, b, a > e loan . a > 1200}
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
18
Page: 2 of 2
E Ex xa am mp pl le e Q Qu ue er ri ie es s
n Find the names of all customers having a loan, an account, or both at
the Perryridge branch:
{< c > | - s,n (< c, s, n > e customer) .
x,y,z (< x, y, z > e branch . y = Brooklyn)
- a,b (< x, y, z > e account . < c,a > e depositor)}
n Find the names of all customers who have an account at all
branches located in Brooklyn:
{< c > | - l ( < c, l > e borrower
. - b,a (< l, b, a > e loan . b = Perryridge))
v - a (< c, a > e depositor
. - b,n (< a, b, n > e account . b = Perryridge))}
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
19
Page: 2 of 2
SQL
Data Definition
Basic Query Structure
Set Operations
Aggregate Functions
Null Values
Nested Subqueries
Complex Queries
Views
Modification of the Database
Joined Relations
Domain Types in SQL
char(n). Fixed length character string, with user-specified length n.
varchar(n). Variable length character strings, with user-specified maximum length
n.
int. Integer (a finite subset of the integers that is machine-dependent).
smallint. Small integer (a machine-dependent subset of the integer domain type).
numeric(p,d). Fixed point number, with user-specified precision of p digits, with n
digits to the right of decimal point.
real, double precision. Floating point and double-precision floating point numbers,
with machine-dependent precision.
float(n). Floating point number, with user-specified precision of at least n digits.
Integrity Constraints
Integrity constraints guard against accidental damage to the database, by ensuring that
authorized changes to the database do not result in a loss of data consistency.
A checking account must have a balance greater than $10,000.00
A salary of a bank employee must be at least $4.00 an hour
A customer must have a (non-null) phone number
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
20
Page: 2 of 2
Advanced SQL
SQL Data Types and Schemas
Integrity Constraints
Authorization
Embedded SQL
Dynamic SQL
Functions and Procedural Constructs
Recursive Queries
Advanced SQL Features
Built-in Data Types in SQL
date: Dates, containing a (4 digit) year, month and date
Example: date 2005-7-27
time: Time of day, in hours, minutes and seconds.
Example: time 09:00:30 time 09:00:30.75
timestamp: date plus time of day
Example: timestamp 2005-7-27 09:00:30.75
interval: period of time
Example: interval 1 day
Subtracting a date/time/timestamp value from another gives an interval value
Interval values can be added to date/time/timestamp values
Can extract values of individual fields from date/time/timestamp
Example: extract (year fromr.starttime)
Can cast string types to date/time/timestamp
Example: cast <string-valued-expression> as date
Example: cast <string-valued-expression> as time
Referential Integrity in SQL Example
create table customer
(customer_name char(20),
customer_street char(30),
customer_city char(30),
primary key (customer_name ))
create table branch
(branch_name char(15),
branch_city char(30),
assets numeric(12,2),
primary key (branch_name ))
create table account
(account_number char(10),
branch_name char(15),
balance integer,
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
21
primary key (account_number),
foreign key (branch_name) references branch )
create table depositor
(customer_name char(20),
account_number char(10),
primary key (customer_name, account_number),
foreign key (account_number ) references account,
foreign key (customer_name ) references customer )
Privileges in SQL
select: allows read access to relation,or the ability to query using the view
Example: grant users U1, U2, and U3 select authorization on the branch
relation:
grant select on branch to U1, U2, U3
insert: the ability to insert tuples
update: the ability to update using the SQL update statement
delete: the ability to delete tuples.
all privileges: used as a short form for all the allowable privileges
Revoking Authorization in SQL
The revoke statement is used to revoke authorization.
revoke <privilege list>
on <relation name or view name> from <user list>
Example:
revoke select on branch from U1, U2, U3
All privileges that depend on the privilege being revoked are also revoked.
<privilege-list> may be all to revoke all privileges the revokee may hold.
If the same privilege was granted twice to the same user by different grantees,
the user may retain the privilege after the revocation.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
22
Page: 2 of 2
Embedded SQL
The SQL standard defines embeddings of SQL in a variety of programming
languages such as C, Java, and Cobol.
A language to which SQL queries are embedded is referred to as a host language,
and the SQL structures permitted in the host language comprise embedded SQL.
The basic form of these languages follows that of the System R embedding of SQL
into PL/I.
EXEC SQL statement is used to identify embedded SQL request to the preprocessor
EXEC SQL <embedded SQL statement > END_EXEC
Note: this varies by language (for example, the Java embedding uses
# SQL { . }; )
From within a host language, find the names and cities of customers with more than
the variable amount dollars in some account.
Specify the query in SQL and declare a cursor for it
EXEC SQL
declare c cursor for
select depositor.customer_name, customer_city
fromdepositor, customer, account
where depositor.customer_name = customer.customer_name
and depositor account_number = account.account_number
and account.balance > :amount
END_EXEC
The open statement causes the query to be evaluated
EXEC SQL open c END_EXEC
The fetch statement causes the values of one tuple in the query result to be placed on
host language variables.
EXEC SQL fetch c into :cn, :cc END_EXEC
Repeated calls to fetch get successive tuples in the query result
A variable called SQLSTATE in the SQL communication area (SQLCA) gets set to
02000 to indicate no more data is available
The close statement causes the database system to delete the temporary relation that
holds the result of the query.
EXEC SQL close c END_EXEC
Note: above details vary with language. For example, the Java embedding defines Java
iterators to step through result tuples.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
23
Dynamic SQL
Allows programs to construct and submit SQL queries at run time.
Example of the use of dynamic SQL from within a C program.
char * sqlprog = update account
set balance = balance * 1.05
where account_number = ?
EXEC SQL prepare dynprog from:sqlprog;
char account [10] = A-101;
EXEC SQL execute dynprog using :account;
The dynamic SQL program contains a ?, which is a place holder for a value that is
provided when the SQL program is executed.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
24
Unit Number & Name: III & DATABASE DESIGN Period: 19 of 45
Page: 2 of 2
Functional Dependencies
Constraints on the set of legal relations.
Require that the value for a certain set of attributes determines uniquely
the value for another set of attributes.
A functional dependency is a generalization of the notion of a key.
Let R be a relation schema
o _ R and | _ R
The functional dependency
o |
holds on R if and only if for any legal relations r(R), whenever any two
tuples t1 and t2 of r agree on the attributes o, they also agree on the
attributes |. That is,
t1[o] = t2 [o] t1[| ] = t2 [| ]
Example: Consider r(A,B ) with the following instance of r.
On this instance, A B does NOT hold, but B A does hold.
K is a superkey for relation schema R if and only if K R
K is a candidate key for R if and only if
K R, and
for no o c K, o R
Functional dependencies allow us to express constraints that cannot be
expressed using superkeys. Consider the schema:
bor_loan = (customer_id, loan_number, amount ).
We expect this functional dependency to hold:
loan_number amount
but would not expect the following to hold:
amount customer_name
A functional dependency is trivial if it is satisfied by all instances of a
relation
Example:
customer_name, loan_number customer_name
customer_name customer_name
In general, o | is trivial if | _ o
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
25
Page: 2 of 2
Functional Dependencies First, Second, Third Normal Forms
First Normal Form
Domain is atomic if its elements are considered to be indivisible units
Examples of non-atomic domains:
Set of names, composite attributes
Identification numbers like CS101 that can be broken up into parts
A relational schema R is in first normal form if the domains of all attributes of R are
atomic
Non-atomic values complicate storage and encourage redundant (repeated) storage of
data
Example: Set of accounts stored with each customer, and set of owners stored
with each account
We assume all relations are in first normal form
Atomicity is actually a property of how the elements of the domain are used.
Example: Strings would normally be considered indivisible
Suppose that students are given roll numbers which are strings of the form
CS0012 or EE1127
If the first two characters are extracted to find the department, the domain of
roll numbers is not atomic.
Doing so is a bad idea: leads to encoding of information in application
program rather than in the database.
Third Normal Form
A relation schema R is in third normal form (3NF) if for all:
o | in F+
at least one of the following holds:
o | is trivial (i.e., | e o)
o is a superkey for R
Each attribute A in | o is contained in a candidate key for R.
(NOTE: each attribute may be in a different candidate key)
If a relation is in BCNF it is in 3NF (since in BCNF one of the first two conditions
above must hold).
Third condition is a minimal relaxation of BCNF to ensure dependency preservation
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
26
Page: 2 of 2
Functional Dependencies First, Second, Third Normal Forms
Third Normal Form: Motivation
There are some situations where
BCNF is not dependency preserving, and
efficient checking for FD violation on updates is important
Solution: define a weaker normal form, called Third Normal Form
(3NF)
Allows some redundancy
But functional dependencies can be checked on individual relations
without computing a join.
There is always a lossless-join, dependency-preserving decomposition into
3NF.
EXAMPLE
Relation R:
R = (J, K, L )
F = {JK L, L K }
Two candidate keys: JK and JL
R is in 3NF
JK L JK is a superkey
L K K is contained in a candidate key
Testing for 3NF
Optimization: Need to check only FDs in F, need not check all FDs in F+.
Use attribute closure to check for each dependency o |, if o is a superkey.
If o is not a superkey, we have to verify if each attribute in | is contained in a
candidate key of R
this test is rather more expensive, since it involve finding candidate keys
testing for 3NF has been shown to be NP-hard
Interestingly, decomposition into third normal form (described shortly) can be
done in polynomial time
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
27
Page: 2 of 2
Dependency Preservation
Let Fi be the set of dependencies F + that include only attributes in Ri.
A decomposition is dependency preserving, if
(F1 F2 Fn )+ = F +
If it is not, then checking updates for violation of
functional dependencies may require computing joins,
which is expensive.
Testing for Dependency Preservation
To check if a dependency o | is preserved in a decomposition of R into
R1, R2, , Rn we apply the following test (with attribute closure done
with respect to F)
result = o
while (changes to result) do
for each Ri in the decomposition
t = (result Ri)+ Ri
result = result t
If result contains all attributes in |, then the functional dependency
o | is preserved.
We apply the test on all dependencies in F to check if a decomposition is
dependency preserving
This procedure takes polynomial time, instead of the exponential time
required to compute F+ and (F1 F2 Fn)+
Example
R = (A, B, C )
F = {A B
B C}
Key = {A}
R is not in BCNF
Decomposition R1 = (A, B), R2 = (B, C)
R1 and R2 in BCNF
Lossless-join decomposition
Dependency preserving
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
28
Page: 2 of 2
Boyce/Codd Normal Form
A relation schema R is in BCNF with respect to a set F of functional
dependencies if for all functional dependencies in F+ of the form
o |
where o _ R and | _ R, at least one of the following holds:
o | is trivial (i.e., | _ o)
o is a superkey for R
Example schema not in BCNF:
bor_loan = ( customer_id, loan_number, amount )
because loan_number amount holds on bor_loan but loan_number is
not a superkey
How good is BCNF?
There are database schemas in BCNF that do not seem to be sufficiently
normalized
Consider a database
classes (course, teacher, book )
such that (c, t, b) e classes means that t is qualified to teach c, and b is a
required textbook for c
The database is supposed to list for each course the set of teachers any
one of which can be the courses instructor, and the set of books, all of
which are required for the course
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
29
Page: 2 of 2
Multi-valued Dependencies and Fourth Normal Form
Let R be a relation schema and let o _ R and | _ R. The multivalued
dependency
o |
holds on R if in any legal relation r(R), for all pairs for tuples t1 and t2
in r such that t1[o] = t2 [o], there exist tuples t3 and t4 in r such that:
t1[o] = t2 [o] = t3 [o] = t4 [o]
t3[|] = t1 [|]
t3[R |] = t2[R |]
t4 [|] = t2[|]
t4[R |] = t1[R |]
Tabular representation of o |
Example
Let R be a relation schema with a set of attributes that are partitioned into
3 nonempty subsets.
Y, Z, W
We say that Y Z (Y multidetermines Z )
if and only if for all possible relations r (R )
< y1, z1, w1 > e r and < y1, z2, w2 > e r
then
< y1, z1, w2 > e r and < y1, z2, w1 > e r
Note that since the behavior of Z and Ware identical it follows that
Y Z if Y W
In our example:
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
30
course teacher
course book
The above formal definition is supposed to formalize the notion that
given a particular value of Y (course) it has associated with it a set of
values of Z (teacher) and a set of values of W (book), and these two sets
are in some sense independent of each other.
Note:
l If Y Z then Y Z
l Indeed we have (in above notation) Z1 = Z2
The claim follows.
Use of Multivalued Dependencies
We use multivalued dependencies in two ways:
1. To test relations to determine whether they are legal under a given set
of functional and multivalued dependencies
2. To specify constraints on the set of legal relations. We shall thus
concern ourselves only with relations that satisfy a given set of functional
and multivalued dependencies.
If a relation r fails to satisfy a given multivalued dependency, we can
construct a relations r' that does satisfy the multivalued dependency by
adding tuples to r.
Fourth Normal Form
A relation schema R is in 4NF with respect to a set D of functional and
multivalued dependencies if for all multivalued dependencies in D+ of
the form o |, where o _ R and | _ R, at least one of the following
hold:
o | is trivial (i.e., | _ o or o | = R)
o is a superkey for schema R
If a relation is in 4NF it is in BCNF
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
31
Page: 2 of 2
Join Dependencies and Fifth Normal Form
Join dependencies generalize multivalued dependencies
lead to project-join normal form (PJNF) (also called fifth
normal form)
A class of even more general constraints, leads to a normal form called
domain-key normal form.
Problem with these generalized constraints: are hard to reason with, and
no set of sound and complete set of inference rules exists.
Hence rarely used
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
32
Unit Number & Name: IV & TRANSACTIONS Period: 28 of 45
Page: 2 of 2
Transaction Concepts - Transaction Recovery
A transaction is a unit of program execution that accesses and possibly
updates various data items.
E.g. transaction to transfer $50 from account A to account B:
1. read(A)
2. A := A 50
3. write(A)
4. read(B)
5. B := B + 50
6. write(B)
Two main issues to deal with:
Failures of various kinds, such as hardware failures and system
crashes
Concurrent execution of multiple transactions
Example of Fund Transfer
n Transaction to transfer $50 from account A to account B:
1. read(A)
2. A := A 50
3. write(A)
4. read(B)
5. B := B + 50
6. write(B)
Atomicity requirement
if the transaction fails after step 3 and before step 6, money will be
lost leading to an inconsistent database state
Failure could be due to software or hardware
the system should ensure that updates of a partially executed
transaction are not reflected in the database
Durability requirement once the user has been notified that the
transaction has completed (i.e., the transfer of the $50 has taken place),
the updates to the database by the transaction must persist even if there
are software or hardware failures.
n Transaction to transfer $50 from account A to account B:
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
33
1. read(A)
2. A := A 50
3. write(A)
4. read(B)
5. B := B + 50
6. write(B)
Consistency requirement in above example:
the sum of A and B is unchanged by the execution of the
transaction
In general, consistency requirements include
Explicitly specified integrity constraints such as primary
keys and foreign keys
Implicit integrity constraints
e.g. sum of balances of all accounts, minus sum of
loan amounts must equal value of cash-in-hand
A transaction must see a consistent database.
During transaction execution the database may be temporarily
inconsistent.
When the transaction completes successfully the database must be
consistent
Erroneous transaction logic can lead to inconsistency
Isolation requirement if between steps 3 and 6, another transaction
T2 is allowed to access the partially updated database, it will see an
inconsistent database (the sum A + B will be less than it should be).
T1 T2
1. read(A)
2. A := A 50
3. write(A)
read(A), read(B), print(A+B)
4. read(B)
5. B := B + 50
6. write(B
Isolation can be ensured trivially by running transactions serially
that is, one after the other.
However, executing multiple transactions concurrently has significant
benefits, as we will see later.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
34
Page: 2 of 2
ACID Properties
A transaction is a unit of program execution that accesses and possibly
updates various data items.To preserve the integrity of data the database
system must ensure:
Atomicity. Either all operations of the transaction are properly reflected
in the database or none are.
Consistency. Execution of a transaction in isolation preserves the
consistency of the database.
Isolation. Although multiple transactions may execute concurrently,
each transaction must be unaware of other concurrently executing
transactions. Intermediate transaction results must be hidden from other
concurrently executed transactions.
That is, for every pair of transactions Ti and Tj, it appears to Ti that
either Tj, finished execution before Ti started, or Tj started
execution after Ti finished.
Durability. After a transaction completes successfully, the changes it
has made to the database persist, even if there are system failures.
System Recovery Media Recovery
Failure Classification
Storage Structure
Recovery and Atomicity
Log-Based Recovery
Shadow Paging
Recovery With Concurrent Transactions
Buffer Management
Failure with Loss of Nonvolatile Storage
Advanced Recovery Techniques
ARIES Recovery Algorithm
Remote Backup Systems
Failure Classification
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
35
Transaction failure :
Logical errors: transaction cannot complete due to some internal
error condition
System errors: the database system must terminate an active
transaction due to an error condition (e.g., deadlock)
System crash: a power failure or other hardware or software failure
causes the system to crash.
Fail-stop assumption: non-volatile storage contents are assumed
to not be corrupted by system crash
Database systems have numerous integrity checks to
prevent corruption of disk data
Disk failure: a head crash or similar disk failure destroys all or part of
disk storage
Destruction is assumed to be detectable: disk drives use checksums
to detect failures
Recovery Algorithms
Recovery algorithms are techniques to ensure database consistency and
transaction atomicity and durability despite failures
Focus of this chapter
Recovery algorithms have two parts
Actions taken during normal transaction processing to ensure
enough information exists to recover from failures
Actions taken after a failure to recover the database contents to a
state that ensures atomicity, consistency and durability
Recovery and Atomicity
Modifying the database without ensuring that the transaction will commit
may leave the database in an inconsistent state.
Consider transaction Ti that transfers $50 from account A to account B;
goal is either to perform all database modifications made by Ti or none at
all.
Several output operations may be required for Ti (to output A and B). A
failure may occur after one of these modifications have been made but
before all of them are made.
To ensure atomicity despite failures, we first output information
describing the modifications to stable storage without modifying the
database itself.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
36
We study two approaches:
log-based recovery, and
shadow-paging
We assume (initially) that transactions run serially, that is, one after the
other.
Two Phase Commit - Save Points
Lock-Based Protocols
A lock is a mechanism to control concurrent access to a data item
Data items can be locked in two modes :
1. exclusive (X) mode. Data item can be both read as well as
written. X-lock is requested using lock-X instruction.
2. shared (S) mode. Data item can only be read. S-lock is
requested using lock-S instruction.
Lock requests are made to concurrency-control manager. Transaction can
proceed only after request is granted.
Lock-compatibility matrix
A transaction may be granted a lock on an item if the requested lock is
compatible with locks already held on the item by other transactions
Any number of transactions can hold shared locks on an item,
but if any transaction holds an exclusive on the item no other
transaction may hold any lock on the item.
If a lock cannot be granted, the requesting transaction is made to wait till
all incompatible locks held by other transactions have been released. The
lock is then granted.
Example of a transaction performing locking:
T2: lock-S(A);
read (A);
unlock(A);
lock-S(B);
read (B);
unlock(B);
display(A+B)
Locking as above is not sufficient to guarantee serializability if A and
B get updated in-between the read of A and B, the displayed sum would
be wrong.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
37
A locking protocol is a set of rules followed by all transactions while
requesting and releasing locks. Locking protocols restrict the set of
possible schedules.
The Two-Phase Locking Protocol
This is a protocol which ensures conflict-serializable schedules.
Phase 1: Growing Phase
transaction may obtain locks
transaction may not release locks
Phase 2: Shrinking Phase
transaction may release locks
transaction may not obtain locks
The protocol assures serializability. It can be proved that the transactions
can be serialized in the order of their lock points (i.e. the point where a
transaction acquired its final lock).
Two-phase locking does not ensure freedom from deadlocks
Cascading roll-back is possible under two-phase locking. To avoid this,
follow a modified protocol called strict two-phase locking. Here a
transaction must hold all its exclusive locks till it commits/aborts.
Rigorous two-phase locking is even stricter: here all locks are held till
commit/abort. In this protocol transactions can be serialized in the order
in which they commit.
There can be conflict serializable schedules that cannot be obtained if
two-phase locking is used.
However, in the absence of extra information (e.g., ordering of access to
data), two-phase locking is needed for conflict serializability in the
following sense:
Given a transaction Ti that does not follow two-phase locking, we can find
a transaction Tj that uses two-phase locking, and a schedule for Ti and Tj
that is not conflict serializable.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
38
Page: 2 of 2
SQL Facilities for recovery
Transaction Definition in SQL
Data manipulation language must include a construct for specifying the
set of actions that comprise a transaction.
In SQL, a transaction begins implicitly.
A transaction in SQL ends by:
Commit work commits current transaction and begins a new one.
Rollback work causes current transaction to abort.
In almost all database systems, by default, every SQL statement also
commits implicitly if it executes successfully
Implicit commit can be turned off by a database directive
E.g. in JDBC, connection.setAutoCommit(false);
Concurrency Need for Concurrency
Timestamp-Based Protocols
Validation-Based Protocols
Multiple Granularity
Multiversion Schemes
Insert and Delete Operations
Concurrency in Index Structures
Lock requests are made to concurrency-control manager. Transaction can proceed
only after request is granted.
Graph-Based Protocols
Graph-based protocols are an alternative to two-phase locking
Impose a partial ordering on the set D = {d1, d2 ,..., dh} of all data items.
If di dj then any transaction accessing both di and dj must access di before
accessing dj.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
39
Implies that the set D may now be viewed as a directed acyclic graph, called a
database graph.
The tree-protocol is a simple kind of graph protocol.
Tree Protocol
1. Only exclusive locks are allowed.
2. The first lock by Ti may be on any data item. Subsequently, a data Q can be
locked by Ti only if the parent of Q is currently locked by Ti.
3. Data items may be unlocked at any time.
4. A data item that has been locked and unlocked by Ti cannot subsequently be
relocked by Ti
Graph-Based Protocols (Cont.)
The tree protocol ensures conflict serializability as well as freedom from deadlock.
Unlocking may occur earlier in the tree-locking protocol than in the two-phase
locking protocol.
shorter waiting times, and increase in concurrency
protocol is deadlock-free, no rollbacks are required
Drawbacks
Protocol does not guarantee recoverability or cascade freedom
Need to introduce commit dependencies to ensure recoverability
Transactions may have to lock data items that they do not access.
increased locking overhead, and additional waiting time
potential decrease in concurrency
Schedules not possible under two-phase locking are possible under tree protocol, and
vice versa.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
40
Page: 2 of 2
Locking Protocols Two Phase Locking Intent Locking
Lock requests are made to concurrency-control manager. Transaction can
proceed only after request is granted.
Lock-compatibility matrix
A transaction may be granted a lock on an item if the requested lock is
compatible with locks already held on the item by other transactions
Any number of transactions can hold shared locks on an item,
but if any transaction holds an exclusive on the item no other
transaction may hold any lock on the item.
If a lock cannot be granted, the requesting transaction is made to wait till
all incompatible locks held by other transactions have been released. The
lock is then granted.
Example of a transaction performing locking:
T2: lock-S(A);
read (A);
unlock(A);
lock-S(B);
read (B);
unlock(B);
display(A+B)
Locking as above is not sufficient to guarantee serializability if A and
B get updated in-between the read of A and B, the displayed sum would
be wrong.
A locking protocol is a set of rules followed by all transactions while
requesting and releasing locks. Locking protocols restrict the set of
possible schedules.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
41
The Two-Phase Locking Protocol
This is a protocol which ensures conflict-serializable schedules.
Phase 1: Growing Phase
transaction may obtain locks
transaction may not release locks
Phase 2: Shrinking Phase
transaction may release locks
transaction may not obtain locks
The protocol assures serializability. It can be proved that the transactions
can be serialized in the order of their lock points (i.e. the point where a
transaction acquired its final lock).
Two-phase locking does not ensure freedom from deadlocks
Cascading roll-back is possible under two-phase locking. To avoid this,
follow a modified protocol called strict two-phase locking. Here a
transaction must hold all its exclusive locks till it commits/aborts.
Rigorous two-phase locking is even stricter: here all locks are held till
commit/abort. In this protocol transactions can be serialized in the order
in which they commit.
There can be conflict serializable schedules that cannot be obtained if
two-phase locking is used.
However, in the absence of extra information (e.g., ordering of access to
data), two-phase locking is needed for conflict serializability in the
following sense:
Given a transaction Ti that does not follow two-phase locking, we can find
a transaction Tj that uses two-phase locking, and a schedule for Ti and Tj
that is not conflict serializable.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
42
Page: 2 of 2
Deadlock- Serializability Recovery Isolation Levels
System is deadlocked if there is a set of transactions such that every
transaction in the set is waiting for another transaction in the set.
Deadlock prevention protocols ensure that the system will never enter
into a deadlock state. Some prevention strategies :
Require that each transaction locks all its data items before it
begins execution (predeclaration).
Impose partial ordering of all data items and require that a
transaction can lock data items only in the order specified by the
partial order (graph-based protocol).
More Deadlock Prevention Strategies
D De ea ad dl lo oc ck k H Ha an nd dl li in ng g
n Consider the following two transactions:
T
1
: write (X) T
2
: write(Y)
write(Y) write(X)
n Schedule with deadlock
T
1
T
2
lock-X on X
write (X)
lock-X on Y
write (X)
wait for lock-X on X
wait for lock-X on Y
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
43
Following schemes use transaction timestamps for the sake of deadlock
prevention alone.
wait-die scheme non-preemptive
older transaction may wait for younger one to release data item.
Younger transactions never wait for older ones; they are rolled
back instead.
a transaction may die several times before acquiring needed data
item
wound-wait scheme preemptive
older transaction wounds (forces rollback) of younger transaction
instead of waiting for it. Younger transactions may wait for older
ones.
may be fewer rollbacks than wait-die scheme.
Both in wait-die and in wound-wait schemes, a rolled back transactions is
restarted with its original timestamp. Older transactions thus have
precedence over newer ones, and starvation is hence avoided.
Timeout-Based Schemes :
a transaction waits for a lock only for a specified amount of time.
After that, the wait times out and the transaction is rolled back.
thus deadlocks are not possible
simple to implement; but starvation is possible. Also difficult to
determine good value of the timeout interval.
Deadlock Detection
Deadlocks can be described as a wait-for graph, which consists of a pair
G = (V,E),
V is a set of vertices (all the transactions in the system)
E is a set of edges; each element is an ordered pair Ti Tj.
If Ti Tj is in E, then there is a directed edge from Ti to Tj, implying
that Ti is waiting for Tj to release a data item.
When Ti requests a data item currently being held by Tj, then the edge Ti
Tj is inserted in the wait-for graph. This edge is removed only when Tj is
no longer holding a data item needed by Ti.
The system is in a deadlock state if and only if the wait-for graph has a
cycle. Must invoke a deadlock-detection algorithm periodically to look
for cycles.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
44
Deadlock Recovery
When deadlock is detected :
Some transaction will have to rolled back (made a victim) to break
deadlock. Select that transaction as victim that will incur
minimum cost.
Rollback -- determine how far to roll back transaction
Total rollback: Abort the transaction and then restart it.
More effective to roll back transaction only as far as
necessary to break deadlock.
Starvation happens if same transaction is always chosen as victim.
Include the number of rollbacks in the cost factor to avoid
starvation
Wait-for graph without a cycle
Wait-for graph with a cycle
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
45
Unit Number & Name: V & IMPLEMENTATION TECHNIQUES Period: 37 of 45
Page: 2 of 2
Overview of Physical Storage Media
Several types of data storage exist in most computer systems. They vary in speed
of access, cost per unit of data, and reliability.
Cache: most costly and fastest form of storage. Usually very small, and managed
by the operating system.
Main Memory (MM): the storage area for data available to be operated on.
General-purpose machine instructions operate on main memory.
Contents of main memory are usually lost in a power failure or ``crash''.
Usually too small (even with megabytes) and too expensive to store the
entire database.
Flash memory: EEPROM (electrically erasable programmable read-only
memory).
Data in flash memory survive from power failure.
Reading data from flash memory takes about 10 nano-secs (roughly as fast as
from main memory), and writing data into flash memory is more
complicated: write-once takes about 4-10 microsecs.
To overwrite what has been written, one has to first erase the entire bank of
the memory. It may support only a limited number of erase cycles
( to ).
It has found its popularity as a replacement for disks for storing small
volumes of data (5-10 megabytes).
Magnetic-disk storage: primary medium for long-term storage.
Typically the entire database is stored on disk.
Data must be moved from disk to main memory in order for the data to be
operated on.
After operations are performed, data must be copied back to disk if any
changes were made.
Disk storage is called direct access storage as it is possible to read data on
the disk in any order (unlike sequential access).
Disk storage usually survives power failures and system crashes.
Optical storage: CD-ROM (compact-disk read-only memory), WORM (write-
once read-many) disk (for archival storage of data), and Juke box (containing a
few drives and numerous disks loaded on demand).
Tape Storage: used primarily for backup and archival data.
Cheaper, but much slower access, since tape must be read sequentially from
the beginning.
Used as protection from disk failures!
The storage device hierarchy is presented in Figure 10.1, where the higher levels
are expensive (cost per bit), fast (access time), but the capacity is smaller.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
46
Figure 10.1: Storage-device hierarchy
Another classification: Primary, secondary, and tertiary storage.
1. Primary storage: the fastest storage media, such as cash and main memory.
2. Secondary (or on-line) storage: the next level of the hierarchy, e.g., magnetic
disks.
3. Tertiary (or off-line) storage: magnetic tapes and optical disk juke boxes.
Volatility of storage. Volatile storage loses its contents when the power is
removed. Without power backup, data in the volatile storage (the part of the
hierarchy from main memory up) must be written to nonvolatile storage for
safekeeping.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
47
Page: 2 of 2
Magnetic Disks RAID
RAID (an acronym for redundant array of independent disks; originally redundant
array of inexpensive disks
[1][2]
) is a storage technology that combines multiple disk
drivecomponents into a logical unit. Data is distributed across the drives in one of several
ways called "RAID levels", depending on what level of redundancy and performance
(via parallel communication) is required.
RAID is an example of storage virtualization and was first defined by David A.
Patterson, Garth A. Gibson, and Randy Katz at the University of California, Berkeley in
1987.
[3]
Marketers representing industry RAID manufacturers later attempted to reinvent
the term to describe a redundant array of independent disks as a means of dissociating a
low-cost expectation from RAID technology.
[4]
RAID is now used as an umbrella term for computer data storage schemes that can divide
and replicate data among multiple physical drives. The physical drives are said to be in a
RAID,
[5]
which is accessed by the operating system as one single drive. The different
schemes or architectures are named by the word RAID followed by a number (e.g., RAID
0, RAID 1). Each scheme provides a different balance between two key goals:
increase data reliability and increase input/output performance.
A number of standard schemes have evolved which are referred to as levels. There were
five RAID levels originally conceived, but many more variations have evolved, notably
severalnested levels and many non-standard levels (mostly proprietary). RAID levels and
their associated data formats are standardised by SNIA in the Common RAID Disk Drive
Format (DDF) standard.
Following is a brief textual summary of the most commonly used RAID levels.
[6]
RAID 0 (block-level striping without parity or mirroring) has no (or zero)
redundancy. It provides improved performance and additional storage but no fault
tolerance. Hence simple stripe sets are normally referred to as RAID 0. Any drive
failure destroys the array, and the likelihood of failure increases with more drives
in the array (at a minimum, catastrophic data loss is almost twice as likely
compared to single drives without RAID). A single drive failure destroys the
entire array because when data is written to a RAID 0 volume, the data is broken
into fragments called blocks. The number of blocks is dictated by the stripe size,
which is a configuration parameter of the array. The blocks are written to their
respective drives simultaneously on the same sector. This allows smaller sections
of the entire chunk of data to be read off the drive in parallel, increasing
bandwidth. RAID 0 does not implement error checking, so any error is
uncorrectable. More drives in the array means higher bandwidth, but greater risk
of data loss.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
48
In RAID 1 (mirroring without parity or striping), data is written identically to
multiple drives, thereby producing a "mirrored set"; at least 2 drives are required
to constitute such an array. While more constituent drives may be employed,
many implementations deal with a maximum of only 2; of course, it might be
possible to use such a limited level 1 RAID itself as a constituent of a level 1
RAID, effectively masking the limitation.
[citation needed]
The array continues to
operate as long as at least one drive is functioning. With appropriate operating
system support, there can be increased read performance, and only a minimal
write performance reduction; implementing RAID 1 with a separate controller for
each drive in order to perform simultaneous reads (and writes) is sometimes
called multiplexing (or duplexing when there are only 2 drives).
In RAID 2 (bit-level striping with dedicated Hamming-code parity), all disk
spindle rotation is synchronized, and data is striped such that each sequential bit is
on a different drive.Hamming-code parity is calculated across corresponding bits
and stored on at least one parity drive.
In RAID 3 (byte-level striping with dedicated parity), all disk spindle rotation is
synchronized, and data is striped so each sequential byte is on a different drive.
Parity is calculated across corresponding bytes and stored on a dedicated parity
drive.
RAID 4 (block-level striping with dedicated parity) is identical to RAID 5 (see
below), but confines all parity data to a single drive. In this setup, files may be
distributed between multiple drives. Each drive operates independently, allowing
I/O requests to be performed in parallel. However, the use of a dedicated parity
drive could create a performancebottleneck; because the parity data must be
written to a single, dedicated parity drive for each block of non-parity data, the
overall write performance may depend a great deal on the performance of this
parity drive.
RAID 5 (block-level striping with distributed parity) distributes parity along with
the data and requires all drives but one to be present to operate; the array is not
destroyed by a single drive failure. Upon drive failure, any subsequent reads can
be calculated from the distributed parity such that the drive failure is masked from
the end user. However, a single drive failure results in reduced performance of the
entire array until the failed drive has been replaced and the associated data rebuilt.
Additionally, there is the potentially disastrousRAID 5 write hole.
RAID 6 (block-level striping with double distributed parity) provides fault
tolerance of two drive failures; the array continues to operate with up to two failed
drives. This makes larger RAID groups more practical, especially for high-
availability systems. This becomes increasingly important as large-capacity drives
lengthen the time needed to recover from the failure of a single drive. Single-
parity RAID levels are as vulnerable to data loss as a RAID 0 array until the failed
drive is replaced and its data rebuilt; the larger the drive, the longer the rebuild
takes. Double parity gives additional time to rebuild the array without the data
being at risk if a single additional drive fails before the rebuild is complete.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
49
Page: 2 of 2
Tertiary storage File Organization Organization of Records in Files
Optical Disks
1. CD-ROM has become a popular medium for distributing software, multimedia
data, and other electronic published information.
2. Capacity of CD-ROM: 500 MB. Disks are cheap to mass produce and also
drives.
3. CD-ROM: much longer seek time (250m-sec), lower rotation speed (400 rpm),
leading to high latency and lower data-transfer rate (about 150 KB/sec). Drives
spins at audio CD spin speed (standard) is available.
4. Recently, a new optical format, digit video disk (DVD) has become standard.
These disks hold between 4.7 and 17 GB data.
5. WORM (write-once, read many) disks are popular for archival storage of data
since they have a high capacity (about 500 MB), longer life time than HD, and
can be removed from drive -- good for audit trail (hard to tamper).
Magnetic Tapes
6. Long history, slow, and limited to sequential access, and thus are used for
backup, storage for infrequent access, and off-line medium for system transfer.
7. Moving to the correct spot may take minutes, but once positioned, tape drives
can write data at density and speed approaching to those of disk drives.
8. 8mm tape drive has the highest density, and we store 5 GB data on a 350-foot
tape.
9. Popularly used for storage of large volumes of data, such as video, image, or
remote sensing data.
File organization is the methodology which is applied to structured computer files. Files
contain computer records which can be documents or information which is stored in a
certain way for later retrieval. File organization refers primarily to the logical
arrangement of data (which can itself be organized in a system of records with correlation
between the fields/columns) in a file system. It should not be confused with the physical
storage of the file in some types of storage media. There are certain basic types of
computer file, which can include files stored as blocks of data and streams of data, where
the information streams out of the file while it is being read until the end of the file is
encountered.
We will look at two components of file organization here:
1. The way the internal file structure is arranged and
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
50
2. The external file as it is presented to the O/S or program that calls it. Here we
will also examine the concept of file extensions.
We will examine various ways that files can be stored and organized. Files are presented
to the application as a stream of bytes and then an EOF (end of file) condition.
A program that uses a file needs to know the structure of the file and needs to interpret its
contents.
Internal File Structure
Methods and Design Paradigm
It is a high-level design decision to specify a system of file organization for a computer
software program or a computer system designed for a particular purpose. Performance is
high on the list of priorities for this design process, depending on how the file is being
used. The design of the file organization usually depends mainly on the system
environment. For instance, factors such as whether the file is going to be used for
transaction-oriented processes like OLTP or Data Warehousing, or whether the file is
shared among various processes like those found in a typical distributed system or
standalone. It must also be asked whether the file is on a network and used by a number
of users and whether it may be accessed internally or remotely and how often it is
accessed.
However, all things considered the most important considerations might be:
1. Rapid access to a record or a number of records which are related to each other.
2. The Adding, modification, or deletion of records.
3. Efficiency of storage and retrieval of records.
4. Redundancy, being the method of ensuring data integrity.
A file should be organized in such a way that the records are always available for
processing with no delay. This should be done in line with the activity and volatility of
the information.
Types of File Organization
Organizing a file depends on what kind of file it happens to be: a file in the simplest form
can be a text file, (in other words a file which is composed of ascii (American Standard
Code for Information Interchange) text.) Files can also be created as binary or executable
types (containing elements other than plain text.) Also, files are keyed with attributes
which help determine their use by the host operating system.
Techniques of File Organization
The three techniques of file organization are:
1. Heap (unordered)
2. Sorted
1. Sequential (SAM)
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
51
2. Line Sequential (LSAM)
3. Indexed Sequential (ISAM)
3. Hashed or Direct
In addition to the three techniques, there are four methods of organizing files. They
aresequential, line-sequential, indexed-sequential, inverted list and direct or hashed
accessorganization.
Sequential Organization
A sequential file contains records organized in the order they were entered. The order of
the records is fixed. The records are stored and sorted in physical, contiguous blocks
within each block the records are in sequence.
Records in these files can only be read or written sequentially.
Once stored in the file, the record cannot be made shorter, or longer, or deleted. However,
the record can be updated if the length does not change. (This is done by replacing the
records by creating a new file.) New records will always appear at the end of the file.
If the order of the records in a file is not important, sequential organization will
suffice, no matter how many records you may have. Sequential output is also useful for
report printing orsequential reads which some programs prefer to do.
Line-Sequential Organization
Line-sequential files are like sequential files, except that the records can contain only
characters as data. Line-sequential files are maintained by the native byte stream files of
the operating system.
In the COBOL environment, line-sequential files that are created with WRITE statements
with the ADVANCING phrase can be directed to a printer as well as to a disk.
Indexed-Sequential Organization
Key searches are improved by this system too. The single-level indexing structure is the
simplest one where a file, whose records are pairs, contains a key pointer. This pointer is
the position in the data file of the record with the given key. A subset of the records,
which are evenly spaced along the data file, is indexed, in order to mark intervals of data
records.
This is how a key search is performed: the search key is compared with the index keys to
find the highest index key coming in front of the search key, while a linear search is
performed from the record that the index key points to, until the search key is matched or
until the record pointed to by the next index entry is reached. Regardless of double file
access (index + data) required by this sort of search, the access time reduction is
significant compared with sequential file searches.
Let's examine, for sake of example, a simple linear search on a 1,000 record sequentially
organized file. An average of 500 key comparisons are needed (and this assumes the
search keys are uniformly distributed among the data keys). However, using an index
evenly spaced with 100 entries, the total number of comparisons is reduced to 50 in the
index file plus 50 in the data file: a five to one reduction in the operations count!
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
52
Hierarchical extension of this scheme is possible since an index is a sequential file in
itself, capable of indexing in turn by another second-level index, and so forth and so on.
And the exploit of the hierarchical decomposition of the searches more and more, to
decrease the access time will pay increasing dividends in the reduction of processing
time. There is however a point when this advantage starts to be reduced by the increased
cost of storage and this in turn will increase the index access time.
Hardware for Index-Sequential Organization is usually Disk-based, rather than tape.
Records are physically ordered by primary key. And the index gives the physical location
of each record. Records can be accessed sequentially or directly, via the index. The index
is stored in a file and read into memory at the point when the file is opened. Also, indexes
must be maintained.
Life sequential organization the data is stored in physical contiguous box. How ever the
difference is in the use of indexes. There are three areas in the disc storage:
Primary Area:-Contains file records stored by key or ID numbers.
Overflow Area:-Contains records area that cannot be placed in primary area.
Index Area:-It contains keys of records and there locations on the disc.
Inverted List
In file organization, this is a file that is indexed on many of the attributes of the data
itself. The inverted list method has a single index for each key type. The records are not
necessarily stored in a sequence. They are placed in the are data storage area, but indexes
are updated for the record keys and location.
Here's an example, in a company file, an index could be maintained for all products,
another one might be maintained for product types. Thus, it is faster to search the indexes
than every record. These types of file are also known as "inverted
indexes." Nevertheless, inverted list files use more media space and the storage devices
get full quickly with this type of organization. The benefits are apparent immediately
because searching is fast. However, updating is much slower.
Content-based queries in text retrieval systems use inverted indexes as their preferred
mechanism. Data items in these systems are usually stored compressed which would
normally slow the retrieval process, but the compression algorithm will be chosen to
support this technique.
When querying a file there are certain circumstances when the query is designed to
be modalwhich means that rules are set which require that different information be held
in the index. Here's an example of this modality: when phrase querying is undertaken, the
particular algorithm requires that offsets to word classifications are held in addition to
document numbers.
]Direct or Hashed Access
With direct or hashed access a portion of disk space is reserved and a hashing
algorithm computes the record address. So there is additional space required for this kind
of file in the store. Records are placed randomly through out the file. Records are
accessed by addresses that specify their disc location. Also, this type of file organization
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
53
requires a disk storage rather than tape. It has an excellent search retrieval performance,
but care must be taken to maintain the indexes. If the indexes become corrupt, what is left
may as well go to the bit-bucket, so it is as well to have regular backups of this kind of
file just as it is for all stored valuable data!
External File Structure and File Extensions
Microsoft Windows and MS-DOS File Systems
The external structure of a file depends on whether it is being created on
a FAT or NTFSpartition. The maximum filename length on a NTFS partition is 256
characters, and 11 characters on FAT (8 character name+"."+3 character
extension.) NTFS filenames keep their case, whereas FAT filenames have no concept of
case (but case is ignored when performing a search under NTFS Operating System).
Also, there is the new VFAT which permits 256 character filenames.
UNIX and Apple Macintosh File Systems
The concept of directories and files is fundamental to the UNIX operating system.
On Microsoft Windows-based operating systems, directories are depicted as folders and
moving about is accomplished by clicking on the different icons. In UNIX, the directories
are arranged as a hierarchy with the root directory being at the top of the tree.
The root directory is always depicted as /. Within the / directory, there are subdirectories
(e.g.: etc and sys). Files can be written to any directory depending on the permissions.
Files can be readable, writable and/orexecutable.
Organizing files using Libraries
With the advent of Microsoft Windows 7 the concept of file organization and
management has improved drastically by way of use of powerful tool called Libraries. A
Library is file organization system to bring together related files and folders stored in
different locations of the local as well as network computer such that these can be
accessed centrally through a single access point. For instance, various images stored in
different folders in the local computer or/and across a computer network can be
accumulated in an Image Library. Aggregation of similar files can be manipulated, sorted
or accessed conveniently as and when required through a single access point on a
computer desktop by use of a Library. This feature is particularly very useful for
accessing similar content of related content, and also, for managing projects using related
and common data.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
54
Page: 2 of 2
Indexing and Hashing Ordered Indices
Indexing mechanisms used to speed up access to desired
data.
E.g. author catalog in library
_ Search key attribute or set of attributes used to look up
records in a le.
_ An index le consists of records (called index entries) of the
form
search-key pointer
_ Index les are typically much smaller than the original le
_ Two basic kinds of indices:
Ordered indices: search keys are stored in sorted order
Hash indices: search keys are distributed uniformly across
buckets using a hash function
Index Evaluation Metrics
Indexing techniques evaluated on basis of:
_ Access types supported efciently. E.g.,
records with a specied value in an attribute
or records with an attribute value falling in a specied range
of values.
_ Access time
_ Insertion time
_ Deletion time
_ Space overhead
Ordered Indices
_ In an ordered index, index entries are stored sorted on the
search key value. E.g., author catalog in library.
_ Primary index: in a sequentially ordered le, the index whose
search key species the sequential order of the le.
Also called clustering index
The search key of a primary index is usually but not
necessarily the primary key.
_ Secondary index: an index whose search key species an
order different from the sequential order of the le. Also called
non-clustering index.
_ Index-sequential le: ordered sequential le with a primary
index.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
55
Page: 2 of 2
B+ tree Index Files B tree Index Files
B+
-tree indices are an alternative to indexed-sequential les.
_ Disadvantage of indexed-sequential les: performance
degrades as le grows, since many overow blocks get
created. Periodic reorganization of entire le is required.
_ Advantage of B+
-tree index les: automatically reorganizes
itself with small, local, changes, in the face of insertions and
deletions. Reorganization of entire le is not required to
maintain performance.
_ Disadvantage of B+
-trees: extra insertion and deletion
overhead, space overhead.
_ Advantages of B+
-trees outweigh disadvantages, and they are
used extensively.
A B+
-tree is a rooted tree satisfying the following properties:
_ All paths from root to leaf are of the same length
_ Each node that is not a root or a leaf has between dn/ 2e and n
children.
_ A leaf node has between d(n 1)/ 2e and n 1 values
_ Special cases: if the root is not a leaf, it has at least 2 children.
If the root is a leaf (that is, there are no other nodes in the
tree), it can have between 0 and (n 1) values.
B+
-Tree Node Structure
_ Typical node
P1 K1 P2 . . . Pn1 Kn1 Pn
Ki are the search-key values
Pi are pointers to children (for non-leaf nodes) or pointers to
records or buckets of records (for leaf nodes).
_ The search-keys in a node are ordered
K1 < K2 < K3 < ... < Kn 1
Example of a B+
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
56
-tree
Perryridge
Mianus Redwood
Brighton Downtown Mianus Perryridge Redwood Round Hill
B
+
-tree for account le (n = 3)
Static Hashing Dynamic Hashing
Static Hashing
_ A bucket is a unit of storage containing one or more records (a
bucket is typically a disk block). In a hash le organization we
obtain the bucket of a record directly from its search-key value
using a hash function.
_ Hash function h is a function from the set of all search-key
values K to the set of all bucket addresses B.
_ Hash function is used to locate records for access, insertion as
well as deletion.
_ Records with different search-key values may be mapped to
the same bucket; thus entire bucket has to be searched
sequentially to locate a record.
Dynamic Hashing
_ Good for database that grows and shrinks in size
_ Allows the hash function to be modied dynamically
_ Extendable hashing one form of dynamic hashing
Hash function generates values over a large range
typically b-bit integers, with b = 32.
At any time use only a prex of the hash function to index
into a table of bucket addresses. Let the length of the prex
be i bits, 0 _ i _ 32
Initially i = 0
Value of i grows and shrinks as the size of the database
grows and shrinks.
Actual number of buckets is < 2
i, and this also changes
dynamically due to coalescing and splitting of buckets.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
57
Page: 2 of 2
Query Processing Overview Catalog Information for Cost Estimation
Basic Steps in Query Processing
1. Parsing and translation
2. Optimization
3. Evaluation
Query
Parser &
Translator
Relational Algebra
Expression
Optimizer
Query
Evaluation Engine Execution Plan
Output
Data Statistics
About Data
Parsing and translation
_ translate the query into its internal form. This is then translated
into relational algebra.
_ Parser checks syntax, veries relations
Evaluation
_ The query-execution engine takes a query-evaluation plan,
executes that plan, and returns the answers to the query.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
58
Page: 2 of 2
Query Processing Overview Catalog Information for Cost Estimation
Statistical Information for Cost Estimation
nr: number of tuples in a relation r.
br: number of blocks containing tuples of r.
sr: size of a tuple of r.
fr: blocking factor of r i.e., the number of tuples of r that fit into one block.
V(A, r): number of distinct values that appear in r for attribute A; same as the size of A(r).
SC(A, r): selection cardinality of attribute A of relation r; average number of records that satisfy eq
uality on A.
If tuples of r are stored together physically in a file, then:
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
59
Page: 2 of 2
Selection Operation Sorting Join Operation Database Tuning
Database tuning describes a group of activities used to optimize and homogenize the
performance of a database. It usually overlaps with query tuning, but refers to design of
the database files, selection of the database management system (DBMS), operating
system and CPU the DBMS runs on.
The goal is to maximize use of system resources to perform work as efficiently and
rapidly as possible. Most systems are designed to manage work efficiently, but it is
possible to greatly improve performance by customizing settings and the configuration
for the database and the DBMS being tuned.
I/O tuning
Hardware and software configuration of disk subsystems are examined: RAID levels and
configuration
[1]
, block and stripe size allocation, and the configuration of
disks, controller cards, storage cabinets, and external storage systems such as a SAN.
Transaction logs and temporary spaces are heavy consumers of I/O, and affect
performance for all users of the database. Placing them appropriately is crucial.
Frequently joined tables and indexes are placed so that as they are requested from file
storage, they can be retrieved in parallel from separate disks simultaneously. Frequently
accessed tables and indexes are placed on separate disks to balance I/O and prevent read
queuing.
DBMS tuning
DBMS tuning refers to tuning of the DBMS and the configuration of the memory and
processing resources of the computer running the DBMS. This is typically done through
configuring the DBMS, but the resources involved are shared with the host system.
Tuning the DBMS can involve setting the recovery interval (time needed to restore the
state of data to a particular point in time), assigning parallelism (the breaking up of work
from a single query into tasks assigned to different processing resources), and network
protocols used to communicate with database consumers.
Memory is allocated for data, execution plans, procedure cache, and work space. It is
much faster to access data in memory than data on storage, so maintaining a
sizable cache of data makes activities perform faster. The same consideration is given to
work space. Caching execution plans and procedures means that they are reused instead
of recompiled when needed. It is important to take as much memory as possible, while
leaving enough for other processes and the OS to use without excessive paging of
memory to storage.
h
t
t
p
:
/
/
c
s
e
t
u
b
e
.
c
o
.
n
r
/
GKMCET
Lecture Plan
60
Processing resources are sometimes assigned to specific activities to
improve concurrency. On a server with eight processors, six could be reserved for the
DBMS to maximize available processing resources for the database.

DataBase Management Systems

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

DataBase Management Systems

Hochgeladen von

Copyright:

Verfügbare Formate

h

Das könnte Ihnen auch gefallen