Query processing and optimization

Definitions
Query processing
translation of query into low-level activities
evaluation of query
data extraction

Query optimization
selecting the most efficient query evaluation

Advanced D

Query processing an

Query Processing (1/2)

SELECT * FROM student WHERE name=Paul
Parse query and translate
check syntax, verify names, etc
translate into relational algebra (RDBMS)
create evaluation plans

Find best plan (optimization)

Execute plan
student

takes

course

cid

name

cid

courseid

courseid

coursename

00112233

Paul

00112233

312

312

Advanced DBs

00112238

Rob

00112233

395

395

Machine Learning

00112235

Matt

00112235

312

Advanced D

Query processing an

Query Processing (2/2)

query

parser and
translator

relational algebra
expression

optimizer

evaluation
engine

output

data

Advanced D

evaluation plan

data

Query processing an

data
statistics

Relational Algebra (1/2)

Query language
Operations:

select:
project:
union:
difference: product: x
join:

Advanced D

Query processing an

Relational Algebra (2/2)

SELECT * FROM student WHERE name=Paul
name=Paul(student)

name( cid<00112235(student) )
name(coursename=Advanced DBs((student

student

cid

takes)

courseid

takes

course) )

course

cid

name

cid

courseid

courseid

coursename

00112233

Paul

00112233

312

312

Advanced DBs

00112238

Rob

00112233

395

395

Machine Learning

00112235

Matt

00112235

312

Advanced D

Query processing an

Why Optimize?
Many alternative options to evaluate a query
name(coursename=Advanced DBs((student cid takes) courseid course) )
name((student cid takes) courseid coursename=Advanced DBs(course)) )

Several options to evaluate a single operation

name=Paul(student)
scan file
use secondary index on student.name

Multiple access paths

access path: how can records be accessed

Advanced D

Query processing an

Evaluation plans

Specify which access path to follow

Specify which algorithm to use to evaluate operator
Specify how operators interleave
name
Optimization:
estimate the cost of each plan (not all plans)
select plan with lowest estimated cost

coursename=Advanced DBs l

name=Paul ; use index i

courseid; index-nested
loop

student

name=Paul
cid; hash join

student

Advanced D

student

course

takes

Query processing an

Estimating Cost
What needs to be considered:
Disk I/Os
sequential
random

CPU time
Network communication

What are we going to consider:

Disk I/Os
page reads/writes

Ignoring cost of writing final output

Advanced D

Query processing an

Operations and Costs

Operations and Costs (1/2)

Operations: , , , , -, x,
Costs:
NR: number of records in R
LR: size of record in R
FR: blocking factor
number of records in page

BR: number of pages to store relation R

V(A,R): number of distinct values of attribute A in R
SC(A,R): selection cardinality of A in R
A key: S(A,R)=1
A nonkey: S(A,R)= NR / V(A,R)

HTi: number of levels in index I

rounding up fractions and logarithms

Advanced D

Query processing an

11

Selection (1/2)
Linear search
read all pages, find records that match (assuming equality search)
average cost:
nonkey BR, key 0.5*BR

Binary search
on ordered field
average cost: log 2 BR m
m additional pages to be read
m = ceil( SC(A,R)/FR ) - 1

Primary/Clustered
Index

average cost:
single record HTi + 1
multiple records HTi + ceil( SC(A,R)/FR )

Advanced D

Query processing an

13

Selection (2/2)
Secondary Index
average cost:
key field HTi + 1
nonkey field
worst case HTi + SC(A,R)
linear search more desirable if many matching records

Advanced D

Query processing an

14

Complex selection expr

conjunctive selections:

1 2 ... n

perform simple selection using i with the lowest evaluation cost

e.g. using an index corresponding to i

apply remaining conditions on the resulting records

cid 00112233courseid 312 (takes)

cost: the cost of the simple selection on selected

multiple indices

select indices that correspond to is

scan indices and return RIDs
answer: intersection of RIDs
cost: the sum of costs + record retrieval

disjunctive selections:

1 2 ... n

multiple indices
union of RIDs

linear search

Advanced D

Query processing an

15

Projection and set operations

SELECT DISTINCT cid FROM takes
requires duplicate elimination
sorting

set operations require duplicate elimination

RS
RS
sorting

Advanced D

Query processing an

16

Sorting
efficient evaluation for many operations
required by query:
SELECT cid,name FROM student ORDER BY name

implementations
internal sorting (if records fit in memory)
external sorting

Advanced D

Query processing an

17

External Sort-Merge Algorithm (1/3)

Sort stage: create sorted runs
i=0;
repeat
read M pages of relation R into memory
sort the M pages
write them into file Ri
increment i
until no more pages
N=i
// number of runs

Advanced D

Query processing an

18

External Sort-Merge Algorithm (2/3)

Merge stage: merge sorted runs
//assuming N < M
allocate a page for each run file Ri

// N pages allocated

read a page Pi of each Ri

repeat
choose first record (in sort order) among N pages, say from page Pj
write record to output and delete from page Pj
if page is empty read next page Pj from Rj
until all pages are empty

Advanced D

Query processing an

19

External Sort-Merge Algorithm (3/3)

Merge stage: merge sorted runs
What if N > M ?

perform multiple passes

each pass merges M-1 runs until relation is processed
in next pass number of runs is reduced
final pass generated sorted output

Advanced D

Query processing an

20

Sort-Merge Example
a 12

d 95
R1

a 12
x 44
s 95

file

d 95

o 73

a 12

t 45

x 44

n 67

x 44

R2

memory

e 87
z 11
v 22
b 38

Advanced D

R3

o 73

run
a 12

d 95

f 12

f 12

pass

a 95
d
12
f 12
a 95
d
12

d 95

e 87

s 95

e 87

b 38

n 67

e 87

t 45

n 67

Query processing an

d 95

o 73
x 44

R4 v 22
z 11

b 38

f 12

s 95

b 38

a 12

t 45
v 22
z 11

pass

f 12
n 67
o 73
s 95
t 45
v 22
x 44
z 11

21

Sort-Merge cost
BR the number of pages of R
Sort stage: 2 * BR
read/write relation

Merge stage:

BR
initially M runs to be merged

each pass M-1 runs sorted

BR
log
thus, total number of passes: M 1 M

at each pass 2 * BR pages are read

read/write relation
apart from final write

Total cost:
2 * BR + 2 * B R *

BR
log

M 1
M

BR

Advanced D

Query processing an

22

Projection
1,2 (R)
remove unwanted attributes
scan and drop attributes

remove duplicate records

sort resulting records using all attributes as sort order
scan sorted result, eliminate duplicates (adjucent)

cost
initial scan + sorting + final scan

Advanced D

Query processing an

23

Join
name(coursename=Advanced DBs((student
implementations

cid

takes)

courseid

course) )

nested loop join

block-nested loop join
indexed nested loop join
sort-merge join
hash join

Advanced D

Query processing an

24

Nested loop join (1/2)

R

for each tuple tR of R

for each tS of S
if (tR tS match) output tR.tS
end
end
Works for any join condition
S inner relation
R outer relation

Advanced D

Query processing an

25

Nested loop join (2/2)

Costs:
best case when smaller relation fits in memory
use it as inner relation
BR+BS

worst case when memory holds one page of each relation

S scanned for each tuple in R
NR * Bs + BR

Advanced D

Query processing an

26

Block nested loop join (1/2)

for each page XR of R
foreach page XS of S
for each tuple tR in XR
for each tS in XS
if (tR tS match) output tR.tS
end
end
end
end

Advanced D

Query processing an

27

Block nested loop join (2/2)

Costs:
best case when smaller relation fits in memory
use it as inner relation
BR+BS

worst case when memory holds one page of each relation

S scanned for each page in R
BR * Bs + BR

Advanced D

Query processing an

28

Indexed nested loop join

R S
Index on inner relation (S)
for each tuple in outer relation (R) probe index of inner relation
Costs:
BR + NR * c
c the cost of index-based selection of inner relation

relation with fewer records as outer relation

Advanced D

Query processing an

29

Sort-merge join
R S
Relations sorted on the join attribute
Merge sorted relations
pointers to first record in each relation
read in a group of records of S with the same values in the join
attribute
read records of R and process
d D
e 67

Relations in sorted order to be read once

Cost:
cost of sorting + BS + BR

e E

e 87

x X

n 11

v V

v 22
z 38

Advanced D

Query processing an

30

Hash join

R S
use h1 on joining attribute to map records to partitions that fit in memory
records of R are partitioned into R0 Rn-1
records of S are partitioned into S0 Sn-1

join records in corresponding partitions

using a hash-based indexed block nested loop join

Cost: 2*(BR+BS) + (BR+BS)

Advanced D

R0

S0

R1

S1

Rn-1

Sn-1

Query processing an

31

Exercise: joins
R S
NR=215
BR = 100
NS=26
BS = 30
B+ index on S
order 4
full nodes

nested loop join: best case - worst case

block nested loop join: best case - worst case
indexed nested loop join

Advanced Databases

Query processing and optimization

32

Evaluation
evaluate multiple operations in a plan
materialization
pipelining

name
coursename=Advanced DBs

courseid; index-nested
loop
cid; hash join

student

Advanced D

course

takes

Query processing an

33

Materialization
create and read temporary relations
create implies writing to disk
more page writes

name
coursename=Advanced DBs

courseid; index-nested
loop
cid; hash join

student

Advanced D

course

takes

Query processing an

34

Pipelining (1/2)
creating a pipeline of operations
reduces number of read-write operations
implementations
demand-driven - data pull
producer-driven - data push

name
coursename=Advanced DBs

ccourseid; index-nested
loop
cid; hash join

student

Advanced D

course

takes

Query processing an

35

Pipelining (2/2)
can pipelining always be used?
any algorithm?
cost of R S
materialization and hash join: BR + 3(BR+BS)
pipelining and indexed nested loop join: NR * HTi
courseid

pipelined

materialized

R
cid

student

Advanced D

takes

coursename=Advanced DBs
course

Query processing an

36

Query Optimization

Choosing evaluation plans

cost based optimization
enumeration of plans
R

T, 12 possible orders

cost estimation of each plan

overall cost
cannot optimize operation independently

Advanced D

Query processing an

38

Cost estimation

operation (, ,
implementation
size of inputs
size of outputs
sorting

)
name
coursename=Advanced DBs

courseid; index-nested
loop
cid; hash join

student

Advanced D

course

takes

Query processing an

39

Size Estimation (1/2)

A v (R)
SC(A,R)

A v (R)

v min(A,R)
max(A,R) min(A,R)

1 2 ... n (R)

NR *

multiplying probabilities
N R *[(s1 N R ) *(s2 N R ) *...(sn N R )]

1 2 v... n (R)

probability that a record satisfy none of :

[(1 s1 N R ) *(1 s2 N R ) *...* (1 sn N R )]
N R * (1 [(1 s1 N R ) *(1 s2 N R ) *...* (1 sn N R )])

Advanced D

Query processing an

40

Size Estimation (2/2)

RxS
NR * NS

R S
R S = : NR* NS
R S key for R: maximum output size is Ns
R S foreign key for R: NS
R S = {A}, neither key of R nor S
NR*NS / V(A,S)
NS*NR / V(A,R)

Advanced D

Query processing an

41

Expression Equivalence
conjunctive selection decomposition

1 2

(R) 1 ( 2 (R))

commutativity of selection

( (R)) ( (R))
1

combining selection with join and product

1(R x S) = R

commutativity of joins
R

S=S

distribution of selection over join

1^2(R S) = 1(R)

2 (S)

distribution of projection over join

A1,A2(R S) = A1(R)

associativity of joins: R

Advanced D

A2 (S)

(S

T) = (R

S)

Query processing an

42

Cost Optimizer (1/2)

transforms expressions
equivalent expressions
heuristics, rules of thumb

perform selections early

perform projections early
replace products followed by selection (R x S) with joins R
start with joins, selections with smallest result

create left-deep join trees

Advanced D

Query processing an

43

Cost Optimizer (2/2)

name
name

coursename=Advanced DBs

cid; hash join

student

takes

Advanced D

ccourseid; index-nested

ccourseid; index-nested

loop

loop

course
student

cid; hash join

takes

Query processing an

coursenam =
Advanced DBs

course

44

Cost Evaluation Exercise

name(coursename=Advanced DBs((student cid takes) courseid course) )
R = student cid takes
S = course
NS = 10 records
assume that on average there are 50 students taking each
course
blocking factor: 2 records/page
what is the cost of coursename=Advanced DBs (R courseid S)
what is the cost of R coursename=Advanced DBsS
assume relations can fit in memory

Advanced Databases

Query processing and optimization

45

Summary
Estimating the cost of a single operation
Estimating the cost of a query plan
Optimization
choose the most efficient plan

Advanced D

Query processing an

46