Module 01

Logical Query Processing

Overview
• Logical Query Processing Order
• Logical Query Processing Example
• Phase Details

© 2003 - 2019 Itzik Ben-Gan 2

Logical Query Processing Order

5 SELECT 5.2 DISTINCT 7 TOP <TOP_spec>

5.1 <select_list>

1 FROM <table_operators:
JOIN, APPLY, PIVOT, UNPIVOT>
2
WHERE <predicate>
3
GROUP BY <definition_of_grouping_sets>
HAVING <predicate>
4
ORDER BY <order_by_list>
6
OFFSET <offset_spec> FETCH <fetch_spec>
7

© 2003 - 2019 Itzik Ben-Gan 3

Logical Query Processing Example
Customers
Customers Return the customer ID and number of
custid
custid city
city
----------
---------- ----------
---------- orders for customers from Madrid who
FISSA Madrid
FISSA
FRNDO
FRNDO
Madrid
Madrid
Madrid
placed fewer than 3 orders.
KRLOS
KRLOS
MRPHS
Madrid
Madrid
Zion
Sort the result by the number of
MRPHS Zion
orders.
SELECT
SELECT
Orders
Orders C.custid,
C.custid,
orderid
orderid custid
custid COUNT(O.orderid)
COUNT(O.orderid) AS AS numorders
numorders
-----------
----------- ----------
---------- FROM
FROM dbo.Customers
dbo.Customers ASAS CC
11 FRNDO
FRNDO LEFT
LEFT OUTER
OUTER JOIN
JOIN dbo.Orders
dbo.Orders AS
AS OO
22 FRNDO
FRNDO ON
ON C.custid
C.custid == O.custid
O.custid
33 KRLOS
KRLOS WHERE
WHERE C.city
C.city == 'Madrid'
'Madrid'
44 KRLOS
KRLOS GROUP
GROUP BYBY C.custid
C.custid
55 KRLOS
KRLOS HAVING
HAVING COUNT(O.orderid)
COUNT(O.orderid) << 33
66 MRPHS
MRPHS ORDER
ORDER BYBY numorders;
numorders;
77 <NULL>
<NULL>
custid
custid numorders
numorders
----------
---------- -----------
-----------
FISSA
FISSA 00
FRNDO
FRNDO 22

© 2003 - 2019 Itzik Ben-Gan 4

1. FROM <table_operators>
JOIN Sub-Phase 1: Cartesian Product
Customers C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
Customers
------------
------------ ------ --------- ------------
------ --------- ------------
custid
custid city
city FISSA
FISSA Madrid
Madrid 11 FRNDO
FRNDO
----------
---------- ----------
---------- FISSA
FISSA Madrid
Madrid 22 FRNDO
FRNDO
FISSA Madrid FISSA
FISSA Madrid
Madrid 33 KRLOS
KRLOS
FISSA Madrid FISSA Madrid 44 KRLOS
FISSA Madrid KRLOS
FRNDO
FRNDO Madrid
Madrid FISSA
FISSA Madrid
Madrid 55 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid FISSA
FISSA Madrid
Madrid 6 MRPHS
6 MRPHS
MRPHS Zion FISSA
FISSA Madrid
Madrid 77 <NULL>
<NULL>
MRPHS Zion
FRNDO
FRNDO Madrid
Madrid 11 FRNDO
FRNDO
FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
Orders
Orders FRNDO
FRNDO Madrid
Madrid 33 KRLOS
KRLOS
FRNDO
FRNDO Madrid
Madrid 44 KRLOS
KRLOS
orderid
orderid custid
custid FRNDO Madrid 55 KRLOS
FRNDO Madrid KRLOS
-----------
----------- ----------
---------- FRNDO
FRNDO Madrid
Madrid 6 MRPHS
6 MRPHS
11 FRNDO
FRNDO FRNDO
FRNDO Madrid
Madrid 77 <NULL>
<NULL>
KRLOS
KRLOS Madrid
Madrid 11 FRNDO
FRNDO
22 FRNDO
FRNDO KRLOS Madrid 22 FRNDO
KRLOS Madrid FRNDO
33 KRLOS
KRLOS KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
44 KRLOS
KRLOS KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
55 KRLOS
KRLOS KRLOS Madrid 6 MRPHS
KRLOS Madrid 6 MRPHS
66 MRPHS
MRPHS KRLOS
KRLOS Madrid
Madrid 77 <NULL>
<NULL>
77 <NULL>
<NULL> MRPHS
MRPHS Zion
Zion 11 FRNDO
FRNDO
MRPHS
MRPHS Zion
Zion 22 FRNDO
FRNDO
MRPHS
MRPHS Zion
Zion 33 KRLOS
KRLOS
FROM
FROM dbo.Customers
dbo.Customers AS
AS CC MRPHS
MRPHS Zion
Zion 44 KRLOS
KRLOS
MRPHS
MRPHS Zion
Zion 55 KRLOS
KRLOS
...
... JOIN
JOIN dbo.Orders
dbo.Orders AS
AS OO MRPHS
MRPHS Zion
Zion 6 MRPHS
6 MRPHS
MRPHS
MRPHS Zion
Zion 77 <NULL>
<NULL>

© 2003 - 2019 Itzik Ben-Gan 5

JOIN Sub-Phase 1
Cartesian Product
• A cross join is performed between left and right tables
• Result table contains all columns from both tables
• Column names in result table prefixed with:
– Full table name if table alias wasn’t provided
– Table alias if such was provided
– If table alias was provided, using full table name as column prefix is not
allowed in subsequent phases
• Must specify column prefix if column name is ambiguous

© 2003 - 2019 Itzik Ben-Gan 6

JOIN Sub-Phase 2
Apply ON Filter
Match?
Match? C.custid
C.custid C.city
C.city O.orderid
C.city
O.orderid
C.city O.custid
O.orderid O.custid
O.custid
O.orderid O.custid
------------
------------ ------
------ ---------
--------- ------------
------------ ON
ON C.custid
C.custid
TRUE
FALSE
TRUE
FALSE FRNDO
FISSA
FRNDO
FISSA Madrid
Madrid 11 FRNDO
FRNDO
TRUE
FALSE FRNDO
FISSA Madrid 2 FRNDO
== O.custid
O.custid
TRUE
FALSE FRNDO
FISSA Madrid 2 FRNDO
TRUE
FALSE
TRUE
FALSE KRLOS
FISSA
KRLOS
FISSA Madrid
Madrid 3 KRLOS
3 KRLOS
TRUE
FALSE
TRUE
FALSE KRLOS
FISSA
KRLOS
FISSA Madrid
Madrid 44 KRLOS
KRLOS
TRUE
FALSE
TRUE
FALSE KRLOS
FISSA
KRLOS
FISSA Madrid
Madrid 55 KRLOS
KRLOS
TRUE
FALSE
TRUE
FALSE MRPHS
FISSA
MRPHS
FISSA Zion
Madrid
Zion
Madrid 66 MRPHS
MRPHS
UNKNOWN
UNKNOWN FISSA
FISSA Madrid
Madrid 77 <NULL>
<NULL>
TRUE
TRUE FRNDO
FRNDO Madrid
Madrid 1 FRNDO
1 FRNDO
TRUE
TRUE FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
FALSE
FALSE FRNDO
FRNDO Madrid
Madrid 33 KRLOS
KRLOS
FALSE
FALSE FRNDO
FRNDO Madrid
Madrid 44 KRLOS
KRLOS
FALSE
FALSE FRNDO
FRNDO Madrid
Madrid 55 KRLOS
KRLOS
FALSE
FALSE FRNDO
FRNDO Madrid
Madrid 66 MRPHS
MRPHS
UNKNOWN
UNKNOWN FRNDO
FRNDO Madrid
Madrid 77 <NULL>
<NULL>
FALSE
FALSE KRLOS
KRLOS Madrid
Madrid 1 FRNDO
1 FRNDO
FALSE
FALSE KRLOS
KRLOS Madrid
Madrid 22 FRNDO
FRNDO
TRUE
TRUE KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
TRUE
TRUE KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
TRUE
TRUE KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
FALSE
FALSE KRLOS
KRLOS Madrid
Madrid 66 MRPHS
MRPHS
UNKNOWN
UNKNOWN KRLOS
KRLOS Madrid
Madrid 77 <NULL>
<NULL>
FALSE
FALSE MRPHS
MRPHS Zion
Zion 1 FRNDO
1 FRNDO
FALSE
FALSE MRPHS
MRPHS Zion
Zion 22 FRNDO
FRNDO
FALSE
FALSE MRPHS
MRPHS Zion
Zion 33 KRLOS
KRLOS
FALSE
FALSE MRPHS
MRPHS Zion
Zion 44 KRLOS
KRLOS
FALSE
FALSE MRPHS
MRPHS Zion
Zion 55 KRLOS
KRLOS
TRUE
TRUE MRPHS
MRPHS Zion
Zion 66 MRPHS
MRPHS
UNKNOWN
UNKNOWN MRPHS
MRPHS Zion
Zion 77 <NULL>
<NULL>

© 2003 - 2019 Itzik Ben-Gan 7

JOIN Sub-Phase 2
Apply ON Filter
• Join predicate is evaluated per each row
• Possible results: TRUE, FALSE, UNKNOWN
• Only rows for which join predicate is TRUE are returned

© 2003 - 2019 Itzik Ben-Gan 8

JOIN Sub-Phase 3
Add Outer Rows
Customers
Customers (Preserved)
(Preserved) Result
Result of
of previous
previous step
step
custid
custid city
city C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
----------
---------- ----------
---------- ------------
------------ ------
------ ---------
--------- ------------
------------
FISSA
FISSA Madrid
Madrid FRNDO
FRNDO Madrid
Madrid 11 FRNDO
FRNDO
FRNDO
FRNDO Madrid
Madrid FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
KRLOS
KRLOS Madrid
Madrid KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
MRPHS
MRPHS Zion
Zion KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
MRPHS
MRPHS Zion
Zion 66 MRPHS
MRPHS

dbo.Customers
dbo.Customers AS
AS CC
LEFT
LEFT OUTER
OUTER JOIN
JOIN dbo.Orders
dbo.Orders AS
AS OO

C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
------------
------------ ------
------ ---------
--------- ------------
------------
FRNDO
FRNDO Madrid
Madrid 11 FRNDO
FRNDO
FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 4 KRLOS
4 KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
MRPHS
MRPHS Zion
Zion 66 MRPHS
MRPHS
FISSA
FISSA Madrid
Madrid <NULL>
<NULL> <NULL>
<NULL>

© 2003 - 2019 Itzik Ben-Gan 9

JOIN Sub-Phase 3
Add Outer Rows
• Rows from preserved table(s) that have no matches based
on the ON predicate are added as outer rows
• NULLs are used as place holders in outer rows

© 2003 - 2019 Itzik Ben-Gan 10

Further Table Operators
• T-SQL supports three proprietary table operators:
– APPLY – applies the right table expression to each row from the left table
– PIVOT – rotates and aggregates data from rows to columns
– UNPIVOT – rotates data from columns to rows
• Operate in the context of the FROM phase
• APPLY operates on two tables; PIVOT and UNPIVOT operate on a single table
• Table operators are processed from left to right; the result of a table operator is
used as the left input of the next table operator (or as input of the next logical
phase)

© 2003 - 2019 Itzik Ben-Gan 11

Further Table Operators
• Each table operator has its own set of sub-phases (details later in the
course):
– APPLY (details in module 03)
• 1: Apply right table expression to each row from the left table
• 2: Add Outer Rows
– PIVOT (details in module 04)
• 1: Group
• 2: Spread
• 3: Aggregate
– UNPIVOT (details in module 04)
• 1: Generate Copies
• 2: Extract Element
• 3: Remove NULLs

© 2003 - 2019 Itzik Ben-Gan 12

2. Apply WHERE Filter
C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
------------
------------ ------
------ ---------
--------- ------------
------------
FRNDO
FRNDO Madrid
Madrid 1 FRNDO
1 FRNDO
FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
MRPHS
MRPHS Zion
Zion 6 MRPHS
6 MRPHS
FISSA
FISSA Madrid
Madrid <NULL>
<NULL> <NULL>
<NULL>

WHERE
WHERE C.city
C.city == 'Madrid'
'Madrid'

C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
------------
------------ ------
------ ---------
--------- ------------
------------
FRNDO
FRNDO Madrid
Madrid 11 FRNDO
FRNDO
FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
FISSA
FISSA Madrid
Madrid <NULL>
<NULL> <NULL>
<NULL>

© 2003 - 2019 Itzik Ben-Gan 13

Phase 2 – Apply WHERE Filter
• WHERE predicate is evaluated per each row
• Possible results: TRUE, FALSE, UNKNOWN
• Only rows for which WHERE predicate is TRUE are returned
• Column aliases provided in the SELECT phase cannot be
used in the WHERE phase since WHERE is processed prior
to SELECT
• Aggregate functions cannot be used in the WHERE clause
since the GROUP BY phase was not processed yet

© 2003 - 2019 Itzik Ben-Gan 14

Reuse of column aliases
• To reuse column aliases
– Can use table expressions (CTEs, derived tables)
– Can use CROSS APPLY and VALUES

SELECT
SELECT orderid,
orderid, orderyear
orderyear
FROM
FROM Sales.Orders
Sales.Orders
CROSS
CROSS APPLY
APPLY (( VALUES(
VALUES( YEAR(orderdate)
YEAR(orderdate) )) ))
AS
AS A1(orderyear)
A1(orderyear)
CROSS
CROSS APPLY
APPLY (( VALUES(
VALUES( DATEFROMPARTS(orderyear,
DATEFROMPARTS(orderyear, 12,
12, 31)
31) )) ))
AS
AS A2(endofyear)
A2(endofyear)
WHERE
WHERE orderdate
orderdate == endofyear;
endofyear;

© 2003 - 2019 Itzik Ben-Gan 15

3. Arrange Rows in Groups
C.custid
C.custid C.city
C.city O.orderid
O.orderid O.custid
O.custid
------------
------------ ------
------ ---------
--------- ------------
------------
FRNDO
FRNDO Madrid
Madrid 11 FRNDO
FRNDO
FRNDO
FRNDO Madrid
Madrid 22 FRNDO
FRNDO
KRLOS
KRLOS Madrid
Madrid 33 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 44 KRLOS
KRLOS
KRLOS
KRLOS Madrid
Madrid 55 KRLOS
KRLOS
FISSA
FISSA Madrid
Madrid <NULL>
<NULL> <NULL>
<NULL>

GROUP
GROUP BY
BY C.custid
C.custid

groups rows

C.custid C.custid C.city O.orderid O.custid

FRNDO FRNDO Madrid 1 FRNDO
FRNDO Madrid 2 FRNDO
KRLOS Madrid 3 KRLOS
KRLOS KRLOS Madrid 4 KRLOS
KRLOS Madrid 5 KRLOS
FISSA FISSA Madrid <NULL> <NULL>

© 2003 - 2019 Itzik Ben-Gan 16

Phase 3 – Arrange Rows in Groups
• Rows are associated with groups according to GROUP BY
list
• A group is formed for each unique combination of values in
GROUP BY list
• Can define multiple grouping sets using the GROUPING
SETS, CUBE and ROLLUP clauses (details in Module 04)

© 2003 - 2019 Itzik Ben-Gan 17

4. Apply HAVING Filter
groups rows

C.custid C.custid C.city O.orderid O.custid

FRNDO Madrid 1 FRNDO
FRNDO
FRNDO Madrid 2 FRNDO
KRLOS Madrid 3 KRLOS
KRLOS KRLOS Madrid 4 KRLOS
KRLOS Madrid 5 KRLOS
FISSA FISSA Madrid <NULL> <NULL>
HAVING
HAVING COUNT(O.orderid)
COUNT(O.orderid) << 33

C.custid C.custid C.city O.orderid O.custid

FRNDO Madrid 1 FRNDO
FRNDO
FRNDO Madrid 2 FRNDO
FISSA FISSA Madrid <NULL> <NULL>
© 2003 - 2019 Itzik Ben-Gan 18
Phase 4 – Apply HAVING Filter
• HAVING predicate is evaluated for each group; only groups for which predicate is
TRUE are returned
• Applicable post-GROUP BY phase:
– Can refer to columns/expressions from the groups section or to aggregate functions
applied to columns from the rows section
– Columns/expressions not included in the GROUP BY list cannot be specified without
an aggregate function
– Column aliases provided in the SELECT list can not be used as the SELECT phase
was not processed yet

© 2003 - 2019 Itzik Ben-Gan 19

Process SELECT Clause
groups raw

C.custid C.custid C.city O.orderid O.custid

FRNDO FRNDO Madrid 1 FRNDO
FRNDO Madrid 2 FRNDO
FISSA FISSA Madrid <NULL> <NULL>
SELECT
SELECT C.custid,
C.custid, COUNT(O.orderid)
COUNT(O.orderid) AS
AS numorders
numorders

result table

C.custid numorders
FRNDO 2
FISSA 0

© 2003 - 2019 Itzik Ben-Gan 20

Phase 5 – Process SELECT Clause
• Determines result columns in query result table
• Result is a multiset/bag
• 5.1: Process expressions in SELECT list
• 5.2: Process DISTINCT clause
• If query groups data, rules that apply post-GROUP BY are
applicable here
• Expressions in SELECT list are conceptually processed all-
at-once; same applies to all phases (e.g., ON, WHERE,
HAVING)

© 2003 - 2019 Itzik Ben-Gan 21

6. Apply Presentation Ordering
groups raw result table

C.custid C.custid C.city O.orderid O.custid C.custid numorders

FRNDO Madrid 1 FRNDO FRNDO 2
FRNDO
FRNDO Madrid 2 FRNDO
FISSA 0
FISSA FISSA Madrid <NULL> <NULL>

ORDER
ORDER BY
BY numorders
numorders

result cursor

C.custid numorders
FISSA 0
FRNDO 2

© 2003 - 2019 Itzik Ben-Gan 22

Phase 6 – Apply Presentation Ordering
• Can refer to column aliases assigned in the SELECT phase
• Can use ordinal positions of columns, though this is
considered a bad practice
• If DISTINCT is specified, ORDER BY elements are restricted
to result columns from the SELECT list; otherwise, can refer
to expressions that would have been allowed in the SELECT
list
• Returns a cursor

© 2003 - 2019 Itzik Ben-Gan 23

Phase 7 – OFFSET-FETCH/TOP Filter
• If the OFFSET-FETCH or TOP option are used, filter rows
based on the specified number and ordering
• The same ORDER BY clause used for presentation ordering
(phase 6) is used to define the ordering specification for the
OFFSET-FETCH and TOP filters
• If OFFSET-FETCH or TOP used in a table expression,
ORDER BY is allowed; outer query has no presentation
ordering guarantees

© 2003 - 2019 Itzik Ben-Gan 24

Logical Query Processing, Revisited
1. FROM
2. WHERE
3. GROUP BY
4. HAVING
5. SELECT
5.1 SELECT list
5.2 DISTINCT
6. ORDER BY (presentation)
7. OFFSET-FETCH / TOP

© 2003 - 2019 Itzik Ben-Gan 25

Logical Query Processing, Revisited

© 2003 - 2019 Itzik Ben-Gan 26

Quiz
Quiz Script: Module 01 - Logical Query Processing - Exercise
Solution Script: Module 01 - Logical Query Processing -
Solution

© 2003 - 2019 Itzik Ben-Gan 27

 End of Module 01
Logical Query Processing

Review:
T-SQL Querying, Chapter 1

Questions?

© 2003 - 2019 Itzik Ben-Gan 28