Lecture 7

Least Squares Method for

Curve Fitting

Process Control
Prof. Kannan M. Moudgalya

IIT Bombay
Monday, 5 August 2013

1/32 Process Control Least squares method for curve fitting

Outline

1. Geometric interpretation of least squares

method
2. An application to identification

2/32 Process Control Least squares method for curve fitting

1. Least squares method

3/32 Process Control Least squares method for curve fitting

Ramp input to first order system or
step input to an integrating process
K a
Y(s) =
s + 1 s2
A B C
= + + 2
s + 1 s s
2
Ka Ka Ka
= + 2
s + 1 s s
Inverting
y(t) = Ka et/ Ka + Kat
 
t/
= Ka e +t
4/32 Process Control Least squares method for curve fitting
An Application: Midterm Exam, 2007
Corridor Consultants, Pvt. Ltd. has deputed you to
identify the transfer function of a plant in a clients
company.

From the initial studies, you have found that this

plant can be modelled as a second order system
with integrating action, with the following unit
step response,

y(t) = K[t + et/ ]

with the unit of time t being, hour.

5/32 Process Control Least squares method for curve fitting

Problem statement - ctd
I You recall from CL-358 days that a step
response experiment can be used to determine
the model parameters K and .
I At a convenient time, you begin a unit step
response experiment.
I You note down the starting time of this
experiment as t = 0.
I You instruct the plant operators to collect the
data of y(t) vs. t.
I Unfortunately, because of delays in getting the
measurement logistics worked out, the initial
data are not collected.
6/32 Process Control Least squares method for curve fitting
Problem statement - ctd
I The operators succeed in collecting data only
after a lapse of a sufficiently long time.
I They obtain following data in consistent units:

t (hr) y(t)
4 7.1
5 8.9
6 11.2
1. Formulate this as a least squares problem to
determine the model parameters K and .
2. Solve the above least squares problem and
determine K and . (30 marks)
y(t) = K[t + et/ ]
7/32 Process Control Least squares method for curve fitting
Solution
After sufficiently long time, the model is:

K(t ) = y(t)

With the definition of C = K , we obtain,

Kt + C = y(t)

t (hr) y(t) 4K + C = 7.1

4 7.1 5K + C = 8.9
gives
5 8.9
6 11.2 6K + C = 11.2
8/32 Process Control Least squares method for curve fitting
Solution - ctd

4K + C = 7.1
5K + C = 8.9
6K + C = 11.2

These equations are in the form Ax = b, with

4 1   7.1
K
A = 5 1 , x = , b = 8.9
C
6 1 11.2

3 equations in two unknowns - how to solve this?

9/32 Process Control Least squares method for curve fitting
Interpretation of linear equations
Consider the following nonsquare system:

a11 a12   b1
a21 a22 x1
= b2
x2
a31 a32 b3

I How do you interpret these equations?

I System of 3 equations in two unknowns
I Overdetermined system
I What is the geometric meaning of such a
system?
10/32 Process Control Least squares method for curve fitting
One Interpretation of linear equations

Consider the following nonsquare system:

a11 a12   b1
a21 a22 x1 = b2
x2
a31 a32 b3

What is the geometric meaning of this system?

1. Three lines intersecting at a point
2. How often can this happen?

11/32 Process Control Least squares method for curve fitting

Another interpretation

a11 a12   b1
a21 a22 x1
= b2
x2
a31 a32 b3

Can be written as:

a11 a12 b1
a21 x1 + a22 x2 = b2
a31 a32 b3

Now, can anyone give a geometric meaning?

12/32 Process Control Least squares method for curve fitting
Meaning of 3 equations in 2 unknowns

a11 a12 b1
a21 x1 + a22 x2 = b2
a31 a32 b3

a11 a12
I A1 = a21 and A2 a22 make a 2 dimensional
a31 a32
space - a plane
I Left hand side is in a 2 dimensional space - a
plane
I b is a 3 dimensional vector
I Can be solved only for very special case of b
13/32 Process Control Least squares method for curve fitting
Geometric Interpretation
b

A plane

I A plane = all combinations of A1 and A2

I b lies outside this plane
14/32 Process Control Least squares method for curve fitting
Revisit the meaning of Ax = b

We are studying the system:

a11 a12   a11 a12 b1
x1
Ax = a21 a22
= a21 x1 + a22 x2 = b2

x2
a31 a32 a31 a32 b3

I Ax is a 2-d vector, lies on A plane

I b is a 3-d vector, lies outside this plane
I Ax = b has no solution
I Is there an approximate solution?

15/32 Process Control Least squares method for curve fitting

Can we solve this at least approximately?
b

A plane

Choose one:
1. Make Ax outside A plane, to coincide with b
2. Approximate b by a vector lying on the A plane
3. Cannot solve this problem, even approximately
Ans: 2, project b on to the A plane
16/32 Process Control Least squares method for curve fitting
 b

Ax0

A plane

17/32 Process Control Least squares method for curve fitting

Approximate solution to Ax = b
b

Ax0

A plane

I Approximate x with x0
I such that Ax0 ' b
I Ax0 is closest to b
I Closest
18/32 means Process
what? Control Least squares method for curve fitting
For best Ax ' b, minimize error, e
b
e = Ax0 b

Ax0

A plane
19/32 Process Control Least squares method for curve fitting
How does smallest e help calculate x0?
b
e = Ax0 b

Ax0

A plane

I How to make e perpendicular to A plane?

I Make e perpendicular to every column of A
I As a linear combination of columns makes A
I Inner product of e and every column of A = 0
20/32 Process Control Least squares method for curve fitting
Solution to Ax0 ' b
b
e = Ax0 b

Ax0

A plane

I Inner product of e and every column of A = 0

I A = [A1 A2 ] and e = Ax0 b
I A1 (Ax0 b) = 0
I AT1 (Ax0 b) = 0
I AT2 (Ax0 b) = 0
21/32 Process Control Least squares method for curve fitting
Solution to Ax0 ' b - ctd

I AT1 (Ax0 b) = 0
I AT (Ax b) = 0
 2 T 0  
A1 0
I
T (Ax0 b) = 0
A2
I AT (Ax0 b) = 0
I AT Ax0 = AT b
I x0 = (AT A)1 AT b

22/32 Process Control Least squares method for curve fitting

Solution to Ax0 ' b

Solution to Ax = b is x = (AT A)1 AT b

23/32 Process Control Least squares method for curve fitting

Condition for the existence of (ATA)1
I Solution to Ax = b is x = (AT A)1 AT b
T
I Inverse of (A
A) should
exist
a11 a12
I Recall A = a21 a22 , a 3 2 matrix

a31 a32
I (AT A) is a 2 2 matrix
I When will (AT A) be not singular?
I When even two out of three equations are not
independent
I Example: the same equation is written three
times!
I Cannot cheat by mathematical jugglery!
24/32 Process Control Least squares method for curve fitting
2. Application to Identification

25/32 Process Control Least squares method for curve fitting

Recall the Identification Problem

Ax = b

with


4 1   7.1
K
A = 5 1 , x= , b = 8.9
C
6 1 11.2

3 equations in two unknowns

26/32 Process Control Least squares method for curve fitting

Solution - ctd

  4 1  
4 5 6 77 15
AT A = 5 1 =
1 1 1 15 3
6 1
 
1 3 15
(AT A)1 =
6 15 77

27/32 Process Control Least squares method for curve fitting

Solution - ctd

The least squares solution is given by,

   7.1
1 3 15 4 5 6
x = (AT A)1 AT b = 8.9
6 15 77 1 1 1
11.2
   
1 3 0 3 1 12.3
= =
6 17 2 13 6 7.1
 
2.05
=
1.18

28/32 Process Control Least squares method for curve fitting

Solution - ctd

We obtain,

K = 2.05, K = 1.18
= 0.577

It can be verified that the assumption of long time

is correct.

29/32 Process Control Least squares method for curve fitting

Why is it called least squares
minimisation?
I Sum of squares of error results in 2 norm
(squared)
I The shortest distance in 2 norm is by taking
perpendicular
b
e = Ax0 b

Ax0

A plane
30/32 Process Control Least squares method for curve fitting
What we learnt today

I Geometric interpretation of Least squares

method
I An application

31/32 Process Control Least squares method for curve fitting

 Thank you

32/32 Process Control Least squares method for curve fitting