Mathematics for chemical engineers

Drahoslava Janovska

Winter Semester 2013-2014

Linear and nonlinear regression
Estimation of experimental data Basic model of linear regression Nonlinear regression

Outlines

1 Estimation of experimental data

2 Basic model of linear regression

Equivalent model
Least squares method

3 Nonlinear regression
Estimation of experimental data Basic model of linear regression Nonlinear regression

Estimation of experimental data

- Solving of chemical-engineering problems usually let us derive the
model of process or phenomenon taking place in the device.
- Frequently, we are not able to identify numerical values of model
parameters.
Function (x) = E(Y (x)) defined on the domain A R is called regression
function. Regression is the relationship between the mean value of random
variable Y (x) and the independent variable x.
Assume that we know the form of the regression function. Based on random
selection, we estimate its unknown parameters:
Choose n values of independent variable xj , j = 1, . . . , n, xj A, and for
each xj we observe (measure) the realization (value) yj of random variable Yj :

xj , j = 1, . . . , n, A yj = Y (xj ) .

Obtained pairs of values (x1 , y1 ), . . . , (xn , yn ) are used for estimation unknown
parameters.
Estimation of experimental data Basic model of linear regression Nonlinear regression

Basic model of linear regression

Model of linear regression must fulfil:

1. (x) is a linear function of the form
p
X
(x) = k fk (x) ,
k=1

where fk (x) are known functions, and k , k = 1, . . . , p unknown

parameters. The function is linear in parameters.
2. Value xj is assigned random variable Yj , for which reads

E(Yj ) = (xj ) , D(Yj ) = 2 , j = 1, . . . , n ,

The second equation means that the variance is independent of xj , and

thus is constant. For example, it corresponds to the case that all
realizations y1 , . . . , yn of random variables Y1 , . . . , Yn are measured with
the same precision.
Estimation of experimental data Basic model of linear regression Nonlinear regression

3. Matrix F = (fij ) , where fij = fi (xj ) , i = 1, . . . , p, j = 1, . . . , n. has the

rank p . Note that the number n of pairs (xj , yj ) must be greater than the
number of unknown parameters p, precisely, it should hold true
n p > 2.
4. Random variables Y1 , . . . , Yn are not correlated, i.e.

cov(Yi , Yj ) = 0, i, j = 1, . . . , n, i 6= j .

Matrix notation
Cy = 2 En ,
where En is the identity matrix of order n, Cy is the covariance matrix of
variables Y1 , . . . , Yn .

Example A regression line, i.e. the regression function of form

(x) = + x has the number of unknown parameters p = 2, and
1 = , f1 = 1, 2 = , f2 = x .
Estimation of experimental data Basic model of linear regression Nonlinear regression

Equivalent model

Described model is in equivalent form

p
X
Yj = (xj ) + j = k fkj + j , j = 1, . . . , n , (1)
k =1

where values x1 , . . . , xn are values of nonrandom variables, values fkj = fk (xj )

fulfil the 3. condition of the model. Random errors j , j = 1, . . . , n , and
covariance matrix C of the random vector = (1 , . . . , n ) fulfil

E(j ) = 0 , j = 1, . . . , n , C = 2 E = Cy .

Equation (1) reads in the matrix form

Y = FT +
.
Estimation of experimental data Basic model of linear regression Nonlinear regression

Least squares method

Least squares method

We find unknown parameters 1 , . . . , p in described model of linear

regression by least squares method . Let these estimations are b1 , . . . , bp ,
which are selection functions of random choice y1 , . . . , yn . We minimize the
sum of squares of deviations from observed values yj and their mean values
j = (xj ), thus the sum of squares is
n n p
!2
X X X
Q(1 , . . . , p ) = (yj j )2 = yj k fkj .
j=1 j=1 k =1

Thus estimations b1 , . . . , bp are found as a solution of the set of equations

Q
= 0, k = 1, . . . , p.
k
This system is called the system of normal equations.
Estimation of experimental data Basic model of linear regression Nonlinear regression

Least squares method

We can rewrite the system of normal equations for searched estimations

b1 , . . . , bp in a lucid form

b1 S11 + b2 S12 + + bp S1p = S1y

b1 S21 + b2 S22 + + bp S2p = S2y
..
.
b1 Sp1 + b2 Sp2 + + bp Spp = Spy ,
where n
X
Ski = fkj fij , i, k = 1, . . . , p ,
j=1
Xn
Sky = fkj yj , k = 1, . . . , p .
j=1

Clearly Sik = Ski for i, k = 1, . . . , p .

Estimation of experimental data Basic model of linear regression Nonlinear regression

Least squares method

Matrix notation:



If y = (y1 , . . . , yn )T , b = (b1 , . . . , bp )T , then normal equations can be
written in form


F FT b = F y . (2)
T T
Assume that h(F ) = p, then also h(F F ) = p and F F is the type p p,
regular = exists (F F T )1 , and thus we can express from equation (2)


vector b :


b = (F F T )1 F y ,


the vector b is uniquely determined and each component is a linear
combination of values y1 , . . . , yn .

Attention! Calculation is extremely numerical unstable, see the lecture

Linear algebra.
Estimation of experimental data Basic model of linear regression Nonlinear regression

Least squares method

Example

Let a regression line goes through the beginning, (x) = a x , then fij = x and


1 = . Denote x = (x1 , . . . , xn )T , F = (x1 , . . . , xn ) . Then
n
X 1
F F T = (x1 , . . . , xn ) (x1 , . . . , xn )T = xj2 = (FF T )1 = Pn .
j=1 j=1 xj2

Estimation of the parameter is

! Pn
T 1
1 T
j=1 xj yj
a = (F F ) Fy = Pn x y = Pn .
j=1 xj2 j=1 xj
2
Estimation of experimental data Basic model of linear regression Nonlinear regression

Least squares method

Unbiased estimation of linear parametric function

Task We find estimation of the linear function of parameters

= (1 , . . . , p )T . Assume a parametric function
p
X

= ck k = c T ,
k =1

where c = (c1 , . . . , cp )T is known nonzero vector ( c 6= 0) .


Statement The best estimation of linear parametric function c T is a
T


function (statistics) g = c b , where b is the solution of normal equations.
E(g) = , and the best means that the variance D(g) is minimal in the class
of unbiased estimations.
For evaluation of the covariance matrix of vector of estimations b1 , . . . , bp ,
where bk are random quantities, let us remind rules for computation of
variance and covariance of linear combination of random quantities, a
random vector (u1 , . . . , un )T , Cu is the covariance matrix of random vector,

Cu = (cov(uj , uk )) , j = 1, . . . , n; k = 1, . . . , n, D(uj ) = cov(uj , uj ) .

Estimation of experimental data Basic model of linear regression Nonlinear regression

Nonlinear regression

Goal: Estimation of parameters a1 , . . . , an in nonlinear empirical formulae

y = f (x, a) .

We will minimize the sum of squares of deviations

m 
X m
2 X
S(a) = f (x j , a) y j = qj2 (a) ,
j=1 j=1

where qj is residuum jth measured point. Denote a+ to be the point where

the sum of squares S(a) has its minimum. Value a+ is obtained as a limit so
called minimizing the sequence ak so that it holds

S(ak+1 ) < S(ak ) .

Estimation of experimental data Basic model of linear regression Nonlinear regression

Taylor series of function f (neglect terms of higher order than 1):

f (x, a) f (x, ak ) + gradTa f (x, ak ) (a ak )

n
X f (x, ak )
f (x, a) f (x, ak ) + (aj ajk ) .
aj
j=1

Evaluate the approximation formulae

n
X f (x, ak )
y f (x, ak ) = 4ajk .
aj
j=1

Let (a) be the Jacobi matrix,

f (x 1 , a) f (x 1 , a) f (x 1 , a)

...
a1 a2 an
.. ..

(a) = .

. .
f (x m , a) f (x 1 , a) f (x m , a)
...
a1 a2 an
Estimation of experimental data Basic model of linear regression Nonlinear regression

We search for solution:

 1
4+ ak = T (ak ) (ak ) T (ak )q(ak ) ,

where q = (q1 , . . . , qm ). Using 4+ ak we calculate the next iteration

ak+1 = ak + 4+ ak , (0, 1i .
k+1 k
The first value: = 1. If S(a ) S(a ), then .
The calculation is performed for

T (ak ) (ak ) 4+ ak = T (ak ) q(ak ) .

| {z }
matrix n n

Process is stopped, if ||4+ ak || is less than a required precision.