Mathematical Preliminary

1. Introduction
1.1 Forms and components of a mathematical programming problems
A mathematical programming problem or, simply, a mathematical program is a mathematical formulation
of an optimization problem.
Unconstrained Problem:
(P) minx f(x)
s.t. x  X ,

where x=(x1, …, xn)T  R n , f(x): R n  R , and X is an open set (usually X  R n ).

Constrained Problem:
(P) minx f(x)
s.t. g i  x   0 i  1,..., m
hi  x   0 i  1,..., l
x X,

where g1(x),…, gm(x), h1(x),…, hl(x): R n  R .

Let g(x)=( g1(x),…, gm(x))T: R n  R m , h(x)=( h1(x),…, hl(x))T: R n  R l . Then (P) can be written as
(P) minx f(x)
s.t. g  x   0
h  x  0 (1)
x X.
Some terminology: Function f(x) is the objective function. Restrictions “hi(x)=0” are referred to as
equality constraints, while “gi(x)  0 ” are inequality constraints. Notice that we do not use constraints
in the form “gi(x)<0”!
A point x is feasible for (P) if it satisfies all the constraints. (For an unconstrained problem, x  X . )
The set of all feasible points forms the feasible region, or feasible set (let us denote it by F ). The goal

of an optimization problem in minimization form, as above, is to find a feasible point x such that


f x  f  x  for any other feasible point x.

1
1.2 Linear Programming and the Simplex Method
Let’s start with an example.
Minimize x1  3 x2

subject to  x1  2 x2  6

x1  x2  5
x1 x2  0

2
 Linear Programming

1. Simplex Method
Let’s start with an example.
Minimize x1 − 3 x2

subject to − x1 + 2 x2 ≤ 6

x1 + x2 ≤ 5
x1 x2 ≥ 0
To make it be a standard format
Minimize x1 − 3 x2
subject to − x1 + 2 x2 + x3 =6
x1 + x2 +x4 = 5
x1 , x2 , x3 , x4 ≥0
where x3 and x4 are slack variables
define x3 = 6 + x1 − 2 x2 (The form illustrated below is called “a dictionary.”)
x4 = 5 − x1 − x2
z= x1 − 3 x2
Initial solution
x1 = 0, x2 = 0, x3 = 6, x4 = 5, z = 0 , x1 x2 are nonbasic variables, x3 x4 are basic variables
Choosing the entering variable x2
x3 = 6 + x1 − 2 x2 ≥ 0 ⇒ x2 ≤ 3 (1)
x4 = 5 − x1 − x2 ≥ 0 ⇒ x2 ≤ 5 (2)
(1) is the most stringent.Increasing x2 up to (1).
∴ x1 = 0, x 2 = 3, x3 = 0, x 4 = 3 ( x3 leaves the basis)
To construct the new system, we shall begin with the new comer to the left hand side, Namely, the
variable x2 . The desired formula for x2 in terms of x1 , x3 , x 4 is
x x
x2 = 3 + 1 − 3
2 2
Next, in order to express x4 and z in terms of x1 , x3, we simply substitute
 x x  3 x
x4 = 5 − x1 −  3 + 1 − 3  = 2 − x1 + 3
 2 2 2 2
 x x  x 3
z = x1 − 3  3 + 1 − 3  = −9 − 1 + x3
 2 2 2 2
Hence our new system
x x
x2 = 3 + 1 − 3
2 2
3 x3
x4 = 2 − x1 +
2 2
x 3
z = −9 − 1 + x3
2 2
Increase x1 ( x1 enters the basis)
1
 x1 x3
x2 = 3 + − ≥0 ⇒ x1 can be infinity
2 2
3 x 4
x4 = 2 − x1 + 3 ≥ 0 ⇒ x1 ≤ ( x4 leaves the basis)
2 2 3
4 11
∴ x1 = , x2 = , x3 = 0, x4 = 0
3 3
2 1 2 2 4 1 2
⇒ x1 = 2 ⋅ + ⋅ x3 − x4 = + x3 − x4
3 2 3 3 3 3 3
14 1 2  x 11 1 1
and x2 = 3 +  + x3 − x4  − 3 = − x3 − x4
23 3 3  2 3 3 3
14 1 2  3 29 4 1
z = −9 −  + x3 − x4  + x3 = − + x3 + x4
23 3 3  2 3 3 3
Hence
4 1 2
x1 = + x3 − x4
3 3 3
11 1 1
x2 = − x3 − x4
3 3 3
29 4 1
z=− + x3 + x4
3 3 3
∴ x3 and x4 in the objective function are with positive coefficients.
∴ Stop.

In general from
Minimize
n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

2
3
Some notes:
xs is chosen to increase the objective function, and xr is chosen to maintain feasibility.
A change from one dictionary to the next dictionary is called a pivot, and the equation
s
corresponding to xr is called the pivoting row. The row with the smallest ratio is called the
r
pivot row and corresponds to the leaving variable.

4
2. Pitfall of the simplex method
The examples illustrating the simplex method in the preceding lecture were purposely smooth. They did
not point out the dangers that can occur. The purpose of this section, therefore is to rigorously analyze
the method by scrutinizing its every step.

INITIALIZATION
Given a standard form LP
Maximize
n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

If all bi are nonnegative the initial dictionary is feasible, the basic simplex algorithm solves the

problem.

If some bi < 0 , how to find an initial feasible solution?

An example
Maximize − +
s.t. 2 − + 2 ≤ 4
2 −3 + ≤ −5
− + −2 ≤ −1
, , ≥ 0.

Following the previous discussion, here is the trouble to find an initial feasible solution.
= −4 − 2 +3 −2
= −5 − 2 +3 −2
= −1 + 2 −3 +2

One way of getting rid of this trouble is solving the auxiliary problem for finding the initial feasible
solution. To avoid unnecessary confusion, we write the auxiliary problem in its maximization form:

5
Maximize −
s.t. 2 − + 2 − ≤ 4
2 − 3 + − ≤ −5
− + − 2 − ≤ −1
, , , ≥ 0.

where x0 is called an artificial variable.

Writing down the formulas defining the slack variables x4, x5, x6 and the objective function w, we obtain
the dictionary
= −4 − 2 +3 −2 +
= −5 − 2 +3 −2 +
= −1 + 2 − 3 + 2 +
= −1 + 2 − 3 + 2 −
which is infeasible. Nevertheless, this infeasible dictionary can be transformed into a feasible one by a
single pivot, with x0 entering and x5 leaving the basis: (x5 is the most infeasible basic variable among x4,
x5, x6)
= −5 + 2 −3 +2 +
= −9 − 2 −2 −2 +
= −4 + 3 − 4 + 3 +
= −5 − 2 + 3 − 2 − .

After the first iteration, with x2 entering and x6 leaving:

= −1 + 0.75 + 0.75 + 0.25 − 0.25
= −2 − 0.25 − 1.25 + 0.25 + 0.75
= −7 − 01.5 − 02.5 + 00.5 + 00.5
= −2 + 0.25 + 1.25 − 0.25 − 0.75 .

After the second iteration, with x3 entering and x0 leaving:

= 1.6 − 0.2 + 0.2 + 0.6 − 0.8
= 2.2 + 0.6 + 0.4 + 0.2 − 0.6
= 3.2 − 1.5 − 2.5 − 0.5 + 0.2
= 2.2 + 0.2 + 1.2 − 0.2 − 0.7 .

6
The last dictionary is optimal. Since the optimal value of the auxiliary problem is zero, the last
dictionary points out a feasible solution of the original problem: x1 = 0 , x2 = 2.2 , x3 = 1.6 .
Furthermore, the last dictionary can be easily converted into the desired feasible dictionary of the
original problem. To obtain the first three rows of the desired dictionary, we simply copy down the first
three rows of the last dictionary, omitting all the terms involving x0 :
= 1.6 − 0.2 + 0.2 + 0.6
= 2.2 + 0.6 + 0.4 + 0.2
= 3.2 − 1.5 − 2.5 − 0.5 .
To obtain the last row, we have to express the original objective function
= − +
in terms of the nonbasic variable x1 , x5 , x6 .

= − 2.2 + 0.6 + 0.4 + 0.2 + 1.6 − 0.2 + 0.2 + 0.6

= −0.6 + 0.2 − 0.2 + 0.4 .
In short, the desired dictionary reads
= −1.6 − 0.2 + 0.2 + 0.6
= −2.2 + 0.6 + 0.4 + 0.2
= −3.2 − 1.5 − 2.5 − 0.5
= −0.6 + 0.2 − 0.2 + 0.4 .
General case
Original problem
Maximize

n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

Auxiliary problem

7
Phase 1:

8
Theorem: The original LP is feasible if and only if the optimum objective value for the auxiliary
problem is 0.
Proof:

ITERATION
Given some feasible dictionary, we have to select an entering variable, to find a leaving variable, and to
construct the next feasible dictionary by pivoting.

Unbounded dictionary
= 5+2 −2 −3
= 7+6 −3 −4
=5+2 −2 −4 .

The entering variable is x3 , but neither of the two basic variables x2 and x5 imposes an upper bound
on its increase. Therefore, we can make x3 as large as we wish (maintaining x1 = x4 = 0 ) and still
retain feasibility: setting x3 = t for any positive t, we obtain a feasible solution with x1 = 0 ,
x2 = 5 + 2t , x4 = 0 , x5 = 7 , and z = 5 + t . Since t can be made arbitrarily large, z can be made
arbitrarily large. We conclude that the problem is unbounded.

Degeneracy
The presence of more than one candidate for leaving the bases has interesting consequences. For
illustration, consider the dictionary.
= 1−2 +2 −2
= 3−2 +4 −6
= 2+1 −3 −4
=6+2 −2 +8 .

9
Having chosen x3 to enter the basis, we find that each of the three basic variables x4 , x5 , x6 limits the
increase of x3 to 1/2. Hence each of these three variables is a candidate for leaving the basis. We
arbitrarily choose x4 . Pivoting as usual, we obtain the dictionary.
= 0.5 − 2 +2 − 0.5
= 0.3 − 2 +4 + 0.3
= 0.2 + 1 −3 + 0.2
= 0.4 + 2 −2 − 0.4 .

Along with the nonbasic variables, the basic variables x5 and x6 have value zero in the associated
solution. Basic solutions with one or more basic variables at zero are called degenerate. Degeneracy
may have annoying side effects. These are illustrated on the next iteration in the example. There, x1
enters the basis and x5 leaves; because of degeneracy, the constraint x5 ≥ 0 limits the increment of
x1 to zero. Hence the value of x1 will remain unchanged, and so will the values of the remaining
variables and the value of the objective function z.

= 0.5 − 2 + 1.5 − 0.5

= 0.5 − 2 − 0.5 + 0.3
= 0.2 − 1 + 3.5 − 0.5
= 0.4 + 3 − 20. − 0.4 .

In this particular iteration, pivoting changes the dictionary as above, but it does not affect the associated
solution at all. Simplex iterations that do not change the basic solution are called degenerate.

10
TERMINATION: CYCLING
Can the simplex method go through an endless sequence of iterations without ever finding an optimal
solution? Yes, it can. To justify this claim, let us consider the initial dictionary

= 0 − 0.5 + 5.5 + 2.5 − 09

= 0 − 0.5 + 1.5 + 0.5 − 00
= 1 − 0.0 + 3.5 − 0.5
= 0 + .10 − .57 − 0.9 − 24 .
and let us agree on the following:
(i) The entering variable will always be the nonbasic variable that has the largest coefficient in the
z-row of the dictionary.
(ii) If two or more basic variables compete for leaving the basis, then the candidate with the
smallest subscript will be made to leave.
Now the sequence of dictionaries constructed in the first six iterations goes as follows.
After the first iteration:
= 0 − .11 + 5.5 − .18 − 02
= 0 − 0.4 − 1.2 + 0.8 + 00
= 1 − .11 − 3.5 + .18 + 02
= 0+. . .53 + .41 − 204 − 20 .
After the second iteration:
= 0 − 0.5 + 1.2 + 0.25 − 0.25
= 0 − 0.5 + 5.4 + 0.75 − 2.75
= 1 + 0.5 − 3.4 − 0.75 − 13.25
= 0+. .14.5 − 98 − 6.75 − 13.25 .
After the third iteration:
= 0−. 8 + 1.5 − 5.5 − 02
= 0 − 0.2 − 0.5 + 2.5 + 00
= 1 − .11 − 3.5 + .18 −0
= 0+. . .18 + .15 − 93 − 29 .
After the fourth iteration:
= 0 − 0.25 + 1.25 + 0.5 − 0.5
= 0 − 00.5 + 14.5 + 0.2 − 0.4
= 1 − 00.5 + 14.5 − 0.2 − 0.4
= 0+. .10.5 − 70.5 − 20 − 9 .
After the fifth iteration:
= 09 +4 −8 −2
= 0 − − 0. 5 + 1.5 + 0.5
= 1 − 0. 5 + 0.2 − 0.4
= 1 + .24 + 22 − 293 − 21 .

11
 After the sixth iteration:
= 0 − 0.5 + 1.5 + 0.5 − 0.5
= 0 − 0.5 + 5.5 + 2.5 − 0.9
= 1 − 0.5 + 4.5 − 0.2 − 0.4
= 1 + .10 − .57 − 2.9 − 24 .

Since the dictionary constructed after the sixth iteration is identical with the initial dictionary, the
method will go through the same six iterations again and again without ever finding the optimal solution
(which, as we shall see later, has z = 1). This phenomenon is known as cycling. More precisely, we say
that the simplex method cycles if one dictionary appears in two different iterations (and so the sequence
of iterations leading from the dictionary to itself can be repeated over and over without end). Note that
cycling can occur only in the presence of degeneracy: since the value of the objective function increases
with each nondegenerate iteration and remains unchanged after each degenerate one, all the iterations in
the sequence leading from a dictionary to itself must be degenerate. Cycling is one reason why the
simplex method may fail to terminate.

Bland’s Rule

The entering variable chosen by the smallest index variables with c j > 0 .

If there is a tie for leaving variables, break the tie by the smallest index.

Recall the previous example if we follow Bland’s rule.

= 0 − 0.5 + 5.5 + 2.5 − 09

= 0 − 0.5 + 1.5 + 0.5 − 00
= 1 − 0.0 + 3.5 − 0.5
= 0 + .10 − .57 − 0.9 − 24 .

12
After the first iteration:
= 0 − .11 + 5.5 − .18 − 02
= 0 − 0.4 − 1.2 + 0.8 + 00
= 1 − .11 − 3.5 + .18 + 02
= 0+. . .53 + .41 − 204 − 20 .
After the second iteration:
= 0 − 0.5 + 1.2 + 0.25 − 0.25
= 0 − 0.5 + 5.4 + 0.75 − 2.75
= 1 + 0.5 − 3.4 − 0.75 − 13.25
= 0+. .14.5 − 98 − 6.75 − 13.25 .
After the third iteration:
= 0−. 8 + 1.5 − 5.5 − 02
= 0 − 0.2 − 0.5 + 2.5 + 00
= 1 − .11 − 3.5 + .18 −0
= 0+. . .18 + .15 − 93 − 29 .
After the fourth iteration:
= 0 − 0.25 + 1.25 + 0.5 − 0.5
= 0 − 00.5 + 14.5 + 0.2 − 0.4
= 1 − 00.5 + 14.5 − 0.2 − 0.4
= 0+. .10.5 − 70.5 − 20 −9 .
After the fifth iteration:
= 09 +4 −8 −2
=0− − 0. 5 + 1.5 + 0.5
= 1 − 0. 5 + 0.2 − 0.4
= 1 + .24 + 22 − 293 − 21 .

After the sixth iteration: (different from the previous example)

= −2 +3 + − 2
=0 +4 + −8
=1+2 −3 − +2 0
= 1+. −20 − .27 + 2. − 44 .

After the seventh iteration: ( x3 enters and x7 leaves)

=1 +2 −3 − +2 0
=⋯
=⋯
= 1 − 18 − .30 − 2. − 42 .

13
Theorem: The simplex method always terminates provided that both the entering and the leaving
variable are chosen according to Bland’s rule.
Proof

14
15
 Linear Programming

1. Simplex Method
Let’s start with an example.
Minimize x1 − 3 x2

subject to − x1 + 2 x2 ≤ 6

x1 + x2 ≤ 5
x1 x2 ≥ 0
To make it be a standard format
Minimize x1 − 3 x2
subject to − x1 + 2 x2 + x3 =6
x1 + x2 +x4 = 5
x1 , x2 , x3 , x4 ≥0
where x3 and x4 are slack variables
define x3 = 6 + x1 − 2 x2 (The form illustrated below is called “a dictionary.”)
x4 = 5 − x1 − x2
z= x1 − 3 x2
Initial solution
x1 = 0, x2 = 0, x3 = 6, x4 = 5, z = 0 , x1 x2 are nonbasic variables, x3 x4 are basic variables
Choosing the entering variable x2
x3 = 6 + x1 − 2 x2 ≥ 0 ⇒ x2 ≤ 3 (1)
x4 = 5 − x1 − x2 ≥ 0 ⇒ x2 ≤ 5 (2)
(1) is the most stringent.Increasing x2 up to (1).
∴ x1 = 0, x 2 = 3, x3 = 0, x 4 = 3 ( x3 leaves the basis)
To construct the new system, we shall begin with the new comer to the left hand side, Namely, the
variable x2 . The desired formula for x2 in terms of x1 , x3 , x 4 is
x x
x2 = 3 + 1 − 3
2 2
Next, in order to express x4 and z in terms of x1 , x3, we simply substitute
 x x  3 x
x4 = 5 − x1 −  3 + 1 − 3  = 2 − x1 + 3
 2 2 2 2
 x x  x 3
z = x1 − 3  3 + 1 − 3  = −9 − 1 + x3
 2 2 2 2
Hence our new system
x x
x2 = 3 + 1 − 3
2 2
3 x3
x4 = 2 − x1 +
2 2
x 3
z = −9 − 1 + x3
2 2
Increase x1 ( x1 enters the basis)
1
 x1 x3
x2 = 3 + − ≥0 ⇒ x1 can be infinity
2 2
3 x 4
x4 = 2 − x1 + 3 ≥ 0 ⇒ x1 ≤ ( x4 leaves the basis)
2 2 3
4 11
∴ x1 = , x2 = , x3 = 0, x4 = 0
3 3
2 1 2 2 4 1 2
⇒ x1 = 2 ⋅ + ⋅ x3 − x4 = + x3 − x4
3 2 3 3 3 3 3
14 1 2  x 11 1 1
and x2 = 3 +  + x3 − x4  − 3 = − x3 − x4
23 3 3  2 3 3 3
14 1 2  3 29 4 1
z = −9 −  + x3 − x4  + x3 = − + x3 + x4
23 3 3  2 3 3 3
Hence
4 1 2
x1 = + x3 − x4
3 3 3
11 1 1
x2 = − x3 − x4
3 3 3
29 4 1
z=− + x3 + x4
3 3 3
∴ x3 and x4 in the objective function are with positive coefficients.
∴ Stop.

In general from
Minimize
n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

2
3
Some notes:
xs is chosen to increase the objective function, and xr is chosen to maintain feasibility.
A change from one dictionary to the next dictionary is called a pivot, and the equation
s
corresponding to xr is called the pivoting row. The row with the smallest ratio is called the
r
pivot row and corresponds to the leaving variable.

4
2. Pitfall of the simplex method
The examples illustrating the simplex method in the preceding lecture were purposely smooth. They did
not point out the dangers that can occur. The purpose of this section, therefore is to rigorously analyze
the method by scrutinizing its every step.

INITIALIZATION
Given a standard form LP
Maximize
n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

If all bi are nonnegative the initial dictionary is feasible, the basic simplex algorithm solves the

problem.

If some bi < 0 , how to find an initial feasible solution?

An example
Maximize − +
s.t. 2 − + 2 ≤ 4
2 −3 + ≤ −5
− + −2 ≤ −1
, , ≥ 0.

Following the previous discussion, here is the trouble to find an initial feasible solution.
= −4 − 2 +3 −2
= −5 − 2 +3 −2
= −1 + 2 −3 +2

One way of getting rid of this trouble is solving the auxiliary problem for finding the initial feasible
solution. To avoid unnecessary confusion, we write the auxiliary problem in its maximization form:

5
Maximize −
s.t. 2 − + 2 − ≤ 4
2 − 3 + − ≤ −5
− + − 2 − ≤ −1
, , , ≥ 0.

where x0 is called an artificial variable.

Writing down the formulas defining the slack variables x4, x5, x6 and the objective function w, we obtain
the dictionary
= −4 − 2 +3 −2 +
= −5 − 2 +3 −2 +
= −1 + 2 − 3 + 2 +
= −1 + 2 − 3 + 2 −
which is infeasible. Nevertheless, this infeasible dictionary can be transformed into a feasible one by a
single pivot, with x0 entering and x5 leaving the basis: (x5 is the most infeasible basic variable among x4,
x5, x6)
= −5 + 2 −3 +2 +
= −9 − 2 −2 −2 +
= −4 + 3 − 4 + 3 +
= −5 − 2 + 3 − 2 − .

After the first iteration, with x2 entering and x6 leaving:

= −1 + 0.75 + 0.75 + 0.25 − 0.25
= −2 − 0.25 − 1.25 + 0.25 + 0.75
= −7 − 01.5 − 02.5 + 00.5 + 00.5
= −2 + 0.25 + 1.25 − 0.25 − 0.75 .

After the second iteration, with x3 entering and x0 leaving:

= 1.6 − 0.2 + 0.2 + 0.6 − 0.8
= 2.2 + 0.6 + 0.4 + 0.2 − 0.6
= 3.2 − 1.5 − 2.5 − 0.5 + 0.2
= 2.2 + 0.2 + 1.2 − 0.2 − 0.7 .

6
The last dictionary is optimal. Since the optimal value of the auxiliary problem is zero, the last
dictionary points out a feasible solution of the original problem: x1 = 0 , x2 = 2.2 , x3 = 1.6 .
Furthermore, the last dictionary can be easily converted into the desired feasible dictionary of the
original problem. To obtain the first three rows of the desired dictionary, we simply copy down the first
three rows of the last dictionary, omitting all the terms involving x0 :
= 1.6 − 0.2 + 0.2 + 0.6
= 2.2 + 0.6 + 0.4 + 0.2
= 3.2 − 1.5 − 2.5 − 0.5 .
To obtain the last row, we have to express the original objective function
= − +
in terms of the nonbasic variable x1 , x5 , x6 .

= − 2.2 + 0.6 + 0.4 + 0.2 + 1.6 − 0.2 + 0.2 + 0.6

= −0.6 + 0.2 − 0.2 + 0.4 .
In short, the desired dictionary reads
= −1.6 − 0.2 + 0.2 + 0.6
= −2.2 + 0.6 + 0.4 + 0.2
= −3.2 − 1.5 − 2.5 − 0.5
= −0.6 + 0.2 − 0.2 + 0.4 .
General case
Original problem
Maximize

n
Z = ∑cjxj
j =1

s.t.
n

∑a x
j =1
ij j ≤ bi , i = 1, 2,..., m

x j ≥ 0, j = 1, 2,..., n

Auxiliary problem

7
Phase 1:

8
Theorem: The original LP is feasible if and only if the optimum objective value for the auxiliary
problem is 0.
Proof:

ITERATION
Given some feasible dictionary, we have to select an entering variable, to find a leaving variable, and to
construct the next feasible dictionary by pivoting.

Unbounded dictionary
= 5+2 −2 −3
= 7+6 −3 −4
=5+2 −2 −4 .

The entering variable is x3 , but neither of the two basic variables x2 and x5 imposes an upper bound
on its increase. Therefore, we can make x3 as large as we wish (maintaining x1 = x4 = 0 ) and still
retain feasibility: setting x3 = t for any positive t, we obtain a feasible solution with x1 = 0 ,
x2 = 5 + 2t , x4 = 0 , x5 = 7 , and z = 5 + t . Since t can be made arbitrarily large, z can be made
arbitrarily large. We conclude that the problem is unbounded.

Degeneracy
The presence of more than one candidate for leaving the bases has interesting consequences. For
illustration, consider the dictionary.
= 1−2 +2 −2
= 3−2 +4 −6
= 2+1 −3 −4
=6+2 −2 +8 .

9
Having chosen x3 to enter the basis, we find that each of the three basic variables x4 , x5 , x6 limits the
increase of x3 to 1/2. Hence each of these three variables is a candidate for leaving the basis. We
arbitrarily choose x4 . Pivoting as usual, we obtain the dictionary.
= 0.5 − 2 +2 − 0.5
= 0.3 − 2 +4 + 0.3
= 0.2 + 1 −3 + 0.2
= 0.4 + 2 −2 − 0.4 .

Along with the nonbasic variables, the basic variables x5 and x6 have value zero in the associated
solution. Basic solutions with one or more basic variables at zero are called degenerate. Degeneracy
may have annoying side effects. These are illustrated on the next iteration in the example. There, x1
enters the basis and x5 leaves; because of degeneracy, the constraint x5 ≥ 0 limits the increment of
x1 to zero. Hence the value of x1 will remain unchanged, and so will the values of the remaining
variables and the value of the objective function z.

= 0.5 − 2 + 1.5 − 0.5

= 0.5 − 2 − 0.5 + 0.3
= 0.2 − 1 + 3.5 − 0.5
= 0.4 + 3 − 20. − 0.4 .

In this particular iteration, pivoting changes the dictionary as above, but it does not affect the associated
solution at all. Simplex iterations that do not change the basic solution are called degenerate.

10
TERMINATION: CYCLING
Can the simplex method go through an endless sequence of iterations without ever finding an optimal
solution? Yes, it can. To justify this claim, let us consider the initial dictionary

= 0 − 0.5 + 5.5 + 2.5 − 09

= 0 − 0.5 + 1.5 + 0.5 − 00
= 1 − 0.0 + 3.5 − 0.5
= 0 + .10 − .57 − 0.9 − 24 .
and let us agree on the following:
(i) The entering variable will always be the nonbasic variable that has the largest coefficient in the
z-row of the dictionary.
(ii) If two or more basic variables compete for leaving the basis, then the candidate with the
smallest subscript will be made to leave.
Now the sequence of dictionaries constructed in the first six iterations goes as follows.
After the first iteration:
= 0 − .11 + 5.5 − .18 − 02
= 0 − 0.4 − 1.2 + 0.8 + 00
= 1 − .11 − 3.5 + .18 + 02
= 0+. . .53 + .41 − 204 − 20 .
After the second iteration:
= 0 − 0.5 + 1.2 + 0.25 − 0.25
= 0 − 0.5 + 5.4 + 0.75 − 2.75
= 1 + 0.5 − 3.4 − 0.75 − 13.25
= 0+. .14.5 − 98 − 6.75 − 13.25 .
After the third iteration:
= 0−. 8 + 1.5 − 5.5 − 02
= 0 − 0.2 − 0.5 + 2.5 + 00
= 1 − .11 − 3.5 + .18 −0
= 0+. . .18 + .15 − 93 − 29 .
After the fourth iteration:
= 0 − 0.25 + 1.25 + 0.5 − 0.5
= 0 − 00.5 + 14.5 + 0.2 − 0.4
= 1 − 00.5 + 14.5 − 0.2 − 0.4
= 0+. .10.5 − 70.5 − 20 − 9 .
After the fifth iteration:
= 09 +4 −8 −2
= 0 − − 0. 5 + 1.5 + 0.5
= 1 − 0. 5 + 0.2 − 0.4
= 1 + .24 + 22 − 293 − 21 .

11
 After the sixth iteration:
= 0 − 0.5 + 1.5 + 0.5 − 0.5
= 0 − 0.5 + 5.5 + 2.5 − 0.9
= 1 − 0.5 + 4.5 − 0.2 − 0.4
= 1 + .10 − .57 − 2.9 − 24 .

Since the dictionary constructed after the sixth iteration is identical with the initial dictionary, the
method will go through the same six iterations again and again without ever finding the optimal solution
(which, as we shall see later, has z = 1). This phenomenon is known as cycling. More precisely, we say
that the simplex method cycles if one dictionary appears in two different iterations (and so the sequence
of iterations leading from the dictionary to itself can be repeated over and over without end). Note that
cycling can occur only in the presence of degeneracy: since the value of the objective function increases
with each nondegenerate iteration and remains unchanged after each degenerate one, all the iterations in
the sequence leading from a dictionary to itself must be degenerate. Cycling is one reason why the
simplex method may fail to terminate.

Bland’s Rule

The entering variable chosen by the smallest index variables with c j > 0 .

If there is a tie for leaving variables, break the tie by the smallest index.

Recall the previous example if we follow Bland’s rule.

= 0 − 0.5 + 5.5 + 2.5 − 09

= 0 − 0.5 + 1.5 + 0.5 − 00
= 1 − 0.0 + 3.5 − 0.5
= 0 + .10 − .57 − 0.9 − 24 .

12
After the first iteration:
= 0 − .11 + 5.5 − .18 − 02
= 0 − 0.4 − 1.2 + 0.8 + 00
= 1 − .11 − 3.5 + .18 + 02
= 0+. . .53 + .41 − 204 − 20 .
After the second iteration:
= 0 − 0.5 + 1.2 + 0.25 − 0.25
= 0 − 0.5 + 5.4 + 0.75 − 2.75
= 1 + 0.5 − 3.4 − 0.75 − 13.25
= 0+. .14.5 − 98 − 6.75 − 13.25 .
After the third iteration:
= 0−. 8 + 1.5 − 5.5 − 02
= 0 − 0.2 − 0.5 + 2.5 + 00
= 1 − .11 − 3.5 + .18 −0
= 0+. . .18 + .15 − 93 − 29 .
After the fourth iteration:
= 0 − 0.25 + 1.25 + 0.5 − 0.5
= 0 − 00.5 + 14.5 + 0.2 − 0.4
= 1 − 00.5 + 14.5 − 0.2 − 0.4
= 0+. .10.5 − 70.5 − 20 −9 .
After the fifth iteration:
= 09 +4 −8 −2
=0− − 0. 5 + 1.5 + 0.5
= 1 − 0. 5 + 0.2 − 0.4
= 1 + .24 + 22 − 293 − 21 .

After the sixth iteration: (different from the previous example)

= −2 +3 + − 2
=0 +4 + −8
=1+2 −3 − +2 0
= 1+. −20 − .27 + 2. − 44 .

After the seventh iteration: ( x3 enters and x7 leaves)

=1 +2 −3 − +2 0
=⋯
=⋯
= 1 − 18 − .30 − 2. − 42 .

13
Theorem: The simplex method always terminates provided that both the entering and the leaving
variable are chosen according to Bland’s rule.
Proof

14
15
16
3. Geometric interpretation of the simplex method

Maximize = 4 +5
s.t. 6 +4 ≤ 24 (1) 6 +4 + + + + = 24
+2 ≤6 (2) +2 + + + + = 6
− + ≤1 (3) − + + + + + = 1
≤2 (4) 6 + + + + + = 2
, ≥0

Choose the entering variable with the most positive objective function coefficient
= 24 − 6 −4
= 6− −2
= 1+ −
= 2 −
= 4 +5
The first iteration:
= 1+ −
= 20 − 10 +4
= 4− 3 +2
= 1− +
= 5+ 9 −5
The second iteration:
= 1+ −
= 2+ −
= 10 − 6 + 10
= 1− +3
= 14 + 4 − 9
The third iteration:
= 2 − +2
= 2 + −
= 4 +6 −8
= 1 − +3
= 18 − 4 +3

17
The fourth iteration:
1 1
= 3 − +
4 2
3 1 3
= + −
2 8 4
5 3 5
= − +
2 8 8
1 1 3
= − +
2 8 4
39 3 7
= − −
2 8 4
z = 4x + 5x : A line describes all point satisfying z = 4x + 5x
The line is perpendicular to the vector # $ (normal vector).
Different values of lead to different lines, and all of them are parallel to each other. Increasing z
corresponds to moving the line along the direction of # $. We would like to move the line as much
as possible in the direction of # $.

18
 Choose the entering variable with the least positive objective function coefficient.
The first iteration:
2 1
= 4− −
3 6
4 1
= 2− +
3 6
5 1
= 5− −
3 6
2
= 2− +
3
7 2
= 16 + −
3 3
The second iteration:
1 1
= 3− +
4 2
3 1 3
= + −
2 8 4
5 3 5
= − +
2 8 8
1 1 3
= − +
2 8 4
39 3 7
= − −
2 8 4

19
3-Dimension graph

20
Degenerate
Maximize = 3 +9
s.t. +4 ≤8 (1) +4 + + + + =8
+2 ≤4 (2) +2 + + + + =4
, ≥0

= 8− −4
= 4− −2
= 5+ 3 +9

The first iteration ( enters leaves)

1 1
= 2− −
4 4
1 1
= − +
2 2
3 9
= 18 + −
4 4

The second iteration ( enters leaves)

1
= − −2
2
1 1
= 2− +
2 2
3 3
= 18 − −
2 2

21
 Lecture 3 Duality Theory

1. Finding upper bounds on the optimal objective value of maximization problems

Consider the following maximization problem.
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 + 1𝑥 + 5𝑥 + 3𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1𝑥 − 1𝑥 − 1𝑥 + 3𝑥 ≤ 1
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 5𝑥 + 1𝑥 + 3𝑥 + 8𝑥 ≤ 55
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡 − 𝑥 + 2𝑥 + 3𝑥 − 5𝑥 ≤ 3
𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 0.
To get a reasonably good lower bound on the objective function value, we need only come up with a
reasonably good feasible solution.

What is upper bound on the optimal objective function value?

Can we push down the upper bound any further?

1
Motivation of the dual problem
Rather than searching for further improvements in a haphazard way, we shall now describe the strategy
in precise and general terms. We construct linear combinations of the constraints. That is, we multiply
the first constraint by some number y1, the second by y2, the third by y3, and then we add them up. (In
the first case, we had y1 = 0, y2 = 5/3, y3 = 0; in the second case, we had y1 = 0, y2 = y3 = 1.) The resulting
inequality reads

1𝑥 − 1𝑥 − 1𝑥 + 3𝑥 ≤ 1
5𝑥 + 1𝑥 + 3𝑥 + 8𝑥 ≤ 55
−𝑥 + 2𝑥 + 3𝑥 − 5𝑥 ≤ 3
𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 0.

2
Of course, we want as small an upper bound on z* as we can possibly get. Thus, we are led to the
following LP problem:
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑦 + 55𝑦 − 3𝑦
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑦 + 5𝑦 − 1𝑦 ≥ 4
−𝑦 + 1𝑦 + 2𝑦 ≥ 1
− 𝑦 + 3𝑦 + 3𝑥 ≥ 5
3𝑦 + 8𝑦 − 5𝑦 ≥ 3
𝑦 , 𝑦 , 𝑦 ≥ 0.

2. The dual problem

This problem is called the dual of the original one; the original problem is called the primal problem. In

general, the dual of the problem

𝑚𝑎𝑥𝑚𝑖𝑧𝑒 𝑐𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑥 ≤𝑏 (𝑖 = 1, 2, … , 𝑚)

(2)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑥 ≥0 (𝑗 = 1, 2, … , 𝑛).

is defined to be the problem

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑏𝑦

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑦 ≥𝑐 (𝑗 = 1, 2, … , 𝑛)

(3)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑦 ≥0 (𝑖 = 1, 2, … , 𝑚).

(Note that the dual of a maximization problem is a minimization problem. Furthermore, the m primal

constraints ∑ 𝑎 𝑥 ≤ 𝑏 are in a one-to-one correspondence with the m dual variable yi; conversely,

the n dual constraints ∑ 𝑎 𝑦 ≥ 𝑐 are in a one-to-one correspondence with the n primal variables xj.

The coefficient at each variable in the objective function, primal or dual, appears in the other problem as

the right-hand side of the corresponding constraint.)

3
Recall the example
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 + 1𝑥 + 5𝑥 + 3𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1𝑥 − 1𝑥 − 1𝑥 + 3𝑥 ≤ 1
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 5𝑥 + 1𝑥 + 3𝑥 + 8𝑥 ≤ 55
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡 − 𝑥 + 2𝑥 + 3𝑥 − 5𝑥 ≤ 3
𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 0.

4
Theorem: Weak duality

For every primal feasible solution (x1, …, xn) and for every dual feasible solution (y1, …, ym), we have

𝑐𝑥 ≤ 𝑏𝑦

Proof:

Theorem: Strong duality

If the primal problem has an optimal solution (x1*, …, xn*), then the dual problem has an optimal

solution (y1*, …, ym*) such that

∑ 𝑐𝑥 =∑ 𝑏𝑦. (4)

Proof:

5
6
Complementary Slackness

Theorem：Let 𝑥 ∗ ⋯ 𝑥 ∗ be a feasible solution of the primal problem and 𝑦 ∗ ⋯ 𝑦 ∗ be a feasible solution

of the dual problem. Both solutions are optimal if and only if

𝑥∗ ∙ 𝑦 ∗ = 0, 𝑖 = 1, ⋯ , 𝑚 (9)

𝑥∗ ∙ 𝑦∗ = 0, 𝑗 = 1, ⋯ , 𝑛 (10)

(9) tells us that either the ith inequality in the primal problem holds at equality or the ith dual variable is

equal to 0.

𝑎 𝑥∗ = 𝑏 𝑜𝑟 𝑦 ∗ = 0, 𝑖 = 1, ⋯ , 𝑚

(10) tells us that either the ith inequality in the dual problem holds at equality or the jth primal variable is

equal to 0.

𝑎 𝑦∗ = 𝑐 𝑜𝑟 𝑥 ∗ = 0, 𝑗 = 1, ⋯ , 𝑛

Proof：

7
8
Theorem A feasible solution 𝑥 ∗ , 𝑥 ∗ ⋯ , 𝑥 ∗ of the primal problem (2) is optimal if and only if there
are numbers 𝑦 ∗ , 𝑦 ∗ ⋯ , 𝑦 ∗ such that

𝑎 𝑦∗ = 𝑐 whenever 𝑥∗ > 0

𝑦∗ = 0 whenever 𝑎 𝑥∗ < 𝑏 . (13)

and such that

𝑎 𝑦∗ ≥ 𝑐 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑗 = 1, 2, ⋯ , 𝑛

𝑦 ∗ ≥ 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖 = 1, 2, ⋯ , 𝑚. (14)

Example
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 18𝑥 − 7𝑥 + 12𝑥 + 5𝑥 + 0x + 8x
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 12𝑥 − 6𝑥 + 12𝑥 + 7𝑥 + 3x + 8x ≤ −1
𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑡𝑜 − 3𝑥 − 1𝑥 + 34𝑥 − 3𝑥 + 0x + 2x ≤ −2
𝑠𝑢𝑏𝑗𝑒 𝑐8𝑥 − 3𝑥 + 15𝑥 − 2𝑥 + 0x + 2x ≤ −4
𝑠𝑢 𝑏𝑗 𝑒𝑐𝑡𝑡𝑜14𝑥 + 1𝑥 + 38𝑥 + 7𝑥 − 0x + 3x ≤ −1
𝑠𝑢𝑏𝑗 𝑒 𝑐𝑡𝑡𝑜15𝑥 + 2𝑥 − 33𝑥 + 6𝑥 − 2x − 2x ≤ −5
𝑥 , 𝑥 , 𝑥 , 𝑥 , x , x ≥ −0.

First, let us consider the claim that

x ∗ = 2, x ∗ = 4, x ∗ = 0, x ∗ = 0, x ∗ = 7, x ∗ = 0

is an optimal solution of the problem

9
Example
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 8𝑥 − 9𝑥 + 12𝑥 + 4𝑥 + 11𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑥 − 3𝑥 + 14𝑥 + 3𝑥 + 13𝑥 ≤ 11
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 5𝑥 + 7𝑥 + 13𝑥 − 2𝑥 + 11𝑥 ≤ 51
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 5𝑥 + 4𝑥 − 16𝑥 + 2𝑥 + 13𝑥 ≤ 22
𝑥 , 𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 10.
Second, let us consider the claim that

x ∗ = 0, x ∗ = 2, x ∗ = 0, x ∗ = 7, x ∗ = 0

is an optimal solution of the problem

10
3. Finding the dual problem in general

Very often linear programs are encountered in equality form with nonnegative variables. For example,

the canonical form, which is used for computing a solution by the simplex method, is in equality form.

It is of interest, then, to find the dual of the equality form:

𝑚𝑎𝑥𝑚𝑖𝑧𝑒 𝑧 = 𝑐𝑥,

subject to:

∑ 𝑎 𝑥 =𝑏 (𝑖 = 1, 2, … , 𝑚), (15)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑥 ≥0 (𝑗 = 1, 2, … , 𝑛).

A problem in equality form can be transformed into inequality form by replacing each equation by two

inequalities. Formulation (15) can be rewritten as

11
Table: Conversion of the primal and dual problems

12
Example

Example

13
4. Economic interpretation

Let us begin with some basic terminology and notation. The second column then introduces more

general terms for these same components that will fit many linear programming problems. The key

terms are resources and activities, where m denotes the number of different kinds of resources that can

be used and n denotes the number of activities being considered. Some typical resources are money and

particular kinds of machines equipment, vehicles, and personnel. Examples of activities include

investing in particular projects, advertising in particular media, and shipping goods from a particular

source to a particular destination. In any application of linear programming, all the activities may be of

one general kind (such as any one of these three examples), and then the individual activities would be

particular alternatives within this general category.

As described in the introduction to this chapter, the most common type of application of linear

programming involves allocating resources to activities. The amount available of each resource is

limited, so a careful allocation of resources to activities must be made. Determining this allocation

involves choosing the levels of the activities that achieve the best possible value of the overall measure

of performance. Certain symbols are commonly used to denote the various components of a linear

programming model. These symbols are listed below, along with their interpretation for the general

problem of allocating resources to activities.

𝑍 = value of overall measure of performance.

𝑥 = level of activity 𝑗 (for 𝑗 = 1,2, … , 𝑛).

𝑐 = increase in Z that would result from each unit increase in level of activity 𝑗.

𝑏 = amount of resource 𝑖 that is available for allocation to activities (for 𝑗 = 1,2, … , 𝑚).

𝑎 = amount of resource 𝑖 consumed by each unit of activity 𝑗.

14
 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑍 = 𝑐 𝑥 + 𝑐 𝑥 + ⋯ + 𝑐 𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑎 𝑥 + 𝑎 𝑥 + ⋯ + 𝑎 𝑥 ≤ 𝑏

𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑎 𝑥 + 𝑎 𝑥 + ⋯ + 𝑎 𝑥 ≤ 𝑏

𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑎 𝑥 +𝑎 𝑥 +⋯ + 𝑎 𝑥 ≤ 𝑏

𝑥 ≥ 0, 𝑥 ≥ 0, … … , 𝑥 ≥ 0.

The dual problem

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑏𝑦

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑦 ≥𝑐 (𝑗 = 1, 2, … , 𝑛)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑎 𝑦 ≥0 (𝑖 = 1, 2, … , 𝑚).

Each 𝒃𝒊 𝒚𝒊 can thereby be interpreted as the current contribution to profit by having 𝒃𝒊 units of

resource 𝒊 available for the primal problem. Thus, the dual variable 𝑦 , is interpreted as the

contribution to profit per unit of resource 𝑖 (𝑖 = 1, 2, . . . , 𝑚), when the current set of basic variables

is used to obtain the primal solution.

In other words, the 𝑦 , values (or 𝑦 ∗ values in the optimal solution) are just the shadow prices of the

resource i. This interpretation of the dual variables leads to our interpretation of the overall dual

problem. Specifically, since each unit of activity 𝑗 in the primal problem consumes 𝑎 units of

resource 𝑖,

∑𝒎
𝒊 𝟏 𝒂𝒊𝒋 𝒚𝒊 is interpreted as the current contribution to profit of the mix of resources that would

be consumed if 1 unit of activity 𝒋 were used (𝒋 = 𝟏, 𝟐, . . . , 𝒏).

For each activity 𝑗, this same mix of resources (and more) probably can be used in other ways as well,

but no alternative use should be considered if it is less profitable than 1 unit of activity 𝑗. Since 𝑐 is

interpreted as the unit profit from activity 𝑗, each functional constraint in the dual problem is interpreted

as follows:

15
∑𝒎
𝒊 𝟏 𝒂𝒊𝒋 𝒚𝒊 ≥ 𝒄𝒋 says that the actual contribution to profit of the above mix of resources must be at

least as much as if they were used by 1 unit of activity 𝒋. Otherwise, we would not be making the

best possible use of these resources.

𝒚𝒊 ≥ 𝟎 says that the contribution to profit of resource 𝒊 (𝒊 = 𝟏, 𝟐, . . . , 𝒎) must be nonnegative:

otherwise, it would be better not to use this resource at all.

The objective function

𝒎

𝐦𝐢𝐧 𝒃 𝒊 𝒚𝒊
𝒊 𝟏

can be viewed as minimizing the total implicit value of the resources consumed by the activities.

This interpretation can be sharpened somewhat by differentiating between basic and nonbasic variables

in the primal problem for any given basic feasible solution (𝑥 , 𝑥 , … , 𝑥 ). Recall that the basic

variables (the only variables whose values can be nonzero) always have a coefficient of zero in z-row.

Therefore, we have

𝑎 𝑦 = 𝑐, 𝑖𝑓 𝑥 > 0 (𝑗 = 1, 2, . . . , 𝑛),

𝑦 = 0, 𝑖𝑓 𝑥 > 0 (𝑖 = 1, 2, . . . , 𝑚).

(This is one version of the complementary slackness property) The economic interpretation of the first

statement is that whenever an activity 𝑗 operates at a strictly positive level (𝑥 > 0), the marginal

value of the resources it consumes must equal (as opposed to exceeding) the unit profit from this activity.

The second statement implies that the marginal value of resource 𝑖 is zero (𝑦 = 0) whenever the

supply of this resource is not exhausted by the activities (𝑥 > 0). In economic terminology, such a

resource is a “free good”; the price of goods that are oversupplied must drop to zero by the law of

supply and demand. This fact is what justifies interpreting the objective for the dual problem as

minimizing the total implicit value of the resources consumed, rather than the resources allocated.
16
 Lecture 4 The Revised Simplex Method

In each iteration of the simplex method, one basic feasible solution is replaced by another. When the old

solution is represented by a dictionary, the new solution is easy to find, and only a small part of the

dictionary is actually used for that purpose. This part may be reconstructed directly from the original data

and the new solution found without any reference to dictionaries. The resulting implementations of the

simplex method are known under the generic name of the revised simplex method; the implementation of

the simplex method that updates a dictionary in each iteration is known as the standard simplex method.

1. Matrix description of dictionaries

maxmize 19𝑥 + 13𝑥 + 12𝑥 + 17𝑥

subject to 13𝑥 + 12𝑥 + 12𝑥 + 12𝑥 ≤ 225

subject to 13𝑥 + 12𝑥 + 12𝑥 + 12𝑥 ≤ 117

subject to 14𝑥 + 13𝑥 + 13𝑥 + 14𝑥 ≤ 420

𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 100
After two iterations of the simplex method
𝑥 = 1754 − 0.5𝑥 − 0.5𝑥 − 0.5𝑥 + 0.5𝑥
𝑥 = 1763 − 0.5𝑥 − 0.5𝑥 + 0.5𝑥 − 1.5𝑥
𝑥 = 1715 + 0.5𝑥 − 0.5𝑥 + 0.5𝑥 + 2.5𝑥
𝑧 = 1782 − 2.5𝑥 + 1.5𝑥 − 3.5𝑥 − 8.5𝑥

1
More generally, consider an arbitrary LP problem in the standard form

maxmize ∑ 𝑐𝑥

subject to ∑ 𝑎 𝑥 ≤𝑏 (𝑖 = 1, 2, … , 𝑚)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 ∑ 𝑎 𝑥 ≥0 (𝑗 = 1, 2, … , 𝑛).

After the introduction of the slack variables xn+1, xn+2,…, xn+m,this problem may be recorded as

maxmize 𝐜𝐱

subject to 𝐀𝐱 = 𝐛

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝐱 ≥ 𝟎.

2
2. The revised simplex method

In each iteration of the simplex method, we first choose the entering variable, then find the leaving

variable, and finally update the current basic feasible solution. An examination of the way these tasks are

carried out in the standard simplex method will lead us to the alternative, the revised simplex method. For

illustration, we shall consider the update of the feasible dictionary on page 1 in the standard simplex

method. The corresponding iteration of the revised simplex method begins with

x∗ 54 3 1 0
𝐱 𝐁∗ = x ∗ = 63 and 𝐁 = 1 1 0.
x∗ 15 4 3 1

The entering variable may be any nonbasic variable with a positive coefficient in the last row of the
𝟏
dictionary. As previously observed, the coefficients in this row form the vector 𝐜𝐍 − 𝐜𝐁 𝐁 𝐀 𝐍 . If the

standard simplex method is used, then this vector is readily available as part of the dictionary; in our

example, we have

𝑧 = ⋯ − 2.5𝑥 + 1.5𝑥 − 3.5𝑥 − 8.5𝑥 .

3
4
Iteration of the revised simplex method
Step 1. Solve the system 𝐲𝐁 = 𝐜𝐁
Step 2. Choose an entering column. This may be any column a of AN such that ya is less than the
corresponding component of cN. If there is no such column, then the current solution is optimal.
Step 3. Solve the system Bd = a.
Step 4. Find the largest t such that 𝐱 𝐁∗ − 𝑡𝐝 ≥ 𝟎. If there is no such t, then the problem is unbounded;
otherwise, at least one component of 𝐱 𝐁∗ − 𝑡𝐝 equals zero and the corresponding variable is leaving the
basis.
Step 5. Set the value of the entering variable at t and replace the values 𝐱 𝐁∗ of the basic variables by
𝐱 𝐁∗ − 𝑡𝐝. Replace the leaving column of B by the entering column and, in the basis heading, replace the
leaving variable by the entering variable.

5
3. How to handle explicit bounds on individual variables

Many linear programming problems involve explicit upper bounds on individual variables. For

instance one might be confronted with the problem

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑐𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑎 𝑥 ≤𝑏 (𝑖 = 1, 2, ⋯ , 𝑚)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤𝑢 (𝑗 = 1, 2, ⋯ , 𝑛)

such that each 𝑢 is a positive number. This problem may be cast in the standard form

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑐𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑎 𝑥 ≤𝑏 (𝑖 = 1, 2, ⋯ , 𝑚)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤𝑢 (𝑗 = 1, 2, ⋯ , 𝑛)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≥0 (𝑗 = 1, 2, ⋯ , 𝑛)

The above standard form can be solved by the simplex method. However, the upper bound or lower

bound on individual variables may be viewed as some special conditions when pivoting rather than adding

constraints. By doing so, we may save some computational time. Let us see the following example.

Consider

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 3𝑥 + 5𝑥 + 2𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 + 𝑥 + 2𝑥 ≤ 14

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑥 + 4𝑥 + 3𝑥 ≤ 43

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤4

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 7 ≤ 𝑥 ≤ 10

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤3

6
You can convert it into the standard form
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 3𝑥 + 5𝑥 + 2𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 + 𝑥 + 2𝑥 ≤ 14
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑥 + 4𝑥 + 3𝑥 ≤ 43
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 + 4𝑥 + 3𝑥 ≤ 4
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑥 − 4𝑥 + 3𝑥 ≤ −7
4𝑥 + 3𝑥 ≤ 10
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2 + 3𝑥 ≤ 3
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 ,𝑥 ,𝑥 ≥ 0
Is there any way to compute it efficiently?
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 3𝑥 + 5𝑥 + 2𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 + 𝑥 + 2𝑥 + 𝑥 + 𝑥 = 14
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑥 + 4𝑥 + 3𝑥 + 𝑥 + 𝑥 = 43
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤4
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 7 ≤ 𝑥 ≤ 10
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 0≤𝑥 ≤3
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 ,𝑥 ≥ 0

7
8
4. Dual-feasible dictionaries

Consider the primal problem.

𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 − 13𝑥 + 7𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 13𝑥 + 2𝑥 + 5𝑥 ≤ 5
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 − 3𝑥 + 2𝑥 ≤ 3
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 , 𝑥 , 𝑥 ≥ 0.

Add the slack variables.

𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 − 13𝑥 + 7𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 13𝑥 + 2𝑥 + 5𝑥 + 𝑥 = 5……….𝑦

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 − 3𝑥 + 2𝑥 + 𝑥 = 3……….𝑦
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 , 𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 0.

Our objective is to point out a certain one-to-one correspondence between the dictionaries the primal and

dual problems. In order to make this correspondence more transparent, we change the way to subscript

the primal variables.

𝑥 → 𝑥 ,𝑥 → 𝑥 ,𝑥 → 𝑥 ,𝑥 → 𝑥 ,𝑥 → 𝑥 ,
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 − 13𝑥 + 7𝑥
𝑚𝑎𝑥𝑚𝑖𝑧𝑒 4𝑥 − 13𝑥 + 7𝑥
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 13𝑥 + 2𝑥 + 5𝑥 ≤ 5
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 13𝑥 + 2𝑥 + 5𝑥 + 𝑥 = 5…… 𝑦
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 − 3𝑥 + 2𝑥 ≤ 3
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥 − 3𝑥 + 2𝑥 + 𝑥 = 3……𝑦
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 , 𝑥 , 𝑥 ≥ 0.
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑥 , 𝑥 , 𝑥 , 𝑥 , 𝑥 ≥ 0.

The dual problem is

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 5𝑦 + 3𝑦 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 − 5𝑦 − 3𝑦
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 13𝑦 + 𝑦 ≥ 4 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 3𝑦 − 𝑦 + 𝑦 + 𝑦 + 𝑦 = −4
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 2𝑦 − 3𝑦 ≥ −13 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 2𝑦 + 3𝑦 + 𝑦 + 𝑦 + 𝑦 = 13
𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 5𝑦 + 2𝑦 ≥ 7 𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 5𝑦 − 2𝑦 + 𝑦 + 𝑦 + 𝑦 = −7
𝑠𝑢𝑏𝑗𝑒𝑐 𝑡𝑦 , 𝑦 ≥ 0. 𝑠𝑢𝑏𝑗𝑒𝑐 𝑡𝑦 , 𝑦 , 𝑦 , 𝑦 , 𝑦 ≥ 0.

The initial dictionaries are

𝑦 = −4 + 3𝑦 + 𝑦
𝑥 = 5 − 3𝑥 − 2𝑥 − 5𝑥
𝑦 = 13 + 2𝑦 − 3𝑦
𝑥 = 3− 𝑥 + 3𝑥 − 2𝑥
𝑦 = −7 + 5𝑦 + 2𝑦
𝑧= 4𝑥 − 13𝑥 + 7𝑥
−𝑤 = −7 − 5𝑦 − 3𝑦

9
𝑥 enters and 𝑥 leaves.
𝑥 =− 4 + 3𝑥 − 11𝑥 + 𝑥
𝑥 = 3− 𝑥 + 3𝑥 − 2𝑥
𝑧 = 12 − 4𝑥 − 𝑥 − 𝑥

By complementary slackness, 𝑥 𝑦 = 0 for all 𝑖 = 1, … 5. y3 leaves and y2 enters. We have

𝑦 =4− 3𝑦 + 𝑦
𝑦 = 1 + 11𝑦 − 3𝑦
𝑦 =1− 𝑦 + 2𝑦
−𝑤 = −12 + 4𝑦 − 3𝑦

Similarly,
𝑦 =1+ 2𝑦 − 𝑦
𝑥 = −5 − 2𝑥 + 5𝑥 − 19𝑥
𝑦 =1− 5𝑦 + 3𝑦
𝑥 = 4+ 𝑥 − 3𝑥 + 11𝑥
𝑦 = 12 + 19𝑦 − 11𝑦
𝑧 = 8− 𝑥 − 𝑥 − 12𝑥
−𝑤 = −8 + 5𝑦 − 4𝑦

In each of these pairs, the primal dictionary is a mirror image of the dual dictionary. The general

representation is

𝑥 =𝑏 − 𝑎 𝑥 (𝑟 ∈ 𝐵)
∈

𝑧 = 𝑑̅ − 𝑐 𝑥
∈

The corresponding dual dictionary is

−𝑦 = −𝑐 + 𝑎 𝑦 (𝑠 ∈ 𝑁)
∈

−𝑤 = −𝑑̅ − 𝑏 𝑦
∈

The primal dictionary is feasible if br  0 for all r  B . Now we refer to the dual dictionary as dual-

feasible if the corresponding dual dictionary is feasible. Thus, the dual dictionary is feasible if and only if

cs  0 for all s  N .

10
5. The dual simplex method

We have observed that solving an LP problem by the simplex method, we obtain a solution of its dual as

a by-product. Vice versa, solving the dual we also solve the primal. This observation is useful for solving

problems such as

𝑚𝑎𝑥𝑚𝑖𝑧𝑒 − 4𝑥 − 8𝑥 − 9𝑥

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 2𝑥 − 𝑥 − 𝑥 ≤ 1

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 3𝑥 − 4𝑥 + 𝑥 ≤ 3 (1)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 5𝑥 − 𝑥 − 2𝑥 ≤ −8

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 𝑥 ,𝑥 ,𝑥 ≥ 0.

The initial dictionary is

𝑥 = 1 − 2𝑥 + 𝑥 + 𝑥

𝑥 = 3 − 3𝑥 + 4𝑥 − 𝑥

𝑥 = −8 + 5𝑥 − 𝑥 + 2𝑥

𝑧 = −8 − 4𝑥 − 8𝑥 − 9𝑥

Since this problem does not have feasible origin, the routine approach calls for the two-phase method.

Nevertheless, we can avoid the two-phase method as soon as we realize that the dual of (1) does have

feasible origin.

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑦 + 3𝑦 − 8𝑦

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 2𝑦 + 3𝑦 − 5𝑦 ≥ −4

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 − 𝑦 − 4𝑦 + 𝑦 ≥ −8 (2)

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 𝑦 + 𝑦 − 2𝑦 ≥ −9

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 𝑦 ,𝑦 ,𝑦 ≥ 0.

Hence we may simply solve the dual and then read the optimal primal solution of the final table for

the dual. In this section, we shall discuss the dual simplex method of solving the dual without actually

doing so. We shall first describe it as a mirror image of the simplex method and then we shall illustrate it

on the presented example. We note that the dual simplex method is nothing but a disguised simplex

method working on the dual. The dual simplex method produces a sequence of dual feasible dictionaries;
11
as soon as it finds one which is also primal feasible, the method terminates. In each iteration of the simplex

method, we first choose the entering variable and then determine the leaving variable. For the entering

variable, we may choose any nonbasic variable with a positive coefficient in the z-row. Then we determine

the leaving variable so as to preserve primal feasibility in our next dictionary. On the other hand, in each

iteration of the dual simplex method, we first choose the leaving variable and then determine the entering

variable. For the leaving variable, we may choose any basic variable whose current value is most negative.

Then we shall determine the entering variable so as to preserve dual feasibility in our next dictionary.

The standard form of the dual problem is

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 − 𝑦 − 3𝑦 + 8𝑦

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 − 2𝑦 − 3𝑦 + 5𝑦 ≤4

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 − 𝑦 + 4𝑦 + 𝑦 ≤8

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 − 𝑦 − 𝑦 + 2𝑦 ≤9

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 1 𝑦 , 𝑦 , 𝑦 ≥ 0.

12
13
16
3. Geometric interpretation of the simplex method

Maximize = 4 +5
s.t. 6 +4 ≤ 24 (1) 6 +4 + + + + = 24
+2 ≤6 (2) +2 + + + + = 6
− + ≤1 (3) − + + + + + = 1
≤2 (4) 6 + + + + + = 2
, ≥0

Choose the entering variable with the most positive objective function coefficient
= 24 − 6 −4
= 6− −2
= 1+ −
= 2 −
= 4 +5
The first iteration:
= 1+ −
= 20 − 10 +4
= 4− 3 +2
= 1− +
= 5+ 9 −5
The second iteration:
= 1+ −
= 2+ −
= 10 − 6 + 10
= 1− +3
= 14 + 4 − 9
The third iteration:
= 2 − +2
= 2 + −
= 4 +6 −8
= 1 − +3
= 18 − 4 +3

17
The fourth iteration:
1 1
= 3 − +
4 2
3 1 3
= + −
2 8 4
5 3 5
= − +
2 8 8
1 1 3
= − +
2 8 4
39 3 7
= − −
2 8 4
z = 4x + 5x : A line describes all point satisfying z = 4x + 5x
The line is perpendicular to the vector # $ (normal vector).
Different values of lead to different lines, and all of them are parallel to each other. Increasing z
corresponds to moving the line along the direction of # $. We would like to move the line as much
as possible in the direction of # $.

18
 Choose the entering variable with the least positive objective function coefficient.
The first iteration:
2 1
= 4− −
3 6
4 1
= 2− +
3 6
5 1
= 5− −
3 6
2
= 2− +
3
7 2
= 16 + −
3 3
The second iteration:
1 1
= 3− +
4 2
3 1 3
= + −
2 8 4
5 3 5
= − +
2 8 8
1 1 3
= − +
2 8 4
39 3 7
= − −
2 8 4

19
3-Dimension graph

20
Degenerate
Maximize = 3 +9
s.t. +4 ≤8 (1) +4 + + + + =8
+2 ≤4 (2) +2 + + + + =4
, ≥0

= 8− −4
= 4− −2
= 5+ 3 +9

The first iteration ( enters leaves)

1 1
= 2− −
4 4
1 1
= − +
2 2
3 9
= 18 + −
4 4

The second iteration ( enters leaves)

1
= − −2
2
1 1
= 2− +
2 2
3 3
= 18 − −
2 2

21
3
2. The Supremum Property of R
The supremum property of the real numbers will help us to ensure that S  R exists as a real number.
The formulation of this property involves upper or lower bounds for sets of real numbers.

SUPREMA AND INFIMA

Definition 2.1
Let S  R.
(a) We say that u  R is an upper bound of S if s  u for all s  S .
(b) We say that w R is a lower bound of S if s  w for all s  S .

Examples

(I) If S   x  R : 0  x  1 , then 1 is an upper bound, but so is any u  1. Likewise 0 is a lower

bound, and so is any w  0 .

(II) If S  R , then S does not have an upper or lower bound.
Remarks
(a) If a set has an upper bound, then we say that it is bounded above;
(b) If a set has a lower bound, then we say that it is bounded below;
(c) If a set has both an upper and lower bound, then we say that it is bounded; otherwise it is
unbounded.
Examples

(I) S   x  R : 0  x  1 is bounded above and below, so is bounded;

(II) P   x  R : x  0 is bounded below, but not above, so is unbounded.

Definition 2.2
Let S  R .
(a) Let S be bounded above. An upper bounded of S is said to be a supremum (or least upper
bound), if it is less than any other upper bound of S . We denote the supremum by sup S .
(b) Let S be bounded below. A lower bounded of S is said to be an infimum (or greatest lower
bound), if it is larger than any other lower bound of S . We denote the infimum by inf S .

4
Remarks
We see that u is a supremum of S if and only if it satisfies the following two conditions:
(i) s  u for all s  S .
(ii) if v is any upper bound for S then u  v.
Thus a supremum is the ”least upper bound” or “ smallest upper bound”. It is then fairly easy to see
that if exists, it must be unique. Similarly for an infimum.

Lemma 2.3
Let S  R be non-empty. Then u R is the supremum of S if and only if both
(i) there are no elements s  S with u  s ;
(ii) If v  u , then there exists s  S with v  s .

Examples

(I) Let S1   x  R : 0  x  1 . Then

sup S  1;inf S  0

(II) Let S2   x  R : 0  x  1 . Then again

sup S  1;inf S  0

Note that S1 does not contain its supremum or infmum, but S 2 contains both.

(III) Let P   x  R : x  0 . Then inf P  0 , but sup P does not exist.

Property 2.4 Supremum Property

Every non-empty set of real numbers which has an upper bound has a supremum.
Property 2.5 Infimum Property
Every non-empty set of real numbers which has a lower bound has a infimum.

5
3. Open and Closed Sets
We shall discuss topological notions such as open and closed sets, interior points. Recall that R p is the set

of p -tuples

x   x1 , x2 ,...x p 


with

x  x12  x22  ...x 2p .



Definition 3.1

A set G in R p is said to be open in R p (or merely open) if for each x  G , there exists r  0 such that
 
 y  R : y  x  r   G
p

   
Remark

Thus G is open if each point in G is the center of some open ball contained in G .

Examples

(I) (0,1) is open in R and ( x, y )  R 2 : x 2  y 2  1 is open in R 2 .

(II)  0,1 and  0,1 are not open in R and ( x, y )  R 2 : x 2  y 2  1 is not open in R 2 . Why?

We now establish some properties of open sets, which are often taken as the starting point for studying
topology:

Theorem 3.3
(a) The empty set is open in R p ; R p is open in R p .
(b) The intersection of any two open sets is open.
(c) The union of any collection of open sets is open.
Proof

6
7
CLOSED SETS
Definition 3.4
A set F in R p is said to be closed if R p \ F is open.

Thus a set is closed if its complement is open.

Examples

(I)  0,1 and  0,  are closed in R and ( x, y )  R 2 : x 2  y 2  1 is closed in R 2

(II)  0,1 is not closed in R .

(III) The empty set is closed in R p

Proposition 3.6 Properties of closed Sets

(a) The empty set  and the whole space R p are closed in R p .
(b) The union of any two closed sets is closed in R p
(c) The intersection of any collection of closed sets is closed in R p .

NEIGHBORHOODS
Definition 3.7

(a) If x  R p , then any set which contains an open set containing x is called a neighborhood of x .
  

Now let A  R p and x  A . (Definition by Marsden and Hoffman, 1993: A neighborhood of a point


is an open set containing that point.)

(b) We call x an interior point of A if there is a neighborhood of x that is contained in A . The
 

interior of A is the collection of all interior points of A and is denoted int(A).

(c) We call x a boundary point of A if every neighborhood of x contains a point in A and a point in
 

Rp \ A.
(d) We call x an exterior point of A if there is a neighborhood of x that is contained in R p \ A .
 

Remark
Note that exactly one of the three holds: x is interior or boundary or exterior, but cannot be more than
one of these.

8
Examples

(I) Let A   0,1 in R . Then any point in  0,1 is an interior point of A , while 0 and 1 are boundary

points. The exterior points of A are R \  0,1 . The same is true if we take A   0,1 or  0,1 .

(II) Let A   y  R p : y  x  r  be an open ball in R p . We see that each point in A is an interior point,
   

while the boundary points of A are those points y with y  x  r . The exterior points of A are those
  

y  R p with y  x  r . Similarly for the closed ball  y  R p : y  x  r  .

  
   

We may characterize open and closed sets in terms of their interior / boundary points:

Theorem 3.10
A set F  R p is closed iff it contains all its boundary points.

Theorem 3.11 The Heine Borel Theorem

Let K  R p . Then K is compact iff it is closed and bounded.
Remark

It was seen the closed set K  1,   is not compact; note that K is not bounded. It was also seen in

Example that the bounded set K = (0,1) is not compact; note that K = (0,1) is not closed.

9
4. Introduction to Sequences
You have discussed sequences of real numbers or vectors in earlier courses. Here we revisit them, and
discuss them more carefully. A somewhat formal definition of the notion of a sequence is given in:

Definition 4.1
Let S be a set. A sequence in S is a function on N and whose range is in S. In particular, a sequence
in R p is a function whose domain is N and whose range is contained in R p . In other words, a
sequence in R p assigns to each natural number n = 1,2,…, a uniquely determined element of R p .

Remarks
(a) We shall denote a sequence by placing brackets around the elements of the sequence. Thus if

xn  S , n  1 , then we denote the sequence by ( xn ). (Many of you have used { xn } in earlier

courses.)

(b) Mostly we shall deal with sequences in R p . Thus  x n  denotes a sequence in R p .

(c) Sequences may be defined by a formula, e.g.

xn  2n , n  1 ,

or finitely many formulas, or inductively: e.g. x1  x2  1 ,

xn  xn 1  xn  2 , n  3 .

(d) We can perform arithmetical operations on sequences:

Definition 4.2
Let  xn  and y 
n be sequences in R p .
(I) We define the sum of  xn  and y 
n to be the sequence x n  yn  . Their difference is  xn  yn  .
(II) The inner product of  xn  and y 
n is defined to be the sequence x n  yn  .
(III) If c  R , we define the sequence c  x n  to be the sequence  cx n  .
(IV) If  zn  is a sequence of real numbers, we define the product of  zn  and  x n  to be the
sequence  zn x n  . If also zn  0, n  1 , we define the quotient of  x n  and  zn  to be the
sequence  xn / zn  .

10
Example

If  xn  and  zn  are sequences in R given by

xn  n;
zn   1 / n,
n

then

 xn   1,2,3,... ;
 zn    1,
1 1 1 
,  , ,... 
 2 3 4 

and

 xn  z n    n   1 

n1 2 
/ n   0, 2 , 2 ,...  ;
 2 3 

 n 1 1 3
 2 3 4 

 xn  z n   n   1 / n   2,1 ,3 , 2 ,...  ;
 xn zn     1    1,1, 1,...;
n

 xn / z     1 n    1, 4, 9,... .

n 2
n

A very important part of analyzing sequences is discussing their convergence:

Definition 4.3 Sequence Convergence

Let  xn  be a sequence in R p and x  R p . We say that x is a limit of  xn  if for each

neighborhood V of x , there is a natural number KV such that

n  KV  xn  V .

If x is a limit of ( xn ), we say that ( xn ) converges to x , and that the sequence is convergent. We also

write
x  lim xn
n 

If the sequence does not converge, we say that it is divergent.

11
Thus, given any neighborhood V of x , xn must lie in V for n large enough (depending on the
particular neighborhood V). We can reformulate this in terms of norm distances:

Theorem 4.4 (Another definition of sequence convergence)

Let ( xn ) be a sequence in R p and x  R p . Then x is a limit of ( xn ) iff for each   0 , there exists

K     N such that

n  K     xn  x   .

Theorem 4.5 Uniqueness of Limits

Let ( xn ) be a sequence in R p . There it can have at most one limit.

Proof

Lemma 4.6 Boundedness of Convergent Sequences

A convergent sequence ( xn ) in R p is bounded.

Proof

12
It follows easily form the definition of the norm in R p that a sequence in R p converges iff all its
component sequences converge:
Theorem 4.7
Let

xn   xn1 , xn 2 , ... xnp   R p , n  1 .

Then  xn  converges in R p iff every one of the component sequence  xnk n 1 , k  1, 2,..., p ,


converges in R.

SOME EXAMPLES
Example 4.8 (a)
Let
1
xn  , n  1.
n
Then we know from experience that this sequence converges to 0. Let us prove this rigorously.

Theorem 4.9
A set F is closed if and only if for every convergent sequence (an ) such that an  F for all n  1, we have

that lim an  F
n 

13
5. Subsequences and Combinations
Very often we can learn a lot of about a sequence by looking at different “parts” of the sequence. These
parts are called subsequences:

Definition 5.1
Let X   xn  be a sequence in R p . Let
r1  r2  r3  ... (*)
be a strictly increasing sequence of natural numbers. Then the sequence given by
     

X '  xrn  xrn  xr1 , xr2 , xr3 ,...
n 1

is called a subsequence of  xn  .

Theorem 5.2
If a sequence X in R p converges to a limit x  R p , then any subsequence of X also has the same limit.

Corollary 5.3

If X   xn  is a sequence that converges to x , and mN , then  xn  m n 1


also converges to x .

Example
Let

xn   1 , n  1.
n

Then ( xn ) is a divergent sequence. On the other hand,

x2 n  1, n  1,
So ( x2n ) converges to 1. Similarly ( x2 n 1 ) converges to 1 .

14
COMBINATIONS OF SEQUENCES
Now we can investigate what happens when we add, multiply, … convergent sequences.
Theorem 5.6
(a) Let X and Y be sequences in R p that converge to x and y respectively. Then

(i) THE LIMIT OF THE SUM IS THE SUM OF THE LIMITS

X  Y converges to x  y .

(ii) THE LIMIT OF THE DIFFERENCE IS THE DIFFERENCE OF THE LIMITS

X  Y converges to x  y .

(iii) THE LIMIT OF THE INNER PRODUCT IS THE INNER PRODUCT OF THE LIMITS
X  Y converges to x  y .

(b) THE LIMIT OF THE PRODUCT IS THEPRODUCT OF THE LIMITS

Let X   xn  be a sequence in R p that converges to x and let A   an  be a sequence in R

that converges to a. Then the sequence  an xn  in R p converges to ax .

(c) THE LIMIT OF THE QUOTIENT IS THE QUOTIENT OF THE LIMITS IF THE
DENOMINATOR LIMIT IS NON-ZERO
Let X   xn  be a sequence in R p that converges to x and B   bn  be a sequence of non-zero

real numbers that converges to a non-zero number b. Then the sequence  bn1 xn  in R p

converges to b 1 x .

Proof

15
Remark
It is essential in (c) that the denominator limit is non-zero. For example, let ( xn ) and ( bn ) be sequences
of real numbers defined by
1
xn  1  , n  1
n
and
1
bn  , n  1.
n
Then it is easy to see that ( xn ) converges to 1 and (bn ) converges to 0, but
 1
bn1 xn  n  1+   n  1
 n

So (bn1 xn ) is not bounded, and cannot converge.

Theorem 5.7
Every sequence ( xn ) has a monotonic subsequence.

Example
Let
1
xn   1 
n
, n  1.
n
Then ( xn ) is a divergent sequence and is not monotonic. On the other hand,
1
x2n  1  , n  1,
2n
So ( x2n ) is a monotonic subsequence. Similarly ( x2 n 1 ) is a monotonic subsequence as well.

Two Criteria for Convergence

In this section, we derive two important general criteria for convergence of a sequence. They are often
simpler to apply than the original definition of convergence.

Theorem 5.8 MONOTONE CONVERGENCE THEOREM

Let  xn  be a sequence of real numbers such that

x1  x2  x3  ...  xn  xn 1  ... .

(We say that  xn  is monotone increasing). Then the sequence converges iff it is bounded. If it is bounded,
then

lim xn  sup  xn : n  1 (*)

n 

16
Corollary 5.9

Let  xn  be a sequence of real numbers such that

x1  x2  x3  ...  xn  xn 1  ...

(We say that  xn  is monotone decreasing). Then the sequence converges iff it is bounded. If it is bounded,

then

lim xn  inf  xn : n  1 .
n 

THE BOLZANO WEIERSTRASS THEOREM

It will be useful to reformulate the Bolzano-Weierstrass Theorem for sequences:
Theorem 5.9 Bolzano-Weierstrass Theorem for sequences
A bounded sequence in R p has a convergent subsequence.

17