The Elimination Form of the Inverse and Its Application to Linear Programming Author(s): Harry M.

Markowitz Reviewed work(s): Source: Management Science, Vol. 3, No. 3 (Apr., 1957), pp. 255-269 Published by: INFORMS Stable URL: http://www.jstor.org/stable/2627454 . Accessed: 10/05/2012 02:14
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Management Science.

http://www.jstor.org

THE ELIMINATION FORM OF THE INVERSE AND ITS APPLICATION TO LINEAR PROGRAMMING*
HARRY M. MARKOWITZ' The RAND Corporation

Introduction It is common for matrices in industrial applications of linear programming to have a large proportion of zero coefficients. While every item (raw material, intermediate material, end item, equipment item) in, say, a petroleum refinery may be indirectly related to every other, any particular process uses few of these. Thus the matrix describing petroleum technology has a small percentage of nonzeros. If spacial or temporal distinctions are introduced into the model the percentage of non-zeros generally falls further. The present paper discusses a form of inverse which is especially convenient to obtain and use for matrices with a high percentage of zeros. The application of this form of inverse in linear programming is also discussed. The Elimination Form of Inverse

The inverse (A-') of a matrix (A) is valuable when a number of sets of equations AX = b are to be solved using different b's and the same A. A-', like any matrix, may be expressed as the product of other matrices A- l-mAm... M,

in an infinite number of ways. E.g. (2) = (2) (4) = (W)(16) etc. If we have such M1, ***, Mm we can solve X = A-b in a series of steps:
X(1=

Mlb

X(2) =M2X(1 X

MmX(m-1)

M1 is referred to as a "product form" of inverse. In The expression Mm ... some problems there may be Mi which are easier to obtain and apply than A-' itself. This paper discusses a particular product form of inverse which is closely related to the Gaussian elimination method of solving a set of simultaneous equations. This "elimination form of the inverse," as we shall call it, is especially valuable when A has a large number of zero coefficients.
* ReceivedOctober,1956.

The writer is indebtedto GeorgeDantzig andAlan Mannefor valuablesuggestions. 255

256

HARRY

MARKOWITZ

The elimination form of inverse can be illustrated in terms of the solution of three equations in three unknowns:
(1) a11Xj + al2X2 + a13X3 = r

(2) (3)

a2lX1 + a22X2+ a23X3= r2 a31X1+ a32X2+ a33X3= r3

For the moment we will let the kth diagonal element be the kth pivotal elements From equation 1) we get the first equation of our back solution (Bi)
X
=

all

ri-

al2X

all

-_

all

X3

We eliminate X1 from equations 2) and 3) by adding equation to the

ith

ail

times the first

equation, thus obtaining

= + b22X2 b23X3

(2') (3')
where

r2*

= + b32X2 b33X3

r3*

r2
r3

=
=

r2r3 -

all i-r1
-3

all

r1

Similarly we get (B2) and where

r3**
=

X2

b22

r2*

b22

2X3

c33X3 =r3**

r3 *

b2

r2

Finally (B3) X=
C33
r3

(B3) gives us X3; X3 and B2) give X2; X2, X3 and B1) give X1. Consider the transformations which occurred to our original right hand side. First we formed
1 0 0

(7r*)

_a' a

()r2

all

ELIMINATION

FORM OF THE INVERSE

257

then (ri\~~
=

0 io 1
0
b32

l \*! 1

then

(H)
then

1 0
0 o1

Is 0 ($
0

/X1\

_ab2

62/r2

1 0

b23 ~~~~~b22

Since the

aibji

do not depend on the rs we havre 1

a12 _?13

[10

0-1=al
X

a1 all

aa l

0-1

258

HARRY

MARKOWITZ

Similarly if A is any mXm non-singular matrix we have .. A-' = B1B2 ... BmRmR, where Rk is a matrix of the form 1

~~0
0

Vk+l,k Vkk

0
Vmk
Vkk

and Bk

iS

of the form

1
1
Vkk Vkk+1 Vkk
_

Vkm Vkk

Although this elimination form of inverse consists of 2m - 1 matrices it contains only n numbers which are not known a-priori to be 0 or 1. It requires the same number of multiplications and additions to apply the elimination form as it does to apply the conventional A-'. The arithmetic operations re-

ELIMINATION

FORM OF THE

INVERSE

259

quired to obtain the B1 ... BmRm_i... R1 and to apply them once are exactly the same as those required to solve a set of linear equations by Gaussian elimination. Suppose that the kth pivot is not vkkk but vi0o (whereV7 is the value of the parameter of ith equation jth variable at the kth step). This step subjects the right-hand side to a transformation of the form r1 7

(r)new

o (r)old

'Yin

1 of the ey are zero. The y vector is in the ioth column of the maAt least k trix. The kl step also gives rise to a back solution of the form
= x To =

0 1 rio +

E niX

where at least m - k of the -j equal zero. The elimination form of the inverse in this case is
... A-' = BBBmPR7n*-i .R

where

ri = 7tl Bk:
=

pk

...

vm

oio

260

HARRY

MARKOWITZ

'Yi

Rk=

7Ym

1 )

and P is permutation matrix such that if vijk is a pivotal element P makes the old ith component of (r) become the new jth component of r. That A-1=B1 * * P * * * R1, i.e., that AX = r implies X = B1*.* P*** R1 r may be seen as follows. The transformations Rm.- ... Rlr give the right hand sides of the back solutions in the original equation order. We can imagine reordering the equations so that new first back solution is the one of the form
1 k + E ,X

and similarly the new kth back solution is the one in which Xk was eliminated. This rearrangement changes the order of the ri exactly as does P. The last back solution is of the form
Xji = V rim

Since rim is now the jmth component of (r), Bm (as described above) will transform (r) into a vector with Xjm in the jmth position. The next back solution is of the form
Xji.1 =

lXjm

Since, thanks to Bm and P, rimi- is the jmlth component and Xj, is the jmth component of (r), Bm., transforms (r) into a vector with Xjm_1 in the j"m_jth position as well as Xjm in the jmth position. And so on. In recording Bk or Rk it is not necessary to write out the entire matrix. In the case of an Rk it is only necessary to record the i eliminated and the nonzero yi . In the case of a Bk it only is necessary to record the j eliminated, the non-zero nj and 1/pivotal element. If the matrix A has a large number of zero aqi, the elimination form of inverse may have appreciably less than n2 non zero ys and vs. This may be so even though the usual A-' has no zeros.

ELIMINATION

FORM OF THE INVERSE

261

The number of non-zero v and y in an elimination form of inverse may depend on which pivotal elements are used. Suppose the *'s below represent the non-zero elements of a 5 x 5 matrix.
* * * * * * * * * * * * * * *

If all is the first pivotal element the non-zeros at step two are (barring accidental zeros) as follows
* * * * * * * * * * * * * * * *

But if a1nor a51is the pivotal element the pattern of non-zeros is

* * * * * * * * *

A table indicating zero and non-zero coefficients is a valuable aid in the choice of pivotal elements. From such a table an agenda (i.e., a complete set of pivotal elements) can be chosen before computation begins. This separation of the choice of the agenda and actual elimination is convenient both in hand and machine computation. There are, however, two dangers attached to deciding on an agenda beforehand. Some pivotal element may accidentally become zero, in which case the agenda cannot be followed to the end. Or some pivotal element may turn out to be so small that its use would adversely affect the accuracy of the results. One solution to these difficulties is to have some test of the acceptability of a pivotal element; form the agenda beforehand and follow it as long as each pivotal element meets the test. If a pivotal element fails the test, a new agenda can be worked out for the remaining equations and variables. An example agenda is presented in Table 1. The X's represent the original non-zero elements of the matrix. The M's represent coefficients which began 43 in the matrix as zeros but ended as non-zeros. The numbers k = 1, *., indicate the kth pivotal element. The number (pi) at the right of each row indicates the number of elements of that row which were not already eliminated when the row was eliminated. The number oj at the bottom of each column indicates the number of elements of that column which were not already eliminated when the column was eliminated. One of the by-products of making an agenda beforehand is foreknowledge of all the variables which will appear in any equation and all the equations in which a variable will ever appear.

__ __

__ __

I
_

I 1 IN I8 1 1 1~ 1 I I8 1 1e 1 1' 1 I II lI l I I

IR I8 I8 1 I8 1I 81 81

1 1 1 1 1 1

_ _

I~I

I I I I I
I I I I I I

_ _ _ _ _ __

_ _

I I
1,

I I I I I I I I I I I I I 1 1I I I l I I I I I I I I I II I I
I I I

I1
1

11__,__,_

II I 1 1 1 1 1 1 1 1

I I
I-L 1 1 1 1 1 1 1 1 1
_ _

?_
__ __ __

I I I I I
1

I1
I8 I

I I I

__ __ _

__ _

I I I I I I I I I I I I I I I

I I I I

I I I I

I I I I

I I I I

I I I I

I I I I I I I I I IIc
1 1 1 1

I II I I I I I I I I IC: I I I I I I II

I 1 1 1 1 1 I It 1
8

e'

'

I 'I 'I I'1 I --

I
--1--

-?1 -I
__ _ _

1- 1 I 1 I I I I I I I I I I I I 1I I I

I I I I I I I I I I I I I I I I I I I I I IN

__

I I II
l_

1 I

I
I 1 1

1~ 1

II
1 1 1 XI

I
1 1 1

I
1

___

004 I
_

I
I

I I I

I II. III I I I I I1 I CD

C4
____I__I__I

114I

Z 1l41

I IHI

I cq 1 1 1 1 1 1HlIq I I I I I I 1 1 I Icq II
I I I I I I I
l locllli

' -C4iXvloobxclllelovIa<

S IIll4llll---cllic
00
eq

00~~~~~~~6 1[ 1 I I 1 1 I 1001 1
1-1e11111111111111111111q-

~I

IIII 1 1 11 1 III

It111111111111111111111126

[I

I1 ItII I II8 I

~ 1I~

cO

cq

Ccq e

C--

co

ce

C-4 c

e IC

I I I I I I I I I8I 1 1 1 I>I8I 18 8 8 I I 1 1O II I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I 1 1O I I I I I I I I I I I I I I I I I I I 1 1O I I I I I I I I I I I I I I I I I I 1' 1 1 0 I I I I I I I I I I I I I I I I I I IcaI 1 _ I I I I I I I I I I I I I I IC I I I I 1 _ s I I I I I I I I I I I I I>I 1 1 1 1 1 1 1 _ I I I I I I I I I I 1 1? I I I I I I I I I _ _ I I I I I I I I I I IaZI I I I I I I I I 1 _ I I I i I I I I I I I I I I I I I I I I 1 _ I I I I I I I I I I I I I I I I1 _ II I I I I I I I I I I I I I I I I I I I I I I 1 _ __ _ I I I I I I I I I I I I I I I I I I I I1 I I I I I I I I I I I I I I I I I I I1 I_ _ I I I I I I I I I I I I I I-1 1 I 1 _ ~8 1 _ _ I I I I I I I I I I I I I I I- I -I I I1 _1 _ _ I I I I I I I I I I I I I I I I I I81 1 I _ I I I I I I I I I I I I I8 1 81 1 1~ 1

I III

III

1~~ 1-1-1-1-1-1 1

I>I11I 11 I I811I 1 1 11 -I I 11 I I I 1111 18811 I 111li I I 1 ~ oII IIII I ~I II ~I I -1_11_--1__11

___

1I oI I81 1 1~ 1I

II I III I~ I I I I 1 11--I 118 ~ ~I

_____1_

11

I1 1I1811111 1I II I I I IIIII I I I I I I 1111111liii11

III118 1 1K1 1 I I I I I Ke II11K

263

264

HARRY

MARKOWITZ

The matrix which Table 1 represents was the optimum basis of a linear programming problem involving a 43-equation model of petroleum refining. This matrix has 197 non-zero elements. As compared with a possible (43)2 = 1849, the number of non-zero elements in the elimination form of inverse is

pi + E (o-j

p + E o-j 43 = 201. rj

= 247 multiplications or To derive this inverse requires (i pivotal) Pij divisions and somewhat less additions or subtractions. It would be desirable to choose an agenda so as to minimize the number of zeros which become non-zero. Generally it is harder to find such an "optimum" agenda than to invert a matrix. An alternative is to choose at each step the pivot which minimizes the number of zeros which become non-zero at that step. A still simpler alternative, which seems adequate generally, is to choose the pivot which minimizes the number of coefficients modified at each step (excluding those which are eliminated at the particular step). This is equiva-

lent to choosingthe non-zeroelementwith minimum(pi

l)(0o

1). William

Orchard-Hays has coded the "Johniac" for the elimination form using the minimum (pi - 1)(oj - 1) selection principle. The small percentage of zeros which became non-zeros in our example is typical of cases run thus far (with approximately ten per cent non-zeros in the original matrix) with the OrchardHays code. Figure 1 illustrates matrices with the following properties: a) all diagonal elements akk are non-zero

b) if an element aik or aki in the kth columnor row is non-zero,then all elements between ak (or akj) and akk are non-zero. It is trivial to find an optimum agenda for such matrices. If the kth diagonal element is used as the kth pivot, no zero coefficient will become non-zero. If a matrix is almost of the above form except that a few zeros are mixed in with the non-zeros, then using the kth diagonal element as the kth pivot may cause the "misplaced" zeros (and only these) to become non-zero.
FIG. 1
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Application to Linear Programming The simplex method for solving linear programming problems has a number of variants. A recent version2 requires the solution of two sets of equations. The first set of equations p'A = ' or A'p = y
2 G. B. Dantzig, Alex Orden, & Philip Wolfe, "The Generalized Simplex Method," RAND P-392-1, 4 August 1953.

ELIMINATION

FORM OF THE INVERSE

265

yields prices p which are used to select a variable X8, currently equal to zero, to be increased until some "basis" variable becomes zero. The second set of equations is Ae = P8 where P, is a column vector associated with the variable X. . e is used to determine which basis variable X, first becomes zero as X. is increased. The matrix A at one iteration differs from that at the next only in that the column vector Pr, (which we will assume to be the rth column of A,) is replaced by the column vector P, . I.e., A (k) becomes
A(k+l) =

(Pjl

...

Pr .

Pji) Pj)

(Pjl

P..

Or el

A?

(k

= (Pj,

P,

Pi,_)
er

em ei

iJ

Letting E

em

266 we have

HARRY

MARKOWITZ

A(k+)_

AkE

The inverse of A(k?l) is

A (k +1)E=g

1 (Kf)

where
(l

~~~elA e.

e. er em

at each step by multiplyingout E(k)

In an early variant of the simplex method the new inverse A (k+l)1 A(k) 3

was obtained

In a morerecentversionthe E(k)

are carried along and used as product form of inverse.4 If the first iteration starts with the identity matrix then
A(k)-1 = - E (k-1) R.E(2)1 k-l)E(k2... E(1)-1.

If the product form of inverse is used, as k increases computing time required per iteration also increases. A point is eventually reached when it is desirable to reinvert the current basis A (k,) and let A (ko +k)
EI= E(ko+k-1)
1 (ko)-'A (ko)-l

At this point the elimination form of inverse can be of value, especially if A has a large number of zeros, since this form requires less time to obtain and apply.
3 See George B. Dantzig, "Maximization of a Linear Function of Variables Subject to Linear Inequalities," pp. 339-347, and Robert Dorfman, "Application of the Simplex Method to a Game Theory Problem," p. 358 in Activity Analysis of Production and Allocation, T. C. Koopmans, Ed., New York, 1951. 4See George B. Dantzig, "The Product Form for the Inverse in the Simplex Method," in Mathematical Tables and OtherAids to Computation,VIII, No. 46, April, 1954.

ELIMINATION

FORM OF THE INVERSE

267

Reinverting A is only part of the operations involved in solving a linear programming problem. We therefore cannot expect to obtain, by the use of the elimination form, the same percentage reduction in computing time for the whole linear programming problem as we obtain for the reinversion of A. When a more convenient form of inverse is available it may be desirable to reinvert more frequently. To see the effect of a more convenient form of inverse on the frequency of reinversion and the time required to solve a linear programming problem, we must explore the question of optimum reinversion policy. We will first derive some neat results under several restrictive assumptions. Afterwards we will show computing procedures for obtaining an optimum reinversion policy under more general assumptions. It has been observed, with stop watch as well as theory, that, with the RAND linear programming code for the IBM 701, the computing time required per iteration increases linearly with the number of iterations. Let us assume, for the moment, that (a) the problem starts with a first basis to be inverted; (b) the time required for this first inversion is the same as that for any subsequent reinversion; (c) computation time per iteration is a linear function of the number of iterations since the last (re)inversion. Hence the time (t) since the start of the last reinversion is a quadratic t = a + 3I + yI2 where a, 3, ry > 0. Let us further assume, for the moment, that the number of iterations I required to solve the problem is known beforehand. Suppose it were decided that there would be n inversions (n - 1 reinversions). Let AIi = the number of iterations between the ith and i + ith inversion (for Let AI, = the number of iterations from the nfth inveri = 1, .., n-1). sion to the end of the problem. Total time (T) required is
n

T
where

Ea

+ fAIi + y(AIi)2
n
i= 1

ZAIi = I

The optimum solution must satisfy the Lagrangian equations

_ i

aE (a + fAIi + y(AI )2)

_ _ _ _ _ _ _ _ _ _

X(AIi)
_ _

= 0

aAIi,
3+ 2-yAIi-X = 0 for all i AIi is the same for all i We can therefore rewrite the expression for T as

T = n(a + fI +
where I = AIi

yI2)

7 n

268 Or

HARRY

MARKOWITZ

T -an
dT dn dT
dn2

+ P3I+I72

a
2-

n2

y 2n3

oo as n -0 0 for Since -w > O all n > 0, and since T any n > 0 withdn d~~~~~~~~~~~~ gives a minimum value of T for all n > 0. If such an n is non-integral the best integral value is either that immediately above A, or that immediately below A, or both

d2T

dT

When

dT n =4/ I

Let us assume that n is integral. Then 'the optimum

z

=1/

T-?aV17? I

3I+

(f a 7

+2rV~)I

The last expression can be used for estimating the time to' be saved by using a more convenient form of inverse. Thus-given our various assumptions-if a new method of inversion could produce an inverse in one-fourth the time (a) and because of its compactness it permitted the first subsequent iteration to be done in one-half the time (p, roughly), the whole linear programming problem could be done in one-half the time. Let us now suppose that I is not known but has an a-priori probability distribution (derived presumably from past linear programming problems). We may as well also drop the quadratic assumption on t. We define

aij = the expected value of qij

where [0when I < i The (expected) time required to (re)invert the matrix at iteration i and iterate through j - 1 without reinverting-if I > j. The expected time from the beginning of the reinversion at i to the end of the problem if i I < j.

qij

ELIMINATION

FORM OF THE INVERSE

269
IK (since rein-

Suppose the points of reinversion are before iterations

1, **.

version points can be chosen with IR so large that there is a zero probability that this iteration will occur, there is no loss of generality in assuming a fixed K). Expected time, to be minimized, is

aulI + ai1r2 +

+ aIK.1IK

The optimal value of Ii of I, could be calculated if 72 were known. It is given by the function I1(I2) which minimizes a,,, + ar1I2 for various values of 12 and which can be readily computed. Define
aI2

11

al,

1(I2)

a1(I2),

I2

We now only need to minimize

= aI2/I1 +

ai2r3 +

aIK.1rK

We repeat the process until we have

E
= a 1K,, .K-i

from which we get IK and work back through

IK-1(IK), K-2(I1-1),

etc.