SCHAUM’S OUTLINE OF THEORY AND PROBLEMS oF PROBABILITY SEYMOUR LIPSCHUTZ, Ph.D. Professor of Mathematics Temple University SCHAUM’S OUTLINE SERIES McGRAW-HILL New York San Francisco Washington, D.C. Auckland Bogotd Caracas Lisbon London Madrid Mexico City. Milan Montreal New Delhi San Juan Singapore Sydney Tokyo TorontoCopyright © 1965 by The McGraw-Hill Companies, Inc. All Rights Reserved. Printed in the United States of America. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. ISBN 07-087982-3 3132 33 34.35 36 BAW BAW 9098765432109 McGraw-Hill ‘A Divison of The McGraw Hill CompaniesPreface Probability theory had its beginnings in the early seventeenth century as a result of investigations of various games of chance. Since then many leading mathematicians and scientists made contributions to the theory of probability. However, despite its long and active history, probability theory was not axiomatized until the twenties and thirties of this century. This axiomatic development, called modern probability theory, was now able to make the concepts of probability precise and place them on a firm mathematical foundation. The importance of probability has increased enormously in recent years, Today the notions of probability and its twin subject statistics appear in almost every discipline, e.g. physics, chemistry, biology, medicine, psychology, sociology, political science, education, economics, business, operations research, and all fields of engineering. This book is designed for an introductory course in probability with high school algebra as the only prerequisite. It can serve as a text for such a course, or as a supplement to all current comparable texts. The book should also prove useful as a supplement to texts and courses in statistics. Furthermore, as the book is complete and self-contained it can easily be used for self-study. The book begins with a chapter on sets and their operations, and follows with a chapter on permutations, combinations and other techniques of counting. Next is a chapter on probability spaces and then a chapter on conditional probability and independence. The fifth and main chapter is on random variables. Here we define expectation, variance and standard deviation, and prove Tchebycheff’s inequality and the law of large numbers. Although calculus is not a prerequisite, both discrete and continuous random variables are considered. We follow with a separate chapter on the binomial, normal and Poisson distributions. Here the central limit theorem is given in the context of the normal approxi- mation to the binomial distribution. The seventh and last chapter offers a thorough ele- mentary treatment of Markov chains with applications. Each chapter begins with clear statements of pertinent definitions, principles and theorems together with illustrative and other descriptive material. This is followed by graded sets of solved and supplementary problems. ‘The solved problems serve to illustrate and amplify the theory, bring into sharp focus those fine points without which the student continually feels himself on unsafe ground, and provide the repetition of basic principles so vital to effective learning. Proofs of most of the theorems are included among the solved problems. The supplementary problems serve as a complete review of the material of each chapter. I wish to thank Dr. Martin Silverstein for his invaluable suggestions and critical review of the manuscript. I also wish to express my appreciation to Daniel Schaum and Nicola Monti for their excellent cooperation. SEYMouR LIPscHUTZ ‘Temple UniversityCONTENTS Page Chapter I SET THEORY . Introduction. Sets, elements. Set operation sets. Classes of sets, Finite and countable sets. Product Chapter 2 TECHNIQUES OF COUNTING ......0.0...0000005 16 Introduction. Fundamental principle of counting. Factorial notation. ‘Permutation Permutations with repetitions. Ordered samples. Binomial coefficients and theorem. Combinations. Ordered partitions. Tree diagrams. ‘Chapter 3 INTRODUCTION TO PROBABILITY . ooeo - 38 Introduction. Sample space and events, “Axioms of probabil jity. Finite probability spaces. Finite equiprobable spaces. Infinite sample spaces. Chapter 4 CONDITIONAL PROBABILITY AND INDEPENDENCE . Conditional probability. Multiplication theorem for conditional probability. Finite stochastic processes and tree diagrams. Partitions and Bayes’ theorem. Inde- pendence, Independent or repeated tri: Chapter 5 RANDOM VARIABLES feed Introduction, Distribution and expectation of a finite random variable. Variance and standard deviation. Joint distribution. Independent random variables. Func- tions of Discrete random variables in general. Continuous random variables, Cumulative distribution function. Tchebycheff’s inequality. Law of large numbers. Chapter 6 BINOMIAL, NORMAL AND POISSON DISTRIBUTIONS .. 105 Binomial distribution. Normal distribution. Normal approximation to the binomial distribution. Central limit theorem, Poisson distribution. Multinomial distribution. Chapter 7 =MARKOV CHAINS .. ceeees Introduction. Probability vectors, stochastic matrice Regular stochastic ‘matrices. Fixed points and regular stochastic matrices. Markov chains. Higher transition probabilities. Stationary distribution of regular Markov chains, Absorbing states. INDEX ....... seeee 152Chapter 1 Set Theory INTRODUCTION This chapter treats some of the elementary ideas and concepts of set theory which are necessary for a modern introduction to probability theory. SETS, ELEMENTS Any well defined list or collection of objects is called a set; the objects comprising the set are called its elements or members. We write p €A_ ifp isan element in the set A If every element of A also belongs to a set B, i.e. if p € A implies p € B, then A is called a subset of B or is said to be contained in B; this is denoted by ACB or BDA Two sets are equal if each is contained in the other; that is, A=B ifandonlyif ACB and BCA The negations of p€ A, ACB and A=B are written p€A, ACB and A~B respectively. We specify a particular set by either listing its elements or by stating properties which characterize the elements of the set. For example, A = (1,3,5,7,9} means A is the set consisting of the numbers 1,3, 5,7 and 9; and B = (x: ais.aprime number, 2 < 15} means that B is the set of prime numbers less than 15. Unless otherwise stated, all sets under investigation are assumed to be subsets of some fixed set called the universal set and denoted (in this chapter) by U. We also use to denote the empty or null set, i.e. the set which contains no elements; this set is regarded as a subset of every other set. Thus for any set A, we have DCACU. Example 1.1: The sets A and B above can also be written as A = {z: cisanodd number, 2 <10} and B = {2,8,5,7,11, 18} Observe that 9€A but 9€B, and 11€B but 11€@A; whereas $€A and 3€B, and 6@A and 6EB.2 SET THEORY [cHAP. 1 Example 12: We use the following special symbols: N = the set of positive integers: Z = the set of integers: R = the set of real numbers. Thus we have NCZCR. Example 13: Intervals on the real line, defined below, appear very often in mathematics. Here a and 6 are real numbers with a <6. Open interval from a to b = (a,b) = (e:a’ —=yF a—=5 =F ‘ —= as The elements of S x TX W are the ordered triplets listed to the right of the tree diagram. 148. Each has 60 elements. 149. Both are true: AXA = (BX B)N (CXC) = (BX C)N (Cx B). 151.) Ais (i) Age (ii) As, (iv) Ai () Ay (Wi) Ane 1.52. P(B) = (B, (1, {2,3}}, {1,4}, {{2,8}, 4}, (1), ((2,8}}, (43, O}. 1.53. (i) no, (ii) no, (iii) yes, (iv) yes. 1.54, ({1,2,3}], ((1}, (2 33), ({23, (1, 8)], [(8}, 1,23] and ((1}, (2), (8)] 15Chapter 2 Techniques of Counting 'TRODUCTION In this chapter we develop some techniques for determining without direct enumeration the number of possible outcomes of a particular experiment or the number of elements in a particular set. Such techniques are sometimes referred to as combinatorial analysis. FUNDAMENTAL PRINCIPLE OF COUNTING We begin with the following basic principle. Fundamental Principle of Counting: If some procedure can be performed in m dif- ferent ways, and if, following this procedure, a second procedure can be performed in nz different ways, and if, following this second procedure, a third procedure can be performed in ns different ways, and so forth; then the number of ways the procedures can be performed in the order indicated is the Product mi+m2*ms.... Example 2.1: Suppose a license plate contains two distinct letters followed by three di the first digit not zero. How many different license plates can be printed? ‘The first letter can be printed in 26 different ways, the second letter in 25 dif- ferent ways (since the letter printed first cannot be chosen for the second letter), the first digit in 9 ways and each of the other two digits in 10 ways. Hence 26+25+9+10+10 = 585,000 different plates can be printed. FACTORIAL NOTATION The product of the positive integers from 1 to » inclusive occurs very often in mathe- maties and hence is denoted by the special symbol n! (read “n factorial”): mY = 1+2+B+-+-+(n—2)(n—1)n It is also convenient to define 0!= 1. Lega = 24, 6+120 = 720 Example 2.2: Qt = 1+2 = 2 Bt = 1268 BI = Bed! = 5+24 = 120, ge7 = 56 aeeane1o = H2eHt=10-91 _ at Example 2.3: 5 a PERMUTATIONS An arrangement of a set of » objects in a given order is called a permutation of the objects (taken all at a time). An arrangement of any r=_n of these objects in a given order is called an r-permutation or a permutation of the n objects taken 7 at a time. Example 24: Consider the set of letters a, 6, ¢ and d. Then: (i) bdea, deba and acd are permutations of the 4 letters (taken all at a time); bad, adb, ebd and bea are permutations of the 4 letters taken 8 at a time; ‘ad, cb, da and bd are permutations of the 4 letters taken 2 at a time, 16CHAP. 2] TECHNIQUES OF COUNTING 17 ‘The number of permutations of x objects taken r at a time will be denoted by P(n,r) Before we derive the general formula for P(n,r) we consider a special case. Example 25: Find the number of permutations of 6 objects, say a,b,¢,d,¢,f, taken three at a time. In other words, find the number of “three letter words” with distinct letters that can be formed from the above six letters, Let the general three letter word be represented by three boxes: | Now the first letter can be chosen in 6 different ways; following this, the second letter can be chosen in 5 different ways; and, following this, the last letter can be chosen in 4 different ways. Write each number in its appropriate box as follows: 6 5 4 Thus by the fundamental principle of counting there are 6+5+4 = 120 possible three letter words without repetitions from the six letters, or there are 120 permu- tations of 6 objects taken 3 at a time. That is, P63) = 120 ‘The derivation of the formula for P(n,r) follows the procedure in the preceding example. The first element in an r-permutation of n-objects can be chosen in n different ways; follow- ing this, the second element in the permutation can be chosen in »—1 ways; and, following this, the third element in the permutation can be chosen in n—2 ways. Continuing in this manner, we have that the rth (last) element in the r-permutation can be chosen in 1” —(r—1 n—r+lways, Thus Theorem 21: P(n,r) = n(n—1)(n—2)---(n-r +1) = The second part of the formula follows from the fact that nn=1(n~2)---(m—r+)(m—7)! _ at @-7t ~ (a=ryt nn 1)(n—2)- (nr +1) In the special case that r=", we have P(n,n) = n(n—1)(n—2)-+-38+2-1 = vt Namely, Corollary 2.2: There are x! permutations of x objects (taken all at a time) Example 26: | How many permutations are there of 3 objects, say, a, b and c? By the above corollary there are 3! = 1*2*3= 6 such permutations. These are abe, acb, bac, bea, cab, cba. PERMUTATIONS WITH REPETITIONS Frequently we want to know the number of permutations of objects some of which are alike, as illustrated below. The general formula follows. Theorem 2.3: The number of permutations of objects of which m are alike, m2 are alike, ...,m, are alike is ~18 TECHNIQUES OF COUNTING [cHAP. 2 We indicate the proof of the above theorem by a particular example. Suppose we want to form all possible 5 letter words using the letters from the word DADDY. Now there are 5!=120 permutations of the objects D:,A,D2,Ds,Y where the three D’s are dis- tinguished. Observe that the following six permutations D:DsD:AY, D2D,D:AY, DsD.:D:AY, DiDsD:AY, D2D:Di:AY, DsD2D:AY produce the same word when the subscripts are removed. The 6 comes from the fact that there are 8!=3+2+1=6 different ways of placing the three D's in the first three posi- tions in the permutation. This is true for each of the other possible positions in which the D's appear. Accordingly there are 51 _ 120 31 6 different 5 letter words that can be formed using the letters from the word DADDY. = 20 Example 27: How many different signals, each consisting of 8 flags hung in a vertical line, can be formed from a set of 4 indistinguishable red flags, 3 indistinguishable white flags, and a blue flag? We seek the number of permutations of 8 objects of which 4 are alike (the red flags) and 3 are alike (the white flags). By the above theorem, there are = 280 different signals. ORDERED SAMPLES Many problems in combinatorial analysis and, in particular, probability are concerned with choosing a ball from an urn containing n balls (or a card from a deck, or a person from a population). When we choose one ball after another from the urn, say r times, we call the choice an ordered sample of size r. We consider two cases: (i) Sampling with replacement. Here the ball is replaced in the urn before the next ball is chosen. Now since there are different ways to choose each ball, there are by the fundamental principle of counting nemeneon = a different ordered samples with replacement of size r. (ii) Sampling without replacement. Here the ball is not replaced in the urn before the next ball is chosen. Thus there are no repetitions in the ordered sample. In other words, an ordered sample of size r without replacement is simply an r-permutation of the objects in the urn. Thus there are at ee different ordered samples of size r without replacement from a population of n objects. Example 28: In how many ways can one choose three cards in succession from a deck of 62 cards (i) with replacement, (ii) without replacement? If each card is replaced in the deck before the next card is chosen, then each card can be chosen in 52 different ways. Hence there are B2+52+52 = 629 = 140,608 different ordered samples of size 3 with replacement,CHAP. 2] TECHNIQUES OF COUNTING 19 On the other hand if there is no replacement, then the first card can be chosen in 52 different ways, the second card in 51 different ways, and the third and last card in 50 different ways. Thus there are 52+51+50 = 192,600 different ordered samples of size 3 without replacement. BINOMIAL COEFFICIENTS AND THEOREM ‘The symbol (*). read “nCr”, where r and n are positive integers with r=_n, is defined as follows: (") _ n(n=1)(n rp 1 (r—Dr ‘These numbers are called the binomial coefficients in view of Theorem 2.5 below. Observe that ( *) has exactly r factors in both the numerator and denominator. Also, \ +1) Mm) _ e(w= Wee (m—=rt Do _ n= Vre(m—rt Ym! _ nt rp > RBar RB (Yr! (nr)! Using this formula and the fact that n—(n—r)=1, we obtain the following important relation. Lemma 2.4: (n+) = (7) or, in other words, if a+b =n then (2) = (6) trams x0; (9) = (8) = (29) = RE = a ‘Note that the second method saves both space and time. Remark: Motivated by the second formula for 3) and the fact that 0!=1, we define: n ! p . 0 o! (6) = Sig] = 1 and, in particular, (0) =om=2 The Binomial Theorem, which is proved (Problem 2.18) by mathematical induction, gives the general expression for the expansion of (a+ b)*. Theorem 2.5 (Binomial Theorem): ery = > (*) canbe ca = at + nab + MRD ar-aoe toes te mabe! + be Example 211: (a+b)8 = ats + Sabi + 6% = a + 5% + 100% + 100% + Sabt + BS 6:5 615+ = 1 ee (a+ 6)® = at + 608 + TOF atet + a 825 gps 5 + oe FBS anes + Tp atl + Gabe + = a® + GaSb + 150%? + 200%? + 15024 + Gabs + be20 TECHNIQUES OF COUNTING (CHAP. 2 The following properties of the expansion of (a+ b)* should be observed: (i) There are n +1 terms. (ii) The sum of the exponents of a and b in each term is n. (iii) The exponents of a decrease term by term from to 0; the exponents of b increase term by term from 0 to n. (iv) The coefficient of any term is (i) where i is the exponent of either @ or b. (This fol- lows from Lemma 2.4.) (v) The coefficients of terms equidistant from the ends are equal. ‘We remark that the coefficients of the successive powers of a +b can be arranged in a triangular array of numbers, called Pascal’s triangle, as follows: (a+b = 1 1 (atop = a + b ied (ato = a? + Bab + BF 124 (a+6)) = a3 + S0% + sab? + Bs eras. at (a+b) = at + dat + Gar? + dads + OF 146 41 (a+b)> = aS + 5atb + 100%? + 10a%* + Sabt + b5 1 @ jo 5 1 (a+b) = aS + Gad + 150d? + 200588 + 15024 + abs + OF 16 @@is 6 1 Pascal’s triangle has the following interesting properties. (a) The first number and the last number in each row is 1. (b) Every other number in the array can be obtained by adding the two numbers appearing directly above it. For example: 10 = 4+6, 15 = 5 +10, 20 = 10+ 10. We note that property (b) above is equivalent to the following theorem about binomial coefficients. nt n ne Theorem 26: ( ; ) = (en) + (*) Now let m1, 22, ..., mr be nonnegative integers such that m+ m+ +--+” =” Then the expression ( 0 ) is defined as follows: n - nt Mi, May os +5 Mr, mine “mrt For example, 7\ om _ 8 - 8. (Yr) Se = Gepp) lorena These numbers are called the multinomial coefficients in view of the following theorem which generalizes the binomial theorem. 2 Mgt... a” Theorem 2.7: (a +a2+ +++ +a)" = arta? «+ alr geo fee) nytngs a) saeCHAP. 2] TECHNIQUES OF COUNTING 21 COMBINATIONS Suppose we have a collection of objects. A combination of these n objects taken r at a time, or an r-combination, is any subset of r elements. In other words, an r-combination is any selection of r of the m objects where order does not count. Example 2.12: The combinations of the letters a,b,c,d taken 3 at a time are {a,b,c}, (a,b, d}, {a,¢,d}, {b,¢,d) or simply abe, abd, aed, bed Observe that the following combinations are equal: abe, aeb, bac, bea, cab, eba ‘That is, each denotes the same set {a,b,c}. The number of combinations of n objects taken r at a time will be denoted by en, r) Before we give the general formula for C(n, r), we consider a special case. Example 213: We determine the number of combinations of the four letters a,6,¢,d taken 3 at a time, Note that each combination consisting of three letters determines 3! = 6 Permutations of the letters in the combination: Combinations Permutations abe abe, acb, bac, bea, cab, cba abd abd, adb, bad, bda, dab, dba acd acd, ade, cad, eda, dae, dea bed bed, bde, ebd, edb, dbe, ded Thus the number of combinations multiplied by 3! equals the number of permu- tations: C(4,3)+8! = P(4,3) or C(4,3) = rus) Now P(4,3) = 4+3+2 = 4 and 3!=6; hence C(4,3)=4 as noted above. Since each combination of n objects taken r at a time determines r! permutations of the objects, we can conclude that P(n,r) = (n, 7) Thus we obtain = Pir) _ nt Theorem 28: C(n,r) = Pt) - Bh Recall that the binomial coefficient (”) was defined to be ——™!—; hence r, Fim—nyi C(n, 7) = () ‘We shall use C(n, r) and (Cy interchangeably.22 TECHNIQUES OF COUNTING [cHAP. 2 Example 2.14: How many committees of 3 can be formed from 8 people? Each committee is essentially a combination of the 8 people taken 3 at a time. Thus 8 “1 08,3) = (3) = EES = 6 different, committees can be formed. ORDERED PARTITIONS Suppose an urn A contains seven marbles numbered 1 through 7. We compute the num- ber of ways we can draw, first, 2 marbles from the urn, then 3 marbles from the urn, and lastly 2 marbles from the urn. In other words, we want to compute the number of ordered Partitions (An, As, As) of the set of 7 marbles into cells A; containing 2 marbles, A: containing 3 marbles and As containing 2 marbles. We call these ordered partitions since we distinguish between (1, 2} (8, 4, 5}, (6,7}) and ({6, 7}, (8, 4,5}, (1, 2}) each of which yields the same partition of A. Since we begin with 7 marbles in the urn, there are @) ways of drawing the first 2 marbles, ie. of determining the first cell A,; following this, there are 5 marbles left in the um and so there are G) ways of drawing the 3 marbles, i.e. of determining As; finally, 2 there are 2 marbles left in the urn and so there are () ways of determining the last cell As. Thus there are 7:6 54-3 2-1 Te Deg tg = 210 7\(5 2b alse) = TF different ordered partitions of A into cells A: containing 2 marbles, A: containing 3 marbles, and A, containing 2 marbles. Now observe that 1\ (5 2! a ()@)G) = Biel Brat’ 201 = 2rsrai since each numerator after the first is cancelled by the second term in the denominator of the previous factor, In a similar manner we prove (Problem 2.28) Theorem 2.9: Let A contain n elements and let m,72,...,% be positive integers with m+mt-+++n,=n. Then there exist nl mai matna! T different ordered partitions of A of the form (A1, Aa, ...,A-) where A: con- tains n: elements, Az contains nz elements, ..., and A, contains n, elements. Example 2.15: In how many ways can 9 toys be divided between 4 children if the youngest child is to receive 3 toys and each of the other children 2 toys? ‘We wish to find the number of ordered partitions of the 9 toya into 4 cells containing 3, 2, 2 and 2 toys respectively. By the above theorem, there are ot Bigtarai = 7560 such ordered partitions.