Chapter 1.

Probability Theory

1. Algebra of Sets

(1) Let  denote the set of all elements under consideration; that is,  is the universal set.

If  is an element of set  , we will write ∈  .

(2) For any two sets  and  , we will say that  is a subset of  , or  is contained in 
(denoted  ⊂  ), if every point in  is also in  .

(3) The empty set, denoted by ∅ , is the set consisting of no points.

(4) If  is a subset of  , the complement of  , denoted by   , is the set of points that are

in  but not in  ;       ∈  and  ∉  .

(5) The union of  and  , denoted by  ∪  , is the set of all points in  or  or both;
 ∪      ∈  or ∈  .

(6) The intersection of  and  , denoted by  ∩  , is the set of all points in both  and  ;

 ∩      ∈  and ∈  


(7) The union of sets   ,   , ⋯ ,   :   ∪   ∪ ⋯ ∪    
 
     ∈   for some   ⋯ 

∞
The union of sets   ,   ,   , ⋯ :   ∪   ∪   ∪ ⋯  
 
     ∈   for some        ⋯ 


The intersection of sets   ,   , ⋯ ,   :   ∩   ∩ ⋯ ∩    
 
     ∈   for all       ⋯   

∞
The intersection of sets   ,   ,   , ⋯ :   ∩   ∩   ∩ ⋯  
 
     ∈   for all   ⋯ 

 ∞
        
  
 
       ,
       
 
        ,
 

 ∞
       
              

           

(8) Commutative laws :  ∪   ∪  ,  ∩  ∩

(9) Associative laws :  ∪  ∪   ∪  ∪  ,  ∩  ∩    ∩  ∩ 

(10) Distributive laws :  ∩  ∪   ∩  ∪  ∩   ,  ∪  ∩   ∪  ∩  ∪ 

(11) DeMorgan’s laws :  ∪     ∩   ,  ∩     ∪  

(12)  and  are said to be mutually exclusive, or disjoint, if  ∩   ∅ .

(13)   ,   ,   , ⋯ are said to be mutually exclusive if  ∩    ∅ for all  ≠  .

1
2. Basics of Probability Theory

(1) A random experiment is an experiment in which

(a) all possible outcomes of the experiment are known,

(b) the outcome cannot be predicted with certainty before the experiment is performed,
(c) the experiment can be repeated under identical conditions.

(2) The sample space  is the set of all possible outcomes of a random experiment.

(3) If  is a subset of  , then  is called an event.

(4) If  contains at most a countable number of elements, finite or countably infinite, we say
that  is a discrete sample space. If  contains an uncountable number of elements,
we say that  is a continuous sample space.

(5) (a) Tossing a die;

The sample space         has  elements.  is a discrete (finite) sample space.

(b) Tossing a coin until a head appears;

The sample space   H  TH  TTH  TTTH  ⋯  has a countable number of elements.
 is a discrete sample space.

(c) Randomly select a real number from the interval    ;

The sample space      has an uncountable number of elements.
 is a continuous sample space.

(6) A function  from a set  to a set  is a rule that assigns a unique (single) element

  ∈  to each element ∈  . The set  of all possible input values is called the domain

of the function. The set of all output values of    as  varies throughout  is called
the range of the function. When the range of a function is a set of real numbers,
the function is said to be real-valued. A set function is a real-valued function defined on
a collection of sets.

(7) Axiom of probability (Kolmogorov)

Probability is a set function  that assigns, to each event  in the sample space  ,
a number    , called the probability of the event  , such that the following properties
are satisfied:

(a)    ≥  ;

(b)      ;

2
 (c) if          ⋯ are mutually exclusive events, then

 
∞ ∞
   ∪   ∪   ∪ ⋯   
 
     .
 
 (countable additivity)

(8) If            ⋯  is a discrete sample space, it is sufficient to assign probability to each

∞
  such that     ,  ≤  ≤  , 

  .

(9) A coin is tossed two times. The four elements of the sample space  are

    ,     ,     ,     .

If the coin is balanced,         are equally likely outcomes. Hence we can assign equal

probabilities; that is,     ,       . If the coin is unbalanced and      ,

it does not seem appropriate to assign equal probabilities to elements of  . Under certain
conditions it is reasonable to assign      ,      ,      ,      .

(10) If  is a continuous sample space, we cannot assign positive probability to each element
of  without violating      . In this case we can assign probabilities to compound
events consisting of intervals.

(11) For each event  ,         .

Proof: We have    ∪   and  ∩    ∅ . Thus, from properties (b) and (c),

            . Hence         .

(12)  ∅    .

Proof: Take   ∅ so that     . Then  ∅          .

(13) If events  and  are such that  ⊂  , then    ≤    .

Proof: We have    ∪  ∩   and  ∩  ∩   ∅ . Hence, from property (c),

        ∩  ≥    because, from property (a),   ∩  ≥  .

(14) For each event  ,    ≤  .

Proof: Since  ⊂  ,    ≤    ≤  .

(15) If  and  are any two events, then   ∪             ∩   .

Proof: The event  ∪  can be expressed as a union of mutually exclusive events, namely,

3
  ∪    ∪  ∩   . Hence   ∪         ∩   . However,    ∩  ∪  ∩   ,

which is a union of mutually exclusive events. Thus       ∩     ∩   and

  ∩         ∩   . If the right-hand side of this equation is substituted into

  ∪         ∩   , we obtain   ∪             ∩   .

(16) Inclusion-Exclusion formula

   ∪   ∪                       ∩        ∩        ∩        ∩   ∩   

            ∩     ∩ ∩
 
         ⋯         ∩   ∩ ⋯ ∩   
       

3. Methods of Enumeration

(1) Let      ⋯     be a finite sample space. If each   has the same probability of occurring,

we say that the  outcomes are equally likely. That is,    

,       ⋯  . If the number

of outcomes in an event  is , under the assumption of equally likely outcomes, we have

   
       ,
   

where    is the number of outcomes in  and    is the number of elements of  .

(2) Multiplication rule

If  experiments   ,   , ⋯ ,   have  ,  , ⋯ ,  possible outcomes, then the composite

experiment     ⋯   has   ⋯  possible outcomes.

(3) The number of possible outcomes of the rolling of a pair of dice and two tosses of a coin
is        .

(4) Suppose that  positions are to be filled with  different objects. There are  choices for

filling the first position,    for the second, ⋯ , and  choice for the last position. So,
by the multiplication principle, there are         ⋯      possible arrangements.
Each of the  arrangements of  different objects is called a permutation of the  objects.

(5) If only  positions are to be filled with objects selected from  different objects,  ≤  , then

the number of possible ordered arrangements is


P           ⋯         .
   

Each of the  P  arrangements is called a permutation of  objects taken  at a time.

4
(6) If  objects are selected from a set of  objects, and if the order of selection is noted,
then the selected set of  objects is called an ordered sample of size  .

(7) Sampling with replacement occurs when an object is selected and then replaced

before the next object is selected.

(8) By the multiplication principle, the number of possible ordered samples of size  taken

from a set of  objects is    ⋯     when sampling with replacement.

(9) Sampling without replacement occurs when an object is not replaced

after it has been selected.

(10) By the multiplication principle, the number of possible ordered samples of size  taken
from a set of  objects without replacement is  P  , the number of permutations of

 objects taken  at a time.

(11) Birthday problem

Consider a class of  students. The birthdays of these  students form a sample of size 
from the  days in the year. We assume that every student has an equal probability of

being born on any day during the year, independent of everyone else, and ignore the
additional complication presented by leap years (that is, assume that nobody is born on
February 29). Then the probability that all  birthdays are different is


 P   
  
 

 
     ⋯   
          
   

.

      
           

(12) Let  denote the number of unordered samples of size  that can be selected from

 different objects without replacement. We can obtain each of the P  ordered subsets

by first selecting one of the  unordered subsets of  objects and then ordering these

 objects. Since the latter ordering can be carried out in  ways, the multiplication

principle yields    ordered subsets; so    must equal  P  . Thus, we have

P  
    
     


 C  . 

5
 
Each of the  (  choose  ) unordered samples is called a combination of  objects taken

 at a time. We could also say that the number of ways in which  objects can be selected

without replacement from  objects when the order of selection is disregarded is  .  

(13) The number of ways in which  objects can be selected out of  objects is equal to the
number of ways in which    objects can be selected out of  objects. That is,

    
(14) Binomial theorem


       

  

The terms  are called binomial coefficients since they arise in the binomial expansion.
(15) An urn contains  red,  green  blue, and  white balls. A sample of size  is selected at
random without replacement. The probability that the sample contains  red,  green,

 blue, and  white balls is

   .
 
 

(16) Pascal’s triangle


 
 
   
    
⋮ ⋮ ⋮ ⋮ ⋮

The  th row of this triangle gives the coefficients for       . To find an entry in the
table other than a  on the boundary, add the two nearest numbers in the row directly
above. The equation

        

explains why the Pascal’s triangle works.

 
(17) If we consider     and    , we can prove that       


and     .



6
(18) Suppose that a set contains  objects of two types:  of one type and    of the other
type. The number of permutations of  different objects is  . However, in this case,

the objects are not all distinguishable. To count the number of distinguishable
arrangements, first select  out of the  positions for the objects of the first type. This


can be done in  ways. Then fill in the remaining positions with the objects of the

second type. Thus, the number of distinguishable arrangements is  . Each of the   

permutations of  objects,  of one type and    of another type, is called a
distinguishable permutation.

(19) Suppose that in a set of  objects,  are of one kind,  of a second kind, ⋯ ,  of

a  th kind, where       ⋯   . Then the number of distinguishable permutations

of the  objects is

    ⋯     
 

   ⋯  
 
.
 


The terms     ⋯  
  

 are often called multinomial coefficients because they occur
in the expansion of the multinomial term     ⋯   raised to the  th power.

  

    ⋯         ⋯     ⋯  ,
   
   

where this sum is taken over all     ⋯   such that       ⋯   .

(20) To count the number of unordered samples of size  selected from  objects with
replacement, think of  defining bins in which we can place  stars. So if   
and    , a possible outcome is

* ** * ***
    

which says that zero  ’s, one  , two  ’s, one  and three  ’s. That is,  . In

general, each outcome is a permutation of  stars and    walls,  bins yield   

walls, but we disregard the two end walls. Each distinguishable permutation is equivalent
to an unordered sample. The number of distinguishable permutations, and hence

the number of unordered samples of size  that can be selected out of  objects
when sampling with replacement, is

7
        
    

 
. 
(21) Number of samples of size  from  objects

without with
replacement replacement

ordered 
    

unordered      

(22) Suppose that we are going to calculate all possible averages of  numbers selected from
       with replacement. The outcomes in the ordered and unordered sample spaces,

averages and probabilities are in the following table.

unordered ordered average probability

              
              
              
           ,    ,         
           ,    ,          
           ,    ,          
           ,    ,          
           ,    ,          
           ,    ,          
           ,    ,    ,     ,     ,        

The probabilities come from considering the    outcomes in the ordered sample space
to be equally likely. This corresponds to the common interpretation of sampling with
replacement; namely, one of the three items is chosen, each with probability  ; the item
is noted and replaced; the items are mixed and again one of the three items is chosen,
each with probability  . If we are only interested in the average of the sampled numbers,
the ordering is unimportant, and thus the total number of distinct samples is obtained by
counting according to unordered, with-replacement sampling. The total number of distinct

samples is        and the  outcomes in the unordered sample space are not

equally likely. To calculate the probability distribution of the sample averages, we must
count the different ways that a particular average can occur. It is possible that two
different unordered samples will result in the same average. For example, the unordered

8
 samples    and    both result in an average value of  . The first sample has
probability  and the second has probability  , giving the value  probability  .

Hence the probability distribution of the sample averages is given by

average           

probability                   .

4. Conditional Probability

(1) The conditional probability of an event  , given that event  has occurred, is defined by

  ∩  
    
  

provided that     .

(2) The conditional probability function  ⋅    satisfies three axioms for a probability function
with      .

(a)    ≥ 

(b)     

  
∞ ∞
(c) 
 
      ,
 
 where         ⋯ are mutually exclusive events.

(3) An urn contains  red,  blue, and  white balls. Four balls are to be selected at random
without replacement. The conditional probability of two red balls (event  ) given that
at least one ball of each color (event  ) is

  ∩  
           .
   /  
          
           /   
    
  

    

(4) Multiplication rule

The probability   ∩   is given by the multiplication rule,

  ∩                 ,

if      and      .

(5) The multiplication rule can be extended to three or more events. In the case of three
events, using the multiplication rule for two events, we have

9
    ∩   ∩       ∩        ∩                  ∩    .

This type of argument can be used to extend the multiplication rule to more than three
events, and the following formula for  events can be proved by mathematical induction.

   ∩   ∩ ⋯ ∩                ∩   ⋯     ∩   ∩ ⋯ ∩     

(6) An urn contains  balls of which  are black and  are white. At each trial a ball is
selected at random, its color is noted, and it is replaced along with two additional balls
of the same color. What is the probability that a black ball is selected in each of the first
three trials?

Solution: Let   be the event that a black ball is selected on the  th trial. Then

      
   ∩   ∩                 ∩         
   


5. Independent Events

(1) Two events  and  are independent if any of the following holds:

  ∩         ,

        ,

        .

Otherwise,  and  are called dependent events.

(2) A red die and a white die are rolled. Let     on the red die ,    sum of dice is odd ,
and    sum of dice is  . Of the  equally likely outcomes,      ,      ,      ,

  ∩     and   ∩     . Then

    
            ∩  
  


and

     
         ≠     ∩   .
   


Hence  and  are independent events but  and  are dependent events.

(3) If  and  are independent events, then the following pairs of events are also independent:

(a)  and   ;

(b)   and  ;

10
 (c)   and   .

Proof: (a) If      , then              . Thus

  ∩                       

                 ,

since          by independence. Consequently,  and   are independent.

(4) Events         ⋯   are independent if and only if the following conditions hold:

  ∩        for  ≠  ,

  ∩  ∩          for  ≠  ≠  ,

   ∩   ⋯ ∩           ⋯     .

(5) Events          ⋯    are pairwise independent if   ∩        for  ≠  .

(6) Independence of          ⋯    implies pairwise independence, but pairwise independence

does not imply independence.

(7) A rocket has a built-in redundant system. In this system, if component   fails, it is

bypassed and component   is used. If component   fails, it is bypassed and component

  is used. Suppose that the probability of failure of any one component is  , and assume

that the failures of these components are mutually independent events. Let   denote the

event that component   fails for    . Because the system fails if  fails and  fails

and   fails, the probability that the system does not fail is given by

    ∩   ∩           ∩   ∩                       

(8) A red die and a white die are rolled. Let    {odd number on the red die},

   {odd number on the white die}, and    {sum of dice is odd}. Of the  equally likely
outcomes,                 ,   ∩      for  ≠  , and    ∩   ∩      . Then


  ∩           for  ≠ ,

and

     ∩   ∩    ≠            


Hence         are pairwise independent, but they are not independent.

11
6. Bayes’ Theorem

(1) The collection of sets       ⋯     is called a partition of  if     ∪   ∪ ⋯ ∪   and

 ∩    ∅ , for  ≠  .

(2) Total probability theorem

Assume that       ⋯     is a partition of a sample space  such that     ,

for     ⋯   . Then for any event 


         .
 
 

Proof: If  is an event, then  is the union of  mutually exclusive events, namely,

   ∩    ∩   ∪   ∪ ⋯ ∪      ∩   ∪  ∩   ∪ ⋯ ∪  ∩    .

Thus
 
       ∩          .
 

 
 

(3) Bayes’ Theorem

Assume that       ⋯     is a partition of  such that     , for     ⋯   . Then

for any event  for which      ,

      
    

,     ⋯   .

     
 

Proof: The proof follows directly from the definition of conditional probability and
the total probability theorem. Note that

  ∩         
      
   


     
 

Remark : The probability    is called the prior probability of   and the conditional

probability     is called the posterior probability of   .

(4) There are five urns, and they are numbered  to  . Each urn contains  balls. Urn  has

 defective balls and    nondefective balls,     ⋯   . For instance, urn  has three

defective balls and seven nondefective balls. Consider the following random experiment:
First an urn is selected at random, and then a ball is selected at random from the

12
selected urn. (The experimenter does not know which urn was selected.) Let us ask two
questions:

(a) What is the probability that a defective ball will be selected?

(b) If we have already selected the ball and noted that it is defective, what is the
probability that it came from urn  ?

Solution: Let  denote the event that a defective ball is selected and   the event that

urn  is selected,    ⋯   . Note that


      ,    ⋯   ,


and

     ,    ⋯   .


Using the total probability theorem, we have

 
 
    
 
       

 
 .


Employing Bayes’ theorem, we find

              
          .
    

     
 

Similarly,

         

         ,    ⋯   .
    

     
 

13