Probability & Random Process

Chapter I
 Basics
Probability & Random Process

2
 PROBABILITY THEORY
Probability theory deals with the study of random
phenomena, which under repeated experiments yield different
outcomes that have certain underlying patterns about them.

Laplaces Classical Definition: The Probability of an event A

is defined a-priori without actual experimentation as
Number of outcomes favorable to A
P( A) , (1-1)
Total number of possible outcomes
provided all these outcomes are equally likely.

3
 Consider a box with n white and m red balls. In this case,
there are two elementary outcomes: white ball or red ball.
n
Probability of selecting a white ball .
nm

Relative Frequency Definition: The probability of an

event A is defined as
nA
P ( A) lim
n n

where nA is the number of occurrences of A and n is the

total number of trials.
The axiomatic approach to probability, is modeling by
4 makes predictions that agree with physical experiments.
 Basic Terms
Let denote the set of all possible distinct,
indecomposable measurements that could be observed.
The set is called the sample space.
Finite/infinite or Countable/ uncountable sets

Elements or points in the sample space are called

outcomes.
1 , 2 ,, k , (1-3)
Collections of outcomes are called events.

5
 Cont
Note: Every point is in is not considered as valid

event; where are every event is part of the Sample

space ,.

We can take dice rolling; with the possible outcomes

,1,2,3,4,5,6
as
Any Combination of the above outcomes can be

considered as an event.
6
 Review of Set Notation
Since probability theory relies on the concept of set

theory heavily, a review of set theory is given in the

following sections.

Element

Subset

Proper subset

Complement
7
 has subsets A, B, C,. Recall that if A is a subset
of , then A implies . From A and B, we
can generate other related subsets A B, A B, A, B,
A B | A or B
etc.

A B | A and B
and

A | A (1-4)

A
A B A B A

A B A B A

8 Fig.1.1
 If A B , the empty set, then A and B are said to be
mutually exclusive (M.E).
The set difference operation is defined by A B B \ A
i.e., { | B and A}

B Fig. 1.2
A A B

A B B \ A A B

9
 A partition of is a collection of mutually exclusive

subsets of such that their union is .

A1
A2
Ai Ai Aj , and A .
i (1-5)
Aj An i 1

10
 Set Operation
Commutative property
A B B A and A B B A
Associative

A B C A B C and A B C A B C

Distributive

A B C A B A C and
A B C A B A C

11
 De-Morgans Laws:
A B A B ; A B A B (1-6)

A B A B A B A B

A B A B A B
Fig.1.3

Often it is meaningful to talk about at least some of the

subsets of as events, for which we must have mechanism

to compute their probabilities.

12
 Example: Consider the experiment where two coins are
simultaneously tossed. The various elementary events
are

1 ( H , H ), 2 ( H , T ), 3 (T , H ), 4 (T , T )
and
1 , 2 , 3 , 4 .

The subset A 1 , 2 , 3 is the same as Head

has occurred at least once and qualifies as an event.
13
 Assuming that the probability pi P(i ) of elementary
outcomes i of are apriori defined, how does one
assign probabilities to more complicated events such as
A, B, AB, etc., ?

The three axioms of probability defined below can be

used to achieve that goal.

14
PILLAI
 Theory of Probability

Probability & Random Process

15
 Axioms of Probability
For any event A, we assign a number P(A), called the
probability of the event A. This number satisfies the
following three conditions that act the axioms of
probability.
(i) P( A) 0 (Probabili ty is a nonnegativ e number)
(ii) P() 1 (Probabili ty of the whole set is unity) (1-7)
(iii) If A B , then P( A B ) P( A) P( B ).

(Note that (iii) states that if A and B are mutually exclusive (M.E.) events, the
probability of their union is the sum of their probabilities.)
16
 Example: In a coin tossing experiment, where the

same coin is tossed indefinitely, there are always

two outcomes; with out including mid point. It can
be heads or tails.

Thus, the Probability of head appearing is calculated

from Laplaces Classical Definition (Eq. 1.1)

P(heads) = 1/2 similarly, P(tails) = 1/2

Where

17
P = P(heads) +P(tails) = 1
 The following conclusions follow from these axioms:
a. Since A A , we have using (ii)
P( A A) P() 1.
But A A , and using (iii),
P( A A) P( A) P( A) 1 or P( A) 1 P( A). (1-8)

b. Similarly, for any A, A .

Hence it follows that P A P( A) P( ) .
But A A, and thus P 0. (1-9)
c. Suppose A and B are not mutually exclusive (M.E.)?
How does one compute P( A B ) ?
18
To compute the above probability, we should re-express

A B in terms of M.E. sets so that we can make use of

the probability axioms. From Fig.1.4 we have

A B A AB, (1-10) A AB Fig.1.4

where A and AB are clearly M.E. events. A B

Thus using axiom (iii)

P( A B ) P( A AB ) P( A) P( AB ). (1-11)

19
 To compute P ( AB ), we can express B as
B B B ( A A)
( B A) ( B A) BA B A (1-12)

P( B ) P( BA) P( B A), (1-13)

BA AB B A AB
From (1-13),
P( AB ) P( B ) P( AB)

and using (1-13) in (1-11)

P( A B ) P( A) P( B ) P( AB). (1-14)
20
 Conditional Probability and
Independence
Probability and random Processes

21
 Two events, A and B are said to be Independent if the
occurrence of one event dose not affect the probability of the
other.
P( A B) P( A) P( B)
For more than two events

P( Ai A j Ak ) P( Ai ) P( A j ) P( Ak )

22
 Example: If two experiments ,rolling a die and tossing a

coin are done simultaneously where the events are

independent. Then, the probability of getting head and six
dots

P ( H 6) P ( H ) P (6)
P ( H 6) 1 / 2 *1 / 6 1 / 12

23
 Example: A coin is tossed three times and the number of heads is

noted. Find the probability that the number of heads is two,

assuming the tosses are mutually independent and that on each
toss the probability of heads is for some fixed 0<<1.
Soln: we can have HHT,HTH,THH

If P(H) = , then P(T) = 1-

P(2 heads) = P(HH T) +P(H T H)+P(T H H)

= 3(2- 3)

24
 Conditional Probability
Conditional probability gives us a mathematically precise way of

handling questions of the form Given that an event B has

occurred, what is the conditional probability that A has also
occurred?

P(A|B) = Probability of the event A given that B has occurred.

We define P( A B)
P( A | B) ,
P( B)
provided P( B) 0.

25
 Total probability Theorem

Let A1 , A2..Am be disjoint events that form a partition of

the
sample space and assume P(Ai) > 1 for i = 1,2 m. Then for
any
Event B, we have
P(B) = P(A1 n B) + .+ P(Am n B)
= P(A1)P(B/A1)+ .+ P(Am)P(B/Am)
A1
B

26 A2 A3
 Example: A box contains 6 red and 4 green balls.

Remove two balls sequentially at random without

replacement. What is the probability that the first one is
red and the second one is green?

Let W1 = first ball removed is red

B2 = second ball removed is green

27
 We need P(W1 B2 ) ? We have W1 B2 W1B2 B2W1.Using the
conditional probability rule,
P(W1B2 ) P( B2W1 ) P( B2 | W1 ) P(W1 ).
But
6 6 3
P (W1 ) ,
64 10 5
and
4 4
P ( B2 | W1 ) ,
54 9
and hence
5 4 20
P (W1 B2 ) 0.25.
9 9 81
Are the events W1 and B2 independent?

28
 Our common sense says No. To verify this we need to compute

P(B2). Of course the fate of the second ball very much depends on
that of the first ball. The first ball has two options: W1 = first ball
is red or B1= first ball is green. Note that W1 B1 , and
Hence W1 together with B1 form a partition.

P( B2 ) P( B2 | W1 ) P(W1 ) P( B2 | R1 ) P( B1 )
4 3 3 4 4 3 1 2 42 2
,
54 5 6 3 10 9 5 3 5 15 5

and
2 3 20
P ( B2 ) P (W1 ) P ( B2W1 ) .
5 5 81

As expected, the events W1 and B2 are dependent.

29
 Bayes Rule
P( A B) P( A | B) P( B).

P( B A) P( A B)
P( B | A) ,
P( A) P( A)

P( A B) P( B | A) P( A).

P ( A | B ) P ( B ) P ( B | A) P ( A).

P( B | A)
P( A | B ) P( A)
P( B )
30
 Suppose A and B are independent, then
P( A B) P( A) P( B)
P( A | B) P( A).
P( B) P( B)
Thus if A and B are independent, the event that B has occurred does not
shed any more light into the event A. It makes no difference to A
whether B has occurred or not. An example will clarify the
situation:

31
 Example: The probability that a cell in a wireless system is
overloaded is 1/3. Given that it is overloaded, the probability
of a blocked call is 0.3. Given that it is not overloaded, the
probability of a blocked call is 0.1. Find the conditional
probability that the system is overloaded given that your call
is blocked.

32
 Soln:

1 2
P (O) P (O ) c

3 3
P ( B | O ) 0.3
P ( B | O ) 0.1
c

P (O | B ) ?
P ( B | O) P (O )
P (O | B )
P ( B | O ) P (O) P( B | O ) P (O )
c c

1
0.3 *
P (O | B ) 3 0.6
1 2
0.3 * 0.1*
33 3 3
 Examples:
1. A letter of the alphabet (az) is generated at

random. Specify a sample space and a

probability measure P. Compute the probability
that a vowel (a, e, i, o, u) is generated.
Soln:

a, b, c, d , e, f , g , h, i, g , k , l , m, n, o, p,
.
q, r , s, t , u, v, w, x, y, z
34
 Probability of any letter,

1
P(i ) .
26
{a, e, i, o, u} an event consisting five outcomes.

P( A) 1 / 26 1 / 26 1 / 26 1 / 26 1 / 26
5 / 26.

35
 2. Two boxes B1 and B2 contain 100 and 200 light bulbs
respectively. The first box (B1) has 15 defective bulbs
and the second 5. Suppose a box is selected at random
and one bulb is picked out.
(a) What is the probability that it is defective?
Solution: Note that box B1 has 85 good and 15 defective
bulbs. Similarly box B2 has 195 good and 5 defective
bulbs. Let D = Defective bulb is picked out.
Then
15 5
P ( D | B1 ) 0.15, P ( D | B2 ) 0.025.
100 200
36
 Since a box is selected at random, they are equally
likely.
1
P ( B1 ) P ( B2 ) .
2
we obtain
P( D) P( D | B1 ) P( B1 ) P( D | B2 ) P( B2 )
1 1
0.15 0.025 0.0875.
2 2

Thus, there is about 9% probability that a bulb

picked at random is defective.

37
 (b) Suppose we test the bulb and it is found to be

defective. What is the probability that it came from

box 1? P( B1 | D) ?

P( D | B1 ) P( B1 ) 0.15 1 / 2
P( B1 | D) 0.8571.
P ( D) 0.0875

Notice that initially P( B1 ) 0.5; then we picked out

a box at random and tested a bulb that turned out to
be defective. Can this information shed some light
about the fact that we might have picked up box
38 1?
 Since P( B1 | D) 0.857 0.5, and indeed it is
more likely at this point that we must have chosen
box 1 in favor of box 2. (Recall box1 has six times
more defective bulbs compared to box2).

39
 3) Suppose box 1 containers x white balls and y black
balls, and box 2 containers w white balls and z black.
One ball of unknown color is transferred from box 1 to
box 2 and then a ball is drawn from the second box.
What is the probability that it will be white?
Soln: If a ball is not transferred from box 1 to box 2 then
w
P (W )
w z

z
P( z )
w z
40
If we consider probablity of the second ball being white, then
A { the drawn ball is white}
B {the transferr ed ball is white}
C {the transferr ed ball is black}
P(A) {the probabilit y of event A given the transferr ed ball is white} or
{the probabilit y of event A given the transferr ed ball is white}
P( A) P( A / B) * P( B) P( A / C ) * P(C )
a b
(( ) * ( w 1) /( w z 1)) (( ) * ( w /( w z 1)))
ab ab

41