Intro Probability Theory

PnRV Lecture # 01
Probability and
Random Variables
Instructor: Engr. Muhammad Umar Khan

Textbooks
Probability & Random Processes Albert Leon-Garcia
for Electrical Engineers, 2nd ed.
Introduction to Probability Models, Sheldon M. Ross

9th ed.
Elements of Information Theory Thomas M. Cover and Joy

A. Thomas
Chaos Theory Tamed Garnett P. Williams
Copyright © Syed Ali Khayam 2008

Course Outline
• Sets Theory
• Basics of Probability
• Conditional Probability
• Random Variables
• Distributions & Density Functions
• Moments
• Linear Regression and Curve Fitting
• Limits Theorems
• Stochastic Processes
• Applications

What will we cover in this lecture?
This lecture is intended to be an introduction to elementary
probability theory
We will cover:
Random Experiments and Random Variables
Axioms of Probability
Mutual Exclusivity
Conditional Probability
Independence
Law of Total Probability
Bayes’ Theorem

Definition of Probability
❑ Probability:
• the quality or state of being possible
• something (as an event or circumstance) that is possible
• the ratio of the number of outcomes in an exhaustive set
of equally likely outcomes that produce a given event to
the total number of possible outcomes, the chance that a
given event will occur
We will revisit these definitions in a little bit …

Definition of a Random Experiment
A random experiment comprises of:
• A procedure
• An outcome
Procedure
(e.g., flipping a coin)
Sample Space
(Set of All Possible
Outcome Outcomes)
(e.g., the value
observed [head, tail] after
flipping the coin)

Outcomes, Events and the Sample Space
An outcome cannot be further decomposed into other outcomes

{s1 = the value 1}, …, {s6 = the value 6}
An event is a set of outcomes that are of interest to us

A = {s: such that s is an even number}
The set of all possible outcomes, S, is called the sample space

S = {s1, s2, s3, s4, s5, s6}

s1 s5
s6
s2 s4
s3

Example of a Random Experiment:

Experiment: Roll a fair dice once and record the number of dots on the
top face
S = {1, 2, 3, 4, 5, 6}
A = “the outcome is even” = {2, 4, 6}
B = “the outcome is greater than 4” = {5, 6}

Axioms of Probability
Probability of any event A is non‐negative:
Pr{A} ≥ 0
The probability that an outcome belongs to the sample space is 1:

Pr{S} = 1
The probability of the union of mutually exclusive events is equal

to the sum of their probabilities:
If A1 ∩ A2=Ø,
=> Pr{A1 U A2} = Pr{A1} + Pr{A2}

Mutual Exclusivity
For mutually exclusive events A1, A2 … AN, we have:
A1 s1
s5
Find Pr{A1 U A2} and

s6 Pr{A1}+Pr{A2} in the fair
s2 s4 dice example
A2
s3

Mutual Exclusivity
In general, we have:
Pr{A1 U A2} = ??
s1 s5
s6
s2 s4
s3

Mutual Exclusivity
In general, we have
Pr{A1 U A2} = Pr{A1} + Pr{A2} – Pr{A1 ∩ A2}
s1 s5
s6
s2 s4
s3

Given that event B has already occurred, what is the probability
that event A will occur?
Given that event B has already occurred, reduces the sample
space of A
s1 s5 s1 s5
Event B has
already occurred
s6 => s2, s4, s3 s6
s2 s4 s2 s4
cannot occur
s3 s3
S S

Given that event B has already occurred, we define a new
conditional sample space that only contains B’s outcomes
The new event space for A is the intersection of A and B:
Event space ‐> EA|B = A ∩ B
s1 s5 s1 s5
Event B has
already
s6 occurred s6
s2 s4 s2 s4
s3 s3
S S
S|B = {s1, s5, s6}
What’s missing here?
EA|B= A ∩ B = {s6}
The probability of an event A in the conditional sample space is:
Pr A∩B
Pr B
Pr{s6 } 1/6 1
= Pr{B} 3/6 3
s1 s5 s1 s5
Event B has
already
s6 occurred s6
s2 s4 s2 s4
s3 s3
S S
S|B = {s1, s5, s6}
Independence
❑ Two events are independent if they do not provide
any information about each other:
Pr(A|B) = Pr(A)
❑ In other words, the fact that B has already happened does

not affect the probability of A’s outcomes
❑ Implications:
Pr(A|B) = Pr(A)
Pr A ∩ B = Pr(A)
Pr(B)
Pr(A ∩ B) = Pr(A) Pr(B)
Independence: Example
Are events A and C independent?
 Assume that all outcomes are equally likely
s2
s6
s4
s5
s1 s3

❑ Are events A and C independent?
s2
s6
s4
s5
s1 s3
S
❑ Are events A and C independent?
Pr{A ∩ C} = Pr{s5} = 1/6
Pr{A}Pr{C} = (3/6)x(2/6) = 1/6
Yes!
s2
s6
s4
s5
s1 s3
S
Are events A and B independent?
 Assume that all outcomes are equally likely
s2
s6
s4
s5
s1 s3
S
❑ Are events A and B independent?
Pr{A ∩ B} = Pr{s5} = 1/6
Pr{A}Pr{B} = (3/6)x(3/6) = ¼
No!
s2
s6
s4
s5
s1 s3
S
Mutual Exclusivity and Independence
Experiment:
Roll a fair dice twice and record the dots on the top face:

Define three events:
A1 = “first roll gives an odd number”

A2 = “second roll gives an odd number”
C = “the sum of the two rolls is odd”
Find the probability of C using probability of A1 and A2

24
A2
S={ (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),
(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

A1
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }







Recap
1. Outcomes, events and sample space:
2. For mutually exclusive events A1, A2,…, AN, we have:
3. In general, we have:

Recap
4. Conditional probability reduces the sample space:
5. Two events A and B are independent only if
6. For independent events:

Four “Rules of Thumb”
1. Whenever you see two events which have an OR relationship (i.e., event A or
event B), their joint event will be their union, {A U B}
Example: On a binary channel, find the probability of error?
An error occurs when
A: “a 0 is transmitted and a 1 is received” OR
B: “a 1 is transmitted and a 0 is received”
Thus probability of error is: Pr{A U B}
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}

2. Whenever you see two events which have an AND relationship (i.e., both
event A and event B), their joint event will be their intersection, {A ∩ B}
Example: On a binary channel, find the probability that a 0 is transmitted and a
1 is received?
A: “a 0 is transmitted” AND
B: “a 1 is received”
Thus probability of above event is: Pr{A ∩ B}
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}

3. Whenever you see two events which have an OR relationship (i.e., A U B),
check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
B: “a 1 is transmitted and a 0 is received” Thus
probability of error is: Pr{error} = Pr{A U B} Are A
and B are mutually exclusive?
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}

3. Whenever you see two events which have an OR relationship (i.e., A U B), check if
they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
B: “a 1 is transmitted and a 0 is received”
Thus probability of error is: Pr{error} = Pr{A U B}
YES!
A and B are mutually exclusive; transmission of a 0 precludes the possibility of
transmission of a 1, and vice versa. Therefore, we can set
Pr{error} = Pr{A U B} = Pr{A} + Pr{B}
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}

4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if
they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?
Probability of above event is: Pr{A ∩ B}
Are A and B independent?
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}
1 e01
Pr A ∩ B = Pr A B Pr B = e01 x = 2
2
1 1 1
Pr A Pr B = x =
2 2 4
4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if
they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?
Probability of above event is: Pr{A ∩ B}
Are A and B independent?
No.
Pr{R0|T0}
T0 R0
T1 R1
Pr{R1|T1}
1 e01
Pr A ∩ B = Pr A B Pr B = e01 x = 2
2
1 1 1
Pr A Pr B = x =
2 2 4
Total Probability
❑ B1, B2,…, BN form a partition of a sample space we have:

 S = B1 U B2 U … U BN
 Bi ∩ Bj = Ø, i ≠ j
B2
s1 s5
B3
B1 B4 s6
s4
s2
s3

Total Probability
❑ If B1, B2,…, BN form a mutually exclusive partition:
What does this imply?
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability
What does this imply? B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability
What does this imply? B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S
How to express A in term of Bi?
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability

B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S
How to express A in term of Bi?
A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability

How to express A in term of Bi? A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)
What is the probability of A?
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability

How to express A in term of Bi? A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)
What is the probability of A? Pr{A} = Pr{A ∩ B1} + Pr{A ∩ B2} + … + Pr{A ∩ BN}
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Total Probability
❑ Using the definition of conditional probability:
Pr{A| Bi} = Pr{A ∩ Bi} / Pr{Bi}
=> Pr{A ∩ Bi} = Pr{A| Bi} Pr{Bi}
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

The Law of Total Probability
The Law of Total Probability states:
If B1, B2,…, BN form a partition then for any event A
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Bayes’ Theorem
❑ Based on the Law of Total Probability, Thomas Bayes decided to
look at the probability of a partition given a particular event, the
so‐called inverse probability.
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Bayes’ Theorem
❑ Based on the Law of Total Probability, Thomas Bayes decided to
look at the probability of a partition given a particular event
Pr{Bi|A} = Pr{A ∩ Bi} / Pr{A}
=> Pr{A ∩ Bi} = Pr{A|Bi} Pr{Bi}
=> Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Bayes’ Theorem
Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
From the Law of Total Probability, we have:
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}
Bayes’ Rule
B2
s1 s5
B1 A A s6
B4 s4
s2
B3
s3

Putting It All Together
❑ Experiment: “Roll the die, observe the number of dots on the
face that comes up.”
Sample/ Outcome
x=3
Sample Space
Random Variable X
1 2 3 4 5 6 Outcomes x
Sample Space SX
Announcement of Quiz
Study Material:
Book: Probability, Statistics and Random Processes for Electrical Engineers 3rd
Edition, Albert Leon Garcia
Chapter 2: Basic Concepts of Probability Theory
• Topic: 2.1, 2.2, 2.4, 2.5 (Examples included)
• End Problems: 2.23, 2.62-2.96

Intro Probability Theory

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Intro Probability Theory

Hochgeladen von

Copyright:

Verfügbare Formate

PnRV Lecture # 01

Instructor: Engr. Muhammad Umar Khan

Introduction to Probability Models, Sheldon M. Ross

Elements of Information Theory Thomas M. Cover and Joy

Chaos Theory Tamed Garnett P. Williams

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

An outcome cannot be further decomposed into other outcomes

An event is a set of outcomes that are of interest to us

The set of all possible outcomes, S, is called the sample space

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Example of a Random Experiment:

Copyright © Syed Ali Khayam 2008

The probability that an outcome belongs to the sample space is 1:

The probability of the union of mutually exclusive events is equal

=> Pr{A1 U A2} = Pr{A1} + Pr{A2}

Copyright © Syed Ali Khayam 2008

Find Pr{A1 U A2} and

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

❑ In other words, the fact that B has already happened does

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Define three events:

A1 = “first roll gives an odd number”

Find the probability of C using probability of A1 and A2

Copyright © Syed Ali Khayam 2008

S={ (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

2. For mutually exclusive events A1, A2,…, AN, we have:

Copyright © Syed Ali Khayam 2008

5. Two events A and B are independent only if

6. For independent events:

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

Copyright © Syed Ali Khayam 2008

❑ B1, B2,…, BN form a partition of a sample space we have:

Copyright © Syed Ali Khayam 2008

What does this imply?

Copyright © Syed Ali Khayam 2008

What does this imply? B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S

Copyright © Syed Ali Khayam 2008

What does this imply? B1 ∩ B2 ∩ ….. ∩ Bn = Ø and B1 U B2 U ….. U Bn = S

How to express A in term of Bi?

Copyright © Syed Ali Khayam 2008

What does this imply?