Beruflich Dokumente
Kultur Dokumente
let
Definition: if
is defined as
Definition: The relative entropy or Kullback Leibler distance between two probability mass functions and is defined as
Relative entropy is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality. Definition: Consider two random variables and with a joint probability mass function and marginal probability mass function and . The mutual information is the relative entropy between the joint distribution and the product distribution , i.e.,
and
and
be drawn according to
Definition: The conditional relative entropy is the average of relative entropies between the conditional probability mass functions and averaged over the probability mass function .
if for
A function is said to be strictly convex if equality holds only if Definition: A function is concave if is convex
or
A function is convex if it always lies below any chord. A function is concave if it always lies above any chord. Theorem: if the function has a second derivative which is non-negative (positive) everywhere, then the function is convex (strictly convex) Theorem: Jensen's inequality : If is a convex function and is a random variable then
If f is a convex function on the interval [ab] , for each xknk=1[ab] and each knk=1 with k0 one has: fknknk=1kxkknknk=1kfxk A common situation occurs when1+2++n=1; in this case, the inequality simplifies to:
fnk=1kxknk=1kf(xk)
where0k1. Iffis a concave function, the inequality is reversed. Corollary: Non-negativity of mutual information: For any two random variables, and ,
with equality if and only if Corollary: with equality if and only if Corollary: , with equality if and only if
Theorem: , where denotes the number of elements in the range of , with equality if and only if has a uniform distribution
Theorem: Conditioning reduces entropy: with equality if and only if and are independent Theorem: Independence bound on entropy: Let to , then
be drawn according
and
with equality if and only if Theorem: is a convex in the pair paris of probability mass functions , then , i.e., if and are two
for all Theorem: Concavity of entropy: is a concave function of outcomes. The concavity of then
where u is the uniform distribution of follows directly from the convexity of Theorem: Let a concave function of fixed
for fixed
is for
Definition: Random variables are said to form a Markov chain in the order (denoted ) if the conditional distribution of only depends on and is conditionally independent of . Specifically, form a Markov chain if the joint probability mass function can be written as
if and only if
implies Theorem: Data Processing inequality: If We have equality if and only if Similarly Corollary: In particular, if Corollary: if then when do not form a , we have , i.e., , then