Beruflich Dokumente
Kultur Dokumente
Maximumentropyprobabilitydistribution
FromWikipedia,thefreeencyclopedia
Contents
1Definitionofentropy 2Examplesofmaximumentropydistributions 2.1Givenmeanandstandarddeviation:thenormaldistribution 2.2Uniformandpiecewiseuniformdistributions 2.3Positiveandgivenmean:theexponentialdistribution 2.4Discretedistributionswithgivenmean 2.5Circularrandomvariables 3AtheorembyBoltzmann 3.1Continuousversion 3.2Discreteversion 3.3Caveats 4Seealso 5Notes 6References
Definitionofentropy
Furtherinformation:Entropy(informationtheory) IfXisadiscreterandomvariablewithdistributiongivenby
thentheentropyofXisdefinedas
IfXisacontinuousrandomvariablewithprobabilitydensityp(x),thentheentropyofXissometimesdefined
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
1/6
4/1/12
as[1][2][3]
Examplesofmaximumentropydistributions
AtableofexamplesofmaximumentropydistributionsisgiveninPark&Bera(2009)[4]
Givenmeanandstandarddeviation:thenormaldistribution
ThenormaldistributionN(,2)hasmaximumentropyamongallrealvalueddistributionswithspecified meanandstandarddeviation.Therefore,theassumptionofnormalityimposestheminimalpriorstructural constraintbeyondthesemoments.(Seethedifferentialentropyarticleforaderivation.)
Uniformandpiecewiseuniformdistributions
Theuniformdistributionontheinterval[a,b]isthemaximumentropydistributionamongallcontinuous distributionswhicharesupportedintheinterval[a,b](whichmeansthattheprobabilitydensityis0outsideof theinterval). Moregenerally,ifwe'regivenasubdivisiona=a0<a1<...<ak=boftheinterval[a,b]andprobabilities p1,...,pkwhichadduptoone,thenwecanconsidertheclassofallcontinuousdistributionssuchthat
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
2/6
4/1/12
Positiveandgivenmean:theexponentialdistribution
Theexponentialdistributionwithmean1/isthemaximumentropydistributionamongallcontinuous distributionssupportedin[0,)thathaveameanof1/. Inphysics,thisoccurswhengravityactsonagasthatiskeptatconstantpressureandtemperature:ifX describestheheightofamolecule,thenthevariableXisexponentiallydistributed(whichalsomeansthatthe densityofthegasdependsonheightproportionaltotheexponentialdistribution).Thereason:Xisclearly positiveanditsmean,whichcorrespondstotheaveragepotentialenergy,isfixed.Overtime,thesystemwill attainitsmaximumentropyconfiguration,accordingtothesecondlawofthermodynamics.
Discretedistributionswithgivenmean
Amongallthediscretedistributionssupportedontheset{x1,...,xn}withmean,themaximumentropy distributionhasthefollowingshape:
whereagaintheconstantsCandrweredeterminedbytherequirementsthatthesumofalltheprobabilities mustbe1andtheexpectedvaluemustbe.Forexample,inthecasethatxk=k,thisgives
Circularrandomvariables
Foracontinuousrandomvariable distributedabouttheunitcircle,theVonMisesdistributionmaximizes theentropywhengiventherealandimaginarypartsofthefirstcircularmoment[5]or,equivalently,the circularmeanandcircularvariance. Whengiventhemeanandvarianceoftheangles modulo theentropy.[5] ,thewrappednormaldistributionmaximizes
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
3/6
4/1/12
AtheorembyBoltzmann
AlltheaboveexamplesareconsequencesofthefollowingtheorembyLudwigBoltzmann.
Continuousversion
SupposeSisaclosedsubsetoftherealnumbersRandwe'regivennmeasurablefunctionsf1,...,fnandn numbersa1,...,an.WeconsidertheclassCofallcontinuousrandomvariableswhicharesupportedonS(i.e. whosedensityfunctioniszerooutsideofS)andwhichsatisfythenexpectedvalueconditions
IfthereisamemberinCwhosedensityfunctionispositiveeverywhereinS,andifthereexistsamaximal entropydistributionforC,thenitsprobabilitydensityp(x)hasthefollowingshape:
Discreteversion
SupposeS={x1,x2,...}isa(finiteorinfinite)discretesubsetoftherealsandwe'regivennfunctionsf1,...,fn andnnumbersa1,...,an.WeconsidertheclassCofalldiscreterandomvariablesXwhicharesupportedonS andwhichsatisfythenconditions
IfthereexistsamemberofCwhichassignspositiveprobabilitytoallmembersofSandifthereexistsa maximumentropydistributionforC,thenthisdistributionhasthefollowingshape:
wheretheconstantscandjhavetobedeterminedsothatthesumoftheprobabilitiesis1andtheabove conditionsfortheexpectedvaluesaresatisfied.
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
4/6
4/1/12
Caveats
Notethatnotallclassesofdistributionscontainamaximumentropydistribution.Itispossiblethataclass containdistributionsofarbitrarilylargeentropy(e.g.theclassofallcontinuousdistributionsonRwithmean0 butarbitrarystandarddeviation),orthattheentropiesareboundedabovebutthereisnodistributionwhich attainsthemaximalentropy(e.g.theclassofallcontinuousdistributionsXonRwithE(X)=0andE(X2)= E(X3)=1[citationneeded]). ItisalsopossiblethattheexpectedvaluerestrictionsfortheclassCforcetheprobabilitydistributiontobezero incertainsubsetsofS.Inthatcaseourtheoremdoesn'tapply,butonecanworkaroundthisbyshrinkingthe setS.
Seealso
Exponentialfamily Gibbsmeasure Partitionfunction(mathematics)
Notes
1. ^Williams,D.(2001)WeighingtheOddsCambridgeUPISBN052100618x(pages197199) 2. ^Bernardo,J.M.,Smith,A.F.M.(2000)BayesianTheory'.'Wiley.ISBN047149464x(pages209,366) 3. ^O'Hagan,A.(1994)Kendall'sAdvancedTheoryofstatistics,Vol2B,BayesianInference,EdwardArnold. ISBN0340529229(Section5.40) 4. ^Park,SungY.Bera,AnilK.(2009)."Maximumentropyautoregressiveconditionalheteroskedasticitymodel" (http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper masterdownload%5C2009519932327055475115776.pdf).JournalofEconometrics(Elsevier):219230. http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper masterdownload%5C2009519932327055475115776.pdf.Retrieved20110602. 5. ^abJammalamadaka,S.RaoSenGupta,A.(2001).Topicsincircularstatistics (http://books.google.com/books? id=sKqWMGqQXQkC&printsec=frontcover&dq=Jammalamadaka+Topics+in+circular&hl=en&ei=iJ3QTe77NK L00gGdyqHoDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CDcQ6AEwAA#v=onepage&q&f=false). NewJersey:WorldScientific.ISBN9810237782.http://books.google.com/books? id=sKqWMGqQXQkC&printsec=frontcover&dq=Jammalamadaka+Topics+in+circular&hl=en&ei=iJ3QTe77NK L00gGdyqHoDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CDcQ6AEwAA#v=onepage&q&f=false. Retrieved20110515.
References
T.M.CoverandJ.A.Thomas,ElementsofInformationTheory,1991.Chapter11.
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
5/6
4/1/12
I.J.Taneja,GeneralizedInformationMeasuresandTheirApplications (http://www.mtm.ufsc.br/~taneja/book/book.html)2001.Chapter1 (http://www.mtm.ufsc.br/~taneja/book/node14.html) Retrievedfrom"http://en.wikipedia.org/w/index.php? title=Maximum_entropy_probability_distribution&oldid=481601486" Categories: Entropyandinformation Continuousdistributions Discretedistributions Typesofprobabilitydistributions
Particlestatistics
en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz
6/6