This article was downloaded by: [Universitaetsbibliothek Dortmund] On: 15 July 2013, At: 09:07 Publisher: Taylor &

Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Statistics: A Journal of Theoretical and Applied Statistics

Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/gsta20

The Hadamard Product and Some of its Applications in Statistics

H. Neudecker , S. Liu & W. Polasek
a a a a

University of Amsterdam and University of Basel, Published online: 27 Jun 2007.

To cite this article: H. Neudecker , S. Liu & W. Polasek (1995) The Hadamard Product and Some of its Applications in Statistics, Statistics: A Journal of Theoretical and Applied Statistics, 26:4, 365-373, DOI: 10.1080/02331889508802503 To link to this article: http://dx.doi.org/10.1080/02331889508802503

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Statistics 26 ( 1995) 365-373 Reprints available directly from the publisher Photocopying permitted by license only

d:

1995 OPA (Overseas Publishers Association) Amsterdam B.V. Published under license by Gordon and Breach Science Publishers SA Printed in Malaysia

THE ZIADAMARD PRODLTCT AND SOME OF ITS APPLICATIONS IN STATISTICS

H. NEUDECKER, S. LIU and W. POLASEK
Unicersiry of Amsterdam and University o f Basel
Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013
f Receiued 3 September 1993: in,final,form29 August 1994 )

Summary. Algehraic properties of the Hadamard product are used to establish some statistical properties.
AMS 1991 subject classifications: 15A69; 62H99. Key words: Kronecker and Hadamard products. selection matrlx. random vector, expectation, variance.

1. INTRODUCTION Some results on Kronecker and Hadamard products are collected in Rao and Kleffe (1988) and Magnus and Neudecker (1991). Early uses of the Hadamard product in statistics can be traced back to e.g. Styan (1973)and Neudecker (1975, 1981).Later on the Hadamard product was applied to the estimation of heteroscedastic linear regression models by Amemiya (1985). Some new applications on asymptotic distributions and image factor analysis were given recently by Kollo and Neudecker (1993)and Neudecker (l993), respectively. Algebraic properties and applications were extensively reviewed by Horn (1990). Some statistical properties of the Kronecker product of random vectors were reported in Magnus and Neudecker (1979) and Neudecker and Wansbeek (1983). It is thought to be relevant and useful to study statistical properties of Hadamard products of vectors. In this paper we shall examine this interesting field of research. In Section 2 new results and new proofs for known results are presented. The expectations, variance matrices, density functions of Hadamard products of random vectors, and the moment-generating function under normality of the Hadamard product x .x are given afterwards.

2. SOME ALGEBRAIC DEFINITIONS. PROPERTIES AND RELATIONSHIPS

Let "vec" indicate the vectorization operator, "0" and "." indicate the Kronecker and Hadamard products respectively. We know that A 0B = {uijB ) and A . B = { a i jhij), where A = { a i j ) and B = { b i j ) . The two products are intimately connected. Three well-known Kronecker properties are (See Rao and Kleffe (1988) or Magnus and

366

H N E U D E C K E R c,t i d

Neudecker (199 1)): (vecA)' vec B = tr ,4' B,

(1

vec uh' = h @ u.

(3)

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

Let furthers: = (1,. . ., 1)' be the p-dimensional summation column vector, A,: = I,- A be the diagonal matrix with diagonal elements aiiof matrix A, A(h) be the diagonal matrix such that A(h)s = h. Let ei be the ith unit column vector of dimension p, E,,:= eiei, i = 1 , . . . ,p. Let K,, (or K) indicate the p2 x p2 commutation matrix which plays an important role in the context involving vecs and the Kronecker product, i.e. Kpp= Cr,= (E,,,@Elj). Let J, indicate the pZ x p selection matrix that links the Hadamard and Kronecker products. It is interesting to mention two seemingly different but identical definitions of J,, viz J,:
=

C (c,8 Eii)= ( E l l . .. E,,,)';

i= 1

J,:=

C (el @ vec E,,) = (vec E l l . . . vec E,,).

i= 1

For (4)see Rao and Kleffe ( 1 988), for (5) see Kollo and Neudecker ( I 993) or Neudecker (1983,1993). I1 is not difficult to prove that the two definitions are identical. We notice by using h' @ a = u @ h' = ah' and (3) that

hence
P
P

(vecE l l . . .vec E,,)

=
i=1

(e;8 vec E,,) =

i= 1

(ei@ E,,) = (E, ,. . . E,,)'.

We now give some properties.

PI. (i) u = JivecA; (ii) vec A, = J,u; (iii) Ji Jp= I , ; (iv) K,, J , = J,; L = K,; (v) J, J (vi) .I, Ji vec A, = vec A,,
where a: = Ads, the p x p matrix K, is the diugonul matrix derivedfrom the commutation matrix K .

THF. HADAMARD PRODlJCT

367

(See also Rao and Kleffe ( I 988) for (i) and (ii).See also Neudecker (1983)for (i) and (iii). See Magnus (1988)for jiii) and (vi). See Kollo and Neudecker (1993) for (iv) and (v).) (vecE,,)'vecA = trE,!A /,a
= = r ( Ar = u,;, by

using (1);

x
P

aiivecEii= vec

x
P

uii Eii = vec A,, using ( 5 ) and thc definilivn of A,;

i=l

i- 1

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

l;l.Ipr = .IAvecA(x)= x,for any vector x, by using (i) and (ii); alternatively, 1,; J p= J,:(1 O I ) J p = 1.1 = I. using P2 (to follow); Using - (5); Using (4); .Ip.I; vec A, = .Ira = vec A,. This establishes the result.

.
Y

P2. J b ( A @ R ) .Iy= A . l3,ji)r uny p x q mufricw A and B. (See also Browne (1974),Pukelsheim (1977)and Kollo and Aeudecker (1993)for p

= q.)

2 a , jbij Eij = A.B, using (4).

(=I j=1

.
1(EiiTe,@EiiAEii)
i,j= 1
P

P3. J, A J ( r @ Al J, = c r @ Al J,.for any diauonal matrices r a n d A of' order p. (See also Kollo and Neudecker (1993) and Magnus (1988).)
P

Proof

J, J b ( r @ A) J,

=
r = l

( E , ,@ Ei,)(rC3A)

j= 1

x
P

(ejOEij)=

x
P
r=l

yilbi(ei @ I E,,) = (I-@ A) J,, using (4).

P4. rlAP2.AlBA, = PIAl(A.B)r,A2, ,for any p x q matrices A and B, and compatible diugonal mutricrs T I , T2, A, and A,. (See also Styan (1973)for PI = A, = I and Theorem 3.8 (c) in Magnus and Neudecker (1991) for r, = A2 and r, = A l =I.)
Proyf Consider p x q matrices A and B. Then T1AT2.A,BA2 = J b ( r l A T , @ A l B A 2 ) J q = J ~ ( r , @ A l ) ( A @ B ) ( ~ 2 @ A 2J), J ( r, l= @Al) JpJ;(A@ B) J, J;(T2@A2)J, = ( r l . A l ) ( A . B ) ( r 2 . A 2 = ) r,Al(A.B)r2A,, by virtue of P2 and P3.

P5. (A@ B).(C@D) = (A.C)@(B.D). (See also Rao and Kleffe ( l988).)
Proof (A@ B ) . ( C @D) = (a,,B)~{c,,D) = (a,,c,,B.D) = (A.C)@(B.D).

368

H. N E U D E C K E R ct

cil

P6. (i) (A @ B), = A, @ B,,for arhitrury square matrices A and B. (ii) ( A @ B),s = LI @ h, bvhere u: = Ads trnd h: = B,s ,f& urhitrurj, squure murriccs A und B.
Proof: (i) By P 5 , ( A @ B ) , , - ( I @ I ) . ( A @ B ) - ( I . A ) @ ( I . B ) = A , @ B , . (ii) Let the vector s ' " ' be the m-dimensional summation column vector. Consider m x m matrix A and n x n matrix B. Then

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

Property P7 is related to Pl(i)and P2:

P7. (i) u.h = J;(u @ h), ,/br any p x 1 wetors u und h; (ii) at.'.hdr = (u.h)(c.d)'= ild'.h~,'.
Proof: (i) follow5 from P2 with y
= 1 after substiiuting u for A and 1) for B; (ii) follows from PS, ac1.hd'= (a@c').(h@d') = (u-h)@(c'.d') = (a-h)(c-d)' = (a-h)(d.c)' = ud'.bcf: alternatively. from P2 and (i) by substituting ac' for A and bd' for B.

P8. J ; ( A @ c ) = Jb(c @ A) = A(c.)A,/br uny p x y mutrix A und p x 1 vector c.. (See Kollo and Neudecker (1993) for A =I.)

P9. (i) af(h-c) = hl(c.a)= cf(u.h)= u'B,c = h'C,u = c'A,h = Cr= u, h,c,, where (I:= Ads and A,h = a.h yenerically. (ii) (A1(BC)),= (Br(C.A)),= (C'(A -B)), = ((A.B)'C),, (iii) t r A 1 ( B C ) = trB1(CA)= trC1(A.B) = tr(A.B)'C.
Proof (i) Trivial. (ii)and (iii) follow from (i) by noting ej A'(B-C)ei= u'(h.c), where a: = A etc.

An application of P9(ii) is the following PIO. It arises in the context of Varimax, see Magnus and Neudecker (1991, p 375).

P10. (B'Q), = (C'MC),, where Q = B.MC, C = B.B, M is unjl symmetric matri.~.

Proof: [B1(B.MC)],
= [(MC)'(B B)], = [C'MC],,

using P9 (ii).

P11. (i) (AA B'),s = (A. B)/, where A is u diagonal mutriv and / = As. (ii) c = (S.S ) / , \t,here C = S A S ' is the spectrul decomposition of (symmetric) C, c = C,s. (See also Rao and Kleffe (1988) for (i). See Horn (1990) for (ii).)

T H E HADAMARD PRODUCT

Proof: (i) By virtue of Pl(i). (2), Pl(ii) and P2, (AABf),s = J ' vecAABf = J ' I B O A ) vecA = J'IB@A!JAs =!B.AI/ =IA.B)/. (ii) By using A = B = S in (i). c = C,s = (SAS1),s= (S.S)/.

369

Pi2. i i j i i . B ~ O / / i i , B > O , o r A,BsO. 2 ( A ' - B' j f A ( A - B),jiir cinj; p x q iriciiricvs A ciild B mu' ji x p (ii) (AIPA)-(B'ABj diugoriul mutrices T, A 2 0. (See Bellman (1970) and Horn (1990)for A, B 2 0 in (i). See Amemiya (1985. p. 205) for A'A fi - fi - B' = B and 1"= A in (ii).)
Prooj: (i) follows from P2, as A @ B 2 0. (ii) (A'fA).(B'AB)= J;[(A'PA)@(B'A B)]J,= J;[(A'@ B')(f@A)(A@B)]J,; (A'.B')PA(A.B)= [J;(A'@B') J,][J;(POA) J,][JJ,(A @ B) J,]. Note that .I,.l;(P@ A) .IpJ;, = J,, J;(PO A) = (POA) J,J,. Hence r @ A - J,Jk(r@~l)J,Jb=(T@11) - .I, Jb(P@A)-(T@A)J,Jb+ J,Jb (P@A)J,Jb ( I - J P J ) ( f @ ) ( IJ P J ; ) 2 0 , a s P , A 2 0 . Thus we estdblish the I-esult.

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

WP shall now look mtn snme stat~rt~cal prnpertief of Hrrdamard prnd~~ctq of random vectors Some algebraic properties glven before are golng to be used

3. THE EXPECTED VALUES O F RANDOM HADAMARD PRODUCTS

We prove the following result:

Lemma I . The euprcted cu1ur.s of the rundoin Hudur~iurdproducts x . x und x.y are
(i) E ( s . x ) = c u + p;p,; (ii) E(.uy) = c + p, .p2,
where ti,: = E(x), p,:= E(j),R:= L)(x), V = E(x - p,)(y p , ) ' , to:= R,s and o:= V,s.
= JrE(.u@x)= ~ ' ( v e c R +p, 8 ~ 1 , E(.ux)= E{J1(.u@x)} Prod. (i) We have = JrvecR + J r ( p I @ p l )= to p, . p i , by virtue of P7(i), (31, Plii) and D(x) = E(sxf) - E(x)E(xl). (ii) Similarly.

Lemma I(i) can, of course, also be obtained in a more pedestrian way, as E ( x ~= ) D(ui)+ p f = (oii+ pz, where toii is the i i t h element of R.

Lemma 2. Lrt x und y he indrprnderit, E(x) = p,, E(y) = p,, E(xxl)= V, and E(yyl)= V,, then
(i) E(xy'-xy') = u,t,;; (ii) E(xy'y.xf)= V , . V,; (iii) E(x.x.y.y)= o;u,. (iv) E(.Y@ x).(y@ y) = vec V, . vec V'.

3 70

H. NEUDECKER er al.

Proof: (i) By using P7(ii) and Lemma l(i), E(xy'.xyl) = E(x.x)(y.y)' = E(x.x) E(y.y)' = (o, p,-pl)(w2 p2.p2)'= v , v;, as Vi = 4 + pip; hence vi = wi + pi.pi ( i = 1,2), where D(x) = R, and D(y) = R2; (ii) By using P7(ii), E(xyr.yx') = E(xx'.yyf)= V, V2; (iii) By Lemma l(i), E(x..u.y.y) = (Ex.x).(Ep.y)= v1.o2; (iv) E(x @ x).(y@ y) = vec Vl . vec V2, as E(x @ x) = E vecxx' = vec Ifl, using (3).

In Section 4 we shall derive the variances of x.x and x.y.

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

4. THE VARIANCES O F RANDOM HADAMARD PRODUCTS

We study the general case first.

Lemma 3. Let p: = E(x),R: = D(x) and o : = R,s, then

Proof: D(x.x)= E(x.x)(x.x)' - E(x.x) E(x.x)'

= E(xxl.xx')-(o

+p.p) (o+ p.p)', by using P7(ii) and Lemma l(i).

Lemma 4. Let x and y be independent, E (x) = pl, E(y) = p2, E(xxl)= Vl and E(yyf)= V2, then D(x.y)= V1.V2-plp;.p2p;.
Proof.
D(x.y) = E(x.y)(x-y)'- E(x.y)E(x.y)'= E(xx1.yy')- (p,-p,)(&.p;)
= Vl .V2 - pl pi .,u2p;, by using P7(ii) and Lemma l(ii).

We now consider the normal case.

Lemma 5. The variance of the Hadamard product xsx, where x (Q + 2w').

Np(p,R) is 2 R

Proof: The basic property to be applied is given in Magnus and Neudecker (1979):

where K,, is the appropriate commutation matrix.

THE HADAMARD PRODUCT

It follows that D(x.x) = JJI,,

+KpP)(R O R + f2 @ ,up' + p p 1 0R )J p

We used the properties Pl(iv) and P2, keeping in mind the commutativity of the Hadamard product. W
Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

Lemma 6. Let x and y be jointly normally distributed with E ( x )= pl, E(y)= p,, E(xx') = V,, E(yyr) = V2,E ( x y f ) = V12and E ( y x f ) = V2,, then

Proof: Using P2, Pl(iv) and Theorem 4.3(ii) in Magnus and Neudecker (1979), viz D ( x O . ) = 1/,0 i72 + f ( V 1 2 @ V 2 1 ) - 2 ~ l 8 ~~ ', 1 ~ ; . After having presented some expectations and variances generally and under normality, we shall especially look into the density functions of x.x.

5. THE DENSITY FUNCTIONS O F THE HADAMARD PRODUCT x.x

A straightforward result is Lemma 7. Let the density o f x be f(x). Then the density of y = x.x is

where A(x),A(y) and A'I2(y) are diagonal matrices such that A(x)s = x, A(y)s= y and > 0 respectively, and xk(y)is the k-th branched inverse transformationfrom y to x: A112(y) x 1 = A1I2(y)s1,. ..,x~~= A1I2(y) s ~where ~ , sk (k = 1,. ..,2") are p x 1 vectors with 1 and - 1 arranged lexicographically such that s, = (1,. . .,1)', . . . ,s2, = (- 1,. . . , - 1)'. Proof: Clearly dy = d(x.x)= 2A(x)dx, where d(.) stands for the differential. Hence axlay'= 1/2AP1(x). As y = x - x has 2P branched inverse functions xk =A'12(y)sk and corresponding Jacobians Jk = dxk/i?yf and IJkI = 2-PIA-1(x)l= 2-PIA-1'2(y)l, k = 1, ...,2', so

This establishes the lemma.

372

H. NEUDFCKER er ul.

A straightforward specialization to normality is

Lemma 8. If .u hus multiuuriute normul density with p = 0 and R = [Id, vlr (271) ""exp( - I j2.~'Q; Is), then J = x.x hu.5 den.>ity Junction (271) ' '"IA cxp( !/2 ;,'<i;), .,i,!y7c < i = ; Qd5.
-

Proof: By Lemma 7 and noting that u ; Q i l u,

= y'to,

1,. . . .2".
Y.Y

1' 2 (s)l

The next step is to find the moment-generating function of

under normality

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

6. T H E MOMENT-GENERATING FUNCTION O F T H E HADAMARD P R O D U C T x.x UNDER NORMALITY

The ~i~oment-generati~lg function of .Y-x is defined as rn(t) = Eexp t l ( s . x ) , t being a vector variable. Its domain of definition will implicitly be given below. We shall prove the following result:

Lemma 9. The moment-ycwcratinyrain function u f ' s . is ~

where W ' : = Q

'-2A(t)und W>O.

Proof: We write t'(x.x) = u'Alt).u and use the result (due to Heijmans. see Neudecker
( 1990)):

where W ': = R ' - 2 ~ ~ 4 such . 7 that W > O and A' = -4. Replacing TA by A(t) in (9)and subsequently transforming prA(t)pand p'A(t) WA(t)p yields the result. The detailed algebra goes as follows. pfA(t)p= fl(p.p), by virtue of P9(i); and

p ' )A(t) = t' J;(W@ppl) J,t plA(t)w A ( t ) p = trA(t)w'A(t) W = j v e ~ A ( t ) ) - ' ( W @ ~vec = tr(W.pp')t, by virtue of ( I ) , (2), Pl(ii) and P2.

[ I ] Amemiya, 7. ( 1985). Arlwnixd Eco~~omerric,.s. Basil Blackwell, Oxford. 121 Bellman, R. (1970). Introduction t o M u r r i ~ Anulysis ( 2 ~ ed). d McGraw-Hill. New York. 131 Browne. M. W . (1974). Generalired least squares estimators in the analysis of covariance structures. S. Afr. Srurrsr. J . 8. 1 24. [4] Horn, K. G . (1990). The Hadamard product. Prot. Sjtiip. Appl. Murh. 40. $7- 169 1-51 Kollo, T. and Neudecker, H. (1993).Asymptotics ofeigenvalues and unit-lengthe~genvectors of sample ~2 8 3 3 0 0 . variance and correlation matrices. J. Mulrr. A J 47,

THE HADAMARD PRODUCT

373

Downloaded by [Universitaetsbibliothek Dortmund] at 09:07 15 July 2013

[6] Magnus, J. R. [1988]. Linear Structures. Griffin's Statistical Monographs, 42. Charles Griffin and Company Ltd, London and Oxford University Press. New York. [7] Magnus, J. R. and Neudecker. H. (1979).The commutation matrix: Some properties and applications. Ann. Stutist. 7. 381-394. C8-j Magnus. J. R.. and Neudecker. H. (1991).M u t r i . ~ Diferential Calculus with Appliccirions in Statistics and Econometrics (2nd ed). Wiley, Chichester. [9] Neudecker. H. (1975).A derivation of the Hessian of the (concentrated)iikelihood function of the factor . 152- 156. model employing the Schur product. Br. J. Math. Statist. P s ~ c h o l28. [lo] Neudecker. H. (1981). On the matrix formulation of Kaiser's varimax criterion. Psjchometrika 46, 343 345 [l 11 Neudccker, H. (1983).On Jacobians oftransformations with skew-symmetric.strictly (lower)triangular or diagonal matrix arguments. Linear Multi. Alg. 14.271-295. [I21 Neudecker, H. (1990).The variance matrix of a matrix quadratic form under normality assumptions. A derivation based on its moment-generating function. Statistics 21,455-459. [13] Neudecker, H. (1993). The Hessian matrixjor Imuqe Factor Analysis. in: Steyer, R., Wender, K. F. and Widaman, K. F. (eds.) Psychometric Methodology. Gustav Fischer, Stuttgart. 361-365. [14] Neudecker. H. and Wansbeek. T. (1983). Some results on commutation matrices, with statistical applications. Can. J. Statist. 11,221-231. [I51 Pukelsheim, F. (1977).On Hsu's model in regression analysis. Statistics 8. 323-331. [16] Rao, C. R. and Kleffe, J. (1988).Estimation vf Variance Components and Applications. North-Holland. Amsterdam. [17] Styan. G. P. H. (1973). Hadamard products and multivariate statistical analysis. Linear Alq. Appl. 6, 217-240. H. Neudecker. S. Liu lnstitute of Actuarial Science and Econometrics. University of Amsterdam Roetersstraat 11 NL-1018 WB Amsterdam W. Polasck lnstitute of Statistics and Econometrics University of Basel Petersgraben 51 CH-4051 Basel