Analysis of 1-D Nested Lattice Quantization and Slepian-Wolf Coding for Wyner-Ziv Coding of i.i.d.

Sources
Lori A. Dalton ELEN 663: Data Compression Dept. of Electrical Engineering Texas A&M University Due: May 9, 2003

Abstract Entropy coded scaler quantization (ECSQ) performs 1.53 dB from optimal lossy compression at high data rates. In this paper, we generalize this result for distributed source coding with side information. 1-D nested scaler quantization with Slepian-Wolf coding is compared to Wyner-Ziv coding of i.i.d. Gaussian sources at high data rates. We show that nested scaler quantization with Slepian-Wolf coding performs 1.53 dB from Wyner-Ziv at high data rates.

Introduction

It has been shown that lossless compression of a scaler quantized source, or entropy coded scaler quantization, performs 1.53 dB from optimal lossy compression for high data rates. In this report, we nd analogous results for distributed source coding of 2 correlated Gaussian sources. Fig. 1 illustrates nested scaler quantization with Slepian-Wolf coding (NSQ-SW). One source is assumed to be imperfectly known to the receiver, while the second source serves as perfectly known side information. Both sources are processed independently from each other at the encoder, but are decoded jointly. The rst source undergoes 1-D nested lattice quantization (a generalization of scaler quantization) and lossless Slepian-Wolf coding. We compare this performance to Wyner-Ziv coding, which is illustrated in Fig. 2. This paper claims that NSQ-SW coding performs 1.53dB from Wyner-Ziv at high data rates.

Figure 1: Distributed source coding with nested scaler quantization and Slepian-Wolf (NSQ-SW) coding.

Figure 2: Distributed source coding with Wyner-Ziv. To motivate this problem, we rst consider the case where both sources are uncorrelated. Thus, the side information at the receiver does not help to decode the rst source. This degenerate case reduces to the classical problem of comparing ECSQ with optimal lossy compression. ECSQ performs 1.53 dB worse (see Section 2.4), indicating that at least in the uncorrelated source case, NSQ-SW performs 1.53 dB worse than Wyner-Ziv coding. 2

'

  $ %
      

    

 !#" $ 

&

     

Preliminaries

In this section, we review concepts that will be used throughout this paper. The source model is described in Section 2.1. In Sections 2.2 and 2.3, we review SlepianWolf coding and Wyner-Ziv coding, respectively. Finally, in Sections 2.4 and 2.5, we cover entropy coded scaler quantization and nested scaler quantization, respectively.

2.1

Source Model

Two i.i.d. sources, X and Y , are jointly Gaussian with correlation coecient . We will consider Y to be known perfectly at the decoder as side information, while X is compressed with loss. To model the correlation between X and Y , we use Y = X +Z, where X and Z are independent Gaussian random variables with
2 X N (0, X ), 2 Z N (0, Z ), 2 Y N (0, Y ), 2 2 2 where Y = X + Z , 2 = 1
2 Z 2 , Y

(1) (2) (3)

2 2 X Z 2 + 2 . X Z

2 2 and X|Y = X (1 2 ) =

We denote the

PDF of X by fX (x).

2.2

Slepian-Wolf Coding

The setup of Slepian-Wolf coding is shown in Fig. 3. The idea is to optimally compress X independently from Y without loss, knowing that X and Y will be decoded jointly. Surprisingly, [2] shows that we can achieve the same rate encoding both sources separately as we can encoding them jointly. That is, the achievable rates using Slepian-Wolf coding and the setup in Fig. 4 are equivalent. The achievable rate region is given by (4)-(6) and shown in Fig. 5. R1 H(X | Y ), R2 H(Y | X), R = R1 + R2 H(X, Y ). (4) (5) (6)

Figure 5: Achievable rate region with Slepian-Wolf or joint source coding.

                               

   ( &! '  ! !  %$#"   

 

 

Figure 4: Lossless joint source coding.

    

Figure 3: Slepian-Wolf coding.

    

   

                      

 

   

   

2.3

Wyner-Ziv Coding

Wyner-Ziv coding [3] generalizes Slepian-Wolf coding, allowing now for lossy compression of X. In general, the Wyner-Ziv coding scheme in Fig. 2 loses rate compared to the joint source encoding scheme shown in Fig. 6. However, when X and Y are zero mean, stationary, and memoryless Gaussian sources, the performance in both cases are identical for the MSE distortion metric. In this case, achievable rates for the source X are lower bounded by RW Z (D) = where log+ x = max{log x, 0}. (8) 1 2 (1 2 ) , log+ X 2 D (7)

Figure 6: Lossy joint source coding.

2.4

Entropy Coded Scaler Quantization

We now consider performance of ECSQ for high rates. The setup for ECSQ is shown in Fig. 7. We nd the high rate performance of ECSQ by following the approach in [4].

Figure 7: Entropy coded scaler quantization (ECSQ). For large rates, it has been shown that the optimal scaler quantizer for ECSQ is uniform for all smooth PDFs [1]. We use the quantizer shown in Fig. 8, with the following properties: 5

& 0) ( '

$ %  #

      

 $ #   " !

    

&

     

1. Quantizer bins have width q. 2. Quantization codeword for bin i is xi = iq, where i = 0, 1, 2, . . .. 3. The thresholds between bins are the midpoints of the quantization levels, namely ti =
xi1 +xi b b 2

= (i 1 )q. 2

4. Quantization intervals are given by Ri = [ti , ti+1 ).

Figure 8: Uniform Quantization for ECSQ. For small q, we compute the MSE distortion of this quantizer. D = E d(X, X)

=
i= Ri

fX (x)d(x, xi )dx
1 (i+ 2 )q

= = = =
i=

(i 1 )q 2

fX (x) (x iq)2 dx
(i+ 1 )q 2
1 (i 2 )q

fX (iq)
i=

fX (iq)
i= 2

q3 12

q fX (iq) q 12 i=

2 q = 12 q2 = . 12

fX (x)dx

  

   

(x iq)2 dx

(9)

Compressing the quantized source losslessly does not change distortion. Let pi be the probability that X is in bin i. Then for high rates, pi fX (iq) q. The = corresponding entropy is given by

H(X) = = = =
i= i=

pi log2 pi fX (iq) q log2 (fX (iq) q)

i=

fX (iq) q log2 fX (iq)

i=

fX (iq) q log2 q

fX (x) log2 fX (x)dx log2 q

fX (x)dx

= h(X) log2 q. With ecient entropy coding, R = H(X). Then from (10), q 2h(X) R. = Combining (11) with (9), the rate-distortion function for ECSQ is given by DECSQ (R) = 1 2h(X) 2R 2 2 . 12

(10)

(11)

(12)

Fig. 9 shows the setup for optimal lossy compression of source X. Performance of the scheme in Fig. 9 is bounded by [5]: DL (R) = 1 2h(X) 2R 2 2 D(R) 2 22R . 2e (13)

Moreover, D(R) DL (R) for high rates. Thus, ECSQ performs e/6, or 1.53 dB = 2 from the optimal lossy encoder. Note for Gaussian sources, DECSQ (R) e 22R =
6

and D(R) = 2 22R .

Figure 9: Lossy encoding of a source, X.

      

   

2.5

Nested Scaler Quantization

Nested scaler quantization is similar to scaler quantization, except now each bin consists of sub-bins uniformly spaced over the range of the source, X. This quantizer is illustrated in Fig. 10 with the following properties: 1. N is the number of bins. Dene NL and NU to be the index of the lowest and highest quantization levels, respectively. Note N = NU + NL + 1. 2. Sub-bins within each bin have width q. Bin-groups have width N q. 3. The quantization codeword for bin i is xi = iq, i = NL , . . . , NU . 4. The thresholds between sub-bins are ti = (i 1 )q, i = 0, 1, 2, . . .. 2 5. Quantization intervals are given by Ri = jZ [ti+jN , ti+jN +1 ), where i is the bin index and j is the bin-group index.

' 2 1 0$0)          54 2 3                   ( % " % ' % &$% $ #"!  

Figure 10: Uniform nested scaler quantization. 8

High Rate Performance of NSQ-SW Coding

In this section, we analyze nested scaler quantization with Slepian-Wolf coding. We rst nd the distortion between X and the estimated source output, denoted by X. We use the mean-square error (MSE) measure of distortion, d(X, Y ) = (X Y )2 . Since Slepian-Wolf coding is lossless, it does not aect distortion. We then nd the achievable rate of the Slepian-Wolf encoder. From these results, we present a ratedistortion bound for NSQ-SW coding. For high rate analysis, the bin-group width, N q, is xed to a constant and q 0.

3.1

Distortion From Nested Scaler Quantization

The joint decoder sees the side information, Y = y, and the nested scaler quantized source, X = xi = iq. Thus the receiver knows the bin index, i, and uses Y to estimate the bin-group index, j, to isolate the specic sub-bin that X is in. We denote the estimated bin-group by j, which corresponds to the sub-bin within bin i that is closest to Y . The nal estimate of X at the receiver is X = xi (y) = (i + N j(y iq))q, where j(z) = Suppose x = (i + N j)q + , | | < j(y iq) = Thus, if
q 2

z 1 + . Nq 2

(14)

and y = x + z. The receiver computes +z 1 + . Nq 2

(i + N j)q + + z iq 1 + =j+ Nq 2 +z >

Nq , 2

+ z < N q or 2
Nq 2

we will choose j incorrectly. i.e., if Y lies

further than

from the scaler quantized version of X, we will choose the incorrect

bin-group. For brevity, we will sometimes denote j(y iq) with simply j.

For a given realization of the side information, Y = y, the distortion is D(y) = E d(X, X)|Y = y)
NU

=
i=NL NU

E d(X, X)|Y = y, X Ri ) Pr(X Ri ) fX|Y (x|y)(x xi (y))2 dx

i=NL NU xRi ti+N j+1 ti+N j ti+N j+1

= =
i=NL j=

fX|Y (x|y)(x xi (y))2 dx (x xi (y))2 dx. (15)

= Note,

NU

fX|Y ((i + N j)q|y)

i=NL j= ti+N j

ti+N j+1 ti+N j

(x xi (y))2 dx = =

q3 + (xi (y) (i + N j)q)2 q 12

q3 + ((i + N j)q (i + N j)q)2 q 12 q3 + (j j)2 (N q)2 q. = 12

(16)

Therefore, at high rates the distortion is given by D(y) =

NU

fX|Y ((i + N j)q|y)

i=NL j= N

q3 + (j j)2 (N q)2 q 12

U q2 = 12 i=N

fX|Y ((i + N j)q|y)q

j= NU

+ (N q)2
i=NL j=

fX|Y ((i + N j)q|y)(j j)2 q.

(17)

10

Also note,
NU

fX|Y ((i + N j)q|y)q = =

NU q

fX|Y (x + N qj|y)dx
j= NL q (N j+NU )q

i=NL j=

fX|Y (x|y)dx
j= (N jNL )q

fX|Y (x|y)dx (18)

= 1. Furthermore, since j(z + N qk) = j(z) k, and plugging (17) into (18),
U q2 2 D(y) = + (N q) 12 i=N

fX|Y ((i + N j)q|y)(j j)2 q

j= NU q NL q

2 q + (N q)2 = 12 j= 2

fX|Y (x + N qj|y)(j(y x) j)2 dx

q + (N q)2 fX|Y (x|y)(j(y (x N qj)) j)2 dx 12 2 q 2 = + (N q) fX|Y (x|y)j(y x)2 dx 12 q2 = + (N q)2 EX|Y j(Y X)2 |Y = y . 12

(19)

The average distortion over all realizations of Y is D = EY [D(Y )] q2 + (N q)2 EX,Y j(Y X)2 12 q2 = + (N q)2 EZ j(Z)2 . 12 = second term is distortion due to choosing the wrong bin-group.

(20)

We may view the rst term in (20) as distortion due to scaler quantization. The

11

In addition, notice E j(Z)2 = =

j=

fZ (z)j(z)2 dz
N q(j+ 1 ) 2 N q(j 1 ) 2 N q(j+ 1 ) 2 N q(j 1 ) 2

fZ (z)j(z)2 dz fZ (z)j 2 dz fZ (z)dz j 1 2 Q Nq Z j+ 1 2 , (21)

=
j=

=
j=

N q(j+ 1 ) 2 N q(j 1 ) 2

=
j=

j2 Q

Nq Z

where Q(x) is the Q-function. (21) may be simplied: u(w) 1 j2 Q w j 2 w 2 j=

L

Q w j+ 1 2

1 2 1 2 (22)

= lim

j2 Q w j
j=L

Q w j+

= lim uL (w),
L

where, 1 uL (w) = j Q w j 2 j=L

2 L L L

j=L

j 2Q w j +
L+1

1 2 1 2

=
j=L

j 2Q w j

1 2 1 2

(j 1)2 Q w j
j=L+1

= L2 Q w L
L

Q w L+ 1 2

1 2 . (23)

(j 2 (j 1)2 )Q w j
j=L+1

12

For large L, the following approximation is accurate. uL (w) L2 + =

2 L

(2j 1)Q w j
j=L+1 0

1 2
L

1 =L + (2j 1)Q w j 2 j=L+1

L1 2

+
j=1

(2j 1)Q w j
L

1 2 1 2

1 =L + (2j 1) 1 Q w j + 2 j=0
L1

+
j=1 L1

(2j 1)Q w j (2j + 1)Q w j + 1 2

= L2 L2
j=0 L1

(2j 1)Q w j + 1 2 .

1 2

+
j=0

=2
j=0

(2j + 1)Q w j +
j=0 (2j

(24)

So

u(w) w2

=2

1 + 1)Q(w(j + 2 )) and, 2

E j(Z)

=2
j=0

(2j + 1)Q

Nq Z

j+

1 2

(25)

Finally, the average distortion is given by q2 D= + 2(N q)2 12

(2j + 1)Q
j=0

Nq Z

j+

1 2

(26)

This is similar to the distortion for ECSQ, with an additional bin-group error term. Note if Z = 0 (Y = X), then Z = 0, D =
q2 , 12

and there is no bin-group error.

Fig. 11 shows the rate-distortion curve for 1-D nested lattice quantization without Slepian-Wolf coding for various N q. Fig. 12 shows the optimal N q that minimizes distortion in (26) for xed N = 2R . It is interesting to see if the optimal N q without Slepian-Wolf coding converges to a constant as N . Note D= (N q)2 2R 2 + 2(N q)2 12

(2j + 1)Q
j=0

Nq Z
j=0 (2j

j+

1 2

.
Nq Z

(27) j+
1 2

If N q does converge, then D has a oor of 2(N q)2 converge to a constant. 13

+ 1)Q

as

N . This oor is minimized (eliminated) when N q , thus N q does not

10

10

10

10 DNSQ(R)

10

10

10

10

12

10

14

Nq = 5 Z Nq = 7.5 Z Nq = 10 Z Nq = 12.5 Z Nq = 15 Z 0 2 4 6 8 10 12 R = log2 N (bits/c.u.) 14 16 18 20

Figure 11: Rate-distortion curve for 1-D nested lattice quantization without Slepian2 Wolf coding, Z = 0.01.

11

10

Optimal Nq/

50

100 N = 2R

150

200

250

Figure 12: Optimal N q for 1-D nested lattice quantization without Slepian-Wolf coding. 14

3.2

Achievable Rate with Slepian-Wolf Coding

Once X is processed by a nested scaler quantizer, we encode X losslessly with minimum rate. Knowing that the decoder will also have side information Y , recall from Section 2.2 that this is a Slepian-Wolf problem, and that the minimum achievable rate for source X is R = H(X|Y ). For a given realization of the side info Y = y,
NU

H(X|Y = y) =
i=NL

pi (y) log2 pi (y),

(28)

where for high rate, pi (y) =

xRi

fX|Y (x|y)dx
ti+N j+1

= =
j= ti+N j

fX|Y (x|y)dx fX|Y ((i + N j)q|y)q,

j=

(29)

for i = NL , . . . , NU . Dene f (x) to be the N (0, 1) distribution. Then fX|Y (x|y) =

1 X|Y

xy X|Y

and pi (y) = q X|Y

f
j=

(i + N j)q y X|Y

(30)

Dene g(x) =

1 X|Y

j=

x+N qj X|Y

so pi (y) = g(iq y)q. Thus, entropy is ap-

proximately given by,

NU

H(X|Y = y) =
i=NL

g(iq y)q log2 [g(iq y)q]

NU q NL q NU q

= =

g(x y) log2 [g(x y)q] dx

NU q

NL q

g(x y) log2 g(x y)dx log2 q

g(x y)dx.
NL q

(31)

15

Furthermore,
NU q

g(x y)dx =
NL q

1 X|Y 1 X|Y 1 X|Y

NU q

NL q j= (N j+NU )q j= (N jNL )q

x + N qj y X|Y f xy X|Y

dx dx

= =

xy X|Y

dx (32)

= 1. Plugging (32) into (31), we have for high rates

NU q

H(X|Y = y) =
NL q

g(x y) log2 g(x y)dx log2 q.

(33)

Notice g(x N qk) = g(x) k Z, so g(x) is periodic with period N q. We simplify (33) further:
NU q

H(X|Y = y) =
NL q

g(x y) log2 g(x y)dx log2 q

NU q

= = =

1 X|Y 1 X|Y 1

NL q j= (N j+NU )q j= (N jNL )q

x + N qj y X|Y f xy X|Y

log2 g(x y)dx log2 q log2 g(x N qj y)dx log2 q

X|Y 1 = X|Y

xy X|Y xy X|Y

log2 g(x N qj y)dx log2 q log2 g(x y)dx log2 q (34)

f (x) log2 g(X|Y x)dx log2 q.

16

Notice H(X|Y = y) is independent of y! Averaging over all side info, Y , we have H(X|Y ) = H(X|Y = y), i.e.,

H(X|Y ) =

f (x) log2 g(X|Y x)dx log2 q f (x) log2 f (x) log2

j=

1 X|Y

f
j=

x+ N qj X|Y

N qj X|Y

dx log2 q (35)

x+

dx + log2 X|Y log2 q.

In general, H(X|Y ) must be computed numerically.

3.3

Summary of Main Results

For high rate and assuming ecient entropy coding (R = H(X|Y )), we have shown D= R=h where

Nq q2 2 + Z u , 12 Z Nq + log2 X|Y log2 q, X|Y

(36) (37)

u(x) = 2x2
j=0

(2j + 1)Q x j +

1 2

(38) (39)

h (v) =

f (x) log2
j=

f (x + vj) dx.

Combining D and R through q, we have for NSQ-SW coding: 1 2h DN SQSW (R) = 2 12


Nq X|Y

2 2 X|Y 22R + Z u

Nq Z

(40)

Recall for Wyner-Ziv coding of Gaussian sources, the rate-distortion bound is

2 given by (7), or DW Z (R) = X|Y 22R . When N q is large enough for u Nq Z 1 2 12

to be neg

ligible, this shows that nested scaler quantization performs 10 log10 dB from Wyner-Ziv coding. 17

2h

Nq X|Y

As N q , the 1-D nested lattice quantizer becomes a regular uniform scaler

q 1 quantizer. We have u( NZ ) = 0 and h ( N q ) = 2 log2 2e. This case yields X|Y

DSQSW (R) =

e 2 22R , 6 X|Y

(41)

which is 1.53 dB from Wyner-Ziv coding. This result is parallel to the performance dierence between ECSQ and optimal lossy encoding.
2 Fig. 13 and 14 show high rate distortion for NSQ-SW coding with X = 1 and 2 2 various values of N q for Z = 0.1 and Z = 0.01, respectively. Notice h (v) controls

the horizontal shift of the rate-distortion curve, while u(x) determines the error oor.

3.4

Choosing N q

Note as N q decreases, H(X|Y ) decreases and distortion increases. Thus we are interested in nding the optimal N q to trade o rate and distortion. First, to minimize distortion we must minimize u(x). A plot of u(x) is shown in Fig. 15. Notice that choosing N q 10Z results in an error oor corresponding to u(x) 105 . To control the horizontal shift of DN SQSW (R), we consider h (v), which is shown in Fig. 16. Notice h (v) is upper bounded by N q 5X|Y results in a rate very close to the maximum. Relatively row rates are achievable by using smaller bin groups, but the distortion oor starts to dominate for a xed N q as we increase rate. To nd the optimal N q, we use the lower convex hull of all rate-distortion curves for dierent N q. At a given rate, the optimal N q corresponds to the rate-distortion curve that lies on the lower convex hull at that rate. In general, for nite N q, high rate NSQ-SW actually performs slightly better than 1.53 dB from Wyner-Ziv. However, this eect becomes
2 2 negligible as we increase rate. Optimal N q for Z = 0.1 and Z = 0.01 are shown in 1 2

log2 2e, and choosing

Fig. 17 and Fig. 18, respectively. For N q 5X|Y , where h (v) h(Z) log2 Z , we can approximate the best = N q by choosing N q just large enough so that for the target rate, u negligible compared to
1 2h 2 12

Nq Z

Nq Z

is barely

22R .

18

10

10

10

10 DNSQSW(R)

10

10

10

10

12

10

14

Nq = 5 X|Y Nq = 7.5 X|Y Nq = 10 X|Y Nq = 12.5 X|Y Nq = 15 X|Y Nq 0 5 10 R (bits/c.u.) 15 20 25

10

16

2 2 Figure 13: High rate D(R) for NSQ-SW with X = 1 and Z = 0.1.

10

10

10

10 DNSQSW(R)

10

10

10

10

12

10

14

10

16

Nq = 5 X|Y Nq = 7.5 X|Y Nq = 10 X|Y Nq = 12.5 X|Y Nq = 15 X|Y Nq 0 5 10 R (bits/c.u.) 15 20 25

10

18

2 2 Figure 14: High rate D(R) for NSQ-SW with X = 1 and Z = 0.01.

19

10

10

10 u(x) 10

10

10

15

10

20

10 x = N q/Z

12

14

16

18

20

Figure 15: Choosing N q: u(x).

0 Entropy (bits/c.u.)

1 h(v) 1/2 log2 2e 2

5 v = N q / X|Y

10

Figure 16: Choosing N q: h (v).

20

22

21.5

21

Optimal Nq/X|Y

20.5

20

19.5

19

18.5

5 R (bits/c.u.)

10

2 2 Figure 17: Optimal N q for X = 1 and Z = 0.1.

21

20.5

20 Optimal Nq/X|Y

19.5

19

18.5

18

5 R (bits/c.u.)

10

2 2 Figure 18: Optimal N q for X = 1 and Z = 0.01.

21

Conclusion

We have shown that high rate NSQ-SW coding for Gaussian sources performs 1.53 dB from Wyner-Ziv coding, which is the same gap between ECSQ and optimal lossy compression. We also computed the rate-distortion function for NSQ-SW for high data rates and xed bin-group widths. An interesting problem to pursue beyond this paper is the performance of 2-D or higher dimensional nested lattice quantization with Slepian-Wolf coding.

References
[1] H. Gish and J. N. Pierce, Asymptotically ecient quantizing, IEEE Trans. Inform. Theory, vol. 14, pp. 676683, Sep. 1968. [2] D. Slepian and J. K. Wolf, Noiseless coding of correlated information sources, IEEE Trans. Inform. Theory, vol. 19, pp. 471480, July 1973. [3] A. Wyner and J. Ziv, The rate-distortion function for source coding with side information at the decoder, IEEE Trans. Inform. Theory, vol. 22, pp. 1-10, Jan. 1976. [4] D. S. Taubman and M. W. Marcellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice. Kluwer Academic Publishers, Norwell, Massachusetts, 2002. [5] T. M. Cover and J. A. Thomas, Elements of Imformation Theory. Wiley, New York, 1991.

22