WIRELESS COMMUNICATIONS AND MOBILE COMPUTING

Wirel. Commun. Mob. Comput. 2008; 8:9399

Published online 12 September 2006 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/wcm.437

Bit error rate for rectangular QAM with arbitrary

constellation mapping in Nakagami-m channels
Hong Xu1 , Mustapha Benjillali2 and Leszek Szczeci ski2 ,
1

TenXc Wireless Inc. Ottawa, Canada

Institut National de la Recherche Scientique INRS-EMT, 800 de la Gaucheti`ere O Suite 6900, Montreal,
H5A 1K6, Canada
2

Summary
We present closed-form expressions for the bit error rate (BER) of rectangular quadrature amplitude modulation
(QAM) in Nakagami-m fading channels. The presented formulas, which are valid for arbitrary bits-to-symbols
mapping, thus may be used when non-Gray mapping is employed, are particularly useful in the low signal-to-noise
ratio (SNR) range and/or for small values of the parameter m. The advantage of the proposed expressions over the
known bounding techniques is illustrated through numerical simulations. Copyright 2006 John Wiley & Sons, Ltd.

KEY WORDS: bit error rate; QAM; performance evaluation; Nakagami-m fading

1.

Introduction

Calculation of the bit error rate (BER) is one of the

fundamental problems in digital communications. Exact expressions for the BER of different modulation
schemes have previously been presented with the assumption of Gray mapping, for example, in [1,2,Ch.
2,3].
Although Gray mapping is very popular in communications systems, other mapping strategies are also
gaining popularity. For example, in [4] non-Gray mapping is used when signaling is non-uniform, and in
[5,6]it is shown to be beneficial in the context of
iterative (turbo) de-mapping.
The expression for the BER in QAM with arbitrary
mapping has been established in [7] for additive white

Gaussian noise (AWGN) channels. In our paper we extend the results to fading channels. The main motivation for this work is the fact that QAM is very popular
in communication systems and a good approximation
of the performance in the region of high BER (where
the bounding techniques fail) is important because the
strong error-correcting codes, for example, the turbo
codes [8], are providing satisfactory results in this very
range of the BER (the so-called waterfall region).
Evaluation of BER for arbitrary constellations and
fading channels was treated in [9]. However, the expressions shown therein are based on non-trivial geometric considerations, therefore, cannot be considered
closed-form.
In this paper, we develop closed-form formulas applying the results of [7] to the case of transmission over

*Correspondence to: Leszek Szczecinski, Institut National de la Recherche Scientifique INRS-EMT, 800 de la Gaucheti`ere O
Suite 6900, Montreal, H5A 1K6, Canada.

E-mail: leszek@emt.inrs.ca
Contract/grant sponsors: PROMPT, Quebec, and NSERC, Canada; contract/grant number: #249704-02.
Contract/grant sponsor: Axiocom Research Inc., Trois-Rivieres, Canada.
Copyright 2006 John Wiley & Sons, Ltd.


H. XU, M. BENJILLALI AND L. SZCZECINSKI

94

Nakagami-m channels. Such a fading model provides

a good fit to land-mobile, indoor-mobile multipath
propagation measurements, as well as to scintillating
ionospheric radio links [10]. In particular, for the
parameter m = 1 this model represents the Rayleigh
channel. We derive the expressions for the uncoded
BER in cases when the parameter m is half-integer, that
is, belongs to the set 0.5 N {0.5, 1, 1.5, 2, . . .}.
Such expressions can be used in the mapping design
[4,6], and as a calibration tool for verification of
numerical simulations.
This letter is organized as follows. The system
model, basic assumptions, and notations are introduced
in Section 2. The exact expressions for the BER are presented in Section 3, while the detailed mathematical
derivations are shown in the Appendix. Numerical examples are given in Section 4 to illustrate the relevance
of the proposed expressions when applied in the highBER range. The conclusions are drawn in Section 5.
2.

System model

Consider a system where the bits, denoted by y, are

mapped into symbols s via a memoryless and arbitrary
mapper {}. As a result, s(t) = {y(t)} A, where t is
the discrete time, y(t) = [y(Bt + 1), . . . , y(Bt + B)]
B is the modulating codeword of length B, B is the
set of all possible codewords
and A = {a1 , . . . , aM } is
1
M
2
the energy-normalized M
l=1 |al | = 1, zero-mean

M
B
l=1 al = 0 modulation constellation with M = 2 .
In the following, we consider rectangular M-ary
quadrature amplitude modulation
(QAM), that is,
A = AR {AI }, where = 1, and is the
Cartesian product. The sets AR = {aR,1 , . . . , aR,MR }
and AI = {aI,1 , . . . , aI,MI } contain, respectively, the
real and imaginary, monotonically increasing parts
of the symbols, where MR > 1, MI > 1, and M =
MR MI . We assume that AR and AI contain equidistant
elements, that is, aR,k+1 aR,k = aI,k+1 aI,k 2.
The channel output is given by r(t) = h(t)s(t) + (t),
where (t) is an AWGN with variance N0 , h(t) is a complex gain whose amplitude follows a Nakagami-m distribution, and which has an average energy E{|h(t)|2 },
so the instantaneous signal to noise ratio (SNR) (t) =
|h(t)|2 /N0 has the probability density function given
by [11,Ch. 2]
1  m m m1 m/
pm, () =

e
,
(1)
(m) 
where (m) is the gamma function and  = (E{|h
(t)|2 })/(N0 )represents the so-called average SNR.
Copyright 2006 John Wiley & Sons, Ltd.

Knowing the instantaneous channel state h(t) (i.e.,

the coherent detection is assumed), the detector makes
decision in favor of the codeword labeling the symbol
that is closest (in the sense of the Euclidean distance)
to the observation r(t), that is,
r(t) Zb = {r : |r h(t){b}|

y(t)
= b B if

< |r h(t){
b}|,

b = 
b},

(2)

where Zb is the decision region associated with the

symbol {b}.
3.

BER in Nakagami-m fading channels

For a given h(t), that is, knowing the instantaneous SNR

(t), the closed-form expression for the BER may be
calculated as [7]:
BER() =

 
1   dH (b, 
b)  
Q (R (b, 
b) ) 2
M bB 
B
bB
 =b
b


 
b) + ) 2
R (b)Q (R (b, 
 
 
b) ) 2
Q (I (b, 

 
b) + ) 2 ,
I (b)Q (I (b, 

(3)

where dH (b, 
b) is the Hamming distance between b
and 
b, R (x, v) = |R {x} R {v}|, I (x, v) =
|I {x} I {v}|, R {x} [{x}], I {x} [{x}],
[] and []
and imaginary parts respectively;
are real

Q(t) = 1/ 2 t exp( 2 /2)d, and R (b) and

I (b) indicate if the symbol {b} is at the constellation
border with respect to its real or/and imaginary parts,
which can be formally expressed as

0 if R {b} = aR,MR R {b} = aR,1

R (b) =
1
otherwise.
(4)
The form of I (b) is obtained changing the subindexing from ()R to ()I in Equation (4).
In order to calculate the average BER BER, we need
to take the expectation of BER() in (3) with respect
to , that is,

BER =

pm, ()BER()d.

(5)

Substituting Equation (3) in Equation (5) results in

a linear combination of terms having the following
Wirel. Commun. Mob. Comput. 2008; 8:9399

BIT ERROR RATE RECTANGULAR QAM

form:

fm, (1 , 2 ) =

and, since the so-called moment generating function in

Nakagami-m fading is given by [14]

Q(1 )Q(2 )pm, ()d

m, (s) =

Q(1 , 2 ; 0)pm, ()d,

(6)

where 1 , 2 R and Q(, ; ) is a bivariate Qfunction [12, Equation (6)].

To obtain a closed form for the average BER
Equation (5) it is then sufficient to evaluate Equation
(6) assuming that 1 , 2 are positive. If one of the
arguments of Equation (6) is negative we may use the
relationship Q(||) = 1 Q(||) to obtain

exp(s)pm, ()d =
0

1
2


exp

2 sin ()

exp

b2

d, a, b 0

2 sin2 ()

(x, y) =

(8)
tan1

 

if y = 0,

x
y

(9)

fm, (1 , 2 )
(
1 ,2 )

1
2


exp

21

2 sin2 ()

pm, () d d

(
2 ,1 )


exp

21

2 sin2 ()


22
2 sin2 ()

= Am, (1 , 2 ) + Am, (2 , 1 ), (12)

(x,y)
 

1
=
2
=

sin2 ()
sin () + x2 /(2m)

(x,y)+ 2 

1
Ic
2

cos2 ()
2
cos () + x2 /(2m)

d
m
d

x2 

, (x, y) + ,
, m . (13)
2
2 2m

otherwise,

we obtain a new version of Equation (6)

1
=
2

m,
0

1
Am, (x, y) =
2

(
2 ,1 )

1
+
2

a2

with

m,

(b,a)


where

1
+
2

(11)

(
1 ,2 )

1
fm, (1 , 2 ) =
2

(7)

Q(a, b; 0) =

1
,
2
we can write Equation (10) as

= 2fm, (0, |2 |) fm, (|1 |, |2 |)

Knowing that [13, Equation (10)]

m
m + s

fm, (|1 |, |2 |) = fm, (|2 |, |1 |)

(a,b)


95

22
2

2 sin ()

pm, () d d

(10)

The function Ic (u, v, a, m) was introduced in Reference [14]

m
v 
cos2 (x)
Ic (u, v, a, m) =
dx,
(14)
a + cos2 (x)
u

where also a closed-form expression for it was shown

for integer values of m. Our derivation for a half-integer
m, presented in more details in the Appendix, complements the existing results. Finally, Equation (14) may
be written as follows:



y=(1+1/a) cot2 (v)

arctan (1 + y)a y=(1+1/a) cot2 (u)

if m = 0.5,






n1
m

m
n1



ank 21

I
(u,
v,
a,
0.5)
+
(1)

c
1

n
k
n=1
k=0 (1+a) 2

 



Ic (u, v, a, m) =
1/a + 1 cot(v) , 1/a + 1 cot(u) , 1 + k
m 0.5 + N
P






m
n1

m
n1



ank 21

u
+
(1)

n
k
n=1

k=0 (1+a) 2

 




1/a + 1 cot(v) , 1/a + 1 cot(u) , 1 + k

m N
P
Copyright 2006 John Wiley & Sons, Ltd.

(15)

Wirel. Commun. Mob. Comput. 2008; 8:9399


H. XU, M. BENJILLALI AND L. SZCZECINSKI

96

We note that in the context of our problem, the notation may be further simplified because the first parameter of the function Equation (14) is constant. Thus,
u = /2 in Equation (14) implies t cot(u) = 0, so
[q(x)]x=0
x=a q(a) in Equation (16). However, we preferred to keep the original structure of Equation (14)
defined by [14] because it may be applied in a variety
of the performance evaluation problems, cf. [14].
4.

Fig. 1. 16-QAM modulation with non-Gray bit-to-symbol

mapping taken from [6].

Numerical Examples

As a double-check for the mathematical derivation, we

contrast the analytical approach with simulation results comparing them also with some known methods,
namely the union bound [16, Ch. 4.3.2], and the expurgated bound [17].
All simulations were done with a 16-QAM constellation. The bit-to-symbols mapping, taken from [6] and
shown in Figure 1, is non-Gray. From Figure 2, we

where m = m 0.5 , = 1 + 2(m m ), x is the

smallest integer greater or equal to x, [q(x)]x=b
x=a
q(b) q(a) and the function P(s, t, n) is defined as
P(s, t, n) =



n1

(nk1)!  y y=t

y=t
1n

2
[arctan(y)]
+
if m N
(2n3)!!
y=s
k
2
nk
(n1)!
2 (2n2k1)!! (1+y )
y=s
k=1

y=s

(y+1)0.5n

1
if m 0.5 + N
(12n)
1+a

(16)

y=t

where (2n 1)!! (2n 1) (2n 3) . . . 3 1 and

for convenience (1)!! 1, [14].
Finally, using the Equation of (15) in Equation (12),
it is straightforward to calculate the average BER in
Equation (5). Slightly different but equivalent derivations based on recursive formulas are shown in [15].

Fig. 2. Simulation results and numerical evaluation of the

BER obtained in Nakagami-m channels; m = 1/2, m = 1
and m = 2.
Copyright 2006 John Wiley & Sons, Ltd.

can see that the simulation results exactly match the

BER predicted by the presented analytical formula for
m = 0.5, 1, and 2.
Both the union- and expurgated bounds as well as the
presented analytical results are practically the same for
m 2 at high SNR, that is, for weak fading. This confirms the results obtained for the AWGN channel [7].
On the other hand, the bounds produce a significant
discrepancy for small values of m. In particular, for
m = 1/2, both bounds are not tight even for a high
SNR providing, respectively, 2 dB and 10 dB offset at BER = 101 . Note that the union bound is also
not tight for m = 1, that is, in a Rayleigh channel.
It is easy to see that given , the function
fm, (1 , 2 ) defined in Equation (6) is analytic with
respect to m so BER is also an analytic function with
respect to m. This implies that BER, as a function
of m, can be globally approximated by a series of
polynomials over any closed subinterval [m1 , m2 ] as
long as 0 < m1 < m2 < +. To approximate BER
for m > 0.5, we can apply Lagrange interpolation
[18, pp. 89].
The interpolation with the knots at m
{0.5, 1, 1.5, 2, 2.5} produces the results which
are shown in Figure 3 for m = 0.75, 1.25, and 1.75.
Wirel. Commun. Mob. Comput. 2008; 8:9399

BIT ERROR RATE RECTANGULAR QAM

97

A.m=0.5
In this case
v

Ic (u, v, a, 0.5) =
u

cos2 (x)
dx
a + cos2 (x)

(A.1)

Denoting
c = 1/a and performing a change of variable
z = 1 + c cot(x) we obtain
Ic (u, v, a, 0.5)

cot(u)
 1+c

1+c

cot(v) 1+c


Fig. 3. Simulation results and the BER obtained via interpolation in Nakagami-m channels; m = 0.75, 1.25, and 1.75.

It is clear that our formula may be used to predict

BER even when m is not in the set 0.5 N . The
BER interpolation error is, for all practical purpose
negligible, as can be seen in Figure 3.

z2
dz
a(1 + c + z2 ) + z2

cot(u)
 1+c

1+c

cot(v) 1+c


5.

dz
1+a+

Conclusion

In this paper we presented closed-form formulas to

calculate the BER of rectangular QAM in Nakagamim channels for m {0.5, 1, 1.5, 2, 2.5, . . .}, which
model a large family of fading scenarios. The proposed
expressions, unlike the other known from the literature,
are not restricted to Gray mapping.
Using numerical simulations and through comparisons with the union and expurgated bounds, we illustrated the advantage of using the derived expression,
especially in the low SNR range and for small values
of the parameter m, that is, for severely fading channels.

(A.2)

Ic (u, v, a, 0.5)

1+c
2

2
(1+c)
 cot (u)

(1+c) cot2 (v)

1
(1 + c + y)

dy
(1 + a) + (1 + a)y

2
(1+c)
 cot (u)

2 a

(1+c) cot2 (v)


=

Copyright 2006 John Wiley & Sons, Ltd.

1+a
z2

1
(1 + c + z2 )

Another change of variable y = z2 gives rise to

Appendix
We present the mathematical derivations of the Equation in (15) when m is not an integer and when u, v > 0,
which is indeed the case in our scenario because u
/2 and /2 v .
For the cases, which may appear in other performance evaluation problems, for example, [14], we may
use the relationships Ic (|u|, |v|, a, m) = Ic (|v|, |u|,
a, m), Ic (|u|, |v|, a, m) = Ic (0, |v|, a, m) + Ic (0, |u|,
a, m), and Ic (|u|, |v|, a, m) = Ic (|v|, |u|, a, m).

1
1 + c + z2


arctan

dy

(1 + c + y) 1 + y

1+y
c

B.m 0.5 + N

(1+c) cot2 (v)

.

(A.3)

(1+c) cot2 (u)

Let us denote c = 1/a and m = m1 + 0.5 with m1

N . Then
Ic (u, v, a, m) =
v 
u

cos2 (x)
a + cos2 (x)

m1 

cos2 (x)
dx
a + cos2 (x)

(A.4)

Wirel. Commun. Mob. Comput. 2008; 8:9399


H. XU, M. BENJILLALI AND L. SZCZECINSKI

98

and, using a binomial identity



cos2 (x)
a + cos2 (x)

m1

2
(1+c)(cot(u))


m1

a
=
+1
a + cos2 (x)


n
m1

m1
a
=
a + cos2 (x)
n

(1+c)(cot(v))2

k=0

n=0

Ic (u, v, a, m)
= Ic (u, v, a, 0.5) +

m1



(1)n

m1

cos2 (x)
1+
a

= Ic (u, v, a, 0.5) +

n 

m1



n

m1

1
1 + c cos2 (x)

1
1 + c cos2 (x)

n 

cos2 (x)
dx
a + cos2 (x)

(A.5)

cos2 (x)
dx
a + cos2 (x)

1+c
 cot(u)

1
(1 + c)n0.5

(1 + c + z2 )n1
(1 + z2 )n

1+c cot(v)

z2
dz
a(1 + c + z2 ) + z2


n1

n1
ck
=
(1 + c)n0.5
k

k=0

1+c
 cot(u)

1+c cot(v)

n1

k=0

n1
k

(1 + z2 )k+1

ank 2

1
2 1 + a(1 + a)n 2
(1+c)(cot(u))2

2(y + 1)0.5k
2k + 1

(A.6)
(1+c)(cot(v))2

We thank the anonymous reviewer for his detailed revision and helpful comments. Research supported by
PROMPT, Quebec, and NSERC, Canada under research grant #249704-02. Hong Xu was with Axiocom
Research Inc., Trois-Rivieres, Canada when the work
was carried out.

We consider now the integration inside the

last equation. Changing variable again, z = cot(x) 1 + c, we
obtain
v 

Acknowledgement

n=1

n 

n1
k

1+y

cos2 (x)
dx
a + cos2 (x)

(1)

n=1

we may write

n1


dy
(1 + y)k+1

zdz
a(1 + c) + (1 + a)z2

ck

2 1 + a(1 + c)n0.5

Copyright 2006 John Wiley & Sons, Ltd.

References
1. Cho K, Yoon D. On the general BER expression of one- and
two-dimensional amplitude modulations. IEEE Transactions on
Communications 2000; 50(7): 10741080.
2. Hanzo L, Wong CH, Yee MS. Adaptive Wireless Tranceivers.
John Wiley and Sons: Chichester, England, 2002.
3. Vitthaladevuni PK, Alouini MS. A closed-form expression for
the exact BER of generalized PAM and QAM constellations.
IEEE Transactions on Communications 2004; 52(5): 698700.
4. Takahara G, Alajaji F, Beaulieu NC, Kuai H. Constellation mappings for two-dimensional signaling of nonuniform sources.
IEEE Transactions on Communications 2003; 51(3): 400408.
5. Li X, Ritcey J. Bit-interleaved coded modulation with iterative
decoding using soft feedback. Electronic Letters 1998; 34(10):
942943.
6. Schreckenbach F, Gortz N, Hagenauer J, Bauch G. Optimization
of symbol mappings for bit-interleaved coded modulation with
iterative decoding. IEEE Communications Letters 2003; 7(12):
593595.
7. Szczecinski L, Gonzalez C, Aissa S. Exact expression for the
BER of rectangular QAM with arbitrary constellation mapping.
accepted for publication in IEEE Transactions on Communications 2006; 54(3): 389392.
8. Berrou C, Glavieux A, Thitimajshima P. Near shannon limit
error-correcting coding and decoding: Turbo codes. In Proceedings of ICC 93, 1993, pp. 10641070.
9. Xiao L, Dong X. A new approach to calculating the exact transition probability and bit error probability of arbitrary twodimensional signaling. In IEEE Globecom 2004, 2004, pp. 1239
1243.
10. Bai H, Atiquzzaman M. Error modeling schemes for fading
channels in wireless communications: a survey. IEEE Communications Surveys and Tutorials. Fourth Quarter 2003 2003; 5(2):
29.
11. Simon MK, Alouini MS. Digital Communications Over Fading
Channels: A Unied Approach To Perfomance Analysis, 1st edn.
John Wiley & Sons: New York, NY, 2000.
Wirel. Commun. Mob. Comput. 2008; 8:9399

BIT ERROR RATE RECTANGULAR QAM

12. Zhong L, Alajaji F, Takahara G. Error analysis for nonuniform
signaling over Rayleigh fading channels. IEEE Transactions on
Communications 2005; 53(1): 3943.
13. Simon MK. A simpler form of the Craig representation for the
two-dimensional joint Gaussian Q-function. IEEE Communications Letters 2002; 6(2): 4951.
14. Annamalai A, Tellambura C. Error rates for Nakagami-m fading multichannel reception of binary and M-ary signals. IEEE
Transactions on Communications 2001; 49(1): 5868.
15. Xu H, Benjillali M, Szczecinski L. Exact expression for the
BER of rectangular QAM with arbitrary constellation mapping
in Nakagami-m fading channel. Technical Report INRS-EMT011-1004, Institut National de la Recherche Scientifique-EMT,
October 2004.
16. Benedetto S, Biglieri E. Principles of Digital Transmission with
Wireless Applications. Kluwer Academic, 1999.
17. Caire G, Taricco G, Biglieri E. Bit interleaved coded modulation.
IEEE Transactions on Information Theory 1998; 44(3): 927
946.
18. Borwein P, Erdelyi T. Polynomials and Polynomial Inequalities.
Springer-Verlag: New York, 1995.

Authors Biographies
Hong Xu graduated from Department of Math in Wuhan University,
PRC in 1991. He obtained a Ph.D in
Mathematics from University of Paris
Dauphine in 1996. By 2002, he got a
MS degree in telecommunications from
INRS, Canada. He worked for Nortel,
Axiocom Inc. since 2000. He recently
joined TenXc wireless Inc as systems
engineer. His research interests are MIMO, space time
coding, channel coding etc.

Copyright 2006 John Wiley & Sons, Ltd.

99

Mustapha Benjillali graduated with

honors as an engineer from INPT,
Morocco, in 2003. He received the
M.Sc. degree in telecommunications
in 2005 from INRS, Montreal, Canada,
where he is working toward the Ph.D.
degree. Since May 2004, he has been
a research assistant with the Wireless
Communications Group at INRS-EMT,
Montreal, Canada. His research interests
include wireless communications with a focus on physical
and link aspects, cross-layer design, packet combining and
performance evaluation.
Leszek Szczecinski received M.Eng.
degree from the Technical University of
Warsaw, Poland in 1992, and Ph.D. from
INRS-Telecommunications,
Canada
in 1997. From 1998 to 2000, he was
with the Department of Electrical
Engineering, University of Chile; since
2001, has been assistant professor at
INRS-EMT, Montreal, Canada. His
research interests are in the area of digital signal processing
for wireless communications with emphasis on iterative
processing.

Wirel. Commun. Mob. Comput. 2008; 8:9399