ICTON 2010 Mo.B1.

Advanced Modulation Formats and Digital Signal Processing

for Fiber Optic Communication
Alik Gorshtein, Dan Sadot, Senior Member, IEEE
Ben Gurion University, Beer-Sheva 85105, Israel
e-mail: alikgor@ee.bgu.ac.il, sadot@ee.bgu.ac.il
ABSTRACT
Optical communications is currently undergoing a real technological revolution based on coherent detection.
In addition to preserving the phase information of the optical field, in coherent detection the optical channel
becomes a linear system, enabling full compensation of optical fiber impairments by the use of digital signal
processing techniques. Here, a coherent transmission and reception scheme for ultra-high data rates (100G and
above), adopting advanced modulation formats is presented. In turn, symbol rate sampling followed by anti-
aliasing filtering and digital signal processing is applied. This allows reducing the analog to digital conversion
rate which forms one of the major bottlenecks at 100G and above data rates.
Keywords: ADC-DSP, symbol rate sampling, MLSE, DP-QPSK, coherent detection.

1. INTRODUCTION
One of the major benefits of coherent communication is the preserving of the phase of the optical field, which in
turn allows a full compensation of optical channel impairments like chromatic dispersion (CD) and polarization
mode dispersion (PMD). The full complex amplitude of the optical field can be obtained by mixing the signal
coming out of the fiber with an electrical field produced by the local oscillator (LO) laser in the receiver.
Coherent optical fiber systems intend to possess up to 80 wavelength division multiplexing (WDM) channels,
each channel carrying more than 100Gbps. To allow such ultra-high bitrates in a single carrier system,
an appropriate advanced modulation format should be carefully chosen. Two leading criteria in the choice of the
proper modulation format are the sampling rate of the analog to digital converter (ADC), dictated by the analog
signal bandwidth, and the overall system performance in terms of bit error rate (BER) and signal to noise ratio
(SNR). Investigations made in [1] show that the most suitable candidate for 40Gbps system is dual polarization
quadrature phase shift keying (DP-QPSK) modulation format, due to its lower BER compared to other
investigated modulation formats. Therefore, DP-QPSK is the leading modulation format for single carrier 100G
(and beyond) coherent long-haul optical fiber communication systems. Here we present 111Gbps DP-QPSK
transmission system, focusing on subsequent digital signal processing (DSP) and equalization of linear
transmission impairments.
The common practice for digital compensation is to use fractional space equalizers, with two samples per
symbol or more. 111 Gbps DP-QPSK system DSP requires ADCs with very high sampling rate, which are
a state of the art even with the last advances in SiGe and CMOS technology. Therefore, it is desirable to reduce
the sampling rate in the designed systems. It is well known that in undistorted media, sampling at the symbol
rate forms sufficient information to recover the digital data [2]. As it was previously shown [3]-[5], the sampling
rate can be reduced down to one sample per symbol. Moreover it does not necessarily have to be an integer
multiple of the symbol rate. Yet, when the channel introduces linear distortions, e.g., CD, full information of the
received analog signal is required in order to apply digital compensation. Sampling the received signal with less
than two samples per symbol without preceding filtering violates the Nyquist sampling criterion, causing
aliasing effect that results in performance degradation. On the other hand, using anti aliasing filter (AAF) prior to
symbol rate sampling introduces heavy low pass filtering (LPF) which, in turn, causes heavy inter symbol
interference (ISI). The subject and rational of sampling rate of M/K where M>K and are both integer numbers,
together with digital CD compensation, are described in [3]. The interplay between the sampling rate of the ADC
and the bandwidth of the analog anti-aliasing filters with subsequent CD equalization for 100G DP-QPSK is
studied in [5]. Symbol rate sampling, followed by digital CD compensation is suggested in [5].
Here we also propose a symbol rate of 111 Gbps DP-QPSK system, with subsequent digital compensation of
linear transmission impairments. Our main contributions are symbol rate sampling and the use of the maximum
likelihood sequence estimation (MLSE) for combating heavy ISI, introduced by the AAF. It is widely known,
that MLSE is the optimal equalizer, in the sense of minimum probability of error, for channels with ISI [2].

2. SYSTEM MODEL
A high level block diagram of single carrier coherent 111 Gbps DP-QPSK system is given on Figure 1.
The transmitter (Tx) and receiver (Rx) front ends are similar to ones presented in [3]. The optical channel is
assumed to be linear, exhibiting CD 1st order PMD [3] and amplified spontaneous emission (ASE) noise coming
from optical amplifiers.

978-1-4244-779-/10/$26.00 2010 IEEE

ICTON 2010 Mo.B1.1

Figure 1. High level block diagram of single carrier coherent 111Gbps DP-QPSK system.
Figure 2 presents the digital signal (post) processing unit (PPU) that is used to combat the transmission
impairments employing coherent DP-QPSK modulation format.

I(
x)
(t ) I(
x)
[ n]

Q(
x)
(t ) Q( x) [ n] CD-1
I(
y)
(t ) I ( y ) [ n] hxx hxy

hyx hyy
Q( y ) ( t ) Q( y) [ n] CD-1

Figure 2. Post processing unit for DP-QPSK system.

At the input of the PPU there are four lanes corresponding to the in-phase and quadrature signals from both
polarizations coming from a polarization diversity receiver, which can be expressed analytically as follows:
j ( x) j ( x) 1 j ( y ) j ( y )
I R( ) ( t ) = K cos 2 rl t e l + sin 2 rl t + e l sin 2 rl t e l rl t + e l + I N( ) ( t )
x x

l 2 l 2 2 l 2 l 2 (1)
j (y ) j (y ) 1 j ( x) j ( x)
I R( ) ( t ) = K cos 2 rl t + e l + sin 2 rl t e l sin 2 rl t e l rl t + e l + I N( ) ( t )
y y

l 2 l 2 2 n 2 l 2
(x) (y )
where I R( x ) ( t ) and I R( y ) ( t ) represent complex baseband currents in each polarization. e jl , e jl and
I N( x ) ( t ) , I N( y ) ( t ) denote the complex information and noise terms in the corresponding polarization respectively.
In addition, equation (1) accounts or the following effects: CD, first order PMD, laser phase noise at both Tx and
local oscillator lasers, intradyne, AAF and additive Gaussian noise (mainly coming from optical amplifiers).
The effect of PMD in (1) is expressed through (the angle between the input state of polarization in the fiber
and the principal states of polarization) and being the differential group delay (DGD) between the modes [3].
The influence of CD, phase noises and intradyne is represented by the shaping pulse:

( )
rl ( t ) = hCD g ( t lT ) e jT (t ) e
j IF t +R ( t )
(2)

where g ( t ) is the shaping pulse at the transmitter, hCD ( t ) designates the impulse responses accounting for CD,
T ( t ) and R ( t ) represent the phase noses at the Tx and Rx respectively, and IF denotes the angular intradyne

frequency, describing the frequency mismatch between Tx and LO lasers.

Each lane, corresponding to the real and imaginary parts of (1), is filtered by an AAF, modeled as a 5th order
Butterworth low-pass filter (LPF). In turn, the signal at each lane is sampled and quantized by an analog to
digital converter (ADC) at sampling rate of 27.75 Gsamples/s with 5-bit resolution. The proposed PPU is similar
to those presented in [1][3][6], when the differences are: sampling and DSP processing rates are equal to symbol
rate, the incoming signal bandwidth is intentionally reduced by the AAFs to comply with the Shannon sampling
theorem, and the introduced ISI is compensated by the MLSE. The CD compensation block is denoted on Fig. 2
as CD-1, Polarization Demux carries out PMD compensation, IFE and CPE blocks perform intradyne frequency
estimation and carrier frequency estimation to compensate for frequency mismatch and lasers' phase noises
respectively.

3. SIMULATION RESULTS
To verify the concept of using an MLSE after the compensation of the transmission impairments, using a single
sample per symbol, a set of simulations has been carried out. At the first stage, only a single polarization and CD
effect have been examined. It is assumed that the same laser serves as the local oscillator and the transmit laser,

2
ICTON 2010 Mo.B1.1

i.e. there is no need to compensate for frequency mismatch and phase noise. A set of Monte-Carlo simulations
with 400,000 bits each, was performed to determine the optimal AAF bandwidth to ensure pre-FEC BER value
of 10-3. The CD equalizer is designed according to a zero forcing criterion, and can be implemented either in the
time or in the frequency domain [7]. The number of states in the MLSE is 16; histogram estimation method is
used for channel estimation with training sequence of 50,000 observations, which are subsequently discarded
from BER calculation. The simulation results are summarized in Fig. 3.
111Gbps DP-QPSK, orderAAF= 5, various CD in [ps/nm] 111Gbps DP-QPSK,12GHz 5th order AAF, CD[ps/nm] + DGD [ps]
18 18
0 0
17.5 200 17.5 20
1000 40
17 10000 17 60
Required OSNR@BER=10-3,[dB]

Required OSNR@BER=10-3,[dB]
40000 80
16.5 100000 16.5 100

16 16

15.5 15.5

15 15

14.5 14.5

14 14

13.5 13.5
9 10 11 12 13 14 15 2 3 4 5
10 10 10 10
AAF cutoff,[GHz] CD,[ps/nm]

(a) (b)
Figure 3. Required ONSR at 10-3 BER; (a) versus AAF cutoff frequency for various CD, and (b) versus
CD for various DGD.

It can be seen in Figure 3(a) that the proposed system with MLSE decoder enables full compensation of CD up
to 100,000 ps/nm. Furthermore, the ISI that is introduced by the AAF is completely compensated with an OSNR
penalty of 1 dB only, as compared to back to back (B-t-B) transmission.
Figure 3(b) reveals that the Polarization Demux fully compensates for DGD up to 100ps with less than
0.5 dB penalty. Its coefficients are updated by two-dimensional LMS criterion [7], whereas 13 complex taps are
used.

4. CONCLUSIONS
A 111Gbps DP-QPSK system with one sample per symbol based on anti-aliasing filtering and MLSE is
proposed. It is shown that the system can tolerate 100,000 ps/nm of CD together with 100 ps of DGD with only
1.5 dB penalty when MLSE is used to compensate the ISI introduced by the AAF. This concept allows the use of
commercially available 28 Gsamples/sec ADC.

REFERENCES
[1] K. Roberts et al.: Performance of dual-polarization QPSK for optical transport systems, J. Lightwave
Technol., vol. 27. pp. 3546-3559, Aug. 2009.
[2] J. G. Proakis: Digital Communications, Ed., New York: McGraw-Hill, 1995.
[3] E. Ip and J.M. Kahn: Digital equalization of chromatic dispersion and polarization mode dispersion,
J. Lightwave Technol., vol. 25, pp. 2033-2043, Aug. 2007.
[4] C. Malouin et al.: Sub-rate sampling in 100 Gb/s coherent optical receivers, in Proc. OFC 2010, San Diego,
California, USA, March 2010, paper OThT3.
[5] C.R.S. Fludger et al.: Coherent equalization and POLMUX-RZ-DQPSK for robust 100-GE transmission,
J. Lightwave Technol., vol. 26, pp. 131-141, Jan. 2008.
[6] M. Kuschnerov et al.: DSP for coherent single-carrier receivers, J. Lightwave Technol., vol. 27, pp. 3614-
3622, Aug. 2009.
[7] S. J. Savory: Digital filters for coherent optical receivers, Optics Express, vol. 16, pp. 804-817, 2008.