A Rapid Introduction to Adaptive Filtering

Leonardo Rey Vega and Hernan ReySpringerBriefs in Electrical and Computer EngineeringA Rapid Introduction to Adaptive Filtering201310.1007/978-3-642-30299-2_1© The Author(s) 2013

1. Introduction

Leonardo Rey Vega¹   and Hernan Rey²  

(1)

School of Engineering, University of Buenos Aires, Paseo Colón 850, C1063ACV Buenos Aires, Argentina

(2)

Department of Engineering, University of Leicester, University Road, Leicester, LE1 7RH, UK

Leonardo Rey Vega (Corresponding author)

Email: lrey@fi.uba.a

Hernan Rey

Email: hgr3@le.ac.uk

Abstract

The area of adaptive filtering is a very importantone in the vast field of Signal Processing. Adaptive filters areubiquitous in current technology. System identificaction, equalization for communication systems, active noise cancellation,speech processing, sonar, seismology, beamforming, etc, are a few examples from a large set of applications were adaptive filters are used to solve different kinds of problems. In this chapter we provide a short introduction to the adaptive filtering problem and to the different aspects that should be taken into account when choosing or designing an adaptive filter for a particular application.

1.1 Why Adaptive Filters?

The area of adaptive filtering has experienced a great growth in the last 50 years, and even today it is constantly expanding. This allowed for the possibility of applying adaptive filters in such a variety of applications as system identification, echo cancelation, noise cancelation, equalization, control, deconvolution, change detection, smart antennas, speech processing, data compression, radar, sonar, biomedicine, seismology, etc. [1, 2]. Certainly, this would not have been possible without the VLSI technology necessary to provide the computational requirements for these applications [3].

Fig. 1.1

Scheme of an adaptive filtering problem

A filter becomes adaptive when its coefficients are updated (using a particular algorithm) each time a new sample of the signals is available. Conceptually, the algorithm chosen should be able to follow the evolution of the system under study. This provides the adaptive filter with the ability to operate in real time and improves its performance with no need to have any users involved [2].

Figure 1.1 shows a scheme of an adaptive filtering problem. Basically, the adaptive filter receives a vector input signal $${\mathbf x }(n)$$ and a reference/desired signal d(n).¹ At time $$n$$ , the filter coefficients

, with $$(\cdot )^T$$ denoting transposition, will be updated according to a specific algorithm.

As we will see in Chap. 2, Wiener filtering provides the linear optimum filtering solution (in mean-square-error sense). However, we will also see that the associated cost function and its optimal solution require knowledge of the input autocorrelation matrix and the cross-correlation vector. These statistics capture the information required about the system under study. Then, we may ask: Why should we seek adaptive algorithms? There are indeed several reasons, e.g.:

Savings in memory. In practice, the statistics are not available, so they need to be estimated. This computation could require the accumulation of a lot of samples from the input and desired signals.

Introduction of delay. The accumulation and post-processing introduce delay in the filter output. This issue is critical in certain applications like echo cancelation, adaptive delay estimation, low-delay predictive coding, noise cancelation, radar, and channel equalization applications in mobile telephony.

No a priori statistical information required. The adaptive algorithm learns the statistics from only one realization of the input and reference processes. Hence, the designer has no need to know a priori the underlying signal statistics.²

Tracking ability. If the statistics (of the input and/or reference signals) change with time, the optimal filter will also change, so it would require its recalculation. The learning mechanism of the adaptive algorithm can track these changes in the statistics. Again, different algorithms will do this at different rates and with different degrees of success.

Ease of implementation. In general, they are easy to implement, making them suitable for real-world applications.

Three basic items are important in the design of an adaptive filter [3]:

1.

Definition of the objective function form. Some examples for the objective function are Mean Square Error (MSE), Least Squares (LS), Weighted Least Squares (WLS), Instantaneous Squared Value (ISV) and Mean Absolute Error (MAE). The MSE, in a strict sense, is only of theoretical value, since it requires an infinite amount of signal samples to be computed (given it has the expectation operator in its definition). In practice, this ideal objective function can be approximated by the LS, WLS, or ISV. They differ in the implementation complexity and in the convergence behavior characteristics. In general, the ISV is easier to implement but it presents noisy convergence properties, since it represents a greatly simplified objective function. The LS is convenient to be used in stationary environments, whereas the WLS is useful in applications where the environment is slowly varying or when the measurement noise is not white. MAE is particularly useful in the presence of outliers and leads to computational simple algorithms.

2.

Definition of the error signal. The choice of the error signal is crucial for the algorithm definition, since it can affect several of its characteristics, including computational complexity, speed of convergence, and robustness.

3.

Definition of the minimization algorithm. This item is the main subject of Optimization Theory, and it essentially affects the speed of convergence and computational complexity of the adaptive process. As we will see in Chap. 3, Taylor approximations of the cost function with respect to the update can be used. The first-order approximation leads to the gradient search algorithm, whereas if the second-order approximation is used, Newton’s algorithm arises (if the Hessian matrix is not positive definite, then the function does not necessarily have a unique minimum). Quasi-Newton’s algorithms are used when the Hessian is estimated, but they are susceptible to instability problems due to the recursive form used to generate the estimate of the inverse Hessian matrix. In these methods, the step size controls the stability, speed of convergence, and some characteristics of the residual error of the overall iterative process.

Several properties should be considered to determine whether or not a particular adaptive filtering algorithm is suitable for a particular application. Among them, we can emphasize speed of convergence, steady state error, tracking ability, robustness against noise, computational complexity, modularity, numerical stability and precision [1]. Describing these features for a given adaptive filter can be a very hard task from a mathematical point of view. Therefore, certain hypotheses like linearity, Gaussianity and stationarity have been used over the years to make the analysis more tractable [4]. We will show this in more detail in the forthcoming chapters.

1.2 Organization of the Work

In the next chapter we start studying the problem of optimum linear filtering with stationary signals (particularly, in the mean-square-error sense). This leads to the Wiener filter. Different values for the filter coefficients would lead to larger mean square errors, a relation that is captured by the error performance surface. We will study this surface as it will be very useful for the subsequent chapters. We also include and example where the Wiener filter is applied to the linear prediction problem.

In Chap. 3 we will introduce iterative search methods for minimizing cost functions. This will be particularly applied to the error performance surface. We will focus on the methods of Steepest Descent and Newton-Raphson, which belong to the family of deterministic gradient algorithms. These methods uses the second order statistics to find the optimal filter (i.e., the Wiener filter) but unlike the direct way introduced in Chap. 2, they find this solution iteratively. The iterative mechanism will lead to the question of stability and convergence behavior, which we will study theoretically and with simulations. Understanding their functioning and convergence properties is very important as they will be the basis for the development of stochastic gradient adaptive filters in Chap. 4

Chapter 4 presents the Stochastic Gradient algorithms. Among them, we find the Least Mean Square algorithm (LMS), which is without doubt the adaptive filter that has been used the most. Other algorithms studied in this chapter are the Normalized Least Mean Square (NLMS), the Sign Data algorithm (SDA), the Sign Error algorithm (SEA), and the Affine Projection algorithm (APA). We show several interpretations of each algorithm that provide more insight on their functioning and how they relate to each other. Since these algorithms are implemented using stochastic signals, the update directions become subject to random fluctuations called gradient noise. Therefore, a convergence analysis to study their performance (in statistical terms) will be performed. This will allow us to generate theoretical predictions in terms of the algorithms stability and steady state error. Simulation results will be provided to test this predictions and improve the understanding of the pros and cons of each algorithm. In addition, we discuss the applications of adaptive noise cancelation and adaptive equalization.

In Chap. 5 we will study the method of Least Squares (LS). In this case, a linear regression model is used for the data and the estimation of the system using input-output measured pairs (and no statistical information) is performed. As the LS problem can be thought as one of orthogonal projections in Hilbert spaces, interesting properties can be derived. Then, we will also present the Recursive Least Squares (RLS) algorithm, which is a recursive and more computational efficient implementation of the LS method. We will discuss some convergence properties and computational issues of the RLS. A comparison of the RLS with the previously studied adaptive filters will be performed. The application of adaptive beamforming is also developed using the RLS algorithm.

The final chapter of the book deals with several topics not covered in the previous chapters. These topics are more advanced or are the object of active research in the area of adaptive filtering. We include a succinct discussion of each topic and provide several relevant references for the interested reader.

References

1.

S. Haykin, Adaptive Filter Theory, 4th edn. (Prentice-Hall, Upper Saddle River, 2002)

2.

D.G. Manolakis, V.K. Ingle, S.M. Kogon, Statistical and Adaptive Signal Processing (Artech House, Norwood, 2005)

3.

P.S.R. Diniz, Adaptive Filtering: Algorithms And Practical Implementation, 3rd edn. (Springer, Boston, 2008)MATH

4.

S. Haykin, Signal processing: where physics and mathematics meet. IEEE Signal Process. Mag. 18, 6–7 (2001)CrossRef

Footnotes

1

Unless otherwise stated, we focus in this book on finite impulse response (FIR) filters, and random signals being time-discrete, real-valued, and zero-mean.

2

Even if some statistical information is available, it can be used combined with the input and reference realization.

Leonardo Rey Vega and Hernan ReySpringerBriefs in Electrical and Computer EngineeringA Rapid Introduction to Adaptive Filtering201310.1007/978-3-642-30299-2_2© The Author(s) 2013

2. Wiener Filtering

Leonardo Rey Vega¹   and Hernan Rey²  

(1)

School of Engineering, University of Buenos Aires, Paseo Colón 850, C1063ACV Buenos Aires, Argentina

(2)

Department of Engineering, University of Leicester, University Road, Leicester, LE1 7RH, UK

Leonardo Rey Vega (Corresponding author)

Email: lrey@fi.uba.a

Hernan Rey

Email: hgr3@le.ac.uk

Abstract

Before moving to the actual adaptive filtering problem, we need to solve the optimum linear filtering problem (particularly, in the mean-square-error sense). We start by explaining the analogy between linear estimation and linear optimum filtering. We develop the principle of orthogonality, derive the Wiener–Hopf equation (whose solution lead to the optimum Wiener filter) and study the error surface. Finally, we applied the Wiener filter to the problem of linear prediction (forward and backward).

2.1 Optimal Linear Mean Square Estimation

Lets assume we have a set of samples $$\{x(n)\}$$ and $$\{d(n)\}$$ coming from a jointly wide sense stationary (WSS) process with zero mean. Suppose now we want to find a linear estimate of $$d(n)$$ based on the $$L$$ -most recent samples of $$x(n)$$ , i.e.,

(2.1)

The introduction of a particular criterion to quantify how well $$d(n)$$ is estimated by $${\hat{d}}(n)$$ would influence how the coefficients $$\textit{w}_l$$ will be computed. We propose to use the Mean Squared Error (MSE), which is defined by

(2.2)

where $$E[\cdot ]$$ is the expectation operator and $$e(n)$$ is the estimation error. Then, the estimation problem can be seen as finding the vector $${\mathbf w }$$ that minimizes the cost function $$ J_{\mathrm MSE }({\mathbf w })$$ . The solution to this problem is sometimes called the