Adaptive Neural Fuzzy Inference Systems (ANFIS): Analysis and Applications

Copyright 2002 by Piero P. Bonissone

Outline
Objective Fuzzy Control
Background, Technology & Typology

ANFIS:
as a Type III Fuzzy Control as a fuzzification of CART Characteristics Pros and Cons Opportunities Applications References
Copyright 2002 by Piero P. Bonissone 2

ANFIS Objective
To integrate the best features of Fuzzy Systems and Neural Networks:
From FS: Representation of prior knowledge into a set of constraints (network topology) to reduce the optimization search space From NN: Adaptation of backpropagation to structured network to automate FC parametric tuning

ANFIS application to synthesize:

controllers (automated FC tuning) models (to explain past data and predict future behavior)
Copyright 2002 by Piero P. Bonissone 3

FC Technology & Typology

Fuzzy Control
A high level representation language with local semantics and an interpreter/compiler to synthesize non-linear (control) surfaces A Universal Functional Approximator

FC Types
Type I: RHS is a monotonic function Type II: RHS is a fuzzy set Type III: RHS is a (linear) function of state
4

Copyright 2002 by Piero P. Bonissone

FC Technology (Background)
Fuzzy KB representation
Scaling factors, Termsets, Rules

Rule inference (generalized modus ponens) Development & Deployment

Interpreters, Tuners, Compilers, Run-time Synthesis of control surface

FC Types I, II, III

Copyright 2002 by Piero P. Bonissone

FC of Type II, III, and ANFIS

Type II Fuzzy Control must be tuned manually Type III Fuzzy Control (Takagi-Sugeno type) have an automatic Right Hand Side (RHS) tuning ANFIS will provide both:
RHS tuning, by implementing the TSK controller as a network and LHS tuning, by using back-propagation
Copyright 2002 by Piero P. Bonissone 6

ANFIS Network

Inputs

IF-part

Rules + Norm

THEN-part

Output

1
& N

Input 1

x1
& N

& N

Output

Input 2

x2
& N

Layers:

Copyright 2002 by Piero P. Bonissone

ANFIS Neurons: Clarification note

Note that neurons in ANFIS have different structures:
Values [ Membership function defined by parameterized
soft trapezoids (Generalized Bell Functions) ]

Rules [ Differentiable T-norm - usually product ] Normalization [ Sum and arithmetic division ] Functions [ Linear regressions and multiplication with , i.e., normalized weights , ] Output [ Algebraic Sum ]

Copyright 2002 by Piero P. Bonissone

ANFIS as a generalization of CART

Classification and Regression Tree (CART)
Algorithm defined by Breiman et al in 1984 Creates a binary decision tree to classify the data into one of 2n linear regression models to minimize the Gini index for the current node c: Gini(c) = where:
1 ! p2 j
j

pj is the probability of class j in node c Gini(c) measure the amount of impurity (incorrect classification) in node c
Copyright 2002 by Piero P. Bonissone 9

CART Problems
Discontinuity Lack of locality (sign of coefficients)

Copyright 2002 by Piero P. Bonissone

10

CART: Binary Partition Tree and Rule Table Representation

x1

x1 a1 > x1
x2

x2 a2 > a2 a2 > a2

y f1(x1,x2) f2(x1,x2) f3(x1,x2) f4(x1,x2)

a1 x1
x2

a1 a1 a2 > x2 f4 (x1,x2 ) a1 > a1 >

a2 x2

a2 > x2 a2 x2

f1(x1,x2)

f2(x1,x2)

f3 (x1,x2)

Partition Tree

Rule Table

Copyright 2002 by Piero P. Bonissone

11

Discontinuities Due to Small Input Perturbations

y3= f3(x11, . )

x1

x1 a1
x2

a1 > x1
x2

y1= f1(x11, . )

x2 a2

a2 > x2 a2 x2 a2 > x2 f4 (x1,x2)

f1(x1,x2 )

f2 (x1,x2)

f3 (x1,x2)

(a )(x1)
1

X1

Let's assume two inputs: I1=(x11,x12), and I2=(x21,x22) such that: x11 = ( a1- ) x21 = ( a1+ ) x12 = x22 < a2 Then I1 is assigned f1(x11,x12) while I2 is assigned f3(x1,x2) 0

X1
x11 a1 x21
Copyright 2002 by Piero P. Bonissone 12

Takagi-Sugeno (TS) Model

Combines fuzzy sets in antecedents with crisp function in output: IF (x1 is A) AND (x2 is B) THEN y = f(x1,x2)
IF X is small THEN Y1=4 IF X is medium THEN Y2=-0.5X+4 IF X is large THEN Y3=X-1

!Y w
j

Y =
Copyright 2002 by Piero P. Bonissone

j =1 n

!w
j =1

j
13

ANFIS Characteristics
Adaptive Neural Fuzzy Inference System (ANFIS)
Algorithm defined by J.-S. Roger Jang in 1992 Creates a fuzzy decision tree to classify the data into one of 2n (or pn) linear regression models to minimize the sum of squared errors (SSE):
SSE = ! e2 j
j

where:
ej is the error between the desired and the actual output p is the number of fuzzy partitions of each variable n is the number of input variables
Copyright 2002 by Piero P. Bonissone 14

ANFIS as a Type III FC

L0: State variables are nodes in ANFIS inputs layer L1: Termsets of each state variable are nodes in ANFIS values layer, computing the membership value L2: Each rule in FC is a node in ANFIS rules layer using soft-min or product to compute the rule matching factor i L3: Each i is scaled into i in the normalization layer L4: Each i weighs the result of its linear regression fi in the function layer, generating the rule output L5: Each rule output is added in the output layer
Copyright 2002 by Piero P. Bonissone 15

ANFIS Architecture
Rule Set:
IF ( x 1 is A 1 ) AND (x 2 is B 1 ) THEN IF ( x 1 is A 2 ) AND (x 2 is B 2 ) THEN ...
x1 x2 x1 x2 A1 A2 B1 B2 1 1 2 2 N 1 N 2 x1 x2 4 1 f1 2 f2 5 y

f 1 = p 1 x 1 + q 1 x 2 + r1 f 2 = p 2 x 1 + q 2 x 2 + r2

Layers: 0

Copyright 2002 by Piero P. Bonissone

16

ANFIS-Visualized (Example for n =2)

where A1: Medium; A2: Small-Medium B1: Medium; B2: Small-Medium

Fuzzy reasoning
A1 B1
w1

y1 = p1*x1 +q1*x2+r1 y2 = p2*x1+q2* x2 +r2

z=

A2 x y

B2

w1*y1+w2*y2 w1+w2

w2

ANFIS (Adaptive Neuro-Fuzzy Inference System)

A1

x y

A2 B1 B2

w1

w1*y1 w2*y2

wi*yi
wi

Y
17

w2

Copyright 2002 by Piero P. Bonissone

Layer 1: Calculate Membership Value for Premise Parameter

Output O1,i for node i=1,2 Output O1,i for node i=3,4

O1,i = Ai (x1 )

where A is a linguistic label (small, large, )

O1,i = Bi 2 ( x 2 )

A ( x1 ) =

1 x1 c i 1+ ai
2b

Node output: membership value of input

Copyright 2002 by Piero P. Bonissone

18

Layer 1 (cont.): Effect of changing Parameters {a,b,c}

(a ) C h a n ging 'a ' 1 0.8 0.6 0.4 0.2 1 0 .8 0 .6 0 .4 0 .2 -5 0 5 10 0 -1 0 -5 0 5 10 (b ) C ha n gin g 'b '

A (x) =
1+

1 x ci ai
2b

0 -10

(c ) C h a n ging 'c ' 1 0.8 0.6 0.4 0.2 0 -10 -5 0 5 10 1 0 .8 0 .6 0 .4 0 .2 0 -1 0

(d ) C ha ngin g 'a ' a n d 'b '

-5

10

Copyright 2002 by Piero P. Bonissone

19

Layer 2: Firing Strength of Rule

Use T-norm (min, product, fuzzy AND, ...)

O2 ,i = wi = Ai (x1 ) Bi (x2 )
(for i=1,2)

Node output: firing strength of rule

Copyright 2002 by Piero P. Bonissone 20

Layer 3: Normalize Firing Strength

Ratio of ith rules firing strength vs. all rules firing strength

O 3 ,i

wi = wi = w1 + w 2
(for i=1,2)

Node output: Normalized firing strengths

Copyright 2002 by Piero P. Bonissone 21

Layer 4: Consequent Parameters

Takagi-Sugeno type output

O4,i = wi fi = wi ( pi x1 + qi x2 + ri )
Consequent parameters {pi, qi, ri} Node output: Evaluation of Right Hand Side Polynomials
Copyright 2002 by Piero P. Bonissone 22

Layer 5: Overall Output

O 5 ,1 =

!
i

wi fi =

!w f !w
i i i i

Note:
Output is linear in consequent parameters p,q,r:
= w1 w2 f1 + f2 w1 + w 2 w1 + w 2

= w 1 ( p 1 x 1 + q 1 x 2 + r1 ) + w

= w 1 x 1 p 1 + w 1 x 2 q 1 + w 1 r1 + w 2 x 1 p 2 + w 2 x 2 q 2 + w

( )

(p 2 x1

+ q 2 x 2 + r2

( )r
2

Node output: Weighted Evaluation of RHS Polynomials

Copyright 2002 by Piero P. Bonissone 23

ANFIS Network

Inputs

IF-part

Rules + Norm

THEN-part

Output

1
& N

Input 1

x1
& N

& N

Output

Input 2

x2
& N

Layers:

Copyright 2002 by Piero P. Bonissone

24

ANFIS Computational Complexity

Layer #
L0 L1 L2 L3 L4 L5

L-Type # Nodes
Inputs Values Rules Normalize Lin. Funct. Sum n (pn) pn pn pn 1

# Param
0 3(pn)=|S1| 0 0 (n+1)pn=|S2| 0

Copyright 2002 by Piero P. Bonissone

25

ANFIS Parametric Representation

ANFIS uses two sets of parameters: S1 and S2
S1 represents the fuzzy partitions used in the rules LHS
S1 = { a11, b11, c11} ,{a12, b12, c12} ,...,{ a1p , b1p ,c1p } ,...,{ anp, bnp, cnp}

S2 represents the coefficients of the linear functions in the rules RHS

S2 =

{c {

10 , c11 ,..., c1n

c p 0 , c p 1 ,..., c p n } },..., {
n n n

}
26

Copyright 2002 by Piero P. Bonissone

ANFIS Learning Algorithms

ANFIS uses a two-pass learning cycle
Forward pass:
S1 is fixed and S2 is computed using a Least Squared Error (LSE) algorithm (Off-line Learning)

Backward pass:
S2 is fixed and S1 is computed using a gradient descent algorithm (usually Back-propagation)

Copyright 2002 by Piero P. Bonissone

27

Structure ID & Parameter ID

Hybrid training method
nonlinear parameters
A1

linear parameters
w1*y1

w1

x1 x2

A2 B1 B2

wi*yi
w4 w4*y4

wi

/
A1 B1 A2 B2

Y
x2
B2 B1

Input space partitioning x1 x2

A1

A2 x1

Forward stroke Backward stroke MF param. (nonlinear) Coef. param. (linear) fixed least-squares steepest descent fixed
28

Copyright 2002 by Piero P. Bonissone

ANFIS Least Squares (LSE) Batch Algorithm LSE used in Forward Stroke:
Parameter Set: S = {S1U S 2}, and {S1 I S 2 = }
Output = F I , S , where I is the input vector

( )

H (Output ) = H o F I , S , where H o F is linear in S2

( )

For given values of S1, using K training data, we can transform the above equation into B=AX, where X contains the elements of S2 This is solved by: (ATA)-1AT B=X* where (ATA)-1AT is the pseudo-inverse of A (if ATA is nonsingular) The LSE minimizes the error ||AX-B||2 by approximating X with X*
Copyright 2002 by Piero P. Bonissone 29

ANFIS LSE Batch Algorithm (cont.)

Rather than solving directly (ATA)-1AT B=X* , we resolve it iteratively (from numerical methods):
" $ Si +1 = Si , T $ 1 + a( i +1) Si a( i+1) $ $ # for i = 0,1,..., K 1 $ T T Xi+1 = Xi + S(i +1) a(i +1) ( b( i+1) a( i+1) Xi ) $ $ $ % Si a(i +1) a(T i +1) Si
where:
X 0 = 0, S 0 = I , (where is a large number )
th aT i = i line of matrix A th bT i = i element of vector B

X * = Xk
Copyright 2002 by Piero P. Bonissone

30

ANFIS Back-propagation
Error measure Ek (for the kth (1kK) entry of the training data)
Ek =
where : N(L) = number
N ( L) i =1 2 ( d x ) ! i L ,i

nodes in layer L of desired of actual output output vector vector

d i = i th component x L , i = i th component

Overall error measure E:

E =
k=1

Ek
31

Copyright 2002 by Piero P. Bonissone

ANFIS Back-propagation (cont.)

For each parameter i the update formula is:

+E = i

where :

+ E ( ) ! ) & & i * i ' is the step size

is the learning rate

+E is the ordered derivative i

Copyright 2002 by Piero P. Bonissone 32

ANFIS Pros and Cons

ANFIS is one of the best tradeoff between neural and fuzzy systems, providing:
smoothness, due to the FC interpolation adaptability, due to the NN Backpropagation

ANFIS however has strong computational complexity restrictions

Copyright 2002 by Piero P. Bonissone

33

ANFIS Pros
Characteristics
Translates prior knowledge into network topology & initial fuzzy partitions Network's first three layers not fully connected (inputs-valuesrules)

Advantages
Faster convergency than typical feedforward NN

Smaller fan-out for Backprop

++

Induced partialorder is usually preserved

Smaller size training set

+++

Uses data to determine rules RHS (TSK model)

Network implementation of Takagi-Sugeno-Kang FLC

Model compactness (smaller # rules than using labels)

Copyright 2002 by Piero P. Bonissone

34

ANFIS Cons
Characteristics
Translates prior knowledge into network topology & initial fuzzy partitions

Disadvantages
Surface oscillations around points (caused by high partition number)

Sensitivity to initial number of partitions "

--

P"

Sensitivity to number of input variables " n "

Spatial exponential complexity: ^n # Rules = P

--

Uses data to determine rules RHS (TSK model)

Partial loss of rule "locality"

Coefficient signs not always consistent with underlying monotonic relations

Copyright 2002 by Piero P. Bonissone

35

ANFIS Pros (cont.)

Characteristics
Uses LMS algorithm to compute polynomials coefficients Uses fuzzy partitions to discount outlier effects Automatic FLC parametric tuning Uses Backprop tune fuzzy partitions to Error pressure to modify only "values" layer

Advantages

+++

Uses FLC inference mechanism to interpolate among rules

Smoothness guaranteed by interpolation

++

Copyright 2002 by Piero P. Bonissone

36

ANFIS Cons (cont.)

Characteristics
Uses LMS algorithm to compute polynomials coefficients Batch process disregards previous state (or IC)

Disadvantages
Not possible to represent known monotonic relations

Uses Backprop tune fuzzy partitions

to

Error gradient calculation requires derivability of fuzzy partitions and T-norms used by FLC

Cannot use trapezoids nor "Min"

Based on quadratic error cost function

Symmetric error treatment & great outliers influence

--

Uses FLC inference mechanism to interpolate among rules

Uses convex sum: f ( X)/ i i i

"Awkward" interpolation between slopes of different sign

Copyright 2002 by Piero P. Bonissone

37

ANFIS Opportunities
Changes to decrease ANFIS complexity
Use dont care values in rules (no connection between any node of value layer and rule layer) Use reduced subset of state vector in partition tree while evaluating linear functions on complete state
X = (X r U X ( nr ) )

Use heterogeneous partition granularity (different partitions pi for each state variable, instead of p)
# RULES = pi
i =1
Copyright 2002 by Piero P. Bonissone 38

ANFIS Opportunities (cont.)

Changes to extend ANFIS applicability
Use other cost function (rather than SSE) to represent the users utility values of the error (error asymmetry, saturation effects of outliers,etc.) Use other type of aggregation function (rather than convex sum) to better handle slopes of different signs.

Copyright 2002 by Piero P. Bonissone

39

ANFIS Applications at GE

Margoil Oil Thickness Estimator Voltage Instability Predictor (Smart Relay) Collateral Evaluation for Mortgage Approval Prediction of Time-to-Break for Paper Web

Copyright 2002 by Piero P. Bonissone

40

ANFIS References
ANFIS: Adaptive-Network-Based Fuzzy Inference System, J.S.R. Jang, IEEE Trans. Systems, Man, Cybernetics, 23(5/6):665-685, 1993. Neuro-Fuzzy Modeling and Control, J.S.R. Jang and C.-T. Sun, Proceedings of the IEEE, 83(3):378-406 Industrial Applications of Fuzzy Logic at General Electric, Bonissone, Badami, Chiang, Khedkar, Marcelle, Schutten, Proceedings of the IEEE, 83(3):450-465 The Fuzzy Logic Toolbox for use with MATLAB, J.S.R. Jang and N. Gulley, Natick, MA: The MathWorks Inc., 1995 Machine Learning, Neural and Statistical Classification Michie, Spiegelhart & Taylor (Eds.), NY: Ellis Horwood 1994 Classification and Regression Trees, Breiman, Friedman, Olshen & Stone, Monterey, CA: Wadsworth and Brooks, 1985
Copyright 2002 by Piero P. Bonissone 41