Self Organising INCREMENTAL Neural Network

Contents What is SOINN Why SOINN Detail algorithm of SOINN SOINN for machine learning SOINN for associative
memory References
Self-organizing incremental neural network and its application

F. Shen1
1 National
O. Hasegawa2
Key Laboratory for Novel Software Technology, Nanjing University Science and Engineering Lab, Tokyo Institute of Technology
2 Imaging
June 12, 2009
F. Shen, O. Hasegawa
Contents What is SOINN Why SOINN Detail algorithm of SOINN SOINN for machine learning SOINN for associative memory References
Contents of this tutorial

1 2 3 4 5 6
What is SOINN Why SOINN Detail algorithm of SOINN SOINN for machine learning SOINN for associative memory References
What is SOINN
1 2
4 5
What is SOINN
What is SOINN
SOINN: Self-organizing incremental neural network Represent the topological structure of the input data Realize online incremental learning
What is SOINN
What is SOINN
What is SOINN
What is SOINN
What is SOINN
What is SOINN
Background Characteristics of SOINN
1 2
4 5
Background: Networks for topology representation
SOM(Self-Organizing Map): predene structure and size of the network NG(Neural Gas): predene the network size GNG(Growing Neural Gas): predene the network size; constant learning rate leads to non-stationary result.
Background: Networks for incremental learning
Incremental learning: Learning new knowledge without destroy of old learned knowledge (Stability-Plasticity Dilemma) ART(Adaptive Resonance Theory): Need a user dened threshold. Multilayer Perceptrons: To learn new knowledge will destroy old knowledge Sub-network methods: Need plenty of storage
Characteristics of SOINN
Neurons are self-organized with no predened network structure and size Adaptively nd suitable number of neurons for the network Realize online incremental learning without any priori condition Find typical prototypes for large-scale data set. Robust to noise
Architecture of SOINN Training process of SOINN Similarity threshold for judging input data Learning rate Simple version of SOINN Simulation results
1 2
4 5
Structure: Two-layer competitive network

Two-layer competitive network First layer: Competitive for input data Second layer: Competitive for output of rst-layer Output topology structure and weight vector of second layer




Training owchart of SOINN
Adaptively updated threshold Between-class insertion Update weight of nodes Within-class insertion Remove noise nodes
First layer: adaptively updating threshold Ti

Basic idea: within-class distance T between-class distance
1 2
Initialize: Ti = + when node i is a new node. When i is winner or second winner, update Ti by
If i has neighbors, Ti is updated as the maximum distance between i and all of its neighbors. Ti = max ||Wi Wc ||
cNi
(1)
If i has no neighbors, Ti is updated as the minimum distance of i and all other nodes in network A. Ti = min ||Wi Wc ||
cA\{i }
(2)

1 2
cNi
(1)
cA\{i }
(2)

1 2
cNi
(1)
cA\{i }
(2)

1 2
cNi
(1)
cA\{i }
(2)

1 2
cNi
(1)
cA\{i }
(2)

1 2
cNi
(1)
cA\{i }
(2)
Second layer: constant threshold Tc

Basic idea 1: within-class distance T between-class distance Basic idea 2: we already have some knowledge of input data from results of rst-layer. Within-class distance: dw = 1 NC ||Wi Wj ||
(i ,j)C
(3)
Between-class distance of two class Ci and Cj : db (Ci , Cj ) =

i Ci ,jCj
min
||Wi Wj ||
(4)

(i ,j)C
(3)

i Ci ,jCj
min
||Wi Wj ||
(4)

(i ,j)C
(3)

i Ci ,jCj
min
||Wi Wj ||
(4)

(i ,j)C
(3)

i Ci ,jCj
min
||Wi Wj ||
(4)

(i ,j)C
(3)

i Ci ,jCj
min
||Wi Wj ||
(4)
Second layer: constant threshold Tc (continue)

1
Set Tc as the minimum between-cluster distance. Tc = db (Ci1 , Cj1 ) =

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(5)
Set Tc as the minimum between-class distance. Tc = db (Ci1 , Cj1 ) =

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(6)
If Tc is less than within-class distance dw , set Tc as the next minimum between-cluster distance. Tc = db (Ci2 , Cj2 ) =
k,l=1,...,Q,k=l,k=i1 ,l=j1
min
db (Ck , Cl )
(7)
Go to step 2 to update Tc until Tc is greater than dw .

F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

1

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(5)

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(6)
k,l=1,...,Q,k=l,k=i1 ,l=j1
min
db (Ck , Cl )
(7)


1

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(5)

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(6)
k,l=1,...,Q,k=l,k=i1 ,l=j1
min
db (Ck , Cl )
(7)


1

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(5)

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(6)
k,l=1,...,Q,k=l,k=i1 ,l=j1
min
db (Ck , Cl )
(7)


1

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(5)

k,l=1,...,Q,k=l
min
db (Ck , Cl )
(6)
k,l=1,...,Q,k=l,k=i1 ,l=j1
min
db (Ck , Cl )
(7)

Updating learning rate 1 (t) and 2 (t)

Update of weight vector Ws1 Wi = 1 (t)( Ws1 ) = 2 (t)( Wi ) (i Ns1 ) (8) (9)
After the size of network becomes stable, ne tune the network stochastic approximation: a number of adaptation steps with a strength (t) decaying slowly but not too slowly, i.e., 2 t=1 (t) = , and t=1 (t) < . The harmonic series satises the conditions. 1 1 (t) = , t
2 (t) =
1 100t
(10)

2 (t) =
1 100t
(10)

2 (t) =
1 100t
(10)

2 (t) =
1 100t
(10)
Single-layer SOINN
For topology representation, rst-layer is enough Within-class insertion slightly happened in rst-layer Using subclass and density to judge if connection is needed.
Single-layer SOINN
Single-layer SOINN
Single-layer SOINN
Single-layer SOINN
Single-layer SOINN
Articial data set: topology representation

Stationary and non-stationary Stationary: all training data obey same distribution Non-stationary: next training sample maybe obey dierent distribution from previous one. Original data Stationary Non-stationary






Articial data set: topology representation (continue)

Original data Two-layer SOINN Single-layer SOINN
Conclusion of experiments: SOINN is able to Represent topology structure of input data. Realize incremental learning. AutomaticallyF.learnO.number of Self-organizing incremental neural network and its application nodes, de-noise, etc. Shen, Hasegawa







Unsupervised learning Supervised learning Semi-supervised learning Active learning
1 2
4 5
Some objectives of unsupervised learning
Automatically learn number of classes of input data Clustering with no priori knowledge Topology representation Realize real-time incremental learning Separate classes with low density overlapped area
SOINN for unsupervised learning: If two nodes connected with one path, the nodes belong to one class
1
Do SOINN for input data, output topology representation of nodes Initialize all nodes as unclassied. Randomly choose one unclassied node i from node set A. Mark node i as classied and label it as class Ci . Search A to nd all unclassied nodes that are connected to node i with a path. Mark these nodes as classied and label them as the same class as node i . Go to Step3 to continue the classication process until all nodes are classied.
2 3
1
2 3
1
2 3
1
2 3
1
2 3
1
2 3
Articial data set: 5 classes with 10% noise

Original data Clustering result
Conclusion of experiments Automatically reports number of classes. Perfectly clustering data with dierent shape and distribution. Find typical prototypes; incremental learning; de-noise; etc. its application F. Shen, O. Hasegawa Self-organizing incremental neural network and





Face recognition: AT&T face data set
Experiment results Automatically reports there are 10 classes. Prototypes of every classes are reported. With such prototypes, recognition ratio (1-NN rule) is 90%.
Prototype-based classier: based on 1-NN or k-NN rule

Nearest Neighbor Classier (NNC): all training data as prototypes Nearest Mean Classier (NMC): mean of each class as prototypes k-means classier (KMC), Learning Vector Quantization (LVQ), and others: predene number of prototypes for every class. Main diculty
1 2
How to nd enough prototypes without overtting How to realize Incremental learning

Incremental of new data inside one class (non-stationary or concept drift); Incremental of new classes.

1 2


1 2


1 2


1 2


1 2


1 2


1 2


1 2


1 2

SOINN for supervised learning: Targets
Automatically learn the number of prototypes needed to represent every class Only the prototypes used to determine the decision boundary will be remained Realize both types of incremental learning Robust to noise
Adjusted SOINN Classier (ASC)
SOINN learns k for k-means. Noise-reduction removes noisy prototypes Center-cleaning removes prototypes unuseful for decision
ASC: noise-reduction & center-cleaning
Noise-reduction If the label of a node diers from the label of majority voting of its k-neighbors, it is considered an outlier. Center-cleaning If a prototype of class i has never been the nearest prototype of other classes, remove the prototype.
Experiment results: articial data (I)

Original data SOINN results ASC results
Test results of ASC No. of prototypes = 6; Recognition ratio = 100%.









Experiment results: articial data (II)










Experiment results: articial data (III)

Test results of ASC No. of prototypes = 87; Recognition ratio = 97.8%.









Experiment results: optdigits

ASC with dierent parameter sets (ad , ), displayed with average of 10 times training and standard deviation Parameter set of {ad , } (50, 50) (25, 25) (10, 10) 97.7 0.2 97.4 0.2 97.0 0.2 377 12 258 7 112 7 9.9 0.3 6.8 0.2 2.9 0.2
recognition ratio (%) No. of prototypes Compression ratio (%)
Compare with SVM and 1-NN LibSVM: 1197 support vectors; Recognition ratio = 96.6%. 1-NN: best classier (98%). All 3823 samples as prototypes.



Experiment results: UCI repository data sets

Comparison results of ASC and other classiers: recognition ratio
Data set Iris Breast cancer Ionosphere Glass Liver disorders Pima Indians Wine Average ASC (ad , ) 97.4 0.86 97.4 0.38 90.4 0.64 73.5 1.6 62.6 0.83 72.0 0.63 82.6 1.55 82.3 0.93
2 NSC (max ) 96.3 0.4 97.2 0.2 91.9 0.8 70.2 1.5 62.9 2.3 68.6 1.6 75.3 1.7 80.4 1.2
KMC (M) 96.2 0.8 95.9 0.3 87.4 0.6 68.8 1.1 59.3 2.3 68.7 0.9 71.9 1.9 78.3 1.1
NNC (k) 96.7 0.6 97.0 0.2 86.1 0.7 72.3 1.2 67.3 1.6 74.7 0.7 73.9 1.9 81.1 0.99
LVQ (M) 96.1 0.6 96.3 0.4 86.4 0.8 68.3 2.0 66.3 1.9 73.5 0.9 72.3 1.5 79.9 1.2
In average, ASC has best recognition performance.

Experiment results: UCI repository data sets

Comparison results of ASC and other classiers: recognition ratio
Data set Iris Breast cancer Ionosphere Glass Liver disorders Pima Indians Wine Average ASC (ad , ) 97.4 0.86 97.4 0.38 90.4 0.64 73.5 1.6 62.6 0.83 72.0 0.63 82.6 1.55 82.3 0.93
2 NSC (max ) 96.3 0.4 97.2 0.2 91.9 0.8 70.2 1.5 62.9 2.3 68.6 1.6 75.3 1.7 80.4 1.2
KMC (M) 96.2 0.8 95.9 0.3 87.4 0.6 68.8 1.1 59.3 2.3 68.7 0.9 71.9 1.9 78.3 1.1
NNC (k) 96.7 0.6 97.0 0.2 86.1 0.7 72.3 1.2 67.3 1.6 74.7 0.7 73.9 1.9 81.1 0.99
LVQ (M) 96.1 0.6 96.3 0.4 86.4 0.8 68.3 2.0 66.3 1.9 73.5 0.9 72.3 1.5 79.9 1.2
In average, ASC has best recognition performance.

Experiment results: UCI repository data sets (continue)

Comparison results of ASC and other classiers: compression ratio
Data set ASC (ad , ) Iris 5.2 (6, 6) Breast cancer 1.4 (8, 8) Ionosphere 3.4 (15, 15) Glass 13.7 (15, 15) Liver disorders 4.6 (6, 6) Pima Indians 0.6 (6, 6) Wine 3.2 (6, 6) Average 4.6 2 NSC (max ) KMC (M ) 7.3 (0.25) 8.0 (4) 1.8 (35.0) 0.29 (1) 31 (1.25) 4.0 (7) 97 (0.005) 17 (6) 4.9 (600) 11 (19) 1.7 (2600) 1.0 (4) 96 (4.0) 29 (17) 34.2 10.0
NNC (k ) 100 (14) 100 (5) 100 (2) 100 (1) 100 (14) 100 (17) 100 (1) 100
LVQ (M ) 15 (22) 5.9 (40) 6.8 (24) 45 (97) 8.4 (29) 3.4 (26) 32 (57) 16.6
In average, ASC has best compression ratio.

Experiment results: UCI repository data sets (continue)

Comparison results of ASC and other classiers: compression ratio
Data set ASC (ad , ) Iris 5.2 (6, 6) Breast cancer 1.4 (8, 8) Ionosphere 3.4 (15, 15) Glass 13.7 (15, 15) Liver disorders 4.6 (6, 6) Pima Indians 0.6 (6, 6) Wine 3.2 (6, 6) Average 4.6 2 NSC (max ) KMC (M ) 7.3 (0.25) 8.0 (4) 1.8 (35.0) 0.29 (1) 31 (1.25) 4.0 (7) 97 (0.005) 17 (6) 4.9 (600) 11 (19) 1.7 (2600) 1.0 (4) 96 (4.0) 29 (17) 34.2 10.0
NNC (k ) 100 (14) 100 (5) 100 (2) 100 (1) 100 (14) 100 (17) 100 (1) 100
LVQ (M ) 15 (22) 5.9 (40) 6.8 (24) 45 (97) 8.4 (29) 3.4 (26) 32 (57) 16.6
In average, ASC has best compression ratio.

Requirement of Semi-supervised learning
Labeled instances are dicult, expensive, or time consuming to obtain. How can a system use large amount of unlabeled data with limited labeled data to built good classiers? New data are continually added to an already huge database How can a system learn new knowledge without forgetting previous learned knowledge?
SOINN used for Semi-supervised learning

1
SOINN:represent topology, incremental learning; Labeled data: label nodes (winner); Division of a cluster
Condition of division Rc1 Rc &Rc > Rc+1 Rc =

aNc
(11) (12)
dis(wa , wc )
c-1: former node c+1: unlabeled neighbors.


1

aNc
(11) (12)
dis(wa , wc )


1

aNc
(11) (12)
dis(wa , wc )


1

aNc
(11) (12)
dis(wa , wc )


1

aNc
(11) (12)
dis(wa , wc )

Experiment: original data
5%, 15%, or 40% overlap training samples 500, validation samples 5,000, and test samples 5,000 labeled samples: 10% and 20% light blue: unlabeled data; others: labeled data - - - - - ideal decision boundary
Experiment results
Separate classes with few labeled samples. For UCI data sets, work better than other typical methods.
Experiment results
Experiment results
SOINN used for active learning
Targets: Actively ask for label of some samples to label all classes Idea:
1 2 3 4
Use SOINN to learn the topology structure of input data. Actively label the vertex nodes of every class Use vertex nodes to label all nodes. Actively label the nodes lie in the overlapped area.
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
Experiment: articial data set under stationary environment
Original data: Four classes in all, with 10% noise. Results: under stationary environment; 10 teacher vectors.
Experiment: articial data set under non-stationary environment
16 teacher vectors are asked.

Experiment: articial data set under non-stationary environment
16 teacher vectors are asked.

Background SOINN-AM Experiments General Associative Memory
1 2
4 5
Background: typical associative memory systems

Distributed Learning Associative Memory:
Hopeld Network: most famous network, for auto-associative memory Bidirectional Associative Memory (BAM), for hetero-associative memory
Competitive Learning Associative Memory

KFMAM: Kohonon feature map associative memory.
Diculties Forget previously learned knowledge when learning new knowledge incrementally. Storage limitation. Memory real-valued data. Many-to-Many associate.




















Objectives of SOINN-AM
Incremental learning of memory pairs. Robust for noise data. Dealing with real-valued data. Many-to-many association.
Architecture of SOINN-AM
Algorithms of SOINN-AM
Basic idea of memory phase
1 2
Combine key vector and associate vector as input data. Use SOINN to learn such input data.
Basic idea of recall phase

1
Using key part of nodes to nd winner node for key vector, the distance is d. If d , output the associative part of winner as the recall results. If d > , report unknown for key vector.
1 2

1
1 2

1
1 2

1
1 2

1
1 2

1
1 2

1
1 2

1
Original data
Binary data
Real-valued data
Comparison with typical AM systems
Robustness of noise
Many-to-Many associate testing
SOINN-AM recalls all patterns perfectly.
Architecture and basic idea of GAM
Input layer: key vector and associate vector. Memory layer: Memory patterns with classes. Associate layer: Build association between classes.
1 2
4 5
References about SOINN

SOINN for unsupervised learning: Furao Shen and Osamu Hasegawa, An Incremental Network for On-line Unsupervised Classication and Topology Learning, Neural Networks, Vol.19, No.1, pp.90-106, (2005) Furao Shen, Tomotaka Ogura and Osamu Hasegawa, An enhanced self-organizing incremental neural network for online unsupervised learning, Neural Networks, Vol.20, No.8, pp.893-903, (2007) SOINN for Supervised learning: Furao Shen and Osamu Hasegawa, A Fast Nearest Neighbor Classier Based on Self-organizing Incremental Neural Network, Neural Networks, Vol.21, No.10, pp1537-1547, (2008)



SOINN for Semi-supervised and active learning Youki Kamiya, Toshiaki Ishii, Furao Shen and Osamu Hasegawa: An Online Semi-Supervised Clustering Algorithm Based on a Self-organizing Incremental Neural Network, IJCNN 2007, Orlando, FL, USA, August 2007 Furao Shen, Keisuke Sakurai, Youki Kamiya and Osamu Hasegawa: An Online Semi-supervised Active Learning Algorithm with Self-organizing Incremental Neural Network, IJCNN 2007, Orlando, FL, USA, August 2007 SOINN for Associative Memory: Sudo Akihito; Sato Akihiro; Hasegawa Osamu, Associative Memory for Online Learning in Noisy Environments Using Self-organizing Incremental Neural Network, IEEE Transactions on Neural Networks, (2009) in press


Download papers and program of SOINN http://www.isl.titech.ac.jp/ hasegawalab/soinn.html

Self Organising INCREMENTAL Neural Network

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Self Organising INCREMENTAL Neural Network

Hochgeladen von

Copyright:

Verfügbare Formate

Contents What is SOINN Why SOINN Detail algorithm of SOINN SOINN for machine learning SOINN for associative

Self-organizing incremental neural network and its application

June 12, 2009

Self-organizing incremental neural network and its application

Contents of this tutorial

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for topology representation

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for topology representation

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for topology representation

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for topology representation

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for incremental learning

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for incremental learning

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for incremental learning

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for incremental learning

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Background: Networks for incremental learning

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Background Characteristics of SOINN

Self-organizing incremental neural network and its application

Self-organizing incremental neural network and its application

Structure: Two-layer competitive network

Self-organizing incremental neural network and its application

Structure: Two-layer competitive network

Self-organizing incremental neural network and its application

Structure: Two-layer competitive network

Self-organizing incremental neural network and its application

Structure: Two-layer competitive network

Self-organizing incremental neural network and its application

Structure: Two-layer competitive network

Self-organizing incremental neural network and its application