NEURAL NETWORK

NEOCOGNITRON
1

OVERVIEW
Theneocognitron is :
proposed by Kunihiko Fukushima in the 1980s
Inspired by the mammalian visual system
Hierarchical
Multilayered
It has been used forhandwritten character
recognition and otherpattern recognitiontasks, and
served as the inspiration forconvolutional neural
networks

OVERVIEW
Different versions of the neocognitron
learning without a teacher

often called self-

organized
learning with a teacher
Themain advantageof neocognitron is its ability to recognize
correctly not only learned patterns

TASK
Hierarchical feature extractionis the basic principle of the
neocognitron
The simplest features are extracted in the first stage and in each of
the following stages the more complex features are extracted
Joining between features in adjoining stages represents what
features from previous layer are used during extraction of the
certain feature
4

TASK

NETWORK STRUCTURE
Structure of the neocognitron arises from a hierarchy of stages
Totalnumber of stagesof the neocognitron depends on the
complexity of recognized patterns

NETWORK STRUCTURE
Each stage of the neocognitron consists of certain number oflayers

NETWORK STRUCTURE
Cell :
we denote the basic operating element
The cell is neuron-like element
processes input values according to the certain rule
and produces corresponding non-negative output
value
there are elementary cells of four distinct types
which differ by their function
8

INPUT LAYER
Input layeris intended forretaining of presented pattern
This layer is formed by array ofreceptor cells
receptor cells : simplest type ofcellin the neocognitron . Its
function is onlyto retain one pixel of presented pattern

S-LAYER
EachS-layerin the neocognitron is intended for extraction
of featuresfrom corresponding stage of hierarchy.
Particular S-layers are formed by distinct number ofS-planes.
Number of these S-planes depends on the number of
extractedfeatures.
EachS-planein the neocognitron is intended for
extractionof certainfeature.
Particular S-planes are formed by array ofS-cells

10

S-LAYER
S-cellis the most important type ofcellin the neocognitron.
Its function isto extractcertainfeatureat certain position
ininput layer, i.e. in itsreceptive field.
All the cells in oneS-planealways extract the same feature
receptive field: we denote area in theinput layer formed by
all the cells which can affect output value of thiscell

11

V-LAYER
EachV-layerin the neocognitron is intended forobtaining of informations about
average activityin previousC-layer(orinput layer).
Particular V-layers are always formed by only oneV-plane
EachV-planein the neocognitron is intended forobtaining of information
about average activityin previousC-layer(orinput layer).
Particular V-planes are formed by array ofV-cells.
Function of eachV-cellin the neocognitron isto obtain information about
average activityofC-cells(orreceptor cells) fromconnection areas. This
information is used by correspondingS-cellfrom the same stage of the network
duringfeature extraction.

12

V-LAYER
EachV-layerin the neocognitron is intended forobtaining of informations about
average activityin previousC-layer(orinput layer).
Particular V-layers are always formed by only oneV-plane
EachV-planein the neocognitron is intended forobtaining of information
about average activityin previousC-layer(orinput layer).
Particular V-planes are formed by array ofV-cells.
Function of eachV-cellin the neocognitron isto obtain information about
average activityofC-cells(orreceptor cells) fromconnection areas. This
information is used by correspondingS-cellfrom the same stage of the network
duringfeature
connection area : we denote area that is formed by group of cells from previous
layer outputs

13

C-LAYER
EachC-layerin the neocognitron is intended for ensuring of tolerance of shifts of
featuresextracted in previousS-layer.
Particular C-layers are formed by distinct number of C-planes . Their number
depends on number offeatures extracted in previous S-layer
EachC-planein the neocognitron is intended for ensuring of tolerance of shifts
of features extractedin correspondingS-plane
Particular C-planes are formed by array ofC-cells
EachC-planein the highest stage of the network contains only onecell

14

V-CELL FUNCTION
EachV-cellin the neocognitronevaluates outputs ofC-cells(orreceptor
cells) from the certainconnection areasfrom previousC-layer(orinput layer)
Size of connection areasis the same for allV-cellsandS-cellsin one stage of
the network
it is determined at construction of the network
V-cell output valuerepresents average activity of cells fromconnection
areasand it is used for inhibition of correspondingS-cellactivity
connection area : we denote area that is formed by group of cells from
previous layer outputs of which the given cell evaluates.
Exact specification of V-cell function is described inmathematical descriptionof
its behaviour.

15

16

V-CELL FUNCTION

17

S-CELL FUNCTION
EachS-cellin the neocognitronevaluates outputs ofC-cells(orreceptor cells)
from the certainconnection areasfrom previousC-layer(orinput layer).
Size of connection areasis the same for allS-cellsin oneS-layerand it is
determined at construction of the network
Function ofeachS-cellis toextractthe certainfeatureat the certain
position in the input layer
For extraction of this feature an S-cell uses only informations obtained from its
connection areas and information about average activity in these areas
obtained from correspondingV-cell

18

Cell plane of S-cells designated forextractionof feature corresponding to vertical line

is shown infigure

19

S-cell output valueis determined exactly by the equation described inmathematical description:

20

 selectivitywe denote parameter by which it is possible to affect the ability ofScelltoextract also deformed representations of the givenfeature

21

C-CELL FUNCTION
EachC-cellin the neocognitronevaluates outputs ofS-cellsfrom the
certainconnection areafrom one ofS-planesfrom previousS-layer. Number of
S-planes, however, can be greater in some cases
Size of connection areasis the same for allC-cellsin oneC-layerand it is
determined at construction of the network

22

C-CELL FUNCTION
EachC-cellin the neocognitronevaluates outputs ofS-cellsfrom the
certainconnection areafrom one ofS-planesfrom previousS-layer. Number of
S-planes, however, can be greater in some cases
Size of connection areasis the same for allC-cellsin oneC-layerand it is
determined at construction of the network
C-cell output valuedepends on activity ofS-cellsfromconnection area. The
greater number of active S-cells is or the greater their activities are the
greater C-cell output value

23