ACOE402NeuralNetworksandFuzzyLogic

ACOE
402 Neural Networks and Fuzzy Logic
ArtificialNeuralNetworks
EfthyvoulosC.Kyriacou(PhD)Assoc.Prof.
ComputerScienceandEngineeringDepartment
Resources:

Dr.Neocleous Costas,CyprusUniv.ofTechnology
NeuralNetworks,S.Haykin,Prenticehall1999
FundamentalsofNeuralNetworks,L.Fausett,Prentice hall1994

ComputationalIntelligence(CI)isaspecialcaseof
A ifi i l I lli
ArtificialIntelligence(AI)
(AI)
What is intelligence?
Whatisintelligence?
It is difficult to be clearly defined
Itisdifficulttobeclearlydefined.
It may not be needed to be defined to be understood
Itmaynotbeneededtobedefinedtobeunderstood.
Terman*suggeststhatitis
gg
thecapacityforabstractthought.
p y
g
Mostpsychologiststhough,agreethatitisthecapacityfor
effectiveadaptation toanenvironment, whichisdonethrough
changeintheorganism,orchangeintheenvironment,oreven
creationofanewenvironment.
ti
f
i
t
__________________________________________________________________________________________________
*TermanL.(1916).Theusesofintelligencetests.nThemeasurementofintelligence.Boston,HoughtonMifflin. 2

Hereisamorepracticalsuggestion:
Intelligenceisthegeneralmentalability involvedin
processessuchascalculation,reasoning,
h
l l ti
i
classification,learning,usinglanguage,
understandingtheenvironment,selfcorrecting,
d
d
h
lf
inventingandadjustingtonewsituations.
ArtificialIntelligencethenishumancreated,non
g
,
biologicalintelligence.
ComputationalIntelligenceisintelligencethatemerges
through some form of computation
throughsomeformofcomputation.
3

AIandCImayleadtosolutionsofdifficult
y
problemsinscienceandengineering.
Theyhavealsoinstigatedarevitalizationof
fundamental philosophical questions
fundamentalphilosophicalquestions.
Theyalsogeneratednewfrontiersofexplorations
They
also generated new frontiers of explorations
andintroducednewquestions.

Fundamental questions in relation to AI

FundamentalquestionsinrelationtoAI
Canabiologicalbraingenerateanartificialbrainand
i t lli
intelligence?
?
C th h
Canthehumanbrainunderstanditsownfunctioning?
b i d t d it
f ti i ?
Will
Willsuchunderstandingbeabsolute?
h d t di b b l t ?
Canourbrain,eithersinglyorcollectivelygenerateanew
C
b i ith i l
ll ti l
t
kindofintelligence?
HowisWILLgenerated?
5

What is consciousness?
Whatisconsciousness?
Can universal consciousness exist?
Canuniversalconsciousnessexist?
How does creativity appear?
Howdoescreativityappear?
How are illusions love hate generated?
Howareillusions,love,hate,generated?
Can matter generate cognition ? And how?
Canmattergeneratecognition?Andhow?

Wh t i
Whatismemory?
?
Can there be olle ti e memor (or national) memor ?
Cantherebecollectivememory(ornational)memory?
How are numbers words meaning codified?
Howarenumbers,words,meaningcodified?
How does the brain process logic?
Howdoesthebrainprocesslogic?
Howcanthebrainhandlewitheasebothcrispandfuzzy
How
can the brain handle with ease both crisp and fuzzy
logic?

Computational Intelligence Tools

ComputationalIntelligence
Expert systems(ExS)
Artificialneuralnetworks (ANN)
Evolutionarysystems(ES)
Geneticalgorithms(GA)
Artificiallife(AL)
Coevolutionaryrobotics
Evolvablehardware
Fuzzysystems(FS)

IbelievethattheArtificialNeuralNetworks are
presentlythemostpromisingtechnologythatmay
lead to nonprogrammed,
leadtonon
programmed,intelligentartificial
intelligent artificial
systems.

ANNs will be presented here in some detail

ANNswillbepresentedhereinsomedetail.

Artificial Intelligence
ArtificialIntelligence
Weak
approach

Humanknowledgeand
specificexpressionsofitcan
besimulatedthrough
computationalsystems

Strong
approach

Humanknowledgeand
specificexpressionsofit,
emergenaturallyin
computationalsystems

ClassicAI (symbolicAI)

ConnectionistAI

Itisexpressedthroughsymbolicentities
thatmaybeproperlycoded

Expressedthroughnonsymbolicmethods such
astheANN.Thevariouscognitiveprocesses
emergenaturallyfromthedynamically
connectedcells,astheyevolvewhen
influencedfromanenvironment.

Someimportantproponentsare:
Dennet,Newell and Simon,Chomski,Minsky,
Fodor and Pylyshyn
Fodorand

Someimportantproponentsare:
Smolenskyand Hameroff.

10

Brief explanation of the methods

Briefexplanationofthemethods
E
Expert
t systems
t
AlsoknownasKnowledgeBasedSystems
Itisasystemthatdescribesthebehaviour ofoneormore
experts insomefield,byusingsymbolsand rules.
The setofrulesisappliedtoaknowledgebase,aimingat
Th
f l i
li d
k
l d b
i i
extractingusefulinformation.
Therearelimitationsbecauseoftheneedtohaverulesfor
the rules of the exponentially growing size and the difficulty
therules,oftheexponentiallygrowingsize,andthedifficulty
infindingtherules.
11

MaincharacteristicsofExpertSystems:
p
y
Theprogrammingandtheconclusionsarebasedonrules.
Theyarenoteasilymadetoleranttomistakes.
Iftheknowledgebasechanges,reprogrammingmaybe
If
the knowledge base changes reprogramming may be
needed.
Explanationofresultsiseasiertoimplementwiththese
systems.

12

Brief explanation of Artificial Neural Networks

BriefexplanationofArtificialNeuralNetworks
Introduction
Biologicalneurons
g
Biologicalneuralnetworks
Artificialneurons
Artificialneuralnetworks
Learning
Applications
pp
13

Othernames fortheArtificialNeuralNetworks
Artificial
A
tifi i l Neural
N
l Systems
S t
Parallel Distribution Processing Systems
Connectionist Systems
Neurocomputing Systems
Adaptive Networks
Associative Networks
Collective Computation Systems
NeuromorphicSystems
14

TheANNsarebasedontheefforttomimicthe
operationalpropertiesofthenatural(biological)neural
(
)
networks.
Theyarecomposedofmanyartificialneurons
connectedinanorganizedsysteminwhichthereis
directorindirectcommunicationandinteraction
amongallitsmembers.
Thereisusuallyprovisionforinformationinput andfor
the desired output
thedesiredoutput.
Groups of neurons may organized into layers orslabs.
Groupsofneuronsmayorganizedintolayers
or slabs
15

AnArtificialNeuralNetwork
An emerging
A
i global
l b l behaviormayappearthroughtheuseof
b h i
th
h th
f
simple,andlocal learningandevolutionrules.
Artificial neural network
Artificialneuralnetwork
Processingelement
SYSTEM
SYSTEM
INPUT

SYSTEM
SYSTEM
OUTPUT

N1

N4

N6

N3

N2

N5

N7

N8

Signalflow

16

Togainabetterunderstandingoftheirfunctioning,letushaveabrief
encyclopaediclookonthebiologicalneuralnetworks.
Synapse

Typical
biological
neuron

Dendriteofreceivingneuron
xonofsendingneuron

Nucleus
Cellbodysoma
Internalvoltage 60to 80mV

Membrane
Thickness 5to10nm=0.000005to0.00001mm
Capacitance 0.944F/cm2
Fieldintensity 12000000V/m=120kV/cm

AXONHILLOCK
INITIALSEGMENT
TRIGGERZONE
NODEOFRANVIER

yelinsheath
xonornervefiber
Diameter:0.5 22minvertebrate
500 1000minthesquidgiantaxon

DENDRITEOFRECEIVINGNEURON

17

INFORMATIONTRANSMISSION
(Generation of the Action Potential)
(GenerationoftheActionPotential)
Excitationfrom
theenvironment

Habituatio
n

Stimulus

Sensory
receptor

Time

Synapse
Sy
apse
mV

Thegradedpotential is
GradedPotential graduallyweakened asitis
propagated towards the soma
propagatedtowardsthesoma.

Time

Neuron

mV
ActionPotentiall
Time

Anotherneuron

Thereceptivesensors convert
changesintheenvironment
(light pressure chemical
(light,pressure,chemical
constituency,...) inGraded
Potential.

Manygradedpotentialsfrom
differentdendritesare
accumulatedtocreateabigger
orsmallerone,which,onceit
reachesathreshold(40mV),
it becomes an Action Potential
itbecomesanActionPotential
(atrainofpulsedvoltages,
nerveimpulses orspikes)

18

Synapse

Diameter 1m=0.001mm
Gap20to40nm
=0.00002to0.00004mm
Diameter/Gapratio 100
Delayinthetransmissionofthe
presynapticpotentialtoapost
synapticpotential 0.3to1.0
ms
Velocity 0.2cm/minute

19

Themediumforthesignaltransmissionisbasicallyof
electrochemicalnature.
Thereexistsagreatvarietyofsynapses.
Eventhepositionofasynapsemakesadifferenceinsignal
transmission.
transmission
y p
Thetransmissionofcodedinformationatthesynapsesis
primarilydonewithchemicalsubstancesknownas
neurotransmitters [e.g. acetylcholine,norepinephrine,
d
dopamine,5hydroxytryptamine
h d
serotonin,aminobutyricacid
b
d
GABA].
Thesephenomenaaremainlyoccurringinmammalnervous
systems.
y
20

Comparisons
Thereareabout1010 1012 neuronsinthebrain
andabout1013 1016 synapses.

Organism

leech

Worm

Fly

Cockroach

Bee

Man

Numberof
Synapses

>104

>105

~109

<1011

>1011

~1014
21

Therealsoexistelectricalsynapses,buttheseoccurmainlyinlower
There
also exist electrical synapses, but these occur mainly in lower
animals.

Thesystematicuseofasynapseisbelievedto
improve its efficacy
improveitsefficacy.

learning,memory
g,
y

HebbsRule

22

Mathematical models of the biological neuron

Mathematicalmodelsofthebiologicalneuron
TheHodgkin
g
Huxleymodel,1952
y
,
Cm(dV/dt)=I
(dV/dt) Img
gNam3h(VV
h(V VNa)g
) gKn4(VV
(V VK)g
) gL(VV
(V VL)
dm/dt = m(1 m) mm
dn/dt = n(1 n) nn
dh/dt = h(1 h) hh

23

The Hodgkin Huxleymodel

TheHodgkin
Huxley model
Where,
Vrest
Vna
VK
VL
gNa
gK
gL

= 60mV
= 50mV
= 77mV
= 54.402mV
54 402 V
= 120mmho/cm2
= 36mmho/cm2
= 0.3mmho/cm
=
0 3 mmho/cm2

E =V Vrest
25 E/10 1 ,
m =0.125 E/e
/ 25E/10

E/18
m =4eE/18

n =0.0110 E/e10E/10 1,

n =0.125eE/80

h =0.07eE/20,

h =1/1+e30E /10

24

The FitzHugh Nagumo model

TheFitzHughNagumomodel
ThisisasimplerformoftheHodgkin Huxleymodel

duj/dt = uj uj 3/3 zj + x
d j/dt = k1j zj + k2j uj + k3j
dz

25

BRAIN NERVOUSSYSTEM

26

BRAIN NERVOUSSYSTEM
9

Thebiologicalneuralnetworksarenonlinear
computational systems thatarecharacterizedbyahigh
computationalsystems
that are characterized by a high
degreeofparallelism,robustness andfaulttolerance.

Theyshowacapacityforlearning,generalization,and
handling vagueness.

Itisestimatedthatthetotallengthofthebrain
It
i ti t d th t th t t l l th f th b i
connectionsareabout 109 meters,whichisabout25
times the earth perimeter!
timestheearthperimeter!

27

Eventhoughthebrainisonlyabout2%ofthetotal
weightofaperson,itneedsabout 15%ofthetotal
bloodandabout 25%oftheinhaledoxygen.

Livingorganismsthatdonothavenervoussystems
h d
h
(bacteria,protozoa,invertebrate)canexhibitsomeform
of behavior memory and even learning
ofbehavior,memoryandevenlearning.
Forinstance,the
For
instance, the paramecium (asingle
(a singlecell
cellorganism)
organism)
candemonstratelearningbehavior,eventhoughitdoes
nothavebrainandsynapses.

28

Everyneuroncommunicates
locally anddirectly withmany
other neurons and indirectly
otherneuronsandindirectly
withallothers.

LAYER

LAYER

Groupsofneuronsform
subsystems,layers,slabs,
y
, y ,
,
LAYER

Theinformationtransmissioninsynapsesisdoneinparallel.Becauseofthis,the
frequencyofchangesisabout1016 persecond.Takingintoconsiderationthefact
thatateachsynapsemanyneurotransmittersarealsotransferred,theinformation
transmissionisimpressive.
29

Comparisonofbiological
and artificial neurons and networks
andartificialneuronsandnetworks
BIOLOGICAL
NEURONS
ANDNETWORKS

ARTIFICIAL
NEURONS
ANDNETWORKS

Denseconnections
~ (1012 neurons)(10
~(10
)(104 synapses)=
)
=1016 connections

Fewconnections

Singleneuronsaredifferenttooneanother
Single
neurons are different to one another
Modularstructures
Autonomouslocalinteraction

Mostlysimilartooneanother
Mostly
similar to one another
Partlymodular
Nonautonomous
Usually supervision is needed
Usuallysupervisionisneeded
Mostlyserialprocessing
Muchenergyconsumption
Mostlymathematicaloralgorithmicdescription
y
g
p

Parallelprocessing
Verylittleenergyconsumption
Nonmathematical
Non
mathematicaloralgorithmicoperation
or algorithmic operation

30

ArtificialNeuralNetworks
Theyareflexible,nonlinearsystems
They
are flexible non linear systems composedofmany
composed of many
cooperatingprocessors,that,amongothers,helpin:
Studyofbiologicalneuralsystems.
Dataprocessingandknowledgemining.
Mappingandrelationextraction.
M
i
d l ti
t ti
Classification,Patternrecognition.
Forecasting.
Forecasting.
Studyofdynamicalsystems.
Adaptivesignalprocessing.
Automaticcontrolofsystems.

31

Itishopedthatthroughanautonomousevolutionof
It
is hoped that through an autonomous evolution of
suchconnectionistsystems,someformofintelligence
may emerge
mayemerge.
Will it lead to artificial brain?
Willitleadtoartificialbrain?
Artificial mind?
Artificialmind?
Artificial
Artificialconsciousness?
consciousness?

32

Artificialneurons
TheyarethebasicbuildingblocksofANNs.
They
are the basic building blocks of ANNs
Theyarealsoknownas:

Units or ProcessingElements

Theirmaincharacteristicsare:
Theyhavemultipleinputs oneoutput (MISO).
Theyarenonlinear.
They exhibit adaptivity
Theyexhibitadaptivity.
33

Theycanbeimplementedthroughappropriate
They
can be implemented through appropriate
software orhardware.
Thehardwareneuronscanbe:
Electronic,chemical,optical,micromechanical,
nanomechanical,...
nanomechanical
FFunctionally,theycanbesimulatedascellular
ti
ll th
b i l t d
ll l
automata.
Hereisageneralizedformofanartificialneuron.
34

Generalformofasingleneuronmodel
Inputinformationfromtheenvironment
orfromotherneurons

Outputtotheenvironmentortoother
neurons

PREACCUMULATOR
PROCESSING
(Subsystemoffunctional
anddynamicalprocessors)
(including cross
(includingcross
correlations),p1()

Main
Accumulator

POSTACCUMULATOR
PROCESSING
(Subsystemoffunctional
anddynamicalprocessors)
,p2()

DISTTRIBUTOR

Learningsystemthatadaptsthe
variousparameters
p

FFeedforward/Feedback
df
d/F db k
subsystem
h()

35

Someusefulpropertiesoftheartificialneuralnetworks
Theycangeneralize inthesensethattheyconcludeaboutanarea
of interest in which they hadnttbeenpreviouslytaught.
ofinterestinwhichtheyhadn
been previously taught
Theyshowrobustness. Theycontinuetobehaveacceptablywell
evenifsomeneuronsaredeleted,orsomeparametersmodified.
if
d l d
difi d
y
Theyarefaulttolerant.
Therulesareexpressedinadistributedprocessingand
representation manner.
manner
Theyusuallylearnthroughappropriatetraining,whichcanbe
guided(supervised)orunguided(unsupervised).
id d (
i d)
id d (
i d)

36

Structuring an ANN
StructuringanANN
Artificialneuralnetwork
Processingelement
SYSTEM
INPUT

SYSTEM
OUTPUT
TN1

TN 4

TN 6

TN 3

TN 2

TN 5

TN 7

TN 8

Signalflow
37

ANNTopologies,Architectures
NETWORKARCHITECTURES

STATICARCHITECTURES

DYNAMICARCHITECTURES

RANDOMCONNECTIONS

AUTONOMOUSLYADAPTED

ORGANIEDCONNECTIONS

EXTERNALLYGUIDED

ORGANIEDFEEDFORWARD
CONNECTIONS
(Withoutfeedback)
SINGLELAYER
TWOLAYER
THREELAYER
MULTI LAYER
ORGANIEDRECURRENT
CONNECTIONS
(With feedback)
(Withfeedback)
SINGLELAYER
TWOLAYER
THREELAYER
MULTI LAYER
CELLULARNETWORKS

38

SomeexamplesofANNarchitectures
Multilayerperceptron(MLP)
x1

y1 =sgn(u
= sgn(u1)

x2

xN

Weight,w

y2 =sgn(u2)

Forclarity,therestoftheweightsarenotshown

39

Multilayer,multislabnetwork

(hidden)
INPUT

OUTPUT
SLAB2

(hidden)

SLAB4

SLAB 5
SLAB5

SLAB1
(hidden)

SLAB3

40

Input
put
Vector

Weight
p(1)
Matrix
p1(1)
w(1a)

Receptive
p
Field

x1
.
.
.
xk
.
.
.
xK

f1
.
.
.
fk
.
.
.
fK

pI((1))

1(1)
.
.
.
i(1)
.
.
.
I((1))

p1(2)
.
.
.
pj(2)
.
.
.
pJ(2)

1(2)
.
.
.
j(2)
.
.
.
J(2)

.
.
.

IxK

p (1)
i

.
.
.

JxK

Weight
Matrix
w(2a)

fk =ake

(1)

p(2)

p(L) =w(La)f

i(1) =

j(2) =[pj(2)]

(2)

Weight
A(1)
Matrix
(1)
W(1b) A1
OxI

.
.
.

Ao(1)
.
.
.
AO((1))

OxJ

Weight
Matrix
W(2b)

A1(2)
.
.
.
Ao(2)
.
.
.
AO(2)

+
+

A1
.
.
.
Ao
.
.
.
AO

y1
.
.
.
yo
.
.
.
yO

A(2)

A(L) =W(Lb)(L)
41

Kohonenarchitectures

42

Recurrentarchitecture

43

Cellularneuralnetwork(CNN)architecture

Theartificialneurons
The
artificial neurons
communicateonlywiththeir
peripheral
peripheral.
44

Generalcommentsonneuralarchitectures
Itisnotnecessarytocodetheinformationprocessinginthe
y
p
g
formalismshowninthepreviousfigures/structures.
Wecoulduseanalgebraic/differentialformalism,orotherkind
We
could use an algebraic/differential formalism or other kind
ofgraphssuchas:

INPUT

HIDDENUNITS

OUTPUT

etc

45

LearninginANNs
Weightadaptation
x1()

Desiredoutput

w1()
d()

x2()

w2()

Error

u( ) = f xiwi
i =1

e()=d() u()

xN()

Comparison

wN(())

Thelearningalgorithmgovernsthewaythattheweightsadaptsothattheerrorgradually
The
learning algorithm governs the way that the weights adapt so that the error gradually
LEARNINGALGORITHM
becomesacceptablysmall.
46

LearninginANNs
Haykin (1994):
Learningisaprocessbywhichthefreeparameters(weights)ofa
neural network are adapted through a continuing process of
neuralnetworkareadaptedthroughacontinuingprocessof
stimulationbytheenvironmentinwhichthenetworkis
embedded.
embedded.

A more general proposition is:

Learningisachievedthroughanychange,inany
g
g
y
g ,
y
characteristic ofanetwork,sothatmeaningful
results are achieved.
resultsareachieved.
47

Thisdefinitionleadstothepossibilityofachieving
learning through:
learningthrough:
Synapticweightmodification.
Networkstructure(topology)modifications.
Bycreatingordeletingneuronsorconnections.
By creating or deleting neurons or connections
Throughtheuseofsuitableattractorsorother
suitablestablepoints.
it bl t bl
i t
Learningthroughforgetting.
Throughappropriatechoiceofactivationfunctions.
Throughmodifyingcontrollableparametersina
Through modifying controllable parameters in a
lookuptabledefininganactivation
transformation.
transformation
Combinationofabove.
48

Learningisusuallyaprocessofminimizinganerror
functionormaximizingabenefitfunction.
Inthisrespect,learningresemblestheoptimization,
whereweseektoextremizeadesiredcriterion
function(Error,Loss,Penalty,Computational
energy Lyapunov Hamiltonfunction)
energy,Lyapunov,
Hamilton function)

49

Learningrules
Some of the learning rules are significantly different than others, while
there are many that are of a minor variation, or are special cases of
more general ones,
ones usually identified by a different name.
name

Some important
p
rules are:
Hebb
Delta and its variations such as:
Backpropagation (the most common)
Adaline
Perceptron

Competitive learning
Hopfield
p
Kohonen
ART

50

Applicationsofartificialneuralnetworks
Iftheobjectiveistofindamappingsuchthatthesystem
capturesthedynamicsrelatinganinputdatasettoanoutput
dataset,theproblemiscalledfunctionapproximationor
systemidentificationincontrolengineeringapplications.
Iftheobjectiveistofindamappingsuchthesystemloosely
i
interpolateswithinaninputoutputdataset,theproblemis
l
i hi
i
d
h
bl
i
calledgeneralization.
Iftheobjectiveistofindamappingsuchthatthesystem
generalizestodiscreteoutputclasses,theproblemiscalled
classification.Insomefieldsthisproblemisgivendifferent
namessuchasassociativememories,patternassociation or
patternrecognition.
51

Applicationsofartificialneuralnetworks
In Engineering
Modeling
manufacturing
processes.
processes
Monitoringand
diagnosisofmachinery
failures.
= x1

Roboticsand
autonomousvehicles.
Automaticcontrol
systems.
Qualitycontrol.

& = x2

l
u

mc

x=x3

x& = x 4

52

InMedicineandHealthSciences
DiagnosticapplicationsinEEG,ECG,EMGandother
medical signatures
medicalsignatures.
Artificialvision.
Artificialhearing.
Artificialnose.
Artificial nose
Braincontrolledtelekinesis.
IdentifyingRNAandDNAinproteins.

53

Neuromusculardiseasediagnosis(UCY,ING)

54

QuantitativeAssessmentofthePrognosticFactorsinBreastCancer(UCY,ING)

Manual, semiquantitative, immunohistochemical score

%ofCells Score Staining Score Total Diagnosti
Positive
Intensity
Score
c
Index
0

Negative

025
25%
%

Weak

14

1+

2650% 2

Moderate

58

2+

5175% 3

Strong

3 912

3+

Very
Strong

4+

76%

>13

55

InSignalProcessing
Classification patternrecognition.
Si
Signalidentification,
l id ifi i
speechrecognition,
signaturerecognition,
i t
iti
imagerecognition,
speakeridentification.
k id tifi ti
Seismicimageanalysis.
Identificationof
explosives and military
explosivesandmilitary
targets.
Opticalcharacter
recognition.

56

In Meteorology
Modeling meteorological processes.
Filli i missing
Fillingin
i i rainfall
i f ll data
d (HTI,
(HTI UCY).
UCY)
y
Creation of isohyets.
In Cognitive studies
InCognitivestudies

AoccdrnigtoarscheearchatCmabrigdeUinervtisy,itdeosn
Aoccdrnig
to a rscheearch at Cmabrigde Uinervtisy it deosn'tt
mttaerinwahtoredrtheltteersinawrodare,theolny
iprmoetnt tihngistahtthefrist
tihng is taht the frist andlsatltteerbeattherghit
and lsat ltteer be at the rghit
pclae.Thersetcanbeatotalmsesandyoucansitllraedit
wouthitporbelm.Tihsisbcuseaethehuamnmniddeosnot
p
raederveylteterbyistlef,butthewrodasawlohe.
57

In Financial Engineering
Indecisionmaking.
Forecastingofmarketindices.(UCY,HTI)
F
i
f
k i di
(UCY HTI)
Stockexchangetrends.(HTI,UCY)
g
(
)
Bankrupsyprediction.(UCY)
Optimization.(HTI,UCY)

58

In Prediction and forecasting

Frauddetection.
Powerplantloadprediction.(HTI,UCY)
p
Potentialmineralexplorationsites.
InOptimization
Inproductdistribution(traveling salesmanproblems).
Indecisionmaking.
I d ii
ki
InSynthesis
Synthesisofmusicandspeech.
Synthesisofnewmaterials.
S h i f
i l
59

InDataclusteringandcompression
InTelecommunications,
Telemedicine,
Telemedicine
Geology,Meteorology

60

PossibleresearchareasinANNs
BasicResearch
Newlearningmethods
l
h d
Newarchitectures
Genetic algorithm learning
Geneticalgorithmlearning
Newsingleneuronmodels
Medicine
Diseasediagnosis
Engineering
Powerloadforecasting
l df
Meteorologicalapplications
Robotic and other complicated system control
Roboticandothercomplicatedsystemcontrol
Shipresistancecalculations

61

Dynamicalsystems
Conflictresolution,conflictavoidance(FuzzyCognitive
Maps?)
Dynamicanalysis (FuzzyCognitiveMaps?)
D ii
DecisionmakingdynamicsintheCyprusFederalState
ki d
i i th C
F d l St t
(FuzzyCognitiveMaps?)

Miscellaneous
Earthquakeprognosis
Athletic capability prognosis
Athleticcapabilityprognosis

62

ANNandStatisticalmethods

Thereexistssomeconflictamongneuroscientists
and statisticians on who invented first a particular
andstatisticiansonwhoinventedfirstaparticular
methodology!
Manyofthenon
Many
of the nondynamical
dynamicalmodelsusedinANNshave
models used in ANNs have
theirrootsinknownstatisticalmethodssuchas:
Generalizedlinearmodels
P l
Polynomialregression
i l
i
Nonparametricregression
Discriminantanalysis
Principalcomponents
Clusteranalysis
63

Comparisonofterminology
STATISTICS

ANN

Variables
Features
Independentvariables
Inputs
Predictedvalues
Outputs
Dependentvariables
d
i bl
Targetsortrainingvalues
i i
l
Residuals
Errors
Estimation
Training,learning,adaptation,orselforganization.
Estimationcriterion
Errorfunction,Costfunction,orLyapunov function
Observations
Patternsortrainingpairs
Parameterestimates
(Synaptic)weights
Interactions
Higherorderneurons
Transformations
Functionallinks
Regressionanddiscriminant analysis
Supervisedlearningorheteroassociation
Datareduction
Unsupervisedlearning,encoding,or
autoassociation
Clusteranalysis
Competitivelearningoradaptivevector
q anti ation
quantization
Interpolationandextrapolation
Generalization

64

Evolutionary Systems ES
EvolutionarySystems
Lately has been given great momentum
Latelyhasbeengivengreatmomentum.
TheyarusuallyknownasGeneticAlgorithms(GA)
Th
Theyarealgorithms
l ith foroptimizationandsearch.
f
ti i ti
d
h
Initiallywerestudiedby JohnHolland ofthe
University of Michigan back in the 70s.
UniversityofMichiganbackinthe70
s.
They resemble the natural evolution
Theyresemblethenaturalevolution.
65

Theymaybeusedin:
Function optimization.
Operationsresearch.
Learninginartificialneuralnetworks.
f l
l
k
Evolutionofnewartificialneuralnetworkarchitectures.
Evolutionoffuzzyrules.

Theyarerelativelysimple
y
y
p toapply,especiallyincomplicated
pp y, p
y
p
optimizationproblems.
They can find global optimization solutions
Theycanfindglobaloptimizationsolutions.
66

Somemoremoderntermsthatarebeingusedare:
EvolutionaryComputationor Evolutionary Algorithms.
Theseinclude:
GeneticAlgorithms GA
Genetic Programming GP
GeneticProgramming
EvolutionaryProgramming EP
Evolutionaryhardware EHW
Evolutionary Strategies ES
EvolutionaryStrategies
LearningClassifierSystems LCS
67

Artificial Life Systems ALS

ArtificialLifeSystems
Itisahighlysimplifiedsimulationofsomedynamicalfeaturesof
living organisms
livingorganisms.
Theyhavealotwiththewellestablishedscienceofcellular
automata.
t
t
Theyusesimplerulesforselforganization.
e.gConwaysGameofLife

ItiscommontoexperienceComplexity andChaotic problems.

68

Theycanshowanemergingbehaviour.
Canbeusedin:
Optimization problems
Optimizationproblems.
Neural network learning
Neuralnetworklearning.
...

ModernParticleSwarmSystems isanextensionofALS.

69

Fuzzy Systems (FS)

FuzzySystems(FS)
Thefuzzylogic isasupersetofthebistateBoolean
l i
logic.
Ithasbeendevisedinordertotacklethesocalled
It
has been devised in order to tackle the so called
halftruths andtheparadoxes,aswellasourneed
to have partly truth values
tohavepartlytruthvalues.

70

Someparadoxes:
Iamaliar!

(Epimenides)

AmIsayingthetruth?
Thebarberwhoshavesallthosewhodontshave
themselves
(Bertrand Russel)
(BertrandRussel)
Wh h
Whoshavesthebarber?
h b b ?

71

Fuzzy Systems
FuzzySystems
Example:Whenissomeonetall?
If P isthesetofallhumansandh theirheight,afuzzysubset(h) that
couldanswertheabovequestionis:
q

(h) =

0
(h 1,5)

if

h < 1,5 m

if

1,5 h 2,2 m

if

h > 2,2 m

0,7
1

Where (h) istheestablishedmembershipfunction

72

Differencebetweenfuzzyandconventional:
Inbinarylogic,theresultcanbetrueorfalse:
e g 1+1=2
e.g.1+1=2
Theworldismostlyfuzzy.
e.g.

Itscold,
It is green
Itisgreen,...

Therearevaguenessbecauseof probableevents and

b
becauseitisdifficulttofinddividinglinesamongeventsand
i i diffi l
fi d di idi li
d
objects.

73

FuzzySystems
Membershipfunctions
(h)
1

0.5

0
0.5

1.85

2.2

Height,h [m]

SSuchfunctionsmaybeconstructedforotherparameters,
hf
i
b
df
h
andgenerateafuzzylogicsystemwithrules:
74

FuzzySystems
SomemembershipfunctionsfromtheMATLABlibrary:
trapmf

gbellmf

trimf

gaussmf

gauss2mf

smf

1
0.8
0.6
0.4
02
0.2
0

zmf

psigmf

dsigmf

pimf

sigmf

0.8
0.6
0.4
0.2
0

75

FuzzySystems
y y
ANDfuzzyrule:

AND

76

MostapplicationsareinthefieldsofAutomatic
,
y
andinsupporting
pp
g
Control,inInformationSystems
Decisionmaking.

77

FuzzyCognitiveMaps
y g
p FCM
Itisakindof
combinationbetween
ANNsandfuzzylogic.
Theyaremadeofasetof
junctionsthatrepresent
conceptsandfromarrows
thatconnectthese
junctions.
Canbeusedincomplicatedproblemsinordertosimulatethe
dynamical interrelationships asinDecisionMaking
dynamicalinterrelationships
as in Decision Making andin
and in
StrategicPlanning.
Essentiallyarerecurrentdiagrams thatusefuzzyvalues.
78

AnattempttostudyinterrelationsintheCyprusProblem
Conceptsthathavebeenconsidered
Concepts
that have been considered
30 influencingparameters(concepts)havebeenexplored.
These are:
Theseare:
INFLUENCING PARAMETERS (Concepts)
C1

Welfare of the Federal State of Cyprus

C2

Welfare of the Greek Cypriot State

C3

yp
State
Welfare of the Turkish Cypriot

C4

Greek Cypriot nationalism

C5

Christian religiousness

C6

Knowledge of Turkish language by the Greek Cypriots

C7

Knowledge of Turkish history by the Greek Cypriots

C8

Educational level of the Greek Cypriots

C9

Turkish Cypriot nationalism

C10

Islamic religiousness
79

Conceptsthathavebeenconsidered

C11

Knowledge of Greek language by the Turkish Cypriots

C12

Knowledge of Greek history by the Turkish Cypriots

C13

Educational level of the Turkish Cypriots

C14

Political interests of Europe

C15

Political interests of USA

C16

Political interests of Russia

C1
C17

Political
li i l interests
i
off UK

C18

Political interests of Israel

C19

Political interests of Greece

C20

Political interests of Turkey

80

Conceptsthathavebeenconsidered

C21

Military interests of Israel

C22

Military interests of Greece

C23

Military interests of Turkey

C24

Military interests of UK

C25

Mili
Military
iinterests off USA

C26

Interests of Anatolian settlers

C27

Level of tourism in the federated state

C28

Oil fields

C29

Quality of environment

C30

Other natural resources

81

300%
Effects
EffectsofchangesinIslamicreligiousness,EuropeanandUSAinterests
of changes in Islamic religiousness European and USA interests

250%

Percentage
g change
g in parameters
p
due to major
j increase in Islamic Religiousness
g

es
rc
ou
te
es n t
sta
l r me
ed
ra
at
tu ron
er
na vi
ed
er f en
th
eF
O ty o
t h rs
li
in ett le
ua
Q elds ri sm an s
u li
fi
o
SA
t
to
il
O l of ana f U
o
ve of ts K
Le sts eres f U ey
o k
e
t
ter in ts ur
In ary eres f T ce
o
t
t
ili in ts ree
M ary res f G
l
e o
t
ili int ts rae
M ry res f Is y
e o ke
ta
r
ili int s
t
M ry res Tu e
f
e
ta
c
ili int s o ee
M ry rest Gr
f
l
ta e
ili nt s o ae
M al i rest f Isr
ic e o
lit nt s K
t
i
Po cal eres f U ia
i
o
lit int ts uss
Po cal eres f R
i
lit int ts o SA
Po cal res f U e
p
e
i
lit int ts o ro
TC
Po cal res f Eu C the C
e
i
lit int ts o e T by e T
Po cal res f th ry y t h
o
e
i
o
lit int el ist e b
Po cal lev ek h uag
i
l re ng
lit na
a
Po atio of G ek l
uc ge re
Ed led of G ss
C
w e ne
s
eG
no g
K led iou
C y th GC
w ig
e G b t he
no el
K i c r ism f th tory by
o is e
m al
la ion vel h h uag
Is
t
le i s g
a
N al urk lan
n
TC atio of T ki sh
uc ge ur
Ed led of T ness
w e us
no g
K ed i o
g
l
w eli
s
r
e
no
ru
K tian li sm Stat
yp
ris na C e
fC
t
Ch atio he T St a te o
a
t C
N
St
C of
G re he G eral
fa t ed
el of
W re he F
fa t
el of
W re
fa
el
W
-50%

Percentage change in parameters due to major increase in European interest

200%

P
Percentage
change
h
iin parameters due
d to major
j increase
i
in
i USA interest
i
150%

100%

50%

0%

82

CaseofincreaseintheIslamicreligiousness
ThisresultsinsignificantincreaseintheGreekCypriot
nationalism(+36%)andmostimportantlyaveryhighincrease
inChristianreligiousness(+283%).
Also,itwillresultinanincreaseinthemilitaryinterestsof
Al
it ill
lt i
i
i th
ilit
i t
t f
Turkey,GreeceandIsrael.
Ofsignificanceistheobservationthatthewelfareofthe
FederalStateofCypruswillreduceby7%andthewelfareof
yp
y
theTurkishCypriotcommunityby4%.
Somereductionintheleveloftourismandthequalityofthe
environmentisalsoobserved.
83

CaseofincreaseintheEuropean politicalinterestsinCyprus
Themodelpredictsasignificantincreaseoftheknowledgeof
GreeklanguageandhistorybytheTurkishCypriotsandvice
versa.
Thisismorepronouncedinthecaseofknowledgeofthe
Thi
i
d i th
fk
l d
f th
TurkishlanguageandhistorybytheGreekCypriots.
Itishoweververyinterestingthatthemodelsuggestsa
significantreductioninChristianreligiousness(28%)and
g
g
(
)
somewhatless(5%)intheIslamicreligiousness.
Mostinterestingthough,isthesuggestionthattheinterestsof
theAnatoliansettlerswillreduceby20%.
84

CaseofincreaseinthepoliticalinterestsofUSA inCyprus
Themodelpredictsa12%increaseinthepoliticalinterestsof
Israelanda12%increaseinthewelfareoftheTurkishCypriots.
Whatismostinterestingthough,andsomewhatunusual,isthe
reductionoftheChristianreligiousnessby16%,oftheTurkish
d ti
f th Ch i ti
li i
b 16% f th T ki h
Cypriotnationalismby18%,oftheIslamicreligiousnessby8%,
of the political interests of Greece and Turkey by 11% and 16%
ofthepoliticalinterestsofGreeceandTurkeyby11%and16%
respectively,andofthemilitaryinterestsofUKby18%.

85