2002-04-

24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Particle Swarm
optimisation: A mini
tutorial
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
The inventors (1)
Russell
Eberhart
eberhart@engr.iupui.edu
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
The inventors (2)
James
Kennedy
Kennedy_Jim@bls.gov
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Part 1: United we
stand
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Cooperation example
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Initialization. Positions
and velocities
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Neighbourhoods
geograp
hical
socia
l
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
The circular
neighbourhood
Virtual circle
1
5
7
6
4
3
8
2
Particle 1s
3-
neighbourh
ood
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Psychosocial
compromise
Here I
am!
The best
perf. of my
neighbour
s
My
best
perf.
x
p
g
p
i
v
i
-
p
r
o
x
i
m
i
t
y
g
-
p
r
o
x
i
m
i
t
y
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
The historical
algorithm
( )
( )
( ) ( ) ( )
( ) ( ) ( )

'

+
+
+
t x p rand
t x p rand
t v
t v
d d g
d d i
d
d
, 2
, 2
, 2
, 2
2

for each particle

update
the
velocity
( ) ( ) 2 ) 2 ( + + + t v t x t x
then move
for each component d
At each time step t
R
a
n
d
o
m
n
e
s
s

in
s
id
e

t
h
e
lo
o
p
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Random proximity
x
p
g
p
i
v
i
-
p
r
o
x
i
m
i
t
y
g
-
p
r
o
x
i
m
i
t
y
Hyperparallelepiped => Biased
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Animated illustration
Globa
l
optim
um
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Part 2: How to choose
parameters
T
h
e
rig
h
t w
a
y
T
h
is w
ay
O
r th
is w
a
y
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Type 1 form
2 2 2 2
' ' ) , 2 ( ) , 2 ( + + rand rand
2 2
2 2
' '
' '


+
+

g i
p p
p

'

+ + +
+ +
) ( ) 2 ( ) 2 (
))) ( ( ) ( ( ) 2 (
t x t v t x
t x p t v t v
with

'

>
+

2 for
2 2
2
2

else
Usual values:
=1
=4.1
=> =0.73
swarm size=20
hood size=3
N
o
n

d
i
v
e
r
g
e
n
c
e

c
r
i
t
e
r
i
o
n
G
lo
b
a
l
c
o
n
s
t
r
ic
t
io
n

c
o
e
f
f
ic
ie
n
t
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
5D complex space
2
2
2
-2
-2
2
2
2
-2
-2
2
2
2
2
2
2
2
}
}
Convergence
Non
diverg
ence
A 3D section

Re(y)
Re(v)
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Move in a 2D section
(attractor)
2
. 22 2 - . 22 -2
. 00
. 00
. 00
. 00
2
- . 22
- . 22
- . 22
- . 22
Im( v)
Re( v)
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Some functions ...
Rosenbrock
Griewank
Rastrigin
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
... and some results
D function 22 PSO Type " 2 Evolutionary
algo.(Angeline ) 22
Griewank [ ] 222 . 0000000 . 22222
Rastrigin [ ] 2 . 0000000 . 222222
Rosenbrock [ ] 22 . 00000000 . 2222222
Optimum=0, dimension=30
Best result after 40 000 evaluations
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Beat the swarm!
Y
o
u
r

c
u
r
r
e
n
t

p
o
s
i
t
i
o
n
Y
o
u
r

b
e
s
t

p
e
r
f
.
B
e
s
t

p
e
r
f
.

o
f

t
h
e

s
w
a
r
m
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Part 3: Beyond real
numbers
0
1
2
3
4
0
1
2
3
0
1
2
3
4
1
2
3
4
5
6
0
1
2
0
1
2
3
4
8
8
Bingo!
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Minimun
requirements
( ) ( ) ( ) ( ) ( ) ( ) ' ' , ) ' , ( x f x f x f x f H H x x <
( )
( )
( )
( ) position velocity position
velocity position position
velocity velocity velocity
velocity velocity t coefficien

,
,
,
,

Comparing
positions in the
search space H
Algebraic operators
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
velocity = pos_minus_pos(position
1
, position
2
)
velocity = linear_combin(,velocity
1
,,velocity
2
)
position = pos_plus_vel(position, velocity)
(position,velocity) = confinement(position
t+1
,position
t
)
Pseudo code form
}
algebr
aic
operat
ors
=>
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Fifty-fifty

x
i
2...N
{ }
i j x
i
x
j
x
i
x
i
D/2+2
D

2
D /2

'









N=100, D=20. Search space: [1,N]
D
105 evaluations:
63+90+16+54+71+20+23+60+38+15
=
12+48+13+51+36+42+86+26+57+79 (=450)
g
r
a
n
u
l
a
r
i
t
y
=
1
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Knapsack

x
i
2...N
{ }
i j x
i
x
j
x
i
S
iI , I D, I 2, N
{ }

'









N=100, D=10, S=100,
870 evaluations:
run 1 => (9, 14, 18, 1, 16, 5, 6, 2, 12,
17)
run 2 => (29, 3, 16, 4, 1, 2, 6, 8, 26, 5)
g
r
a
n
u
l
a
r
i
t
y
=
1
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Graph Coloring
Problem
1
4 5
2
1
1
5
5
3
2
2
1
1
5
5
5
0
2
0
-
1
4
-3
-1
-1
+
=
p
o
s
-
p
l
u
s
-
v
e
l
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
The Tireless Traveller
Example of position: X=(5,3,4,1,2,6)
Example of velocity: v=((5,3),(2,5),(3,1))
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
n
1
n
3
n
2
Apple trees
( ) ( )
2
2 2
2
2 2
n n n n f +
Evaluation n2 n2 n2
2
2
2
2
Swarm size=3
Best position
7 7 6
3 11 6
6 4 10
3 0 17
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Part 4: Some variants
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
2
. 00
. 00
. 00
. 00
2
. 22
2 2 2 2 2 22 22 22 22 22 22 22 22 22 22
Unbiased random
proximity
x
p
g
p
i
v
i
-
p
r
o
x
i
m
i
t
y
g
-
p
r
o
x
i
m
i
t
y
Hyperparallelepiped => Biased
Dimension
V
o
l
u
m
e
Hypersphere
vs hypercube
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Clusters and queens
Each particle is
weighted by its
perf.
Dynamic clustering
Centroids =
queens =
temporary
new
particles
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Think locally, act
locally (Adaptive
versions)
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Adaptive swarm size
There has been enough
improvement
but there has been not enough
improvement
although I'm the worst
I'm the best
I try to kill myself
I try to generate a
new particle
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Adaptive coefficients
The better I
am, the more I
follow my own
way
The better is my best
neighbour, the more I
tend to go towards
him
v
rand(0b)(p-x)
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Energies: classical
process
Rosenbrock D. Swarm size= , constant coefficients 2 22 .
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
Number of evaluations
2
2
22
22
22
22
Potenti al energy
Kinetic energy
Swarm size
= . 222222
after evaluations 2222
Rosenbrock 2D. Swarm size=20, constant coefficients
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Energies: adaptive
process
Rosenbrock D. Adaptive swarm size, adaptive coefficients 2 .
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
, E+ 222 22
Number of evaluations
2
2
22
22
22
22
Potenti al energy
Kinetic energy
Swarm size
= . 222222
after evaluations 0000
Rosenbrock 2D. Adaptive swarm size, adaptive coefficients
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Part 5: Real
applications (hybrid)
Medical diagnosis
Industrial mixer
Electrical generator Electrical vehicle
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Real applications
(stand alone)
Cockshott A. R., Hartman B. E., "Improving the fermentation medium for
Echinocandin B production. Part II: Particle swarm optimization", Process
biochemistry, vol. 36, 2001, p. 661-669.
He Z., Wei C., Yang L., Gao X., Yao S., Eberhart R. C., Shi Y., "Extracting
Rules from Fuzzy Neural Network by Particle Swarm Optimization", IEEE
International Conference on Evolutionary Computation, Anchorage, Alaska,
USA, 1998.
Secrest B. R., Traveling Salesman Problem for Surveillance Mission using
Particle Swarm Optimization, AFIT/GCE/ENG/01M-03, Air Force Institute of
Technology, 2001.
Yoshida H., Kawata K., Fukuyama Y., "A Particle Swarm Optimization for
Reactive Power and Voltage Control considering Voltage Security
Assessment", IEEE Trans. on Power Systems, vol. 15, 2001, p. 1232-1239.
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
To know more
Clerc M., Kennedy J., "The Particle Swarm-Explosion,
Stability, and Convergence in a Multidimensional
Somplex space", IEEE Transaction on Evolutionary
Computation, 2002,vol. 6, p. 58-73.
Clerc M., "L'optimisation par essaim particulaire.
Principes et pratique", Herms, Techniques et
Science de l'Informatique, 2002.
Particle Swarm Central, http://www.particleswarm.net
THE site:
Self advert
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Appendix
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Canonical form
( ) ( ) ( ) ( )
( ) ( ) ( )

'

+ + +
+ +
2 2
2
t v t x t x
t x p t v t v
( ) ( ) t x p t y
v t +2 ( ) v t ( )+y t ( )
y t +2 ( ) v t ( )+ ( )y t ( )

'



( )
( )
( )
( )
1
]
1

1
]
1

1
]
1

+
+
t y
t v
t y
t v

2 2
2
2
2
M
Eigen values e
1
and e
2
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Constriction
( ) ( ) ( )
( ) ( ) ( )

'

+
+ + +

+
+ + + +

2 2
2 2
2 2
2 2
2
2 2
2
2
2 2
2
Constriction coefficients
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Convergence criterion

6 5 4 3 2 1
3.5
3
2.5
2
1.5
1
0.5

2
e
2
<2

2
e
2
<2

'

2
<2

2
e
2
<2

'




2
e
2
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Magic Square (1)
1
1
1
]
1

D D D
j i
D
m m
m
m m
, 2 ,
,
, 2
2 , 2

( )
{ }

'

+
l k j i
j i
D i j
j i
m m
N m
x m
, ,
,
2
,
2
( )
( )
2
2
2
2
2
2 , ,
2
2
2
2
, 2 ,

,
_

,
_

+
D
j
D
i
j i j i
D
i
D
j
j i j i
m m
m m
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Magic Square (2)
D=3x3, N=100
10 runs
13430 evaluations
10 solutions
55 30 68
42 49 62
56 74 23
30 61 53
89 32 23
25 51 68
27 96 39
73 40 49
62 26 74
22 70 58
40 75 35
88 5 57
50 43 67
58 55 47
52 62 4
43 51 78
75 33 64
54 88 30
50 65 68
69 42 72
64 76 43
18 25 59
32 53 17
52 24 26
80 3 30
22 72 19
11 38 64
65 28 64
63 55 39
29 74 54
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation
Non linear system
( )

'

+
2 22 sin
2 2
2 2
2
2
2
2
x x
x x
Search space
[0,1]
2
1 run
143 evaluations
1 solution
10 runs
1430 evaluations
3 solutions
2002-04-
24
Maurice.Clerc@Write
Me.com
Particle Swarm optimisation