Appendix B

Genetic Algorithm
Implementation
I
n this appendi x, a genet i c al gor i t hm i mpl ement at i on is present ed.
Software for it is on t he web site for t he book at www.engr.iupui.edu/
-~eberhart/web/PSObook.html. The i mpl ement at i on includes files neces-
sary to t rai n a neural net wor k to classify t he Iris Data Set i nt r oduced by
Anderson (1935) and popul ari zed by Fisher (1936). The genetic algo-
r i t hm is used to evolve t he wei ght s of t he net work.
The dat a set cont ai ns 150 pat t erns, each wi t h four i nput s and one
out put . Each i nput is a feature of an iris flower: sepal l engt h, sepal wi dt h,
petal l engt h, or petal wi dt h. The out put is t he species of flower. There are
t hree species, each represent ed by 50 pat t erns in t he dat a set. It is a not o-
riously hard dat a set to classify well.
The genetic al gor i t hm (GA) i mpl ement at i on is basically a "pl ai n va-
nilla" GA, but wi t h a few i nt erest i ng opt i ons available. It i mpl ement s
one-poi nt , t wo-poi nt , or uni f or m crossover, as well as mut at i on, wi t h an
i nt erest i ng set of opt i ons available for mut at i on, one of whi ch is reminis-
cent of evol ut i on strategies. To learn mor e about t he basics of genet i c al-
gori t hms, refer to Chapt er 4.
Source code is provi ded on t he web site al ong wi t h t he execut abl e
code needed to evolve neural net wor k weights. We provi de t he source
code because of t he somewhat unusual nat ure of genet i c al gor i t hm
i mpl ement at i ons.
In t he case of genetic al gor i t hm i mpl ement at i ons t he fitness f unct i on
bei ng opt i mi zed is t he mai n user-suppl i ed component . This fitness func-
t i on is an integral part of t he genet i c al gori t hm, and so must be coded
and compi l ed wi t h it. For each maj or change in fitness funct i on, a new
451
Appendix B--Genetic Algorithm Implementation
execut abl e GA must be compi l ed. Therefore, for you to appl y t he GA to
your probl em, you will need to subst i t ut e your fitness f unct i on i nt o t he
source code and recompi l e.
The Run File
To r un t he genet i c al gor i t hm i mpl ement at i on on t he web site requires
t he execut abl e file ga. exe and an associ at ed r un file, for exampl e, ga. run.
To r un t he i mpl ement at i on f r om wi t hi n t he di rect ory cont ai ni ng ga. e x e
and ga. run, at t he syst em pr ompt t ype ga ga. run at t he DOS pr ompt .
One way to pr esent t he genet i c al gor i t hm i mpl ement at i on is to exam-
i ne and discuss t he cont ent s of a typical r un file t hat can be i nvoked wi t h
t he execut abl e:
c ' \ g a \ i r i s . d a t
c ' \ ga\ wei ght 1
c ' \ g a \ c o r e c t i
4
2O00
16
4
3
150
7O
0.60
0.01
0.02
I
0
0
2
The first entry, i ri s . dat , is t he name of t he dat a file. The numbe r of
targets appears on t he first line of t he file, and targets are i ncl uded wi t h
each pat t er n. The next entry, wei ght _l , is t he name of t he file i nt o whi ch
t he "evol ved" wei ght s are pl aced at t he end of t he run. Next is t he name
of t he file i nt o whi ch t he results (out put s) are wri t t en, cor ect _l i n this
case.
The Run File
Following t he file list is a list of numbers: t he number of hi dden neu-
ral net work processing elements (PEs) (4), t he maxi mum number of gen-
erations (2,000), the number of bits per variable (16), the number of in-
put PEs (4), t he number of out put PEs (3), t he number of patterns in the
pat t ern file (150), the popul at i on size (70), t he percent probability of
crossover divided by 100 (0.60), the percent probability of mut at i on di-
vided by 100 (0.01), the acceptable sum-squared error to whi ch t he net-
work weights are to be evolved (0.02), t he "mut at e according to bit posi-
tion" flag (1), t he fitness shift flag (0), the "mut at e according to variance"
flag (0), and t he crossover flag (2).
The numbers of input, hi dden, and out put PEs define the configura-
t i on of the neural net work for whi ch the weights are to be evolved. The
activation funct i on utilized in the software is ( t anh0 + 1)/2, whi ch has
an out put range from 0 to 1. The maxi mum number of generat i ons is t he
maxi mum number of epochs, t hat is, the maxi mum number of times t he
entire pat t ern set will be evaluated for fitnesses of individuals.
The number of bits per variable allows the user to set the resolution
for each vector element; in this case, each el ement represents one net-
work weight. The trade-off here is t hat a relatively hi gh number of bits
provides the resolution needed to successfully adjust weights on a com-
plex error surface, but also increases comput at i onal compl exi t y sig-
nificantly. A debate has raged for years wi t h respect to weight resolution
needed to guarantee successful net work training. This GA i mpl ement a-
t i on provides a tool to investigate this quest i on wi t h a variety of data sets
represent i ng various problems.
The size of t he pat t ern file can of course be varied to investigate t he ef-
fect on test sets. If this value n is less t han t he number of pat t erns actually
in t he designated pat t ern file, onl y the first n pat t erns will be utilized for
evolving t he net work weights; all pat t erns are utilized, however, when
the net work is tested.
The number of popul at i on members (70 in this case) can be varied ac-
cording to t he problem. A hi gher number allows a more t hor ough explo-
ration of the probl em domai n, but increases comput i ng time. Typically,
t he value shoul d be set bet ween 20 and 200, but values outside t he range
may be appropriate for relatively simple problems t hat involve relatively
short individuals (<20) or for hi ghl y complex problems i nvol vi ng very
long chromosomes (>200).
The probability of crossover shoul d be set bet ween 60 and 80 per-
cent for many problems. Straightforward t wo-poi nt crossover as de-
scribed in Chapt er 4 can be i mpl ement ed, as can one-poi nt and uni form
crossover.
Appendix B--Genetic Algorithm Implementation
The next value (0.01 in t he list) is t he probabi l i t y of mut at i on. Op-
t i ons for mut at i on i mpl ement ed in this GA are expl ai ned later. The value
listed here is a sort of baseline mut at i on value, as it can be modi fi ed in up
to t wo ways. If not modified, however, t he value represent s t he chance
t hat mut at i on will occur, det er mi ned bit by bit.
The next value, 0.02, is t he sum-squared error target for t he perfor-
mance of t he "evolved" neural net work. The GA will t er mi nat e when
this error level is achi eved or when t he ma xi mum number of genera-
t i ons have been calculated, whi chever occurs first. In ei t her case, t he re-
sult and wei ght files are wri t t en.
The next value (1) is t he "mut at i on accordi ng to bit posi t i on" flag.
Whe n this flag is 0 (disabled), mut at i on is carried out in t he nor mal way:
mut at i on is done bit by bit wi t h t he probabi l i t y of mut at i on listed ear-
lier. When it is 1 (enabled), t he probabi l i t y of mut at i on mb varies wi t h
t he bit posi t i on in each variable. The vari at i on in mut at i on across each
variable appr oxi mat es a Gaussi an funct i on; t hat is, it is muc h mor e prob-
able t hat t he least significant bit will be mut at ed t han it is t hat t he most
significant bit will. It is i mpl ement ed accordi ng to t he fol l owi ng equa-
t i on, wher e b is t he bit posi t i on (b = 0 for t he least significant bit, b = 1
for t he next -t o-l east significant bit, etc.) and m0 is t he probabi l i t y of mu-
t at i on in t he r un file:
1 -b/2
~e (B.1)
mb = m~ ~/2x
Not e t hat this is onl y an approximation of a Gaussi an funct i on. Not e
also t hat t he cal cul at i on is done across each variable. So, in t he above r un
file, for a variable represent ed by 16 bits, t he resul t i ng probabi l i t y of mu-
t at i on is mo(1/2~) 1/2, or about (.01)(.40) for t he least significant bit and
about (. 01)(. 40)exp(-7. 5) - (.004)(.00055) for t he most significant bit.
The vari ance for t he quasi -Gaussi an f unct i on can t hus be seen to depend
upon t he variables' dynami c range ( - 1 0 to 10 in this case) and how t he
variables are represent ed by t he bi nar y string.
The next value in t he r un file (0) provides an opt i on to i mpl ement a
shift in fitness values (for purposes of r epr oduct i on only). When this flag
is 0, t he opt i on is disabled, and "raw" fitness values are used to calculate
nor mal i zed fitness values for t he r epr oduct i on step. When set to 1, all
fitness values are shifted so t hat t he mi n i mu m value is 0.1; t he
rai n_val ue to max_val ue fitness range is preserved. For exampl e, if all raw
fitness values are bet ween 0.90 and 0.99, i mpl ement i ng fitness shift will
scale t hem bet ween 0.10 and 0.19. This provides an expanded fitness
Recompiling ( - ~ , ~ @
differential t hat is oft en useful when fitness values become bunched to-
get her near 1.0 late in a GA run.
The next -t o-l ast i t em in t he r un file (0) allows a change in mut at i on
rate if t he vari ance of t he fitness values drops bel ow 0.05. This change
can be in addi t i on to (on t op of, so to speak) t he change caused by mut at -
i ng accordi ng to bit posi t i on, if it is enabl ed. If this opt i on is disabled (0),
no change in mut at i on rate is triggered by a low vari ance value. If it is
enabled, t he mut at i on rate mlv is i ncreased for low vari ance values, ac-
cordi ng to t he fol l owi ng equat i on, wher e mprev is t he previous (exist-
ing) mut at i on rate, modi fi ed by bit posi t i on or not, dependi ng on t he
modi fy-by-bi t -posi t i on flag:
miv = mprev {(int)[(0.05 - var) * 100] + 1} for var < 0.05 (B.2)
This opt i on allows t he mut at i on rate to increase si gni fi cant l y t owar d
t he end of a r un when t here are oft en ma ny i dent i cal popul at i on mem-
bers and fitness value vari at i on is qui t e low. In t he l i mi t as vari ance ap-
proaches 0, t he mut at i on rate mul t i pl i er will appr oach 6. This sig-
ni fi cant l y "bumps up" mut at i on, facilitating expl orat i on of more of t he
pr obl em space.
The last value in t he list (2) is t he crossover flag. The GA i mpl ement a-
t i on allows t he user to choose ei t her of t hree ki nds of crossover. If t he
flag is set to 0, one- poi nt crossover is i mpl ement ed; if it is set to 1, uni -
form crossover is i mpl ement ed; and a val ue of 2 i mpl ement s t wo- poi nt
crossover.
Recompiting
Because source code for t he genet i c al gor i t hm is i ncl uded on t he web site,
it is possible to use t he GA to opt i mi ze f unct i ons ot her t han neural net-
work weights. To do so, t he fol l owi ng al t erat i ons to t he source code t hat
are tailored to t he pr obl em must be made.
First, in t he file ga. c t he i ndi vi dual popul at i on l engt h cal cul at i on
must be changed to reflect t he new probl em. The l engt h is current l y cal-
cul at ed as
l e n g t h =( ( N+ 1) x H+ ( H+ 1) x P) x K
whi ch reflects t he number of wei ght s in a neural net work.
Second, in t he file f i t ne s s , c it is necessary to do t he following:
@ : ~ Appendix BmGenetic Algorithm Implementation
1. Arrange and allocate memor y to t he paramet ers for t he new prob-
lem. In t he exampl e provi ded on t he web site, t he connect i on
wei ght s are arranged i nt o t wo wei ght matrices in accordance wi t h
t he t opol ogy of t he backpr opagat i on net work.
2. Change t he getWeight0 f unct i on source code to correspond wi t h
t he new paramet er ar r angement .
3. Write a new eval uat i on f unct i on to replace t he one (bp. c) t hat is
provi ded.
4. Modi fy t he final fitness cal cul at i on source code accordi ng to t he
meani ng of t he ret urn value of t he new eval uat i on funct i on. In
t he exi st i ng boO funct i on, t he r et ur n value is t he mean sum-
squared error, not t he fitness. This can be i ncorporat ed i nt o t he
new eval uat i on f unct i on source code.
Third, t he r un file must be modi fi ed as appropri at e in l i ght of t he new
variable structure.
Running the Program
Whe n t he pr ogr am is run, values of t he best, average, and worst individ-
ual fitnesses are pl ot t ed on t he screen. (Be sure t hat t he file egavga, bgi is
in t he same di rect ory as t he execut abl e file ga. exe.) By observi ng t he pl ot
a number of i nt erest i ng t hi ngs can be learned. For exampl e, wi t h t he
vari ance flag set, whi ch i mpl ement s a hi gher mut at i on rate for low vari-
ance, it is possible to see an al most i mmedi at e effect on t he best, average,
and worst values when t he vari ance drops bel ow 0.05.