DECLARATION

We hereby declare that this submission is our own work and that, to the best of our knowledge

and belief, it contains no material previously published or written by another person nor material

which to a substantial extent has been accepted for the award of any other degree or diploma of

the university or other institute of higher learning except where due acknowledgement has been

made in the text.

Names:

AnshuSaxena (1100210011)

KanikaVerma(1100210024)

KanikaVij (1100210025)

Riya Jain (1100210057)

1
 CERTIFICATE

This is to certify that Project Report entitled Performance Analysis Of Pattern Storage And

Recalling With HOFFIELD NEURAL NETWORK For Handwritten and Distorted English

Capital Alphabets which is been submitted by AnshuSaxena, KanikaVerma,

KanikaVij,and Riya Jain in partial fulfillment of the requirement for the award of degree

ofB.Tech in Computer Science And Engineering Of Uttar Pradesh Technical University. It is a

record of the candidates own work carried out by them under my supervision. The matter

embodied in this thesis is original and has not been submitted for the award of any other degree.

Under the guidance of:

Dr. R.K Sharma Er.Anurag Sharma
Asst. Prof, Comp.Sc.&Engg.Dept H.O.D,Comp.Sc.&Engg.Dept
F.E.T Agra College F.E.T Agra College,
Agra Agra

2
 ACKNOWLEDGEMENT

It gives us a great sense of pleasure to present the report of the B.Tech project undertaken during

B.Tech final year. We owe special debt of gratitude to Department of Computer Science and

Engineering, F.E.T. Agra College, Agra for its constant support and guidance throughout the

course of our work. Its sincerity, thoroughness and perseverance have been a constant source of

inspiration for us. It is only its cognizant efforts that our endeavors have seen light of the day.

We take the opportunity to acknowledge the contribution of Dr.R.K Sharma for his full support

during the development of the project.

We also take the opportunity to acknowledge the contribution of Mr. Anurag Sharma, Head of

Department of Computer Science and Engineering, for his full support and assistance during the

development of the project.

We also do not like to miss the opportunity to acknowledge the contribution of all faculty

members of the department for their kind assistance and cooperation during the development of

our project. Last but not the least, we acknowledge our friends for their contribution in the

completion of the project. I hope our work is appreciated and we are excused for any

inconvenience caused to anybody unintentionally.

NAME:

AnshuSaxena (1100210011)

KanikaVerma(1100210024)

KanikaVij (1100210025)

Riya Jain (1100210057)

3
 ABSTRACT

This project is designed to analyze the performance of a Hopfield Neural Network for storage

and recall of images. The study implements a form of unsupervised learning. The project first

discusses the storage and recall via Hebbian learning and the problem areas or the efficiency

issues involved and then the performance enhancement via the pseudo-inverse learning.

Performance is measured with respect to storage capacity, recall of distorted or noisy patterns i.e.

association of a noisy version of a stored pattern to the original stored pattern for testing the

accretive behavior of the network and association of new or noisy / distorted patterns to some

stored pattern.

The performance evaluation for the feed forward neural network with three different soft

computing techniques to recognition of hand written English alphabets. Evolutionary algorithms

for the hybrid neural network are showing the numerous potential in the field of pattern

recognition.Wehave taken five trials and two networks of each of the algorithm: back

propagation, Evolutionary algorithm, and Hybrid Evolutionary algorithm respectively. These

algorithms have been taken the definite lead on the conventional approaches of neural network

for pattern recognition.

It has been analyzed that the feed forward neural network by two Evolutionary algorithms makes

better generalization accuracy in character recognition problems. The problem of not

convergence the weight in conventional backpropagation has also eliminated by using the soft

computing techniques. It has been observed that, there are more than one converge weight matrix

in character recognition for every training set. The results of the experiments show that the

hybrid evolutionary algorithm can solve challenging problem most reliably and efficiently.

4
 PROJECT OBJECTIVE

Performance Analysis Of Pattern Storage And Recalling With HOPFIELD NEURAL

NETWORK For Handwritten And Distorted Capital English Alphabet"

This is a MATLAB based project in which we are using MATLAB 7.0 to store and recall images

using Hopfield Neural Network. In this we are going to analyze the performance of Pattern

Storage using Hopfield Neural Network for Handwritten and Distorted English Capital

Alphabets.

We will first of all take all the scanned images and then with the help of Matlab store those

images, which include the Training set of images as well as the Testing set of images. All the

images have been scanned and saved in the JPG format. The images are all handwritten,

distorted and also we have used paint brush to draw them. The training set is properly written

using stencils while the testing set has been written by different individuals in their own writing

styles.

The code will analyze the testing set and then recall and recognize it with the training set of data.

After this a regression line will be plotted to mark the deviation between the input image and the

recalled image

For example:

5
 TABLE OF CONTENTS

Page

DECLARATION ii

CERTIFICATE iii

ACKNOWLEDGEMENT iv

ABSTRACT v

PROJECT OBJECTIVE vi

CHAPTER 1: INTRODUCTION

CHAPTER 2: NEURAL NETWORK

CHAPTER 3: PATTERN RECOGNITION & HOPFIELD MODEL

CHAPTER 4: IMPLEMENTATION

CHAPTER 5: RESULT

CHAPTER 6: SOFTWARE AND HARDWARE

CHAPTER 7: CONCLUSION

REFRENCES

6
 CHAPTER 1

INTRODUCTION

7
 Chapter 1

INTRODUCTION

Pattern Storage is one of the prerequisite for the pattern recognition task to be realized using

ANN.Feedback neural networksare the most commonly used networks for implementing the

pattern storage task. The fast reliable and computerized recognition of handwritten and distorted

images is a remarkable problem in patternrecognition that has not to this date received a

complete solution.

ANN is an information processing paradigm that is inspired by the way biological nervous

systems, such as the brain, process information.

Some attractive features of biological neural network that make it superior to even the most

sophisticated AI computer system for pattern recognition task are the following:

robustness and fault tolerance

flexibility

collective computation

Pattern recognition aims to classify data (patterns) based on either a priori knowledge or on

statistical information extracted from the patterns.

The patterns to be classified are usually groups of measurements or observations, defining points

in an appropriate multidimensional space.Acomplete pattern recognition system consists of a

sensor that gathers the observations to be classified or described; a feature extraction mechanism

that computes numeric or symbolic information from the observations; and a classification or

description scheme that does the actual job of classifying or describing observations, relying on

the extracted features. Character recognition plays an important role in todays life.

8
It can solve many complex problems in of real life. An example of character recognition is

Handwritten English alphabets. The classic difficulty of being able to correctly recognize even

typed optical language symbols is the complex irregularity among pictorial representations of the

same character due to variations in fonts, styles and size. This irregularity undoubtedly

widenswhenone deals with handwritten characters. These are the concepts:

Member-roster concept: Under this template-matching concept, a set of patterns belonging to a

same pattern is stored in a classification system.When an unknown pattern is given as input, it is

compared with existing patterns and placed under the matching pattern class.

Common property concept: In this concept, the common properties of patterns are stored in a

classification system.

When an unknown pattern comes inside, the system checks its extracted common property

against the common properties of existing classes and places the pattern/object under a class,

which has similar, common properties.

Clustering concept: Here, the patterns of the targeted classes are represented in vectors whose

components are real numbers. So, using its clustering properties, we can easily classify the

unknown pattern. If the target vectors are far apart in geometrical arrangement, it is easy to

classify the unknown patterns. If they are nearby or if there is any overlap in the cluster

arrangement, we need more complex algorithms to classify the unknown patterns.

Bayesian decision theory is a system that minimizes the classification error. Bayesian decision

theory has a conceptual clarity leading to an elegant numerical recipe. This algorithm can deal

with a broader scope of stochastic models than classical algorithms. Nearest neighbor ruleis used

to classify the handwritten characters. The distance measured between two character images is

needed to use this rule.

9
Here we are using Hopfield Neural Network for performance analysis of storage and recalling

for handwritten and distorted english capital alphabets.

Hopfield networks are constructed from artificial neurons.

These artificial neurons have N inputs. With each input i there is a weight associated.

They also have an output.

The state of the output is maintained, until the neuron is updated.

Regression: Introduction
regression

Linear regression

Syntax
[r,m,b] = regression(t,y)
[r,m,b] = regression(t,y,'one')

Description
[r,m,b] = regression(t,y) takes these arguments,

t Target matrix or cell array data with a total of N matrix rows

y Output matrix or cell array data of the same size and returns these outputs,

r Regression values for each of the N matrix rows

m Slope of regression fit for each of the N matrix rows

b Offset of regression fit for each of the N matrix rows

[r,m,b] = regression(t,y,'one') combines all matrix rows before regressing, and returns single
scalar regression, slope, and offset values.

10
11
 CHAPTER 2

NEURAL NETWORK

12
 Chapter 2

NEURAL NETWORK

ARTIFICIAL NEURAL NETWORKS: BIOLOGICAL INSPIRED PARADIGM

Artificial Neural Networks are relatively crude electronic models based on the neural structure of

the brain. The brain learns from experience and acquires knowledge. It is natural proof that some

problems that are beyond the scope of current computers are indeed solvable by small energy

efficient packages. This brain modeling also promises a less technical way to develop machine

solutions. This new approach to computing also provides a more graceful degradation during

system overload than its more traditional counterparts.

These biologically inspired methods of computing are thought to be the next major advancement

in the computing industry. Even simple animal brains are capable of functions that are currently

impossible for computers. Computers do rote things well, like keeping ledgers or performing

complex math. However, computers have trouble recognizing even simple patterns much less

generalizing those patterns of the past into actions of the future.

Now, advances in biological research promise an initial understanding of the natural thinking

mechanism. This research shows that brains store information as patterns. Some of these patterns

are very complicated and allow us the ability to recognize individual faces from many different

angles. This process of storing information as patterns, utilizing those patterns, and then solving

problems encompasses a new field in computing. This field, as mentioned before, does not

utilize traditional programming but involves the creation of massively parallel networks and the

training of those networks to solve specific problems. This field also utilizes words very different

from traditional computing, words like behave, react, self-organize, learn, generalize, and forget.

13
NEURAL NETWORK:
The study of the neural networks is generated by the study of human brain, that how the brain

works to make the decision, to store the patterns and how a brain learns for the recognition of

objects. The study defines that a human brain contains an average of10 10 neurons of various

types, which are basically known as decision elements, with each neuron connecting to up to 104

synapses. Artificial Neural Networks are relatively crude electronic models based on the neural

structure of the brain. Neural networks are attractive since they consist of many neurons, each of

the neurons processes information separately and simultaneously. All the neurons are connected

by synapses with variable weights. Thus, neural networks are actually parallel distributed

processing systems.

14
During the decades of 1940s McCulloch and Pitts found that the neuron can be modeled as a

simple threshold device to perform logic function . After that during the late 1940s, Hebb

proposed the Hebbian rule to describe how learning affects the synaptics between two neurons.

In the late 1950s and early 1960s, Rosenblatt proposed the perceptron model, and Widrow and

Hoff proposed the adaline (adaptive linear element) model, trained with a least mean squares

(LMS) method. Thevarious studies and development of neural networks have been proposed by

the researchers. The landmark of the field is the multilayer perceptron (MLP) model trained with

the backpropagation (BP) learning algorithm published in 1986 by Rumelhartet al. Later it

turned out that the BP algorithm had already been described in 1974 by Werbo when he studied

social problems.

15
NEURONS:
The human body is made up of a vast array of living cells. Certain cells are interconnected in a

way that allows them to communicate pain, or to actuate fibers or tissues. Some cells control the

opening and shutting of minuscule valves in the veins and arteries. Others tell the brain that they

are experiencing cold, heat, or any number of sensations. These specialized communication cells

are called neurons. Figure shows the biological structure of neurons.

The biological neurons are equipped with long tentacle-like structures that stretch out from the

cell body, permitting them to communicate with other neurons. The tentacles that take in signals

from other cells and the environment itself are called dendrites, while the tentacles that carry

signals from the neuron to other cells are called axons. The interaction of the cell body itself with

the outside environment through its dendrite connections and the local conditions in the neuron

itself cause the neuron to pump either sodium or potassium in and out, raising and lowering the

neurons electrical potential. When the neurons electrical potential exceeds a threshold, the

neuron fires, creating an action potential that flows down the axons to the synapses and other

neurons. The action potential is created when the voltage across the cell membrane of the neuron

becomes too large and the cell fires, creating a spike that travels down the axon to other

16
neurons and cells. If the stimulus causing the buildup in voltage is low, then it takes a long time

to cause the neuron to fire. If it is high, the neurons fire much faster. The neuron is the basic

information processing unit of a NN consists of:

i. A set of links, describing the neuron inputs, with weights W1, W2, , Wm

ii. An adder functions, for computing the weighted sum of the inputs.

iii. Activation function (squashing function) for limiting the amplitude of the neuron output.

The activation function represents a linear or nonlinear mapping from the input to the output.

The output of the neuron is calculated by

n
u wi xi
(2.1)
i 1

and the final output z is calculated as:

z f u (2.2)

There are some functions that can be used as activation functions as follows:

1. Hard limiter

1 u0


f u (2.3)
1 u0

2. Piecewise linear function

1, u 1
2


f u u 1 u 1 (2.4)
2 2

0 u1
2

17
3. Logistic Sigmoid function

f u
1
(2.5)
1 e u

4. Hyperbolic tangent function

f u tanh u (2.6)

Hopfield Neural Networks:

Hopfield networks are constructed from artificial neurons. These artificial neurons have N

inputs. With each input i there is a weight wi associated. They also have an output. The state of

the output is maintained, until the neuron is updated.

Updating the neuron entails the following operations:

The value of each input, is determined and the weighted sum of all Inputs, is

calculated.

The output state of the neuron is set to +1 if the weighted input sum is larger or equal to

0. It is set to -1 if the weighted input sum is smaller than 0.

18
 A neuron retains its output state until it is updated again.

1: 0
Written as a formula: =
1: < 0

A Hopfield network is a network of N such artificial neurons, which are fully connected. The

connection weight from neuron j to neuron i is given by a number wij . The collection of all such

numbers is represented by the weight matrix W , whose components are wij .

Now given the weight matrix and the updating rule for neurons the dynamics of the network is

defined if we tell in which order we update the neurons. There are two ways of updating them:

Asynchronous: one picks one neuron, calculates the weighted input sum and updates

immediately. This can be done in a fixed order, or neurons can be picked at random, which is

called asynchronous random updating.

Synchronous: the weighted input sums of all neurons are calculated with-out updating the

neurons. Then all neurons are set to their new value, according to the value of their weighted

input sum. The lecture slides contain an explicit example of synchronous updating.

19
Use of the Hopfield network

The way in which the Hopfield network is used is as follows. A pattern is entered in the network

by setting all nodes to a specific value, or by setting only part of the nodes. The network is then

subject to a number of iterations using asynchronous or synchronous updating. This is stopped

after a while. The network neurons are then read out to see which pattern is in the network.

The idea behind the Hopfield network is that patterns are stored in the weight matrix. The input

must contain part of these patterns. The dynamics of the network then retrieve the patterns stored

in the weight matrix. This is called Content Addressable Memory (CAM). The network can also

be used for auto-association. The patterns that are stored in the network are divided in two parts:

cue and association. By entering the cue into the network, the entire pattern, which is stored in

the weight matrix, is retrieved. In this way the network restores the association that belongs to a

given cue.

The stage is now almost set for the Hopfield network, we must only decide how we determine

the weight matrix. We will do that in the next section, but in general we always impose two

conditions on the weight matrix:

symmetry: wij = wji

no self connections: wii = 0

20
 CHAPTER 3

PATTERN RECOGNITION AND

ITS TECHNIQUES

21
 Chapter 3

PATTERN RECOGNITION AND ITS TECHNIQUES

The concept of recognition is simple in the real world environment, but in the world of computer

science, recognizing any object is an amazing feat. The functionality of the human brain is

amazing; it is not comparable with any machines or software. Recognition is a basic property of

all human beings; when a person sees an object, he or she first gathers all information about the

object and compares its properties and behaviors with the existing knowledge stored in the mind.

If we find a proper match, we recognize it . Pattern recognition, as a subject, spans a number of

scientific disciplines, uniting them in search for a solution to the common problem of

recognizing the pattern of a given class and assigning the name of identified class. Pattern

recognition is the categorization of input data into identifiable classes through the extraction of

significant attributes of the data from irrelevant background detail. A pattern class is a category

determined by some commonattributes. Therefore, a pattern is the description of a category

member representing a pattern class. A pattern class is a family of patterns that shares some

common properties.Pattern recognition problems may be logically divided into two broad

categories. First involves the study of pattern recognition capabilities of human beings and other

living organism, while the second involves the development of theory and techniques for the

design of device capable of performing a given recognition task for a specific applications. The

first area deals with the subjects of psychology, physiology and biology, but the second area

deals with the computer, information science and artificial intelligence or machine learning

methods. The act of recognition with machine can be divided into two broad categories:

22
1. Concrete item recognition, it involves the recognition of spatial samples such as

fingerprints, weather maps, pictures and physical objects and the recognition of temporal

samples such as, waveforms and signatures.

2. Abstract item recognition, it involves the recognition of a solution to a problem, an old

conversation or argument.

STATISTICAL METHODS

This is an approach to machine intelligence which is based on statistical modeling of data. In a

statistical model, one applies probability theory and decision theory to get an algorithm. During

the past there has been progress in theory and applications of 'Statistical Pattern

Recognition'. The three major issues encountered in the design of a statistical pattern

recognition system are sensing, feature extraction, and classification. The primary issue is

the representation of the input data which can be measured from the objects to be recognized

and it is called sensing problem. Each measured quantity describes the characteristics of the

pattern or objects.

The number of features of the pattern samples is usually very large. The features of the

pattern samples are reduced by considering their salient characteristics. This process is

referred to as feature extraction. Several approaches for feature extraction have been

proposed by the various researchers such as feature extraction by moments invariants,

feature extraction by autoregressive models, and feature extraction by KL transformation.

The last issue of the statistical pattern recognition is the pattern classification or

development of the classifier. The pattern classifier is defined as a device or a process that

sorts the given data into identifiable categories and classes. The pattern classification is an

information transformation process, i.e., the classifiers transforms relatively large set of

23
mysterious data into a smaller set of useful data. Trainable classifier is one that can improve

its performance in response to the information it receives as a function of time. Training is a

process by which the parameters of the classifiers are adjusted. The classifier is trained

using the reduced pattern samples. It is often assumed that the pattern samples of a given

class occupy a finite region in a pattern space and it is called a class region.

A good classification is the main object of recognition system. Generally, the statistically pattern

recognition problems fall into two main categories: supervised classification (discriminate

analysis) problems and unsupervised classification (clustering) problems. When the samples

have known classification (labeled samples) then the classification is called the supervised

otherwise it is unsupervised. So in the supervised classification, the data is labeled; and the

problem is to design a classifier to predict the class label for any given sample. In the

unsupervised classification, data are not labeled and to classify the data into groups with features

that distinguish one group from another. In the supervised classification, each sample is assumed

to come from one of the c possible classes j , j 1,2,....., c . Suppose, we are given n training

samples xi , yi , i 1,2,.....n , where xi R d represents the data and yi 1,2,....., cthe

corresponding class labels. The objective is to design a decision function g x from these

training samples such that g x can accurately predict the class label for any given test sample x .

A classical way to tackle this problem is to use Bayes decision rule [16], which assigns a test

sample x to the class with the maximum posterior probability p j x , that is,

x k , if , pk x max p j x (3.1)
j 1, 2,......c

where p j x is defined as the conditional probability that the observed object belongs to class

j given the associated sample data, x . In practice these posterior probabilities are usually

unknown. By applying Bayes theorem, they may be calculated as,

24
 p j x

p x j p j
(3.2)
p x

where px is the probability density that the sample x occurs, p j is the prior probability

that class j occurs and p x j is the class-conditional density of a sample x occurring given

that the sample is known to come from class j . Then the decision rule becomes,

x k if px k pk p x j p j , j 1,2,....., c (3.3)

Generally, the prior probability p j and the class-conditional densities p x j are unknown

and need to be estimated. Compared with the estimation of the prior probability p j , the

estimation of the class-conditional densities p x j is much more difficult.

If we know the functional form of the class-conditional densities p x j (e.g., multivariate

Gaussian), but do not know the parameters (e.g., mean and covariance), we face a parametric

decision problem. In this case, a common strategy is to replace the unknown parameters in the

density functions with some estimated values, resulting in the so-called Bayes plug-in classifier.

If the form of the class-conditional densities is not known, then we operate in a non-parametric

mode. In this case, we may either directly estimate the density for a given sample (e.g., Parzen

windows and k-Nearest Neighbor) or construct a decision boundary based on the training

samples. In the latter case, assuming that the decision boundary is linear leads to a linear

discriminate method (e.g., perceptrons and Fishers rule). Assuming that the decision boundary is

not linear leads to a nonlinear discriminate method (e.g., feedforward neural networks and

support vector machines).

25
Parametric Methods

In the parametric approach the underlying densities of the pattern classes are known but the

values of the parameters might be unknown. If the parameters were known then the discriminate

functions based on them can be readily specified. In practice when large numbers of pattern

samples are available, class density function can be estimated or learned from the samples. These

parameters are then used for specification of discriminate functions. Important situations in

which pattern classes are characterized by set of parameters occur when the patterns in each of

the N classes are random variables governed by N distinct probability functions. The most widely

accepted parametric type classifier in pattern recognition is Bayes classifier. This classifier assigns

zero loss to correct classifications and equal loss to incorrect classifications. The optimal

decision of the Bayes classifier minimizes the probability of error in classification. For a two-

class problem the Bayes discriminator gives an optimum decision by assigning the pattern

sample.If the functional form of the class-conditional densities p x j is known, but the

parameters of the density functions are unknown, we may estimate the unknown parameters from

the available samples. Without loss of generality, we may drop the class label and write p x j
as px; , where is a parameter vector to be estimated. The commonly used parametric

models are the multivariate Gaussian distribution for continuous variables, the binomial

distribution for binary variables, and the multi normal distribution for integer-valued (or

categorical) variables.

Applying the most widely used estimator, namely the maximum likelihood estimator, we select

as to estimate the particular value of , which gives the greatest value of the likelihood, which

is defined by,

26
 n
Lx1 ,....., xn ; pxi ; (3.4)
i 1

An important result of using this method is that, when px; is a multivariate Gaussian

distribution, if the covariance matrices for different classes are assumed to be different, the

optimal classifier provides a quadratic decision boundary. More interestingly, if the covariance

matrices are assumed to be identical, then the optimal classifier has a linear decision boundary.

In general, the parametric approach works well when the number of features is small.

Unfortunately, the parametric approach suffers from a phenomenon known as Curse of

Dimensionality, which states that the number of required samples is an exponential function of

the feature dimension. To address this problem, various regularization techniques were proposed

to obtain a robust estimate for situations with small sample size and/or high-dimension. Among

them, the most famous one is the Regularized Discriminate Analysis proposed by Friedman.In

many real world problems, data may form subgroups in each class. In this case, we may

approximate the density px; by a finite mixture model with,

p x; j px; j
g
(3.5)
j 1

where g is the number of mixture components, j is a mixing proportion j 0, j 1 j 1 ,
g

and px; j is a component density function which depends on parameters j . There are three

sets of parameters to be estimated: j , j and g . Generally, the value of g is the most difficult

to estimate. Wolfe proposed a modified likelihood ratio test to estimate the number of

components. This approach was later investigated by Everitt and Anderson . Other approaches

include Bozdogan and Celeux and Soromenho. Their approaches are based on the information-

27
theoretic criteria such as entropy. If the value of g is already known, the values of j and j

can be estimated by using the Expectation Maximization (EM) method , which is an extension of

the maximum likelihood estimator for the uni-modal case (each class has only one group).

Although the EM method is very easy to implement, its convergence rate can be poor, depending

on the data distribution and the initial parameters. Jamshidian and Jennrich proposed a

generalized conjugate gradient method to speed up the EM method. Lindsay and Basak described

a method to initialize the EM model. In addition to EM, several authors also advocated the

Markov Chain Monte-Carlo method, but this method is computationally demanding.

STRUCTURED METHODS

Structural pattern recognition, sometimes referred to as syntactic pattern recognition due to its

origins in formal language theory, relies on syntactic grammars to discriminate among data from

different groups based upon the morphological interrelationships or interconnections present

within the data. Structural pattern recognition techniques are effective for the data which contain

an inherent, identifiable organization such as image data and time series data. The usefulness of

structural pattern recognition systems, however, is limited as a consequence of fundamental

complications associated with the implementation of the description and classification tasks. In

many recognition problems involving complex patterns, it is more appropriate to adopt a

hierarchical perspective where a pattern is viewed as being composed of simple sub patterns

which they built from yet simpler sub patterns. The elementary sub patterns to be recognized are

called primitives and the given complex pattern is represented in terms of the interrelationships

between these primitives. In syntactic pattern recognition, a formal analogy is drawn between the

structure of patterns and the syntax of a language. The patterns are viewed as sentences

belonging to a language, primitives are viewed as the alphabet of the language, and the sentences

28
are generated according to a grammar. Thus, a large collection of complex patterns can be

described by a small number of primitives and grammatical rules. The grammar for each pattern

class must be inferred from the available training samples. Structural pattern recognition is

intuitively appealing because, in addition to classification, this approach also provides a

description of how the given pattern is constructed from the primitives. This paradigm has been

used in situations where the patterns have a definite structure which can be captured in terms of a

set of rules, textured images, and shape analysis of contours. The implementation of a syntactic

approach, however, leads to many difficulties which primarily have to do with the segmentation

of noisy patterns and the inference of the grammar from training data.The notion of attributed

grammars which unifies syntactic and statistical pattern recognition. The syntactic approach may

yield a combinatorial explosion of possibilities to be investigated, demanding large training sets

and very large computational efforts .

Structural pattern recognition systems are difficult to apply to new domains because

implementation of both the description and classification tasks requires domain knowledge.

Knowledge acquisition techniques necessary to obtain domain knowledge from experts are

tedious and often fail to produce a complete and accurate knowledge base. Consequently,

applications of structural pattern recognition have been primarily restricted to domains in which

the set of useful morphological features has been established in the literature and the syntactic

grammars can be composed by hand. To overcome this limitation, a domain independent

approach to structural pattern recognition is used that is capable of extracting morphological

features and performing classification without relying on domain knowledge. A hybrid system

that employs a statistical classification technique to perform discrimination based on structural

features is a natural solution. While a statistical classifier is inherently domain independent, the

domain knowledge necessary to support the description task can be eliminated with a set of

29
generally useful morphological features. Such a set of morphological features is suggested as the

foundation for the development of a suite of structure detectors to perform generalized feature

extraction for structural pattern recognition in time series data.

Identification problems involving time series (or waveform) data constitute a subset of pattern

recognition applications that is of particular interest because of the large number of domains that

involve such data . Both statistical and structural approaches can be used for pattern recognition

of time series data: standard statistical techniques have been established for discriminate analysis

of time series data, and structural techniques have been shown to be effective in a variety of

domains involving time series data.

A domain independent structural pattern recognition system is one that is capable of acting as a

black box to extract primitives and perform classification without the need for domain

knowledge. Such a system would automatically describe and classify data, thereby eliminating

the overhead associated with traditional approaches to structural pattern recognition. A domain

independent structural pattern recognition system for time series data must incorporate

techniques for the description and classification tasks that are not dependent on domain

knowledgei.e., generalized description and generalized classification. Since syntactic

grammars are inherently tied to the domain and application, a sensible approach to generalized

classification for time series data is a statistical classifier that performs discrimination based on

structural features extracted from the data. Generalized description can be implemented using a

foundation of generally useful morphological features that are effective regardless of the domain.

The field of signal processing offers a suggestion for morphological features which can provide

the foundation for generalized description of time series data. Six fundamental types of

modulation commonly used in signal processing systemsconstant, straight, exponential,

30
sinusoidal, triangular, and rectangularentail morphologies deliberately introduced into a

continuous medium with the intent of conveying information regardless of the domain or

application . Moreover, these six modulation types subsume the small set of domain independent

morphological features commonly extracted by structural pattern recognition systemsstraight

lines, parabolas, and peaks. A suite of feature extractors which identify morphological features

based on these six modulation types, therefore, would constitute a first pass at implementing

generalized feature extraction to support domain independent structural pattern recognition in

time series data.

Structural approaches, while supported by psychological evidence which suggests that structure

based description and classification parallels that of human perceptual and cognitive processes

have not yet been developed to the fullest potential due to fundamental complications associated

with implementing structural pattern recognition systems. Shiavi and Bourne summarize the

problems of applying structural methods for pattern recognition within the context of analyzing

biological waveforms: There are obvious problems with the use of [structural techniques]. First,

rather deep knowledge about the problem is required in order to successfully identify features

and write [grammar] rules. While it is conceptually interesting to consider the possibility of

using some automated type of grammatical inference to produce the rules, in practice no

technique of grammatical inference has proved robust enough to be used with real problems

involving biological waveforms. Hence, the writing of rules is incumbent on the designer of the

analysis system. Similarly, the selection of features must be accomplished essentially by hand

since automated techniques usually cannot provide the guidance necessary to make a useful

feature selection. Second, the control strategy of typical parsing systems is relatively trivial and

cannot deal with very difficult problems. Typical parsing techniques consist of simple repeated

application of a list of rules, which is often equivalent to forward chaining, an elementary

31
concept in knowledge based rule systems. Formation of a robust control strategy for guiding

syntactic parsing of strings appears somewhat problematic. There is no general solution for

extracting structural features from data. Friedman addresses the issue by saying, The selection

of primitives by which the patterns of interest are going to be described depends upon the type of

data and the associated application. Nadler seems to support this position when he states,

...features are generally designed by hand, using the experience, intuition, and/or cleverness of

the designer. The lack of a general approach for extracting primitives puts designers of

structural pattern recognition systems in an awkward position: feature extractors are necessary to

identify primitives in the data, and yet there is no established methodology for deciding which

primitives to extract. The result is that feature extractors for structural pattern recognition

systems are developed to extract either the simplest and most generic primitives possible, or the

domain and application specific primitives that best support the subsequent classification task.

NEURAL NETWORK BASED METHODS

The pattern recognition approaches discussed so far are based on direct computation through

machines. Direct computations are based on statistical analysis techniques. The neural approach

applies biological concepts to machines for pattern recognition. The outcome of this effort is

invention of artificial neural networks. Neural networks can be viewed as massively parallel

computing systems consisting of an extremely large number of simple processors with many

interconnections. Neural network models attempt to use some organizational principles (such as

learning, generalization, adaptively, fault tolerance, distributed representation, and computation)

in a network of weighted directed graphs in which the nodes are artificial neurons and directed

edges (with weights) are connections between neuron outputs and neuron inputs. The main

characteristics of neural networks are that they have the ability to learn complex nonlinear input-

32
output relationships, use sequential training procedures, and adapt themselves to the data. The

most commonly used family of neural networks for pattern classification tasks is the feed-

forward network, which includes multilayer perceptron and Radial-Basis Function (RBF)

networks. These networks are organized into layers and have unidirectional connections between

the layers. Another popular network is the Self-Organizing Map (SOM), or Kohonen-Network ,

which is mainly used for data clustering and feature mapping. The learning process involves

updating network architecture and connection weights so that a network can efficiently perform a

specific classification/clustering task. The increasing popularity of neural network models to

solve pattern recognition problems has been primarily due to their seemingly low dependence on

domain-specific knowledge (relative to model-based and rule-based approaches) and due to the

availability of efficient learning algorithms. Neural networks provide a new suite of nonlinear

algorithms for feature extraction (using hidden layers) and classification (e.g., multilayer

perceptrons). In addition, existing feature extraction and classification algorithms can also be

mapped on neural network architectures for efficient (hardware) implementation. In spite of the

seemingly different underlying principles, most of the well known neural network models are

implicitly equivalent or similar to classical statistical pattern recognition methods. Ripley and

Anderson et al.also discuss the relationship between neural networks and statistical pattern

recognition. Most neural networks conceal the statistics from the user. Despite these similarities,

neural networks do offer several advantages such as, unified approaches for feature extraction &

classification and flexible procedures for finding good, moderately nonlinear solutions. It

consists of massive simple processing units with a high degree of interconnection between each

unit. The processing units work cooperatively with each other and achieve massive parallel

distributed processing. The design and function of neural networks simulate some functionality

of biological brains and neurons systems. The advantages of neural networks are their adaptive-

33
learning, self-organization and fault-tolerance capabilities. For these outstanding capabilities,

neural networks are used for pattern recognition applications. The goal in pattern recognition is

to use a set of example solutions to some problem to infer an underlying regularity which can

subsequently be used to solve new instances of the problem. Examples include hand-written digit

recognition, medical image screening and fingerprint identification. In the case of feed-forward

networks, the set of example solutions (called a training set), comprises sets of input values

together with corresponding sets of desired output values. The training set is used to determine

an error function in terms of the discrepancy between the predictions of the network, for given

inputs, and the desired values of the outputs given by the training set. A common example of an

error function would be the squared difference between desired and actual output, summed over

all outputs and summed over all patterns in the training set. The learning process then involves

adjusting the values of the parameters to minimize the value of the error function. This kind of

feedback would be used to reconstruct the input patterns and make them free from error; thus

increasing the performance of the neural networks. These kinds of networks are called as auto

associative neural networks. As the name implies, they use back-propagation algorithms.

However, effective learning algorithms were only known for the case of networks in which at

most one of the layers comprised adaptive interconnections. Such networks were known

variously as perceptrons and Adalines, and were seriously limited in their capabilities.Research

into artificial neural network was stimulated during the 1980s by the development of new

algorithms capable of training networks with more than one layer of adaptive parameters.

34
HANDWRITTEN CHARACTER RECOGNITION

For more than thirty years, researchers have been working on handwriting recognition. As in the

case of speech processing, they aimed at designing systems able to understand personal encoding

of natural language. This new stage in the evolution of handwriting processing results from a

combination of several elements: improvements in recognition rates, the use of complex systems

integrating several kinds of information, the choice of relevant application domains, and new

technologies such as high quality high speed scanners and inexpensive powerful CPUs. Methods

and recognition rates depend on the level of constraints on handwriting. The constraints are

mainly characterized by the types of handwriting, the number of scripture, the size of the

vocabulary and the spatial layout. Obviously, recognition becomes more difficult when the

constraints decrease. Considering the types of roman script (roughly classified as hand printed,

discrete script and cursive script), the difficulty is lower for handwriting produced as a sequence

of separate characters than for cursive script which has much in common with continuous speech

recognition. For other writing systems, character recognition is hard to achieve, as in the case of

Kanji which is characterized by complex shapes and a huge number of symbols. The

characteristics which constrain hand writing may be combined in order to define handwriting

categories for which the results of automatic processing are satisfactory. The trade-off between

constraints and error rates give rise to applications in several domains. These new challenges

bring the ongoing studies closer to unconstrained handwritten language processing which is the

ultimate aim. The reading of all of the handwritten and printed information present on a

document is necessary to process it automatically, to use content dependent criteria to store,

access and transmit it and to check its content. Automatic handwritten language processing will

also allow one to convert and to handle manuscripts produced over several centuries within a

computer environment.

35
Intense activity was devoted to the character recognition problem during the seventies and the

eighties and pretty good results have been achieved. Character Recognition techniques can be

classified according to two criteria: the way preprocessing is performed on the data and the type

of the decision algorithm. Preprocessing techniques include three main categories: the use of

global transforms (correlation, Fourier descriptors, etc.), local comparison (local densities,

intersections with straight lines, variable masks, characteristic loci, etc.) and geometrical or

topological characteristics (strokes, loops, openings, diacritical marks, skeleton, etc.). Depending

on the type of preprocessing stage, various kinds of decision methods have been used such as:

various statistical methods, neural networks, structural matching (on trees, chains, etc.) and

stochastic processing (Markov chains, etc.). Many recent methods mix several techniques

together in order to provide a better reliability to compensate the great variability of handwriting.

Neural network computing has been expected to play a significant role in a computer-based

system of recognizing handwritten characters. This is because a neural network can be trained

quite readily to recognize several instances of a written letter or word, and can then be

generalized to recognize other different instances of that same letter or word. This capability is

vital to the realization of robust recognition of handwritten characters or scripts, since characters

are rarely written twice in exactly the same form. There have been reports of successful use of

neural networks for the recognition of handwritten characters , but we are not aware of any

general investigation which might shed light on the systematic approach of a complete neural

network system for the automatic recognition of cursive character.

36
 CHAPTER 4

SOFTWARE AND HARDWARE

REQUIREMENTS

37
 Chapter 4

SOFTWARE AND HARDWARE REQUIREMENT

Hardware Requirement

1. I5 with 2.5 GHz,

2. 1 Gb RAM,

3. 500 Gb Hard Disk,

4. Graphic Adaptor,

5. Mathcore Processor,

6. High Speed Cache Memory,

7. Scanner.

Software Requirement
1. 32 bit Operating System,

2. Window 7, 8.

3. Matlab version 7.

38
 CHAPTER 5

IMPLEMENTAION

39
 Chapter 5

IMPLEMENTATION

Hopfield algorithm

The Hopfield has the following steps.

Step 1. One knows the input set I(x) and store it by computing the matrix W = Wnxn with
formula (1) or (2). One or more test vectors xr are given. Put t = 1.

Step 2. (optional) Compute the complement set I(xc) and the Hamming distance between the
vectors of I(x) and the test vectors xr. Check if the input set I(x) is orthogonal or not.

Step 3. At the time t, propagate the information from the Sx layer to the Sx layer by the formulas
(3) and (4). So we
nd the vector xr(t + 1) 2 Hn

Step 4. Put t + 1 instead of t and repeat the step 3 until there are no further changes in the
components of xr(t + 1).

40
Hebbian Rule

where : learning rate

t: spike time of postsynaptic neuron

: spike time of presynaptic neuron

: set of all firing times of postsynaptic neuron i

occurring after .

: set of all firing times of presynaptic neuron j,

adjusted for spike travel time (delay factor).

T: the current time-step

: learning interval (default is 100ms)

: learning window (see figure )

s:

41
 TRAINING SET OF CAPITAL ALPHABETS

All the images are handwritten with the help of stencil and are saved in .jpgformat , with

200200 pixel size.

42
TESTING SET OF CAPITAL ALPHABETS

These alphabets are handwritten individually by us and distorted, and after that saved in .jpg

format.with 200200 pixel size.

43
CODING FOR TRAINING SET
clearall; close all; clc;

total = 26;

%total1 = 52;

for counter = 1 : total

double myvar; double P; myasc=65;

charfname=char(myasc);

count = dec2base(counter,10);

myimg = imread( strcat(count,'.jpg')); % here Reading 1.jpg

fname = rgb2gray(myimg); % here converting A to gray.

fname = im2bw(fname,graythresh(fname));

%char fname1 = strcat(fname,'X');

%fname1 = edge(uint8(fname)); % imshow(fname1); pause; to see what

happening

%se = strel('square',2);

%fname1 = imdilate(fname1,se); % AX = imdilate(AX,se);

charfname2=strcat('P',fname);

fname2 = imresize(fname, [50 50]); % PA = imresize(AX,[50 50]);

fname2 = reshape(fname2, [2500 1]); % PA = reshape(PA,[2500,1]);

fname2=fname2*2-1; % Convering to bipolar

%imshow(fname2)

P{counter} = fname2; % to store 26 patterns but P is of Cell Type Array. P=[P1 P2 P3 P4 P5

P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 P21 P22 P23 P24 P25 P26];

end

44
p1=cell2mat(P); % Here Cell type array is converted to double type array to pass in hopfiled

network.

%P1=2*P1-1; % this is done at line no 17.

net = newhop(p1);

y = sim(net,26,[],p1);

Rg = plotregression(p1,y);

flag1 = 2;

while flag1 ~= 0

Ip = input('enter the number of image between 1 to 26');

Yp = y(:,Ip);

Yp = reshape(Yp,[50 50]);

imshow(Yp);

R1 = input('do you want to continue(1-yes/0-no)')

if R1 == 0

flag1 = 0;

break;

end

end

45
CODING FOR TESTING SET

total = 52;

doublepe;

for counter_1 = 27 : total

double myvar_1; myasc=65;

charfname1=char(myasc);

count_1 = dec2base(counter_1,10);

myimg1 = imread(strcat(count_1,'.jpg')); % here Reading 1.jpg

fname_1 = rgb2gray(myimg1); % here converting A to gray.

fname_1 = im2bw(fname_1,graythresh(fname_1));

% char fname_2 = strcat(fname_1,'X');

% fname_2 = edge(uint8(fname_1)); % imshow(fname1); pause; to see what

happening

% se = strel('square',2);

% fname_2 = imdilate(fname_2,se); % AX = imdilate(AX,se);

charfname_2=strcat('pe',fname_1);

fname_2 = imresize(fname_1, [50 50]); % PA = imresize(AX,[30 30]);

fname_2 = reshape(fname_2, [2500 1]); % PA = reshape(PA,[900,1]);

fname_2=fname_2*2-1; % Convering to bipolar

pe{counter_1} = fname_2; % to store 26 patterns but P is of Cell Type Array. P=[P1 P2 P3 P4

P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 P21 P22 P23 P24 P25 P26];

end

p2=cell2mat(pe); % Here Cell type array is converted to double type array to pass in hopfiled

network.

46
%net = newhop(p2);

yt = sim(net,26,[],p2);

rg1 = plotregression(p1,yt);

Yt = [y yt];

flag=1;

while flag ~= 0

Ipe = input('enter the no. of image b/w 27 to 52');

Ype = Yt(:, Ipe);

Ype = reshape(Ype, [50 50]);

imshow(Ype);

R= input('Do you want to continue (1-Yes / 0-No')

if R==0

flag=0;

break;

end

end

47
CODE FOR REGRESSION OF INDIVIDUAL CHARACTERS

x=imread('27.jpg'); % e.g here 27.jpg is name of the testing image A.

imshow(x)

x=rgb2gray(x);

x=im2bw(x,graythresh(x));

x=imresize(x,[50 50]);

x=reshape(x,[2500 1]);

x=2*x-1;

yt=sim(net,1,[],x);

pa=p1(:,1); % e.g here 1 is the name of training image A.

plotregression(pa,yt)

48
CHAPTER 6

RESULTS

49
 Chapter 6
RESULT

RESULT 1: FOR TRAINING SET

Regression line for original alphabets which is common to all, i.e. R=1.

50
Snapshots of training set(stored).

51
52
53
54
55
RESULT 2: FOR TESTING SET

Snapshots of testing set for each alphabets with its regression line.

56
57
58
59
60
61
62
63
64
CHAPTER 7

CONCLUSION

65
 Chapter 7

CONCLUSION

Pattern Storage net works have been used to efficiently store and recall the proto type patterns.

Efficiency of the Pattern Storage networks depends on a number off act or as like pre processing

techniques, learning method sad optedand the activation dynamics followed by the network. The

tradition all earning method i.e. Hebbian learning suffers from number of limitations. This can be

sufficiently taken care of by using the modified Pseudo inverse rule.

A machine initially cannot identify and recognize the handwritten English capital alphabets. But

with our coding, which we have done in MATLAB, our machine will both identify the character

and recall the original character.

66
REFERENCES

67
 REFERENCES
1. Aiyer, SVB, Niranjan, N, Fallside, F, 1990. A theoretical investigation into the
performance of the Hopfield model. IEEE Transactions Neural Networks 1, 204215.
2. Amari, S.I., 1972. Learning patterns and pattern sequences by self organizing nets of
threshold elements. IEEE Transactions on Computing 21, 11971206.
3. Atithan, G., 1995. A comparative study of two learning rules for associative memory.
PRAMANAJournal of Physics 45 (6), 569582.
4. Chartier, S., Lepage, R., 2002. Learning and extracting edges from images by a modified
hopfield neural network, In: Proceedings of the 16th International Conference on Pattern
Recognition 3, pp. 431434.
5. Chikkerur, S., Govindaraju, V., Pankanti, S., Bolle, R.M., Ratha, N.K., 2005. Novel
approaches for minutiae verification infingerprint images, In: Proceedings of
WACV/MOTION, pp. 111116, doi: eeecomputersociety.org/10.1109/
ACVMOT.2005.85.
6. Davey, N., Hunt, S.P., Adams, R., 2004. High capacity recurrent associative
memories.NeurocomputingIJON 62, 459491.
7. He, Y., Tian, J., Luo, X., Zhang, T., 2003. Image enhancement and minutiae matching in
fingerprint verification. Pattern Recognition Letters 24 (910), 349360.
8. He, Y., Tian, J., Li, L., Chen, H., Yang, X., 2006. Fingerprint matching based on global
comprehensive similarity. IEEE Transactions on Pattern Analysis and Machine
Intelligence 28 (6), 850862.
9. Hillar, C., Dickstein, J.S., Koepsell, K., 2012, Efficient and optimal little-Hopfield
autoassociative memory storage using minimum probabilityflow, arXiv:1204.2916, v1,
[nlin.AO].
10. Hrechak, A.K., McHugh, J.A., 1990. Automatedfingerprint recognition using structural
matching. Pattern Recognition 23, 893904.
11. Huang, Ke.,Aviyente, S., 2004. Fingerprint verification based on wavelet subbands, In:
Proceedings of SPIE 5404, Biometric Technology for Human Identification,
doi:10.1117/12.541912.
12. Huang, Z., Kuh, A., 1992. A combined self-organizing feature map and multilayer
perceptron for isolated word recognition. IEEE Transactions on Signal Processing 40,
26512657.

68
13. Isenor, D.K., Zaky, S.G., 1986. Fingerprint identification using graph matching. Pattern
Recognition 19 (2), 113122.
14. Jain, A., Hong, L., Bolle, R., 1997. On-line fingerprint verification. IEEE Transactions on
Pattern Analysis and Machine Intelligence 19, 302314.
15. Jain, K., Hong, L., Pankanti, S., Bole, Ruud, 1997. An identity authentication system
usingfingerprints. Proceedings of the IEEE 85 (9), 13651388.
16. Jeffrey, W., Rosner, R., 1986. Optimization algorithms: simulated annealing and neural
network processing. Astrophysical Journal 310 (1), 473481.
17. Kohonen, T., 1988.Self-organizing formation of topologically correct feature maps.
Biological Cybernetics 43 1982, 5969. (Reprinted in 1988, Anderson and Rosenfeld 1,
511-521).
18. Kohonen, T., 1989.Self Organization and Associative Memory.Springer,
Berlin.Kohonen, T., Ruohonen, M., 1973.Representation of associated data by matrix
operators. IEEE Transactions on Computers C-22 (7), 701702.Kumar, S., Singh, M.P.,
2012.Study of Hopfield neural network with sub-optimal and random GA for pattern
recalling of English characters. Elsevier-Applied Soft Computing 12, 25932600.
19. Labiouse, Christophe L., Salah, Albert, A., Starikova, I., 2002. The impact of
connectivity on the memory capacity and the retrieval dynamics of Hopfield Fig.
6.Comparative analysis of the pattern correction and recall efficiency of the Hopfield
network using standard Hopfield recall algorithm and with simulated annealing for the
FFT and DWT filtered feature vectors produced by SOM for 20%, 30% and 40%
PageFPCFromClipboardUntitled distorted patterns.
20. M.P. Singh, R.S. Dixit / Engineering Applications of Artificial Intelligence ( ) 13
Please cite this article as: Singh, M.P., Dixit, R.S., Optimization of stochastic networks
using simulated annealing for the storage and recalling of compressed images using
SOM. Eng. Appl. Artif.Intel. (2013),http://dx.doi.org/10.1016/j.engappai.2013.07.003i -
type networks.Proceedings of the Santa Fe Complex Systems Summer School, 7784.

69