Vinay Varghese

Introduction:
This report contains the work done to filter out the serial number from a South African bank note
and to store it digitally as text. The steps required to do this will require basic knowledge of image
processing as well as skills in working with Matlab and Python. All work in his report was done using
Python 2.76 and Matlab 2012b.

General remark:
Most of the work needed to be done in this report can very easily be done using built in functions
already present in Matlab. However, I have done as much as I could by using my own code and
functions that were constructed manually.




















Cropping of the Image:

Figure1: A digital image of a fifty rand bank note along with its dimensions in pixels

Ideally, only the segment of the digital image containing the serial code should be inputted into the
image to text converter that is to be made.
From looking at this example of a scanned note, one can see all the dimensions of critical sections.
The code to crop the note was made looking at the ratio of the required segment to the dimensions
of the whole note. The note was cropped 10% of its width to the right, around 40% down of its total
height and finally the note was cropped about 5% its total height from the bottom.
The code required to do this was implemented using python and is shown below.



The above code is the code used to manually crop out the segment with the serial code and to store
it was an image named croppedimage.jpg.
Below is an image that was the output to the cropping code.


Figure2: The cropped segment containing the serial code

This image was then send through into the next function, the filter.

Filtering of the Image:
Although there are many built in functions available in matlab for image filtering, I decided to make
my own filtering code in python. The two filters I managed to code were the median filter and the
mean filter.
The purpose of the filter is to try removing the background noise in the image and to end up with
just the serial code.
The two filters built will be discussed below:

1) The median Filter

The principle of the median filter is to run through the image pixels, replacing each pixel
with the median of the neighbouring pixels. This will results in some of the noise
disappearing in the image.

For my code for the median filter, I made a square matrix, and ran it through the entire
image, while the matrix runs through the image the code will continuously be taking the
median and placing it in the image. Two median filters were made, a 9x9 median filter and a
15x15 median filter. The filter was made in python by using for loops to run through the
image. The code for just the 9x9 median filter is shown in the following page:














The output to this median filter when the input is the cropped note is shown below:





Figure3: The output of a 9x9 median filter






Figure4: The output of a 15x15 median filter
As can be seen above, the output of the median filters are not good enough. The 9x9 filter results in
an image where the text has lost some of its quality and the 15x15 filter results in a unusable image.

2) The mean Filter
The mean filter works in a similar manner to the median filter described above. The mean filter was
made using 9x9 matrices and 15x15 matrices. The filter goes through all the pixels in the image and
replaces it with the average of the neighbouring pixels. For this reason it is safe to assume that the
smaller the matrix used, the more defined the picture will be, but at the same time one must know
that if the chosen matrix is too small in dimensions none of the noise in the background will be
filtered out.
The code made to implement this mean filter was made in Python 2.76. It is shown below:





Below are the images from the mean filter:





Figure5: The output of the 15x15 mean filter





Figure6: The output of the 9x9 mean filter




Figure7: The output of a 3x3 mean filter.

As you can the 15x15 mean filter results in too much of the image being blurred out. The 9x9 image
has text that can be recognized by eye but still doesnt look too clear. And finally, the 3x3 filter
results in a clear sharp image. The one downside to the 3x3 image is that some of the noise is still
present in the background. So from here on out the coding of the image to text program will be
done assuming that the output of the 3x3 mean filter is used.





Text recognition:
For the text recognition section, the coding will be done using Matlab 2012b. This is because Matlab
is far more than Python and offers more image processing modules.

There are several methods that can be used for text recognition. For the purpose of detecting serial
numbers on bank notes I have chosen the cross correlation method using a template. This was
chosen because it seems like the most reasonable option considering my limited knowledge in image
processing.

Creating a template:
The first step that needs to be done when detecting letters in this method is to make a template.
The template will be the set of images that will be used to compare to when the program is trying to
recognize text. In this case the template will be a series of images containing the possible characters
present on a serial code. The possible characters on a South African bank note is
ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890.
Note that the process of creating a template is shortened by the realization that only capital letters
can be found on the notes.
Below are some examples of the images present in the template:


Figure8: Template image of the letter A

Figure9: Template image of the letter S

Figure9: Template image of the number 2



Cross correlation:
The code now needs to somehow match up the figures in the template folder with patterns in the
segment containing the serial code. For this it is neccesary that the orientation of the figures in the
template be in the same orientation as the image containing the serial code.
Cross correlation will compare the image in the template with the original image and check for
matches. In other words the code will search for a segment of the image that will match up with a
image in the templates folder.
The cross correlation function will be used in Matlab and it will produce a graph that will show to
what extent the image in the templates folder is a match to the segment in the cropped and filtered
bank note.
The code will do this cross correlation for all images in the templates folder to map the serial code.
The images in the templates folder where made using the each letter from the already filtered bank
note.
Below is the code used to cross correlate. It will be used just to detect the number 4 for now:

This code has the following outputs:







Figure10: Image showing the detected character 4










Figure11: Graph showing correlation. Peaks represent the match to the template image used.

One can see from above that the 4 has been detected in the image using the template of the
character. The graph produced shows the amount of correlation. The location of the peak is related
to the location of the character found in the filtered image. This fact is of significant use. The
locations of peaks can be used to put the detected characters in order to compile the serial code in
text. This will be done in a latter step.
This form of template matching needs to be done for every character ranging from A to Z and 0 to 9.
In this way all characters present in the filtered image will be detected.

From Image to Text:
Now one of the final things that needs to be done is to convert the matches to the templates into
text. From tedious work I have concluded that if the template matches the letter on the bank note,
the correlation value was around 0.8 or more. So from this I made a function in matlab that looks for
peaks above the threshold value of 0.8. The letter that corresponds to the template as well as the
peaks x position is stored as variables in Matlab.
The code in the following page will look for the letter A and find its correlation value as well as its x
position. The outputs are shown in the next page:




Figure12: Code that correlates the letter A from template and finds its x position

The variable above named xbegin is very importent as it shows the letters position horizontally. By
arranging all the letters from smallest to biggest the program will automatically arrange all detected
values in order.
The above arranging of all xbegin values were done by placing them in an array then using a sort
function to automatically place them in order.
There are some problems that could arise from the above text recognition code. Firstly, the main
problem is if a character is repeated in the serial code. The problem of repeated characters is solved
by the use of threshold detection. By using this, the code will look for all peaks above the threshold
value, hence it will find all three 9 characters as is seen in the example code. The second problem
is that the code will not find the space between the second last and last character of the serial code.
This is not a problem that needs solving in my opinion and hence I have left that as it is.
Below is the array of values that represent X positions of the letters. The output was from matlab.




Figure13: Array with all x positions of letters from the template.

Now that the x values are acquire, the next step would be to arrange them from smallest to largest.
This will the represent the order of the letter. Below I have shown the arranged values.









Figure14: Array representing x positions of detected letters in an arranged manner from left to
right.

Now all that is needed to complete this task is to match up these x positions to the letters found
from the template.
This is done by a simple code section in MatLab. The code will simply ask MatLab if the letters
position corresponds to the first value of the array named arrayofxvaluessorted. If it is, then the
first letter is that corresponding template image. If not, the code must try the next letter and so on.
This is done using forloops and if statements.

Figure15: Code that compares if B is the first letter, if it is it stores it in the array.

The code above is made in conjunction with forloops in the actual code. If for example, the first
letter isnt B, then it will test C, then D then so on. In this manner it will go through every possible
character and place it in order as an array. In the end the array will just be converted into a string
and displayed.
The final output of the example code is shown below:

Figure16: Final output of the program.

This is the end of the character recognition code. As one can see above, the text was correctly
recognized and printed as a string. This is as far as I went because I am working through this project
individually and not in a group. The code worked moderateley well for most of the note I tested.
The code however, did not work for the image with the note in the wrong orientation. This problem
could be easily solved in a number of ways. The simplest way being, if no text is recognized by
Matlab, it is almost safe to assume that the note is in the wrong orientation. I will use the rotate
function in the python section of my code to put the digital bank note back in the same orientation.
By implementing that simple step, all note can be detected irrespective of its its positional
orientaion.
For the sake of completion I have decided to try input my filtered image into a external image to text
module in python. This module is named Tesseract.
Tesseract is a module that is available in Python and is easily implemented. I have just added this
section so that I can compare the performance of Tesseract vs the performance of the code that I
have made.
The output is shown in the following page:


Figure17: The output of my python code used in conjuction with Tesseract.

As one can see, the program converts the filtered image into text with zero error. This is a fantastic
result and is quite clearly better than my image to text script that I made with matlab and python.
The performance time is unfortunately a bit slow. This is most likely because the entire code is done
in Python 2.76 and its performance speeds cannot be compared to that of Matlab.

Throughout this report, steps have been documented to show what was done to implement an
image to text recognition program. Additionally, the alternate method of using an external module
to do this task is also shown to make a comparison.

Comments:
This coding could have greatly been simplified by using the built in filters in Matlab instead of coding
one manually in Python, however, for the sake of doing every step manually in the process of text
recognition I have made my own filters. The performance of the matlab filters would have most
likely been better than the python mean and median filters that were created.
Refferences
http://www.mathworks.com/help/images/ref/normxcorr2.html;jsessionid=6a0df9d726b90dba9dfef
8ce3418