Czech Technical University in Prague

Faculty of Electrical Engineering

Department of Computer Graphics and Interaction

Bachelor thesis

Electronic Keyboard Tutorial System

Petra Kalinov

Supervisor: Ing. Adam Sporka, Ph.D.

Study Programme: Software Engineering and Management

Field of Study: Web and multimedia

June 11, 2009

iv
Declaration
I hereby declare that I have completed this thesis independently and that I have listed all
the literature and publications used.
I have no objection to usage of this work in compliance with the act 60 Zkona
. 121/2000Sb. (copyright law), and with the rights connected with the copyright act
including the changes in the act.

V Praze dne 11. 6. 2009 .............................................................

v
vi
Abstract
The purpose of this work is to design and implement an application used by beginner
pianists to play songs using an electronic keyboard connected to the computer via a Musical
Instrument Digital Interface (MIDI). The program can serve as an educational tool as well
as a means of entertainment for people who have already mastered playing the piano, due
to its style diering from standard musical notation by the way it symbolizes the tones
which are being played. An educational element is represented by the set of editable lessons
containing individual instructions and references to particular songs in Standard MIDI File
Format. Each session may focus on a specic piano playing problem and while the song is
practised an evaluation is provided.

Abstrakt
Zmrem tto prce je navrhnout a implementovat aplikaci umoujc zanajcm pi-
anistm hrt skladby na elektronick klvesy pipojen k potai pomoc rozhran MIDI.
Tento program me slouit nejen jako vukov pomcka, ale tak jako zdroj zbavy pro
uivatele, ktei u zkladn techniku hry na klavr ovldaj, protoe zpsob, jakm je
skladba zapsna se li od bn pouvan hudebn notace tm, jak jsou symbolizovny
jednotliv noty, kter se maj hrt. Vukov element je reprezentovn formou editovatel-
nch lekc, kter obsahuj konkrtn pokyny a odkazy na skladby uloen v MIDI souboru.
Kad lekce se tak me zamit na specick problm. Hodnocen hre probh jen-
dotliv po kad pehran skladb a slou jako rozhodovac faktor pro jeho postup na
dal obtnostn rove.

vii
viii
 Contents

List of Figures xi
1 Introduction 1
1.1 Traditional music education . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Specication of the goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Relevant techniques and technologies 3

2.1 Alternative music notations and music storage systems . . . . . . . . . . . . 3
2.1.1 Piano roll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 Klavarskribo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.3 Standard MIDI File format . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.4 MusicXML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Embedded solutions in electronic keyboards . . . . . . . . . . . . . . . . . . 5
2.2.1 Electronic keyboard . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Yamaha Education Suite . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Music video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 Konami KeyboardMania . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3.2 Guitar Hero and Rock Band . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.3 Synthesia - Piano for Everyone . . . . . . . . . . . . . . . . . . . . . 8
2.4 Research summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 System requirements specication 11

3.1 System requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Requirements specication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.1 Evaluating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.2 Playback system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 User interface design 13

4.1 Representation of music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Representation of piano keyboard . . . . . . . . . . . . . . . . . . . . . . . . 15

5 System architecture 17
5.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.1.1 MIDI module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.1.2 System module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2 Management of input and output . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.1 Screen management . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.2 Data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.3 Graphics management . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6 Implementation 21
6.1 Software development tools and libraries . . . . . . . . . . . . . . . . . . . . 21
6.1.1 Programming language . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.2 Graphics framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.3 Integrated Development Environment . . . . . . . . . . . . . . . . . 22
6.1.4 MIDI Application Programming Interface . . . . . . . . . . . . . . . 22
6.2 Implementation highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

ix
 6.2.1 Application graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6.2.2 PianoRoll library overview . . . . . . . . . . . . . . . . . . . . . . . . 24
6.2.3 XML object loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2.4 Input from keyboard . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2.5 External tools for lesson authoring . . . . . . . . . . . . . . . . . . . 25

7 Testing 27
7.1 Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
7.2 Testing process and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

8 Conclusion 29
8.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

9 Bibliography 31
A List of acronyms 33
B UML diagrams 35
C Application screenshots 39
D Usability testing questionnaire 41
E User manual 43
F CD Content 45

x
List of Figures
2.1 Original piano roll, source [16] . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Piano roll in MIDI software . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Example of Klavar notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Yamaha's lighted keyboard, source [17] . . . . . . . . . . . . . . . . . . . . . 6
2.5 Screenshot of KeyboardMania video game . . . . . . . . . . . . . . . . . . . 7
2.6 Guitar Hero video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.7 Synthesia video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.1 Final design of playback UI . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.2 2D design of playback UI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.3 Playback screen storyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.4 Other possible designs of playback UI . . . . . . . . . . . . . . . . . . . . . 15

5.1 Hierarchy of modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2 Diagram of application's menu ow . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 Diagram of objects in XML le . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.4 Graphics lifecycle in application . . . . . . . . . . . . . . . . . . . . . . . . . 20

6.1 Scheme of connection between computer and electronic keyboard . . . . . . 22

6.2 Orthogonal views of playback scene with camera viewports . . . . . . . . . . 24
6.3 Relationships between PianoRollLibrary namespaces . . . . . . . . . . . . . 24
6.4 Mockup of external authoring tool . . . . . . . . . . . . . . . . . . . . . . . 26

7.1 Illustration of test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

B.1 User's use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

B.2 Design of data model classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
B.3 Interaction and relationships between PianoRollLibrary members . . . . . . 37
B.4 Screen management classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

C.1 Main menu screen with static background image . . . . . . . . . . . . . . . 39

C.2 Playback scene before tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
C.3 Playback scene adjusted according to test results . . . . . . . . . . . . . . . 40

xi
xii
CHAPTER 1. INTRODUCTION 1

1 Introduction
Playing the piano is a skill usually learnt during childhood from professional teachers,
typically in weekly sessions. Although the grand piano has limited mobility and is often
expensive, it is still one of the most familiar musical instruments and the most favoured to
begin with. In recent years, popularity of keyboard instruments has been boosted by the
increasing quality and the decreasing price of more portable versions of piano  the digital
piano and the electronic keyboard.
Looking at the age of beginner pianists there is a signicant amount of adults who decide
to learn to play the piano. However, most potential students give up the idea even without
trying. The reasons might dier from case to case but the majority are frightened by
the necessity of the traditional education system which could appear to be too pedantic.
Furthermore, there is the musical notation which might scare o another group of people,
although reading music is not as hard as it seems at rst sight.
Having a talent for music is not the key point to successfully learn playing any musical
instrument as it requires lot of patience as well as regular exercising and nothing can express
the whole principle more accurately than the phrase: Practise makes perfect. In addition
the gained skills must be maintained by occasional training and can be even completely
forgotten after some time.

1.1 Traditional music education

In the Czech republic, a type of school has been established called Zkladn umleck
kola (ZU) where the education of dierent art forms including music is learnt. These
schools have a long lasting tradition in every major town and are also responsible for
organizing various local competitions of young artists. Although they are administrated
by the Ministry of Education, Youth and Sports (MMT), the education expenses are
partly covered by students' tuition fees.
Most of the time spent in front of the piano keyboard during the rst few sessions is
used for learning the position of each key on the sta and by the time the student gets this
into practise, he might be already bored or fed up enough to discontinue further training.
While children are usually forced to attend the lessons by their parents, the adults who
decide to play on their own are sacricing their free time and spending their money, and if
they feel that both could be used for something more enjoyable, they simply stop and the
negative experience may make them possibly never try it again.
Before beginners rst sit at the piano they have to attend a few music theory lessons and
become familiar with specic terminology and the principal of notation. The student then
continues with acquiring the basic playing technique and the identication of the correct
piano key according to the notation. The learning process as a whole consists of the main
topics which are: ear training, rhythm, notation, technique, improvisation, sight reading
and memorization [2]. The actual teaching method has to adapt according to student's
dierent personality, learning style, education level, and so on.
The requirement of individual education plus the high purchase cost of the instrument
itself makes piano training expensive and therefore is not aordable for everyone. The idea
of a complete solution for teaching oneself to play the piano using either printed or an
2 CHAPTER 1. INTRODUCTION

electronic tutorial is not the right one. As it applies mostly to motor skills; the bad habits
gained at the beginning are later hard to get rid of. At least an occasional supervision by
a professional piano teacher who checks the technique and recommends suitable exercises
for further training is necessary.

1.2 Specication of the goal

The aim of this thesis is to implement a tool which would make the rst steps of beginner
pianists more comfortable and enjoyable than it would have been while applying the classic
learning methods. The plan is to nd another way of displaying the notes in the music
being played so the user can avoid the necessity of learning the common notation in the
rst place. The requirement for frequent practise highlights the importance of the system's
ability to maintain the user's motivation to repeat the songs over and over. The motivation
used for the same purpose in the computer games is to record an amount of points gained
during the song playback in a form of score.
As stated in the project's assignment, the tutorial system requires an electronic keyboard
connected to the computer using the Musical Instrument Digital Interface (MIDI) which
will be described in more detail further in the text. The possibility of communication be-
tween the instrument and the application brings forward an essential interaction with the
user and provides them with feedback from the system to their actions which is a funda-
mental principle of each learning method. These two elements, interaction and feedback,
are the reasons for the decision to connect the devices together and bring the piano lessons
into another level.
The development of our application will begin with background research of technolo-
gies which have something in common with our system. According to the results of our
investigation, we will be able to dene the requirements of the system. Then we specify
its concrete functions which will be further designed and implemented. The qualities and
limitations of nal product should be revealed during the tests.
CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 3

2 Relevant techniques and technologies

This chapter consists of an explanation of the situation regarding piano playing lessons
which will help us with identifying the problems which we should be aware of. Then we
look closer at the notation system and analyze the non traditional ways of displaying the
music. The description of existing either educational or entertainment systems follows.

2.1 Alternative music notations and music storage systems

Music notation is a system which describes music using written symbols. The way of nota-
tion has been evolving since the prehistoric ages similarly to the writing system. Modern
music notation has its origin in European classical music of 18th century and has been
widely used throughout the world until now. However, there has always been the inten-
tion to discover an alternative which would facilitate music-reading. With an increasing
improvement in the area of technology, music production has moved, during the centuries,
from mechanical systems such as orchestrion to electronic devices capable of not only re-
playing but also storing and editing the music. Therefore the need for translation of the
paper notation into its electronic form has appeared. Common audio formats used for the
sound storage and by audio players usually manipulate the actual audio wave unlike the
descriptive music le formats. Examples of mentioned technologies follow in this chapter.

2.1.1 Piano roll

Piano roll is the rst industrially reproduced music storage medium used by self-playing
pianos, i.e. Pianola. It is made of the roll of paper with perforations which determine
the position and length of the note played on the piano. The roll moves over a specic
device with holes, one for each piano key. When a perforation passes over the hole, the
note sounds. This mechanism now belongs to the museum but the term piano roll is still
used in MIDI data editing software as a name for another option of showing the data
in dierent way from the standard musical notation. Piano roll typically provides colour
coding, variable-length symbols to indicate note duration, and a chromatic keyboard-style
layout. The examples of the historical and modern look of piano roll style are shown in
Figure 2.1 and Figure 2.2.

Figure 2.1: Original pi- Figure 2.2: Piano roll in

ano roll, source [16] MIDI software
4 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.3: Example of Klavar notation

2.1.2 Klavarskribo
Klavarskribo is a music notation introduced in 1931 by the Dutchman Cornelis Pot. It
distinguishes from the conventional notation by being easily readable. However, it hasn't
met with great success among music teachers and the only signicant group of people using
the notation in practice nowadays are the members of the promoting organizations in the
Netherlands [10]. Although the notation is universal for all instruments and for singing it
suits keyboard instruments the most since the design logic is based on the piano keyboard.
The example of Klavarscribo can be seen in Figure 2.3.
The klavar notation itself is unique in the way the stave is placed on the page and
corresponds to the real piano keyboard more than the traditional layout which consists
of ve horizontal lines and four spaces whereas in the klavar stave, the notes are written
vertically and the music is read from top to bottom. The lines match the position of the
black keys on the keyboard thus each note has its own individual place and no sharps, ats
or even any key signature are needed. In other words, Klavarskribo presents a notation in
its straightforward form, stripped of the music theory. Whether the note is played by right
or left hand is indicated by the orientation of note's stem. The Klavarskribo Foundation
has transcribed over 25,000 pieces and custom composition can be transcribed easily with
the aid of specialized software, the KlavarScript.

2.1.3 Standard MIDI File format

The Musical Instrument Digital Interface is a specication which was rst announced in
1982 and denes a communication protocol used for transfer of music data in a form of
MIDI messages through MIDI devices. Although the standard was primarily designed for
keyboard instruments only, it was soon extended to special drums, guitars and even enables
the lighting to be synchronized with the music during live performances [1]. Apart from the
specic protocol, the MIDI Manufacturers Association (MMA) also dened a specication
for the format for storing MIDI messages, formally called a Standard MIDI File (SMF)
format.
Remarkably this le format is small in size in comparison with the Waveform audio
format (.wav) where the actual sound wave is written directly in the le contrary to the
MIDI le format (.mid) which contains just the data describing the music and the specic
CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 5

representation of the sound depends on the current synthesizer. Another advantage over
the wave storage system is ease of editing, composing or recording the music data including
the possibility of its conversion to the printable standard notation eventually.
SMF is dierent from native MIDI protocol, because the events are time-stamped for
playback in the proper sequence. The data are stored in hexadecimal form and are struc-
tured into two dierent types of chunks, header and track. The le does not contain only
the notes but also stores the information about instrument used to replay the track in
a form of specic instrument patch which is identied by a Program Number parameter
with a range of 128 possible values where each number corresponds to one melodic sound.
The portability between several synthesizers is ensured by the instrument table dened in
General MIDI (GM) standard.

2.1.4 MusicXML
This type of music notation is not meant to be used by the players directly but can serve as
a storage format further used by music production software. The transcription is based on
Extensible Markup Language (XML) which is the format widely used by automated tools
to be parsed and manipulated. MusicXML was designed for the interchange of scores.
Although it can be possibly written by hand, a more practical approach is to use an
interactive score writing program. More details can be found in [12].

2.2 Embedded solutions in electronic keyboards

After the examination of the paper and digital representation of music we can move to
its actual producing in a form of sound output from electronic device such as keyboard,
which will serve as an input for our application. In addition, one interesting attempt of
an educational system which is built-in and shipped with the instrument as a complete
solution will be presented.

2.2.1 Electronic keyboard

A keyboard is an electronic device resembling a grand piano by having similar looking
musical keyboard. While it lacks not only the piano's size and weight but also its timbre
and keystroke, it can actually sound like more than one instrument by a simple change of
the settings. Although the generated reproduced audio limits the nal credibility of the
instrument's sound, it oers another big advantage against the classic piano which is the
possibility of sound volume regulation or even the redirection of the output into a headset.
Today, a variety of dierent types of keyboards exist on the market. From inexpen-
sive instruments targeted at amateurs and children to truly professional synthesizers rich
in many features. Nonetheless even the mainstream is able to show o many functions
and oers reasonable overall quality due to recent advances in computer and electronics
technology.
In addition to a large tone bank, the instrument typically contains pre-programmed
rhythm patterns for several musical styles presented as an auto-accompaniment. The key
function in the sense of piano imitation is the touch response, also called touch sensitivity
6 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.4: Yamaha's lighted keyboard, source [17]

and does not have to always be included. The optional parts of the keyboard are additional
sockets placed on the back which allow the user to plug MIDI or audio inputs and outputs.

2.2.2 Yamaha Education Suite

Yamaha belongs to the group of electronic piano manufacturers and its products are well
known among the musical public. Various keyboard series contain the built in Yamaha
Education Suite (YES) which is aimed at helping the user with piano playing. Several
dierent versions have been developed since 1998 [17].
The suite contains a system of lessons, built-in songs and a chord dictionary which is
there to display the notes and chord names on the LCD screen. The lesson itself can be
displayed in numerous modes in order to provide more ways to practice. Short descriptions
of some modes follows. Timing mode allows the user to play the melody by pressing any
key and only the correct timing is important. Waiting mode stops the playback of a song
until the player nds the correct note and then continues. Minus One mode plays the song
at the normal tempo minus the part the player has chosen to play. Each hand can be
trained separately and in addition, the correct ngering is indicated on the LCD display
while the lesson is running. The lesson system wouldn't be complete without the progress
monitor to present lesson grading.
The Yamaha EZ series oers a helpful feature which is a keyboard with LED embedded
keys allowing interactive learning. Figure 2.4, shows such an instrument. On these models
a red light operates as a guide to tell the player which note to play and when. In earlier
versions of keyboards the lights were placed right above the keys while the latest ones are
made to have the eect of the key itself emitting the light. To facilitate the learning process
even more, the song stops and waits for the player to nd the correct key thus letting them
comfortably play the song on their own. Not only Yamaha but also Casio, another well
known brand producing electronic keyboards, oers similar illuminated keyboards.

2.3 Music video game

This section gives a brief overview of products of the gaming industry which belong to the
music-themed video game genre and can be closely specied as a rhythm game, a subgenre
of action game. These games are based on simulating the playing of musical instruments
where the player's accuracy is further evaluated. Achieving a high score is the main game-
play factor, further boosted by the support of multiplayer mode. The specic instrument
CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 7

Figure 2.5: Screenshot of KeyboardMania video game

used has a function of game controller and is either in its convenient or simplied version.
The players follows the instructions on a scrolling display which in principle is similar to
piano rolls. The short description of arcade and console games in this chapter is followed
by a presentation of a software application which has its origin in the previously mentioned
games and the development of which was stimulated by their increasing popularity. These
games have a lot in common and can inspire us in the playability of the elements they
contain.

2.3.1 Konami KeyboardMania

Keybaordmania is a rhythm video game published in 2000 by Japanese company Konami,

the producer of video games and toys and is actually an owner of several music game-
related patents. Since 1997, the company's Bemani division, has released many dierent
series of music games which have met with greater or lesser success but denitely served as
inspiration for other products of the genre further described in the next section. Benami`s
most successful rhythm game which, unlike the other Konami's products, has crossed the
border of Japan and was released in North America and Europe in 1999, is Dance Dance
Revolution (DDR) where players control the game by the movement of their feet on a
special pad, in the order dictated by on-screen instructions. it is remarkable to notice that
this game is now used not only for its entertainment purpose but also as a an exercise tool
[5].
The notes in Keyboardmania are represented on-screen by small bars that scroll down-
ward above an image of the keyboard itself. The goal is to play the matching key when
a note bar descends to the red play point line. The game has several Japanese releases
intended for arcade cabinets, two home versions for Playstation 2 which use a special key-
board controller and a Windows PC version shipped with the Yamaha EZ-250i lighted
keyboard a screenshot of which can be seen in Figure 2.5. The last mentioned software
brings the concepts of Yamaha's YES and allows users to practice their playing in dierent
levels and even load any General MIDI song. However, the game is not included with the
latest version of Yamaha's lighting keyboard, EZ-200 [17].
8 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES

Figure 2.6: Guitar Hero video game

2.3.2 Guitar Hero and Rock Band

Guitar Hero, released in November 2005, and Rock Band, released 2 years later, both belong
to the same category of modern music video games distributed on various video game
consoles. Each of them requires special controllers in the shape of a musical instrument
to play built-in songs of various genres. The increasing popularity of these games led to
the development of several custom clones which mostly interact with MIDI les and are
controlled using a PC or MIDI keyboard, i.e. Synthesia [11].
The most widely known is a guitar peripheral with ve or more colored buttons on the
neck and a strumming controller. The mechanics of the game lies in pushing down the
instrument's color matching buttons according to the indicators rolling on the board in
music's rhythm, at the right time. The rating is represented on a meter which should stay
green and only goes red when a player misses too many notes which can even cause the
song to stop completely if the performance is too bad. For better imagination, the picture
of the playback screen can be seen in Figure 2.6.
The game supports not only single player modes but also various multiplayer modes
where the songs are performed by more players at the same time. Unlike most rhythm
games, Guitar Hero and Rock Band have a career mode that guides the player through
the diculty settings and oers decently conceived learning curve which makes the game
suitable not only for experienced gamers but also for entire beginners. Another remarkable
point witch makes those two games dierent from most of their predecessor and clones, is
the graphics design suiting the genre and a party game mode has been made for it as well.
Although the board with rolling notes dominates the layout, there are other elements such
as animated background which does not lead the player's attention away and only makes
the screen more interesting for observers.

2.3.3 Synthesia - Piano for Everyone

Synthesia is a computer game developed by Nicholas Piegdon with the rst release pub-
lished in November 2006 and the latest one which is dated from September of 2008. It
allows the user to play songs in a MIDI format using a musical keyboard connected to
the computer. The project was initially named Piano Hero due to the inuence of Guitar
Hero; however, it soon had to be renamed as the author received a cease and desist letter
from Activision, the publisher of the Guitar Hero [11]. A player determines which piano
key should be played from the position of the note falling from the top of the screen and
travelling towards a piano displayed at the bottom as is shown in Figure 2.7, thus no sheet
music reading is required. The users can choose whether the track or the tracks of the
MIDI song will be played by them or automatically by the program. A basic evaluation
CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES 9

Figure 2.7: Synthesia video game

system in a form of per-song scoreboards which enables the players to keep track of their
progress is included. While the basic functionality is still free, a Learning Pack key can be
purchased to unlock additional features, such as a sheet music display mode.

2.4 Research summary

Looking closely at all the described products the YES ts our goals the most. Its users
can largely prot from oered modes while practicing their playing but are heavily limited
not only by the built-in selection of songs but also by the small screen with low resolution,
placed on the keyboard. The lighting key feature can serve well when playing an easy song
but with an increasing diculty the player would appreciate the knowledge of incoming
notes in order to prepare the ngers in time.
Piano roll notation seems to be suited more to music editing than learning and music
performance due to lack of guidance on which hand or nger plays the note, the elements
which are present in standard notation and Klavarskribo. The piano roll style is widely
used in the mentioned video games but those systems are not designed to teach the user to
play the real instruments unlike the educational purpose of our application. We will take
inspiration from the gameplay elements which we think should not be missing even in a
tutorial application.
Synthesia, which also uses a piano roll style for representing the music, seems to be the
application which suits our ideas about the system. Simple and well-arranged 2D layout
with the set of horizontal equidistant lines, following which the user should get used to
playing quite fast seems to be the right representation of the piano roll style. Although
Synthesia oers the mentioned optional learning pack, it consists of MIDI les of several
diculties only and, apart from sheet music display, it does not contain any additional
instruction info about piano playing technique.
While designing our application we will try to follow the user interface (UI) design
concept of previously mentioned systems together with an attempt to keep the program
highly educative and interactive at once with a regard of remarks gained in this research.
10 CHAPTER 2. RELEVANT TECHNIQUES AND TECHNOLOGIES
CHAPTER 3. SYSTEM REQUIREMENTS SPECIFICATION 11

3 System requirements specication

In this section the requirements of the system dened in the project's assignment are listed
and described in more detail, together with other requirements which follows the summary
from the previous section.

3.1 System requirements

Following functional requirements dene what the system should do.

The system allows the user to play the songs without the player's knowledge of
musical notation.

The system provides the instruction information about piano playing in the form of
complete lessons.

Lessons have a specied number of songs which are required for their completion.

Lessons consist of an individual amount of songs and it is up to user which songs

they choose to play.

A song is considered completed when a required percentage of score points is gained

by the player.

Before the song starts playing, the user can choose to play the song in a slower tempo
but is penalized as the score is evaluated.

While playing the song the user is informed about the remaining time of the song
and the amount of score points gained.

The song's playback can be paused by the user.

When the song is nished the user is informed of their achievement.

An electronic keyboard and PC are connected together via a MIDI interface.

Lessons data are dened in an external le which is possible to edit.

The songs are identied by their names; the song's author might be added as well.

The application can be exited at any point by pressing and conrming the escape
command.

The system's behavior from the user's point of view is described in a use case diagram
placed in Figure B.1. End users of the system are people of any age who are willing to
connect two devices together and spend their free time playing the keyboard in front of
their computer.
12 CHAPTER 3. SYSTEM REQUIREMENTS SPECIFICATION

3.2 Requirements specication

First of all, the user needs to have an electronic keyboard connected to the computer
via a USB MIDI interface in order to be able to participate in the educational process
which consists of several lessons where new players can register and participate. The non-
educational mode is called Quickplay mode and while performed, no results related to the
player are stored in the system. After the system's launch the user is informed if any MIDI
input device have not been recognized. The user controls the application using PC or
piano keyboard to navigate through the menu. The congurable options of the system are
located in an options menu and consist of resolution, MIDI input and output settings and
a toggle full screen mode entry. The song data are kept in the SMF while the additional
information like the song's name, its author, the default tempo, the diculty and the track
being played are stored in the system's data structures. The user commands and external
les can be considered as system's input.

3.2.1 Evaluating system

The player's achievement is expressed in a form of score points. Several possible diculty
modes of song exist. The simplest mode is when only the correct rhythm is evaluated and
the key user has pressed is not important allowing him to pay attention at the song itself,
not the positions of played notes. Another mode is when the correct oncoming note is
pressed but the exact time of its press is not evaluated. Normal mode is when both the
rhythm and correct note is rated with a tolerance which diers with the playback tempo.
The tolerance is lowest at the slowest tempo. The diculty mode currently used is dened
in the lesson's data and the user is given a notice about the actual system of evaluation
before playback starts.

3.2.2 Playback system

As already mentioned in Subsection 2.1.3, the SMF song consists of one or more parallel
tracks where each track actually represents one part of the overall piece of music and is
assigned to a specic instrument. For example the melody performed on the trumpet is
stored in the rst track while the piano accompaniment is stored in another. The system
currently supports only one graphical representation at a time.
The interpretation of the playback as it is presented and designated to be played by
the user lies in the conversion of a specied track to its real time graphics representation.
While the designs of various playbacks used in researched programs aimed to be performed
on the keyboard are in two-dimensional (2D) view, we have decided to originate our play-
back by using an attractive looking three-dimensional (3D) view. The aim is to make
our design intuitive and easily readable for the players who are not familiar with any of
described similar tools. The challenge rests in representing the whole keyboard containing
several keys unlike the described guitar or drums peripherals with maximum of ve dif-
ferent positions to watch and play. Although a few of designs presented further had been
outlined before the actual application implementation, several details have been adjusted
in the functioning program prototype and after the application usability tests described in
Chapter 7.
CHAPTER 4. USER INTERFACE DESIGN 13

4 User interface design

This part of the product design process usually consists of an activity called prototyping.
A prototype is a form of early product's model with limited functionality the purpose of
which lies in verifying the suitability of a specic design approach and allows comparison
between alternatives. The advanced version of prototype is a mockup which provides at
least part of the functionality of a system and enables testing of an actual design. In the
following section a few drafts of our system which can further serve as a basis of the paper
prototype will be presented.
Because the playback screen is the key point of the whole application, the following text
is focused on its design. The arrangement of the playback screen's graphical user interface
(GUI) depends primarily on the position of the keyboard model, since it is the largest
static element on display. As mentioned in the previous chapter, the scene is planned to
be created in 3D space rendered onto 2D screen from a static position of a camera. The
perspective view of the piano keyboard has been chosen because of the more realistic look
and also because the object will take less space on the screen. Although the keyboard
is not meant as the dominant part of the screen, its position in the lowest bottom part
ensures that it is still well visible and also as close as possible to the real piano keyboard,
placed in front of the computer screen.
Apart from rendered 3D playback scene, the screen used for playback contains the Heads-
Up Display (HUD), the static part of user interface which in video games is used to display
information important for player during the game. Our HUD contains only two elements
which is the amount of score points and the remaining time of the song as can be seen in
Figure 4.1. Figure 4.2 shows the 2D concept, used in Synthesia application mentioned in
Subsection 2.3.3, to enable the reader straight comparison between two dierent views.

4.1 Representation of music

Another part of the 3D scene is the graphical representation of the notes the user will play
for the musical part. As stated at the end of the research chapter our application will stick
with the piano roll style as it is easy to transform from the SMF and intuitive enough for
the player to read. Each note of the played track is shown as a box directly proportional
in length to the note's duration above the corresponding piano key. The box moves toward
the piano key model and when its near edge touches the key, the key should be pressed

Figure 4.1: Final design of playback UI

14 CHAPTER 4. USER INTERFACE DESIGN

Figure 4.2: 2D design of playback UI

Figure 4.3: Playback screen storyboards

and when the far edge leaves the screen the key should be released. The illustration of the
just described process can be seen in Figure 4.3.
The boxes appear on the screen in advance before they should be actually played to
give the player time to prepare. When playing an advanced piece of music the technique
of preparing ngers to incoming notes applied while playing from the standard notation
needs to be used [2]. The beginner user should follow the lesson system carefully to get
used to more dicult pieces automatically. The success of the overall education system
lies in the lesson's design, which is not the exact purpose of this work and brings out a
suggestion for further work.
The lines heading to the keys on which the boxes trace have a dierent colour according
to the octave where they end. Another line crosses the line where the keyboard begins at
the time where the downbeat appears in the song bringing in a replacement for regular bar
line known from standard notation. The position of the lines is in the same plane as the
CHAPTER 4. USER INTERFACE DESIGN 15

Figure 4.4: Other possible designs of playback UI

keys are and the perspective view makes the boxes get bigger when they are come closer
or in other words are going to be played. This design has been chosen over the other two
concepts shown in Figure 4.4 which were rejected due to their being less able to adapt to
the scene requirements.

4.2 Representation of piano keyboard

If the electronic keyboard is connected and the system is receiving the MIDI input messages
the models of keys on the screen move up and down as the user presses or releases them.
Timestamps of the notes in the songs and of the played keys are the data for evaluating
the user's score. The number of keys which can be displayed on the screen is limited by the
current screen resolution which can be customized in the Options menu. The minimum of
octaves displayed is the number to cover the actual song range and others are shown to
ll the space on the screen and since not used, they are shaded to express their inactivity.
Already referenced Figure 4.3 also shows the various cases of the piano key states in the
combination with the dierent position of corresponding note box.
16 CHAPTER 4. USER INTERFACE DESIGN
CHAPTER 5. SYSTEM ARCHITECTURE 17

5 System architecture
After we got through the specication of the program's functions and we have a rough idea
of its graphical look, it is time to plan the solution from the software architectural view
and to build preliminary data models. The rst steps involve the division of the system
into its separate objects so the application can be later easily implemented using object
oriented programming. The goal of the design is extensibility so the new capabilities can
be added to the software without major changes to the underlying work. The ideas for
future work are introduced in the nal part of this thesis.

5.1 Components

Beginning with the design from the global view, we have decided to split the system
as a whole into three logical parts where each part has its own functionality capability.
The hierarchy of those parts including the MIDI API is shown in Figure 5.1. The rst
module, which uses services of the MIDI API is the library managing the MIDI commands,
the second is the system library containing the graphical and MIDI related objects and
methods. The last but not least is the part where the application's entry point is placed
and which is responsible for the screen and data management.

Figure 5.1: Hierarchy of modules

5.1.1 MIDI module

The purpose of this library is to encapsulate the low-level functions from the system's
MIDI API library for their later manipulation in the program. It should be able to handle
the data retrieved from the library on the level of the objects as well as read the data from
the SMF. All the abstract MIDI entities such as input and output devices, sequencers,
sequences, tracks, clocks etc. should have their object equivalents in this library and the
methods streaming the MIDI data should be provided. The discussion about usage of the
external solution for this task is discussed in Subsection 6.1.4.

5.1.2 System module

This library represents a bridge between the MIDI objects and the application's main
components and as it performs dierent functional assignments. It is essential to divide it
into more parts:
18 CHAPTER 5. SYSTEM ARCHITECTURE

MIDI  methods and objects which communicate directly with the MIDI library.

Recognizing, identifying and communicating with MIDI input and output devices.
Classes for sending and receiving various MIDI messages.

Song  methods working with objects storing the data from MIDI song le.

Parsing the content of the SMF and saving them into tracks which consist of notes
where each note has its pitch and duration.
Class which gathers needed data and perform the MIDI song playback with its own
MIDI clock. Also includes the displaying of 3D models.

3D model  contains parts of the playback scene.

Loading the graphical objects from les and displaying them.

Input  handling the input from the user's electronic keyboard, if connected.

Exception  specic exception to be thrown in potentially unstable places further caught

on the higher level.

Score  calculating the score points when the piano key is pressed. The amount depends
on the actual timestamp and other parameters.

5.2 Management of input and output

5.2.1 Screen management
Screen manager is a class responsible for displaying various screens according to the current
state of the system in which it occurs and which depends on players input mainly. The user
can navigate the menu either using the PC or the piano keyboard following the descriptive
icons which are part of the background image. The concrete menu ow diagram [4] of
our system can be seen in Figure 5.2. Various types of screens with dierent functionality
which are described further were arranged into a tree hierarchy. The class diagram of these
screens can be found in Figure B.4.

Screen manager  maintains a stack of screens and calls their update and draw methods
and routes the input to the active screen.

Game screen  screen which has its own update and draw logic and serves as a parent
screen for all screens which description follows.

Splash screen  displayed as a rst screen when the application is launched.

Message box  a popup screen which communicates with the user.

Info screen  a screen which enables displaying the data on several pages.

Menu screen  base class for screens that contain a menu of options which can be navi-
gated and selected by the user
CHAPTER 5. SYSTEM ARCHITECTURE 19

1 Main Menu 2 Players 3 Songs 4 Options 5 About

Training 2 Player #1 6 Song #1 8 Input device

Quickplay 3 Player #2 Song #2 Output device

Options 4 - New Player - Song #3 Resolution

About 5 - New Player - Song #4 Full screen

Exit 0 Back 1 Back 6/1 Back 1 Back 1

6 Lessons 7 Lesson #1 8 Loading 9 Playback 10 Results

Continue 10
Lesson Song
Lesson #1 7
Instructions Instructions
Lesson #2

Lesson #3 Continue 3 Tempo 9 Replay 8

Back 2 Back 6 Back 6/1 Back 3 Back 3

Figure 5.2: Diagram of application's menu ow

y z
Background screen  stays behind all the menu screens and draws the background im-
age.

Playback screen  the scene where the MIDI song is replayed.

Loading screen  coordinates transition between menu and playback screens which takes
a longer
z time do to the data load. x

HUD  a screen which displays graphics loaded in 2D textures onto the playback screen.

Whole concept of managing the screen states has been designed according to the sample
created by Microsoft and published on their community web pages of XNA Creators Club
[8].

5.2.2 Data management

According to the dened requirements, we are not going to operate with score writing
therefore we have decided to store the songs replayed in the sessions in a MIDI format for
easy data manipulation and communication with the electronic keyboard. Also, since the
MIDI is older than MusicXML, more supporting software and music les exist in the .mid
le format. The MusicXML is still a possible way and its usage might be reconsidered
during the application's further development when other functionalities may be added.
The data which the system uses as parameters for its object are stored in a le system
in their appropriate form. The rst types of data are the MIDI les themselves which
are simply placed on concrete le path such as .\Music\*.mid where the appropriate
loading function seeks for them. Next information needed for the application run is the
player's details such as the nickname and the table of completed songs so the program
can decide from what point continue the player's training. Those data should not be
editable externally by the user; therefore the byte stream of the internal application object
is suitable.
Last but not least are the lessons data. To meet the easy editable requirement, any
standardized structure should be used. We have decided to create a simple XML document
20 CHAPTER 5. SYSTEM ARCHITECTURE

Figure 5.3: Diagram of objects in XML le

Figure 5.4: Graphics lifecycle in application

which can be parsed afterwards in the program by the object manager into application's
internal objects. The set of entities is shown as a diagram in Figure 5.3. The strict
syntax which can be dened in Document Type Denition (DTD) enables the possibility
of later connection with other applications such as an editor tool. The separate data also
makes easier the additional upgrades of content and allows the possibility of interchanges
of lessons between users.

5.2.3 Graphics management

The graphical content of the application is stored in the appropriate le formats and is of
two types. The 3D models representing the virtual piano keyboard and the other elements
of a playback scene are created in 3D modeling software. The next type is the 2D image
used as the background of the screens and for HUD components and can be drawn in any
bitmap or raster graphics editor. Those objects are loaded in the program the way in
which depends on the graphics library used in an implementation.
The approach of displaying the graphical information has been realized using the update
and draw methods that are usually implemented when handling the graphical data in
applications such as video games where they represent the part, so called game loop. As
the names suggest, in the update method the changes of the graphical parameters which
are visible in the draw method are made. The draw method manages to render the models
in 3D world into the 2D according to certain matrices and displays the nal image on the
screen. The whole lifecycle of the graphical content in the running application describes
Figure 5.4.
CHAPTER 6. IMPLEMENTATION 21

6 Implementation
6.1 Software development tools and libraries

Before we move toward the implementation part itself, the chosen technologies will be pre-
sented together with the particular reasons for their selection from other possible options.
The research has been focused on technologies supporting Rapid Application Development
(RAD), a type of software development methodology. This technique is suitable for our
project since its principle is based on minimal planning in favor of rapid prototyping and
allows an iterative development process.

6.1.1 Programming language

Looking at the designing part, the class-based and component-oriented programming lan-
guage with support for 3D graphics is essential. The data are manipulated on the object
level thus there is no need for any low-level programming language. A high-level language
with integrated garbage collector and ability to assist the programmer with type checking
is suitable. According to the project's assignment, the platform-dependency has not been
dened so there is no need for using a platform independent programming language such
as Java. Since the author's currently used operating system is Microsoft Windows we have
decided to use C# which responsibly fulls all the previously mentioned requirements.
C# is an object-oriented programming language developed by Microsoft to work under
its .NET Framework. The syntax is based on C++ and contains elements similar to those
of Java. A point worth mentioning is the modularity in C#, represented by support of an
independent part of code in a form of dynamic-link library (DLL); Microsoft's implementa-
tion of the shared library which allows the programmer to import unmanaged code written
in other languages such as C or C++. We will take an advantage of this functionality
while wrapping the low-level MIDI library in the code written in C#.

6.1.2 Graphics framework

Handling the graphics in the program requires a set of tools which provide the functionality
in the form of a framework or library. Graphical APIs commonly used are cross-platform
OpenGL and Microsoft's Direct3D, the part of DirectX. Considering our aim of using RAD
tools and developing our application in C# it is essential to use Microsoft XNA Framework
which is in fact a managed version of DirectX that is intended to assist development of
games and also supports porting the code on the Microsoft's video game console Xbox 360.
Unlike the DirectX runtime, XNA Framework is not shipped as a part of Windows and
developers are expected to redistribute the runtime components along with their games or
applications which can be distributed free of charge under Microsoft's current licensing.
There is also an ocial site containing useful samples, making the framework easy to learn
and understand. In addition, a community of XNA developers has grown on the Internet
with its increasing popularity so another portion of samples is available online accelerating
the development by using solved solutions even more.
22 CHAPTER 6. IMPLEMENTATION

Figure 6.1: Scheme of connection between computer and electronic keyboard

6.1.3 Integrated Development Environment

When we decided to use C# together with XNA, the choice of integrated development
environment (IDE) is pretty straightforward since the only IDE ocially supporting XNA
is XNA Game Studio which can be run only under its corresponding version of Microsoft
Visual Studio (MSVS). MSVS exists in many versions where the one supporting C# de-
velopment uses its own compiler, Microsoft Visual C#. Apart from the integrated browser
of ocial Microsoft's online documentation [7], the IDE supports the C#'s documentation
system, which is similar to Java's Javadoc, but based on XML. When present in the code,
even the custom code notes and comments are visible in the tooltip of auto completion
future of MSVS called IntelliSense.

6.1.4 MIDI Application Programming Interface

MIDI devices are of two main types. The rst, MIDI input, is the device also called
controller and is capable of sending MIDI messages unlike the second, MIDI output, which
receives MIDI messages from the input and plays them. MIDI keyboard or a digital piano
connected to the computer through the USB MIDI interface can serve as both MIDI input
and output device. The way of connection describes Figure 6.1. USB MIDI interfaces do
not pass any sound or audio data through them, just the MIDI data represented by MIDI
messages.
Considering playing the electronic keyboard, when a piano key is pressed a signal dening
a Note-on message is transmitted. When the key is released a Note-o message is sent.
These messages consist of a status byte which denes the message type and up to two data
bytes which in the case of Note-on/o message are the number representation of the note's
pitch and its velocity. Both mentioned messages along with the Program change message
belong to the group of the channel voice messages and are processed by the MIDI output
device in some way unlike the Meta Events commands which do not result in any MIDI
message being sent [3].
The handling of the MIDI commands while developing a software product requires the
use of an application programming interface (API). In Windows-based computers there
are two APIs oered by the operating system which allow accessing hardware ports at
the low level. The rst is a module of Windows Multimedia API, the winmm.dll library
and the second is DirectMusic, a component of the MS DirectX API. DirectMusic has
been, together with other older APIs declared by Microsoft as deprecated, cannot be used
CHAPTER 6. IMPLEMENTATION 23

by 64-bit applications and is no longer part of the latest's versions of DirectX SDK [7].
Another option for how to deal with MIDI data is the package javax.sound.midi provided
by Java 2 Standard Edition Platform as a part of Java Sound API. While implementing
our application we have decided to use the winmm.dll library regarding the selection of
other development tools discussed earlier in this chapter.
To encapsulate the function of low level library, wrapping a frequently used DLL function
in a managed class is an eective approach to follow. In our application we needed to
nd a way to realize communication between a computer and MIDI devices using MIDI
messages on the programming level. We are going to use the winmm.dll library, written in
C programming language. Instead of writing our own C# wrapper we will take advantage
of the complete solution to speed up the process of development.
We have found two remarkable .NET MIDI libraries. The rst is NAudio [6], the library
of which development started in 2001, with currently released 1.2 version from June 2008
published under the Microsoft Public License. NAudio is a complete solution for working
with variety of audio le formats, not only MIDI. The second is C# MIDI Toolkit [13]
which has been developed by the author since February 2004 now in its fth version from
April 2007 licensed under the MIT License.
Both toolkits have their source code available to browse together with demo applications
thus we could have made our decision according to the suitability of the projects capabilities
to our application by looking at available objects and methods in detail and did not have
to rely on their documentation. Although NAudio is robust project it is still marked as in
beta version. At the end we have decided to go with C# MIDI Toolkit due to its MIDI
orientation only, its complex structure and standalone sequencer.

6.2 Implementation highlights

Further text in this chapter covers the implementation details of the most important parts
of our application which is called PianoRoll ; the name which has been chosen for the
program.

6.2.1 Application graphics

The XNA framework supports both 2D and 3D graphics in the form of images loaded in the
2D sprites and 3D meshes using the ContentManager class which handles the management
of the graphics content and the content pipeline. Content pipeline is used to convert the
content assets into an XNA specic .xnb format. These objects are rendered in a Draw
method where the sprites are passed to the graphics device using a SpriteBatch object
whereas the meshes of 3D object are rendered one by one using a BasicEffect class which
supports vertex colors and lightning [9].
Models used in our application have been created in Autodesk Softimage Mod Tool
which is a free version of professional 3D computer graphics application recommended for
using with XNA as it natively includes an export function to Microsoft XNA Game Studio.
At the current stage of the system's development the models of black and white key are
low-poly objects tting the parameters of the real keys. The note boxes are simple cubes
with default unit size resized in the program according to the length of the represented
note. The complete scene drawn in the program using cloned key, box and line models is
24 CHAPTER 6. IMPLEMENTATION

Figure 6.2: Orthogonal views of playback scene with camera viewports

Figure 6.3: Relationships between PianoRollLibrary namespaces

together with highlighted camera viewport shown in Figure 6.2.

6.2.2 PianoRoll library overview

Objects and methods from C# MIDI Toolkit used by our program are placed in Sanford.
Multimedia.Midi namespace. The relationship between toolkit's and system's namespaces
is outlined in Figure 6.3 and can be seen in more detail in Figure B.3. The toolkit's objects
and methods used for sending and receiving the MIDI messages between MIDI devices are
handled within the PianoRollLibrary.MIDI namespace classes. The SongData class from
PianoRollLibrary.Song uses the toolkit's objects Sequence and Track to parse the data
from the MIDI le and the instance of MidiInternalClock is used as the main timer for
the playback. The need of the separate timer is because the song does not start playing
until the rst note's box hits the keyboard line.
The PianoRollLibrary.3DModel consists of classes which contains the update and
draw methods of playback scenes' graphics objects. Those methods are called inside the
CHAPTER 6. IMPLEMENTATION 25

SongPlayer's class method PlayTrack() since they are dependent on the parameters of
currently played track. Lines representing the boxes' path are static and drawn together
with the keyboard whereas the note boxes and bar lines are dynamically added or removed
from the lists according the actual position of the timer in the song.

6.2.3 XML object loader

As mentioned in Subsection 5.2.2, we are going to load the lesson data from the XML le
to enable the possibility of editing and sharing the data. Therefore the requirement for
the object manger to change properties can be made without recompiling the program.
An object manager used in our program has been taken from the online sample found
on Ziggyware web pages [15]. Unlike other existing object loaders this one does not use
the built-in XML serialization but denes its own reection API. It is able to instantiate
a class dened in attribute type of node object and lls the elds with the content of
child nodes. For example if we want to add a new song into the lesson's list of songs using
AddSong(Song song) method, we insert the <Song> and its child element into a <AddSong>
tag. The object loader also supports loading resources therefore the images can be loaded
into lessons and displayed further in the program on the screen with lesson instructions.

6.2.4 Input from keyboard

The user's input performed by pressing and releasing the keys on electronic keyboard is
used not only for playing the song but also for navigating the menu screen. For that
purpose there is a set of classes which store the piano keyboard states and manage the
communication between the piano keyboard 3D model and the real electronic keyboard
represented by MIDI input device.
When the song is replayed, the MIDI input device sends the MIDI messages to the system.
These messages contain data about currently pressed and released keys and are received
in the MidiInput class where event handlers are declared which invoke the appropriate
methods in the PianoKeyboard class. These methods contain calls for other actions such
as piano key animation, score evaluation and passing the message to the MIDI output
device and use as parameters the data of the channel message received, i.e. the Data1
property gets the note's pitch and Data2 note's velocity.

6.2.5 External tools for lesson authoring

When the song data are created it is essential to know the parameters of the SMF which is
going to be used like the default tempo or tracks the le holds. For this reason there is a
need of an external tool. We have created a simple Windows Forms application using our
PianoRollLibrary and the C# MIDI Toolkit. The screenshot can be seen in Figure 6.4.
For now the capabilities of this program are limited so while creating the testing data we
also used Music MasterWorks music editing application produced by Aspire Software [14].
Since only the 30-day trial version is free to use, we would like to expand our editor in order
to become a complex tool shipped with our tutorial system with all needed functionalities
including the lessons data editor.
26 CHAPTER 6. IMPLEMENTATION

Figure 6.4: Mockup of external authoring tool

CHAPTER 7. TESTING 27

7 Testing
While testing our application, we will focus on usability tests which examine if the product
meets the requirements from the user's point of view and how easy and intuitive its usage
is. The aim is to discover errors and areas of improvement while observing people using the
product. The tracked aspects are eciency, accuracy, recall and emotional response. The
person who leads the test, a moderator, rst nds out information about the participant
in the form of questionnaire, then he gives him concrete instructions and instead of asking
questions or showing what to do. Moderator watches the reactions and the actual steps
the user makes. An interview about person's feelings and impressions of the system closes
the testing process. Participants are identied in the report by alphabet letters since their
true identity is not relevant to the test and should stay annonymous.

7.1 Test setup

Our tutorial system was tested on the small group of users with dierent piano playing
skills and experience with similar products to cover the target group of users as complex as
possible from the sample. The tests were not run in one round and after each test changes
to signicant problems were corrected. This procedure has been chosen to increase the
eciency of upcoming tests since we assume that if the mistake was left, it would be
harder to analyse other potential problems covered by the detected problem.
We expected the test to reveal the playback screen's defects as we were aware of a lack
of preliminary prototype tests which would take extra time to create. Instead of early
prototyping, we have decided to implement the program to the designed form and wait for
the state of development, where the system currently occurs, which allows us to make the
changes on the graphical design on the run.
To make the tests possible, the test data had to be created. These data contain the
beginning lessons with simple melodies and some advanced songs placed in Quickplay
mode. For the nal version of the application the data should be created in co-operation
with a professional music master in order to guarantee success in the system's purpose
since our aim is to develop a tool which would allow it. The songs on which the users
train his skills should be chosen according to certain criterion consulted with the experts
at pedagogy.
Before and after test questionnaire can be found in Appendix D. An explanation of the
questions and the summary of the results gained while testing our 5 test subjects in the
average age of 22 follows. In Figures C.2 and C.3, there are the playback scene designs
before and after testing to show the dierences made to the program.

7.2 Testing process and results

Since our application is targeted at piano players of dierent levels either to be used as an
educational or an entertainment tool, we asked the participants about their piano playing
experience. The reactions to the piano roll style of representing the notes from the players
used to the typical notation were observed. Also we considered it useful to know if they
were familiar with the concept of playback from similar products and score used in video
28 CHAPTER 7. TESTING

Figure 7.1: Illustration of test setup

games. When the participant answered these questions he was navigated to the song on a
level appropriate to his skill and asked to follow the onscreen instructions.
The questions from the after-test questionnaire were pointed at user's impressions and
opinions about our application. We were interested if the score motivated the player to
replay the song which was conrmed by users which played the video games. According
to answers to question about adequate lesson level we adjusted the test data before we
approached the next test. The participants were asked about their desired use of our
system. Most of them would use the application for their entertainment, only a few because
of its educational element. We suppose the rst group to be the people who would learn
themselves without even noticing it. Some of the participants expressed their intention
to play their favorite songs. Participant A was sure that he would denitely prefer this
way of learning to the classic way which he would have never started by himself but our
system seems to be more of an incentive. None of the participants would hesitate with
buying electronic keyboards capable of connection to their PC which they all own and
would invest from 2 000 to 5 000 CZK.
Most of the problems occurred with the playback screen. Participant A had diculties
with recognizing if the position of his hand corresponds to the range of virtual keyboard
and was not sure when the key should be pressed. To correct this problem each octave was
distinguished with a specic color and the key which was about to be played was highlighted
by the reector from the note box which darkens when closer to the corresponding piano
key. Participants B and C did not recognize what key should be played therefore the
adjustments in the colors of dierent key states was made and the lines were prolonged
even in front of the keyboard. Participant D only had a problem recognizing when the two
same tones follows immediately but during the second replay he played it right. Because
this user was the type of player which can play the songs by ear he suggested a replay
function of the song which is going to be played. The last one, Participant E liked the
keyboard input when navigating the menu but needed some time to get familiarized with
the concept as he was without the previous experience with similar products. Although he
stated that the incoming boxes are better than if the keys would just change colour.
CHAPTER 8. CONCLUSION 29

8 Conclusion
I have implemented a tutorial system which is targeted at users who are going to learn
to play the piano using an electronic keyboard and a computer. The application supports
playback of songs stored in MIDI le format and oers the user the possibility of partici-
pating either in Training or Quickplay mode. The player's knowledge of musical notation
is not required since the program represents music in a visual form.
I have chosen the subject of this work because of my personal knowledge of the piano
playing educational system. I found there is a lack of encouragement which prevented me
from returning to playing after a few years of pause. Therefore my goal was to create a
piano playing tutorial system which would be educative and entertaining at the same time.
When designing the application itself I was mainly inspired by my experience with Guitar
Hero whose concept seemed to me to be applicable for educational purposes if applied to
the real musical instrument.
First of all I had to look closely at the current situation regarding available tools and have
learnt from their drawbacks and took advantage from their successful points. I studied in
detail the MIDI technology to learn how to handle communication between an instrument
and a computer program. I also searched for the most suitable option of handling the
MIDI commands in my program.
Through testing I have conrmed that players who are not familiar with standard musical
notation can use the program prociently and with a clear fun element. Though the
system's data needs to be precisely adjusted to ensure their adequate diculty. I am
aware of the necessity of cooperation with a professional teacher for creating the lesson
data to ensure the eectiveness of the educational element.

8.1 Future work

In order to provide users of our educational system with additional services some sugges-
tions for the future work follows. First of all the existing editor tool could be extended
with the capability of preparing the MIDI songs and levels for the application. In the
playback scene the advanced graphics using particle eects and shaders is likely to make
the playback more intuitive and attractive to the players. Especially the visualization at
the moment the key should be pressed needs to be improved. To satisfy the largest range
of target group of users I would suggest letting the user choose between dierent genres
of songs played to ensure their enjoyment of the application as a whole. The score adjust-
ments in dierent modes need to be revised and tested. Finally, an animated hand showing
how to play a certain song correctly can be used as a guiding tool for the user.
30 CHAPTER 8. CONCLUSION
CHAPTER 9. BIBLIOGRAPHY 31

9 Bibliography
[1] MIDI Manufacturers Associatio. MIDI.
http://www.midi.org/.
[2] Jan Dostal. Dt u klavru. Supraphon, Praha, R, 1th edition, 1977.
[3] Daniel Forr. MIDI komunikace v hudb. Grada, Praha, R, 1th edition, 1993.
[4] Brent Fox. Game Interface Design. Course Technology PTR, Boston, USA, 1th
edition, 2004.

[5] DDR Game. DDR Distributor.

http://www.ddrgame.com/.
[6] Mark Heath. NAudio.
http://naudio.codeplex.com/.
[7] Micfosoft. Microsoft Developer Network.
http://msdn.microsoft.com/en-us/library/.
[8] Microsoft. Game state management sample.
http://creators.xna.com/en-US/samples/gamestatemanagement.
[9] Benjamin Nitschke. Professional XNA Programming. Wrox, Boston, USA, 1th edition,
2008.

[10] Klavar Music Foundation of Great Britain. Klavarskribo.

http://www.klavarmusic.org/Explanation.pdf.
[11] Nicholas Piegdon. Synthesia game.
http://www.synthesiagame.com/.
[12] Recordare. MusicXML.
http://www.recordare.com/xml.html.
[13] Leslie Sanford. C# MIDI Toolkit.
http://www.codeproject.com/KB/audio-video/MIDIToolkit.aspx.
[14] Aspire Software. Music MasterWorks.
http://www.musicmasterworks.com/.
[15] Ziggyware team. Advanced Object-Loading through XML and Reection.
http://www.ziggyware.com/readarticle.php?article_id=160.
[16] Dalian Piano roll.
http://website.lineone.net/~agr/24.jpg.
[17] Yamaha. Yamaha lighted keyboards.
http://www.yamaha.com/.
32 CHAPTER 9. BIBLIOGRAPHY
APPENDIX A. LIST OF ACRONYMS 33

A List of acronyms
2D Two-Dimensional

3D Three-Dimensional

API Application Programming Interface

DDR Dance Dance Revolution

DTD Document Type Denition

GM General MIDI

GUI Graphical User Interface

HUD Heads-Up Display

MIDI Musical Instrument Digital Interface

MMA MIDI Manufacturers Association

RAD Rapid Application Development

SMF Standard MIDI File

UI User Interface

XML Extensible Markup Language

YMS Yamaha Education Suite

34 APPENDIX A. LIST OF ACRONYMS
APPENDIX B. UML DIAGRAMS 35

B UML diagrams
Use cases

Figure B.1: User's use cases

36 APPENDIX B. UML DIAGRAMS

Class diagrams

Figure B.2: Design of data model classes

APPENDIX B. UML DIAGRAMS 37

Figure B.3: Interaction and relationships between PianoRollLibrary members

38 APPENDIX B. UML DIAGRAMS

Figure B.4: Screen management classes

APPENDIX C. APPLICATION SCREENSHOTS 39

C Application screenshots
Menu screen

Figure C.1: Main menu screen with static background image

40 APPENDIX C. APPLICATION SCREENSHOTS

Playback screen

Figure C.2: Playback scene before tests

Figure C.3: Playback scene adjusted according to test results

APPENDIX D. USABILITY TESTING QUESTIONNAIRE 41

D Usability testing questionnaire

Questions asked before test
1. Age:

2. Piano playing experience (years active/years inactive):

3. Similar products experience:

4. Video games experience:

Questions asked after test

1. First impression about the tutorial system:

2. Adequate level of lessons:

3. Motivation to replay a song via score:

4. Negative feelings invoked:

5. Appealing things:

6. Clear principle of playback screen:

7. Purpose of using this application:

8. Electronic keyboard purchase, limit:

42 APPENDIX D. USABILITY TESTING QUESTIONNAIRE
APPENDIX E. USER MANUAL 43

E User manual
System requirements
Supported operating systems: MS Windows XP SP3, MS Windows Vista SP1.

A graphics card that supports DirectX 9.0 and Shader Model 1.1.

Installed Microsoft XNA Framework Redistributable 3.0.

Installed Microsoft .NET Framework 2.0 or higher.

Electronic keyboard connected to the computer via MIDI.

Installation instructions
Application installation is not required, user can run the le directly by opening the le
PianoRoll.exe.

Navigating the menu

Command PC keyboard / Electronic keyboard

Select Enter / two white keys next to each other

Cancel Esc / two black keys next to each other

Down Down arrow / any white key

Up Up arrow / any black key

Left Left arrow / two white keys separated by one white

Right Right arrow / two black keys separated by one black

44 APPENDIX E. USER MANUAL
APPENDIX F. CD CONTENT 45

F CD Content
CD
|-- PianoRoll - Executable file
|-- src - Source files of the application
|-- text
| |--latex - Source files of this text
| \--thesis.pdf - PDF version of this text
\-- readme.txt - User manual
46 APPENDIX F. CD CONTENT