Providing Mark-up and Feedba
k to Students with Online Marking
David V. Mason
dmason@sarg.ryerson.ca
S
hool of Computer S
ien
e
Ryerson Polyte
hni
University
350 Vi
toria Street
Toronto, Ontario
Canada
Abstra
t the first author started teaching all 150 students in the CS1
Online marking of assignments can lead to improved mark-
ing consistency and integrate well with on-line mark report-
ing. It can also be easier for the marker. Unfortunately,
most such systems do not include good feedback mecha-
nisms for the students. This paper describes an environment
that provides online marking with convenient, structured and
detailed feedback.
1 Introdu
tion
There is an increasing trend to automate as many aspects of
course operation as possible. This promises to be more effi-
cient for markers and more convenient for students because
they will be able to access information at their convenience.
There are many aspects of course operation that are amen-
able to some degree of computerization. Systems such as
WebCT[1, 2] provide WWW access to course content, ad-
minister quizzes, and provide some tools to manage grades.
Other systems such as [5, 4] handle on-line grading of pro-
gramming examinations. BOSS[3] is designed to partially
automate marking of student assignments and labs. As well
as reducing marking time, this provides more consistent mark-
ing, particularly over a range of markers. The main problem
with the BOSS system is that feedback is limited to grades
and out-of-context notes attached to the returned evaluations.
The system described in this paper uses the WWW to
provide in-context annotation of work returned to students.
1.1 Ba
kground
The system described here has gone through several incar-
nations over the past couple of years. It was created when
Denise M. Woit
dwoit@scs.ryerson.ca
M5B 2K3
course with a dozen teaching assistants. The original system
was programmed as a CGI script written primarily in Bourne
shell script. It was functional, but extremely slow. Over the
Christmas break it was re-programmed in Scheme as a CGI,
resulting in a system that was usable but still a little slow
because of the CGI start-up time. During summer’97 it was
re-implemented as an HTTP server in Scheme. This ver-
sion had perfectly acceptable performance, but crashed oc-
casionally because of problems with the underlying Scheme
implementation (scm) and because of a monolithic imple-
mentation. It is currently undergoing a (hopefully last!) re-
implementation, being modularized and implemented on a
more robust version of Scheme.
2 Operation
Although the program includes many parts that perform other
functions such as allowing students to upload files and to ac-
cess their marks, only the part which is relevant to marking
assignments is described in this paper. We will describe the
current implementation and leave most comments about im-
provements to the section, “Lessons Learned”.
The files that make up an assignment are submitted to the
system through either an interactive “submit” command
on our Unix systems or via the WWW. The students do not
have to modify their files in any way. All of the annota-
tion and recognition of parts of the assignment that are de-
scribed below are automatically determined by the marking
environment itself, based on the file extension (“ .scm”,
“.c”, etc.).
The examples here are from marking a Scheme program,
but the annotation program also has rules for marking C and
simple prose. While we have not included examples for the
other assignment types, they are very similar and it is fairly
simple to extend the annotator.
Figure 1 shows the first page presented to the marker for
a particular assignment and a given student. The annota-
tion program will generate a hyperlink for each significant
feature of the assignment to provide marking and annotation
for that section. In this case, each comment block is indepen-
dently markable and the top-level define is also markable.
In addition to the markable sections, there is also a com-
ment field and assignable mark for the total assignment. The

Figure 1: Initial marking page
name of the file is shown just below that with the date of its
last modification, and in this case there are also the results of
an automatic execution which can be reviewed by the marker
(in some cases, the execution can be set up to calculate part
of the mark automatically).
Figure 2: Marking of Headers
From the previous image, if the marker clicked on the
header comment, they would be presented with figure 2. On
this page, the comment has been isolated, and the type of
the section has been identified as “header documentation”
(we treat the initial comment on assignments as identify-
ing information and mark for different things). There are
2 pull-down menus here: one to assign a mark (-3: : :+3),
and one to assign a mark and a comment. In this snap-
shot, we have selected one of the predefined mark/comment
choices, “-1 No course number”, and will now click
on “Submit Query”. On this page, the other predefined
comments include “-2 Missing description” and
“-10 No header comment block”, among others. As
mentioned in the next section this is probably more restric-
tive than it should be.
Figure 3: Marking of Code
From figure 1, if the marker had instead clicked on the
define, they would be presented with the page shown in
figure 3. On this page, the code block has been isolated, and
the type of the section has been identified as “Code”. The
same pull-down menus are provided, but a different set of
pre-defined mark/comment choices are available (including
“-2 Bad indentation”, “-3 Poor algorithm”,
“-5 Wrong interface”, and “-2 Useless code”).
In this case, the marker has typed in a comment and decided
to give a bonus mark to the student.
After the marking for an assignment has been completed
(the “Submit Mark” link on the first page has been clicked)
the page looks like the image in figure 4. This shows the
mark, the entered comment, who marked it, and when it was
marked. There are many other parts of this system, and all
of this information is provided to the student on a different
page. From here, the assignment can be remarked, or it can
be examined as the student would see it. The current system
just keeps the latest marking of an assignment, but it should
probably be extended to keep a log of all markings of an
assignment.
Figure 5 shows the student view. Here the underlining
of the hyperlinks makes it is obvious that annotations have
been provided by the marker for the header comments and
the first code block. The student can click on either of these
 to see the particular annotation.

Figure 6: Annotation seen by student

Figure 4: Remarking page
Assuming the student has clicked on the “define” block,
they would be presented with figure 6. This shows the type
of the block, the mark and the comment from the marker, as
well as the source text of the block and some miscellaneous
identifying information.

3 Lessons Learned

To evaluate the system, there are 2 relevant aspects: how

well it works for students, and how well it works for markers.

3.1 Students

No formal poll has been done, but the student perception of

the system is positive. The active links are very visible to
the student so they can see where there were problems in
the program. By following the links, they can easily see the
comments applicable to the particular section of the assign-
ment.
The other improvement from the perspective of the stu-
dent is that because all of the markers use the same set of
marking rules consistency is improved. Unlike in [3], the
markers knew the students they were marking, but we never
received a complaint that markers were picking on, or being
easy on, any students.

3.2 Markers

The markers comments were also mostly positive. They

Figure 5: Student view found the system (when it was working well) to be easy to
use and fairly efficient. The complaints from the markers
were of two types: speed and customization.
Speed When a marker wants to make an annotation about creation of sophisticated web-based learning environ-
part of the assignment, they are only willing to wait a second ments. In Proceedings of NAUWeb ’97 - Current Prac-
or two before being able to enter the annotation. Any longer tices in Web-Based Course Development (Flagstaff, Ari-
delay led to the markers became frustrated. The CGI ver- zona, June 1997).
sions of the program took about 5 seconds, which became a
major irritation. [3] J OY, M., AND L UCK , M. Effective electronic marking
for on-line assessment. In Proceedings ITiCSE’98 (3rd
Annual Conference on Integrating Technology into Com-
Customization and Convenien e The CGI version of puter Science Education) (Dublin, Ireland, Aug. 1998),
the program enforced security by only accepting connec- ACM, pp. 134–138.
tions from particular machines. This meant that the mark-
ers could only mark from an on-campus lab, which meant [4] M ASON , D., AND W OIT, D. Integrating technol-
that the markers sometimes had difficulty finding the time ogy into computer science examinations. In Proc. of
to mark expeditiously. In addition to speed, this was one 1998 ACM Conference on Computer Science Education
of the main reasons to switch to the server implementation. (SIGCSE’98) (Atlanta, Georgia, Feb. 1998).
The current system can be accessed by students and mark-
ers from anywhere on the Internet with security enforced by [5] W OIT, D., AND M ASON , D. Lessons from on-line pro-
passwords. gramming examinations. In Proc. of 3rd Annual Confer-
Markers also found that, for a particular assignment, they ence on Integrating Technology into Computer Science
were often repeating the same comments for several stu- Education (ITiCSE’98) (Dublin, Ireland, Aug. 1998).
dents. If the instructor had thought of that annotation, then
it would be on a pull-down menu, but otherwise the mark-
ers would have to repeatedly type in the same comment.
This was inconvenient and also diminished the consistency
of marking. The new version will pool the annotations so
that an annotation used by one marker for a particular as-
signment will be automatically added to the pull-down list
provided to all markers. The list will also be sorted by usage
frequency so that the most common comments will appear
at the top of the pull-down.

4 Con lusions and Future Work

This paper has presented one of the possible design points

in an on-line marking system to provide detailed feedback
to students. One of the most significant lessons learned was
that finding a good interface for the markers was a fairly
difficult, iterative process. It was only after each problem
was resolved that the next set of problems was recognized,
despite all of the work being done by experienced teachers
and computer scientists.
As mentioned earlier, we are currently working on the
fourth version of the system. As we feel that the marking en-
vironment is now fairly stable we will be adding other course
management features to this system, as well as convenience
features for instructors (the current system requires a lot of
knowledge and editing of control files to set up properly).
Once this is complete, we will be making this system avail-
able for use by others.

Referen es

[1] G OLDBERG , M. W. Calos: An experiment with

computer-aided learning for operating systems. In Pro-
ceedings of the ACM’s 27th SIGCSE Technical Sympo-
sium on Computer Science Education (1996).
[2] G OLDBERG , M. W., AND S ALARI , S. An update
on webct (world-wide-web course tools) a tool for the