Education Tech Research Dev (2013) 61:197215

DOI 10.1007/s11423-013-9286-5
DEVELOPMENT ARTICLE

The Assessment Agent System: design, development,

and evaluation
Jianhua Liu

Published online: 13 February 2013

Association for Educational Communications and Technology 2013

Abstract This article reports the design, development, and evaluation of an online
software application for assessing students understanding of curricular content based on
concept maps. This computer-based assessment program, called the Assessment Agent
System, was designed by following an agent-oriented software design method. The
Assessment Agent System is composed of five types of software agents: instructor agent,
student agent, management agent, assessment agent, and reporting agent. Software agents
in the system, through communication and cooperation, collectively provide the functionalities of user-system interaction, user management, task authoring and management,
assessment delivery, task presentation, response collection, automatic assessment, and
reporting. One-to-one evaluations and group evaluations were conducted to reveal students perceptions of the Assessment Agent System. Measures of visual clarity, system
functionality, consistency, as well as error prevention and correction indicate that the
Assessment Agent System is a useful tool for large-scale assessment based on concept
maps. Through the process of design, development, and evaluation of the Assessment
Agent System, this study demonstrates the agent-oriented approach for producing educational software applications. Furthermore, this research explored the concept map
assessment method for the Assessment Agent System. When node terms and linking
phrases are provided, the assessment of student concept maps can be conducted automatically by comparing student concept maps with the criterion concept map, proposition
by proposition. However, the validity of the proposition-comparing method depends on the
accuracy and thoroughness of the criterion propositions. Therefore, assessment criteria
need to be continually refined and improved through the examination of student-created
propositions.
J. Liu (&)
Department of Learning Sciences and Technologies, School of Education, Virginia Polytechnic
Institute and State University, Blacksburg, VA 24061, USA
e-mail: jhliu@vt.edu; jxl88@psu.edu
Present Address:
J. Liu
Center for Workplace Learning and Performance, Office of Human Resources, The Pennsylvania State
University, University Park, PA 16802, USA

123

198

J. Liu

Keywords Computer-based assessment
Automated assessment
Concept map

Assessment agent
Software agent

Introduction
Educational researchers have made efforts to explore effective forms of large-scale
assessment due to dissatisfaction with traditional assessment methods (e.g., Pirnay-Dummer
et al. 2010). Concept maps are regarded as an alternative assessment tool (Ruiz-Primo and
Shavelson 1996). A concept map is a diagram in which nodes represent concepts, and
labeled directional links that connect nodes represent relationships between concepts.
Concept mapping is a straightforward way for students to demonstrate, with graphical
representation, their understanding of concepts as well as relationships among concepts
(Novak and Gowin 1984). Thus, concept maps offer ways to assess students integrated
conceptual understanding. Research suggests that concept maps can be a valuable tool for
assessing students understanding of curricular content (Herl et al. 1999; Yin et al. 2005).
There are several advantages in the use of concept maps for evaluating student learning
on the level of understanding (Fisher 2000). First, assessment tasks are generative, not
simply responsive. The construction of concept maps requires students to precisely
understand content and to explicitly express their understanding. Second, concept mapping
tasks not only assess the identification of isolated concepts and relations but also the
organization of concepts and relations. Third, the process of creating concept maps
facilitates students to shape their systematic and holistic views on learning content.
However, it is not efficient to create, evaluate, and manage paper-based concept maps in
large-scale assessment. In particular, manually assessing concept maps is time-consuming,
subjective, inconsistent, and error-prone (Luckie et al. 2011; Taricani and Clariana 2006).
These factors negatively impact the wide employment of concept mapping in assessment.
Therefore, it is worthwhile to develop a computer-based assessment system to assure the
reliability and validity of assessment, to reduce workload, and to increase the efficiency
when concept maps are used for large-scale assessment.
Computer systems for large-scale assessment need to provide service to a large number
of concurrent users with satisfactory system performance. Distributed or geographically
dispersed users, including assessment developers and assessment takers, may create and
submit various tasks and responses. Moreover, the needs of real-time decision making for
scoring and providing feedback require a dynamic system. Further, systems usually require
upgrades with reasonable cost and minimal system impact. The requirements of scalability,
accessibility, dynamism, as well as time- and cost-efficiency of large-scale assessment
programs call for employing appropriate technology to build high quality computer-based
assessment systems.
The use of software agent technology in learning assessment has received attention from
researchers (e.g., Grundspenkis 2011; Tecuci and Keeling 1999). A software agent is a
software entity that functions autonomously to meet its design objectives in an environment. A number of agents constitute a multi-agent system through interaction with each
other in an environment to pursue goals or perform tasks (dInverno and Luck 2004;
Wooldridge 2009). Each agent in a multi-agent system is an independent problem-solver,
and these agents collaborate to constitute a whole system.
Four main advantages in building computer-based systems for large-scale assessment
with software agent technology can be derived from research on multi-agent systems (e.g.,

123

The Assessment Agent System

199

Sterling and Taveter 2009; Sycara 1998; Wooldridge 2009). First, software agents
autonomously carry out tasks to reduce the workload of users. Second, the distribution of
system functionalities to various software agents increases modularity and flexibility,
thereby making the system easier to maintain and upgrade. Third, the use of a multi-agent
structure can enhance computational efficiency, reliability, extensibility, robustness,
responsiveness, and reuse, thus strengthening system performance. Fourth, the concept of
software agents provides a suitable modeling technique for the design of large-scale
assessment systems.
Software agent technology enriches the means and increases the possibility of developing a large-scale assessment mode based on concept mapping. However, there have been
few attempts to systematically explore the necessary functionalities and system architecture in building software agent-oriented systems for large-scale assessment.
The purpose of this study was to explore the feasibility of employing software agent
technology to support large-scale assessment based on concept maps. The research
included the design, development, and evaluation of a computer-based system for assessing
students understanding of curricular content based on concept maps. More specifically,
this study aimed to analyze and determine system functionalities, design the system
architecture, and explore the method of assessing concept maps for the system. It was
designed to answer the following research questions:
1. What functionalities are necessary for large-scale assessment based on concept maps
in a software agent-oriented system?
2. What system architecture is appropriate for large-scale assessment based on concept
maps in a software agent-oriented system?
3. What design provides a reliable and valid assessment method for large-scale
assessment based on concept maps in a software agent-oriented system?

Research methodology
An approach that makes research more directly impact educational practice is designing,
developing, and evaluating curricula, programs, software, tools, models, and school
organizations (Edelson 2002; Richey and Klein 2007). Sharing similar goals and approaches but with various focuses, this type of research has been labeled as design research
(van den Akker et al. 2006) or developmental research (Richey et al. 2004). More recently,
Richey and Klein (2007) coined the term design and development research to refer to
this type of study, and defined it as:
The systematic study of design, development and evaluation processes with the aim of
establishing an empirical basis for the creation of instructional and non-instructional
products and tools and new or enhanced models that govern their development. (p. 1).
Richey and Klein (2007) classified design and development research projects into two
categories: (a) product and tool research, and (b) model research. Product and tool research
pertains to studies of designing, developing, and evaluating instructional or non-instructional products, programs, or tools. Model research involves constructing, validating, and
using design and development models or processes. The current study adopted the product
and tool research approach. The major steps in the research process included literature
review, design, development, one-to-one evaluation, and group evaluation (see Table 1).

123

200

J. Liu

Table 1 The research process

Phase

Research activity

Literature
review

Reviewed and analyzed studies of concept maps as an assessment tool to decide the
assessment task format and scoring method for the assessment system
Reviewed and analyzed existing computer-based assessment systems based on concept
maps to assist decision making on the system design and development
Reviewed research of software agents and identified features of software agents that should
be addressed in the system design
Investigated agent-oriented system design techniques and selected one appropriate method
for designing the assessment system

Design

Analyzed requirements for computer-based assessment systems

Determined system functionalities
Designed system architecture
Designed each component of the system

Development

Developed a database
Developed user interface
Developed system functionalities

Evaluation

Conducted one-to-one evaluations

Identified problems and improved the system
Conducted group evaluations
Analyzed data and summarized results

Literature review
Concept maps as an assessment tool
The basic components in a concept map are nodes and links (Novak and Gowin 1984). A
node is composed of a node symbol and a node term. A node symbol is a graphic shape,
such as a circle, oval, or rectangle. A node term labels a node. The combination of a node
symbol and a node term represents a concept. A link includes a linking line and a linking
phrase. A linking line connects two nodes to indicate that there is a relationship between
the two concepts. The arrowhead of the linking line indicates the direction of the relationship between the two concepts. A linking phrase specifies the relationship. A nodelink-node combination is called a proposition, which is the basic unit of a meaningful
statement in a concept map (Ruiz-Primo and Shavelson 1996).
Ruiz-Primo and Shavelson (1996) suggested that a concept map assessment is composed of a task, a response format, and a scoring system. There are many types of concept
map assessment techniques that result from the variation of tasks, response formats, and
scoring methods in concept map assessment (Ruiz-Primo 2004).
Assessment tasks
Concept map assessment tasks may assume a variety of formats. Some examples of concept
mapping task format are fill-in-nodes, fill-in-links, construct-a-map with provided node terms
and linking phrases, construct-a-map with provided node terms and suggested structure,
construct-a-map with provided node terms, and construct-a-map freely (Ruiz-Primo 2004).
However, different concept mapping formats may provide different information about
students knowledge structures. Ruiz-Primo et al. (2001) examined the effects of fill-in-the-map

123

The Assessment Agent System

201

and construct-a-map-from-scratch formats on assessing students connected understanding.

In the fill-in-the-map condition, students were asked to fill blank nodes or blank linking phrases
in skeleton maps. Lists of concept terms and linking phrases were provided. In the constructa-map-from-scratch condition, students were asked to construct-a map with provided concept
terms. Linking phrases and map structure were not provided. Results indicate that the constructa-map-from-scratch format is more effective than the fill-in-the-map format for revealing differences among students knowledge structures.
In another study, Yin et al. (2005) investigated construct-a-map with created linking
phrases (C) and construct-a-map with selected linking phrases (S) formats by comparing
map products and map construction processes. In the construct-a-map with created linking
phrases (C) condition, students constructed maps with provided concept terms, but they were
asked to generate linking phrases. In the construct-a-map with selected linking phrases
(S) condition, students constructed maps with provided concept terms and linking phrases.
Student generated maps were analyzed and compared on the variables of total accuracy score,
individual proposition scores, proposition choice, map structure complexity, proposition
generation rate, and proposition generation procedure.
Yin et al. (2005) concluded that the C and S mapping formats have different strengths
for assessing students knowledge structures. The C format more effectively indicates
students partial knowledge and reveals their misunderstandings. However, the S format is
more efficient for scoring concept maps. A subsequent study indicated that the S format is
more reliable under the same conditions (Yin and Shavelson 2008). Therefore, the C
format would be an effective tool for formative assessment in classroom settings, and the S
format is more appropriate for large-scale summative assessment.
Scoring methods
Ruiz-Primo and Shavelson (1996) classified concept map scoring strategies into three
categories: (a) scoring a student map against a list of rubrics or criteria, (b) comparing a
student map with a criterion map, and (c) using a combination of both methods. Chung
et al. (2006) further categorized concept map scoring methods across two dimensions:
referent-free versus referent-based methods, and methods based on map structure versus
those based on semantic content.
In referent-free methods, student maps are evaluated by using a list of rubrics or criteria. In
the scoring method developed by Novak and Gowin (1984), a concept map is scored by
counting the number of valid propositions, levels of the hierarchy, cross links, and examples.
The scoring rules in this method address both semantic and structural aspects of concept maps.
Another referent-free scoring approach is assessing the accuracy of each proposition in a
concept map against a proposition inventory (Ruiz-Primo et al. 2001). This method is suitable
for the assessment task format where students construct their concept maps with provided node
terms and linking phrases. The proposition inventory includes all possible concept-relationship-concept combinations of the concept terms and linking phrases given to students.
In referent-based scoring methods, expert concept maps are used as criteria for assessing
student maps. The quality of student maps can be assessed by comparing them with the
expert map, including both semantic and structural comparison (Herl et al. 1996). Computers can assess student concept maps by matching the referent map, proposition by
proposition (Chung et al. 2006); by comparing distance data and link data of the expert
map, without assessing linking phrases (Taricani and Clariana 2006); or by measuring the
semantic and structural similarity between student maps and the criterion map (Cline et al.
2010).

123

202

J. Liu

McClure et al. (1999) examined the reliability and validity of six concept map scoring
methods. The six scoring methods include (a) holistic, (b) holistic with master map,
(c) relational, (d) relational with master map, (e) structural, and (f) structural with master map.
Results indicate that the relational with master map method yielded the most reliable scores.
All methods, except the structural with master map, are valid. Research conducted by Herl
et al. (1999) confirms that the relational with master map scoring method is reliable and valid.
Agent-oriented software design
The concept of software agents not only provides a new way to develop computing
technologies, but also offers a novel perspective on software design (Jennings 2001; Luck
et al. 2004). In an agent-oriented system, software agents autonomously interact with each
other and with the environment to pursue goals or perform tasks. With agents as a means of
abstraction, the agent-oriented approach provides a natural and intuitive perspective on
modeling and designing complex, distributed software systems.
The agent-oriented approach deals with complex software design in three ways:
decomposition, abstraction, and organization (Jennings 2001). A complex system is
decomposed into a set of problem-solving components, which are in turn designed as
various types of agents. A set of agent-oriented concepts and notions is used to abstract and
model computing systems. Some basic elements in the set include agents, goals, roles,
environments, interactions, and organizations of agents (Sterling and Taveter 2009).
Agents are organized in a multi-agent structure, and an application is modeled as a society
of interacting agents. An individual agent or a group of organizational agents can be
designed and incrementally added into a multi-agent system. A variety of agent-oriented
analysis and design methodologies have been created for developing agent-based systems,
and studies have been conducted to compare and evaluate these methods (Bergenti et al.
2004; Henderson-Sellers and Giorgini 2005). Each has its strengths and weaknesses. The
most appropriate methodology is dependent on the target application. Prometheus
(Padgham and Winikoff 2004) is a well-documented software design method aimed at
developing intelligent multi-agent systems. It defines a practical process to design agentoriented software systems and offers a set of detailed guidelines in each step of the design.

The design of the Assessment Agent System

An assessment agent is a type of software agent that may autonomously produce items for
assessment or assess responses to items. The assessment agent in this study assesses
concept maps created and submitted by students electronically. The Assessment Agent
System is composed of an assessment agent and other types of software agents. The
Prometheus agent-oriented software design method (Padgham and Winikoff 2004) was
adopted to guide the system design. The design process included three phases: system
specification, architecture design, and detailed design.
Phase 1: system specification
Step 1: analyzing system requirements
There are three types of activities in the process of concept map assessment in this study.
First, experts or instructors create assessment tasks and criterion maps, which are stored in

123

The Assessment Agent System

203

a database. Second, students create concept maps as responses to the assessment tasks and
submit their concept maps for assessment. To strike a balance between effectively
revealing students understanding of curricular content and efficiently assessing student
concept maps with a computer-based assessment system, the assessment task format where
students construct concept maps with a given set of node terms and linking phrases was
selected. The node terms and linking phrases provided to students include both appropriate
and inappropriate terms and phrases; however, inappropriate terms and phrases serve as
distracters. Third, the system assesses student-created concept maps by comparing them
with the criterion map, proposition by proposition. The use of criterion maps is to reduce
the arbitrariness in the assessment of student concept maps, and to increase the reliability
and validity of assessment results.
The first step in the system specification phase involved analyzing the system
requirements. This process began with an analysis of the generic requirements of computer-based assessment systems. The generic analysis was followed by an analysis of the
specific Assessment Agent System requirements associated with this study. The requirements of the Assessment Agent System were specified based on the expectations for
computer-based assessment systems (Bull and McKenna 2004; Burkhardt and Pead 2003)
and research on concept maps as an assessment tool (Chung et al. 2006; Yin et al. 2005).
The Assessment Agent System requirements reflect the most important expectations for
computer-based assessment, including task creation and management, task delivery, student working and responding to assessment tasks, automatic assessment, assessment
reporting, and assessment security.
Step 2: specifying system functionalities
To meet the specified system requirements identified in Step 1, the Assessment Agent
System should possess corresponding functionalities and goals. The functionalities that
should be addressed by the system in this study include user-system interaction, user
management, task authoring and management, assessment delivery, task presentation,
response collection, automatic assessment, and reporting. The corresponding goals of the
functionalities are presented in Table 2.
Step 3: developing scenarios
The scenario development technique is used to identify and refine functionalities, goals,
percepts (input), actions (output), and the data that the system uses and produces. In this
study, the main scenarios in the Assessment Agent System include: (a) creating an assessment task, (b) assigning a task, (c) presenting a task, (d) creating a student map, (e) submitting a student map, (f) assessing a student map, and (g) requesting an assessment report.
Step 4: specifying interface and data
The interface between the Assessment Agent System and its users includes percepts and
actions, which are input and output of the system. Percepts of the system include:
(a) entering text, (b) requesting task submission, (c) editing a task, (d) requesting task
assignment, (e) requesting task presentation, (f) selecting an item, (g) selecting and moving
a node, (h) drawing a linking line, (i) dragging and dropping a linking phrase, (j) requesting
student map submission, (k) requesting an assessment report, and (l) exiting. Actions of the

123

204

J. Liu

Table 2 Functionalities and goals of the Assessment Agent System

System functionality
User-system interaction

System goals
Provide text areas for receiving input from users
Provide menus and/or buttons for receiving commands
Support operations of dragging and dropping to build concept maps
Display information

User management

Accept, maintain, and update user profiles

Identify users

Task authoring and management

Accept and store data in a database

Provide course and assessment task lists for selection
Provide student name lists for selection

Assessment delivery

Deliver assessment tasks to users via the Internet

Task presentation

Provide course and assessment task lists for selection

Present assessment task description in text
Provide nodes and linking phrases

Response collection

Recognize student concept maps

Accept and store propositions of student concept maps in a database
Notify student map reception

Automatic assessment

Compare student propositions with the criterion propositions

Send results and feedback to a database

Reporting

Query assessment results in a database

Generate, deliver, and present assessment reports

system include: (a) displaying text, (b) confirming task submission, (c) confirming task
assignment, (d) displaying a task, (e) displaying nodes, (f) displaying linking lines,
(g) displaying linking phrases, (h) confirming student map submission, and (i) displaying
an assessment report.
Three types of data are involved in the system and stored in a database, including user
data, assessment task data, and student concept map data. User data contains user profiles,
tasks created or taken, assessment results and feedback. Assessment task data contains
course titles, task titles, task descriptions, node terms, linking phrases, criterion propositions, and task creator names. Student concept map data contains student propositions,
student names and identification numbers.
Phase 2: system architecture design
The Assessment Agent System architecture design defined software agents, as well as their
interactions with each other and with the environment, to provide the specified system
functionalities. The architecture design included three steps: (a) specifying agent types,
(b) describing interactions between agents, and (c) designing the overall system structure.
Step 1: specifying agent types
Through analyzing data coupling and grouping functionalities, the researcher designed five
types of software agents for the system, including instructor agent, student agent, management agent, assessment agent, and reporting agent (see Fig. 1).

123

The Assessment Agent System

205

Fig. 1 Five types of software agents in the Assessment Agent System

Instructor agents interact with instructors who initiate the interaction. The functionalities included in an instructor agent are user-system interaction, task authoring and management, and reporting. With an instructor agent, an instructor can create and edit
assessment tasks, assign tasks to students, and obtain assessment reports.
Student agents provide functionalities of user-system interaction, task presentation, response
collection, and reporting. Through interacting with a student agent, a student can request presentation of assessment tasks, create and submit concept maps, and request assessment reports.
The management agent responds to all requests from instructor agents and student
agents, communicates with the assessment agent and reporting agent, and manages data in
the database. The functionalities included in the management agent are user management,
task authoring and management, assessment delivery, response collection, and reporting.
The assessment agent assesses submitted student maps and generates feedback on
student maps. The reporting agent queries the database, generates assessment reports, and
delivers assessment reports to the management agent.
Step 2: describing interactions between agents
Interactions between agents were specified based on the scenarios developed in the system
specification phase. The interactions between agents are illustrated and defined with
interaction diagrams, messages, and message groups. An interaction diagram illustrates
interactions between agents (Padgham and Winikoff 2004). A message is information sent
from an agent to another agent. A message group includes a series of messages, responding
to a request, exchanged between two agents.
Step 3: designing the overall system structure
The overall system structure was designed based on the defined agents, messages, message
groups, percepts, actions, and data. The Assessment Agent System includes one management

123

206

J. Liu

agent, one assessment agent, one reporting agent, instructor agents and student agents. The
management agent, assessment agent, and reporting agent are situated in a server computer.
Instructor agents and student agents are situated in instructors or students computers. The
distribution of instructor agents and student agents in local computers reduces the communication workload between users and the server in large-scale assessment.
Phase 3: detailed design of agents
The detailed design of agents includes designing agents capabilities to provide the
assigned functionalities, and designing plans for actualizing agents capabilities. The
detailed design of the five types of software agents can be found in Liu (2010).

The development of the Assessment Agent System

The Assessment Agent System was developed with Dreamweaver, Flash, ColdFusion, and
MySQL. The system development process included developing a database, user interface,
and system functionalities, as well as testing, improving, and finalizing the system.
The student interface for creating concept maps is presented in Fig. 2. It displays nodes
and linking phrases. The bottom area of the student interface houses buttons for returning
to the student menu, viewing task descriptions, submitting concept maps, and exiting the
system.
An instructor uses existing software, including Excel and Navicat, to create, submit, and
edit tasks; to assign tasks to students; and to obtain assessment reports. Assessment tasks
can be created with Excel and transferred into the database with Navicat. An instructor
creates or edits task descriptions, node terms, linking phrases, and assessment criterion
propositions through interacting with the database.

Fig. 2 Student interface for creating concept maps

123

The Assessment Agent System

207

A programmer coded and implemented the database, user interface, and system functionality. The researcher tested the system on various computers and platforms. The system
was tested on Microsoft Windows and Mac OS X with popular Internet browsers, including
Firefox, Internet Explorer, and Safari. After the initial version of the Assessment Agent
System was developed, the researcher conducted three sessions of one-to-one evaluations
with graduate students in a large land-grant university to test user interface and system
functionality. The one-to-one evaluations revealed some problems with the student interface. The results of one-to-one evaluations were analyzed and used to improve the user
interface and system functionality. After the one-to-one evaluation, the user interface was
redesigned and redeveloped. Subsequently, the researcher conducted group evaluations on
the finalized Assessment Agent System.

Results
This study addressed three research questions. Responses to those questions were derived
partially from a review of relevant literature. This review was followed by the design,
development, and evaluation of the Assessment Agent System based on information
obtained from the literature review. The results of this process are presented below.
Research question 1 What functionalities are necessary for large-scale assessment based
on concept maps in a software agent-oriented system?
Through the literature review, the researcher identified eight categories of functionality.
These functionalities are user-system interaction, user management, task authoring and
management, assessment delivery, task presentation, response collection, automatic
assessment, and reporting. Details of these functionalities can be found in Table 2. The
developed Assessment Agent System provides these functionalities, which are carried out
by five types of software agents. Table 3 presents the developed functionalities of the
Assessment Agent System.
Research question 2 What system architecture is appropriate for large-scale assessment
based on concept maps in a software agent-oriented system?
The literature review suggested multi-agent structures would be appropriate to create
sophisticated, distributed computing systems (dInverno and Luck 2004; Wooldridge
2009). The Assessment Agent System in this study is composed of five types of software
agents. They are instructor agent, student agent, management agent, assessment agent, and
reporting agent. Each of these agents was designed to possess relevant capabilities. Software agents in the system, through communication and cooperation, collectively provide
the necessary functionalities.
The operability of the five types of software agents was examined as part of a group
evaluation, and all agents functioned as designed. The results of the group evaluation
appear later in this article.
Research question 3 What design provides a reliable and valid assessment method for
large-scale assessment based on concept maps in a software agent-oriented system?
The literature review suggested that when node terms and linking phrases are provided,
it should be possible to automate the process of assessing student concept maps if student
responses can be compared with the assessment criteria on a proposition by proposition
basis (e.g., Chung et al. 2006). The assessment is reliable. However, the validity of the

123

208

J. Liu

Table 3 The developed functionalities of the Assessment Agent System

Category of functionality

Developed system functionality

User-system interaction

Through the student interface, the system allows students to input their user
names and passwords; to select courses and assessment tasks; to create and
submit concept maps; and to request assessment reports. Assessment reports
are displayed on computer screens
Through the instructor interface, the system allows instructors to input their user
names and passwords, as well as student account information; to create and
edit assessment tasks; and to request assessment reports. Assessment reports
are displayed on computer screens

User management

The system accepts, maintains, and updates user data in the database. The system
allows authorized users to log into the system

Task authoring and

management

The system accepts and stores assessment task data and student account data in
the database

Assessment delivery

The system delivers assessment tasks to students computers via the internet

Task presentation

After a student selects a course and an assessment task, the assessment task is
delivered to the students computer. Nodes and linking phrases are displayed
on the students screen

Response collection

After a student creates a concept map and clicks the Submit Concept Map
button on the student interface, the system extracts the node terms and linking
phrases of the student map, and saves the node terms and linking phrases in the
database

Automatic assessment

The system compares propositions in student concept maps with the criterion
propositions and stores assessment results in the database. Assessment results
generated by the assessment agent fall into three categories: correct
proposition, different proposition, and missing proposition

Reporting

The system queries assessment results in the database, as well as generates,

delivers, and presents assessment reports

proposition-comparing method depends on the accuracy and thoroughness of the criterion

propositions.
This study resulted in the development of the Assessment Agent System. Students using
the system were presented with node terms and linking phrases and asked to create concept
maps demonstrating their understanding of a particular topic. The following section presents the group evaluation of the system.

Group evaluation of the Assessment Agent System

The purpose of the group evaluation was for students to evaluate the user interface and
system functionality.
Participants
Participants in the group evaluation were 34 students in a large land-grant university. They
were recruited through email and face-to-face invitations. Their study disciplines were
highly varied, including science, technology, engineering, education, business, social
sciences, and humanities. Twenty-nine participants had experience taking computer-based

123

The Assessment Agent System

209

assessments. Fifteen participants of the thirty-four had experience evaluating computerbased educational products.
Materials
Materials used in the group evaluation included (a) the student directions for evaluating the
Assessment Agent System, (b) a task description, and (c) a reading material. The student
directions material included fourteen steps that guided participants to perform the whole
process of evaluation, from electronically signing the consent form, to learning basic
knowledge of concept maps, to carrying out a concept map assessment task, to completing
an evaluation questionnaire. The reading material was about the concept of health and was
adapted from an introductory-level college textbook. The task description explained the
concept map assessment task and listed twenty-five node terms and nine linking phrases. It
was developed by the researcher based on the content of the reading material.
Data collection instrument
The data collection instrument was an evaluation questionnaire. The researcher developed
this questionnaire based on the usability checklist for computer-based testing programs
created by Harmes and Parshall (2000). The questionnaire included 29 items, which were
divided into four sections: participant background, visual clarity, system functionality,
consistency, as well as error prevention and correction.
Procedure
The evaluation was conducted in computer labs. First, participants were provided with
materials used in the evaluation, introduced to the research project, and told that they were
invited to evaluate the Assessment Agent System from the perspective of students. Second,
the researcher explained the evaluation procedure, provided the basic knowledge of concept maps, and demonstrated how to use the system. Third, participants began to read the
provided materials and carried out a concept map assessment task by following the
directions in the materials. Fourth, after participants completed their assessment tasks and
obtained their assessment reports, the researcher explained the content in the assessment
reports. Fifth, participants completed the evaluation questionnaire on their computers.
Results of the group evaluation
The collected data are participants responses to items in the evaluation questionnaire, as
well as their comments and suggestions. The participants responses to the evaluation
questions were counted, and the means and standard deviations of these responses were
calculated. Suggestions from participants were analyzed and summarized.
The means and standard deviations of the responses to evaluation questions from all
participants are reported in Table 4. The numbers represent responses on a 4-point scale: 4
for always, 3 for most of the time, 2 for some of the time, and 1 for never. A
different scale was used for the evaluation question, Did the assessment report provide
beneficial information about your concept map? The 4-point scale is: 4 for provided all
information needed, 3 for provided most information needed, 2 for provided some
information needed, and 1 for provided none of information needed. The range of

123

210

J. Liu

Table 4 Means and standard deviations of students responses to evaluation questions

Question

SD

Visual clarity
Was the information on the screen easy to read?

3.53

0.61

Did the color arrangement of the screen help make the displays clear?

3.71

0.58

Was it easy to find the needed information on the screen?

3.59

0.61

Were the different information settings clearly separated from one another?

3.53

0.66

Did all the buttons and menus function correctly?

3.76

0.43

Were you able to arrange nodes to create your concept map?

3.82

0.39

Were you able to remove nodes from your concept map?

3.74

0.67

Were you able to create linking lines?

3.91

0.29

Were you able to delete linking lines?

3.65

0.60

Were you able to drag and drop linking phrases into relationship boxes?

3.85

0.36

Were you able to change linking phrases that were already in relationship boxes?

3.88

0.54

Were you able to delete linking phrases?

3.59

0.92

Were you able to delete links?

3.76

0.50

Were you able to rearrange and organize nodes in your concept map?

3.88

0.33

Did the assessment report provide beneficial information about your concept map?

2.97

0.67

Were icons used consistently in the student interface?

3.85

0.36

Was the color arrangement consistent in the student interface?

3.97

0.17

Did the same type of information (e.g. navigation) have consistent placement in the student
interface?

3.91

0.29

Was the Assessment Agent System consistent in responding to a particular user action?

3.91

0.29

System functionality

Consistency

Error prevention and correction

Was it easy for you to correct a mistake?

3.53

0.61

Could you view the entire concept map on the screen?

3.53

0.75

Were you able to check your concept map before it was submitted?

3.79

0.59

Note N = 34

means is between 2.97 and 3.97. The results suggest that participants were generally
satisfied with the Assessment Agent System regarding visual clarity, system functionality,
consistency, as well as error prevention and correction. Their primary suggestions and
concerns are provided in the following section.
Visual clarity
On average, 93 % of the participants rated always or most of the time for the items
related to visual clarity. Suggestions for improving the user interface focused on the
concept map creation space and color options. The concept map assessment task contained
25 nodes. As more and more propositions were created, the area for building the concept
map on the screen became more and more crowded. Participants suggested enlarging the
concept map working space or making the size of nodes smaller. Participants also suggested providing ways to change the node shape and select the color of nodes, linking
phrases, and linking lines.

123

The Assessment Agent System

211

System functionality
On average, 95 % of the participants rated always or most of the time for the items
related to button functions, concept map creation, and assessment reports. Many participants expressed their concerns about propositions that were assessed as different proposition in their concept maps. They thought that some of those different propositions
were correct.
Consistency
All participants rated always or most of the time for the items related to consistency,
including use of icons, arrangement of colors, placement of information, and system
responses to user actions.
Error prevention and correction
On average, 93 % of the participants rated always or most of the time for the items
related to error prevention and correction, including correcting mistakes, viewing the entire
concept map on the screen, and checking their concept maps before submission. Participants complained that they were not allowed more screen space for creating concept maps.
They noted that nodes and links overlapped each other when more and more propositions
were created.

Discussion
System functionality
The Assessment Agent System possesses eight categories of functionality, which are usersystem interaction, user management, task authoring and management, assessment delivery, task presentation, response collection, automatic assessment, and reporting. These
functionalities are cooperatively carried out by five types of software agents in the
Assessment Agent System.
To evaluate the system functionalities, a concept map assessment task and assessment
criteria were developed and entered into the system, student accounts were created in the
system, and one-to-one evaluations as well as group evaluations were conducted. With the
system, users could be managed and assessment tasks could be created and updated.
Student users were able to interact with the system to request that the assessment task be
presented on their computer screens, to create and submit concept maps, and to obtain
assessment reports. The results of group evaluations indicate that the functionalities
designed and developed for the Assessment Agent System are necessary in order to meet
users needs for large-scale assessment based on concept maps.
Existing computer-based tools for assessing concept maps, such as Concept Connector
(Luckie et al. 2011), Concept Mapping Tool (Cline et al. 2010), and Human Performance
Knowledge Mapping Tool (Chung et al. 2006), possess a variety of functionalities. These
functionalities fall into the eight categories of functionality identified in this study. The
findings about necessary functionalities of this study are in agreement with those of other
studies on computer-based concept map assessment.

123

212

J. Liu

Students who evaluated the Assessment Agent System suggested more functions for the
system. Most of the suggestions related to user-system interaction and automatic assessment. For example, they suggested that the system could provide more advanced functions
for creating and modifying concept maps. They were also concerned about their concept
map assessment reports and wanted more personal feedback on their concept maps. In the
future, the Assessment Agent System can be improved on the functionalities of usersystem interaction and automatic assessment.
System architecture
The Assessment Agent System was designed as a multi-agent system, which is composed
of five types of software agents. They are instructor agent, student agent, management
agent, assessment agent, and reporting agent. These software agents provide the designed
functionalities through communication and cooperation. The architecture of a computer
system for large-scale assessment affects its performance, maintenance, and usability
(Higgins et al. 2003). Architectures of existing computer systems for assessing concept
maps were designed as either clientserver architecture or modeled as a set of modules
(e.g., Cline et al. 2010; Gouli et al. 2004; Luckie et al. 2011). Compared to these structures,
the Assessment Agent System was modeled as a group of software agents, which is a
higher level abstraction and a more reusable structure (Bergenti 2003). Therefore, although
the Assessment Agent System architecture was designed based on assessing concept maps,
the system architecture may serve as a template for designing other computer-based
assessment systems.
Concept map assessment method
When node terms and linking phrases are provided, the assessment of student concept
maps can be conducted automatically by comparing student concept maps with the criterion map, proposition by proposition. The reliability of assessment is assured in two
ways. First, the concept map assessment method in this study is reliable according to the
research on assessment tasks (Yin and Shavelson 2008) and scoring methods (McClure
et al. 1999). Second, the computer automatically compares student-created propositions
with the criterion propositions without introducing subjective errors in the assessment
process.
However, the validity of the proposition-comparing method depends on the criterion
propositions. In practice, students may create some propositions that do not match the
criterion propositions, but that may be considered correct. To overcome this problem, the
assessment process may include two phases. The assessment begins with expert-created
propositions as the assessment criteria. In the first phase, the assessment agent assesses the
first set of student-created propositions, and the assessment result includes three categories:
correct proposition, different proposition, and missing proposition. However, the assessment result is temporary. In the second phase, an expert examines the different propositions of the student concept map. The different propositions are identified as either
correct or incorrect. In this way, the final assessment results are produced, partly done by
the assessment agent and partly done by an expert.
Next, the expert adds the identified correct and incorrect propositions (assessed as
different by the assessment agent) of the student concept map to the assessment criterion
propositions pool. The assessment criteria are improved and include expert-identified
correct and incorrect propositions. Subsequent student concept maps will be assessed

123

The Assessment Agent System

213

against the improved assessment criteria, and assessment results generated by the assessment agent will include four categories: correct proposition, incorrect proposition, different
proposition, and missing proposition. The expert continually reviews the different
propositions of the student concept map, produces final assessment results and improves
the assessment criteria.
As the process continues, the pool of correct and incorrect propositions in assessment
criteria becomes larger and larger and covers more and more correct propositions, either
created by experts or students, along with incorrect propositions, created by students.
Although the assessment process relies on an expert to improve the assessment criteria at
the beginning, once the proposition pool in the database encompasses all possible propositions, the assessment agent can automatically produce valid assessment results.
Future research
The current study has focused on the design and development of the Assessment Agent
System. More work is needed in this line of research. For example, some functionalities of
the system can be improved to enhance its usability. The following are some possibilities
for future development:
(a)

Improve the student interface. Provide a means for changing the node shape,
adjusting the curve of linking lines, as well as selecting the color of nodes, linking
phrases, and linking lines.
(b) Design and develop a more user-friendly instructor interface. Allow instructors to
create and edit tasks and assessment criteria more conveniently, and to obtain
assessment reports that include more detailed information.
(c) Improve the functionality of the assessment agent by allowing it to identify incorrect
nodes, linking phrases, and direction of links.
Additionally, studies should be conducted to examine the large-scale effectiveness of
the Assessment Agent System in assessing students comprehensive understanding of
curricular content. Finally, assessment and learning mutually influence one another. Many
students acknowledge that assessment drives their learning. In particular, the format of
assessment impacts how students learn. Therefore, future research should explore how the
Assessment Agent System can be used to benefit students in their learning. The Assessment Agent System is thus a potential tool for helping students to build their comprehensive understanding of complex curricular content.
Acknowledgments The author would like to thank Drs. Kenneth Potter, Barbara Lockee, Mike Moore,
and Todd Ogle for their insightful advice on conducting this study, as well as Mr. Todd Bowden for his
knowledgeable assistance in developing the Assessment Agent System.

References
Bergenti, F. (2003). A discussion of two major benefits of using agents in software development. In P. Petta,
R. Tolksdorf, & F. Zambonelli (Eds.), Engineering societies in the agents world III (pp. 112). Berlin:
Springer-Verlag.
Bergenti, F., Gleizes, M.-P., & Zambonelli, F. (Eds.). (2004). Methodologies and software engineering for
agent systems: The agent-oriented software engineering handbook. Boston: Kluwer Academic.
Bull, J., & McKenna, C. (2004). Blueprint for computer-assisted assessment. New York: RoutledgeFalmer.
Burkhardt, H., & Pead, D. (2003). Computer-based assessment: A platform for better tests? In C. Richardson
(Ed.), Whither assessment? (pp. 133148). London: Qualifications and Curriculum Authority.

123

214

J. Liu

Chung, G.K.W.K., Baker, E.L., Brill, D.G., Sinha, R., Saadat, F., & Bewley, W. L. (2006). Automated
assessment of domain knowledge with online knowledge mapping (CSE Tech. Rep. No. 692). Retrieved
from University of California, Los Angeles, National Center for Research on Evaluation, Standards,
and Student Testing website: http://www.cse.ucla.edu/products/reports/r692.pdf. Accessed 26 Apr
2012.
Cline, B. E., Brewster, C. C., & Fell, R. D. (2010). A rule-based system for automatically evaluating student
concept maps. Expert Systems with Applications, 37, 22822291. doi:10.1016/j.eswa.2009.07.044.
dInverno, M., & Luck, M. (2004). Understanding agent systems (2nd ed.). Berlin: Springer-Verlag.
Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning
Sciences, 11, 105121.
Fisher, K. M. (2000). SemNet software as an assessment tool. In J. J. Mintzes, J. H. Wandersee, &
J. D. Novak (Eds.), Assessing science understanding: A human constructivist view (pp. 197221). San
Diego, CA: Academic Press.
Gouli, E., Gogoulou, A., Papanikolaou, K., & Grigoriadou, M. (2004). COMPASS: An adaptive web-based
concept map assessment tool. In A. J. Canas, J. D. Novak, & F. M. Gonzalez (Eds.), Concept maps:
Theory, methodology, technology. Proceedings of the first international conference on concept mapping. Retrieved from http://cmc.ihmc.us/papers/cmc2004-128.pdf. Accessed 26 Apr 2012.
Grundspenkis, J. (2011). Concept map based intelligent knowledge assessment system: Experience of
development and practical use. In D. Ifenthaler, P. Isaas, J. M. Spector, Kinshuk, & D. G. Sampson
(Eds.), Multiple perspectives on problem solving and learning in the digital age (pp. 179197). New
York: Springer. doi:10.1007/978-1-4419-7612-3_12.
Harmes, J. C., & Parshall, C. G. (2000, November). An iterative process for computerized test development:
Integrating usability methods. Paper presented at the annual meeting of the Florida Educational
Research Association, Tallahassee.
Henderson-Sellers, B., & Giorgini, P. (Eds.). (2005). Agent-oriented methodologies. Hershey, PA: Idea
Group Publishing.
Herl, H. E., Baker, E. L., & Niemi, D. (1996). Construct validation of an approach to modeling cognitive
structure of U.S. history knowledge. The Journal of Educational Research, 89, 206218.
Herl, H. E., ONeil, H. F, Jr, Chung, G. K. W. K., & Schacter, J. (1999). Reliability and validity of a
computer-based knowledge mapping system to measure content understanding. Computers in Human
Behavior, 15, 315333. doi:10.1016/S0747-5632(99)00026-6.
Higgins, C., Hegazy, T., Symeonidis, P., & Tsintsifas, A. (2003). The CourseMarker CBA system:
Improvements over Ceilidh. Education and Information Technologies, 8, 287304. doi:10.1023/A:
1026364126982.
Jennings, N. R. (2001). April). An agent-based approach for building complex software systems. Communications of the ACM, 44(4), 3541. doi:10.1145/367211.367250.
Liu, J. (2010). The assessment agent system: Assessing comprehensive understanding based on concept
maps (Doctoral dissertation). Retrieved from http://scholar.lib.vt.edu/theses/. Accessed 26 Apr 2012.
Luck, M., Ashri, R., & dInverno, M. (2004). Agent-based software development. Boston: Artech House.
Luckie, D., Harrison, S. H., & Ebert-May, D. (2011). Model-based reasoning: Using visual tools to reveal
student learning. Advances in Physiology Education, 35, 5967. doi:10.1152/advan.00016.2010.
McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475492.
Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. New York: Cambridge University Press.
Padgham, L., & Winikoff, M. (2004). Developing intelligent agent systems: A practical guide. Chichester,
UK: John Wiley & Sons.
Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010). Highly integrated model assessment technology and
tools. Educational Technology Research and Development, 58, 318. doi:10.1007/s11423-009-9119-8.
Richey, R. C., & Klein, J. D. (2007). Design and development research: Methods, strategies, and issues.
Mahwah, NJ: Lawrence Erlbaum Associates.
Richey, R. C., Klein, J. D., & Nelson, W. A. (2004). Developmental research: Studies of instructional design
and development. In D. H. Jonassen (Ed.), Handbook of research on educational communications and
technology (2nd ed., pp. 10991130). Mahwah, NJ: Lawrence Erlbaum Associates.
Ruiz-Primo, M. A. (2004). Examining concept maps as an assessment tool. In A. J. Canas, J. D. Novak, &
F. M. Gonzalez (Eds.), Concept maps: Theory, methodology, technology. Proceedings of the first international conference on concept mapping. Retrieved from http://cmc.ihmc.us/papers/cmc2004-036.pdf.
Accessed 26 Apr 2012.
Ruiz-Primo, M. A., Schultz, S. E., Li, M., & Shavelson, R. J. (2001). Comparison of the reliability and
validity of scores from two concept-mapping techniques. Journal of Research in Science Teaching, 38,
260278.

123

The Assessment Agent System

215

Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science
assessment. Journal of Research in Science Teaching, 33, 569600.
Sterling, L. S., & Taveter, K. (2009). The art of agent-oriented modeling. Cambridge, MA: MIT Press.
Sycara, K. P. (1998). Multiagent systems. AI Magazine, 19(2), 7992.
Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps.
Educational Technology Research and Development, 54, 6582. doi:10.1007/s11423-006-6497-z.
Tecuci, G., & Keeling, H. (1999). Developing an intelligent educational agent with disciple. International
Journal of Artificial Intelligence in Education, 10, 221237.
van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (Eds.). (2006). Educational design
research. New York: Routledge.
Wooldridge, M. (2009). An introduction to multiagent systems (2nd ed.). Chichester, UK: John Wiley &
Sons.
Yin, Y., & Shavelson, R. J. (2008). Application of generalizability theory to concept map assessment
research. Applied Measurement in Education, 21, 273291. doi:10.1080/08957340802161840.
Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison of two
concept-mapping techniques: Implications for scoring, interpretation, and use. Journal of Research in
Science Teaching, 42, 166184. doi:10.1002/tea.20049.

Jianhua Liu received his Ph.D. in Curriculum and Instruction, with a specialization in Instructional Design
and Technology, from Virginia Polytechnic Institute and State University, Blacksburg, Virginia. His
research has focused on the use of software agent technology in education.

123

Copyright of Educational Technology Research & Development is the property of Springer Science & Business
Media B.V. and its content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email articles for
individual use.