COMSATS INTITUTE OF INFORMATION TECHNOLOGY ISLAMABAD

Testing In Agent Base Systems

Hafsah
Qaisar Afzal
Ahsan Azam
Tahir Mehmood
Junaid Kiani

SP14-RSE-012
SP14-RSE-013
SP14-RSE-014
SP14-RSE-016
SP14-RSE-017

Testing In Agent Base Systems

Testing In Agent Base Systems

1. What is Agent?
One that acts or has the power or authority to act or represent another

1.1. What is Software Agent?

According to Z. Houhamdi:
A software agent is a computer program that works toward goals in a dynamic context on behalf
of another entity (human or computational), perhaps for a long period of time, with
discontinuous direct supervision or control, and exhibits a significant flexibility and even
creativity degree in how it tries to transform goals into action tasks [3].Software agents
have (among others) the following properties:

1. Reactivity: agents are able to sense contextual changes and react appropriately;
2. Pro-activity: agents are autonomous, so they are able to select which actions to take in
order to reach their goals in given situations;
3. Social ability: that is, agents are interacting entities, which cooperate, share
knowledge, or compete for goal achievement
OR
A software agent is defined as a software program that can perform specific tasks for a user and
possessing a degree of intelligence that permits it to perform parts of its tasks autonomously and
to interact with its environment in a useful manner.

COMSATS institute of Information Technology

Page 2

Testing In Agent Base Systems

1.2. Agent Types:

The following are the types of Agents:
Intelligent Agents
Collaborative Agent
Interface Agent
Mobile Agent
Information/Internet Agent
Reactive Software Agent
Heterogeneous Agent System

1.2.1. Intelligent Agent

Intelligent agents continuously perform three functions: perception of dynamic conditions in the
environment; action to affect conditions in the environment; and reasoning to interpret perceptions,
solve problems, draw inferences, and determine actions.

Sensor Input

Agent

Action Output

Environment

1.2.1.1.

Characteristics of Intelligent Agent

Autonomy: The ability to take goal directed autonomous decisions without intervening with
human or other agents.
Goal Oriented: Does not simply act in response of environment.
Communicative: communicate with other agents including human.
Learning: changes its behavior based on previous experience.
Flexible: Actions are not scripts
COMSATS institute of Information Technology

Page 3

Testing In Agent Base Systems

Pro-activity: The ability to initiate actions to accomplish the goals rather than responding to the
unpredictable environment.
Reactivity: The ability to respond in selective and timely fashion by perceiving the environment.
Social ability: The ability to work in collaboration with other agents and human in order to
achieve the goal.
Mobility: The ability to move from one platform/environment to another in self-directed way.

1.2.2. Collaborative Agents:

Collaborative agents emphasize autonomy and cooperation (with other agents) in order to
perform tasks for their owners. They may learn, but this aspect is not typically a major emphasis
of their operation. In order to have a coordinated set up of collaborative agents, they may have to
negotiate in order to reach mutually acceptable agreements on some matters.
In brief the key general characteristics of these agents include autonomy, social ability,
responsiveness and pro-activeness. Hence, they are (or should/would be) able to act rationally
and autonomously in open and time-constrained multi-agent environments.

1.2.3. Interface Agents:

Interface agents emphasize autonomy and learning in order to perform tasks for their owners. A
key proponent of this class of agents, points out that the key metaphor underlying interface
agents is that of a personal assistant who is collaborating with the user in the same work
environment.
Note the subtle emphasis and distinction between collaborating with the user and collaborating
with other agents as is the case with collaborative agents. Collaborating with a user may not
require an explicit agent communication language as one required when collaborating with other
agents.

Figure: How Interface Agents Work

COMSATS institute of Information Technology

Page 4

Testing In Agent Base Systems

Figure depicts the functioning of interface agents. Essentially, interface agents support and
provide assistance, typically to a user learning to use a particular application such as a
spreadsheet or an operating system. The agent observes and monitors the actions taken by the
user in the interface), learns new short-cuts, and suggests better ways of doing the task. Thus, the
interface agent acts as an autonomous personal assistant which cooperates with the user in
accomplishing some task in the application. As for learning, interface agents learn typically to
better assist its user in four ways all shown in Figure:

By observing and imitating the user (i.e. learning from the user);
Through receiving positive and negative feedback from the user (learning from
the user);
By receiving explicit instructions from the user (learning from the user);
By asking other agents for advice (i.e. learning from peers)

Their cooperation with other agents, if any, is limited typically to asking for advice, and not in
getting into protracted negotiation deals with them as is the case with collaborative agents. The
learning modes are typically by rote (memory-based learning) or parametric, though other
techniques such as evolutionary learning are also being introduced.

1.2.4. Mobile Agents

Mobile agents are computational software processes capable of roaming wide area networks
(WANs) such as the WWW, interacting with foreign hosts, gathering information on behalf of its
owner and coming back home having performed the duties set by its user. These duties may
range from a flight reservation to managing a telecommunications network.
Mobile agents are agents because they are autonomous and they cooperate, differently to
collaborative agents. For example, they may cooperate or communicate by one agent making the
location of some of its internal objects and methods known to other agents. By doing this, an
agent exchanges data or information with other agents without necessarily giving all its
information away.

1.2.5. Information/Internet Agents:

Information agents perform the role of managing, manipulating or collating information from
many distributed sources.

COMSATS institute of Information Technology

Page 5

Testing In Agent Base Systems

1.2.6. Reactive Software Agents:

Reactive agents represent a special category of agents which do not possess internal, symbolic
models of their environments; instead they act/respond in a stimulus-response manner to the
present state of the environment in which they are embedded.
There are three key ideas which underpin reactive agents.
Firstly, emergent functionality which we have already mentioned, i.e. the dynamics of the
interaction leads to the emergent complexity. Hence, there is no a priori specification (or plan) of
the behavior of the set-up of reactive agents.

1.3. Agent Based Systems

Agent based systems are the Ones in which the key abstraction used is that of an agent. It might be
conceptualized in terms of agents or analogy with OO Systems. A number of software tools exist that
allow a user to implement software systems as agents, and as societies of cooperating agents.

1.3.1. Agents Based System Example

a) Autonomous Airplanes
Currently large airplanes have most of the technologies for autonomous flight like
control of individual airplanes
Coordination of activities with other airplanes relies on central authorities.
requires dealing with non-autonomous airplanes older or smaller ones
Proposals have been put forward to allow more autonomy

UAVs for military and security applications are in use. Most of them are to a large
degree remotely controlled.

b) Apples Siri:
Siri is the intelligent personal assistant that helps you get things done just by asking. It allows
you to use your voice to send messages, schedule meetings, place phone calls, and more. But Siri
isnt like traditional voice recognition software that requires you to remember keywords and
speak specific commands. Siri understands your natural speech, and it asks you questions if it
needs more information to complete a task.

2. Multi-Agent System:

COMSATS institute of Information Technology

Page 6

Testing In Agent Base Systems

A multi-agent system (MAS) is a computational context in which individual software agents

interact with each other, in a collaborative (using message passing) or competitive manner, and
sometimes autonomously trying to attain their individual goals, accessing resources and services
of the context, and occasionally producing results for the entities that initiated those software
agents [4].Consequently, they are difficult to debug and test.

2.1. Testing In Multi Agent Systems

2.1.1. Test Levels
Over the last years, the view of testing has evolved, and testing is no longer seen as a step which
starts only after the implementation phase is finished. Software testing is now seen as a whole
process that filters in the development and maintenance activities. Thus, each development phase
and maintenance phase should have a corresponding test level.
Work in testing software agents and MAS can be classified into different testing levels: unit,
agent, integration, system, and acceptance. Here we use general terminologies rather than using
specific ones used in the community like group, society. Group and society, as called elsewhere,
are equivalent to integration and system, respectively. The testing objectives, subjects to test, and
activities of each level are described as follows:
Unit testing: tests all units that make up an agent, including blocks of code, implementation of
agent units like goals, plans, knowledge base, reasoning engine, rules specification, and so on;
make sure that they work as designed.
Agent testing: tests the integration of the different modules inside an agent; test agents'
capabilities to fulfill their goals and to sense and effect the environment.
Integration or Group testing: tests the interaction of agents, communication protocol and
semantics, interaction of agents with the environment, integration of agents with shared
resources, regulations enforcement; Observe emergent properties, collective behaviors; make
sure that a group of agents and environmental resources work correctly together.
System or Society testing: tests the MAS as a system running at the target operating
environment; test the expected emergent and macroscopic properties of the system as a whole;
test the quality properties that the intended system must reach, such as adaptation, openness,
fault tolerance, performance.
Acceptance testing: tests the MAS in the customer's execution environment and verifies that it
meets stakeholder goals, with the participation of stakeholders.

COMSATS institute of Information Technology

Page 7

Testing In Agent Base Systems

2.1.2. Multi Agent Systems are Difficult to Test

Software agents and MAS testing is a challenging task because these systems are distributed,
autonomous, and deliberative. They operate in an open world, which requires context awareness.
In particular, the very particular character of software agents makes it difficult to apply existing
software testing techniques to them. There are issues concerning communication and semantic
interoperability, as well as coordination with peers. All these features are known to be hard not
only to design and to program [1], but also to test.
There are several reasons for the increase of the difficulty degree of testing MAS:

Increased complexity, since there are several distributed processes that run
autonomously and concurrently;

Amount of data, since systems can be made up by thousands of agents, each owning its
own data;

Irreproducibility effect, since we cant ensure that two executions of the systems will
lead to the same state, even if the same input is used. As a consequence, looking for a
particular error can be difficult if it is impossible to reproduce it each time [2].

They are also non-deterministic, since it is not possible to determine a priori all
interactions of an agent during its execution.

Agents communicate primarily through message passing instead of method

invocation, so existing object-oriented testing approaches are not directly applicable.

Agents are autonomous and cooperate with other agents, so they may run correctly by
themselves but incorrectly in a community or vice versa.

3.Continuous Testing of Multi Agent System:

Multi-Agent systems are systems composed of multiple software agents that interact with one
another in order to achieve their intended goals, and the goals of the systems as a whole. A MAS
is usually a distributed system, its agents can be located in different hosts, and they communicate
mainly through message passing. Each host provides a specific environment for the agents
located at that host. Due to these peculiar properties, testing MAS is a challenging task.

COMSATS institute of Information Technology

Page 8

Testing In Agent Base Systems

4. Testing Techniques
4.1.

Role Oriented Approach

Role is an important attribute of an agent that has a huge scope and support for the analysis,
design and implementation of Multi-Agent System (MAS). A role can be defined as the
capability enabler that exposes to the agent that plays it a set of actions. Roles are created to do
something and it has the responsibility of achieving specific system goals and sub goals. Roles
provide a well-defined interface between agents and cooperative processes.
Role Oriented Testing
Testing the responsibilities corresponding to their respective role ensures that the agent plays its
role as per the intended goal. As long as the goals are getting satisfied, it is understood that the
individual agent is performing well that suits the system.

An = Gi ( Rj ( Rek ) ) where i = j = k = x = 1
Following are the steps involved in role-oriented testing.

STEP 1: Select the Agent to be tested.

STEP 2: Identify Goals (Gi), Roles (Rj) and their corresponding Responsibilities (Rek)
STEP 3: Design Role Model Diagram (Ax (Gi Rj Rek)
STEP 4: Analyze role model (Ax(Gi Rj Rek)
STEP 5: Define the interacting agents and situations.
STEP 6: Identify environmental factors pre-conditioning input trigger Rek
STEP 7: Identify fulfillment criteria that satisfies Responsibility
STEP 8: Run test cases

Role Oriented Unit Testing

Role Oriented Unit Testing focuses primarily on the smallest building block of the Multi-Agent
System. The basic idea behind the unit testing in MAS is to verify whether the individual agent
(unit) in isolation performs its responsibilities under various conditions. Every individual agent
has its own goal to be achieved and plans to do to fulfill the goal. In addition to goal and plan,
role is one important mental state of the agent, which is defined as a set of capabilities and
expected behavior. A role can be represented as <Goal, Responsibilities, Protocol, and
Permissions>
Goal, for which the agent playing this role is responsible

Responsibilities, Which indicates the functionalities of agents playing such roles

COMSATS institute of Information Technology

Page 9

Testing In Agent Base Systems

Protocol, which indicates how an agent playing such role can interact with agents playing
other role

Permissions, which are a set of rights associated with the role.

<Agent>
<Goal>
<Role1>
<Responsibilities 1>
<Protocol>
.
<Responsibilities n>
<Protocols>
</Role1>
<Role n>
<Responsibilities 1>
<Protocol>
.
<Responsibilities n>
<Protocol>
</Role n>
</Goal>
</Agent>
Role Oriented Integration Testing
Integration testing in conventional software development is that to progressively integrate the
tested units (module, program, procedure, function).After ensuring that the individual agent in
isolation is working as per the requirement; the next immediate step is to integrate the agents
involved in the MAS so as to test the interaction, and communication among agents.
Role Oriented System Testing
System testing verifies that all elements (hardware, people, and databases) are integrated
properly so as to ensure whether the overall product met its requirement and achieved the
expected performance. System testing also deals with non-functional requirements of the
software such as recovery testing, security testing, stress testing and performance testing. System
testing in agent oriented approach will test the complete functionality and test the system as a
whole.
Role Schema
Role schema provides a well-defined interface between agents and cooperative processes. This
allows an agent to read and follow, normative rules established by the cooperation process even
if not previously known by the agent. Their major motivation to introduce such roles is to
increase the agent systems adaptability to structural changes. Role schema involves role name,
COMSATS institute of Information Technology

Page 10

Testing In Agent Base Systems

agent name, goal to be achieved, description of the role, protocol and related activities,
permissions and responsibilities.
Role Schema

Advantages of Role Based Approach

Role based approach helps the multi agent system developers in testing the individual unit of the
agent based system.
Disadvantages of Role Based Approach
Testing whether the agent performs its role properly is a challenging task.

4.2. Unit Testing in Multi-Agents Systems using Mock

Agents
In a multi-agent system it may not always be possible to scan/gather the complete agent
requirements, due to the fact that agent changes its functionalities based on the environmental
need so as to achieve the goal. Agent is an important building block of a multi-agent system and
it is very essential to verify whether the agent in isolation matches with the specification under
normal and abnormal condition. Thus in an agent-oriented programming, the smallest building
block/unit is an agent itself. Unit testing with respect to agent based system is to individually test

COMSATS institute of Information Technology

Page 11

Testing In Agent Base Systems

all the agents involved in the system. Testing an agent is based on its own mental attributes such
as goal, plan, role and action. Test cases are to be generated so as to ensure the following,
1. To test whether the goal of an agent is achieved.
2. To test whether the plan get triggered by the event that is supposed to handle.
3. To test whether the agent takes appropriate role as expected by the environment

4. To test whether the agent performs the activity as per the functionality.
The primary aim of unit testing is to uncover errors within a given unit
A Mock Agent is a regular agent that communicates with just one agent: the AUT. It has just one
plan to test the AUT. The Mock Agents plan is equivalent to a test script, since it defines the
messages that should be sent to the AUT and the messages that should be received from it. By
testing an agent in isolation using Mock Agents the programmer is forced to consider the agents
interactions with its collaborators (or competitors), possibly before those collaborators (or
competitors) exist.
Agent unit test approach that is composed of five participants:
1- Test Suite: which consists in a set of Test Cases and a set of operations performed to
prepare the test environment before a Test Case starts.
2- Test Case: defines a scenario a set of conditions to which an Agent under Test is
exposed, and verifies whether this agent obeys its specification under such conditions.
3- Agent under Test (AUT): is the agent whose behavior is verified by a Test Case.
4- Mock Agent: consists in a fake implementation of a real agent that interacts with the
AUT. Its purpose is to simulate a real agent strictly for testing the AUT.
5- Agent Monitor: is responsible for monitoring agents life cycle in order to notify the TestCase about agents states.

COMSATS institute of Information Technology

Page 12

Testing In Agent Base Systems

Each agent unit test follows the common structure depicted in Figure.
In step 1, the Test Suite creates the agents platform and whatever element needed to set up the
test environment.
After that, a Test Case is started. The Test Case creates Mock Agents to every role that interacts
with the Agent under Test - in the scenario defined by the Test Case (step 2).
Next, it creates the Agent under Test (step 3) and asks the Agent Monitor to be notified when the
interaction between the AUT and the Mock Agents finishes (step 4).
At this point, the AUT and the Mock Agent start to interact. The Mock Agent sends a message to
the AUT, and it replies (steps 5 and 6) or vice-versa. They can repeat steps 5 and 6 as many times
as necessary to perform the test codified in the Mock Agents plan (for instance, the Mock
Agent can reply three messages before finalizing its test activity).
During all this interaction process, the Agent Monitor keeps track of changes in agents life
cycle. In order to do that it uses three lists as illustrated in Figure : (i) Created Agents List:
maintains IDs of the agents that have been created, but are not running yet an ID is any
information that uniquely identifies an agent.(ii) Running Agents List: maintains IDs of the
agents in the running state. (iii) Work Done Agents List: maintains IDs of the Mock Agents that
have completed their plan (equivalent to a test script).When a Mock Agent concludes its plan, the
Agent Monitor includes the Mock Agents ID in the Work done list.
And then notifies the Test Case that the interaction between the Mock Agent and the AUT have
concluded (step 7).
Lastly, the Test Case asks the Mock Agent whether or not AUT acted as expected (step 8).
This agent unit testing approach has two main concerns: (i) the design of a Test Case based on
the use of Mock Agents; (ii) and the Test Case execution which relies on the Agent Monitor to
notify when the test script (codified in Mock Agents plan) was concluded. As each Mock Agent
exercises just one role of the AUT, rather than the wide interface that comprises all the features
provided it, we call this approach Role Driven Unit Testing.
Pros:
1- In MASs testing approach Unit Testing aims to remove faults along the
application development.
2- Unit testing is independent of any development life cycle used.
3- Unit testing in MAS verifies whether each agent in isolation respects its
specifications under normal and abnormal conditions.
COMSATS institute of Information Technology

Page 13

Testing In Agent Base Systems

4- It is a role based approach, we can find out the individual behavior of

agents.
Cons:
1- It doesnt create complete test cases, this technique needs to be
combined with other strategies to make a thorough set of test cases.
2- This approach only tests individual agent, group testing of agents or
system of agents as a whole is not considered

5. Conclusion:
Most of the existing research work on testing software agent and MAS focuses mainly on agent
and integration level. Basic issues of testing software agents like message passing,
distributed/asynchronous have been considered; testing frameworks have been proposed to
facilitate testing process. And yet, there is still much room for further investigations, for instance
[5]:
A complete and comprehensive testing process for software agents and MAS.

Testing MAS at system and acceptance level: how do the developers and the end-users
build confidence in autonomous agents?

Test inputs definition and generation to deal with open and dynamic nature of software
agents and MAS.

Test oracles, how to judge an autonomous behavior? How to evaluate agents that have
their own goals from human tester's subjective perspectives?

Testing emergent properties at macroscopic system level: how to judge if an emergent

property is correct? How to check the mutual relationship between macroscopic and
agent behaviors?

Deriving metrics to assess the qualities of the MAS under test, such as safety, efficiency,
and openness.

Reducing/removing side effects in test execution and monitoring because introducing

new entities in the system, e.g., mock agents tester agents, and monitoring agent as in
many approaches, can influence the behavior of the agents under test and the
performance of the system as a whole.

COMSATS institute of Information Technology

Page 14

Testing In Agent Base Systems

References:
1. F. Bergenti, M. Gleizes, and F. Zambonelli, Methodologies and Software Engineering
for Agent Systems, The Agent-Oriented Software Engineering Handbook, Springer, Vol.
11, 2004.
2. I. Sommerville, Software Engineering, 9th edition, Addison Wesley, 2011.
3.

Z. Houhamdi, Test Suite Generation Process for Agent Testing, Indian Journal of
Computer Science and Engineering, Vol. 2, 2, 2011.

4. J. Botia, A. Lopez-Acosta, and G. Skarmeta, ACLAnalyser: A tool for debugging multiagent systems, Proceeding of the 16th European Conference on Artificial Intelligence,
pp. 967-968, IOS Press 2004.
5. Multi-Agent System Testing: A Survey
6. Sivakumar, N., and K. Vivekanandan. "Exploring the Need for Specialized Testing
Technique for an Agent-Based Software." PROMETHEUS 2002.7 (2002).
7. Agent Oriented Software Testing Role Oriented approach (IJACSA) International

Journal of Advanced Computer Science and Applications, Vol. 3, No.12, 2012 by

N.Sivakumar and K.Vivekanandan
8. Unit Testing in Multi-agent Systems using Mock Agents By Roberta
Coelho, Uir Kulesza, Arndt von Staa, Carlos Lucena.
9. Computer Science Department, Pontifical Catholic University of Rio de Janeiro
PUC-Rio, Brazil
10.{roberta, uira, arndt, lucena}@inf.puc-rio.br
11. Agent Oriented Software Testing Role Oriented approach (IJACSA) International

Journal of Advanced Computer Science and Applications, Vol. 3, No.12, 2012 by

N.Sivakumar and K.Vivekanandan
12. http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs01-introduction-2x2.pdf Last
Accessed at 27-11-2014

13. http://www.apple.com/iphone/features/siri.html Last Accessed at 25-11-2014

COMSATS institute of Information Technology

Page 15

Testing In Agent Base Systems

Resources for Further Reading

Research Papers:
1. Kissoum, Y., Sahnoun, Z., & Barkaoui, K. (2009, April). An approach for testing
mobile agents using the nets within nets paradigm. In Research Challenges in
Information Science, 2009. RCIS 2009. Third International Conference on (pp. 207216). IEEE.
2. Padgham, L., Zhang, Z., Thangarajah, J., & Miller, T. (2013). Model-based test oracle
generation for automated unit testing of agent systems. Software Engineering, IEEE
Transactions on, 39(9), 1230-1244.
3. Kalboussi, S., Bechikh, S., Kessentini, M., & Said, L. B. (2013, November). On the
Influence of the Number of Objectives in Evolutionary Autonomous Software Agent
Testing. In Tools with Artificial Intelligence (ICTAI), 2013 IEEE 25th International
Conference on (pp. 229-234). IEEE.
4. Malz, C., & Jazdi, N. (2010, May). Agent-based test management for software system
test. In Automation Quality and Testing Robotics (AQTR), 2010 IEEE International
Conference on (Vol. 2, pp. 1-6). IEEE.
5. Moser, T., Merdan, M., & Biffl, S. (2010, February). A pattern-based coordination and
test framework for multi-agent simulation of production automation systems. In
Complex, Intelligent and Software Intensive Systems (CISIS), 2010 International
Conference on (pp. 526-533). IEEE.
6. Gurcan, O., Dikenelli, O., & Bernon, C. (2011, September). Towards a generic testing
framework for agent-based simulation models. In Computer Science and Information
Systems (FedCSIS), 2011 Federated Conference on (pp. 635-642). IEEE.
7. Miller, T., Padgham, L., & Thangarajah, J. (2011). Test coverage criteria for agent
interaction testing. In Agent-Oriented Software Engineering XI (pp. 91-105). Springer
Berlin Heidelberg.
8. Jing, G., & Yuqing, L. (2009, December). Automatic Test Task Allocation in AgentBased Distributed Automated Testing Framework. In Computational Intelligence and
Software Engineering, 2009. CiSE 2009. International Conference on (pp. 1-5). IEEE.

COMSATS institute of Information Technology

Page 16

Testing In Agent Base Systems

9. Paydar, S., & Kahani, M. (2011). An agent-based framework for automated testing of
web-based systems. Journal of Software Engineering and Applications, 4(02), 86.

Web Links:
1.
2.
3.
4.

http://www.agentbuilder.com/Documentation/whyAgents.html
http://www.sce.carleton.ca/netmanage/docs/AgentsOverview/ao.html
http://www.ai.mit.edu/projects/humanoid-robotics-group/cog/capabilities.html
http://www.slideshare.net/ToniMorenoURV/introduction-to-agents-and-multiagentsystems
5. http://www.agentbuilder.com/Documentation/product.html
6. www.askjeeves.com
7. http://ti.arc.nasa.gov/tech/asr/planning-and-scheduling/remote-agent/
8. http://www.agentbuilder.com/AgentTechnology/agentApplications.html
9. http://www.agentbuilder.com/AgentTools/index.html
10. http://en.wikipedia.org/wiki/Distributed_artificial_intelligence

COMSATS institute of Information Technology

Page 17