Replication of the MacroABM Model

Replication Issues in the Context of Economic Agents

Richard Oliver Legendi
AITIA International and Etvs Lornd University
Budapest, Hungary
rlegendi@aitia.ai

Laszlo Gulyas
AITIA International and Etvs Lornd University
Budapest, Hungary
lgulyas@aitia.ai

Abstract Agent-based approaches are getting more and more attention

recently. In our current work we replicated the initial model of Domenico Delli
Gatti et al. described in their work entitled Macroeconomics from the Bottomup. We address the question of validating and verifying simulations in general,
also in the context of economic modelling, and summarize the lessons we
learnt from the replication of the aforementioned model. The results highlight
the importance of explicit documentation of the actors, timing of the events,
and partial results that replicate the hallmarks of the model which can be
verified independently with a set of simulation runs.
replication, simulation, complex adaptive systems, emergent economics

Introduction
Classical economy created well defined approaches and several tools that seemed to serve us for the
last decades. However, recent economic fluctuations emphasise the need of alternative tools,
approaches, frameworks and methodologies to be able to describe such phenomena. Recent works
emphasize their weaknesses [1].

Econometric models are in principal statistical models used to predict the state of a system,
supposing that the trends are generally unchanged. In other words, they work only if the
system stays (more or less) in the same state, but fails in the face of wild oscillations we are
experiencing recently.
Another common criticism stated against widely used tools (specifically: against dynamic
stochastic general equilibrium models) is that actors are homogeneous, the system has an
optimal equilibrium state, and assume a perfect world (where great oscillations like crises
are ruled out inherently).

Recent years show substantial growth in interest among scientists for alternative approaches,
especially towards agent-based modelling. The reason is twofold. First, the agent-based framework
allows us to move away from the representative agent paradigm into a model of heterogeneous
actors, agents. Second, the bottom-up approach enables the system to change in a non ad-hoc way.

Page | 1

This approach is popular in various fields like political science, social science and economics as well,
because it allows us to conduct experiments with the modelled system. This computational model
helps us find emergent behaviour that propagates from lower (micro) level interactions to higher
(macro) level regularities in the formalized phenomena.

Replication: Verification, Validation and Certification

In the case we consider simulations as experiments, reproducibility is a crucial question. In the
context of models created to describe real-world phenomena (for instance in physics and natural
sciences), emphasis must be put on the reproducibility of experiments to validate the results as a
scientific result. If a model cannot generate the same output for the exactly same conditions, its
scientific value is questioned.
In the context of computer simulations, there is a growing list of publications covering the topics of
docking (the alignment of different models to demonstrate they produce similar results) and model
replications (the reimplementation of an original model, typically in a different environment). An
extensive review of these approaches can be found in [22].
Alignment of computational models, or docking, as referred commonly (in analogy to orbital docking
of dissimilar spacecrafts), is a process where we fit a series of models against each other, and try to
verify if they can produce the same results which is crucial for equivalence testing of models. The
ability to tell if two different models targeting the same phenomena can (or cannot) generate the
same results is essential. A few examples of successful docking experiments include [6], [18] or [19].
Replication is another critical issue, which is an important method of scientists. These models were
designed to describe, explain or predict different real-world phenomena, and without being able to
replicate artificial models it is impossible to target real-world systems. Unfortunately, most of the
agent-based models were never replicated by anyone but the original author, with the exception of
some classic models [15]. Questions like how to replicate a model, and how to evaluate the
replication is often raised at several forums.
One of the major problems is ambiguity: there are several concepts how a model should be defined
in order to minimize uncertainty. Several different approaches exist, from textual frameworks like
the Overview, Design concepts, and Design details (ODD, [20]), extensions on top of standardized
modelling languages like AgentUML [21], but there is still no consensus on using them. Until these
frameworks become widely accepted and applied, replication is an issue of concern.
An example for replication of a classic agent-based model on the platform we are also approaching is
[16], while several notable examples can be found in [22].
The authors of the Bottom-up Adaptive Macroeconomics (BAM) model [5] applied empirical external
validation based on real-world data. In our current work, we validate the simulation through
replication. Our goal is to create an alternative implementation in a different environment and verify
if these models produce the same results in other words, we test their identicality.In our recent
work, we successfully created an alternative implementation of the original BAM model. The
replicated model resembles the results and hallmarks of the original one. First, we briefly describe
the structure of the original model, and discuss the reasons for replication. Then, we demonstrate an
Page | 2

approach we used to be able to generate comparable simulations, and summarize the problems we
faced and what options we had to solve them regarding to Validation. In the last sections, we present
our results and conclude our findings.
Replication of models helps us get a deeper understanding of the relevant properties, identify the
key issues of the analysed phenomena, and to deploy simulation as a research tool successfully.

Overview of the Original Model

The original model considers three different actors, Households, Firms and Banks: Households supply
labour, buy consumption goods and hold deposits, Firms demand labour, produce and sell
consumption goods, give shares to their owner, and Banks receive deposits from households and
extend loans to Firms.
There are three different markets in the model between these participants: the labour market (which
is homogeneous), the consumption market (which is also homogeneous, and contains non-durable
goods) and the credit market. An overview of the model can be seen on Figure 1.

Figure 1. Structure of the model [24]

In the model, the behaviour of the agents is not (necessarily) the outcome of an optimization
process: generally behaviour changes adaptively according to rules of thumb. The markets are fully
decentralized and characterized by a continuous search and matching process.

Households
The jth worker (j = 1, 2, ..., J) supplies inelastically one unit of labour. Once employed, the worker
receives a wage for the entire duration of the labour contracts. The individuals receive wage and
dividends as owner of the firms. This income (wage and dividends) is spent in part on consumption
goods, and savings are deposited at the bank (even involuntary savings are added to voluntary ones).
The propensity to consume is a nonlinear decreasing function of accumulated savings (wealth).
The worker visits a finite number z of firms to ascertain the selling prices, and then it buys first from
the lowest price selling firm. If the most preferred firm is in short supply, the household can resort to

Page | 3

the remaining (z 1) firms. If the jth worker does not succeed in satisfying its consumption plan, it
saves involuntarily.

Firms
The ith firm (i=1, 2, ..., I) produces a consumption good using only labour at the tth (t=1, 2, , T) time
step. Technology is strictly linear (Yit = itLit, where Yit is the level of production, it is the labour
productivity, Lit is the labour). The ith firm (i=1, 2..., I) knows the status quo in the tth time step, Ait =
(Pit, Yit). At the price Pit, given the average price Pt, demand Dit for the products of the firm can be
different from production Yit. Hence the output is the inventories: Sit = Yit - Dit.
Based on the status quo, the firm decides to change the price, and decides to update the expectation
of future demand and therefore the desired scale of activity Yit+1d. The decision on the "desired" scale
of production Yitd, determines the demand for labour Litd = Yitd / itd.

Figure 2. Price and quantity adjustments for the firms [5]

Firms also post labour vacancies if the demand for labour to reach the desired scale of production is
greater than the operating workforce. Moreover, if internal financial resources (net worth: Ait ) are in
short supply with respect to the wage bill witLit - i.e. if there is a financing gap - the firm asks for a
bank loan Bit.
If funds Bits + Ait are still not sufficient to pay for the wage bill (credit rationing), the firm will fire
redundant workers. If the firm ends up with excess supply, it gets rid of the unsold (non-durable)
goods at zero costs.

Banks
The bank will extend a total amount of credit to firms equal to a multiple of its equity base: Ct = Et / v,
where 0 < v < 1 can be interpreted as a capital requirement coefficient. Hence 1/v is the maximum
allowable leverage for the bank. Ct represents the amount of credit vacancies posted by the bank
on the credit market.

Page | 4

We assume that bank k offers to the ith firm a standard debt contract, which consists of an interest
rate rit and the corresponding repayment schedule rit is a markup (over the policy rate r), which
increases with the borrowers leverage lit = Bit/Ait (external finance premium).

Market Processes
In each simulation step, the following events take place in order:
1. At time t, each firm computes net worth Ait. If Ait < 0 the firm goes bankrupt and exits. If Ait >
0 the firm survives and proceeds to the next time step. If all the firms goes bankrupt, the
simulation ends.
2. Starting from the status quo, the firm decides (adaptively) the quantity to be produced
(hence the demand for labour) and the price to be charged.
3. Firms post labour demand approximated by excess.
4. The credit market opens:
a. The bank decides credit conditions, and posts credit vacancies (supply of loans) at a
certain offer interest rate.
b. Each firm contact the banks and decide to whether take a loan or not.
5. Firms redefine labour demand, and determine the effective number of vacancies.
6. The job market opens:
a. Firms publish their vacancies:
i. Firms with excess workforce fire random workers at a certain offer wage.
ii. Firms with insufficient workforce hire and try to fill the empty workspaces
with unemployed workers.
In both cases firms and workers randomized so that no one is privileged.
b. Once the matching has been completed, the firm computes the wage bill and the
financing gap, which has to be filled by means of loans.
7. Market for the consumption goods opens:
a. Workers receive their wages for the current period.
b. Consumers compute their consumption budget.
c. Firms post their offer price.
d. Each consumer contacts a given number of randomly chosen firms to buy
consumption goods, starting from the one which offers the lowest price.
e. Once the matching has been completed, the firm assess whether there is sufficient
demand to sell its goods. If not - i.e. in case of excess supply (inventories) - it gets rid
of the unsold goods (at zero costs) because they are non-durable.
f. Unspent money is added to involuntary savings.
g. Production takes the whole period tth time period.
8. Accounting and surviving:
a. At the end of the period, firms collect revenues and calculate profits.
b. If gross profits are big enough, the firm validates financial commitments.
c. Earnings after interest payments and dividends are retained profits, which are used
to update net worth.
9. The cycle starts again with the next simulation step (t+1).

Page | 5

Bankrupcy
If the ith firm goes bankrupt there is a negative demand spillover due to the loss of employment.
Other firms will experience a reduction of demand. Moreover, the bank will record a non performing
loan, which will affects the banks net worth and therefore, through capital regulation, also the total
amount of credit extended: Ct = Et / v. Credit extended to other firms will be curtailed.

Reasons for Replication

In the introduction of this paper we described that model replication might help us validate the
scientific results of an agent-based model. However, there were some additional issues we
considered.

Efficiency
In our work, we tried to focus our efforts not just on creating a validated replication for the Mark 1
model, but tried to make it as efficient as it is possible on the Java platform. The reason why we
chose Java as the basis of our work is its ease of use and support of various simulation tools. That is
also a reason why we chose Mason, a fast discrete-event multiagent simulation library core in Java,
designed to be the foundation for large custom-purpose Java simulations [] [17] as our simulation
platform, whose main design goal and philosophy focuses on supporting the creation of efficient
computer simulations.

Tool Support: Model Exploration Module

The Model Exploration Module (MEME) is an automated, generic tool (supporting various simulation
platforms like NetLogo, Repast, Fables and Mason) designed to help modellers dealing with
enormous parameter spaces in a reasonable time [13].
One of the approaches to handle such amount of required simulation runs is to exploit a private grid
or distributed cloud computing resources in order to execute large-scale parameter space
explorations or sensitivity analyses [12].
MEME also offers statistical tools to reduce the number of simulation runs needed for collecting
enough information (i.e., samples) from a given parameter space region to judge the behaviour of
the simulation. These statistical methods are based on the Design of Experiments [14] literature that
was developed for the experimental sciences with the goal to reduce the number of costly
experiments without jeopardizing the amount of knowledge that can be collected from the
experiments.
MEME has a growing list of classic designs of experiments, including (fractional) factorial design(s),
the Box-Behnken design, the central composite design and Latin hypercube designs, as well as it
offers a number of more advanced, dynamic exploration strategies, based on heuristic optimization
methods. These latter include Genetic Algorithms and Iterative Interpolation. A good example how
these techniques can be exploited in computer simulations is [11].

Page | 6

Our Chosen Approach

During our work, we heavily used the advantages of different software testing tools. These
frameworks are commonly referred when model replication and validation is an issue to solve [2],
and there are even tools specifically designed to help modellers to write test their simulations [23].
First, we identified different parts of the original model which can be isolated and verified separately
(e.g., initialization of the model and the agents, different utility functions, the various markets, etc.).
Then, we carefully examined the parameters and state variables having influence on the behaviour of
the subsystem (input), the state variables that were changed during the activation (output), and
prepared several black-box unit tests for each component: it the same input is given for the
subsystem, when executing it, then the result must contains the exactly same output.

Ensuring Comparable Results

We had a few issues to solve, and one of them was the different handling of pseudo-random
generators in Java and Matlab. To be able to generate comparable results, we had to synchronize
them.
We exploited the Matlab-Java interoperability here: with some minor restrictions, Matlab is able to
use any Java component. It was a simple solution to use the random generator in the Java core
libraries from Matlab: this way we were even able to determine the exact number of random
function calls at any point in the original model.
JAVA_COMPATIBLE_RANDOM_GENERATOR = 1;
randCallCtr = 0;
javaRandom = java.util.Random(0);
function randVal = jrand()
randCallCtr = randCallCtr + 1;
if JAVA_COMPATIBLE_RANDOM_GENERATOR
randVal = javaRandom.nextDouble;
else
randVal = rand;
end
end
Code Listing 1. Changing the default random generator in Matlab

There was another issue with the randperm(n) function which was heavily used in the model. The
source code of the function is accessible from the Matlab distribution, so it was easy to change the
new random generator:
function randPerm = jrandperm(n)
if JAVA_COMPATIBLE_RANDOM_GENERATOR
dummy = zeros(1,n);
for act = 1:n
dummy(act) = jrand();
end
[junk,randPerm] = sort(dummy);
else
randPerm = randperm(n);
end
end
Code Listing 2. Generating random sequences

Page | 7

Results Comparison
When we finished the replication model, preliminary results showed that it replicates well the
behaviour of the original model. We demonstrate the output of a typical simulation run in Appendix
1: Sample Simulation Run (the parameter vector used for this sample run is shown in Table 1. in the
column entitled Sample). However, a rigorous evaluation is appropriate.
Authors of the original model used two approaches to explore the parameter space because of the
limited computational resources. With the tool support available for the replicated model these
constraints are not entirely removed, but extended considerably.
The model verification procedure consisted of two separate steps, namely internal validity and
sensitivity analysis.
First, a set of runs were executed with exactly the same parameter combination, but with different
seed values. The random seed employed by the pseudo-random generator did not cause large
variability in the aggregated outcome - it was proven that the model is sufficiently accurate.
As a second step, the authors performed a univariate sensitivity analysis. During these experiments,
the model was analysed by changing the value of a specific parameter, leaving the other input values
untouched. For each of these parameters, different scenarios were evaluated.
Authors claim that the parameters having the most significant effect on the simulation results are the
ones related to the duration of labour contracts, to the number of opportunities to locally explore
searches for market transactions, and the total size of the economy.
In our current work, we also concentrate on parameters that prove to be crucial considering the
results. These are the ones influencing the number of trials in the goods market (z), and the number
of households/capitalists in the simulation (I and J). Note that the number of banks did not have any
major effect on the results, so we fixed it on a default value.
Name

Parameter

T
I
J
Bk
div
refi

wb
z
c

Number of time periods

Number of workers
Number of firms/capitalists
Number of banks
Share of dividends
General refinancing rate
Labour productivity
Credit demand contraction
Rate of debt reimbursement
Wage rate
Number of trials in the goods market
Propensity to consume

Value
(Sample)
100
650
100
1*
0.2
0.02
0.1
0.8
0.05
1
2
0.8

Value
(Validation)
1000
650/1300/3250/6500
100/200/500/1000
1*
0.2
0.02
0.1
0.8
0.05
1
2/3/4/5/6
0.8

Table 1. Parameter values used in simulations

During the simulations, we changed the value of parameter z incrementally on the interval [2,6], and
used different agent numbers by multiplying the total number of agents by 1, 2, 3, 5 and 10 (resulting

Page | 8

in a total of 7500 agents in the latter case). Note that the overall ratio of the agents remained at a
constant level. As a total, we executed 40 different simulation runs for the models.

Benchmarking
During the implementation, we paid special attention to efficiency. We tried to use one of the most
efficient modelling libraries available for Java, Mason, and we tried to use the most efficient language
constructs (until it was against readability). Note that this was not a design concept with the original
Matlab model. The execution time of 40 runs is averaged on Figure 3 where each measurement point
shows the average of 5 different runs.

Performance Comparison
execution time (in seconds)

1400
1200
1000
800
600
400
200
0

750 agents

1500 agents

3750 agents

7500 agents

Matlab

39,8406

94,2619

370,6098

1164,0148

Mason

2,4436

8,8554

52,8372

206,8896

Figure 3. Averaged execution time comparison for both the original and replicated model

As a result we got that the Java model is faster by several magnitudes (usually between a factor of 6
to 10) compared to the original model.

Verification
There are several hallmarks of the original model, a number of emergent properties that can be
identified (stylized facts).
First, the economy shows a tendency towards self-organization, i.e. a spontaneous order
("equilibrium") emerges. This order is generally characterized by a persistent involuntary
unemployment and unsold production and/or credit rationing. This order occasionally displays
remarkable and irregular upswings and downswings ("coordination failures") so that a great
depression can occur because of the transmission of an idiosyncratic shock, i.e. in the absence of a
major negative aggregate shock.
In the original model, the authors used 1.000 simulation time steps, and (in order to get rid of
transients), only the last 500 periods have been considered. In our work, we included the initial
phase as well because of two reasons: i) there are heavy oscillations in several statistics during that
period, and ii) allows us evaluate the model on a wider interval. The most relevant results are
summarized on Table 2, and they verify the equivalence of the original and the replicated model.

Page | 9

Table 2. Summarizing the average results of 40 different simulation runs

Page | 10

Conclusion
In our current work, we were able to successfully replicate the results of an agent-based economic
model. We also performed a basic exploration of the model, and got results that are comparable to
the original results.
The original model is roughly about 350 lines of code. On the other hand, our replicated model is
about 2500 lines of code, from which 1500 line is the core model, and an additional 1000 line is
required to use a generic modelling library and a graphical interface. Another interesting fact is that
we created about 6700 lines of test code that ensures completely identical simulation results, which
is more than 4 times longer than the core replicated model.
We also described our simple approach how an existing model might be translated into another
language, and (in the context of Matlab and Java) we documented some issues model replicators
might face.
The benefits we gained by an alternative implementation is at one hand the speedup (the
performance gain is over 5 magnitude), and the window of widely used simulation tools is opened,
allowing us to perform more extensive parameter space explorations by exploiting cloud systems or
by using sophisticated Design of Experiments techniques.

Acknowledgements
This work was partially supported by the Hungarian Government (KMOP-1.1.2-08/1-2008-0002), the
European Union Seventh Framework Programme FP7/2007-2013 under grant agreement CRISIS-ICT2011-288501 (CRISIS Complexity Research Initiative for Systemic InstabilitieS) and mOSAIC 2011256910 (Open-Source API and Platform for Multiple Clouds). These supports are gratefully
acknowledged.

Rererences
[1] Farmer, J. Doyne, and Duncan Foley. The Economy Needs Agent-based Modelling. Nature 460,
no. 7256 (August 5, 2009): 685686.
[2] Gilbert, G N, and K G Troitzsch. Simulation for the Social Scientist. Open University Press, 2005.
[3] Tesfatsion, Leigh. Agent-Based Computational Economics: Growing Economies From the Bottom
Up. Artificial Life 8, no. 1 (2002): 5582.
[4] Tesfatsion, Leigh Agent-Based Computational Economics: Modeling Economies as Complex
Adaptive Systems. Information Sciences 149 (2003): 263269.
[5] Gatti, Domenico Delli, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati.
Macroeconomics from the Bottom-up. 1st ed. Springer, 2011.
[6] Epstein, Joshua M., Robert Axtell, and 2050 Project. Growing Artificial Societies: Social Science
from the Bottom Up. Brookings Institution Press, 1996.
[7] Moss, Scott, Bruce Edmonds and Centre For Policy Modelling. Sociology and Simulation:
Statistical and Qualitative Cross-Validation. American Journal of Sociology 110 (2005):10951131.
[8] Troitzsch, Klaus G. Validating Simulation Models. Proceedings of 18th European Simulation
Multiconference on Networked Simulation and Simulation Networks, SCS Publishing House
(2004): 265270.
Page | 11

[9] Grimm, V. et al. (2006). A standard protocol for describing individual-based and agent-based
models. Ecological Modelling 198(1-2):115-126
[10]Bernhard Bauer, Jrg P. Mller, James Odell. Agent UML: A Formalism for Specifying Multiagent
Interaction. Agent-Oriented Software Engineering, Paolo Ciancarini and Michael Wooldridge
eds., Springer-Verlag, Berlin, pp. 91-103, 2001.
[11]Szab, A., L. Gulys, I. J. Tth (2009) "Sensitivity Analysis of a Tax Evasion Model Applying
Automated Design of Experiments" in Seabra Lopes, L.; Lau, N.; Mariano, P.; Rocha, L.M. (Eds.)
Progress in Artificial Intelligence, pp. 572-583., LNAI 5816., ISBN 978-3-642-04685-8.
[12]Mhr, Tams, Bocsi, Rajmund, Gulys, Lszl: Simulation as a Service: The Model Exploration
Service, 3rd World Congress on Social Simulation, Kassel, Germany, 2010.
[13]Mrton Ivnyi, Lszl Gulys, Rajmund Bocsi, Gbor Szemes, and Rbert Mszros: Model
exporation module, In Agent 2007: Complex Interaction and Social Emergence Conference,
Evanston, IL, USA, November 2007.
[14]George E. P. Box, J. Stuart Hunter, and William G. Hunter: Statistics for Experimenters: Design,
Innovation, and Discovery, 2nd Edition. Wiley-Interscience, July 2005.
[15]Uri Wilensky, and William Rand. Making Models Match: Replicating an Agent-Based Model.
Journal of Artificial Societies and Social Simulation vol. 10, no. no. 4 2 (2007).
http://jasss.soc.surrey.ac.uk/10/4/2.html.
[16]Bigbee, Anthony, Claudio Cioffi-Revilla, and Sean Luke. Replication of Sugarscape Using
MASON. In Agent-Based Approaches in Economic and Social Complex Systems IV, edited by
Takao Terano et al., 3:183190. Agent-Based Social Systems. Springer Japan, 2007.
[17] Luke, Sean, Claudio Cioffi-Revilla, Liviu Panait, Keith Sullivan, and Gabriel Balan. MASON: A
Multiagent Simulation Environment. Simulation 81, no. 7 (July 1, 2005): 517527.
[18]Grimm, Volker, Eloy Revilla, Uta Berger, Florian Jeltsch, Wolf M. Mooij, Steven F. Railsback, HansHermann Thulke, Jacob Weiner, Thorsten Wiegand, and Donald L. DeAngelis. Pattern-Oriented
Modeling of Agent-Based Complex Systems: Lessons from Ecology. Science 310, no. 5750
(November 11, 2005): 987991.
[19]Miodownik, Dan. Cultural Differences and Economic Incentives: An Agent-Based Study of Their
Impact on the Emergence of Regional Autonomy Movements.. October 31, 2006.
http://jasss.soc.surrey.ac.uk/9/4/2.html.
[20]Grimm, Volker, Uta Berger, Finn Bastiansen, Sigrunn Eliassen, Vincent Ginot, Jarl Giske, John
Goss-Custard, Tamara Grand, Simone Heinz, and Geir Huse. A Standard Protocol for Describing
Individual-based and Agent-based Models. Ecological Modelling 198, no. 12 (2006): 115126.
[21]Bauer, Bernhard, Jrg P. Mller, and James Odell. Agent UML: A Formalism for Specifying
Multiagent Interaction. In In: Ciancarini, P.; Wooldridge, M. [eds.], Agent-oriented Software
Engineering, 91103. Springer, 2001.
[22]Miodownik, Dan, Britt Cartrite, and Ravi Bhavnani. Between Replication and Docking: Adaptive
Agents, Political Institutions, and Civic Traditions; Revisited. Journal of Artificial Societies and
Social Simulation 13, no. 3 (2008): 1.
[23]Gurcan, O., O. Dikenelli, and C. Bernon. Towards a Generic Testing Framework for Agent-based
Simulation Models. In Computer Science and Information Systems (FedCSIS), 2011 Federated
Conference On, 635 642, 2011.
[24]Domenico Delli Gatti, personal communications

Page | 12

Appendix
Sample Simulation Run

Page | 13

Page | 14

Page | 15