A Data-Driven SVAR Approach to Model Validaton: an Application to a Macroeconomic Agent-based Model

(1)

Master of Science in Economics

A Data-Driven SVAR Approach to

Model Validation

an Application to a Macroeconomic Agent-Based

Model

Supervisor:

Prof. Alessio MONETA Co-Supervisor: Dr. Andrea VANDIN

Candidate: Damiano DI FRANCESCO

(2)

(3)

List of Figures

2-1 The output structure of an ABM. Source: Fagiolo et al. (2007). . . 18

2-2 Representational scheme of empirical validation . . . 28

3-1 A directed causal graph (Demiralp and Hoover, 2003) . . . 38

3-2 Example of a causal DAG . . . 42

3-3 A directed acyclic graph of Eq. 3.4 . . . 42

3-4 Faithfulness violation (Shimizu, 2014). . . 45

3-5 The five steps of the validation method. . . 50

4-1 Average trend dynamics of the main aggregate variables . . . 61

4-2 Auto- and cross-correlations . . . 63

4-3 RW and AB time series . . . 67

4-4 Example of Markov equivalence class of DAGs . . . 72

4-5 Averaging procedure . . . 76

4-6 Average aggregate time series . . . 77

4-7 Agent-based and real-world contemporaneous causal structures . . . 78

A-1 PC algorithm . . . 87

B-1 VAR-LiNGAM algorithm . . . 88

(6)

List of Tables

4.1 Percentages of non-rejection of statistical equilibrium . . . 68

4.2 Augmented Dickey-Fuller test . . . 69

4.3 Gaussianity tests on VAR residuals for average time series . . . 71

4.4 Similarity measures between RW and AB causal structures. . . 73

4.5 Augmented Dickey-Fuller test for average artificial time series . . . 78

4.6 Gaussianity tests on VAR residuals . . . 78

4.7 Similarity measures between RW and (average) AB causal structures . . . 79

4.8 M & G (2017) similarity measures with penalization . . . 79

4.9 Similarity measures using confidence intervals . . . 81

4.10 Similarity measures between RW and (average) AB causal structures (190 obs) . . . 82

(7)

Abstract

The rise to the top of macroeconomic agent-based models can be easily explained by their ability to endogenously generate business cycles fluctuations and financial crisis as emergent properties of a complex adaptive system populated by heterogeneous bounded-rational agents. However, this flexibility in accounting for out-of-equilibrium dynamics and non-trivial processes makes the empirical validation of policy-oriented macro-ABMs a very challenging operation. Thus, the aim of this work is to (i) first provide an epistemological characterization of ABMs empirical validation and then (ii) to tackle the issue of model validity in a structural manner, by applying the validation method proposed by Guerini and Moneta (2017) to the SFC-AB model by Caiani et al. (2016a). This is done by esti-mating a data-driven structural vector autoregressive (SVAR) model, identified by means of causal search algorithms and Independent Component Analysis, on both real-world and model-generated time series . The artificial and real causal structures so inferred are then compared to assess whether the macroeconomic model under scrutiny is good enough to perform counterfactual and policy analysis.

(8)

Chapter 1 Introduction

"Essentially, all models are wrong, but some are useful.” — George Box (Box and Draper, 1987)

In a relatively recent paper, Christiano et al. (2018) reflect upon the state of health of contemporary macroeconomics and confidently affirm that "DSGE models will remain central to how macroeconomists think about aggregate phenomena", spotting no credible alternative to policy modelling in macro. This future centrality of DSGE models, even in their new guise embedding financial frictions and mild agent heterogeneity, is questioned by economists defending a complexity approach to macroeconomics. Advocates of agent-based modelling in macro (for a comprehensive read on the subject, see Dawid and Delli Gatti, 2018) believe that this novel approach provides more solid foundations to reconstruct a new theoretical and empirical framework in economics (Arthur, 2021; Caiani et al., 2016a; Gallegati and Kirman, 1999; Tesfatsion, 2006). Crucially, whether this ‘new turn’ in macroeconomics is possible is also a methodological issue and concerns (1) the empirical adequacy of the proposed approach and (2) its explanatory potential, i.e. the ability of macro ABMs to effectively capture the causal mechanisms at work in the real world.

The dissatisfaction with the representative-agent framework led many economists to tackle the issues of aggregation and heterogeneity in a non-trivial manner, by investigat-ing the micro-to-macro relations by means of computer simulations.1 _{In contrast to the}

reductionist representative-agent hypothesis typical of mainstream DSGE models, macroe-conomic ABMs establish that there is no isomorphism between the behaviours of the micro-entities and the upper-level aggregate dynamics so that the latter must not be conceived

1_{For an extensive treatment of the fallacy of composition related to the representative-agent framework}

see Gallegati et al. (2006); Kirman (1992); Stoker (1993). Other macroeconometric studies have proved that aggregation of very simple individual behaviour may lead to aggregate complex dynamics (Forni and Lippi, 1997).

(9)

as the sum across the micro-entities but as the emergent outcome of their disequilibrium interactions (Dosi and Roventini, 2019). The idea that some emergent macro-phenomena non-reductively superveneon the underlying micro-structure is largely compatible with the literature on complex systems in economics (e.g. Arthur, 2006, 2021) and with recent de-bates in philosophy of science and economics (Hoover, 2001b, 2015; Machamer et al., 2000; Sawyer, 2004; Simon, 1991). This theoretical setting opens up two major methodological problems for the scientific status of macro ABMs.

Type of explanation. Epstein (2006) calls the type of explanation allowed by ABM ’generative explanation’, meaning that the macro phenomena are obtained (generated) as emergent properties of the specification at the micro-level, i.e. entities behavioural char-acteristics and interactions. In other words the generandum is the explanandum (Grüne-Yanoff, 2009). This bottom-up explanation is well summarized by the famous formula “if you did not grow it, you did not explain it” (Epstein, 2006). However, contrasting Ep-stein’s view, some authors claim that agent-based simulation - and simulation in general - is incapable of providing explanation about the mechanisms at work in reality and as a consequence it is impossible to make any type of (causal) inference about the real world (see, among others, Grüne-Yanoff, 2009).

Empirical validation i.e. whether or not agent-based simulation in fact captures the empirical reality of the emergence. Notably, many macroeconomic ABMs can account for complex micro and macroscopic phenomenology, in the sense that they are able to replicate many stylized facts at different levels of aggregation (see Caiani et al. 2016a; Cincotti et al. 2010; Dosi et al. 2010 for notable examples). At the same time, it is generally held that replication of stylized facts is not enough for assessing the empirical adequacy of ABMs – especially if we compare this approach with more advanced validation techniques in the DSGE literature. This is because stylized facts are statistical regularities that provide no clues about the (causal) structure that has generated them. For this reason, in the last years many AB modellers are striving to move from mere qualitative resemblance (and ad hoc calibration techniques) towards more stringent validation and estimation procedures.

By tackling these two issues, with a special focus on empirical validation, the overall analysis seeks to understand whether agent-based models in macroeconomics can provide an adequate theoretical framework to unveil the underlying mechanisms of macroeconomic aggregate phenomena, thus allowing ABM researchers to perform meaningful structural macroeconomic analysis. From an applied viewpoint, this is done through an application of the causality-based model validation method proposed by Guerini and Moneta (2017)

(10)

to the large-scale macroeconomic model proposed by Caiani et al. (2016a).

As a final remark, it should be emphasized that albeit macroeconomic relationships possess a causal structure themselves - that allows aggregates to be put in causal relationships with other aggregates (as we will do in our applied exercise in Chapter 3) -these macro-causal relationships are not independent of their lower-level constituent parts. Hence, to adequately capture the granular and microscopic origin of many complex macro phenomena (e.g. business cycles, financial instability or inequalities) as well as the feedback effects of socio-economic aggregate dynamics on the behaviour of individuals, a fully-fledged bottom-up explanation of upper-level phenomena grounded on empirical validation at dif-ferent levels of aggregation is needed. This latter methodological step is crucial to coun-teract reductionist explanatory accounts in economics and to support instead what has been called a macrofoundation of the micro (Colander, 1993; Dosi and Roventini, 2017; Gallegati and Kirman, 2019).

The rest of the work is structured as follows. Chapter 2 provides a general overview of agent-based models in macroeconomics and briefly presents the relevant contributions with regard to ABM empirical validation. The last part of this chapter aims instead at investigating the concept of empirical validation in agent-based modelling from an epis-temological viewpoint, by reviewing the existing methodologies under a causal inference perspective. Chapter 3 introduces the methodology used. In particular, the issue of SVAR identification is presented and two data-driven approaches to the problem are discussed. The first one is grounded on graph-theoretic approaches to causation whereas the second is based on Independent Component Analysis. Starting from these two data-driven meth-ods, the validation method by Guerini and Moneta (2017) is presented. Chapter 4 presents an application of this validation approach to a well-known macroeconomic computational model: the Agent Based-Stock Flow Consistent macro model by Caiani et al. (2016a). In the same section, the preliminary findings of the analysis are presented and some robustness checks are finally performed. To conlcude, Section 5 presents a summary of the results and outline the major limitations as well as further developments of the proposed approach.

(11)

Chapter 2 Macro-ABMs and Empirical Validation

”[...] economists are permanently indifferent to matters bearing on whether "the assumptions are sufficiently close to the truth to make the conclusions approximately true", and this means that their peculiar use of models ... remains as much a mystery as ever. ”

—Alexander Rosenberg, (Rosenberg, 1978)

2.1 ABMs in macroeconomics

Agent based Computational Economics (ACE) is a growing modelling approach in eco-nomics and social sciences in general. In ecoeco-nomics, ABMs have been used in a number of different fields such as finance, industrial dynamics and only recently in macroeconomics. Their increasing role in macroeconomic research is largely due to the the need "to move beyond DSGE models" with the purpose of building "a more empirically based macroe-conomics" (Colander et al., 2008), less concerned with analytical tractability and more committed to a realistic representation of reality. Indeed, over the last decade it has been widely certified that traditional Dynamic Stochastic General Equilibrium models (DSGE), based on the restrictive (to say the least) representative-agent assumption, failed to spot the growing macroeconomic imbalances culminated in the 2007-08 financial crises, thus paving the way for new forms of macroeconomic theorizing. Not only were DSGEs inca-pable of predicting the crises - something that, at least to a certain extent, cannot be really demanded to economic models - but they also structurally ignored the possibility of its occurrence (Stiglitz, 2018).1 _{Admittedly, it must be noted that "not all DSGEs are created}

equal" (Ghironi, 2017) and that there is new stream of research that definitely deserves

1

On the inadequacy of DSGE models and the need for agent-based models in macro see, among others, Farmer and Foley (2009), Colander et al. (2008), Dosi and Roventini (2019), Fagiolo and Roventini (2016) and Arthur (2021).

(12)

much more attention: from DSGE models embedding a large set of financial frictions and distortionary wedges (though sometimes accused of ad hocery) to the so-called Heteroge-neous Agent New Keynesian (HANK) approach, in which bounded-rationality and mild forms of heterogeneity are introduced (Kaplan and Violante, 2018).

However, in light of the consideration that the DSGE paradigm is hopelessly flawed and that "any meaningful model of the macro economy must analyze not only the characteris-tics of the individuals but also the structure of their interactions" (Colander et al., 2008), ABM modelers attempt to provide a fully-fledged alternative framework since the early 2000s (Dosi and Roventini 2019 trace back the prodromes of ACE modelling to the evolu-tionary tradition of Nelson and Winter 1982 and the works of Herbert Simon). The general modelling philosophy of the complexity approach to economics is that the economy is a complex evolving system populated by heterogenous bounded-rational agents (e.g. firms, workers, banks) whose far-from equilibrium local interactions lead to some collective order, even if the micro- and meso-structure of the system continuously change (Dosi and Roven-tini, 2019). Interestingly enough, this theoretical framework opens up new methodological possibilities for policy modelling in macroeconomics. The main concern of the present work is exactly to conceptualize the role of ABMs in macro - by stressing their stance as alter-native models with respect to the orthodox DSGE paradigm - and especially to tackle the issue of their empirical validation from both a methodological and applied perspective.

In what follows the main building blocks of the ABM approach to macroeconomic analysis are described and the most important works on macro agent-based modelling are briefly presented. Then, in the next sections of this chapter the concept of empirical validation will be extensively analyzed.

Building blocks of macro-ABMs

Building on Delli Gatti et al. (2018) and Fagiolo and Roventini (2016) the ABM framework can be succinctly described by the following ten characteristics2_:

1. A bottom-up perspective. Instead of directly modelling the dynamics of macro vari-ables, AB modelers typically start by specifying the behaviour of heterogeneous agents at the micro level (from consumers to banks), whose non-trivial interactions generate an upper-level dynamics for the aggregate variables. The latter are typically determined by means of summation or averaging across the population of agents and their properties emerge from the micro-dynamics of lower-levels entities (Delli Gatti

2_{One of the most recent and complete overviews on the complexity approach to economics can be found}

(13)

et al., 2018). This approach explicitly contrasts with the top-down nature of neo-calisscal models whose aggregate consistency is dictated by equilibrium conditions (Kaplan and Violante, 2018).

2. Heterogeneity. In contrast with the representative-agent assumptions typical of (old-fashioned) mainstream macro models, ABMs economies are populated by heteroge-neous agents that can differ in multiple aspects, from initial endowments and location to behavioural rules, abilities, rationality, and computational skills.

3. The evolving complex system (ECS) approach. Agent-based models stem from what has been called the ’new science of complexity’, an interdisciplinary approach born in the late 80s at the Santa Fe Institute (see, e.g. Arthur, 2006, 2021). In this view, the economy is seen as a complex system where agents continuously evolve through time and adapt to changing circumstances and whose decentralized interactions generate significant emergent macro-patterns that qualitatively differ from the individual en-tities. This aspect has important methodological and epistemological implications. In particular, the complexity approach implies that agent-based models adopt an emergentist account of explanation according to which the macro behaviour can-not be straightforwardly deduced from the behaviour of the lower-level multitude of agents, much less from a representative individual (Forni and Lippi, 1997; Kirman, 1992; Machamer et al., 2000). Moreover, the complexity approach leads to another important characteristic of agent-based models, namely their out of equilibrium dy-namics: equilibrium is not imposed as a state of the system but rather it is defined in statistical terms as the situation in which aggregate variables (or the distributions of their statistics) are somewhat stable.

4. Non-linearity. Locally dispersed interactions follow a non-linear dynamics and show feedback loops between micro and macro entities. This means that the micro-to macro transition is a non-trivial, non-additive, complex aggregative process in which even small-scale shocks can produce significant modifications in the macro-structure.3

5. Direct (endogenous) interactions. Agents interactions in ABMs are not mediated by a centralized mechanism that transmits the effects of other agents’ behavior (like the neoclassical Walrasian auctioneer) but instead interact directly in a fully-decentralized market through search and matching protocols and their choices are

3_{Interestingly enough, the macroeconomic consequences of microeconomic shocks is an hot issue also in}

(14)

directly affected - e.g. via expectations - by other agents’ decisions. The endogeneity of interactions stems from the fact that the artificial economy, lacking any coordinat-ing top-down market clearcoordinat-ing mechanism, evolves through time in a self-organizcoordinat-ing order generating emergent macro-dynamics (e.g. business cycles or persistent un-employment). Some examples of endogenous interactions are provide by Delli Gatti et al. (p. 119-124 2018).

6. Bounded rationality. The idea to replace the ’olympic’ hyper-rationality of neoclassi-cal agents with a more realistic form of cognitively limited rationality is of the Nobel prize Herbert Simon who claimed that: "the task is to replace the global rationality of economic man with the kind of rational behavior that is compatible with the access to information and the computational capacities that are actually possessed by or-ganisms, including man, in the kinds of environments in which such organisms exist" (p. 99 Simon, 1955). Thus, agents in ABMs typically behave according to (possibly) empirically based heuristics, myopic optimisation rules and adaptive expectations. Interestingly enough, a lot of today’s neoclassical contributions build upon the con-cept of bounded-rationality (see, among others, Farhi and Werning, 2019).

7. The nature of learning. As noted by Fagiolo and Roventini (2016), neoclassical learn-ing takes place in an equilibrium framework in which the focus is on the analysis of inter-temporal optimal coordination and, in those cases in which asymmetric in-formation is introduced, on the ways of dealing with multiple equilibria. On the contrary, ABMs get rid of the hyper-rationality and perfect information assumptions and model cognitively-limited agents that are not able to fully capture the rationale of their environment, mastering only a subset of the total information that limit their decisional capabilities. Moreover, the nature of learning is affected by the persistent introduction of novelty that forces agents to be engaged in an "open-ended search of dynamically changing environments" (Fagiolo and Roventini, 2016)

8. Endogenous and persistent novelty. This feature of agent-based models is rooted in the evolutionary tradition on which it is grounded. In this framework, the introduc-tion of radical innovaintroduc-tion completely alters in a non-staintroduc-tionary manner the model structure and the environment in which agents interact. A key consequence of this fact is that uncertainty is not ontologically reduced to probabilistic risk but is rather ’Knightian’ in nature, meaning that agents cannot quantitatively measure the plau-sibility of occurrence of a given phenomenon. This together with bounded-rationality

(15)

and adaptive expectations leads to non-trivial aggregation and dynamics.

9. ‘True’ dynamics. As a consequence of adaptive expectations and the introduction of persistent novelty ABMs often display what is called a ’true’ dynamics, i.e. an evo-lutionary dynamic process marked by a non-reversible and strongly path-dependent character.

10. Selection-based market mechanisms. This is another particular feature of the evolu-tionary tradition and is linked to the idea that "agents are typically selected against" (Fagiolo and Roventini, 2016). In more detail, the idea is that there are several mech-anisms that act at different temporal level (ex ante and ex post) and along different dimensions (demand-side and market based mechanisms but also science-based and supply-side mechanisms) that progressively increase the degree of determinatess of agents and technology selection (Dosi, 1982). This leads to turbulent patterns in the landscape of the technologies to be adopted and in the agents’ entry and exit dynamics.

The richness of agents’ characteristics and the complexity of the environment in which they are bound to leave are able to generate interesting macro-patterns that allowed the complexity approach to be extended to macroeconomic analysis.

Overview of the macro-ABM literature

The need for an all-encompassing complexity-based macroeconomic framework led many economists to develop small, medium and large-scale macro models of artificial economies that, in one way or another, embed the features listed in the previous section. So far, the most complete systematization of the macro-ABM literature is the one provided by Dawid and Delli Gatti (2018). In the latter contribution, the main features of macroeconomic agent-based models are carefully studied and the main challenges for this new approach are outlined. Moreover, the authors provide a systematic comparison of seven macro-ABM archetypes along several different dimensions. For the sake of completeness, in what follows we briefly recall the most important macro-ABMs. For a detailed introduction to the functioning of macro-ABMs, their history and their structural characteristics, please refer to Dawid and Delli Gatti (2018).

Dawid and Delli Gatti (2018) distinguish between seven broad families of macroeconomic agent-based models:

(16)

2. the family of models proposed by Delli Gatti, Gallegati and co-authors in Ancona and Milan, labelled Complex Adaptive Trivial Systems (CATS). This group of mod-els builds on the pioneering work of Hyman Minsky and is mainly (but not only) concerned with the interaction between business cycles and financial factors. Just to make an example, the Caiani et al. (2016a) model presented in the last chapter of this work can be seen as extension of this family of ABMs;

3. the framework developed by Herbert Dawid and co-authors in Bielefeld, known as EUBI (Dawid et al., 2016);

4. the EURACE framework maintained by Cincotti and co-authors in Genoa (EUGE) (Cincotti et al., 2010);

5. The Java Agent based Macroeconomic Laboratory developed by Salle and Seppecher (JAMEL) that associates Keynesian thinking and an agent-based approach (Seppecher, 2012);

6. the LAGOM model developed by Jager and co-authors (Wolf et al., 2013);

7. the family of models developed by Dosi, Fagiolo, Roventini and co-authors in Pisa, known as the “Keynes meeting Schumpeter” framework, that bridge the Schumpete-rian theory of innovation with the Keynesian theory of aggregate demand (Dosi et al., 2015, 2010);

Most of these models have been extensively applied to actual policy issues, from general fiscal and monetary policies experiments to more specific settings such as macroprudential policy, European cohesion policies and policies tackling wealth and income inequality. However, the legitimacy of the agent-based approach to macroeconomics cannot be as-sessed only on theoretical and modelling considerations. On this ground, macro-ABMs already passed the test having proved to be capable of effectively tackling the fallacies of mainstream macro-models. Arguably, whether agent based models can be fruitfully ap-plied to structural macroeconomic policy is also a methodological issue and concerns the way these models are taken to the data. The process of taking the model to the data is commonly labelled as ’empirical validation’ and is by far one of the hottest topics in the macro agent-based models research program.

(17)

2.2 Empirical validation in macro agent-based modelling

Empirical validation is a challenging but fundamental field of research given the expanding role that agent based macro-models are assuming in policy making (Haldane and Turrell, 2019).4 _{The rise to the top of macro-ABMs can be easily explained by their ability to}

en-dogenously generate business cycle fluctuations and financial crisis as emergent properties of a complex adaptive system populated by heterogenous bounded-rational agents. How-ever, this flexibility in accounting for out-of-equilibrium dynamics and non-trivial processes comes at a cost: non-linearities, multiple interactions and emergent macro outcomes make the empirical validation of macro-ABMs a very challenging operation.5

To better clarify what we mean with empirical validation, imagine that the phenomena we want to study originate from a (causal) data generating process, labelled real-world DGP (rwDGP), that can be seen as an unknown "complicated, multiparameter, stochastic process that governs the generation of a unique realization of some time series and stylized facts that we can empirically observe" (Fagiolo et al., 2019). An ABM is meant to provide a simplified data generating process (that we call mDGP) that is able to capture the salient features of the rwDGP. In this respect, with empirical validation of a model we typically refer to the process by which the model is taken to the empirical data with the aim to assess to what extent the mDGP provides a meaningful approximation of the rwDGP or, in other words, how much the model-generated data embed a similar statistical and causal structure with respect to the real-world ones.

Most notably, ABMs are able to generate artificial data at both micro and macro level and can thus be validated at different stages of aggregation. More formally, once the in-puts of the model (initial conditions and parameters) have been properly specified, the microlevel output of an ABM is composed of Monte Carlo simulations panel datasets con-taining different micro variables for a set I of agents over a specified time window T :6

𝑍𝑚,𝑘 ∈ R𝐾×𝑀 𝐶, 𝑍𝑚,𝑘 = {𝑧𝑚,𝑘,𝑖,𝑡: 𝑖 = 1, . . . , 𝐼; 𝑡 = 𝑡0, . . . , 𝑇 }, ∀𝑘 ∈ 𝐾 (2.1)

where m denotes the specific Monte Carlo simulation, k indicates the micro-variable of interest (e.g. household consumption level or firms’ profits), i is the agent cross-section dimension and t indexes the time dimension. The macrolevel output is instead obtained by

4

In fact, as noted by Dawid and Delli Gatti (2018), if macroeconomic ABMs were only little explored before 2006, after the great financial crisis and the consequent failure of traditional DSGE models to account for its occurring they have become increasingly important in the economic community.

5_{This sections extensively draws on Fagiolo et al. (2007) and Fagiolo et al. (2019).} 6

(18)

aggregating or averaging out the micro-time series variables across the agent cross-section dimension:

¯

𝑍𝑚,ℎ ∈ R𝐻×𝑀 𝐶, ¯𝑍𝑚,ℎ = {¯𝑧𝑚,ℎ,𝑡: 𝑡 = 𝑡0, . . . , 𝑇 }, ∀ℎ ∈ 𝐻 (2.2)

where h is the aggregate variable observed at different time steps t. Just to make an example, aggregate consumption is obtained by summing levels of consumption across households. The typical micro-to-macro structure of a macro-ABM is depicted in Fig. 2-1. Crucially, empirical validation can take place at both input and output level and, in each of the two cases, at both micro and macro level.7_{More rigorously, taking as benchmark}

Fig. 2-1, input validation concerns the setting of initial conditions 𝑥𝑖,0, micro parameters 𝜃𝑖

and macro parameters Θ. Once the model has been parametrized, a set of micro and macro time series are generated through computer simulations until the model reaches a somewhat stable configuration. The information contained in this stable configuration is stored in a set of statistics 𝑆 = {𝑠1, . . . , 𝑠2}, whose values vary across different Monte Carlo realizations.

To properly take into account the stochastic nature of the model, a sufficiently high number of independent MC simulations is performed so that for each statistics 𝑠 of interest we are able to estimate the corresponding Monte Carlo distribution, from which moments can be computed. Output validation refers to the comparison between the statistical properties of the model-generated micro and macro time series (or their longitudinal moments) with those of the unique observable set of real-world time series. Arguably, this comparison can be made along different levels of aggregation and especially along different dimensions, from mere statistical resemblance to more rigorous structural similarity.

In the remainder of the chapter, the concept of empirical validation and the related literature will be analyzed from a methodological/epistemological perspective.

2.3 Macroeconomic ABMs Validation: a methodological

as-sessment

Overall this section attempts to critically assess macroeconomic ABMs empirical valida-tion in light of the methodological challenges that this new modelling approach faces, especially for what concerns causal inference and explanation. Curiously, there is little

dis-7

At this stage, one important point must be clarified. In what follows we are mainly concerned with the empirical validity of a model. However, as noted by Fagiolo et al. (2019) also theoretical validity (i.e. the validity of the theory relative to the simulation) and model validity (i.e. the validity of the model relative to the theory) must be taken into account. We must also further discriminate between model validation, i.e. the ability of the model to capture real-world properties and model verification (or program validity), i.e. whether "the model does what it is supposed to be doing" from a simulation perspective (Gräbner, 2018).

(19)

Fig. 2-1. The output structure of an ABM. Source: Fagiolo et al. (2007).

cussion among economists on the explanatory role of ABMs as a tool for causal discovery.8

Therefore, our main concern is about the type of (causal) explanation supported by ABMs in macroeconomics and how the new approaches to empirical validation that are prolifer-ating in the literature tackle this issue.

In general, empirical validation methods attempt to respond to the following broad question: How good is your model? In this sense, with empirical validation we generally define the process by which a particular model is taken to the data ‘meaningfully’. But what does ‘meaningfully’ mean? What kind of relation exists between models and data in the validation process? Of course, it will depend on the purpose of the validation proce-dure. For example, is the mere replication of stylized facts a satisfactory result for model validation? Should we instead go beyond mere replication and get into the often neglected world of (explicit) causal analysis? If so, what account of causality should we adopt in the ABM context?

In order to answer to such questions it is useful to recall the main differences between calibration, estimation and model validation. In general, there are two different conceptions of the ‘estimation vs calibration’ dichotomy and the way in which the term ‘validation’ is related to it (Delli Gatti et al. 2018). In Kydland and Prescott’s view, for instance, calibration aims at tuning the model’s parameters with the aim of fitting the real data ‘as close as possible’ while estimation in conceived as the quantitative determination of the

8_{In other social sciences where ABM are used, especially in analytical sociology, there is a more intense}

(20)

size of the parameter values. This latter step, methodologically speaking, involves a major “inferential effort to learn about the underlying values of the parameters” (Grazzini et al. 2017). On the other hand, Hansen and Heckman refuse the ‘division-of-labour’ conception and see calibration and estimation as “two approaches both aiming at solving the same class of problems” (Fagiolo et al. 2019).9 _{To sum up, even if the aim is the same – i.e.}

reducing the uncertainty surrounding the model’s outcome – calibration and estimation contribute to this aim in different degrees:

“calibration aims at maximising the fitness of the model with the observed data in a distance metric arbitrarily chosen by the modeller, without bothering about the ‘true’ value of the parameters of the real world data generating process, or the un-certainty surrounding them; estimation aims at learning about the ‘true’ value of the parameters of the rwDGP by evaluating the fitness of the model with the observed data in a carefully chosen distance metric, such that the estimator has well known (at least asymptotically) properties [. . . ] estimation is nothing else than more conscious calibration”. (Delli Gatti et al. 2018).

This clarification is necessary to correctly frame the relation between empirical vali-dation, calibration and estimation. In this light Indirect Calibration, the Werker-Brenner approach and the History-friendly approach are seen as calibration methods since they aim at making the model qualitatively similar to the target system of interest (they are also referred to as qualitative output validation techniques).10 _{On the contrary, Simulated}

Minimum Distance methods (SMD) (Grazzini and Richiardi, 2015),11 _{and Bayesian}

ap-proaches (Grazzini et al., 2017) are considered estimation methods since they try to infer the quantitative values of the model’s parameters with well-behaved – in terms of e.g. asymptotic properties – estimators. Interestingly enough, there is a growing number of contributions that cannot be clearly defined along the previous dimensions either because the mathematical properties of the resulting estimators have yet to be fully explored or because they tackle the validation issue from a completely different perspective. Some ex-amples are: Lamperti (2018a) and Lamperti (2018b) in which the goodness of the model is evaluated by comparing the similarity between the dynamics of model-generated time series with that of observed time series, using an information criterion called GSL-div; Barde and Van Der Hoog (2017) and Barde (2020) in which the authors apply univariate

9_{In his recent review of calibration methods in ABMs, for instance, Platt uses the two terms}

inter-changeably (Platt, 2020).

10_{See Delli Gatti et al. (2018, pp. 166-172) and Fagiolo et al. (2007) for an extensive treatment.} 11

SMD methods are the most common ones and embrace Simulated Method of Moments (SMM), Indirect Inference (II) and to some extent Simulated Maximum Likelihood (SML). See Delli Gatti et al. (2018) for an extensive treatment.

(21)

and multivariate versions of the so-called Markov Information Criterion (MIC) to score the simulated data against empirical data; Salle and Yıldızoğlu (2014) where a computational technique called Kriging Meta-Modeling is used to approximate the original ABM in order to study the effects of inputs on the model’s output, thus significantly reducing the com-putational burden typical of sensitivity analysis (see Bargigli et al., 2020; Dosi et al., 2017, for recent applications); Lamperti et al. (2018) in which traditional model calibration is coupled with supervised machine-learning and intelligent sampling to build a meta-model for extensive parameter space exploration; Guerini and Moneta (2017), in which the causal structures of the artificial and observed time series are compared via SVAR estimation and causality-based similarity measures (more on this method in the last section of Chapter 2).

Given the difficulty to frame this "new wave of validation approaches" (Fagiolo et al., 2019) into the usual ’calibration vs estimation’ framework, Grazzini et al. (2017) refer to them as ‘advanced calibration’ techniques (see also Platt 2020 on this point). Actually, Fa-giolo et al. (2019) provide a more refined systematization of the previous works, considering them as different ways of evaluating an agent-based model along different but complemen-tary dimensions: matching of causation (Guerini and Moneta, 2017), replication of time series dynamics (Barde, 2020; Lamperti, 2018b), meta-modelling (Bargigli et al., 2020) and parameter space exploration (Lamperti et al., 2018). In this work we explicitly endorse this latter ‘ecumenical’ perspective on model validation by suggesting a multi-dimensional def-inition of empirical validation. In particular with empirical validation we mean the process by which (i) the model is taken to the data via calibration and estimation techniques and (ii) the model performance is evaluated along a broad range of dimension, from replica-tion of univariate time series dynamics to matching of real-world causal structure. The ’calibration and estimation’ phase along with the ’evaluation’ step exhaust the concept of empirical validation.

Empirical validation through the lens of causal inference

In what follows we try to integrate our definition of validation with the methodological analysis made by Casini and Manzo (2016) in which agent based models are related the so-called ’generative account of explanation’. In summary, we will argue that the key role of empirical validation is to move from ‘just-so-stories’ explorative models whose validity is evaluated in terms of the interesting (possibly counter-intuitive and realistic) generated phenomena to explanatory (possibly causal) models, whose validity is evaluated in terms

(22)

of a severe confrontation with data. In particular, starting from the definition given in the previous section we aim to integrate the classification of ABMs with respect to their degree of empirical validity made by Fagiolo et al. (2019) with an epistemological analysis of agent based modelling with respect to causality, elaborated by Casini and Manzo (2016). We believe that merging these two visions can be particularly useful to build a bridge between the empirical validity of a model and its causal power. Four levels are considered for assessing the empirical validity of an ABM:

• Level 0: the model is a caricature of reality, as established through the use of simple graphical devices

• Level 1: the model is in qualitative agreement with empirical macro and micro-structures, as established by plotting, e.g., the distributional properties of agent population.

• Level 2: the model produces quantitative agreement with empirical macro-structures, as established through on-board statistical estimation routines.

• Level 3: the model exhibits quantitative agreement with empirical micro-structures, as determined from cross-sectional and longitudinal analysis of the agent population. Note that this is a progressive classification i.e. the higher the level the lower the uncer-tainty of the model’s outcome. In particular, the level proportionally increases with the quantitative content of the validation method used. However, no attempt is made by Fa-giolo et al. (2019) to interpret this classification in a causal fashion. Hence, we suggest that this scheme can be easily complemented with the theoretical framework provided by Casini and Manzo (2016), who make the first systematic treatment of the relation between causality, mechanisms, and ABMs. In their analysis, ABMs are seen as inferential devices that “implement one specific understanding of mechanism, thereby supporting a specific view of causality”. This consideration is based on the belief that there is a distinction between causal inference based on observations (as in the traditional potential outcome and regression approaches) and causal inference based on simulation modelling. The first is labelled as “dependence” or “difference making” account of causality while the latter as “production account of causality”. In particular, in their production perspective of causal-ity, ABMs are seen to provide a “credible narrative” of how a set of causes (possibly at the micro-level) brings about a set of effects (the macro emergent outcomes) through a multi-level aggregative process. This ‘credible narrative’ is seen as a formal demonstration of the existence in the real world of a similar underlying process that cannot be directly grasped

(23)

with traditional inferential techniques that act at the observational level. On this ground macro-ABMs can be seen as the right tools for providing a generative and counterfactual account of causality. Moreover, to effectively solve the empirical problem of discovering real-world causal relations, this philosophical account must be integrated with two other phases. The first is empirical validation along the classification made before. In this context, the injection of empirical information into macro-ABMs enables us to establish that the causal mechanisms triggered by the model are actually at work also in the real-world (ex-ternal validity). The second issue is linked with the counterfactual dimension and regards the internal functioning of the model. Indeed, once the model has been properly validated the effects of a given alteration of the parameter set (and by this way of the mechanism at hand) on aggregate dynamics of the model can be interpreted in a counterfactual sense. ABM methodology naturally performs these exercises through sensitivity and robustness analysis (as argued by Casini and Manzo 2016 and Fagiolo et al. 2007). To better clarify this account of agent-based models, let us illustrate two well-known empirical validation methods recently appeared in the macroeconomic ABM literature. Our ultimate aim is to assess to what extent these methods can make the validated macro models reliable formal devices for inferring causality from data.

Critical evaluation of two validation methods: macro-ABMs as tools for gener-ative causal explanation

1. Bayesian Approach. The first work we aim to investigate is Delli Gatti and Grazzini (2020), whose methodology is likely to change the way medium-sized macro ABM are ‘taken to the data’, as it can be detected by the title “Rising to the Challenge: Bayesian Estimation and Forecasting Techniques for Macroeconomic Agent-Based Models”. Indeed, the contribution of this paper is twofold. First, it is one of the first attempts to estimate a model of that dimension (in particular the CATS model with capital goods) using a Bayesian approach.12 _{Second, this work is also the first example, as claimed by the}

au-thors, in which an estimated macro-AB model is used for forecasting economic aggregate time series, using a VAR model as a benchmark.13_{Without going into details, the authors}

12

Interestingly enough, in his recent review Platt (2020), who compares different ABM estimation and calibration techniques, applies this method to a large scale AB model (the INET Oxford Housing Market Model ) and finds that the Bayesian estimation adopted by Grazzini et al. (2017) generally out-performs alternative procedures such as Simulated Minimum Distance. Moreover, Platt (2020) notes that most calibration and estimation methods are applied to rather simple models such us the Brock and Hommes’ one and calibration of larger model is still rare.

13

In this respect, it is worth stressing two issues. First, given the encouraging results in terms of forecast-ing capability of the AB model considered, though still underperformforecast-ing VAR models, forecastforecast-ing exercises can represent a novel dimension along which the empirical validity of a model is evaluated. Second, the

(24)

propose an augmented Bayesian estimation procedure with respect to that used by Grazz-ini et al. (2017), in which the parameters of a macro ABM are estimated by explicitly introducing an appropriate time structure to the simulated data. Their aim is to find the set of numerical values of the model’s parameters that make the output of the model (the simulated aggregate time series) ‘as close as possible’ – according to some distance metrics – to the observed time series. To evaluate the reliability of the procedure the authors imple-ment a "pseudo-estimation exercise”. In particular they compare the simulated time series with the ones obtained from U.S. data for the period 1948–2018. They find that, under the assumption that the model is well-specified, the estimation procedure using Bayesian method is very accurate, since the 72.5% of the estimates correctly identify the parameters.

2. Matching of causation approach. The second contribution that we intend to analyse is the work by Guerini and Moneta (2017), in which a ‘validation as matching of causation’ approach is adopted. This novel method significantly differs from traditional estimation techniques used in ABM since, as far as we know, this is the only empirical validation procedure in which causality is the relevant dimension along which the explanatory power of an ABM is judged. Given that this will be the method adopted in the validation exercise conducted in the present work, a detailed explanation of the procedure and the relative results can be found in the last section of Chapter 3 and in Chapter 4.

Critical review of the two approaches. An interesting exercise is to review the relative strengths and weaknesses of the proposed validation methods in light of their ability to make macro-ABMs efficient tools for providing causal claims based on a generative account of explanation. In other words we critically assess to what extent the proposed methods ensure that the model results “reflect persistent aspects of real-world systems under study rather than an overfitting of model parameters to temporary aspects of these systems” (Fagiolo et al., 2007). Starting from the work by Delli Gatti and Grazzini (2020), two particular aspects of their proposed methodology are important. First, as mentioned be-fore, the authors explicitly take into account the time structure of the simulated data to improve the comparison with real-world ones. Arguably, this refinement increases the ‘de-gree of realism’ of the model under inquiry. Second, the validity of the model is assessed from a usually unexplored dimension i.e. the ability to forecast time series dynamics. This

forecasting exercise is used to recover the probability of a recession. In this way, ABMs can be used as kinds of ‘thermometers’ for future crisis or in the author’s words as “early warning indicators” (Delli Gatti and Grazzini, 2020).

(25)

‘predictive output validation’ through out-of-sample forecasting exercises (see Tesfatsion, 2017, for details) turns out to be very important for the ability of the model of producing causal relevant information. In fact, if the model is able to match some future statistical properties of the data it means that the model’s structure is robust: it can account for something more than temporary features of the studied system. Even if they set up only a benchmark forecasting procedure - on which future research can build upon - the results on this regard seem very promising. Notably, this exercise will also please all those instrumen-talists à la Friedman who claim that what counts for a model is not its realism (something that cannot be really tested, they argue) but its capability to successfully predict what might happen.

However, the goodness of the method is assessed only taking into account its ability to replicate the statistical properties observed in real data. The authors claim that “in general the statistical properties of observed time series are matched by those of the artificial time series generated by simulations” and this is considered a satisfactory result. Arguably, this is not enough for an ABM to be treated as an efficient causal device. Indeed, as noted by Casini and Manzo (2016) statistical inference is different from causal inference: the former aims at estimating “the value of a set of parameters with its associated uncertainty, from a limited set of observations” while the latter involves a deeper understanding of the un-derlying mechanism and aims at “establishing the existence of a non-spurious connection between two properties of the world”. Even if the two are connected they should not be confused. The point is that under a statistical regularity there could be multiple differ-ent causal structures that may have generated it (this is known in the literature as the under-determination problem). In other words, the statistical micro or macro regularities that the model is able to quantitatively match are only the prima facie manifestation of an underlying causal data generating process. If recovering the latter in all its details is of course impossible (we would end up building a one-to-one mapping to the real world) it is still possible to augment statistical knowledge with a more causal-centric one. This latter step is crucial to shed some lights on the ‘epistemic opacity’ of ABMs and could further refine our knowledge about the model similarity with the real world (Casini and Manzo, 2016). This further step is explicitly addressed by Guerini and Moneta (2017). In the latter work, the authors complain with the fact that when output validation enters the stage, macro-ABMs are often evaluated according to their ex-post ability to match a number of macro regularities. However, paraphrasing Tobin’s famous characterization of the Philips Curve, stylized facts are statistical regularities in search for a theory (i.e. a set

(26)

of causal explanations) like Pirandello’s characters are in search for an author. Certainly, as noted by Hirschman (2016), identifying an empirical regularity is an important step per se because it means "to have identified parts of the world that can be analytically sep-arated and whose movements can be usefully observed". Nevertheless, what Guerini and Moneta (2017) claim is that without an explicit causal interpretation of those facts, struc-tural macroeconomic policy analysis cannot be properly performed through macro-ABMs. Methodologically speaking, the rationale behind this validation method can be found in the need to move from ‘dynamic sufficiency’ of a model, i.e. its capacity to reproduce patterns observed in the target, to ‘mechanistic adequacy’, i.e. its ability to capture the (causal) mechanistic structure of the target (Gräbner, 2018). The attitude of moving beyond the mere “phenomenological analogy” of simulated and real outcomes is further supported by the chosen identification procedure of the SVAR, that is based on causal search algorithms. These algorithms have been explicitly created to augment the statistical evidence obtained at the observational level with formal causal knowledge. So, even if both methods (the one proposed by Delli Gatti and Grazzini 2020 and the one by Guerini and Moneta 2017) allow the validated ABMs to pass from level 1 to level 2 of our classification, the SVAR-method approach by Guerini and Moneta (2017) that focuses on the dynamic causal structures of the variables of interest has a more direct link with causal inference. This aspect is particu-larly important to appreciate the need to complement the ’empirical validity’ classification with causality-based methodological considerations.

Moreover, the two analyzed methods should not be seen as competing and it would be profitable to shift the debate from defending the superiority of particular methods to discussing how different empirical validation procedures “can provide a variety of eviden-tial sources that compensate for each other’s weaknesses” (Casini and Manzo 2016). So we tend to agree that it does not exist a ’self-sufficient causal machinery’ and that, if we believe in evidential pluralism„ the right way to properly deal with causal inference is by “methodological synergy” (Casini and Manzo 2016). This position is also supported by Guerini and Moneta (2017) who maintain that “other researchers instead [. . . ] have focused on estimation, or on the analysis of the emergent properties stemming from ABMs [. . . ] The estimation approach to the confrontation of model to data is an interesting aspect and should be considered as complementary to the validation approach”.

Two last points are worth mentioning. First, all the previous issues are fundamental for policy and counterfactual analysis.14 _{Indeed, once the model has been properly}

vali-14

(27)

dated (e.g. the parameters are carefully estimated and the causal structure of the model are proved to resemble the real-world one) we can make comparative exercises that aim at understanding how different configurations of the parameter set, that ‘resisted’ estimation and calibration procedures, affect the aggregate patterns of interest (e.g. the dynamics of the simulated time series). In this respect, Casini and Manzo (2016) note that sensitivity and robustness analysis provide two natural instruments to investigate the functioning of the causal mechanisms in a deeper way. In fact, before validation the multiple outcomes linked with the degrees of freedom of the parameters space and initial conditions can only be considered as interesting proof-of-concepts on which no reliable causal claims can be made. After validation (i.e. with the injection of empirical information), the parallel worlds generated by the ABM can be seen as credible worlds and the modification of this or that parameter is not only a theoretical exercise but turns out to provide valuable insights in terms of policy experiments.15

The second point is related to the fact that the counterfactual interpretation of alter-native parameter settings still faces an unresolved problem, perfectly illustrated by the following question: How can one interpret “alternative parameter values in an evolution-ary world where history, indeterminacy, and non-linear feedbacks between the micro and macro levels may strongly affect the outcomes?” (Fagiolo et al., 2007). As well said by Stock (2001) maybe "the workings of the economy are too subtle and evolve too rapidly to be divined statistically" and, perhaps, even structurally. If output validation as ‘matching of causation’ partially solves the under-determination problem mentioned above - especially if it is properly complemented by careful estimation procedures - an equifinality problem may equally arise with respect to the micro-mechanism underlying the simulation out-come: different explanations entailed by different micro-structures can be consistent with the same aggregate data and possibly also with the same macro-causal structure (Casini and Manzo, 2016; Fagiolo et al., 2019). This multiple-realizability problem (as it is known in philosophy of science) is an issue for macro ABMs. Indeed, our quick analysis of two validation procedures shows the difficulties for the ABM modeller to provide severe tests for model outputs. The problem, noted by Fagiolo et al. (2007) with respect to the Brock’s ‘unconditional objects’ critique, is well summarized by the motto ‘replication does not nec-essarily imply explanation’: it is difficult to extract relevant information on the dynamics

but also on what we have not observed but that may have happened if we had intervened on the system.

15

It is worth underlying the importance of running sensitivity analysis after the validation procedure. Fagiolo et al. (2007) claim that “the extent to which sensitivity analysis is performed prior to empirical validation has important implications for the universality of the simulation results that are obtained”. In other words, doing sensitivity before engaging in empirical validation can be an interesting exercise that however does not bear any causal relevant evidence.

(28)

of the stochastic processes from the data that we actually observe (Brock, 1999). Even if we obtain good results in terms of parameters estimation (as it is done by Delli Gatti and Grazzini 2020) and matching of aggregate real-world causal structures (as for example in Guerini and Moneta 2017) we still do not really know which low-level causal mechanism has triggered the macro outcome observed.16 _{In this respect, we agree with the idea that a}

possible solution could be “to not only validate the macro-economic output of the model, but also its micro-economics structure” (Delli Gatti et al. 2018). We argue that this last suggestion is naturally implied by our treatment of macro-ABMs as causal-machineries that provide a ’production’ account of causality. This is particularly important if we consider the greater potentiality of ABM vis à vis DSGE for what concerns input and microlevel output validation. On this latter field DSGE are silent by construction (the aggregation problem is avoided by the representative agent assumption and the interplay of multi-level processes is ruled out a priori) and it is arguably the most important weakness of their theoretical construct.17 _{However, the view according to which the target to replicate is}

not only the aggregate time series but also the actual set of micro-level processes implies that the “limitations imposed by data availability becomes exorbitant” (Casini and Manzo, 2016).

2.3.1 Taking stock: model explanation as ’progressive empirical valida-tion’

In the previous analysis we saw that different validation procedures have been proposed in the macro-ABM literature, each tackling different methodological targets, from quali-tative statistical resemblance and quantiquali-tative estimation to causal structures similarity. On closer inspection, this set of validation methods fits quite well with the proposed bi-dimensional definition of empirical validation given at the beginning of the section. In particular, some of the methodologies presented (e.g. Grazzini et al., 2017) belong to ’cal-ibration and estimation’ part of the definition, in which particular attention is devoted to make the parameters of the model as close as possible to the real-world ones; on the other hand, other mentioned approaches (Guerini and Moneta, 2017; Lamperti, 2018b) pertain to what we loosely called ’model evaluation’, in which the focus is on assessing whether the model mechanisms resemble the real-world ones along statistical and/or structural

as-16

“To show the generative sufficiency of a mechanism is not to show that it is in fact this mechanism that is in some particular instance at work” (Casini and Manzo 2016).

17

In analytical DSGE models the interaction and heterogeneity is very limited, functional forms are linearized and as a result aggregation is not an issue. By this way, macroeconomic DSGE models are validated only on the output side.

(29)

Fig. 2-2. Representational scheme of empirical validation as model calibration/estimation + model evaluation.

pects. This general, though admittedly crude, representation of the concept of empirical validation can be graphically represented as in Fig. 2-2.

The y-axis represents the ’calibration/estimation’ dimension and the top-down direc-tion of the arrow indicates that the more the model inputs are calibrated and/or estimated the more the model is closer to the target system T (i.e. the explanandum-phenomenon) of interest. The x-axis represents instead the ’evaluation’ dimension and the left-to-right direction of the arrow indicates the degree of resemblance of the model mechanisms with the real-world counterparts. An important thing to notice is that though being highly inter-dependent, these two dimensions are susceptible to be analyzed autonomously: one can in principle calibrates the parameters of the model without analyzing its statistical performance and likewise, the statistical properties of a particular model can be investi-gated whether or not the inputs of that model are calibrated or estimated. Actually, it is very likely to have in-between cases as it is shown in Fig. 2-2. Take for instance the work by Delli Gatti and Grazzini (2020) (DG & G (2020) in the figure): this is arguably an hybrid methodology because the authors first estimate the model through a Bayesian approach - a quite advanced estimation technique that justifies its position at the bottom of the y-axis - and then evaluate the performance of the model by comparing artificial and observed stylized facts and especially by running out-of-sample forecast exercises. However, it can

(30)

be noticed that the work by Guerini and Moneta (2017) (M & G (2017) in the image) is more to the right with respect to Delli Gatti and Grazzini (2020) because, as we discussed in the previous section, it compares artificial and observed causal structures rather than simply evaluating the matching with statistical regularities. By the same token, M & G (2017) is in a higher position in the x-axis with respect to DG & G (2020) because no attempt is made "to maximize the similarity of the two models by means of parameter estimation" (Gallegati et al., 2017). To sum up, both methods enable the models under scrutiny to pass from level 1 to level 2 of the ’empirical validity ladder’. However, under a causal inference perspective (Casini and Manzo, 2016), the M & G (2017) methodology takes explicitly into account a causality-based approach whereas DG & G (2020) restrict themselves to statistical and forecasting analysis.

With this framework in mind, we are now able to conceptualize the espistemological status of macro-ABMs by suggesting a tentative account of model explanation as ’progres-sive empirical validation’: in a nutshell, the more a model is empirically validated along the two aforementioned dimensions (calibration/estimation and evaluation), the more it explains the explanandum-phenomenon (or target system T ) it is supposed to explain. In other words, if a model’s parameters are properly calibrated and/or estimated and its mechanisms are extensively evaluated from both a statistical and causal perspective, than the model under investigation is supposed to be a quite reliable ’causal device’ that can be used to effectively carry out policy and structural macroeconomic analysis.

The core idea of this approach is that ’model explanation’ is not a binary concept (i.e. a model either explains or it does not) but rather it comes in degrees. This suggestion is represented in the two shaded areas in Fig. 2-2. The movement from the left to the right side of the graph marks the progressive passage from the dynamic sufficiency of a model, i.e. its ability to produce an output reasonably similar to that of its target (e.g. qualitative matching of micro and macro stylized facts), to its mechanistic adequacy, i.e. the capa-bility of a model "to mimic the causal of its target adequately" (Gräbner, 2018). To be more precise, dynamic sufficiency overlaps in some respects to what Tesfatsion (2006) calls ’descriptive output validation’ that asks to what extent the output of a model can actually replicate existing data. Following Gräbner (2018), this form of validation is rather weak because of (i) overfitting problems, i.e. the model has so many free parameters that it can be easily calibrated on existing data although it may performs poorly for new data and (ii) equifinality problems, related to the fact that "the mechanism-to-function mapping is many-to-one" (Gräbner, 2018). By performing ’predictive output validation’ (Tesfatsion,

(31)

2017), Delli Gatti and Grazzini (2020) solves the overfitting problem but their work is sub-ject to the equifinality issue. On the other hand, the structural output validation adopted by Guerini and Moneta (2017) solves the equifinality problem but only partially: here em-pirical validation is conducted only at the aggregate level meaning that, in principle, the method is not able to rule out different competing micro-configurations that may have generated the resulting macro-outcome. This consideration marks the need for a structural empirical validation methodology that takes into account the different levels of aggregation represented in a macro-ABM.

As a final remark, the proposed approach to model validation can be blamed for adopt-ing an empiricist perspective, in which the only thadopt-ing that really matters is what we can learn from empirical reality. Admittedly, empirical validation is not the only dimension along which the explanatory potential of a model should be assessed: for instance, the fact that macro-ABMs are able to account for a richer micro-dynamics with respect to DSGEs enhances the explanatory power of ABMs though being mainly a theoretical (rather than empirical) consideration.

2.3.2 Concluding remarks

The previous analysis supports the thesis that AB modellers engaged in policy analysis should go towards a causality-based approach to explanation, provided that this concept is consciously outlined in a non-trivial, rigorous and pluralistic framework. Of course, applied methodologies to pursue this task - from causal discovery methods to structural econometric analysis to model exploration in general - are not able to fully take into account the multi-faceted character of causation. Still, the previous analysis shows that great improvements have been made in the last years suggesting that AB modellers are (and must be) in the business of establishing causal claims. This idea is shared also by some economists involved in empirical validation and policy analysis through macro-ABMs who claim that "developing more rigorous methods to compare [. . . ] causal mechanisms is one of the most important open problems in the agent-based modelling community"(Barde, 2020).

Given the difficulties underlined in the previous sections regarding the validation of macroeconomic agent based models, three interrelated issues emerge as particularly urgent in tackling the problem of causal inference and empirical validation more broadly.

First, the lack of commonly established protocols in the ABM community (Fagiolo et al., 2019; Richiardi et al., 2006). This point has been recently illustrated by Platt (2020)

(32)

who claims that the literature on this theme is “over-compartmentalised”: the modeller is left with a plenty of methods without really knowing which of them is the best.

Second, there is a clear ambiguity in macro-ABMs analysis for what regards causality. On the one hand, there is evidence that establishing causal claims is the goal of a vast majority of ABM papers, especially policy-oriented ones. On the other hand, all these causal claims are not supported by an explicit account of causality and no genuine discussion is made on using ABMs as a tools for causal inference. In this regard, we suggested to build a bridge between the agent based approach to macroeconomic modelling and the mechanism-based account of causality (or generative approach to explanation). This connection – that has proved to be fruitful in other social sciences such as analytical sociology - can serve as a sort of ex-ante validation of the micro-to-macro relations typical of complex macroeconomic systems.

The third issue is the lack of suitable data for a proper micro-validation of macro-ABMs. The feeling that macro-ABMs validation could progress if only suitable datasets were available is widespread in the literature. For instance, Delli Gatti et al. (2018) claim that “the most common reason for under-determination in economics is the incompleteness of the available data sets” and that “sometimes a model is disregarded on the basis of existing empirical data, but other types of data could provide a better test and potentially support the model, if they had been collected”. This latter point addresses a fundamental meta-methodological issue concerning the “inherent conservativeness” (Fagiolo et al., 2007) of established calibration and estimation techniques. This leads to a methodological bias since the ABM modeller is discouraged to build models characterized by the interplay of multiple (and multi-level) processes for which data are not readily available.18 _{As a}

consequence, in the current situation of lack of data, we are in the paradoxical situation that “the more weight is given to full empirical calibration and validation, the less ABMs can be used to gain insight about those phenomena for which data are missing” (Casini and Manzo, 2016). To overcome this issue and avoid a priori restrictions to the models new high quality datasets should be constructed. In this sense, Fagiolo et al. (2019) feel optimistic in claiming that “the increased data availability and computing power will push toward the systematic inclusion of more informative micro-level data”. Arguably, this is fundamental to achieve more fine-grained lower-level empirical validation of macro ABM

18

The issue is well summarized in Fagiolo et al. (2007) where it is noted that the aforementioned bias “supports the continuation of orthodox theories and models for which empirical data is readily available. It disadvantages new theories and new models for which empirical research has not yet caught up, and mitigates against the study of qualitative phenomena that are either difficult to measure or are inherently immeasurable by their very nature” (Fagiolo et al., 2007).

(33)

(34)

Chapter 3 Methodological approach

”I will take the attitude that a piece of theory has only intellectual interest until it has been validated in some way against alternative theories and using actual data .”

—Clive Granger, (Granger, 1999)

Inspired by the the famous book Causality in Macroeconomics by Kevin D. Hoover (2001a), this chapter seeks to answer to the following methodological questions: How can we know about causal relationships using macroeconomic data? In particular, how might we infer causal direction and measure causal strength starting from macroeconomic time series data?

After a short presentation of the so-called identification problem in Structural Vector Autoregressive Models, two methodologies to discover causal relations by analyzing sta-tistical properties of purely observational time series data are presented: graphical causal models and independent component analysis. Finally, in the last section we show how these techniques can be fruitfully exploited to compare model-generated causal structures with real world ones.

3.1 The SVAR identification problem

Since the famous article by Sims (1980), Vector Autoreggressive Models (VARs) have become standard tools in empirical macroeconomics. Before this pioneering work, applied macroeconomists extensively used multivariate simultaneous equations models, inspired by the research program of the Cowles Commission during the 1950s and the work of Haavelmo (1944). The core of the Sims’ critique to this methodological approach pointed to the arbitrariness in the determination of the endogenous and exogenous variables in the

A Data-Driven SVAR Approach to Model Validaton: an Application to a Macroeconomic Agent-based Model