Beyond full rationality: modeling tradeoff dynamics in multi-objective water management

(1)

Politecnico di Milano

Department of Electronics, Information, and Bioengineering

Doctoral Program in Information Technology

BEYOND FULL RATIONALITY:

MODELING TRADEOFF DYNAMICS

IN MULTI-OBJECTIVE WATER MANAGEMENT

From tradeoff identification to modeling its time evolution

Doctoral Dissertation of:

Emanuele Mason

Advisor:

Prof. Andrea F. Castelletti

Co-Advisor:

PhD. Matteo Giuliani

Tutor:

Prof. Luca Bascetta

The Chair of the Doctoral Program:

Prof. Andrea Bonarini

(2)

(3)

πρὸς ἐµαυτὸν δ᾿ οὖν ἀπιὼν ἐλογιζόµην ὅτι τούτου µὲν τοῦ ἀνθρώπου ἐγὼ σοφώτερός εἰµι: κινδυνεύει µὲν γὰρ ἡµῶν οὐδέτερος οὐδὲν καλὸν κἀγαθὸν εἰδέναι, ἀλλ᾿ οὗτος µὲν οἴεταί τι εἰδέναι οὐκ εἰδώς, ἐγὼ δέ, ὥσπερ οὖν οὐκ οἶδα, οὐδὲ οἴοµαι

and so, as I went away, I thought to myself, “I am wiser than this man; for neither of us really knows anything ﬁne and good, but this man thinks he knows something when he does not, whereas I, as I do not know anything, do not think I do either.” Apology, section 21d, in “Plato in Twelve Volumes, Vol. 1 translated by Harold North Fowler. Cambridge, MA, Harvard University Press; London, William Heinemann Ltd. 1966.”

道可道, 非常道

名可名, 非常名

The truth that may be told is not everlasting Truth. The name given to a thing is not the everlasting Name. Dàodé J¯ıng 道德, translated by Yáng Ji ¯aluò 楊家駱 in 老子 truth and nature. Hong Kong. 1962

"Forty-two," said Deep Thought, with inﬁnite majesty and calm. "The Answer to the Great Question, of Life, the Universe and Everything" "I checked it very thoroughly," said the computer, "and that quite deﬁnitely is the answer. I think the problem, to be quite honest with you, is that you’ve never actually known what the question is."

(4)

(5)

Abstract

A

nthropocene denotes the scale and intensity of the anthropic inﬂuence on natural processes and ecosystems. Recent trends in the scientiﬁc lit-erature on natural resources acknowledge this issue by putting forward the concept of Coupled Human-Natural System (chns). It denotes systems where the human and natural components are so entangled that a correct as-sessment of system resilience or sustainability require a comprehensive study of both parts. Within the human component, a crucial role is played by decision making, which mediates human interactions at the various levels of system governance, ranging from institutional to operational decisions that directly impact the natural resource. Modeling of decision making has been a daunting challenge for researchers. However, they developed various approaches, among which the normative approach.

In the normative approach, it is assumed that an agent’s decisions seek to rationally achieve a certain goal. The rationality hypothesis enabled a plethora of theoretical studies in various fields, e.g., economics, or optimal control theory, which supported the broad adoption of this modeling approach to provide prescriptions. However, the same hypothesis has been strongly criticized on the basis of empirical evidence of behavior deviations. In particular, full rationality have thwarted modelers’ efforts to deal with systems operated for multiple objectives. In these systems, the operating policy has to balance multiple goals, by reflecting the preferences of the decision maker and/or of the stakeholders. Another issue arises from the time dynamics of the tradeoff and preferences that are unlikely to remain stationary but are instead adjusted in response to various changes. Triggers of the change may be exogenous influences that modify the conditions at the system boundary, or extreme events, such as floods or droughts, originated by the inherent variability within the system.

The objective of this thesis is to advance algorithms adopting the normative approach to develop behavioral models of system operators. The proposed algorithms are able to cope with tradeoﬀs among multiple objectives, and with the time evolution of preferences.

A ﬁrst eﬀort has been devoted to formalize the modeling of multiple objec-I

(6)

us to adopt the idea of rival framings, each representing a set of objectives for-mulations, to rationalize the search for the candidate set of objective functions that represents the operator of the modeled system.

On this premise, we then propose two different algorithms to identify the tradeoff among the multiple objectives that best represents the historical opera-tions in the modeled chns. The first algorithm adopts Inverse Reinforcement Learning to efficiently identify a set of weights that measure the preferences of the system operator as they can be presumed by a time series of system opera-tions. This algorithm is able to achieve high quality, above 0.9 goodness-of-fit, in a synthetic application. We also applied it to a real case study, effectively improving the operator’s behavioral model with respect to single-objective counterparts. Moreover, we were able to quantify the effect of an exogenous transition on the system in terms of change in the weights of operating objec-tives.

We also developed a second algorithm for tradeoff identification, inspired by multi-agent negotiation protocols, called Set-based Egocentric Concession protocol (sec). Operator’s behavioral models identified with this algorithm prove to be accurate, as we tested on a synthetic case study. Moreover, sec identifies the tradeoff as a function of a set of parameters, named attitudes, that can be used to model tradeoff evolution in time. To this end, we propose an autoregressive model of attitude evolution driven by the recent system performance as they reflect the extreme variability of the system, e.g., in terms of droughts and floods. We found this model a promising start to explain the evolution of the tradeoff of a timeserie of decisions with dynamic preferences, developed for the synthetic case study. More significantly, we framed the testing of the proposed model of preference evolution in a scientific approach that has significant implications for the construction of reliable projections of the future evolutions of chns.

Part of this research has appeared (or has to appear) in the following journal publications:

• Amigoni, F., Castelletti, A., Gazzotti, P., Giuliani, M., Mason, E., 9-13 May 2016. Using multiagent negotiation to model water resources systems operations. In: Proceedings of the international workshop of the 2016 Autonomous Agents and MultiAgent Systems International Conference on Issues with Deployment of Emerging Agent-based Systems. Singapore. • Mason, E., Giuliani, M., Castelletti, A., Amigoni, F., 2017a. Identifying and modelling evolving tradeoﬀs in multipurpose water resources systems via agent-based negotiation. Water Resources Research, under review. • Mason, E., Giuliani, M., Pirotta, M., Restelli, M., Castelletti, A., 2017b.

(7)

Inverse reinforcement learning to unveil management tradeoﬀ. Environ-mental Modeling & Software, in preparation.

(8)

(9)

Sommario

L’

antropocene, il nome dell’attuale era geologica, sottolinea la scala e

l’in-tensità dell’influenza antropica sui processi naturali e sugli ecosistemi. I recenti sviluppi della letteratura scientifica sulle risorse naturali tengo-no conto di questa influenza con il concetto di Coupled Human-Natural System (chns). Questo concetto si riferisce a quei sistemi dove le componenti antropi-che e naturali sono così interconnessi da necessitare uno studio comprensivo di entrambe le parti per valutare correttamente la resilienza o la sostenibilità del sistema. All’interno della componente antropica, i processi decisionali svolgono un ruolo cruciale, essendo parte di ogni interazione umana ai vari livelli della go-vernance, dalle decisioni istituzionali fino alle scelte operative che direttamente impattano sul destino della risorsa naturale. La modellizzazione dei processi decisionali ha da sempre messo in difficoltà i ricercatori. Ciò nonostante, sono stati sviluppati diversi approcci, tra cui il cosiddetto approccio normativo.

Nell’approccio normativo, le decisioni di un certo agente sono razionalmen-te volrazionalmen-te al raggiungimento di un obiettivo. L’iporazionalmen-tesi di razionalità ha aperto la strada a numerosi studi teorici in vari campi, dall’economia alla teoria del con-trollo ottimo, che hanno supportato l’adozione di questo approccio modellistico per delineare prescrizioni gestionali. Tuttavia, la medesima ipotesi è stata forte-mente criticata sulla base di prove empiriche delle deviazioni comportamentali. In particolare, l’ipotesi di completa razionalità ha rallentato gli sforzi volti a modellizzare i sistemi gestiti secondo molteplici obiettivi. In questi sistemi, la politica operativa deve bilanciare più obiettivi, riflettendo le preferenze del decisore e/o dei portatori d’interesse. Un altro problema per l’ipotesi di com-pleta razionalità nasce dalla dinamica temporale del bilanciamento tra obiettivi e delle preferenze, che molto raramente rimangono stazionari ma vengono invece adattati in risposta ai cambiamenti. Questi possono essere innescati da influenze esogene che modificano le condizioni al contorno, oppure da eventi estremi come piene o siccità, che nascono invece dall’intrinseca variabilità del sistema.

L’obiettivo di questa tesi è di sviluppare ed estendere algoritmi che adottano l’approccio normativo per comporre modelli comportamentali degli operatori

(10)

molteplici obiettivi e l’evoluzione temporale delle preferenze.

Un primo sforzo è stato dedicato a formalizzare la modellizzazione di mol-teplici obiettivi, riconoscendo l’incertezza intrinseca alla loro formulazione. Partendo da questa consapevolezza, si è adottata l’idea dei cosiddetti rival

fra-mings, ovvero cornici interpretative alternative, ciascuna rappresentante un

insieme di obiettivi e loro formulazioni. Questo approccio razionalizza la ri-cerca della rappresentazione delle funzioni obiettivo che meglio riproducono l’operatore del sistema oggetto del modello.

A partire da questo fondamento, vengono proposti due differenti algoritmi che identificano quel bilanciamento tra molteplici obiettivi che meglio rap-presenta la serie storica di gestione del sistema chns modellizzato. Il primo algoritmo si serve dell’apprendimento per rinforzo inverso per identificare efficientemente un insieme di pesi numerici che misurano le preferenze dell’o-peratore del sistema così come possono essere desunte da una serie temporale di misure delle azioni operative. Tale algoritmo ottiene prestazioni di alta qua-lità, riproducendo oltre il 90% della serie di calibrazione in un’applicazione sintetica. L’algoritmo è stato applicato anche ad un caso reale, ottenendo un miglioramento del modello comportamentale dell’operatore rispetto ad una controparte a singolo obiettivo. Inoltre, ha permesso di quantificare l’effetto di una transizione esogena che ha interessato il sistema in termini di cambiamento dei pesi relativi agli obiettivi operativi.

È stato anche sviluppato un secondo algoritmo per l’identificazione del bilanciamento, ispirato ai protocolli di negoziazione multiagente, chiamato Set-based Egocentric Concession protocol (sec). I modelli comportamentali degli operatori identificati con questo algoritmo si sono rivelati accurati, così come dimostrato dall’applicazione ad un caso di studio sintetico. Inoltre, sec permette di identificare il bilanciamento degli obiettivi in funzione di certi parametri, chiamati attitudini, che possono servire come base per modellizzare l’evoluzione temporale del bilanciamento. Riguardo quest’ultimo scopo, viene proposto in questo lavoro di tesi un modello autoregressivo dell’evoluzione temporale delle attitudini in funzione delle prestazioni recenti del sistema, le quali riflettono l’estrema variabilità del sistema, per esempio in termini di piene o magre. Tale modello si è dimostrato essere un primo promettente passo verso la spiegazione dell’evoluzione temporale del bilanciamento tra gli obiettivi, qui verificato per il caso di studio sintetico su una serie temporale di decisioni costruita con preferenze dinamiche. Non solo, la sperimentazione e lo sviluppo di questo modello dell’evoluzione temporale delle preferenze sono stati affrontati con un approccio scientifico che ha significative conseguenze sullo sviluppo e la costruzione di proiezioni credibili dell’evoluzione futura di chns.

(11)

Parte di questa ricerca è apparsa, o apparirà a breve, nelle seguenti pubblica-zioni scientiﬁche:

• Amigoni, F., Castelletti, A., Gazzotti, P., Giuliani, M., Mason, E., 9-13 May 2016. Using multiagent negotiation to model water resources systems operations. In: Proceedings of the international workshop of the 2016 Autonomous Agents and MultiAgent Systems International Conference on Issues with Deployment of Emerging Agent-based Systems. Singapore. • Mason, E., Giuliani, M., Castelletti, A., Amigoni, F., 2017a. Identifying and modelling evolving tradeoﬀs in multipurpose water resources systems via agent-based negotiation. Water Resources Research, in revisione.

• Mason, E., Giuliani, M., Pirotta, M., Restelli, M., Castelletti, A., 2017b. Inverse reinforcement learning to unveil management tradeoﬀ. Environ-mental Modeling & Software, in preparazione.

(12)

(13)

Acknowledgements

A number of people contributed directly or indirectly to this research and supported me during these years of PhD studies.

First of all, I would like to express my sincere gratitude to my supervisor, Professor Andrea Castelletti. He ﬁrst arouse my curiosity towards the use of new models for designing the management of water resource systems. He enabled and supported this research with inestimable dedication and kindly gave me motivation, trust, and resources. I would like to thank also Dr. Matteo Giuliani for supervising this research with Professor Andrea Castelletti. He has been a great example for me, and I really appreciated his commitment, patience, and his ability to approach science with extreme clarity and eﬀectiveness.

I am thankful to Professor Francesco Amigoni, with whom I collaborated in the initial part of this research, for his rigorousness and availability under incoming deadlines. I also want to thank Paolo Gazzotti, to whom I wish all the best for his doctoral studies. I wish to express my gratitude to Professor Marcello Restelli and to Dr. Matteo Pirotta, who kindly shared their latest algorithms and have been always interested in exploring new applications. Their acumen is of great inspiration to me.

I would like to acknowledge Professor Andrea Rizzoli, SUPSI, and Professor Jon D. Herman, UC Davis, for reviewing this PhD thesis and for their construc-tive comments and suggestions. Their inputs have been essential to add value to the ﬁnalized version of this thesis.

This research has been supported by the scholarship for PhD studies oﬀered by the Italian Ministry of Education, that I enjoyed for these three years.

I am full of admiration for all the members of the Natural Resources Manage-ment group at Politecnico di Milano and to my PhD fellows at the DepartManage-ment of Electronics, Information and Bioengineering. Our discussions and friend-ship enriched me as a person and as a professional. They made my time here pleasant, and knowing that young researchers like them are out there makes me feel hopeful.

Lastly, I would like to express my deepest gratitude to those for whom words are never enough. My family, my friends, and above all, Cecilia, who gave me the support to complete this research.

(14)

(15)

List of Figures

1.1 Precautious releases during the irrigation season to avoid

ﬂood-ings along the shores of Lake Como. . . 4

1.2 Chns conceptualization. . . 5

1.3 Simpliﬁed overview on the three world of decisions. . . 7

1.4 Simpliﬁed overview of methods for modeling static and dynamic operator’s behavior with reference to the sections where they are described. . . 13

1.5 Overview of the research contribution of this thesis. . . 18

2.1 Scheme of a mdp. . . 20

2.2 Schematization of the dps method. . . 22

2.3 Methodology for tradeoﬀ identiﬁcation with cpirl. . . 27

2.4 Depiction of cutting planes construction in cpirl. . . 28

2.5 Schematization of the tradeoﬀ identiﬁcation using sec. . . 31

2.6 Schematization of how sec is similar to the reference point method. . . 33

2.7 Example of a negotiation among two agents over a problem resulting in a concave Pareto front. . . 34

2.8 Schematization of the tradeoﬀ evolution modeling. . . 35

3.1 Diagram of the numerical case study used across this thesis. . . 40

3.2 Performance of Pareto optimal operating policies in terms of Flooding (x-axis) and Irrigation (y-axis) objectives. . . 42

3.3 Cumulative distribution functions for the synthetic inﬂows and relevant statistics for the ﬂooding and irrigation objectives. . . 44

3.4 Location of the Premadio station within the A2A system in upper Valtellina, north of Italy. . . 45

3.5 Historical data of the Premadio operations. . . 46

3.6 Hourly prices of electricity in the regulated and free energy market from the most proﬁtable to least one. . . 47

3.7 Histogram of the San Giacomo inﬂows, and their autocorrela-tion funcautocorrela-tion. . . 50

(18)

3.8 Mathematical transformation of the San Giacomo inflows . . . 51 3.9 Root Mean Square Error for different inflow models and moving

window. . . 52 3.10 Analysis of residuals from four models. . . 53 3.11 Autocorrelation function of the residuals of the model

t_ar-mas0_364with window size 3. . . 53 4.1 Policy of the suboptimal expert, and its performance. . . 56 4.2 Release pattern of Premadio’s optimal operations for the

diﬀer-ent objectives. . . 61 4.3 Comparison of simulation trajectories of the optimized policies

with respect to historical behavior from 1996 to 2002. . . 63 4.4 Comparison of simulation trajectories of the optimized policies

with respect to historical behavior from 2010 till 2014. . . 65 5.1 Excerpt of the sec negotiation in a single time period. . . 70 5.2 Model accuracy over a 100 time periods scenario obtained

through repeated sec negotiations. . . 71 5.3 Accuracy of the dynamic attitudes model and correlation

anal-ysis between the trajectories of the attitudes parameters and some features of the inﬂow scenario. . . 73 5.4 Excerpt of the time evolution of preferences under diﬀerent

parameters. . . 74 5.5 Accuracy of the attitudes’ dynamic model on the three-objective

scenario. . . 75 5.6 Comparison of performances obtained on the inﬂow scenario

used in Section 5.1.2 with diﬀerent tradeoﬀ selection strategies. 78 6.1 Pareto optimal alternatives to the historical management of

Lake Como in three recent time periods. . . 83 A.1 Main drivers of operations in Lake Como. . . 90

(19)

List of Tables

3.1 Parameters used in the inﬂow scenario generation. . . 43 4.1 Cpirl experiments to select most promising dm’s objectives and

their formulations. . . 57 4.2 Objective values of single objective and multi-objective optimal

policies, compared with the historical Premadio operations from 1996 to 2002. . . 62 4.3 Weights identiﬁed by the diﬀerent query-point selection

meth-ods of cpirl to best explain the historical 1996-2002 operations’ goal. . . 62 4.4 Weights identiﬁed by the diﬀerent query-point selection

meth-ods of cpirl to best explain the historical 2010-2014 operations’ goal. . . 64

(20)

(21)

Acronyms

A

ann Artiﬁcial Neural Network. 21

arma Autoregressive Moving-Average model. 49, 50, 52

C

chns Coupled Human-Natural System. 5–7, 12– 16, 20, 34, 81, 82, 84

cpirl Cutting-Plane Inverse Reinforcement Learn-ing. 15–17, 26–29, 32, 35, 55, 57–59, 61–66, 82 D dm Decision Maker. 8–11, 13–17, 22–25, 29, 36, 39, 40, 42, 55–59, 68, 69, 71, 72, 75–79, 82 dp Dynamic Programming. 21, 25

dps Direct Policy Search. 16, 20–22, 32, 40, 42 dss Decision Support System. 12, 14

E

emodps Evolutionary Multi-Objective Direct Policy Search. 24, 56, 68, 78, 79, 87

G

gme Gestore dei Mercati Energetici, authority of

energy markets. 47, 63

I

irl Inverse Reinforcement Learning. 15, 16, 25, 26, 55, 58, 66, 82

(22)

M

mas Multi-Agent Systems. 30, 77

mdp Markov Decision Process. 14, 19, 20, 25, 43, 52

N

nfe Number of Function Evaluations. 56, 87

P

pun Prezzo Unico Nazionale, unique national

[elec-tricity energy] price. 63

R

rl Reinforcement Learning. 14, 25, 26 rmse Root Mean Squared Error. 51, 52

S

sdp Stochastic Dynamic Programming. 16, 20, 21, 26, 40, 41, 49, 52, 60, 63, 66

sec Set-based Egocentric Concession protocol. 15–17, 30–33, 35, 67–72, 75–80, 82

sop Standard Operating Policy. 21, 40, 41, 56, 68, 69, 85, 88, 89

(23)

1

Introduction

A

nthropocene is the epoch we are nowadays living in, and takes its name from the signature that human activities are leaving in stratigraphical records (Crutzen, 2002; Steffen et al., 2007).1_{Anthropogenic impacts} have in fact reached the planetary scale and are threatening the resilience of many Earth processes (Steffen et al., 2015). Among these, surface runoff pro-duction, freshwater storage and other components of the water cycle, which are key to support ecosystems and other natural processes, are altered by the interaction with the society. This is a great concern, as human-induced cli-mate change is expected to aggravate and exacerbate water scarcity as well as flood risk (Hirabayashi et al., 2013; Schewe et al., 2014). Increase in the latter has already been observed, but the connection of these evidences to climate change remains unclear (Kundzewicz et al., 2010; Guhathakurta et al., 2011). Regardless of climate change, direct human pressures are expected to be at least of the same order of magnitude of climate change impacts (Haddeland et al., 2014). Population growth, for instance, increases water demand for domestic consumption as well as for agricolture (Zhou et al., 2016), and pushes towards more water withdrawals and dams construction (Wada et al., 2014). Recent estimates (Lehner et al., 2011) suggest that existing dams control around 46% of the world largest rivers, i.e., rivers having an average flow above 1000 m3_/s. This figure is expected to grow rapidly following the renewed interest in dams as a primary mean to secure water and energy in fast developing African and Asian countries (World Bank, 2009; Zarfl et al., 2014). Massive infrastructure expansion will further enlarge the number of river systems whose dynamics is

1_{This word was ﬁrst recorded in the work of the Italian geologist Stoppani (1873, Chapter XXX).}

(24)

not only determined by their streamﬂow regime according to naturally varying rainfall, climate, and hydrologic conditions (Botter et al., 2010), but is mediated by human decisions driven by one or multiple operating purposes (Caldas et al., 2015). Moreover, these alterations are so fast that their time scale is compa-rable with the eﬀects of water resources management decisions (Montanari et al., 2013; Sivapalan and Blöschl, 2015). Indeed, anthropic management is increasingly substituting the natural self-regulation of ecosystems (Vitousek et al., 1997), promoting mankind to the role of the crew of the spaceship Earth (Boulding, 1966).2

To prolong our spaceflight, the navigation should remain in the safe oper-ating space (Rockström et al., 2009), without crossing planetary boundaries that would cause irreversible and dangerous changes. The location of these boundaries is a product of environmental and human dimensions. For instance, global freshwater supply is determined by the available resource at least as much as by the institutions and infrastructures through which societies collect the benefits of the use (Padowski et al., 2015). A governing system that couples human and environment requires an adaptive form that addresses multiple scales (Ostrom, 2010), e.g., the panarchy proposed in Gunderson and Holling (2001). This new governance asks for procedures and information to identify suitable actions, condensed in usable tools such as decision support systems (Nilsson et al., 2008). However, current knowledge is ill-equipped for construct-ing reliable and credible projections of the future evolutions under change (e.g., Wagener et al., 2010). Disciplinary boundaries have to be crossed to come up with specific, although partial solutions and tools to support each arena of decision-making (Redman, 1999; DeFries and Nagendra, 2017).

This doctoral dissertation develops within this picture and aims at advancing tools for modeling decision-making in complex water systems. To provide a comprehensive example of the challenges we address, the box below illustrates the last years of management in a complex water system, Lake Como.

An example: Lake Como management

Since 1946, Lake Como has been operated to supply water to 1400 km2 of irrigated agriculture. Operations also provide protection from ﬂood-ing along lake shores, especially in the lowest point situated in piazza

Cavourin the city of Como. Regulation of lake level is realized through

a system of dam gates that modulate the releases when the lake is be-tween−0.50 m and 1.30 m with respect to the hydrometric station in Malgrate. This is a discretionary operating space deﬁned by normative

2_{Although some underline that men across and within societies do not equally share impacts and responsibilities} (Malm and Hornborg, 2014).

(25)

constraints, rather than by the infrastructure. For instance, the regu-lation license enforces a complete opening of the dam gates above the maximum level (Galelli and Soncini-Sessa, 2010).

The historical operations effectively used the dam to avoid floods while providing more water to downstream users (Moisello et al., 2013). Down-stream water demand peaks in summer, thus water is stored and pre-served in the spring. However, increasing lake storage means also in-creasing flood risk, and this conflict has been throughly analyzed in the literature (Guariso et al., 1986; Castelletti et al., 2010; Anghileri et al., 2013; Culley et al., 2016). Figure 1.1 shows the tradeoff between the two objectives, visualized by plotting lake levels during the high demand season when the release exceeds the downstream demand. In fact, the only rational reason for this to happen is to lower lake levels for buffering any incoming flood.

During the first years, the operator was activating these precautious releases only when the level was above the upper boundary of 1.30 m, in accordance with the original sizing of the infrastructure. However, exploitation of groundwater in Como initiated a process of subsidence that involved piazza Cavour, and by 1975 the square level reached the lake regulation upper boundary. Consequently, lake operations were adjusted, and precautious releases started at lower levels, just above 1.2 m. Then, two major flood peaks in 1993 (highest level since dam construction), and in 1997 put the system under stress, and triggered another adjustment of the operations, as one can appreciate by looking at the dots aligning around 0.6 m from 1998 to 2002. This adjustment was caused by an event endougenous to the system, namely the inflows variability. More recently, no dots make their way into Figure 1.1 from 2003 until 2007, because severe droughts hit the system, and water shortage during the irrigation season caused damages to downstream stakeholders. This again triggered another adjustment of the precautious releases to start from a level around 0.8 m, in an attempt to restore part of the lake storage capacity.

Analysis of the recent droughts underlined the extreme conditions of 2003-2006, eﬀectively characterizing these years as hydrological droughts, which also damaged agriculture (Zaniolo et al., 2017). How-ever, better management and coordination among stakeholders reduced the scarcity in 2007 despite facing again adverse hydrometeorogical con-ditions (Riva, 2010). In particular, the regional government forced the upstream hydroelectric reservoirs to release more water, and lowered the bottom boundary of the lake regulation to release additional water

(26)

(ddG 11321/19 luglio 2005). 1975 1980 1985 1990 1995 2000 2005 2010 2015 -0.5 0 0.5 1 1.5 2 2.5 3 Lake level [m]

Figure 1.1: Precautious releases during the irrigation season to avoid ﬂood along the shores of Lake

Como. The red lines indicate the ﬂooding threshold, the upper one being the original1.3 m, while

the lower one is adapted to the subsidence in Como, around1.24 m.

The importance of including the management in estimating the oper-ations effects in water resources system should be now clear. In Lake Como, the system balance between providing water for agriculture and protecting the lake shores from flooding resulted in different effects of high and low flows. This tradeoff has been adjusted during the his-tory of dam operations, usually in response to adverse conditions: the mechanism behind the adjustment is of special interest for this thesis. The case of Lake Como management underlines the intricate network of interplays between natural processes, stakeholders, and governance system that can influence the daily operations, e.g., the water released from the lake. The interaction along most of the linkages is modulated by human decisions, from the level of daily operations to regulatory actions. In particular, daily operations are adjusted to the evolving tradeoff chosen by the system operator. However, the complexities arising from the tradeoff among multiple objectives, its evolution in time, and the dynamics of the evolution, are a complicate matter that is faced in the following section.

1.1 The wicked problem of modeling human decisions

Managing natural resources in complex and changing systems requires prag-matic tools that adopt new paradigms of hydrological prediction as well as emerging understanding of the interdependence on other resources (Wagener

(27)

1.1. The wicked problem of modeling human decisions

CHNS

H

HUMAN COMPONENT

N

NATURAL COMPONENT feedbacks external disturbances system response

CONSTITUTIONAL

COLLECTIVE

OPERATIONAL

CHOICE

resource use operational policy

mechanisms

CHOICE CHOICE

Figure 1.2: chns conceptualization adapted from (Li, 2016; Polhill et al., 2016).

et al., 2010; Thompson et al., 2013; Scanlon et al., 2017; Wada et al., 2017). These systems are labelled as Coupled Human-Natural Systems (chns; Liu et al., 2007) to highlight the complex feedback among socio-economic and natural components that determines the system evolution (see Figure 1.2 for a concep-tualization). While a long legacy of studies focused on the natural processes (e.g., Fanning, 1896), understanding and representing the human component is crucial for characterizing the observed complex dynamics of these systems (e.g., Emmerik et al., 2014; Elshafei et al., 2015), or for retrospectively assessing the behavior of the system managers and the associated performance (e.g., Hejazi and Cai, 2011).

This evidence has been the inspiring principle for the development of a number of modeling frameworks explicitly representing both the natural and the human component, along with their reciprocal interactions, feedbacks, and coevolution in time, such as Socio-hydrology (Sivapalan et al., 2012; Troy et al., 2015a), Coupled Environmental-Human Systems (Horan et al., 2011), Socio Ecological Systems (Redman et al., 2004; Binder et al., 2013), Socio-Environmental Systems (Filatova et al., 2016) and iSAW (integrating Structure,

Actors and Water; Hale et al., 2015).

These frameworks focus on the key properties that diﬀerentiate chns sys-tems. One of the most peculiar is complexity (Kauﬀman, 1993; J.H. Holland, 1996) that arises from non linear responses to internal and external forces com-bined to feedback across time and space, and from the inherent impredictability due to interactions of multiple actors (DeFries and Nagendra, 2017). Non linear responses can also trigger catastrophic transitions, irreversibly changing the system properties, e.g., the carrying capacity of an ecosystem, or the potential for a lake self-depuration (Carpenter et al., 1999; Lade et al., 2013).

(28)

over, it is not clear how to model these feedback interactions: recent literature on socio-hydrology focuses on water systems where feedback was previously overlooked, e.g. the human-ﬂood interaction (Baldassarre et al., 2013a,b), the interplay with agricultural systems (Elshafei et al., 2015), or in the adoption of green infrastructures (Hale et al., 2015).

Managing systems with these properties poses unique challenges, and has been recognized as a wicked problem (DeFries and Nagendra, 2017). First coined by Rittel and Webber (1973) to describe the problem of planning public policies, wicked problems require a political judgement for resolution as no single technical solution is conclusive. The problem definition itself depends on which view is adopted among conflicting stakeholders, and it is not even clear when the problem can be considered solved.3_{The interplay with societies} and governance systems means that two wicked problems are never alike, and thus no panaceas can be looked for (Meinzen-Dick, 2007): the past history of each social system determines the specific setting of the managing institution. A multilevel governance is instead advocated (Ostrom, 2010), where overlapping institutions fit the systems scale (Epstein et al., 2015),4_{and employ adaptive} management plans with feedback loops to evaluate the effects of the decision taken, and possibly trigger amelioration (for an engineered approach, see Os-trom et al., 2007; Haasnoot et al., 2013). However, these are not straightforward answers to management challenges, and the concept of wickedness exposes the issues that researchers find in understanding and modeling chns (Levy et al., 2016; Kwakkel et al., 2016).

In order to design eﬀective management, the human and natural components are often decomposed as nested hierarchies of entities, and then the interlacing among the parts is studied (Gunderson and Holling, 2001). For instance, Em-merik et al. (2014) modeled the agriculture development in the Murrumbidgee River basin, Australia, by combining ﬁve subsystems, namely population, irri-gated area, environmental awareness, ecology and hydrology. Each subsystem is represented by a lumped dynamic equation, and relevant feedbacks connect the equations. In another Australian river system, the Murray-Darling, droughts and overuses of the water drove the local Basin Authority to regulate water abstraction starting in 2010 (Walker et al., 2015). System regulation plays an important role, and its downscaling to the physical level is mediated by institu-tions and managers’ behavior, the latter being a relevant source of modeling uncertainty (Fulton et al., 2011).

Behavior of human components in chns are formed by repeated

decision-3_{An interesting consequence is that wicked problems are not scientific, since no hypothesis can be falsified by} experiments (Troy et al., 2015b). In fact, wickedness is more of an epistemological problem, due to its peculiar interplay with the knowledge and culture. Therefore, planners should not care about finding the truth, but about producing good solutions (E. Winsberg, 2006).

4_{An implementation is attempted within the European Union Water Framework Directive by promoting a river} basin approach (Moss, 2004).

(29)

1.2. Embedding water operators’ behavior into Coupled Human-Natural Systems models

CHNS

H

HUMAN COMPONENT

N

NATURAL COMPONENT feedbacks external disturbances system response

CONSTITUTIONAL

COLLECTIVE

OPERATIONAL

resource use rules

institutions

DECISION DECISION DECISION

Figure 1.3: Simpliﬁed overview on the three world of actions, a synthesis of studies on multilevel governance.

Adapted from Kiser and Ostrom (2000) and Ostrom et al. (2009, p. 82)

making processes, which are at the core of the various arenas of governance. Ostrom (1990) defines three levels of decision in studying common resources, depicted in Figure 1.3. Operational decisions interact directly with the physical layer, for instance when the system operator defines the daily release from a water reservoir. Collective decisions delimit, instead, the operational space, defining rules for the use of the resource, for instance regarding the annual partition rule among the water users. At the highest level of costitutional decisions, the institutions that mediate the collective decisions are designed according to general principles, such as equal rights for water.

In this dissertation, we focus on the operational level of decision-making to provide actionable numerical models, but further developments are desirable to expand the modeling of human decision-making to the other levels, as this is key to an holistic view of chns (a comprehensive analysis of chns modeling can be found in Liu et al., 2014; Blair and Buytaert, 2016).

1.2 Embedding water operators’ behavior into Coupled

Human-Natural Systems models

In the studies on chns, human behavior and decision-making are generally represented according to two distinct perspectives (Smith, 1991): descriptive approaches, which describe the internal decision mechanisms, and normative approaches, which focus on motivation-based actions that maximize idealized objective functions.5 _{These perspectives correspond to diﬀerent model} im-plementations: the former leads to explicit models of action, while according to the latter, actions are implicitly derived from goals or objectives. In the following, we present few theories that underpin the models brieﬂy reviewed in the subsequent Section 1.2.2.

5_{Behavioral modeling should not be confounded with Beven’s notion of “behavioral models” that focus on a style} of modeling based on organizing principles (Beven and Freer, 2001; Schaeﬂi et al., 2011).

(30)

1.2.1 Theories of decision-making

A modeling-oriented review of decision-making theories is presented in Schlüter et al. (2017). According to the descriptive approach, models implement behav-ioral rules that describe human actions in response to changing forcing, e.g., hydro-meteorological, as in the following Equation (1.1),

ut = f (It), (1.1)

where utis the vector of actions taken at time t, f(·)is the relation that imple-ments the behavioral rule, and It contains the set of information to base the decision on. Rules are either inferred from observational data, in accordance to modeler’s choices, (e.g., Schaeﬂi et al., 2007; Hejazi et al., 2008), or are in-spired by general theories that investigate the mental path toward the decision, drawing from the ﬁeld of cognitive sciences, (e.g., Giacomoni and Berglund, 2015; Sanderson et al., 2017). The latter approach is largely adopted in health education and promotion studies, under the name of Human Behavioral Repre-sentation (Mavor et al., 1998), leading to Health Belief Model (Rosenstock et al., 1988), Social-Cognitive theory (Bandura, 2001), Theory of Reasoned Action (M. Fishbein and Ajzen, 1977), Theory of Planned Behavior (Ajzen, 1991), and Protection Motivation Theory (Rogers, 1975).

These models are usually easier to accept in participatory processes, and to be trusted by system operators. However, they often include a large number of context-speciﬁc assumptions and parameters that limit the possibility of generalizing these behaviors to study future decisions under altered boundary conditions. Besides, a rigorous validation of behavioral rules against observa-tional data not used in model calibration is often impossible or missing, and thus detrimental to the reliability of the models’ outputs (Ligtenberg et al., 2010).

On the other side, the normative approach postulates a goal the actions of a rational Decision Maker (dm) should aim at. This approach is largely adopted in neoclassical economics and engineering sciences (Neumann and Morgenstern, 1944; Keeney and Raiﬀa, 1976; Becker, 1978), where is often applied to provide prescriptions. However, there are also early examples of its application to provide a description of actions (Friedman and Savage, 1948). Within this approach, the optimal action characterizing the dm’s behavior can be calculated as

ut =arg max J, (1.2)

where J, often named value or utility function, expresses the dm’s satisfaction as a function of the outcome of each action. The normative approach implies that the rational agent is able to solve the maximization problem to ﬁnd the best decision (Becker, 1978; Kagel and Roth, 1995).

(31)

This approach is extended by the Expected Utility Theory to handle uncer-tain outcomes. In this case, the utility function is formulated as J = ∑ pE,

where E is a possible outcome that happens with probability p. Probabilities can be estimated from historical records of the uncertainty source, such as meteorological conditions. The approach can also include the deterministic effects of the presence of multiple decision-makers (see Game Theory in Neu-mann and Morgenstern, 1944). Different arrangements in power relationships are translated in corresponding decision-making structures, for instance by encoding a preferential sequence determining how the various dms take their decisions. A key step of the normative approach is the composition of the utility function that, building on the efforts of economists and mathematicians, can rely on well-defined axioms and strong logical deduction. Algorithms to numerically solve the optimization problem (1.2) are well developed in the field of mathematical programming and operations research (Bertsekas, 1995).

Moreover, the concept of utility constitutes a ﬂexible layer of abstraction from context details, and allows to simulate the decision model under diﬀerent external forcings, e.g., climate change. This is a key feature of the normative approach: by assuming that agents maximize the utility function, modelers can exclude all the contextual variables that are not argument of the utility function. For instance, if a reservoir is operated with the goal of providing water supply, while water quality is not an argument of the maximization problem (1.2), then future behavior under climate change can be correctly reproduced by adopting a scenario of water availability. The evolution of water quality can be a priori excluded, as it is a consequence of how the utility function is modeled. In fact, the normative approach assumes that the utility function J captures the real interest driving the dm and that such function is time-invariant.

Beyond these potentials, normative approaches also collected a number of critiques on the basis of observations of individual behaviors (Baron, 1998). Several psychological biases are found to hinder the general rationality principle in people’s decision-making (Gigerenzer et al., 1999), such as the confirmation bias (Nickerson, 1998). Moreover, specific issues affect the decision problem formalized in Equation (1.2), and are briefly described in the following.

Value under risk: Starting with the Allais’s paradox (Allais, 1970), it has been

shown that dm’s probability assessment of uncertain outcomes can not be approximated with the real outcomes probabilities. In such a case, the utility function can be represented as J =∑ wE, where w is the weight

the dm assigns to the outcome E. Prospect theory (Kahneman and Tversky, 1979) thoroughly underlines the differences between probability p and weight w arising from psychological biases, such as the certainty effect, the isolation effect, and the framing effect.

Bounded rationality: Simon (1955, 1982) underline that few dms possess the 9

(32)

information and the computational capability to solve the optimization problem in (1.2) considering all the possible decisions. They introduce the term satisficing, a portmanteau between satisfy and suffice, to indicate a limited problem in which a subset known by the dm of all available options is analyzed to find the one that satisfy his or her target.

Multiple objectives: Water systems are often operated to meet multiple

com-peting purposes, ranging from environmental protection to hydropower production. The presence of multiple objectives is common also in other ﬁelds, and prompted the development of Multi-Attribute Value Theory (Keeney and Raiﬀa, 1976) to deal with the case when the multiple ob-jectives are irreducible to a single measure. Prescriptive advices to the decision maker are derived by solving the following,

ut =arg max h

J1, J2, . . . , Jni, (1.3) where Ji_{measures each incommensurable objective. This latter} formula-tion, however, does not lead to a single optimal solution. Rather, in the multi-objective context the single optimal solution is replaced by a set of Pareto optimal solutions, where each alternative represents a different tradeoff between the considered objectives and, thus, a different behavior. An alternative is said to be Pareto optimal if no improvements can be achieved according to one objective without the detriment of another. In a typical decision-making context, this tradeoff represents the outcome of a negotiation process with the stakeholders (e.g., Swartz, 2006) and is ex-pected to ensure a fair resource allocation among the competing demands. However, when the formulation is adopted to model decisions and not to prescribe them, the dm’s behavior is not univocally defined but strongly depends on the relative importance assigned to the different operating objectives, namely his or her set of preferences. Preferences are especially tricky if one considers that no fully rational procedure can select a single alternative (see the Arrow’s impossibility theorem; Arrow, 1950). This issue leads to the development of various Multi Criteria Decision Aiding methods (Roy and Vanderpooten, 1996) to provide prescriptive support to dms, such as promethee (Behzadian et al., 2010). However, these methods have no descriptive capabilities, and cannot be included in a model of decision-making.

Time dynamics: A utility function is not a static behavioral attribute (Guiso

et al., 2013). Rather, it may evolve in time when exposed to changing external forcing (e.g., extreme drought or ﬂood events), which can make the decision less satisfactory in one or more operating objectives. By

(33)

reacting to this imbalance, the dm’s preferences shift towards a new equi-librium to reduce the frequency of unsatisfactory system states, bringing the short-term performance closer to the one expected on the long-term (Simpson et al., 2016). For example, an extreme flood event may raise concerns about flood risk (e.g., Haasnoot and Middelkoop, 2012; Baldas-sarre et al., 2013a; Viglione et al., 2014), and suggests to increase dike heights or to enlarge the flood pool for augmenting the reservoir buffering capacity. On the contrary, prolonged and intense drought events may amplify human sensibility towards water scarcity (e.g., AghaKouchak et al., 2014), promoting an increase in the efficiency of the water supply system by modernization of the infrastructure and by more effective hedging strategy. Economic studies of dynamic choice problems label as naïve or myopic those dms who are unaware of their future changing preferences (Hammond, 1976). In fact, the key issue here is how long the horizon taken in consideration by the dm is.

To recap, the use of a normative approach in modeling decision-making is limited by the assumption of full rationality. Institutional decisions or average behaviors of groups of individuals (e.g., group of farmers) can overcome some of the limitations (Giuliani et al., 2016c), but the modeling still requires a utility function J that embeds both biased assessment of risky outcomes, and the identiﬁcation of preferences among competing objectives, and that might evolve in time. On top of this, utility functions are sensitive information, which the dm might want to strategically hide (Hershey et al., 1982; Cameron et al., 2002), although techniques for interviewing stakeholders and assessing their stated preferences do exist, (e.g., Andreopoulos et al., 2015).

1.2.2 Models of decision-making

As seen, neither descriptive nor normative approaches are free of issues, and models that can be found in the scientiﬁc literature locate on a continuous line connecting the two, according to the research question and to the mod-eler’s background (Schreinemachers and Berger, 2006). Here we present some examples from the water related literature.

Decision-making according to the descriptive approach can be rendered via

explicitmodels, such as decision trees, often used in agent-based models. For

in-stance, Hu et al. (2017) use directed information graphs and boosted regression trees to identify behavioral rules from a database of decisions. Other predefined rules are found in Ali et al. (2017) to represent a urban water manager, and in Schaefli et al. (2007) to represent an hydropower reservoir operator. Prede-fined rules are also embedded in simulation software and distributed models, such as the rule-curve used to model dam operations in hydrological models,

(34)

e.g., topkapi-eth (Finger et al., 2012; Fatichi et al., 2013), or the pool zones

in hec-ResSim (Wurbs, 2005). They may introduce relevant simplifications: Majone et al. (2016) calibrate a monthly-constant release rule for hydropower reservoirs over 6 years, and then use this rule to validate a distributed geo-morphological model (geotransf) over 30 past years, and to assess alterations induced by climate change over 40 future years. However, these simplifications may be acceptable when the spatial and temporal scale of analysis is very large (Kao et al., 2015). A more flexible approach to calibration against decision data often leads to black-box models, whose internal structure might be difficult to analyze. Data mining approaches are used to extract knowledge from data: Bessler et al. (2003) use an induction tree technique to extract an operational rule from optimization results, while fuzzy logic is used in Macian-Sorribes and Pulido-Velazquez (2016) to construct an operating rule embedded in a Decision Support System (dss) based on historical operations. Corani et al. (2009) adopt a lazy learning with variable time step to model the management of Lake Lugano (CH) from 1982 to 2002.

On the other hand, the normative approach is adopted to produce implicit decision-making models, where the focus lies on the utility function. In an agent-based model, Giuliani et al. (2016c) simulate farmer’s cropping decision by maximizing a crop yield function, and validate the result according to the cropping pattern across the overall study area. Implicit models can be used to assess decision-making under different conditions and to show how man-agement can cope with the variability of new conditions (Minville et al., 2010; François et al., 2015). For example, hydropower operators take decisions by maximizing the resulting revenue for the energy company or by minimizing the deficit with respect to an energy demand. Energy price and demand, and the hydrology will change in the future, while the overall objectives of the companies will very likely remain the same, i.e., the hydropower operators will continue maximizing the revenue or satisfying the demand (e.g., Anghileri et al., 2011; Gaudard, 2015). Moreover, normative models are computationally efficient, and may be applied to regional case studies (Madani and Lund, 2010; Pereira-Cardenal et al., 2014) or even global ones (Turner et al., 2017).

1.3 Thesis objectives

Within this scattered picture, we propose a data-based behavioral model in-spired by normative approaches, adopting the methodology depicted in Fig-ure 1.4. The model aims at reproducing the decision-making process that underlies the daily operations in a chns, by combining how the preferences of the system operator shape the evolution of the physical system at small temporal scales. Although the methodology can be applied to most natural resources

(35)

1.3. Thesis objectives

1: STATIC MODEL OF TRADEOFF SELECTION

2: DYNAMIC MODEL OF TRADEOFF EVOLUTION

EXOGENOUS TRIGGER:

switch

change in the boundary conditions, e.g, energy market structure, is reflected in the

ENDOGENOUS TRIGGER:

drift

The decision maker adapts the tradeoff, e.g. because of reaction to the performance

OBJECTIVE FUNCTIONS

goal modelling of chapter 3

TRADEOFF IDENTIFICATION

goal modelling of chapter 3 DOMAIN EXPERTISE

goal modelling of the reservoir control system

corresponds to a reverse engineering of decision support systems section 2.1.3

OPERA

TOR’S BEHA

VIORAL MODELING

goal advance algorithms for operator’s behavioral modeling

according to the normative approach

1.a assess rival framings for decomposing the utility function 1.b identify tradeoff among the objectives by mixing computer

science paradigms with water resources management

1.b.1 reinforcement learning and its inverse

1.b.2 multiagent systems and negotiation protocols

goal capture the dynamic evolution of tradeoff 2.a in response to exogenous triggers 2.b in response to endogenous triggers

  §2.2 §2.3 §2.3.1  §2.3.2 §2.4 §2.4.1

OPERA

TOR’S BEHA

VIORAL MODELING

1: MODELING

STATIC PREFERENCES

1.b.2 data-based tradeoff identification with multilateral negotiations  ⁊ _{literature material} collection on multilateral negotiation protocols ⁊ design of a negotiation

protocol for tradeoff identification 

⁊ _{Parametric tradeoff} identification: new

negotiation protocol (SEC)

⁊ _{Application to test case} study 2.a analysis of operations in response to exogenous triggers   

⁊ _{selection among rival} framings

⁊ _{modeling operations}

before the trigger

⁊ modeling operations after

the trigger 

⁊ Model of Premadio’s

operations in the

regulated energy market ⁊ Model of Premadio’s

operations in the free energy market

2: MODELING 

DYNAMIC PREFERENCES

2.b modeling evolving operations in response to endogenous triggers  ⁊ repeated tradeoff identification with SEC negotiations

⁊ _{retrospective calibration of}

a behavioural parameter 

⁊ Modeling approach for time-varying preferences ⁊ Dynamic model of

preferences based on the

availability bias 1.b.1 data-based tradeoff identification with inverse reinforcement learning  ⁊ _{literature material} collection on inverse reinforcement learning ⁊ _{adaptation of CPIRL as a} non-parametric model of tradeoff identification 

⁊ Selection among rival framings

⁊ Non-parametric tradeoff

identification: via inverse reinforcement learning 1.a compose rival

framings of candidate objective functions  ⁊ _{turning multi-objective} normative design of operations to descriptive modeling  ⁊ Framework to move

beyond full rationality

with normative approaches

1: MODELING

STATIC PREFERENCES

2: MODELING 

DYNAMIC PREFERENCES

GOAL  RESEARCHSTAGES  OUTPUT

Figure 1.4: Simpliﬁed overview of methods for modeling static and dynamic operator’s behavior with reference

to the sections where they are described.

systems, all the applications developed in the following regard water resources management. In this context, the proposed model aims at reproducing the daily operations, such as the release from a water reservoir. This decision has an immediate eﬀect on the resource, that can be measured and for which data are available to calibrate the model. In the following, we interchangeably use dm and operator to refer to the agent who takes this kind of decisions: according to the classiﬁcation of Ostrom et al. (2009), they correspond to the operational decisions (see Figure 1.3).

By adopting the normative approach, our methodology is affected by the four issues of dealing with value under risk, bounded rationality, multiple objectives, and time dynamics. The first two issues are less relevant in the context of managing chns where multiple stakeholders are affected by the frequent operations. On one hand, stakeholders may interact in the higher

(36)

arenas of decision-making to enforce the dm’s daily operations with rules that prevent significant deviations from efficient system performance. On the other, dss are often adopted to preserve the value of natural resources, and assist the dm in considering a large range of possible decisions and their effects. Therefore we assume that the issues of value under risk and bounded rationality are not the prominent obstacle to model daily chns operations.

Instead, the presence of multiple objectives strongly limits the development of behavioral models. In this case, the overall utility of chns operations de-pends on achieving multiple objectives. We propose a normative model of the following type,

ut =arg max J =arg max

∑

i

wiJi, (1.4)

where utis the vector of decision, and wiis a numerical weight that captures dm’s preference over the objective Ji_{. We devote a signiﬁcant part of this} disser-tation to discuss how the modeler can identify the objectives J of a real operator (1.a in Figure 1.4). In particular, we acknowledge that objective formulation is an uncertain issue and we adopt the concept of rival framings to deal with this uncertainty (Quinn et al., 2017a). Rival framings indicate alternative formula-tions, in this case of the objectives, that are likely to represent the goal of the chns operations.

1.3.1 Identifying tradeoffs among objectives with preferences

Having deﬁned the operator’s objectives, we focus on how the vector w of weights can be calibrated to represent his or her tradeoﬀ (step 1.b in Figure 1.4). We take advantage of the historical time series of his or her actions to perform the calibration, thus avoiding explicit elicitation of preferences. In this way, we also evade Arrow’s impossibility theorem to directly represent the (not fully rational) dm’s assessment of relative importance among objectives. The methodology is resumed by the following equation,

wE =arg min w {ˆrt+1} H t=0− {rt+1}tH=0 , (1.5)

where wE_{represents the tradeoﬀ of the historical operations}_{_r

t+1}tH=0, while

{ˆrt+1}tH=0are the releases obtained by simulating the optimal policy for w over the historical horizon t ∈ [0, H). We introduce two methodologies to

solve the problem (1.5), by taking inspiration from two paradigms of computer science, namely Reinforcement Learning and Negotiation protocols.

Reinforcement Learning (rl) is based on Markov Decision Process (mdp) to model decision problems and prescribe optimal decisions (Sutton and Barto, 1998). Rl has been only relatively explored in the water domain to inform the

(37)

1.3. Thesis objectives

operation of water infrastructures (Bhattacharya et al., 2003; Lee and Labadie, 2007; Castelletti et al., 2010; Rieker and Labadie, 2012; Castelletti et al., 2013a). Here, we reverse engineer the approach to identify the unknown vector of preferences represented by w that leads to the historical decisions, and map this problem into the class of Inverse Reinforcement Learning (irl). We adapt a newly developed algorithm named Cutting-Plane Inverse Reinforcement Learn-ing (cpirl) (step 1.b.1 of Figure 1.4) that identifies the vector w of weights for which the corresponding optimal decisions resemble the historical operations. Another approach to identify the tradeoff as a parametric problem adopts negotiations protocol. In the computer science literature, negotiation frame-works are traditionally designed to model autonomous agents that competi-tively bargain to identify the solution of a cooperative problem (e.g., Rubinstein, 1982), with several applications also developed in environmental problems (e.g., Frisvold and Caswell, 2000; Thoyer et al., 2001; Šauer et al., 2003; Madani, 2011, 2013). We contribute a new Set-based Egocentric Concession protocol (sec) (step 1.b.2 of Figure 1.4) that identifies the tradeoff as a function of a set of parameters, named attitudes, which can be subsequently used to represent the operator’s biases related to the temporal evolution of the tradeoff.

1.3.2 Modeling dynamic preferences

As in the real world example of Lake Como, water resources systems and other chns constantly evolve, as boundary conditions and internal components change. This time dynamics is an obstacle for the adoption of the normative approach to construct behavioral models of system operators. In fact, if the util-ity function evolves over time, the tradeoﬀ expressed by the weights wi_would no longer represents the dm’s choice. To address this case, the problem (1.4) is extended as

ut =arg max J =arg max

∑

i

wi_tJi, (1.6) where the vector wtof weights is allowed to vary with time. We distinguish two cases with respect to the modeling steps required (Janssen et al., 2007). In case that preferences evolution arises in response to changes in the system boundaries, stationarity of the operations model is falsified. This is what we call an exogenous change, and we repeat the tradeoff identification problem to compose two models of the dm, corresponding to operations pre- and post-change (step 2.a of Figure 1.4).

However, the trigger for the time evolution can arise within the system boundaries, as the variability of inflows influenced the tradeoff in the example of Lake Como (Section 1). This case, which we call endogenous, develops within the chns, and thus, the modeling domain. All the elements for modeling this process are available, so in the last step of this thesis we propose a dynamic

(38)

model of preferences that drives the tradeoﬀ evolution. Here, preferences are linked with recent system performance, as they are measured by the objective functions, and with a representation of the dm’s internal state that captures the irrational biases. This model is then calibrated over historical data to identify the parameters that best match a given dm (step 2.b in Figure 1.4).

1.4 Overview

Figure 1.5 summarizes the main contributions of this PhD thesis to compose a framework that turns multi-objective, prescriptive approaches into a descrip-tive tool: (1.a) the application of the rival framing concept to deal with the uncertainties in objective formulation; (1.b.1) a non-parametric method for tradeoff identification based on irl; (1.b.2) an alternative parametric method based on a new agent-based negotiation protocol; (2.a) the analysis of a real case study where a change of the operations tradeoff is triggered by an exogenous factor; and finally, (2.b) a dynamic model that embeds the tradeoff evolution within the chns as caused by the recent system performance.

The rest of the dissertation is organized as follows:

Chapter 2 introduces the problem of identifying an optimal operating

pol-icy with a speciﬁc focus on water resources systems, and presents the methods that are adopted in this dissertation, namely Stochastic Dynamic Programming (sdp) and Direct Policy Search (dps), in Section 2.1. In Sec-tion 2.2, we discuss how to select objectives formulaSec-tion and compose rival framings that possibly capture the dm’s actual goals. Then, we formal-ize the problem of preference calibration on the time series of decisions in Section 2.3, and we propose the use of irl algorithms, and especially of cpirl, in Section 2.3.1. We also describe an alternative approach based on multilateral negotiation protocols in Section 2.3.2, where the sec is intro-duced. Finally, we analyze the issue of dynamic preferences in Section 2.4, and we introduce a new model of preference evolution in Section 2.4.1 to link recent system performances to the dm’s choice of operating policy.

Chapter 3 introduces the case studies of the two applications. The ﬁrst case

study is presented in Section 3.1: it is a numerical model of a reservoir inspired by Lake Como, northern Italy, that serves as a controlled envi-ronment to test the negotiation-based method for tradeoff identification. Generators of synthetic inflows and regulation are shown in Section 3.1.2, as they are used to create a scenario of non-stationary operations that are subsequently used. The second case study, in Section 3.2, regards the management of a hydropower reservoir in the Italian Alps, named Premadio, during the transition from regulated to unregulated energy

(39)

1.4. Overview

market. In Section 3.2.1, we formalize the design of optimal operating policies for this reservoir, while in Section 3.2.2, we detail the procedure to devise an appropriate inﬂow model.

Chapter 4 then introduces the ﬁrst application. In Section 4.1, we present

few use cases to discuss how cpirl can identify a vector of weights that best represent the dm’s preferences. This methodology is then applied to the case study of Premadio in Section 4.2, in order to devise a behavioral model of the operator. Then, we show that this model gets unvalidated by the transition from regulated to unregulated energy market in Sec-tion 4.2.2, and how the re-calibraSec-tion of the behavioral model highlights the response of reservoir management to the exogenous change. Con-tent of this chapter will be included in a paper under preparation for Environmental Modelling & Software (Mason et al., 2017b).

Chapter 5 introduces the application of the new sec protocol in Section 5.1,

ﬁrst to identify a static set of preferences in Section 5.1.1, and then to model evolving preferences in Section 5.1.2. In Section 5.2, we produce a brief discussion of the connection between sec protocol and Arrow’s impossibility theorem, and of its Pareto optimality in long term scenario. Part of this chapter is adapted from a publication currently under review for Water Resources Research (Mason et al., 2017a).

Beyond full rationality: modeling tradeoff dynamics in multi-objective water management

BEYOND FULL RATIONALITY:

MODELING TRADEOFF DYNAMICS

IN MULTI-OBJECTIVE WATER MANAGEMENT

From tradeoff identification to modeling its time evolution

Abstract

A

Sommario

L’

Acknowledgements

Contents

List of Figures

List of Tables

Acronyms

1

Introduction

A

1.1 The wicked problem of modeling human decisions

CHNS

H

N

CONSTITUTIONAL

COLLECTIVE

OPERATIONAL

CHNS

H

N

CONSTITUTIONAL

COLLECTIVE

OPERATIONAL

1.2 Embedding water operators’ behavior into Coupled

Human-Natural Systems models

1.3 Thesis objectives

1: STATIC MODEL OF TRADEOFF SELECTION

2: DYNAMIC MODEL OF TRADEOFF EVOLUTION

EXOGENOUS TRIGGER:

switch

ENDOGENOUS TRIGGER:

drift

OBJECTIVE FUNCTIONS

TRADEOFF IDENTIFICATION

OPERA

TOR’S BEHA

VIORAL MODELING

OPERA

TOR’S BEHA

VIORAL MODELING

1: MODELING

STATIC PREFERENCES

2: MODELING

DYNAMIC PREFERENCES

1: MODELING

STATIC PREFERENCES

2: MODELING

DYNAMIC PREFERENCES

∑

∑

1.4 Overview

2: MODELING 

2: MODELING 