Department of Electronics, Information, and Bioengineering Doctoral Program In Information Technology
ADVANCING JOINT DESIGN AND OPERATION
OF WATER RESOURCES SYSTEMS
UNDER UNCERTAINTY
Doctoral Dissertation of: Federica Bertoni Advisor:
Prof. Andrea Castelletti Co-Advisor:
Dr. Matteo Giuliani Prof. Patrick M Reed Tutor:
Prof. Luigi Piroddi
The Chair of the Doctoral Program: Prof. Barbara Pernici
the primary reason for feedback in control is uncertainty.
G
lobally, many countries are actively seeking to maximize the hydropower potential of major river basins, yielding proposals for constructing ap-proximately 3,700 major dams in the near future. The planning of new water reservoir systems raises several major challenges that must be conjunc-tively accounted for within the system design phase, namely (i) potentially con-flicting and heterogeneous objectives; (ii) interdependency between dam size and operations; (iii) future uncertainties in the main external drivers (e.g., cli-mate, human demands); and (iv) vast amount of information that is becom-ing increasbecom-ingly available to system planners at different temporal and spatial scales. Such issues must be jointly addressed through novel, integrated approaches in order to design efficient yet sustainable infrastructures able to satisfy multi-ple water needs and perform well under a wide range of external future changes. Building on these research challenges, the main goal of this thesis is to ad-vance the current planning and operation of water reservoir systems, focus-ing on the couplfocus-ing of dam sizfocus-ing and operation design in order to thoroughly capture their interdependencies also with respect to uncertainty in the main external drivers. In addition, the role of exogenous information (e.g., stream-flow forecasts) in dam design is investigated to further analyze how dam design is shaped by information feedbacks. We contribute novel methodological ap-proaches as the primary outcome of our research, which have been developed by extending and integrating existing optimization techniques traditionally ap-plied to the water management field in order to additionally account for the planning dimension of the problem, cover all the challenges of current plan-ning and operation of water resource systems, and eventually provide support-ing tools to water system planners intended to design water reservoir systems in complex, highly uncertain decision making contexts.The first outcome of this research is a novel Reinforcement Learning (RL)-based approach to integrate dam sizing and operation design, while signifi-cantly containing computational costs with respect to alternative state-of-the-art methods. Our approach is tested on a numerical case study, where the wa-ter infrastructure must be sized and operated to meet downstream users wawa-ter
signs with respect to uncertainties in the main external drivers is addressed, by developing a robust dam design framework that jointly considers sizing and operations while explicitly accounting for key human and hydro-climatic un-certainties. Bridging the Multi-Objective Robust Decision Making and Evolu-tionary Multi-Objective Direct Policy Search methods, our dam design frame-work jointly optimizes planning (i.e., dam size) and management (i.e., reser-voir release policy) in a single optimization process. In the end, we explore the added benefits of including valuable information, and in particular streamflow forecasts, during the optimal dam design phase to identify more efficient sys-tem configurations. Building on the robust dam design framework mentioned above to solve coupled dam sizing and operation design problems, in this third contribution we want to assess whether searching for more flexible operating policies informed by streamflow forecasts allows to design smaller reservoir sizes with respect to solutions that do not rely on forecast information. The potential of the last two contributions is demonstrated through an ex-post de-sign analysis of the Kariba dam in the Zambezi river basin.
Part of this research has appeared (or has to appear) in the following journal publications:
• Bertoni, F., Giuliani, M., Castelletti, A., 2019c. Integrated design of dam size and operations via Reinforcement Learning. Journal of Water Resources Planning and Management (in press)
• Bertoni, F., Castelletti, A., Giuliani, M., Reed, P. M., 2019a. Discovering de-pendencies, trade-offs, and robustness in joint dam design and operation: An ex-post assessment of the Kariba dam. Earth’s Future (in press) • Bertoni, F., Castelletti, A., Giuliani, M., Reed, P. M., 2019b. Informed
wa-ter infrastructure design: Improving coupled dam sizing and operation by streamflow forecasts. Water Resources Research (under review)
G
lobalmente, molti paesi stanno attivamente cercando di massimizzare il loro potenziale idroelettrico, dando vita a proposte per la costruzione di circa 3,700 grandi dighe nel prossimo futuro. La pianificazione di nuovi sistemi di dighe solleva molteplici sfide che devono essere considerate congiuntamente, tra cui (i) obiettivi potenzialmente in conflitto ed eterogenei; (ii) interdipendenza tra dimensione della diga e relativa gestione; (iii) incertez-ze future nei principali driver esterni (ad es., clima, domanda); e (iv) una gran-de quantitá di informazioni che stanno diventando sempre piú disponibili ed accessibili dai pianificatori di sistemi idrici su diverse scali temporali e spazia-li. Questi elementi devono essere considerati congiuntamente attraverso nuovi approcci integrati al fine di progettare infrastrutture efficienti e sostenibili in grado di soddisfare molteplici esigenze idriche ed essere robuste rispetto ad una vasta gamma di cambiamenti futuri.Partendo da queste sfide di ricerca, l’obiettivo principale di questa tesi é for-nire un contributo all’attuale pianificazione e gestione dei sistemi di dighe, con-centrandosi sull’accoppiamento del dimensionamento di dighe e relativa gestio-ne al figestio-ne di catturare a fondo le loro interdipendenze anche rispetto all’incer-tezza nei principali driver esterni. Inoltre, il ruolo di informazioni esogene (ad es., previsioni di flusso) nella progettazione di dighe é investigato per analizzare ulteriormente come il loro design possa essere modellato attraverso gli effetti di tali informazioni. In particolare, questa tesi contribuisce allo sviluppo di nuo-vi approcci metodologici, che sono stati snuo-viluppati estendendo e integrando le tecniche di ottimizzazione esistenti applicate tradizionalmente al campo della gestione delle risorse idriche al fine di includerne il problema di pianificazione, risolvere tutte le sfide della pianificazione e relativa gestione ed infine forni-re strumenti di supporto ai pianificatori volti a progettaforni-re sistemi di dighe in contesti decisionali complessi e altamente incerti.
Il primo risultato di questa ricerca é un nuovo approccio basato sul Reinfor-cement Learning (RL) per integrare il dimensionamento delle dighe e la rela-tiva gestione, contenendo in modo significativo i costi computazionali rispet-to a merispet-todi giá esistenti nello starispet-to dell’arte. Il nostro approccio é testarispet-to su
per soddisfare la domanda idrica degli utenti a valle minimizzando i costi di costruzione. In secondo luogo, la robustezza delle dighe rispetto ad incertezze nei principali driver esterni é analizzata attraverso lo sviluppo di un framework robusto per la progettazione di dighe, che permette di risolvere congiuntamen-te il problema di dimensionamento e relativa gestione, considerando esplicita-mente l’incertezza nei driver socio-economici e climatici esterni. Combinan-do Multi-Objective Robust Decision Making ed Evolutionary Multi-Objective Direct Policy Search, il nostro framework identifica congiuntamente la pianifi-cazione (dimensione della diga) e la gestione (politica di gestione) in un singolo processo di ottimizzazione. Infine, esploriamo i vantaggi aggiuntivi di inclu-dere informazioni rilevanti, in particolare previsioni di afflusso, durante la fase di progettazione delle dighe per identificare configurazioni di sistema piú effi-cienti. Basandoci sul framework robusto per la progettazione di dighe sopra-citato, in questo terzo contributo vogliamo valutare se la ricerca di politiche di gestione piú flessibili informate da previsioni di afflusso consenta di progettare dighe piú piccole rispetto a tecniche che non fanno affidamento su alcun tipo di previsione. Il potenziale degli ultimi due contributi é dimostrato attraverso un’analisi ex post della diga di Kariba nel bacino del fiume Zambezi.
Parte della ricerca presentata in questa tesi appare (o apparirá prossimamen-te) nelle seguenti pubblicazioni scientifiche:
• Bertoni, F., Giuliani, M., Castelletti, A., 2019c. Integrated design of dam size and operations via Reinforcement Learning. Journal of Water Resour-ces Planning and Management (in press)
• Bertoni, F., Castelletti, A., Giuliani, M., Reed, P. M., 2019a. Discovering de-pendencies, trade-offs, and robustness in joint dam design and operation: An ex-post assessment of the Kariba dam. Earth’s Future (in press) • Bertoni, F., Castelletti, A., Giuliani, M., Reed, P. M., 2019b. Informed
wa-ter infrastructure design: Improving coupled dam sizing and operation by streamflow forecasts. Water Resources Research (under review)
1 Introduction 1
1.1 Past, present and future of hydropower systems . . . 1
1.2 Challenges and opportunities of modern planning and opera-tion of water resources systems . . . 2
1.3 Thesis motivation and objectives . . . 4
1.4 Research methods . . . 5
1.5 Thesis outline . . . 5
2 Case studies 9 2.1 Zambezi river basin . . . 10
2.1.1 Kariba dam . . . 12
2.1.2 Model description . . . 13
2.2 Synthetic regulated lake . . . 14
2.2.1 Model of the system . . . 15
3 Reinforcement Learning for designing water reservoirs 17 3.1 Introduction . . . 18
3.2 Methodology . . . 21
3.2.1 Problem formulation . . . 21
3.2.2 Traditional dam design . . . 23
3.2.3 Nested dam design . . . 24
3.2.4 Novel RL dam design . . . 25
3.3 Computational experiments . . . 28
3.4 Results . . . 31
3.4.1 Least cost planning vs Novel RL dam design approach . 31 3.4.2 Influence of dam size and objectives trade-offs on system dynamics . . . 32
3.4.3 Nested vs Novel RL dam design approach . . . 33
3.4.4 Computational costs . . . 37
4 A novel robust assessment framework for designing water reservoir
systems 41
4.1 Introduction . . . 42
4.1.1 Traditional design frameworks . . . 43
4.1.2 A robust design framework . . . 45
4.2 Methodology . . . 45
4.2.1 Alternatives generation . . . 48
4.2.2 States of the world sampling . . . 53
4.2.3 Robustness measures . . . 55
4.2.4 Robustness controls . . . 55
4.3 Computational experiments . . . 55
4.4 Results . . . 56
4.4.1 Baseline forensic solution . . . 56
4.4.2 Design and operational trade-offs based on historical records 57 4.4.3 Trade-offs under stationary hydro-climatic variability . 61 4.4.4 Influence of dam size and operational preferences on re-lease decisions . . . 64
4.4.5 Assessing robustness for changing demands and hydrology 67 4.4.6 Factor mapping of exogenous variables driving system robustness . . . 69
4.5 Discussion and final remarks . . . 71
5 Role of information in designing water reservoir systems 75 5.1 Introduction . . . 76
5.2 Methodology . . . 78
5.2.1 Forecasts generation and selection . . . 79
5.2.2 Design optimization . . . 81
5.2.3 Forecast value . . . 82
5.3 Computational experiments . . . 83
5.4 Results . . . 86
5.4.1 Influence of dam size on the Expected Value of Perfect Information . . . 86
5.4.2 Informed infrastructure design using perfect seasonal fore-casts . . . 87
5.4.3 Informed infrastructure design using perfect inter-annual forecasts . . . 91
5.4.4 Informed infrastructure design using realistic forecasts . 94 5.5 Discussion and final remarks . . . 97
A Appendix A 105
A.1 Gaussian RBFs vs Piecewise linear operating policies . . . 106
A.2 Diagnostic verification step . . . 108
A.3 Runtime evolution of the Borg MOEA search . . . 109
A.4 Net Present Value . . . 110
A.5 Evolution of the nominal hydraulic head with dam size . . . 111
B Appendix B 113 B.1 Runtime evolution of the Borg MOEA search . . . 114
B.2 Iterative Input Selection for seasonal and inter-annual forecasts 115 B.3 System dynamics under Informed Infrastructure Design . . . . 118
B.4 System dynamics under over-estimated forecasts . . . 120
B.5 Sensitivity analysis with respect to forecasts Pbias . . . 121
1
Introduction
1.1
Past, present and future of hydropower systems
Hydropower has been employed as the first renewable energy source for elec-tricity generation back in the 19th century and today it still plays a major, mul-tidimensional role in the electricity sector worldwide for a variety of reasons. Firstly, it is a clean and renewable source of energy that generates local, afford-able power fostering sustainafford-able development, as promoted under the Sustain-able Development Goals (SDGs) (United Nations, 2019). Secondly, it allows to reduce dependence upon imported fuels, associated to high risks of price volatility and supply uncertainty. Then, hydropower dams can offer multiple co-benefits, from storing water for drinking and irrigation, to being used for drought-preparedness, flood mitigation and recreation. In the end, hydropower is very competitive with other electricity sources from a costs point of view and provides a rapid-response when intermittent energy sources (e.g., solar) are off-line (Cole et al., 2014; IFC, 2015).
Hydropower is currently responsible for about 16% of global electricity pro-duction, a percentage that is projected to substantially increase due to the dou-bling of the total installed hydropower capacity expected by 2050 (IEA, 2012). Since developed countries already exploited more than 50% of their hydropower potential, most of the future hydropower expansion is predicted to occur in de-veloping countries, which still present a vast untapped potential. Among oth-ers, Africa represents an extreme case with its almost 90% of undeveloped hy-dropower potential, with respect to a 25% global exploitation rate on average (Cole et al., 2014; Berga, 2016). This has motivated potential investments in the construction of approximately 3,700 new dams in the near future (Zarfl et al.,
2015), a large share of which will be built in Africa, Asia and Latin America (IEA, 2012). For example, in the Amazon, Congo and Mekong river basins, character-ized by a limited hydropower development over the last decades mainly due to limited infrastructures and low energy demands, more than 450 new dams are planned in the next years, with many already under construction. The Amazon will experience the largest total number of new dams in South America, with 334 planned dams compared to the 416 existing/under construction, leading to significant losses of fish biodiversity, human settlements and deforestation that will likely exceed potential benefits derived from increased energy supply and jobs creation (Zarfl et al., 2015; Winemiller et al., 2016). As for the Mekong basin, more than 25% of the existing/under construction dams are planned for the future, with plans for at least 11 new impoundments on the middle and lower river reaches. There, dam construction will disconnect the Mekong delta from its sources of sediments, interrupt fish migrations and therefore cause fishery losses, requiring substantial agricultural areas expansion to guarantee regional food security (Orr et al., 2012; Schmitt et al., 2018). In the end, the Congo, which presents only 15% of the Amazon and Mekong total number of dams, highly relies on hydropower production with 13 planned dams and 51 existing/under construction. These water infrastructures will impound about 83% of the annual Congo river discharge and produce energy to be mostly ex-ported (Brunn, 2011).
In the future, changes in water availability and extreme events (e.g., droughts) due to climate change coupled with high rates of population growth will con-tribute to an increase in both migration rates within a single region, as well as energy and food demands, putting additional pressures on already stressed wa-ter resources. Globally, both existing and planned dams will thus have to face a vast array of future challenges, such as water scarcity, and growing resource conflicts in their demands (e.g., hydropower production vs irrigation water sup-ply).
When planning new dams, integrated, strategic approaches must be there-fore employed to find a balance between key economical, social and environ-mental objectives, while accounting for different water users and changes in external conditions that might strongly impact water resources systems in the future.
1.2
Challenges and opportunities of modern planning
and operation of water resources systems
The planning of large dams traditionally consists in basin-wide assessments of the potential economic outcomes of different designs via financial metrics (e.g., net present value) to evaluate their corresponding financial value (e.g., Jeuland,
2009; Ray et al., 2011). This approach combines costs and monetized down-stream impacts of large water infrastructures into a single aggregate monetary value, disregarding potentially conflicting objectives and trade-offs among dif-ferent water users within the basins of focus. In addition, most of the aggregated cost functions consider hydropower revenue and capital/operating costs only. For example, the Sambor dam on the Mekong river was originally located and designed to maximize hydropower production, disregarding the severe chal-lenges posed to fish migration and sediments passage needed to stabilize the Mekong delta (Wild et al., 2018). Since decisions taken with respect to a single objective usually perform poorly when evaluated over multiple objectives, the planning of large dams should always account for trade-offs among multiple objectives (Kareiva, 2012).
Secondly, over the last fifty years the interdependency between dam size and operations has been largely neglected by traditional engineering approaches relying on the widespread Rippl method (Rippl, 1883), aimed at identifying a single optimal dam size based on a sequence of pre-defined releases and ob-served inflows (U.S. Army Corps of Engineers, 1975, 1977). More recently, novel approaches able to jointly consider dam size and operations have been developed. By carefully considering alternative operating schemes, these meth-ods successfully explore the interdependent impacts of different short-term op-erating strategies on the long-term system design (Tian et al., 2018), avoiding the over-sized, under-performing dams designed via traditional engineering approaches (Moran et al., 2018). The advantages of explicitly accounting for the interdependency between dam size and operations are particularly evident for large dam systems, characterized by a greater operational flexibility.
Third, the long design life of large dams critically exposes them to future un-certainties related to climatic and socio-economic changes. Yet, their planning is usually performed assuming stationarity in the long-term natural processes and without accounting for uncertainty in the external drivers. This is partic-ularly true for new African dams, which are being constructed without assess-ing how climate change will affect them and regardless of many existassess-ing hy-dropower dams that are already experiencing power shortages due to extreme drought events (Imhof and Lanza, 2010). For example, Burundi, Congo, and Rwanda are planning the Ruzizi III dam along their shared border, although they are all suffering from power cuts due to very low river levels (Imhof and Lanza, 2012). In 2013 in Brazil, where extreme climate events are expected to cause a 36% reduction in hydropower production in the near future, water reservoirs reached their lowest levels since 2001 (Prado Jr et al., 2016). Since the assumption of a stationary climate is unlikely to be valid in the future (Milly et al., 2008), uncertainties in the main external drivers must be taken into ac-count during dam planning in order to design robust infrastructures that are
able to perform satisfactorily in the future with respect to multiple sources of uncertainty.
Last but not least, the increasing number and spread of environmental mon-itoring systems (e.g., sensor networks, remote sensing, crowdsourcing) are con-tributing a vast amount of data available at different temporal and spatial scales (The Economist Editorial, 2011). However, both the planning and subsequent operation of many dams still rely on a basic set of information, typically consist-ing of basic statistical analysis of historical data and personal experience of the system planner/operator (Giuliani et al., 2015). Water system planners’ under-standing of current conditions and future evolution of water resources systems must be therefore improved by exploiting increasingly available information, which allows them to design more efficient systems and, finally, take better de-cisions.
1.3
Thesis motivation and objectives
Building on the above mentioned research challenges and opportunities in the planning and operation of water resources systems, in this thesis we contribute a set of modelling and optimization tools converging in multiple, novel inte-grated frameworks for thoroughly capturing interdependencies between plan-ning and operation in non-linear systems, also with respect to uncertainty in the main external drivers (e.g., hydro-climatology, human demands) and the wide range of exogenous information that is becoming increasingly available. The main focus is on water resources systems and specifically on coupling dam siz-ing and operation design. The main goal of this thesis is therefore to develop novel methodological approaches for optimally designing water reservoir sys-tems, namely methods that advance the current dam sizing and operation de-sign of water reservoir systems and that are:
• multi-objective;
• capable of joint planning and operation, capturing the interdependencies between dam size and the associated trade-offs across candidate operating policies;
• integrated with state-of-the-art stochastic optimization, yielding system configurations that are less vulnerable to intrinsic, stationary hydro-climatic variability;
• directly accounting for robustness to long term deep uncertainties (i.e., the probability of occurrence of uncertain future conditions is unknown and a set-membership description is the only information available; Lempert and Schlesinger (2000); Lempert (2002); Dessai et al. (2009)), clarifying
how alternative system configurations perform with respect to uncertain drivers (e.g., inflows and demands);
• explicitly considering the most valuable sources of information upon which to condition system design, exploring how joint dam sizing and operations frame the value of information to improve the overall system design. There are two main innovative aspects to this research. First, we contribute novel methodological approaches, developed by extending and integrating ex-isting optimization techniques traditionally applied to the water management field in order to additionally account for the planning dimension of the prob-lem. Secondly, the set of integrated frameworks that we developed covers all the challenges of current planning and operation of water resource systems, advancing the traditional approaches commonly employed. Our comprehen-sive methodologies can be, therefore, considered as supporting tools for water system planners intended to design water reservoir systems in complex, highly uncertain decision making contexts. The following paragraphs contain further insights into the research method adopted in this thesis and the content of each chapter.
1.4
Research methods
Figure 1.1 illustrates the three main phases followed to develop this research and the corresponding methodological contributions, namely:
• Integrated dam design via Reinforcement Learning (RL); • Robust framework for dam design;
• Informed dam design by forecasts.
Here, each research stage is classified based on the challenges it addresses for the design of water resources systems, namely multiple objectives, interdepen-dency between planning and operation, uncertainty, and information (specif-ically forecasts). In addition, it is briefly presented in terms of activities, i.e., methodological research stages, and outputs, i.e., peer-reviewed publications contributing to this thesis. As regards the methodological research stages, each methodological contribution comprises different sub-tasks, ranging from the implementation of novel algorithms and integrated frameworks to the testing of their effectiveness with respect to traditional, state-of-the-art engineering approaches to dam sizing. As for the written contributions of this thesis, each of the following chapters corresponds to a specific research stage and method-ological contribution, whose tools and numerical results are discussed. The the-sis outline and further details about the content of each chapter are summarized in the next section.
RE S E A RC H S TA G E S 1. Li te ra tu re r ev ie w o n da m s iz in g te ch ni qu es 2. Al g o ri th m d e s ig n a n d i m p le m e n ta ti o n 3. De s ig n o f e x p e ri m e n ts 4. Be n c h m a rk w it h t ra d it io n a l m e th o d s 5. As s e s s m e n t o f a lg o ri th m co m put at io na l ef fi ci en cy In te g ra te d d am d e si g n v ia R L OU TP U T Ch a p te r 1: In tr o d u c ti o n Ch a p te r 3 : R e in fo rc e m e n t Le a rn in g f o r de s ig ni ng w at er re s e rv o irs PE ER -RE VI EW ED P U B LI C A TI O N S Be rt on i e t al . ( 20 19 c) RE S E A RC H S TA G E S 1. Li te ra tu re r ev ie w o n da m s iz in g 2. Ro b us t as se ss m e n t fr am e w or k im pl em en ta tio n 3. Be n c h m a rk w it h t ra d it io n a l m e th o d s 4. An a ly s is o f d a m s ize in fl u e n c e o n op e ra ti on s 5. As s e s s m e n t o f u n c e rt a in ty o n s y s te m pe rf or m a nc e Ro b us t fr am e w or k f or d am d e si g n OU TP U T Ch a p te r 1: In tr o d u c ti o n Ch a p te r 4 : A n o v e l ro b u s t a s s e s s m e n t fr a m e w o rk fo r d e s ig n in g w a te r re s e rv o ir s y s te m s PE ER -RE VI EW ED P U B LI C A TI O N S Be rt on iet a l. (2 01 9a ) RE SE A RC H ST A GE S 1. Ge n e ra ti o n o f p e rf e c t a n d r e a li s ti c fo re c a s ts 2. In fo rm a ti o n s el e ct io n o f fo re c as t le ad ti m es 3. Ba s ic s y s te m d e s ig n u n d e r b a s ic in fo rm a tio n 4. In fo rm e d s ys te m d e si g n u n de r b ot h pe rf e c t a n d re al is ti c fo re c a st s In fo rm e d d am d e si g n b y fo re c as ts OU TP U T Ch a p te r 1: In tr o d u c ti o n Ch a p te r 5 : R o le o f in fo rm a ti o n i n d e s ig n in g wa te r re se rv oi r sy st em s PE ER -RE V IE W ED P U B LI C A TI O N S Be rt on iet a l. (2 01 9b ) Fo re ca st s No i nf or m at io n P la nn in g a n d o p e ra ti o n in te rd e pe nd e n cy M ul ti p le o b je ct iv es U n ce rt ai n ty
1.5
Thesis outline
Chapter 2 In Chapter 2, we present the two application contexts of this the-sis. In order to test the effectiveness of our novel integrated frameworks, we perform an ex-post analysis of the Kariba dam in the Zambezi river basin in south-eastern Africa, mainly shared among Zambia, Zimbabwe and Mozam-bique. Here, multiple conflicting water users exist and uncertainty related to future climate and socio-economic conditions is high. On the other hand, we focus on a synthetic regulated lake to test the actual effectiveness of a novel al-gorithm for dam design that we developed, conducting experiments in a fully controlled environment.
Chapter 3 In Chapter 3, we contribute a novel Reinforcement Learning (RL)-based approach to integrate dam sizing and operation design, while signifi-cantly containing computational costs with respect to alternative state-of-the-art methods. Our approach first optimizes a single operating policy parametric in the dam size and, then, searches for the best reservoir size operated using this policy. The parametric policy is computed through a novel batch-mode RL al-gorithm, called Planning Fitted Q-Iteration (pFQI). The proposed RL approach is tested on a numerical case study, where the water infrastructure must be sized and operated to meet downstream users’ water demand while minimizing con-struction costs. Results show that our RL approach is able to identify more ef-ficient system configurations with respect to traditional sizing approaches that neglect the optimal operation design phase. However, when compared with other integrated approaches, we prove the pFQI algorithm to be computation-ally more efficient.
The content of this chapter is adapted from Bertoni et al. (2019c).
Chapter 4 In Chapter 4, we present an integrated framework to solve cou-pled dam sizing and operation design problems. The framework combines Multi-Objective Robust Decision Making and Evolutionary Multi-Multi-Objective Direct Policy Search into a novel approach to dam sizing, which internalizes the op-eration design problem and explicitly considers uncertainty in external drivers. We demonstrate the potential of this integrated dam design framework through an ex-post design analysis of the Kariba dam in the Zambezi river basin. Our results show that careful exploration of the coupled planning/operation search space yields designs that significantly outperform the existing Kariba system. Moreover, we demonstrate that our framework leads to a significant reduc-tion in capital costs (i.e., smaller reservoir sizes) while simultaneously improv-ing system robustness with respect to changimprov-ing hydro-climatology and human irrigation demands.
The content of this chapter is adapted from Bertoni et al. (2019a).
Chapter 5 In Chapter 5, we present a novel integrated framework to inves-tigate the value of streamflow forecasts in informing at the same time the de-sign of a water reservoir and its operations. Building on the robust dam dede-sign framework proposed in chapter 4 to solve coupled dam sizing and operation design problems, in this study we want to assess whether searching for more flexible operating policies informed by streamflow forecasts allows to design less costly reservoir sizes with respect to solutions that do not rely on forecast information. We demonstrate the potential of our novel framework through an ex-post design analysis of the Kariba dam in the Zambezi river basin. Results show that our framework successfully identifies the most valuable information to improve the Kariba system design, covering about 20% of the space for im-provement estimated under exact knowledge of the future and reducing capital costs by 20% while still attaining a satisfactory performance.
The content of this chapter is adapted from Bertoni et al. (2019b).
Chapter 6 In Chapter 6, we summarize the main achievements and general conclusions of this thesis, together with further developments and ideas to be implemented in future research.
2
Case studies
This chapter introduces the two case studies employed in this thesis. The first case study described in section 2.1 is the Zambezi River Basin in Southern Africa, with a particular focus on the Kariba dam, which is the largest man-made reser-voir in Africa and accounts for about 35% of the total installed hydropower capacity within the basin. The Kariba dam was built in 1960 and sized via standard design methods that assumed a pre-defined operating rule to maxi-mize hydropower production (Soils Incorporated Ltd, 2000). It is therefore a paradigmatic example of an existing water infrastructure designed via classic sizing methods, carrying all the challenges of traditional planning of water re-sources systems presented in chapter 1. An ex-post analysis of the Kariba dam is conducted in chapter 4 in order to rigorously validate the effectiveness of the contributed integrated framework with respect to classic dam design tech-niques and its potential in overcoming the challenges of traditional planning. Being a large dam with a significant active storage and an inter-annual carry-over capacity, the Kariba dam is also employed to test the effectiveness of the second integrated framework presented in chapter 5, exploring the added ben-efits of including valuable streamflow forecasts during the dam design phase to identify more efficient dam designs.The second case study described in section 2.2 is a numerical case study that builds upon the Lake Como system in Northern Italy. Since the choice of a syn-thetic case study allows to perform experiments in a controlled environment, we use it to test the novel Reinforcement Learning-based algorithm contributed in chapter 3 to integrate dam sizing and operation design.
2.1
Zambezi river basin
Draining 1.37 million km2, the Zambezi River Basin (ZRB) is the 15th-largest
basin worldwide based on drainage area extent, and the 4th largest basin in Africa. The river is shared among eight countries (Angola, Botswana, Malawi, Mozambique, Namibia, Tanzania, Zambia, and Zimbabwe), with Zambia, Zim-babwe and Mozambique encompassing nearly 70% of the entire basin, and in-habited by almost 40 million people (SADC, 2012). It originates in eastern An-gola and north-west Zambia at an almost 1,600 m asl altitude, flows south-east for 3,000 km through plains and gorges, and in the end discharges 2,600 m3/s
on average into the Indian Ocean in Mozambique (Figure 2.1).
The ZRB can be divided into three major areas based on a well-defined geo-morphological pattern: (i) the upper Zambezi from its source down to the Vic-toria Falls, where both very low gradient stretches (e.g., Barotse Floodplain) and very steep gorges coexist; (ii) the middle Zambezi from Victoria Falls down to the boundary with the Mozambican coastal plain (downstream of Cahora Bassa gorge), characterized by much steeper gorges than the upstream section before widening into the Kariba and Cahora Bassa basins; and (iii) the lower Zambezi from the Cahora Bassa gorge down to the Indian Ocean, where the lower river reach forms an alluvial delta extending 120 km inland from the Indian Ocean coast (Teodoru et al., 2015).
Both temperature and rainfall present a seasonal pattern typical of a humid sub-tropical climate, namely a rainy season from November to April (during which up to 95% of the annual rainfall occurs) and a dry season from May to Oc-tober. Even if the average annual rainfall is quite high, reaching about 950 mm, it is unevenly distributed both across the basin and within a same year, ranging from 1,400 mm/yr in the northern and eastern regions to 400 mm/yr in the western and southern parts. This seasonality causes the hydrological cycle of the Zambezi river to have a bimodal distribution with a single maximum flood peak in April/May and a minimum discharge peak in November. Many tribu-taries contribute to the annual average discharge of the Zambezi river, which is extremely variable from year to year ranging from 3,400 to 4,200 m3/s due to
the high frequency of occurrence of extreme flood and drought events (World Bank, 2010; Beilfuss, 2012). Draining a basin of over 156,000 km2that entirely
lies in Zambia, the Kafue river represents its major tributary with a mean annual discharge of 320 m3/s (Teodoru et al., 2015).
The natural course of the Zambezi has been highly altered by the construc-tion of large dams, especially in the middle and lower Zambezi regions, primar-ily designed for hydropower production (grey circles in Figure 2.1). Here, two major reservoirs have been built: the Kariba reservoir, officially commissioned in 1960, is located 170 km downstream the Victoria Falls between Zambia and Zimbabwe and represents the largest man-made reservoir in Africa with a
sur-Itezhi-Tezhi
120 MW
Victoria Falls
108 MW Kariba
2000 MW
Kafue Gorge Upper
990 MW Cahora Bassa 2075 MW Nkula Falls 122 MW Tedzani 88 MW Kapichira 132 MW
Figure 2.1:Topography of the Zambezi River Basin based on the Digital Elevation Model (DEM) derived from SRTM data. Grey circles correspond to the existing hydropower infrastructures, where the marker size is proportional to the power plant installed capacity [MW].
face area of about 5,600 km2and a total storage capacity of about 180 km3, and
the Cahora Bassa reservoir, completed in 1974 in Mozambique about 300 km downstream of Kariba, is the fourth largest reservoir in Africa with a surface area of 2,675 km2and an active storage of about 52 km3(World Bank, 2010).
Along with these two large impoundments, two other smaller dams have been constructed on the Kafue river, namely the Itezhi-Tezhi reservoir that stores water for guaranteeing constant water supply to the more downstream Kafue Gorge Upper reservoir. Besides altering the natural bimodal discharge of the Zambezi, dams construction caused the percentage of water lost for evapora-tion from large reservoirs to increase up to more than 11% of the annual average river discharge (Beilfuss, 2012).
At present, water needs within the ZRB are lower than the available re-sources, with the largest consumptive water uses (amounting to about 15/20% of the available runoff) located in the middle and lower Zambezi and corre-sponding to evaporation losses from the reservoirs, irrigated agriculture, and water supply. In the future, extreme flood and drought events are likely to in-tensify due to climate change (Pachauri et al., 2014), together with high rates for population growth that will contribute to an increase in both energy and food demands within the region, driving the expansion of irrigated agriculture and the construction of additional reservoirs for hydropower production (Figure
Figure 2.2: Elevation profile of the major existing (grey) and planned (red) hy-dropower plants along the ZRB (SADC, 2007).
2.2) (World Bank, 2010). A wide range of future challenges, e.g. reduced water availability, rising frequency of extreme events (e.g., floods, droughts), will thus highly impact both existing and planned infrastructures and eventually exacer-bate regional and international tensions over water use.
2.1.1 Kariba dam
About 35% of the total hydropower capacity within the ZRB is installed at the Kariba dam, which provides storage for two hydropower plants, namely the North Bank Station in Zambia and the South Bank Station in Zimbabwe for a total nameplate capacity of about 2,000 MW, which are jointly operated by the Zambezi River Authority (ZRA). The Kariba dam was designed via tradi-tional engineering approach for dam sizing, assuming a pre-defined operating rule curve (i.e., an end-of-the-month storage target to be tracked; Soils Incorpo-rated Ltd (2000)). Built using historical streamflow records from the 1925-1955 time frame, the rule for tracking the prescribed storage is intended to maximize hydropower production under stationary hydro-climatic conditions, neglect-ing the subsequent emergence of irrigation user demands upstream as well as downstream from the reservoir. In order to follow the operating rule curve, the real system operator is supposed to adopt a bang-bang operating strategy, which consists of closing/opening the dam when the storage is below/above its desired monthly value. The historical storage trajectories recorded over the
a2 t+1 Irr 1 Irr 2 Kariba Zambezi River qt+1 a1t+1 st it+1 rt+1 Hydropower plant Irrigation district a) b)
Figure 2.3:Panel a: Schematic representation of the Kariba multi-purpose reservoir system, with the main variables explained in the text. Panel b: Monthly storage trajectories of the existing Kariba dam (dark coloured line) recorded over the 1986-2005 period. The light coloured line identifies the official rule curve to be tracked, which repeats itself the same every year.
1986-2005 period, however, show that the real Kariba system operator was not able to track the rule curve due to the non-deterministic nature of external driv-ing forces, such as unexpected drought periods that were not taken into account during the design phase, and other contingencies (e.g., civil war; World Com-mission on Dams (2000). Please refer to Figure 2.3b).
In the future, climate change coupled with high population growth will put additional pressure on already stressed water resources (Hallegatte et al., 2015; Rigaud et al., 2018). Existing and planned water reservoirs will thus have to confront a broad array of future challenges, namely water scarcity and rising frequency of extreme events (e.g., floods, droughts) induced by climate change, while requiring effective operations for meeting growing resource conflicts in their demands (e.g., hydropower production vs irrigation water demand vs flood protection). These threats will have a potentially large impact on the Kariba dam and, more generally, on the existing water infrastructures that were originally designed under the myopic assumption that they are only intended to maximize hydropower production under stationary hydro-climatology.
2.1.2 Model description
As shown in Figure 2.3a, the Kariba reservoir system modeled consists of three main components, the reservoir with its hydropower plant and two irrigation districts upstream and downstream. A monthly modeling time-step is employed to capture the Kariba reservoir’s dynamics through the following water mass
balance equation:
st+1= st+ it+1− rt+1− et· St (2.1)
where stis the storage at the beginning of month t, it+1 is the inflow to the
reservoir, rt+1 is the volume of water released and et· Stis the water
evapo-rated in the time interval [t, t + 1). In particular, etis the mean monthly
evap-oration rate, while Stis the reservoir surface uniquely defined by a non-linear
relation given st. The actual release rt+1 = f(st, ut, it+1, et)is formulated
ac-cording to the non-linear, stochastic relation f(·) between rt+1and the release
decision ut(Piccardi and Soncini-Sessa, 1991), which is constrained within a
certain zone of operation discretion by the maximum and minimum feasible release functions, due to the presence of physical (i.e., spillway activation) con-straints. In the adopted notation the time subscript of a variable indicates the exact moment in time at which its value becomes deterministically known.
Note that the Kariba reservoir provides storage for two hydropower plants, namely the North Bank Station in Zambia and the South Bank Station in Zim-babwe, both operated by the Zambezi River Authority. In particular, the South Bank is equipped with deeper, more efficient yet smaller turbines than the North Bank (Gandolfi and Togni, 1997). Since both power plants are operated by the same authority, in our model the actual release rt+1is calculated as the total
release from the reservoir. Given that the two power plants possess different turbine capacities, efficiencies and turbine depths, however, their hydropower potential differs. Consequently, we simplistically assume the total release to be split into rN
t+1 = rt+1· ηN for the North and rSt+1 = rt+1· ηS for the South
Bank, based on their respective turbines efficiency ηN < ηS, to calculate the
exact hydropower production at each of the two power plants.
As for the two irrigation districts (id=1,2), they can abstract water aid
t+1from
the river through a regulated water diversion channel. The volume of water they can abstract is calculated according to a non-linear hedging rule (Celeste and Billib, 2009). For example, the upstream diversion channel is regulated as follows: a1t+1= min(qt+1, w1t· [ qt+1 h1 ] m1) if q t+16 h1 min(qt+1, w1t) else (2.2)
where qt+1 is the volume of water available in the river, w1t is the monthly
water demand, whereas h1and m1are the parameters regulating the diversion
channel. The diversion rules allow hedging the water abstractions to account for downstream users.
2.2
Synthetic regulated lake
Real case studies can present a vast array of complexities ranging from the defi-nition of the system boundaries to include all the relevant drivers and the unfea-sibility of conducting various experiments, to the lack of information regard-ing the decision makers’ objective functions and the actual economic value of planned infrastructures and associated operations (Mason, 2018). The choice of a synthetic case study can address these issues while allowing to perform exper-iments in a controlled environment and therefore assess the actual potential of novel methodological contributions. We designed a numerical case study that builds upon the Lake Como system in Northern Italy presented in Giuliani et al. (2016b), whose similar version already appeared in Bertoni et al. (2017). Similar synthetic models can also be found in the existing literature, such as the classi-cal shallow lake problem dealing with water quality (Quinn et al., 2017b), and the Y-shaped river system for analyzing competition among different decision makers (Yang et al., 2009; Giuliani et al., 2014a).
100 200 300 400 500 600 700 Time [days] 300 320 340 360 380 400 420 Inflow [m 3 /s] Mean = 369 m3/s
Figure 2.4:Daily inflow trajectory feeding the synthetic reservoir. The average flow [m3/s] is reported in the bottom left corner.
2.2.1 Model of the system
The synthetic water system is composed of a synthetic water reservoir that must be sized and operated for ensuring reliable water supply to downstream users while minimizing construction costs. A daily modelling time-step is employed. The reservoir’s dynamics is described through a state-transition function
sim-ilar to the water mass balance formulated in equation 2.1 for the Kariba case study:
st+1= st+ it+1− rt+1 (2.3)
where the same variables employed for the Kariba reservoir dynamics still hold, except for the evaporation term that is now neglected. Moreover, we feed the synthetic reservoir with a single realization of 730 daily reservoir inflow values, as in Giuliani et al. (2016b) (Figure 2.4). Here, the system is assumed as station-ary (i.e., the seasonality is removed) in order to simplify the problem, adapting the values of inflows and downstream irrigation demands. Even then, in the adopted notation the time subscript of a variable indicates the exact moment in time at which its value becomes deterministically known.
3
Reinforcement Learning for
designing water reservoirs
Abstract
1In the water systems analysis literature and practice, planning (i.e., dam siz-ing) and management (i.e., operation design) have been for long time addressed as two weakly interconnected problems, and this often resulted in over-sized, poorly performing infrastructures. Recently, several authors started explor-ing the interdependent nature of these two problems introducexplor-ing new inte-grated approaches to simultaneously design water infrastructures and their op-erations. Yet, the high computational burden is a likely downside of these meth-ods, a large share of which requires to solve one optimal operation design prob-lem for every candidate dam size, making it unfeasible to explore the entire planning and associated operation decision space. In this chapter, we contribute a novel Reinforcement Learning (RL)-based approach to integrate dam sizing and operation design, while significantly containing computational costs with respect to alternative state-of-the-art methods. Our approach first optimizes a single operating policy parametric in the dam size and, then, searches for the best reservoir size operated using this policy. The parametric policy is com-puted through a novel batch-mode RL algorithm, called Planning Fitted Q-Iteration (pFQI). The proposed RL approach is tested on a numerical case study, where the water infrastructure must be sized and operated to meet downstream users’ water demand while minimizing construction costs. Results show that
1Bertoni, F., Giuliani, M., Castelletti, A., 2019c. Integrated design of dam size and operations via Reinforcement
our RL approach is able to identify more efficient system configurations with respect to traditional sizing approaches that neglect the optimal operation de-sign phase. However, when compared with other integrated approaches, we prove the pFQI algorithm to be computationally more efficient.
3.1
Introduction
As already detailed in chapter 1, hydropower is currently responsible for a sub-stantial share of the electricity generated worldwide, providing over 16% of global electricity production (REN21, 2016). Yet, a significant untapped hy-dropower potential still persists in many river basins, which has mobilized in-vestments for the construction of about 3,700 new dams globally (Zarfl et al., 2015). Projected increasing energy demand is also contributing to boost such a large hydropower expansion in many energy markets, especially in Asia and Africa (World Energy Council, 2016). To mention just a few examples, 68 new hydroelectric dams are planned to be built in the Lower Mekong river basin (Mekong River Commission, 2013), whereas the current installed hydropower capacity in the Zambezi river basin is forecasted to double by 2025 (World Bank, 2010).
When planning such water infrastructure systems, many practitioners al-ready account for both operations and impacts of new dams in addition to siz-ing decisions (e.g., South Florida Water Management District, 2018). However, existing studies often present limited formal approaches, where planning (i.e., dam sizing) and management (i.e., operation design) are traditionally consid-ered as two separate problems, disregarding their potentially interdependent nature. Within the infrastructure sizing domain, several methodological con-tributions have been developed over the past decades, which can be divided into three main categories (Jain and Singh, 2003; Adeloye, 2012): (i) Critical Period Techniques (e.g., Hall et al., 1969; Hall and Dracup, 1970); (ii) Behavior (or Sim-ulation) Analysis (e.g., Turner and Galelli, 2016; Turner et al., 2017); and (iii) Optimization methods. Critical Period Techniques, namely the period during which an initially full reservoir empties without spilling (Jain and Singh, 2003), are the precursor of storage-yield analysis and include the Mass Curve (Rippl, 1883) and Sequent Peak Algorithm (SPA) (Thomas Jr and Burden, 1963; Sound-harajan et al., 2016), together with several modified versions of this latter (e.g., Lele, 1987; Montaseri, 1999). In particular, the Mass Curve approach requires only observed streamflow records, whereas the SPA method uses either histor-ical or synthetic data to calculate the smallest dam size needed to meet a given downstream demand, i.e. the volume that would be depleted during the most severe shortfall between demand and water flow (Klemeš, 1979; Montaseri and Adeloye, 1999). Even though both approaches are simple and straightforward,
they present a few limitations: the Mass Curve does not provide any informa-tion about the probability of failure (Schultz, 1976) and the dam size identified is optimal only with respect to the historical period considered in the analysis. Yet, the recorded time-series might not fully characterize the intrinsic variabil-ity of hydrologic processes, due to the limited number of observed data avail-able compared to the longer return periods of hydrologic extremes, leading to strong biases in the water infrastructure design (Adeloye, 2012). On the other hand, Thomas Jr and Burden (1963)’s SPA does not model water spillage from the dam (Koutsoyiannis, 2005) and cannot employ a different dam operating policy apart from the standard one (Lele, 1987). The standard operating pol-icy (SOP), which constrains the releases to be always equal to either the target release or all the available water (McMahon and Adeloye, 2005), might cause severe water deficits downstream during prolonged drought periods, inducing the system to strongly under-perform. A second dam sizing technique is Be-havior (or Simulation) Analysis, a variant of the Mass Curve and SPA (Pretto et al., 1997) that calculates the minimum storage capacity required via itera-tive simulation procedure (e.g., bisection method) of the reservoir behavior (i.e., mass balance equation) (e.g., McMahon et al., 2007; Turner and Galelli, 2016; Turner et al., 2017). Unlike the previous two methods, Behavior Analysis is able to handle multi-reservoir systems (Adeloye, 2012), however it still requires a pre-specified operating policy (e.g., SOP) (Koutsoyiannis, 2005) and is highly in-fluenced by the length of the inflow time-series considered (Pretto et al., 1997). Recently, optimization techniques are being increasingly employed for sizing water infrastructure systems (e.g., Manikkuwahandi et al., 2019; Zhang et al., 2019), with special attention to evolutionary algorithms, particularly in the wa-ter distribution and supply design field (e.g. Vairavamoorthy and Ali, 2005; Wu and Walski, 2005; Matrosov et al., 2015). Alike the other approaches presented so far, the focus is on the identification of optimal water infrastructure sizes only, assuming pre-defined operating policies, consisting of a target release to be discharged at each time-step. The vast majority of these studies relies on a static downstream water demand, while all of them neglect the interdependency between dam size and operation, and the impacts of different short-term oper-ating strategies on the long-term system design (Tian et al., 2018). By failing to carefully explore alternative reservoir operating strategies, such methods are likely to result in over-sized, under-performing water infrastructures (Moran et al., 2018). In order to avoid these biases, other works started to investigate sizing and operation in a more integrative way.
Among others, Houck and Cohon (1978); Opricović et al. (1991); Mousavi and Ramamurthy (2000); Yang et al. (2007) for water reservoir systems and Chang et al. (2009) for a groundwater planning problem employ a two-stage optimization procedure, nesting an optimal control problem - to design the
optimal operating policy for a given dam size into a global optimization -to explore the space of the sizes. This nested approach is employed -to solve a Multi-Objective (MO) problem, whose solutions are alternative system con-figurations (size and operating policy) that vary with the selected trade-off be-tween least cost planning and other operating objectives (e.g., downstream wa-ter supply). When the number of objectives considered increases linearly, how-ever, the number of sub-problems to be solved grows factorially (e.g., in order to solve a four objective problem, four single-objective, six two-objective, and four three-objective sub-problems must also be solved) (Reed and Kollat, 2013; Giuliani et al., 2014b).
In order to overcome this limitation, other studies started to address the sizing and operation design as two strictly interconnected faces of the same problem and to solve them jointly in a fully integrated framework. Among oth-ers, Stedinger et al. (1983); Lall and Miller (1988); Afzali et al. (2008); Afshar et al. (2015) employ Linear Programming (LP) to identify optimal dam sizes and the associated time sequences of releases, namely open-loop optimal release se-quences. A core technical limitation of this approach, however, is in the repre-sentation of reservoir releases and storage dynamics. Since state feedbacks on release decisions are ignored, the open-loop release sequence is optimal only with respect to the single realization of external drivers (e.g., inflows) under which it was optimized. Therefore, the operation is not able to effectively adapt to possible changes in the external drivers. On the other hand, storage release dynamics are typically highly non-linear, yielding strongly simplified abstrac-tions when they are modelled via LP methods (Loucks et al., 2005). Other works have developed more sophisticated, policy search-based versions of the joint problem, constraining the dam operating policy to fall within a pre-specified linear class of functions (i.e., Linear Decision Rule (LDR)), whose parameters are thus optimized together with the dam size via LP (e.g., Revelle et al., 1969; Houck et al., 1980; Stedinger et al., 1983; Afshar and Mariño, 1989; Afshar et al., 1991; Malek-Mohammadi, 1998; Satishkumar et al., 2010). A further improve-ment to this integrated planning and optimal operation problem formulation has been brought by most recent studies, which either employ a less simplistic operating policy class of functions (e.g., Afshar et al., 2009), while still solving the integrated optimization problem via LP, or use Evolutionary Algorithms (EA) for the optimization process, despite using a linear class of functions for the operating policy (e.g., Geressu and Harou, 2015). However, some limitations still hold, namely the operating policy is linear in the reservoir storage - thus the releases cannot depend upon other informative exogenous variables - and time-invariant. From a computational perspective, both LP and EA techniques suffer from a high computational burden when solving complex problems with a large number of objectives. Since LP requires the problem to be single-objective, one
optimization must be performed for each objective trade-off, causing the com-putational costs to significantly increase. The same holds for single-objective EAs, which aggregate multiple, potentially conflicting objectives into a single one for each considered trade-off. As for multi-objective EAs, they are able to solve multi-objective problems by considering multiple objectives simultane-ously. Yet, their computational demands are still significant, due to the fact that (i) several test optimizations must be run in order to identify the most suitable parameter settings for each specific problem; (ii) a large number of function evaluations must be performed in order to guarantee the convergence of the approximation set to the Pareto-optimal solutions; and (iii) multiple seeds must be run in order to improve solutions diversity and avoid randomness depen-dence (Reed et al., 2013; Maier et al., 2014).
In this chapter, we contribute a novel reinforcement learning (RL) approach to conjunctively design dam size and operations, overcoming the shortcomings just illustrated. The method here proposed relies on a novel algorithm, called Planning Fitted Q-Iteration (pFQI), which extends the batch-mode RL Fitted Q-Iteration (FQI) algorithm developed by Ernst et al. (2005) by enlarging the original FQI state space to include the discrete planning decision (i.e., dam size) as an additional state variable. The key idea behind pFQI originates from the Multi-Objective Fitted-Q Iteration (MOFQI) algorithm developed by Castel-letti et al. (2013): the continuous approximation of the action-value function originally performed by FQI over the state-action space is now enlarged to the planning space by including dam sizes as new variables within the arguments of the action-value function. This enables pFQI to approximate the optimal op-erating policy associated to any dam size within a single learning process.
This new algorithm therefore overcomes the limitations and biases intro-duced by traditional sizing methods by directly addressing the strict interde-pendency between dam size and operation within an integrated framework through an operating policy parametric in the dam size. Secondly, it overcomes the high computational costs associated with state-of-the-art nested and inte-grated approaches by solving a single operation optimization via pFQI, as the resulting policy can be used to simulate the optimal operations of all the pos-sible dam sizes associated to alternative trade-offs between least cost planning and operating objectives (e.g., downstream water supply). This characteristic contributes to significantly reducing the computational burden of pFQI.
The pFQI algorithm is tested on the multi-objective numerical case study presented in section 2.2, consisting of a synthetic water reservoir that must be sized and simultaneously operated to satisfy the water demand of downstream users. We demonstrate the effectiveness of our novel RL approach in identify-ing more efficient system configurations by direct comparison with the system performance achieved via traditional sizing techniques (i.e., Behavior Analysis)
in order to highlight the importance of capturing dam size and operation in-terdependency within an integrated framework. We then compare the system configurations obtained via nested approach and pFQI algorithm in terms of operating objective performance in order to prove the ability of this latter to ef-ficiently approximate the optimal operating policy over an enlarged state space and to perform satisfactorily also for dam sizes not directly experienced dur-ing its learndur-ing phase. In the end, we compare the computational costs of both state-of-the-art nested approaches and pFQI in order to demonstrate the com-putational efficiency of this latter.
3.2
Methodology
3.2.1 Problem formulation
We formulate the integrated design problem of dam size and operations as the following multi-objective, stochastic, non-linear, closed-loop, optimal planning and operation problem (e.g., Castelletti et al., 2008a,b, and references therein):
Problem P3.1 (Integrated design problem) θ∗, π∗=arg min θ,π | J 1 θ,π,. . . , Jnθ,π | (3.1a) where Jdθ,π = Ψ ε1,...,εh Φ 0,...,h gd1(x0, u0, ε1) ,. . . , gdh(xh) d=1,...,n (3.1b) subject to xt+1 = ft(xt, ut, εt+1, θ) t =0, . . . , h − 1 (3.1c) εt+1∼ φt t =0, . . . , h − 1 (3.1d) ut= mt(xt)∈Ut(xt, θ) t =0, . . . , h − 1 (3.1e) π, {mt(·) ; t = 0, . . . , T − 1} (3.1f) θ∈ Θ (3.1g)
where equation 3.1a identifies a vector of n planning (e.g., construction costs) and operation (e.g., downstream water supply deficit) objective functions to be minimized with respect to the planning (i.e., dam size θ) and operation (i.e., operating policy π) decision variables. Equation 3.1b provides the extended formulation for the d-th objective, where gd
t+1for t = 0, 1, . . . , h − 1 is the
value of the d-th immediate cost at time t + 1, gd
h(xh) is a penalty function
over the final state xh, Φ
0,...,his a time-aggregator operator over the entire
evalu-ation horizon [0, h] (e.g., average), and Ψ
ε1,...,εh
un-certainty introduced by the stochastic disturbances εt+1(e.g., expected value).
Equation 3.1c represents the state transition function according to which the current state xt∈Rnxof the system evolves throughout the evaluation horizon,
affected by a stochastic external driver εt+1∈Rnεdescribed by its probability
density function φt. Being equations 3.1c-3.1e periodic of period T, the optimal
policy π∗is periodic with the same period, namely π∗ ={m∗
0(·) , . . . , m∗T −1(·)}.
This policy is therefore defined as a periodic sequence of operating rules mt(·)
that in turn map the current state xtinto the release decision ut∈Ut(xt, θ)
⊆Rnu within its feasibility space, which depends upon the planning decision
θ(i.e., reservoir size). Equation 3.1g identifies the feasibility space Θ the plan-ning decision θ must belong to. Solving Problem P3.1 means finding a set of Pareto-efficient solutions in the multi-objective space rather than a unique op-timal solution.
In the next sections, we will adapt the general problem formulation P3.1 to fit three different solving approaches, namely traditional dam design, nested dam design, and our novel RL approach relying on the pFQI algorithm. Three different problem instances will be thus formulated, obtained by varying the number of decision variables, and consequently the number of optimization problems to be solved (from a least cost planning with a pre-defined operating policy to an integrated dam size and operations optimization). For the sake of simplicity, we will consider a single planning objective Jc
θas a function of dam
size θ only, and a single operating objective Jirr
θ,πdependent upon both dam size
θand operating policy π, which can be formulated as follows: • Minimization of dam construction costs Jc
θ[$] (planning objective):
Jcθ =1.2 · θ (3.2)
where the construction costs increase linearly with the reservoir size θ according to a multiplication factor equal to 1.2, which is arbitrary taken within the range of reservoir construction costs proposed by Weatherhead et al. (2009);
• Minimization of the downstream squared water deficit averaged over the evaluation horizon h Jirr
θ,π[m3/s]2(operating objective): Jirrθ,π = 1 h h−1 X t=0 (max (w − rt+1,0))2 (3.3) where w = 370 m3/s is the daily, time invariant irrigation demand,
com-pared to an average inflow of 369 m3/s feeding the reservoir. As already
discussed in section 2.2, the system is assumed as stationary (i.e., the sea-sonality is removed) in order to simplify the problem, adapting the values
of inflows and downstream irrigation demands. The quadratic formula-tion penalizes severe deficits in a single time step, while allowing for more frequent, small shortages to occur (Hashimoto et al., 1982).
3.2.2 Traditional dam design
Traditional dam design approaches aim at identifying the least cost dam size subject to an operating policy defined a priori. Problem P3.1 can be thus refor-mulated as a single-objective, least cost planning optimization:
Problem P3.4 (Least cost planning problem) θ∗ =arg min θ J c θ (3.4a) subject to Jirrθ,π¯ 6 1 − ¯r (3.4b) ¯ π given (3.4c) 3.1c, 3.1d, 3.1e, 3.1f, 3.1g where Jc
θ represents the planning objective (i.e., dam construction costs) as a
function of the dam size θ, whereas Jirr
θ,π¯ corresponds to the operating objective
(i.e., downstream water deficit) attained under the pre-defined operating pol-icy ¯π. The traditional sizing method of Behavior (or Simulation) Analysis may be used to solve this optimization problem, whose outcome is the optimal least cost dam size θ∗. This sizing technique employs the bisection method to vary
the infrastructure size and converge on a pre-defined reliability rate ¯r for the operating objective Jirr
θ,π¯, which is evaluated via simulation of the reservoir
be-havior under the pre-defined operating policy ¯π. The reliability quantifies the ratio of non-fail days (i.e., downstream water demand fully met) over the to-tal length of the evaluation horizon. Given a pre-defined reliability rate ¯r, the outcome of the least cost optimization is therefore the optimal dam size θ∗
op-erated under the pre-defined operating policy ¯π that minimizes construction costs while satisfying the reliability constraint.
3.2.3 Nested dam design
The nested approach consists of a two-stage optimization procedure nesting an optimal operation problem (designing the optimal operating policy π∗for
a given dam size) into a planning problem (finding the optimal dam size θ∗).
This second problem instance therefore optimizes both dam size and operation within an integrated framework and can be formulated as follows:
Problem P3.5 (Nested design problem) θ∗=arg min
θ | J
irr
θ,π∗, Jcθ | (3.5a)
where π∗=arg min
π J
irr
θ,π with θ given (3.5b)
subject to 3.1c, 3.1d, 3.1e, 3.1f, 3.1g
where θ∗ is the optimal dam size that minimizes both planning Jc
θ and
oper-ating Jirr
θ,π objectives, this latter evaluated under the optimal operating policy
π∗ for a given dam size θ. π∗is hence the optimal operating policy that min-imizes the operating objective Jirr
θ,π (eq. 3.5b) and is obtained from a pure
op-timal operation problem, which must be solved for each candidate dam size θ. This inner problem is usually solved via Dynamic Programming (DP) (Bellman, 1957) and, more recently, Approximate Dynamic Programming (ADP) (Powell, 2007) and Reinforcement Learning (Barto and Sutton, 1998) algorithms. Then, the external planning problem is solved by exploring the set Θ of planning deci-sions via exhaustive search, when feasible, or via a suitable optimization method (e.g., the gradient-descent or the Newton method), in order to identify the op-timal dam size θ∗ (Castelletti et al., 2008a). In the end, alternative dam sizes
and corresponding optimal operating policies associated to different trade-offs between the planning (i.e., construction costs) and operating (i.e., downstream water supply deficit) objectives are identified. For example, the least cost dam size will be associated to the optimal policy attaining the highest water supply deficit, whereas the minimum deficit will be achieved by the largest, thus most costly reservoir identified. In order to explore all the possible system configura-tions, the nested optimization problem P3.5 must be solved for all the objectives trade-offs considered. Being the computational costs increase factorial in the number of objectives, a likely disadvantage of the nested approach is the signif-icant increase in the computational burden when a large number of trade-offs, and thus a large number of two-stage nested optimizations must be performed, making it unfeasible to explore the entire planning and associated operation decision space.
3.2.4 Novel RL dam design
The proposed RL approach based on the pFQI algorithm integrates the plan-ning and operation problem using a reversed perspective with respect to Prob-lem P3.5. In particular, it first solves a pure operation optimization (ProbProb-lem P3.6a), whose outcome is a parametric operating policy π∗(·, θ)that minimizes
the operating objective Jirr
π(·,θ),θ. Being parametric in the dam size θ, this policy
is defined over the entire planning decision space Θ, and can be thus used to operate all the candidate dam sizes. As a consequence, Problem P3.6a must be
