Crisis maps quality check and a proposal of integration with data from social media sources

(1)

POLITECNICO DI MILANO

SCUOLA DI INGEGNERIA CIVILE, AMBIENTALE E TERRITORIALE M.Sc. of CIVIL ENGINEERING FOR RISK MITIGATION

Crisis maps quality check and a

proposal of integration with data

from social media sources

Supervisor: prof. Daniela Carrion

Thesis by: Katarina Spasenovic Student ID: 832857

(2)

2

Abstract

The Rapid Mapping for Emergency Management has a key role in making efficient response and diminishing possible consequences from a disaster. Crisis map production helps all stakeholders to understand type and severity of damages, impact on the area and its population, having in mind that maps need to be produced within the first 24-72 hours after the event. The quality of crisis maps is of a great importance, since the disaster response depends on provided information. In some cases, in order to deliver a map on time, its quality can be lower than expected. The first part of thesis work is focused on the evaluation of a sample of rapid maps produced between 2013 and 2017 collected from open source Copernicus Service. The evaluation has been preformed by visual analysis, observing around 36 parameters defined in the validation protocol designed at the Joint Research Centre (JRC) of the European Commission. The evaluation process will be presented and the results will be analysed in detail. The second part of thesis will explore the new powerful sources of information such as social media are, presenting their potential as a support in Rapid Mapping and Emergency Management. The concept of getting information from post‟s geolocation, related to a certain event, obtained from the social media will be explored and further spatial analysis from obtained data will be preformed and conclusions will be made.

Abstract

La cartografia di emergenza riveste un ruolo chiave per fornire un efficiente risposta nella gestione delle conseguenze derivanti da un disastro naturale (o catastrofe). Le carte di emergenza sono di supporto agli operatori per analizzare i vari tipi di danni, l‟impatto su una determinata zona e sulla sua popolazione, tenendo presente che le mappe necessitano di un minimo di 24-72 ore per essere prodotte. La qualità delle mappe assume un ruolo di estrema rilevanza dal momento che la risposra al disastro dipende dalle informazioni fornite. In alcuni casi la qualità delle carte può risultare di livello inferiore rispetto alle aspettative a causa del poco tempo dedicato alla sua produzione. La prima parte della tesi si concentra sulla valutazione di un campione di carte di emergenza prodotte dal 2013 al 2017 selezionate tra quelle messe liberamente a disposizione da Copernicus. La valutazione si basa su un‟analisi visiva su 36 parametri definiti a partire protocollo di validazione definito dal Joint Research Centre (JRC) della Commissione Europea. Verrà presentato il processo di valutazione e verrranno analizzati i risultati nel dettaglio. La seconda parte della tesi si pone come obiettivo quello di esplorare il nuovo importante strumento di informazione costituito dai social media, esponendo il suo potenziale come supporto per la cartografia di emergenza. Verrà analizzata la possibilità di acquisire informazioni dalla geolocalizzazione deipost nei social media, con riferimento ad uno specifico evento e e da ultimo verranno tratte le conclusioni.

(3)

3

1 Introduction

Natural disasters, multiple-casualty accidents, man made accidents, terrorist attacks present events with a scenario of great challenges for responsible organizations engaged in management of national, regional or local emergencies. The event leads to the emergency state in which normal procedures are interrupted, and immediate actions and measures need to be taken in order to prevent further propagation of the event turning it into disaster and causing great damage and or loss of life. Involved stakeholders should collect data and transform it into information that will help in creating a picture of current situation and support decision making for rescue operation planning and resource allocation in order to minimize the loss of human life, built-up, infrastructure and deal with long time effects such as sanitary and health conditions. According to the FEMA (Federal Emergency Management Agency of USA, https://training.fema.gov/emiweb/downloads/is111_unit%204.pdf) the process of disaster management can be presented in four phases; two phases as a part of pre-disaster: mitigation and preparedness and two of post-disaster: response and recovery. In Figure 1 below, it can be seen that Emergency Management (EM) is a cycle of actions that influence each other, no matter of their implementation time. From the past events it is possible to conclude where errors have been made and improvements in phases for future emergency management can be implemented.

Figure 1 The Emergency Management cycle

1.1 Preparedness

Preparedness phase is implemented before the event in order to prepare the residents of potentially harmful zone to handle an emergency. It is focused on making plans and having needed resources in a zone of potential risk. According to

the FEMA training module

(https://training.fema.gov/emiweb/downloads/is111_unit%204.pdf) some of measures are evacuation plans, stocking supplies, trainings that can decrease risk of loss and improve rescue operations.

1.2 Mitigation

Disaster mitigation actions or preventions are those that exclude or reduce the impacts and long-term risks from hazards and their effects, through proactive

(8)

8

measures taken before the event occurs. Preventive or mitigation measures take different forms for different types of disasters. Mitigation measures may include: installation of generator – ensuring continuous power supply; seismic retrofitting of property; forbidding building in a high risk zone, adaptation of structure from a lesson learnt last event. It can be seen from a measures that mitigation phase can occur before the event or after as a lesson learnt product.

1.3 Response

When the disaster occurs, response phase is active, which main aim is to save life and property. Response phase involves all stakeholders, assistance from organisations and agencies may be provided by national and in some cases international level. This may include warning, evacuating or sheltering the public, activating evacuation plans, search and rescue, providing medical treatment etc. Effective coordination of disaster assistance is of a great importance, having many involved parties in the emergency management it is significant that each stakeholder knows its role and tasks in order to diminish errors and increase efficiency of disaster response.

1.4 Recovery

The recovery phase will start when human life is no more under risk. Main goal of recovery phase is to bring the affected area back to normality as quickly as possible. According to FEMA recovery phase can be divided in two sections, short term and long term recovery ( https://training.fema.gov/hiedu/docs/fem/chapter%2011%20-%20community%20disaster%20recovery.doc). Short term recovery includes bringing power, telephone and other lifelines back to functionality ensure social needs of individuals. Long-term goals present restoring economic activity and rebuilding community facilities and housing.

Overviewing all phases of EM cycle it can be concluded that quality information regarding the affected area and its condition as well as safety of human life has a key role on which depend further development and activation of phases. One of the widely used tools is crisis map which presents digital representation of disaster situation that proved to be very helpful for disaster management.

(9)

9

2 Rapid Mapping as a support for Emergency Management

Rapid Mapping presents digital representation of maps with geospatial information providing an overview of current situation in affected area with crisis information for response (IWG-SEM 2014). In particular Rapid Mapping has a great role in response phase, providing in short-term (within hours or days) geospatial information, which will be used for supporting emergency management activities. The process of map production is generally based on exploitation of satellite images. Main topographic features representing pre-event situation should be provided (e.g. transportation network, build-up, land cover, hydrography, and utilities) in order to form better general image of the event impact. As stated before time is crucial for EM, to minimize the time required for post-event map production, Rapid Mapping should be based on consolidated operational workflows: a simplified workflow, highlighting the main processing steps generally adopted in the context of Rapid Mapping shown in Figure 2 (Ajmar et al. 2015). It has to be stressed that the proposed workflow should be accepted as general, presenting the main operational steps. There are more than one service providers involved in disaster Rapid mapping such as United Nations UNOSAT Rapid Mapping Service that provides timely and high-quality geo-spatial information to UN decision makers, member states, international organizations and non-governmental organizations (https://unitar.org/unosat/unosat-rapid-mapping-service). International charter space and major disasters have good network of earth observation agencies worldwide that

provide them with assessment information.

(https://www.esa.int/Our_Activities/Observing_the_Earth/The_International_Charter_ Space_and_Major_Disasters). The European Union's Earth Observation Programme Copernicus, even being a service of EU it is open to humanitarian organisations working worldwide. Even though all previously presented Rapid Maps providers are independent services they collaborate in a partnership with each other and with other agencies and organisations of interest in order to deliver good quality product in time. In this work focus is on Copernicus Rapid Mapping Service and quality of its products.

(10)

10 Figure 2 Simplified Rapid Mapping general flow-chart highlighting the main processing steps

and the activity timeline

According to Carrion et al. (2012) Rapid Mapping have contributed significantly disaster management in the last decades, reaching different actors from local organizations (police station, fire brigade), to national (civil protection) and international institutions, such as World Bank, the United Nations and the European Commission. Copernicus Rapid Mapping Service has shown evident support during dramatic events like Harvey and Irma hurricanes 2017 (Copernicus, visit http://copernicus.eu/news/hurricanes-harvey-and-irma-summer-all-records).

Hurricane Harvey has been considered as one of the costliest tropical cyclone, inflicting $125 billion in damage. With an interval of less than two weeks, it was followed by Irma, another record-breaking hurricane that spread havoc along its path

(11)

11

in the Caribbean. For nearly three weeks, the Rapid Mapping team of the Copernicus Emergency Management Service (EMS) worked on nearly continuous shifts to supply the many end-users in need of timely geo-spatial information while dealing with the devastating impact of Harvey, which was to be soon followed by Irma. In this unheard-of situation, the EMS, its operator, the European Commission‟s Joint Research Centre as well as the involved contractors, demonstrated their resilience and ability to cope with simultaneous and concurrent disasters on a worldwide basis.

2.1 Copernicus Emergency Management Service for Rapid Mapping

Copernicus EMS presents online platform with an open access to all users which provides geospatial information for emergency response in relation to different types of disasters. As mention in previous chapter geospatial information is derived from satellite remote sensors and fulfilled by available in situ or open data sources details. The service is based on the acquisition, processing and analysis, in rapid mode, of satellite imagery and other geospatial raster and vector data sources. The service produces two types of maps: Rapid Maps and Risk Recovery Maps. For the purpose of the research in thesis focus will be on Rapid Maps, which presents on-demand delivery of geospatial information in a short time after an emergency event. The products are standardised and they are three: Reference Maps, Delineation Maps (providing an assessment of the event range) and Grading Maps (providing an assessment of the damage grade and its spatial distribution).

2.1.1 Reference Maps

Reference map shows general review of the territory and assets prior to the crisis that is up to date. The content of map consists selected topographic features on the affected area, exposed properties and other helpful information that can assist the users during their crisis management tasks. A reference map is usually based on a pre-event image captured as close as possible before the event. As stated in Copernicus Service, time needed for reference map production after satellite image acquisition is minimum 9 hours. Example of a reference map of an earthquake in Nepal 2015 is shown in a Figure 3 below.

(12)

12 Figure 3 Reference map of earthquake in Hetauda city of Nepal on April 2015

2.1.2 Delineation Maps

Delineation map provides an assessment of the event range. Delineation maps are derived from satellite post-disaster images. They depend on the disaster type and the delineation of the areas impacted by the disaster, for example delineation of flooded area, earthquake impact area, estimation on the exposed or affected population and assets. Delivery time for delineation map is around 12 hours, but First Available Map (FAM) that is a lower quality is available within 3 hour (Copernicus, visit www.jrc.ec.europa.eu). Example of a delineation map of fire in Spain in 2015 is shown in a Figure 4 below.

(13)

13 Figure 4 Delineation map of fire in city Lujar of Spain on July 2015

2.1.3 Grading Maps

Grading map presents an assessment of the damage grade and its spatial distribution. Maps are derived from post-event satellite images and include the level, magnitude or damage grades specific to each disaster type. They may also provide relevant and up-to-date information that is specific to affected population and assets, e.g. settlements, transport networks, industry and utilities. Delivery time for grading map is around 12 hours, but FAM of a lower quality is available within 3 hour (Copernicus, visit www.jrc.ec.europa.eu). Example of a grading map of fire in Spain in 2015 is shown in a Figure 5 below.

(14)

14 Figure 5 Grading map of fire in Acebo city of Spain in 2015

(15)

15

3 The quality of crisis maps

During the emergency, crisis map presents the source of information and medium for its exchange between different users, which are independent one from the other. Information that map shows should be clear, complete and provided together with metadata1. Complete provided information of crisis maps would allow all users to contribute the emergency actions in the best way improving the response quality. When fundamental details are missing from geographic data, such as scale and information sources‟ resolution, or when it is incomplete, for example when the legend is not clear enough, often it is not possible to exploit them, e.g. for integration with other layers into Geographic Information Systems (GIS) (Carrion et al. 2012). The work done in the first part of thesis analysed the quality of a representative number of crisis maps, produced between 2013 and 2017 by world leading providers, downloaded from Copernicus Service that has an open access. Original evaluation procedure relies on the analysis presented in paper by Carrion et al. (2012), where a sample of crisis maps produced between 2005 and 2010 by world leader providers has been evaluated through around 40 parameters assessed by visual analysis. Merging results obtained in paper and in thesis, twelve years wider overview can be made. In general the quality of data is defined as „fitness for use‟ and some parameters commonly used to measure it are positional and thematic accuracy, consistency and completeness [see US National Committee Digital Cartographic Data Standards Task Force, (DCDSTF 1988)]. According to Borrough and McDonnell (2000) the factors which affect spatial data quality are:

 Currency (nowadays maps are in electronic form that makes them out-dated; data provided can be used in a long period of time)

 Completeness

 Consistency (connection and dependence between data)

 Accessibility (easy access to the data information)

 Accuracy and precision (correctness of provided data)

 Source of errors in original data

 Source of errors in derived data and in the results of modelling and analysis

These quality parameters have been taken into account in the validation protocol (Broglia et al. 2010, Corbane et al. 2011a) for crisis maps, that has been developed at Joint Research Centre (JRC) and applied in the framework of SAFER FP7 (Services and Applications For Emergency Response, http://safer.emergencyresponse.eu) project. Quantitative and qualitative parameters of the protocol have been grouped into four categories: reliability of the information content, consistency of the information support, usability of the product and efficiency of the service. The protocol presents very detailed analysis of data, metadata. The target of thesis work was to explore the crisis maps data quality in a more general approach, overviewing large number of crisis maps over five years, applying cost and time effective approach instead of in-depth approach. Comprehensive analysis of this big sample would take a lot of time and would bring challenges of finding needed information and data connected to the events all around the world. Evaluation process relied on the idea to analyse the subsets of

1

(16)

16

quality parameters, considering minimal requirements needed for further utilization of crisis maps, for example integration into GIS or obtaining information for evacuation plans. Some examples of these minimum requirements are the possibility to understand the geographic position and time of the disaster, legends, understanding maps‟ metadata, such as accuracy and scale. Evaluated crisis maps have had an open access and were free for download, making them source of information available to everyone. During the evaluation process of crisis maps, it was taken in consideration that maps should be understandable for all types of users, experts and non-experts groups. In this perspective a checklist including 36 parameters has been derived from JRC validation protocol.

(17)

17

4 The sample of crisis maps

The oldest crisis map available for free download on Copernicus Service platform dates from 2013. As a starting point Flooding of Refugee Camp in Mafraq, Jordan that occurred on 7th of January 2013. As end point, Floods in Lower Saxony, Germany that happened on 14th of December 2017 has been taken into account. The total amount of 2635 published maps covering the time period under the study, have been downloaded and considered as the population for verification process for the thesis.

It should be noted that population of 2635 downloaded maps, have been classified with respect to the type of crisis and the year of map production (see Figure 6). Out of the population of downloaded and classified maps, a sample of 760 maps (30% of total amount) has been extracted, taking in consideration all types of events of interest as well as all affected zones. For example for a flood in Slovenia happened on 6th of November 2014, there were three affected cities that were presented with crisis maps LJubljana, Ptuj and Ilirska Bistrica. Situation in the city of Ljubljana was presented in seven maps (delineation overview map, delineation detail 1,2 and 3, reference overview map and reference detail map 1 and 2), from which one of them was extracted randomly and have presented the significant sample. Cites Ptuj and Ilirska Bistrica were presented with four crisis maps (delineation overview and detail maps as well as reference overview and detail maps), the same random method has been performed and one map for each city has been chosen. This extraction principle was applied through all downloaded maps. Finally, a sample of 760 maps, presenting almost 30% of total, has been obtained. This significant sample was used for further quality analysis check of the crisis maps.

Figure 6 Population of totally downloaded crisis maps considered in the study 0 200 400 600 800 1000 1200 1400 2013 2014 2015 2016 2017 N u m b e r o f m ap s Year Typhone Wind Storm Tsunami Industrial accident Hurricane Tropical Storm Fire Earthquake Landslide Flood

(18)

18

Classification of significant sample per year and type of event could be seen in Figure 7. Comparing Figure 6 with Figure 7 it could be noticed that the sample reflects the population behaviour, making it representative. From both graphs it was very noticeable increasing trend of the total produced number of the maps per year. The number of produced maps in 2017 was five times more than in 2013. It was also possible to notice that the most dominant type of the event that prompt the biggest map production was Flood (42% of representative sample), followed by Fire and Earthquake (20% and 14% of sample, respectively) (for more details see Figure 11).

Figure 7 Representative sample of the maps considered in the analysis

Observing the dispersion over continents of the sampled maps (Figure 8), it was possible to see that there was no homogeneous behaviour over time, but there was a noticeable dominance of the blue colour that presents Europe, representing 58% of total maps, over the period of five years. An explanation of big amount of events in Europe was not, because this continent have had the biggest rate of natural disasters, it was because Copernicus is EU service, even if it is open to humanitarian organizations working worldwide, their biggest focus is on crisis rapid mapping located in Europe. 0 50 100 150 200 250 300 2013 2014 2015 2016 2017 N u m b e r o f m ap s Year Typhone Wind Storm Tsunami Industrial accident Hurricane Tropical Storm Fire Earthquake Landslide Flood

(19)

19 Figure 8 Distribution of representation sample maps per continent during time

Comparing the previous analysis with the analysis done for the time period from 2005 till 2010 by Carrion et al. 2012, it could be noticed that nowadays the number of maps had increasing trend meanwhile before it was randomly oscillating (see Figure 9). The reason lies on the fact that today system is more stabile and standardized and providers are more expertized.

Figure 9 Distribution of sample of the maps over the time of 2005-2010 with classification by event made by Carrion et al 2012

0 50 100 150 200 250 300 2013 2014 2015 2016 2017 N u m b e r o f m ap s Year Oceania Europe Asia South-America Central-America North-America Africa

(20)

20

An effective and very short summary of the event information should be present in the description of crisis maps, since these data could help to the user to understand if the obtained map would be relevant for further work or not. The further text will present general information that was present in the sampled set of crisis maps.

 Service provider

There were six world service providers on Copernicus (see details in Table 1) that made maps of all types of events. Difference between providers has not been examined in depth.

Table 1 List of the world leader providers of crisis maps

Service

Provider Description Origin

DLR

German Space Agency, research establishment for engineering sciences in five main research areas: aerospace, aeronautics, energy, transport and security. DLR operates the “Center for Satellite Based Crisis Information” which provides a 24/7 service for the rapid provision, processing and analysis of satellite imagery during natural and environmental disasters, for humanitarian relief activities and civil security issues worldwide.

Germany

SERTIT

The provider of rapid mapping since 1999 within the International Charter and multiple projects is a service of the University of Strasbourg developing EO applications covering the full crisis cycle.

France

e-Geos

Leading international player in the Earth Observation and Geo-Spatial Information business. The company offers a unique portfolio of application services, and has acquired a leading position within European Copernicus Program.

(21)

21 Table 2 continued

Service

Provider Description Origin

GAF AG

Subsidiary of e-GEOS, is one of the major Earth observation service provider with a European and Global footprint, and leader in geo-information technology and technical assistance consultancy.

Germany

ITHACA

The non-profit association ITHACA is a center of applied research devoted to support humanitarian activities in response to natural disasters by means of remote sensing techniques.

Italy

SIRS

An independent consulting and GIS engineering company, specialist in the production of geographic data from satellite or aerial images.

France

The most active provides on Copernicus service were GAF and e-GEOS producing more than 26% of crisis maps, in contrary DLR made just 1.32% of total maps in a period of five years. It is important to note that activity of producers over the five years was mostly uniform, except the production of maps from DLR that have been inactive for four years but in 2017 it gave contribution to the service. An overall productivity of crisis map producers for a period of five years have been presented in Table 2 below.

Table 2 Activity of crisis map producers for a period from 2013 till 2017

Provider SIRS ITHACA GAF e-GEOS DLR SERTIT

number of

maps 108 105 205 200 10 132

percentage 14.21% 13.82% 26.97% 26.32% 1.32% 17.37%

 Type of crisis event

Copernicus Service for rapid mapping covers around fifteen types of events. For the purpose of thesis research the following events have been taken in consideration:

 Flood

 Landslide

(22)

22  Fire  Tropical Storm  Hurricane  Industrial Accident  Wind storm  Typhon

The listed events were the most frequent, spreaded around the world, what made them good candidates for building general picture of crisis maps quality. The most frequent natural disaster was flood with a 42.24% of total events for observed period of five year, a bit less present but still in the top were earthquakes and fire with 13.55% and 19.74%. On the other way the rarest events were industrial accident and typhoons, both having 0.39% of total number of events (see Figure 10).

Figure 10 Presence of the events for a period of five years

Information about the type of the event could be found in the title of crisis map as presented in Figure 11.

Figure 11 Crisis map title components 42,24% 1,97% 13,55% 19,74% 5,79% 2,50% 0,39% 13,42% 0,39% Flood Landslide Earthquake Fire Tropical Storm Hurricane Industrial accident Wind Storm

(23)

23

 Type of map

Crisis map could be produced in three standardized forms: reference, delineation and grading map. Examples of maps with detailed explanation have been presented in Chapters 2.1.1., 2.1.2. and 2.1.3. Type of the map is present in title of the crisis map (see Figure 11). The representative sample had 21% of reference maps, 45% of delineation and 34% of grading maps. It was good that majority maps were delineation and grading maps, so the quality of crisis representation in digital format could be more easily checked.

 Map language

Most of the geo-information products for emergency response consider English as a reference language for communication (Broglia M. et al. 2010). Sometimes for non English speaking countries it is necessary to have bilingual map. Bilingual map in most of the cases would be combination of English and other language as it could be seen in Figure 12 below where the title, legends and name of section were written in German and description parts in English.

Figure 12 Example of bilingual map description

The representative samples have shown that most of the produced crisis maps were in English having 743 maps in this language, the rest of 17 maps were bilingual (3 Spanish-English, 4 German-English and 10 Portuguese-English).

 Time gap between crisis event and crisis map production

The smaller gap time between the event activation and the source of mapped information, the more reliable the map is to the user (Broglia M. et al. 2010). Crisis map should be produced within first 24-72 hours after event, sometime for events that have long duration e.g. flood, hurricane, crisis maps can be used even later (after one week or two), depending on event duration. In general it is expected that the biggest map production is concentrated within first 72 hours. The sample has shown that the results are matching with predictions, having 8% of maps produced at the day of the event, 29% (highest peek) within first 72hours. During the first two

(24)

24

weeks it have shown high production of crisis maps, after what production is decreasing, as shown in chart below.

Figure 13 Time gap between crisis event and crisis map production

5 The quality assessment checklist

The quality analysis of spatial data ensures that the product meets the needs of end-users. The delivery of rapid maps has a high priority topic to users and providers, but in the crisis situations, the delivered geo-information is consumed without having the opportunity to check its accuracy and quality. According to the JRC protocol the quality of spatial data can be affected by reliability and consistency of information contents, usability of the product and efficiency of the service. Guided by the parameters presented in JRC validation protocol, more synthetized version of evaluation framework has been designed. Parameters that can be verified by a visual analysis have been considered and finally adapted to the purpose. The biggest attention has been devoted to the maps readability and usability. Following Table 3 presents the final quality assessment checklist presenting the 36 parameters that will be controlled.

8% 29% 28% 17% 7% 4% 6% 1%

Maps production over the time

First 24h 72 h 1 week 2 weeks 3 weeks 1 month 3 months

(25)

25

Table 3 List of 36 parameters considered for the rapid map quality check.

Validation protocol

category Parameter

Usability general information

Service provider Type of crisis event Type of map

Date of crisis (month and year) Place of crisis event - continent Place of crisis event - country Place of crisis event - town First language map

Second language map Reliability of the

information content

Information on occlusion of EO sources (clouds)

Time gap between crisis event and crisis map production Legend semantic definition for thematic data (e.g. Corine Landcover)

Consistency of the information support

Consistency between map and legend symbols

Consistency between declared scale and resolution of the images used to produce the map

Usability of product Contrast between background and thematic entities Symbols easily differentiable

Scale bar

Declared nominal scale Presence of overview map

Coordinate graticules/grid and its labels Presence of interpretation text

Presence of map title

Completeness of title: information on geographical area, date of event, thematic content

Type of map background

Information on conditions related to access, use and information sharing

Responsibility assumption (on data set or information sources)

(26)

26 Table 3 continued

Validation protocol

category Parameter

Usability metadata Metadata - Description of data sources used Metadata - description of processing steps

Metadata - information on quality control procedure used Metadata - information on known sources of error

Metadata - information on spatial accuracy Metadata - information on thematic accuracy Metadata - point of contact

Metadata - reference datum Metadata - reference projection Metadata - coordinate system

(27)

27

6 The process of validation of parameters and results

An effective validation process does not mean specifying the right parameters it takes into consideration understanding the purpose of each parameter and how it has been presented. To ensure uniformity in terminology and to minimize mistakes in data entry, for each parameter range of answers have been defined (e.g. Yes, Partial, No), so during the assessment the answer had to be chosen from a drop-down list.

In the following text each parameter of the validation protocol was evaluated on representative sample of 760 downloaded maps, where the steps that should be made was explained and the results have been presented.

For the purpose of simplicity and better understanding, an example map has been taken randomly and will be divided in five sections A, B, C, D, E that would be recalled in the further explanation steps. Division of the crisis map in sections could be seen in Figure 14 below.

Figure 14 Crisis map division per sections

6.1 Usability general information

When a user starts to use the map it is very important that he is able to clearly understand the main information related to the event and map production. An effective and very short summary of the event presented by the map should be shown in the title and, possibly, in the subtitle.

 Date and place of crisis event

The event has started, the crisis map production can be activated. It is important to note that date of the crisis in most of the cases was not the same as the map production date. Information regarding the date and place could be found in the title of crisis map (see Figure 15).

(28)

28 Figure 15 Crisis map title components

Validation process

Presence has been checked of all parameters from the list defined in Table 3 which were making the group of the category, usability general information.

Results

All parameters from the group have been present in all evaluated crisis maps, only Date of the crisis event shown to be missing in 4 maps out of 760.

6.2 Reliability of the information content

Reliability is generally defined as the degree to which the information presented in a product is similar to a reference, i.e. the degree to which the result or specification can be depended on to be accurate; it is free of errors and complete. From the user point of view, reliability is a statement about how much the user is confident with the information that he is getting from product. Reliability is characteristic that user evaluates and his judgment may depend on the purpose of the map.

How much information content is reliable in general can be checked by comparing the product information with the reference source. In the following text, the crisis maps‟ parameters that can be used for evaluation of the reliability of product by visual interpretation will be presented.

 Information on occlusion of EO sources (clouds) expressed as a percentage of the spatial extent

(29)

29

Determination of areas that have not been analysed due to the presence of clouds is essential for the user to better understand the information content. Crisis maps have been made on a base of satellite images, where clouds could be obstacle for the interpretation of crisis situation on land cover.The information could be expressed as percentage of the spatial extent of the mapped area and/or directly represented on the map (i.e. occluded areas represented with a mask and defined in the legend) (Broglia M. et al. 2010). In a crisis map information about the cloud occlusion was present in a Data Sources section that could be found in a section D of crisis map and its exact location could be seen in Figure 16 below. If information about the cloud coverage is absent it was consider to be 0%.

Figure 16 Information on clouds occlusion Results

Validation of cloud coverage gave the result that 46% of observed maps have presence of clouds, showing that almost half of the sample have had this type of obstacle that could be limitation in the interpretation of the ground situation after the event.

(30)

30

 Legend semantic definition for thematic data

Every feature represented on a map should be defined in understandable form. This could be done in the legend, where it has been explained the meaning of symbology used on a map. This parameter was analysed by checking absence or presence of semantic definition of all symbols that are defined in the legend, section C (see Figure 17).

Results

The sample has shown that all crisis maps were providing this parameter completely.

Figure 17 Position of legend, semantic definition for thematic data

6.3 Consistency of the information support

A consistent product is the one that does not contain contradictions. Consistency checks internal contradictions of the product, between different components of the same product. When a contradiction between two or more elements is detected, it shows that the product contains error and it can lead to further investigations and products improvements.

 Consistency between map and legend symbols

The map and the legend should contain a consistent set of symbols. Symbols used in the map should be named in the legend, and vice versa symbols named in the legend should be used in the map. The following Figures 18 and 19 are showing some examples of inconsistency that have been found during the verification process. In the first example symbol describing highly damaged buildings was present in the legend but was not shown in the map, on contrary second example have had railway station symbol presented in the map, but its description in the legend was missing

(31)

31 Figure 18 The legend highly damaged for building grading is missing in the map

Figure 19 The railway station symbol is present in the map but its explanation is missing in legend symbols

Detail level of this parameter was performed by observing and comparing symbols presented in map and legend. Absence or presence of this parameter was expressed by following list:

1. Present – crisis map satisfied all aspects of consistency between map and legend symbols

2. Partially missing symbols in map – there was one or few symbols that were not present in the map, but were present in the legend

3. Partially missing symbols explanation in legend – one or few symbols were present in the map but in a legend section they were not explained

4. Absence of symbols in map - there were more than half or all symbols that were not present in the map, but were present in the legend

5. Absence of symbols explanation in legend – more than half or all symbols were present in the map but in a legend section they were not explained

(32)

32 Results

After performing the assessment it could be observed that the sample have had high presence of completed parameter having 89% of maps that satisfied all aspects of consistency (see Table 4), even this parameter was very good there is still space for improvements.

Table 4 Presence of consistency between map and legend symbols over the sampled maps

Present Partial (missing some symbols in the map) Partial (missing some symbols in the legend) No (missing symbols in the map) No (missing symbols in the legend) no. of maps 675 39 1 39 6 percentage 89% 5% 0% 5% 1%

 Consistency between declared scale and resolution of the images used to produce the map

The representation scale could be declared with scale bar or by numeric ratio e.g. 1:100,000. Adequacy of information sources with the declared nominal scale could be verified only if the physical dimension of paper sheet of the crisis map was provided. This information could be provided with resolution of the input imagery that could be found in the general information of the map layer e.g. GSD 2 0.5 m see Figure 20.

Figure 20 Information regarding the resolution of base map imagery

Having information regarding the resolution it was possible to calculate the range of map scale in which the map could be represented. In Table 5 the acceptable scale range with respect to the imagery resolution have been presented (table source presentation at Politecnico di Milano of Copernicus EMS presented by Broglia M. visit www.jrc.ec.europa.eu): checking the range scale of given GSD in crisis map it was possible to realize if the information sources are adequate with respect to the declared nominal scale (were in the range from table) or not.

2_{GSD stands for Ground sample distance}_{and is used}_{in a digital photo of the ground from air or space where} presents the distance between pixel centers measured on the ground

(33)

33 Table 5 Range of acceptable nominal scale according to the imagery resolution values

Resolution of input imagery

[m]

Maximum

scale Ideal scale

Minimum scale 0.25 1: 500 1: 1000 1: 5000 0.5 1: 1000 1: 2000 1: 10000 1 1: 2000 1: 4000 1: 20000 2.5 1: 5000 1: 10000 1: 50000 5 1: 10000 1: 20000 1: 100000 10 1: 20000 1. 40000 1: 200000 15 1: 30000 1: 60000 1: 300000 Results

Analysing this parameter it has been found out that 70% of products were consistent with respect to the declared scale and resolution. In Table 6 the distribution of this parameter per producer was explored: almost all producers have shown analogous behaviour considering this parameter. The only producer that has lower percentage of adequacy of information source with the declared nominal scale is DLR, this could be caused by the lack of standardization practice.

Table 6 Adequacy between declared scale and resolution of the images of producers expressed in percent.

SIRS ITHACA GAF e-GEOS DLR SERTIT

69% 69% 71% 67% 50% 73%

6.4 Usability of the product

Usability is the degree to which something fits to be used. It is directly related to the communication of information to users. When usability was evaluated, users‟ point of view was taken in consideration. The crisis map use includes reading, interpreting, analysing and integrating information provided by map. Therefore, it is crucial to eliminate any misunderstanding and ambiguities. Checking the usability of the product was one crucial aspect of the validation, because its aim was narrowing the gap between the service provider and the end user.

(34)

34

 Contrast between background and thematic entities

Colour conventions were checked and symbols visibility in comparison with the background map. The logic of use of different colours should be appropriate with respect to user‟s habits and common sense: e.g. Flood should be represented with

Blue colour; Burned area should be represented in Red. The following rules should

be checked when evaluating this parameter:

 The selected legend colour should be easily noticeable when reviewing map

 The colour should give the correct impression of what feature represents and colour shades should show how features changes in magnitude across the map

Figure 21 Example of bad contrast between background and thematic entities

In a Figure 21 above examples of bad representation of legend colours with respect to background could be seen. In the first map (Figure 21a), grey colour was present in high percent: rooftops of houses, in the background map, are light grey, transportation network of the area is presented with grey, the points of interests such as the airport, bridge and heliport are shown as well with the same tones, therefore it was very difficult to distinguish and single out all features. The second map (Figure 21b) presented fire in the UK and at the first look not experienced user would say that red areas were burnt, but it is not true the orange area was. Settlements are chosen to be represented with pink and red colours that are very similar with the colour of background map, thus they are hardly noticeable.

A

(35)

35 Validation process

For the evaluation purpose scale Good, Fair and Bad contrast have been made. Good contrast between background and thematic entities has been assigned to maps where all features were easily recognisable, fair contrast has been assigned to maps where some of the features were hard to be distinguished or recognized and bad contrast maps with very poor choice of colours.

Results

Contrast between background and thematic entities have shown to be at good, since there were no maps with bad contrast during the five years period, 98% of maps have had all features easily recognizable and in just 2% of observed maps some features were hard to be distinguished.

 Symbols easily differentiable

According to Broglia M. et al. (2010) the differentiability of symbols is related to spacing, size, orientation, shape, arrangement and texture (combining the visual variables spacing and size) of visual variables. The relevance of map symbols was evaluated taking into account these basic visual variables. A significant example found during the validation process where symbols were hardly differentiated was shown in the first map of Figure 21a of previous paragraph, where transportation network and points like airport, bridge and helipad were presented with the same range of colours. Another characteristic that has been taken into account for evaluation of this parameter was symbols overlapping causing difficulties to interpret them, as it is shown in Figure 22.

Figure 22 Undetermined symbols due to their overlapping Validation process

Evaluation of this parameter was consistent of detailed observation of all symbols presented in the crisis map. It could be expressed as Good (all symbols were easily differentiable), Fair (some symbols were hard to be distinguish) and Bad (most of the features presented in the map were hard to be differentiated).

Results

The results have shown presence of maps with symbols that were overlapping in 9% of the sample maps. The biggest percentage of 80% had maps with symbols that are easy to differentiate and 1% of those where it was very hard to make difference between symbols.

 Scale bar

Scale bar provide a visual indication of the size of features, and distance between features, on the map.The representation scale could be declared in different ways, the most used representation is with scale bar (see Figure 23), that in crisis map could be found in section C.

(36)

36 Figure 23 Scale bar

Results

All crisis maps that have been validated had presence of Scale bar.

 Presence of overview map

An overview map is a smaller map included within the context of a larger map. Its purpose is to show the primary mapped area in relation to large, more recognizable area.

In the evaluation process not only the presence of overview map (in most cases it was present in section B after the title see Figure 24) has to be checked but also was controlled if the location of the mapped area was highlighted and easily identified.

Results

All crisis maps validated for a period between 2013 and 2017 have presence of overview map.

Figure 24 Overview map

 Coordinate graticules/grid and its labels

The graticule represents the projected position of the geographic coordinates at constant intervals, or in other words the projected position of selected meridians and parallels. As presented in Validation protocol for Emergency response (Broglia M. et al. 2010) the shape of the graticule depends largely on the scale e.g. on the 1:50,000 topographic map, graticule lines can be represented at every 5 minutes and grid lines at every kilometer.

The presence or absence of graticules and grids on a map as well as their labelling was checked during the validation and size of piece of the map area covered by them.

(37)

37 Results

Almost all crisis maps being part of representative piece had suitable coordinate graticules only 3 out of 760 maps did not have it

 Presence of interpretation text

The interpretation text is a descriptive part of crisis map it provides information about the elements included by map (location, description of data etc) and it contributes to the usability of a product. In addition to the presence/absence of the interpretation text, its completeness and clearness will be checked.

Results

All maps have shown to have clear and completed interpretation text.

 Presence of map title and its completeness

Map title is usually present in the top of text section of the crisis map, more precisely in the top of section B. The appropriateness and completeness of the titles was assessed as well. The title should have information on geographical area (city and country), date of event, type of the event and thematic content (type of crisis map) as shown in Figure 25.

Figure 25Components of the title of crisis map Results

750 maps out of 760 had complete map title the remaining 10 crisis maps were missing one of the title elements, but title in general was present.

 Type of map background

The most common method used for representing the background of crisis map was by satellite image. When it is possible the use of a topographic map is preferable, because it is characterized by interpreted content, adequate to the map scale. In view of the importance of the type of background for map readability this aspect has been checked.

Results

In general, only 21.18% of the 760 checked maps displayed a Topographic map as a background, while most of maps (75.61%) displayed a satellite image as a

(38)

38

background. Exploring more obtained results it has been found outthat the presence of DEM (Digital elevation model) as a background was significant only for flood events, specifically they were used in 2% of the flood maps (Figure 26). Taking in consideration type of background map with respect to the event type, as shown in graph, it could be seen that landslide and fire events were almost equally presented with satellite and topographic maps. Industrial accidents were having more topographic maps as a background (67%). For all other events satellite image was present in more than 70% of the maps.

In Figure 27 the type of map background with respect to the continent where the event occurred has been shown. It was possible to notice that the type of layer used as background have changed with respect to the developed continents. In continents as Europe, North-America and Asia there was much higher presence of topographic maps.

Figure 26 The use of different map background in the sample of maps, with respect to the type of event 0 50 100 150 200 250 300 350 N u m b e r o f m ap s

Type of the event

no background DEM

topographic map satellite map

(39)

39 Figure 27 The use of different map backgrounds in the sample of maps, with respect to the

place of event.

 Information on conditions related to access, use and information sharing Access limitations are restrictions and requirements for accessing the information resource. This information describes any constrains or legal limitations for accessing the information resource or its component products or services. This includes any access constraints applied to assure the protection of privacy or intellectual property and any other special restrictions or limitation on accessing the information. It is recommended to explicitly specify whether the product is governed by access limitations. Copernicus service has an open access, it could be expected that majority of maps have had an access without restrictions. This parameter could be checked in section D paragraph (see Figure 28) Dissemination/Publication where the statement about the restriction could be found.

Figure 28 Information on conditions related to access, use and information sharing 0 50 100 150 200 250 300 350 400 450 500 N u m b e r o f m ap s Continent DEM topographic map satellite map

(40)

40

Evaluating this parameter it has been observed that 4 out of 760 maps had restricted access and 94 did not have restriction, the rest of the evaluated maps did not have any information about the access and use of information provided by crisis map. Analysing results more in depth it showed that most of maps produced in 2013 and 2014 have hade information on conditions related to access, use and information sharing, all other maps produced after did not have information about this parameter, from where it could be assumed that the convention regarding this parameter have been changed in 2015.

6.5 Metadata

Metadata in general is a set of data that describes and gives information about other data. The user should be aware of the quality and the characteristics of the product that he would like to use. Metadata could help the user to choose product understanding if it suits to his purpose.

 Metadata - Description of data sources used

This metadata section was showing from where the information used for maps production were taken, as well as where the background layer and/or other layers were acquired. That information could be really helpful to understand the reliability of a product. Parameter related to description of data sources used is checked in section D, paragraph named Data Source, see Figure 29.

Figure 29 Information of data source used

Information about the data source from the picture above was an example of good description of data source, since it provides detailed information related to the source of used data.

 Metadata - description of processing steps

Metadata describing processing steps define how the crisis map has been made, taking in account description of tools and techniques used for obtaining data from sources. In a crisis map this information could be found in ”Map Production” paragraph of section E (see Figure 30).

(41)

41 Figure 30 Information about the processing steps

Information about the processing steps from the figure above is an example of good description where details of processing steps were presented.

 Metadata - information on quality control procedure

These metadata section explored if the final product (in this case the crisis map) have been checked and who was an inspector that performed the quality check, it could be found in the “Map Production” paragraph of section E. Good example of quality control check was considered when name of inspector was present, partially good information was considered for cases when there was statement that quality check was performed but without further information.

Figure 31 Metadata – information on quality control procedure

 Metadata - information on accuracy

Accuracy could be defined as the degree or closeness to which the information on a map matches the “real” values, which corresponds in practice to a higher precision value. Accuracy is a measure of correctness. For crisis maps both spatial and thematic accuracy were of a great importance.

(42)

42

Spatial accuracy

Spatial accuracy (or „positional accuracy‟) refers to the accuracy of the spatial component of a dataset, it could tell with what precision objects were presented in map with respect to their position on a reference dataset.

Thematic accuracy

Thematic accuracy could show how well the class feature on the map corresponds to what was really on the ground.

Both accuracies should be checked performing in-depth analysis of metadata documentation. Satellite based crisis maps could present big constraints to the accuracy computation: the significant added value of satellite products could give the possibility to map the area without visiting it and the only way to correctly compute the accuracy, in particular thematic accuracy, was possible by checking a certain number of points in the field, or at least with respect to better source. This activity could not be performed, in most of the cases, in the few days or hours dedicated to the map production and without considering significant additional costs. However, an indication on map accuracy or reliability should be provided to the user. Good information regarding accuracy corresponded to the information where the accuracy measure was provided with addition how the producer came to that value, partial information provides user just an accuracy value without saying how the producer obtained it. Information about both spatial accuracy and thematic accuracy were presented in “Map Production” paragraph of Section E (see figure below).

Figure 32 Metadata – Information about the spatial and thematic accuracy

 Metadata - reference datum and projection

A map projection is mathematical method for transforming the curved surface of the Earh into a flat surface 2-D . A datum is a point of reference. A projection uses the datum as a point of reference, it's location on earth. For example WGS84 is map

(43)

43

 Metadata - coordinate system

A geographic coordinate system uses a three-dimensional spherical surface to define locations on the earth. The most used coordinate system for crisis maps is WGS84.

During verification process the focus was on presence of coordinate system definition. Information about the coordinate system and reference datum and projection were placed in “Cartographic Information” that could be found in section C of crisis map (see Figure 33)

Figure 33 Information regarding the coordinate system and reference datum and projection

 Results related to Metadata

Parameters connected to metadata (presented previously) have been evaluated (see Figure 34). On the 760 sampled maps it has been assessed that:

 4% of maps missed completely description of data source used and 6% description of processing steps.

 12% of sampled maps did not have information about known source of error (thematic accuracy)

 Spatial accuracy was the weakest parameter, since 48% of maps did not have any information about spatial accuracy and 15% had it but it was not complete.

 all other parameters of metadata satisfied all criteria and were presented good in almost all sampled maps

Crisis maps quality check and a proposal of integration with data from social media sources

POLITECNICO DI MILANO