A critical analysis of autonomous robotics competitions as experiments

(1)

Politecnico di Milano

Scuola di Ingegneria Industriale e dell'Informazione

Corso di Laurea Magistrale in Ingegneria Informatica Dipartimento di Elettronica, Informazione e Bioingegneria

A CRITICAL ANALYSIS OF AUTONOMOUS

ROBOTICS COMPETITIONS AS EXPERIMENTS

Relatore: Prof. Viola SCHIAFFONATI Correlatore: Prof. Francesco AMIGONI

Tesi di Laurea Magistrale di:

Selenia Vincenza RUSALEN Matr. 815974

(2)

(3)

Computer science is not a science, and its ultimate signicance has little to do with computers - Hal Abelson and Gerry Sussman

(4)

(5)

Acknowledgments

I would like to express my gratitude to my supervisor Prof. Viola Schiaonati for her continuous support to my work, for her insightful suggestions, and for her indus-try in guiding me in the selection of the most appropriate and interesting material.

I would also like to thank my assistant supervisor Prof. Francesco Amigoni, especially for his valuable advice and for his meticulous attention in revising my thesis.

(6)

(7)

Abstract

By virtue of its hybrid nature, combining mathematical, scientic, and engi-neering activities, computer science encountered diculties in developing a unied rigorous experimental methodology, since some of its peculiarities (e.g., dealing with artifacts) do not t easily into the traditional conception of experiment borrowed from natural sciences namely a set of observations and actions, performed in a controlled context, to support a given hypothesis. Indeed, taking for instance the robotics eld into consideration, we observe that predicting the behaviour of robotic systems in their interaction with complex environments is deemed unfeasible due to excessive modeling and computational complexity and to the impossibility of achiev-ing full control on the experimental environment. However, since the development of experimental procedures is deemed benecial in order to strengthen the scientic reputability of the area, we can observe a growing interest for this topic, in par-ticular in autonomous robotics. In this regard, robot competitions (i.e., events in which robots engage in a contest to determine the best performing one in solving a specic task), which have been developed for decades as a valid research aid and which already provide rather standardized test beds allowing for direct comparison of dierent robotic solutions, have been recently suggested as potential candidates for the role of experiments.

The aim of this thesis is to analyze similarities and dierences between experi-ments and robot competitions in order to ascertain the meaningfulness of this pro-posal, as well as the boundaries and the conditions of application.

In order to do so, we select some particularly interesting robot competitions, evidencing the discrepancies between them and experiments and suggesting counter-measures whenever possible. After observing that most of the encountered diculties in equating the two activities such as for instance the diculty of recreate the same scenarios for independent testing are attributable to the aforementioned peculiari-ties which are typical of the whole eld of robotics such as for instance the diculty of controlling the environment , we also suggest a dierent approach to the notion of experiment in order to privilege the aspect of gaining knowledge with reasonable eort over the focus on controlling the experimental environment, thus possibly clas-sifying conveniently adjusted robot competitions as a particular type of experiments,

(8)

(9)

Sommario

L'informatica non dispone attualmente di una metodologia sperimentale consoli-data, principalmente in virtù del fatto che la sua natura ibrida, comprendente atti-vità riconducibili sia alla matematica e alla scienza sia all'ingegneria, non consenta di adottare agevolmente una pratica sperimentale mutuata dalle scienze naturali ed improntata sul concetto di esperimento controllato ossia una serie di osservazioni ed attività eettuate in ambiente controllato per fungere da supporto ad una data ipo-tesi. Nella fattispecie, la branca della robotica autonoma ha recentemente registrato un crescente interesse per la possibilità di sviluppare una metodologia sperimentale rigorosa che consenta di accrescere la credibilità scientica del settore e formulare riferimenti comuni per la valutazione delle prestazioni dei sistemi robotici. A tal ne è stata avanzata l'ipotesi di impiegare come esperimenti, previa opportuna riformu-lazione, degli strumenti di supporto alla ricerca già ampiamente consolidati quali le competizioni robotiche, in considerazione del fatto che queste ultime costituiscano già un banco di prova discretamente standardizzato per confrontare le prestazioni di soluzioni robotiche dierenti.

Lo scopo di questa tesi è valutare la percorribilità di una simile ipotesi, identi-candone condizioni e limiti di applicabilità.

Per perseguire tale obiettivo sono state selezionate alcune competizioni roboti-che, ritenute di particolare rilevanza per dimensione, longevità o scopo, e si sono indagate somiglianze e dierenze con gli esperimenti controllati. Dopo aver riscon-trato alcune divergenze sostanziali pregiudizievoli per una completa identicazione delle due attività, si è osservato che tali divergenze siano in gran parte riconducibili ad alcune peculiarità del settore della robotica quali ad esempio la manipolazione di manufatti e la dicoltà nell'esercitare un controllo completo sulle condizioni am-bientali e si è pertanto proposto di dirigersi verso una denizione di esperimento che privilegi l'acquisizione di conoscenza rispetto alla controllabilità delle condizioni di svolgimento, ossia l'esperimento esplorativo. In tale accezione si è potuto quindi concludere che con gli opportuni aggiustamenti le competizioni robotiche possano essere impiegate come esperimenti esplorativi.

(10)

(11)

List of Figures

3.1 RoboCup Soccer Humanoid League sample match . . . 19 3.2 RoboCup@Home sample contendant performing a Wake me up test . 21 3.3 DARPA Robotic Challenge sample contendant cutting a hole in a wall 22 4.1 Neobotix MM-500 . . . 28 4.2 Sample RoboCup Soccer League contendant . . . 29 4.3 RoboCup Soccer Humanoid League 2015 - Arena structure . . . 33

(16)

(17)

List of Tables

4.1 Public . . . 31

4.2 RoboCup Soccer Humanoid League - Arena classication . . . 35

4.3 RoboCup Soccer Humanoid League - Rules classication . . . 36

4.4 RoboCup@Home - Arena classication . . . 36

4.5 RoboCup@Home - Rules classication . . . 36

4.6 DARPA Robotic Challenge - Arena classication . . . 37

4.7 DARPA Robotic Challenge - Rules classication . . . 37

4.8 RoCKIn@Home - Arena classication . . . 37

4.9 RoCKIn@Home - Rules classication . . . 38

4.10 RoCKIn@Work - Arena classication . . . 38

4.11 RoCKIn@Work - Rules classication . . . 38

4.12 ASC - Arena classication . . . 39

4.13 ASC - Rules classication . . . 39

4.14 HRATC - Arena classication . . . 40

4.15 HRATC - Rules classication . . . 40

4.16 MMC- Arena classication . . . 40

4.17 MMC - Rules classication . . . 41

(18)

(19)

Chapter 1

Introduction

The diculties in developing experimental procedures in computer science can be explained considering the hybrid nature of the area itself. In fact, as suggested by Denning and Freeman [20], computer science combines the analytic activity of discovering general laws and the study of abstract processes and structures, as well as the synthetic activity of building new artifacts. Consequently, the application of the traditional notion of controlled experiment borrowed from natural sciences is not properly tting. In order to overcome this situation, we consider some more innovative notions of experiment such as the ones proposed by Hansson [25] and Tedre [39], focusing respectively on the importance of action and on the engineering component of computer science.

Notwithstanding the diculties, the adoption of a systematic experimental prac-tice has been often advocated over the years. For instance, we could take into consideration the recommendations of Denning [18, 19], Langley [30], Morrison and Snodgrass [32], and Tichy [41], suggesting the adoption of a systematic experimental practice in order to strengthen the scientic reputability of the eld, accelerating its progress, and building a systematic and reliable base of sectorial knowledge.

Focusing on the robotics eld, we observe a growing interest in experimental methodology, in particular with regard to autonomous robotics, aiming at aligning with scientic disciplines and developing objective benchmarks for product compari-son. For instance, we consider the trends summarized by Amigoni and Schiaonati [9] regarding the recent increment of the number of experiments and the increasing stan-dardization of the platforms as well as the diculty in comparing dierent systems on equal ground and the little attention to statistical analysis.

In this context, robot competitions have been considered a valid research aid for a long time, by virtue of the fact that they contribute notably to the promotion of the research in the eld, to a positive impact on public opinion, to an ecient fund raising, to the education and recruitment of the next generation of researchers, and to the building and maintenance of a community, as remarked for instance by

(20)

1. Introduction

Bonasso and Dean [13] and Braunl [16].

In addition, since competitions already provide rather stardardized test beds allowing for direct comparison of dierent robotic systems, they have been recently suggested as candidates to be recasted as experiments, for instance by Anderson, Jenkins, and Osentoski [11], or to be used as benchmarks for research, for instance by Behnke [12].

Starting from this debate, the aim of this thesis is to analyze similarities and dif-ferences between experiments and competitions in the eld of autonomous robotics, in order to ascertain the meaningfulness of the proposal of using competitions as experiments, its limits, and its conditions of application. In order to do so, we select some particularly interesting robot competitions in terms of longevity, popularity, pervasiveness, and variety of themes and purposes, evidencing the discrepancies be-tween them and experiments and suggesting countermeasures whenever possible. Then, we observe that most of the encountered diculties in equating the two ac-tivities for instance the diculty of recreate the same scenarios for independent testing are attributable to some peculiarities which are typical of the whole eld of robotics for instance the fact that dealing with artifacts is involved and the di-culty of controlling the environment. Consequently, we suggest a dierent approach to the notion of experiment in order to privilege the purpose of gaining knowledge with reasonable eort over the focus on controlling the experimental environment, thus possibly classifying conveniently adjusted robot competitions as a particular type of experiments, called explorative experiments [8].

The structure of this thesis is described hereafter. In Chapter 2, we present the concept of experiment and its possible inections with regard to specic areas of our interest, namely computer science and robotics. We also present the main character-istics and purposes of experimentation, as well as the recent trends concerning robot competitions.

In Chapter 3, in order to consider the possibility of assimilating robot competi-tions to experiments, we introduce some particularly interesting robot competicompeti-tions, providing a detailed description of their aims, characteristics and organization. The selected competitions are deemed relevant in terms of popularity, pervasiveness, and variety of themes and purposes. In particular, we consider the worldwide largest robot competition, namely RoboCup [31,34], a competition held by a major agency for the development of new technology for military use, namely the DARPA Robotic Challenge [17], a recent eort to enhance the scientic value of robot competitions, namely the RoCKIn Robot Challenge [35,36], and a series of competitions with varied application elds and purposes, namely the competitions held by IEEE International Conference on Robotics and Automation [13,22].

(21)

1. Introduction according to some criteria we propose, in order to formalize the description of the current situation. In fact, we identify the most characterizing aspects of robot com-petitions, such as the presence of robots performing various kinds of tasks in an arena according to a set of rules. Consequently, we introduce a taxonomy based on these aspects, formulated according to the existing literature [13, 44] and to direct observation of the considered competitions. Since a thorough documentation of every aspect of experimental environment and procedures is required in order to comply with the standards of rigorous experimentation, we devote particular attention to the description of the competition environment, namely the arena, and the challenge modes, namely the set of rules. Thereby, we analyze any described aspect of arenas with regard to the completeness and the quantiability (i.e., the use of a quantiable measurement process to express quantity) of the descriptions. Also, we isolate some distinguishing features in sets of rules, such as the presence of human commands issued to the robots, the comparison mode (e.g., the possibility for the robots to interact with each other during the performance), the scoring system (e.g., the allo-cation of scoring points based on human judgement or measurements), the presence of a ranking and the evaluation scope (e.g., the evaluation of the performance of the whole robot or of a single ability of its). Hence, we apply our taxonomy to the previously selected robot competitions, with the aim of assembling an overall picture of the present situation enabling us to evaluate the assimilability of competitions to experiments.

Finally, in Chapter 5, we discuss the main similarities and dierences between experiments and robot competitions in relation to the precedent analysis, identifying some possible strategies aiming at dealing with the current problems. After observ-ing the discrepancies of the present competitions with experiments, we draw some general conclusions on the possibility of assimilating the two activities. Indeed, we observe that the diculties encountered in the development and the enactment of experimental procedures are typical of the whole eld of robotics. As a consequence, we assume that these diculties originate from the fact that the peculiarities of the eld cannot easily being accommodated under the traditional concept of experiment. Therefore, we introduce the notion of a specic kind of experiment, such as the ex-plorative experiment, in order to take the aforementioned peculiarities, such as for instance the fact that handling of artifacts is involved and achieving full control on the experimental environment is often unfeasible, into consideration. As a result, we conclude that classifying conveniently adjusted robot competitions as explorative ex-periments could be possible, provided that their present gap is conveniently narrowed by addressing the observed problems.

Future development of this idea may involve the formalization of an organized framework of guidelines for the design of a competition with a purposeful experi-mental value, as hinted in Chapter 5. In fact, while acknowledging the limits of

(22)

1. Introduction

competitions, for instance with respect to the possibility of recreating the same sce-narios in dierent times and places for independent crosschecks, we glimpse scope for improvement with regard to the accuracy of documentation and the objectivity of the evaluation.

(23)

Chapter 2

Experiments: Conception and

inections

The traditional notion of experiment as a controlled experience i.e., a set of observations and actions, performed in a controlled context, to support a given hy-pothesis does not describe accurately what needs to be performed in computer science, which is an area combining the analytic activity of discovering general laws and the study of abstract processes and structures, as well as the synthetic activity of building new artifacts. Thus, some elds belonging to this area more specically the branch of autonomous robotics have diculties in developing a systematic ex-perimental practice. As a result, a growing interest for this topic can be observed. Among the various research trends, we consider a particularly interesting one sug-gesting to employ an already existing research aid, namely robotics competitions, as experimental activity. This chapter provides an overview on this subject.

2.1 Denition and basic principles

A widely accepted conception of experiment has emerged during the Scientic Revolution of the XVII century, dening an experiment as a controlled experience, namely as a set of observations and actions, performed in a controlled context, to support a given hypothesis. Borrowing from Hansson [25], we could focus on the centrality of action to distinguish between experiments and non-experimental obser-vations (i.e., the acquisition of information about a phenomenon without actively interacting with it), dening an experiment as a procedure in which some object of study is subjected to interventions that aim at obtaining a at least partially pre-dictable outcome. For instance, we could consider the dierence between observing a drop of water through a microscope and observing the same drop of water after having stained some specimen microorganisms with a colored chemical reagent in order to highlight them [38].

(24)

2. Experiments: Conception and inections

We could also underline the fact that traditionally taking place under controlled conditions is also a characteristics of experiments, since this kind of approach brings some advantages such as the possibility to study a larger number of combinations of circumstances than under unmanipulated conditions, to achieve better control over these circumstances, and to couch investigations in the terms of cause-eect relationships [25].

Generally, the main purposes of experiments are testing theories, choosing be-tween opposing positions, and verifying hypotheses, but they can also serve the purpose of improving instruments and our abilities to manipulate objects, therefore enhancing scientic knowledge.

Indeed, experiments can for instance be performed in order to: investigate some phenomena devoid of supporting theories;

show that an accepted theory could be incorrect, by detecting some anomalies; provide evidence of the existence of the entities involved in a theory;

conrm, refute, or give hints about the mathematical structure of a theory, for example by means of empirical measurements;

choose between competing theories, by showing compliance of a phenomenon with one of them;

carry out checks and calibration, by means of reproduction of known phenom-ena by the experimental apparatus.

Anyway, although their purposes may dier, some shared basic principles consti-tuting the core of the modern conception of experiments can be identied [5] as follows:

comparison, i.e., the possibility to know exactly what has been already done in the past within the same eld of research in order to accurately compare new results with old ones. This principle serves the dual purpose of avoiding to repeat uninteresting experiments and suggesting new research directions. Consequently, a necessary feature to be comparable with other experiments is a thorough documentation of every aspect;

reproducibility and repeatability, i.e., respectively the possibility for researchers to verify independently of each other the results of a given experiment and the possibility to perform a number of trials possibly at dierent times and in dierent places to assess the validity of the outcome. This dual principle aims at guaranteeing, as far as possible, that the result has not been achieved by chance, but is systematic. As in the case of comparison, reproducibility requires documentation and stardardization of initial conditions and instrumentation;

(25)

2. Experiments: Conception and inections justication and explanation, i.e., the possibility to explain the obtained data in the light of a conceptual framework and to interpret it in order to derive the correct implications. In this respect, we could also underline the fact that experiments are enabled to discover any potential deciency or error in that very same conceptual framework.

A distinction between dierent kinds of experiment is also possible [25]. In fact, we could distinguish between directly action-guiding and epistemic experiments, namely between respectively experiments whose main purpose is action knowledge (i.e., the ability to intervene on phenomena, inuencing the outcome in a desired way), and experiments whose main purpose is factual knowledge (i.e., cognition of mechanisms underlying natural processes). Hansson states that an experiment is directly action-guiding if and only if it satises the following two criteria:

the outcome looked for should consist in the attainment of some desired goal of human action;

the interventions studied should be potential candidates for being performed in a non-experimental setting in order to achieve that goal.

Typical examples of this kind of experiment are clinical trials, since they are usually performed in order to get a benecial eect of human health (e.g., pain reduction in the case of analgesics) and they are meant to result in a treatment of established and ordinary use in a clinical context. Similarly, we could consider agricultural eld trials such as the selection of specic kind of seeds, or many technological tests such as tests of the longevity of light bulbs.

We could underline the fact that, despite not having the same amount of devoted attention in literature both from a historical and a philosophical point of view as epistemic experiments, directly action-guiding experiments are historically well rep-resented in non-academic elds. Indeed, this kind of experimentation was extensively performed by farmers and craftspeople of dierent trades since prescientic times. For instance, we could consider the systematic experimentation on the composition of glass performed in the early Islamic period in Raqqa, in eastern Syria, including a systematic search for optimal proportions of the main ingredient by means of a chemical dilution line [25].

The distinction between epistemic experiments and directly action-guiding ex-periments is particularly useful for comparing exex-periments with robot competitions, as we can see in the following chapters of this thesis (Chapter 5).

2.2 Experiments in Computer Science

Unsurprisingly the inherent role of experiment in computer science is debated, since the classication of computer science as science is argued itself. In fact,

(26)

com-2. Experiments: Conception and inections

puter science can be regarded as a combination of the the traditional paradigms of science, engineering, and mathematics [20], considering these disciplines as fol-lows [21]:

science, i.e., the analytic activity of uncovering the laws of the universe; engineering, i.e., the synthetic activity of building new artifacts which are

prac-tically useful;

mathematics, i.e., the construction and study of abstract processes and struc-tures independently of any potential use.

However, the development of a systematic experimental practice as well as exper-imental facilities and benchmarks in computer science has been recommended by many [30, 32, 41] over the years by virtue of some expected benecial eects such as strengthening its scientic reputability [18], building up a reliable base of knowledge, getting new unexpected insights, accelerating progress by means of pruning of fruit-less approaches [41], and coping with the lack of theoretical evidence when dealing with high complexity [20,30].

Despite being often hoped for, the aforementioned systematic experimental prac-tice encountered some diculties, as the very practical transposition of the concept was subject to many dierent interpretations over the years [38]. In fact, even if in general terms experiments in computer science can be intended as the empirical practice to gain and check knowledge about a system, there are multiple ways in which implementation can be articulated [4].

Historically, an oversimplied conception of experimentation in computer science spread, possibly fueled by some inuential opinion such as the denition given by Newell and Simon in 1976, dening computer science as an experimental science in the following terms:

Computer science is an empirical discipline. We would have called it an experimental science, but like astronomy, economics, and geology, some of its unique forms of observation and experience do not t a nar-row stereotype of the experimental method. None the less, they are experiments. Each new machine that is built is an experiment. Actu-ally constructing the machine poses a question to nature; and we listen for the answer by observing the machine in operation and analyzing it by all analytical and measurement means available. Each new program that is built is an experiment. It poses a question to nature, and its behavior oers clues to an answer. Neither machines nor programs are black boxes; they are artifacts that have been designed, both hardware and software, and we can open them up and look inside. We can relate their structure to their behavior and draw many lessons from a single

(27)

2. Experiments: Conception and inections experiment. We don't have to build 100 copies of, say, a theorem prover, to demonstrate statistically that it has not overcome the combinatorial explosion of search in the way hoped for. Inspection of the program in the light of a few runs reveals the aw and lets us proceed to the next attempt. [33].

However, we could suggest that this kind of view, borrowing a traditional concept of experimentation from natural sciences without further investigation on the pe-culiarities of the eld, could not lead to the denition of a satisfying framework of practical guidelines. Indeed, this topic originated a problem which has been ad-dressed in successive waves over the years until today. For instance, some suggested that the process of supporting and testing hypotheses was of utmost importance in order to apply the traditional standards of science [18], while others underlined the centrality of performance evaluation in order to discover system limits and predict future behavior [32].

In brief, experimentation in computer science has been mainly associated with rather dierent practices such as:

the branch of research devoted to the realization of concrete systems and mostly concerned with the demonstration of the feasibility of these systems, whether software or hardware [21];

the mathematical modeling of the behaviour of complex systems by means of the implementation of an apparatus for collecting data, a hypothesis, and systematic analysis to see whether the data supports the hypothesis [19]; the evaluation of computer systems, using the standard methodologies of the

natural sciences [41].

Nevertheless, among the most recent conceptualizations we select a specic and par-ticularly innovative one categorising experiments in computing according to some common conceptions which are practically traceable, although not generally ac-cepted, and can be summarized as follows [40]:

feasibility experiment, i.e., the implementation of a novel technical solution with the aim of assessing its feasibility;

trial experiment, i.e., the assessment of the performance of a feasible system in terms of speed, resource consumption, robustness an so forth;

eld experiment, i.e., the exposure of a system to its intended context of use (e.g., having a robot car driving on a real road under various weather condi-tions);

(28)

comparative experiment, i.e., a comparison between competing systems with the aim of assessing the best performing one according to some selected metrics (e.g., comparing a processor and its ancestor on the base of the instructions per second rate);

controlled experiment, i.e., a test of a system under controlled conditions. These conceptions mostly reect the inuence of the engineering component in com-puter science, since in the ordinary practice in experiments performed in engineering disciplines eorts are mostly focused on concrete results such as verifying the eective functioning of some proposed systems, comparing their performance, and identifying their boundaries. Still, it is said that generally the rigor of experimental methods in computer science is often below the standards of traditional sciences [41].

It is anyway interesting to observe that this sort of classication focusing on im-plementation and performance assessment, even if not congruent with the distinction between epistemic experiments and directly action-guiding experiments operated by Hansson, could be yet compatible.

2.3 Experiments in Autonomous Robotics

Being computer science an agglomeration of related but relatively independent areas, it is not surprising that dierent areas are diversely aected by the various trends or research. Among the aforementioned areas, we focus on robotics, and in particular on the presently growing interest in experimental methodology shown by the autonomous robotics eld. Actually, the adoption of experimental methods is currently skewed among the various areas of the robotics eld. On one hand, in industrial robotics standardized testing procedures have been developed with the aim of measuring the robot performance and assessing the compliance with non-functional requirements such as safety. On the other hand, in other areas of robotics research, such as autonomous robotics, experimental activities are addressed with a less rigorous attitude [9]. In fact, despite the growing interest of the autonomous robotics community in the development and the enactment of experimental proce-dures, with the aim of aligning with scientic disciplines and developing objective benchmarks for product comparison, the aforementioned principles of experimental activity (Section 2.1) are not yet part of the research practice [10]. The following subsections present an overview of the current situation.

2.3.1 State of the art

A recent analysis [10] identies some modern experimental trends in autonomous robotics as follows:

(29)

2. Experiments: Conception and inections recent increment in the number of experiments, possibly due to a shift in the community toward experimentation or because of a reviewing process privileg-ing experiment-endorsed papers;

majority of simulations over real robot experiments, probably because of the increased reliability of the simulations qualifying them as a cost-eective valid alternative to real robot experiments;

increased use of standard platforms, due to cost and time eciency of the use of a well-documented exiting tools with respect to a new self-produced one; weakness of experimental comparison for systems, possibly because of the

dif-culty to compare dierent systems on equal ground. This problem might be overcome by means of the adoption of the aforementioned standard platforms; increased attention towards non-functional parameters (i.e., those aspects con-cerning the qualities of the overall system, such as security, cost eectiveness and so on, instead of its specic behaviour), since autonomous robotics has still a limited impact on industry and it has only recently begun to take a interest in the eciency of commercialized product;

little attention to statistical analysis, which is a necessary requirement for a meaningful interpretation of data;

low availability of data and code, which may be overcome by means of the creation of a repository for additional material of the papers and the use of publicly available datasets.

It could be anyway useful to underline the fact that the identied trends emerged from direct observation of published material and they are therefore a consequence of a spontaneously evolving scenario, currently lacking of generally accepted guide-lines. In the following subsection a very specic trend which we deemed of particular interest is presented.

2.3.2 Emerging trend: Robot competitions as experiments

Robot competitions are events in which robots are required to solve some selected tasks in order to be ranked according to some specic metric, thus determining the most performing one. They have been developed for decades with the aim of promoting research in the eld, positively impacting on public opinion, recruiting the next generation of researchers, and building and maintaining community [13]. In addition, it has been recognized that they can also be a useful tool for boosting education and that they can play a role in raising funds, qualifying them as valid research aid, provided that they are part of a larger, well organised research program

(30)

[16]. Moreover, a recent trend suggests the possibility that competitions could be recasted as experiments [11] or used as benchmarks, since they provide stardardized test beds allowing for direct comparison of dierent approaches for solving a same task [12]. In truth, the fact that competitions involve some detailed specications [27] about their organization such as:

the specication of competition environment;

the specication of robot requirements and constraints; specic performance metrics in order to rank the participants information about their planning and scheduling;

may sound particularly appealing, even if some adjustment in precision to comply with the standards of scientic rigor would be obviously required. While undoubtedly sharing some similarities with experiments, competitions pose some problems about their lack of reproducibility and of focus on the single robot abilities and subsystems which are requirements for rigorous experimentation as an obvious consequence of the fact that they are not intended to evaluate a specic hypothesis, to explore phenomena and to share results as experiments are. In fact, some dierences that can be immediately noticed are the following:

while experiments should aim principally at verifying and measuring in terms of performance metrics the ability of a system to solve a specic problem, robot competitions mainly aim at directly comparing the performances of some alternative solutions on a predened test bed;

experiments are more likely to measure the performance of a single component in solving a specic task rather than the ability of the overall system, while competitions are usually mainly concerned with the performance of a robot, or a team of robots, as a whole;

some basic principles of experimentation, namely reproducibility and repeata-bility, do not apply to robot competitions, since respectively recreating the very same conditions in dierent locations is typically not feasible and the repetition of each test within a competition is normally not considered for organizational reasons;

in contrast with experiments, robot competitions are subject to a temporal evolution due to the necessity to maintain a reasonable level of engagement of the public and the participants [27].

Current proposals for improving the scientic rigor of robot competitions imply in-deed a renewed attention to the functionality of the single robotic modules, a more

(31)

2. Experiments: Conception and inections detailed description of the environment and robot specications, and an attempt to introduce some kind of repeatibility within the single competition. However, even though this idea has already been partially implemented in some competitions such as the RoCKIn Robot Challenge [35,36], it is still largely unexploited [27].

In the following chapters further analysis of the present situation and related problems is presented.

(32)

(33)

Chapter 3

Robot competitions: State of the

art

In this chapter we provide an overview on the current state of the art of robot competitions by describing a set of robotic challenges which have been selected ac-cording to their relevance in terms of popularity, pervasiveness, and variety of themes and purposes. In particular, we consider the worldwide largest robot competition, such as RoboCup (Section 3.2.1), a competition held by a major agency for the de-velopment of new technology for military use, such as DARPA Robotic Challenge (Section 3.2.2), a recent eort to enhance the scientic value of robot competitions, such as RoCKIn Robot Challenge (Section 3.2.3), and a series of competitions with varied application elds and purposes, such as the competitions held by IEEE Inter-national Conference on Robotics and Automation (Section 3.2.4).

3.1 Overview

Over a quarter of a century many robot competitions have been developed. O-cially, their main purpose is to be a technological push in various application elds of robotics such as industrial machinery or logistic aid. Indeed, their involved technol-ogy, both for civilian and military use, ranges from unmanned vehicles, re-ghting robots, and autonomous underwater vehicles to maze-solving, sport playing, and ser-vice robots. Nevertheless, they result in many other benets. In the rst place, they spur the commercial development of inexpensive innovative solutions. In the second place, they draw public attention on robotics, consequently improving the public opinion and attracting investors. In the third place, they play a fundamental role in building and keeping alive a community revolving around robots.

(34)

3. Robot competitions: State of the art

3.2 Description of the sample competitions

We considered the following competitions as sample ones: RoboCup Soccer Humanoid League 2015;

RoboCup@Home 2015;

DARPA Robotic Challenge 2015; RoCKIn Robot Challenge 2015;

ICRA 2016 IEEE International Conference on Robotics and Automation developed robot challenges, such as:

◦ Airbus Shop Floor Challenge;

◦ 2016 Mobile Microrobotics Challenge;

◦ 2016 Humanitarian Robotics and Automation Technology Challenge; ◦ Formal Methods for Robotics Challenge.

The aforementioned competitions have been selected in order to analyze a range of dierent specimens, spanning from the most popular multiple-year ones to the newest and highly-specialized ones. In particular:

RoboCup is probably the most famous robot competition, and having evolved over the years it has achieved a remarkable level of structuring complexity and geographical pervasiveness;

DARPA Robotic Challenge is funded by the US Defense Advanced Research Projects Agency, which aim is the development of emerging technologies for use by the military;

RoCKIn is a recent eort to organize robot competitions focusing signicantly on benchmarking, by means of an evaluation methodology aiming at better statistical analysis and of a deeper analysis of the relation between functionality and task performance;

Robot competitions held by ICRA 2016 present us with a variety of dier-ent purposes, practical applications, organisational structures and evaluation methods.

(35)

3.2.1 RoboCup

RoboCup is an international competition aimed at promoting AI and robotics research. Although being originally concerned with robot soccer tournaments, it now also includes dedicated leagues for rescue, domestic, and industrial applications. Consequently, it consists of various competition branches, such as:

RoboCup Soccer, focusing on teams of autonomous robots competing in the game of soccer in dierent variants such as:

◦ Humanoid League, in which robots with a human-like body plan and human-like senses are involved;

◦ Middle Size League. in which middle-sized robots of no more than 50 cm diameter are involved;

◦ Simulation League, focusing on articial intelligence and team strategy and involving software players playing soccer on a virtual eld inside a computer, in the following subleagues:

2D Soccer Simulation; 3D Soccer Simulation;

◦ Small Size League, involving small robots tting within an 180 mm dia-meter circle and no higher than 15 cm;

◦ Standard Platform League, involving the use of standard robots for all teams, thus enabling concentration on software development only; RoboCup Rescue, focusing on disaster rescue and thus including:

◦ Rescue Robot League, involving multi-agent team work coordination of physical robotic agents for search and rescue;

◦ Rescue Simulation League, aiming at providing emergency decision sup-port by means of the implementation of a disaster simulator;

RoboCup@Home, focusing on the introduction of service and assistive robot technology (for instance robotic nurses) with high relevance for future personal domestic applications;

RoboCup@Work, focusing on industrial robots cooperating with human wor-kers for complex tasks ranging from manufacturing, automation, and parts handling up to general logistics;

RoboCup Logistics League, focusing on in-factory logistics applications in or-der to achieve a exible solution of material and informational ow within industrial production using coordinated teams of autonomous mobile robots;

(36)

RoboCupJunior, focusing on education by means of sponsorship of local, re-gional and international robotic events for young students. allowing them to compete in challenges of a variety of interests such as:

◦ Soccer Challenge, involving 2-on-2 teams of autonomous mobile robots tracking a special light-emitting ball in an enclosed, landmarked eld; ◦ Dance Challenge, involving one or more robots coming together with

music, dressed in costume and moving in creative harmony;

◦ Rescue Challenge, involving robots identify victims within re-created di-saster scenarios.

For our purposes, only the RoboCup Soccer Humanoid League [34] and the Robo-Cup@Home [31] have been selected as samples to be analyzed, since they are respec-tively the most popular robot competition and the largest worldwide competition for domestic service robots.

3.2.1.1 RoboCup Soccer Humanoid League

It consists [34] of a soccer tournament played by teams of one to four humanoid robots, divided according to their size in three dierent classes such as:

KidSize class; TeenSize class; AdultSize class.

It includes also some technical challenges implying dynamic walking, running, and kicking the ball while maintaining balance, such as:

Push Recovery, i.e., returning to a stable standing or walking posture after be-ing strongly pushed from the front and from the back by a swbe-ingbe-ing pendulum; Goal-Kick from Moving Ball, i.e., kicking a ball which is rolling down a ramp

into the goal, before the ball comes to a stop;

High Jump, i.e., terminating ground contact and staying in the air as long as possible, remaining upright for a minimum of three seconds after landing with-out leaving the measuring device, namely a contact device of approximately 40Ö40 cm;

(37)

3. Robot competitions: State of the art AdultSize robots can also compete in the obstacle-avoidance one versus one Dribble and Kick Competition, in which the striker has to acquire the ball and score a goal while the goalkeeper tries to block the ball.

At the end the Best Humanoid robot is selected by evaluating its overall perfor-mance according to the following criteria:

Robustness; Walking ability; Ball handling; Soccer skills.

Figura 3.1: RoboCup Soccer Humanoid League sample match

3.2.1.2 RoboCup@Home

It focuses [31] on personal domestic applications, aiming at developing service and assistive robot technology. It is organized in multiple ability and integration tests, as well as open demonstrations for the audience, as follows:

Stage I:

◦ General Purpose Service Robot, in which the robot has to understand a random set of speech commands and solve the corresponding tasks; ◦ Manipulation and Object Recognition, in which the robot has to identify,

(38)

◦ Navigation Test, in which the robot must be able navigate through an apartment, avoiding or interacting obstacles along the way;

◦ Person Recognition Test, in which the robot has to identify a specic person within a crowd and report information about him/her and the crowd;

◦ RoboZoo, in which robots have to perform a show for the audience; ◦ Speech Recognition & Audio Detection Test, in which the robot must be

able to properly recognize and answer to a specic set of questions from a moving sound source without ask for conrmation;

Stage II:

◦ Open Challenge, in which teams can demonstrate recent research results and the best of the robots' abilities;

◦ Restaurant, in which robots have to act as waiters in a restaurant; ◦ Robo-Nurse, in which robots have to assist an elderly person;

◦ Wake me up Test, in which robots have to help their owners through their morning routine;

Final demonstration, which is a demonstration of the team's choice. The competition features the following awards:

Winner of the competition; Innovation award;

Winner of the RoboZoo; Best Test Score Certicate; Best looking robot;

(39)

3. Robot competitions: State of the art Figura 3.2: RoboCup@Home sample contendant performing a Wake me up test

3.2.2 DARPA Robotic Challenge

The DARPA Robotic Challenge is a competition whose aim is to develop ro-bot systems capable to assist humans in responding to natural and human-made disasters. It consists in a circuit of consecutive physical tasks, such as [17]:

Drive the vehicle; Egress from the vehicle;

Open door and travel through opening; Open valve;

Use a cutting tool to cut a hole in a wall; Surprise manipulation task;

Traverse rubble, i.e., either cross debris eld (by moving the debris or traversing it) or negotiate irregular terrain;

Climb stairs.

Robots are ranked according to their performance in task completion and task completion time.

(40)

Figura 3.3: DARPA Robotic Challenge sample contendant cutting a hole in a wall

3.2.3 RoCKIn Robot Challenge

The RoCKIn Robot Challenge [35,36] is a project including robot competitions, workshops and symposiums with the aim to act as a catalyst for smarter and more dependable robots. It develops two competitions, such as:

RoCKIn@Home, for domestic service robots; RoCKIn@Work, for industrial robots.

3.2.3.1 RoCKIn@Home

More in detail, the RoCKIn@Home consists of a series of tasks such as:

Catering for Granny Annie's Comfort, i.e., executing a list of natural language commands;

Welcoming visitors, i.e., handling a series of visitors according to their role, for example guiding them to a specic room;

Getting to know my home, i.e., mapping the position of the objects in the house.

In addiction, a series of functionality benchmarks is carried out in order to assess specic capabilities of the robots. Every task is performed multiple times, within a maximum execution time allowed, and the performance evaluation is based on the number and percentage of successful trials.

The selected capabilities to assess are: Object Perception, which includes:

(41)

3. Robot competitions: State of the art ◦ Object Detection;

◦ Object Recognition; ◦ Object Localization;

Object Manipulation, i.e., operating manual command (both digital and ana-log) on common domestic appliances;

Speech Understanding, i.e., translating natural language into a formal language command.

3.2.3.2 RoCKIn@Work

The RoCKIn@Work consists in a series of tasks such as:

Prepare assembly aid tray for force tting, i.e., collecting bearing boxes from shelves and inserting them into specialized aid trays, waiting for them to be force tted and performing a nal examination before delivering the nal prod-uct;

Plate drilling, i.e., handling incomplete or faulty parts from an external com-ponent supplier by drilling a missing cone sink on a cover plate or discarding faulty components;

Fill a box with parts for manual assembly, i.e., composing boxes with parts for the manual, nal assembly of a drive axle.

And the associated functionality benchmarks are: Object Perception, which includes:

◦ Object Detection; ◦ Object Recognition; ◦ Object Localization;

Manipulation, i.e., identifying, grasping, lifting and notifying object acquisi-tion;

Control, i.e., moving the manipulator in a precise and continuous manner, following a given path in Cartesian space.

(42)

3.2.4 ICRA Robot Challenges

The IEEE International Conference on Robotics and Automation in 2016 features the following robot competitions:

Airbus Shopoor Challenge [3]; Mobile Microrobotics Challenge [2];

Humanitarian Robotics and Automation Technology Challenge [1]; Formal Methods for Robotics (FMR) Challenge [22].

3.2.4.1 Airbus Shopoor Challenge

The Airbus Shopoor Challenge aims at nding new promising projects to de-velop lightweight robotic system able to perform accurate drilling compliant with aeronautic standards. In order to show their accuracy, robots are required to per-form several rounds of a simplied drilling task on an artefact representing part of the aircraft fuselage. Scoring is based upon the number of correctly drilled holes, considering their diameter and position.

3.2.4.2 Mobile Microrobotics Challenge

The Mobile Microrobotics Challenge features three events, such as:

Autonomous Mobility & Accuracy Challenge, in which microrobots must track predened micro-scale trajectories multiple times;

Microassembly Challenge, in which microrobots must assemble multiple mi-croscale components inside a narrow channel;

MMC Showcase & Poster Session, in which each team can show some capabil-ities of their choice.

3.2.4.3 Humanitarian Robotics and Automation Technology Challenge The Humanitarian Robotics and Automation Technology Challenge's mission is the application of robotics and automation technologies for promoting humanitarian causes. In particular, the 2016 edition focuses on promoting the development of new strategies for autonomous landmine detection. Robots will be ranked accord-ing to the number of correctly detected, undetected or exploded mines, incorrect classications, percentage of swept area and area coverage time.

(43)

3. Robot competitions: State of the art 3.2.4.4 Formal Methods for Robotics Challenge

The challenge aims at motivating advancement of the state of the art of formal methods (i.e., techniques for the verication and automatic synthesis of transition systems that satisfy desirable properties exactly or within some statistical tolerance) toward practical realization (e.g., motion planning in robotics). It is organized into three problem domains such as:

Arbitrary dimensional chains of integrators (i.e., controlling acceleration or a higher order derivative of a point-mass in high-dimensional spaces so as to visit goal regions and avoid obstacles);

Trac network with Dubins cars, i.e., computing the optimal path;

(44)

(45)

Chapter 4

Robot competitions: Analysis and

classication

In this chapter we identify the most characterizing aspects of robot competitions, namely the presence of robots performing some kinds of tasks in an arena according to a set of rules. We then introduce a taxonomy based on these aspects, formu-lated according to the existing literature and to direct observation of the considered competitions. Finally, we apply our taxonomy to the robot competitions previously selected in Chapter 3, with the aim of assembling an overall picture of the present situation enabling us to evaluate the assimilability of competitions to experiments.

4.1 Main identied aspects of a competition

Although the competitions may be highly variable in popularity, longevity, pur-poses and modalities, some common traits can be identied. We consider the most interesting ones to be mainly the presence of:

robots, which are a characterizing element and basically the main concern of the competition;

an arena, i.e., the bounded area in which robots are supposed to perform; rules, i.e., a set of rules determining the allowed interactions between robots,

robots and environment (time and space), and robots and humans;

a public, which is an inuential element, despite not being directly involved in the competition.

(46)

4. Robot competitions: Analysis and classication

4.1.1 Robots

A rst evident basic element is the presence of robots, since they are the core of the competition. Depending on the purpose of the competition, such as implement-ing some new software modules for a specic application or proposimplement-ing an innovative design for the physical implementation of the robots, the level of standardization of the contenders may vary dramatically (as shown in Figure 4.1 and Figure 4.2), since focusing on a specic functionality usually requires a standardized underlying hardware, while a redesign of the whole robot should not impose to many restrictions on the ingenuity of the participants. In general, we could adopt a broad denition of robot as an active, articial agent whose environment is the physical world [37]. Traditionally, robots are used for manufacturing and material handling (i.e., storage, transport, and delivery of material), patrolling, action in hazardous environments (e.g., cleanup, exploration, or mining), telepresence and virtual reality, and augmen-tation of human abilities (for instance prosthetic limbs). Consequently, most of the considered competitions address one of these elds of application.

From a physical point of view, we could assume that a robot has some sort of rigid body, with rigid links linked together by means of joints in order to allow motion. They are normally equipped with:

eectors, namely any device which is able to aect the environment as desired (e.g., grippers, screwdrivers, welding guns and so on), which mainly allow for locomotion of the robot and manipulation of the surrounding objects. In order to have an impact on the physical world, they must be equipped with actuators, such as typically electric motors or hydraulic or pneumatic cylinders, that convert software commands into physical motion;

sensors, namely anything that can change the computational state of the agent in response to a change in the state of the world (e.g., cameras, encoders, force sensors and so on), which mainly allow for various kinds of perception such as vision, proprioception, force sensing, and tactile sensing [37].

(47)

4. Robot competitions: Analysis and classication Figure 4.2: Sample RoboCup Soccer League contendant

4.1.2 Arena

A second basic element is the arena, i.e., the bounded area along with its content in which robots are supposed to perform. Despite the fact that its description in the rule books can be diversely detailed, some aspects that recur more often can be identied. Taking the sample competitions [13, 17, 22, 31, 3436] into consideration we observed that, in the most exhaustive cases, the aspects of the arena including objects and humans within it which are regulated are usually the following ones:

shape; size;

internal structure, i.e., the positioning of the content; material;

texture; colour; lighting;

telecommunication medium for robot-robot and human-robot interaction; weight.

Since accurate description of the environment conditions is a basic requirement to comply with core principles of experimentation such as comparison, repeatability, and reproducibility, we pay a special attention to the current practice of describing

(48)

competition arenas. Indeed, we deem a thorough documentation of the aforemen-tioned current practice potentially useful in order to estimate the distance between present robot competitions and experiments. Therefore, a detailed analysis of the present condition can be found in the following sections.

4.1.3 Rules

A third element is the set of rules governing the interactions between robots, robots and environment, and robots and humans, aiming basically at picking a win-ner. In fact, some common traits we could recognize were:

a goal the robots must achieve;

a scoring system to evaluate the degree of achievement of the goal; a human supervisor.

Being competitions focused on determining the most performing robot for solving a given task, it is not surprising that time is given particular attention in the evaluation process, being indeed considered a performance metric. We thus observe that time plays generally three roles within competitions, which are:

superior bound for performance duration; ranking criterion;

threshold for qualication.

In the sample competitions [13,17,22, 31,3436] only the rst two roles occur. A consideration similar to the one regarding arenas applies also for rules. In fact, sets of rules could be easily compared to experimental procedures, thus re-quiring themselves a through documentation when factoring into the calculations that competitions could be recasted as experiments, in order to comply with the aforementioned principles of comparison, repeatability, and reproducibility.

4.1.4 Public

Despite not being directly involved in the competition, the presence of the public should be taken into consideration because of the side eects it implies, since compe-titions often act as advertisement for the robotics eld targeting a general audience. A correct management of the public can actually aect fund raising, community building and education of the next generations.

Within the public, it is possible to distinguish some categories of spectators with specic features and dierent aims. A further analysis of the public conguration [13]

(49)

4. Robot competitions: Analysis and classication resulted in the classication depicted in Table 4.1.

Tabella 4.1: Public

Public

Audience Backers _Participants Attendees Press Organizations Sponsors Manufacturers

It is also possible to identify the specic aim of each distinguishable category as follows:

conference attendees expect information about the state of the art and the latest contributions to science and engineering, meanwhile being entertained to some extent;

press reporters are typically interested in technological glitz coupled with com-petitive drama, particularly regarding to relevant benecial eects on human life;

professional organizations mainly aim at promoting science, increasing atten-dance at the conferences, and improving public opinion;

sponsors, i.e., government agencies and industrial sponsors, are concerned about outcome of funded projects and seeking promising new projects;

equipment manufacturers are interested in good advertisement resulting in pur-chase increment.

4.2 Classication criteria

4.2.1 Reasons for selection

We decide to classify the sample competitions described in Chapter 3 according to the description of the arena and the set of rules. This classication is preparatory to the comparison of robot competitions to experiments with regard to the environ-ment description and the denition of the activities, respectively. Actually, a rigorous description of the subject of study is also expected in preparing an experiment. For this purpose, a classication based on the characteristics of robots could be expected as well. However, we deem this classication unfeasible due to the fact that the level of detail of the rule book descriptions varies from almost zero to a complete descrip-tion - like for instance those competidescrip-tions in which a standard hardware is available with a blurred range of intermediate solutions in between. This is primarily an eect

(50)

of the fact that on one hand, robot competitions usually allow for direct comparison of dierent approaches to solving a task, therefore avoiding placing restrictions on the ingenuity of the participants. On the other hand, when focusing only on the software side, the hardware side tends to be standardized in order to enable a fair comparison.

4.2.2 Criteria based on arena

4.2.2.1 Completeness

We observe that the descriptions of the arena usually do not encompass every as-pect. Thus, when considering the completeness of a selected aspect of the description we can dene the prescription as:

partial, if only some of the internal elements are taken into consideration; complete, if all given elements are taken into consideration.

For instance, in the RoboCup Soccer Humanoid League 2015 [34] specications are given with respect to the material of every component of the arena, whereas in the RoCKIn Robot Challenge@Home 2015 [35] the material of the furniture is often undened.

4.2.2.2 Quantiability

We observe that every aspect of the arena can be regulated by means of an informal description or by means of a quantiable unit of measure. Thus, we consider the prescription relative to an aspect to be:

qualitative, if it is associated with a subjective quality;

quantitative, if it is based on quantities obtained using a quantiable measure-ment process;

In the latter case, the prescription can be further classied into: range-based, if a range of options is available;

punctual, if a single value is given.

For instance, in the DARPA Robotics Challenge 2015 the internal structure of the arena is described by means of a quantitative, range-based prescription since for example the size of a valve is dened as a diameter between 4 inches (10 cm) and 16 inches (40 cm) [17], whereas in RoboCup Soccer Humanoid League 2015 [34] the prescription for the internal structure of the arena is punctual, as shown in Figure 4.3. In this very same competition we could also nd an example of a qualitative

(51)

4. Robot competitions: Analysis and classication description when taking the lighting into consideration, since it is described as The eld is illuminated presuming a sucient bright and constant lighting on the eld (i.e., no daylight).

Figure 4.3: RoboCup Soccer Humanoid League 2015 - Arena structure

4.2.3 Criteria based on rules

4.2.3.1 Human instructions

We could classify competitions according to the need of the robots to be in-structed by humans, distinguishing:

autonomous ones, if no direct human instruction is needed for the robot to take action:

human-driven ones, if human command is needed.

For instance, the RoboCup Soccer Humanoid League 2015 is an autonomous com-petition, being teleoperation, remote control, or remote brain of any kind not al-lowed [34], whereas DARPA Robotics Challenge 2015 [17] is a human-driven compe-tition, since its main purpose is to assess the capability of the robots to communicate with operators and to execute their commands.

4.2.3.2 Comparison mode

Depending on the possibility for the robots to interact with each other during the performance, possibly aecting its outcome, we deem [44] the comparison mode to be classied into:

head-to-head, if robots interact with each other; separate run, if every robot performs alone.

(52)

As an evident example of head-to-head competition we could mention the RoboCup Soccer Humanoid League 2015, since robots perform in opposing teams [34]. Instead, the RoboCup@Home 2015 [31] could be mentioned as an example of a competition with a separate run comparison mode, since every robot tries to perform a task multiple times alone.

4.2.3.3 Scoring system

We observe that the scoring system could be based on: automatic measurement;

human judgement, that can relate to:

◦ task assessment, if the achievement of a goal must be assessed; ◦ rating, if a personal opinion is required.

On the one hand, an example of automatic measurement can be found in the 2016 Humanitarian Robotics and Automation Technology Challenge [1], since the team scores are computed through the combination of the following measures:

Number of detected mines;

Number of undetected mines in the covered area, swept by the robot; Number of undetected mines in the uncovered area, not swept by the robot; Number of non-mines object classied as mines;

Number of exploded mines (known and unknown); Number of exploded wrongly classied mines; Number of obstacle collisions;

Percentage of swept area in relation to the total arena area; Arena coverage time.

On the other hand, an example of a scoring system based on human judgement could be found in RoboCup@Home 2015 [31], in which both task assessment (e.g., successfully navigating through the apartment) and rating (e.g., RoboZoo) could be found.

4.2.3.4 Ranking

If ranking is implied, then we can have:

no threshold, if the contendants are ranked only according to their score; a threshold, if a lower bound on performance is set.

A threshold was set, for example, for the RoboCup Robot Rescue Event in 2001 [44], when nding all the victims of an earthquake in a specic area is required. Due

(53)

4. Robot competitions: Analysis and classication to the fact that statistical analysis is a commonly adopted experimental strategy (see Section 5.4) we deem the prior exclusion of the least performing solutions to be potentially detrimental for such analysis. However, since none of the sample competitions denes a threshold for qualication, this aspect is not further considered in this study.

4.2.3.5 Scope

A distinction regarding the performance scope is also possible, since the subject of the test could be:

the whole robot;

a single module of the robot.

For instance, the RoCKIn Robot Challenge@Home 2015 [35] focuses both on the performance of the whole robot when evaluating complex tasks such as handling dierent kinds of visitors (e.g., Task Benchmarks) and on the single functionalities such as object perception or manipulation (e.g., Functionality Benchmarks).

4.3 Classication of the sample competitions

We proceed to classify the sample competitions according to the previously iden-tied criteria.

4.3.1 RoboCup

4.3.1.1 RoboCup Soccer Humanoid League

The RoboCup Soccer Humanoid League competition is classied with respect to the 2015 edition rule book [34]. The classications based on the description of the arena and of the set of rules are respectively depicted in Table 4.2 and Table 4.3.

Table 4.2: RoboCup Soccer Humanoid League - Arena classication

Aspect Criterion

Completeness Quantiability Shape Complete Quantiable/Punctual

Size Complete Quantiable/Punctual Internal structure Complete Quantiable/Punctual

Material Complete Qualitative Texture Partial Qualitative Colour Complete Qualitative Lighting Complete Qualitative Communication medium Complete Quantiable/Punctual