Copyright © 2015 McGraw-Hill Education (Italy), S.r.l. Via Ripamonti, 89 – 20141 Milano
Rights of translation, reproduction, electronic storage and total or partial adaptation by any mean whatsoever (including microfilms and Photostat copies) are not allowed.
Given the intrinsic features of the Internet, the Publisher is not responsible for any possible change in both the addresses and the contents of the mentioned Internet websites.
Names and brands mentioned in the text are generally registered by the respective producers.Copertina: Rebecca Squires
Programme and Portfolio Manager: Natalie Jacobs Programme Manager: Marta Colnago
Programme Manager: Daniele Bonanno Produzione: Donatella Giuliani
Prestampa e postproduzione digitale: digitaltypes.it
Prooceedings of the Cumulus Conference, Milano 2015
The
Vi
r
t
uous
Ci
r
cl
e
Des
i
gn
Cul
t
ur
e
and
Exper
i
ment
at
i
on
3-7 June 2015, Milano, Italy
Editors
Luisa Collina, Laura Galluzzo, Anna Meroni Publisher: McGraw-Hill Education Italy
Politecnico di Milano
Design Department School of Design Poli.Design
Fondazione Politecnico
For further information on Cumulus Association: Cumulus Secretariat
Aalto University
School of Arts, Design and Architecture PO Box 31000, FI-00076 Aalto
Finland
E: cumulus@taik.fi
W: http://www.cumulusassociation.org
- v
Conference Colophon
President of Cumulus International Association of Universities and Colleges of Art, Design and Media.
Conference Chair
Luisa Collina / Design Department, Politecnico di Milano.
Conference Manager
Laura Galluzzo / Design Department, Politecnico di Milano.
Scientific Committee Chairs
Ezio Manzini / DESIS Network
Anna Meroni / Design Department, Politecnico di Milano.
Tracks Chairs Nurturing
Eleonora Lupo / Design Department, Politecnico di Milano.
Sarah Teasley / Royal College of Art
Paolo Volonté / Design Department, Politecnico di Milano.
Envisioning
Giulio Ceppi / Design Department, Politecnico di Milano.
Stefano Marzano / THNK, School of Creative Leadership.
Francesco Zurlo / Design Department, Politecnico di Milano.
Experimenting/Prototyping
Banny Banerjee / Stanford University
Paola Bertola / Design Department, Politecnico di Milano.
Stefano Maffei / Design Department, Politecnico di Milano.
Incubating/Scaling
Anna Meroni / Design Department, Politecnico di Milano.
Cabirio Cautela / Design Department, Politecnico di Milano.
Gjoko Muratovski / Auckland University of Technology.
Assessing
Lia Krucken / Universidade do Estado de Mina Gerais.
Pier Paolo Peruccio / Politecnico di Torino.
Paolo Tamborrini / Politecnico di Torino.
Disseminating/Communicating
Elena Caratti / Design Department, Politecnico di Milano.
Paolo Ciuccarelli / Design Department, Politecnico di Milano.
Mark Roxburgh / University of Newcastle.
Training/Educating
Luca Guerrini / Design Department, Politecnico di Milano.
Pablo Jarauta / IED, Istituto Europeo di design.
Lucia Rampino / Design Department, Politecnico di Milano.
International Affairs
Anne Schoonbrodt / Design Department, Politecnico di Milano.
Visual Communication
Andrea Manciaracina, Umberto Tolino / Design Department, Politecnico di Milano.
Pictures
Massimo Ferrari
Translations and English Editing
Rachel Anne Coad
Graphic and Interior Design
International Review Board
The conference adopted double blind peer review.
Yoko Akama, Royal Melbourne Institute of Technology Jose Allard, Pontificia Universidad Catolica de Chile
Zoy Anastassakis, Universidade do Estado do Rio de Janeiro Nik Baerten, Pantopicon
Giovanni Baule, Politecnico di Milano Elisa Bertolotti, Politecnico di Milano Alessandro Biamonti, Politecnico di Milano Massimo Bianchini, Politecnico di Milano Luigi Bistagnino, Politecnico di Torino Sandy Black, University of the Arts London Spyros Bofylatos, University of the Aegean
Gustavo Borba, Universidade do Vale do Rio dos Sinos Brigitte Borja de Mozota, Paris College of Art Clare Brass, Royal College of Art
Caelli Brooker, University of Newcastle
Graeme Brooker, Middlesex University London Sam Bucolo, University of Technology Sydney Daniela Calabi, Politecnico di Milano
Barbara Camocini, Politecnico di Milano Angus Campbell, University of Johannesburg Daria Cantù, Politecnico di Milano
Michele Capuani, Politecnico di Milano Michelle Catanzaro, University of Newcastle Manuela Celi, Politecnico di Milano
Eunji Cho, Hunan University
Jaz Choi, Queensland University of Technology Matteo Ciastellardi, Politecnico di Milano
Carla Cipolla, Universidade Federal do Rio de Janeiro Luisa Collina, Politecnico di Milano
Chiara Colombi, Politecnico di Milano Sara Colombo, Politecnico di Milano Marta Corubolo, Politecnico di Milano
Vincenzo Cristallo, Sapienza University of Rome, Italy Robert Crocke, University of South Australia
Heather Daam, Institute without Boundaries Toronto Chiara Del Gaudio, Universidade do Vale do Rio dos Sinos Alessandro Deserti, Politecnico di Milano
Loredana Di Lucchio, Sapienza Università di Roma Jonathan Edelman, Stanford University
Davide Fassi, Politecnico di Milano David Fern, Middlesex University London Silvia Ferraris, Politecnico di Milano Venere Ferraro, Politecnico di Milano Alain Findeli, University of Nimes Elena Formia, Università di Bologna
Marcus Foth, Queensland University of Technology Silvia Franceschini, Politecnico di Milano
Carlo Franzato, Universidade do Vale do Rio dos Sinos Karine Freire, Universidade do Vale do Rio dos Sinos Marisa Galbiati, Politecnico di Milano
Laura Galluzzo, Politecnico di Milano Giulia Gerosa, Politecnico di Milano Miaosen Gong, Jiangnan University
Carma Gorman, University of Texas at Austin Francesco Guida, Politecnico di Milano Ashley Hall, Royal College of Art Michael Hann, University of Leeds
Denny Ho, The Hong Kong Polytechnic University Stefan Holmlid, Linkoping University
Lorenzo Imbesi, Sapienza Università di Roma Ayelet Karmon, Shenkar - Engineering. Design. Art Martin Kohler, HafenCity University Hamburg Cindy Kohtala, Aalto University
Ilpo Koskinen, The Hong Kong Polytechnic University Peter Kroes, TU Delft
Peter Gall Krogh, Aarhus University
Carla Langella, Seconda Università degli Studi di Napoli Yanki Lee, Hong Kong Design Institute
Elisa Lega, University of Brighton Wessie Ling, Northumbria University
Cyntia Malagutti, Centro Universitário Senac Naude Malan, University of Johannesburg Ilaria Mariani, Politecnico di Milano Tuuli Mattelmaki, Aalto University Alvise Mattozzi, Università di Bolzano Mike McAuley, University of Newcastle
Lisa McEwan, Auckland University of Technology
Stuart Medley, Edith Cowan University Western Australia Massimo Menichinelli, Openp2pdesign
Cynthia Mohr, University of North Texas Nicola Morelli, Aalborg University
Afonso Morone, Università degli Studi di Napoli Federico II Francesca Murialdo, Politecnico di Milano
Andreas Novy, WU Vienna University of Economics and Business Marina Parente, Politecnico di Milano
Raffaella Perrone, ELISAVA Escola Superior de Disseny Margherita Pillan, Politecnico di Milano
Francesca Piredda, Politecnico di Milano Marco Pironti, Università di Torino Paola Pisano, Università di Torino
Giovanni Profeta, Scuola Universitaria Professionale, Svizzera Italiana Agnese Rebaglio, Politecnico di Milano
Livia Rezende, Royal College of Art Dina Riccò, Politecnico di Milano Francesca Rizzo, Università di Bologna Rui Roda, University of Aveiro
Liat Rogel, Nuova Accademia di Belle Arti Valentina Rognoli, Politecnico di Milano
Dario Russo, Università di Palermo Fatina Saikaly, Cocreando
Giuseppe Salvia, Nottingham Univesity Daniela Sangiorgi, Lancaster University Daniela Selloni, Politecnico di Milano Anna Seravalli, Malmo University Giulia Simeone, Politecnico di Milano Michele Simoni, Università Parthenope
Eduardo Staszowski, Parsons The New School for Design Cristiano Storni, University of Limerick
Shehnaz Suterwalla, Royal College of Art
Kate Sweetapple, University of Technology Sydney Virginia Tassinari, MAD Faculty Genk
Carlos Teixeira, Parsons The New School for Design Adam Thorpe, Central Saint Martin
Paola Trapani, Unitec Institute of Technology Auckland Raffaella Trocchianesi, Politecnico di Milano
Federica Vacca, Politecnico di Milano Fabrizio Valpreda, Politecnico di Torino Francesca Valsecchi, Tongji University Beatrice Villari, Politecnico di Milano
Katarina Wetter Edman, University of Gothenburg Robert Young, Northumbria University
915
Assessing European Design Policy:
Towards an Evaluation Culture
Venanzio Arquilla, Associate Professor - venanzio.arquilla@polimi.it
Design Department
Politecnico di Milano, Italy
Stefano Maffei, Associate Professor - stefano.maffei@polimi.it
Design Department
Politecnico di Milano, Italy
Marzia Mortati, Research Fellow - marzia.mortati@polimi.it
Design Department
Politecnico di Milano, Italy
Beatrice Villari, Research Fellow - beatrice.villari@polimi.it
Design Department
Politecnico di Milano, Italy
The paper describes the results of a European co-funded project aimed at creating an eval-uation framework for design policy in Europe. In the first part the European Agenda for Design is described pinpointing the important steps that the EC followed to begin support-ing explicit inclusion of design in innovation policy. In this arena, evaluation emerges as an important issue that can support policy makers and beneficiaries to better use investments and resources, towards widening the understanding of design beyond R&D and marketing.
Reflecting on the above, the paper discusses the approach adopted and the main results developed within DeEP - Design in European Policy, one of the six projects co-funded within the EDII initiative. In particular, DeEP has produced a set of design indicators to evaluate the effectiveness of design policy on beneficiaries and national contexts. Recognizing the lack of evidences specific for design in all current innovation rankings, the evaluation approach proposed is based on the collection of new data using tailored design indicators that are integrated in a web-based tool to propose evaluation through info-graphics.
The paper concludes discussing the challenges and limitations of the results currently obtained and few relevant discussions for the wider support of design policy evaluation across Europe.
Keywords
916 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
THE EUROPEAN AGENDA FOR DESIGN
Over the last few years, design has become increasingly embedded in the debate on innovation at European level. This emphasis has derived from the process of advocacy promoted by the European Commission, aimed at highlighting the importance of design innovation and design approaches as levers for growth in different fields, from business to public sectors. The explicit commitment in Europe is to encourage the inclusion of design in innovation policies, and has begun in 2009 when the document “Design as a driver of user-centred innova-tion” was released (EU, 2009). Here, the issues addressed have derived from a direct consultation with the design community, and describe - through the col-lection and analysis of the relevant sources - the contribution of design in inno-vation and competitiveness as well as the possibilities for barriers’ removal to the better use of design in companies, policies, and the public sector. The hypothesis proposed considers design as “a driver and tool for user-centred and sustainable innovation and differentiation, complementary to technological R&D”, underlin-ing that “increased use of design could increase European competitiveness” (EU, 2009, p.7). This also refers to fostering non-technological innovation to promote the vision of “an innovation-friendly, modern Europe” (Barroso, 2006).
Following the guidance provided in these documents, European policy makers and intermediaries have begun putting particular emphasis on few issues: – Design has no commonly agreed definition and it is often associated with the
aesthetic aspect of objects;
– Design is partly included in the definition of R&D processes and often consid-ered as a part of marketing or other forms of innovation;
– European companies, and particularly SMEs, lack experience of design, and are often unsure of where to find design help;
– A different level of awareness of design is recognizable across Europe that dis-tinguishes ‘leaders’ and ‘followers’ to identify countries that consider design in their national strategies for innovation at different degrees.
To reinforce the promotion of design as lever for competitiveness, the European Design Leadership Board (EDLB) was set up in May 2011. This was composed by fifteen members representing design organizations, design promotion institutions, academics, design businesses, and designer practitioners, who were asked to pro-pose a “joint vision, priorities and actions to enable design to become an integral part of European innovation policy” (Tomson & Koskinen, 2012, p. 15). The main result of the Board’s activity was the report “Design for Growth and Prosperity” published in 2012 and containing a series of recommendations to support the development of design innovation capabilities in European education, innovation, research, public sector, and enterprise systems. This was based on the consider-ation that design is a process of “people-centred innovconsider-ation by which desirable and usable products and services are defined and delivered” (Tomson & Koskinen, 2012, p. 15), sustaining the centrality of people and considering design in the larger perspective of problem setting activity and innovation driver.
Assessing - 917
During the EDLB mandate, the European Commission put into practice the suggestions emerged by launching EDII (European Design Innovation Initiative) first call for proposals, in 2011. This has been the first European policy explicitly funding design innovation. As a result, six projects were funded involving 46 organisations from 19 countries, covering a wide array of design issues from the idea of strengthening the link between policy makers and the design community, the introduction of metrics for design in innovation scoreboards, the inclusion of design thinking and management in the public sector, and the diffusion of an evaluation culture for design policy.
To further these efforts, the most recent step has regarded the release of the Staff Working Document “Implementing an Action Plan for Design-Driven Innovation” aimed at promoting the understanding of design’s impact on innovation and at supporting design-driven innovation in companies and the public sector. The document describes the main strategic areas to address design as a key lever for European development, proposing a series of action lines emerged from the on-going results of the six EDII projects.
EVALUATION AS A KEY ISSUE IN THE EUROPEAN AGENDA
FOR DESIGN
Following on from the European debate described, there is no doubt that design and design policy have increasingly acquired recognition and relevance. The European debate begun in 2009 has de facto resulted in an increase in the number of discussions around design in many European arenas that has pushed policy makers in many regions and nations to declare their intention to foster innovation through design as a key priority in their industrial and innovation policies. A growing number of EU Member States have developed design policies recently, for example Estonia in 2012, Finland in 2013, and Latvia in 2014. The topic is emergent and fresh, both in the academic and in the policy debate where design is discussed as a means to address societal challenges and public sector innovation as well as to help SMEs build new capabilities.
However, the potential of design was recognized at political level only five years ago and is still looking for development in many areas. In particular, referring to the policies for innovation and the creative industries, the value of design still needs to be acknowledged and proved especially through measuring the impacts and effects produced by existing design policies. Evidences of the real benefits coming from including design in innovation are still unclear to many civil servants, public bodies, and policy beneficiaries. Examples provided are often pilot programmes with little evidence on real impact – economical or social – and in most cases slightly adapting the same type of policy. Research shows that the most relevant and recent design policy programmes are primarily based on coaching or mentoring and couple designers and businesses, thus lacking to experiment with different possibilities both in terms of approach and type of beneficiary (AA.VV, 2014). Proper evaluation of design policy could
918 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
be a way to gather relevant evidence, justify investments, and spot the most interesting areas for future policy, although this practice is seldom incorporated into programmes and initiatives due to high costs and uncertainties in the most appropriate metrics for design. Measuring design and design policy value is currently a major challenge, encompassing difficulties in defining which data is meaningful for design, which indicators can collect this data, and through which channels. In addition, data currently available is fragmented across Member States and measures for innovation performance are either not explicitly including design, or are intending it as a sub-set of R&D and/or marketing activities. For example, important statistics like the Community Innovation Survey include design only implicitly, and in only one question. The European Innovation Union Scoreboard 2014 (EU, 2013) includes design activities only implicitly, and only if the EU definition for design as driver for non-technological innovation is adopted. The measurement framework proposed distinguishes between 3 main types of indicators (enablers, firm activities, and outputs) and 8 innovation dimensions, capturing in total 25 different indicators. Within these, design could be interpreted in the third category – thus among the outputs forgetting its role as process for problem setting and solving.
A different approach is proposed by the Innovation Index launched by NESTA to prompt different paths to measure investments in innovation in the UK. A recent working paper linked to this (Goodridge, Haskel & Wallis, 2014) interestingly documents how investments in intangible assets and knowledge are contributing to national economic growth. This sheds light on the recent interests in the creative industries, including software, design, film/television, literary, music, etc. Among other findings, the paper arguments how the majority of studies proceed in the investigation by selecting a number of creative industries, and then documenting their employment or GVA from published sources. This understates the output of creative assets, since much intangible creation is not developed in the usual linear innovation funnel model, but in iterative and non-routine activities hardly measured only through the output indicators proposed in the previous scoreboard.
Further, other studies exist that try to measure specifically the economic contribution of design and design policy. Another example worth noting is the International Design Scoreboard (Moultrie & Livesey, 2009), a framework for ranking nations that describes a model of ‘national design system’, clustered into four categories: enabling conditions, inputs, outputs and outcomes. This offers 7 indicators that frame a picture of national design capabilities, assessing design’s value in 12 countries worldwide. However, it represents a very promising initial effort to measurement that has not found – until today – a second version or update.
The reports and surveys mentioned, whilst possessing varying degrees of coverage, granularity and relevance to design, provide a useful base for discussion to support the development of an informed debate around two crucial topics: the measurement of design value, and the evaluation of the impact and effectiveness of design policy support. These are both fundamental to help
Assessing - 919
policy makers understand the real potential of design, SMEs incorporate better design activities in their innovation process, and design professionals in all areas (research, promotion, development of products, services, systems) advocate with real evidence the relevance of their contribution.
The evaluation of the effectiveness of design policy is also the central topic developed by the EU co-funded research DeEP – Design in European Policy. This will be the main subject of the remainder of the paper, specifically describing the efforts that the research has directed at elaborating an evidence-based argument to prove design policy effectiveness and to developing an evaluation framework, and relevant guidelines for future design policy.
THE DEEP RESEARCH: ADVOCATING FOR EVALUATION
IN DESIGN POLICY
DeEP – Design in European Policy is an attempt to help policy makers and enter-prises develop a tool to analyse the effectiveness of design policies at micro level (the effect of policy initiatives on beneficiaries) and macro level (the overall effect on the national design eco-system). The final aim of the research has been to create an understanding of these actions by building an evaluation framework and a series of indicators to measure design policies’ effectiveness.
The key objectives for DeEP have been to:
– Identify and understand the presence of tacit and explicit design innovation policies in Europe;
– Develop design indicators to measure the effectiveness of design innovation policies both at a macro and micro level;
– Activate a process of mutual learning and transnational cooperation in the field of design policies, also through the creation of an open platform and knowledge repository as well as through contacting important stakeholders directly;
– Actively disseminate research results to the European Commission and to all interested stakeholders (researchers, managers, SMEs, policy makers) in as many European countries as possible.
In particular, the structure of the research could be summarized in a design pro-cess encompassing four phases:
i. Discover, a predominantly explorative phase aimed at: – defining design and describing its link with innovation
– describing design policies and their state of advancement in Europe, pro-ducing a list and taxonomy of design innovation policies.
ii. Define, expanding and applying previous knowledge to specific research topics, including:
– a description of the Design Policy Evaluation Cycle, connecting the policy making cycle with the policy evaluation cycle
920 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
– the definition of the DeEP Evaluation Principle, as the core of the research and engine for the development of evaluation tools.
iii. Develop, applying theoretical knowledge to: – defining and describing micro design indicators – defining and describing macro design indicators
– testing indicators directly with target users in all countries of the consortium iv. Deliver, condensing key concepts into the DeEP Evaluation Tool and design
policy evaluation strategies through:
– designing, testing, and implementing the tool and the evaluation engine – mapping design policies across Europe through the Design Policy
Landscape
– making recommendations and developing concepts for national bench-marks and firm outlines as tools for policy makers to develop better design policy in the future.
THE TOPIC OF POLICY EVALUATION WITHIN DEEP
The crucial topic addressed in the research has been policy evaluation, with spe-cial focus on design innovation policy and their outputs, outcomes, and impacts. A wide range of approaches, tools, data and indicators exist to facilitate policy evaluation in general, the aim being to evaluate the effectiveness of a given policy in line with the objectives defined as part of the political agenda. This type of evaluation could use three different approaches: Ex-Ante, Monitoring, and Ex-Post Evaluation. Usually, ex-ante precedes decision-making, and pre-assesses the effects and consequences of planned policies in order to ‘feed’ the information into the on-going decision-making process; monitoring identifies the (interim) effects and results of policies and measures implementation and realization while this is still under way; ex-Post assesses the impact of the policy intervention, and provides a feedback on the degree of accomplishment of the policy objectives.
The specific context for DeEP has referred to past experiences of policy initiatives with SMEs as main beneficiaries. The method for investigation adopted has collected first hand data on the initiative, looking at understanding the political objectives and the socio-economical context for the development of the policy, while interviewing policy makers and beneficiaries directly to recollect any evaluation conducted. The evidences emerged have highlighted the lack of previous evaluations and the high difficulties in quantifying the benefits obtained through the programme. On the contrary, the majority of interviewees underlined the need of quantitative data to justify investments and advocate for new and improved iterations of the programme, which often lead to the difficulties of moving beyond pilot initiatives. This is evident also when analyzing the landscape of design policy initiatives across Europe, where the most relevant actions do remain pilots.
Interpreting these evidences, DeEP has proposed a specific evaluation principle for design policy directed at SMEs. This states that the effectiveness
Assessing - 921
of a design policy can be measured through the positive change and/or transformation in the stock of design capabilities observed in design policy beneficiaries.
The focus on the transformation that happens on beneficiaries comes from the consideration that design is a set of capabilities that enable people-centred innovation. Therefore, design capabilities have been hypothesized as the central concept to measure the effects of design in innovation processes. Further, design is considered a pervasive human activity and the output of that activity that needs more than quantitative measures to be properly evaluated. These should incorporate both hard numbers and a qualitative evaluation based on narratives and profiles explaining how design can help create value in culture and strategic vision as well as on performance and return on investment. On these concerns, the research found that both approaches need much development, and the only evaluations conducted and available are strongly based on reports made by experts on the basis of qualitative interviews and participants’ feedback. Further, the few available experiences are quite fragmented and difficult to be compared due to the difference in approaches adopted, contexts represented, beneficiaries and type of intermediaries involved, and evaluators’ competences and knowledge. DeEP has tried to move beyond this fragmentation, surveying the existing examples (the two most quoted experiences include Designing Demand in the UK and Design 2005! in Sweden) and proposing an original approach that chooses to measure policies not only at the level of larger systems (nations/regions called in the research macro level), but including also the changes provoked in beneficiaries (SMEs called in the research micro level).
This has been proposed in terms of a capability approach to measure the transformation of the use/skills for design in companies. Design capabilities in SMEs have thus been investigated through analysing beneficiaries after their participation in the program. 16 companies have been interviewed considering 5 design policies in 4 European countries (Un Designer per le Imprese - A designer for enterprises, Italy – Lombardy Region; DEA | Design e Artigianato per il Trentino - Design for Craft, Italy – Trentino Alto Adige Region; Designing Demand, UK; Design som Utvecklingskraft - Design as a Development Force, Sweden; Design your Profit, Poland).
THE DESIGN INDICATORS: MACRO AND MICRO LEVELS
To apply the evaluation principle stated earlier, the research has progressed by defining a set of design indicators, divided in two groups of measures. Mac-ro indicators were identified fMac-rom a range of existing sources, (e.g. Innovation index, International Design Scoreboards, OECD reports, etc.) to describe the complexity of the macro socio-economical context where design policies and programmes are launched. Micro indicators were built considering the specific contribution and skills that design integrates in firms, in terms of technical, man-agerial, strategic/cultural capabilities.
922 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
– Design Investment, representing a governments’ investment in design in both financial (€) and policy terms;
– Design Supply, reflecting the education, training and supply of design practi-tioners – including wider education and training provision;
– Design Sector, relating to the national design industry as providers of design skills and expertise.
– Micro design indicators have considered three design capabilities:
– Design Leadership that relates to the presence of a multi-faceted understand-ing of design inside the organisation and a clear focus on the user to define the offer system as well as the production and distribution process;
– Design management that includes the idea of: resourcing design expertise and embedding them at all levels of the firm; managing the overall firm port-folio (products/services); accessing design collaborations (in terms of people, money, facilities) for the success of the firm;
– Design execution that describes the technical skills that design contributes to companies.
Finally, the evaluation principle and the design indicators have been translated into a practical solution to evaluate design policy initiatives. The DeEP Evalua-tion Tool envisages a twofold objective for design policy evaluaEvalua-tion in the near future: from the one side it is proposed for adoption to collect data and build the European design policy landscape, on the other it is aimed at supporting policy makers to gather data and justify investment for design while understanding through real proof the value of this activity. In the next paragraph, the tool is described in more details.
A WEB TOOL FOR DESIGN POLICY EVALUATION
The DeEP Evaluation Tool is a web-based instrument based on the collection of data related to macro and micro design indicators. It is principally based on monitoring policy beneficiaries within the context of individual programmes or initiatives, and with a link to existing national datasets.
The tool is structured to acquire data before and after the policy is delivered. In particular, it aims to support policy makers and firms to:
– Understand better the Design Policy Landscape in Europe,
– Monitor the Design Policy process and collect data on firms to justify design innovation policy investments,
– Evaluate and compare project outcomes and types of policy to understand which works best,
– Create and promote a European platform for the development of better design policy.
Assessing - 923
HOW THE TOOL WORKS
The DeEP Evaluation Tool is an online platform structured into four main areas: i. A description of the concepts driving the DeEP approach;
ii. Observations of the effects of design policies on firms (micro level);
iii. A description of the enabling conditions for design policies in Europe (macro level);
iv. Recommendations for future design policies. 1. A description of the concepts driving the DeEP approach
This section is divided in three main areas:
Understanding Design Policy - presents the definition of design adopted throughout the research (i.e. design as a set of capabilities that enable people-centred innovation) and the definition of design policy to describe the area of interest of the Tool;
Design Policy in Practice describes the Policy Cycle – the main phases for developing design policy are described in order to further emphasise the link with each evaluation step.
Design Policy Evaluation Process - explains the evaluation process described by DeEP to frame the function of theTool (ex-ante, monitoring, and ex-post stages described and connected with the use of micro and macro design indicators in the Tool).
2. Observations of the effects of design policies on firms (micro level)
On Implementing Design Policy Evaluation the DeEP Tool collects original data on the basis of design indicators clustered into three design capabilities (lead-ership, management and execution). The section is dedicated to managing and evaluating the effectiveness of design policies on beneficiaries, and has separate access for policy makers and firms. The former can register, monitor and evaluate policies; the latter can register and apply for policies; access their private profile and assess their progress through micro design indicators.
SMEs are evaluated for the transformation their design capabilities undergo as a result of the policy intervention. Their individual data is compared with a control sample (e.g. the other firms applying to the policy but not selected as beneficiaries), which allows policy makers to understand better the effectiveness of the policy directly considering contextual limitations, constraints and peculiarities.
Further, the Tool manages and visualises micro design indicators through info-graphics to facilitate understanding of evaluation results.
3. A description of the enabling conditions for design policies in Europe (macro level)
The Design Policy Landscape connects design policies to national ecosystems by visualising data in interactive maps on past and present design policies and initi-atives registered in the platform. This area also visualises data collected through macro design indicators (Design Investment, Design Supply, Design Sector) by
924 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
extracting design-relevant information from existing baselines. In particular, the Landscape provides two main outputs: Benchmarks and national scenarios com-paring national design performances across Europe and against EU benchmarks; Design Policy Map, a visual and interactive repository of data on EU design inno-vation policies, organisations and initiatives.
4. Recommendations for future design policies
The Developing Better Design Policy section proposes future perspectives on the connection between design policy evaluation and making. The objective is to promote an evaluation culture within organisations, including recommenda-tions for policy makers involved in promoting design innovation across Europe. CURRENT CHALLENGES AND LIMITATIONS OF THE TOOL
The DeEP EvaluationTool is part of an on-going process aimed at spreading an evaluation culture for policies across Europe. Currently, it represents the interim result of a much broader path and many challenges still exist to its full devel-opment and adoption, part of which are linked to politics and governmental decisions, while part are technical constraints requiring a top level decision in institutions to be fully implemented. In particular, the main challenges in the broader path of advocacy for design policy evaluation can be listed as follows: – Promotion and adoption of the Tool in all new design policies developed by
the EC;
– Adoption of the Tool by the widest pool of European countries/policy systems; – Lack of existing national data directly linked to design innovation;
– Need for further testing with policy makers coming from as many Europe-an policy systems as possible Europe-and considering different types of policy Europe-and beneficiaries;
– Need for expert qualitative interpretations of the data collected.
Further, the tool is not to be intended as a neutral delivery machine once the indi-cators are in place, but will require constant human understanding to translate statistical results into storytelling and business narratives. A close consideration of context, type of beneficiaries, and connected constraints is necessary to pro-duce evaluation models that go beyond the ladder type, and move into a more useful evidence for evaluating and proving the impact of design on innovation.
PROPOSALS FOR THE FUTURE OF DESIGN POLICY EVALUATION
DeEP can be considered a first step to promote design policy and policy eval-uation more concretely. The model and the tool proposed could foster debate and advancement in the design innovation community, as well as in the policy making one. This should begin considering the idea of evaluating design value in qualitative terms and through storytelling, but not always and not necessarily
Assessing - 925
using hierarchical models and rankings, which are the most popular today (the Design Ladder is one of the best known examples of this type of approach). One of the critical issues with existing ladders is that, often, beneficial inclu-sion of design is compared to having a strategic type of integration of design capabilities. This can be true for some, while for the majority the best way to incorporate it depends on the context, the market, the user, the organizational skills and mission, which concur to creating a much wider spectrum of profiles. Here also companies with technical design skills could be considered leaders in their own sector.
Elaborating further on the open issues emerged from the project, it is important to state that design innovation can not be measured only in relation with the production of goods (products and services) and the return on investment these produce. It also has strong social, environmental, territorial concerns. Understanding which are the most appropriate quantitative and qualitative metrics (indicators) to measure the outcomes of the contribution of design in innovation processes is therefore a delicate task that will keep raising many questions, for example:
– Is it possible to imagine qualitative indicators as well as quantitative ones? How can we collect data for these? Are they transferable between different contexts?
– No single ‘best’ evaluation method and technique exists. The variety of tools available is a signal of the diversity of effects that a policy intervention can achieve. Consequently, each method is fit to analyze one specific impact/ effect, and should be chosen for each policy case individually. How is it possi-ble to build the best evaluation approach considering that a mix of different evaluation methods is required?
REFERENCES
AA.VV. (2014). DeEP Design in European Policy. Lancaster, Great Britain: Lan-caster University press. Retrieved 17 November, 2014, from http://www.deep-initiative.eu/wp-content/uploads/2012/12/DEEP_FINAL-PUBLICATION.pdf European Commission (2013). Implementing an Action Plan for Design-Driv-en Innovation. Retrieved 17 November, 2014, from http://ec.europa.eu/ enterprise/policies/innovation/policy/design-creativity/index_en.htm
European Commission (2013) European Innovation Scoreboard 2013. Europe-an Commission. Brussels, Belgium. Retrieved 17 November, 2014, from http:// ec.europa.eu/enterprise/policies/innovation/files/ius-2013_en.pdf
European Commission (2014). Design for innovation. Retrieved 17 Novem-ber, 2014, form http://ec.europa.eu/enterprise/policies/innovation/policy/ design-creativity/index_en.htm
926 - The Virtuous Circle - Cumulus conference June 3-7, 2015, Milan
European Commission (2009). Design as a driver of user-centred innovation. Commission Staff Working Paper, Brussels: Commission of the European Com-munity. Retrieved 17 November, 2014, from http://ec.europa.eu/enterprise/ policies/innovation/policy/design-creativity/index_en.htm.
Barroso J. M. (2006) An Innovation-friendly, Modern Europe. Speech at the European Technology Platforms seminar, Retrieved 17 November, 2014, from http://europa.eu/rapid/press-release_SPEECH-06-784_en.htm?locale=en Goodridge P., Haskel J., Wallis G. (2014), UK Innovation Index 2014, Nesta Working Paper No.14/07, Retrieved 17 November, 2014, from http://www. nesta.org.uk/sites/default/files/1407_innovation_index_2014.pdf
Hobday, M., Boddington, A. Grantham, A. (2012). Policies for design and policies for innovation: Contrasting perspectives and remaining challenges, Technova-tion, 32(5), 272-281.
Moultrie J. Livesey F. (2009). International Design Scoreboard: Initial indi-cators of international design capabilities, Institute for Manufacturing, UK, Retrieved 17 November, 2014, from http://www.designcouncil.org.uk/sites/ default/files/asset/document/InternationalDesignScoreboard.pdf
OECD (2013) Innovation Union Scoreboard 2013. Organisation for Eco-nomic Co-operation and Development. Retrieved 17 November, 2014, from http://ec.europa.eu/enterprise/policies/innovation/facts-figures-analysis/ innovation-scoreboard/index_en.htm
Raulik-Murphy G., Cawood G., Lewis A. (2010). Design Policy: An Introduction to What Matters, Design Management Review, 21(4), 52-59.
Tomson, M., & Koskinen, T. (2012). Design for Growth and Prosperity, Retrieved 17 November, 2014, from http://ec.europa.eu/enterprise/policies/ innovation/files/design/design-for-growth-and-prosperity-report_en.pdf
