Sie sind auf Seite 1von 522

Research and Innovation Conference Proceedings 2018

ENoLL Office Pleinlaan 9 B-1050 Brussels Belgium T: +32 2 614 85 47 F: +32

ENoLL Office

Pleinlaan 9 B-1050 Brussels Belgium

T: +32 2 614 85 47 F: +32 2 629 17 00 E:


This report is a compilation of the papers presented between the 22 nd and 23 rd of August 2018, in Geneva, Switzerland, as part of the OpenLivingLab Days 2018 conference. The publications here contained are a result of the double-blind review and evaluation procedure launched on February of 2018 as part of the “Call for Papers” responding to the theme of the OpenLivingLab Days 2018 conference: “Living Labs and the Sustainable Development Goals: From Theories to Practice”

This year’s “Call for Papers” encourage contributions from three different paper categories to encourage a diverse participation of actors: Research Papers – scientific research; Innovation Papers – practitioner case studies; and Research in-progress – scientific research not yet completed.

ISBN (e-book): 9789082102789 © 2018 ENoLL - European Network of Living Labs All rights reserved

Review Panel


Anna Ståhlbröst – Botnia Living Lab

Steering Committee

Pieter Ballon – ENoLL Secretary

Evaluation Committee

Abdorasoul Habibipour – Botnia Living Lab Alen Keirnan – RMIT Ana Salas – Consorcio Fernando de los Rios Anna Ståhlbröst – Botnia Living Lab Annabel Georges - iMEC Anotonis Billis - Thessaloniki Active & Healthy Ageing Living Lab Carolyn Hassan – Knowle West Media Centre David Guimont – Living Lab en Innovation Ouverte Diana Chronéer - iMEC Dimitri Schuurman - iMEC Francesco Molinari – Lunigiana Amica Gareth Priday – Australian Network of Living Labs Gavin McCrory – Chalmer University of Technology Hanna-Greta Puurtinen – TAMK Living Lab Idoia Muñoz – GAIA & Bird Living Lab Johan Wenngren – Botnia Living Lab Lynn Coorevits - iMEC Mikko Julin – Laurea University of Applied Sciences Natasja Van Buggenhout - VUB Ömer Onur - Ba ş ak ş ehir Municipality / Ba ş ak ş ehir Living Lab Parvaneh Westerlund – Botnia Living Lab Shaofu Huang – University of Bristol Sonja Pedell – Future Self and Design Living Lab Tanguy Coenen – iMEC

ENoLL Office Contributors

Clara Mafé

Table of Contents

Top 7 Papers selected by the Evaluation Committee

Constraints on upscaling and social inclusion in smart city living lab experiments and ways to anticipate them: lessons from four “smarter” labs by

Francesca Cellina, Roberta Castri, Mario Diethart, Thomas Höflehner, Nicola Da

Schio, Marc Dijk

Innovation Management in Living Lab projects: the Innovatrix Framework by



International Innovation Sprint Bridging the Sustainability Gap between

Metropolitan Core and Peripheries by Tuija Hirvikoski, Kaisla Saastamoinen and Mikael Uitto

Learning with Community: Developing Citizen-Led Housing by Shaofu Huang,


Dimitri Schuurman, Aron-Levi Herregodts, Annabel Georges, Olivier Rits

Ruth Deakin Crick, Melissa Mean, Carolyn Hassan, Colin Taylor


Participatory software development via Living Labs: A case of Deajeon by Hee-Sook. Yoo, Su-jin. Jung, Seong-gyu. Park, Miran. Cho


Transnational piloting for smooth internationalization of health-tech start-ups

by Päivi Haho, Metropolia and Virpi Kaartti

User needs and expectations as a challenging factor for successful living lab research initiatives involving older adults: The DDRI Experience by Tiziana C. Callari, Louise Moody, Nikki Holliday, Ed Russell, Janet Saunders, Gill Ward, Julie




Actors motivations, needs and expectations

Living labs as instrument for societal change: the role of intermediary actors and activities by Jos van den Broek, Timo Maas, Isabelle van Elzakker, Jasper


Models for Living Lab’s sustainability. Evidences from Italy and the Netherlands


Edoardo Gualandi and Luca Leonardi

Motivational Goals for using Electronic Health Record Applications by Rachel

Burrows, Sonja Pedell, Leon Sterling, Tim Miller, Antonette Mendoza

Smart Campus: a route using 4G and 5G by Esmat Mirzamany and Joe Barrett





Spectrum Analysis Digital Arts-and-Culture Assets and the Future of Co Creating Decentralised Technologies by Olivier Zephir, Soenke Zehle, Olivier Buchheit


The Austrian Real-World Laboratories - How to manage multi-stakeholder

engagement in the wide field of mobility research by Lina Mosshammer, Doris


The Developing Process of the Digital Game Which Supports Well-being of


Small Children by Päivi Marjanen, Janne Pirttikoski, Vesa Valkonen and Viktoria Tiainen


Governance and process-related areas

Daegu Living Labs” as City Innovation Platform by Hee Dae Kim


Listening to locals? Question-oriented approach in Field Lab Amsterdam East

by Sandra Bos …………………………………………………………………………

Living Labs – an Approach for achieving Sustainable Change in City Logistics


Living-Lab-as-a-Service: Exploring the market and sustainability offers ofLiving


Schools4energy: a living laboratory for energy awareness in schools by Carmelina Cosmi, Filomena Pietrapertosa, Giuliano Sarricchio, Michele Giordano,

Monica Proto, Marco Tancredi, Monica Salvia

The Circle of Mediators: Towards a governance model for tackling sustainability challenges in a city by Anne Äyväri, Annukka Jyrämä, Tuija Hirvikoski



by Nina Nesterova, Hans Quak, Tariq van Rooijen


Labs in Germany by Justus von Geibler, Julius Piwowar, Annika Greven

The management of information and knowledge flows in Urban Living Labs: a

constructivist approach by Carolina Campalans


Theoretical and Methodological Challenges

Applying the Living Lab Approach for the Design of Public Spaces– A Living

Lab Case Study by Sonja Pedell, Gareth Priday, Alen Keirnan, Flavia Marcello, and

Andrew Murphy


Comparison of Health and Wellbeing Living Lab Business Models – Preliminary result based on Business Model Canvas Evaluation by Teemu Santonen, Mikko


Developing a mutualized R&D for network organizations based on Living Lab methods: the transformation of Sion military airport into a commercial airport by Emmanuel Fragnière, Benjamin Nanchen, Marine Héritier, Randolf Ramseyer,

Magali Dubosson and Joelle Mastelic



Evaluating the Performance of the Regional Living Lab Concept on Integrated

Sustainable Energy Planning by Giannouli Ioanna, Tourkolias Christos, Georgiou

Paraskevas, Cantero Celada Sergio, Fernández Maroto Miguel

Evolving Community Living Labs through Dual Safety Knowledge Circulation Systems to Prevent Injuries in Children by Mikiko Oono, Yoshifumi Nishida, Koji

Kitamura, Kimiko Deguchi

Initiating Participation: Methodological and Practical Challenges of Living Lab Projects for Early Stages of Research and Development by Andreas Bischof,

Albrecht Kurze, Sören Totzauer, Michael Storz, Kevin Lefeuvre, Arne Berger





Living Lab 65+ – Participatory testing of technical assistance systems in the

natural home environment of senior citizens by Misoch, S., Lehmann, S., Pauli, C.,

Hämmerle, V., Guggenbühl, U. & Konstantas, D

Living Lab Research: A State-of-the-Art Review and Steps towards a Research

Agenda by Abdolrasoul Habibipour

Making open innovation work for Sustainable Development Goals:

Sustainability-orientation and assessment based on the SDG-Check by Justus

von Geibler, Julius Piwowar, Annika Greven

Solar Living Lab: implementing sustainable experimentation process,

responsible innovation and community engagement in a solar energy research

facility by Silvana Di Bono, Filippo Paredes and Fabio Maria Montagnino

The benefits and challenges of co-creation with seniors: an interdisciplinary social innovation project designed to improve quality of life by D. Campisi, N.

Nyffeler, D. Roulet Schwab, V. Le Fort, L. Bergeron

The Island Approach (TIA): a suited projective technique for evaluating a bevy

of attributes by Natasja Van Buggenhout, Wendy Van den Broeck, Iris Jennes









This publication is a collaborative effort of several individuals representing the European Network of Living Labs and its network members.


Constraints on upscaling and social inclusion in smart city living lab experiments and ways to anticipate them: lessons from four “smarter” labs

Francesca Cellina 1 *, Roberta Castri 1 , Mario Diethart 2 , Thomas Höflehner 2 , Nicola Da Schio 3 , Marc Dijk 4

* Corresponding author 1 University of Applied Sciences and Arts od Southern Switzerland (SUPSI), ISAAC 2 University of Graz, Regional Centre of Expertise Graz-Styria 3 VUB, Cosmopolis, Department of Geography, Vrije Universiteit Brussel, 4 Maastricht University, International Centre for Integrated Assessment & Sustainable Development (ICIS)

Category: Research in-progress

Development (ICIS) Category: Research in-progress Abstract The ‘Smart City Living Lab’ is an emerging

Abstract The ‘Smart City Living Lab’ is an emerging approach in European cities, referred to projects devised to design, test and learn from innovative socio-technical practices (i.e. ‘new ways of doing something’) in real-time and in urban contexts, with a diversity of stakeholders. However, successful implementation of new practices in the reality of LLs does not guarantee the large-scale adoption required to reach their full effect in resource efficiency. Also, there is a risk of exclusion of social groups not matching the required ‘smart citizen’ profile. Acknowledging such drawbacks, we focused on how to foster upscaling and avoid social exclusion, developing a novel approach that anticipates such problems, and testing it through ‘smarter’ LL experiments addressing mobility-related topics in four European cities. In this paper we summarize the key constraints we have identified and comment on the strategies we implemented in our Living Lab activities in order to cope with them.

Keywords: living lab; inclusion; upscaling; smart city; mobility.


Introduction The ‘Smart City Living Lab’ is an emerging approach in European cities, referred to


The ‘Smart City Living Lab’ is an emerging approach in European cities, referred to projects devised to design, test and learn from innovative socio-technical practices (i.e. ‘new ways of doing something’) in urban contexts, with a diversity of stakeholders. A Living Lab (LL) was defined as an institutional environment for open innovation that supports experimentation with real users in real contexts (Folstad 2008; Hillgren 2013). It may be organized in a variety of ways (long-term or short- term, independent from or embedded in the municipal organization (Kemp and Scholl 2016), provider-driven or user-driven (Leminen 2013)), but commonly characterized by situated experimentation, diversity and participation, learning, and evaluation.

The interaction of the social and technical dimensions often makes urban infrastructure quite resistant to change and requires specific attention when new practices are to be introduced (Hommels 2005, 2010). The current approach to LLs focuses on small-scale performance tests and technology-user interactions, mostly neglecting the larger social-institutional context (Karvonen & van Heur, 2014; Karvonen et al., 2013). Therefore, successful implementation of new practices in the reality of LLs is not a warrant for the large-scale adoption required to reach their full effect in resource efficiency. Another limitation of the current LL approach to smart urban technologies is its focus on ‘smart citizens’ as users and partners, namely citizens with both the cognitive and material resources to consume and co-produce the smart services of the smart city. Citizens lacking these resources will normally not be included as users and co-creators in LLs, nor are they likely to be able to use the smart services once these are implemented on a larger-scale (Dutilleul et al., 2010). The consequence is not only limited adoption and use of these smart technologies but also social inequality and exclusion (Evans & Karvonen, 2014).

In the ‘SmarterLabs’ project we focused our attention on how to foster upscaling and avoid social exclusion, developed a novel approach that anticipates such problems and tested it through ‘smarter’ LL experiments addressing mobility-related topics in four cities: Bellinzona (CH), Brussels (BE), Graz (AT), and Maastricht (NL). Action research activity in such LLs is still going on and we are now performing final evaluations of effectiveness. However, the elements collected so far allow us to identify a set of recurrent constraints on inclusion and upscaling and to comment on the strategies we implemented in order to cope with them. After a brief introduction to our ‘smarter’ pilots (Section 2), in this paper we present the constraints we identified and the lessons we gathered (Sections 3 – 14), concluding with considerations on future work (Section 15).


Main constraints on social inclusion and upscaling In the framework of the SmarterLabs project we

Main constraints on social inclusion and upscaling

In the framework of the SmarterLabs project we performed a literature research, developed a retrospective analysis on mobility and land planning processes and directly engaged in action research activity by designing, managing and evaluating four ‘smarter’ LL processes. In Bellinzona citizens were involved in co-designing a smartphone app aimed at promoting individual behaviour change and rewarding those who reduce car use. In Brussels citizens were involved in participatory measurements of air quality, with the aims of increasing awareness on the impact of urban traffic flows on local air pollution and co-designing more sustainable mobility scenarios. In Graz citizens and local stakeholders were engaged in the ‘smart’ redesign of Griesplatz, a large square in the centre of the City, especially important as a traffic hub. Similarly, in Maastricht a series of focus group meetings exploiting a web-based design tool were held to engage stakeholders in co-designing the renovation of the central station area. Table 1 summarizes the constraints we identified in our LL pilots. We refer to upscaling as the emergence and expansion of an innovative practice (i.e. a new way of doing something) in the particular urban area and to social exclusion as a multidimensional, multi-layered and dynamic deprivation that people may suffer because of new urban practices. Note that we understand social exclusion as a key constraint affecting upscaling itself: addressing constraints on social inclusion is a pre-condition to effective upscaling.

Table 1 Constraints precluding social inclusion and upscaling.

Constraints on social inclusion

Constraints on upscaling

Exclusion from the LL

Exclusion in the

Related to LL

Related to




Citizen’s lack of financial, intellectual and human resources

Reproducing existing power structures inside the Lab

Limited learning

Low stakeholder receptiveness

Mismatching goals between the citizens and the Lab

Poor timing

Low institutional receptiveness


High institutional

Overlooking people outside lab context


fragmentation Sticky urban assemblage Neglecting effects outside project locality


Citizen’s lack of financial, intellectual and human resources Living Labs can be complex and long

Citizen’s lack of financial, intellectual and human resources

Living Labs can be complex and long lasting. To participate meaningfully, citizens need to have time and energy, a certain level of understanding of the discussion and sometimes also specific economic and intellectual resources. People with no, low or very discontinuous revenues might be excluded from the Lab, since earning their living can leave little space to other activities. Also, people with precarious employment or residential conditions might lack the possibility to plan for long term and therefore commit to participate in a Lab. People responsible for taking care of elderly or children, as well as people working during non-office shifts might lack the material time to join the Lab. Foreigners and new-comers can be excluded because of their limited proficiency in the language. In addition, people lacking a minimum understanding of the issue at stake or acquaintance with the technology used in the Lab (e.g. because of low education level or age) are also at risk of exclusion. Socially marginalized groups may tend not to participate in community initiatives due to a lack of self-determination, of financial or educational resources, or both.

While it is virtually impossible for LLS to be inclusive of all relevant groups, it is desirable to minimise exclusion. It is important to reflect on desired outcomes and apply stakeholder and requirement analysis tools to identify potential types of exclusion and adequate coping strategies. This exercise is essential in the design phase, though it requires ongoing reflections at different stages of the Lab. Also, involving all Lab participants (not only the ‘institutional’ initiators) in explicit reflections concerning causes and outcomes of exclusion and seeking for solutions would be beneficial.

For example, efforts to minimise exclusion were at the core of the Brussels Lab since the beginning. Different adjustments were also made in progress, to cope with unexpected circumstances. At very early stages, the organizers (one of the local universities, and a network of neighbourhood committees) identified potential barriers to inclusion and opted for establishing different sub-groups, precisely to include the broadest variety of population. Throughout the process, regular outreach efforts were made towards groups at potential risk of exclusion, also relying on a ‘focal person’ identified in each group. For instance, venues and schedules for each group were strategically selected: for EU officers, meetings were convened in EU premises at lunch time, for groups of parents and shopkeepers, small meetings were organised in the early morning (just after leaving the children in school/before opening the shop), for young professionals, meetings were organised at early evening in a central neighbourhood. Several smartphones were purchased to ensure everybody could still take part in the Lab, as well as tablets, used for demonstrative purposes. Less acquainted people with smart technologies where dedicated more training time. In


some cases, however, these efforts were not enough to bridge the gap, resulting in participants not using the technology.

Similarly, the City of Graz aimed to act in a district with challenging circumstances:

high proportion of migrants, various cultures and ethnics, education levels and incomes below average. The strategy to reach out to marginalized groups such as migrants, elderly people and children was to offer different formats of LL activities:

workshops, social safaris, online questionnaires, mental maps, etc. Lab organizers did not wait for people to show up, but actively approached them on the street, literally bringing the Lab to the people. By repeatedly offering possibilities for stakeholders to participate and actively approaching them, over a long period of time also marginalized groups were included.

long period of time also marginalized groups were included. Mismatching goals between citizens and the Lab

Mismatching goals between citizens and the Lab

Due to their experimental character, LLs often struggle to define concrete goals, especially in initial stages. Trial-and-error and learning-by-doing approaches can be a challenge when it comes to communication. On the one hand, it is important to be transparent and inform stakeholders and potential participants about LL objectives in advance. On the other hand, participatory processes are more beneficial when a common vision for the Lab is created together with participants. What happens in practice is that stakeholders often feel information is lacking (due to little or wrong communication) or – even worse – disagree with Lab objectives (due to not being sufficiently involved in their creation). The latter strongly depends on the facilitators. Aiming to involve a variety of people, special attention needs to be paid to their individual demands and desires. Communication, including awareness raising campaigns to inform people and establish a connection between them and the LL facilitators, is the key to success throughout a Lab’s lifetime. Through a constant exchange, expectations can be kept realistic and results are better accepted.

For example, the LL in Graz was initiated by the city government which aimed to improve the quality of life in the traffic-dominated area of Griesplatz. Although changes in traffic infrastructure were not supposed to be part of the participation process, the residents around Griesplatz attributed the LL ambitious desires in that aspect too. The divergent goals between citizens and the LL could be partly attributed to legal circumstances that prevented rapid changes in infrastructure (i.e. concession for bus operators). However, also communication turned out to be misleading, as external policy-makers interfered with the information provided by the facilitators of the LL. Whether they were justified or not, it was the task of the organizers to address high expectations among the residents and clarify certain issues: they clearly pointed out that the Lab itself was an approach to a solution of existing problems and not


another initiative to fight against – the Lab would facilitate dialogue between city decision makers and stakeholders. To this purpose, various communication channels were used, including newspapers, Facebook, public events and direct interaction with people via the Lab’s district office.

Overlooking people outside Lab contextinteraction with people via the Lab’s district office. Living Labs are experiments situated in a specific

Living Labs are experiments situated in a specific geographic context, ranging from a building block to a neighbourhood, a municipality or a whole urban area. While there is a certain flexibility in choosing the scale within which to operate, any choice implies the definition of boundaries excluding people living beyond them. Often, people living outside or faraway the project context self-exclude themselves, relinquishing to join the Lab either because of the effort to reach Lab meetings or because they do not feel immediately concerned - though they might be impacted by the project. When setting a LL, therefore, thorough reflections on the multiple scales relevant to the LL, and on the actors that at any scales might be included/excluded, are needed. Adequate logistic arrangements can help minimise exclusion. This includes communicating LL purposes, adapting them and adjusting the frame. Overall, constantly discussing with (potential) participants about the objectives and the frame of the Labs can be particularly helpful in defining a shared vision, thereby increasing motivations and obtaining a broader audience.

For example, in Brussels the place of residence was one of the most solid barriers to broad inclusion. As indicated above, to minimise exclusion based on participants’ residence, different LL ateliers were held in different locations, depending on the participants’ home and daily schedules. However, despite the outreaching efforts, the Lab was eventually not successful in including participants from all neighbourhoods, nor participants living outside the regional borders, many of whom are commuters towards Brussels – thus largely contributing to, and being impacted by, air pollution in the city. Main reasons for failing in engaging them were the lack of time and resources to identify suitable locations at the urban periphery and their relatively lower concern for the issue at stake (i.e. widespread perception that suburban living is less impacted by air pollution). To complement for this shortcoming, constant efforts of networking and coordination with other organisations were made, to share good practices and lessons from the Lab: by experience-sharing with organisations in nearby cities, the conditions were created for replication in other contexts.

Reproducing existing power structures inside the Labconditions were created for replication in other contexts. One fundamental aim of LLs is to establish

One fundamental aim of LLs is to establish a democratic structure that guarantees that every voice is heard and considered. However, in practice, instead of achieving


real participation, various circumstances can lead to mere reproductions of the power structures already existing in real life. This could be the result of deliberate management in the LL, if run as an alibi activity. Or, LL organizers might not be aware of the heterogeneity of stakeholders and precautions needed to provide any group with equal opportunities.

To avoid reproducing existing power structures, first these need to be assessed by carrying out a stakeholder analysis. Then LL organizers have to design a communication strategy to address all identified target groups, applying tailor-made methods for each of them. Flexibility in the use of methods is a key requirement (e.g. not only conversation or only ICT tools). Also inviting people at various levels and occasions and building trust and social cohesion plays an important role. Organizers should facilitate development of activities along different tracks and allow each group to adapt to their speed. Next to the methodology, also the locations should be neutral and unbiased, providing access to everybody. If that is not possible, meetings should change locations over time.

For example, the Lab in Graz involved residents, shop owners, bus operators, city entities and politicians. All of them filled out certain roles that contained different levels of power. Moreover, a couple of persons repeatedly ‘sabotaged’ events by excessively raising their voices and acting as opinion leaders. Lab organizers aimed to blur the borders between them, enabling each person to participate equally. This was achieved by offering different formats of LL activities (social safaris, questionnaires, mental maps, etc.) and carefully selecting locations: the city district office next to Griesplatz remained a neutral place for diverse activities throughout the whole project, complemented by outdoor activities in the district. These measures created awareness and social cohesion among the people involved.

awareness and social cohesion among the people involved. Limited learning Living Lab processes are frequently run

Limited learning

Living Lab processes are frequently run by actors already engaged with other compelling duties, being them civil servants, for city-owned LLs, or voluntary citizens, for civil society-owned LLs. Therefore, they often lack time or resources to perform explicit monitoring of the lessons learnt throughout the process. Even when single actors draw their assessments and conclusions, they often lack a comprehensive view of the process, and therefore a comprehensive knowledge. If no single actor has an overview of all options, mechanisms and impacts emerged during Lab activities, limited transfer of learning is possible to future users, precluding upscaling. Explicit learning strategies are needed, capable of capturing and monitoring knowledge creation and transferring it to all actors. To this purpose, first goals and ambitions of the actors need to be understood. Then, knowledge exchange can be


favoured through people-to-people real-life interactions (i.e. physical meetings), which make learning more rewarding and comprehensive to all and also ensure tacit knowledge to emerge.

For example, the Lab in Bellinzona was a pilot project, run on a voluntary, politically non-binding base. On one hand, this favoured acceptance of the LL approach by the City, but on the other hand it made also responsibilities and commitment by the City to contribute to the participatory knowledge-sharing process less pressing. This made the process of capitalizing on the ‘lessons-learnt’ from the Lab and integrating them into the City’s policies more difficult.

Therefore, a learning strategy was explicitly designed, with the aim of monitoring knowledge creation. The strategy included analysing the project’s impacts, assessing the level of engagement and satisfaction by Lab participants, and reporting and communication of results, both internally to all actors involved, as well as externally, through local media. In parallel, it was avoided external experts driving a one-way learning process, by defining ‘their problem’, providing ‘their knowledge and technology’, and preparing ‘their solutions’. This would have meant limiting lab participants as testers of the app functionalities, not assimilating the knowledge developed during the LL, nor transferring it to future users. To avoid this, a user-centered approach was adopted and the app was directly co-designed within the Lab meetings. This helped increasing intrinsic motivation, enduring participation and learning and knowledge-sharing between participants.

and learning and knowledge-sharing between participants. Wait-and-see attitude It is not infrequent that Living Labs

Wait-and-see attitude

It is not infrequent that Living Labs are managed as routine projects, with no special attention to diffusion of knowledge and learning, during and after the pilot: either upscaling effects are expected to occur by themselves or strategies are put into place after the pilot ended.

At the beginning of Lab activities, what can reasonably be up-scaled should be identified, and an upscaling strategy should be designed, together with the relevant communication and dissemination measures – keeping it flexible and open to the evolution of activities in the Lab, which, for their open and participatory nature, might impose adjustments respect to initial plans. Such a strategy might be developed with the actors engaged in the Lab, who might get engaged also for its practical implementation. Also, the identified dissemination and communication activities need to be tailored to the specific context where Lab results are up-scaled, by choosing the right channels, time and language.


The Lab in Bellinzona is emblematic: initially, the process was approached as a sequence of closed and separate steps: first, the app is developed, then a plan is made to promote it to the population, finally it is assessed whether additional citizens need to be engaged. Namely, no specific upscaling strategy would have been devised by the City. Promoting app use to the population was initially expected to follow a rather traditional communication plan: press conference and distribution of information leaflets. No particular efforts would have thereafter been planned to actively advertise the initiative and promote app use.

However, the very choice of engaging citizens in app co-design triggered their intrinsic motivation and commitment, thus innately generating communication and dissemination possibilities versus the outside. Therefore, the ‘multiplier effect’ triggered during LL experiences thanks to committed participants was explicitly exploited: Lab participants were actively engaged in promoting app use among their circle of family and friends. Also, specific functionalities were explicitly included in app-design (‘collective challenges’), with the aim of periodically launching them and continuously attracting new citizens to join app use. The product of the Lab itself (the app), was therefore endowed with an inbuilt mechanism to favour its diffusion and counteract the ‘wait-and-see’ dominant approach.

and counteract the ‘wait-and-see’ dominant approach. Poor timing ‘Poor timing’ in the implementation of the

Poor timing

‘Poor timing’ in the implementation of the LL precisely refers to a situation where broader dynamics, namely the particular social, economic, cultural and political conjuncture, are disregarded and the experiment is designed as if it takes place in a vacuum. No immediate replication of LL best practices is likely to be successful without adequate customisation and adaptation to local conjuncture. This includes accounting for broader socio-economic, cultural and political considerations, ensuring links with the existing public debate, with what communities consider their priorities and with what stakeholders consider feasible.

In practice, avoiding ‘poor timing’ involves maintaining flexibility throughout the Lab, to ensure that both its objectives and frame can be adjusted and continuously re- defined by all actors. An important precondition is to place citizens at the core of the process, as they are likely to have the most detailed understanding of the local context. Also, it also requires active coordination with other societal developments and initiatives regarding related contents. This can be done at different levels, from simple information sharing to building bridges and identifying possibilities of cooperation. Ensuring that Labs are well linked to the broader societal debate is also a way to ensure that participants feel recognised, thereby strengthening internal dynamics.


In Brussels, an effort was made to link the Living Lab with the broader public debate. To begin with, the original frame around 'smart mobility' was immediately adapted by LL initiators, so that air quality and people health were at the core. Soon after, the organizers also engaged in open dialogue with all stakeholders active on the topic, contributing to establishing a platform for discussion for all civic movements striving for better air and a network of researchers working on air quality and citizen science. Both efforts contributed to reaching a broad audience and ensuring the LL was part of a broader discussion.

Low stakeholder receptivenessand ensuring the LL was part of a broader discussion. Even though results produced within the

Even though results produced within the Lab are aligned with the original plans and expectations, the policy climate might no longer support adoption of the innovation pursued in the Lab. Alternatively, outcomes of the Lab might not find consensus beyond Lab participants. In both cases, Lab outcomes would lack support or agreement by the population, as well of the political majority needed to activate the envisioned upscaling measures.

To avoid this, Labs should open to participation as much and as early as possible, by activating participatory processes already from the development of visions, selection of methodologies and identification of actions to be performed. In doing so, ‘the common’ should always be emphasized and already existing networks and coalitions between groups of stakeholders should be exploited. Also, building relationships with successful initiatives already developed by other actors would be beneficial.

For example, when the City of Maastricht started re-designing the station area, a LL visioning workshop was organised around the vision for mobility in Maastricht in 2040. Around thirty people, stakeholders from different backgrounds and interests (residents, entrepreneurs, commuters, urban planners, mobility operators), came together to picture a vision of the transportation system in 2040, with attention for accessibility, affordability and quality of the living environment. Participants could experience their own design in a 3D visualization model (SketchUp), learn about the designs of other stakeholder groups, obtain feedback and then adjust their own vision. Involving actors since the visioning level favoured later consensus.

Low institutional receptivenessactors since the visioning level favoured later consensus. Sometimes barriers might be due to the lack

Sometimes barriers might be due to the lack of open-mindedness and receptiveness by institutions and policy-makers. Decision-makers might in fact be unfamiliar with,


or not open to co-design approaches, believing that interaction with stakeholders adds unneeded complexity to policy development. Favouring expert-driven ways of thinking and agreement with powerful lobbies, policy-makers might not show (or indeed not have) real commitment: in the best-case Lab outcomes would not get full support from City government; in the worst case, the Lab itself would be an alibi activity.

To cope with such constraints, early inclusion of policy-makers should be sought for. Provided that activities in the Lab are adequately designed, namely that Lab organizers show genuine commitment and give voice, role and responsibility to diverse groups of citizens, civil society organizations and experts, policy-makers and institutions might start appreciating the approach and its benefits. Then, it would be a matter of repetition: once multiple successful pilot processes are carried out, institutions and policy-makers will embrace approaches and processes, supporting their outcome.

For example, the City of Bellinzona was formally owning the LL process; however, they were unaware of the potential of participatory LL projects in supporting policy development. Therefore, they lacked leadership and predominantly relied on advice and superintendence by the local university. They mainly perceived the Lab as a technology innovation testing ground: a single, small-scale, closed and controlled process, aimed at developing and evaluating the mobile app prior to its roll-out at city- level. In particular, City decision-makers tended to cling to authoritative governance styles, rather than opening up to more consultative, cooperative or even facilitative approaches (Ker Rault, 2008), mainly due to the fear of losing formal power and responsibility on the decision. Their main concern was to avoid possible financial and personal drawbacks and, inadvertently or not, the tendency was to keep the living Lab in the policy periphery.

However, leadership can only be learnt through experience: providing first-hand opportunities of experiencing public participation processes is a first start. Thus, researchers involved in Lab organization tried to promote a new political culture by ensuring the presence and active participation of representatives of the Municipality (civil servants, politicians) in LL meetings. This helped getting local authorities and decision-makers gradually acquainted with the concept that LLs may represent a valuable learning-by-doing tool and a constructive and enriching means for reflection on practices or policy.

Also, to favour Lab acceptance by decision-makers, the strategy was to focus at first on an app development: practical and technologically oriented, this was perceived as a low-conflict topic and therefore easily supported. Later on, capitalizing on the actor- and context-dependent knowledge created while Lab participants were testing the


app and concretely experiencing new mobility behaviors, discussion in the Lab was upscaled to policy-related topics regarding future mobility scenarios (‘What would we need to make mobility more sustainable in Bellinzona?’). This way, also potentially scaring, far-reaching and conflicting discussions were spontaneously introduced in the Lab with the support of the institutions.

High institutional fragmentationintroduced in the Lab with the support of the institutions. Even when policy-makers embrace the LL

Even when policy-makers embrace the LL participatory approach, its outcomes might suffer from limited diffusion due to fragmented institutional arrangements, which hinder clear distribution of responsibility and effective cooperation between involved city departments. Fragmentation into units and departments (‘silo compartments’) within and between public administration institutions, in fact, makes both horizontal and vertical dissemination of results rather difficult.

Transparency and collaboration between administrative units and organizations must therefore be actively fostered. In Bellinzona, for example, administrative organization at the City level was the main obstacle preventing diffusion of the LL approach to other fields than mobility and institutionalization of new governance practices. The strategy to overcome ‘silo compartments’ barrier was to actively engage councillors and civil servants, instead of waiting for them to spontaneously express interest in process or results. Thus, it was planned to invite them to attend LL meetings, in order to personally experience how they work and the effort needed, and guess their potential in addressing complex or conflictual topics. In the end, the envisioned strategy was not put into practice, mainly due to ‘low institutional receptiveness’ mentioned above. However, this gap will at least partially be closed, by inviting councillors and civil servants to a workshop aimed at presenting the approach and discussing its opportunities and limitations, as emerged from final assessment of the whole LL process.

Sticky urban assemblageas emerged from final assessment of the whole LL process. Changes in urban contexts are sometimes

Changes in urban contexts are sometimes tricky to achieve, due to technical, infrastructural, legal or financial aspects. There is an obduracy to urban assemblages that can result from persisting infrastructure, long-term contracts or legal ‘lock-ins’. In such cases decisions need to be taken by multiple stakeholders or entities on a political level and cannot be attached to the outcome of a participatory process only. For example, the Living Lab in Graz aimed to improve the quality of life in the traffic- dominated area of Griesplatz. However, due to its purpose as traffic hub, not all


infrastructural elements could be replaced according to citizens’ desires: long-term contracts with bus operators forced to wait or find alternative locations for bus stops which occupy a big part of the square.

If circumstances don’t allow big changes, a LL should focus on what is actually possible. Communication strategy and methodology have to be designed accordingly, in order to avoid wrong expectations among LL participants. A focus on behavioural measures can be fruitful in order to trigger structural change over time. Also collecting ideas and concepts to apply in future, under different circumstances, can be a strategy.

In the case of Griesplatz, people started to complain that elaborated discussions ended up in little outcome. Thus, the organizers remained flexible and changed their strategy by focusing on small steps: in order to deliver visible outcomes of the participatory process, they provided small and quick improvements for the Griesplatz area, such as a bike lane, enlargement of a green area, benches, or painting temporary zebra crossings as awareness-raising measures. They released press articles ensuring that ‘no idea was lost’ and would be put into place at a later stage within a public architectural competition.

at a later stage within a public architectural competition. Neglecting effects outside project locality Replicating

Neglecting effects outside project locality

Replicating pilot projects in the broader urban area can be prevented either because generated knowledge is very much related to the specific context of the Living Lab or because the whole Lab process only focused on the pilot project, neglecting or forgetting the effects on its boundaries. To avoid this, it is important to always consider projects’ indirect and cross-scale effects, also outside the boundary of analysis, by actively engaging stakeholders of the broader urban context.

This aspect was not considered in Maastricht, when in 1971 the first parking garage was situated below the main square that until then had served as an open-air public parking. The preparation plan for the garage was developed in consultation with business representatives from the city centre, and intended to secure the economic attractiveness of the city as well as the spatial attractiveness of an open square. At first, parties involved were pleased with the results: parking capacity had increased and the square was cleared of cars. Along time, however, when parking tariffs strongly increased, urging people to park shortly to allow high circulation, negative effects started to become manifest. Not at the square, this remained clean, but the growing flows of in- and outbound traffic increased congestion, noise and air pollution at the inner-ring with its neighbouring residents.


To avoid replication of similar errors, when the City activated the redevelopment process for the station area and the related Living Lab visioning workshop, the project of the station area was explicitly put in the broader urban context of Maastricht, thus preventing that an improved train station area would go at the expense of other urban areas.

Conclusionsstation area would go at the expense of other urban areas. In this paper we have

In this paper we have presented lessons learnt from action research in four ‘Smart City Living Labs’ in the field of mobility, held in Bellinzona (CH), Brussels (BE), Graz (AT), and Maastricht (NL). We were in particular interested in understanding which factors typically hinder effective social inclusion and upscaling possibilities, thus conditioning LL’s overall impacts. Once we identified such ‘constraints’, we developed ‘smarter’ methodologies aimed at anticipating them, in both design and management of Living Lab activities, and practically tested them. At the time of writing, final assessments are still being performed. However, insights obtained so far already allow us to identify simple coping strategies that, if implemented, could produce tangible improvements in the effort towards wider social inclusion and upscaling.

Acknowledgmentsin the effort towards wider social inclusion and upscaling. Part of this research project is financially

Part of this research project is financially supported by the Swiss Innovation Agency Innosuisse, within the Swiss Competence Center for Energy Research SCCER Mobility.



Dutilleul, B., F. A. J. Birrer, W. Mensink (2010). Unpacking European Living Labs:

Analysing Innovation's Social Dimensions. Central European Journal of Public Policy, 4(1): 60-85. Evans, J., & Karvonen, A. (2014). ‘Give Me a Laboratory and I Will Lower Your Carbon Footprint!’—Urban Laboratories and the Governance of LowCarbon Futures. International Journal of Urban and Regional Research, 38(2), 413-


Følstad, A. (2008). Living labs for innovation and development of information and communication technology: a literature review. eJOV: The Electronic Journal for Virtual Organizations and Networks. Volume 10, “Special Issue on Living Labs”, 99-131. Hillgren, P. A. (2013). Participatory design for social and public innovation: Living Labs as spaces for agonistic experiments and friendly hacking. Public and collaborative: Exploring the intersection of design, social innovation and public policy, 75-88. Hommels, A. (2005). Studying obduracy in the city: toward a productive fusion between technology studies and urban Studies. Science, Technology and Human Values, 30, 323-351. Hommels, A. (2010). Changing obdurate urban objects: the attempts to reconstruct the highway through Maastricht. In I. Farias & T. Bender (Eds.), Urban Assemblages: How Actor-Network Theory Changes Urban Studies. Routledge. Karvonen A, Evans J & Van Heur B (2013). The Politics of Urban Experiments. In:

Hodson M, Marvin S (eds) After Sustainable cities? Pp 104-115. Routledge, London Karvonen, A., & Heur, B. (2014). Urban laboratories: Experiments in reworking cities. International Journal of Urban and Regional Research, 38(2), 379-392. Kemp, R., & Scholl, C. (2016). City labs as vehicles for innovation in urban planning processes. Urban Planning, 1(4), 89-102. Ker Rault, P.A. (2008). Public Participation in Integrated Water Management – a Wicked Concept for a Complex Societal Problem. Dissertation, Cranfield University, UK. Leminen, S. (2013). Coordination and participation in living lab networks. Technology Innovation Management Review, 3(11).


Innovation Management in Living Lab projects: the Innovatrix Framework

Dimitri Schuurman 1 , Aron-Levi Herregodts* 1 , Annabel Georges 1 , Olivier Rits 1

*Corresponding author 1 imec.livinglabs

Category: Research Paper

author 1 imec.livinglabs Category: Research Paper Abstract Despite being described as ‘orchestrators’ and

Abstract Despite being described as ‘orchestrators’ and innovation intermediaries, the Living Labs literature on concrete guidelines and tools for innovation project-related innovation management is scant. Within this paper, we propose the Innovatrix, an innovation management framework built upon existing business model and innovation management tools and frameworks and iterated based on practical experience in Living Lab projects. We illustrate its added value within three practical case studies that lead to three propositions regarding innovation management in Living Lab projects. First, Innovatrix helps to scope the user involvement activities, which leads to a more efficient use of resources. Second, Innovatrix forces the project owner to focus on a limited number of customer segments, which increases the efficient spending of the scarce entrepreneurial resources. Third, Innovatrix allows to capture the iterations and pivots that were made during an innovation project, which helps to link outcomes with certain Living Lab activities.

Keywords: Living Labs; Innovation Management; Business Modeling; User research; Assumption; Validation; Testing.


Introduction Although we seem to be living in an era where founding a start-up company


Although we seem to be living in an era where founding a start-up company has never been easier, studies point out to high mortality rates of these organizations. A Harvard Business School study put forwards a failure rate of 75% (Gage, 2012), whereas a CB Insight study even proposes 90% for digital innovations (CB Insights, 2018). This pleads for more research and insights into innovation management and innovation intermediaries that can help these failure rates go down. One specific type of innovation intermediary are the so-called Living Labs.

Living labs are complex partnerships, as they facilitate university-industry relationships but also relationships between large companies, SMEs and startups. Living Labs are often referred to as public- private-people partnerships (4P’s) (Westerlund and Leminen, 2011). Based on a meta-review of the Living Labs literature, Schuurman (2015) defines Living Labs as an organized approach (as opposed to an ad hoc approach) to innovation consisting of real-life experimentation and active user involvement by means of different methods involving multiple stakeholders, as is implied in the Public-Private-People character of Living Labs. Moreover, he also concludes that Living Labs are emanations of both Open Innovation and User Innovation practices, as external inputs, including end-user contributions, are used to iteratively design and cocreate the innovation in development. This opening of the innovation process and the involvement of external actors in a structural process have the potential to increase the value and sustainability of the business model of the innovation (Baccarne et al., 2013). However, there is only a limited amount of literature available that combine Living Labs with business models. Rits et al. (2015) note that the majority of the papers in this field deal with the business model of Living Labs themselves, but that explicit integration of business model research in Living Labs is very rare. Moreover, in his literature review of the most influential Living Lab papers, Schuurman (2015) discovered that the majority dealt with the Living Lab organizations. This feels contra- intuitive as Living Labs are regarded as innovation instruments and innovation intermediaries that are capable of closing the gap between research and market introduction (Almirall & Wareham, 2011). Therefore, we would expect much more attention for the Living Lab project activities, and practical guidelines how to approach innovation projects in a Living Lab setting. In this paper, we focus on this project level and look for innovation management guidelines in Living Labs. After a review of Living Labs literature, we propose the Innovatrix framework, based on existing innovation management and business model tools and frameworks. We then investigate the practical implementation of Innovatrix by means of four case studies, selected from a sample of 40 Living Lab projects that used the Innovatrix framework.


Innovation management in Living Labs

Living Labs are regarded as complex phenomena where three analytical levels can be distinguished: the organizational level, the project level and the individual user interactions level (Schuurman, 2015). The defining elements of Living Labs, real-life context, multi-stakeholder, multi-method, active user co- creation and medium- to long term (Schuurman et al., 2013), are situated among these three separate, but interlinked layers. The multi-stakeholder characteristic especially applies to the organizational level. In this domain, Leminen (2015) provides a very diverse overview of actor roles and management implications for Living Lab networks. Managing value capture and value creation processes within Living Lab organizations is crucial for their sustainability, but also cumbersome (Schaffers et al., 2007). This also resonates with the medium- to long-term element. On the user interactions level, end- user co- creation is regarded as the way to involve users. The literature describes various ways and strategies to facilitate the process of co-creation (e.g. Kristensson et al., 2008) and provides an overview of different user characteristics and user roles (Leminen et al., 2014; Schuurman & De Marez, 2012) of Living Lab participants.

In terms of the project level, the real-life aspect and the multi-method approach are characteristics. There is some literature on the real-life aspect and on context (e.g. Bergvall-Kåreborn et al., 2009). The other element is the multi-method nature of Living Lab projects. These Living Lab projects are described as a structured approach to open and user innovation (Almirall & Wareham, 2008; Leminen et al., 2012; Schuurman et al., 2016). This way, we look at Living Lab projects from an innovation management perspective. However, Living Lab papers on methodology tend to describe a very specific methodology, which is specific for a certain Living Lab, or an innovation process with rather fixed elements and building blocks (e.g. Bergvall- Kåreborn et al. , 2010). The most concrete are the works of Pierson and Lievens (2005) and Schuurman et al. (2016) who put forward a quasi- experimental design, with a pre- test, an intervention and a post-test. Next to this, there is little to no literature that looks at innovation management in Living Lab projects, with the exception of some studies on ‘Living-Labs-as-a-service’ (see the next section). This is surprising, as already in 2006, at the start of the Living Labs movement, Niitamo et al. state that “[i]n Living Labs there is a need to combine highly self-organized and self-managed processes with multi-disciplinary R&D and innovation management processes”. Ståhlbröst (2013) also defines a Living Lab as “an orchestrator of open innovation processes focusing on co- creation of innovations in real-world contexts by involving multiple stakeholders with the objective to generate sustainable value for all stakeholders focusing in particular on the end users”. Nonetheless, this has not lead to an abundance of papers and studies that unravel or describe this process of orchestration. Coupled to this lack of innovation management guidelines for Living Lab projects, there are only few studies


that present concrete results of the outcomes of Living Lab projects, and even fewer that compare Living Lab projects with other innovation projects. Ballon et al. (2018) relate this to the complexity of the Living Labs phenomenon, but also urge for more studies in this domain. Therefore, within this paper, we will propose a tool that can be used as innovation management approach and that also enables to elicitate impact and outcomes of Living Lab interventions.

elicitate impact and outcomes of Living Lab interventions. Business model components as innovation management elements

Business model components as innovation management elements

A notable exception of the search for innovation management anchor points for Living Labs can be found in the scant literature on Living-Labs-As-A-Service. These Living Labs, focused on delivering specific services to external customers, play the role of innovation intermediary between entrepreneurs and end-users (Ståhlbröst, 2013). Coorevits and Schuurman (2014) argue that the Validation board (, from the Lean Start-up methodology, can be used as a tool to structure Living Lab projects as it is focused on planning and executing user research. Rits et al. (2015) argue for the integration of business model research with user research in Living Labs. In this context, they refer to established tools linked to business modeling and technology entrepreneurship, such as the Business Model Canvas (Osterwalder & Pigneur, 2013), the Lean Canvas (Maurya, 2012), and the Value Proposition Canvas (Osterwalder, Pigneur, Bernarda & Smith, 2015). D’Hauwers et al. (2015) proposed the iLLAB, a hypothesis driven Living Lab framework incorporating both user and business model learning, based on elements from the above business model tools. They see the iLLAB tool as an aggregation of principles from Ries (2011), the Osterwalder Value Proposition Design (2015), the business model matrix of Ballon (2007), the business model canvas of Osterwalder (2010) and Porter’s five forces model (1985) and translated into a set of strategic components. They developed their own framework to gather assumptions for user research, as the input from the other frameworks remained too high-level to define and execute user research. The validation board (Ries, 2011) functioned as main framework as it puts the customers at the core and focuses on customer hypothesis, problem hypothesis and solution hypothesis. This is also in line with the work of Wildevuur et al. (2013) who designed the People Value Canvas (PVC) tool to help build value propositions during user-centric service development processes. The PVC consists of nine building blocks, describing the input that has to be provided to establish the value proposition. The PVC is an iteration on the business model canvas. This facilitates a process-oriented approach, more specifically for highly iterative (and lean) innovation processes allowing for structured learning and pivoting.


However, based on practical experience with the validation board and the iLLAB- tool, we felt that more structure could be added to the existing elements. This is acknowledged by Herregodts et al. (2017) who develop a framework on knowledge uncertainties. Within these uncertainties, a major distinction can be made between knowledge related to the current environment versus knowledge related to the innovation under development. While the first is closely related to problem and opportunity identification, the second is related to the formulation and evaluation of solutions. This framework is based on the metaphoric use of ‘states’. States relate to reference points, either from the perspective of the organization or the individual (Gourville, 2005). Where the existing, ‘current state of being’, the ‘as-is’ or ‘status quo’ is opposing ‘possible future states’ (Alasoini, 2011). These insights led to the Innovatrix- framework (figure 1), which used elements from the previously discussed frameworks and tools, but that also considers this dichotomy between current and future state-knowledge. This resonates with a more process-oriented approach and should facilitate innovation management in an optimal way. We now briefly introduce and discuss the elements that compose the Innovatrix: customer segment, current practices, needs, value proposition, solution, barriers, value capture & key partners. The eight components will be discussed in more detail.

The eight components will be discussed in more detail. Figure 1. Innovatrix assumption & validation matrix

Figure 1. Innovatrix assumption & validation matrix


Customer segment – Current state

As used in the Validation Board (Ries, 2011) and the Business Model Canvas, Innovatrix starts from customer segments. However, there is room for multiple customer segments and the other elements are all linked to customer segments, and do not necessarily apply for all segments. This enables more fine-grained assumption development. In the Innovatrix framework there’s room for three customer segments (the grey areas in the framework) to cater to the need of clear focus through limited scope. The first column is used as an overarching column to map similarities between defined segments. Following Innovatrix checks can be used to gauge for relevant input to the Customer Segment criteria: What customer segments to focus on? What are key characteristics? What is the use-context?

Needs – Current state

Osterwalder (2015) includes customer jobs, pains, and gains in the Value Proposition Design canvas, which is the basis for the needs identification in the Innovatrix framework. Furthermore, Ries (2011) links customer segments - customer problems and the fit with the potential solution and/or value proposition. Following Innovatrix checks can be used to gauge for relevant input to the Needs criteria: What are the needs of the customer segment? How do we prioritie these needs?

Current practices – Current state

One missing pillar in Ries (2011), Osterwalder (2010) and in Ballon (2007), is the competition and the differentiation of an SME/start-up/innovator. Competition refers to the Five Market Forces of Porter (1985), which draws from five forces model. The five forces make up the attractiveness of a market. The five forces can be defined as (1) rivalry within the industry, the (2) threat of new entrants, (3) the threat of substitutes, (4) bargaining power of suppliers and, (5) bargaining power of buyers. Assessing the rivalry within the industry can help identify the difficulties of entering the market. If for example, the market consists of multiple strong players (i.e., Oligopoly market), the need to diversify can lead to high barriers to entry. On the other hand, if several new entrants enter the market (Monopolistic competition), it could indicate that it’s an attractive market with lower barriers of entry. For some products and/or services one can find possible substitutes, which can serve as an alternative to the specific service and/or product. Following Innovatrix checks can be used to gauge for relevant input to the Current Practices criteria: Who or what are competitors, alternatives, customer behaviour? What are the pains and gains of these current practices?


Value proposition – Current & future state

The value proposition is covered by the Lean Matrix of Ries (2011), the Value Proposition and the Business Model Canvas of Osterwalder (2010,2015) and, by the Business Model Matrix of Ballon (2007). The value proposition is the match between the needs of customer segments, and how this can be solved with the solution provided by the innovator. Following Innovatrix check can be used to gauge for relevant input to the Value Proposition criteria: What (measurable) impact will you create for this customer segment?

Solution – Future state

The solution refers to ‘the functional architecture’ of Ballon in the Business Model Matrix (2007). The functional architecture comprises of the technical systems composed of at least one building block (or module), governed by specific rules (or intelligence) that interwork (or not) with other technical systems through predetermined interfaces. The composition of the solution in the key modules and technical systems enables the researcher and the innovator to identify the unique selling point of the innovation compared to the competition. This division is less explicitly included in Osterwalder (2010), even though the difference can be significant in certain innovations. Following Innovatrix checks can be used to gauge for relevant input to the Solution criteria: What are the components of your (digital) solution? How do these components differ for the different customer segments?

Barriers – Future state

According to Steinkühler, Mahlendorf & Brettel (2014) self-justification is the most

empirically supported explanation for escalation of commitment, the “

become locked-in to a course of action, throwing good money after bad or committing new resources to a losing course of action” (Staw, 1981). Therefore, Steinkühler and his colleagues (2014) argue that self-justification cannot be totally avoided but for de- escalation of the commitments, the search for disconfirming evidence can help. Therefore, it was decided to explicitly include ‘barriers’ as an element to look for this disconfirming evidence. Following Innovatrix checks can be used to gauge for relevant input to the Barriers criteria: What are the barriers for adoption, usage and market entry?

tendency to

Value Capture – Future state

Ballon (2007) included the financial model in the Business Model Matrix, which described the revenue model and the revenue sharing model. Osterwalder (2010) also considers the revenues model, where the pricing level and the pricing model are mentioned. The researchers opted to utilize the definition of ‘Value capture’, which


comprises of the pricing model and the pricing level, and in cases where revenue sharing is applicable, this section can be utilized. The application of the Innovatrix matrix in different projects shows that partners can face difficulties identifying their pricing model and pricing level, and thus this needs to be included in the Innovatrix matrix. It is important to note that Value capturing has an important link with how pressing the customer need is, and to the associated value the partner promises to deliver. Following Innovatrix checks can be used to gauge for relevant input to the Value Capture criteria: What value (monetary and non-monetary) do I receive in return? What price should I set (and how)?

Key Partners – Future state

The value network definition is an alternative to the broad market-based approach of the Business Model Matrix of Ballon (2007). In the value network analysis, though the applicability is more adapted to innovations in the form of partnerships required to deliver the innovation to the customers and with whom do innovators need to collaborate. Following Innovatrix checks can be used to gauge for relevant input to the Key Partners criteria: Who are your key partners? How to interact with stakeholders?

are your key partners? How to interact with stakeholders? Innovatrix put to practice: from workshop to

Innovatrix put to practice: from workshop to innovation management process

In practice, the Innovatrix is used twofold: 1) as an innovation framework in a hands- on workshop session and 2) as an innovation management process.

First, as an innovation framework, the Innovatrix is used at the start of an innovation project, during a kick-off workshop. The most important roles in such a workshop are the trained Innovatrix facilitator and the innovator (or the innovator’s team). The workshop starts with an innovation pitch provided by the innovator. After this pitch, the eight distinctive Innovatrix criteria – as described above – are ‘filled’ with relevant input from the innovator. Here, the facilitator plays an important role in the gathering of all relevant input through the means of very specific probing questions in the form of ‘Innovatrix checks’. The gathered input is then awarded one of initially two possible statuses, based on the nature and the strength of the input: either assumption (the input has not yet been validated and is thus hypothesized by the innovator) or validated assumption (the input has already been validated in previous activities). Depending on the assumption status, the input is mapped on different- coloured post-its: yellow (assumptions) or green (validated assumptions). The outcome of an Innovatrix workshop is the mapping of assumptions and validated


assumptions, followed by marking the most important assumptions as ‘key uncertainties’. The focus of following research activities should then put on these key uncertainties.

Second, the Innovatrix is used as support of the innovation management process. Here, the the Innovatrix framework is used as the starting point of a living lab innovation project. The outcome of the Innovatrix workshop, key uncertainties, is then translated into testable assumptions and matched with adequate research and innovation activities. Research and innovation activities are then carried out. In a next Innovatrix workshop, the Innovatrix update, the focus is placed on these key uncertainties that were object of research and innovation activities. Here, the assumption status is under debate based on the outputs of the research and innovation activities. In an Innovatrix update, the status of an assumption can be changed in dialogue with the entrepreneur with following possible statuses:

assumption (the research has not been validated), validated assumption (the input was validated), new insight (new information arose from the research and innovation activities) and invalidated (the assumption proof to be invalidated). The Innovatrix is thus used in support of the innovation management process through the mapping and changing of assumption statuses in a structured dialogue with the innovator throughout the entire living lab innovation project. This process is repeated until the end of the living lab project.

process is repeated until the end of the living lab project. Methodology Within this paper, we


Within this paper, we adopt a mixed design with quantitative and qualitative data. For the quantitative part, we look at all innovation projects carried out by the user research team of imec.livinglabs from 2011 up till 2018, which makes for a sample of 86 projects. For this sample we coded whether Innovatrix was used or not, based on the project deliverables. In practice, the older projects did not use Innovatrix, whereas the more recent projects all made use of Innovatrix. We also coded the status of the project in terms of project outcome: ‘on the market’ if the innovation is available for adoption by end-users, ‘abort’ if the innovation project is stopped and the team members disband, ‘reboot’ if the innovation project is stopped, but the team members continue with a new innovation project, based on the insights, and ‘in development’ is used to indicate that the innovation is not launched yet. This can be regarded as an ‘in between’ category, as over time these projects will either become available on the market, be aborted or be rebooted. The data for the initial coding of the projects is taken from a post assessment interview at the end of the project. However, every year this database is updated based on an online search and a personal follow-up with the project owners to assess changes. The last update of the status dates from May 2018.


All of these projects were innovations with a digital component. The majority (58) had a focus on B2C, whereas the remaining 28 projects could be labelled as B2B. For an idea of some of the projects, see Schuurman (2015) and Schuurman et al.


For the quantitative analysis, we simply compared the numbers of the projects with Innovatrix and the (older) ones without Innovatrix (see Table 1). Because of the relatively small sample size, as compared to the outcome categories, no chi-square tests could be performed as the expected cell numbers were less than 5 for more than 20% of the cells. Therefore, we simply report the percentages. For the qualitative study, we selected cases where the Innovatrix had explicit value. These cases were chosen after a review of all projects by the co-authors, so the cases can be considered as illustrative case studies (Yin, 2017).

can be considered as illustrative case studies (Yin, 2017). Results The dataset consisted of in total


The dataset consisted of in total 86 entrepreneurial innovation projects. Almost half made use of Innovatrix, whereas 46 projects did not use the framework. When we look at the outcomes over all, we already see a few striking results. Roughly 1 out of 4 of the projects is stopped after the project and almost 1 out of 10 is rebooted based on the project insights, whereas 1 out of 10 is still ‘in development’ or implementing the lessons learned. However, it is remarkable that almost 60% ends up in a market launch. Referring back to the CB Insights study, where only 10% made in to the market, the type of projects is rather different. In that study, entrepreneurs from an incubator were studied, whereas in our study, the entrepreneurs should be considered as more mature as they contacted an innovation intermediary (our Living Lab) for support, defined a project proposal and paid for the project. However, this big difference is still remarkable.

Table 1



Living Lab

Living Lab


projects without

projects with





N: 19 – 41%

2 – 5%

N: 21 – 24%


N: 5 – 11%

2 – 5%

N: 7 – 8%

In development

N: 0 – 9%

9 – 23%

N: 9 – 10%

On the market

N: 22 – 48%

27 – 67%

N/ 49 – 57%


N: 46 – 53%

N: 40 – 47%

N: 86

As we look to the split between projects with and without Innovatrix, there are quite some interesting differences as well. Only two projects were aborted, whereas in the non-Innovatrix sample this is over 40%. Also, the percentage of reboots is also half as high in the Innovatrix-category. All of the projects that are still ‘in development’ belong to the Innovatrix category, which illustrates the point that this is a ‘transit’ category because over time all projects end up in one of the other categories. Finally, in this data set, two out of three innovation project that used innovatrix during their Living Lab project resulted in a market launch. As a lot of these projects were finished quite recently, these numbers will undoubtfully change, but still this contrasts heavily with the 9/10 failure ratio.

We now turn over to three case studies where Innovatrix was used and provided specific value to the innovation project.

Motosmarty: This project was about a mobile application that detected the driving behaviour of young people in order to give feedback and assess their risk profile. In term of end-user focus, there were no issues as the target population were young people and students. Co-creation sessions, surveys and user tests were performed to iterate the application. However, in terms of customer segment for the generated data of the application, there was no focus at all. Here, the Innovatrix was used to explicate all knowledge and assumptions regarding these customer segments (17 in total!). This led to discussions inside the team and made them realize that focus was needed, otherwise they would burn all their resources without finding a paying (B2B) customer. They are now on the market as Road Skippers and as Smart Drivers and focus on insurance companies.

Spott: This Living Lab project was about a new way to recognize, like, share and buy products with your smartphone based on what you see during a TV show or TV commercial. The Innovatrix workshop at the start of the project indicated three types of end-user segments and the television stations. In terms of value capture, the assumption was that an affiliate marketing fee was the main source of income for Spott. However, based on the co-creation sessions and field trials with the


application, it appeared that the ‘buying’ of items recognized during the TV-show was not that common, but that more adept TV viewers felt more connected to the TV content as they received more information on the objects that were used or worn by their favorite characters. This newly discovered evidence made Spott delete one of the end-user segments, that was focused on buying, and turned more towards the ‘heavy’ TV-viewers, with as main income TV station paying to use the application for their TV shows. The higher attraction of the content could attract advertisers, which made the application interesting for them and increased their willingness-to- pay. In this project, Innovatrix brought scope, identified unexpected outcomes and enabled to focus on a limited number of segments. At the moment, Spott has already been launched in multiple countries worldwide and is growing rapidly.

Lab Box: Lab Box is the organization behind Pikaway, a multi-modal transport application that is able to plan and book trips, and is not restricted to one or a few means of transport. At the starting workshop, it appeared that the three customer segments were still rather high level and in need of specification. To tackle this, we did a segmentation survey and subsequently conducted a field trial with representatives of the user segments. This enabled to create persona, which provided focus for the developers to prioritize their development backlog. Moreover, one of the key assumptions captured during the first workshop, the need for a one- stop shop application, could be iteratively validated by capturing the frustrations with the current practices in the segmentation survey and co-creation session, but this was also expressed in the field trial so this was also reflected in the future state (solutions element). At the end of the project, the people from Lab Box asked for the main take-aways and next steps for them based on the project. By going through the Innovatrix and the modifications step by step, we could easily extract the main learnings and key elements to work on before the market launch. At this moment, Pikaway is available in the app store and a launch in the Play Store is planned for the near future.

a launch in the Play Store is planned for the near future. Discussion & Conclusion Although

Discussion & Conclusion

Although Living Labs are regarded as orchestrators, which hints at an innovation management approach, there is a lack of literature and studies that further explicate this role of the process. Rather, Living Labs are described and studied in terms of defining characteristics (such as real-life experimentation, active user co-creation and public-private-people partnership), but it is left untouched how these elements should be managed and utilized according to the needs and characteristics of a specific innovation project. For Living Labs to take the next step in becoming mature and established innovation organizations, we feel that this innovation management role should be further elaborated and that this is even crucial in the complex entities that Living Labs are. The three-layered model of Schuurman (2015) provides a


useful framework to anchor these elaborations. In our literature review, we noticed that the largest gap in terms of the orchestration role in Living Labs is situated on the project level. Therefore, within this paper we focused on the question how innovation management in Living Lab projects can be facilitated and supported by tools or frameworks.

As a result, we presented the Innovatrix framework, consisting of eight elements, derived from existing business model tools and frameworks, with as specific characteristic the fact that all elements of the framework should be specified for each customer segment that is identified. Moreover, the Innovatrix also has a clear distinction between the current state elements (the top three) and the future state elements (the bottom four), which gives it a more dynamic, process-like feeling.

Based on an analysis of 86 Living Lab projects, where just over half did not use Innovatrix and just under half did use Innovatrix, we discovered that two out of three projects using Innovatrix resulted in a market launch. More data is needed to validate these findings, but this gives a first indication of the effectiveness of the framework. Moreover, based on three illustrative case studies of projects from the sample that used innovatrix, we can derive three propositions regarding the use and implications of the Innovatrix framework.

First, Innovatrix helps to scope the user involvement activities, as it clearly explicates assumptions, related to the different customer segments, and it also enables indicating which assumptions are key for taking the next steps in the project. This leads to a more efficient use of resources and facilitates the selection of representative users for the given customer segments and also guides the choice of method to validate the assumption.

Second, Innovatrix forces the project owner to focus on a limited number of customer segments, as there is only room for three to four segments maximum. If there are more segments, the elements of the Innovatrix help to choose between different segments in terms of focus. This increases the efficient spending of the scarce entrepreneurial resources and helps decision making for the innovation teams. Third, Innovatrix allows to capture the iterations and pivots that were made during an innovation project. This is esecially useful to link outcomes with certain Living Lab activities, as this has appeared rather problematic to this date (see Ballon et al., 2018). Innovatrix serves as a visual summary of key elements and assumptions regarding an innovation project from the viewpoint of the end-user. By capturing snapshots of its state before and after a research activity, the modifications and alterations become apparent. To this end, at the moment a digital version of Innovatrix is being built that enables to fill out Innovatrix digitally and keep track of all changes during a project. Moreover, Innovatrix is also an interesting tool to facilitate the discussion between Living Lab researchers and the project owners.


To further explore and validate these propositions, more research is needed and more cases are needed to assess the value of Innovatrix. Also, Innovatrix represent a specific view on innovation management and Living Lab activities. It has been used in a ‘Living-Labs-as-a-service’-context, but might be applicable in other contexts as well. We encourage uses and tests in other contexts, other types of projects and other organizations in order to increase the knowledge on innovation management in a Living Lab-context and to help building a more structural, encompassing Innovatrix for all kinds of Living Lab projects and activities to increase the impact and position of Living Labs as innovation intermediaries.



Almirall, E., & Wareham, J. (2008). Living labs and open innovation: Roles and applicability. eJOV: The Electronic Journal for Virtual Organization & Networks, 10. Almirall, E., & Wareham, J. (2011). Living Labs: arbiters of mid-and ground-level innovation. Technology Analysis & Strategic Management, 23(1), 87-102. Baccarne, B., Logghe, S., Veeckman, C., & Schuurman, D. (2013). Why collaborate in long-term innovation research? An exploration of user motivations in Living Labs. In 4th ENoLL Living Lab Summer School 2013. European Network of Living Labs. Ballon, P., Van Hoed, M., & Schuurman, D. (2018). The effectiveness of involving users in digital innovation: Measuring the impact of living labs. Telematics and Informatics. Bergvall-Kåreborn, B., Eriksson, C. I., Ståhlbröst, A., & Svensson, J. (2009). A milieu for innovation: defining living labs. In ISPIM Innovation Symposium:


Bergvall-Kåreborn, B., Howcroft, D., Ståhlbröst, A., & Wikman, A. M. (2010, March). Participation in living lab: Designing systems with users. In IFIP Working Conference on Human Benefit through the Diffusion of Information Systems Design Science Research (pp. 317-326). Springer, Berlin, Heidelberg. Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63-72. Blank, S. G. (2006). The Four Steps to the Epiphany. Foster City, Calif.: CB Insights (2018). 253 Startup Failure Post-Mortems. Retrieved at Coorevits, L., & Schuurman, D. (2014). Hypothesis Driven Innovation: Lean, Live and Validate. In ISPIM Conference Proceedings (p. 1). The International Society for Professional Innovation Management (ISPIM). D'Hauwers, R., Rits, O., & Schuurman, D. (2015). A hypothesis driven tool to structurally embed user and business model research within Living Lab innovation tracks. In Open Living Lab Days 2015.

Gage, D. (2012). The Venture Capital Secret: 3 Out of 4 Start-Ups Fail. Retrieved at


Gourville 2005 - Gourville, J. T. (2005). The curse of innovation: a theory of why innovative new products fail in the marketplace. HBS Marketing Research

Paper, (05-06). Herregodts, A. L., Baccarne, B., Conradie, P., & Schuurman, D. (2017, June). Managing Innovation Uncertainties: a User-Oriented Knowledge Typology. In ISPIM Innovation Symposium. The International Society for Professional


Innovation Management (ISPIM). Kristensson, P., Matthing, J., & Johansson, N. (2008). Key strategies for the successful involvement of customers in the co-creation of new technology- based services. International journal of service industry management, 19(4),


Leminen, S., Westerlund, M., & Nyström, A. G. (2014). On becoming creative consumers–user roles in living labs networks. International Journal of Technology Marketing, 9(1), 33-52. Leminen, S. (2015). Living Labs as Open Innovation Networks-Networks, Roles and Innovation Outcomes. Lovallo, D. & Kahneman, D. (2003). Delusions of success. Harvard business review, 81(7), 56-63. Maurya, A. (2012). Running Lean: Iterate from Plan A to a Plan That Works. Sebastopol, CA: O'Reilly McPhee, C., Schuurman, D., Ballon, P., Leminen, S., & Westerlund, M. (2017). Editorial: Innovation in Living Labs (January 2017). Technology Innovation Management Review, 7(1): 3-6. Osterwalder, A. & Pigneur, Y. (2013). Business model generation: a handbook for visionaries, game changers, and challengers. John Wiley & Sons. Osterwalder, A., Pigneur, Y., Bernarda, G., & Smith, A. (2015). Value Proposition Design: How to Create Products and Services Customers Want. John Wiley & Sons. Ries, E. (2011). The Lean Startup: How Constant Innovation Creates Radically Successful Businesses. New York: Crown Business. Rits, O., Schuurman, D., & Ballon, P. (2015). Exploring the Benefits of Integrating Business Model Research within Living Lab Projects. Technology Innovation Management Review, 5(12). Schaffers, H., Cordoba, M. G., Hongisto, P., Kallai, T., Merz, C., & Van Rensburg, J. (2007, June). Exploring business models for open innovation in rural living labs. In Technology Management Conference (ICE), 2007 IEEE International (pp. 1-8). IEEE. Schuurman, D., & De Marez, L. (2012). Structuring user involvement in panel-based Living Labs. Technology Innovation Management Review, 2(9). Schuurman, D., De Marez, L., & Ballon, P. (2013). Open Innovation Processes in Living Lab Innovation Systems: Insights from the LeYLab. Technology Innovation Management Review, 3(11): 28-36. Schuurman, D. (2015). Bridging the gap between Open and User Innovation?:

exploring the value of Living Labs as a means to structure user contribution and manage distributed innovation (Doctoral dissertation, Ghent University). Ståhlbröst, A. (2013). A living lab as a service: creating value for micro-enterprises through collaboration and innovation. Technology Innovation Management Review, 3(11).


Staw, B. M. (1981). The escalation of commitment to a course of action. Academy of management Review, 6(4), 577-587.

Steinkühler, D., Mahlendorf, M. D., & Brettel, M. (2014). How Self-justification Indirectly Drives Escalation Of Commitment - A Motivational Perspective. Schmalenbach Business Review: ZFBF,66(2), 191. Wildevuur, S., van Dijk, D., Hammer-Jacobsen, T., Bjerre, M., Äyväri, A. and Lund, J. (2013), Connect. Design for an empathic society, BIS Publishers, Amsterdam, NL. Yin, R. K. (2017). Case study research and applications: Design and methods. Sage publications.


International Innovation Sprint Bridging the Sustainability Gap between Metropolitan Core and Peripheries

Tuija Hirvikoski 1 , Kaisla Saastamoinen 2 and Mikael Uitto 3

1,2 Laurea University of Applied Sciences

Category: Innovation paper

Abstract Generally, RDI work takes place at the core of Metropolitan areas where the best innovation resources, Universities, and Research Institutions normally situate. At the same time, the periphery of Metropolitan that provides the core area with many vital resources can itself suffer from migration loss, brain drain, loss of jobs, and the many challenges related to ageing population and long distances or poor logistics. In this paper we aim to contribute to the discussions on the role of science, technology and innovation in society and particularly in making cities and communities inclusive, resilient and sustainable (UN SDG 11). We adopt perspectives and concepts from innovation literature and policy documents to introduce Innovation Sprint as an innovation intermediary tool. We explain how the sprints were designed and experimented first in Taiwan and then in Finland. In both cases, multidisciplinary and international Sprints were taken to a remote community to observe, understand, and then to co-create innovative solutions with and for the local stakeholders. We also discuss the ways the Sprint might bridge the sustainable development gaps between the urban, peri-urban and rural areas where the intensity of knowledge, technology, and monetary resources can vary substantially.

Keywords: Innovation Sprint; Living Labs; sustainability gap; science, technology and innovation in society; ICT solutions making cities and communities inclusive, resilient and sustainable


Theoretical framework: The role of science, technology and innovation in society The role of science

Theoretical framework: The role of science, technology and innovation in society

The role of science and technology in society has been discussed since 1971, when the OECD report Science, Growth and Society (OECD 1971) was published. Since then, science and innovation policies have tried to reconcile the curiosity driven science and its inner need of autonomy with the need driven science (Wessner 2005) and society’s wish (EC 2014, 2016 & 2018) to enjoy the fruits of science. Decades ago, the OECD innovation policy stressed that the linear science-technology innovation (STI) was to be replaced by interactive or systemic innovation (Miettinen et al. 2006). The systemic nature of innovation refers (e.g. Fagerberg 2006) to the collective achievement of innovation through interlinking actors, activities, and innovation system.

As a consequence of the OECD innovation policy, the connections between science, technology and economy, quality of life, and societal challenges have become important research topics. In 2018, the role of science and innovation in society is emphasised with such policies as Responsible Research and Innovation (EC 2014), Open Innovation, Open Science Open to the World (EC 2016), Sustainable Development Goals (UN 2015) and Mission oriented research and Innovation (EC


Moreover, examples as the Innovation Manifesto (ENoLL 2018) show how the locus of emphasis is simultaneously changing from business innovation to social digital innovation.

As urban areas expand and interweave, the role of science and innovation has been discussed in new urban configurations, and urbanization has been considered an open process, determined by constant innovation and inventiveness (e.g. Diener et al 2005). In the urban context, interactive or systemic innovation has emerged parallel to the linear science-technology innovation (STI). Concepts such as Cities as Urban Labs (Evans et al 2016) have been used to illuminate how the ST–innovation can be tested and validated in the urban context. At the same time, interactive innovation has been connected to such notions as Doing, Using and Interacting (DUI) (Lundvall 1985), Mode two of knowledge production (Gibbons & al. 2005), Quadruple Helix (Curley & Salmelin 2018) based on Etzkowitz’s (2002) Triple Helix of Academia, Industry and State, or Open Innovation 2.0 (Curley & Salmelin 2018), Living Labs (The Helsinki Manifesto 2006), and City as a Living Lab (ENoLL 2016), or the city as an open innovation platform (6Aika).


Finland can be considered an example of sustainable optimization through technology and Living Labs. The Smart and Clean Helsinki Metropolitan (2017) claims the capital is the best testbed or living lab in the world for smart and clean solutions, “providing a unique environment for getting things done”. They aim at changes that will lead to new, permanent processes that will improve quality of life, are carbon positive and value adding, and will boost circular economy with the world’s most resource-wise citizens.

Although such regional policies as the Regional Cities Programme (2018) rely on effective cooperation between the regional cities and the state, less attention has, however, been paid on how science, technology and innovation can help to bridge the sustainability gap between Metropolitan agglomerations and their immediate hinterlands, where the leading science and innovation organisations seldom exist or have any activities.

organisations seldom exist or have any activities. Methods and Case Studies In this chapter, we will

Methods and Case Studies

In this chapter, we will introduce the method Innovation Sprint as a living lab and two cases bringing international experts with scientific knowledge to the Metropolitan periphery to collectively solve hinterland challenges with the local stakeholders. The Sprint also epitomizes how education, science, and innovation in society can be successfully re-contextualised.

Living Labs with and for sustainable ICT solutions making cities and communities inclusive, resilient and sustainable

With Living Labs and sustainable ICT solutions making cities and communities inclusive, resilient and sustainable we refer to the UN Agenda 2030 and the Sustainable Development Goals (SDGs) a global sustainability agenda with 17 ambitious goals, agreed in 2015 by 193 countries within the United Nations framework.

Westerlund and Leminen (2011) define Living Labs as “physical regions or virtual realities, or interaction spaces, in which stakeholders form public–private–people partnerships (4Ps) of companies, public agencies, universities, users, and other stakeholders, all collaborating for creation, prototyping, validating, and testing of new technologies, services, products, and systems in real-life contexts”.

Making ICT deliver sustainability - Co-design for sustainable lifestyles (G-Stick conference 2017) emphasized the importance of testing, experiencing, and co- creating innovation through Living Labs as intermediary orchestrators in


multidisciplinary and cross-polluting sectors, by making citizens the driving factor of innovation creation. The conference defined Living Labs as user-centered, open innovation ecosystems based on a systematic user co-creation approach. They integrate research and innovation processes in real life communities and setting, placing the citizen at the center of innovation. Living Labs have in this way demonstrated their ability to better mold the opportunities offered by new ICT concepts and solutions to the specific needs and aspirations of local contexts, cultures, and creativity potentials.

In this paper, we consider Innovation Sprint as a full-scale urban, peri-urban or rural living labs, proving ground for learning, inventing, prototyping, analyzing, assessing, and creating market opportunities for new digital social innovation or ICT technology applications.

Due to the crosscutting nature of ICT and Living Labs, they jointly have the capacity to create and support systemic transitions leading to an inclusive, safe, resilient and sustainable ‘smart’ city development. This capacity is based on the fact that Living Labs are based on trust capital whereas the citizens are perceived as innovation actors or prosumers, not as research factors. Prosumers refer to consumers who become involved with designing or customizing e.g. digital products for their own needs as consumers.

Living Labs capturing the needs of a broader range of users ensures a higher rate of inclusion of technology. They collaborate with the lead users to face the market needs months or years before the bulk of that marketplace encounters them. They also co- create with the vulnerable users or users at-risk parts of the society with a poorer access to services and technologies and therefore are at the risk of exclusion. Often Living Labs aim to provide equal learning, innovating, and capacity building opportunities for all citizens; therefore, they create trust-capital into the society and local communities.

The original idea of Living Labs was with the help of co-creation and rapid experimentation to shorten time between digital development and market deployment. Living Labs can be utilized as a tool for changing behaviour towards sustainable actions of consumers.

Innovation Sprint as an innovation intermediary tool for Living Labs

Innovation Sprint is based on the idea of Living Labs as an open innovation ecosystem or a platform allowing sharing and seamless interaction between all stakeholders such as cities, citizens, companies, and academia (Curley & Salmelin


2017). The Sprint offers a physical place and virtual space bringing together the different stakeholders. During the Sprint, co-creation and real-time experimentation is conducted in real-world situations, allowing simultaneous technical and societal innovation.

As we explain in this paper, international university students are a crucial element of the Living Lab based on Sprint. During the Sprint, the students link the scientific knowledge from the Universities to the market-based knowledge from the hinterland citizens and local authorities. However, the Sprints sometimes have received negative feedback from students, when run by big corporations. A Sprint can be seen as a way of drafting a lot of unpaid talent to solve a problem for the financial gain of the corporation while leaving the Sprinters with nothing but warm thanks for their ideas.

To avoid this problem, our Sprints focused on creating public good. Sprint, when internalizing the challenges related to making cities and communities inclusive, resilient, and sustainable and externalizing public good, makes it easier for everyone to find inspiration without the risk of feeling that one’s intellectual property rights have been exploited.

Next, we will briefly introduce the two Innovation Sprints, both of which were based on service design thinking (Ramaswamy and Goiullart 2010; Ojasalo, Koskelo, Nousiainen, 2015). Moreover, they applied Wenger’s (1998) Social theory of learning and Living Lab approach to integrate international universities’ scientific and innovation know-how to the real-live challenges and resources situating in the hinterlands of Metropolitan areas.

Innovation Sprints as a transdisciplinary and agile development process

An Innovation Sprint is a transdisciplinary and agile development process where design thinking and co-creation play a key role. In a Sprint, a team or teams are working on tackling a real-life challenge through rapid prototyping and testing, making the problem-solving process considerably faster and more human-centric than in traditional project work. Sprint teams are multidisciplinary to optimise benefitting from each team members' diverse individual knowledge and skills as well as from the multidisciplinary co-creation as a team. A Sprint can follow service design, design thinking, or other agile development and innovation processes. It is important that the participants will be provided with enough information on both the challenge at hand and the way of working, as well as the reasons for selecting the process in question, so that even in a short time of the Sprint they can trust the chosen methods and their functionality in the process.


The Asian Smart Living Sprint

The first case comes from Taiwan. The first Asian Smart Living Summer School was organised in 2011 in Taiwan and the Sprint title was “Innovation and Connection:

Crafting Smart Journey in Formosa.” The design of the Sprint was based on the double diamond method with 4Ds: discover, define, develop, and deliver (British Design Council).

First, the Sprint took the participants on a Journey to observe and understand the challenges and opportunities related to the communities situating at the Metropolitan hinterlands. Three communities were selected to facilitate creative connections between cultures and technologies, the young and the elderly, and people and the environment: 1) a Taiwanese community of aboriginals reviving traditional industries with innovation, 2) an ageing village outside Taipei, offering an ageing life without limitation on body, mind and spirit, and 3) a Zen Buddhist community integrating cutting-edge technology and rich cultural content.

In Taiwan, the aim was to create a prototype of a new program modernising academic education and design-based innovation and to experiment the program in practice. Therefore, the Ministry of Education of Taiwan sponsored the program. It was jointly organised by the Talent Cultivation Program for Smart Living Industry and National Taiwan University, National Chiao Tung University, and Dharma Drum Buddhist College. Laurea University of Applied Sciences and ENoLL joined the prototype development and its experimentation.

The first, 5-day edition of the Sprint offered an organised innovation and learning program for almost 100 people from Finland, Japan, Malaysia, Russia, Singapore, Taiwan, USA, and Vietnam. Forming six teams, the participants defined and then worked on the challenge they deemed most important and most viable to be solved with the resources they had access to during the one-week encounter with the mentoring university professors and staff, other students, and with the locals. As a result, six technology-enhanced prototypes and their value propositions were introduced to the jury and the representatives of the local communities, researchers, and companies. (Sung & Jou 2011)

The European Digital Wellbeing Sprint

The second case is from Finland, where a small municipality situated at the outer skirts of Helsinki Metropolitan hosted a Digital Wellbeing Sprint in 2017. In Finland,


the Sprint was initiated by Laurea and jointly organised by the strategic alliance of three Universities of Applied Sciences (Laurea, Metropolia and Haaga-Helia).

In the 6-day Sprint, students, the municipality at the outer skirts of Helsinki Metropolitan area, and the organising UASs worked in collaboration to tackle the municipality’s challenge “Living well all the way”. The purpose was to gather deep understanding of the every-day life and challenges of the inhabitants, especially those with memory disorders, and then ideate and prototype proposed solutions based on the gathered understanding. The teams followed the Service Innovation process introduced by Ojasalo et al. (2015): Map & Understand; Forecast & Ideate; Model & Evaluate; and Conceptualize & Influence.

More than 30 Bachelor and Master students from the 3 UAS alliance and partner universities, as well as exchange students, formed eight Sprint teams. The participants were from Finland, Romania, Germany, Vietnam, Morocco, Philippines, Chile, China, Kenya, and Pakistan; with their fields of study ranging from Healthcare to Service Design, and Business to IT. The Sprint teams had more senior Masters students as well as staff members of the three UASs as Sprint mentors. Part of the teams spent the majority of the Sprint in the challenge owner municipality, immersing themselves in the inhabitants’ life and environment in practice.

At the end of the Sprint, each team pitched their proposed solution to the jury consisting of members of the municipality administration, a member of staff in a local care home, and a UAS teacher and facilitator of the Sprint. Each proposed solution pitch was evaluated on the criteria of 1) the usability, potential of realisation in reality, and originality of the team’s idea or concept; 2) how the team delivered the pitch; 3) how the team had used the different methods and followed the Sprint process, especially paying attention to consideration and usage of Futures Thinking and Value Proposition Design; and 4) focus and attention on customer, the municipality’s resident with memory disorder. The scalability of the teams’ propositions was also considered. As the end result, the municipality gained 8 diverse prototyped service propositions and concepts that the municipality could further refine.

and concepts that the municipality could further refine. Results An Innovation Sprint can work as a


An Innovation Sprint can work as a means for bridging the sustainability gap between a Metropolitan area and its peripheries. The Sprint offers an invaluable opportunity for the periphery to gain access to a skilled, motivated workforce with fresh ideas and proposed solutions to their challenges. The process being facilitated by a University ensures the quality of theoretical background and novel approaches. The participants


gain a learning experience in a multidisciplinary team, learning-by-doing in a real-life setting. The collaboration between the stakeholders enables the periphery to benefit from the Metropolitan higher intensity of knowledge and technology without expenditure of their monetary resources.

For an Innovation Sprint to produce potentially usable proposed solutions to real-life challenges, it is vital that all stakeholders share the mindset of open innovation. The challenge provider is prepared for collaboration and sharing of information with both the organisers as well as the participants of the Sprint; the organiser provides a space and time to connect the stakeholders and offers theoretical and facilitative support on the process, and the participants do their best in sharing and using their individual strengths to work together effectively. In our example cases, besides helping formulate the challenge to suit a short Sprint, the challenge owners provided Sprint organisers and teams with the time of some of their key staff as well as with the opportunity for participants to immerse themselves in the local life in the real setting of the challenge. Part of the teams were working the majority of the Sprint in the hinterland municipalities where they not only observed the general setting but also attended visits organised by the municipalities to observe the daily life of e.g. local service providers and to interview their inhabitants and staff. This provided participants with invaluable insight into the challenge and led to prominent proposed solutions for the challenge owner peripheries.

In total, the example case Sprints offered the small municipalities on the outskirts of Helsinki, Taipei, and Hsinchu Metropolitan areas about 700 full days of work by motivated, multidisciplinary, and multi-cultural participants. Besides the working hours, expertise, and insights gained by the municipality during the Sprint, even a short Sprint can also help build longer relationships between different stakeholders, benefitting all parties involved. Some of the concepts or ideas that result from a Sprint could immediately be developed further or spark ideas for prototyping and testing others, and the Sprint participants – already immersed in the challenge – could be perfect for the job. Ideas and concepts born during a Sprint can also grow into Start- ups or improved products or services of an existing organisation. Typically, start-ups are concentrated in Metropolitan areas, however their operation can benefit a wider audience and in turn continue to take part in revitalising the periphery.

Innovation Sprints can help universities develop their operation and its effectiveness by actively facilitating open collaboration and its benefits for the various stakeholders. As for the roles of universities, Innovation Sprints have clear benefits also on both pedagogic and regional development level: Sprints help generate opportunities and obtaining of new competences for both the student participants as well as the region as a whole. A Sprint also enhances the university’s competitiveness, contributing to regional development objectives and showcasing the utilizing of methods such as


open innovation, co-creation and rapid prototyping – skills indispensable to modern working-life and its development. The results imply that innovation-in-interaction between and within higher education, science, and society can be fruitful to all parties while supporting positive social and economic links between the Metropolitan and its peripheries (UN SDG 11). Benefits for the periphery

Through collaboration with Sprint participants, the municipality providing the challenge has the opportunity to gain both out-of-the-box new solutions as well as potential development ideas of current situation with a fresh outlook. At the end of the Sprint, in Taiwan 6 and in Finland 8 prototypes or concepts were introduced to the jury and for the stakeholders to utilize.

Besides the presented end results of the Sprint in the form of a pitch or similar, a Sprint can work as a valuable base for longer, productive relationships between its stakeholders. In the case of the Sprint 2017, together with the Finnish municipality two student participants have continued working further on one of the concepts born in the Sprint. Another student started their Master’s thesis on a topic related to the Sprint theme: their thesis topic was redefined based on the Sprint experience in the real-life setting, and the connection with the municipality staff established during the Sprint enabled them to conduct their thesis research with the actual inhabitants. Also, the inhabitants of the municipality who were part of the Sprint process e.g. in the role of interviewees or observees reported gaining a feeling of inclusiveness in being seen and heard in a process of tackling the challenges that affect their every-day lives.

Benefits for the Metropolitan

Solutions born as a result of innovation Sprints can generate commercial activities and tax revenue that benefit the society as a whole – both in the Metropolitan area and its peripheries. Start-ups are born both on ideas sparked during Sprints as well as a result of the Sprint collaboration: two team members from the winning team of Digital Wellbeing Sprint 2017 went on to attend an international entrepreneurship and start-up accelerator bootcamp where teaming up with 2 others they founded a health- related start-up company.

Method Development

Innovation Sprints as a new Living Lab method integrate University’s curricular and RDI activities with its stakeholders’ needs and resources. Through Sprints, Universities can offer their theoretical knowledge and facilitation skills to enable a meaningful and practical learning experience for students, and at the same time offer


valuable resources and input for the other stakeholders and the region as a whole. Innovation Sprints can also facilitate longer-term relationships between the stakeholders, benefitting them and the local area long after the Sprint has finished.

Innovation Sprints create spaces for dialogue and orchestrate co-innovation among academy, industry, SMEs, Civil Society Organisations (CSOs), and policy makers. This contributes to sustainable regional development and allows the peripheries of Metropolitans to benefit from the higher intensity of knowledge and technology traditionally typical of the core area.

If integrated into Universities’ other Living Lab orchestration activities, Sprints can effectively explore systematic and institutional adaptation models to future regional priorities and synthesise new solutions.

Benefit for the participants

Innovation, co-creation, and experimentation with different national and international stakeholders provide students and other participants with excellent learning and employment opportunities with a potential of new Start-ups. Participants attending Innovation Sprints gain skills and practise vital in the modern working-life: open innovation, rapid prototyping and testing, and co-creation. They can practise these skills in a real-life environment with a real challenge with skilled facilitators, and take their learnings – as well as the connections they have gained – into their studies or working life.

Reconciliation of curiosity driven and need driven science

The Innovation Sprint brings the curiosity and need driven science into mutual interaction. An Innovation Sprint organised in the outer skirts of Metropolitan area provides a wide variety of observations and data on inclusion, resilience, and sustainability, allowing the researchers to decentre their focus of analysis to illuminate the wider context of an urban territory. The Sprint, intertwining the interactive and iterative need driven approach with the curiosity driven science is an additional tool to bridge the sustainability gap between the Metropolitan centralities and peripheries.

Challenges and Limitations

To succeed in producing viable proposed solutions to real-life challenges, innovative operative methods such as co-creation and innovation Sprints and Living Labs call for innovative leadership and freedom within framework from all stakeholders. Additionally, to succeed these methods call for professional facilitation. An academic


institution or another impartial organisation is in a prime position to create trust and facilitate gathering of relevant stakeholders to be involved, as well as to provide needed background theory and support on the chosen framework. Whereas some scholars (for example Kivistö and Pihlström 2018) are sceptical about agile methods to be used for research and innovation purposes, a viable Business Model can be created thanks to the fact that the Sprints provide a learning opportunity to the University students. However, the costs related to trips, accommodation, venues and equipment is a challenge. Different Business Models has been applied for different challenges; the most common ones are based on external project funding or the challenge owners’ compensations.

project funding or the challenge owners’ compensations. Conclusions and Further Suggestions for the practitioners,

Conclusions and Further Suggestions for the practitioners, researchers, and policy makers

During the physical Innovation Sprint, the collaboration between the stakeholders enables the periphery to benefit from the Metropolitan higher intensity of knowledge and technology without expenditure of their monetary resources. Today, the multiple disruptive technologies, all arriving at the same time, can be used in the Living Labs and to improve the Innovation Sprint’s benefits to all stakeholders. With the following three suggestions based on the EC (2016) and OECD (2017) policies, we anticipate to balance the sustainable development between the Metropolitan periphery and core areas. Firstly, the Sprint stakeholders and participants should be equipped with basic knowledge on the possibilities provided by different disciplines of computer science (such as Artificial Intelligence (AI), Data Mining, Bloc Chain, or Cloud Computing) relevant to the Innovation Sprint. Whilst, during the Innovation Sprint, the computer scientists carry the responsibility to compose the actual algorithms for digital innovation, creating innovation in multidisciplinary Sprint teams would become easier if all the participants shared an intuition of the potentials related to digitalisation. That is, all the participants should understand what is possible and not possible through programming, and what are the challenges and major implications of the major digital innovations such as AI or Block Chain. Therefore, we would recommend such knowledge to become a prerequisite for all the Sprint participants. Such virtual courses are available on internet free of charge 1. Secondly, as big data and data-driven innovation is creating significant business opportunities, data should also be used to solve the sustainability challenges of the municipalities situating in the outer skirts of the Metropolitan areas. Aligned with the Findable, Accessible, Interoperable and Reusable (FAIR) Data policy



(Implementation Roadmap for the European Open Science Cloud), the Innovation Sprints should create and agree on the Data Management Plans, to help the Sprint participants in different countries and times to collect, share, and use data before, during, and after the Innovation Sprints. Innovation Sprints and Living Labs sharing data would leverage better solutions for local authorities responsible for urban, peri- urban and rural areas to become inclusive, resilient and sustainable. Therefore, Digital Innovation Sprints should be included into local, regional and EU strategies to support science based economic, social, and environmental links between the Metropolitan cores and peripheries and to promote territorial inclusion within EU. Moreover, FAIR Data provides new Business opportunities for the Living Labs contributing and sharing data on same phenomenon. Thirdly, based on the previous two suggestions we recommend transnational co- creation and experimentation Innovation Sprints to be organised simultaneously in different countries and to connect them digitally. With the help of internet connections, the Innovation Sprint participants in urban, peri-urban, and rural areas could create and exchange more data to develop better solutions and to receive immediate feedback from the rapid experiments in different legislative, cultural, language, climate, and logistic conditions. Internationalization would have one further benefit. The delocalized but simultaneous Innovation Sprits would decrease the travel costs of the Innovation Sprints without losing the access to cultural diversity. The data collected during the various Innovation Sprints could operate as an attraction factor for the world’s leading researchers. Their involvement before, during, and after the Sprints would create better conditions for the cities and rural areas at the hinterland to benefit from the combination of the agile and fast face-to-face Innovation Sprints and their permanent connections to fundamental research and innovation institutions in any global metropolis. Moreover, continuous research is needed to assess the impacts of the Digital Innovation Sprints on learning and innovation results. With these recommendations, we believe the Digital Innovation Sprint will promote Commissioner Carlos Moedas’ (2017) vision: “The year is 2030. Open Science has become a reality and is offering a whole range of new, unlimited opportunities for research and discovery worldwide. Scientists, citizens, publishers, research institutions, public and private research funders, students and education professionals as well as companies from around the globe are sharing an open, virtual environment called The Lab”.



6Aika. Six Finnish cities join forces to become better and smarter. english/ British Design Council. The Design Process: What is the Double Diamond.

diamond Curley, M. ja Salmelin, B. (2018). Open Innovation 2.0. The New Mode of Digital Innovation for Prosperity and Sustainability. Springer, Cham. Diener, R., Herzog, J., Meili. M., Meuron,P., & Schmid, C. (2005). Switzerland – an Urban Portrait. EC (2006). The Helsinki Manifesto 20.11.2006. We have to move fast, before it is too

late. Finland EU Presidency

EC (2014). Rome Declaration on Responsible Research and Innovation in Europe. Italian Presidency of the Council of the European Union

EC (2016). Open innovation, open science, open to the world - a vision for Europe. Produced by European Commission's Directorate-General for Research & Innovation (RTD). EC (2018). Mission-oriented research and innovation policy. A RISE perspective – Study. Directorate-General for Research and Innovation. EC Committee of the Regions. Innovation Camp.

trust-in-europe.aspx ENoLL (2018). The Manifesto for Innovation in Europe. Evans, J., Karvonen, A. & Raven, R. (Eds.) (2016). The Experimental City. Routledge

research in sustainable urbanism. New York: Routledge. Etzkowitz, H. (2002). The Triple Helix of University - Industry – Government Implications for Policy and Evaluation. Science Policy Institute Working paper


Fagerberg, J. (2006). Innovation: A Guide to the Literature. In Fagerberg, J., Mowery, D. C. & Nelson, R. R. (Eds). The Oxford Handbook of Innovation (pp. 1-26). Oxford: Oxford University Press. Gibbons, M., Limoges, C., Nowotny, H., Schwatzman, S., Scott, P. & Trow, M. (2005). The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies. London: SAGE G-STIC 2017, key conference findings. findings Hirvikoski, T. & Saastamoinen, K. (Forthcoming). Innovation Sprint – Manual for organisers. Laurea publications.


Kivistö, S. & Pihlström, S. (2018). Sivistyksen puolustus. Miksi akateemista elämää tarvitaan? Talinna: Gaudeamus Lundval, B-Å. (1985). Product innovation and producer-user interaction. Industrial Development Research Series, 31, 79-91. Aalborg University Press. Miettinen, R., Tuunainen, J., Knuuttila, T. & Mattila, E. (2006). Tieteestä tuotteeksi? Yliopistotutkimus muutosten ristipaineessa. Helsinki: Yliopistopaino

OECD (1971). Science growth and society: a new perspective. Report of the Secretary-General's Ad Hoc Group on New Concepts of Science Policy. OECD (2017). Making Innovation Benefit All: Policies for Inclusive Growth. Paris:

OECD Publishing. Ojasalo, K., Koskelo, M., & Nousiainen, A. K. (2015). Foresight and service design boosting dynamic capabilities in service innovation. In The Handbook of Service Innovation (pp. 193-212). Springer, London. Ramaswamy V. and Goiullart, F. (2010). The power of cocreation. New York: Free Press. Regional Cities Programme secretariat and advisory group (2018). Regional cities programme – Independent development and multilateral partnerships. Ministry of Finance publications 21/2018. Helsinki.


The Smart and Clean Helsinki Metropolitan (2017). Sung, T. J. & Jou, S.C. (2011). 2011 Asian Smart Living Summer School Report. Innovation and Connections. The Talent Cultivation Program for Smart Living Industry. Ministry of Education R.O.C. UN (2015). The 2030 Agenda for Sustainable Development.

Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. Cambridge University Press. Westerlund, M., & Leminen, S. (2011). Managing the Challenges of Becoming an Open Innovation Company: Experiences from Living Labs. Technology Innovation Management Review, 1(2): 19-25. Wessner, C. W. (2005). Entrepreneurship and the Innovation Ecosystem Policy Lessons from the United States. In Audretsch, D.B., Grimm, H. & Wessner W.C. (Eds.). Local Heroes In the Global Village: Globalization and the New Entrepreneurship policies. New York: Springer science & Business media.


Learning with Community:

Developing Citizen-Led Housing

Shaofu Huang 1 , Ruth Deakin Crick 1,3 , Melissa Mean 2 , Carolyn Hassan 2 , Colin Taylor 1

1 University of Bristol 2 Knowle West Media Centre 3 University Technology Sydney

Category: Research Paper

3 University Technology Sydney Category: Research Paper Abstract Creating a flourishing city and delivering the

Abstract Creating a flourishing city and delivering the Sustainable Development Goals in urban development is a complex challenge because of the interconnectedness between the different goals as well as the diverse perspectives and prioritisation of values among stakeholders. To navigate through this complexity and effectively orchestrate changes at the scales needed, innovation programmes have to be co-produced by and with stakeholders. Effective co-production requires mindful leadership and skilful facilitation. We introduce a learning framework which helps innovators and change programme leaders to facilitate an integrated approach to stakeholder engagement, so they can lead their project effectively and stimulate desired, community-led transformation. We examine the conceptual underpinnings of the framework, drawing from evidence in the literature about motivation and decision theories, co-production, learning, and resilient agency. We then describe how this framework has been developed in a case study of a citizen- led housing project in Knowle West, Bristol, UK and how the solution developed responds to the urban challenge themes of the Urban ID project: mobility & accessibility, health & happiness, equality & inclusion, carbon-neutral city, and environment & ecology. We highlight the importance of framing co- production as a learning journey and the need for the effective facilitation of both the behavioural processes of getting things done and the learning facilitation processes which enable people to change their minds and behaviours. These lead to the argument that mindful leadership in both thought and action and the development of a collaborative learning infrastructure are important success factors for co- production in Living Labs.

Keywords : citizen-led housing, urban diagnostics, learning infrastructure, stakeholder engagement


IntroductionCreating a flourishing city and delivering the Sustainable Development Goals (SDGs) in urban development is

Creating a flourishing city and delivering the Sustainable Development Goals (SDGs) in urban development is a complex challenge which requires a holistic systems approach to diagnose problems, innovate solutions and sustain community directed change. This complexity is rooted in the inter-connectedness between the different goals as well as the diverse perspectives, disciplines, domains and prioritisation of values among stakeholders. To navigate through this complexity and effectively orchestrate changes at the scales needed, the change programme has to be co- produced by and with stakeholders so that there is a sustainable commitment from the people and groups who will be responsible for delivering it.

There is an established consensus manifested in the SDGs regarding what needs to be achieved. The pressing question is how we, as individuals, teams and organisations from all sectors and domains, can deliver these goals in a manner which is itself sustainable. From the authors’ experience in working with stakeholders in the urban areas of Bristol and South Gloucestershire, we knew that many people and organisations are passionate and committed about leading positive change and have organically developed valuable assets, know-how and networked communities. We therefore aimed to understand whether, and what, barriers might exist that block this positive, organic energy for change from developing into authentic outcomes. This was the challenge of the UrbanID research project, which aimed to develop a framework to help stakeholders to take an integrated view on the issues or places of their concern and to surface, and then address, any systemic barriers.

In this paper, we first attend to the conceptual understanding that underpinned the development of the UrbanID framework, then turn the focus to the challenge and barriers encountered in community engagement, using the case study of a citizen- led housing project in Knowle West. We conclude by a reflexive analysis of lessons learned in the co-production process.

Urban Integrated Diagnostics and Stakeholder Engagementanalysis of lessons learned in the co-production process. The UrbanID project was one of the five

The UrbanID project was one of the five Urban Living Partnership (ULP) pilots funded by Research Councils UK and Innovate UK to explore the transdisciplinary research challenges of future urban living. UrbanID aimed to create a novel Integrated Diagnostics framework and methods to address these interconnected urban challenges.


Whilst there was a serious commitment from stakeholders, the UrbanID project also recognised that there was a collective ‘business as usual’ mindset amongst a broader set of stakeholders in the city, which reinforced existing siloed practices and made it difficult to engage thematically across different domains, disciplines, and commercial and political interests. The ability to engage stakeholders in the development of a more user-centred and joined-up view was seen as a critical success factor for the delivery of the Sustainable Development Goals across the city.

The UrbanID framework was built upon three methodological components: systems thinking, learning design, and co-production methods. The project design was co- produced with a high degree of multi-stakeholder participation from the quadruple helix of public, private, voluntary and academic sector organisations across the City of Bristol and the South Gloucestershire local authority areas 2 . To stimulate a holistic, user-led approach, five challenge themes were identified by the group:

Mobility & accessibility;

Health & happiness;

Equality & inclusion;

Carbon neutral city;

Environment & ecology.

The UrbanID framework, as an analytical tool, focused on the challenge themes through the lens of authentic, user-led case studies, exploring how the challenges manifest themselves in the life- narratives of citizens, business, government, third sector, and academic actors and impact on wider ecological systems. The framework was developed and tested through five case studies, including two transportation infrastructure projects, two local neighbourhood areas projects, and a leadership organisation focused on environmental sustainability.

organisation focused on environmental sustainability. Learning Together to Co-produce The emergence of

Learning Together to Co-produce

The emergence of co-production as a public service planning and delivery approach was primarily driven by the need for fiscal austerity. However, co-production has since been recognised as a more effective way to support citizens to achieve their wellbeing (Meijer, 2016; Nabatchi Tina, Sancino Alessandro, & Sicilia Mariafrancesca, 2017). Advocates of co-production believe that all stakeholders will participate and contribute from their particular viewpoints, drawing on their unique capabilities, if they can envision some value that they might gain from doing so. This belief is based on

2 For more details about this project visit


the assumptions that stakeholders can align around a shared goal, are motivated to work together and are open and ready to engage with relevant diverse perspectives, data, information and practices and to struggle together to find workable solutions. These assumptions call for a learning design approach because what is required is a way of facilitating or ‘scaffolding’ a process of learning together to solve complex problems when the target outcome is not known in advance (Bauman, 2001).

This process has at least six collective core tasks. Stakeholders need to:

i) Align sufficiently around a shared purpose;

ii) Analyse the problem and any ‘blockers’ to achieving that purpose;

iii) Decide how to go about a change initiative that will achieve the purpose;

iv) Identify what a good outcome would look like and how it might be measured;

v) Select, collect, curate and re-structure relevant data and information into a new solution which achieves the purpose;

vi) Deliver and evaluate the outcome.

These are key tasks of collaborative learning understood as a process, or a journey, which operates in authentic contexts, rather than in the ‘training room’ or the ‘lecture theatre.’ Co-production refers to the involvement of all stakeholders in the process:

the learning design principles provide a ‘context-neutral’ way of moving beyond the Why and addressing the How of going about achieving a collaborative purpose.

the How of going about achieving a collaborative purpose. Understanding Motivation and Engagement Given that

Understanding Motivation and Engagement

Given that collaborative learning takes time and effort and, by definition, entails a deviation from business-as-usual, it depends significantly on stakeholders’ motivation and willingness to engage in what is likely to be a disruptive process. A study of motivation to learn in a performance- oriented culture shows that motivation is influenced significantly by external factors in the immediate environment and the cultural context of the person who is learning. Furthermore, people’s sense of agency, choice, identity and belief structures are four key mechanisms that drive deep engagement and achievement (Harlen & Deakin Crick, 2003; Sebba et al., 2008).

More general research into human motivation has linked people’s behaviour to their (unmet) needs, which are understood to have a hierarchy (Maslow, 1943), as well as being wide ranging, from subsistence and protection to creation, identity and freedom (Max-Neef, Elizalde, & Hopenhayn, 1991). While the satisfaction of needs explains the motivation underpinning people’s actions, how those needs are satisfied and the extent to which those needs have been satisfied are perceived differently from person to person. These perceptions inform a person’s construction of their goals – their


internal representations of pathways leading towards a desired state (Austin & Vancouver, 1996). Goal constructs will recursively create a lens through which a person understands the world and the values they live by. This self-reinforcing process leads to the formation of a person’s worldview over time and will have already shaped a person’s identity, relationships and expectations in his or her social network by the time they encounter a community problem which they want to address collaboratively.

Engagement decisions in authentic contextsproblem which they want to address collaboratively. Despite need satisfaction being a fundamental motivator,

Despite need satisfaction being a fundamental motivator, people do not generally make direct, or rational, choices about which need they are to satisfy next because needs are complex and interdependent and sometimes outside a person’s awareness. The choice is made at a more visceral and practical level that concerns how a person deploys effort, time or resource. It is the consequences of a person’s actions that will lead to the satisfaction or deprivation of needs, which means the behavioural choices are connected to needs in a many-to-many manner (Ballard, Yeo, Loft, Vancouver, & Neal, 2016).

In authentic, real-life contexts, as opposed to experimental settings or training rooms, people usually make decisions about their next best action amongst multiple options, each of which has consequences which relate to multiple needs. People do not have sufficient mental capacity to assess all the consequences of all possible actions systematically; they consider the consequences sequentially, one after another (Ballard et al., 2016), and do not always make deliberate choices but depend on intuition and routines (Betsch & Haberstroh, 2005; Marchiori, Adriaanse, & Ridder, 2017). Therefore, the attention a person gives to the consequence of a choice accounts for a significant part of a decision process, while the preferences underpinning a decision are rooted in agency, goal or belief constructs, and needs.

A layered framework of motivationare rooted in agency, goal or belief constructs, and needs. These motivation related factors and mechanisms

These motivation related factors and mechanisms can be grouped into at least four layers in an inside-outside model of motivation, in that the inside layers are more strongly driven by internal processes than external ones whilst the outer layers are more open to external influence factors (see Figure 1).


Figure 1. Layers of sources of influence on participant's motivation to engage in Living Lab

Figure 1. Layers of sources of influence on participant's motivation to engage in Living Lab activities

Need Satisfaction represents an underpinning value system, which is largely a result

of the co-evolution of the human species. Fundamental human needs are qualitatively

stable and broadly similar across populations, but need satisfaction is subject to cultural influences and will be assessed subjectively with reference to the extent and ways in which needs are satisfied in a specific time and place (Deci & Ryan, 2000). Most human moral judgements have also been incorporated into our basic psychology (Greene, 2014) and therefore it is reasonable to consider “value” as an

integrated part of the complex system of human needs (Pink, 2011).

Goal Construct is a person’s internal cognitive, affective and volitional model of how the world works and thus how a person can satisfy his or her needs. Hence, goal constructs imply at least two sets of knowledge relating to needs: i) declarative knowledge which characterises what is needed and the options available, and ii) procedural knowledge about the set of actions that constitute an option. Even though people have similar set of needs, they assign differing weights of importance to those needs and have differing understandings of the options for achieving satisfaction.

Agency is the capability of a person to utilise the structure of a system to achieve his

or her goal (Giddens, 1984). As far as motivation is concerned, it is the person’s belief

in whether he or she has agency that influences the person’s decision to engage with

a goal. There are two agency-related beliefs: self-belief that the person can act him

or herself, and belief and trust in the agents the person needs to work with in order to act and achieve a goal. Even though a person may over or under estimate his or her


actual agency, their agentic belief will strengthen their actual agency and the increase of actual agency will strengthen their belief.

Framing is the interpretation of the relationships the person has with other people, the material environment and their social context, which informs the person’s attention and temporal importance weighting of goals (Ballard et al., 2016). To give an example, Steele (2011) describes how female students feel the pressure to prove their abilities in solving difficult maths problems because there is a stereotype cultural belief that female students are less able in maths than their male peers. Alternative task framing can remove the identity threat and nudge the students to attend to the goal of, say, stretching their ability in solving difficult math problems, instead of proving wrong the stereotype about their maths ability (Steele, 2011).

the stereotype about their maths ability (Steele, 2011). The learning framework of UrbanID Framing co-production

The learning framework of UrbanID

Framing co-production activities as learning opportunities is of critical importance to overcome the siloed and fragmented thinking and practices embedded in stakeholders’ business- as-usual modes of operation. This is because doing something differently, by definition, involves learning. If we already know what to do, we don’t need to learn and ‘business-as-usual-practices’ will be the default modus operandi. However, innovation and change require moving into the unknown and this involves learning. To approach new learning opportunities (i.e. change) productively requires learning power, which is ‘the relational and embodied process through which we regulate the flow of energy and information over time’ in pursuit of a purpose (Deakin Crick, Huang, Ahmed Shafi, & Goldspink, 2015, p. 121). Eight dimensions of learning power emerged from five studies with more than 50,000 cases: Mindful Agency; Hope and Optimism, Sense-Making, Creativity, Curiosity, Belonging, Collaboration and Openness to Learning. These are malleable qualities which emerge from the interactions between the learner and their environment (ibid.). Learning power is influenced by relationships and social culture – the data suggests that where a culture is characterised by trust, affirmation and challenge then people are more likely to move into new ways of thinking and acting, mobilising their learning power in pursuit of their goal (Deakin Crick & Joldersma, 2007; Deakin Crick, McCombs, Haddon, Broadfoot, & Tew, 2007). The implication of this for a Living Lab is that when activities are designed in a way that enable people to utilise and develop their learning power e.g. creativity, curiosity, mindful agency, sustainable learning relationships etc, it will not only help people learn better together but it also incentivises future participation and goal achievement.

The layered framework of motivation clarifies the depth of influence required for


significant behavioural change at scale within a community and highlights the importance of learning design in stakeholders’ engagement in change and innovation. Substantive change occurs when a person’s understanding of their needs and their worldview is modified, and when they have formed sufficient agency to pursue a new goal; and a person’s agency and goal constructs change only through learning.

This layered framework of motivation, the tasks of collaborative learning, and holistic systems thinking constitute the learning framework of UrbanID. In the following sections we describe how the framework was developed and applied through a particular UrbanID case study focused on citizen-led housing development.

We Can Make: The Citizen-Led Housing Projectcase study focused on citizen-led housing development. Knowle West Media Centre (KWMC) is an arts centre,

Knowle West Media Centre (KWMC) is an arts centre, founded in 1996 and rooted in the community of Knowle West, Bristol, UK. KWMC works across different disciplines using art and technology to address issues including health, housing, and smart cities. KWMC runs Bristol Living Lab and as such fosters civic innovation by collaborating with residents, artists, local authorities, business and academia to support people to make positive change and explore new ways of living better together.

The We Can Make project is a response to community demands and concerns about housing need in the area. The project team carried out extensive participative research in Knowle West, a housing estate on the outskirts of Bristol consisting of 5,500 households and 12,000 residents. Knowle West is a low-density estate – below 25 dwellings per hectare which is below the threshold to justify a regular bus service. There are few sources of local employment, public transport is poor, and the neighbourhood remains geographically isolated. The low population density makes it difficult to sustain shops and services, with many closing down or moving elsewhere, including the cinema, and swimming pool, and presently the library which is under threat of closure.

Nevertheless, the neighbourhood has a high level of housing need: 522 people in the Knowle West area are on HomeChoice, the Council’s official housing waiting list. Knowle West represents 2.9% of the total population of Bristol, and 5.8% of people registered on HomeChoice are from the area – double its proportionate size.

Co-identification of the needare from the area – double its proportionate size. Over a year, in 2017, We Can

Over a year, in 2017, We Can Make brought together local people, artists, architects,


policy- makers, academics and industry professionals to collaborate between and beyond traditional professional and social silos. The aim was to understand the needs of people from their perspective, identify the nature of the problem, collate and map the assets and resources the community had available, and co-construct new citizen- led approaches to enable people to supply their own affordable housing at the point of need. This process maps on to the six learning design principles described above. The diagnosis of the issue of housing need was developed through a door-to- door- survey, narrative interviews, and semi-structured conversations at workshops and on doorsteps. Four types of unmet housing need were identified:

Down Shifters: individuals or couples who want to stay in their neighbourhood but whose house is now too big for them and are looking for a smaller home. This could be because children have grown up and flown the nest, changing mobility needs, or a need to reduce costs in retirement.

New Shoots: families where there is an urgent need for more space because children are growing up and seeking independence or are having offspring of their own, or caring responsibilities are expanding to include additional elderly or disabled family members.

Better Fits: families where one or more member(s) is experiencing changing mobility and the family home needs to be adapted to enable them to stay close to their support networks.

Making Ends Meet: individuals and families that need extra income or reducing their housing costs due to changing circumstances.

reducing their housing costs due to changing circumstances. Co-diagnosis of barriers The project team researched the

Co-diagnosis of barriers

The project team researched the assets and resources people had in the neighbourhood and found there was no shortage of change-ready attitude, know-how in construction and trade, spare capacity and land ( Figure 2). The project also identified the barriers to citizens and their communities in applying some of these assets and resources to how people could go about supplying and securing their affordable housing at the point of need:

Conventional strategies to access housing are reductively competitive. They either require people to divert ever more of their wages and savings to getting on a property ladder, where the bottom rungs are missing, or compel people to prove how ‘weak’ or ‘incapable’ they are in order to win eligibility for austerity rationed social housing. These dominant strategies can cause further damages, such as diverting capital to be locked in housing stock rather than other forms of investment that can circulate in the economy more productively, and


undermining an individual’s sense of personal agency.

An emerging alternative is the “third sector” – the civic or community sector. However, despite enabling legislation at national level and new forms of low- interest finance, the take-up so far has been slow – self-build and custom build account for just 7% of new homes delivered each year in the UK, compared to 80% in Austria and 60% in France.

A significant barrier is caused by the deliberately veiled nature of the development process. Access to land, technical know-how, and financial resources continue to be controlled by professional interests and has disproportionate costs and barriers to entry; the current housing system is more often working against than supporting the civic or community approach. For example:

i) Land: most citizen-led schemes range from 1-10 homes, a size that is just too small and carries disproportionate transaction costs for most council site allocation and land disposal processes to bother with. At the same time the long-term social and economic impact of citizen-led housing struggles to make its added value count in a highest-sale- price-wins market. The result is that communities are often left with the most difficult sites, which commercial developers have rejected or are overly dependent on increasingly stretched local authorities as a route to get land at a discounted price.

ii) Finance: at each stage of the development process – from a lack of pre-planning development finance, the high cost of social investment, to a lack of capital reserves to accommodate construction cost overruns – risk and uncertainty accumulates. The high transaction costs of sticking together small pots of multiple funds each with different strings attached also creates substantial drag on community- led projects.

iii) Technical know-how: from planning permission to finding contractors, the process of making homes is dominated by costly professionals. The inequity this creates is clear – in Knowle West, planning applications are twice as likely to be rejected compared to wealthier neighbourhoods in Bristol.

iv) No repeats: Most community-led projects remain non-replicable and


the experience gained – often through years of ‘hard graft’ – by individual residents stays stuck to specific sites with few people wishing to repeat the experience. The result is high per- project overhead costs in terms of both time and money as each new amateur has to learn and master the process from scratch. The isolation of individual projects also means the citizen-led asset base – which might be able to fund the next project – fails to grow.

The systemized opaqueness and uncertainty mean many individuals and communities do not know what tools they need, cannot afford them anyway, and ultimately, do not have the faith that they can achieve the outcome they want – a safe, secure and affordable home. Using the terms of the layered framework of motivation, community-led housing was not an option in people’s existing goal constructs nor did they have the agency required to act upon this option. These findings informed the project’s next key task which was to co-design the collective tools that could help overcome these barriers.

collective tools that could help overcome these barriers. Figure 2. Assets the Knowle West people and

Figure 2. Assets the Knowle West people and community had to utilise as captured by the WCM project.

and community had to utilise as captured by the WCM project. Learning together to create an

Learning together to create an alternative approach

To lead the community through the transformative journey from a business-as-usual mindset to an innovative new approach, We Can Make utilised a mix of open research


and development tools and approaches to stimulate learning:

Co-design of workshops which mixed residents, artists, and housing professionals to explore issues and ideas together in an open, learner- centred and non-hierarchical way;

The involvement of artists to help open up learning conversations, invite new perspectives, translate between different communities and types of participant and create shared learning experiences, spaces, and new knowledge;

Using visualisations to unravel and de-mystify complex issues and ideas and thus empower learning;

Testing and re-testing emerging ideas and propositions with a wide range of stakeholders – including residents – learning and evaluating together as the project developed.

learning and evaluating together as the project developed. Figure 3. An illustrative graphic which helps citizens

Figure 3. An illustrative graphic which helps citizens understand the community-led housing process from the perspective of a user whose need is Better Fits, which is one of the four types of needs identified in We Can Make.

Overall, We Can Make developed a different kind of user-led learning journey for people to find and realise ways to access affordable homes, attuned to their needs and resources, and created collective tools to support the individual journeys as a


process of learning. These processes were facilitated and captured using creative artefacts. For example, the project represented these individual learning journeys through illustrative graphics which help citizens understand the community-led housing process from the user perspective (Figure 3).

To make the community-led housing option an authentic experience and thus deepen the community learning, We Can Make then built a prototype home to pilot the development process from end to end, demonstrating with the community how some

of the barriers could be overcome in practice. The pilot process involved the following


Identifying a micro-site next to a community centre;

Development of community criteria for construction and design including local materials, local labour, sustainability, and affordability;

Use of existing planning rules to permit the development;

Employing local construction teams;

Artists and local people designing and making fittings and furniture;

A rapid prototype home built over 3 months;

Local people experiencing and testing the home by staying in it.

The prototype home created an embodied space where people could see, feel and help make it happen, learning through experience. This is of critical importance in helping people incorporate this innovative approach of housing into their existing goal constructs; it demonstrated that a different way of imagining, making and accessing an affordable home is possible, as opposed to just a concept or topic for consultation. The community learning was deep and transformative through ‘playing, making and doing’ (Thomas & Brown, 2011).

‘playing, making and doing’ (Thomas & Brown, 2011). Co-generation of community agency A co-design approach was

Co-generation of community agency

A co-design approach was used throughout the research project with residents

working alongside professionals to develop and test ideas, which inspired the community to develop their own ‘agency’ and become the developer. The prototype home has now had over 100 people staying in it, including locals and people from as far away as Berlin, San Francisco, and Melbourne. Local people stay for free to test out living in the home, and non-locals pay through AirBnB. The income is shared with a community centre who provided the micro-site, and in four months has contributed 10% of the centre’s annual income.

KWMC also seeks funding and support to increase the resourcefulness of the


community and generate the ‘community agency.’ The We Can Make project has now been identified as a priority pathfinder project by Bristol City Council. The project has now secured support from Power to Change and the Nationwide Foundation to develop the collective tools, including a land assembly mechanism, financial tools, community design code, community supplier’s framework, and digital learning platform, to support the development and delivery of We Can Make in the pilot neighbourhood during 2018 and 2019.

We Can Make in the pilot neighbourhood during 2018 and 2019. Discussion We Can Make ’s


We Can Make’s achievements have arisen because sufficient people became motivated to work together to achieve a mutually beneficial common purpose. Such motivation is not a given; it is an outcome of a learning process that each individual participant and organisation goes through. The success of We Can Make is the result of the creative, purposeful and engaging approach to orchestrate the stakeholders’ collective learning journey, not simply following the script of the UrbanID Learning Framework presented in this paper. Nevertheless, the framework provides a useful working theory and set of design principles which explain the stakeholder engagement demonstrated in the We Can Make project and the quality of its outcomes.

First, establishment of a shared set of learning design principles helped to direct stakeholders’ attention to the intended shared goal. Framing the project’s co- production process – from identifying the problem, diagnosing the ‘blockers’, co- constructing and achieving an alternative goal – created a social environment that generated collaborative learning power, motivation and agency and therefore led to deep stakeholder engagement. Using the challenge themes as an analytical lens also prompted stakeholders to think systemically, leading to the development of a solution that is expected to have positive impact on most of the challenges themes. These links are presented in Table 1 below. Second, there were three processes running in parallel throughout the We Can Make project:

i) The identification of the problem and the ‘job to be done’;

ii) The behavioural processes of co-production, including but not limited to those key tasks of collaborative learning discussed in section 3, which signpost the pathway forwards for stakeholders to navigate and so identify, collect and curate the necessary resource, data and information which will enable them ‘co-construct’ a solution and thus achieve their purpose – or ‘do the job’;


iii) The intra- and inter-personal learning processes, as discussed in the layered framework of motivation: a meta-level, internal (and often invisible) element of co-production which is necessary to sustain engagement and lasting changes.

The behavioural processes and the learning processes, as we observed in the case study, were coupled like a double helix around the ‘job to be done’. When both processes are facilitated and integrated appropriately around the ‘job to be done’ they are mutually reinforcing. Conversely, if either of the two processes is neglected or if they are managed separately as in isolated silos, it will limit the transformative potential of co-production. The job may still get done, but it will be less sustainable because the learning and knowledge co-production processes will not be embedded in the hearts and minds of the community and its stakeholders.


Table 1. The expected impact of We Can Make approach on the UrbanID examplar challenge theses

Challenge theme


Increase of local

Mobilisation of

housing capability






Mobility &

More variations of house designs to suit the needs of people with varied mobility and accessibility

Higher density makes it more cost and energy effective to run infrastructure services, including public transport, and other locally facilities, providing easier and less discriminated

access to essential


Health &

User-centred development means the new houses better meet the quality required for a health and happiness

WCM Financial Tools and Community Supplier Framework offers job and investment opportunity for local people to generate and retain value against the potential raise of house price as a result of development


Equality &



services for well- being.

Carbon neutral

The prototype home was built to high environmental standards (it is built of compressed straw); Developing carbon saving techniques and methods is part of the community learning process and will be embedded into the WCM assets


Environment &

Better use of land in existing residence reduces the development pressure on wild


Enhances collective and individual agency and relatedness

Enabling much more diverse ways to contribute to and benefit from urban living than an exclusive mode of job earning and money spending

Increased resourcefulness means greater ability and willingness to adopt latest technologies which in general are more environment- friendly

green space


The third reflection is about the role that digital and visual artefacts and assets played in the We Can Make project. In a related community engagement study with Hunter Water Corporation 3 in NSW, Australia, the components of a digital ‘always on’ community engagement platform have been developed based on the learning framework identified in this study. In its early stages, based on a data architecture which offers a ‘single view of the user’ it maps out the key tasks of the behavioural processes of co-production – or learning with the community - framing these around the ‘job to be done’, whilst identifying key trigger points for critical learning processes. The ‘job to be done’ in this example is to secure a resilient water future for the community – the content-neutral learning framework discussed here provides the ‘scaffolding’ (see Figure 4).

5. How Create Establish Create Reflect and Take Action Enjoy benefit Awareness Purpose learn Purposeful
5. How
Reflect and
Take Action
Enjoy benefit
Purposeful ‘Learning With‘ Journeys

© Decisioning Blueprints Ltd 2017

Figure 4. The community engagement platform architecture developed by Learning Emergence Lab 4

The importance of assets has been well understood by KWMC who have sought support and investment to advance their development of assets as part of the learning infrastructure of the community. The notion that this learning infrastructure has to include both learning facilitation services and physical and digital assets is consistent with the findings from a study of innovative business model for infrastructure (Crick, Huang, Godfrey, Taylor, & Carhart, 2017).

While it is widely accepted that co-production is incompatible with managerial control


4 Learning Emergence Lab is the innovation engine of Learning Emergence Partners, a global collaborative of practitioners, data architects, business consultants and academics and a Community Interest Company. It is based in Scotland, UK.


exercised through a ‘top-down’ power structure (Meijer, 2016), it takes more than a simple reverse of power into a ‘bottom-up’ power structure to orchestrate effective co- production. What we have highlighted in this case study was the Living Lab as an organisation committed to the provision of a community service, which introduced mindful leadership in thought, action and relationships to energise and scaffold a co- production process which led to the emergence of a new social and industrial structure. This was the ‘job’ they set out to do: enabling the community to solve its housing problems. They did this through the intentional and visible use of (i) learning design principles in their mode of communications and relationships and (ii) the deployment of invited behavioural processes – key tasks that structured the journey from purpose to performance.

Conclusionsthat structured the journey from purpose to performance. In the paper we presented the Learning Framework

In the paper we presented the Learning Framework of the UrbanID project which was developed through participatory, user-led research. Through the case study of citizen- led housing, we illustrated the importance of framing and scaffolding co-production as a learning journey. We argued for the value of using learning journey design principles to facilitate the ‘job to be done’ through the interconnected behavioural processes and learning processes which empower stakeholder engagement in co- production. An additional insight emerging from our analysis is the critical role digital and visual artefacts and assets play in harnessing learning across the community over time. These insights led us to recognise that mindful leadership in thoughts and actions – leading to the provision of a community focused learning infrastructure - are important success factors for co-production in Living Labs.

Acknowledgementsimportant success factors for co-production in Living Labs. The Authors gratefully acknowledge the financial support of

The Authors gratefully acknowledge the financial support of UKRI through the Urban Living Partnership project, UrbanID, grant reference EP/P002137/1. The UrbanID project involved a large team of collaborators who worked on other interconnected case studies. The Authors gratefully acknowledge the contributions of their project colleagues to the background discussions that informed the learning framework development described herein.



Austin, J. T., & Vancouver, J. B. (1996). Goal constructs in psychology: Structure, process, and content. Psychological Bulletin, 120(3), 338. Ballard, T., Yeo, G., Loft, S., Vancouver, J. B., & Neal, A. (2016). An integrative formal model of motivation and decision making: The MGPM*. Journal of Applied Psychology, 101(9), 1240–1265.

Bauman, Z. (2001). The individualized society. Cambridge: Polity Press.

Betsch, T., & Haberstroh, S. (Eds.). (2005). The Routines of Decision Making (1 edition). Mahwah, N.J: Psychology Press. Crick, R., Huang, S., Godfrey, P., Taylor, C., & Carhart, N. (2017). Learning Journeys and Infrastructure Services: a game changer for effectiveness (advanced copy). In T. Dolan & B. Collins (Eds.), ICIF White Paper Collection (in Press). London: UCL Press. Deakin Crick, R., Haigney, D., Huang, S., Coburn, T., & Goldspink, C. (2013). Learning power in the workplace: the effective lifelong learning inventory and its reliability and validity and implications for learning and development. The International Journal of Human Resource Management, 24(11), 2255–2272.

Deakin Crick, R., Huang, S., Ahmed Shafi, A., & Goldspink, C. (2015). Developing Resilient Agency in Learning: the internal structure of learning power. British Journal of Educational Studies, 63(2), 121–160.

Deakin Crick, R., & Joldersma, C. W. (2007). Habermas, lifelong learning and citizenship education. Studies in Philosophy and Education, 26(2), 77–95.

Deakin Crick, R., McCombs, B., Haddon, A., Broadfoot, P., & Tew, M. (2007). The Ecology of Learning: Factors Contributing to Learner Centred Classroom Cultures. Research Papers in Education, 22(3), 267–307.

Deakin-Crick, R., Sebba, J., Harlen, W., Yu, G., & Lawson, H. (2005). Systematic review of research evidence of the impact on students of self-and peer assessment. EPPI Centre Social Science Research Unit Institute of Education University of London, London Google Scholar. Deci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268.

Giddens, A. (1984). The constitution of society: outline of the theory of structuration. Cambridge: Polity Press.


Greene, J. (2014). Moral tribes: Emotion, reason, and the gap between us and them. Penguin. Harlen, W., & Deakin Crick, R. (2003). Testing and motivation for learning. Assessment in Education: Principles, Policy & Practice, 10(2), 169–207. Marchiori, D. R., Adriaanse, M. A., & Ridder, D. T. D. D. (2017). Unresolved questions in nudging research: Putting the psychology back in nudging. Social and Personality Psychology Compass, 11(1), e12297.

Maslow, A. H. (1943). A theory of human motivation. Psychological Review, 50(4), 370–396.

Max-Neef, M. A., Elizalde, A., & Hopenhayn, M. (1991). Human scale development: conception, application and further reflections. New York: The Apex Press.

Meijer, A. (2016). Coproduction as a structural transformation of the public sector. International Journal of Public Sector Management, 29(6), 596–611. 01-2016-0001 Nabatchi, T., Sancino A., & Sicilia M. (2017). Varieties of Participation in Public Services:The Who, When, and What of Coproduction. Public Administration Review, 77(5), 766– 776.

Pink, D. H. (2011). Drive: The surprising truth about what motivates us. Penguin.

Steele, C. M. (2011). Whistling Vivaldi: how stereotypes affect us and what we can do. New York: WWNorton & Company.

Thomas, D., & Brown, J. S. (2011). A new culture of learning: Cultivating the imagination for a world of constant change (Vol. 219). CreateSpace Lexington, KY. Thomas, D., & Seely-Brown, J. (2009). Learning for a World of Constant Change:

Homo Sapiens, Homo Faber & Homo Ludens revisited. Presented at the 7 th Glion Colloquium. Retrieved from for a World of Constant Change.pdf


Participatory software development via Living Labs:

A case of Deajeon

Hee-Sook. Yoo 1 , Su-jin. Jung 2 , Seong-gyu. Park 3 , Miran. Cho 4

1,2 National IT Industry Promotion Agency (NIPA), the Republic of Korea 3,4 Deajeon Information & Culture Industry Promotion Agency (DICA)

Category: Innovation Paper

Industry Promotion Agency (DICA) Category: Innovation Paper Abstract Over the past few years in Korea (Republic

Abstract Over the past few years in Korea (Republic of Korea), demands for welfare policies to prepare for an aging society and to secure a social security have been grown. Responding to the growing needs of people, Korea Government is running the research and development (R&D) projects using Information Technologies (eg. Software, IoT, AI, big data etc.) to make public services which resolve the social issues efficiently. The Living labs is considered as one of research and development methods, which aims to close the gap between technical and social innovation by enhancing collaboration between different relevant stakeholders (Kopp, Haider, & Preinesberger, 2017, p.61). Hence, NIPA (National IT Industry Promotion Agency) has tried applying the Living Labs on software development projects to make software solutions for local community. Compared to the conventional research and development projects, Living Labs has a slower pace spending more time and efforts on communication with relevant stakeholders (issue exploration, demand definition etc.), but it’s more effective way to reflect concrete demands of users on research and development projects.

Keywords: Living Labs, software development, co-creation, software solution, stakeholders


Introduction The Ministry of Science and ICT (referred to “MSIT”) is responsible for the policies


The Ministry of Science and ICT (referred to “MSIT”) is responsible for the policies on Science and Information communication technology research and development in Korea (The Republic of Korea). National IT Industry Promotion Agency (referred to “NIPA”) is a government affiliated organization that makes action plans for the ICT Research and Development Projects and supports organizations who conduct ICT research and development projects - such as research institutions, universities, business entities, local government agencies - to put MSIT policies into practice.

Recently, NIPA has initiated the research and development projects using Living Lab methodology, which use software technology to resolve issues for a local community. The projects are in progress as a pilot. Based upon the result of it, NIPA is planning to expand the adoptions of Living Lab methodology to the software research and development project. In every project using Living Labs, the participations of different stakeholders are crucial factors. Among the projects NIPA supports, with the case of Deajeon, this paper explains how to apply Living Lab methodology to software research and development project, and how to draw the stakeholders’ collaboration on Living Lab process.

R&D toward Technological Advancement- Versus Social Innovation

Since 1990s, Korea Government has invested huge amount of funds on Information Technology (eg. software, big data, cloud, IoT, artificial intelligence etc.) research and development. Majority of them led by technology professionals - such as research institutes, universities, corporations etc. - , and aim to produce economic value by developing advanced technologies dealing with industrial issues.

In the 2000s, people have been concerned about social welfares for an aging society and a daily living safety. Responding to these national demands, Korea Government has tried to take advantage of ICT (Information and Communication Technology) to make a better life of its people. Government driven projects to develop public services using ICT were carried out in a wide range of areas, such as healthcare, education, emergency management, food sanitation etc. However, the outcomes of them had difficulties in applying to people’s real life and commercialization. The conventional research and development projects are led by technology professionals, such as research institute, universities, corporations etc. In such way, embracing the opinions of ordinary people to the development project was limited. The majority of public service end-users is ordinary people. Many of them are not familiar with cutting-edge technologies, and reluctant to try new things required much learnings how to use. First and most important thing to ordinary people is not whether up-to-date


technologies are applied in it, but whether the services (the result of development project) could resolve their inconveniences or not.

Eriksson, Niitamo, and Kulkki (2005, p.6) and Lepik, Krigul, and Terk (2010, p.1091) consider the Living Labs as a research and development methodology, which aims to close the gap between technical and social innovation and to enhance collaboration between different relevant stakeholders. In this regard, NIPA went into applying Living Labs to software service development project to resolve social issues of local governments. Considering that Living Labs was not familiar to employee of local government agencies, NIPA provided a three-month “Design Thinking” and Living Labs methodology training program to local government agencies’ employee beforehand, in 2016. After the training program, NIPA encouraged those trainees to become Living Lab facilitators and to do applying Living Labs to their software research and development projects. As a trial basis, Living Labs has been implemented to a part of their projects.

Deajeon City Case: The Household Wastes Treatment in Old Town

Deajeon is a distinctive area that national research and development institutes are clustered. People of Deajion have a positive attitude to adopt technologies relatively. Deajeon City wanted to resolve the inconvenience of citizens efficiently using software technologies. Deajeon Information & Culture Industry Promotion Agency (reffered to as “DICA”) is a local government affiliated organization that executes the ICT Research and Development policies for Deajeon City. Since 2016, NIPA and DICA have conducted “Soft Town” Program which aims to enhance Deajeon citizens living by applying software technologies. NIPA and DICA agreed that the conventional way of research and development was not fit to deal with social issues, and not suitable for embracing opinions of people sufficiently. In that regard, they decided to adopt Living Labs to a part of their software service development project in “Soft Town” Program from 2017.

Software Service Development via Living Labs

To gather the opinions of citizens and made an effective solution to citizen, NIPA and DICA would like to make process software development project along with local people with a systematic research and development framework. iLab.o’s methodology is based on the social construction of technology framework, which suggests that technology is shaped by the user (in this case, “deajeon citizens”) and highlights the importance of context in the process of endowing technologies with social meanings (Almirall, Lee, and Wareham, 2012, p.13-14). Users are considered the central focus and facts and meanings are the results of social process (Almirall


et al., 2012, p.13-14). The methodology consists of four phases aimed at understanding the context where the technology will be adopted and emphasizing the changes in meanings that this adoption will produce.

The object of the project is drawing the positive changes to Deajeon citizens with software technologies. Going through the Living Labs process with Deajeon people, NIPA and DICA wanted to understand what kind of problem they really have, and draw the necessary and effective software solution. In this sense, we decided to apply adapted version of iLab.o’s methodology to the Deajeon Living Lab projects.

iLab.o’s methodology to the Deajeon Living Lab projects. Figure 1. Adapted iLab.o’s methodology Contextualization

Figure 1. Adapted iLab.o’s methodology

Lab projects. Figure 1. Adapted iLab.o’s methodology Contextualization The contextualization phase aimed to get


The contextualization phase aimed to get the issue relevant information. The information was used to organize a group of stakeholders for the project process (Almirall et al., 2012, p.13-14). To collect the information sufficiently, in this phase, we conducted Documentary Survey, Deajeon City officials Interview, Issue- relevant-spot investigation and Fact-check, 1st Professional Group Discussion and Citizen Survey, 2nd Professional Group Discussion and Panel Group Define. In this process, we focused on having a good grasp what are the real issues deajeon citizens have and who are the issue relevant stakeholders.

i) Documentary Survey

To grasp the need of Deajeon Citizen preliminary, survey reports, statistical data, and


policy materials that were published by Deajeon City within two years were reviewed. As a result of the literature review, sixty two themes of Deajeon City were listed-up. These were categorized into six fields (industry, culture, transportation, community welfare, senior citizens, environmental pollution). Then, DICA found out that the six fields were relevant with twenty two departments at Deajeon City Hall.

ii) Deajeon City official Interview

In accordance with the documentary survey analysis, managers at the twenty two departments were interviewed. During the interview, we inquired what policies they conducted, what kind of inconveniences that citizens asked to them, how much the inconveniences were resolved. Then, we identified the issue of each department and

the stakeholders of such issues.

iii) Issue-relevant-spot investigation and Fact-check

Based upon interviews, people in charge of Living Lab in DICA visited the issue relevant spots, and did interview with the stakeholders at the spots. Conducting on- the-spot investigation, they did cross-check the analysis of documentary survey and official interview with real situation. Through this activity, they checked whether the information from documentary survey and city official interview was true or not.

iv) 1st Professional Group Discussion and Citizen Survey

The professional group was organized with people from various fields (public administration, engineering, economist, business administration, statistics etc.). They

discussed about urgency, feasibility, and influence of the issues gathered from the above a, b, and c phases. Going through five discussions, 4 priority issues - residential environment improvement, public transportation, school violence, senior citizen safety - were defined. And the survey to identify the severity and causes of those issues was designed by the professional group. For two weeks, we conducted a survey of 1,000 Deajeon citizens aged 15 years or older, and analyzed the specific issue condition and inconvenience etc.

v) 2nd Professional Group Discussion and Defining Panel Group

As the last phase of Contextualization, based upon the analysis above mentioned in a, b, c, and d phases, the professional group compared with what kinds of problems happens now (real situation), and how do citizens think what problem they have (perception). Based upon this analysis, considering the severity of issues and the feasibility to resolve, the professional group discussed about the above survey results, and selected residential environment improvement as a top priority issue to be resolved. Then, they identified issue relevant stakeholders, and defined the details of the living Lab panel organization.


Figure 2. Citizen Survey Result Concretisation Almirall et al. (2012, p.13-14) state that the key

Figure 2. Citizen Survey Result

Figure 2. Citizen Survey Result Concretisation Almirall et al. (2012, p.13-14) state that the key element


Almirall et al. (2012, p.13-14) state that the key element in this phase is capturing the issue condition that can be later compared with the changed condition with the introduction of the new technology or the innovation to be validated. In this regard, during the Living Lab process, we tried to define the real issue condition that should be resolved, and draw the positive stakeholders’ participants efficiently. To make specific issue condition to get solved and to define the problem Living Lab solve, DICA organized the panel of Living Lab with local residents, local government associates, NGOs, and professionals from various fields.

At the beginning of DICA Living Lab, when the panels exchanged their opinion and shared their ideas, they provided much unconfirmed and false information. Some wanted to bring personal complaints to the table, and some talked about the issue condition which had already resolved. Caused of the difference of individual view point, sometime small conflicts were aroused between panels.

To reduce inefficiency of co-works and to make them understand what the Living Lab is, DICA - as a facilitator - provided a program that introduced the concept and cases


of Living Labs. In addition, to make the panels more easily interchange each thoughts and opinion to explore the solution, DICA also provided “Design Thinking” - a problem solving methodology that emphasizes user’s participation on research and development - training program to them. Consequently, as the Living Labs processed, the participation of panels was more active, and their communication became more focused on the issue Living Lab deals with.

became more focused on the issue Living Lab deals with. Figure 3. Living Labs & Design

Figure 3. Living Labs & Design Thinking Training

Through these efforts, the top priority problem of Living Labs was defined as ‘The Household Wastes Treatment in Old Town. Household Wastes should be separated to recyclable, non-recyclable, and food waste. In old town, waste was not properly separated, and also not collected in time. These make spoil the residential environment, and cause pollution and safety accident.


Figure 4. The Household Wastes Disposals in Old Town Panels reviewed the materials from contextualization

Figure 4. The Household Wastes Disposals in Old Town

Panels reviewed the materials from contextualization phase, and shared their experiences, thoughts, and opinions. Simultaneously, they visited the most serious spot “Shinsung-dong” to grasp the real specific problem condition. They met 250 residents and interviewed them.

During the visit, professionals on Living Labs, Design Thinking and NGO made residents actively give their opinion. Local government officials made residents notice the relevant information they wanted to get. Technicians considered how technology could be used to solve the problem.

With this visit, panels summarized the causes of the problem that residents pointed out. Many residents did not have enough information how to separate the waste, and they did not know where the waste should be put out. Also, they confused the time to collect the separated waste.

should be put out. Also, they confused the time to collect the separated waste. Figure 5.

Figure 5. Drawing solution process


As a result, panels agreed that the most effective way to resolve the inconvenience in short time was distributing software service application that provide the information – how to separate, where the waste is put out, the waste collection schedule.

DICA has made action plans for Deajeon City ICT research and development policies. They have a lot of experiences making and conducting software research and development project. Drawing co-work from panels, they designed requirement specification for the application that would be able to resolve the problem. A local software company was selected to develop the application, and the prototype development was completed in Dec. 2017.

and the prototype development was completed in Dec. 2017. Figure 5. The requirement specification for the

Figure 5. The requirement specification for the application & the prototype application

for the application & the prototype application Implementation and Feedback The test and validation in real

Implementation and Feedback

The test and validation in real life was carried out in the implementation phase. In the feedback phase, the ex post measurement was conducted. The results were compared with those obtained in the contextualization and implementation phases and used to infer and produce recommendations on the concrete diffusion and implementation of the technology (Almirall et al., 2012, p.13-14).

Deajeon City and DICA selected Shinsung-dong, as a testbed. Since the first quarter of 2018, the application which provide the information – how to separate, where the


waste is put out, the waste collection schedule etc. – has been distributed to Shinsung-dong residents.

As the distribution of the app was done recently, we don’t find rapid changes that the application makes. However, during this phase, the more feedbacks panels gather and have a communication with users (residents), the newer service demand and idea are drawn. As the phase is processed, users’ commitments to Living Lab have been grown. They positively recognize Living Lab as a place where they are able to talk about their problems with local government decision makers, and actively share their experiences and opinions. Residents want that Living Lab deals with other local community issues.

In addition, in the process of implementation, the wastes collectors asked the software application for them. This was discussed on Living Labs, and DICA Living Lab has begun to design requirement specification for the application that senses the condition of piling up and give the information to the collectors. Besides the waste issue, DICA also accept the opinions from stakeholders (panles, residents etc.) that Living Lab should be applied to other issues. They make a plan to build a big data system to draw social issues and define problem for local community more precisely.



As a result of conducting Living Lab project for one year, we identify that the followings are important to enhance stakeholders’ commitments and Living Lab efficiency.

First, the shared concept and objective of Living Labs is the efficient way to give Living Labs motivation to stakeholders. As mentioned above, at the beginning phase, inefficient communication and conflicts among stakeholders existed. We resolved such conditions with Living Labs and Design Thinking training program. With the program, we tried to make understood stakeholders what they do, what kinds of result they can draw, how they make changes to their community with Living Labs. The more clear-cut mission Living Lab facilitators give to stakeholders, the stronger reasons they commitment to Living Lab are formed.

Second, the on-spot investigation is the way of filtering inefficiency for Living Labs. In Contextualization and Concretization phase, we visited issue-relevant-spot to define what condition is real problem by checking whether what we discussed and analyzed with stakeholders. Going through the on-spot investigation, some distorted or polluted


information can be filtered. It is also very helpful to gather enough resources to decide for setting priorities among the several issues. Furthermore, by communicating with people on the spot, many ideas and issues - not be discussed, but should be dealt – can be gathered. In this regard, the on-spot investigation is a good information filtering tool, as well as an ideation method.

Third, data is the key to draw real issues. In the processing contextualization and concretization, DICA Living Lab tried to compare the reality with the perception of people to define real problem. People would sense their condition more or less serious than actual condition. Data, such as statistical data, data from e-government etc., can be the criteria to determine real situation. In this project, we used such data to check out fact condition and to filter distorted information. Defining real problem is the phase of building a Living Lab project objective. The use of data can be helpful to set a direction Living Lab heads to. In recent, big data technology have been rapidly enhanced, and widely used in various fields. Considering this big wave and the Living Lab experience, DICA begins to build a big data system for Living Labs.

As a short implementation period, we are not able to find out the drastic changes Living Lab lead to though, we did many experimental attempts with a systematic Living Labs framework. As a result, some factors that make enhanced commitments of each stakeholders and Living Lab Efficiency – sharing Living Lab concept and cases with panels, problem solving methodology (ex. design thinking) training program etc. - are identified. To build knowledge to make an advancement of Living Labs, the details of the above founding should be dealt in later studies.



This work was supported by National IT Industry Promotion Agency (NIPA) grant funded by the Korea government (The Ministry of Science and ICT).



Almirall, E., Lee, M., & Wareham, J. (2012). Mapping living labs in the landscape of innovation methodologies. Technology innovation management review, 2(9). Dutilleul, B., Birrer, F. A., & Mensink, W. (2010). Unpacking european living labs:

analysing innovation’s social dimensions. Central European journal of public policy, 4(1), 60-85. Eriksson, M., Niitamo, V. P., & Kulkki, S. (2005). State-of-the-art in utilizing Living Labs approach to user-centric ICT innovation-a European approach. Lulea:

Center for Distance-spanning Technology. Lulea University of Technology Sweden: Lulea. Kopp, U., Haider, C., & Preinesberger, J. (2017). Systematic tools to better identify and understand stakeholder roles and relations in Living Labs. In Research Day Conference proceedings 2017 (p. 60). Lepik, K. L., Krigul, M., & Terk, E. (2010). Introducing Living Lab's Method as Knowledge Transfer from one Socio-Institutional Context to another: Evidence from Helsinki-Tallinn Cross-Border Region. J. UCS, 16(8), 1089-1101.


Transnational piloting for smooth internationalization of health-tech start-ups

Päivi Haho 1 , Metropolia and Virpi Kaartti 2

1 Metropolia University of Applied Science 2 Laurea University of Applied Science

Applied Science 2 Laurea University of Applied Science Category: Innovation Paper Abstract Health-tech business

Category: Innovation Paper

Abstract Health-tech business is increasing all over the global markets. Startups need agile and cost-efficient methods to study the needs and opportunities in foreign markets. Thus, the purpose of this paper is to advance the understanding of internationalization of health-tech startups and to discuss the practical issues related to transnational living lab practices based on a case study of health-tech startups. Finally, a preliminary model for transnational health and wellbeing living labs is introduced.


internationalization, health-tech












Lean start-up and lean innovation terms have been adapted from lean manufacturing meaning eliminating waste, the non-value-creating efforts, to emphasize the core idea behind the lean innovation and lean startup methodology (Rasmussen & Tanev, 2015). Later, lean term has been applied in similar contexts, e.g. in software development, lean development, and lean enterprise (e.g. Ojasalo & Ojasalo, 2018). Besides business development, lean startup methodology can be used to support internationalisation of “lean global startups” (Rasmussen & Tanev, 2015). Living labs, as local, agile and networked operators in the target market, could support “lean global startups” to validate their business model, learn from customer experiences and cultural aspects and identify the right partners and channels for marketing.

This study follows from the authors’ aspiration for practice and preliminary research into the transnational piloting for smooth internationalization of health-tech startups. The results and key findings are reported in this stage.

The paper has four sections. The first describes the theoretical framework consisting of internationalization of startup companies, lean startups, and living labs. The second section introduces a case study of transnational piloting. The third section presents the preliminary results and findings. The fourth and final section discusses the contributions of the paper, notes the study’s limitations, and presents the roadmap for the future development needs of the transnational health and wellbeing living labs.

2 Theoretical framework

Digital and technological solutions for health-tech business is increasing all over the global markets. Startups need lean and cost-efficient methods to find out, create and test market needs and opportunities in foreign markets. Lean startup approach (Blank, 2007, 2013; Ries, 2011; Maurya, 2012) has been identified as a structured and efficient process, which may help startups to achieve their strategic and business goals in internationalization (Neubert, 2017). Many startups aim at internationalizing early and fast to become profitable and therefore they seek for markets where it is easy and fast to enter. Based on Johanson and Vahlne (2009), the Uppsala model can be applied to firms that begin to internationalize fast after their founding, if they seek low-risk and low-cost market-entry modes such as exporting. However, in many cases, this is not enough or it is not even an option, and the business model has to be created within the cultural premises and local market conditions. On the other hand, internationalizing early and fast is very challenging for startups and entrepreneurs because it requires specific competences, networks, special


preparation, high experience, and willingness and readiness to enter international markets (Neubert, 2016, 2017).

Rasmussen and Tanev (2015) linked the two research streams: lean startups and born-global firms, and introduced the new concept of lean global startup, which is typically a high-tech startup aiming at creating a new product with innovative solution for a specific market niche. In lean global startups, the internationalization strategy is part of the initial business plan (Neubert 2017). The lean startup methodology may also be used in internationalization by applying incremental and iterative product development cycles to develop minimum viable products (MVPs) and test them with quick feedback in the market (Tanev, 2017; Coviello & Tanev, 2017; Neubert 2017; Blank 2013). According to Neubert (2017) and Johanson and Vahlne (2009) the speed of learning in small, iterative steps defines the speed of early internationalization.

In the literature, social networks and networking ability as well as ability to learn have been recognized as the main drivers of fast internationalization (Coviello, 2015; Neubert, 2017). Based on Ciravegna, Lopez and Kundu (2014) the social networks of an entrepreneur as a driver of the speed of internationalization is essential. For the lean startups, the networking in the internationalization context is especially the ability to create market opportunities to acquire new clients and distribution partners with local networks.

In their article, Ojasalo and Ojasalo (2018) formulate how lean service innovation approach and process can focus and solve the needs of early identification of core customer value with business potential, especially for new or potential customers utilizing latent needs in service innovation. Their article build on the idea of lean innovation (Blank, 2007, 2013; Ries, 2011), and they borrow this idea for the service innovation process to fulfil the knowledge gap in service innovation research. In addition to identifying the knowledge gap, the article presents a managerial framework for applying service-dominant (S-D) logic in practice. Ojasalo and Ojasalo (2018) argue that the existing lean development models focus on an early understanding of customer needs and value, and thus the lean approach has a lot to offer for the research of S-D logic. They also underline that much more than knowing what presents customer value is required to turn into a profitable business. That is a scalable and profitable business model. In their framework, lean service innovation approach is used throughout the service process, and case-specific development methods are applied for solution development, testing, and experimentation. Living labs are intermediaries for innovations. They can be characterized in multiple ways and they serve several purposes. In a Living Lab Methodology Handbook Anna Ståhlbröst (2017) define: “A Living Lab is an orchestrator of open innovation

processes focusing





innovations in real-world contexts by involving multiple stakeholders with the objective to generate sustainable value for all stakeholders focusing in particular on the end-users”.

Based on Westerlund and Leminen (2011) “living labs are physical regions or virtual realities, interaction spaces, in which stakeholders from public-private-people partnership of companies, public agencies, universities, institutes, users, and others that follow the philosophies of open and user innovation to collaborate for improving, developing, creating, prototyping, validating, and testing of current or new technologies, services, products, and systems in real-life contexts”. Regardless of the multiple different definitions and implementations, living labs share certain common elements that are essential to the approach: 1) multi-method approaches, 2) user engagement, 3) multi-stakeholder participation, 4) real-life setting and 5) co-creation (Malmberg & Vaittinen, 2017). In addition, the living labs strives for mutually valued outcomes that are results of all stakeholders. Schuurman (2017) describes that three main elements have been distinguished within living lab projects, following the innovation development stages: 1) exploration; getting to know the current state and designing possible future states, 2) experimentation; real-life testing of one or more proposed future states and 3) evaluation; assessing the impact of the experiment with regards to the current state in order to iterate the future state.

In Living Lab Methodology Handbook (2017) the role of living labs is described as

mediators to build and strengthen the European Open Innovation ecosystem that enables the internationalization of SMEs supporting validation of products and services in other markets throughout living labs’ co-operation and consultancy. However, this role is new and emerging, and the living labs needs to advance their international collaboration and commercial offering for transnational validation of services for SMEs and startups.

Living labs has been used specifically by startups and SMEs, and they offer a structured approach to open innovation (Schuurman, 2015) in user innovation paradigm (von Hippel, 2009). Schuurman et al. (2016) have explored the value of a living lab approach for open innovation in small and medium-sized enterprises with

comparative case study in 27 SME projects, and they argue based on the results that

a real-life intervention and a multi-method approach increase the chance of

generating actionable user contributions for the innovation under development. In this study, a transnational living lab has been defined as a living lab, which serves companies or other institutions in an international context, i.e. at least in two countries. In literature, the terms cross-border or transregional living lab have been

used in similar contexts. The mediator living lab, i.e. Laurea, is seeking transnational health and wellbeing living labs as testing partners to accomplish the needs of local startups in their internationalization processes. Transnational testing partners are


requested to act in commercial basis performing expertise in their functions, and they should follow the lean start-up approach in their services.

The overarching aim of this case study is to create better opportunities and efficient methods and models for health-tech startups to internationalize by following the lean startup principles in transnational living labs. Finally, the results are described as preliminary findings and a model for transnational health and wellbeing living labs.

3 Transnational piloting

The living lab practices and experiences presented in this study are based on two cases. They are a part of the publicly funded project, in which one aim is to create and pilot a model for the transnational health and wellbeing living lab. In order to achieve this aim three pilots were planned to be carried out. The piloting focuses on the service ideas/solutions that are trying to solve one or several of the following challenges: 1) how might we increase physical activity in everyday life, 2) how might we enable people to take their health and well-being in their own hands and to manage them and 3) how might we support active independent living of the elderly. The challenges were defined based on the needs in the market. First three pilots were selected but finally two of them continued for actual piloting phase.

Next, the case companies are presented. Company A creates differentiating products by connecting everyday goods with mobile applications and accessories. They create tangible digital experiences in everyday products using wireless connectivity, sensors and cloud services with hardware, smartphone middleware and services platform. They also offer consulting services. The company is currently developing its own product portfolio of wellness and rehabilitation products. The company wants to pilot their product in order to test its suitability to the target market. They are looking for partners with whom they can develop technologies, services and products focusing on rehabilitation and wellbeing and gather feedback from local users. They are interested to work with cities and public sector. In addition, a living lab as a part of university might be a potential partner for them. They are also interested in to join a bigger consortium. By doing this testing, they want to develop their service design further and internationalize their service. For the piloting, there was a need to recruit end users who represented three different customer segments plus a professional to support the testing with end users.

Company B has two product lines of which product x is the newest. Several healthcare providers and companies in Finland use its predecessor, product y. This has allowed the company to collect data from hundreds of thousands of users over the years, which led to the development of the product x. While the product y is


designed for occupational health care professionals, product x is created for its end- users and employers. It is not just a mobile version of its predecessor but rather a scalable, more intelligent, user-friendly tool with several new features and more content. The company is heading to international markets and they are willing to test the service in a company/ies operating in their target market. They want to learn if there is a need to make significant changes to the product/service and how it works outside of Finland. The objective is to get a test data of a couple of hundred users in order to fine-tune the algorithm used in the application for the target market. In this case there was first a need to recruit a piloting company/ies who was/were willing to pilot the product with their employees. Thus, the focus is in business-to-business market.

Before explaining the piloting processes of the case companies further, some background information is given in order to present the whole development process of transnational health and wellbeing living lab model. The first draft of the transnational living lab model was planned based on the workshop done in the Open Living Lab Days 2017. The main organizers of the workshop were Laurea Living Labs and Licalab. The goal of the workshop was to move forward in setting up a Transnational Living and Care Lab as a unique innovation instrument to support SMEs in developing and scaling up innovations for 'living and care' and 'active and healthy ageing'. The transnational living lab model was discussed and considered through the lenses of the customer (e.g. SME). The workshop provided insights how to promote the transnational living lab, and revealed that there are development needs for the coordination of the international projects. It was identified that there is a need to improve the awareness of the healthcare living labs and their services. In addition, the contact information should be easily available, preferable one contact point. Further, cultural differences were discussed.

The process of piloting included several phases: call for ideas, selection of the pilots (inc. selection criteria), invitation to tender, selection of the living lab, planning, testing, results and next steps. Before explaining the phases in more detail, the stakeholders and their roles are presented. The project coordinator, Laurea University of Applied Sciences (Laurea Living Labs), works as a mediator in the piloting. Laurea is an official contracting body and facilitates the process between the company and the living lab. The company gives the brief and practical instructions and support for the piloting. In this pilot support is for example providing the products for testing and organizing the training for the living lab staff and the users in the target market. The living lab is responsible of the implementation of the pilot, analysis and reporting the results.

Call for ideas was a necessary step in this publicly funded project in which all the organizations who fulfilled the criteria should have a similar possibility to leave their


application and participate. The content of the call for ideas was: description of the challenges, the focus and the limitations, responsibilities, selection criteria, timetable, monetary support and contact information. The project partners who had a direct access to their startup networks were valuable messengers during this phase. In total nine applications were received before the deadline.

Selection of the pilots were done based on the criteria, which was planned before the call for ideas were published. First, the best applicants were selected based on the data they had provided, then the meetings were organized to discuss the piloting plan and confirm the common interest and understanding. A lot of practical information and new insights were received in these meetings. The meetings had also an important role to get acquainted with each other’s and to build trust.

Invitations to tender were planned based on the applications for call for ideas and the meetings during the selection process. Thereafter it was sent to the case companies for comments. Some iterations were done, and legal aspects were checked before sending invitations to tender forward. Simultaneously with this phase, potential service providers (living labs) were searched. That happened by the help of the European Network of Living Labs (ENoLL) and its Special Interest Group in healthcare, project partners from different ongoing or past projects and colleagues. The name and country of origin of the living labs were quite easy to get but what was more difficult was to find out their contact information and descriptions of them, their expertise and services. In some cases, there were web pages describing the living lab but not always in the language you could understand.

Selection of the living lab was done based on their offers. The selection criteria were explained in the invitation to tender. Before signing, the agreement there was a negotiation together with the service provider, Laurea and the startup company. For company A, the service provider is an experienced living lab having expertise in the business sector in question. For company B, the service provider is a private company offering partly similar services than living labs. The first drafts of the agreements were done by Laurea and they were sent to the service providers and startups for comments. After finalizing the agreements, they were signed by Laurea and the service providers. In the case of the company B a separate confidentiality agreement was signed with the sub-contractor of the service provider. In this phase the support from the legal services was needed and it was easily available as an in- house service at Laurea. The time for planning the agreements was not considered when making the initial schedules for testing.

The initial plan for testing was introduced in the invitation to tender. This plan covered following aspects: description of the case company and its product/service to be tested (what, to whom, how), implementation of testing (when, objectives, guidelines),


outcomes and the role of the testing partner (living lab). The plan was finalized together with Laurea, the startup and the testing partner. The planning was done in the skype meetings. Some modifications were done later in the process. The timeframe for the implementation of testing was first between January and May 2018. Because of the delays in the invitation to tender process, time used for the agreements and delays in the recruiting phase in the case of the company B the deadline was postponed until September 2018.

Actual testing phase has now ended with the company A. The testing was implemented quite independently by the testing partner: there were some communication between the company A and the testing partner regarding to the information needs and functionalities of their product and application. Further, Laurea and the startup visited the testing partner in this phase to discuss the process and see the testing in action.

The results were presented in the skype meeting. The final report was shared in advance to be able to discuss in more detail in the meeting. The expected outcomes in the report were analysis of the usability and the end user experience, needs for local adaptation and business model (potential for licensing, service as a part of the package, value proposition, marketing channels). These aspects were discussed and further information was asked, especially by the company A. For the company it is crucial to understand the context of testing and how it has been implemented in order to interpret the data correctly and to be able to apply the new knowledge in the similar circumstances elsewhere.

Testing phase with the company B proceeded with one piloting company to their human resource department. The research on that is still in progress and the results are not yet in use. Thus, the results and findings are based on the experiences so far. They are presented in the next chapter.

4 Preliminary results and findings

The preliminary results of the ongoing case studies provide guidelines to create a practical collaboration model for transnational living labs to support startups’/SMEs’ internationalization. These results and findings are presented in the table 1 and discussed in more detail thereafter.


Table 1: The results and findings of the case studies






Tasks of the startup/SME

Tasks of the Living Lab, mediator


Tasks of the Living Lab, testing partner (LL2)


Need to internationalize their business


LL services have to be visible in the

Call for ideas for international piloting


ernational startup



Searches for support to test their product/service in their target market

Contact point for startups to discuss if their need match with the service offering


Permanent contact point for the inquiries of LL services is needed



ready made

Describes a preliminary brief

Modifies the brief of startup/SME

template helps when preparing the brief


Searches for potential living labs for collaboration from the target market

Services are visible/known in the living lab network

Information needs to be available in English. More information of the focus areas (content, geographical area), services and expertise of LLs is often needed. Need to get an access to B2B customers.


Consultation with

Accepts the invitation to tender (matches with the brief)

the legal services.

Plans the invitation to tender



template helps when preparing a call for tender.

Gives further

Sends the invitation to tender and replies to the inquiries

Receives the invitation to tender and asks further questions if needed


information if

Timing is critical (holiday seasons)




Enough time for leaving a tender (quality)




Evaluates the offers together with the

Evaluates the offers and asks further questions if needed

Prepare an offer (implementation plan for testing, expertise, costs, contact details)

information of the LL2 is essential in the offer because otherwise the information is not available.


Negotiates with the LLs

Negotiates with the

Negotiates with the

Readiness for online







agreement between

the startup and the Living Lab/s or their


Signs agreement

Signs agreement

might be needed. Notify the time for planning of an agreement. Legal services might be needed.


Depending on the testing partner, a mediator can have a smaller/bigger role



Support the process

Facilitates the process, quality/cost/schedul e control

Implements the testing as agreed

Depending on the brief a startup/SME

can have a bigger


role in the recruiting



Longer time period to recruit B2B


Gets the results and plans the next steps

Accepts the results and pays the invoice. Discuss the possibilities for the future collaboration.

Presents the results, sends the invoice. Discuss the possibilities for the future collaboration.

Quality check of the



The findings are focusing on roles and responsibilities of the different actors and the whole process of piloting starting from the identification of the need to the results and planning of the next steps.

Several roles were identified in the project: living lab as a mediator, living lab as a testing partner, a piloting company (a company who is willing to test the product/service), end users, and a funding organization. The roles and responsibilities may vary depending on the expertise and available resources in the collaborating living labs (mediator, testing partner). Another thing, which defines the roles and responsibilities, is a possible project funding which was the situation in this particular case presented in this paper.

Process of piloting includes several steps: searching for companies, negotiations between a mediator and a company interested to pilot their product/service, preliminary plan for testing, invitation to tender, searching for living labs, sending the invitations to tender, analysing offers, negotiations, selecting a living lab for piloting, making a contract, modifying the plan for testing, training the living lab to use the product/service, implementation of the pilot, observing and following the project of piloting, results, and planning the next steps.

Contracts were done between a mediator and a living lab/s and the legal services of the mediator organisation were used. When having this service in-house it is easier and cheaper to use. There might also be a need to make e.g. confidentiality agreements between the startup/SME and the sub-contractor of the living lab (testing partner). The needed agreements with the end users and other test participants were in the responsibility of a testing partner. The process of making and accepting the contracts may take time, especially if there is not a ready-made contract templates that can be modified. There are many other moments, which take time in the process as well, and not all of them have been thought about thoroughly beforehand. Thus, there is a risk for delays in the process.

Information of the living lab services is not easily available whether it is on regional, national or international level. Even if the potential for the creation of new international growth companies has been created through the startup associations, entrepreneurship societies, business accelerators and incubators they are not aware of the living lab services and how to use them. If living lab services are not found in the certain country or region other service provides will be searched. That might be an only option also in the project where service has to be tested in business organization.

Transnational health and wellbeing living lab model could benefit different actors.


Startups would get an access to the international living lab network, get support from the local living lab, and utilize easy and agile way to implement piloting in their target market. Living lab as a mediator or as a testing partner could get new clients and collaboration possibilities with other living labs where to learn and broaden their network.

Building blocks for the preliminary transnational health and wellbeing living lab model consists of the structure of the ecosystem, different roles, process and methodology. In this case, the living labs had a regional and industry specific (healthcare) focus. Furthermore, the living lab who worked as a testing partner had a cross border and international scope in their existing ecosystem. Long-term relationships within the network offers a platform for testing and validating services in agile manner. The collaboration between the industry specific, regional and international ecosystems provide not only the expertise in the substance area (industry) but also the expertise to analyse the cultural aspects and overcome the language barriers. The role of living labs are a mediator/facilitator and a testing partner. Besides testing and validating the product/service, there is a need to act as a business developer. The process of working can be divided to a pre-testing phase, a testing phase and a post-testing phase. The methodology follows service design and lean startup approach even if not always identified as such. Interesting is that the aim of promoting regional development and industry is strong. Compared to that the business orientation of the living lab is not so much in focus. Together with business orientation, the commercialization of the services is still in its infancy. In continuously changing business environment and rules and legislation, the legal expertise may not be forgotten of the skillsets of the living lab.



The main contribution of this paper is the preliminary description of a model for collaboration of transnational living labs and startups in the validation, testing and local adaptation of service innovations of startups. In previous studies of transnational living labs (e.g. Bódi et al., 2015), description of experiences of living lab methodology and maturity of innovation and highlights and lessons learned from the internationalization aspect of transnational cases of startups have been elaborated, but any assumptions or a model have not been presented.

Based on the learnings and experiences in the workshop a year ago and two case studies during the past six months there is a clear need to conceptualize the living lab services and make them visible both regionally, nationally and internationally. This development is a necessary step towards commercialization and sustainable


business with startups or any other paying customer. The collaboration between the living labs increases the potential to offer the services to startups in several countries. Startups could contact transnational living labs by themselves when the contact information is easily available and they have resources to do that, or through the local living lab if that is more convenient solution for them.

Transnational context brings new expectations to the living lab that is used to operate locally. They have to be ready to work in a different language and there is a need of understanding the cultural aspects in the business relationships as well as when analysing the test results. Product or service to be tested might have texts that have to be translated to the local language, thus translation services might be essential add on in the service package. Furthermore, they have to have enough knowledge about the legal issues in international context. Transnationality in services brings also new roles and responsibilities. A living lab can have a role of a mediator when it is connecting a startup and a living lab abroad. Besides acting as a mediator, it can be a facilitator and support the process in various levels depending on the resources of other actors.

In this paper, we have presented through a case study how transnational living labs can be applied for validation and testing of heath-tech products and services in international markets, even in commercial bases. However, the role is new and emerging for living labs, and only few cases (e.g. Bódi et al., 2015) have been realized in transnational settings so far. Initially, a focus of living labs has been in user innovation and open innovation, and the validation of business models, business potential, marketing channels, and needs for local adaptation have not been in the scope. In lean startup approach, service development occurs cyclically in incremental steps at the same time when testing and validating a product, thus, transnational living labs are expected to offer this service as well (compared to the agencies). Based on this case study, the transnational living labs can serve the startups in their internationalization process in the means of mediators of lean startup or even lean global startup approach.

The study is in its’ initial phase, and the conclusions cannot be completed, yet. Knowing the limitations and benefits of a case study, in the next phase of research a model for collaboration of transnational living labs and start-ups is demonstrated.



Blanck, S.2007. The Four Steps to the Epiphany – Successful strategies for Products That Win. Raleigh, NC:Lulu Enterprises. Blank, S. 2013. Why the Lean Start-Up Changes Everything. Harvard Business Review, 91(5), 63–72. Bódi, Z, Garatea, J., Robles, A. & Schuurman, D. 2015. Living Lab Services for Business Support & Internationalisation. ENoLL. Ciravegna, L., Lopez, L. & Kundu, S. 2014. Country of Origin and Network Effects on Internationalization: A Comparative Study of SMEs from an Emerging and Developed Economy. Journal of Business Research, 67(5): 916–923. Coviello, N. 2015. Re-Thinking Research on Born Globals. Journal of International Business Studies, 46(1): 17–26. Coviello, N. & Tanev, S. 2017. Initiating a New Research Phase in the Field of International Entrepreneurship: An Interview with Professor Nicole Coviello. Technology Innovation Management Review, 7(5), 52–56. Johanson, J. & Vahlne, J. 2009. The Uppsala Internationalization Process Model Revisited: From Liability of Foreignness to Liability of Outsidership. Journal of International Business Studies, 40(9), 1411–1431. Malmberg, K. & Vaittinen, I. 2017. (Eds.) Living Lab Methodology Handbook. Maurya, A. 2012. Running Lean: Iterate from Plan A to a Plan That Works. Sebastopol, CA: O’Reilly Media. Neubert, M. 2016. Significance of the speed of internationalisation for born global firms – a multiple case study approach. International Journal of Teaching and Case Studies, 7(1), 66–81. Neubert, M. 2017. Lean Internationalization: How to Globalize Early and Fast in a Small Economy. Technology Innovation Management Review, 7(5), 16–22. Ojasalo, J. & Ojasalo, K. 2018. Lean Service Innovation. Service Science, 10(1),


Rasmussen, E. & Tanev, S. 2015. The Emergence of the Lean Global Startup as a New Type of Firm. Technology Innovation Management Review, 5(11), 12–


Ries, E. 2011. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Business. New York: Crown Business. Schuurman, D. 2015. Bridging the Gap between Open and User Innovation? Exploring the Value of Living Labs as a means to Structure User Contribution and Manage Distributed Innovation. Doctoral dissertation, Ghent University, Belgium. Schuurman, D. 2017. Living Lab methodologies in Malmberg, K. & Vaittinen, I. (Eds.) Living Lab Methodology Handbook. Schuurman, D., De Marez, L. & Ballon, P. 2016. The Impact of Living Lab


Methodology on Open Innovation Contributions and Outcomes. Technology Innovation Management Review, 6(1), 7–16. Tanev, S. 2017. Is There a Lean Future for Global Startups? Technology Innovation Management Review, 7(5), 6–15. von Hippel, E. 2009. Democratizing Innovation: The Evolving Phenomenon of User Innovation. International Journal of Innovation Science, 1(1), 29–40. Westerlund, M. & Leminen, S. 2011. Managing the Challenges of Becoming an Open Innovation Company: Experiences from Living Labs. Technology Innovation Management Review, 1(1), 19–25.


User needs and expectations as a challenging factor for successful living lab research initiatives involving older adults: The DDRI Experience

Tiziana C. Callari* 1 , Louise Moody 1 , Nikki Holliday 2 , Ed Russell 3 , Janet Saunders 1 , Gill Ward 4 , Julie Woodley 5

*Corresponding author 1 School of Art and Design, Coventry University 2 Centre for Innovative Research Across the Life Course, Coventry University 3 WCS Care Group 4 Royal College of Occupational Therapists 5 Faculty of Health and Applied Science, University of West England

Category: Research Paper

University of West England Category: Research Paper Abstract Despite the number of Living Labs now in

Abstract Despite the number of Living Labs now in existence worldwide and in Europe, there is still a limited amount of academic study and literature to guide their set-up and running. We were particularly interested in ethical guidelines and frameworks to guide research engagement and participation as Coventry University along with partners sought to develop and establish a new LL. This paper therefore intends to shed light on the ‘user engagement’ aspect, in particular when older adults, and adults with cognitive and physical impairments are involved. To do so, we recruited n=6 residents and n=2 family members from two residential living environments partners of the DDRI programme, a LL pilot initiative led by Coventry University. Methodologically, semi-structured interviews were conducted and the Qualitative Content Analysis (QCA) method used to make sense of the data. A deductive coding frame was built informed by the study goals, and its categories included (among others) ‘user engagement’, ‘user commitment, ‘user expectation’, ‘user needs’, ‘nature of participation’. The results report on the participant views and concerns about these LL research topics, and provide (ethical) insights for future research collaborations.

Keywords: User needs, User expectations, Nature of participation, Ethical issues, Living lab, Older adults, Family




Living Labs (LLs) are defined as “user-centred, open innovation ecosystems based on systematic user co-creation approach, integrating research and innovation processes

in real life communities and settings” (ENoLL, 2006). Involvement of users is seen as

a “participative force for co-creating value”, and facilitating the creation of an

innovation ecosystem based on a public-private-people partnership in the form of collaborative processes and knowledge sharing (Pallot, Trousse, Senach, & Scapin, 2010, p. 2). This innovation process becomes user-centred (with user as subject) and participatory (with users as partners) increasing user-acceptance of new products, services or processes, and hence reducing the failure rate at the market (Dell'Era & Landoni, 2014; Habibipour, Padyab, Bergvall-Kåreborn, & Ståhlbröst, 2017). Multi- stakeholder collaboration and knowledge sharing is a critical factor to successful LLs (Buitendag, van der Walt, Malebane, & de Jager, 2012; Wenger, McDermott, & Snyder, 2002). For example, Buitendag et al (2012) argue that knowledge creation and knowledge sharing within and outside rural LLs motivate and encourage communication and enhance long-term collaboration between members of different communities of practice. This also supports the idea that multiple perspectives can bring value to each of these partners in an integrative way, and contribute to the LL innovation process and outcome. Ultimately, this becomes a requirement for the success of LLs (Habibipour, Padyab, et al., 2017; Pino et al., 2014; Ståhlbröst, 2012). LLs involve a multi-method approach in which the co-design/co-creation criterion can depend on the degree of involvement and the actual utilization of user feedback and suggestions for the (re-)shaping of the innovation (Schuurman, De Marez, & Ballon, 2015; Schuurman, Mahr, De Marez, & Ballon, 2013); the process of user involvement and participation will also impact the ethical and operational concerns of the LL. However, there is limited academic literature and exploration of how LLs should be

developed through stakeholder involvement, to ensure that stakeholder involvement

is maintained and optimised. Central to participant and stakeholder involvement is

consideration of the key ethical issues as well as how communication is managed through the lifetime of the LL and individual projects within it.

Researchers agree that there is still scarce literature addressing specifically the ethical guidelines involved in the design, development, and implementation of R&D projects (Pino et al., 2014; Sainz, 2012). This is certainly due in part to the nature and characteristics of LLs and the heterogeneity (e.g. in the way the LL concept and approach has been understood and applied) of LL projects shared within the scientific community (Burbridge, 2017; Müller & Sixsmith, 2008; Novitzky et al., 2015; Schuurman et al., 2015; Yazdizadeh & Tavasoli, 2016). The involvement in LLs of older adults, and adults with reducing cognitive and physical capacity poses additional ethical challenges, which include fluctuating capacity or loss of capacity to provide informed consent, participation opt-in and opt-out, involvement of third-parties as


decision makers e.g. children and carers of participants (Novitzky et al., 2015; Panek, Rauhala, & Zagler, 2007; Pino et al., 2014; Sanchez, Taylor, & Bing-Jonsson, 2017). Given this context, this study has been undertaken within the framework of the Data Driven Research and Innovation (DDRI) Programme, a LL initiative which has involved Coventry University and sector partners, and two residential living environments. The first residential environment (later on referred to as ‘Setting A’) offers day care, long- term residential and short-term respite care for older people and people living with dementia or a range of different needs. The second (later on referred to as ‘Setting B’) offers independent living for adults over 55 years, extra care is available for those who need it. The overarching objective of the DDRI Programme is to use data driven analytics and insights to support: independent living; healthy ageing; improved quality of life, well-being and care (self-care or care from others); improved management of particular health conditions; and enabling people to stay in their own homes for longer. A number of DDRI/LL projects have been developed and launched within the two living environments over the last 18 months (see Table 1). One of those projects specifically focused on co-creating with stakeholders’ mechanisms for engaging older adults in individual research studies, with a particular focus on how they are approached to take part, how they are engaged and how they are thanked. The project involved a set of interview studies, followed by co-design workshops to address some of the issues raised.

Table 1. List of some of the DDRI Projects that the participants were briefed on

DDRI Project Title




Applied sleep interventions for elderly residents in a care home setting

To explore ways to improve sleep and innovative ways of responding to night-time waking

Setting A

Development of wearable technology to detect dehydration in older adults

To explore the feasibility and design of body worn devices to detect hydration levels and feed-back information to care staff

Setting A & B

Innovation for Dementia Care:

To explore the potential for digital innovations to improve health and wellbeing for a frail elderly population, including people living with dementia

Setting A & B

Evaluation of Digital Health and Wellbeing Apps in ‘Real-Life’ Living Labs


The effects of an acoustic monitoring system on night time falls by care home residents with dementia and care staff response

To trial the use of an acoustic monitoring system and assess its effectiveness in preventing night-time falls within a care home for people with dementia

Setting A

With this study, we aimed to collect the user needs and expectations in relation to participation and engagement in the above LL projects. Participants in this context were the older adults living in the two residential living environments, and their family members. As participation was voluntary, we also investigated the extent to which rewards/incentives could be a possible additional lever for recruitment in research projects.

2 Literature review

In line with this study objective, the literature review here has focussed on the following topics only: i.e. (1) user engagement, motivations and expectations, (2) payment and participation.

User engagement, motivations and expectations are critical inputs to create and sustain a fertile LL context (van Geenhuizen, 2018). User engagement can be driven by a number of activities, which may include promoting clear communication about the project objectives and outcomes, shaping research goals based on user needs and concerns, collaborating with users to help define future research direction, policy or implementation of research outcomes. LLs could experience different degrees of user engagement, running from users as leading co-creators at one extreme to users as passive subjects at the other extreme (i.e. only involved to test/evaluate LL products and/or services) (Almirall, Lee, & Wareham, 2012; van Geenhuizen, 2014). However, engaging users throughout the lifetime of a project is not an easy task, as attention, motivations and expectations tend to decrease over time, and users tend to drop-out from research activities, regardless of the specific phase on the project development process, before they are completed (Habibipour & Bergvall-Kåreborn, 2016; Habibipour, Georges, Schuurman, & Bergvall-Kåreborn, 2017). In particular, user motivation in LL research activities tends to be higher at the beginning, and decreases towards the end of the activities (Habibipour, Padyab, et al., 2017). This trend may be caused by the user’s internal factors (intrinsic motivation) or external factors (extrinsic motivation) (Habibipour & Bergvall-Kåreborn, 2016). Researchers now agree that


keeping users motivated and having sustainable user engagement during the whole process of open innovation is of crucial importance. In fact, user drop-out has been acknowledged as having significant (negative) impact on project time and cost efficiency, quality assurance, overall loss of trust and motivation with project participants and stakeholders (Habibipour, Georges, et al., 2017). In their taxonomy for drop-out in LL field tests, Habibipour and Bergvall-Kåreborn (2016) identify three main categories: (1) Innovation-related category, (2) Research-related category and (3) Participant-related category. Within the first category, beside the technological problems (e.g. stability and maturity of prototype), two subcategories involve understanding user needs and participant motivation. The first regards the ‘perceived usefulness’ of an innovation by the target user; the second ‘perceived ease of use’, in which complexity in user-interaction, and or readiness of a prototype might affect user motivation. Within the second high-level category (i.e. ‘Research-related’), which includes ‘Task design’, ‘Interaction’ and ‘Timing’, in the ‘Interaction’ subcategory the attention is focused on the level of communication established in the LL between researchers and users and the expectation set. Finally, within ‘Participant-related’ drop-out are listed the personal issues related to the participant – i.e. the attitude, the context and resources.

LLs, especially with the involvement of older adults, face a range of ethical issues which need to be adequately addressed. The approach to research in this context (especially where academic partners or health or social care partners are involved) will need to undergo a process of formal ethical review and approval. There is clear guidance of research involving older adults and working in high-risk environment, and from specific professional bodies (Bollig, Schmidt, Rosland, & Heller, 2015; BPS, 2009; NSW, 2015; Walsh, 2009). We would argue though that the issues brought together in LLs are complex and multidisciplinary, and navigating the route to clear ethical principles and practice is an ongoing challenge. One of the challenges for example faced in LLs is how to handle an appropriate rewarding and incentive system for users involved in LLs. We are aware of the ethical debate around the use of rewards and incentives to research participation. Here we report what authors in the field have commented in relation to ‘rewarding’ LL participants for their involvement in LL research activities. For example, Dutilleul et al. (2010) argue that as users in LLs actively involved in co-design and co-creation activities to produce value, there should be rewards in place that secures pay-back to all the actors involved. Incentives may be of different kinds (e.g. vouchers, free tickets, etc.), and can be assigned either at the beginning, after recruitment, or during the trial activities (Georges, Schuurman, & Vervoort, 2016). Others suggest that an inappropriate incentive mechanism, together with issues in building a positive relationship with and mutually between the LL stakeholders, may be among the reasons for participant drop-out (Habibipour, Georges, et al., 2017). Buitendag et al. include this aspect in their definition of LLs, as “the active participation by the community of practice (CoP) and other stakeholders


in some or all living lab activities, which may also include in sharing the reward” (Buitendag et al., 2012, p. 221). They argue that ‘rewards’ do not only refer to tangible or financial incentives, but can encompass the underpinning concept of open society for LLs (Buitendag et al., 2012). Along this line, it is argued that LLs support open innovation built “on voluntary collaboration” in which each participant has a similar role and relevance (Nyström, Leminen, Westerlund, & Kortelainen, 2014, p. 484; Ståhlbröst & Bergvall-Kåreborn, 2013). Along with Sainz (2012), we suggest that the innovative LL approach would benefit from guidelines and regulations on the ethical aspects inherent to its voluntary and participatory nature, this was the driver for the study described here



The full study was carried out in 2017 over a 12-month period employing a qualitative research approach (Blaikie, 2009; Ritchie, Lewis, Nicholls, & Ormston, 2014). The study received ethical approval from Coventry University Research Ethics Committee. A letter of support was provided by the residential organisations (i.e. Setting A and B). The principles of the British Psychological Society Code of Ethics and Conduct and UK Research Integrity Office’s Code of Practice for Research guided the research.


Recruitment of residents followed two strategies: in ‘Setting A’ participants were recruited via the recommendation of the staff team, who facilitated the identification of the interested residents; in ‘Setting B’, coffee morning events were organised, and flyers distributed to advertise the research initiative with the proposed projects, and collect resident interests. Overall, n=6 residents agreed to take part in this study, of which n=5 were female (we only recruited residents who were able to provide their informed consent). The average age was M=79.5, where the youngest was 56 years old, and the oldest 90 years old. Two participants currently lived in the care facility (i.e. Setting ‘A’), whilst the other four have been living in independent apartments in Setting B for an average period of 17 months. Further, n=2 family members were recruited. Their parents live in Setting A and at the time of this study were diagnosed with cognitive impairments. For details see Table 2 below.


Table 2. Participants’ demographics



Male /

Setting #

Time at residential setting



Resident - brain damage age 56, hoping to regain independence.



4 months


Resident, age 90, no dementia






Resident, age 88






Resident, age 87






Resident, age 84






Resident, age 72






Family member - mother has dementia, father was also in Setting A






Family member - father has Parkinson’s



5 months


Semi-structured interviews were conducted with residents and family members. The interview guideline included probing questions to collect opinions (both positive feedback and concerns) about (1) the ongoing DDRI projects in the two living environments (namely in Setting A and Setting B), and (2) ethical issues and concerns. Participants were given short summaries of exemplar DDRI projects by the interviewer of a and questions explored for example their views on each project, concerns about getting involved in it if invited, concerns about giving consent, views on ‘thank you’s’ and the use of incentives for participation. Overall, the average interview length was an hour; resident participants were allowed to take breaks if requested. The interviews were audio-recorded and transcribed verbatim; all transcripts were saved, coded and analysed using NVivo (v.11 Plus for Windows, ©QSR International) (Bazeley, 2007). An NVivo project (entitled ‘DDRI-Driven Research and Innovation’) was created, which contained all interview transcripts; the selected literature articles; the memo journal keeping track of all activities and decision-making points agreed throughout the development of the research.


Data Analysis

The Qualitative Content Analysis (QCA) method was used to make sense of the data (Schreier, 2012). Critically, the coding frame built to interpret the raw material was informed by the study objectives, and hence followed a concept-driven strategy (Saldana, 2012; Schreier, 2012). In the NVivo project a tree-node structure was created: the parent-nodes (i.e. high-level categories) reflected the relevant themes to the research. A number of child-nodes (i.e. the subcategories) specified each parent node. See below Figure 1 for an overview of the coding frame used.

See below Figure 1 for an overview of the coding frame used. Figure 1. Concept-driven coding

Figure 1. Concept-driven coding frame used to categorise the study data (from the NVivo project)

All interviews were coded using the proposed coding frame. The segmentation strategy used as coding unit was the ‘meaning unit’ (that is, any portion of text, regardless of length, to which it was believed a code may apply) (Coffey & Atkinson, 1996; Grbich, 2013; Saldana, 2012).



Involving older adults, and adults with physical (e.g. sight or hearing impairments) or mental (e.g. dementia) disorders poses additional challenges for the researcher. Some of these challenges include the design of an inclusive consent form and participant information; the involvement of family, or others to provide consent should capacity to consent be diminished. Mechanisms for sharing information and gaining consent were discussed. Information about the DDRI projects was circulated in the two residential settings by means of Project Information Sheets (PIS). Colour images and short paragraphs of text (using Arial 14 font) were used to describe the different


active projects, and assess the resident interest before asking them to sign the informed consent. Overall, this approach was commented positively. Further, participants highlighted the need to consider individual needs:

“Sight and hearing are important. We all look normal enough but everyone’s got some kind of underlying problem. For the people who are partly blind or blind, it could be read out to them. You would have here a few with sight problems”.

The involvement of third parties in the process of consent to participate in a study was also discussed. Involving the family, or a ‘carer network’ is well accepted by all residents, as they are conscious of their deteriorating physical and mental conditions, but recognised participant wishes and attitudes should still very much be considered. This also may include the possibility to withdraw from the LL research initiative due to health factors.

“Well I mean if you as the researcher thinks that the person is no longer quite capable of doing it, then I think that’s reasonable [to involve the family]”.

“Family or carers then need to know what the person concerned attitude was in relation to research”.

Challenges around the location of family members were raised and how this might affect recruitment to studies if they were not easy to get hold of and consult:

“I don’t know my family is fairly well scattered around the country, there’s none of them that are local. But they are already aware of different aspect concerning my health, so this involves not only this project”.

All of the presented DDRI projects (see Table 1) were found to be of interest to the participants’ and indeed the feedback was positive. Reactions like ‘This sounds interesting’, ‘Well, I think that’s a brilliant idea’, ‘I’d very much welcome something like that’ were collected. The project related to sleep interventions raised particular interest among the residents in both settings, as it was commented that sleep is an issue at their age. Hence, a number of questions in relation to potential side effects, dosage, and food sensitivities were asked. The project about hydration level detection also raised attention and the resident comments – particularly in relation to wearing the proposed technology – were positive. Overall, the discussion about the LL projects


showed how the residents (in particular in Setting ‘B’) can discern their needs and what could be good (or not) for their health.

“This is another one that’s right in my area of concern. I can’t get to sleep at night without an intake of zopiclones, […]. I notice that in your project you’ve got separate ideas, like the milky drink – but I’m sensitive to milk and any dairy, cheese or anything of that sort so that rules that out for me but I get the tryptophan from bananas and dates, dried dates. You know, I got a lot of faith in a nutrition book that I’ve got down here”.

“I recognise the importance of hydration and my general health dictates that I do have a good intake of, water on one hand but then again I have prostrate problems so that is quite a problem for me getting it right…but a research into it is wonderful”.

It was interesting to gain a sense of user motivation to take part in research. The older adult residents, identified being motivated to be engaged in something challenging that could help maintain their mental capabilities:

“I like to get involved with these sorts of things because I think it keeps my brain working. To be honest with you, it’s just like if you just sat here in this flat and did nothing. I couldn’t do that I have got to be doing something, and I say it I don’t mean physically, I mean mentally!”.

The intrinsic motivation linked to specific recognition that participation in studies might support their own health and satisfy their interests. Their perception of the value of the study and participation encouraged influenced overall resident attitude in relation to research participation. Residents commented that supporting research and the ‘general good’ are important aspects to them, either because they believe in what the research is aiming to, or given their personal and educational background.

“Well if they were told by doing research that they were likely to get better, have better sleep they would, should be taking part. And even if they didn’t I mean it would help somebody somewhere”.

“See, when you came we didn’t really know what we were taking part in, we didn’t really understand what it was but it’s really future research that will help people in the future more than us probably”.

“But then it depends on the background of the person, you know they wouldn’t normally, well they’ve just never heard of it. I think that’s for your educational background and what sort of research”


Many factors can affect the residents’ attitudes towards research participation, both positively and negatively. These may include culture (e.g. what it means to get engaged in research), period of life (e.g. living in senior living settings), personal beliefs, etc. For example, in more than one interview it was highlighted that the word ‘research’ seems to convey a negative connotation, and consequently led to a feeling of distance from the issue and closure.

“I can only tell you the impression I get from talking with people here. I feel that quite a lot of people, they are not really interested. They got to the stage in life where they just really don’t want to be bothered”.

“I feel that as soon as you say research they’ll say “Oh no-- I’m not interested, thank you. I think so because they think of researchers are really going inside you”.

“I don’t tend to do surveys at all, particularly on the phone. They ring you up and say we’re just doing a survey. I just say --no. I don’t do surveys on the phone and put the phone down!”.

“Getting involved with things like these I wouldn’t do it, not knowing your background, knowing where you came from sort of thing I mean I wouldn’t do it to anybody who just came to the door and asked me to do it”.

Further, the interviews highlighted that those wanting to involve older adults in research should also consider how older people can personally manage being part of a LL research initiative and how researchers should take their needs into account in study design and delivery, particularly with pace of all procedures involved, etc. One resident commented about some possible unexpected reactions, pointing at the potential vulnerability and variation in capabilities of older people.

“You see, there are people with slight irritable problems, get het up very quickly, the sight of a piece of paper that they have to listen to and do anything with it is beyond them. Apart from that, don’t put any pressure on them. You know, it’s how much they can cope with and you don’t know whether it’s because of their underlying illness or not, you just accept them as they are and then just work around them. […] You know, refer back to what you did and if somebody has a better feeling than they might do anyway”.

The participants had clear views on the benefits to them of participating. They would like acknowledgement and to know about the impact of the input they made or the valuable future impact in society. This also included their attitudes toward research participation. Although during the recruitment process it was suggested to consider


rewards as a motivation factor for participating, all participants were volunteers. When prompted about the topic, both residents and family members confirmed that participation should happen without the need of rewards.

“We should all do our bit and not expect a reward”.

“I think if you’re interested, you do it, just do it. I mean I can’t see why we need to have a reward’.

“No. No rewards. That drives the wrong behaviour doesn’t it?

They do not necessarily expect a ‘personal’ thank you, but they expected to receive information about the outcomes the project had achieved, any next step, etc. The same view was shared by the family members

“You know having participated in that little bit of research, it obviously links into something else and it could be nice if you can hear about it and think:

“well I feel quite proud of that because I helped”.

“It would be nice to get the outcome be it in the form of an e-mail, a general e-mail to everyone and what the contribution was and how it’s resulted”.

In line with the above, the interviewed family members were very aware of their parent’s health condition, and what this entails in terms of communication and interaction (e.g. the need to repeat a number of times the research questions - posed in a Yes/No way - to get the answer), and how both physical and mental conditions may deteriorate as the research proceeds.

“I don’t think this would be a concern, it would be good because I know how Mum communicates […] by her pointing, so even without saying anything that’s communicating. […] The only thing that I think that would be difficult is if you were to spend an extended period of time with her to get her to do one single thing because of her concentration levels, she’d get tired very quickly. So, it should be a gently, gently approach really”.

“That’s part of the thing -- if this is no longer suitable for my Dad’s condition or somebody else’s condition, we just need to step outside the trial please”.




This study was part of a bigger research initiative carried out within the DDRI Programme that involved a number of LL studies carried out in two residential living environments with older adults, and adults with physical and cognitive impairments. A qualitative research strategy was used, and interviews conducted with the research participants to explore user needs and expectations in engaging in LL research initiatives, and ethical issues connected to this.

Overall, LL research initiatives leveraging greater interest among older people address health topics close to the current user needs (e.g. sleeping issues, body hydration, etc.). This can have an impact on the residents’ intrinsic motivation towards research participation. Other factors may drive the overall motivation, such as cultural aspects, which resulted in several valuable feedback responses. One concern was related to the connotation that the word ‘research’ may convey to older people, and possible alternative ‘labels’ to define LL ‘research’ initiatives have been advanced, including “Adult caring research” or “We want your views”. Further, as LL projects involve medium-, long-term collaborations with research participants, it is argued that it is vital to maintain the ongoing interest and cooperation, as well as expectations, of research participants, family and other stakeholders for successful research initiatives.

More light has also been shed in relation to critical ethical concerns when LL initiatives engage with older adults, and adults with physical and mental impairments. This included the nature of participation in LL research initiatives (i.e. voluntarily. rewarded), opting out from research, or involving family in the decision-making process when the physical and mental condition of elderly persons may put their participation at risk. The findings informed subsequent co-design sessions to develop acceptable briefing information and research processes for gaining consent and meaningful participation within the DDRI model that are reported elsewhere.



Almirall, E., Lee, M., & Wareham, J. (2012). Mapping Living Labs in the Landscape of Innovation Methodologies. Technology Innovation Management Review,


Bazeley, P. (2007). Qualitative Data Analysis with NVivo. London: SAGE Publications. Blaikie, N. (2009). Designing Social Research. Malden, MA: Wiley. Bollig, G., Schmidt, G., Rosland, J. H., & Heller, A. (2015). Ethical challenges in nursing homes – staff's opinions and experiences with systematic ethics meetings with participation of residents’ relatives. Scandinavian Journal of Caring Sciences, 29(4), 810-823. doi:10.1111/scs.12213 BPS. (2009). The British Psychological Society Code of Ethics and Conduct. Retrieved from guidelines Buitendag, A. A. K., van der Walt, J. S., Malebane, T., & de Jager, L. (2012). Addressing Knowledge Support Services as Part of a Living Lab Environment. Issues in Informing Science and Information Technology, 9, 221-241.


Burbridge, M. (2017). If Living Labs are the Answer – What's the Question? A Review of the Literature. Procedia Engineering, 180, 1725-1732.


Coffey, A., & Atkinson, P. (1996). Making sense of qualitative data: complementary research strategies. Thousand Oaks, CA: Sage Publications. Dell'Era, C., & Landoni, P. (2014). Living Lab: A Methodology between User-Centred Design and Participatory Design. Creativity and Innovation Management, 23(2), 137-154. doi:10.1111/caim.12061 Dutilleul, B., Birrer, F., & Mensink, W. (2010). Unpacking European Living Labs:

Analysing Innovation’s Social Dimensions In K. Müller, S. Roth, & M. Zak (Eds.), Social Dimension of Innovation. Prag; Linde. ENoLL. (2006). European Network of Living Labs (ENoLL). Retrieved from Georges, A., Schuurman, D., & Vervoort, K. (2016). Factors Affecting the Attrition of Test Users During Living Lab Field Trials. Technology Innovation Management Review, 6(1). Grbich, C. (2013). Qualitative Data Analysis: An Introduction. London: SAGE Publications, Ltd. Habibipour, A., & Bergvall-Kåreborn, B. (2016). Towards a User Engagement Process Model in Open Innovation


Paper presented at the The International Society for Professional Innovation Management (ISPIM) Innovation Summit: Moving the Innovation Horizon (4-7 December 2016), Kuala Lumpur, Malaysia Habibipour, A., Georges, A., Schuurman, D., & Bergvall-Kåreborn, B. (2017). Drop- out in Living Lab Field Tests : A Contribution to the Definition and the Taxonomy. Paper presented at the Research Day Conference proceedings 2017: OpenLivingLab Days 2017,, Krakow. Habibipour, A., Padyab, A., Bergvall-Kåreborn, B., & Ståhlbröst, A. (2017) Exploring factors influencing participant drop-out behavior in a living lab environment. In:

Vol. 294 (pp. 28-40): Springer Verlag. Müller, S., & Sixsmith, A. (2008). User requirements for ambient assisted living:

results of the SOPRANO project. Gerontechnology, 7(2). Novitzky, P., Smeaton, A. F., Chen, C., Irving, K., Jacquemard, T., O’Brolcháin, F., . Gordijn, B. (2015). A Review of Contemporary Work on the Ethics of Ambient Assisted Living Technologies for People with Dementia. Science and Engineering Ethics, 21(3), 707-765. doi:10.1007/s11948-014-9552-x NSW. (2015). NSW Health Code of Conduct. Retrieved from

Conduct.pdf Nyström, A.-G., Leminen, S., Westerlund, M., & Kortelainen, M. (2014). Actor roles and role patterns influencing innovation in living labs. Industrial Marketing Management, 43(3), 483-495.


Pallot, M., Trousse, B., Senach, B., & Scapin, D. (2010, 2010-08-25). Living Lab Research Landscape: From User Centred Design and User Experience towards User Cocreation. Paper presented at the First European Summer School "Living Labs", Paris, France. Panek, P., Rauhala, M., & Zagler, W. L. (2007). Towards a Living Lab for Old People and their Carers as Co-Creators of Ambient Assisted Living (AAL) Technologies and Applications. Paper presented at the 9th Europ Conf for the Advancement of Assistive Technology in Europe (AAATE). Pino, M., Benveniste, S., Kerhervé, H., Picard, R., Legouverneur, G., Cristancho-

Lacroix, V.,

to the development, assessment and provision of assistive technologies for

supporting older adults with cognitive disorders. Stud. Inform. Univ., 11(2), 34-


Ritchie, J., Lewis, J., Nicholls, M. C., & Ormston, R. (Eds.). (2014). Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: SAGE Publications. Sainz, F. J. (2012). Emerging Ethical Issues in Living Labs. Ramon Llull Journal of Applied Ethics, 3, 47-62.

Rigaud, A.-S. (2014). Contribution of the Living Lab approach


Saldana, J. (2012). The Coding Manual for Qualitative Researchers. Thousand Oaks, CA: SAGE Publications. Sanchez, V. G., Taylor, I., & Bing-Jonsson, P. C. (2017). ETHICS OF SMART HOUSE WELFARE TECHNOLOGY FOR OLDER ADULTS: A SYSTEMATIC LITERATURE REVIEW. Int J Technol Assess Health Care, 33(6), 691-699.


Schreier, M. (2012). Qualitative Content Analysis in Practice. London: SAGE Publications. Schuurman, D., De Marez, L., & Ballon, P. (2015). Living Labs: a systematic literature review. Paper presented at the Open Living Lab Days 2015. Schuurman, D., Mahr, D., De Marez, L., & Ballon, P. (2013). A Fourfold Typology of Living Labs: An empirical investigation amongst the ENoLL community. Paper presented at the 2013 International Conference on Engineering, Technology and Innovation (ICE) & IEEE International Technology Management Conference (24–26 June 2013), The Hague, The Netherlands. Ståhlbröst, A. (2012). A set of key principles to assess the impact of Living Labs. International Journal of Product Development, 17(1-2), 60-75.


Ståhlbröst, A., & Bergvall-Kåreborn, B. (2013). Voluntary Contributors in Open Innovation Processes. In J. S. Z. Eriksson Lundström, M. Wiberg, S. Hrastinski, M. Edenius, & P. J. Ågerfalk (Eds.), Managing Open Innovation Technologies (pp. 133-149). Berlin, Heidelberg: Springer Berlin Heidelberg. van Geenhuizen, M. (2014). Living labs: concepts and critical factors, with case studies from health care. Paper presented at the The 14th ICTPI Conference - Building Sustainable R&D Centers in Emerging Technology Regions (9-12 September 2014), Brno, Czech Republic. van Geenhuizen, M. (2018). A framework for the evaluation of living labs as boundary spanners in innovation. Environment and Planning C: Politics and Space, 0(0), 2399654417753623. doi:10.1177/2399654417753623 Walsh, S. A. (2009). Conducting Research with the Elderly: Ethical Concerns For a Vulnerable Population. Southern Online Journal of Nursing Research, 9