May 2016 - Aug 2017


UX Researcher & Designer
Full-stack Web Developer


Research through Design (RTD)
Field Study
Live Web Prototype


Abobe XD


Reveal the novelty of the performance and bring unique audience experiences


Audiences find the experience unique and interesting and appreciate the innovative design of the performance

Design Process



The work is introduced in the publication "Designing the audience journey through repeated experiences" as part of the entire Climb! project. The paper was presented in CHI conference in Montreal, Canada 2018. By the end of April 2018, Climb! has been performed in four big concerts and it will keep going further. You can find recordings of past performances if interested.



Climb! is a musical composition that combines the ideas of a classical virtuoso piece and a computer game produced by a composer and pianist Maria Kallionpää.  Written for a Disklavier grand piano, the work engages the pianist in an unusual musical dialogue where there is no fixed musical form: the performer’s/gamer’s actions define how the story continues. It has a branching structure that guide the performer along different routes through the work. The performance can be different each time depending on how the performer interact with the piano, whether they continue to progress with their current path or be diverted onto another one.

Maira Performing Climb!

Maira Performing Climb!


Needs Identification

As I introduced earlier, Climb! consists of three macro-compositions, or we call them, paths. Each of them contains 8 micro-compositions. The arrowed lines connecting micro-compositions are possible destinations of that stage. Each micro-composition was composed around a narrative. By interacting with the piano differently,  the musician determines how the story flows, in other words, the experience that audiences go through.

Climb! consists of 3 macro-compositions (path) with 8 micro-compositions in each of them. Each micro-composition tells a story

Climb! consists of 3 macro-compositions (path) with 8 micro-compositions in each of them. Each micro-composition tells a story

When I was attending a rehearsal of Climb! at the first time, I realized that, as an audience, I was not able to appreciate the unique design of the performance since the special structure and interactions between the piano and the player are not perceivable to me. Additionally, I was not able to distinct this performance with others before I was introduced with more details. I gave my feedback to Maria and my supervisor Steve Benford, who brought me into this project, we had a conversation and later came into the agreement that it would be an interesting design opportunity. The goal is to reveal the novelty of the performance with potential to improve the overall audience experience.



Level Map of Games

Level Map of Games

Since it is a unique case to design a software for a specific performance, there are not many examples online that we can learn from. My keywords for research includes "live performance technology", "music games", "digital story telling", "concerts application" etc. After reviewing many examples and discussing with Maira, eventually I borrowed the idea of level map from games and came up with the first version of the design. It was live map to indicate stage changes and communicate stories of the performance. To match the leitmotiv of the performance, I used a mountain as a background and put lines and dots on it to represents paths and micro-compositions. Different color represents different status of the micro-composition, i.e. visited, missed, active, revealed and hidden. The map expand as the performer performing, showing the progress of the performance, its future directions and the stories behind.

1. Brainstorm ideas: a dynamic map

1. Brainstorm ideas: a dynamic map

2. Define different status of micro-compositions using color coding

2. Define different status of micro-compositions using color coding

3. Add narrative panel and change to a dark background to suit the dim environment during the performance

3. Add narrative panel and change to a dark background to suit the dim environment during the performance



I was the only designer and programmer in the team. There was a debut scheduled early June, left me one month to design and realize my design so that I can test the idea with real-world users. With time and resource constraints in mind, I programmed a web prototype within one month working in an agile environment. It was a progressive web application (PWA) and audiences can access it on their mobile phones during the performance. 


Premiere - User Test

The premiere of Climb! took place in a classical recital hall as part of a regular series of contemporary music recitals at the University of Nottingham on June 8th. Maria performed the work in front of around 40 audiences. The audience was a mixture of regular attendees for the event and those who had been invited by the research team. The regular attendees had a specific interest in contemporary music. Other attendees had much more diverse (mainstream) musical preferences, but most attendees reported at least “good” expertise in music.

The experiment was designed as two run in the premiere to compare audience experiences. In the first round, we gave audiences minimized information about the performance with the mobile application, while in the second round we offered a vocal introduction about the performance including its unique structure and special settings, and introduce how the mobile application connected to the performance. We gave out paper questionnaires after each round of performance and interviews were conducted on five selected audiences. In two performances the musician chose different paths.


Feedback Analysis

One of my team members, Chris, conducted data analysis on quantitive data collected from questionnaires and I was responsible for extracting information from qualitative ones. 83% (34/41) people used the app during the first performance and 74% (29/39) during the second.

Based on the analysis result on quantitative data, the app was seen as useful and an enhancement (4/5) as far as it went, although there were many suggestions for how it might be extended. Here we presented several quotes collected from our audiences in the premiere. After reflecting on analysis results and further discussions with Maria, more design insights are brought up regarding to the mobile application.



Design Iteration

Based on users' feedback, I made a revision on the last design. Here are some highlights.


Digital Program Note

Integrate a digital program note into the application. Audiences with various past experience in music can choose to check information that they are interested.


Composition Overview

Offer an overview of the performance structure so that audiences get a sense of how the performance would progress.


Journey Summary

The system documented the narratives that audiences went through and form a complete story for audiences to review at the end of the performance.

Performance Comparison

Performances are likely to be different due to the special design of the performance. The feature of performance comparison would be interesting to audiences who repeatedly go back and listen to them.



The Trap of Innovation

It is difficult to convince users to try out something new while keeping them comfortable and not scared. I found that telling a good story and listing examples help release user's nerve. Also, people tend to be nice and avoid criticism so that they don't break your innovative heart. This factor should be taken into consideration while reflecting on the design. Combining both quantitative and qualitative data gives a more subjective view. Additionally, since there is not much experience that we can learn from other people, we need to get prepared for any unforeseen issues either resulted form users or technical perspectives to come to a success.


Even if many users like the design, an issue raised up was that as described by one user: "I felt being distracted from the performance itself with the mobile application. So I gave it up after opening it for three minutes". It is a challenge, for individuals, to balance attention between the digital world and the real world. And it is a question for designers to assist, or against users to achieve that. More user tests are needed to modify the design and incorporating theories with regard to human behavior would help with the design.


Inspired by some previous research, I am interested in adding interactive features for audiences. For instance, the performance might not be determined solely by the performer, but also audiences: they may vote for future direction, or perform with the performer together in some way. In this way, they will be more engaged and feel as part of the performance. Also, maybe more distracted :|