Lucidity: A Movement-based Interactive Dance Performance
Click here to download pdf version. (91kb)

by Jodi James
jodi.james@asu.edu

Todd Ingalls
todd.ingalls [at] asu.edu

Gang Qian
gang.qian@asu.edu

Loren Olson
loren.olsen [at] asu.edu

Thanassis Rikakis
thanassis.rikakis@asu.edu

Arts, Media and Engineering Program
Arizona State University
http://ame.asu.edu

Keywords  
Interactive dance, movement analysis, real-time visual and audio feedback, 3D, animation, context-aware computing, interaction design, live performance, system design

Abstract
Movement-based interactive dance has recently attracted great interest in the performing arts. While utilizing motion capture technology, the goal of this project was to design the necessary real-time motion analysis engine, staging, and communication systems for the completion of a movement-based interactive multimedia dance performance. The movement analysis engine measured the correlation of dance movement between three people wearing similar sets of retro-reflective markers in a motion capture volume. This analysis provided the framework for the creation of an interactive dance piece, Lucidity, which will be described in detail. Staging such a work also presented additional challenges. These challenges and our proposed solutions will be discussed. We conclude with a description of the final work and a summary of our future research objectives.

	Figure 1: Section three "Lucidity" from same titled work Copyright © Loren Olson

Introduction
Motion capture systems are a highly complex but useful tool designed to function in a variety of ways. The development of such technology in the past two decades has opened up many possibilities for advanced digitization and analysis of movement. The fields of biomechanics, motor control and computer vision use motion capture to investigate quantitative nuances of human movement, in order to understand such principles as how the brain organizes itself to grasp an object or how to calculate the center of rotation of the knee or shoulder joint. Motion capture is also becoming increasingly popular within the film and video game industries in order to animate more realistic physical actions such as playing football, practicing martial arts or skateboarding. Most recently, motion capture technology is being integrated into art-making practices, particularly choreography and interactive-dance. In this latter case, real-time motion analysis generates immediate image and sound composition and synthesis to create unified platforms for interactive, multi-modal environments.

One recent area of application of our work in interactive art was the project motione. This undertaking involved the creation of three new pieces for dance and interactive multimedia using motion capture and analysis systems developed by our research in movement analysis and recognition. Two of these works involved internationally renowned guest artists. The first of these was a dance work, 22, choreographed by Bill T. Jones with interactive music composed by Roger Reynolds and interactive imagery by Paul Kaiser, Marc Downie, and Shelley Eshkar. The second work, how long does the subject linger on the edge of the volume…, was choreographed by Trisha Brown with music by Curtis Bahn, and interactive imagery, again, by Paul Kaiser, Marc Downie, and Shelley Eshkar. Both of these pieces premiered at Arizona State University’s (ASU) Galvin Playhouse on 9 April 2005. The interactive version of how long does the subject linger on the edge of the volume… was also staged at the 35th anniversary season of Trisha Brown Dance Company at Lincoln Center Rose Theater and both were just more recently performed at the Monaco Dance Forum in December 2006.

The third work, Lucidity, was created in response to our work with motione. Lucidity, choreographed by Jodi James, with interactive music composed by Todd Ingalls, and interactive animation created by Loren Olson, premiered at ASU's Galvin Playhouse on 10 April 2005. This paper will discuss our own explorations, challenges and discoveries in creating Lucidity and producing it for a traditional proscenium theater setting.

Overview of System
The main sensing infrastructure was provided by a 16 camera optical motion capture system made by Motion Analysis Corporation. This system allowed for real-time capture of multiple movers with this data then being multicast over a local network.

A recognition engine was developed to extract group movement features of multiple dancers from this motion capture data, particularly proximity between dancers; correlation in translational velocity among the dancers through space; grouping of dancers (e.g. pairs or trios) and correlation of limb trajectories to determine similarity of movement vocabulary. This real-time analysis was then multicast to the visual and audio rendering engines. Further discussion of these algorithms can be found in [1], [2], [3], and [4].

A custom built piece of software, designed specifically for interactive environments, called Dash, was utilized as the animation engine. It connects to the motion capture system to receive real-time marker position data while also receiving the real-time multicast motion analysis stream that recognizes the targeted movement vocabulary. For Lucidity, the visual effects were created from the movement vocabulary that was then projected on a scrim in front of the stage.

The sound engine was built using a combination of custom-built software and two open source software projects. The main program was written using the Common Lisp environment OpenMCL. Custom extensions to this include code for dynamically creating GUI elements and a real-time scheduler for control of musical events. In addition, code from Common Music was used for additional algorithmic generation (the real-time scheduler and other code from this piece is now being integrated into the Common Music code). A real-time sound synthesis program, SuperCollider, was used to generate audio.

Staging Challenges
Motion capture is usually done in highly controlled environments with the goal to achieve the cleanest data for later offline processing and animation. Our project was a departure from this since the quality of the capture had to compete with several other considerations that are important in staging works in traditional theater settings. This required us to find a balance between the factors related to motion capture quality and the needs of lighting, projection, number of dancers, costuming, and the performance aesthetic of proscenium stage. This impacted many of our decisions such as what types of motion capture cameras to use, the cameras’ placement, the marker sets used for the dancers, and what type of data we were going to extract from the motion capture system.

The placement of cameras for motion capture in a theater space presented one of the most challenging problems in the staging of these works. The first concern was the question of camera location so that they neither disrupted the aesthetic for the audience nor impacted the projection on the downstage scrim. After experimenting with several configurations, we determined that hanging the cameras from a 20x30 foot grid suspended above the stage was the best solution. The ideal height for this grid for data collection was between 10 and 13 feet, however, this would have cut the vertical space in half, blocked the theatrical lighting, and bisected the front projection. The grid was raised to 22 feet, which removed its visual impact but placed the cameras at their outer effective distance boundary.

Originally we intended to hang the cameras on each of the four sides of the grid, but quickly realized that this was unworkable due to the side-lighting. This presented a real problem because if a camera saw any side-lighting, the image was literally “blown out” because of the light’s high intensity. Therefore, we had to carefully position the cameras so none of them looked into these sidelights. The solution was to place cameras only on the front and rear of the grid and place single cameras on each corner of the stage on 13 foot stanchions. Further complicating camera placement and resolution was that the cameras with the highest range emitted a distinctive red glow towards the audience. For aesthetic purposes, we decided to use infrared emitters on the cameras facing the audience since they give off no visible light. These camera emitters lower the effective distance of the camera by 1/3, thereby affecting the robustness of the captured data.

The angle of the cameras on the grid also had to be reconciled because the Marley dance floor naturally reflected both the infrared or near-infrared cameras as well as the overhead lighting. A great deal of time was spent determining the optimal cameral angle to avoid reflection. Finally, the speed of the system is affected by the number of markers in the capture volume which presented another challenge. Multiple movers wearing 30-40 markers each diminishes the system's ability to accurately capture and label data in real time. Therefore we established the minimum amount of markers and their optimal placement for capturing the essential elements of
the movement vocabulary. In this case, we focused on capturing only upper body movement since the markers would be closer to the cameras, thereby reducing the incidence of occlusion and enhancing the reconstruction process.

Due to the complexity of the choreography itself and the length of each work losing markers was a concern. Unlike other motion capture scenarios, once started, the performance could not be stopped in order to replace markers. Attempts to sew and permanently secure the markers to the material proved futile. Therefore the possibility of losing markers during the performance was an accepted if not tenuous component of this endeavor because the movement analysis was dependent on the integrity of the marker set.

The Creative Process
In what one might call ‘conventional’ choreography, the movement material is fairly fixed. This is how the choreographer began her process; relying on traditional compositional tools to build choreography for Lucidity. Several short movement sequences were created and then chained together to build longer phrases from which an even larger work would emerge. Under usual circumstances this is an acceptable means of creating choreography. However, after considerable frustration, the choreographer understood that utilizing traditional methods for creating movement material in this context would not work. Only improvisation would accommodate interactivity. Nevertheless our earlier investigations were not in vain. The original movement sequences were incorporated into the final renderings of Lucidity.

One stage of the creative process in building Lucidity was to rehearse while the motion capture system collected data. This served several purposes. First and foremost it allowed us to acquire preliminary motion capture data that could then be used at a later time to test the motion analysis engines and experiment with sound and visual feedback. For the dancers, it gave them an opportunity to grow accustomed to wearing markers while dancing and sort out any moments in the phrase material where marker location might be problematic. This process also allowed us to extract components of the movement material that we felt were particularly salient while simultaneously adapting or eliminating those which were ineffective for the motion analysis and interactivity.

The dancers themselves found the process both intriguing and frustrating. Very few dancers have the opportunity to participate in an endeavor such as this, and fortunately our dancers were quite willing to experiment within a realm of uncertainty. The dancers, on many occasions, lent us their suggestions to the creative process, motivated by their own curiosity about the technology itself and its potential for art-making. One crucial element in building an interactive dance work with motion capture is that of time. Fortunately, the dancers were able to devote one year this project. It takes time and exposure in an interactive environment to truly acquire an embodied understanding of how it works, how their movement affects the feedback, how to perceive the very environment they are creating through their movement and how to create a dance piece within that framework.

At the same time, working with motion capture technology can be wearisome. The preparation and calibration for a capture session takes a significant amount of time, particularly if there is more than one dancer which was true in this case. Therefore we would often schedule 3-4 hour rehearsals to allow for this, leaving the first hour for preparation. In addition, the interactive sound was not always mapped directly to their movement so the dancers frequently expressed that they could not sense how their movement was affecting the sound.  Finally the interactive images were projected onto a scrim at the front of the stage. Because dancers, in performance, are attending to the whole space around them rather than strictly to the front of the stage, they felt that they could not fully realize how their movement was affecting the visuals. Therefore they could not establish a relationship with this aspect of the interactive environment.

Integration of Sound, Visuals and Choreography
The inspiration for Lucidity became the sensations that particular dreams leave us when we wake in the morning, whether it is clarity, confusion, mystery, happiness, fear, and attempting to represent these sensations through movement.

In order to elicit interactivity between the dancers’ movements and the sound and visual environment, there must exist elements of improvisation. In this case, structured improvisation provided an underlying framework to the choreography while also allowing the dancers the opportunity to invent their own material and contribute their own personal uniqueness to the work. It also allows the performers to interact with and manipulate the very environment they are creating through their movement.

Aside from revealing the capabilities of our motion analysis engine we hoped to create the diversity in form necessary to provide a rich landscape for the sound and visuals that would accompany the work. In order to accomplish this, the choreographer placed particular emphasis on how to organize the dancers in space, how they would occupy it, and travel through it as well as how they would relate to each other. The choreographer also concentrated on accentuating the dynamics in the movement itself, interlacing stillness with sudden and sustained phrasing.

Section one, titled Oppression, focused on those dreams we wake from with feelings of anxiety, oppression, fear, or torment. This section is comprised of a duet and a solo. In the beginning of this section the duet is hugging the wings on opposite sides of the stage. They are quite distanced from the soloist who is situated at center stage. Over the course of this section, this space slowly collapses, as the two dancers move closer and closer into the center, eventually pushing her to the ground, representing the domination and suppression of the individual spirit. Accompanying this movement vocabulary is an interactive animation that draws a chaotic image of swirling lines that appear to circle the dancers on stage. As the dancers move closer together, the lines swirl faster and faster, and draw together in a tighter pattern of movement. Adding to this sense of foreboding was an interactive sound score which uses a complex network of frequency modulation synthesis and delay lines to create a heavy texture. The proximity of the dancers and their velocity characteristics were used to affect the density of events and length of delay lines to create a more dense and oppressive sonic environment.

Section two, titled Ambiguity focused on those dreams we wake from with feelings of confusion, obscurity, mystery, and uncertainty. The dancers in this section begin by creating a line upstage, spread out across the back. They remain with their backs always to the audience. They never reveal their identity. The animation for this section is based on a 3D curved surface swept out by the movement of the dancers on stage. The color of the surface is a gradient, based on the velocity profiles between dancers determined by the analysis. The surface slowly dissolves away as it gets older, creating a colorful swirling, moving pattern that follows the dancers. The sounds in this section are mostly reminiscent of some type of bowed instrument. The pitch content was driven by a simple genetic algorithm whose fitness function was affected when the dancers moved on the floor and also by individual movement velocities. The overall affect was of a wondering texture that began at a high range and decreased in register over the course of the section.

The final section, titled Lucidity focused on those dreams we wake from with feelings of resolution, clarity, and lightness. It is comprised of two parts. The first part is set and retrogrades or reverses, in a sense, sections one and two. Metaphorically this represents an unwinding, or unraveling, of the confusion of the two previous sections. The second part begins with all three dancers moving, for the first time, in unison and is performed as choreographed to suggest harmony and self-revelation. The piece resolves with a pool of light falling around the clustered dancers and the stage fading to black. The unwinding of the first two sections and the simplicity of the material performed in unison represent to me the resolution, clarification and transcendence that can arise from dreaming – how often very complicated life issues can reconcile themselves in our unconscious. These concepts are mirrored in the visuals, which create a complex network of nodes. Every few seconds a history node is created that saves the values of the analysis at that point in time. A 3D shape is drawn for the node, based upon the position of the dancers. The node then establishes connections to previous history nodes that have similar attribute values. Connections are visualized by twisting, vibrating curves that link the two nodes. In addition, nodes pull similar nodes gently towards themselves, setting the whole network into motion (Figure 1, see page 1). This section has a more rhythmic drive with an opening up of register, compared to the previous section, based on the dancers’ velocity characteristics or dynamics. Keeping with the theme of this final section, the music tended towards more clearer rhythmic material. Also unison movement was punctuated by high bell-like sounds.

Conclusion  
This paper describes the challenges, solutions, and creative and scientific methodologies for the creation of an interactive dance piece utilizing motion capture technologies. The framework for this creative endeavor was provided by a motion analysis engine that tracked the proximity, translation and body motion correlations, grouping and formations of three dancers in real-time. We discussed the challenges in staging and system communications that surfaced and how we solved them. Finally we detailed how the sound, visuals and choreography utilized the output from the movement analysis engine for the production of an interactive dance work and what these elements looked and sounded like for the final performance.

Nevertheless, we realize that this is just a beginning. Our research in movement analysis and recognition and interactive art-making will continue as we seek other paradigms for analyzing human movement, perceiving and understanding multi-modal environments and encouraging creative expression in human-computer interaction. More specifically, our future work will integrate Laban Movement Analysis, a system which provides a comprehensive vocabulary and structure for movement, as a framework for our motion analysis research. We will also incorporate the development and use of video-based movement analysis and a 22’ x 12’ pressure sensitive floor. The addition of these elements and modes of analysis will provide a more complete understanding of the dynamics of human movement while enriching the experience of creating interactive art work through these systems.

Acknowledgements
Work discussed in this paper was part of the motione project of the Arts, Media and Engineering Program through the Herberger College of Fine Arts and the Fulton School of Engineering at Arizona State University. It was presented by ASU Public Events with additional commissioning support from Lincoln Center for the Performing Arts. It was supported in part by the National Endowment for the Arts, the National Science Foundation (grant #0403428), Motion Analysis Corporation, Arizona Public Service (APS) and City of Tempe Cultural Services.

22
Choreography and performance — Bill T. Jones;
Interactive Imagery — Paul Kaiser, Shelley Eshkar, Marc Downie;
Composition and Sound Design — Roger Reynolds;
Composition and Sound Design Assistant — Pei Xiang
Set and Lighting Design — Robert Wierzel;
Costume Design — Galina Mihaleva;
Motion Capture and Analysis and Interactive Systems — Gang Qian, Todd Ingalls, Daniel Whiteley, Jodi James, Thanassis Rikakis, Loren Olson, Marc Downie, Curtis Bahn, Siew Wong;
Project Directors — Thanassis Rikakis, Colleen Jennings-Roggensack;
Choreographic Assistant — Janet Wong;
Assistant for Lighting Design — Greg Emetaz;

how long does the subject linger on the edge of the
volume...
Choreography — Trisha Brown;
Interactive Imagery — Paul Kaiser, Shelley Eshkar, Marc Downie;
Composition and Sound Design — Curtis Bahn;
Set and Lighting Design — Robert Wierzel;
Costume Design — Galina Mihaleva;
Motion Capture and Analysis and Interactive Systems — Gang Qian, Todd Ingalls, Daniel Whiteley, Jodi James, Thanassis Rikakis, Loren Olson, Marc Downie, Curtis Bahn, Siew Wong;
Dancers — Neal Beasley, Sandra Grinberg, Brandi Norton, Cori Olinghouse, Stacy Spence, Todd Stone, Katrina Thompson;
Choreographic Assistant — Carolyn Lucas;
Assistant for Lighting Design — Greg Emetaz;

References
1. G. Qian, F. Guo, T. Ingalls, L. Olson, J. James, T. Rikakis, “A Gesture-Driven Multimodal Interactive Dance System”, Proceedings of the International Conference on Multimedia and Expo, Taipei, Taiwan, China (2004).

2. T. Ingalls, T. Rikakis, J. James, G. Qian, L. Olson, F. Guo, S. Wong. “A Movement Recognition Engine for the Development of Interactive Multimedia Works”, AISB 2004 Convention: Motion, Emotion and Cognition — ConGAS Symposium on Gesture Interfaces for Multimedia Systems, Leeds, United Kingdom (2004).

3. D. Whiteley, G. Qian, T. Rikakis, J. James, T. Ingalls, S. Wong, L. Olson, “Real-Time Tracking of Multiple People from Unlabelled Markers and Its Application in Interactive Dance”, Proceedings of British Machine Vision Conference, Oxford, United Kingdom (2005).

4. J. James, T. Ingalls , G. Qian , L. Olson, D. Whiteley , S. Wong , T. Rikakis, “Movement-based Interactive Dance Performance”, Proceedings of SIG ACM Multimedia Santa Barbara, California, U.S.A. (2006).

Author Biographies
Jodi James is an artist and scientist, and attempts to bridge these diverse disciplines through her teaching and research. She is an assistant research professor in dance and computation with the Arts, Media and Engineering Program at Arizona State University. She participates in a number of the unit’s research groups including Interactive and Generative Systems for the Arts, Experiential Education for Children and Motion Sensing and Analysis.

Todd Ingalls is a composer and computer artist who has been working with interactive performance and installations for the past several years. He is an assistant research professor at the Arts, Media and Engineering Program at Arizona State University where he is an active researcher. He currently leads the Interactive Arts group and is heavily involved with the Biofeedback for Rehabilitation project.

Gang Qian is an assistant professor jointly with the Department of Electrical Engineering and the Arts, Media and Engineering Program (AME) at Arizona State University. Dr. Qian’s research interests include multimodal human movement sensing and analysis, computer vision, machine learning, and statistical signal processing and their applications in activity-based mediated experiential systems for interactive arts, education and rehabilitation. He has published over 30 peer-reviewed journal and conference articles in these areas. In 2006, he was elected the World Technology Network (WTN) Fellow in Entertainment for his contribution in applying state-of-the-art movement sensing and analysis technology in interactive dance performance.

Loren Olson is an associate research technologist for Multimedia in the Arts, Media and Engineering Program at Arizona State University. Loren is working on interactive visualization systems and chairs the AME Technology Panel. Before joining AME, Loren spent the previous 14 years creating animation and effects for film and television. Working as an animator and technical director, he has created imagery for projects ranging from feature films to commercial television spots, direct to home video childrens' programs to TV show opens. His awards include eight Rocky Mountain Emmy Awards. Loren received his B.S. in Computer Science from Arizona State University.

Thanassis Rikakis is professor and director of the Arts Media and Engineering (AME) Program at Arizona State University. His research work and publications are in the areas of computer mediated arts and experiential systems, interdisciplinary graduate education, biofeedback for rehabilitation, pitch perception, and computer music tools for arts education. His educational background is in music composition and computer music. He is principal investigator of a recently awarded NSF IGERT grant for interdisciplinary research and education in experiential media, Co-PI of a current NSF CISE Research Infrastructure grant for motion analysis and PI for the motione project that premiered in April 2005 and has received an NEA Technology: Resources for Change grant.

Citation reference for this Leonardo Electronic Almanac Essay

MLA Style
James, Jodi; Ingalls, Todd; Qian, Gang; Oldon, Loren; Rikakis, Thanassis. “Lucidity: A Movement-based Interactive Dance Performance.” “LEA-ACM Multimedia” Special Issue, Leonardo Electronic Almanac Vol. 15, No. 3 - 4 (2007). 10 May 2007 <http://leoalmanac.org/journal/vol_15/lea_v15_n05-06/jjames.asp>.

APA Style
James, Jodi; Ingalls, Todd; Qian, Gang; Oldon, Loren; Rikakis, Thanassis. (Apr. 2007) “Lucidity: A Movement-based Interactive Dance Performance,” “LEA-ACM Multimedia” Special Issue, Leonardo Electronic Almanac Vol. 15, No. 5 - 6 (2007). Retrieved 10 May 2007 from <http://leoalmanac.org/journal/vol_15/lea_v15_n05-06/jjames.asp>.