Proposed Research Threads
We describe some of TML’s principal inter-related areas of research: gesture and performance, realtime media choreography, active garments, responsive media, hybrid architecture.
Performance and Phenomenology of Gestures
We investigate what performance means at the fine scale of gestures inside a responsive media space. The approach is not typical of what is traditionally called performance theory, but synthesizes insights from activity theory, studies of material agency, user experience design, the study of writing, and other domains.
TGarden Projects. We investigate what meaningful, compelling gestures are possible with sensor-augmented experimental clothing modifying computer-generated fields of sound and video.
The series of TGarden projects has already yielded a path-breaking hardware / software / garment / media system – TG2001 – that has been deployed under harsh performance conditions at Ars Electronica and V2 Rotterdam. Since then, in 2001-2002, we have built a lab testbed for media choreography and have extended it to incorporate live synthesis of rich video transformations. Its elements been extended in the txOom responsive spaces (Italy, UK 2002) and a physical game system by other groups working in the public cultural sphere (FoAM, Belgium; Times Up, Austria).
(See videos of past TGarden installation experiments Siggraph New Orleans 7/2000, Medi@terra Athens Greece 11/2000, Ars Electronica Linz Austria 9/2001, V2 Rotterdam Belgium 11/2001, plus an extension, txOom, Future Physical 12/2002.)
Inspired by the on-going cycle of the TGarden responsive environments, we will focus on developing wireless and sensor technologies that work well embedded in clothing in socially marked ritual spaces. We investigate how gestures can be mapped in compelling ways to continuously varying fields of sound and video. We are particularly interested in how people collectively and individually habituate themselves to dynamically varying responsive media-fusion play environments without resorting to ordinary language.
Some of the technical questions concern the mechatronics of body-borne sensors and sensate clothing, un-tethered gesture and body tracking, wireless systems. Some of the most interesting challenges lie in the realtime statistical analysis and interpretation of improvised gesture.
Based on the TGarden testbed, the Topological Media Lab is extending the continuous evolution state engine as an infrastructure that can marshall, hint, coordinate the synthesis of rich video and audio material in realtime response to people’s gestures. At the same time, this authoring paradigm allows designers or composers of responsive spaces to give a potential shape to the dynamics and the aesthetics without mapping them into a procedural programming language. The key conceptual contribution here is to make the representations and transformations from topological dynamical systems usable in a fully controllable but not overly finely deterministic and metrized mechanism. The research is predicated on the conjecture that the “qualitative” but rigorous logic employed by designers of interactive media systems can be well-represented and even effectively computably represented by realtime systems using this topological approach. Preliminary results over the past year and a half encourage us to develop this approach into an authoring system for a greater range of designers and responsive media spaces. Of course interaction designers can control, or choreograph the media choreography via the authoring system, but an essential aspect of the TGarden is the distribution of agency to the inhabitant players and to autonomous computational or even material/physical processes for live, realtime expression. Usually this sort of relinquishing of control raises great chances of system failure due to unanticipated input, but our topological approach is also designed to produce qualitative effects robustly in the face of perturbations of input. Indeed, this sort of robustness plus richness of behavior is one of the strategic long-term research interests of the TML. In sum, our experimental approach to managing complex media logics and environments extends to the interaction designers as well as inhabitants a way to manipulate media using dynamical metaphors that are rich and simultaneously operational.
This work is substantively informed by deep expertise derived from experimental theater, performance and dance.
Softwear: Responsive Garments as Expressive Instruments
Another research project evolving from the TGarden project is the construction of sensate and active garments. The active garments research capitalizes on engineering advances in sensor technology, wireless technology and experimental textiles. However, beyond the wearable technology, this work depends critically on ideas from experimental performance, costume design and choreography, and most importantly, on live musical performance.
In fact, this work capitalizes on the past century of expertise and rich design intuition from the domain of live electronic music and electronic instrument design. We continue our collaboration with J. Ryan and M. Waisvisz and colleagues in STEIM and other institutions for electronic music and sound (e.g. CNMAT at University of California Berkeley, and Sony Computer Science Laboratory in Paris). This work also builds on collaboration between the Topological Media Lab and experts in the domain of active fabrics. A recent result is the TCostume garment with sensate liner provided by Prof. S. Jayaraman that conducts signals from TGarden sensors along signal-conducting fiber embedded into the garment to the TGarden wireless computer.
In the domain of active textiles, we are collaborating with J. Berzowska at Concordia who is one of the pioneers in active material and aesthetic computation and media. With such techniques we propose to create a set of expressive, symbolically charged garments that can be tested singly or in conversational settings.
This represents the first of several technical stages of the wearable instruments initiative:
(1) Embed current TGarden sensors and wearable gear using conductive fiber or cable on custom costumes.
(2) Experiment with alternate garment body affordances and constraints, designed with alternate sensors, gesture-tracking, maps to sound and video processes.
(3a) Adapt realtime visual/audio synthesis system to emissive, electronically addressable imaging fabric-based materials, and establish new designs for such materials.
(3b) Adapt realtime visual/audio synthesis system to kinetic, electronically deformable fabrics, and establish new designs for active materials.
(4) Stage integrated experiments with multiple people wearing active garments in responsive environments.
These garments are designed as instruments that can work alone, but come alive in most compelling patterns in the presence of TGarden-based environments. HYBRID ARCHITECTURE AND HABITATION OF URBAN SPACE
In this macro-scale research thread, we investigate how people build, destroy, modify and inhabit city environments using embedded computational systems. The first part of this study is social and historical, employing methods of field observations as well as insights from phenomenological and anthropological studies. We intend to combine this work with insights of colleagues from the domains of urban design and architecture to design computer-mediated, responsive environmental systems for urban space.
The HUBBUB research series presents an foray in this domain of urban responsive architecture. As you walk through a Hubbub space, your speech is picked up by microphones, your speech is partially recognized and converted to text. Associate text is projected onto the walls, furniture and other surfaces around you as animated glyphs whose dancing motion reflects the energy and prosody of your speech. Hubbub is an investigation of how accidental and non-accidental conversations can take place in public spaces, by means of speech that appears as glyphs projected on public surfaces. The installation takes its meaning from the social space in which it is embedded, so its “function” depends on the site we select. Some of the technical issues concern realtime, speaker-independent, training-free, speech recognition; realtime extraction of features from speech data; multi-variate continuous deformation and animation of glyphs in open-air public display systems, such as in projection or in LED displays. We will investigate how embedding such responsive media as the Hubbub speech-painting technology as well as TGarden technologies into the urban environment can modestly support rich, playful forms of sociality.
References
Hubbub, http://www.gvu.gatech.edu/people/sha.xinwei/topologicalmedia/hubbub
Melting Pot or Fruit Salad: Experiment in Autonomously Producing Culture, in preparation for CODE volume, 2003. (PDF)
Pliant Research, http://www.pliant.org
Resistance Is Fertile: Gesture and Agency in the Field of Responsive Media, Configurations 10.2, 2002 (PDF).
Sauna, http://hexagram.concordia.ca/~xinwei/sponge.org/projects/m3_sauna_intro.html
TGarden Media Choreography System, with Y. Visell and B. MacIntyre, 2003 (Technical Report).
TGarden Performance, http://hexagram.concordia.ca/~xinwei/sponge.org/projects/m3_tg_intro.html
Sha, Xin Wei, Michael Fortin, Navid Navab, Tim Sutton, “Ozone: Continuous State-based Media Choreography System for Live Performance,” ACM Multimedia, October 2010, Firenze (http://hexagram.concordia.ca/~xinwei/papers/texts/ACMMM10/acmmm2010_art06601-sha.pdf)
Sha, Xin Wei, Michael Fortin, Jean-Sebastien Rousseau, “Calligraphic Video: A Phenomenological Approach to Dense Visual Interaction,” ACM Multimedia, October 2009, Beijing (http://hexagram.concordia.ca/~xinwei/papers/texts/ACMMM09/art1231-sha.pdf)