Category Archives: Gestural Sound

Story Telling Space

Gestural Sound, Movement

as

Story Telling Space

This research project is an in depth investigation for the realization of a system, able to combine gesture and vocal recognition for interactive art, live events and speech based expressive applications. Practically, we have created a flexible platform for collaborative and improvisatory storytelling combining voice and movement. Our work advances both conceptual and technical research relating speech, body, and performance using digital technologies and interactive media.

We create an immersive audio-visual Story Telling Room that responds to voice and sound inputs. The challenge is to set up an efficient speech feature extraction mechanism using as good microphone conditions as possible. Leveraging the TML’s realtime media choreography framework, we map speech to a wide variety of media such as animated glyphs, visual graphics, light fields and soundscapes. Our desiderata for mapping speech prosody information centre on reproducibility, maximum sensitivity, and nil latency. Our purpose is not to duplicate but to supplement and augment the experience of the story that is being unfolded with a performer and audience in ad hoc, improvised situations using speech and voice for expressive purposes.

[wpsgallery]

 

We are exploring the possibilities of Natural Language Processing in the context of live performance. As the performer speaks the system analyzes the spoken words and with the help of the Oxford American Writer’s Thesaurus (OAWT), each semantically significant lexical unit initiates its own semantic cluster.

As the story is being unfold by the performer the environment is shifting from a state to another according to the censoring data that occur from the system analysis. Furthermore, we are exploring the possibilities of transcribed text from spoken utterances. The spoken words of the performer already burry a communicative value as they already have attached to them a semantic component that has been evolved and transform through out the history of language. The objective of this prototype is to see what happens when light, imagery and texture is added in the text and how it is perceived by the performer.

The pieces of text when encountered as audio-visual forms animated by quasi- intelligent dynamics in digital media become widely perceived as animate entities. People tent to regard animated glyphs as things to be tamed or played with rather than a functional and abstract system of communicative symbols.

[youtube]http://www.youtube.com/watch?v=tLPw1WjHoic[/youtube] [youtube]http://www.youtube.com/watch?v=TyF_-m6RWkY[/youtube]

TECHNIQUE

We have create two stand-alone Java application that perform the Speech Recognition and Speech Analysis tasks. For the mapping techniques we have used partially the already existing ozone state engine and we have create a new state engine in MAX/ MSP for better results in the context of improvisatory story telling

COLLABORATORS

Nikolaos Chandolias, real-time Speech Recognition & Analysis, System Design
Jerome DelaPierre, real-time video
Navid Navab real-time sound
Julian Stein, real-time lights

Michael Montanaro, choreographer/director
Patricia Duquet, Actress
Sha Xin Wei, Topological Media Lab
Jason Lewis, Obx Labs

MORE INFO

Story Telling Space Documentation

 

WYSIWYG

Architecture, Gestural Sound, Movement

ss

WYSIWYG

WYSIWYG was an investigation of sonified soft materials that encourage playful interaction. The group was a diverse mix of artists, scientists and musicians from McGill University’s Input Devices and Music Interaction Lab and Concordia University’s Topological Media Lab. In the first phase of the project, a large, stretchy, light-sensitive square “blanket” was developed, which was shown at a public exhibition on October 31st 2006. At the show, visitors interacted with the interface by standing under it and lifting it up. The tension of the fabric was such that shapes and waves could be made, producing rich, multichannel sound. Detailed discussion of this installation can be found in the publication Mapping and dimensionality of a cloth-based sound instrument

In the second phase, a tapestry was designed and woven with conductive thread which was used to generate an electric field. At its public exhibition on July 18th, 2007, visitors could touch various parts of the tapestry to generate sound. The interplay of narrative image on the tapestry and the abstract sound associated with it encouraged discovery and experimentation.

 

The following overview is from the Topological Media Lab’s WYSIWYG page:

As an extension of the research work conducted with the Topological Media Lab (TML), Sha Xin Wei and his team are creating textile objects such as wall hangings, blankets, scarves, and jewelry that create sound as they are approached or manipulated. These sonic blankets can be used for improvised play. A phonetic pun on the old acronym for What You See is What You Get from the era of the Graphical User Interface, WYSIWYG (for wearable, sonic instrument, with gesture) draws on music technology, dance, children’s group games, textile arts, and fashion. Created first and foremost to sustain social play for people of all ages, WYSIWYG allows players to express themselves whether enjoying time in a park, dancing at a club, passing the time during a long car trip, or just playing at home.

The custom-designed digital instruments embedded in the cloth sample movement to transform ambient body movement and freehand gestures into new sounds or “voices” associated with a player or transmitted to other players in the vicinity.

When the project was launched in November 2006, the WYSIWYG team experimented with a prototype ”blanket” able to sense how it is handled. During the presentation, eight people manipulated this photo-sensitive blanket to produce a spatial sonic landscape. In July 2007, dancers performed a semi-choreographed movement improvisation around a 20’ suspended “tapestry” and a 6-foot “tablecloth” woven with conductive thread on a Jacquard loom by Joey Berzowska’s XS Labs.

Dancer Marie Laurier with 20’ sounding cloth woven by Marguerite Bromley during Ouija workshop. © 2007 Topological Media Lab.

Custom electronics by Elliot Sinyor, McGill University. © 2007 Topological Media Lab.

David Gauthier with capacitive proximity sensor in the form of a bird woven from conductive fiber. © 2007 Topological Media Lab.

Principal investigators: Sha Xin Wei, Marcelo Wanderley
Physical materials advisor: Rodolphe Koehly
Mechatronics, feature extraction: David Gauthier
Mapping, feature extraction: Doug van Nort
Sound instruments: Freida Abtan, David Birnbaum, Elliot Sinyor
Assistant project technical coordinator: Harry Smoak

Ubicomp

Gestural Sound, Movement

asvc

Ubicomp

Projecting live video modified by physically-models video texture synthesis, nuanced by the activity of passersby. The membrane was steel mesh, allowing people to see each other through the projected image.

[wpsgallery]

Tabletap

Gestural Sound, Movement

gestural

Tabletap

A performance choreographed around a chef and sonified objects: fruit, vegetables, meat, knives, pots and pans, cutting board and table.

Cooking*, the most ancient art of transmutation, has become over a quarter of a million years an unremarkable, domestic practice. But in this everyday practice, things perish, transform, nourish other things. Enchanting the fibers, meats, wood and metal with sound and painterly light, we stage a performance made from the moves(gestures) of cooking, scripted from the recipes of cuisine both high and humble…

A performance choreographed around a chef and sonified objects: fruit, vegetables, meat, knives, pots and pans, cutting board and table.

[wpsgallery]

Cooking*, the most ancient art of transmutation, has become over a quarter of a million years an unremarkable, domestic practice. But in this everyday practice, things perish, transform, nourish other things. Enchanting the fibers, meats, wood and metal with sound and painterly light, we stage a performance made from the moves(gestures) of cooking, scripted from the recipes of cuisine both high and humble. Panning features virtuosic chefs who are also movement artists, such as Tony Chong.

Within our responsive scenography system, every cooking process is transformed into an immersive multimedia environment and performance; A multi-sensory experience composed of scent(smell), light, video, sound, movement, and objects. Every process is experienced across many senses at once. The sizzling sound of hot oil, and the mouthwatering aroma of onion and garlic hits the audience within an audio-visual thunderstorm. At the very end, the audience is invited to taste a sample of the dish within the accumulated sonic environment.

The acoustic state evolves via transmutations of sound, light and image in an amalgam of, not abstract data, but substances like wood, fire, water, earth, smoke, food, and movement. Panning allows the performers modulate these transmutations with their fingers and ears and bodies – the transmutation of movement into sound, chemical reaction into sound, and sound into light and image.

Panning is the first part in a series of performances exploring how everyday gestures/events could become charged with symbolic intensity.

[vimeo]http://vimeo.com/51474504[/vimeo]

[vimeo]http://vimeo.com/42057497[/vimeo]

Materials:

Self contained responsive kitchen set embedded into our portable table8.1 speaker system, projectors, food

Meteor Shower

Gestural Sound, Movement

a1

Meteor Shower

Meteor Shower was initially built as a simple responsive environment, its next incarnation it will [TM1] incorporate state-aware behaviour, and further explore ideas of nature/artifice by building narrative structures involving “lunar characters.”

As deployable installation, Meteor Shower holds potential as an environment for architectural installations, play spaces, and performance events – it is being designed with such flexibility in mind.

[wpsgallery]

TECHNIQUE [SOFTWARE]

PEOPLE

Sha Xin Wei – concept & meta-physics
Jean-Sébastien Rousseau – video and particle programming
Timothy Sutton – sound design and programming
Emmanuel Thivierge – state evolution and video feature extraction
Louis-Andre Fortin – visual design and programming
Freida Abtan – sound and systems design advisor

Hubbub

Gestural Sound, Movement

[instant_gallery]

Hubbub is one application of TML research treating speech as a computational substance for architectural construction, complementary to its role as a medium of communication.

Success will be measured by the extent to which strangers who revisit a Hubbub space begin to interact with one another socially in ways they otherwise would not. Hubbub is a part of a larger cycle called URBAN EARS, which explores how cities conduct conversations via the architecture of physical and computational matter.

Hubbub installations may be built into a bench, in a bus stop, a bar, a cafe, a school courtyard, a plaza, a park. As you walk by a Hubbub installation, the words you speak will dance in projection across the surfaces according to the energy and prosody of your voice. For example loud speech produces bold text, whispers We’ll capitalize on recognition errors to give a playful character to the space.

HYBRID ARCHITECTURE AND HABITATION OF URBAN SPACE

In this street-scale research thread, we investigate how people build, destroy, modify and inhabit city environments using embedded computational systems. The first part of this study is social and historical, employing methods of field observations as well as insights from phenomenological and anthropological studies. We intend to combine this work with insights of colleagues from the domains of urban design and architecture to design computer-mediated, responsive environmental systems for urban space.

The HUBBUB research series presents an foray in this domain of urban responsive architecture. As you walk through a Hubbub space, your speech is picked up by microphones, your speech is partially recognized and converted to text. Associate text is projected onto the walls, furniture and other surfaces around you as animated glyphs whose dancing motion reflects the energy and prosody of your speech. Hubbub is an investigation of how accidental and non-accidental conversations can take place in public spaces, by means of speech that appears as glyphs projected on public surfaces. The installation takes its meaning from the social space in which it is embedded, so its “function” depends on the site we select. Some of the technical issues concern realtime, speaker-independent, training-free, speech recognition; realtime extraction of features from speech data; multi-variate continuous deformation and animation of glyphs in open-air public display systems, such as in projection or in LED displays. We will investigate how embedding such responsive media as the Hubbub speech-painting technology as well as TGarden technologies into the urban environment can modestly support rich, playful forms of sociality.

TECHNIQUE [SOFTWARE]

2003 Architecture. We use a custom speech recognizer which can recognize continuous speech, independent of speakers. Moreover, this speech recognition application uses the Windows SAPI engine which allows us to word-spot for a restricted vocabulary and avoid training. This way anyone in a language group can freely speak without first preparing the software system. We have developed a new portable animation system called Commotion which supports some kinetic text animation in general OpenGL and Objective-C GNUStep open source environments. In parallel we use MAX/MSP to perform feature extraction on the speech and use the features to send animation hints to Commotion.

PEOPLE

Vincent Fiano, Commotion animation system.
Stephen Ingram, Word-spotting, grammar-driven speech recognition system.
Graham Coleman, MSP speech feature extraction and Max animation choreography.