[-empyre-] Welcome to Week 3: New Year:/New Tools and Technologies in performance based work

Wed Feb 25 12:02:23 AEDT 2015

Hi Renate

The technical specification for Crosstalk is as follows:

Computers are all Mac OS.

Voice and speech recognition is done using Nuance's Dragon software. This allows the generated text to function across all compliant applications on the computers.

There are two computers doing speech recognition, one for each performer. There is another computer managing the sound, so three in all.

3D sensing of the performers is done with a pair of Kinects.

Speech and other audio is acquired using head-microphones (we have used bone-conductive systems to do this - which means the sound is acquired from the skull and jaw of the performer, so as to minimise feedback from environmental sound and ensure accurate speech recognition).

All of the other software we are using is written by the team (Garth, the sound oriented stuff, and myself, the grammar systems and visual, as well as a programmer called Hadi Mehrpouya, who wrote the physics engine). The software is mainly written in Processing with some C++. The system employs a custom physics engine, which is controllable through voice command. The performers can do things like move the virtual camera, that determines point of view for the projections that envelope them, from one of their heads (a dynamic first person POV) to the other performer's head, or placing it anywhere in space that they wish (third person). They can turn gravity on and off and make graphical and textual objects return to the locations where they were created, amongst other functions.

Everything the performers say (except special command words) appear as 3D live-text (editable) objects in the 3D space that surrounds them on the walls of the space. The text objects are individually interactive, within the physics engine, with the performers and each other. They will move when interacted with, responding just as they would in real-space if they were real objects. Much of the time gravity is turned off, so they easily fly around the space, bouncing off performers and each other. With gravity on they fall to the floor. When the texts touch each other they use an interpretative grammar engine (coded by us) to 'read' one another and form a grammatical index of the other text. They then employ a generative grammar engine (again, custom) to rewrite themselves, swapping elements with the other text object. This creates an ever evolving textual environment. The performers take their directions from the texts, so they are part of this recombinant process and thus a key part of the computational system.

The sound system uses a Kyma Pacarana unit for advanced sound processing and granular synthesis with surround sound output. All sound heard in the space originates with the performers (speech and sound of movement) but is heavily processed. Words the performers speak can trigger specific sound events, as do their physical movements. Performers can move sounds around the space with a sweep of their hand, for example, or set certain words as memorised samples at specific locations, so that when they go back to that location the sample is played (a kind of memory theatre effect). They can 'squeeze' sounds with the arms or legs and achieve all sorts of effects, as they wish.

Not sure if I'm forgetting anything. It's all pretty complicated, really. Garth and Sue are on empyre so might want to add something?

best

Simon

> On 24 Feb 2015, at 13:41, Renate Terese Ferro <rferro at cornell.edu> wrote:
> 
> ----------empyre- soft-skinned space----------------------
> Thanks Simon for sharing your work especially Crosstalk. I am wondering if
> there are other folks on empyre that also construct augmented
> environments.  It would interesting to hear a bit more about the
> logistics.  We have  been focusing this month on some of the tools and
> technologies that enable our conceptual ideas.  Simon would you mind
> giving us a cue as to how all this works.  I am particularly curious about
> how the voice and the interpretative language systems work.  I presume the
> engine is programmed or relies on a coded system.  Curious if your are
> using any proprietary software or something that you have innovated?  Just
> giving us a few more logistics would be interesting for our subscribers
> who may also use similar systems.  Best.   Renate
> 
> 
> 
> On 2/22/15, 11:17 PM, "Simon Biggs" <simon at littlepig.org.uk> wrote:
> 
>> ----------empyre- soft-skinned space----------------------
>> Sue Hawksley (choreographer), Garth Paine (composer) and myself (media
>> artist) have completed a couple of works in this area, most recently
>> Crosstalk.
>> 
>> Documentation can be found here:
>> http://www.littlepig.org.uk/installations/crosstalk/index.htm
>> 
>> Crosstalk is a collaborative interactive performance work for two
>> performers whose movement and speech are re-mediated within an augmented
>> environment employing real-time 3D motion tracking, multi-source voice
>> recognition, interpretative language systems, 3D visualisation employing
>> a custom physics engine, large scale projection and surround-sound audio
>> synthesis. The acquired speech of the dancers is re-mediated through
>> projected digital display and sound synthesis, the performers causing
>> texts to interact and recombine with one another through their physical
>> actions. The elements in the system all effect how each adapts, from
>> state to state, as the various elements of the work - people, machines,
>> language, image, movement and sound - interact with one another.
>> Crosstalk reveals social relations, as articulated in language acts of
>> various kinds (the performative), as a generative ontology of self-hood:
>> an assemblage for 'making people'.
>> 
>> In respect of technological inspiration, the work borrows from Alan
>> Turing's concept for the original Turing Machine. Turing wrote:
>> 
>> ..an unlimited memory capacity obtained in the form of an infinite tape
>> marked out into squares, on each of which a symbol could be printed. At
>> any moment there is one symbol in the machine; it is called the scanned
>> symbol. The machine can alter the scanned symbol and its behavior is in
>> part determined by that symbol, but the symbols on the tape elsewhere do
>> not affect the behavior of the machine. However, the tape can be moved
>> back and forth through the machine, this being one of the elementary
>> operations of the machine. Any symbol on the tape may therefore
>> eventually have an innings.
>> 
>> Alan Turing, 1948, "Intelligent Machinery." Reprinted in "Cybernetics:
>> Key Papers." Ed. C.R. Evans and A.D.J. Robertson. Baltimore: University
>> Park Press, 1968. p. 31.
>> 
>> In Crosstalk the 'symbols' consist of people, their actions and the words
>> they utter, combined with the sounds, texts and images generated by the
>> system. These are all recursive elements that are all part of a feedback
>> loop, often composed of other feedback loops, which ultimately determine
>> the performance. The 'tape' is simply the duration of the performance in
>> the interactive space. The output of the system consists of texts, sounds
>> and the relational dynamics between all the elements, including the
>> performers. The audience can also be part of the system.
>> 
>> best
>> 
>> Simon
>> 
>> 
>>> On 23 Feb 2015, at 14:27, Renate Terese Ferro <rferro at cornell.edu>
>>> wrote:
>>> 
>>> ----------empyre- soft-skinned space----------------------
>>> Over the next few days I thought I would introduce a new thread and open
>>> it up to our subscriber list especially those of you interested in
>>> performance based work and technology.  Just this past week the
>>> choreographer William Forsythe was in residence at Cornell University;
>>> Bill¹s latest work revolves around the choreographic object.
>>> 
>>> Forsythe writes:
>>> A choreographic object is not a substitute for the body, but rather an
>>> alternative site for the understanding of potential instigation and
>>> organization of action to reside.  Ideally, choreographic ideas in this
>>> form would draw and alternative, diverse readership that would
>>> eventually
>>> understand and, hopefully, champion the innumerable manifestations of,
>>> old
>>> and new, of choreographic thinking.²
>>> 
>>> Several of his latest works use objects such as the 400 pendulums
>>> suspended on overhanging sleds in his piece Nowhere and Everywhere at
>>> the
>>> Same Time
>>> https://www.youtube.com/watch?v=59QRpcLgPOQ
>>> 
>>> 
>>> Or the robotic arms used in his piece Black Flags
>>> https://www.youtube.com/watch?v=NDVLfuQTafQ
>>> 
>>> 
>>> Additionally, Forsythe in collaboration with others at  Ohio State
>>> University created a motion databank which can be seen at the website
>>> http://motionbank.org/en/content/about
>>> 
>>> Hope those of you involved in somatic practices and technology will
>>> share
>>> some of the tools, technologies and conceptual ideas that are inspired
>>> by
>>> them with us over the next few day.s
>>> Renate
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Renate Ferro
>>> Visiting Assistant Professor of Art,Cornell University
>>> Department of Art, Tjaden Hall Office:  306
>>> Ithaca, NY  14853
>>> Email:   <rferro at cornell.edu <mailto:rtf9 at cornell.edu>>
>>> URL:  http://www.renateferro.net <http://www.renateferro.net/>
>>>     http://www.privatesecretspubliclies.net
>>> <http://www.privatesecretspubliclies.net/>
>>> Lab:  http://www.tinkerfactory.net <http://www.tinkerfactory.net/>
>>> 
>>> Managing Co-moderator of -empyre- soft skinned space
>>> http://empyre.library.cornell.edu/
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> empyre forum
>>> empyre at lists.artdesign.unsw.edu.au
>>> http://empyre.library.cornell.edu
>>> 
>> 
>> 
>> Simon Biggs
>> simon at littlepig.org.uk
>> @_simonbiggs_
>> 
>> http://www.littlepig.org.uk
>> http://amazon.com/author/simonbiggs
>> 
>> simon.biggs at unisa.edu.au
>> Professor of Art, University of South Australia
>> http://www.unisanet.unisa.edu.au/staff/homepage.asp?name=simon.biggs
>> 
>> s.biggs at ed.ac.uk
>> Honorary Professor, Edinburgh College of Art, University of Edinburgh
>> http://www.ed.ac.uk/schools-departments/edinburgh-college-art/school-of-ar
>> t/staff/staff?person_id=182&cw_xml=profile.php
>> 
>> _______________________________________________
>> empyre forum
>> empyre at lists.artdesign.unsw.edu.au
>> http://empyre.library.cornell.edu
> 
> _______________________________________________
> empyre forum
> empyre at lists.artdesign.unsw.edu.au
> http://empyre.library.cornell.edu

Simon Biggs
simon at littlepig.org.uk
@_simonbiggs_

http://www.littlepig.org.uk
http://amazon.com/author/simonbiggs

simon.biggs at unisa.edu.au
Professor of Art, University of South Australia
http://www.unisanet.unisa.edu.au/staff/homepage.asp?name=simon.biggs

s.biggs at ed.ac.uk
Honorary Professor, Edinburgh College of Art, University of Edinburgh
http://www.ed.ac.uk/schools-departments/edinburgh-college-art/school-of-art/staff/staff?person_id=182&cw_xml=profile.php