S'i' fosse suono
A video mosaicing of audio self-portraits
The idea came from Andrea Cera as a result of a discussion on how to communicate the essence of the project http://skatvg.eu/ through an audiovisual, possibly interactive, composition.
Then the idea evolved and took the form of an interactive checker board, with a face per square cell. The visitor, upon touching a square cell, would trigger video playback of vocalizations, with a soundtrack that is randomly chosen between three different options:
- Original audio of the vocalization;
- Physically-based synthetic sound produced with the Sound Design Toolkit (http://soundobject.org/SDT/);
- Synthetic sound produced with granular/concatenative techniques.
Andrea Cera took a recording of Davide Rocchesso, cleaned up the original audio, and produced a synthetic version of the proposed vocalization.
The synthetic sound is made by using the Sound Design Toolkit, namely 4 different combustion engines, 1 friction model, 2 instances of the bubble model, and 1 wind/cavity model.
The questions for this step where:
For question 1 we thought about a good background for the checker board composition. At the first floor of our lab we have a large blackboard that woul provide a black background. By putting on it a white foam board we created the white background. We positioned two seats, one in front of the black background, the other in front of the white background. Additional seats where positioned around those, just to attenuate the ringing reverb due to multiple sound reflections between floor and ceiling. Subjects have been miked with a Beyerdynamic MC840 condenser microphone, with a hyper-cardioid pattern and low-frequency rolloff. Subjects have been illuminated with two studio lamps with diffusive umbrellas. One light was positioned on the front-right of the subject. The other light was positioned on the left-back of the subject.
The answer to question 2 came naturally out of a collaboration that we have had with Cantiere Teatro Ca' Foscari for production of "Nozze di Sangue" by Federico Garcia Lorca (http://www.unive.it/nqcontent.cfm?a_id=187720). Most of the students/actors and of the crew enthusiastically came and recorded two vocalizations, one on the black and one on the white background. Following a suggestion from Andrea Cera, we asked them to produce a sound of a few seconds that would represent them. Speech, singing, or imitations of musical instruments were not allowed. Each participant signed a declaration, accepting to be filmed and to appear in an audio-visual installation, also possibly on the web.
The video recordings were cropped in size to make them square (1080x1080) and cropped in time to cut silent tails before and after the vocalization.
Since the beginning, we thought that the audio-visual composition should be installed on a touch screen, to have maximal directness between action and feedback. The visitor should immediately feel the ability to play with the installation as if it was an instrument.
Large touch screens are expensive. We found a reasonable compromise between size and price with the HANNS-G HT271HPB 27” touch monitor (http://www.hannspree.eu/en/monitors/ht-series/ht271hpb). It has FULL HD 1920 x 1080 resolution and gives ten active contact points under Windows 10.
Other strategies to detect "touch" have been discarded after trying touch emulation with the Leap Motion sensor. As compared to a real touch screen, solutions based on computer vision are definitely slower and less accurate.