Oralcle - Devilish Fortunes
Live AV Performance at 370J Garage

Performed live at the 370J Garage venue, as part of NYU ITP/IMA Programs “Designing Interfaces for Live Performance”, event can be viewed here.
All visuals, sculptures, instruments, and hardware were made by me except for the LeapMotion 2, which I connected via Touchdesigner.

Vertical poster designed and riso printed by me, landscape poster designed by David Liu.

Before this performance, we actually had a midterm which was building up to this final performance, and I decided to theme my project around the visuals of a rotary telephone transformed into a midi controller, as well as a midi instrument stuffed animal. This was my first time using like “arms” as a gestural piece, and it lead into me switching from a teddy bear into proper hands, both physically and also with gestural elements from the leap motion. You can see the teddy bear as well as my performance piece below. I did not enjoy speaking or singing live for an audience... so that was removed for the final, but if I knew someone more confident in singing then im sure they would love the telephone!

Back to this final performance, it features multiple moving parts, including a peppers ghost illusion, two individual projections, a video playing inside of the sculptures crt, three midi controllers (L Hand, R Hand, and Leapmotion 2 using GECO midi software). The midi controllers send data to both touchdesigner, which then alters some sine waves for modulation purposes with LFO, and Ableton, which features a live performance that I composed in Ableton Live that has the midi controlling pitch, LFO, some notes, and other aspects of performance. I used serum to create the backings and leads for the song.

The main focus of the performance is on the custom interfaces however, which uses the Leapmotion 2 sensors to track my distance, rotation, and other placements of my hands in order to control the performance, and then tracks are played and controlled in addition by the two physical hand controllers that are hooked up to 2 Arduino Nano 33 IoTs, with FSRs and Flex sensors on the fingers allowing me to press down on the silicone hands nails and bend the fingers to modulate aspects of the sound. The silicone is from a hand casting I did on my hands, and then flipped inside out and reversed with the hands to get the skin texture, and are physically soldered onto the hand.

The background visuals backlighting the sculpture are from a chladni figure altered to have more vertical lines, which are responsive with shape and color depending on the backings that accompany them. On the physical sculpture, I have it tracked to the distance and rotation of my own physical hands, which uses points on a sphere to build together an image that is on the sculpture. It switches between three different visuals that are projected onto the sphere, and then obscured/revealed throughout the song.

From there, the hands also use a sprinkle node from touchdesigner, which is applied to a 3D model of a hand that rotates on the screen, and is then put into a peppers ghost illusion. This orb is also where I place the sensor for the LeapMotion 2, and the hands are placed on its sides. I was completely out of I/O ports at this point on my laptop, so I was forced to use my ipad in order to fit all of my connections together, since it allowed for airplay between devices (thank god).

The silicone hands were casted using silicone mold material, which is usually used to then cast stuff in plastic, but for my case, I was looking into a material that had the flexibility to move without tearing, so I opted to just use the mold itself as the piece. This created an issue however as I could not use a mold of dipping my hand into a bucket of the stuff, so I had to physically paint silicone onto my hand while keeping it completely still, and let it dry for 5 layers for enough structural integrity while maintaining the thinness to put it onto a wooden mannequin hand. From there, I then drilled out holes into a giant slab of redwood that I bought secondhand, and installed it into its base for the table that holds the fortune set with the hands and crystal ball.

With all of these pieces together, I began rehearsals, which lasted roughly two weeks, and I had around 3 iterations. The actual use of the sculpture for my piece was after the first rehearsal, as I felt I needed to add more visual feedback to make my hand gestures more obvious, if you want to compare the visual usage in the first rehearsal vs the final performance! This is what it looked like 2 weeks before the show. Likewise, I also have an alternative angle which is decently similar to the rehearsal angle, for a better side by side comparison in how they looked.

That covers the visuals and audio of my performance, and then the sculpture is from my previous projection mapping fabrication sculpture that I made for Fabrication class, which I used as the placement for my projected visuals contrasting front and back. I am really happy with the way this piece turned out, and as I was experimenting with audioreactive visuals in the original iteration of the sculpture, adding the performance element to this piece really allowed me to explore creating hardware, visuals, and sculpture into one cohesive piece, and I would really like to incorporate more projection mapping with sculptural elements for future pieces!

matt toups

works

about

blog