Synesthetic Art and Games – Experimental Game Design: No More Video Games

There have been many attempts to blur the boundaries of sound and visual arts. Instruments for the eyes have been conceptualized since the obscure ocular harpsicord in 1725.

In the early 20th century many artists claim to be affected by synesthesia and tried to incorporate their condition into their art making.

Color Organ invented by composer Alexander Scriabin in 1915 for his Prometheus: Poem of Fire.

Wassily Kandinsky – composition VIII (1923)

“Color is the keyboard, the eyes are the harmonies, the soul is the piano with many strings. The artist is the hand that plays, touching one key or another, to cause vibrations in the soul.”

He worked with other composers, making librettos and stage designs.

Film and video

Experimental filmmakers embrace the idea of visual music. Music as a pathway to abstraction.

Oskar Fischinger – An Optical Poem (1938)

Norman McLaren – Dots (1940)
“Made by painting directly onto clear frames of film, as pictured in the frame enlargements below. Interestingly, the music was created in the same way, painting directly into the area on the film strip usually reserved for the soundtrack.”

Norman McLaren – Synchromy 1971
“To produce the film’s musical soundtrack, McLaren photographed rectangular cards with lines on them. He arranged these shapes in sequences on the analog optical sound track to produce notes and chords.”

John Whitney Sr – Arabesque (1975)
Early computer animation, he applied musical principles such as the harmonic progression to the generation of images.

Music Visualizers

Using real-time sound analysis (fft) control parameters of a generative composition

Atari video music, 1976. The first (analog) electronic music visualizer.

Jeff Minter, Virtual Light Machine 1990, Atari Jaguar

Milkdrop

Live visuals or VJing

Visual artist creating visuals live in response to a live musical performance.

Exploding Plastic Inevitable was a series of multimedia events organized by Andy Warhol in 1966 and 1967, featuring musical performances by The Velvet Underground, dancers, and projections. Related to the notion of expanded cinema

The interest in enhancing musical experiences with visuals re-emerged in the 60s and 70s with psychedelia and the “expanded cinema”

Liquid light shows – overhead projector + live manipulation of fluids. Late 60s – today.

Hexstatic+Coldcut 1997 in collaboration with Greenpeace.
This is a 1997 music video but gives an idea of the visuals/sound relationship.

Ryoji Ikeda, musician and installation artist. a/v performance

Midi controller + real time engine

Look at Kejiro’s vimeo and github for advanced Unity stuff

Tech heavy VJing with real time capture, surround screens etc – they’ve got interesting web based stuff as well

Speedrunning as live AV performance

Rhythm/music games

Currently dead genre… or not?

Rez – Tetsuya Mizuguchi – 2001
Inspired by a techno street parade

Thumper – Drool – 2016

beyond the tunnel/rail gameplay

Soundself – VR based guided meditation

Interactive/dynamic Music Videos

Bob Dylan – Like A Rolling Stone

Wilderness Downtown – Arcade Fire + Chris Milk + Google

Installation / Digital Art

Jacolby Satterwhite
Country Ball is the attempt to recreate a home video from the late 80’s of my family’s mother’s day cookout. My process involved looking for 35 of my mothers drawings that illustrated outdoor recreational utilities, then I trace my mothers drawings by hand onto the computer, import them into a 3D program: build them to create a computer generated landscape. I perform in front of the camera and green screen 100 times; later inserting those videos into the virtual space to create a Hieronymus Bosch “Garden of Earthly Delights” inspired landscape. This gesture is an attempt to use drawing, performance and technology as a device to translate and document a personal mythology.

Touching a cactus

Re-Animated by Jakob Steensen, 2D video/VR work. “RE-ANIMATED is based on the last Kaua’i ʻōʻō bird that died in 1987, marking the extinction of its species. In 2009, its mating call – first recorded in 1975 and later digitized in an ornithology lab in New York – was uploaded to YouTube. Since then, the song of the Kaua’i ʻōʻō calling in vain to a mate who was no longer there has been played more than half a million times. RE-ANIMATED is a response to this mating call. It is a magnificent reanimation of the bird and its song, inhabiting a distorted digital reconstruction of the Kaua’i ʻōʻō’s original habitat on the Hawaiian island of Kaua’i.”

Music videogames/apps

Built around a song, album, or composition

Band/promo games used to be more common than you may expect

Play it here

Love Birds, Night Birds, Devil-Birds by cassie mcquater

Jessica Mak music video

Not interactive but shot within an engine in real time.

“Lately, I’ve been working with 3d animation using Maya and the unreal game engine. I think creating animation in a game engine gives you a special malleability in your work, stemming from being able to work in an environment that is already rendered, alive, textured, lit up. It gives what you’re doing at the moment more opportunity for mistakes. It illuminates directions that wouldn’t ordinarily have been taken if you were rendering out every frame through an ordinary 3d renderer.”
-Jack Wedge