The Project
Make an application that creates a meaningful relationship between sound and images.
It can be an audiovisual instrument, a tool to VJ a music performance, an interactive music video or anything in between.
Make it thematically related to your current or previous artistic research.
We’ll approach this unit with a series of exploratory technical assignments.
The delivery for each exercise are:
– 1 video capture of the application, full screen, HD (1080p), at least 1 minute.
– link to the project exported as .unitypackage and uploaded on the cloud
– each project must have a title
– bring the project to class and be prepared to play it
Exercise 0
If you haven’t worked with Unity before (or in a long time) watch my introduction or something equivalent.
Topics: What is Unity? How do I move things around? How do I move my view? What are all these buttons and panels? What’s a component? What’s an object? How do I set up Visual Studio and Unity? How do I import assets?
Exercise 1: Symmetry
Create a scene using at least 10 assets, in which the keyboard controls at least 8 parameters of the assets (position, scale, rotation material etc).
The theme is: Symmetry.
-Not all assets have to be animated.
-You can have multiple copies of the same asset.
-One input can control multiple things
-Don’t create a diorama or doll house! Use the objects as raw forms, play with scale, color, visual resonances
-Feel free to change the materials (Note: the models aren’t textured).
Due Thursday August 31: Watch the videos, write down your questions and bring them to class
Due Tuesday September 5: Exercise delivery
Download the UnityBasics project
Download and import this selection of assets from the game Everything.
Watch the videos below about the “Interaction” and the “Tweens” scene.
Exercise 2: Sound/Music Visualizer
Starting from simple abstract 3D models you made, create a scene that responds to a soundtrack of your choice.
The theme is: whatever song or soundtrack you pick.
-Find 3 images that serve as color/palette inspiration
– Use at least 5 assets, either made from scratch or found online
– If musical find a tune that is instrumental (possibly no vocals) and not well known (no pre-existing associations or music videos etc). Stay under 3 minutes.
– Have at least 10 parameters controlled by sound. Other things can be scripted, time based or input based if you want.
Due Thursday 7
Watch the videos below and look into the related scenes: Instantiation, Iteration
In class we will do some exercises to see if you all have understood these basic concepts.
Due Tuesday 12
Watch the video “sound control”, start collecting or making assets and sketching out your ideas, choose a soundtrack or two.
In class we’ll discuss your ideas and talk a tiny bit about 3D modeling.
Due Tuesday 19
Project is due
How to use the Fast Fourier Transform to analyze audio sources, extract frequencies, and control objects with them.
Timeline events: how to synchronize events to a timeline, how to trigger functions with markers.
*You can download only the timeline scene here (make sure you aren’t overwriting anything important!)
How to detect beats per minutes (BPMs) by tapping, how to set a coroutine that follows the beat.
More advanced resources (I haven’t tested them):
Beat synchronization – github
Algorithmic Beat Mapping – github
Intro to box modeling in blender
If you don’t have 3D modeling experience get familiar with Blender with these two tutorials:
Blender: What is it? Why is the interface so weird? What are all these panels? How do I move/scale/rotate thing? What’s object mode and edit mode? How to snap to object and grids? What’s parenting? How to select things? What are vertices edges and faces?
Blender box modeling: How do a model my own stuff? Can I turn a box into the car of my dreams? What are all those modeling tools? Extrude? Loop cut? Bevel? How do I make things symmetrical?
Blendshapes/keyshapes, how do I animate by morphing a mesh
Finding assets online
Where do I find 3D models online? Is it cool? What should I look for? How do I import them in Blender?
Blender: Help! the model I imported is: untextured/ animated / all joined together in one object / not smooth / too detailed / too BIG / too SMALL / all messed up!
Exercise 4: Cityscape
Starting from this generic modern city model
Give it a character or a theme by:
-changing lighting: on scene, global illumination, skybox, fog etc.
-applying materials (including textures)
-using effects, such as special shaders or particles
-adding post processing effects
-changing the camera view
-you can add a maximum of 3 extra models or assets (which may appear repeated, like trees)
-you can change the UV mapping of the city objects if you know how.
– if Unity struggles with the large number of objects you can join the objects in Blender.
– you don’t have to assign materials to the objects, you can only change what is seen from the camera
-Don’t modify geometry of the city model.
Deliverable: unitypackage of the scene + short video or 1 image.
Examples of themes:
Friendly city, scary city, underwater city, Mars colony, cartoon city, glitch city, dream city, heaven, hell, utopia, dystopia, paper city, cheese city, trippy, etc…
Exercise 5: Performance
In teams of two.
Given a song and a controller, create an interactive app that can be played along with the song in a live audio/visual performance.
– you will be given 3 song options to choose from.
– it must have an element of sound reactivity, even if minimal
– it should be compelling for the duration of the tune.
– it can use found assets but don’t rely on models with very personal styles, it should feel like your piece
– include at least one technical solution that has not been covered in the class tutorials
– choose a visual, thematic, conceptual angle that makes sense for the song and explain it in a one-page statement
Brainstorming I: drawing sounds
Using pencil and paper draw the shapes of these sounds. You’ll have a minute or so for each sound. It can be more or less abstract, there is no wrong way to do this.
5 simple tones.
Now let’s try with some tunes. It can be a sequence of drawings as well.
Brainstorming II: live mood board
While I play a tune, use river (AI sorting of human curated images) to find shapes and textures that resonate with the tune.
Take a screenshots when it feels right. Compile the screenshots in a mood board.
Brainstorming III: stock video VJ
While I play a tune find a couple of video clips that work well as visual accompaniment.
shutterstock
pixabay
storyblocks
Youtube and archive.org is fine too.
Brainstorming IV: free associations
While I play a tune, write down words that come to your mind. It doesn’t have to be a sorted list, the words can float around the page like visual poetry.
Compare the words with the classmate next to you.
During the break we’ll post them in midjourney to see what the collective (stolen) intelligence comes up with.
Brainstorming V: research
Groups are formed. Each of team member picks a song among the 3 assigned, look up the author and song/album. Get a sense of the historical and cultural context, meaning and intention, process and backstory, read reviews.
Which findings could conceptually inform the audiovisual application?
Assignment:
Each team member creates a miro board for one of the 3 songs, compiling some reference images and videos, sketches, and written findings about the author/tune. In class we’ll discuss the ideas and help you pick the most promising.
You’ll be assigned a “special” interface.
Interfaces
Download UnityMIDI if you have a MIDI interface and watch the video tutorial.
TouchOSC mk1 on the device
OSCext on the Unity side
Events
Deep listening workshop
From 4-6pm on Thursday October 5th in the iDeATe Media Lab (HL A10A), Stephanie Loveless from the Deep Listening Institute will offer a participatory workshop introducing the sounding and listening meditations of composer and sound pioneer Pauline Oliveros. All are welcome. We will explore the difference between passive hearing and active listening, collaborate in group sonic meditations, and develop our own site-responsive scores for listening and sounding. With these activities, we will work towards cultivating attunement to, and agency within, our sonic environment.
LIVE! Art&&Code
site – sign up and participate to at least one of these
- LIVE! Art&&Code’s VDMX & ISF Workshop with David Lublin and Friends
- Friday, October 13th 1-5PM
- Apply Here (Application Closes October 1st at 11:59PM)
- LIVE! Art&&Code’s TouchDesigner Workshop with Zoe Sandoval and Matthew Ragan
- Saturday, November 11th from 1-6PM, and Sunday November 12th from 1-5PM
- Apply Here (Application Closes October 29th at 11:59PM)
- LIVE! Art&&Code’s LIVE CODE Workshop with Olivia Jack, Sarah Groff Hennigh-Palermo, Char Stiles, and Dr. Kate Sicchio
- Saturday 2nd 11AM-6PM
- Apply Here (Application Closes November 19th at 11:59PM)
Exercise 6: Homeplay
Show and tell time.
Play a game/app either among the ones presented in the synesthesia lecture or a interactive artwork or game of your choice that relates to the topics of this unit or your project.
Present an aspect of it that you find interesting: a mechanic, an effect, a solution… you can give an overview of the work but try to focus on a narrow aspect and research it as much as you can.
All presentations should be between 5 and 10 minutes and have audiovisual material to support it.
Exercise 7: Skillshare
Show and tell time.
Research a technical topic (tool, technique, library, algorithm, 3D modeling practice…) not covered in class but relevant to this project and present it to your peers.
All presentations should be between 5 and 10 minutes and have audiovisual material to support it.