Exploring Mental Health through Augmented Reality Sculptures

AI Tool(s) Used

• Augmented Reality (AR): Snapchat’s Lens Studio was likely used to create and display the AR sculptures. These AR lenses allow viewers to interact with the sculptures in both physical and digital spaces.
• 3D Modeling Software: Tools like Blender or ZBrush were probably used to sculpt the busts of Robert Schumann, J.S. Bach, and contemporary models.
• Audio Visualization Tools: Specialized software that maps sound waveforms to 3D models was likely used to translate Schumann’s music into visual patterns. Tools such as TouchDesigner or Processing might have been employed for this.
• Snapchat: For integrating AR and immersive experiences, the Snapchat platform was utilized to allow users to view and interact with the sculptures via its AR lens.

Description of Result

“Ghost Variations” is a large-scale augmented reality installation that merges art, mental health, and sound. The project uses AR to display monumental sculptures of historical figures like Robert Schumann and J.S. Bach, alongside contemporary models. The sculptures, visible through Snapchat’s AR Lens, are visual representations of Schumann’s deteriorating mental state, inspired by his time in a psychiatric asylum, where he found solace in Bach’s music.

The project also uniquely visualizes sound by mapping waveforms from Schumann’s music onto the surfaces of the sculptures, symbolizing how we experience and internalize music. Visitors can view these sculptures in-person at Lincoln Center’s David Geffen Hall or from anywhere via Snapchat, making the experience widely accessible.

Step-by-Step Breakdown

1. Collaboration and Concept Development: Sophie Kahn and Alexis Zerafa conceptualized the project to explore the connection between mental health, music, and AR. The focus was on the life of composer Robert Schumann and his relationship with Bach’s music during his time in a psychiatric asylum.
2. 3D Sculpting of Busts: Kahn created large-scale 3D sculptures of Schumann, Bach, and contemporary models using 3D modeling software such as Blender or ZBrush. These models were designed to be “larger than life” in both digital and AR spaces.
3. Mapping Music to Visuals: Kahn played Schumann’s music underwater, recorded the waveforms, and used audio visualization tools to map these waveforms onto the surface of the 3D busts. This step represents the connection between sound and mental health in a tangible form.
4. AR Integration with Snapchat: Using Snapchat’s Lens Studio, the digital sculptures were turned into interactive AR experiences. Users can engage with these sculptures via Snapchat, seeing sound-animated textures and even “looking through the eyes” of the figures.
5. Exhibition at Lincoln Center: The AR sculptures were made available at David Geffen Hall during the Summer For The City event at Lincoln Center. For those unable to attend in person, the sculptures were also accessible via Snapchat, making the project more inclusive.

Tips & Tricks

• Leverage AR for Accessibility: By using a platform like Snapchat, AR art installations can reach audiences globally, even those who can’t physically attend the event.
• Map Sound to Visuals: Transforming sound into visual elements, like waveforms mapped onto sculptures, creates a unique sensory experience. Use audio visualization tools to enhance art installations with sound-based textures.
• Merge Historical and Contemporary Subjects: Combining historical figures, such as Schumann and Bach, with modern technology and contemporary themes (mental health) creates a bridge between the past and present, making the artwork more relatable.

Annotation

“Ghost Variations” reflects a powerful intersection of art, technology, and mental health. By using augmented reality, Sophie Kahn and Alexis Zerafa create an immersive experience that visualizes the emotional and mental struggles of 19th-century composer Robert Schumann, during his time in a psychiatric asylum. The project artfully connects Schumann’s use of J.S. Bach’s music to navigate his mental decline with the larger theme of how music and mental health are intertwined.

Through AR, viewers can interact with monumental busts of Schumann, Bach, and contemporary figures—each of which represents not just physical forms, but emotional states and sound visualizations. This combination of music and sculpture via augmented reality, especially when accessible on Snapchat, makes for a deeply reflective and engaging experience. The project serves as an artistic exploration of how technology can deepen our understanding of historical figures and our emotional landscapes through interactive art.

Instagram Post