Abstract

In Rhythmic Conversations we explore the opportunity for AI-driven musicians to enhance audience participation in interactive performances. In this paper we explore an interactive performance paradigm whereby virtual musician percussionists “listen” to drums in the audience and generate beats that compliment the audience members at the designated drums in real-time. This interaction then offers a musical accompaniment for an improvising guitar player and other audience members/percussionists. In this paper we reflect on several performance types given settings for Rhythmic Conversations and outline the advantages and disadvantages of the various paradigms.

 Motivation

Musical performances have the potential to evoke powerful emotions and create unique connections between performers and audiences. However, traditional performances often rely on a one-way flow of music from the performers to the audience, limiting the level of engagement and interaction that can take place. The use of AI-driven virtual musicians in musical performances offers a promising solution to this problem, enabling audiences to actively participate in the performance and potentially influence the direction of the music.

The use of AI technology in music also offers the potential for increased creativity and accessibility. By removing the need for expertise with physical instruments and creating new forms of musical expression, AI assistance can enable individuals who may not have access to traditional musical instruments or may have physical limitations to participate in musical performances [1]. Additionally, the use of AI-driven technology can facilitate collaborations between human and non-human performers, leading to novel and innovative musical compositions [2].

Furthermore, the use of AI-driven virtual musicians in musical performances presents an opportunity to explore the role of visual representations of AI for the purpose of engaging the audience and fellow performers. By examining the ways in which graphical representations of AI can be integrated into musical performances, we can gain a better understanding of how technology can be used to enhance audience participation and engagement.

Overall, the motivation behind this paper is to explore the potential of AI-driven virtual musicians in enhancing audience participation in musical performances, as well as the broader implications of using AI technology in the creative process. By understanding the opportunities and challenges presented by these technologies, we can gain insights into how they can be effectively integrated into musical performances to create new forms of artistic expression and engagement.

Related Work

The use of AI-driven musical experiences have gained significant attention in recent years due to their potential to create new musical experiences. We see the potential to utilize the use of visual representations of AI for audience engagement and participation. In this section, we review previous studies and projects that have explored both the use of AI in musical performance and audience participation.

Applications of AI in Real Time

Artificial intelligence has been used to create new musical compositions and generate improvisations in real-time. Some performances have included systems that were created for a call and response interaction [3][4], while more recent advances have incorporated real-time improvisation in other contexts and genres of music [2][5][6].

Notably, the Shimon project has developed a robot musician that uses AI to improvise and play with human musicians including a piano and guitar player [6]. Robotic systems such as Shimon incorporate real-time musicianship and do an excellent job of captivating an audience. Careful attention to detail is spent on the gestural capabilities of Shimon so as to engage the audience and fellow musicians. These performance robotic systems demonstrate the value in visual representation of AI musicianship.

Enhancing Audience Participation

Several studies have explored the use of technology to enhance audience participation in musical performances. One such project is the Beatbug system, which enables participants to create and share rhythmic motifs using tangible and playable bugs [7]. Other projects have used mobile devices, screens, and manipulation of music notations to involve the audience in performance [8][9]. These projects address the opportunity to engage the audience in non-traditional ways and provide the audience members with a sense of agency over the performance.

In summary, previous studies and projects have explored the use of virtual musicians and AI in musical performances, as well as ways to enhance audience participation. Our research builds on these previous works by introducing a novel approach that combines virtual musicians with AI to create a unique and engaging musical experience for both performers and audience members.

Performance System and Methods

The underlying system code and model choices for Rhythmic Conversations is in constant development, but the general system architecture remains the same (see Figure 1).

The system uses Ableton Live and MAX for Live to handle all MIDI routing. MIDI drum pads are used to provide input while a python backend manages the generation of new music. A model trained on Magenta’s Groove Database [10] is passed a short rhythmic phrase from the audience member and is saved as a local file and as clip in Ableton.

The model then generates one or more complimentary drum beats. The generated beats are managed by MAX and played through a track containing drum samples as well as saved as a clip. The clip can be further edited in real time by the performer or can be left to listen and generate new beats.

The MIDI messages are sent over a local network via OSC message using Ableton’s connection pack. The messages are received in Unity and are used to trigger drum animations in the virtual musician. Several other musically influenced idle animations occur during pauses in the performance.

Audience members are encouraged to play instruments during the performance as well as provide input for the virtual musician throughout the performance.

Interaction Opportunities and Constraints

The use of virtual musicians and AI technology in musical performances offers a range of opportunities and constraints. On one hand, these projects can increase creativity and accessibility, as well as enhance audience participation and lead to new collaborations and performance venues. However, the limitations of the technology used to create virtual musicians and AI-driven musical performances, as well as the associated costs and technical expertise required, can pose significant constraints. Despite these constraints, virtual musicians and AI technology offer exciting possibilities for the future of music and musical performances.

Venue Types

The Rhythmic Conversations performance has been performed and is scheduled to be performed in several different settings. In this section we review the benefits and constraints of differing venue types for audience interaction with virtual musicians.

Traditional Performance Venue

An interactive performance on a traditional stage is subject to several limitations that can hinder its effectiveness. One of the primary constraints is the physical layout of the stage, which may not allow for large groups of performers or audience members to move around freely. This can limit the scope of interaction between performers and audience members, and may not create the sense of community that interactive performances aim to foster. Additionally, audience members may feel hesitant to participate, or may not have the opportunity to do so in a structured way.

Technical limitations can also impact the effectiveness of an interactive performance on a traditional stage. Many interactive performances rely on technology to facilitate communication and collaboration between performers and audience members. However, traditional stages may not have the necessary technical infrastructure to support these interactions, such as high-speed internet connections or specialized software. This can limit the types of interactions that are possible and may hinder the overall effectiveness of the performance.

Black Box Theater

An interactive audience performance in a black box theater can offer several benefits, as well as some limitations. Some of the benefits include flexibility in the performance space, intimacy, and more direct interaction between performers and audience members. Black box theaters can be configured in various ways, allowing for more creative freedom and a tailored space for the performance. The smaller size of black box theaters can create a more intimate and immersive experience for the audience, fostering a greater sense of community and collaboration between performers and audience members. This can also facilitate more direct interaction, which can enhance the overall experience for both performers and audience members.

Black box theaters are typically smaller than traditional stages, which can limit the number of audience members that can attend a performance. This can make it difficult to reach a large audience or may limit the number of attendees at the performance.

Informal Settings

Interactive performances involving projections can offer several benefits, as well as some constraints. When held in an outdoor environment, such as a drum circle, the projections can add an element of visual interest and engagement to the performance, enhancing the overall experience for participants. The projections can also serve as a tool for communication and education, providing information about the performance or the cultural significance of the event. However, outdoor environments also present several constraints, such as limitations in power and technical equipment, weather and environmental conditions, and the potential for unwanted interference from external factors, such as sunlight or noise pollution.

When held in a house show setting, interactive performances involving projections can create a more immersive and intimate experience for the audience. The projections can be tailored to the specific space and enhance the ambiance of the performance. Additionally, projections can provide an opportunity for collaboration between musicians and visual artists, adding a multidisciplinary aspect to the performance. However, there are also some constraints associated with house shows, such as limited space and resources, potential technical limitations, and safety concerns for both performers and audience members.

In these more informal settings, it is important to carefully consider the technical and logistical requirements of the performance, as well as the potential impact on the audience experience. Overall, interactive performances involving projections can offer a unique and engaging experience for audiences, but must be carefully planned and executed to ensure success.

Conclusion

The use of AI-driven virtual musicians presents a range of exciting possibilities for enhancing audience participation and engagement in musical performances. While the technology behind virtual musicians and AI-driven performances has its limitations, it also offers opportunities for increased creativity and accessibility, new collaborations, and innovative performance venues. Moreover, the exploration of the role of technology in music can provide valuable insights into how technology can be used to enhance human artistry and creativity.

Future research in this field should focus on addressing the limitations of AI-driven virtual musicians and exploring their full potential in different musical contexts and genres. Finally, efforts should be made to ensure that the integration of AI-driven virtual musicians in musical performances aligns with the aesthetic preferences and values of musicians and audiences.

In summary, the use of AI-driven virtual musicians is an exciting and evolving field that has the potential to transform the way we experience and create music. By exploring its opportunities and constraints, we can develop a better understanding of how these technologies can be effectively integrated into musical performances to enhance audience participation, increase creativity, and promote new forms of artistic expression.

Appendix

Pictures and video will be provided upon deanonymization