Skip to main content
SearchLoginLogin or Signup

Cartographic (Live A/V piece)

Published onAug 29, 2023
Cartographic (Live A/V piece)

Cartographic
Live audiovisual performance
Composer: Tasos Asonitis

 

Abstract

Cartographic is a live audiovisual piece that focuses on a three-dimensional map created from sound. Leveraging Machine Learning techniques, an original sound source is broken into small segments and these segments are scattered in 3D space according to their timbral qualities. The piece unfolds as we navigate through this space and explore the different neighborhoods and their sonic character. Fluctuating between descriptive and poetic, the journey takes us through various stages as we observe the system gradually falling apart and what before stood like a map, eventually dissolve into an abstraction.

 

Context

Cartographic belongs to the broad category of live audiovisual performances and live coding. Specifically, it employs a data-driven musical interface in order to reinvent an original sound source and, at the same time, create live visuals from it. At the heart of this system lies an unsupervised learning algorithm called UMAP (Uniform Manifold Approximation and Projection) that undertakes this task . The product of this algorithm is used as the foundation of the live performance and the main material carrier throughout the piece.

UMAP was implemented via a custom Max4Live device that brings together a set of the Flucoma tools. The Flucoma research project (Tremblay et al. 2021) brought dimensionality reduction algorithms (such as UMAP and PCA) to popular audio programming environments, as a way of exploring large sound corpora. Associated composers and researchers that have explored this pathway involve Gerard Roma (The Big Fry-Up, 2021), Ted Moore (quartet, 2021) Alex Harker (Drift Shadow, 2021) and Rodrigo Costanzo (Kaizo Snare, 2020). The ability to visualise and interact intuitively with large sound corpora , introduces new possibilities in the ways musicians and sound artists think of and explore the sonic material. Applications like Edyson (Fallgren & Edlund 2021), AudioStellar (Garber, Ciccola & Amusategui 2020), CataRT (Schwarz et al. 2006) and XO (commercially available by xln audio) have substantially contributed towards this direction.

 

Methodology

The audiovisual system that forms the basis of the piece is utilizing a dimensionality reduction and clustering algorithm, called UMAP (McInnes et al. 2028), to create a 3D point cloud from a sound file. In detail:

1. The original sound file is sliced into small segments - with duration up to 1 second each - based on onset detection. The maximum constrain of 1 second was imposed in order to ensure the obtaining of short, flexible sounds that could articulate dynamic phrases when played juxtaposed or in sequence.

2. Each sound slice is spectrally analysed using various audio descriptors (loudness, spectral shape, MFCC analysis) that produce 21 analysis values for each slice.

3. The 21 values for each slice are reduced to 3, using UMAP. The dimensionality reduction (21 dimensions to 3), still maintains the information from the original analysis that produced the 21 values.

4. The 3 values for each slice are treated as xyz coordinates, resulting in a visual plot where each set of 3 values is a point in three-dimensional space. Each of these points is associated with its corresponding sound slice through a shared identifier. Since these 3 values carry the information of the 21 values stemming from the audio analysis, the points that appear close in the 3D space correspond to sound slices with similar audio descriptor analyses, and therefore similar sonic character. This results in a 3D point cloud in which we can observe clusters of timbral similarities.

The 4 stages described above are taking place offline, in order to prepare the system. The next stage takes place in real time, forming the core of the live performance.

5. By navigating through this point cloud and selecting different sets of points, the respective sound slices are triggered in real time.

The algorithm, leveraging dimensionality reduction, clustering, and unsupervised learning techniques, creates a data-driven musical interface that offers new opportunities for real time sound exploration. The addition of the visualised plot provides an additional perspective on the sonic space under examination, while at the same time allowing the audience to peek into the central process that delineates the narrative and the sound generation mechanism of the piece. From an aesthetic point of view, the visualised output of UMAP, foremostly used in the computer science community due to its informative outcome, is hereby explored for its aesthetic value poeticising, thus, the algorithmic product.

Source material

The sound file that was used as an input to UMAP, is a 6-minute improvisation on the analogue drum synthesizer Pulsar 23. The recording includes extensive noise and glitch elements produced through various feedback loops . These sonic characteristics greatly influenced the aesthetic direction of the piece. The reason such material was chosen is the desire to experiment with the analysis and re-arrangement of a sound source which includes various types of sonic “accidents”. In a more traditional compositional approach, these sparsely occurring elements might be edited or masked depending on the musical context. However, the implementation of UMAP, by grouping these microsounds together, creates unified spectromorphologies that help recontextualise these isolated “accidents” into cohesive sonic territories that offer themselves to intuitive exploration through navigation in virtual space. An additional reason behind the use of a clustering algorithm as a sound re-arranging tool is to remove the temporal character of the original source and rethink the sonic possibilities of the existing material using instead the timbral qualities as the principle component of sound organisation.

Structure

In terms of structure, the piece unfolds in 4 parts. During the first part, the main idea described above is introduced, by focusing on specific areas of the audiovisual cluster and exploring their sonic character. The areas explored during the performance have been selected through prior experimentation. Noisy timbres are succeeded by resonant ones as the dynamic range build ups to lead to the next section. The second part transitions towards the bigger picture, revealing visual aspects of the audiovisual cluster, where precomposed sonic elements coexist with the sound produced from the real time navigation across the cluster. In this section, as the navigation between different sonic neighborhoods takes place, the resulting spectra interpolate accordingly resulting in morphing textures that would not be possible without a clustering based on timbral similarities. This leads to the generation of longer sonic phrases, instead of the punctual, percussive microsounds that were present during the first part. In the last two sections a drum beat attached to a fixed rhythmic grid is introduced, created primarily from pre-selected sound slices of the original audio file. Eventually the integrity of the rhythmic pattern is broken apart, as noise and heavy effect transformations are applied to the overall sound palette. The two different carriers of material, the drum beat on one hand and the audiovisual system on the other, are fused together into a unified noise/glitch mayhem. The visual component follows the same premise.

The live performative elements involve the navigation across the audiovisual cluster, the application of effects processing on the resulting sounds, the triggering of various sonic and visual cues, and the sound and rhythmic variations on the drum beat.

References

Fallgren, P., & Edlund, J. (2021). Human-in-the-loop efficiency analysis for binary classification in edyson. Interspeech, 3685–3689.

Garber, L., Ciccola, T., & Amusategui, J. (2021). AudioStellar, an open source corpus-based musical instrument for latent sound structure discovery and sonic experimentation. Proceedings of the International Computer Music Conference, 62–67.

McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018). UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 3(29), 861.

Schwarz, D., Beller, G., Verbrugghe, B., & Britton, S. (2006). Real-time corpus-based concatenative synthesis with catart. 9th International Conference on Digital Audio Effects (DAFx), 279–282.

Tremblay, P. A., Roma, G., & Green, O. (2021). Digging it: Programmatic data mining as musicking. International Computer Music Conference 2021, 295–300.

 

Concert theme

Although the piece is not a live coding performance with the strict definition of the term (ie. it does not involve the writing of code in real time), it borrows significantly from the aesthetics and premises of the practice. The audiovisual system behind the work has been designed prior to the performance, however the interaction with it takes place in real time and involves elements of improvisation. The Algorave concert seems to be the most fitting for the performance of Cartographic, with the AI Music Theater concert being a suitable event as well.

 

Previous performance documentation

Can be found in this link:
https://youtu.be/aLpmwAL-xeQ

Technical rider for Cartographic:

 Required by the venue:

1.       Two speakers

2.       One projector and one projector screen along with a HDMI cable

3.       One audio interface (no inputs are necessary) with a USB cable (type c also supported)

4.       One stand with dimensions at least 50cm by 50cm that can support a laptop and a small midi controller and that can be tall enough for a standing performer

5.       Two power plugs for the laptop and the audio interface

Provided by the artist:

1.       One laptop with the power supply

2.       One midi controller with a USB cable

Artist bio:

Tasos Asonitis is an audiovisual artist whose activities encompass various facets of digital arts. His primary interest is on virtual environments that can be described by the creative entanglement of computer generated music and 3D graphics. Asonitis' artistic output however is not limited to the audiovisual domain, and includes multi-channel fixed media pieces, sonic installations and compositions for moving image. Recipient of the EPSRC doctoral scholarship, he is currently doing a PhD in Composition at NOVARS Research Center. His works have been exhibited in MANTIS Festival, People’s History Museum (Manchester, UK), Science and Industry Museum (Manchester, UK), Athens Digital Arts Festival, METS Fest (Cuneo, Italy), among other places.

Comments
0
comment
No comments here
Why not start the discussion?