Abstract

This paper presents early development of a novel method of affecting inter-player co-ordination in a shared music making environment. Highlighting different approaches to encouraging inter-player collaborative a potential method of using statistical inference to compare player movement is highlighted. Questioning whether such individualistic quantification is relevant within an entanglement, the paper goes on explore whether an observation method can be adapted to become a system of affecting what it observes. The paper presents user studies, highlighting user experience that shows this may be possible. Finally, issues are raised and future work to develop the experience further is identified.

Introduction

From individual to entangled agency

There are several documented multiplayer digital instruments or systems that interconnect players [1] however apart from a few notable exceptions ([2], [3]) they create a shared musical experience either through allowing one player to control parts of another’s instrument or through editing a shared score. Exemplified by Weinbergs [1] and Linson et al [4], this isn’t that surprising considering the underlying history of individualisation that underpins historical instruments and playing alongside each other. Mutual Engagement [5], focusing on Interactional Sound and Music (ISM), extends this position to a degree accepting that ISM systems involve as much engagement with the co-producers than the outcome of production. However the work-station environments that the research explored still continues the single player individual instrument/interface metaphor. The practice research that this paper is part of seeks to question such individualisation through developing collective musical experiences that are mutually resistive [6] or enactive [7] or entangling [8] [9].

An ’entanglement’, infers some form of coupling between objects. Typical examples being the direct causation of quantum physics or interface mediated, say two people climbing a cargo net. Physicist and feminist philosopher Karan Barad highlights entanglement and encourages an examination of the nature and re-enactment of our inter-connections within the world, through the term intra-action [10]. Frauenberger reflects on networks and interconnections whilst considering whether entanglement theories can "grow" a new HCI [11]. The multi-player instruments of mutual entanglement discussed here seek to use these ideas to emphasise co-player awareness by coupling players’ actions either directly or systemically.

Figure 1 shows two players experimenting with one of the research’s early entangled instruments, based on Granular Synthesis. Their hands are being tracked within a 1 cubic meter, using a GameTrack [12] interface comprised of two joysticks controlled by retractable nylon cord. The source file for grain sampling is mapped to the left/right axis in front of the players, with players either setting the start or end points for this sample. Grain sample length is controlled in the opposite direction, see figure 2.

Players in this system, were observed stopping to allow their partner to explore the context this created (termed place-making), tracking each other around the space or crisscrossing their paths. Interviews with the participants revealed disagreement about who felt that they were ‘leading’ musically; the moving player, for example, saw the place-maker as subservient to their independence. The same player, however, was not aware of tracking the place-maker, as they shifted the whole soundscape by moving slowly in one direction. The question arose whether statistics, coupled with observation and interviews, could be used to cast light on these interactions. Then subsequently whether such information could be used to guide the experience, reducing an overly leading player’s ability to influence the music making or boosting the other player more for example.

It is worth pausing at this point to reflect on the wider project’s motivation. If the aim of the research is to identify design metaphors and systems that facilitate collective behaviour and awareness of inter-dependency then why focus on assessing the actions of individual players? To do seems to making a cut such that players’ individual decisions matter more than actions they take together. If we take the position that the collective act matters more than the individual action then there is some irony, and creative satisfaction, in exploring whether a system intended to assess one individual’s actions in respect of their influence on another, can somehow be used to encourage a mutual experience to emerge. The difference is key, one approach leads to measuring then punishing or rewarding individuals while the other allows collective agency to emerge within the entangling system.

Real-time Granger Causality

What is Granger Causality

Granger Causality is a linear regression based test designed to provide a statistical measure of whether one signal can be said to cause a second, though there are multi-variate libraries available. A measurement of error, p, is generated for given conditions of sample size and the number of lagged tests, when a series’s own history is used to predict it’s next value and compared to the prediction when same regressive method is used but a degree of a second series’s past is also included . I refer the reader to Anil Seth [13] for a detailed mathematical explanation, however the key looked for feature for suggesting causality is a low reported error, indicating that the second series can be used a predictor; it is therefore a null hypothesis test.

How has it been used in HCI and performance

Researchers have used Granger Causality to look for links between time series data-sets, such as listener perceptions and affect while listening to music [14]or in response to dancer movement [15]. Typically studies ask audiences to continuously log their response to a performance, which the researchers retro-actively compare to data extracted from the performers or aspects of the music being listened to. Saha et al [16] consider this whilst exploring affect aware agents. Reasoning that the link between musical cues and audience emotional response is documented, they explored whether it is possible to build a reference set of affective responses that an agent can use to tailor it’s own response. Granger Causality was used to try to correlate features of data from an electrodermal sensor and and musical features.

Notably these examples are not real-time systems and causality is used to test a theory or tease out potential findings, rather than being placed at the heart of a reactive system. As Dean and Dunsmuir [17] point out there has been an increased interest in real-time causal analysis especially around inter-brain coupling and perceptual responses. However their main purpose is to highlight key pitfalls in its use, such as auto and cross-correlation. Auto-correlation is where a time series is slowly changing within the sampling window and can actually be considered as, at least partly, being predictable on its own history. Using the smooth movement of a mouse pointer on screen as an example, they go to explain how if two highly auto-correlating time series are compared then non-meaningful correlations can naturally emerge, this is cross-correlation.

Why use it here?

There are strong similarities between Dean and Dunsmuir’s mouse pointer example and the interface system used in this study that question the use of Granger Causality as an analytical tool in this paper’s context. In fact it is a miss-application of the process that makes it useful as a reactive musical system. The purpose here is not to suggest a new tool but to illuminate moving from ‘trying to measure’ to ‘trying to influence’ player interaction using statistical methods.

Figure 3 shows a tidied screen shot of 100 second extract of two players (a dyad) playing an instrument on just one plane (Z), standing as in figure 7. The graph to the right of the movement plot is of just the dyad’s Z data plotted against the data-time-code (or sample number), the colour codes match the player in the movement plot, however an increasing Z value in the graph is represented as increasing vertically rather than downwards as in the movement plot.

The top of the figure shows the output for two Granger Causality processes over the same time period. Both are comparing the Z axis data stream from the two players, the left hand has a data window of 50 samples and the right 100. This equates to 2.5 and 5 seconds of activity respectively. The red line is the plot for whether the left hand input (note the real-world orientation not the visual plotted position) Z axis as predictor for the right, with the blue being the converse.

Noting the concern of auto and cross-correlation there is clear evidence, however, of some form of interplay happening between the participant experience and the mathematics. The following figure 4 shows the data from figure 3 with the LR axis movement graph aligned with the resulting Granger Causality plot.

The skill of interpreting a particular point on the Granger plot is to look upwards and then scan a block of data to the left in the upper graph as the regression process does. Statistically a high causality (low p) of Blue on Red, for example, would mean that the Blue data points have a significant influence on the Red, taking the area of the Red values (past history) highlighted in the dotted box into account. A statistical researcher would only be interested in a binary hypothesis result and, we can immediately see points where the mathematics is reporting significance where one or both of the Granger p values drop below the green line. Whereas the practice researcher seeking to sonify the data will be interested ebb and flow of the data seeking to ascertain whether it is consistent and could be interpreted sonically by the players.

The p data plots are very noisy, producing rapid spikes and changes in error (p), despite showing an over all trend, as the data moves through the testing window. The period between points A and B and to the left of point E are examples. Just before point B (E in the 100 sample window plot) the two players have been tracking each other in the same direction for about half a second with blue player approaching by moving slightly faster. This contingency is something the synthesis engine should be responding to in terms of the project’s aim of encouraging inter-player engagement and we can see this reflected in the red Granger reading for the 50 samples and both outputs in the 100 sample window results.

Likewise considering the player action at B/E ; at this point Red slows as they approach Blue but then changes direction, Blue also changes direction and both players continue together. The Granger Causality test reacts to these movements in an interesting way with both the shorter and longer windowed versions (B to C and E to F) reporting a low p value, both window sizes also show a rapid increase when red changes direction and plummet when blue responds.

Whereas the p values to the left of B and E show a trend of falling, as the two players follow each other more closely, the different window sizes show a difference in the way they respond blue’s slight change of direction.

The resulting soundscape was found to be confusing when the individual Granger p values where used as synthesis input. This increased when a second axis of movement was included, adding two more sound sources . The causal p value for the two players, in a single plane, was therefore averaged before mapping to a single control ( see figure 5). This smoothed data fluctuations and simplified the audio. It also increased inter-player coupling emphasising the conceptual shift to players working together rather than individuals leading or following each other.

Implementation in real-time

The Causilo system 1 uses MAX/MSP, and the statsmodels [18] Python module used for the statistical processing, which is accessed via an existing MAX to Python external object [19] . A buffer of windowed data which, once full, is passed to the statistical library for analysis, subsequent new data is rolled into the end of the buffer and the first entry removed. This structure allows for the easy implementation of other statistical analysis Python modules.

Two Causality tests are performed on the input streams A and B (broadly, is A influenced by B and conversely) and the p-values are returned to MAX/MSP, as are ‘buffer filling’ and error messages. External controls in MAX/MSP are provided to set the window size and a system to graph the output written in Javascript.

Options are included to select which streams of sensor data from the interface to compare. A optional script implements a composite method that combines the three sensor readings from each player by treating them as a 3 point vector and calculating its magnitude, as detailed in Seth.

Studying Causal entanglement

Four dyad (two player) tests were conducted with an equal gender split and range of musical experience, subject to the conditions agreed through the University of Sussex’s ethical review process. The whole study process involved testing four different player entanglements, however due to the focus of this paper, only the data relating to the Causilo system is summarised in table 1. Participants played with the system for up to 10 minutes and where then interviewed about their experience. Movement data and video were recorded, whilst playing, for subsequent analysis.

Table 1 : Extracted participant comments.

Movement plots are a common qualitative data source in collaborative play psychology research [20]. Here it is interesting to compare between the Causilo study and another more reactive instrument, called the Elastiphone; based on a virtual elastic band between the players hands. Both studies used the same synthesis system (see figure 8) and the plots shown are of a similar period, selected after the participants have had a few minutes learning and adjusting to the instrument. The Elastiphone and the Causilo where the last instruments played in the hour long study, so the participants had become quite familiar with the limits of the soundscape they were controlling and comfortable with the interface.

Discussion and Conclusions

Participants can guide the Causilo by moving at the same constant speed near to each other. However as discussed earlier there is an additional noise that possibly comes from the inter-play of the data window’s nature (continuity, size) and false correlations that creates the Causilo’s characteristics. The players’ interpreted this in several ways, firstly strongly characterising the Causilo as being different to a reactive instrument and having some autonomy or organism-like qualities (see table 1, A5, A6, A7, C1, C3). The system was often described in very anthropomorphic terms (A1) which may underpin a willingness to accept the system as a active third agent in the study, which could lead to the errors being noticed however being ascribed to the instrument as being inattentive (A2) or irritable (C3).

The players’ reactions to how they felt about their own agency within the system were mixed, ranging from showing a desire to control it (ironically through deliberately annoying the system, C3, though sometimes with difficulty B4), to some frustration at trying to understand and play something that slips away from them (A4, C1, C2). One group (interestingly the only pair with musical experience as solo performers) had a strong reaction and felt they had little agency or autonomy at all (C4, C5).

The participants also reported a shift in the way they play together (B1) needing to react less to their co-player and mirror or co-ordinate their actions (B3), or not (B4) to affect the synthesis. This is reflected in comparison of the movement plots (figure 9. The clearest difference between these being that the player movements in the Elastiphone produce spikes and heavy lines rather than continuous scribbles. These lines are produced when individuals ’place-make’ 2Actively gift a turn to their co-player by pausing; effectively to constrain the sonification either to create a musical context for the shared performance, or to simplify the shared experience and aid understanding in the learning phase., and their absence in the Causilo plots show there is little or no turn-taking happening, or contingent 3 action. The only breaks in the players’ continuous and coordinated movement is when one person (or both) decides to try to irritate the system by changing their movement, or moving asynchronously, in respect to the other; breaking away from entrainment can be surprisingly hard to do (B4).

Drawing form from conclusions from such an early study are risky, especially given the exploratory nature of this one, and often it is safer to highlight a few of the myriad of possible future directions for the research. It does appear, however, that it is possible to (mis)use a casual inference statistical method to create a musical playing surface that is different other digital musical instruments. The resulting player experience could said to be one of playing alongside another agent rather than controlling an instrument directly, and so potentially parallel to experiences of open mixer performances [8], vibro-tactile feedback resonators [21] or even playing alongside AI/aLife agents [22] [23]; however more work needs to be done on system stability before this can be conformed.

The p values in one axis of movement were averaged in order to simplify audio production and emphasise player entanglement. Combining the data also aligned the Causilo specific test with the other tests in the same study, such that they all used the same synthesis system that had just two controls. It should be noted that this system (see figure 8 had a hierarchy built into it that effectively coupled the two axis of averaged Granger analysis, future tests should disentangle this and have a clear audio voice for each axis test. Synthesis techniques that work well with the player behaviour and data texture also need to be identified.

The levels of contingency and coordination (or not) between the players also bears much more consideration, both conceptually and mathematically. Using different statistical tools in the Causilo might provide more stable or accurate measurements of inter-player engagement. However, as this paper demonstrates a judicial misuse of mathematics, it remains to be seen whether increased consistency would still create an interesting playing experience.

Acknowledgments

This research was funded by the Leverhulme Doctoral Scholarship Programme: “be.AI (biomimetic embodied AI) Center”, at the University of Sussex.