Real-Time Proxy-Control of Re-Parameterized Peripheral Signals using a Close-Loop Interface

The Natural Close-Loop Controller
Sensory-motor information flows continuously between the brain and the body to produce well-organized, coordinated behaviors. Such behaviors can be studied while focusing on the person's actions alone, as in a monologue style (Figure 1A), or during complex dynamic actions shared between two agents in a dyad, as in a dialogue style (Figure 1B). Yet, a third option is to assess such complex interactions through a proxy controller, within the context of a human-computer close-loop interface (Figure 1C). Such interface can track the moment-by-moment movements' fluctuations contributed by each agent in the dyad, and by the type of cohesiveness that self-emerges from their synchronous interactions, helping steer the dyad's rhythms in desirable ways.

Figure 1: Different forms of control. (A) Self brain-controlled interfaces rely on the close-loop relations between the person's brain and the person's own body, which can self-regulate and self-interact in "monologue" style. This mode attempts the control of self-generated motions, or it may also aim to control external devices. (B) "Dialogue" style control is introduced for two dancers that interact with each other and through physical entrainment and turn-taking to attain control over each other's motions. (C) "Third party" dialogue control of the dyad is introduced as mediated by a computer interface that harnesses in tandem the bio-signals from both dancers, parameterizes it and feeds it back to the dancers in re-parameterized form using audio and/or vision as forms of sensory guidance. The re-parameterization in the examples presented here were attained using audio or visual feedback, enhanced by the real time kinesthetic motor output of one of the dancers to influence the other; or of both dancers, taking turns in some alternating pattern. Please click here to view a larger version of this figure.

The overall goal of this method is to show that it is possible to harness, parameterize and re-parameterize the moment-by-moment fluctuations in biorhythmic activities of bodies in motion, as two agents engage in dyadic exchange that may involve two humans, or a human and his/her self-moving avatar.

Investigations on how the brain may control actions and predict their sensory consequences have generated many lines of theoretical enquiries in the past^1,2,3 and produced various models of neuromotor control^4,5,6,7,8. One line of research in this multi-disciplinary field has involved the development of close-loop brain-machine or brain-computer interfaces. These types of setups offer ways to harness and adapt the CNS signals to control an external device, such as a robotic arm^9,10,11, an exoskeleton¹², a cursor on a computer screen¹³ (among others). All these external devices share the property that they do not have own intelligence. Instead, the brain trying to control them does have, and part of the problem that the brain faces is to learn how to predict the consequences of the motions that it generates in these devices (e.g., the cursor's motions, the robotic arm's motions, etc.) while generating other supportive motions that contribute to the overall sensory motor feedback in the form of kinesthetic reafference. Often, the overarching aim of these interfaces has been to help the person behind that brain bypass an injury or disorder, to regain the transformation of his/her intentional thoughts into volitionally controlled physical acts of the external device. Less common however has been the development of interfaces that attempt to steer the movements of bodies in motion.

Much of the original research on brain-machine interfaces focus on the control of the central nervous system (CNS) over body parts that can accomplish goal-directed actions^{9,14,15,16,17}. There are, however, other situations whereby using the signals derived from activities of the peripheral nervous systems (PNS), including those of the autonomic nervous systems (ANS), is informative enough to influence and steer the signals of external agents, inclusive of another human or avatar, or even interacting humans (as in Figure 1C). Unlike with a robotic arm or cursor, the other agent in this case, has intelligence driven by a brain (in the case of the avatar that has been endowed with the person's motions, or of another agent, in the case of an interacting human dyad).

A setup that creates an environment of a co-adaptive close-loop interface with dyadic exchange may be of use to intervene in disorders of the nervous systems whereby the brain cannot volitionally control one's own body in motion at will, despite not having physically severed the bridge between the CNS and the PNS. This may be the case owing to noisy peripheral signals whereby the feedback loops to aid the brain continuously monitor and adjust its own self-generated biorhythms may have been disrupted. This scenario arises in patients with Parkinson's disease^18,19, or in participants with autism spectrum disorders with excess noise in their motor output. Indeed, in both cases, we have quantified high levels of noise-to-signal ratio in the returning kinesthetic signals derived from the speed of their intended movements^20,21,22 and from the heart²³. In such cases, trying to master the brain-control of external signals, while also trying to control the body in motion, may result in a self-reactive signal from the re-entrant (re-afferent) stream of information that the brain receives from the continuous (efferent) motor stream at the periphery. Indeed, the moment-by-moment fluctuations present in such self-generated efferent motor stream contain important information useful to aid the prediction of the sensory consequences of purposeful actions²⁴. When this feedback is corrupted by noise, it becomes difficult to predictably update the control signals and bridge intentional plans with physical acts.

If we were to extend such feedback loop to another agent and control the person and agent's interactions through a third party (Figure 1C), we may have a chance to steer each other's performances in near real time. This would provide us with the proof of concept that we would need to extend the notion of co-adaptive brain-body or brain-machine interfaces to treat disorders of the nervous systems that result in poor realization of physical volition from mental intent.

Purposeful actions have consequences, which are precisely characterized by motor stochastic signatures that are context-dependent and enable inference of levels of mental intent with high certainty^25,26. Thus, an advantage of a new method that leverages dyadic exchange over prior person-centered approaches to the brain machine or brain computer interfaces, is that we can augment the control signals to include the bodily and heart biorhythms that transpire largely beneath the person's awareness, under different levels of intent. In this way, we dampen reactive interference that conscious control tends to evoke in the process of adapting brain-cursor control¹⁷. We can add more certainty to the predictive process by parameterizing the various signals that we can access. Along those lines, prior work exists using brain and bodily signals in tandem^27,28,29; but work involving dyadic interactions captured by brain-bodily signals remains scarce. Further, the extant literature has yet to delineate the distinction between deliberate segments of the action performed under full awareness and transitional motions that spontaneously occur as the consequence of the deliberate ones^30,31. Here we make that distinction in the context of dyadic exchange, and offer new ways to study this dichotomy³², while providing examples of choreographed (deliberate) vs. improvised (spontaneous) motions in the dance space.

Because of the transduction and transmission delays in the sensory-motor integration and transformation processes³³, it is necessary to have such predictive code in place, to learn to anticipate upcoming sensory input with high certainty. To that end, it is important to be able to characterize the evolution of the noise-to-signal ratio derived from signals in the continuously updating kinesthetic reafferent stream. We then need protocols in place to systematically measure change in motor variability. Variability is inherently present in the moment-by-moment fluctuations of the outgoing efferent motor stream³⁴. Since these signals are non-stationary and sensitive to contextual variations^35,36, it is possible to parameterize changes that occur with alterations of the tasks' context. To minimize interference from reactive signals that emerge from conscious CNS control, and to evoke quantifiable changes in the efferent PNS motor stream, we introduce here a proxy close-loop interface that indirectly alters the sensory feedback, by recruiting the peripheral signal that is changing largely beneath the person's self-awareness. We then show ways to systematically measure the change that ensues the sensory manipulations, using stochastic analyses amenable to visualize the process that the proxy close-loop interface indirectly evokes in both agents.

Introducing a Proxy Close-Loop Controller
The sensory-motor variability present in the peripheral signals constitute a rich source of information to guide the performance of the nervous systems while learning, adaptation and generalization take place across different contexts³⁷. These signals partly emerge as a byproduct of the CNS trying to volitionally control actions but are not the direct goal of the controller. As the person naturally interacts with others, the peripheral signals can be harnessed, standardized and re-parameterized; meaning that their variations can be parameterized and systematically shifted, as one alters the efferent motor stream that continuously re-enters the system as kinesthetic reafference. In such settings, we can visualize the stochastic shifts, capturing with high precision a rich signal that is otherwise lost to the types of grand averaging that more traditional techniques perform.

To achieve the characterization of change under the new statistical platform, we here introduce protocols, standardized data types and analytics that permit the integration of external sensory input (auditory and visual) with internally self-generated motor signals, while the person naturally interacts with another person, or with an avatar version of the person. In this sense, because we are aiming at controlling the peripheral signals (rather than modifying the CNS signals to directly control the external device or media), we coin this a proxy close-loop interface (Figure 2). We aim at characterizing the changes in the stochastic signals of the PNS, as they impact those in the CNS.

Figure 2: Proxy control of a dyadic interaction using close-loop multi-modal interface. (A) Indirect control of two dancers (dancing salsa) via a computer co-adaptive interface vs. (B) an interactive artificial person-avatar dyad controlled by harnessing the peripheral nervous systems signals and re-parameterizing it as sounds and/or as visual input. (C) The concept of sonification using a new standardized data type (the micro-movement spikes, MMS) derived from the moment-by-moment fluctuations in biorhythmic signals amplitude/timing converted to vibrations and then to sound. From Physics, we borrow the notions of compressions and rarefactions produced by a tuning fork outputting soundwave as measurable vibrations. Schematics of soundwaves represented as pressure modulated over time in parallel to spike concentrations for sonification. Example of a physical signal to undergo the proposed pipeline from MMS to vibrations and sonification. We use the heart rate signal as input to the interface. This takes fluctuations in the signal's amplitude aligned to the movement onset every 4 seconds of motion and builds MMS trains representing the vibrations. The spike trains from the MMS are standardized from [0,1]. The color of the spikes as per the color bar, represents the intensity of the signal. We then sonify these vibrations using Max. This sonified signal can be used to play back in A, or to alter in B the interactions with the avatar. Further, in B it is possible to embed the sound in the environment and use the body position to play the sound back at a region of interest (RoI), or to modulate the audio features as a function of distance to the RoI, speed or acceleration of a body part anchored to another body part, when passing by the RoI. Please click here to view a larger version of this figure.

The PNS signals can be harnessed non-invasively with wearable sensing technologies that co-register multi-modal efferent streams from different functional layers of the nervous systems, ranging from autonomic to voluntary³². We can then measure in near real time the changes in such streams and select those whose changes enhance the signal-to-noise ratio. This efferent motor signal can then be augmented with other forms of sensory guidance (e.g., auditory, visual, etc.) Because the PNS signals scape full awareness, they are easier to manipulate without much resistance ³⁸. As such, we use them to help steer the person's performance in ways that may be less stressful to the human system.

Building the Interface
We present the design of the proxy control mediated by a close-loop co-adaptive multimodal interface. This interface steers the real-time multisensory feedback. Figure 3 displays the general design.

The close-loop interface is characterized by 5 main steps. The 1st step is the multi-modal data collection from multiple wearable instruments. The 2nd step is the synchronization of the multi-modal streams through the platform of LabStreamingLayer (LSL, https://github.com/sccn/labstreaminglayer) developed by the MoBI group ³⁹. The 3rd step is the streaming of the LSL data structure to a Python, MATLAB or other programming language interface to integrate the signals and to empirically parametrize physiological features (relevant to our experimental setup) in real-time. The 4th step is to re-parameterize the selected features extracted from the continuous stream of the bodily signal studied and augment it using a sensory modality of choice (e.g., visual, auditory, kinesthetic, etc.) to play it back in the form of sounds or visuals, to augment, substitute or enhance the sensory modality that is problematic in the person's nervous system. Finally, the 5th step is to re-assess the stochastic signatures of the signals generated by the system in real time, to select which sensory modality brings the stochastic shifts of the bodily fluctuations to a regime of high certainty (noise minimization) in the prediction of the sensory consequences of the impending action. This loop is played continuously throughout the duration of the experiment with the focus on the selected signal, while storing the full performance for subsequent analyses (as depicted in the schematics of Figure 3 and see^{40,41,42,43,44,45,46,47} for an example of a posteriori analyses).

Figure 3: The architecture of the multi-modal peripherally driven close-loop interface concept. Various bodily signals are collected -kinematic data, heart and brain activity (step 1). LSL is used to synchronously co-register and stream the data coming from various equipment to the interface (step 2). Python/MATLAB/C# code is used to continuously parameterize the fluctuations in the signals using a standardized data type and common scale that enables selecting the source of sensory guidance most adequate to dampen the system's uncertainty (step 3). This real-time enhancement of signal transmission through selected channel(s) then allows re-parameterization of the re-entrant sensory signal to integrate in the continuous motor stream and enhance the lost or corrupted input stream (sensory substitution step 4). Continuous re-assessment closes the loop (step 5) and we save all data for additional future analyses. Please click here to view a larger version of this figure.

The following sections present the generic protocol of how to build a close-loop interface (as described in Figure 3) and describe representative results of two experimental interfaces (elaborately presented in Supplementary Material) involving physical dyadic interaction between two dancers (real close-loop system) and virtual dyadic interaction between a person and an avatar (artificial close-loop system).