Visualization Method for Proprioceptive Drift on a 2D Plane Using Support Vector Machine

In recent years, research about the sense or experience of the self-body, that is, one's own body, has increased in the context of embodiment. Embodiment refers to the idea or concept of having a physical or virtual body that can interact with the environment, such as reaching, grasping, and touching. For instance, humans can touch an object or another human positioned in the environment by moving their own body, in this case their own arm and hand. Nowadays, this interaction or communication is not limited to using one's own natural body. Due to inventions and development of human-like robots or avatars in the virtual world, the natural human body can be substituted by an artificial body, such as a humanoid, remote control robot, electric prosthesis, or computer-graphics avatar in virtual reality. For example, researchers developed a robot whose operator can "grasp" an object placed in front of the robot via its mechanical body, even if the robot is placed far away from the operator's body position^1,2. Similar to this example, if a human could perform an action via an artificial body, which body would hold the attribution of the operator's self-body?

We can easily find topics related to this discussion on the attribution or projection of "self" from our own natural body to an artificial, non-flesh-and-bone body. One example can be found in the medical field; for example, in the field of medical rehabilitation, treatments that "trick" the patient's self-body sensation using mirrors are being explored for reducing pain and improving motor function of a missing or paralyzed limb, called mirror therapy^3-6. In this therapy, the mirrored image of the unaffected body part or limb can mislead the patient's brain into believing that the missing or paralyzed limb corresponds to the one displayed in the mirror and lead to the feeling that it is still in its former condition (i.e., before the accident). It is still under discussion how this illusion affects the brain's resilience related to body representation. In addition to this type of discussion on our natural body, we can find similar discussions on embodiment, especially of human-system interaction design issues in the field of engineering. The sense of self for an artificial or virtual body has been well investigated in the context of telepresence, brain-machine interface, and brain-computer interface^1,2,7-9. Some researchers reported that a humanlike robot, which can transfer the tactile sensation from its robot hand to the operator's hand, can capture the operator's sense of self-body to the robot as well as the sense of being at a place where the robot is positioned rather than where the operator actually exists, called tele-existence¹. Other researchers reported that a virtual avatar reflecting the operator's body movements strongly transfers the operator's sense of self-body from the operator's own body to the virtual body⁹. These findings indicate how users can project their sense of self-body into an artificial body, such as a humanoid, remote control robot, electric prosthesis, or computer-graphics avatar in virtual reality, even if the artificial body is not directly connected to their brain and body.

Basic scientific research on this type of self-body sensation for non-flesh-and-blood, artificial body-like objects examined the underlying brain mechanisms for the experience of self-body using the rubber hand illusion (RHI)^10-13 and mirror illusion (MI)^14-16 in the medical and engineering fields as well as in psychophysics and neuropsychology. The RHI is the sensation that a rubber hand belongs to one's own body and is evoked by simultaneously stroking a visible rubber hand and the participant's obscured hand. In the MI, a hand image in a mirror positioned along the midsagittal axis visually captures the participant's perceived position of the unseen opposite hand. Moreover, synchronous movements of the reflected and unseen hand evoke the strong sensation as if the reflected hand image were the unseen opposite hand. According to the research on these illusions, the consistency between multimodal information and the prediction and sensory feedback about body movements seems to play an important role for the judgment of self-body attribution. Thus, these two illusions can be simple but powerful evidence and tools for scientists to investigate the brain mechanisms underlying our sensation of being tricked or believing that some artificial object or image can subjectively be our own body part, and that our self-body sensation does not have to be tied to our natural physical body.

In all these studies listed above, the discussion has been based on the concept of "self" consisting of two types of sensation proposed by the philosopher Gallagher¹⁷: the sense of ownership and the sense of agency. The sense of ownership refers to the sensation that an observed body part is one's own. The sense of agency corresponds to the sensation that body movement is self-caused. These two sensations are defined as the minimal self, that is, an immediate sense of the self¹⁶. According to this concept, the attribution of the "self" for the natural, damaged, virtual, and mechanical bodies can be evaluated by the same indexes: the sense of ownership and agency. In order to use this sensation for scientific evaluation, the question arises of how to measure the sense of ownership and agency robustly. Currently, the estimation of the sense of ownership and agency mainly relies on questionnaires, originally proposed by Botvinick⁹. In addition to questionnaires, we can attempt to measure them in quantitative ways. For instance, the skin conductance response (SCR) has been used as a physiological index of ownership in cases where the rubber hand is suddenly cut by a knife¹⁸. The SCR is calculated by measuring the electrical characteristics of the skin and is a sensitive and valid indicator for arousal¹⁹. Since this method is typically applied for single trials per participant, measuring SCR is not suitable as a physical index during psychophysics experiments that require repetitive measurements within participants. One of the most successful behavioral indexes for the sense of ownership is proprioceptive drift. Proprioceptive drift is the change in the perceived position of the unseen real hand toward the position of an object that looks like a hand, such as the rubber-made prosthesis or computer graphics^10-13. Since this change can be estimated repetitively and robustly by measuring the distance between the unseen real hand and the visual image of the hand, proprioceptive drift is a suitable physical index for psychophysical measurements. However, this usage needs to be evaluated carefully, because recent discussions have questioned whether proprioceptive drift can always be used as a behavioral index of ownership¹².

Typically, proprioceptive drift is measured in only one of the three directions, such as height, width, or depth. Proprioceptive drift has rarely been measured in multiple directions due to the difficulty of estimating and visualizing multi-dimensional data. This metrological limitation is not critical for basic research exploring the mechanisms that process multisensory information, because experimental conditions can be easily designed and controlled to limit the measured dimensions. However, in daily life, our hands move freely in 3D to follow our intentions. In this situation, it is difficult and inadequate to measure a participant's behavior with questionnaires, which severely limits movement and positions of the hands. Thus, considering the potential applications for sense of ownership and agency in engineering and rehabilitation, a measurement that includes multiple directions and allows free hand movement is needed to evaluate the spatial relationship between visual and proprioceptive feedback in daily life situations. If such measurement were possible, the measured distance between real and observed hands could be utilized as a guideline for the sense of self-body. This could not only become an indicator for the progress of rehabilitation but also a criterion for the spatial offset between the manipulated target in the display and the operating hand. The question remains as to how this measurement can be implemented reliably and effectively.

To address this question, we introduce a novel method to estimate proprioceptive drift, which corresponds to the shift from the position of the participant's unseen real hand to that of a visible hand-like object, on a 2D plane using the mirror illusion by combining a psychophysical procedure and an analysis using machine learning. Compared to a rubber hand, the hand image in a mirror strongly captures the participant's perceived position of the unseen real hand. Moreover, a mirrored image immediately reflects voluntary hand movements for hand placement. Thus, a mirror image was selected as the visual feedback of the participant's hand. In addition, to measure proprioceptive drift similar to daily life situations, the participants positioned their hidden hand trial-by-trial at their will, and the number of trials was increased. Although any combination of directions could have been used, the combination of height and depth was chosen due to the ease of placing the mirror vertically. To check consistency between our method and previous research¹³, two visual conditions were implemented: with and without visual feedback. In the condition with visual feedback, the mirror was positioned along the midsagittal plane to create a reflected image of the left hand, as if it were seen as the right hand. In the condition without visual feedback, a matte blackboard was used in order to hide the participant's real right hand. We assessed the effectiveness of this novel method by comparing the results to those obtained with a questionnaire on the sense of ownership and agency.