Here we present a protocol which characterizes the sense of agency developed over the control of sensate virtual or robotic prosthetic hands. Psychophysical questionnaires are employed to capture the explicit experience of agency, and time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.
This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the neural-machine interface (NMI) control of sensate virtual or robotic prosthetic hands. The formation of agency is fundamental in distinguishing the actions that we perform with our limbs as being our own. By striving to incorporate advanced upper-limb prostheses into these same perceptual mechanisms, we can begin to integrate an artificial limb more closely into the user's existing cognitive framework for limb control. This has important implications in promoting user acceptance, use, and effective control of advanced upper-limb prostheses. In this protocol, participants control a virtual prosthetic hand and receive kinesthetic sensory feedback through their preexisting NMIs. A series of virtual grasping tasks are performed and perturbations are systematically introduced to the kinesthetic feedback and virtual hand movements. Two separate measures of agency are employed: established psychophysical questionnaires (to capture the explicit experience of agency) and a time interval estimate task to capture the implicit sense of agency (intentional binding). Results of this protocol (questionnaire scores and time interval estimates) can be analyzed to quantify the extent of agency formation.
As robotic prostheses become increasingly advanced, the importance of relevant sensory feedback will continue to grow. Sensory feedback affects how humans perceive, interact with, and even integrate machines into their body schema. Recent NMI techniques can now provide prosthetic limb users with intuitive control and achieve sensations associated with touch1,2,3,4,5,6,7 and kinesthesia (movement sense)8,9 in missing limbs. When this sensory information is paired with the visual information provided by watching the artificial limb during operation, we have access to key elements that inform the distinction of self-versus-other. Leveraging this access may help bring prosthetic limb users a step closer to operating an artificial limb as a part of their body, rather than just a tool.
Body awareness and the sense of being embodied arises from the establishment of agency (the experience of authorship over a limb's actions) and ownership (the feeling that a limb is a part of the body)10,11. Ownership is primarily mediated through the integration of touch and visual information12. Agency emerges from the integration of intent, movement sensation (kinesthesia), visual information, and predictive cognitive models11. During the performance of a voluntary action, agency is formed when the sensory consequences of that action align with the performer's intent and predictions from the performer's internal models13. Agency is separate and distinct from ownership. The concept of limb ownership has been studied frequently in prosthesis literature14. A sense of limb ownership forms in NMI participants when touch feedback is spatially and temporally appropriate, as measured explicitly through questionnaires or implicitly through changes in residual limb temperature, or temporal order judgments15. However, fewer opportunities have existed to explore agency in the context of NMI16. Recent work with NMI participants has demonstrated that agency can be purposefully promoted and is separate from the experience of ownership8.
Agency is particularly significant in the operation of robotic prostheses as it is a cognitive link to the control of the artificial limb's physical actions through experiences of causality, the feeling of controlling the artificial limb or causing something to happen17. Robotic prostheses are advanced computerized machines that the user must cooperate with to effectively complete tasks. Some prosthetic limbs have incorporated autonomous functions, such as grip-slip detection and correction; yet these systems have seen limited adoption as functionality running outside the user's control can be viewed as frustrating if not appropriately implemented8,18. This presents a fundamental challenge that is echoed throughout applications of human cooperation with autonomous machines. That is, humans often trust their own actions over those resulting from a collaboration with computers or machines, and this trust directly influences an operator's likelihood to use the autonomous functions19,20. As humans, we innately trust ourselves and our bodies to perform the actions we intend; when this is achieved, we establish an intrinsic sense of agency. Interestingly, the formation of agency is impacted in cooperative human-computer actions. During human-human cooperative tasks, a shared sense of agency may be formed over movement21; yet, the literature suggests that shared agency is encumbered during human-computer cooperation22,23. These challenges are reflected in prosthetic upper-limb use, and the rejection rates of robotic devices remain high, with 23% – 39% of users discontinuing their use24. In fact, many prosthesis users still prefer body-powered systems25. These systems remove the computerized machine from the control loop and more intimately couple the user's body movement to the prosthesis movement via wire cables. This further reinforces the importance of cognitive integration in the use of advanced prosthetic devices. We suggest that NMI systems can provide a number of the necessary sensory and motor pieces to help move artificial limbs closer to establishing a cooperative sense of agency, and this will be instrumental in promoting the acceptance and the true integration of these computerized machines with their users.
Agency may be measured in a number of ways. The simplest measures use psychophysical questionnaires or scales that explicitly ask participants to whom or what they attribute an event17,26,27. This relies on an individual's existing perception of "self" by requiring participants to make inferential judgments of self-attribution (i.e., to explicitly judge whether "I" or another entity was responsible for an action or event). Implicit measures provide insight into the background cognitive processes that occur during motor action and sensory events. This view of agency attempts to measure that which is not explicitly perceived by an individual. Typically this is achieved by having participants characterize a perceived difference in self- and externally-generated actions, for example having participants report the duration of time they perceived to occur between a self- and externally-generated event17,28. During the performance of self-generated actions, agency implicitly manifests as a perceptual compression in time between actions and their sensory consequences, known as intentional binding28. When individuals report the time they perceived to occur between an action and its outcome, a shorter perceived duration of time corresponds to a more strongly formed sense of agency29,30. Interestingly, it has been demonstrated that explicit and implicit measures may not directly correlate as they are likely characterizing different perceptual mechanisms17 that together inform the sense of agency. As such, establishing a more comprehensive understanding of agency formation during prosthesis use will likely require experimental protocols employing both explicit and implicit measures.
This work describes a methodological framework that can be used to explicitly and implicitly characterize the sense of agency developed over the NMI control of sensate virtual or robotic prosthetic hands. Two techniques to measure agency during the performance of a sensorimotor object-grasping task are highlighted. Established psychophysical questionnaires are employed to capture the explicit experience of agency, while time interval estimates (intentional binding) are employed to implicitly measure the sense of agency.
The scope of this protocol is to evaluate the sense of agency in the context of an NMI that provides physiologically relevant active motor control and kinesthetic feedback. These techniques are generalizable to virtual or physical prosthetic NMI systems. There are minimal restrictions on the populations that may be recruited to perform this protocol. For instance, the mobility of the participant's upper limbs cannot be bilaterally affected (they must have one sound limb), and they must possess the cognitive ability to make time-based judgments and articulate experienced sensations.
This protocol has been previously approved and follows the guidelines of the Cleveland Clinic’s human research ethics committee.
1. Hardware and Software of the NMI
2. Experimental Setup
3. Experimental Conditions
4. Performance of the Experiment
The experimental protocol was performed with three amputee participants operating a sensate virtual prosthesis via their NMI8 (Figure 1). The setup used a participant-controllable virtual hand moving through preprogrammed kinematic profiles using the MuJoCo HAPTIX physics engine31. The virtual hand was displayed on a horizontal monitor in front of the participants at a location spatially congruent with their missing limb. The NMI participants had previously undergone a surgical neural rewiring procedure (targeted reinnervation), which was coupled with standard prosthetic limb myoelectric (EMG) control strategies to provide intuitive control of the virtual hand32; thus, the participants could drive the virtual hand by 'thinking' about opening and closing their missing hand. Strategic vibrations of the participants' surgically rewired muscles induced illusory perceptions of hand movement, providing a platform for kinesthetic sensory feedback8. Through custom software, EMG hand control signals and virtual prosthesis renderings were integrated with the output of a vibration feedback device. When the participant initiated a movement of the displayed virtual hand, the vibration would induce a corresponding matched sensation of a complex grip movement in the missing hand.
Figure 1: An example setup that satisfies the requirements to characterize agency. This setup provides the user with intuitive control and kinesthetic feedback of a displayed virtual hand. Virtual hand control and feedback are achieved through the neural-machine interface using myoelectric control and vibration stimulation (eliciting illusory movement percepts of the missing limb) of the amputees' reinnervated musculature. Control and feedback are coordinated through a data acquisition system and computer running custom software. Virtual hand kinematics are displayed to the user on a horizontal monitor. Please click here to view a larger version of this figure.
Figure 2A is provided for the comparison of explicit measures of agency under each of the feedback conditions. Here, the average score for the four agency questions (and four control questions) are plotted for each participant and by each feedback condition. In Figure 2B, these individual participant scores are averaged and plotted for each feedback condition, with the error bars representing the average standard deviation. An average rating greater than 1 indicates an agreement with a given statement and 0 indicates neutrality of agreement26. Higher agreement ratings (≥ 1) for the agency questions indicate a greater experience of agency. The responses to the control questions should be negative or neutral (≤ 0) and a score between 0 and 1 is taken as inconclusive. The 'baseline', 'too fast', and 'no feedback' condition demonstrated the lowest average interval estimates indicating the strongest sense of agency formed, whereas the 'passive' and 'opposite movement' conditions demonstrated the weakest sense of agency.
Figure 2: Explicit measures of agency under each feedback condition. (A) The average score for the four agency and four control questions provided to each participant under each feedback condition. (B) The average scores across participants under each feedback condition. The error bars represent the standard deviation. In both plots, an average rating greater than +1 indicates agreement and, for the agency questions, the formation of agency, while 0 indicates neutrality. This figure has been modified from Marasco et al.8. Please click here to view a larger version of this figure.
For a comparison of intentional binding in each feedback condition, Figure 3A shows the time interval estimates for each participant, averaged according to feedback condition. Differences between the actual and perceived time intervals were then averaged across the three participants and are presented in Figure 3B relative to the baseline feedback condition. The error bars denote the average standard deviation. Lower time interval estimates (Figure 3A) and larger negative differences (Figure 3B) are an indication of a stronger implicit sense of agency. The 'too fast' condition followed by the baseline condition demonstrated the lowest average interval estimates, indicating the strongest sense of agency formed, whereas the 'opposite movement' condition demonstrated the weakest sense of agency.
Figure 3: Implicit measures of agency via time interval estimates under each feedback condition. (A) The average interval estimates for the delay interval between the completion of the virtual hand close and the auditory tone plotted for each participant across each randomly presented actual interval. The results are plotted for each feedback condition, and a lower time estimate indicates a stronger sense of agency. These panels have been modified from Marasco et al.8. (B) The average difference (across participants and delay intervals) between the actual time delay and the participant's estimated time interval relative to the baseline feedback condition. The results are plotted for both. Here, a more negative value indicates a stronger average sense of agency, and CI denotes 95% confidence interval (CI). Please click here to view a larger version of this figure.
Figure 4 allows for a comparison of explicit and implicit agency measures. The average difference between actual and perceived time intervals are plotted relative to the results of the baseline feedback condition and with respect to the averaged agency questionnaire scores for each feedback condition. In this presentation of data, moving from left to right on the x-axis indicates a decrease in the explicit experience of agency and moving from bottom to top on the y-axis indicates a decrease in the implicit sense of agency. As in Figure 2 and Figure 3, the 'too fast' condition demonstrated the strongest formation of agency, both explicitly and implicitly.
Figure 4: Average explicit and implicit measures of agency for each feedback condition, combining the results presented in Figure 2B and Figure 3B. The average explicit agency results are plotted on the x-axis and the average interval estimates are plotted on the y-axis. The error bars denote standard deviation. Moving from left to right on the x-axis indicates a decrease in the explicit experience of agency, and moving from bottom to top on the y-axis indicates a decrease in the implicit sense of agency. This figure has been modified from Marasco et al.8. Please click here to view a larger version of this figure.
Here a methodological framework is presented to characterize the experience of agency formed while operating sensate prostheses via NMIs. In this context, agency is particularly relevant as it bridges physical action to the background cognitive processes that shape perception. Through a participant's prosthesis and NMI, we have direct access to a number of key elements that establish the sense of agency: intent, motor output, and movement sensation. Of importance to advanced prosthetic limb control, the tools provided in this work leverage this direct access to help unlock an understanding of how these elements may promote the user's sense of control over, and the cognitive integration of, the actions of their prosthesis.
The techniques highlighted are flexible in that they can be employed with any NMI research and clinical prosthetic system so long as they meet the criteria for real-time perceptually relevant control and kinesthetic feedback. The advantage inherent to many NMIs is the potential for intuitive control achieved by leveraging the neural pathways that remain postamputation. This allows for the measurement of residual physiological activity that once accompanied intact limb movement, which can, in turn, be decoded and mapped to the appropriate virtual or prosthetic limb movement. Therefore, most NMI techniques should satisfy the requirement for perceptually relevant control, provided that the recorded neural activity and the accompanying digital interface can produce reliable output signals that can be appropriately mapped to the virtual hand. The experimental setup also requires a system providing investigators with the ability to actively initiate kinesthetic sensations in real-time with the displayed virtual hand kinematics. This is a critical requirement as a sense of agency over movements is established when we engage in an action and appropriate sensory feedback is returned during the completion of that action13. Again, as long as this criterion is met, most any NMI kinesthetic feedback system will be appropriate.
The techniques presented here have the advantage of evaluating both explicit and implicit cognitive-perceptual measures of agency. There is evidence to suggest that each may be a result of separate cognitive mechanisms that, together, form a complete sense of agency17; however, there is still not a complete understanding of this relationship. The results from these measures are quantitative and easily interpreted. Decreases in time interval estimates suggest that a stronger implicit sense of agency was formed. Similarly, higher questionnaire scores on agency statements indicate a stronger explicit experience of agency. It is suggested that these quantitative values can provide a basis to evaluate and tune NMI control and sensory feedback. For example, in a previous work8 that is reported here in Figure 4, participants often reported smaller perceived time intervals and explicitly reported stronger perceived agency when a virtual hand was displayed that closed slightly faster than the kinesthetic sensation they experienced. This indicates that the user felt a stronger sense of control over the actions of the hand, as reported explicitly, but also suggests that the cognitive processes that establish this sense of control more strongly associate with this faster kinematic display. As such, an adjustment to the NMI control scheme of a clinical prosthesis to accommodate faster hand closing may help improve the user's perceptions of control over their physical device and encourage the user to identify their device's actions as self-generated.
The techniques presented may also be employed to form a more complete understanding of how multiple sensory modalities may influence perceptions of ownership over artificial limbs. For example, touch sensory feedback (or other sensory modalities) may be incorporated into the paradigm presented here to evaluate their possible individual roles in potentiating the sense of agency. Additionally, the techniques presented here may be paired with measures of ownership to more comprehensively characterize the interrelationships between agency, embodiment, and individual sensory modalities. The methods provided may also have broader applicability beyond NMI-controlled devices. Similar experimental tasks could be implemented with complex control systems (such as myoelectric pattern recognition), traditional myoelectric prostheses, and body-powered systems, as well as systems without NMI sensory feedback. This may allow for a unique perspective in understanding how cognitive processes respond to less 'natural' control and feedback paradigms and provide insights into how agency and perceptions of control may act during the operation of more traditional research or clinical prosthetic systems.
As robotic prostheses grow increasingly sophisticated, so too does the need for effective control and cognitive integration of these devices. Sensation is a pathway to addressing a number of critical barriers, and being able to assess the underpinning mechanisms that process movement sensation and information is an important piece. The tools provided here can help facilitate the integration of devices with users by characterizing the explicit and implicit formation of agency. These techniques help quantify the benefits of the innate access to intuitive motor control and sensation that NMIs may provide and can offer a platform for assessment and tuning, to ultimately improve the user's perception of being in control of their artificial limb.
The authors have nothing to disclose.
The authors would like to thank Madeline Newcomb for her contributions to the figure generation. This work was funded by the U.S. taxpayers through an NIH, Office of the Director, Common Fund, Transformative R01 Research Award (grant #1R01NS081710-01) and the Defense Advanced Research Projects Agency (contract number N66001-15-C-4015 under the auspices of Biology Technology Office program manager D. Weber).
LabVIEW 2015, Service Pack 1, Version 15.0.1f2 64-bit | National Instruments, Austin, TX, USA | Full or Pro Version | We wrote custom software in LabVIEW to coordinate virtual prosthesis control with kinesthetic feedback as well as to present experimental conditions and record data. |
8-Slot, USB CompactDAQ Chassis | National Instruments, Austin, TX, USA | cDAQ-9178 | |
±60 V, 800 kS/s, 12-Bit, 8-Channel C Series Voltage Input Module | National Instruments, Austin, TX, USA | NI-9221 | |
100 kS/s/ch Simultaneous, ±10 V, 4-Channel C Series Voltage Output Module | National Instruments, Austin, TX, USA | NI-9263 | |
Custom Wearable Kinesthetic Tactor | HDT Global, Solon, OH, USA | N/A | This item was custom made. Other methods of delivering kinesthetic feedback are acceptable as long as the participant feels the sensation of the hand moving in real-time with the movements of the virtual hand. |
MuJoCo Physics Engine, HAPTIX Version | Roboti LLC, Redmond, WA, USA | mjhaptix150 | Newer versions of MuJoCo should be acceptable as well. We used the MPL Gripper Model. |
Myobock Electrodes, powered by Otto Bock EnergyPack in MyoBoy Battery Receptacle | Ottobock, Duderstadt, Germany | electrodes: 13E200=60 battery: 757B21 battery receptacle: 757Z191=2 |
Any setup that provides an amplified, filtered, and rectified EMG or neural control signal could be used. |