The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs to enable the study of brain-robot interaction with humanoid robots, which is prospective in assisting the sick and elderly as well as performing unsanitary or dangerous jobs.
Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives. The overall goal of our method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, and Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots.
This is achieved by first placing EEG electrodes on a human subject to measure the brain responses through an EEG data acquisition system. A user interface is used to elicit SSVEP responses and to display video feedback in the closed-loop control experiments. The second step is to record the EEG signals of first-time subjects, to analyze their SSVEP features offline, and to train the classifier for each subject. Next, the Online Signal Processor and the Robot Controller are configured for the online control of a humanoid robot. As the final step, the subject completes three specific closed-loop control experiments within different environments to evaluate the brain-robot interaction performance.
The advantage of this approach is its reliability and flexibility because it is developed by integrating multiple software programs. The results show that using this approach, the subject is capable of interacting with the humanoid robot via brain signals. This allows the mind-controlled humanoid robot to perform typical tasks that are popular in robotic research and are helpful in assisting the disabled.
Brain-Robot Interaction (BRI), which provides an innovative communication pathway between human and a robotic device via brain signals, is prospective in helping the disabled in their daily lives 1,2. A variety of methods are able to acquire brain signals either invasively or non-invasively, such as electrocorticography (ECoG), electroencephalograph (EEG), functional magnetic resonance imaging (fMRI), etc. The most commonly used non-invasive method for building the BRI system is to acquire EEG signals from electrodes placed on the scalp. This method is inexpensive, easy to use, and provides an acceptable temporal resolution 3. Among a variety of robotic devices, humanoid robots are advanced as they are created to imitate some of the same physical and mental tasks that humans undergo daily. BRI with a humanoid robot will play an important role in assisting the sick and elderly, as well as performing unsanitary or dangerous jobs. But control of a humanoid robot through BRI system is highly challenging, as the humanoid robot with full body movement is developed to perform complex tasks such as personal assistance 4, 5.
Steady-State Visual Evoked Potential (SSVEP) is a type of brain signal evoked by the modulation of visual stimulus at a given frequency 6. It contains sinusoids at the fundamental and harmonic frequencies of the flickering stimulus, and prominently appears throughout the visual cortex in the occipital region of the scalp 7. The reason for choosing the SSVEP signals is that the SSVEP-based BRI system yields relatively high information transfer rate and requires less training 8. Other types of brainwaves, such as event-related potentials (ERPs) 9 or motor-imagery (MI) potentials 10, can also be embedded into this experimental procedure.
Our procedure for brain-robot interaction with humanoid robots is based on Cerebot – a mind-controlled humanoid robot platform – consisting of an EEG data acquisition system and a humanoid robot 11. The EEG system is able to record, pre-process and display bio-potential signals acquired by various types of electrodes. It provides multiple analog I/Os and digital I/Os and is capable of recording up to 128 signal channels simultaneously at a sampling rate of 30 kHz with 16-bits resolution. Its software development kits in C++ and MATLAB are easy for users to design the experimental procedures. The humanoid robot has 25 degrees of freedom and is equipped with multiple sensors, including 2 cameras, 4 microphones, 2 sonar rangefinders, 2 IR emitters and receivers, 1 inertial board, 9 tactile sensors, and 8 pressure sensors. It provides Choregraphe and C++ SDK for creating and editing movements and interactive robot behaviors.
The overall goal of this method is to establish an SSVEP-based experimental procedure by integrating multiple software programs, such as OpenViBE, Choregraph, Central software as well as user developed programs written in C++ and MATLAB, to enable the study of brain-robot interaction with humanoid robots 11. Figure 1 shows the system structure. The dedicated stimulus presentation computer (SPC) displays the User Interface to provide the subject with visual stimuli, instructions and environmental feedbacks. The dedicated data processing computer (DPC) runs the Data Recorder and the Offline Data Analyzer in the offline training process, and runs the Online Signal Processor and the Robot Controller for the online control of the humanoid robot. Compared with other SSVEP-based control systems, our system is more reliable, more flexible, and especially more convenient to be reused and upgraded as it is developed by integrating a number of standardized software packages, such as OpenViBE, Choregraph, Central software, and modules written in C++ and MATLAB.
The following procedure was reviewed and approved by Tianjin medical university general hospital ethics committee, and all subjects gave written consent.
1. Acquiring EEG Signals
2. Offline Analyzing SSVEP Features
3. Online Processing Brain Signals
4. Connecting the Humanoid Robot
5. Conducting Closed-loop Control Experiments
The results presented here were obtained from a male subject having corrected-to-normal version. Figure 7 shows the procedure of processing EEG data, including extracting a multichannel data epoch (Figure 7A), spatially filtering the data using CCA coefficients (Figure 7B), and calculating the normalized PSD (Figure 7C).
Figure 8 shows the normalized PSD obtained using single trials in which the subject stared at the targets flickering at different frequencies. The prominent peak appears at the fundamental or harmonic frequency of the flickering stimulus. The BRI system maps these patterns of SSVEP responses into commands for the control of different robot behaviors.
Figures 4-6 show the three closed-loop control tasks performed to evaluate the brain-robot interaction performance. These tasks are popular in robotic research and are helpful in assisting the disabled and elderly in their daily lives. Figure 4 shows the telepresence control experiment of walking through obstacles and pushing a switch to turn on the light via brainwaves. Figure 5 shows the telepresence control experiment of walking toward the stair case following the exit sign. Figure 6 shows the telepresence control experiment of picking up a balloon and delivering it to the subject's hand.
Figure 1. System Structure for Brain-Robot Interaction with a Humanoid Robot. The brain signals are measured through the EEG data acquisition system. The user interface elicits SSVEP responses and displays live video feedback in the closed-loop control experiments. For first-time subjects, the Data Recorder and the Offline Data Analyzer are used in the offline training process to analyze their SSVEP features offline and to train the classifier for each subject. Then the Online Signal Processor and the Robot Controller are configured for the online control of a humanoid robot. Please click here to view a larger version of this figure.
Figure 2. User Interface for the SSVEP-based BRI System. The User Interface displays live video feedback in the middle window and flickers four images on the periphery representing humanoid robot behaviors at four frequencies. The 3D representation at the right panel indicates the current posture of the humanoid robot. Please click here to view a larger version of this figure.
Figure 3. Online Signal Processor implemented in the OpenViBE programming environment. The Acquire and Process Signals box marked with the red cycle invokes the processing algorithm written in MATLAB scripts. The Starts button on the menu panel starts up the program. Please click here to view a larger version of this figure.
Figure 4. Telepresence Control Experiment of Walking Through Obstacles and Pushing a Light Switch. Please click here to view a larger version of this figure.
Figure 5. Telepresence Control Experiment of Walking Toward the Staircase Following the Exit Sign. Please click here to view a larger version of this figure.
Figure 6. Telepresence Control Experiment of Delivering a Balloon to the Subject. Please click here to view a larger version of this figure.
Figure 7. Procedure of Processing Multichannel EEG Data. (A) Multichannel data epoch extracted from the trial in which the subject is staring at the stimulus at 4.615 Hz; (B) Spatially filtered data using CCA coefficients; (C) Normalized PSD of the spatially filtered data. Please click here to view a larger version of this figure.
Figure 8. Normalized PSD Obtained in Single Trials in Which the Subject Is Staring at Stimuli Flickering at Different Frequencies. Please click here to view a larger version of this figure.
This paper presents an SSVEP-based experimental procedure to establish the brain-robot interaction system with humanoid robots by integrating multiple software programs. Because human intent is perceived by interpreting real-time EEG signals, it is critical to verify the electrode connections and EEG signal qualities before conducting the experiment. If signals acquired from all the electrodes are of poor qualities, it is necessary to check the connection of the ground and reference electrodes first. If there are problems with parts of the electrodes, re-inject conductive gel to adjust the impedance of those channels.
Another common issue while acquiring EEG signals is the interference of artifacts and noises 17. As the EEG signal has small signal to noise ratio (SNR), artifact and noise can easily obscure changes in PSD of the SSVEP responses. It is important to keep the signal amplifier of the EEG system away from the power supplies and other noise sources. During the experiment, eye blink and body movement should be kept to a minimum to reduce artifacts. To further improve the robustness against such interferences, our method utilizes a CCA-based technique to interpret multichannel EEG data. The results show that this technique is effective in extracting features from EEG signals containing noises and artifacts.
To begin a new task of closed-loop control of humanoid robots, we need to manually configure the definition and parameter of each robot behavior to fit the new task requirement. This is due to the limited number of control commands available in the current BRI system, and thus can be improved by evoking more types of brainwave patterns. Hwang et al.18 proposed a dual-frequency stimulation method for producing more visual stimuli. Wang et al. 19, Allison et al. 20, Pan et al. 21, and Li et al.22 proposed several hybrid methods to combine the SSVEP-based model with other brainwave patterns, including ERPs and MI. It is also feasible to adopt switching techniques based on machine learning or hierarchical architecture to control full body movement of the humanoid robot using limited brainwave patterns 23.
Considering the available flashing frequencies of the LCD monitor 24 and the influence among harmonic components of SSVEPs 25, we scanned all the possible flashing frequencies from 1 to 60 Hz and found using the four frequencies, i.e., 4.615, 12, 15, and 20 Hz, are likely the best choice as they achieved the highest average accuracy rate for our subjects. Therefore, we used the four stimuli on the interface to control the humanoid robot behaviors, including walking forward, turning left, turning right, and stopping walking/pushing the switch/picking up objects, which are feasible enough to control a humanoid robot to accomplish the tasks presented in this manuscript.
The advantages of the BRI system are its reliability and flexibility as it is developed by integrating multiple software programs, such as OpenViBE, Choregraph, Central software and user developed programs in C++ and MATLAB. It is efficient and reliable for designing different experimental procedures using the standardized software. Our system is a powerful tool to investigate new algorithms and techniques for the brain-robot interaction with a humanoid robot. It can be easily upgraded to explore BRI applications in assisting the sick and elderly, and performing unsanitary or dangerous jobs.
The authors have nothing to disclose.
The authors would like to express their gratitude to Mr. Hong Hu for his assistance in performing the experiments reported in this paper. This work was supported in part by the National Natural Science Foundation of China (No. 61473207).
Cerebus EEG Data Acquisition System | Blackrock Microsystems | 4176-9967 | |
NAO humanoid robot | Aldebaran Robotics | H25 | |
EEG cap | Neuroscan | 8732 | |
Ten20 Conductive gel | Weaver and company | 10-20-8 |