Combining Computer Game-Based Behavioural Experiments With High-Density EEG and Infrared Gaze Tracking

1. Design an Entertaining and Scientifically Informative Video Game

1. Apply an iterative game design process in which concerns of scientific value and playability inform each other. As an experimenter, you have ideas as to the stimuli and behavioural paradigms that you want to see built into a computer game. Because you are not a game designer, the task of building these paradigms into games may seem a detail that can be addressed after most of the work has been completed. Nothing could be further from the truth. Whether or not a game will attract motivated players – and thus whether or not your data will be collected under conditions of maximum ecological validity – depends fundamentally on a good design. Game design is distinct from game programming and implementation, and usually is done by different people with different expertise. Your budget may allow for professional designers, or it may allow only for student designers – whichever the case, though, design should be treated as a task distinct from implementation, and a task conversant with, but not synonymous with, specification of the experimental paradigm. Not all of your ideas and constraints will be realisable as a playable game. A good designer will come back to you with questions and suggestions as to how your experimental paradigm might be made more flexible in order to produce a playable, entertaining game. In our experience, game design for experimental science is an iterative process in which the experimenters hand the game designers a set of constraints, the game designers hand back design ideas and suggestions for modifying these constraints, the experimenters reframe the constraints in response to this feedback, and so on.
2. Design for players of both sexes. At a population level, culturally and biologically based cognitive traits distinguish males and females (Valla et al., 2010). Many standard video game formats – in particular the “first-person shooter” – appeal to a typically male cognitive profile. Standard game designs, then, introduce a male recruitment bias and a confound between behavioural performance and sex. The unfortunate reality is that people who specialise in ungendered game design (Graner Ray, 2004) are in short supply. Beware that most gaming professionals (and students) are male, and that when left unsupervised with an experimental paradigm they will, almost inevitably, end up designing a first-person shooter around it.
3. Use game time to collect repeated trials efficiently. Game-based event-related potentials experiments require collection of many repeated trials (typically at least a hundred per condition) of many separate cognitive tasks to be performed during a single experimental session without fatiguing the experimental subject. Consider the proportion of game play time during which experimental trials will actually be being performed. How much of the player’s contact time with the game will be directly scientifically useful in providing experimental data, and how much of this contact time will be “filler” that links these experimental trials together into a game narrative? Design so as to maximise the proportion of game time that will be useful for data acquisition. To avoid frank repetition, consider interspersing experiments of different types, for instance an active behavioural task punctuated by a passive sensory stimulus. After rejecting any bad trials, can you accumulate enough trials for physiological averaging, without the player’s becoming impatient or bored with the game? If the answer to this question is no, you must modify the design.
4. Avoid bloating experiments with additional factors or conditions. It may seem tempting to add conditions and variations within experimental paradigms, so as to address related questions – for instance, in an attention task how might electrophysiology and behavioural performance be affected in a context in which distractors can appear, versus a context in which every stimulus is task-relevant? Or in a context of multimodal versus unimodal sensory cues? In the best case, such factoring would add useful information, and physiologically adequate numbers of trials would be acquired within each factor. In the worst case, though, factoring leads to an “experiment bloat” in which no individual condition carries a sufficient number of observations, the analysis must therefore collapse observations across conditions, and the only result is a problematic increase in within-sample variance. This problem of “experiment bloat” becomes more significant when experiments are implemented in a game format, because variety is a desirable property in a game. Add factors only if you can be certain that each factor individually will contain a number of trials sufficient for averaging, without driving the player to boredom.
5. Avoid timed events; give the player control of when things happen; whenever possible, prompt the player. Many neuropsychiatric populations have difficulty with executive function, that is, in rapidly planning and executing actions in response to sensory inputs. They may have a great deal of skill and in some tasks even exceed normal performance – but this can be an intensely studied and considered, deliberate style of skill, often not expressed under time pressure. So it’s important to make certain that the timing of events is controlled not by the computer (except in cases where the experiment requires it) but by the player. Small additions such as a “next” button or a “ready” button can make all the difference. For example, part of our game (Figure 1) implements a “go/no-go” test of executive inhibition, in which the player is required to respond or to withhold a response based on the identity of a target that emerges from a space “wormhole.” Rather than causing the target to emerge at a completely arbitrary time, our software waits for the player to launch a beam that opens the wormhole. The target then emerges after a random delay. This approach allows measurement of anticipatory and response-related brain electrical events whilst still permitting the player to prepare for the behavioural context. Thus, wherever the constraints of the experimental paradigm permit, games should be player-centred rather than computer-centred, and game play should be event-driven rather than time-driven.
6. Do not depend on a player’s memory for instructions; prompt the player every time. A corollary of the executive function issue is that the player may have trouble remembering a sequence of steps. Even if (s)he has learnt in a tutorial that key A triggers action X and key B triggers action Y, these arbitrary associations might not be remembered, unless the player has had a chance to practise these actions actively, many times over.
7. A further corollary: Do not make input-output mappings depend on the game state. Designers and especially implementors might be tempted to hide functions inside sub-menus, access to which depends on the player’s clicking on the correct primary menu, or to make a drag of the mouse do something different after a click than after no click (or even worse, something different after a left-click than after a right-click). Avoid such sequential logic in the user interface. Wherever possible, use purely combinational logic. Some sequential logic is of course unavoidable – for instance in the transition from one game context or mini-game to another – but it ought to be used sparingly and only when absolutely required.
8. Instead of a sequence of actions, ask for one action at a time. Rapid actions can be difficult enough by themselves, but when neuropsychiatric patients face the additional demand of performing several of these actions rapidly and in the proper sequence they can feel very much overwhelmed. Instead of requiring sequences of inputs in response to a single prompt, try to prompt separately for each input.
9. Use pictures, not exclusively words. Players with deficits in language, reading, attention, or memory might not comprehend textual instructions – not because the player is incapable of comprehending but because (s)he’s concentrating so much on decoding the individual words that (s)he can’t spare much effort to put those words together into the meanings of complete sentences and narratives. Sometimes text is unavoidable; if text is used, avoid verbosity, don’t clutter the display with words, do include series of “next” prompts that separate passages into manageable chunks, and do allow the player to go backwards through these prompts to review text that (s)he has already seen.
10. Players should learn by doing, not just by observing or reading or listening. In this regard, patient populations are no different from people in general: we all learn best when we can be active rather than passive learners. The challenges faced by neuropsychiatric patients make it even more crucial that game activities involve learning-by-doing, rather than depending on learning-by-reading or learning-by-listening. This is particularly true of the game tutorials or instructions.
11. Avoid depending on simultaneous or near-simultaneous events in different perceptual channels (e.g. different places on the screen, or different senses such as video with audio). Some neuropsychiatric patients may have difficulty with perceptual integration, and may focus on only one perceptual channel at a time. When focusing on one point or area of the display, events away from this spatial focus of attention may not register. Anathema for this sort of player would be a cockpit display with many gauges indicating different quantities which all need to be observed simultaneously – or a visual display that needs to be observed at the same time as a spoken or other auditory signal. Instead, either information should be displayed in one region of the display or one sensory channel, or ample time should be allowed to shift attention between points in visual space or between sensory channels. In players with autism, for instance, shifts of attention can take 2 to 3 seconds (Belmonte, 2000) – ten times as long as for other players! Think about what it would be like to be looking at the display through a long telescope that magnifies a small area but shuts out the periphery.
12. Avoid evoking unnecessary anxiety. Neuropsychiatric populations can be much more prone to anxiety than other players – especially when faced with a new and unpractised task, or with a timed task, or with an interactive scenario that’s out of their control. Do everything possible to make certain that the player, not the computer, is the one controlling what happens next, and that the player has every opportunity to practise and to become comfortable with the demands of the game. Consider including a tutorial that allows the player to go through all the actions of the game – e.g. clicks and key presses – as (s)he receives instruction.
13. For repeating events, vary the timing slightly so that the time between two successive instances of the event is not constant. Signal processing experts will know the phenomenon of aliasing, in which a discrete sampling of a high-frequency signal at too low a sampling rate produces an artefactual low-frequency oscillation. The issue surrounding EEG measures of repeated events has much in common with aliasing. Consider as an example the situation that exists when a movement key is pressed and held: the player’s avatar will move at a certain rate, say, once every 500 ms. Suppose that one is interested in the brain response to the movement effect. Suppose also, though, that there is an ongoing, endogenous (that is, internally driven) 10 hz oscillation in visual cortex that has nothing directly to do with this exogenous phenomenon. Since 500 ms is an integral multiple of the 100 ms period of this oscillation, sampling the exogenous brain response to each movement will also sample the endogenous oscillation at the same point in its phase every time, and the analysis thus would misattribute the endogenous signal as an exogenous response to the movement stimulus. To forestall this ambiguity in EEG analysis, one can add a small amount of temporal jitter to the intervals between stimuli (Luck, 2005, p. 135) – not so much as to make them seem unnaturally variable to the player, but enough to get rid of this phase artefact. The exact amount depends on what seems natural given the event separation; in this example of a 500 ms event we might deem that any variation more than 10% of the interval in either direction would seem unnatural, so we might then choose to vary the intervals over a uniform distribution from 450 ms to 550 ms. Add as much temporal jitter as seems natural, without sacrificing playability.
14. Design with extensibility in mind. Bringing a complete computer game from concept to realisation is a time-consuming and labour-intensive undertaking – you may find yourself, effectively, working a day job as a researcher and a night job as a game designer and project manager! It makes sense, therefore, to make the game flexible and extensible, so that experiments that may not have been conceived when the game was first designed can be added without defining and implementing an entirely new game system. This goal of extensibility can be realised partly in the game design and partly in the software design.
    
    In the game design, consider a system that allows addition of new game modules. In our system, a main game is supported by an extensible set of mini-games: players must enter each of the mini-games in order to acquire or to keep resources that are of value in the main game. Each mini-game embeds two to three experiments. For example, having designed a space colony, players enter an executive-function and visual perception mini-game in which they steer a spaceship through a drifting star field and decide how to respond to friendly or opponent spaceships (Figure 1), a visual attention mini-game in which they detect asteroids that can be mined for raw materials to build that colony (Figure 2), an emotion-perception mini-game in which they help in a diplomatic negotiation by matching up the faces of different people who are showing the same emotions (Figure 3), and a social cognitive mini-game in which they thwart pirates who want to steal their colony’s supplies (Figure 4). In practice, it’s fairly simple to find a conceit within which a new experimental task can be integrated into the game narrative – but the general facility to allow such integration must be designed in a priori.
    
    Although players are aware of the general fact that they’re performing experiments, the experimental data collection is unobtrusive: During the steering task a maximum-likelihood estimator (Pentland, 1980; Lieberman & Pentland, 1982) computes the player’s psychophysical threshold for perception of coherent motion of the drifting star field. During the asteroid task, gaze is maintained at the bottom centre of the display in order to watch for impurities in the ore processor (and this gaze direction is verified by the gaze tracker), whilst each of the four sectors in which asteroids might appear flickers at a different fundamental frequency (the least common multiple of which is large) and covert attention is directed to one of these sectors in response to a spatial cue. Changes in the spectral content of EEG can then assess the time course of covert attention based on the EEG amplitudes of the flicker frequencies that tag each sector (Morgan et al., 1996; Belmonte, 1998).
    
    In the software, the “game engine” should provide not only the usual core elements for graphical display (e.g. a particle engine), but also all the common facilities for experimental control and data logging that will be needed by all experiments. In particular, the game engine should provide methods to display externally supplied still and moving-picture graphical assets and sounds, and an event logger that writes to a local disc file and also through an output port (we use the standard parallel port) to synchronise with a behavioural or physiological data recorder such as a gaze tracker, an EEG system, or an fMRI scanner.
15. Provide a method for logging game events. In our system, the log file on disc contains a superset of the data sent through the parallel port: whereas the parallel port receives only unsigned 8-bit event codes 1 through 255, the disc file includes (1) a time stamp in clock ticks, (2) a time stamp in microseconds, (3) the numeric event code sent through the parallel port, (4) a string mnemonic unique to this event code, and (5) a list of (parameter name, parameter value) pairs. For instance, the appearance of a stimulus (e.g. an asteroid) at particular absolute or angular display coordinates could be denoted with an appropriate event code and mnemonic with the two coordinates as parameters. Because there may be more than 255 unique events, consider issuing event codes that mark the beginnings and ends of self-contained contexts (e.g. individual mini-games or game scenarios) within the main game, and re-using event codes from one context to another. The time series of numeric event codes logged by the external data recorder and the time series of detailed and parameterised event codes logged in the local log file then can be used to place the log file and the external data file(s) in temporal register.
16. Log event codes for absolutely everything. Did the player’s avatar just move (either because the movement key has just been depressed, or because it’s been held down constantly and is repeating)? That’s an event. Did some sort of motion start or stop or change speed? That’s an event. Absolutely everything that happens in the game should be reported with an event code. Experimenters can always ignore event codes if they decide that they aren’t of interest in the analysis. What one can’t do, of course, is to go back and insert event codes after the data have been recorded. So put everything in – you never know what might be useful, perhaps not immediately but in some post hoc data mining.

2. Prepare Equipment

1. An hour before the subject arrives, balance the electrodes by soaking them in a salt bath (1 teaspoon table salt per 1 litre distilled water) for 5 to 10 minutes. Do not leave electrodes in any liquid for longer than 10 minutes at a time. Balanced electrodes have small (+/- 20μV) offsets.
2. Just before subject arrives, switch on the gaze tracking camera, converter boxes, and computers.
   
   Our setup uses four computers (see Figure 5): one dedicated gaze-tracking computer (GC), one dedicated EEG acquisition computer (EC), one dedicated stimulus presentation computer (SC), and one computer for video acquisition and data analysis (VC).
   
   Two converter boxes (Figure 6) manipulate VGA outputs from GC and SC and send a spliced video signal to VC. In this way, the screen seen by the subject overlain with a cursor representing current fixation and a timestamp can be recorded to a video file on VC.

3. EEG Setup

1. Measure the circumference of the subject’s head around the brow and inion. Select an electrode cap such that the subject’s circumference is near the middle of the measurement range for the cap.
2. Record the measurement from the subject’s nasion to inion and left pinna to right pinna.
3. Place the cap on the subject’s head. Ensure that the tag is outside the cap, resting on the subject’s neck. Reposition the cap until A1 (the electrode at the vertex) is centred with respect to the nasion-inion and pinna-pinna axes, and the cap midline (A25-C17) is parallel to the midline of the subject’s head.
4. Apply an adhesive ring to the plastic housing of electrodes EX5 and EX6. Align the opening of the ring with the electrode pellet. Remove the paper backing from the adhesive ring and cover the electrode contact with conductive gel. Place EX6 on the subject’s right mastoid and EX5 on the subject’s left mastoid.
5. Use a syringe to place conductive gel in every electrode housing. Wiggle the syringe tip to part the subject’s hair, then simultaneously depress the plunger and pull the syringe away from the head. Fill until gel is flush with the top of the plastic housing.
   
   It is better to have too little gel than too much. In the case of too little gel, more can always be added. With too much gel, the excess can bleed between electrode sites, causing electrode bridging. If electrodes become bridged, remove the cap, have the subject wash and dry their hair, and begin again.
6. With the plug end over the shoulder and the sensors in one hand, gently place each electrode in its corresponding housing. Grasp only the plastic housing of the electrodes and be careful not to crimp the wires.
   
   It is critical not to touch the electrode tips. Contact with skin or clothing will degrade the quality of electrodes.
7. Place conductive gel on EX1-EX4 and use adhesive rings to attach them to the subject’s face. Place EX1 and EX2 about 1cm horizontally from subject’s left and right outer canthi, respectively. Place EX3 and EX4 about 1cm below the middle of the subject’s left and right eyes on the zygomatic bones.
8. Gently collect electrode leads behind the subject and loosely wrap the CMS/DRL lead around the others to create a pony-tail. Place Velcro ties at the top and bottom of the pony-tail to hold the leads in place. Use medical tape to affix the pony-tail to clothing on the subject’s back.
9. Apply 0.5% KCl solution (Lykken & Venables, 1971) to each GSR electrode. Use medical tape to affix GSR electrodes to the index and ring fingers of the subject’s non-dominant hand.
10. Have subject sit in a non-reclining, stationary chair in front of the stimulus presentation monitor. Plug all electrodes into the EEG converter box.
    
    Small (+/- 40 μV) offsets are acceptable. If any electrode shows an offset greater than +/- 40 μV, gently remove the electrode from the cap, apply more gel, and return the electrode.

4. Gaze Tracking Setup

1. Affix a target sticker above the subject’s eyebrow medial to the eye.
2. Launch the EyeLink Popup-calibration application on VC. Begin a new session and use the CMD interface to switch on event logging for the gaze tracker. Set the value ‘file_event_filter’ equal to ‘LEFT,RIGHT,FIXATION,BLINK,MESSAGE,BUTTON,SACCADE,INPUT’ A full list of commands can be found in the file DATA.INI which is provided with the EyeLink software.
3. From the EyeLink Popup-calibration application, launch Camera Setup. Position the camera so that the target sticker and the subject’s eye are centred. Adjust the focus until the eye to be tracked is clear.
4. Calibrate and validate the gaze tracking system to the subject using the nine-point dot matrix. After validation, the EyeLink software labels each calibration point with the error in degrees of visual angle between the calibrated and validated measures. In an acceptably good calibration, the mean error across all the calibration points does not exceed 1°, and the error at any single point does not exceed 1.5°. In a very good calibration, mean error does not exceed 0.5° and the greatest single-point error does not exceed 1°.
   
   If calibration fails, ensure that pupil and corneal reflection thresholds are appropriate. If adjusting these values does not alleviate calibration problems, switch eyes and recalibrate. Ensure that the sampling rate is set to 500 hz by clicking on the 500 hz button on the left of the camera setup screen.
   
   The optical and computational processes by which gaze position is figured are internal to the gaze tracker and need not be known by users of this procedure. Briefly, the technique works by illuminating the eye with infrared light. Light impinging on the retina is reflected out of the eye on the same path along which it entered – this is the optical property that causes “red eye” in flash photographs taken with compact cameras. To a camera positioned well away from the light source, though, the pupil will appear dark. At the same time, some of the illumination is reflected from the cornea as a small, intense glint, the position of which depends only on the position of the head and not on the direction of gaze of the eye. The positional difference between the dark pupil and the corneal glint then can be mapped mathematically to direction of gaze (Ebisawa, 1998). The gaze-tracking computer logs a time series of the resulting point-of-regard coordinates, integrated with event codes from the stimulus-presentation computer.

5. Begin Computer Game

1. From the Popup-Calibration application, begin recording gaze data. Start video recording on VC and EEG recording on EC.
2. On SC, launch the video game.
3. For auditory stimuli, swap out the passive speakers in SC for powered speakers. Then plug in and turn on the amplifier. Using a sound level meter, set the volume to a level sufficient to attain the maximum amplitude (e.g. 80 dB) required by the experimental paradigm.

6. Clean Equipment

1. After the subject has finished the games, exit the game and stop recording data by pressing the ‘Stop’ button within the graphical interface on EC, GC and VC. Switch off and unplug the amplifier and replace the passive speakers.
2. Fill a small plastic bucket 4-5 cm deep with distilled water and microwave on high for 90 seconds.
3. Switch off the EEG converter box and unplug all leads from the converter box. Remove the medical tape and the Velcro ties from the pony-tail of EEG leads.
4. Grasping the electrodes by the plastic housing only, remove all electrodes and place the sensors in the warm distilled water. After the cap electrodes have been removed, remove the cap from the subject’s head. Remember to remove the adhesive rings from electrodes before immersing them in the water.
5. Use a spray bottle filled with distilled water to remove any gel left on the electrodes.
6. Use warm tap water and soap to remove gel from the electrode cap.

7. Data Analysis

1. On SC, convert the gaze-tracking EDF data file to ASCII text using the EyeLink edf2asc application.
2. Consolidate data files onto VC, then launch the Astropolis Processing Toolkit (APT).
   
   The APT is an open source MATLAB (R2008a, The MathWorks, Natick, MA) toolkit built on the open source EEGLAB environment (Delorme & Makeig, 2004). It integrates the various data files produced by this experimental paradigm and automates behavioural and EEG analysis. This automation includes preprocessing and artefact rejection, extended infomax independent component analysis (ICA) as implemented with the runica algortihm (Makeig et al., 1997) in EEGLAB, and equivalent dipole localisation.

8. Representative Results

The results presented here were obtained from pilot data examining children ages 10-15 years in three groups: individuals with an autism spectrum condition (ASC), siblings without any clinical diagnosis (SIB), and typically developing children (TYP). Gaze tracking data have been used to reject trials in which the subject’s gaze has deviated from the stimuli of interest. (More intricate applications of gaze data also are possible, for instance using gaze as a parameter in computing physiological and behavioural averages.)

Figure 7 shows an event-related spectral perturbation obtained from a frontal midline electrode (Fz). PresentEnemy is corresponds to the appearance of an enemy ship (Go) and PresentFriendly corresponds to the appearance of a friendly ship (No-Go). During the No-Go condition, the TYP group demonstrated significantly higher gamma power (30-75 hz) 500-1500 ms post-stimulus.

The APT allows for easy comparisons between physiological and psychometric measures. For instance, in our analysis, alpha power (8-12 hz) during a similar time period (300-1500ms post-stimulus) for No-Go trials during this non-social attentional task was negatively correlated with performance on a measure of social perception, the Benton Facial Recognition test (Benton et al., 1994).

Figure 1: video of Maritime Defender (go/no-go and dot-motion coherence tasks)
Click here to watch video.

Figure 2: video of Stellar Prospector (modified Posner visual spatial attention task)
Click here to watch video.

Figure 3: video of FaceOff (facial emotion recognition)
Click here to watch video.

Figure 4: video of StarJack (“Sally-Anne test” of theory-of-mind)
Click here to watch video.

Figure 5: Cabling diagram of the laboratory setup.

Figure 6: Converter box cabling diagram of the laboratory setup.

Figure 7: Event-related spectral perturbations of Go/No-Go task recorded at Fz. PresentEnemy = Go stimulus; PresentFriendly = No-Go stimulus; ASC = autism group; SIB = sibling group; TYP = control group.