The present protocol combines functional near-infrared spectroscopy (fNIRS) and video-based observationto measure interpersonal synchronization in quartets during a collaborative drawing task.
Functional near-infrared spectroscopy (fNIRS) is a noninvasive method particularly suitable for measuring cerebral cortex activation in multiple subjects, which is relevant for studying group interpersonal interactions in ecological settings. Although many fNIRS systems technically offer the possibility to monitor more than two individuals simultaneously, establishing easy-to-implement setup procedures and reliable paradigms to track hemodynamic and behavioral responses in group interaction is still required. The present protocol combines fNIRS and video-based observation to measure interpersonal synchronization in quartets during a cooperative task.This protocol provides practical recommendations for data acquisition and paradigm design, as well as guiding principles for an illustrative data analysis example. The procedure is designed to assess differences in brain and behavior interpersonal responses between social and non-social conditions inspired by a well-known ice-breaker activity, the Collaborative Face Drawing Task. The described procedures can guide future studies to adapt group naturalistic social interaction activities to the fNIRS environment.
Interpersonal interaction behavior is an important component of the process of connecting and creating empathic bonds. Previous research indicates that this behavior can be expressed in the occurrence of synchronicity, when biological and behavioral signals align during social contact. Evidence shows that synchronicity can occur between people interacting for the first time1,2,3. Most studies on social interactions and their underlying neural mechanisms use a single person or second person approach2,4, and little is known about transposing this knowledge to group social dynamics. Evaluating interpersonal responses in groups of three or more individuals is still a challenge for scientific research. This leads to the necessity of bringing to the laboratory the complex environment of social interactions in everyday human beings under naturalistic conditions5.
In this context, the functional near-infrared spectroscopy (fNIRS) technique is a promising tool for assessing the relationships between interpersonal interaction in naturalistic contexts and its brain correlates. It presents fewer restrictions on participant mobility compared to functional magnetic resonance imaging (fMRI) and is resilient to motion artifacts6,7. The fNIRS technique works by assessing hemodynamic effects in response to brain activation (changes in blood concentration of oxygenated and deoxygenated hemoglobin). These variations can be measured by the amount of diffusion of infrared light through scalp tissue. Previous studies have demonstrated the flexibility and robustness of the technique in ecological hyperscanning experiments and the potential to expand knowledge in applied neuroscience6,8.
The choice of an experimental task for the naturalistic assessment of the neural correlates of social interaction processes in groups is a crucial step in approaching applied neuroscience studies9. Some examples already reported in the literature with the use of fNIRS in group paradigms include music performance10,11,12, classroom interaction8, and communication13,14,15,16,17.
One of the aspects not yet explored by previous studies is the use of drawing games that have as the main feature the manipulation of empathic components to assess social interaction. In this context, one of the games frequently used to induce social interaction in dynamics among strangers is the collaborative drawing game18,19. In this game, sheets of paper are divided into equal parts, and the group participants are challenged to draw shared self-portraits of all members. In the end, each member has their portrait drawn in a collaborative way by several hands.
The objective is to promote quick integration among strangers, provoked by directing visual attention to the faces of the group partners. It can be considered an "ice-breaking" activity due to its ability to support curiosity and consequent empathic processes among the members19.
One of the advantages of using drawing tasks is their simplicity and ease of reproduction20. They also do not require any specific technical training or skills, as seen in the studies using musical performance paradigms21,22,23,24. This simplicity also enables the choice of a more naturalistic stimulus within a social context4,9,25.
Besides being an instrument for inducing social behavior in groups, drawing is also considered a tool for psychological evaluation26. Some graphic-projective psychological tests, such as House-Tree-Person (HTP)27,28,29, Human Figure Drawing – Sisto Scale27, and Kinetic Family Drawing30 are used in a complementary way for qualitative and quantitative diagnoses. Their results usually express unconscious processes, giving clues about the individual's symbolic system and, therefore, their interpretations of the world, experiences, affections, etc.
The practice of drawing makes one think and helps create meaning for experiences and things, adding sensations, feelings, thoughts, and actions31. It gives clues about how to perceive and process these life experiences26. Drawing uses visual codes to allow one to understand and communicate thoughts or feelings, making them accessible to manipulation and, thus, creating the possibility for new ideas and readings31.
In art therapy, drawing is a tool to work on attention, memory, and organization of thoughts and feelings32, and it can be used as means to produce social interaction33.
This study aimed to develop a naturalistic experimental protocol to assess vascular and behavioral brain responses during interpersonal interaction in quartets using a collaborative drawing dynamic. In this protocol, the evaluation of the brain responses of the quartet (individually and the synchronicity between partners) and the possible outcome measures, such as behavioral measures (drawing and gaze behavior) are proposed. The aim is to provide more information on social neuroscience.
The methodology was approved by the Hospital Israelita Albert Einstein (HIAE) Ethics Committee and is based on a procedure for collecting neural data (fNIRS), as well as gaze behavior data, with young adults during a collaborative drawing experience. All collected data were managed on the Redcap platform (see Table of Materials). The project was audited by the Scientific Integrity Committee of the Hospital Israelita Albert Einstein (HIAE). Young adults, 18-30 years old, were selected as subjects for the present study. Written informed consent was obtained from all participants.
1. Preparation for the study
Figure 1: The setting. The setup includes a squared table, four stools, and two wire supports (e.g., tripod), fNIRS equipment, a computer, and the cameras. (A) The setting scheme: Green numbers (1-4) correspond to the participants' labels and their stools/positioning at the table during the experimental run. Yellow numbers: 1 = fNIRS wiring supports, 2 = fNIRS signals' notebook receiver, 3 = NIRSport, 4 = 360° camera, 5 = support cameras. (B) Setting ready for the experimental run. Please click here to view a larger version of this figure.
Figure 2: Collaborative portraits-examples of portraits drawn in a collaborative way. Please click here to view a larger version of this figure.
2. Experimental paradigm
Table 1: Collaborative drawing condition. S1 = Subject 1, S2 = Subject 2, S3 = Subject 3 and S4 = Subject 4. Drawing dyads represents who is drawing who, and the drawing strip represents the writing paper's position for drawing in each condition. For example, for the first Block, use a blue paper sheet. C1, C2, and C3 represent 40 s of the paradigm of drawing social conditions that complete one portrait. C1 (drawing the forehead area, drawing dyads: S2 and S4; S1 and S3), C2 (drawing the nose area, drawing dyads: S1 and S4; S2 and S3) and C3 (drawing the chin area, drawing dyads: S3 and S4; S1 and S2). Follow the diagram for Blocks 2 and 3. This randomization maintains the order of drawing among volunteers (drawing the frontal partner, then the front-side partner, and lastly, the partner sitting next to them) and alters the order of the sheet strips to be drawn. Please click here to download this Table.
Table 2: Sequence 1-task randomization (social, non-social, and resting). Please click here to download this Table.
3. Video setup and data acquisition
Figure 3: Examples of individual drawing patterns. Please click here to view a larger version of this figure.
4. fNIRS setup and data acquisition
Figure 4: Distribution of optodes on the Subject 1 cap. The letters S and D represent the sources and detectors, respectively. S1 on AF7 coordinate of the 10-20 system; S2 on AF3; S3 on AF8; S4 on AF4; D1 on Fp1; D2 on F5; D3 on Fp2; and D4 on F6. The channels are placed in the following configuration: channel 1 between S1-D1; 2 between S1-D2; 3 between S2-D1; 4 between S2-D2; 5 between S3-D3; 6 between S3-D4; 7 between S4-D3; and 8 between S4-FD4. Please click here to view a larger version of this figure.
The protocol was applied to a quartet composed of young women (24-27 years old), all of them students on postgraduate programs (Hospital Israelita Albert Einstein, São Paulo, Brasil), with master's or doctorate level education. All participants were right-handed, and only one reported having previous drawing experience. No participants had a reported history of neurological disorders.
For the scales and psychological test results, two participants (2 and 4) showed high scores for anxiety (17 and 15 against the reference value of 9)44 and the cutoff value for depression (9)44. All participants' scale results for attention and hyperactivity showed scores below the cutoff values
The participants' social skills repertoire was also measured. Subjects 2, 3, and 4 obtained scores higher than 70% (well-developed repertoire of social skills). Subject 1 presented a score of 25% (related to a deficit in social skills). This test also analyses specific social skills such as F1, coping and self-assertion with risk; F2, self-assertion in expressing positive feelings; F3, conversation and social resourcefulness; F4, self-exposure to strangers and new situations; and F5, self-control and aggressiveness. For these factors, all participants showed low scores for F1, F2, and F3 (1% to 3%) and high scores for F4 (20% to 65%) and F5 (65% to 100%).
The fNIRs preliminary results (Figure 5) showed typical brain activation for subjects 1, 2, and 3 in both, social and non-social drawing conditions in both channels located in the left and right hemispheres; however, the activation patterns were distinct. Participant 4, on the other hand, showed atypical brain activation.
Figure 5: Results of the group average of fNIRS data. (A) Block average of fNIRS signals over subject 1. Left-side and right-side channels are displayed separately in x-axes for both conditions (social and non-social). (B) Block average of fNIRS signals over Subject 2. Left-side and right-side channels are displayed separately in x-axes for both conditions (social and non-social). (C) Block average of fNIRS signals over Subject 3. Left-side and right-side channels are displayed separately in x-axes for both conditions (social and non-social). (D) Block average of fNIRS signals over Subject 4. Left-side and right-side channels are displayed separately in x-axes for both conditions (social and non-social). Please click here to view a larger version of this figure.
The oxyhemoglobin signal (Figure 6) was significantly synchronized between subjects only during the last part of the task for both conditions, social and control (Figure 6: median correlation of 0.14; t-value = 1.77 and p-value = 0.046) and in the collaborative drawing condition (median correlation of 0.12; t-value = 2.39 and p-value = 0.028).
Figure 6: Boxplot of subjects' brain (oxyhemoglobin) correlations throughout the experiment. Each box contains a horizontal line (that indicates the median). The top edge represents the 75th percentile and the bottom edge the 25th percentile. For error bars, the boxplot is based on the 1,5 IQR value, above the third quartile, and below Q1, the lower quartile. The asterisk (*) indicates a statistically significant difference from zero. Part 1 corresponds to the first one-third of the experimental block (comprising C1, C2, and C3-the first full collaborative design-interspersed with resting periods and the non-social design condition); Part 2: C4-C6; Part 3: C7-C9. Please click here to view a larger version of this figure.
The video analysis showed that participants looked at each other while they were drawing partners and sometimes shared their gazes with one another. As with the amount of total gaze, a greater amount of synchronous gaze was observed in the positions where the partner was in a frontal position. From the middle to the end of the experiment, the synchronous gazes decreased significantly, and in C9, they did not occur.
Regarding drawing analysis, only Subject 3 reported having previous drawing experience (a 6 year course). Subjects 1, 2, and 4 had similar production in quantity and continuity of strokes. Participant 3 showed a drawing manner of short, non-continuous strokes and a greater number of total strokes. All four participants apparently maintained a constant pattern of figures in their drawings (previous drawing patterns), although subjects 3 and 4 reproduced a greater number of observed details. Even when drawing different partners, there were eye, mouth, and nose drawing patterns that the participants repeated in the social condition. For the participant with prior drawing experience, previous drawing patterns (see 3.4.3.) were also observed (e.g., eyebrows and eyes).
Supplementary Figure 1: Code interface and video. This figure represents video codification, segmentation, the timeline of events, and the coding scheme. Please click here to download this File.
Supplementary Figure 2: Intersection of gaze and drawing tasks. Condition Rest: resting period of 20 s; Condition Draw: the social condition of the experiment (40 s); Condition Dots: the control condition of connecting dots (40 s); Interaction Gaze 1: Subject 1 looking to drawing partner; Interaction Talk 1: Subject 1 speaking; Interaction Body 2: Subject 2 moving hands, shoulders, and head in non-verbal communication. Please click here to download this File.
This study aimed to create a protocol using hyperscanning on four brains concurrently under naturalistic conditions. The experimental paradigm used different drawing tasks and the correlation of multiple outcome measures, drawing metrics, behaviors, and brain signals. The critical steps within this protocol are the consideration of the challenges arising from its high complexity and the maintenance of its ecological and naturalistic conditions.
Video observation was key to this study. It allowed the coding and segmentation of non-verbal communication behaviors on a timeline50. The video's qualitative analysis allowed the observation of gaze behavior in the search for signs that could indicate the functions of encoding (gathering information from the other) and signaling (communicating with others)39,40. It also helped refine the observational information by highlighting individual and group patterns13. Quantitative measures of gaze alone do not provide an understanding of the quality of gaze. The gaze may represent only an automatic behavior of repetition or thinking in the cognitive mode of comparison and structuring18,19. However, the small number of event sequences did not allow this study to perform sequential analysis and statistical validation42. The results showed a significant decrease in synchronous and asynchronous glances in the social conditions from the middle to the end of the experiment. One of the hypotheses for this behavior may be the very process of visual perception and the formation of symbolic representations inherent to the recognition of the other49,51. This hypothesis is in line with the theory regarding face-to-face observation drawing19 and with the results of studies that have associated longer observation times of an object with difficulty in deciding what and how to draw52. Another possible hypothesis is associated with the acquisition of visual memory of the objects drawn throughout the experimental task18,53. In the results, the gaze behavior measures showed a decreasing quantity associated with the drawing partner position. In frontal drawing positions, the amount of asynchronous and synchronous gaze was greater than in the conditions where the partner was to the side. This result is in line with studies suggesting that stimulus location affects visual perception54. The alignment between attentional focus and saccade behavior seems to suggest a certain "automatic" component to attentional focus regarding proximity and/or that the movement constraint acts as a distractor55. Furthermore, recent studies have shown that the interaction perspective can alter gaze patterns. When participants are engaged in a task, social attention on others decreases; this suggests that people in complex social environments look at each other less, and information gathering does not necessarily use direct gaze25. Differences in stroke, drawing, and gaze behavior were also observed between participants who did and did not have prior drawing experience. The participant with previous drawing experience showed far less pattern repetition than other participants and a greater number of drawn details. The number of drawing strokes was also higher, and this trend was accompanied by the number of dots followed in the tracing game and the number of direct gazes to partners. However, the small number of participants did not allow the resulting sequential data analysis and statistical validation to be performed. Drawing practice involves constant gaze shifts between the figure and the paper and the use of visual memory56. Previous studies suggest that drawing practitioners have an easier time encoding visual shapes than non-practitioners57. Still, the results of the study by Miall et al.52 also suggest that people with drawing training modulate their perception to the experience of observation, while prior knowledge (e.g., stable mental images) seems to direct the perception of non-practitioners. These aspects deserve further study, especially with regard to underlying neural networks and differences in attentional processing and gaze behavior25.
Many challenges arise from performing hyperscanning on a four-person group with fNIRS, especially considering a dynamic paradigm; therefore, modifications and technique troubleshooting were performed while refining the protocol methodology. The first challenge was the adequacy of the target brain area with the limitation of the number of optodes and the difficulty of dealing with the signal capture calibration of fNIRS. This protocol envisaged the investigation of two brain areas, the temporo-parietal junction (TPJ) and the medial prefrontal cortex (mPFC). Signal capture of the TPJ was discarded due to the non-controllable difficulty of the density and color of the participants' hair58 versus the number of caps to be administered simultaneously. There was also a major concern for the participants' comfort and availability of time. The second challenge concerns the recording of the experiment. Initially, the protocol foresaw using only one 360° camera located at the center of the table for the experiment. The use of auxiliary cameras proved essential, though. Another difficulty was addressing drawing technique issues to create a robust protocol. Most participants represented body and clothes, areas that were not foreseen in the game's rules, despite the careful explanation, including exposure to previously drawn examples. Some of the participants verbalized that they had difficulty portraying the sizes of the shapes and continued the drawing where the previous partner had stopped due to the proportions. This verbalization is in line with the results of studies suggesting that visual perception reduces the impact of perspective, causing perceptual distortions52. Other drawing studies focusing on the relationship between visual perception and motor commands have also suggested that distortions due to multiple factors intervene in the gaze/hand process20,43. Gowen's study paradigm43, for example, used two expert drawers, one copying and the other drawing from memory. Their results suggested the use of different neural strategies for each drawing technique. Copying seems to depend on comparisons, visual feedback, and closer tracking of the pen tip.
Concerns about the limitations of the technique also involve the ecological condition of the experiment, as well as the use of a collaborative drawing paradigm. One of them concerns participant proximity (due to fNIRS wiring and stability issues) and its possible interference in gaze measures. Different gaze behavior patterns can be produced in social contexts of forced proximity (such as elevator situations)59. Nevertheless, synchronous gazes were observed, facial expressions and smiles emerged throughout the quartet collection, and they possibly indicate a sense of engagement. These results align with results from experiments where "bottom-up" stimuli, such as looking at the other's face, provoked the creation of shared representations among the participants. Non-verbal communication may result from the conceptual alignment that occurs throughout the interaction4. Using drawing techniques "per se" is challenging since most people stop drawing in the realistic phase (after 11 or 12 years of age). The perception of drawing as not expressing reality generates frustration or the discomfort of self-judgment and, consequently, resistance to the act of drawing18. Face drawing could be yet another factor of discomfort. Despite this, the condition of collaborative drawing done in 40 s sessions proved effective for this protocol. When asked about the experience, all participants responded positively, laughed, and commented on the shared portraits. The structured form for the experimental task seems to have lessened the burden of individual production and facilitated interaction between participants, as in Hass-Cohen et al.19.
The correlation of signals or synchronicity of brain signals occurred most strongly in the last part of the experiment for both conditions, social and non-social. The hypothesis was that the social (collaborative design) and non-social (connecting the dots) conditions would result in distinct brain signal synchronicities at different moments in the timeline. In the non-social condition, the synchronicity was expected to result from the cognitive correspondence of the task common to all participants8. Even if they did not interact directly, the synchronicity of signals was expected to occur earlier in the timeline of the experiment8,10,11,12. In the social condition, on the other hand, synchronicity was expected to occur later due to a possible social interaction between different unknown individuals with diverse personal strategies13,14,16,49.
Although many factors may contribute to the preliminary results, a possible interpretation of them relates to the time participants needed to become familiar with each other and the tasks and, finally, create a sense of group. Drawing in itself may generate reactivity and anxiety through self-judgment or the feeling of being evaluated18. Previous studies have associated negative evaluations with variation in group interpersonal brain synchronization (IBS)17. Also, the impact of proximity between participants in this context is not known yet59. Alternatively, the familiarity acquired individually with the tasks and with the group, albeit late, may have generated a cognitive correspondence of engagement, like an "automatic" engagement that occurred beyond the differences between the two proposed tasks.60 The tasks seem to have worked in a block. Therefore, a longer rest period could possibly result in different brain responses by ensuring decreased brain signals picked up by fNIRS between conditions. Another result that needs further attention is the inverse relationship between the amount of synchronous and asynchronous gaze (which decreased throughout the experimental task) versus individual brain activity (which increased throughout the experimental task). A possible interpretation for this result may lie in the high cognitive demand of the experimental task52, but we cannot fail to consider the neurobiological processes involved in human empathic bonding1.
This protocol has an innovative character, first by applying drawing techniques used in art therapy to provoke empathic bonds between participants; second by the dynamic character of the social-ecological situation; and third by the concomitant measurement of four heads using the fNIRS hyper-scanning technique. The drawing social condition promoted eye contact between the participants, allowing the protocol to explore how gaze behavior supports interpersonal interaction behavior in naturalistic situations and the different personal strategies used to recognize others49. It is also a promising tool to study whether gaze behavior is, indeed, associated with the process of attention18 and engagement between partners61 under the same conditions. Relating all these issues, especially in a paradigm conducted in a group under naturalistic conditions, is a challenge for social neuroscience.
Many factors may have contributed to these preliminary results. All of these different variables deserve further study, and the use of this protocol can provide important clues for better understanding group social relationships in a naturalistic context.
The authors have nothing to disclose.
The authors thank Instituto do Cérebro (InCe-IIEP) and Hospital Israelita Albert Einstein (HIAE) for this study support. Special thanks to José Belém de Oliveira Neto for the English proofreading of this article.
2 NIRSport | NIRx Medizintechnik GmbH, Germany | Nirsport 88 | The equipment belong to InCe ( Instituto do Cérebro – Hospital Israelita Albert Einstein). two continuous-wave systems (NIRSport8x8, NIRx Medical Technologies, Glen Head, NY, USA) with eight LED illumination sources emitting two wavelengths of near-infrared light (760 and 850 nm) and eight optical detectors each. 7.91 Hz. Data were acquired with the NIRStar software version 15.2 (NIRx Medical Technologies, Glen Head, New York) at a sampling rate of 3.472222. |
4 fNIRS caps | NIRx Medizintechnik GmbH, Germany | The blackcaps used in the recordings had a configuration based on the international 10-20 | |
Câmera 360° – Kodak Pix Pro SP360 | Kodak | Kodak PixPro: https://kodakpixpro.com/cameras/360-vr/sp360 | |
Cameras de suporte – Iphone 8 | Apple | Iphone 8 | Supporting Camera |
fOLD toolbox (fNIRS Optodes’ Location Decider) | Zimeo Morais, G.A., Balardin, J.B. & Sato, J.R. fNIRS Optodes’ Location Decider (fOLD): a toolbox for probe arrangement guided by brain regions-of-interest. Scientific Reports. 8, 3341 (2018). https://doi.org/10.1038/s41598-018-21716-z | Version 2.2 (https://github.com/nirx/fOLD-public) | Optodes placement was guided by the fOLD toolbox (fNIRS Optodes’ Location Decider, which allows placement of sources and detectors in the international 10–10 system to maximally cover anatomical regions of interest according to several parcellation atlases. The ICBM 152 head model parcellation was used to generate the montage, which was designed to provide coverage of the most anterior portion of the bilateral prefrontal cortex |
Notebook Microsoft Surface | Microsoft | Notebook receiver of the fNIRS signals | |
R platform for statistical computing | https://www.r-project.org | R version 4.2.0 | R is a free software environment for statistical computing and graphics. It compiles and runs on a wide variety of UNIX platforms, Windows and MacOS |
REDCap | REDCap is supported in part by the National Institutes of Health (NIH/NCATS UL1 TR000445) | REDCap is a secure web application for building and managing online surveys and databases. | |
software Mangold Interact | Mangold International GmbH, Ed. | interact 5.0 | Mangold: https://www.mangold-international.com/en/products/software/behavior-research-with-mangold-interact.html. Allows analysis of videos for behavioral outcomes and of autonomic monitoring for emotionally driven physiological changes (may require additional software, such as DataView). Allow the use of different camera types simultaneously and hundreds of variations of coding methods. |
software NIRSite | NIRx Medizintechnik GmbH, Germany | NIRSite 2.0 | For creating the montage and help optode placement and location in the blackcaps. |
software nirsLAB-2014 | NIRx Medizintechnik GmbH, Germany | nirsLAB 2014 | fNIRS Data Processing |
software NIRStar | NIRx Medizintechnik GmbH, Germany | version 15.2 | for fNIRS data aquisition: NIRStar software version 15.2 at a sampling rate of 3.472222 |
software NIRStim | NIRx Medizintechnik GmbH, Germany | For creation and organization of paradigm blocks |