Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

In the following, results obtained with the proposed method are reported. Three experiments measured the influence of different attentional manipulations with three highly different types of stimulus material. The stimuli are simple line segments in pop-out patterns, action space objects in natural images, and cued letter targets.

Experiment 1: Salience in pop-out displays
Experiment 1 aimed at measuring the influence of visual salience on the processing speed of line segments in a synthetic pattern. Subjects judged which of two target line segments (left or right) in a background pattern of oriented line segments flickered first. In half of the trials, the probe was a color pop-out (see Figure 2a). More background on TOJ-based assessment of pop-out can be found in a study by Krüger and colleagues⁸, where local orientation salience was manipulated instead of color. The distribution and frequency of the SOAs are shown in Figure 2b.           

A Bayesian power analysis was conducted as described in the protocol step 2. For typical group overall rates C (M = 70 Hz, SD = 20) and a hypothetical advantage from about 7 Hz for the salient target in the attention condition (resulting from an attentional weight of M = 0.55, SD = 0.02), 200 simulations were performed. The success rate for detecting the advantage was calculated for the lower boundary of the 95 % HDI (highest density interval) being above 4 Hz and fulfilling additional requirements regarding the difference to the control condition (see TVATOJ example “power-exp1.R” for all details). The success rate for reaching this goal under the hypothetical conditions with 25 participants turned out to be 0.88 with a 95 % HDI ranging from 0.82 to 0.92.

For the actual experiment, 30 participants were recruited. One participant was excluded from the analysis because he did not follow the instructions but always pressed the same key.

Figure 2: Experiment 1. (a) Targets (marked with circles for illustration) in the neutral (upper part) and attention (lower part) condition. (b) SOA distributions. (c) Three exemplary subject-level response counts (points) and posterior predictive curves (shaded area; intensity represents likelihood with regard to 100 simulated repetitions at fine-grained SOAs). Blue indicates control and green attention condition. (d) Group-level posterior predictive curves. (e) Posterior distributions of the overall rate C and attentional weights w_p and w_r. (f) Posterior distributions of v_p and v_p and their differences. Please click here to view a larger version of this figure.

The data was fitted with the TOJ model derived from TVA (as described in the Introduction) using a hierarchical Bayesian procedure implemented in JAGS¹⁰. Figure 2c shows three exemplary subject-level plots of the raw data and posterior predictive estimates obtained from subject-level samples of the fitted model. Group-level posterior predictive curves are shown in Figure 2d and parameter estimates in Figure 2e and 2f. Depending on the research question, parameters C and the relative attentional weights w_p and w_r (Figure 2e) or the individual rate parameters v_p  and v_r  (Figure 2f) can be assessed. If the overall processing capacity C was changed by the manipulation, the latter parameters can show whether and how the individual stimulus processing rates have changed.

The proposed hierarchical Bayesian estimation procedure offers a wealth of results. For example, all parameters can be assessed for each participant on the subject level. Typically, there is interest in tendencies in the population. Hence, the results on the group level are discussed. The histograms show distributions over the parameter space. The modes of the distributions are stated to indicate the parameters' central tendencies. The 95 % HDIs mark the ranges in which the true values lie with a probability of 95% according to model and data (for further details on how to interpret the Bayesian statistics, refer to Kruschke^11,13).

Figure 2e shows estimates of the means across subjects for the weight and overall rate parameters. An attentional benefit for the salient stimulus can be seen in the attention condition. The central tendency of parameter w_p is 0.59, and its 95 % HDI ranges from 0.55 to 0.63. Hence, salience shifted the attentional weight away from the neutral value of 0.5. In the control condition, where none of the targets was salient, a neutral weight of w_p= 0.5 was obtained (95 % HDI: 0.48 to 0.51). The corresponding row “Comparison” shows that the difference between the w_p  weights across conditions is 0.09, and the 95 % HDI of this difference ranges from -0.11 to 0.07. Hence, there is a reliable difference between the two weights in favor of the salient stimulus.

However, does this mean that the salient target was processed faster? The difference in weights together with the shared overall rate C in the attention condition indicate that it was processed faster than the non-salient target in this condition. However, an important question is whether it was also processed faster than the targets of the control condition. Taking the estimates of the processing rates into account, the answer must be no. The C estimate shown in Figure 2e is lower in the attention condition by a difference of almost 17 Hz. In the corresponding “Comparison” plot, 0, no difference, is just at the fringe of the 95 % HDI; hence it is highly unlikely. Considering the individual rates of the probe (v_p) and reference (v_r) stimulus in both conditions (Figure 2f), it is clear that the advantage of the salient stimulus results from a 16 Hz reduction of the processing rate of the non-salient stimulus in the attention condition. A possible interpretation of these results is that the salient target leads to a suppression of the non-salient target in the attention condition and thereby benefits in relation.

Note that in this experiment, even though the appearances of probe and reference were identical in the neutral condition, the delay between trial start and probe event was constant. Therefore, participants could have directed attention toward this point in time, thus shifting the attentional weight away from the neutral value 0.5. Consequently, the actual attentional weight of the probe stimulus in the control condition must be estimated and fixed at 0.5. Fixing the parameter is possible when the participant cannot tell even in principle which is the probe and which the reference stimulus, as in the control condition in Experiment 3.

Experiment 2: Action space advantages in natural images
The second experiment measured attentional advantages for objects in the action space in natural images. From change blindness studies it is known that center-of-interest objects benefit from meaning-driven orienting in natural images¹⁴. This effect is absent when images are obscured by upside-down presentation. In unpublished change blindness experiments, we found an action space advantage with a set of images with changes in action-space and background objects (there is a published replication with similar stimulus material in reference¹⁵). We hypothesize that these action space objects, which are close to the observer and possibly graspable, exhibit a similar advantage in their processing rates.

Thus, the proposed TOJ-based method is tested with natural images. Action space (probe) and more distant (reference) objects, which appeared abruptly in natural images, constituted the targets for the TOJ procedure (see Figure 3a). In a between-subjects control condition, upside-down versions of the same images were used. These are known to have reduced context effects in change detection experiments^15,16. The SOA between the onsets was varied according to the distribution shown in Figure 3a.

A power estimation was performed exactly as for Experiment 1, except that between-subject comparisons between experimental and control condition were conducted. The success rate for achieving the goal was estimated with 0.92 (95 % HDI: 0.88 to 0.96) with 35 simulated participants per condition (details can be found in the TVATOJ example “power-exp2.R”).

There were 39 subjects in the attention condition and 38 in the control condition of the actual experiment. (Some subjects participated in both conditions. To our knowledge, this does not compromise the Bayesian statistical analysis. Treating the mixed data as between-subjects reduces the power compared to considering the within-subject differences.) Again, one participant (the same person in both conditions) was removed from the analysis of each condition, due to having given intentionally random responses throughout the experiment.

Figure 3: Experiment 2. (a) Action space (marked with white arrows) and background (black arrow) targets in the neutral (left) and attention (right) condition. (b) SOA distribution. (c) Two exemplary subject-level plots from the neutral (blue) condition and two plots from the attention condition (green) with response counts (points) and posterior predictive curves shaded area; intensity represents likelihood with regard to 100 simulated repetitions at fine-grained SOAs). (d) Posterior distributions of the overall rate C and attentional weights w_p and w_r. (e) Group-level posterior predictive curves. (f) Posterior distributions of  v_p and v_r and their differences. Please click here to view a larger version of this figure.

This data was fitted in the same manner as in the first experiment. The only difference was that because of the between-subjects design, parameter differences between the two conditions could not be calculated during the sampling at the subject level. This reduces the power in comparison to within-subject differences.

The exemplary subject-level and the group-level posterior predictive distributions in Figure 3c show distributions that are barely shifted against each other. The two conditions overlap almost perfectly in the group-level posterior predictive plot (Figure 3e), so that it may seem as if the attention manipulation had not worked at all. Inspecting the posterior distributions of the parameters, however, reveals that there indeed is an advantage for the action space objects. The w_p estimate in the attention condition is shifted away from the neutral state of 0.5, which is only at the very left end of the 95 % HDI. Curiously, however, it is also shifted in the control condition, with the 95 % HDI even excluding 0.5, suggesting that the inversion of the images did not remove the potential action space advantage.

Considering the comparison of the individual stimulus processing rates (Figure 3f, “Comparison”), an effect of attention can be seen for the rate of the reference stimulus v_r. However, the difference points in the direction opposite to the hypothesis and is small, reflecting a rate change of only 2 Hz (95 % HDI: -3.36 to 0.66).

Therefore, it must be concluded that either (a) the attentional advantage of action space objects is due to a factor that is not affected by the scene inversion, such as salience or visibility. Alternatively, (b) the scene inversion does not reduce the action space effects as intended, or (c) the power of the present experiment was too small to detect the effect. Explanations (a) and (b), or a combination, are the likely ones. In our unpublished change-blindness experiments mentioned earlier, which were conducted with the same images, there was still an advantage (though reduced) for the action space objects in inverted scenes.

In the context of this method-centered paper, however, alternative (c) may be the most interesting one. Therefore, the magnitudes of possibly overlooked effects will be briefly discussed. Looking at the comparison of the attentional weights, the lower bound of the 95% HDI, which reflects the hypothesized direction, is at -0.01. Hence, only weights larger by 0.01 in the attended compared to the control condition are probable. This difference is small compared to the other experiments, and the odds are against even such a small effect. This is reflected by the upper HDI bound reaching 0.04. Looking at the processing rates is helpful because the rates in Hz can be readily interpreted as processing speed.

The differences between the two conditions are shown in the “Comparison” row in Figure 3f. Difference Δv_r between the reference stimuli is negative, -2.03 Hz, and the 95 % HDI excludes 0. The negative difference reflects an increase in the processing rate of the reference targets, the background objects, which too is against the action space advantage hypothesis. A small attentional advantage is still possible in the processing rates of the probe targets, Their difference Δv_p is estimated close to zero, but the 95 % HDI  ranges from -1.64 Hz to 1.51 Hz. Even though a value close to zero is most probable, rate effects up to 1.64 Hz in favor of the hypothesis, and up to 1.51 Hz against it, remain possible concerning the 95% HDI. Overall, these results are not favorable for the original hypothesis, but their discussion showed how meaningful sizes of possibly missed effects can be conveniently extracted from the results. Note that for accepting null results, such as the lacking reduction of the action space advantage by rotating the images, regions of practical equivalence can be defined and their overlap with the 95 % HDI can be tested¹¹ (see Discussion Section).

Experiment 3: Spatial cueing in letter recognition
The third experiment investigated the limits of the proposed TVA-based TOJ model and shows how the model can be extended to deal with these difficulties.

What is it that gets the proposed method into trouble? In the two previous experiments, the participants had to judge two temporal events. Now we add a third temporal event, a peripheral cue that is shown 110 msec before the probe stimulus to direct attention toward it. This third event presents difficulties to simple the TVA-based TOJ model, for which only two stimuli are explicitly modeled.

Figure 4: Typical effects of cues on perceptual latencies. Magnitudes of attention effects typically found in TOJs with peripheral cues (horizontal lines). Magnitudes predicted by the TVA-based TOJ model for increasing attentional weights of the probe stimulus (curves). The solid curve corresponds to typically observed parameters. Please click here to view a larger version of this figure.

Magnitudes of attentional benefits reported in the TOJ literature already hint at these difficulties. The latency differences induced by peripheral cues are often as high as 50% and sometimes even as high as 80% of the cueing interval¹⁹. As illustrated in Figure 4, such a large shift would require unlikely attentional weights close to w_p = 0.9  at typical processing rates. Furthermore, such extreme weights lead to highly skewed psychometric distributions. These would have a steep slope at one end and a shallow slope at the other end. In a weaker manifestation, this can be seen in the posterior predictive plots of the first experiment (Figure 2c and d). Such strongly distorted curves are rarely reported. When the data from the present peripheral cueing experiment is fitted with the TVA-based TOJ model, the posterior predicted curves strongly deviate from the actual data pattern.

Importantly, however, peripheral cues produce the strongest and most reliable effects of attention in psychophysical TOJ²⁰. Therefore, it is worthwhile to apply a model-based assessment with an extended version of the proposed model. Alcalá-Quintana and García-Pérez²¹ proposed a TOJ model based on general assumptions of exponential stimulus encoding. This model contains an additional parameter that allows for large shifts without altering the slopes of the psychometric curves. Alcalá-Quintana and García-Pérez used it for crossmodal TOJs, where such shifts originate from delays between modalities. Hence, to model data from cued TOJs, we include their parameter τ. A delay between the start of the encoding processes could account for the expected large lateral shifts. The parameter τ may even have a TVA-compatible interpretation. However, this is not entirely unproblematic and will be discussed later. To keep the model parsimonious, other parameters suggested by Alcalá-Quintana and García-Pérez (response biases, lapses, and a minimum possible temporal resolution) were not included.

Formally, the original psychometric model in Equation 4 is modified by replacing the term SOA by an adjusted term SOA_adj = SOA + τ. This adjustment also reflects the interpretation of τ: the starts of exponential encoding processes are now not only separated by the SOA, but an additional constant delay is added. In the hierarchical Bayesian model, subject level τ is sampled from a group-level normal distribution.

An explicit power analysis was not performed for this experiment. Because the within-subjects design is similar to the one from Expriment 1, a similar power is expected for effects in the rates and attentional weights. The expected large lateral shift to be captured by the τ parameter is much larger and more stable than the rate and weight effects typically are, so that no power problem can be expected for detecting it either.

Data was collected for 32 participants (among them the three authors) according to the experimental procedure described in the protocol step 4. Participants had to report the order of two letters. In half of the trials the probe stimulus was preceded (110 msec) by a peripheral four-dot cue (see Figure 5a). A detailed description of the stimulus material can be found in Tünnermann, Petersen, and Scharlau's study⁷. The SOAs and their frequencies are shown in Figure 5b. Each participant performed one or two sessions.

Figure 5: Experiment 3. (a) Targets, arbitrarily designated as probe and reference in the the neutral condition (upper part). In the attention condition (lower part) the probe stimulus was preceded (110 msec) by a four-dot cue. (c) Three exemplary subject-level response counts (points) and posterior predictive distributions (shaded area; intensity represents likelihood with regard to 100 simulated repetitions at fine-grained SOAs). Blue represents the neutral and green the attention condition. (d) Group-level posterior predictive curves. (e) Posterior distributions of the overall rate C and attentional weights w_p and w_r. (f) Posterior distributions of τ and its difference from zero. (g) Posterior distributions of v_p and v_r and their differences. Please click here to view a larger version of this figure.

The extended model as described above was applied in the hierarchical Bayesian estimation procedure. (For participants who produced very steep psychometric curves in the first session, smaller SOAs were used in the second session. This can be seen, for example, in the leftmost plot Figure 5c, which contains additional data points at small SOAs.) Because of the more complex model, the powerful NUTS sampler from the Stan software package was employed in this analysis²².

In the other experiments, the probe stimulus could have a different processing rate than the reference stimulus, even in the control condition. In Experiment 1, this was because the participants could have allocated attention to its predictable point in time relative to the trial onset. In Experiment 2, the inversion of images was not expected to entirely remove the advantage of action space objects. In this letter-based cueing experiment, however, participants would not even in principle be able to identify which is the probe and which the reference stimulus, because the same random letters were used and the time between trial and target onset did not allow to conclude the target type. Therefore, a truly neutral control condition is expected and w_p fixed at 0.5 and τ at 0 in the neutral condition.

As can be seen in Figure 5c and 5d, the cue leads to a substantial shift of the psychometric function compared to the other experiments. Furthermore, the posterior plots in Figure 5f show that τ is estimated as a 53.27 ms benefit for the cued target. The 95% HDI on the difference (“Comparison” row) excludes all differences smaller than 47.56 (or larger than 57.73), rendering them highly improbable.

Interestingly, there is a change in attentional weights in favor of the uncued target (Figure 5e). The posterior distribution of w_p has its mode at 0.42. The neutral weigth of 0.5 is not included in the 95 % HDI. For the C parameter, there is an increase by 4.69 Hz for the attention condition. Expressed in v-parameters (Figure 5f), it is most notable that the rate v_r of the reference stimulus in the attention condition increases.

In the previous two experiments, it was observed that the attention manipulations increased the attentional weight of the probe stimulus. In the present experiment, however, the pattern could reflect an interference of the cue with the target, thereby reducing its rate in the race for encoding. At the same time, the cued target benefits from the faster processing due to the τ parameter. The latter can be linked to the reduction of the cued target's delays before or after the exponential races. Note, however, that concerning τ, an prolongation of a delay associated with the uncued stimulus explains the relative difference equally well.