Perceptual and Category Processing of the Uncanny Valley Hypothesis’ Dimension of Human Likeness: Some Methodological Issues

The core prediction of the uncanny valley hypothesis is that positively or negatively valenced experience can be evoked as a function of perceived human likeness ⁷⁷ (for an informative overview, see ⁷⁸). Careful examination of how human likeness is actually perceived is in itself therefore an important research undertaking. Similarly important is how the DHL is represented in experiments of uncanny experience. This protocol focuses therefore on the DHL. One approach is to represent human likeness using morph continua, as already implemented in “uncanny” research ^{5, 6, 26, 27, 28}. The advantage of morph continua is that their use permits experimentally controlled differences in humanlike appearance to be brought into relationship with behavioural measures of subjective perception and experience (e.g. category decisions, uncanny feelings) and with underlying neural processes ⁷. This fine-grained approach is particularly important because the uncanny valley hypothesis does not predict the actual degree of human likeness at which the transition between positively valenced and uncanny experience should occur ⁷⁸. If Mori’s conjectures are correct, the findings relating to category processing along the DHL ⁷ would suggest that uncanny experience is most likely to occur at the category boundary where perceptual decision ambiguity is greatest. This has still to be tested.

To be able to interpret the investigated relationship between the DHL, as represented using morph continua, and other variables of interest, a single morph continuum rather than two or even three different juxtaposed continua should be used ^5,28. The juxtaposed continua fail to represent and, in effect, alter Mori’s concept of human likeness by introducing discontinuities to the DHL. This could affect performance in a perceptual discrimination task, because the point of the discontinuity and that of any disparities resulting from the morphing procedure might be used as a reliable but experimentally unintended point of reference for guiding perceptual discrimination (see, ²⁹). Within each morph continuum all morphs should be carefully controlled so that equivalent increments of physical change are represented along the entire continuum ^5,28. This is especially important in this protocol, because experimental control of morph distance along the continua enables examination of whether the sensory information relating to linear differences in physical human-like similarity along the DHL is cognitively represented in a linear or nonlinear way. Nonlinearity is reflected in the step-like function in the slope of the categorization responses (Figures 2A and 5A) and in differences in perceptual sensitivity to stimulus attributes along the DHL (see Figures 4 and 5B). This protocol uses faces as endpoints without applying any further experimental manipulations. Further studies of CP and human likeness could examine for example how specific features such as eye realism compared with the realism of other facial features or manipulations of facial geometry compared with facial texture (cf. ^30,38) differentially influence category processing along the DHL.

The morphing procedure enables smooth blending together of corresponding features of the continuum’s endpoints such as facial configural cues. Difficulty in morphing facial information like upper facial features and hair profile ²⁶ can potentially bias participant’s responses by drawing attention to disparities in the alignment of features during the morphing procedure. This bias is likely to be systematic in that morphing disparities are related to the morph distance from the continua endpoints, the disparities being greatest at the midpoint of morph continua. For our morph continua, the midpoint of the continua corresponds with the category boundary around which there is greatest perceptual sensitivity. Reanalysis of data from one of our pilot studies (a forced choice categorisation task) compared continua in which the eye region was either well or poorly morphed (poor morphing resulted in a very slight inconsistency in the alignment of eye texture between morphs). The reanalysis confirmed a systematic bias in the categorisation decision responses of the poorly morphed continua such that poor morphing effectively caused a relative shift of the category boundary toward the human end of the dimension. This was presumably because the morphing disparity was perceived as a “nonhuman-defining” feature.

A response bias might result also from using continua generated on the basis of endpoint stimuli in which non-facial information such as head attire and facial jewelery are only present in one endpoint stimulus ²⁷. In this case, facial images could be cropped so that participants attend to the stimulus information of research interest rather than to other salient features presented in an image. A systematic response bias can result also from using an image as a continuum endpoint in which nonhuman attributes are presented together with human attributes, even though this image is intended to represent the human end of the DHL ⁶. In this case, any relationship between human likeness and variables such as subjective measures of uncanny experience are not interpretable in terms of Mori’s conception of the DHL and of the hypothesized uncanny valley.

CP can occur along dimensions other than human likeness ^{31, 10, 22, 32, 33, 34, 35}, and category-relevant information can be automatically processed upon exposure to others ³⁶. In this protocol, care should be taken therefore to control for the effects of visual cues indicating differences along the DHL in terms of other category-relevant dimensions on participants’ responses concerning human likeness. These cues might for example relate to ethnicity, gender, facial distinctiveness, familiarity and identity, and facial expression (cf. ^{5, 26, 27, 28}). The present protocol seeks to minimize perception of biological motion between face morphs presented in rapid succession in the perceptual discrimination task and fMRI study by closely matching the facial geometry and configuration of facial features of images used as continuum endpoints. This approach (together with the relative position along the continua of morphs used in the stimulus conditions) helps also to minimize any perception of different identities between morphs of a continuum.

The forced choice classification task determines which morphs of a continuum are clearly categorized as an avatar and as human in order to select morphs for use in the perceptual discrimination task and the fMRI study. We selected the four morphs M0, M4, M8 and M12 from each of the continua (Figures 2B and 2C). In addition to controlling for the degree of physical change along the DHL, the choice of M4 and M8 is based on the following theoretical consideration. Mori described perceptual uncertainty (and associated uncanny experience) as occurring at levels of realism that correspond to the region along the DHL between the two positive peaks in the slope of the valence-human likeness relationship (see Figure 1). At these peaks, objects are regarded as either nonhuman or human. In reframing his considerations in terms of the framework of category processing, these peaks may be seen as reflecting degrees of human likeness at which correctly classified category instances (i.e. nonhuman and human) straddle the category boundary. But Mori did not specify how efficient this classification (i.e. perceptual certainty) must be at these peaks, though the identification of objects at each peak is clearly considered to be relatively efficient and effortless. For this reason, the two morph positions along the continua considered as defining the transition between the two categories and as reflecting the two positive peaks was determined using a more conservative criterion than often otherwise used in CP research (e.g. 66%, as in ^{32, 34}). Thus, morph M4 was identified on average as an avatar in more than 85% of trials and morph M8 as a human in more than 85% of trials. Please note that this criterion applies to both morphs M4 and M8 of any one continuum. Using this approach, this choice of morphs seeks to capture a sense of category change along the DHL between nonhuman and human objects in accordance with both an understanding of CP and Mori’s description of the hypothesis.

This protocol uses a variant of the same-different perceptual discrimination task ¹⁰ to examine CP. The advantage of this task is that participants do not need a description as to what specific similarities and differences must be identified. It is sufficient that they simply identify stimuli as being the same or different. In addition, participants do not need to know the category labels. Labels might be used as a strategy to discriminate between stimuli when the memory load required by a discrimination task such as the ABX task increases ⁴². The same-different task has the advantage that the memory load is comparatively low and that the task encourages direct comparison of stimuli. To reduce the potential influence of labelling, discrimination tasks are normally presented before the forced choice decision task ⁴⁰. The present protocol is based on two different participant groups for the discrimination and forced choice decision tasks ^{7, 41}. This is because the forced choice task is used to select stimuli for the discrimination task. Should however the same participants be tested in both tasks, the protocol should be modified so that the discrimination task is conducted before the forced choice decision task.

A fixed discrimination design is applied in the same-different discrimination task of this protocol (for roving designs, see e.g. ³⁹). This means that M4 and M8 are always shown as the first stimulus of each stimulus pair in the “same”, “within” and “between” conditions of the avatar and human trials, respectively. This protocol includes the experimental constraint that each participant views only the morph stimuli of either avatar or human trials from a given continuum but not both. Using the avatar trials as an example, this means that the first stimulus of each stimulus pair is always M4, that the second stimuli in the “within” (i.e. M1) and “between” (i.e. M8) conditions are presented equally often for a given continuum, and that no further stimuli are drawn for human trials from that particular continuum. This approach aims to avoid selectively inducing stronger representation of and facilitating therefore discrimination of the cross-category faces of a given continuum. To exclude or, for purposes of comparison, to investigate any possible effect on cross-category representation and discrimination of presenting the described avatar and human trials in one experimental block, a design could be implemented in which the described avatar and human trials are presented in separate blocks (with blocks counterbalanced in order across participants).

The present same-different discrimination task has a ratio of same-to-different trials of 1:2. This ratio might induce a response bias in favour of “different” decisions (though other factors can also influence this bias ^{44, 51}). Measures derived from Signal Detection Theory (SDT) are often used to disentangle response bias (β or c) for selecting one response over another from the participant’s sensitivity (A’ or d’) in discriminating sensory stimuli (for an overview see, ⁴⁴). As d’ can vary with response bias due to violation of SDT assumptions ⁵², we used the nonparametric measure of sensitivity A’ ⁵³. For response bias we used β”_D ⁴⁷. Alternatively c has been recommended by ^{43, 44}, partly because it is independent of change in d’ ⁵⁴. Overall, the present results indicate greater perceptual sensitivity for morph stimuli straddling the category boundary than for within-category stimuli.

The selection of morphs for the discrimination task in this protocol means that the task requires discrimination between morphs that are four steps apart along the continua (i.e. a four-step discrimination, see Figure 2B). But this four-step degree of dissimilarity between morphs is too large to allow better specification of the actual morph position at which discrimination is most enhanced (i.e. the discrimination boundary) (Figure 5B). An important criterion for CP (for the other criteria, see e.g. ¹¹) is that there is alignment between the category boundary in the forced choice task and the discrimination boundary in the discrimination task. In other words, the morph position of the category boundary should predict the morph position of the discrimination boundary. One approach to verifying the specific point of alignment would be to use a discrimination task in which the morph distance between pairs of morphs is reduced. For the purpose of illustration, Figure 5B shows results of pilot data using, as a possible alternative to the same-different discrimination task, a traditional ABX discrimination task ^{12, 13}. The figure indicates clearly that there is a peak in perceptual discrimination sensitivity at the morph position predicted by the category boundary. Such results in a study with a larger number of participants and application of SDT in analyses would further verify the finding of effects of CP along the DHL. The actual choice of stimuli for the continua endpoints, the number of morphs generated in a continuum, and the size of the step in morphs to be discriminated will strongly influence the cognitive demands placed on the participant and his or her ability to discriminate morphs along the continua.

One classical criterion of CP is that the position of the category boundary predicts the position of the peak in actual discrimination performance (i.e. the discrimination boundary) ⁸⁰. This is arguably the most important criterion of CP ⁸¹. Conclusive testing of this prediction requires an experimental design in which all morph pairs that together represent the entire length of the morph continuum are presented in the discrimination task in order to determine the actual position of the peak. In ³⁸, discrimination performance was examined on the basis of only certain segments of the morph continua. This could mean that the true position of the actual peak in performance may have been missed, this in turn rendering it difficult to conclusively verify CP. It should be noted that even the early CP study of Lieberman et al. ⁸² failed to meet the studies own stringent criterion that predicted and actual peak in discrimination performance converge, and that other researchers have not applied this criterion stringently (e.g.¹¹, see also ⁸⁰). Determining the actual position of peak performance is nevertheless critical, even if a more liberal interpretation of this criterion is applied. Examining the entire length of the morph continuum also has the advantage of enabling inspection of the data as to whether there is a peak in performance at a point contrary to expectation due for example to an artefact resulting from the morphing procedure.

In addition to the responses, the response time (RT) data in the forced choice classification task is useful as an indicator of difficulty in cognitive processing of stimulus information and of the competing response tendencies to categorize a stimulus as “avatar” or “human” ^{70, 71}. RT should thus be longest for categorization judgments of stimuli positioned at or nearest to the category boundary. Figure 4 shows that this is the case. Taken together, the shape of the response function and the RT data for category judgments show that assignment of a stimulus to a discrete category is subject to large differences in processing difficulty. To assess RT, this protocol instructs participants to respond during categorization as quickly and accurately as possible. Given the potential impact of a speed-accuracy trade-off on responses ^{72, 73}, we examined and found in pilot testing that the shape and position of the avatar-human category response function is very robust, being unaffected by instructions to identify the presented morph stimulus either as quickly and accurately as possible or simply as accurately as possible. This would suggest that participants generally use a decision strategy weighted for accuracy, though this suggestion could be tested more thoroughly. In keeping with Mori’s hypothesis that difficulty in distinguishing a humanlike object from the human image might evoke negatively valenced experience, it would be interesting to establish whether longer RT for humanlike stimuli is associated with measures of negative affect. RT data was also collected and analyzed for the same-different discrimination task. RT has been used to support response data ⁸⁰. In contrast to the ABX task, the same-different task provides a clear time point for RT measurement. The RT of correct responses should be shorter for between- than for within-pairs ⁷⁴, though the interpretation of RT data can be complicated for same-different judgments because RT can be influenced by a number of factors in this task ^{75, 76}. The RT data are however consistent with the idea that less difficult cross-category decisions are made more quickly than within-category decisions (see Figure 7).

It should be pointed out that Mori’s hypothesis does not consider the possibility that physical features might actually vary along the DHL within the human category (Figure 2) ⁷. This is the reason why the second positive peak in the hypothesis’ original valence-human likeness relationship is located at the human end of the DHL (Figure 1). The emphasis on the nonhuman aspect of the DHL has been influential in studies guided by the hypothesis, including studies that have not used morph continua ^{4, 37}, while other studies have used a single human face to represent the human aspect of the DHL ³. Such studies have sought to examine uncanny experience, with unclear results. The findings relating to CP suggest that these studies might not have presented the stimuli needed to evoke implicit or explicit processes of perceptual decision making and processes of conflict resolution in response to category ambiguity along the DHL.

This protocol illustrates an example of how morphs drawn from continua representing the DHL can be used to identify, with fMRI and using the effect of repetition suppression, brain regions sensitive to change in physical humanlike similarity and to change in category-related information. The effectiveness of the fMRI design is influenced strongly by careful generation and selection of the morph stimuli. The forced choice and perceptual discrimination tasks were thus used to ensure comparability between continua in the shape of the avatar-human classification curves (i.e. slope of the response function) and in discrimination performance. The advantage of this fMRI design is that it allows the stimulus conditions described by Mori (i.e. passive observation of novel non-human objects that are subtly different in physical appearance from that of their human counterpart) to be simulated within the constraints of fMRI methodology, using stimuli selected according to the hypothesis’ definition of human likeness, and investigation of effects of category processing while controlling for effects of physical change along the DHL. The fMRI paradigm is not designed to examine uncanny experience, but it could be adapted to investigate affective experience associated for example with the category boundary itself. This would be an important step toward examining in the brain the effects of category processing and category ambiguity in association with affective experience for stimuli drawn from the DHL.