Here, we describe virtual-hand and virtual-face illusion paradigms that can be used to study body-related self-perception/-representation. They have already been used in various studies to demonstrate that, under specific conditions, a virtual hand or face can be incorporated into one's body representation, suggesting that body representations are rather flexible.
Studies investigating how people represent themselves and their own body often use variants of "ownership illusions", such as the traditional rubber-hand illusion or the more recently discovered enfacement illusion. However, these examples require rather artificial experimental setups, in which the artificial effector needs to be stroked in synchrony with the participants' real hand or face—a situation in which participants have no control over the stroking or the movements of their real or artificial effector. Here, we describe a technique to establish ownership illusions in a setup that is more realistic, more intuitive, and of presumably higher ecological validity. It allows creating the virtual-hand illusion by having participants control the movements of a virtual hand presented on a screen or in virtual space in front of them. If the virtual hand moves in synchrony with the participants' own real hand, they tend to perceive the virtual hand as part of their own body. The technique also creates the virtual-face illusion by having participants control the movements of a virtual face in front of them, again with the effect that they tend to perceive the face as their own if it moves in synchrony with their real face. Studying the circumstances that illusions of this sort can be created, increased, or reduced provides important information about how people create and maintain representations of themselves.
According to Western philosophy, the human self consists of two aspects1: For one, we perceive our own body and our activities in the here and now, which creates a phenomenal self-representation (often called the minimal self). For another, we create more enduring representations of ourselves by storing information about our personal history, integrating new information into the emerging self-concept, and present ourselves to our social environment accordingly, which amounts to the creation of a so-called narrative self. The minimal or phenomenal self has been argued to emerge from two sources of information. One is top-down information about longer-lasting aspects of our body, such as information about the effectors we own or the shape of our face. The other is bottom-up information provided by self-perception in the current situation.
Investigations of the latter were strongly inspired by a clever study of Botvinick and Cohen2. These authors presented human participants with a rubber hand lying in front of them, close to one of their real hands, which however was hidden from view. When the real hand and the rubber hand were stroked in synchrony, so to create intermodal synchronous input, participants tended to perceive the rubber hand as part of their own body-the rubber-hand illusion. Further studies revealed that perceived ownership even went so far that participants would start sweating and trying to withdraw their real hand when the rubber hand was being attacked by a knife or otherwise being "hurt"3.
While Botvinick and Cohen have interpreted their findings to demonstrate that self-perception arises from the processing of bottom-up information, other authors have argued that the rubber-hand illusion results from the interaction between intermodal synchrony of input, a bottom-up source of information, and stored representations of one's own hands, a top-down source of information4. The idea is that the stimulus synchrony creates the impression that the real and the rubber hand are one and the same thing, and given that the rubber hand looks like a real hand, this impression is considered reality.
Later research by Kalckert and Ehrsson5 added a visuo-motor component to the rubber hand paradigm, which allows for the investigation of both perceived ownership (the impression that the artificial effector belongs to one's own body) and perceived agency (the impression that one is producing observed movements oneself). Participants were able to move the index finger of the rubber hand up and down by moving their own index finger, and the synchrony between real and rubber hand finger movements, the mode of movement (passive vs. active mode), and the positioning of the rubber hand (incongruous vs. congruous with regard to the participant's hand) were manipulated. The findings were taken to provide support for the notion that agency and ownership are functionally distinct cognitive phenomena: while synchrony of movement abolished both sense of ownership and agency, mode of movement only affected agency, and congruency of the rubber hand position had an effect on ownership only. The latter two result were replicated in a follow-up study in which the distance between real and rubber hand in the vertical plane varied6: ownership for the rubber hand decreased as its position increasingly mismatched the participant's real hand. However, agency was not affected by misplacements of the rubber hand in any condition.
However, recent research using virtual-reality techniques, which provide the participant with active control over the artificial effector, suggests that the role of the top-down part and the distinction between ownership and agency may have been overestimated7,8. These techniques have replaced the rubber hand by a virtual hand presented to participants on a screen in front of them or by means of virtual-reality glasses9. Participants commonly wear a dataglove that translates the movements of the participant's real hand into the movements of the virtual hand, either synchronously or asynchronously (e.g., with a noticeable delay). Similar to the rubber-hand illusion, synchronous translation strongly increases the participant's impression that the virtual hand becomes part of his or her own body10.
Employing virtual-reality techniques to create the rubber-hand illusion has several advantages over both the traditional rubber-hand paradigm and the combination of the rubber-hand paradigm with the visuo-motor components11. Moving one's hand and seeing an effector moving in synchrony with it creates a much more natural situation than facing a rubber hand and being stroked by an experimenter. Moreover, the virtual manipulation provides the experimenter with much more experimental flexibility and much more control over the perceptual relation between perceiving and moving one's real hand and one's perception of the event created by the artificial effector. In particular, using virtual techniques facilitates the manipulation of factors that are likely to influence perceived ownership and agency. For instance, the shape of the virtual hand can be modified much easier and faster than the shape of a rubber hand, and the movements of the virtual hand can be of any kind and for instance involve biologically impossible movements. Among other things, this facilitates exploring the limits of the illusion, as the artificial effector need not look like a hand but may be replaced by any kind of static or dynamic event. Of both practical and theoretical interest, a virtual effector is arguably a lot more immersive and feels a lot more real than a rubber hand, which is likely to reduce the necessity to invoke top-down interpretations to make sense of the present situation.
Ownership illusions have, however, not been restricted to hands. Tsakiris12 was the first to use the stroking technique to create the impression of participants that a static face in a picture presented in front of them is their own. Sforza et al.13 have also found evidence for this phenomenon, to which they refer as enfacement: participants incorporated facial features of a partner when their own and their partner's face were touched in synchrony. The neural mechanism underlying the enfacement illusion has recently been investigated by various researchers; for a comprehensive commentary and interpretation of the findings see Bufalari et al.14. We have recently turned the regular enfacement illusion design into a virtual-reality version (the virtual-face illusion), in which participants are controlling the movements of a virtual face in front of them by moving their own head15.
Here, we describe two experiments that used the virtual-hand illusion7 and the virtual-face illusion15 paradigms, respectively, to investigate self-representation. The virtual-hand experiment included three, completely crossed experimental factors: (a) the synchrony between (felt) real-hand and (seen) virtual-effector movements, which was either close to zero to induce ownership and agency or three seconds as a control condition; (b) the appearance of the virtual effector, which looked either like a human hand or like a rectangle (so to test the effect of real-virtual effector similarity on the ownership illusion); and (c) the opportunity to control the behavior of the virtual effector, which was either nonexistent in a passive condition or direct in an active condition. The virtual-face experiment included two, completely crossed experimental factors: (a) the synchrony between real-face and virtual-face movements, which was either close to zero to induce ownership and agency or three seconds as a control condition; and (b) the facial expression of the virtual face, which was either neutral or showing a smile, to test whether positive mood would lift the mood of the participant and improve his or her performance in a mood-sensitive creativity task.
In this article we described two detailed protocols for the virtual-hand and virtual-face illusion paradigms, in which our virtual-face study was the first to replicate the traditional stroking-induced face-ownership illusion in virtual reality, together with representative results from the two paradigms.
The significant synchrony effects indicate that we were successful in inducing illusory ownership for the virtual hand and the virtual face, similar to more traditional illusion paradigms. Being able to reproduce these effects by means of virtual reality techniques has considerable advantages11,24. Virtual-reality techniques are freeing the experimenter from the rather artificial and interruptive stroking procedure and opens new possibilities for experimental manipulations. For example, morphing virtual effectors allowed us to systematically manipulate the impact of the appearance of the virtual hand and the similarity between the virtual and the participant's real hand, or the facial expression of the virtual face. The impact of agency can also be systematically explored by varying the degree (e.g., immediacy) to which participants can control the movements of the artificial effector.
Another promising avenue for future virtual reality research are first person perspective (1PP) virtual reality experiences. 1PP experiences can create an immense sense of immersion and feeling of presence, on a completely different scale than a third person perspective virtual reality experience25,26,27,28. In 1PP experiences one can truly feel like one is the avatar, that one is literally embodying the avatar. This opens up possibilities for all kinds of manipulations such as detaching parts of a person's body28, elongating29, rescaling body parts30, or changing a person's skin color31,32.
As the present and many other findings demonstrate, controlling virtual events in a synchronous fashion strongly increases the perception of these events belonging to one's own body. For example, our findings from the hand study suggest that immediate control is an important cue for distinguishing between self-produced and other-produced events (i.e., personal agency) and between self-related and other-related events (i.e., body ownership). The findings presented here and elsewhere suggest that bottom-up information plays a decisive role in the emergence of phenomenal self-representation, even for body parts that are not as identity-related as one's own body part4.
The most critical part of the described protocols is the induction process, which introduces correlations between visual, tactile and motor (i.e., proprioceptive) information-these correlations allows the cognitive system to derive ownership and agency. As these correlations rely on the relative timing of the respective events, such as the delay between the participant's own movements and the movements of the artificial effector, it is crucial to keep processing delays (especially with regard to the translation of data from the dataglove to the motion of the virtual effector on the screen) to a minimum. With our experiment setup the maximum time delay is around 40 ms, which is hardly noticeable and does not hamper the perception of causality and agency. Shimada, Fukuda, and Hiraki33 have suggested that the critical time window for the occurrence of multisensory integration processes constituting the self-body representation is 300 ms, which means that longer delays are likely to reduce the perception of control over virtual events.
Another important aspect of the protocol is tight experimental control over the participant's hand or face movements, depending on the paradigm. During induction, active movements of the respective factor are essential, as the required intersensory correlations rely on active explorative movements on the side of the participant. It is thus important to encourage participants to move frequently and to engage in active exploration. In other phases of the experiment, movements can impair the measurement however. For instance, in the virtual-hand illusion paradigm, moving the left hand (from which SCR was recorded) is likely to render measurements of the SCR level noisy and unreliable.
A limitation of the virtual-hand illusion paradigm technique is that, for practical reasons, participants commonly wear a dataglove and orientation tracker during the entire experiment (so to minimize distraction). This may not be comfortable, which in turn may affect the mood or motivation of the participant. One possible solution for that problem would be the use of lighter equipment or custom-made wearables. Another limitation of our current virtual-face illusion paradigm technique is that the equipment only registers head movements but no changes in facial expression. Allowing participants to control the facial expressions of a virtual face is likely to contribute to ownership illusions, but this would require hardware and software that provides reliable detection and categorization of facial expressions in humans-which we do not yet have available in our lab. The use of for example real-time (facial) motion capture utilities would be of great benefit in overcoming these limitations and would allow us to increase sense of agency and ownership of avatars to significantly higher levels.
As suggested by the findings from our study8, people consider various sources of information and update their body representation continuously. They seem to use bottom-up information and top-down information in a compensatory fashion, in the sense that one source of information plays a stronger role in the absence of the other-similar to what has been assumed for the sense of agency34. This provides interesting avenues for future research, as it for instance suggests that ownership can be perceived even for artificial effectors in awkward postures, provided a sufficient degree of surface similarity, or vice versa (i.e., if the artificial effector perfectly aligns with the real effector but differs from it in terms of surface features). The available findings also suggest that the boundaries between self and others are rather plastic, so that features of another person or agent can be perceived as a feature of oneself, provided some degree of synchrony between one's own behavior and that of the other35,36.
The authors have nothing to disclose.
This work was funded by the Chinese Scholarship Council (CSC) to K. M., and an infrastructure grant of the Netherlands Research Organization (NWO) to B. H.
Vizard (Software controlling the virtual reality environment) | Worldviz | Vizard allows importing hand models and integrating the hand, dataglove and orientation tracker modules through self-written command scripts. These scripts can be run to control the presentation of the virtual hand in the virtual environment, the appearance of the hand and the way it moves; they also control vibrator activities. | |
Cybertouch (Dataglove) | CyberGlove Systems | Cybertouch | Participants wear this dataglove to control the movements of the virtual hand in the virtual environment. Measurement frequency = 100 Hz; Vibrator vibrational frequency = 0-125 Hz. |
Intersense (Orientation tracker) | Thales | InertiaCube3 | Participants wear the Intersense tracker to permit monitoring the orientation of their real hand (data that the used dataglove does not provide). Update rate = 180 Hz. |
Biopac system (Physiological measurement device) | Biopac | MP100 | The hardware to record skin conductance response. |
Acquisition unit (Physiological measurement device) | Biopac | BN-PPGED | The hardware to record skin conductance response. |
Remote transmitter (Physiological measurement device) | Biopac | BN-PPGED-T | Participants wear the remote transmitter on their left hand wrist; it sends signals to the Biopac acqusition unit. |
Electrode (Physiological measurement device) | Biopac | EL507 | Participants wear the electrode on their fingers; it picks up skin conductance signals. |
AcqKnowledge (Software controlling acquisition of physiological data) | Biopac | ACK100W, ACK100M | The software to record skin conductance responses. |
Box | Custom-made | Participants put their right hand into the box | |
Computer | Any standard PC + Screen (could be replaced by VR glasses/devive) | Necessary to present the virtual reality environment, including the virtual hand. | |
Cape | Custom-made | Participants wear this cape on their right shoulder so they cannot see their right hand and arm. | |
Kinect (Head position tracker) | Microsoft | Kinect tracks the X-Y position of the participant's head. Recording frame rate = 30 Hz. | |
FAAST (Head position tracker software) | MXR | FAAST 1.0 | Software controls Kinect and is used to track the position of the participant's head. |
Intersense (Head orientation tracker) | Thales | InertiaCube3 | Intersense tracks rotational orientation changes of the participant's head. Update rate = 180 Hz. |
Facegen (Face-model generator software) | Singular Inversions | FaceGen Modeller | Facegen allows creating various virtual faces by varying various parameters, such as male/female-ness or skin color. |
Cap | Any cap, e.g., baseball cap | The cap carries the Intersense orientation tracker. | |
Computer | Any standard PC + Screen | Necessary to present the virtual reality environment, including the virtual head. |