Auditory, Visual, and Tactile Interactions During Active, Dynamic Touch

Overview

Even though extensive research on the multisensory integration of auditory and visual stimuli has been carried out, the mechanisms of integration of tactile stimuli with other senses remain less known and understood. Furthermore, the brain mechanisms associated with active and dynamic tactile exploration of a surface have not been extensively studied. In the context of the development of human-computer interaction (HCI) technologies, understanding these mechanisms remains of vital importance for the realization of multisensory devices and to improve the user experience of the general population, but also to benefit the use by clinical populations, e.g., people with visual impairments. The planned experiment aims to study multisensory integration during the active and dynamic tactile exploration of a surface (a natural texture or the screen of a multisensory tactile device). The primary hypotheses are that simultaneous auditory and/or visual stimulation during active tactile exploration of a surface will help participants form a mental representation of the shape or texture they are exploring, and that the recorded brain activity will be compatible with multisensory integration mechanisms at the level of the cerebral cortex. The planned project will include (1) behavioral (psychophysical) experiments, to assess participants' performance in discriminating the spatio-temporal location of tactile, auditory, visual, audio-tactile, audio-visual, and audio-visual-tactile on the screen of a multisensory tactile device and (2) surface electroencephalography (EEG) recording experiments, which will be employed to study the cortical mechanisms of multisensory integration during active and dynamic tactile exploration .

This study, entitled "InTOUCH: Auditory, visual, and tactile interactions during active, dynamic touch" is conducted within the overarching framework of a multi-partner European project (Marie Curie Initial Training Network MULTITOUCH, entitled "Multimodal haptic with touch devices"). The MULTITOUCH project aims to train researchers in the field of haptics, and gain scientific knowledge on how tactile feedback can be integrated with auditory and tactile feedback in human-computer interfaces (HCIs). For this purpose, the ''InTOUCH'' study will use psychophysics and non-invasive electroencephalography (EEG) in healthy human subjects to characterize how the brain integrates tactile information with auditory and visual information during active touch. While much work has been done to investigate somatosensation, both at the perceptual and neural level, studies investigating touch from a more naturalistic view, taking into account its nature of an active, dynamic exploration process, are limited. Furthermore, the neural mechanisms that underlie integration of tactile and visual/auditory information under conditions of active tactile exploration remain largely unknown. The ''InTOUCH'' study will investigate, in conditions of active dynamic touch (e.g. the active tactile exploration of a display) and passive touch, the multisensory interactions between tactile, auditory, and visual stimuli using psychophysics, and exploit EEG-based approaches to isolate and characterize cortical activity related to the processing of (1) somatosensory input produced by the mechanical interactions between the contacting finger pad and tactile displays, (2) somatosensory input produced by an ultrasonic mid-air haptic stimulator allowing to stimulate mechanoreceptors of the hand and (3) concurrent auditory and visual stimulation. Conventional tactile stimulators used in somatosensory research rely on surface vibration. To deliver tactile stimuli in conditions of active dynamic touch, tactile displays based on controlled friction will be used. Controlled friction devices operate by active modulation of the frictional effects between a finger and a surface. Minute and imperceptible vibrations of the plate at ultrasonic frequencies via piezoelectric actuation induces a squeeze film that reduces friction between the fingertip and display. Transient modulations of friction while the finger is sliding on the display can generate tactile sensations, including the sensations of sliding the finger against an edge or texture. Another novel approach to activate skin mechanoreceptors in conditions of active touch are mid-air ultrasonic speakers organized in an array to emit ultrasound waves onto the skin to create sensations of touch on the hand palm. Such devices are commercially available and used for virtual reality setups (e.g. Ultrahaptics Stratos Explore; https://www.ultraleap.com/haptics/). An important advantage of such stimuli is that they can generate naturalistic but nevertheless controlled tactile sensations during free exploration with the hands. Because timing of the stimuli is controlled experimentally, EEG can be used to sample, non-invasively, the cortical activity elicited by the mechanical stimuli in various conditions of active and passive touch. Specifically, two EEG approaches will be exploited: the recording of transient sensory-evoked brain potentials (SEPs) and the recording of steady-state evoked potentials (SS-EPs). Unlike conventional transient SEPs which reflect a phasic cortical response triggered by the occurrence of a brief stimulus, SS-EPs reflect a sustained cortical response induced by the long-lasting periodic repetition of a sensory stimulus, thought to result from an entrainment of neuronal populations responding to the periodically-modulated feature of the stimulus. It is expected that SS-EPs will offer a unique mean to isolate the sustained cortical activity induced by the sustained presentation of a sensory stimulus, and how this activity is modulated by concurrent auditory and/or tactile stimuli. Frequency-domain analysis of the EEG activity elicited by such stimuli is thus particularly well suited to investigate the temporal dynamics underlying the cortical representation of sustained active touch. Furthermore, as compared to other non-invasive approaches to sample brain activity, investigating brain function using SS-EPs offers several advantages. First, SS- EPs exhibit a high signal-to-noise ratio. Second, SS-EPs allow isolating neural activity related specifically to each of several, concurrently applied streams of sensory stimuli to study the processes involved in multisensory integration.