Photo-Tactile VR

About the Project

Researchers at the University of Rochester have been exploring a way to create touch sensations from light through a thermoelastic process. Short pulses of light are absorbed at the fingertip and converted into rapid, controlled vibrations, all without any physical contact with the skin. This is known as Photoacoustic Haptics. In recent experiments, the researchers confirmed that this sensation works in a psychophysical study, in which human participants reported the sensation as mostly vibration with a secondary feeling of heat. To follow this up, the group is now exploring whether the same approach can be used to represent real-world surfaces.

Real-world surfaces are distinguished by their texture, which is largely encoded in the frequency response of a surface as the finger moves across it. The researchers aim to reproduce this by controlling the frequencies felt at the fingertip, combining multiple light sources pulsed at different rates as shown above. To evaluate how faithful these sensations are, the effect is compared against the virtual textures in the Penn Haptic Texture Toolkit, a standard library that has been used to benchmark non-contact haptic devices for applications such as remote surgery. The researchers plan to test this in a further psychophysical study, measuring how well participants can feel these virtual surfaces in a VR environment and, ultimately, how well real-world textures can be represented through light.

The VR experience serves to display visual materials and explore perception of photo-tactile stimuli focusing on roughness. Current XR experiences are weak in the tactile sensory dimension, this research can enhance multimodal presence in XR. 

Key Features

• Minimal virtual environment: Provides a controlled setting designed specifically for the research experiment.
• Hand-tracking integration: Captures users’ hand movements and interactions within the virtual environment.
• Collision detection: Detects contact with virtual materials to synchronize the experience with haptic feedback.
• Two experimental phases: Represents distinct stages of the research experiment.
• Interactive interface: Supports experiment setup and transitions between trials.

Studio X designed a minimal environment to draw focus onto the texture tiles presented. This allows the user to intuit exploring the presented texture tiles by touch. 

Hand tracking was implemented as the primary input. The experiment revolves around sensations on the hands, using hand tracking eliminates the need for controllers and frees up the hands for application of the haptic device.

Contact detection is implemented between contact surfaces of the hands with texture tile representing what material is being simulated. Contact cues are visible to researchers but not participants, keeping the immersive integrity of experience and still providing sufficient visual cues to researchers running the trials, allowing for synchronized haptic feedback in the virtual environment. 

The current design accommodates two separate phases: Exploration and Questioning. Exploration presents consecutive texture tiles for the user to explore and receive haptic feed back from to anticipate testing in the Questioning Phase. The question phase presents a mystery tile where only haptic sensation is given for identification.

A researcher control interface is provided to configure and manage experiment sessions. Researchers are able to specify the materials used for the experiment both before and during runtime, control which materials are currently being displayed and freely switch between the phases.

Video Playthrough

Next Steps

Next, the team will integrate the VR experience with the laser system so that contact with a virtual surface automatically triggers the corresponding haptic response, eliminating the need for manual researcher input. Once the system is fully integrated, the team will begin user testing to evaluate its reliability, usability, and ability to represent distinct material textures. Findings from these initial trials will inform further improvements to the experimental workflow, virtual environment, and synchronization between visual and haptic stimuli.