Tasks Reflected in the Eyes: Egocentric Gaze-Aware Visual Task Type Recognition in Virtual Reality

IEEE TVCG (ISMAR Journal-track), 2024.

Zhimin Wang, and Feng Lu

PDF PPT Dataset Code


Abstract

With eye tracking finding widespread utility in augmented reality and virtual reality headsets, eye gaze has the potential to recognize users’ visual tasks and adaptively adjust virtual content displays, thereby enhancing the intelligence of these headsets. However, current studies on visual task recognition often focus on scene-specific tasks, like copying tasks for office environments, which lack applicability to new scenarios, e.g., museums. In this paper, we propose four scene-agnostic task types for facilitating task type recognition across a broader range of scenarios. We present a new dataset that includes eye and head movement data recorded from 20 participants while they engaged in four task types across 15 360-degree VR videos. Using this dataset, we propose an egocentric gaze-aware task type recognition method, TRCLP, which achieves promising results. Additionally, we illustrate the practical applications of task type recognition with three examples. Our work offers valuable insights for content developers in designing task-aware intelligent applications. Our dataset and source code will be released upon acceptance.

Presentation Video:

Supplymental Video:

Related Work


Our related work:

Gaze-Vergence-Controlled See-Through Vision in Augmented Reality

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

Edge-Guided Near-Eye Image Analysis for Head Mounted Displays

Bibtex


@ARTICLE{Wang_TVCG2024A, author={Wang, Zhimin and Lu, Feng}, journal={IEEE Transactions on Visualization and Computer Graphics}, title={Tasks Reflected in the Eyes: Egocentric Gaze-Aware Visual Task Type Recognition in Virtual Reality}, year={2024}, volume={30}, number={11}, pages={7277-7287}, keywords={Visualization;Switches;Extended reality;Measurement;Tracking;Annotations;Gaze tracking;Virtual reality;eye tracking;visual task type recognition;deep learning;intelligent application}, doi={10.1109/TVCG.2024.3456164}}