teXR

Touch and text input for mixed reality

2026–present · In active development

I am building teXR with Prof. Christian Holz (ETH Zürich) and Alexander Zank.

My focus is model architecture, training, and synthetic data generation.

TouchInsight [1] addresses the problem of inferring touch input from egocentric hand tracking. The following sections summarize its published approach.

1. Touch from egocentric vision

A table provides physical feedback but no electronic contact measurement. The system must infer when contact occurs, which finger touches, and where.

Occlusion, tracking inaccuracies, and rapid hand and head motion obscure contact. Two errors matter: the finger can miss its intended target, and tracking can misplace the finger itself.

TouchInsight diagram separating user error from hand-tracking sensing error.
Tracking uncertainty and user error are distinct. Paul Streli, Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and Christian Holz, TouchInsight (UIST 2024). Unmodified figure; CC BY-NC-ND 4.0, as linked by the paper’s arXiv record.

2. TouchInsight’s uncertainty model

Let Ht be a window of tracked hand features and xt ∈ ℝ² the surface touch location. TouchInsight predicts touch events and finger identity, alongside a location distribution:

p(xt∣Ht)=N(xt;μHt,ΣHt).

Here μ is the predicted mean and Σ a diagonal covariance matrix with strictly positive diagonal entries. This represents sensing uncertainty. For typing, the published decoder combines this distribution with per-key touch distributions representing user error and language priors representing context. These methods and the figures describe the published TouchInsight system. [1, §3]

TouchInsight pipeline from tracked hands through touch estimation to probabilistic text decoding.
Touch estimation and contextual decoding in the prior TouchInsight system. Paul Streli, Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and Christian Holz, TouchInsight (UIST 2024). Unmodified figure; CC BY-NC-ND 4.0, as linked by the paper’s arXiv record.

Reference

  1. Paul Streli, Mark Richardson, Fadi Botros, Shugao Ma, Robert Wang, and Christian Holz. TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision. ACM UIST, 2024. Paper PDF.