MouseReplayer

MouseReplayer captures how a specific human moves a mouse — not just where, but the velocity profile, micro-tremor, overshoot pattern, and approach angle. It replays those trajectories through the Linux kernel input stack at the same temporal precision they were recorded.

The Pipeline

Human → [Capture] → Parquet → [Replay] → OS Input Stack → [Validate] → Fidelity Score

The framework is built around four stages, each with its own architectural decisions documented in this wiki:

• Capture — A threaded sampler decoupled from the render loop records cursor position at 120 Hz alongside 18 kinematic features, persisted as self-describing Apache Parquet files. See Capture.
• Persist — Sessions are serialised with full target-layout metadata, making every Parquet file a self-contained record reproducible without external state. See Data Model and Seeded Layouts.
• Replay — A two-phase injection strategy (absolute warp, then relative deltas) drives a plugin-based backend architecture. The primary backend uses Linux uinput for kernel-level injection, producing isTrusted: true browser events. See Playback and Why evdev.
• Validate — Eleven kinematic features are compared between human and replay sessions; each metric must stay within 20% relative error for the replay to pass. See Fidelity Scoring.

Design Principles

1. Kernel-Level Injection for Trusted Events

Browsers expose event.isTrusted to JavaScript — events injected through Playwright or Selenium always carry isTrusted: false, detectable by a single line of client-side code. To produce events the browser treats as genuine hardware, injection must happen below the browser. See Why evdev and Not xdotool/ydotool.

2. Sampling Decoupled from Rendering

Vsync-based frame capping limits capture rate to the display refresh rate (e.g., 60 Hz on a 60 Hz panel), aliasing the kinematic signal. A dedicated SamplerThread fires at perf_counter-based deadlines independently of the render loop, maintaining 120 Hz capture even on a 60 Hz display. See Why Sampling Is Decoupled.

3. Deterministic, Self-Contained Sessions

If a recorded session's target layout depends on external state — an unseeded RNG, a config file that may not exist at replay time — the environment cannot be faithfully reconstructed. Every recording starts from a hex seed that deterministically drives target layout generation via numpy.random.default_rng(). The seed, the target positions, and the arena dimensions are stored in the Parquet file's metadata. Replaying a session reconstructs the exact layout from metadata alone — no RNG, no external config. See Seeded Layouts.

4. Minimum-Jerk as a Comparison Baseline

To measure what makes a human trajectory distinct from the shortest-path ideal, you need a reference baseline. The ideal trajectory between two points is modelled as a 5th-order polynomial minimising integrated jerk (rate of change of acceleration), producing a symmetric bell-shaped velocity profile with zero velocity at start and end. This is not a generative human model — it is that baseline. Deviation from the ideal is the signal. See The Minimum-Jerk Trajectory.

5. Two-Phase, Not Single-Curve

Real human pointing has a ballistic (open-loop) phase followed by a corrective (closed-loop) phase, producing multi-peak velocity profiles. The current phase detection labels samples as grab, transit, approach, dwell, or click based on spatial proximity to the target, capturing the transition from gross transport to fine correction. See Movement Phases and Two-Phase Pointing.

Guide to This Wiki

The wiki is organised into five sections. If you're new to the project, read in this order:

1. Concepts — The motivation and theory. Start with Why Mouse Movement Fidelity Matters, then Fitts' Law, Minimum-Jerk Trajectory, Two-Phase Pointing, and Jitter.
2. Data Model — The schema, phase labels, and layout reproducibility. Reference pages you will return to while working with recorded data.
3. Capture — Two architectural decisions exclusive to the capture path: the threaded sampler and the vsync-off display config.
4. Playback — How a Parquet file becomes injected hardware events. Read in sequence: backend architecture, evdev rationale, absolute+relative injection, and why stored dx/dy are ignored.
5. Analysis — What you do with recorded data: fidelity scoring, corpus construction, and cross-session biometric consistency.

Project Layout

src/mousereplayer/
├── arena/            # Rendering, target primitives, deterministic canvas generation
├── telemetry/        # Sampling, kinematic recording, feature extraction
├── storage/          # Parquet I/O, date-based session organisation
├── analytics/        # Kinetic baseline model, validation, segment comparison
├── playback/         # PlaybackEngine, uinput injection driver, backend registry
├── observability/    # Coloured console and file logging
├── config.py         # Frozen dataclass configuration
├── controller.py     # AppController — recording lifecycle coordinator
├── main.py           # CLI entry point
└── __main__.py       # `python -m mousereplayer` entry point

Quick Start Reference

pip install -r requirements.txt

# Interactive capture session
python -m mousereplayer record

# Replay via hardware injection (dry-run for non-Linux)
python -m mousereplayer playback sessions/*.parquet --dry-run

# Validate playback fidelity
python -m mousereplayer validate sessions/human.parquet sessions/replay.parquet

# List and inspect recorded sessions
python manage.py list
python manage.py inspect sessions/*.parquet