STATUS: closed — superseded on the energy claim

EPIC (arXiv 2606.15859) reports a 24.3× average energy reduction for gated always-on perception on AR glasses, using reference-frame comparison to skip frames during head stability. This project is in the same family and does not compete on that metric. No energy-reduction figure is claimed here.

Energy numbers in this project come from an analytic FLOP model (FLOPs × [J/FLOP]), not from a power meter. That is not a limitation to be fixed later — it is the reason no reduction figure is claimed.

What remains measured and uncontested is the trust gate: a correction for selection bias under gating. See "The trust gate" below. This is the part of the work that continues.

Saccade patch predictor

A tiny forward predictor for always-on edge vision: given a motion-compensated cache of ViT patch embeddings plus a pseudo-IMU signal, it predicts what each patch embedding will look like in the next frame — so the encoder only has to re-run on patches whose change the prediction could not explain.

This is the learned component of Saccade (prediction-error gating + energy-budget control for an always-on edge VLM).

• 💻 Code: https://github.com/NagaYu/saccade
• 📊 Benchmark dataset: https://huggingface.co/datasets/NagaYu/saccade-egomotion-bench
• 🚀 Interactive demo: https://huggingface.co/spaces/NagaYu/saccade

The trust gate: correcting selection bias under gating

This is the main contribution of this artifact.

The problem

At deployment, ground truth only exists for patches the gate chose to encode (the surprising ones). But the prediction is applied to the patches the gate skipped (the unsurprising ones). A correction fitted on the former is wrong for the latter.

It is not specific to Saccade

Any system that decides for itself when to look has this problem, because the decision that saves energy is the same decision that selects which outcomes get observed. A frame-skipping pipeline only sees full-quality truth on the frames it processed — the ones that looked different. A duty-cycled sensor only reports while it is awake, and it is woken by events. A per-patch encoder only re-measures the patches whose cheap proxy fired. In every case, any model of "what happened while we weren't looking" that is trained on what was looked at is trained on a biased sample, and the bias points exactly where it will be applied: toward large changes, when it is used on small ones. Nothing about this needs a learned predictor to go wrong; it only needs a correction that is fitted on observed outcomes and applied to unobserved ones.

What happened here

An earlier version of this predictor made things worse: fidelity decreased monotonically with learning rate, and doing nothing (pure motion-warped reuse) beat it at every setting. Measured on the held-out stream (seed 99, SyntheticBackbone, before the fix):

The learnt head converged to an almost constant correction: fitted on surprising patches, added to every unsurprising one.

The two fixes (both are in this checkpoint)

1. Residual conditioning — the gate's cheap per-patch appearance residual is an input feature, so the network can tell which regime a patch is in.
2. An earned trust scalar — the applied correction is warped + trust · residual, where trust ∈ [0,1] is raised or lowered by a counterfactual check ("would the full residual have beaten doing nothing?") evaluated on a handful of randomly explored patches drawn from the skipped population — i.e. the distribution the prediction is actually used on. Exploration costs real energy (explore_frac, ~1% of patches) and is counted as encoding.

Result

Trust settles around 0.35–0.50 where the residual helps and decays toward 0 where it does not, so turning the predictor on is never worse than pure motion-compensated reuse — even at a learning rate that would otherwise be destructive. This is an empirical property, checked by regression tests (test_trust_gate_prevents_predictor_from_backfiring at a benign and a destructive learning rate; test_trust_decays_when_residual_is_useless), not a theorem.

trust and gain_ema are saved as buffers, so a loaded checkpoint keeps the trust it earned. The shipped value is calibrated on the held-out stream, not on the training streams: during offline training the head starts untrained, so the counterfactual check correctly drives trust to 0 early on and recovers only slowly. Saving that transient would hand you a predictor that applies no correction until it re-earns trust online.

What this is (and honestly, what it is not)

The predictor is not what produces most of Saccade's energy saving — motion-compensated cache reuse is. The predictor's job is to recover the part of the change that the spatial warp gets wrong. Its value therefore scales with how lossy the warp is in a given embedding space, and we measured exactly that:

(mean cosine fidelity of the reconstructed patch-token map vs a Full encoder, on a held-out ego-motion stream, seed 99.)

The pattern is the point. Where each patch embedding is an independent function of its own pixels, a spatial warp is nearly exact, the residual left over is by definition the unpredictable innovation, and there is almost nothing to learn. Where tokens mix global context through self-attention (any real ViT), warping is a cruder approximation and the predictor recovers a meaningful chunk of the loss.

Files

The GRU's weights are shared across patches, so a checkpoint is independent of the patch grid (196 patches at 14×14 and 256 at 16×16 both load fine). What a checkpoint is tied to is the embedding space it was trained on — use the matching file.

Usage

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from saccade import GRUPatchPredictor, SaccadeEngine, EngineConfig, make_backbone

backbone = make_backbone("hf:facebook/dinov2-small")
predictor = GRUPatchPredictor(backbone.embed_dim, backbone.n_patches, lr=1e-3)
predictor.load_state_dict(load_file(hf_hub_download(
    "NagaYu/saccade-predictor", "predictor_dinov2-small.safetensors")))

cfg = EngineConfig(mode="saccade", budget_watts=0.02, fps=10.0)
engine = SaccadeEngine(backbone, cfg)
engine.predictor = predictor          # warm start instead of learning from scratch

for frame in your_video_stream:       # HxWx3 uint8 RGB
    r = engine.step(frame)
    print(r.n_encoded, r.encoded_fraction, r.cum_joules, r.budget_ok)

Install the package first: pip install git+https://github.com/NagaYu/saccade.

Training

Self-supervised, no labels: targets are simply the vision encoder's own outputs. The engine trains the predictor online on whichever patches the surprisal gate already paid to encode; for this published checkpoint we additionally remove the gate's selection bias by supervising on every patch offline (train_predictor_on_all=True, valid only when ground truth is being computed anyway).

• Streams: 3–5 scripted ego-motion sequences (synthetic_walking_stream, seeds 0/2/3[/4/5])
• Objective: ||prediction − true embedding||², SGD(momentum=0.9), lr 1e-3
• Held-out evaluation: seed 99, never seen during training

Reproduce with hf/export_model.py.

Limitations

• Evaluated on scripted synthetic ego-motion, not natural video. The regimes (static / walk / turn / independently-moving object) are controlled on purpose so the claim is reproducible, but they are not a substitute for a real wearable-camera study.
• The predictor is tied to its embedding space; there is no cross-backbone transfer.
• Energy figures throughout the project come from an analytic FLOP model (FLOPs × [J/FLOP]), not from a power meter.
• The gain on the synthetic space (+0.0008) is small enough to be within run-to-run noise for practical purposes — it is reported because the shape of the result matters, not because that checkpoint is impressive.

References

These are prior or concurrent work in the same problem area.

• EPIC — T. Xia, H. Wang, J. Zheng, S. Chen, S. Q. Zhang. EPIC: A System Framework for Efficient Egocentric Perception on Embodied AR Glasses. arXiv:2606.15859, 2026. https://arxiv.org/abs/2606.15859
• EgoTrigger — A. Paruchuri et al. EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glasses. arXiv:2508.01915, 2025. https://arxiv.org/abs/2508.01915
• Ego-METAS — M. Santos-Villafranca et al. Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark. arXiv:2606.02246, 2026. https://arxiv.org/abs/2606.02246

License

MIT