Force-aware visuo-tactile world model

How to read the figures. For each stream the upper strip is ground truth and the strip directly beneath it is the model's prediction. One column = one latent frame, so every strip column lines up with the force circle below it.
The bottom panels show normal force per gel as two rows of circles — GT above, PRED below (the head reading the model's generated latents). Circle area is proportional to newtons, so radius goes as √F; area is what the eye integrates, and a radius-proportional encoding would exaggerate large forces roughly quadratically. The scale is fixed at 7.87 N (the react_calib ceiling) across every figure, so circles are comparable between samples. A hollow ring means no contact — distinct from nothing being drawn. The thin red trace underneath is the predicted contact probability, with the 0.5 threshold dotted, so a confident miss is distinguishable from a borderline one.
The blue shaded region is the clean ground-truth context the sampler was seeded with (2 latent frames = 8 fps6 frames). Everything right of the blue line is what the model actually generated. The rows marked “predicted frames only” in the tables below exclude that context — on a short window it is half the frames, so the unrestricted numbers flatter the rollout.

What the circle value actually is. One circle = one latent frame = 4 fps6 frames, and its value is the mean force conditioned on contact — the model's training target. So a 2.2 N circle can mean "2.2 N for the whole span" or "one 2.2 N touch in four frames". To keep that readable: circle opacity encodes the contact fraction (faint = brief touch), the fraction is printed above any circle below 1.0, and the grey band at the top of each panel is the raw 6 Hz force straight from force_fps6.pt, so the underlying signal the circles summarise is always visible.

What the head is and is not. It is trained as auxiliary supervision only and is never called during sampling — rollouts are unchanged and need no force at test time. Because force is itself recovered from the GelSight image by react_calib, the head partly re-derives that mapping rather than adding independent information, and react_calib's own held-out MAE (1.23 N) is roughly the accuracy floor any readout can beat.