<video> sampled between two source frames shows the NEXT frame (ceil), not the frame containing t

[vanceingalls]

Summary

When a composition is sampled at a time t that falls between two frames of a <video> source, the snapshot shows the frame at or after t (ceil(t · videoFps)), not the frame whose interval contains t (floor(t · videoFps)). On-grid times are exact. Every off-grid time shows the next source frame, not only times whose fraction is ≥ 0.5.

Other renderers (After Effects, in our comparison) show the containing frame, so the two disagree by one source frame at every off-grid time. For a baked video plate, that shows up as content one frame early or late: in our case PSNR 21.77 against ≈ 99 at a nearby on-grid time.

Observed on hyperframes 0.8.92, macOS (arm64): hyperframes snapshot . --no-browser-gpu --at <t> --no-end.

Minimal reproduction (synthetic)

1. A 24 fps clip where each frame shows its own index. Frame N has background grey 5·N for machine readout, plus its index in large digits.

# make_frames.py — 48 frames at 24 fps (2 s)
from PIL import Image, ImageDraw, ImageFont
import os
os.makedirs("frames", exist_ok=True)
font = ImageFont.truetype("/System/Library/Fonts/Helvetica.ttc", 200)
for n in range(48):
    im = Image.new("RGB", (640, 360), (n * 5,) * 3)
    d = ImageDraw.Draw(im); t = str(n); w, h = d.textbbox((0, 0), t, font=font)[2:]
    d.text(((640 - w) / 2, (360 - h) / 2 - 20), t, fill=(255, 255, 255), font=font)
    im.save(f"frames/f{n:03d}.png")

python3 make_frames.py
ffmpeg -framerate 24 -i frames/f%03d.png -c:v libvpx-vp9 -pix_fmt yuv420p -b:v 1M frames24.webm

(If your ffmpeg has freetype, drawtext=text='%{frame_num}' over a color source works too.)

2. A one-element composition (index.html):

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>video frame selection repro</title>
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
<style>html,body{margin:0;padding:0;background:#000}</style>
</head>
<body>
<div id="root" data-composition-id="main" data-width="640" data-height="360" data-duration="2" style="position:relative;overflow:hidden;width:640px;height:360px;background:#000">
<video id="v" class="clip" data-start="0" data-duration="2" data-track-index="1" src="frames24.webm" muted style="position:absolute;left:0;top:0;width:640px;height:360px"></video>
</div>
<script>
window.__timelines = window.__timelines || {};
window.__timelines["main"] = gsap.timeline({ paused: true });
</script>
</body>
</html>

3. Snapshot at times on the 1/30 s grid that fall between 24 fps frames. Pass full-precision times: 0.7333 would floor to the previous 1/30 frame, see #4430.

hyperframes snapshot . --no-browser-gpu --at 0.4,0.7333333333333333,0.3333333333333333 --no-end

Results

--at	source frame position	expected (containing frame)	observed
12/30 = 0.4	9.6	9	10
22/30 = 0.7333…	17.6	17	18
10/30 = 0.3333… (control, on both grids)	8.0	8	8

The digits and the machine-read background grey agree (50, 90, 40).

Related

#4430 (snapshot --at renders on a 30 fps grid) is separate: the times above are already on that grid, and the frame chosen for them is still the next one.

Ask

Select the source frame as floor(t · videoFps + ε), the frame whose interval contains t. Or document the rule, so callers can align their sample times.

[Dante-dan]

I traced the snapshot path to extractVideoFrameToBuffer in packages/cli/src/commands/snapshot.ts: it passes the requested source time directly to FFmpeg's -ss. The code currently allows ±1 source frame for the fast extraction path. Your on-grid control and both off-grid results suggest that the source-frame selection needs an explicit containing-frame contract; changing the composition's sampling grid would not address this.

I'd like to make snapshots select the source frame whose presentation interval contains the resolved source time, while retaining exact on-grid selection. I would use source presentation timestamps rather than rounding with the composition FPS, so variable-frame-rate input is handled consistently, and keep the existing trim, playback-rate, and loop time mapping. Verification would cover your 24 fps / 30 fps example (frames 9, 17, and 8), exact boundaries, and the reference-pair path. Moving -ss after -i alone would still need verification against the containing-frame contract.

Does that sampling rule match the intended behavior? Please assign this to me so I can implement it within the agreed scope.

[vanceingalls]

@Dante-dan yes, that rule matches the intended behavior: pick the source frame whose presentation interval contains the resolved source time, and keep on-grid times exact. Go ahead and implement it.

Notes for the PR:

• Use source presentation timestamps, not the composition fps, so variable-frame-rate input behaves the same.
• Keep the existing trim, playback-rate, and loop time mapping.
• --against goes through the same extractor, so cover it too.
• Allow for float round-off. A time like 4.1 * fps can land just below an integer and would wrongly pick the previous frame.
• Tests: the 24 fps / 30 fps example from the issue (frames 9, 17, 8), exact frame boundaries, and the reference-pair path.