Frame Finder

Find the closest video frame to a screenshot or photographed screen. The matcher compares spatial patterns of shapes and colors, then exports the strongest match and a side-by-side comparison.

Setup

Requires Python, FFmpeg, and FFprobe on your PATH.

./setup.sh
./check.sh

Code layout

• frame_finder/search.py: frame decoding, candidate search, image exports, and search CLI.
• frame_finder/scoring.py: batched image correlation.
• frame_finder/pipeline.py: generic full-image descriptors, verification, and repeated-view grouping.
• frame_finder/index.py: reusable disk-backed feature indexes.
• frame_finder/compact_index.py: compact descriptor discovery and exact shortlist verification.
• frame_finder/monitor.py and monitor.html: local search dashboard.
• frame_finder/shard_decoder.py: parallel decoding and worker selection.
• frame_finder/hardware.py: hardware-decoder options.
• frame_finder/align_livestream.py: livestream alignment and clock reports.
• tests/: decoding, scoring, search, hardware, and shard regression tests.
• Root search.py and align_livestream.py: thin CLI entry points; existing commands are unchanged.

Run ./check.sh from any directory to lint, check formatting, and run the tests. Package CLIs also support .venv/bin/python -m frame_finder.search and .venv/bin/python -m frame_finder.align_livestream from the project directory.

Find the source from an uncropped photo

The search dashboard, search.py, frame_finder.search, frame_finder.library, and frame_finder.workflow all use the same workflow. Livestream alignment uses that workflow before adding clock calculations. Search every video in a local folder without supplying a source video or screen corners:

.venv/bin/python search.py --photo assets/reference.jpg

By default, the library is inputs/ (including subdirectories). Use --videos /path/to/broadcasts for another collection. Supported extensions are MP4, WebM, MKV, MOV, M4V, and AVI. Every automatic search decodes the source videos afresh, using up to four parallel decoders over five-minute segments. Frame descriptors are compared in batches, and only a bounded shortlist of candidate pixels is kept in memory. No reusable frame or descriptor cache is read or written. The legacy --cache option is ignored for automatic searches.

The automatic baseline proposes screen quadrilaterals from color regions and edges, tries several lower-edge positions to handle foreground occlusion, and includes a whole-image fallback for screenshots. It uses no fixed calibration, pretrained weights, or training. The same generic pipeline handles photos, screenshots, and already-cropped references. Cropped references skip screen detection. Discovery and verification compare the full image rather than a game-specific fixed region. Each crop searches every video's sampled descriptors. Half of each bounded shortlist protects the highest scores; the remaining slots prioritize visually different views so repeated camera angles do not crowd out other scenes. High-scoring occurrences at distant timestamps remain candidates. Candidates are verified at 192×108, then grouped by visual similarity while retaining every distinct shortlisted source/timestamp occurrence. Duplicate crop hypotheses for the same source frame keep the highest score. Refinement retains top-scoring moments and reserves slots for other view groups, then scans their neighborhoods frame by frame at that resolution. result.json records candidate_groups, their occurrences, and each refined match's view_group. Grouping applies to the bounded candidate pool; it does not claim to catalogue every frame in the video. Different files containing the same broadcast can both appear in the results.

--discovery-mode keyframes is an experimental faster option, also available under Advanced options → Discovery in the dashboard. It first searches keyframes across the entire source, then selects visually diverse seed moments (at least 12, or three times the refinement count). Each seed expands to five neighboring keyframes on either side, using actual source timestamps. Overlapping windows merge before sampled discovery runs inside them; the strongest sampled candidates then receive frame-by-frame refinement. The dashboard reports this nearby-video search separately from source traversal, refinement, and export. Reports include expansion windows, sampled seconds, and frame counts. All source data is decoded afresh without a reusable source cache.

On the local livestream, the previous keyframe-only approach selected the wrong moment. Context expansion now recovers the known 7032.466 s match using both the cropped reference and full photo. The cropped-reference run scored 5,378 discovery frames (3,718 keyframes plus 1,660 regional samples), versus 37,340 for sampled discovery, and sampled 830 seconds of expanded context within the 18,670-second video. A single expanded run took 10.0 seconds; timings are workload-dependent and this fixture does not establish general recall. Sampled discovery remains the CLI default. The dashboard defaults to improved keyframe discovery with a candidate shortlist of 32 and 16 moments to refine. Both modes can miss a brief scene; keyframe neighborhoods are a heuristic rather than complete frame coverage.

A separate one-run comparison of the old and new sampled pipelines on that target took 14.14 s and 13.91 s, respectively, retaining the same winning timestamp. That difference is too small to establish a speed improvement; the sharper verifier and scene-diverse pool are implemented, but broad speed/accuracy improvements still need independent references. Benchmark reports are in outputs/performance/live-search-speedups.json and outputs/performance/scene-search-comparison.json.

Results include source file paths, video-relative timestamps in seconds, detected corners in original-photo pixels, and comparison images under outputs/source-search/. screen-proposals.jpg shows the proposed crops; result.json includes ranked matches and discovery settings. The result is an unverified candidate, not a calibrated confidence or proof that the source exists in the library. Similar scores at different moments/files set an ambiguity flag. Inspect the comparisons even when that flag is false.

Discovery is approximate: the default --step 0.5 samples source frames at roughly half-second intervals, --shortlist 64 verifies up to 64 distinct five-second buckets per crop per video, and --refine 6 refines up to six distinct video/time candidates. Increase these counts or reduce the step to broaden discovery. Brief scenes can be missed. Screen proposals can miss grayscale, heavily occluded, very small, or unusually rotated displays. This baseline needs evaluation on independent real photos before assuming broad accuracy.

Search a video

For an already cropped reference image:

.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg

For a photographed screen, provide its four corners in the original photo's pixel coordinates, clockwise from the top-left:

.venv/bin/python search.py --video /path/to/video.mp4 --photo /path/to/photo.jpg --corners 100 120 800 100 850 550 90 580

Local defaults use inputs/match.webm and assets/reference.jpg. Photos, videos, metadata, and generated outputs are excluded from Git and must be supplied locally. Always provide --corners for a new photographed screen; the built-in calibration is specific to a local reference.

Each run saves best-frame.jpg, comparison.jpg, candidates.jpg, and JSON reports in outputs/. Use --output to keep separate results. Default paths are relative to the project; explicit relative paths are relative to the current directory.

Coverage and speed

The default --mode all scores every decoded frame and preserves its timestamp. Complete coverage applies to the input video's representation; a low-resolution or lower-frame-rate copy cannot provide coverage of frames absent from that copy.

Scoring processes small batches of 64 frames, reusing transient conversion buffers and using the same per-channel reductions as scalar scoring. FFmpeg chooses its software-decoder thread count automatically. Neither change skips frames or changes the comparison region. Equivalence checks require identical scalar and batch scores, including constant images and a final partial batch. Buffer reuse improved scoring throughput by approximately 5% in repeated microbenchmarks; decoding still dominates full-resolution searches.

Raw decoder output is also read in batches of 64 frames. These are temporary in-memory buffers, with no matching cache. Bulk reads improved decoding by approximately 8% on a low-resolution sample and had no measurable effect on the 1080p sample.

On measured local samples, scanning a 30-second 1080p60 window improved from a median 2.55 seconds to 2.28 seconds; a three-minute 144p window improved from 0.73 seconds to 0.66 seconds. Both produced identical candidate rankings, scores, timestamps, frame counts, and decoded-pixel hashes. A full 144p archive scan checked all 559,542 frames in 55.0 seconds, versus 68.6 seconds with scalar scoring and four decoder threads: about 25% higher throughput, with identical candidate rankings and coverage. Both benchmark implementations used the corrected timestamp parser. Speed depends on the codec and machine; the 1080p measurement covers a window, not a full-video scan.

For a faster approximate search, use --mode sampled --step 0.5 or --mode keyframes. These modes refine every frame around promising candidates, but they can miss a brief scene elsewhere.

To restrict the search to a known neighborhood:

.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --start 100 --duration 8 --output outputs/verification

Refinement stays within the interval specified by --start and --duration. Reports include the initial scan, each refinement window, and their frame counts. total_frames_scored counts all scoring operations, including frames scanned again during refinement.

Repeated searches with a feature index

Build once, then search without decoding the archive again:

.venv/bin/python -m frame_finder.index --video /path/to/video.mp4 --output outputs/video-index
.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --index outputs/video-index

Both search.py and align_livestream.py accept --index.

The index records every decoded frame's source timestamp, float32 centered channels, and channel norms. Indexed searches use the same reductions and candidate ordering as live searches, rather than an approximate descriptor. Features are read through memory maps in batches of 64 frames. --start and --duration can restrict indexed searches; sampled/keyframe modes and hardware options cannot be combined with --index. Original frames are still decoded to export the winning images.

This trades storage and initial indexing time for faster repeated searches: about 40 KB per frame, or 22.5 GB for 559,542 frames. Building is currently serial. Each build and search hashes the source video to detect stale indexes, so searches still read the compressed video once; they do not decode it. Feature settings are versioned and validated. Existing index directories are never overwritten, and failed builds do not publish partial indexes. Delete and rebuild an index when the source or feature settings change. This initial format does not checksum feature-file contents; preserve index files unchanged.

Compact discovery with exact verification

For repeated searches where approximate discovery is acceptable:

.venv/bin/python -m frame_finder.index --video /path/to/video.mp4 --format compact --output outputs/compact-index
.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --index outputs/compact-index

Compact indexes store a 12×6 descriptor for each of four channels, the original decoder's resized comparison-crop pixels, and source timestamps: approximately 8.7 KB/frame rather than 40.2 KB/frame. Every frame's descriptor is examined. The best three descriptor candidates per five-second bucket are retained; the best 128 buckets are shortlisted. Cached source pixels for those candidates are rescored with the unchanged exact matcher. No source-video seeks are needed for verification; exports still use the original video.

--index-shortlist increases the number of verified buckets, and --index-per-bucket increases the number of verified frames per bucket. Discovery is approximate: a matching frame outside the shortlist can be missed. Exact verified scores do not imply an exhaustive winner or identical full rankings. Reports set approximate_candidate_discovery: true and complete_frame_coverage: false, and distinguish descriptor coverage from exact verification counts. Existing exact indexes remain supported and are the default build format.

To benchmark speed and accuracy against an exact index:

.venv/bin/python scripts/benchmark_index.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --exact-index outputs/exact-index --compact-index outputs/compact-index --output outputs/index-benchmark.json

On the local five-hour archive (559,542 frames), three interleaved repeats measured a median 18.98 seconds for the exact index and 0.727 seconds for the compact index: approximately 26× faster, with the same winning frame and exact score for the existing reference. Three additional references also retained their exact winners, with single-run speedups of 12–18×. The compact index was 4.86 GB versus 22.49 GB for the exact index. The exact index exceeds available RAM on this machine, so much of the advantage is reduced I/O; these are workload-specific results, not a universal speed or accuracy guarantee. Raw local results are under ignored outputs/performance/compact-loop/.

The benchmark includes source validation, discovery, and verification, but excludes index construction and image exports. It alternates search order across repeats and reports storage, exact winning-frame agreement, and speedup. It exits unsuccessfully if any supplied reference misses the exact winner or fails the 2× speed target; this is a regression check for those queries, not an accuracy guarantee for all references.

Search dashboard

The dashboard uses TypeScript, React, Tailwind, TanStack Router, and TanStack Query. Install frontend dependencies with bun install after running ./setup.sh for the Python environment.

bun run dev

This starts the Python API at http://127.0.0.1:8765 and the web dashboard at http://127.0.0.1:5173. Open the dashboard URL. Stop any standalone API already using port 8765 first. The API automatically restarts when Python source files or its fallback HTML change, with a 300 ms debounce. Outputs and input videos are outside the watch paths. Reloading the API cancels an active search. Both servers stop together when either exits or you press Ctrl+C.

The dashboard defaults to the uncropped assets/reference.jpg and inputs/livestream.mp4 broadcast. Video defaults and presets use project-relative paths. Relative paths submitted to the dashboard resolve from the project root. Choose another image, video, or video folder if needed, then Search. The dashboard fits the window, with controls and image panels in one view. The TV crop preview appears as soon as screen detection produces it. Results show Reference Target and Most Likely Match, including the found frame's source filename and timestamp. Advanced options contain image type, local examples, and tuning; Diagnostics contains counts and logs. Automatic searches decode videos afresh, without a reusable video cache.

For a compiled dashboard served directly by Python, run bun run build, then .venv/bin/python -m frame_finder.monitor --port 8765. Python serves dashboard/dist/ when available. Searches save their results under outputs/monitor/. Both servers bind to loopback.

Frontend checks: bun run check runs TypeScript, Oxlint, Oxfmt, and component tests. Use bun run format to apply formatting. Python checks remain ./check.sh.

Search jobs publish atomic per-worker telemetry at most twice per second. The dashboard polls each second. During automatic searches, scan progress sums decoded source-time spans across every video and divides by their combined duration. Each worker updates during decoding; completed videos retain their contribution. The display shows scanned video time and total video time. If any duration is unknown, it shows scanned time without a percentage. Refinement and export have separate phases and item counts, while completed scan time remains visible. Diagnostics shows frames scored, descriptor counts, and current best scores/timestamps. Timeline percentage measures source-time ranges scanned, not elapsed-time prediction or a claim of exact coverage for approximate indexes. Cancelled/failed jobs retain their last partial progress. Cancellation terminates the subprocess group and escalates if necessary; shutting down the monitor also cancels its active job.

Vite updates frontend code automatically during development. Python source changes automatically restart the API; the dashboard reconnects and clears any expired job selection. Previously started subprocesses cannot publish the newly added telemetry.

Parallel full-video decoding

Both commands automatically choose between one and sixteen workers to decode and score separate time ranges in parallel. Use --workers auto explicitly or override the choice:

.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --workers 4

Automatic selection uses the CPU count available to this process, codec, video duration, resolution, and frame rate. It requires at least 15 seconds and one billion source pixels per worker. Low-resolution H.264 inputs (at most 320×180 source pixels) lasting at least ten minutes can use up to sixteen workers, allowing two workers per three available CPUs; other inputs allow one worker per two CPUs and are capped at four. Short or small inputs and missing workload metadata use one worker. This is a conservative estimate, not a runtime benchmark; no probing decode or cache is needed. Use --workers 1 to force serial scanning or a positive count to override the estimate.

Parallel scanning applies to complete --mode all software scans of MP4/MOV and Matroska/WebM files. Bounded searches, approximate modes, hardware decoding, other containers, and inputs with negative or unavailable start times or unavailable durations use the serial path. Reports record the actual worker count in coverage.decoding_workers.

Each worker asks FFmpeg to seek to its range's start; the demuxer seeks back to the required keyframe. There is no additional fixed three-second preroll. Half-open ranges prevent overlap, and the first and last ranges preserve the video's edges. Workers retain original source timestamps, score every frame in their ranges, and merge the best candidate from each time bucket using the same ordering as serial scanning. Decoder thread counts are divided among workers, except low-resolution H.264 workers use one decoder thread to avoid frame-threading overhead.

On a one-minute 1080p VP9 clip, scanning took 4.03 seconds with one worker, 3.06 with two, and 2.87 with four: about 29% less elapsed time with four workers. All decoded pixels, source timestamps, candidate scores, rankings, and coverage matched in the tested clips. Short low-resolution clips were slower because process startup outweighed the work saved; use one worker for those. Performance depends on video length, codec, storage, and available CPU cores.

A complete five-hour 144p H.264 archive took 52.1 seconds with the previously published serial implementation and 18.7 seconds with four workers: approximately 2.8× faster. Both scored all 559,542 frames with identical retained candidates, scores, rankings, first and last timestamps, and coverage. The winning timestamp also matched. This full-file comparison includes the bulk-read improvement as well as parallel decoding.

Further tuning tested separate decoder thread counts and removal of fixed preroll. Low-resolution H.264 benefited from one decoder thread per worker; the 1080p VP9 sample performed best with its existing divided thread budget. Six workers helped the full archive but slowed the shorter 1080p sample, which motivates the workload-specific automatic limits. Pixel and timestamp checks include keyframe boundaries, open and closed GOPs, reordered frames, and six-worker partitions.

In interleaved comparisons with two runs per setting, the previously published automatic defaults took a median 21.72 seconds for the full archive and the tuned defaults took 16.87 seconds, about 22% less time. The archive automatically changed from four workers to six, using one decoder thread each. The roughly one-minute 1080p sample retained four workers and improved from 3.31 to 3.00 seconds, about 9% less time, after removing fixed preroll. Both comparisons preserved exact candidates, scores, rankings, timestamps, frame counts, and coverage. Elapsed times vary between benchmark sessions; these comparisons used the same session and inputs.

The latest full-archive worker sweep on 24 available CPUs measured medians of 15.17 seconds (six workers), 14.67 (eight), 14.60 (twelve), 13.15 (sixteen), and 13.81 (twenty-four). Sixteen workers preserved all 559,542 frames and exact retained candidates/timestamps: about 1.15× faster than the six-worker baseline, not the requested 2×.

GPU pipeline evaluation

See the GPU evaluation for local capabilities, correctness trade-offs, and results. The experimental benchmark does not change production defaults:

.venv/bin/python scripts/benchmark_gpu.py --video /path/to/video.webm --reference /path/to/cropped-reference.jpg --duration 30 --repeats 3 --output outputs/gpu-benchmark.json

It compares software scaling, full-size VAAPI downloads, GPU-side VAAPI scaling, and Vulkan decode with libplacebo scaling. Run it without competing benchmarks. It buffers raw frames for the requested window; keep windows short. Timings exclude image exports and are not comparisons against the automatic parallel full-video scan. GPU resizing can change pixels and scores even when frame/timestamp coverage matches.

Optional hardware decoding

Both commands accept --hwaccel vaapi on Linux with a compatible GPU, driver, and render device:

.venv/bin/python search.py --video /path/to/video.mp4 --reference /path/to/reference.jpg --hwaccel vaapi --hw-device /dev/dri/renderD128

Decoding requests the source pixel format and downloads frames at their original size. Resizing and scoring use the existing software path. No frame skipping, GPU resizing, or disk caching is introduced. hardware_acceleration_requested records the selected option; it is not an independent verification of GPU activity.

Software decoding remains the default. On the tested Radeon RX 7800 XT, VAAPI preserved every tested pixel, timestamp, and score in VP9 and H.264 samples but was slower: approximately 3.1 versus 2.2 seconds for a 30-second 1080p60 window, and 0.69 versus 0.64 seconds for a three-minute 144p window. Transfer-format and decoder-thread variations did not improve that result. Other GPUs, drivers, formats, and larger resolutions need their own comparisons. Ten-bit format preservation has argument tests but has not been validated on real ten-bit hardware input. Use --hwaccel none if the driver cannot decode or download the input format.

Parallel scoring with two or four processes also preserved the tested results but was about 12% slower than serial scoring. A structured timestamp-transport prototype changed interval frame counts and rejected negative timestamps, so it was excluded. Disabling diagnostic checksums gave no measurable benefit. These experiments are kept locally under ignored outputs/performance/; they are not application dependencies or enabled search modes.

A direct FFmpeg library prototype using PyAV 19.0.1 also preserved the tested pixels, timestamps, and full-archive candidates, but took approximately 19.55 seconds with six workers versus 15.41 seconds for the tuned pipe implementation. It was excluded, and PyAV is not an application dependency. A fully compiled decode-and-score loop would be a separate experiment; replacing only the transport introduced additional per-frame Python/native calls.

Livestream alignment

Compare a known frame against an archived livestream:

.venv/bin/python align_livestream.py --video /path/to/archive.mp4 --reference /path/to/frame.jpg --metadata /path/to/broadcast.json

Broadcast metadata must contain ISO timestamps with timezone offsets:

{
  "startTimestamp": "2000-01-01T12:00:00+00:00",
  "endTimestamp": "2000-01-01T13:00:00+00:00"
}

Clock reports use UTC by default; --timezone accepts an IANA timezone name. The report separates archive-relative timestamps from conditional clock times anchored to the metadata's start and end. Those calculations are not bounds on true airtime. Exact wall-clock airtime requires an independently verified archive-to-clock mapping. --verified-origin applies a supplied continuous archive origin without claiming its accuracy has been verified by this tool.

Use --expected for a known-timestamp regression check; it checks the result after searching and does not bias the search.

How matching works

1. Straighten a photographed screen using its corners, or accept a cropped reference.
2. Reduce frames to 96×54 pixels and compare the central visible area.
3. Blur and normalize grayscale and color channels to tolerate lighting differences and screen blur.
4. Score spatial correlation and retain the strongest candidate from every five-second interval.
5. Rank candidates and export the winner and comparison image.

COMPARISON_CROP in frame_finder/search.py defines the shared comparison region. Its default excludes upper and lower screen areas; adjust it for content that needs those regions. Similarity is not a probability. Verify the exported frame visually.

Checks cover timestamp preservation, complete frame coverage, candidate diversity, decoder log interleaving, bulk-read cleanup and partial batches, shard boundaries and candidate merging, score equivalence, bounded refinement in both commands, coverage reporting, and airtime uncertainty.

Final results include a strict visual check with sparse geometric evidence and independent regional validation. Both images are reduced to their shared available resolution (never upsampled to manufacture evidence); local contrast is normalized separately in each color channel before conversion to grayscale. Mutual SIFT correspondences propose a constrained rotation/scale/translation alignment, with identity alignment as a fallback. Geometry is fitted only from feature points; nonoverlapping image regions validate it without fitting to their correlations. Each supporting region requires at least 0.8 normalized correlation. Flat fields and one-dimensional gradients cannot provide evidence.

High certainty requires at least 20 agreeing regions, support covering at least 30% of the entire image, and agreement in at least 60% of informative regions. Areas with detail in either image count in the denominator, so occlusion and disagreement cannot inflate certainty by hiding difficult regions. Missing borders remain unconfirmed. Sparse feature agreement alone cannot pass the strict check. Medium and low ratings do not pass; similar-scoring alternatives cap the rating at medium. Reports retain the original feature metrics and add region bounds, support, comparison resolution, and alignment. The dashboard shows regional support. These are conservative heuristic ratings, not calibrated probabilities or proof of a unique source; severe occlusion and lost source detail can still prevent confirmation of a correct match.

Additional similarity checks validate edge direction, multiscale contrast/structure, and local pixel ordering using the same native-resolution alignment and informative-region denominator. Each check must support at least 20 regions, 30% of the image, and 60% of informative regions to pass. High certainty requires the original regional correlation check plus at least two of these three checks. Correlated checks are corroborating evidence rather than independent probabilities; their scores are never averaged into a confidence percentage. Reports include every check's region counts, area, agreement fraction, and mean similarity. Expand Similarity checks in the dashboard to inspect their outcomes.

Camera comparison now refines residual perspective distortion with a bounded homography and fits video blur to the photograph using alternating training tiles. The blur search is capped at three pixels of sigma at 256-pixel width; its choice uses training correlation, never confirmation thresholds. A small shared anti-alias filter and a single bounded positive contrast gain handle resolution and exposure differences. Edge and ordering checks operate at the fitted blur's spatial scale. Original, unblurred texture determines the evidence denominator, so smoothing cannot remove difficult regions. All existing per-region and overall acceptance thresholds remain unchanged, and each corroborating check additionally requires 60% agreement in unused verification tiles. Reports include the camera model, held-out agreement, and explicit failed requirements; expand Similarity checks in the dashboard for the details. Low-resolution detail lost in the photograph still cannot be recovered by these corrections, so a known matching timestamp can remain below High certainty.