Add frame dedup and Whisper auto-chunking
Frame extraction now runs a perceptual dedup pass (default on, --no-dedup to disable) that drops near-identical frames before the budget cap, so the budget goes to distinct content instead of held slides/static recordings. The Frames report line notes how many near-duplicates were dropped. Whisper transcription splits audio over the 25 MB upload cap into evenly sized chunks, transcribes each, shifts segment timestamps back into source time, and tolerates partial failures (only fails if every chunk fails) — length alone no longer fails transcription. token-burner is exempt from the long-video sparse-scan warning since it keeps every scene-change frame. README/SKILL.md updated. Adds test_dedup.py and test_whisper.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30,7 +30,7 @@ With Claude Video `/watch` you can paste a URL or a local path, ask a question,
|
|||||||
|
|
||||||
## Why this exists
|
## Why this exists
|
||||||
|
|
||||||
I built this because I'm constantly using video to keep up with content. If I see a YouTube video that's blowing up, I want to know how the creator structured the hook — what's on screen in the first 3 seconds, what they said, why it worked. That used to mean watching it myself with a notepad. Now I just paste the URL and ask.
|
I run a [YouTube channel](https://www.youtube.com/@bradbonanno) and a business, [Solaris Automation](https://www.solarisautomation.io/), so I'm constantly using video to keep up with content. If I see a YouTube video that's blowing up, I want to know how the creator structured the hook — what's on screen in the first 3 seconds, what they said, why it worked. That used to mean watching it myself with a notepad. Now I just paste the URL and ask.
|
||||||
|
|
||||||
The other half is summarization. Most YouTube videos don't deserve 20 minutes of my attention. I hand the URL to Claude, it pulls the transcript, and tells me what actually happened. If the visual matters, frames come along too. If it's a podcast or a talking head, transcript is enough.
|
The other half is summarization. Most YouTube videos don't deserve 20 minutes of my attention. I hand the URL to Claude, it pulls the transcript, and tells me what actually happened. If the visual matters, frames come along too. If it's a podcast or a talking head, transcript is enough.
|
||||||
|
|
||||||
@@ -44,12 +44,20 @@ Claude is great at reading and synthesizing — but until now, video was the one
|
|||||||
|
|
||||||
**Summarize a video.** `/watch https://youtu.be/<long-thing> summarize this` does the obvious thing — pulls the structure, the key moments, what was actually said and shown. Faster than watching at 2x.
|
**Summarize a video.** `/watch https://youtu.be/<long-thing> summarize this` does the obvious thing — pulls the structure, the key moments, what was actually said and shown. Faster than watching at 2x.
|
||||||
|
|
||||||
|
**Cut the hype out of an update video.** `/watch https://youtu.be/<launch-video> what's actually new — skip the hype` Strip a "game-changer" feature drop down to the few things that matter, so you get the substance without ten minutes of intro and overselling.
|
||||||
|
|
||||||
|
**Jump to the one answer in a long tutorial.** `/watch https://youtu.be/<2hr-tutorial> how do they set up the webhook?` Instead of scrubbing an hour of video for a 30-second answer — or matching a vague transcript and watching anyway — Claude finds the moment and tells you what's on screen there.
|
||||||
|
|
||||||
|
**Read a screen-share recording the transcript can't.** `/watch standup.mp4 what was on the shared screen when they decided?` When a meeting transcript collapses into "if you look here" and "make it look like this," the words stop carrying the meaning — the frames recover what was actually being shown.
|
||||||
|
|
||||||
|
**Turn a playlist into notes.** `/watch https://youtu.be/<video> summarize this to a note` Run it across a series and file a per-video summary, so a channel or course becomes a searchable set of notes instead of hours you have to sit through.
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
1. **You paste a video and a question.** URL (anything yt-dlp supports — YouTube, Loom, TikTok, X, Instagram, plus a few hundred more) or a local path (`.mp4`, `.mov`, `.mkv`, `.webm`).
|
1. **You paste a video and a question.** URL (anything yt-dlp supports — YouTube, Loom, TikTok, X, Instagram, plus a few hundred more) or a local path (`.mp4`, `.mov`, `.mkv`, `.webm`).
|
||||||
2. **`yt-dlp` checks captions first.** At `transcript` detail, captioned URLs return without downloading video. Otherwise, or when Whisper needs audio, it downloads only what the run needs.
|
2. **`yt-dlp` checks captions first.** At `transcript` detail, captioned URLs return without downloading video. Otherwise, or when Whisper needs audio, it downloads only what the run needs.
|
||||||
3. **`ffmpeg` extracts frames at the chosen detail.** `efficient` decodes keyframes only (near-instant); `balanced`/`token-burner` prefer scene-change frames and fall back to the duration-aware uniform sampler when they under-produce. JPEGs are 512px wide by default and clamped to 1998px tall for Claude Read compatibility.
|
3. **`ffmpeg` extracts frames at the chosen detail.** `efficient` decodes keyframes only (near-instant); `balanced`/`token-burner` prefer scene-change frames and fall back to the duration-aware uniform sampler when they under-produce. JPEGs are 512px wide by default and clamped to 1998px tall for Claude Read compatibility.
|
||||||
4. **The transcript comes from one of two places.** First try: `yt-dlp` pulls native captions (manual or auto-generated) from the source. Free, instant, accurate-ish. Fallback: extract a mono 16 kHz audio clip and ship it to Whisper — Groq's `whisper-large-v3` (preferred — cheaper and faster) or OpenAI's `whisper-1`.
|
4. **The transcript comes from one of two places.** First try: `yt-dlp` pulls native captions (manual or auto-generated) from the source. Free, instant, accurate-ish. Fallback: extract a mono 16 kHz 64 kbps mp3 audio clip (~480 kB/min) and ship it to Whisper — Groq's `whisper-large-v3` (preferred — cheaper and faster) or OpenAI's `whisper-1`.
|
||||||
5. **Frames + transcript are handed to Claude.** The script prints frame paths with `t=MM:SS` markers and the transcript with timestamps. Claude `Read`s each frame in parallel — JPEGs render directly as images in its context.
|
5. **Frames + transcript are handed to Claude.** The script prints frame paths with `t=MM:SS` markers and the transcript with timestamps. Claude `Read`s each frame in parallel — JPEGs render directly as images in its context.
|
||||||
6. **Claude answers grounded in what's actually on screen and in the audio.** Not "based on the description" or "according to the title." It saw the frames. It heard the transcript. It answers the way someone who watched the video would.
|
6. **Claude answers grounded in what's actually on screen and in the audio.** Not "based on the description" or "according to the title." It saw the frames. It heard the transcript. It answers the way someone who watched the video would.
|
||||||
7. **Cleanup.** The script prints a working directory at the end. If you're not asking follow-ups, Claude removes it.
|
7. **Cleanup.** The script prints a working directory at the end. If you're not asking follow-ups, Claude removes it.
|
||||||
@@ -64,10 +72,28 @@ Token cost is dominated by frames. Every frame is an image; image tokens add up
|
|||||||
| 30 s - 1 min | ~40 frames | Still dense |
|
| 30 s - 1 min | ~40 frames | Still dense |
|
||||||
| 1 - 3 min | ~60 frames | Comfortable |
|
| 1 - 3 min | ~60 frames | Comfortable |
|
||||||
| 3 - 10 min | ~80 frames | Sparse but workable |
|
| 3 - 10 min | ~80 frames | Sparse but workable |
|
||||||
| > 10 min | 100 frames | "Sparse scan" warning — re-run focused |
|
| > 10 min | 100 frames (capped modes) | "Sparse scan" warning — re-run focused, or `--detail token-burner` for full uncapped coverage |
|
||||||
|
|
||||||
When the user names a moment ("around 2:30", "the last 30 seconds", "from 0:45 to 1:00"), pass `--start` / `--end`. Focused mode gets denser per-second budgets, capped at 2 fps. Far more useful than a sparse pass over the whole thing.
|
When the user names a moment ("around 2:30", "the last 30 seconds", "from 0:45 to 1:00"), pass `--start` / `--end`. Focused mode gets denser per-second budgets, capped at 2 fps. Far more useful than a sparse pass over the whole thing.
|
||||||
|
|
||||||
|
## Frame deduplication — why you don't pay for held slides
|
||||||
|
|
||||||
|
Sampling spreads frames evenly across time, but time isn't where the information is. A 10-minute screen recording that sits on one slide for 90 seconds gives you a dozen *pixel-for-pixel identical* frames of that slide — each one a separate image, each one billed in tokens. Even-sampling doesn't know they're the same; it just hands them all over.
|
||||||
|
|
||||||
|
So before the frames are handed to Claude, a dedup pass drops the ones that didn't change. It runs by default on every frame mode (`--no-dedup` turns it off). Here's the logic, end to end:
|
||||||
|
|
||||||
|
1. **Decode each frame to a tiny fingerprint.** One `ffmpeg` call scales every extracted JPEG down to a 16×16 grayscale thumbnail — 256 brightness values per frame. ffmpeg does the pixel decoding; everything after is pure-stdlib Python, no image libraries.
|
||||||
|
2. **Measure how much changed.** For each frame, compute the **mean absolute difference** against the *last frame that was kept* — the average per-pixel brightness change on a 0–255 scale. This is a literal "how different are these two frames" number, not a heuristic.
|
||||||
|
3. **Drop it if nothing meaningfully changed.** If that difference is at or below the threshold (`2.0` — i.e. the picture moved less than ~1% on average), the frame is a near-duplicate: delete it. Otherwise keep it, and it becomes the new reference to compare the next frame against.
|
||||||
|
4. **Sample the survivors down to the budget.** Only after duplicates are gone does the frame-budget cap apply — so the budget is spent on *distinct* frames, not diluted by repeats.
|
||||||
|
|
||||||
|
Two design choices worth calling out:
|
||||||
|
|
||||||
|
- **Compare against the last *kept* frame, not the immediately previous one.** A slow fade across ten frames never trips a frame-to-frame threshold (each step is tiny) but is obviously different end-to-end. Comparing against the last *kept* frame catches that gradual drift and still collapses a dead-static hold to a single frame.
|
||||||
|
- **It's deliberately conservative, and it measures absolute brightness — not structure.** The threshold is low on purpose: a code diff with one changed line, a terminal scrolling a single row, or a slide that just gained a bullet all clear it and survive. Measuring absolute brightness (rather than a perceptual/structure hash) is what lets it tell two *flat* frames apart — a solid blue section slide and a solid green one differ in luma, so they're correctly kept, where a structure-only hash would see "no internal detail" in both and wrongly merge them.
|
||||||
|
|
||||||
|
The result is reported, not silent: the **Frames** line shows e.g. `6 selected from 14 candidates (… 8 near-duplicates dropped …)` so you can always see what was collapsed. On a clip that holds a slide and then cuts to motion, that's the difference between paying for 14 frames and paying for 6 — with the transition and every distinct moment preserved. On footage that's genuinely always moving, nothing gets dropped and you pay exactly what you would have anyway.
|
||||||
|
|
||||||
## Detail modes — measured
|
## Detail modes — measured
|
||||||
|
|
||||||
The `--detail` dial trades speed and token cost for visual fidelity. Numbers below are from a real run against a **49:08 (2948 s)** YouTube video (1280×720 source, English auto-captions, 1394 caption segments) — a long, mostly-static screen recording, which is the case that stresses the caps hardest. Frame-extraction times are measured against a pre-downloaded local copy so they reflect the *mode's* CPU cost, not network speed. The one-time video download for this clip was **~37 s** / 76 MB (shared by `efficient` / `balanced` / `token-burner`).
|
The `--detail` dial trades speed and token cost for visual fidelity. Numbers below are from a real run against a **49:08 (2948 s)** YouTube video (1280×720 source, English auto-captions, 1394 caption segments) — a long, mostly-static screen recording, which is the case that stresses the caps hardest. Frame-extraction times are measured against a pre-downloaded local copy so they reflect the *mode's* CPU cost, not network speed. The one-time video download for this clip was **~37 s** / 76 MB (shared by `efficient` / `balanced` / `token-burner`).
|
||||||
@@ -196,13 +222,13 @@ Other knobs (passed to `scripts/watch.py`):
|
|||||||
- `--fps F` — override the auto-fps calculation (still capped at 2 fps).
|
- `--fps F` — override the auto-fps calculation (still capped at 2 fps).
|
||||||
- `--whisper groq|openai` — force a specific Whisper backend.
|
- `--whisper groq|openai` — force a specific Whisper backend.
|
||||||
- `--no-whisper` — disable transcription entirely; frames only.
|
- `--no-whisper` — disable transcription entirely; frames only.
|
||||||
|
- `--no-dedup` — keep near-duplicate frames. By default a frame-delta pass drops frames that are visually near-identical to the one before them (held slides, static screen recordings, paused video), so the frame budget is spent on distinct content; this flag turns that off.
|
||||||
- `--out-dir DIR` — keep working files somewhere specific (default: auto-generated tmp dir).
|
- `--out-dir DIR` — keep working files somewhere specific (default: auto-generated tmp dir).
|
||||||
|
|
||||||
## Limits
|
## Limits
|
||||||
|
|
||||||
- **Best accuracy: under 10 minutes.** Past that the script prints a "sparse scan" warning — re-run focused on the part you actually care about with `--start`/`--end`.
|
- **Long-video accuracy depends on the detail mode.** On the capped modes (`efficient`, default `balanced`) coverage thins out past ~10 minutes — the frame cap spreads across the whole clip, so the script prints a "sparse scan" warning and you're better off re-running focused with `--start`/`--end`. `token-burner` lifts the cap and keeps *every* scene-change frame across the full video, so it stays complete on longer clips at the cost of more image tokens. The 10-minute mark is guidance for the capped modes, not a hard ceiling.
|
||||||
- **Detail is one dial.** Defaults are balanced: scene-aware frames, 2 fps max, 100-frame cap. Use `--detail efficient` for a fast 50-frame keyframe pass, or `--detail token-burner` for uncapped scene candidates. Set `WATCH_DETAIL` in `~/.config/watch/.env` to change the default.
|
- **Detail is one dial.** Defaults are balanced: scene-aware frames, 2 fps max, 100-frame cap. Use `--detail efficient` for a fast 50-frame keyframe pass, or `--detail token-burner` for uncapped scene candidates. Set `WATCH_DETAIL` in `~/.config/watch/.env` to change the default.
|
||||||
- **Whisper upload limit: 25 MB.** At mono 16 kHz that's about 50 minutes of audio. Longer videos need either captions or `--start`/`--end` to a smaller window.
|
|
||||||
|
|
||||||
## Structure
|
## Structure
|
||||||
|
|
||||||
@@ -248,16 +274,18 @@ MIT license.
|
|||||||
|
|
||||||
Built on `yt-dlp`, `ffmpeg`, and Claude's multimodal `Read` tool. Whisper transcription via [Groq](https://groq.com) or [OpenAI](https://openai.com).
|
Built on `yt-dlp`, `ffmpeg`, and Claude's multimodal `Read` tool. Whisper transcription via [Groq](https://groq.com) or [OpenAI](https://openai.com).
|
||||||
|
|
||||||
|
Built by Brad Bonanno — I make content about building with AI on [YouTube (@bradbonanno)](https://www.youtube.com/@bradbonanno), and build AI operating systems for businesses at [Solaris Automation](https://www.solarisautomation.io/). If `/watch` saves you from scrubbing through a video, come say hi on the channel.
|
||||||
|
|
||||||
## Star History
|
## Star History
|
||||||
|
|
||||||
<a href="https://star-history.com/#bradautomates/claude-video&Date">
|
<a href="https://www.star-history.com/?repos=bradautomates%2Fclaude-video&type=date&legend=top-left">
|
||||||
<picture>
|
<picture>
|
||||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=bradautomates/claude-video&type=Date&theme=dark" />
|
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=bradautomates/claude-video&type=date&theme=dark&legend=top-left" />
|
||||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=bradautomates/claude-video&type=Date" />
|
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=bradautomates/claude-video&type=date&legend=top-left" />
|
||||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=bradautomates/claude-video&type=Date" />
|
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=bradautomates/claude-video&type=date&legend=top-left" />
|
||||||
</picture>
|
</picture>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
[github.com/bradautomates/claude-video](https://github.com/bradautomates/claude-video) · [LICENSE](LICENSE)
|
[github.com/bradautomates/claude-video](https://github.com/bradautomates/claude-video) · [@bradbonanno](https://www.youtube.com/@bradbonanno) · [Solaris Automation](https://www.solarisautomation.io/) · [LICENSE](LICENSE)
|
||||||
|
|||||||
@@ -149,6 +149,7 @@ Optional flags:
|
|||||||
- `--out-dir DIR` — keep working files somewhere specific (default: an auto-generated tmp dir)
|
- `--out-dir DIR` — keep working files somewhere specific (default: an auto-generated tmp dir)
|
||||||
- `--whisper groq|openai` — force a specific Whisper backend (default: prefer Groq if both keys exist)
|
- `--whisper groq|openai` — force a specific Whisper backend (default: prefer Groq if both keys exist)
|
||||||
- `--no-whisper` — disable the Whisper fallback entirely (frames-only if no captions)
|
- `--no-whisper` — disable the Whisper fallback entirely (frames-only if no captions)
|
||||||
|
- `--no-dedup` — keep near-duplicate frames. By default a frame-delta pass drops frames that are visually near-identical to the previous kept one (held slides, static screen recordings, paused video) so the frame budget goes to distinct content; the report's **Frames** line notes how many were dropped. Pass this only if the user needs every sampled frame (e.g. judging subtle frame-to-frame motion).
|
||||||
|
|
||||||
### Focusing on a section (higher frame rate)
|
### Focusing on a section (higher frame rate)
|
||||||
|
|
||||||
@@ -234,7 +235,7 @@ Both keys live in `~/.config/watch/.env`. The script prefers Groq when both are
|
|||||||
- **No transcript available** → captions missing AND (no Whisper key OR Whisper API failed). Script prints a hint pointing to setup. Proceed frames-only and tell the user.
|
- **No transcript available** → captions missing AND (no Whisper key OR Whisper API failed). Script prints a hint pointing to setup. Proceed frames-only and tell the user.
|
||||||
- **Long video warning printed** → acknowledge it in your answer. Offer to re-run focused on a specific section via `--start`/`--end` rather than a sparse full-video scan.
|
- **Long video warning printed** → acknowledge it in your answer. Offer to re-run focused on a specific section via `--start`/`--end` rather than a sparse full-video scan.
|
||||||
- **Download fails** → yt-dlp's error goes to stderr. If it's a login-required or region-locked video, tell the user plainly; do not keep retrying.
|
- **Download fails** → yt-dlp's error goes to stderr. If it's a login-required or region-locked video, tell the user plainly; do not keep retrying.
|
||||||
- **Whisper request fails** → the error is printed to stderr (likely: invalid key, rate limit, or 25 MB upload limit on a very long video). The report will say "none available" for transcript. You can retry with `--whisper openai` if Groq failed (or vice versa).
|
- **Whisper request fails** → the error is printed to stderr (likely: invalid key or rate limit). Audio over the API's 25 MB upload cap is split into chunks and transcribed automatically, so length alone won't fail it; if some chunks fail the transcript is partial and the dropped chunks are noted on stderr. The report will say "none available" only if every chunk fails. You can retry with `--whisper openai` if Groq failed (or vice versa).
|
||||||
|
|
||||||
## Token efficiency
|
## Token efficiency
|
||||||
|
|
||||||
|
|||||||
+145
-17
@@ -28,6 +28,14 @@ SCENE_MIN_FRAMES = 8
|
|||||||
# (very short or oddly encoded), so the cheap tier falls back to uniform.
|
# (very short or oddly encoded), so the cheap tier falls back to uniform.
|
||||||
KEYFRAME_MIN = 4
|
KEYFRAME_MIN = 4
|
||||||
MAX_READ_DIMENSION = 1998
|
MAX_READ_DIMENSION = 1998
|
||||||
|
# Frame-delta dedup: downscale each frame to a DEDUP_THUMB x DEDUP_THUMB
|
||||||
|
# grayscale thumbnail and treat two frames as near-identical when their mean
|
||||||
|
# per-pixel difference (0-255) is at or below DEDUP_THRESHOLD. Conservative on
|
||||||
|
# purpose: only collapses frames that are visually the same shot, so a code diff
|
||||||
|
# / scrolling terminal / slide-gaining-a-bullet survives. Unlike a within-frame
|
||||||
|
# perceptual hash, this distinguishes flat frames (solid slides, fades) by luma.
|
||||||
|
DEDUP_THUMB = 16
|
||||||
|
DEDUP_THRESHOLD = 2.0
|
||||||
SHOWINFO_TS_RE = re.compile(r"pts_time:([0-9.]+)")
|
SHOWINFO_TS_RE = re.compile(r"pts_time:([0-9.]+)")
|
||||||
|
|
||||||
|
|
||||||
@@ -404,6 +412,101 @@ def _even_sample(candidates: list[dict], n: int) -> list[dict]:
|
|||||||
return selected
|
return selected
|
||||||
|
|
||||||
|
|
||||||
|
def _frame_delta(a: bytes, b: bytes) -> float:
|
||||||
|
"""Mean absolute per-pixel difference (0-255) between two grayscale
|
||||||
|
thumbnails. Mismatched lengths are treated as maximally different so a
|
||||||
|
decode hiccup never collapses distinct frames."""
|
||||||
|
if not a or len(a) != len(b):
|
||||||
|
return float("inf")
|
||||||
|
return sum(abs(x - y) for x, y in zip(a, b)) / len(a)
|
||||||
|
|
||||||
|
|
||||||
|
def _thumb_frames(paths: list[Path]) -> list[bytes]:
|
||||||
|
"""Decode every frame in ``paths`` to a small grayscale thumbnail via one
|
||||||
|
ffmpeg pass over the JPEG sequence.
|
||||||
|
|
||||||
|
ffmpeg does the pixel decode (keeps us pure-stdlib); we slice the raw
|
||||||
|
grayscale stream into one ``DEDUP_THUMB``-square thumbnail per frame.
|
||||||
|
Fail-open: any ffmpeg error, an unrecognized name, or a byte-count mismatch
|
||||||
|
returns ``[]`` so the caller skips dedup rather than breaking extraction.
|
||||||
|
"""
|
||||||
|
if not paths:
|
||||||
|
return []
|
||||||
|
paths = [Path(p) for p in paths]
|
||||||
|
m = re.match(r"(.*?)(\d+)(\.[A-Za-z0-9]+)$", paths[0].name)
|
||||||
|
if m is None:
|
||||||
|
return []
|
||||||
|
prefix, digits, ext = m.group(1), m.group(2), m.group(3)
|
||||||
|
pattern = str(paths[0].parent / f"{prefix}%0{len(digits)}d{ext}")
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"ffmpeg",
|
||||||
|
"-hide_banner",
|
||||||
|
"-loglevel", "error",
|
||||||
|
"-start_number", str(int(digits)),
|
||||||
|
"-i", pattern,
|
||||||
|
"-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB},format=gray",
|
||||||
|
"-f", "rawvideo",
|
||||||
|
"-",
|
||||||
|
]
|
||||||
|
result = subprocess.run(cmd, capture_output=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
chunk = DEDUP_THUMB * DEDUP_THUMB
|
||||||
|
data = result.stdout
|
||||||
|
if len(data) != chunk * len(paths):
|
||||||
|
return []
|
||||||
|
return [data[i * chunk:(i + 1) * chunk] for i in range(len(paths))]
|
||||||
|
|
||||||
|
|
||||||
|
def dedupe_perceptual(
|
||||||
|
candidates: list[dict], threshold: float = DEDUP_THRESHOLD
|
||||||
|
) -> tuple[list[dict], int]:
|
||||||
|
"""Drop near-identical frames from a chronological candidate list.
|
||||||
|
|
||||||
|
Thumbnails the extracted JPEGs and greedily removes frames whose mean
|
||||||
|
per-pixel difference from the last kept one is within ``threshold``. Returns
|
||||||
|
``(survivors, dropped_count)``; a no-op (unchanged list) when thumbnails are
|
||||||
|
unavailable or there are fewer than two candidates.
|
||||||
|
"""
|
||||||
|
if len(candidates) <= 1:
|
||||||
|
return candidates, 0
|
||||||
|
thumbs = _thumb_frames([Path(c["path"]) for c in candidates])
|
||||||
|
return _dedupe_by_deltas(candidates, thumbs, threshold)
|
||||||
|
|
||||||
|
|
||||||
|
def _dedupe_by_deltas(
|
||||||
|
candidates: list[dict], thumbs: list[bytes], threshold: float = DEDUP_THRESHOLD
|
||||||
|
) -> tuple[list[dict], int]:
|
||||||
|
"""Greedily drop frames within ``threshold`` mean per-pixel difference of the
|
||||||
|
last *kept* frame. Deletes dropped JPEGs and reindexes survivors 0..n-1 (same
|
||||||
|
cleanup contract as :func:`_even_sample`). Fail-open: if ``thumbs`` does not
|
||||||
|
line up 1:1 with ``candidates``, return them unchanged.
|
||||||
|
"""
|
||||||
|
if len(thumbs) != len(candidates) or len(candidates) <= 1:
|
||||||
|
return candidates, 0
|
||||||
|
|
||||||
|
kept = [candidates[0]]
|
||||||
|
last = thumbs[0]
|
||||||
|
dropped: list[dict] = []
|
||||||
|
for cand, thumb in zip(candidates[1:], thumbs[1:]):
|
||||||
|
if _frame_delta(thumb, last) <= threshold:
|
||||||
|
dropped.append(cand)
|
||||||
|
else:
|
||||||
|
kept.append(cand)
|
||||||
|
last = thumb
|
||||||
|
|
||||||
|
for cand in dropped:
|
||||||
|
try:
|
||||||
|
Path(cand["path"]).unlink()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
for i, frame in enumerate(kept):
|
||||||
|
frame["index"] = i
|
||||||
|
return kept, len(dropped)
|
||||||
|
|
||||||
|
|
||||||
def extract_scene_or_uniform(
|
def extract_scene_or_uniform(
|
||||||
video_path: str,
|
video_path: str,
|
||||||
out_dir: Path,
|
out_dir: Path,
|
||||||
@@ -413,17 +516,19 @@ def extract_scene_or_uniform(
|
|||||||
max_frames: int | None = 100,
|
max_frames: int | None = 100,
|
||||||
start_seconds: float | None = None,
|
start_seconds: float | None = None,
|
||||||
end_seconds: float | None = None,
|
end_seconds: float | None = None,
|
||||||
|
dedup: bool = True,
|
||||||
) -> tuple[list[dict], dict]:
|
) -> tuple[list[dict], dict]:
|
||||||
"""Prefer scene selection, falling back to uniform only when the video is
|
"""Prefer scene selection, falling back to uniform only when the video is
|
||||||
effectively static (fewer than ``SCENE_MIN_FRAMES`` detected shots).
|
effectively static (fewer than ``SCENE_MIN_FRAMES`` detected shots).
|
||||||
|
|
||||||
Scene cuts are detected across the *whole* range (uncapped) and then
|
Scene cuts are detected across the *whole* range (uncapped), near-identical
|
||||||
even-sampled down to ``max_frames`` via :func:`_even_sample`, exactly like
|
frames are dropped (:func:`dedupe_perceptual`, unless ``dedup`` is False),
|
||||||
the keyframe engine. This costs a full decode, but it guarantees coverage
|
and the survivors are even-sampled down to ``max_frames`` via
|
||||||
spans the entire clip — capping detection with ``-frames:v`` instead would
|
:func:`_even_sample`, exactly like the keyframe engine. This costs a full
|
||||||
keep only the first ``max_frames`` cuts and drop the tail of long videos
|
decode, but it guarantees coverage spans the entire clip — capping detection
|
||||||
(and could even fall below ``SCENE_MIN_FRAMES`` and misfire the uniform
|
with ``-frames:v`` instead would keep only the first ``max_frames`` cuts and
|
||||||
fallback on a cut-heavy clip).
|
drop the tail of long videos (and could even fall below ``SCENE_MIN_FRAMES``
|
||||||
|
and misfire the uniform fallback on a cut-heavy clip).
|
||||||
"""
|
"""
|
||||||
scene_frames = extract_scene_candidates(
|
scene_frames = extract_scene_candidates(
|
||||||
video_path,
|
video_path,
|
||||||
@@ -435,11 +540,13 @@ def extract_scene_or_uniform(
|
|||||||
)
|
)
|
||||||
scene_count = len(scene_frames)
|
scene_count = len(scene_frames)
|
||||||
if scene_count >= SCENE_MIN_FRAMES:
|
if scene_count >= SCENE_MIN_FRAMES:
|
||||||
cap = scene_count if max_frames is None else max_frames
|
deduped, n_dropped = dedupe_perceptual(scene_frames) if dedup else (scene_frames, 0)
|
||||||
selected = _even_sample(scene_frames, cap)
|
cap = len(deduped) if max_frames is None else max_frames
|
||||||
|
selected = _even_sample(deduped, cap)
|
||||||
return selected, {
|
return selected, {
|
||||||
"engine": "scene",
|
"engine": "scene",
|
||||||
"candidate_count": scene_count,
|
"candidate_count": scene_count,
|
||||||
|
"deduped_count": n_dropped,
|
||||||
"selected_count": len(selected),
|
"selected_count": len(selected),
|
||||||
"fallback": False,
|
"fallback": False,
|
||||||
}
|
}
|
||||||
@@ -454,9 +561,13 @@ def extract_scene_or_uniform(
|
|||||||
start_seconds=start_seconds,
|
start_seconds=start_seconds,
|
||||||
end_seconds=end_seconds,
|
end_seconds=end_seconds,
|
||||||
)
|
)
|
||||||
|
n_dropped = 0
|
||||||
|
if dedup:
|
||||||
|
frames, n_dropped = dedupe_perceptual(frames)
|
||||||
return frames, {
|
return frames, {
|
||||||
"engine": "uniform",
|
"engine": "uniform",
|
||||||
"candidate_count": scene_count,
|
"candidate_count": scene_count,
|
||||||
|
"deduped_count": n_dropped,
|
||||||
"selected_count": len(frames),
|
"selected_count": len(frames),
|
||||||
"fallback": True,
|
"fallback": True,
|
||||||
}
|
}
|
||||||
@@ -469,13 +580,15 @@ def extract_keyframes(
|
|||||||
max_frames: int | None = 50,
|
max_frames: int | None = 50,
|
||||||
start_seconds: float | None = None,
|
start_seconds: float | None = None,
|
||||||
end_seconds: float | None = None,
|
end_seconds: float | None = None,
|
||||||
|
dedup: bool = True,
|
||||||
) -> tuple[list[dict], dict]:
|
) -> tuple[list[dict], dict]:
|
||||||
"""Decode only keyframes (I-frames) — the cheap, near-instant tier.
|
"""Decode only keyframes (I-frames) — the cheap, near-instant tier.
|
||||||
|
|
||||||
``-skip_frame nokey`` makes ffmpeg reconstruct only keyframes, skipping all
|
``-skip_frame nokey`` makes ffmpeg reconstruct only keyframes, skipping all
|
||||||
P/B frames. Encoders emit keyframes at scene cuts, so these already
|
P/B frames. Encoders emit keyframes at scene cuts, so these already
|
||||||
approximate "distinct moments". Over-cap → even-sample first→last; too few
|
approximate "distinct moments". Near-identical frames are dropped
|
||||||
keyframes → uniform fallback.
|
(:func:`dedupe_perceptual`, unless ``dedup`` is False); over-cap →
|
||||||
|
even-sample first→last; too few keyframes → uniform fallback.
|
||||||
"""
|
"""
|
||||||
if shutil.which("ffmpeg") is None:
|
if shutil.which("ffmpeg") is None:
|
||||||
raise SystemExit("ffmpeg is not installed. Install with: brew install ffmpeg")
|
raise SystemExit("ffmpeg is not installed. Install with: brew install ffmpeg")
|
||||||
@@ -543,21 +656,27 @@ def extract_keyframes(
|
|||||||
start_seconds=start_seconds,
|
start_seconds=start_seconds,
|
||||||
end_seconds=end_seconds,
|
end_seconds=end_seconds,
|
||||||
)
|
)
|
||||||
|
n_dropped = 0
|
||||||
|
if dedup:
|
||||||
|
frames_out, n_dropped = dedupe_perceptual(frames_out)
|
||||||
return frames_out, {
|
return frames_out, {
|
||||||
"engine": "uniform",
|
"engine": "uniform",
|
||||||
"candidate_count": len(candidates),
|
"candidate_count": len(candidates),
|
||||||
|
"deduped_count": n_dropped,
|
||||||
"selected_count": len(frames_out),
|
"selected_count": len(frames_out),
|
||||||
"fallback": True,
|
"fallback": True,
|
||||||
}
|
}
|
||||||
|
|
||||||
# Detect-all then even-sample down to the cap (first + last always kept).
|
# Detect-all, drop near-duplicates, then even-sample down to the cap (first +
|
||||||
# ``max_frames is None`` (uncapped) keeps every keyframe.
|
# last always kept). ``max_frames is None`` (uncapped) keeps every keyframe.
|
||||||
candidate_count = len(candidates)
|
candidate_count = len(candidates)
|
||||||
cap = candidate_count if max_frames is None else max_frames
|
deduped, n_dropped = dedupe_perceptual(candidates) if dedup else (candidates, 0)
|
||||||
selected = _even_sample(candidates, cap)
|
cap = len(deduped) if max_frames is None else max_frames
|
||||||
|
selected = _even_sample(deduped, cap)
|
||||||
return selected, {
|
return selected, {
|
||||||
"engine": "keyframe",
|
"engine": "keyframe",
|
||||||
"candidate_count": candidate_count,
|
"candidate_count": candidate_count,
|
||||||
|
"deduped_count": n_dropped,
|
||||||
"selected_count": len(selected),
|
"selected_count": len(selected),
|
||||||
"fallback": False,
|
"fallback": False,
|
||||||
}
|
}
|
||||||
@@ -567,7 +686,7 @@ if __name__ == "__main__":
|
|||||||
if len(sys.argv) < 3:
|
if len(sys.argv) < 3:
|
||||||
print(
|
print(
|
||||||
"usage: frames.py <video-path> <out-dir> [--fps F] [--resolution W] "
|
"usage: frames.py <video-path> <out-dir> [--fps F] [--resolution W] "
|
||||||
"[--max-frames N] [--start T] [--end T]",
|
"[--max-frames N] [--start T] [--end T] [--no-dedup]",
|
||||||
file=sys.stderr,
|
file=sys.stderr,
|
||||||
)
|
)
|
||||||
raise SystemExit(2)
|
raise SystemExit(2)
|
||||||
@@ -581,6 +700,7 @@ if __name__ == "__main__":
|
|||||||
max_frames = 100
|
max_frames = 100
|
||||||
start_arg = None
|
start_arg = None
|
||||||
end_arg = None
|
end_arg = None
|
||||||
|
dedup = True
|
||||||
i = 0
|
i = 0
|
||||||
while i < len(args):
|
while i < len(args):
|
||||||
if args[i] == "--fps":
|
if args[i] == "--fps":
|
||||||
@@ -593,6 +713,8 @@ if __name__ == "__main__":
|
|||||||
start_arg = args[i + 1]; i += 2
|
start_arg = args[i + 1]; i += 2
|
||||||
elif args[i] == "--end":
|
elif args[i] == "--end":
|
||||||
end_arg = args[i + 1]; i += 2
|
end_arg = args[i + 1]; i += 2
|
||||||
|
elif args[i] == "--no-dedup":
|
||||||
|
dedup = False; i += 1
|
||||||
else:
|
else:
|
||||||
i += 1
|
i += 1
|
||||||
|
|
||||||
@@ -622,7 +744,13 @@ if __name__ == "__main__":
|
|||||||
start_seconds=start_sec,
|
start_seconds=start_sec,
|
||||||
end_seconds=end_sec,
|
end_seconds=end_sec,
|
||||||
)
|
)
|
||||||
|
deduped_count = 0
|
||||||
|
if dedup:
|
||||||
|
frames, deduped_count = dedupe_perceptual(frames)
|
||||||
print(json.dumps(
|
print(json.dumps(
|
||||||
{"meta": meta, "fps": fps, "target": target, "focused": focused, "frames": frames},
|
{
|
||||||
|
"meta": meta, "fps": fps, "target": target, "focused": focused,
|
||||||
|
"deduped_count": deduped_count, "frames": frames,
|
||||||
|
},
|
||||||
indent=2,
|
indent=2,
|
||||||
))
|
))
|
||||||
|
|||||||
@@ -60,6 +60,12 @@ def main() -> int:
|
|||||||
default=None,
|
default=None,
|
||||||
help="Force a specific Whisper backend. Default: prefer Groq, fall back to OpenAI.",
|
help="Force a specific Whisper backend. Default: prefer Groq, fall back to OpenAI.",
|
||||||
)
|
)
|
||||||
|
ap.add_argument(
|
||||||
|
"--no-dedup",
|
||||||
|
action="store_true",
|
||||||
|
help="Disable near-duplicate frame removal. Keeps visually identical "
|
||||||
|
"frames (static screen recordings, held slides) instead of collapsing them.",
|
||||||
|
)
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
config = get_config()
|
config = get_config()
|
||||||
@@ -203,6 +209,7 @@ def main() -> int:
|
|||||||
max_frames=detail_budget,
|
max_frames=detail_budget,
|
||||||
start_seconds=start_sec,
|
start_seconds=start_sec,
|
||||||
end_seconds=end_sec,
|
end_seconds=end_sec,
|
||||||
|
dedup=not args.no_dedup,
|
||||||
)
|
)
|
||||||
else: # balanced, token-burner
|
else: # balanced, token-burner
|
||||||
frames, frame_meta = extract_scene_or_uniform(
|
frames, frame_meta = extract_scene_or_uniform(
|
||||||
@@ -214,6 +221,7 @@ def main() -> int:
|
|||||||
max_frames=detail_budget,
|
max_frames=detail_budget,
|
||||||
start_seconds=start_sec,
|
start_seconds=start_sec,
|
||||||
end_seconds=end_sec,
|
end_seconds=end_sec,
|
||||||
|
dedup=not args.no_dedup,
|
||||||
)
|
)
|
||||||
|
|
||||||
if cue_frames:
|
if cue_frames:
|
||||||
@@ -282,9 +290,11 @@ def main() -> int:
|
|||||||
cap_label = "unlimited" if detail_budget is None else str(detail_budget)
|
cap_label = "unlimited" if detail_budget is None else str(detail_budget)
|
||||||
engine = frame_meta.get("engine", "scene")
|
engine = frame_meta.get("engine", "scene")
|
||||||
fallback = " with uniform fallback" if frame_meta.get("fallback") else ""
|
fallback = " with uniform fallback" if frame_meta.get("fallback") else ""
|
||||||
|
deduped = frame_meta.get("deduped_count", 0)
|
||||||
|
dedup_note = f", {deduped} near-duplicate{'s' if deduped != 1 else ''} dropped" if deduped else ""
|
||||||
print(
|
print(
|
||||||
f"- **Frames:** {detail_count} selected from {frame_meta.get('candidate_count', detail_count)} "
|
f"- **Frames:** {detail_count} selected from {frame_meta.get('candidate_count', detail_count)} "
|
||||||
f"candidates ({engine}{fallback}, {range_mode} range, budget {target}, cap {cap_label})"
|
f"candidates ({engine}{fallback}{dedup_note}, {range_mode} range, budget {target}, cap {cap_label})"
|
||||||
)
|
)
|
||||||
elif not cue_frames:
|
elif not cue_frames:
|
||||||
print("- **Frames:** skipped (transcript detail)")
|
print("- **Frames:** skipped (transcript detail)")
|
||||||
@@ -313,13 +323,14 @@ def main() -> int:
|
|||||||
"This may use a large number of image tokens."
|
"This may use a large number of image tokens."
|
||||||
)
|
)
|
||||||
|
|
||||||
if not focused and full_duration > 600 and detail != "transcript":
|
if not focused and full_duration > 600 and detail not in ("transcript", "token-burner"):
|
||||||
mins = int(full_duration // 60)
|
mins = int(full_duration // 60)
|
||||||
print()
|
print()
|
||||||
print(
|
print(
|
||||||
f"> **Warning:** This is a {mins}-minute video. Frame coverage is sparse at this length — "
|
f"> **Warning:** This is a {mins}-minute video. Frame coverage is sparse at this length "
|
||||||
"accuracy degrades noticeably on anything over 10 minutes. For better results, "
|
f"under `{detail}` detail — its cap spreads thin across the full clip. For better results, "
|
||||||
"re-run with `--start HH:MM:SS --end HH:MM:SS` to zoom into a specific section."
|
"re-run with `--start HH:MM:SS --end HH:MM:SS` to zoom into a section, or use "
|
||||||
|
"`--detail token-burner` to keep every scene-change frame across the whole video."
|
||||||
)
|
)
|
||||||
|
|
||||||
print()
|
print()
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
|
import math
|
||||||
import mimetypes
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
@@ -31,6 +32,35 @@ GROQ_MODEL = "whisper-large-v3"
|
|||||||
OPENAI_ENDPOINT = "https://api.openai.com/v1/audio/transcriptions"
|
OPENAI_ENDPOINT = "https://api.openai.com/v1/audio/transcriptions"
|
||||||
OPENAI_MODEL = "whisper-1"
|
OPENAI_MODEL = "whisper-1"
|
||||||
|
|
||||||
|
# Both Groq's free tier and OpenAI whisper-1 cap uploads at 25 MB. We target a
|
||||||
|
# margin under that so multipart framing overhead never pushes a chunk over.
|
||||||
|
MAX_UPLOAD_BYTES = 24 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
def plan_chunks(
|
||||||
|
total_seconds: float,
|
||||||
|
total_bytes: int,
|
||||||
|
max_bytes: int = MAX_UPLOAD_BYTES,
|
||||||
|
) -> list[tuple[float, float]]:
|
||||||
|
"""Split a duration into contiguous (offset, duration) chunks under max_bytes.
|
||||||
|
|
||||||
|
Size scales linearly with duration (constant-bitrate mono mp3), so an even
|
||||||
|
time split yields evenly-sized chunks. Returns a single full-length chunk
|
||||||
|
when the audio already fits.
|
||||||
|
"""
|
||||||
|
if total_bytes <= max_bytes or total_seconds <= 0:
|
||||||
|
return [(0.0, total_seconds)]
|
||||||
|
|
||||||
|
n = math.ceil(total_bytes / max_bytes)
|
||||||
|
chunk = total_seconds / n
|
||||||
|
plan: list[tuple[float, float]] = []
|
||||||
|
for i in range(n):
|
||||||
|
offset = i * chunk
|
||||||
|
# The last chunk absorbs any rounding remainder so durations sum exactly.
|
||||||
|
duration = (total_seconds - offset) if i == n - 1 else chunk
|
||||||
|
plan.append((round(offset, 3), round(duration, 3)))
|
||||||
|
return plan
|
||||||
|
|
||||||
|
|
||||||
def load_api_key(preferred: str | None = None) -> tuple[str, str] | tuple[None, None]:
|
def load_api_key(preferred: str | None = None) -> tuple[str, str] | tuple[None, None]:
|
||||||
"""Return (backend, api_key). Prefers Groq, falls back to OpenAI.
|
"""Return (backend, api_key). Prefers Groq, falls back to OpenAI.
|
||||||
@@ -109,6 +139,65 @@ def extract_audio(video_path: str, out_path: Path) -> Path:
|
|||||||
return out_path
|
return out_path
|
||||||
|
|
||||||
|
|
||||||
|
def audio_duration(audio_path: Path) -> float:
|
||||||
|
"""Return the duration of an audio file in seconds via ffprobe."""
|
||||||
|
if shutil.which("ffprobe") is None:
|
||||||
|
raise SystemExit("ffprobe is not installed. Install with: brew install ffmpeg")
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
[
|
||||||
|
"ffprobe",
|
||||||
|
"-v", "quiet",
|
||||||
|
"-print_format", "json",
|
||||||
|
"-show_format",
|
||||||
|
str(audio_path.resolve()),
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise SystemExit(f"ffprobe failed: {result.stderr.strip()}")
|
||||||
|
fmt = json.loads(result.stdout or "{}").get("format", {})
|
||||||
|
return float(fmt.get("duration") or 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def split_audio(
|
||||||
|
full_audio: Path,
|
||||||
|
work_dir: Path,
|
||||||
|
plan: list[tuple[float, float]],
|
||||||
|
) -> list[tuple[Path, float]]:
|
||||||
|
"""Slice full_audio into per-plan chunk files, returning (path, offset) pairs.
|
||||||
|
|
||||||
|
Uses stream copy (`-c copy`) so there is no re-encode and no quality loss;
|
||||||
|
mp3 frame boundaries are close enough for transcription's purposes.
|
||||||
|
"""
|
||||||
|
if shutil.which("ffmpeg") is None:
|
||||||
|
raise SystemExit("ffmpeg is not installed. Install with: brew install ffmpeg")
|
||||||
|
|
||||||
|
work_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
chunks: list[tuple[Path, float]] = []
|
||||||
|
for index, (offset, duration) in enumerate(plan):
|
||||||
|
out_path = work_dir / f"chunk_{index:03d}.mp3"
|
||||||
|
cmd = [
|
||||||
|
"ffmpeg",
|
||||||
|
"-hide_banner",
|
||||||
|
"-loglevel", "error",
|
||||||
|
"-y",
|
||||||
|
"-ss", f"{offset:.3f}",
|
||||||
|
"-i", str(full_audio.resolve()),
|
||||||
|
"-t", f"{duration:.3f}",
|
||||||
|
"-c", "copy",
|
||||||
|
str(out_path.resolve()),
|
||||||
|
]
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0 or not out_path.exists() or out_path.stat().st_size == 0:
|
||||||
|
raise SystemExit(
|
||||||
|
f"ffmpeg failed to split audio chunk {index + 1}: {result.stderr.strip()}"
|
||||||
|
)
|
||||||
|
chunks.append((out_path, offset))
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
|
||||||
def _build_multipart(fields: dict[str, str], file_path: Path) -> tuple[bytes, str]:
|
def _build_multipart(fields: dict[str, str], file_path: Path) -> tuple[bytes, str]:
|
||||||
"""Assemble a multipart/form-data body the Whisper APIs accept.
|
"""Assemble a multipart/form-data body the Whisper APIs accept.
|
||||||
|
|
||||||
@@ -240,6 +329,24 @@ def _retry_after(exc: urllib.error.HTTPError) -> float | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def shift_segments(segments: list[dict], offset_seconds: float) -> list[dict]:
|
||||||
|
"""Return a copy of segments with start/end shifted by offset_seconds.
|
||||||
|
|
||||||
|
Each chunk is transcribed in isolation, so Whisper returns 0-based timestamps
|
||||||
|
per chunk; shifting by the chunk's offset stitches them into source time.
|
||||||
|
"""
|
||||||
|
if offset_seconds == 0:
|
||||||
|
return segments
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"start": round(seg["start"] + offset_seconds, 2),
|
||||||
|
"end": round(seg["end"] + offset_seconds, 2),
|
||||||
|
"text": seg["text"],
|
||||||
|
}
|
||||||
|
for seg in segments
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _segments_from_response(data: dict) -> list[dict]:
|
def _segments_from_response(data: dict) -> list[dict]:
|
||||||
"""Convert Whisper verbose_json into our {start, end, text} segment format."""
|
"""Convert Whisper verbose_json into our {start, end, text} segment format."""
|
||||||
out: list[dict] = []
|
out: list[dict] = []
|
||||||
@@ -261,6 +368,49 @@ def _segments_from_response(data: dict) -> list[dict]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def transcribe_chunks(
|
||||||
|
chunks: list[tuple[Path, float]],
|
||||||
|
transcribe_one,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Transcribe each chunk, shift its segments by the chunk offset, concatenate.
|
||||||
|
|
||||||
|
A chunk that fails after its own retries is logged and skipped so one bad
|
||||||
|
slice doesn't discard the whole transcript. Raises only if every chunk fails.
|
||||||
|
"""
|
||||||
|
segments: list[dict] = []
|
||||||
|
failures = 0
|
||||||
|
for index, (path, offset) in enumerate(chunks):
|
||||||
|
try:
|
||||||
|
chunk_segments = transcribe_one(path)
|
||||||
|
except SystemExit as exc:
|
||||||
|
failures += 1
|
||||||
|
print(
|
||||||
|
f"[watch] chunk {index + 1}/{len(chunks)} failed — skipping ({exc})",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
segments.extend(shift_segments(chunk_segments, offset))
|
||||||
|
print(
|
||||||
|
f"[watch] chunk {index + 1}/{len(chunks)} → {len(chunk_segments)} segments",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
|
||||||
|
if failures == len(chunks):
|
||||||
|
raise SystemExit("Whisper failed on every audio chunk")
|
||||||
|
return segments
|
||||||
|
|
||||||
|
|
||||||
|
def _transcribe_file(backend: str, api_key: str, audio_path: Path) -> list[dict]:
|
||||||
|
"""Upload one audio file and return its 0-based segments."""
|
||||||
|
if backend == "groq":
|
||||||
|
response = _post_whisper(GROQ_ENDPOINT, api_key, GROQ_MODEL, audio_path)
|
||||||
|
elif backend == "openai":
|
||||||
|
response = _post_whisper(OPENAI_ENDPOINT, api_key, OPENAI_MODEL, audio_path)
|
||||||
|
else:
|
||||||
|
raise SystemExit(f"Unknown whisper backend: {backend}")
|
||||||
|
return _segments_from_response(response)
|
||||||
|
|
||||||
|
|
||||||
def transcribe_video(
|
def transcribe_video(
|
||||||
video_path: str,
|
video_path: str,
|
||||||
audio_out: Path,
|
audio_out: Path,
|
||||||
@@ -286,17 +436,28 @@ def transcribe_video(
|
|||||||
|
|
||||||
print(f"[watch] extracting audio for Whisper ({backend})…", file=sys.stderr)
|
print(f"[watch] extracting audio for Whisper ({backend})…", file=sys.stderr)
|
||||||
audio_path = extract_audio(video_path, audio_out)
|
audio_path = extract_audio(video_path, audio_out)
|
||||||
size_kb = audio_path.stat().st_size / 1024
|
audio_bytes = audio_path.stat().st_size
|
||||||
print(f"[watch] audio: {size_kb:.0f} kB — uploading to {backend} Whisper…", file=sys.stderr)
|
|
||||||
|
|
||||||
if backend == "groq":
|
def transcribe_one(path: Path) -> list[dict]:
|
||||||
response = _post_whisper(GROQ_ENDPOINT, api_key, GROQ_MODEL, audio_path)
|
return _transcribe_file(backend, api_key, path)
|
||||||
elif backend == "openai":
|
|
||||||
response = _post_whisper(OPENAI_ENDPOINT, api_key, OPENAI_MODEL, audio_path)
|
if audio_bytes <= MAX_UPLOAD_BYTES:
|
||||||
|
print(
|
||||||
|
f"[watch] audio: {audio_bytes / 1024:.0f} kB — uploading to {backend} Whisper…",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
segments = transcribe_one(audio_path)
|
||||||
else:
|
else:
|
||||||
raise SystemExit(f"Unknown whisper backend: {backend}")
|
duration = audio_duration(audio_path)
|
||||||
|
plan = plan_chunks(duration, audio_bytes, MAX_UPLOAD_BYTES)
|
||||||
|
print(
|
||||||
|
f"[watch] audio: {audio_bytes / (1024 * 1024):.0f} MB exceeds "
|
||||||
|
f"{MAX_UPLOAD_BYTES // (1024 * 1024)} MB — splitting into {len(plan)} chunks…",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
chunks = split_audio(audio_path, audio_out.parent / "chunks", plan)
|
||||||
|
segments = transcribe_chunks(chunks, transcribe_one)
|
||||||
|
|
||||||
segments = _segments_from_response(response)
|
|
||||||
if not segments:
|
if not segments:
|
||||||
raise SystemExit("Whisper returned no transcript segments")
|
raise SystemExit("Whisper returned no transcript segments")
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,162 @@
|
|||||||
|
"""Frame-delta dedup: per-pixel difference, greedy de-duplication, integration."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import frames
|
||||||
|
|
||||||
|
|
||||||
|
# --- _frame_delta: mean absolute per-pixel difference ------------------------
|
||||||
|
|
||||||
|
def test_frame_delta_identical_is_zero():
|
||||||
|
a = bytes([10] * 16)
|
||||||
|
assert frames._frame_delta(a, a) == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_frame_delta_is_mean_absolute_difference():
|
||||||
|
a = bytes([0, 0, 0, 0])
|
||||||
|
b = bytes([4, 0, 0, 0])
|
||||||
|
assert frames._frame_delta(a, b) == 1.0 # (4+0+0+0)/4
|
||||||
|
|
||||||
|
|
||||||
|
def test_frame_delta_mismatched_length_is_infinite():
|
||||||
|
assert frames._frame_delta(bytes([1, 2]), bytes([1, 2, 3])) == float("inf")
|
||||||
|
|
||||||
|
|
||||||
|
# --- _dedupe_by_deltas: greedy drop vs last *kept* thumbnail ------------------
|
||||||
|
|
||||||
|
def _touch(dirpath: Path, n: int) -> list[dict]:
|
||||||
|
dirpath.mkdir(parents=True, exist_ok=True)
|
||||||
|
out = []
|
||||||
|
for i in range(n):
|
||||||
|
p = dirpath / f"frame_{i:04d}.jpg"
|
||||||
|
p.write_bytes(b"x")
|
||||||
|
out.append({"index": i, "timestamp_seconds": float(i), "path": str(p), "reason": "scene-change"})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
FLAT0 = bytes([0, 0, 0, 0])
|
||||||
|
FLAT255 = bytes([255, 255, 255, 255])
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_collapses_identical_run(tmp_path: Path):
|
||||||
|
cands = _touch(tmp_path, 5)
|
||||||
|
thumbs = [FLAT0, FLAT0, FLAT0, FLAT0, FLAT0]
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(cands, thumbs, threshold=2.0)
|
||||||
|
assert dropped == 4
|
||||||
|
assert len(survivors) == 1
|
||||||
|
assert survivors[0]["index"] == 0
|
||||||
|
assert sorted(p.name for p in tmp_path.glob("frame_*.jpg")) == ["frame_0000.jpg"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_keeps_all_distinct(tmp_path: Path):
|
||||||
|
cands = _touch(tmp_path, 4)
|
||||||
|
thumbs = [FLAT0, FLAT255, FLAT0, FLAT255]
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(cands, thumbs, threshold=2.0)
|
||||||
|
assert dropped == 0
|
||||||
|
assert [s["index"] for s in survivors] == [0, 1, 2, 3]
|
||||||
|
assert len(list(tmp_path.glob("frame_*.jpg"))) == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_compares_against_last_kept_not_previous(tmp_path: Path):
|
||||||
|
"""A,A,B,B,A with A/B far apart -> keep A0, B2, A4 (drops the repeats)."""
|
||||||
|
cands = _touch(tmp_path, 5)
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(
|
||||||
|
cands, [FLAT0, FLAT0, FLAT255, FLAT255, FLAT0], threshold=2.0
|
||||||
|
)
|
||||||
|
assert [s["index"] for s in survivors] == [0, 1, 2] # reindexed survivors
|
||||||
|
assert dropped == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_threshold_is_inclusive(tmp_path: Path):
|
||||||
|
"""Delta exactly == threshold is treated as a duplicate (<=)."""
|
||||||
|
cands = _touch(tmp_path, 2)
|
||||||
|
a = bytes([0, 0, 0, 0])
|
||||||
|
b = bytes([8, 0, 0, 0]) # mean abs diff == 2.0
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(cands, [a, b], threshold=2.0)
|
||||||
|
assert dropped == 1
|
||||||
|
assert len(survivors) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_empty_and_single_are_noops(tmp_path: Path):
|
||||||
|
assert frames._dedupe_by_deltas([], [], threshold=2.0) == ([], 0)
|
||||||
|
one = _touch(tmp_path, 1)
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(one, [FLAT0], threshold=2.0)
|
||||||
|
assert dropped == 0
|
||||||
|
assert len(survivors) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_mismatched_thumb_count_is_noop(tmp_path: Path):
|
||||||
|
"""Fail open: if thumbs don't line up with candidates, change nothing."""
|
||||||
|
cands = _touch(tmp_path, 3)
|
||||||
|
survivors, dropped = frames._dedupe_by_deltas(cands, [FLAT0], threshold=2.0)
|
||||||
|
assert dropped == 0
|
||||||
|
assert len(survivors) == 3
|
||||||
|
|
||||||
|
|
||||||
|
# --- _thumb_frames + dedupe_perceptual: real ffmpeg over extracted JPEGs ------
|
||||||
|
|
||||||
|
def test_thumb_frames_match_candidate_count(cut_clip: Path, tmp_path: Path):
|
||||||
|
out = frames.extract_scene_candidates(str(cut_clip), tmp_path / "f", max_frames=None)
|
||||||
|
thumbs = frames._thumb_frames([Path(fr["path"]) for fr in out])
|
||||||
|
assert len(thumbs) == len(out)
|
||||||
|
assert all(len(t) == frames.DEDUP_THUMB * frames.DEDUP_THUMB for t in thumbs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_perceptual_collapses_static_clip(static_clip: Path, tmp_path: Path):
|
||||||
|
out = frames.extract(str(static_clip), tmp_path / "f", fps=4.0, max_frames=10)
|
||||||
|
n_before = len(out)
|
||||||
|
survivors, dropped = frames.dedupe_perceptual(out)
|
||||||
|
assert n_before > 1
|
||||||
|
assert len(survivors) == 1
|
||||||
|
assert dropped == n_before - 1
|
||||||
|
assert len(list((tmp_path / "f").glob("frame_*.jpg"))) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedupe_perceptual_keeps_distinct_cuts(cut_clip: Path, tmp_path: Path):
|
||||||
|
"""Distinct color shots differ in luma, so frame-delta keeps them all."""
|
||||||
|
out = frames.extract_scene_candidates(str(cut_clip), tmp_path / "f", max_frames=None)
|
||||||
|
n_before = len(out)
|
||||||
|
survivors, dropped = frames.dedupe_perceptual(out)
|
||||||
|
assert dropped == 0
|
||||||
|
assert len(survivors) == n_before
|
||||||
|
|
||||||
|
|
||||||
|
# --- engine integration: dedup runs before the cap, reports deduped_count -----
|
||||||
|
|
||||||
|
def test_scene_engine_reports_zero_dedup_on_distinct(cut_clip: Path, tmp_path: Path):
|
||||||
|
out, meta = frames.extract_scene_or_uniform(
|
||||||
|
str(cut_clip), tmp_path / "f", fps=2.0, target_frames=50, max_frames=100,
|
||||||
|
)
|
||||||
|
assert meta["engine"] == "scene"
|
||||||
|
assert meta["deduped_count"] == 0
|
||||||
|
assert len(out) == len(list((tmp_path / "f").glob("frame_*.jpg")))
|
||||||
|
|
||||||
|
|
||||||
|
def test_uniform_fallback_dedupes_static(static_clip: Path, tmp_path: Path):
|
||||||
|
out, meta = frames.extract_scene_or_uniform(
|
||||||
|
str(static_clip), tmp_path / "f", fps=4.0, target_frames=12, max_frames=100,
|
||||||
|
)
|
||||||
|
assert meta["engine"] == "uniform"
|
||||||
|
assert meta["fallback"] is True
|
||||||
|
assert meta["deduped_count"] > 0
|
||||||
|
assert meta["selected_count"] == 1 # identical frames collapse to one
|
||||||
|
assert len(out) == 1
|
||||||
|
assert len(list((tmp_path / "f").glob("frame_*.jpg"))) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_keyframe_uniform_fallback_dedupes_static(static_clip: Path, tmp_path: Path):
|
||||||
|
out, meta = frames.extract_keyframes(str(static_clip), tmp_path / "f", max_frames=50)
|
||||||
|
assert meta["engine"] == "uniform"
|
||||||
|
assert meta["deduped_count"] > 0
|
||||||
|
assert len(out) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_false_disables_collapse(static_clip: Path, tmp_path: Path):
|
||||||
|
out, meta = frames.extract_scene_or_uniform(
|
||||||
|
str(static_clip), tmp_path / "f", fps=4.0, target_frames=12, max_frames=100,
|
||||||
|
dedup=False,
|
||||||
|
)
|
||||||
|
assert meta["deduped_count"] == 0
|
||||||
|
assert meta["selected_count"] > 1 # no collapse without dedup
|
||||||
|
assert len(out) > 1
|
||||||
@@ -67,3 +67,19 @@ def test_timestamps_with_transcript_detail_is_cue_only(cut_clip: Path):
|
|||||||
assert "reason=transcript-cue" in out
|
assert "reason=transcript-cue" in out
|
||||||
assert "reason=scene-change" not in out
|
assert "reason=scene-change" not in out
|
||||||
assert "reason=keyframe" not in out
|
assert "reason=keyframe" not in out
|
||||||
|
|
||||||
|
|
||||||
|
def _frame_lines(out: str) -> int:
|
||||||
|
return sum(1 for line in out.splitlines() if "/frames/frame_" in line and "(t=" in line)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_collapses_static_by_default(static_clip: Path):
|
||||||
|
out = _run(static_clip) # solid blue → identical frames collapse to one
|
||||||
|
assert "near-duplicate" in out
|
||||||
|
assert _frame_lines(out) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_dedup_preserves_static_frames(static_clip: Path):
|
||||||
|
out = _run(static_clip, "--no-dedup")
|
||||||
|
assert "near-duplicate" not in out
|
||||||
|
assert _frame_lines(out) > 1
|
||||||
|
|||||||
@@ -0,0 +1,157 @@
|
|||||||
|
"""Whisper auto-chunking: plan, split, and timestamp stitching."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import whisper
|
||||||
|
|
||||||
|
|
||||||
|
MB = 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
class TestPlanChunks:
|
||||||
|
def test_under_limit_is_single_chunk(self):
|
||||||
|
plan = whisper.plan_chunks(total_seconds=600.0, total_bytes=5 * MB, max_bytes=24 * MB)
|
||||||
|
assert plan == [(0.0, 600.0)]
|
||||||
|
|
||||||
|
def test_at_limit_is_single_chunk(self):
|
||||||
|
plan = whisper.plan_chunks(total_seconds=600.0, total_bytes=24 * MB, max_bytes=24 * MB)
|
||||||
|
assert plan == [(0.0, 600.0)]
|
||||||
|
|
||||||
|
def test_over_limit_splits_into_enough_chunks(self):
|
||||||
|
# 71 MB against a 24 MB cap → ceil(71/24) = 3 chunks.
|
||||||
|
plan = whisper.plan_chunks(total_seconds=3600.0, total_bytes=71 * MB, max_bytes=24 * MB)
|
||||||
|
assert len(plan) == 3
|
||||||
|
|
||||||
|
def test_chunks_are_contiguous_and_cover_full_duration(self):
|
||||||
|
total = 3600.0
|
||||||
|
plan = whisper.plan_chunks(total_seconds=total, total_bytes=71 * MB, max_bytes=24 * MB)
|
||||||
|
# Offsets start at 0 and each picks up where the previous ended.
|
||||||
|
assert plan[0][0] == 0.0
|
||||||
|
for (off, dur), (next_off, _) in zip(plan, plan[1:]):
|
||||||
|
assert math.isclose(off + dur, next_off)
|
||||||
|
last_off, last_dur = plan[-1]
|
||||||
|
assert math.isclose(last_off + last_dur, total)
|
||||||
|
|
||||||
|
def test_each_chunk_estimated_under_limit(self):
|
||||||
|
total_seconds, total_bytes, cap = 3600.0, 71 * MB, 24 * MB
|
||||||
|
plan = whisper.plan_chunks(total_seconds, total_bytes, cap)
|
||||||
|
bytes_per_second = total_bytes / total_seconds
|
||||||
|
for _off, dur in plan:
|
||||||
|
assert dur * bytes_per_second <= cap
|
||||||
|
|
||||||
|
def test_zero_duration_is_single_chunk(self):
|
||||||
|
plan = whisper.plan_chunks(total_seconds=0.0, total_bytes=0, max_bytes=24 * MB)
|
||||||
|
assert plan == [(0.0, 0.0)]
|
||||||
|
|
||||||
|
|
||||||
|
class TestShiftSegments:
|
||||||
|
def test_adds_offset_to_start_and_end(self):
|
||||||
|
segs = [{"start": 0.0, "end": 2.5, "text": "hi"}, {"start": 2.5, "end": 4.0, "text": "there"}]
|
||||||
|
shifted = whisper.shift_segments(segs, 1800.0)
|
||||||
|
assert shifted == [
|
||||||
|
{"start": 1800.0, "end": 1802.5, "text": "hi"},
|
||||||
|
{"start": 1802.5, "end": 1804.0, "text": "there"},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_zero_offset_is_identity(self):
|
||||||
|
segs = [{"start": 1.0, "end": 2.0, "text": "x"}]
|
||||||
|
assert whisper.shift_segments(segs, 0.0) == segs
|
||||||
|
|
||||||
|
def test_does_not_mutate_input(self):
|
||||||
|
segs = [{"start": 0.0, "end": 1.0, "text": "x"}]
|
||||||
|
whisper.shift_segments(segs, 10.0)
|
||||||
|
assert segs[0]["start"] == 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def _make_mp3(path: Path, seconds: float) -> None:
|
||||||
|
"""Synthesize a mono 16k 64k mp3 of a sine tone — mirrors extract_audio's format."""
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
"ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||||
|
"-f", "lavfi", "-t", str(seconds), "-i", "sine=frequency=440:sample_rate=16000",
|
||||||
|
"-acodec", "libmp3lame", "-ar", "16000", "-ac", "1", "-b:a", "64k",
|
||||||
|
str(path),
|
||||||
|
],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSplitAudio:
|
||||||
|
def test_creates_one_file_per_plan_entry(self, tmp_path: Path):
|
||||||
|
full = tmp_path / "audio.mp3"
|
||||||
|
_make_mp3(full, 6.0)
|
||||||
|
plan = [(0.0, 3.0), (3.0, 3.0)]
|
||||||
|
|
||||||
|
chunks = whisper.split_audio(full, tmp_path, plan)
|
||||||
|
|
||||||
|
assert len(chunks) == 2
|
||||||
|
for chunk_path, _offset in chunks:
|
||||||
|
assert chunk_path.exists() and chunk_path.stat().st_size > 0
|
||||||
|
|
||||||
|
def test_returns_plan_offsets(self, tmp_path: Path):
|
||||||
|
full = tmp_path / "audio.mp3"
|
||||||
|
_make_mp3(full, 6.0)
|
||||||
|
plan = [(0.0, 3.0), (3.0, 3.0)]
|
||||||
|
|
||||||
|
chunks = whisper.split_audio(full, tmp_path, plan)
|
||||||
|
|
||||||
|
assert [offset for _path, offset in chunks] == [0.0, 3.0]
|
||||||
|
|
||||||
|
def test_chunks_are_smaller_than_full(self, tmp_path: Path):
|
||||||
|
full = tmp_path / "audio.mp3"
|
||||||
|
_make_mp3(full, 6.0)
|
||||||
|
plan = [(0.0, 3.0), (3.0, 3.0)]
|
||||||
|
|
||||||
|
chunks = whisper.split_audio(full, tmp_path, plan)
|
||||||
|
|
||||||
|
full_size = full.stat().st_size
|
||||||
|
for chunk_path, _offset in chunks:
|
||||||
|
assert chunk_path.stat().st_size < full_size
|
||||||
|
|
||||||
|
|
||||||
|
class TestAudioDuration:
|
||||||
|
def test_reads_duration_of_synthesized_clip(self, tmp_path: Path):
|
||||||
|
audio = tmp_path / "audio.mp3"
|
||||||
|
_make_mp3(audio, 5.0)
|
||||||
|
assert whisper.audio_duration(audio) == pytest.approx(5.0, abs=0.5)
|
||||||
|
|
||||||
|
|
||||||
|
class TestTranscribeChunks:
|
||||||
|
def test_shifts_and_concatenates_each_chunk(self):
|
||||||
|
chunks = [(Path("a.mp3"), 0.0), (Path("b.mp3"), 100.0)]
|
||||||
|
|
||||||
|
def fake_transcribe(path: Path) -> list[dict]:
|
||||||
|
return [{"start": 0.0, "end": 2.0, "text": path.stem}]
|
||||||
|
|
||||||
|
out = whisper.transcribe_chunks(chunks, fake_transcribe)
|
||||||
|
|
||||||
|
assert out == [
|
||||||
|
{"start": 0.0, "end": 2.0, "text": "a"},
|
||||||
|
{"start": 100.0, "end": 102.0, "text": "b"},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_keeps_successful_chunks_when_one_fails(self):
|
||||||
|
chunks = [(Path("a.mp3"), 0.0), (Path("b.mp3"), 100.0)]
|
||||||
|
|
||||||
|
def flaky(path: Path) -> list[dict]:
|
||||||
|
if path.stem == "b":
|
||||||
|
raise SystemExit("chunk b failed")
|
||||||
|
return [{"start": 1.0, "end": 2.0, "text": "a"}]
|
||||||
|
|
||||||
|
out = whisper.transcribe_chunks(chunks, flaky)
|
||||||
|
|
||||||
|
assert out == [{"start": 1.0, "end": 2.0, "text": "a"}]
|
||||||
|
|
||||||
|
def test_raises_when_every_chunk_fails(self):
|
||||||
|
chunks = [(Path("a.mp3"), 0.0), (Path("b.mp3"), 100.0)]
|
||||||
|
|
||||||
|
def always_fail(path: Path) -> list[dict]:
|
||||||
|
raise SystemExit("boom")
|
||||||
|
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
whisper.transcribe_chunks(chunks, always_fail)
|
||||||
Reference in New Issue
Block a user