/watch

Give Claude the ability to watch any video.

Claude Code (recommended — auto-updates via marketplace):

/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video

Codex, Cursor, Copilot, Gemini CLI, or any of 50+ Agent Skills hosts:

npx skills add bradautomates/claude-video -g

(-g installs globally for your user, available across all projects. Drop it to scope per-project.)

More install options (claude.ai web, manual) in the Install section below.

Zero config to start — yt-dlp and ffmpeg install on first run via brew on macOS (Linux/Windows print exact commands). Captions cover most public videos for free. Whisper API key is only needed when a video has no captions.

---

Claude can read a webpage, run a script, browse a repo. What it can't do, out of the box, is watch a video. You paste a YouTube link and it has to either guess from the title or pull a transcript that's missing 90% of what's on screen.

With Claude Video /watch you can paste a URL or a local path, ask a question, and Claude fetches captions first, downloads only what it needs, extracts frames (scene-aware, or fast keyframes at efficient detail), pulls a timestamped transcript (free captions when available, Whisper API as fallback), and Reads every frame as an image. By the time it answers, it has seen the video and heard the audio.

/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30 second mark?

Why this exists

I run a YouTube channel and a business, Solaris Automation, so I'm constantly using video to keep up with content. If I see a YouTube video that's blowing up, I want to know how the creator structured the hook — what's on screen in the first 3 seconds, what they said, why it worked. That used to mean watching it myself with a notepad. Now I just paste the URL and ask.

The other half is summarization. Most YouTube videos don't deserve 20 minutes of my attention. I hand the URL to Claude, it pulls the transcript, and tells me what actually happened. If the visual matters, frames come along too. If it's a podcast or a talking head, transcript is enough.

Claude is great at reading and synthesizing — but until now, video was the one input I couldn't hand it. Pasting a YouTube link got you nothing useful. /watch closes that gap.

What people actually use it for

Analyze someone else's content. /watch https://youtu.be/<viral-video> what hook did they open with? Claude looks at the first frames, reads the opening transcript, breaks down the structure. Same for ad creative, competitor launches, podcast intros, anything where the how matters as much as the what.

Diagnose a bug from a video. Someone sends you a screen recording of something broken. /watch bug-repro.mov what's going wrong? Claude watches the recording, finds the frame where the issue appears, describes what's on screen, often catches the cause without you ever opening the file.

Summarize a video. /watch https://youtu.be/<long-thing> summarize this does the obvious thing — pulls the structure, the key moments, what was actually said and shown. Faster than watching at 2x.

Cut the hype out of an update video. /watch https://youtu.be/<launch-video> what's actually new — skip the hype Strip a "game-changer" feature drop down to the few things that matter, so you get the substance without ten minutes of intro and overselling.

Jump to the one answer in a long tutorial. /watch https://youtu.be/<2hr-tutorial> how do they set up the webhook? Instead of scrubbing an hour of video for a 30-second answer — or matching a vague transcript and watching anyway — Claude finds the moment and tells you what's on screen there.

Read a screen-share recording the transcript can't. /watch standup.mp4 what was on the shared screen when they decided? When a meeting transcript collapses into "if you look here" and "make it look like this," the words stop carrying the meaning — the frames recover what was actually being shown.

Turn a playlist into notes. /watch https://youtu.be/<video> summarize this to a note Run it across a series and file a per-video summary, so a channel or course becomes a searchable set of notes instead of hours you have to sit through.

How it works

1. You paste a video and a question. URL (anything yt-dlp supports — YouTube, Loom, TikTok, X, Instagram, plus a few hundred more) or a local path (.mp4, .mov, .mkv, .webm).
2. yt-dlp checks captions first. At transcript detail, captioned URLs return without downloading video. Otherwise, or when Whisper needs audio, it downloads only what the run needs.
3. ffmpeg extracts frames at the chosen detail. efficient decodes keyframes only (near-instant); balanced/token-burner prefer scene-change frames and fall back to the duration-aware uniform sampler when they under-produce. JPEGs are 512px wide by default and clamped to 1998px tall for Claude Read compatibility.
4. The transcript comes from one of two places. First try: yt-dlp pulls native captions (manual or auto-generated) from the source. Free, instant, accurate-ish. Fallback: extract a mono 16 kHz 64 kbps mp3 audio clip (~480 kB/min) and ship it to Whisper — Groq's whisper-large-v3 (preferred — cheaper and faster) or OpenAI's whisper-1.
5. Frames + transcript are handed to Claude. The script prints frame paths with t=MM:SS markers and the transcript with timestamps. Claude Reads each frame in parallel — JPEGs render directly as images in its context.
6. Claude answers grounded in what's actually on screen and in the audio. Not "based on the description" or "according to the title." It saw the frames. It heard the transcript. It answers the way someone who watched the video would.
7. Cleanup. The script prints a working directory at the end. If you're not asking follow-ups, Claude removes it.

Frame budget — why it matters

Token cost is dominated by frames. Every frame is an image; image tokens add up fast. The script's auto-fps logic exists so you don't blow your context budget on a sparse scan of a 30-minute video that would have been better answered by a focused 30-second window.

Duration	Default frame budget	What you get
≤30 s	~30 frames	Dense — basically every key moment
30 s - 1 min	~40 frames	Still dense
1 - 3 min	~60 frames	Comfortable
3 - 10 min	~80 frames	Sparse but workable
> 10 min	100 frames (capped modes)	"Sparse scan" warning — re-run focused, or --detail token-burner for full uncapped coverage

When the user names a moment ("around 2:30", "the last 30 seconds", "from 0:45 to 1:00"), pass --start / --end. Focused mode gets denser per-second budgets, capped at 2 fps. Far more useful than a sparse pass over the whole thing.

Frame deduplication — why you don't pay for held slides

Sampling spreads frames evenly across time, but time isn't where the information is. A 10-minute screen recording that sits on one slide for 90 seconds gives you a dozen pixel-for-pixel identical frames of that slide — each one a separate image, each one billed in tokens. Even-sampling doesn't know they're the same; it just hands them all over.

So before the frames are handed to Claude, a dedup pass drops the ones that didn't change. It runs by default on every frame mode (--no-dedup turns it off). Here's the logic, end to end:

1. Decode each frame to a tiny fingerprint. One ffmpeg call scales every extracted JPEG down to a 16×16 grayscale thumbnail — 256 brightness values per frame. ffmpeg does the pixel decoding; everything after is pure-stdlib Python, no image libraries.
2. Measure how much changed. For each frame, compute the mean absolute difference against the last frame that was kept — the average per-pixel brightness change on a 0–255 scale. This is a literal "how different are these two frames" number, not a heuristic.
3. Drop it if nothing meaningfully changed. If that difference is at or below the threshold (2.0 — i.e. the picture moved less than ~1% on average), the frame is a near-duplicate: delete it. Otherwise keep it, and it becomes the new reference to compare the next frame against.
4. Sample the survivors down to the budget. Only after duplicates are gone does the frame-budget cap apply — so the budget is spent on distinct frames, not diluted by repeats.

Two design choices worth calling out:

• Compare against the last kept frame, not the immediately previous one. A slow fade across ten frames never trips a frame-to-frame threshold (each step is tiny) but is obviously different end-to-end. Comparing against the last kept frame catches that gradual drift and still collapses a dead-static hold to a single frame.
• It's deliberately conservative, and it measures absolute brightness — not structure. The threshold is low on purpose: a code diff with one changed line, a terminal scrolling a single row, or a slide that just gained a bullet all clear it and survive. Measuring absolute brightness (rather than a perceptual/structure hash) is what lets it tell two flat frames apart — a solid blue section slide and a solid green one differ in luma, so they're correctly kept, where a structure-only hash would see "no internal detail" in both and wrongly merge them.

The result is reported, not silent: the Frames line shows e.g. 6 selected from 14 candidates (… 8 near-duplicates dropped …) so you can always see what was collapsed. On a clip that holds a slide and then cuts to motion, that's the difference between paying for 14 frames and paying for 6 — with the transition and every distinct moment preserved. On footage that's genuinely always moving, nothing gets dropped and you pay exactly what you would have anyway.

Detail modes — measured

The --detail dial trades speed and token cost for visual fidelity. Numbers below are from a real run against a 49:08 (2948 s) YouTube video (1280×720 source, English auto-captions, 1394 caption segments) — a long, mostly-static screen recording, which is the case that stresses the caps hardest. Frame-extraction times are measured against a pre-downloaded local copy so they reflect the mode's CPU cost, not network speed. The one-time video download for this clip was ~37 s / 76 MB (shared by efficient / balanced / token-burner).

Mode	Engine	Frames	Cap	Extraction time	Temporal coverage	Est. image tokens
transcript	none (captions)	0	—	~4.5 s (network-bound — one yt-dlp call, no video download)	full (text)	0 (transcript ≈26.6k text tokens)
efficient	keyframe (-skip_frame nokey)	50	50	~0.5 s	0:00 → 49:04 (full)	~9.8k
balanced	scene-change	100	100	~20.9 s	0:00 → 48:38 (full)	~19.7k
token-burner	scene-change	116	uncapped	~21.0 s	0:00 → 48:38 (full)	~22.8k

Image-token estimate uses Anthropic's (width × height) / 750 per image. At the default 512px width these 720p frames are 512×288, so ≈197 tokens/frame; total image tokens ≈ frames × 197. Bumping --resolution to 1024 roughly 4×s that. The transcript (~26.6k tokens here) is surfaced in every mode that has captions, so on the frame modes it adds to the image tokens above — on long videos the transcript, not the frames, is often the larger cost.

Note on the transcript time. It looks slower than efficient only because the two numbers measure different things: transcript's ~4.5 s is a network round-trip to the source (its only step — no video download, no ffmpeg), while the frame modes' times are local CPU with the shared ~37 s download billed separately. End-to-end from a cold URL, transcript is the cheapest mode by far; efficient from a cold URL would be ~37 s of download + 0.5 s of decode.

What the numbers show:

• One consistent sampling rule across every frame mode. All three detect all candidates across the full range, then even-sample (first + last always kept) down to the cap via a shared _even_sample helper — transcript excepted. Keyframes (efficient) and scene-cuts (balanced/token-burner) differ only in the candidate source and the cap, never in how coverage is spread.
• efficient is the speed tier — ~0.5 s because it only reconstructs keyframes (P/B frames are skipped). For this clip it decoded 675 keyframes and evenly sampled down to its 50-frame cap, spread across the whole 49 minutes.
• Caps are enforced and coverage spans the full clip. efficient (675 → 50) and balanced (116 → 100) both sample first→last, so the last frame lands at 48:38–49:04, not partway through. Verified on the full video and on focused ranges (--start/--end of 30 s → 10 frames, 3 min → 37 frames). The even-sample step (len(indices) == cap) cannot return more than the cap. (This addresses the "sometimes returns too many / drops the tail" concern: the N selected from 675 candidates line shows the pre-sample candidate count, not what gets surfaced — only the sampled frames are written and Read.)
• balanced now full-decodes like token-burner (~21 s vs efficient's ~0.5 s). Detecting every scene cut requires decoding the whole video; the old -frames:v early-exit was ~3× faster but kept only the first 100 cuts and dropped the tail of long videos — so it was removed in favor of even coverage.
• token-burner only diverges from balanced past the cap. This recording had 116 cuts over 49 min, so balanced sampled 100 of them and token-burner kept all 116 — both spanning the full video. On a high-motion video with hundreds of cuts, token-burner keeps everything (and the >250-frame token warning kicks in) while balanced thins to 100.
• efficient can return more frames than balanced on low-motion footage (50 keyframes vs. few scene cuts) — the tiers differ by extraction method and cap, not by a guaranteed frame-count ordering. "Efficient" means near-instant extraction, not always fewer frames.

Bottom line on timing: every mode finished its own work in under 22 s (plus the shared ~37 s download for the frame modes). efficient is ~40× faster than the scene modes here because the scene detector decodes every frame of the full 49 minutes, while efficient only reconstructs keyframes.

Install

Surface	Install
Claude Code	/plugin marketplace add bradautomates/claude-video then /plugin install watch@claude-video
Codex, Cursor, Copilot, Gemini CLI, +50 more	npx skills add bradautomates/claude-video -g
claude.ai (web)	Download watch.skill → Settings → Capabilities → Skills → +
Manual / dev	git clone then symlink skills/watch into your host's skills dir (see below)

Claude Code

/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video

Update later with /plugin update watch@claude-video.

Codex, Cursor, Copilot, Gemini CLI, and 50+ other hosts

The Agent Skills CLI installs the skill into whatever agents it detects:

npx skills add bradautomates/claude-video -g

-g installs globally for your user (~/.codex/skills, ~/.cursor/skills, etc.); drop it to install into the current project instead. Useful flags:

• -a, --agent <names…> — target specific hosts, e.g. -a codex -a cursor
• -l, --list — list the skills in this repo without installing
• --copy — copy files instead of symlinking (for filesystems without symlink support)

The CLI discovers the skill from skills/watch/SKILL.md and copies the whole folder — SKILL.md plus its scripts/ runtime — as a self-contained unit. SKILL.md resolves its own scripts relative to wherever it was installed, so it works the same on every host.

Update later with npx skills update watch -g.

claude.ai (web)

1. Download watch.skill from the latest release.
2. Go to Settings → Capabilities → Skills.
3. Click + and drop the file in.

Enable "Code execution and file creation" under Capabilities first — the skill shells out to ffmpeg and yt-dlp, so it won't run without it.

Manual (developer)

Clone the repo and symlink the self-contained skill folder into your host's skills directory — the symlink keeps the install in sync with your working tree as you edit:

git clone https://github.com/bradautomates/claude-video.git
ln -s "$(pwd)/claude-video/skills/watch" ~/.claude/skills/watch   # or ~/.codex/skills/watch

For claude.ai, build the .skill bundle from source: bash skills/watch/scripts/build-skill.sh produces dist/watch.skill.

First run

On the first /watch call, the skill runs scripts/setup.py --check. If ffmpeg / yt-dlp aren't on your PATH, or no Whisper API key is set, it walks you through fixing it:

• macOS — auto-runs brew install ffmpeg yt-dlp.
• Linux — prints the exact apt / dnf / pipx commands.
• Windows — prints the winget / pip commands.
• API key — scaffolds ~/.config/watch/.env (mode 0600) with commented placeholders for GROQ_API_KEY (preferred) and OPENAI_API_KEY.

After setup, preflight is silent and /watch just works. The check is a sub-100ms lookup, so it doesn't slow you down on subsequent runs.

Bring your own keys

Captions cover the majority of public videos for free. The Whisper fallback only kicks in when a video genuinely has no caption track — typically local files, TikToks, some Vimeos, and the occasional caption-less YouTube upload.

Capability	What you need	Cost
Download + native captions	yt-dlp + ffmpeg	Free
Whisper fallback (preferred)	Groq API key — whisper-large-v3	Cheap, fast
Whisper fallback (alt)	OpenAI API key — whisper-1	Standard pricing
Disable Whisper entirely	--no-whisper	Free, frames-only when no captions

Usage

/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30 second mark?
/watch https://www.tiktok.com/@user/video/123 summarize this
/watch ~/Movies/screen-recording.mp4 when does the UI break?
/watch https://vimeo.com/123 what tools does she mention?

Focused on a specific section — denser frame budget, lower token cost:

/watch https://youtu.be/abc --start 2:15 --end 2:45
/watch video.mp4 --start 50 --end 60
/watch "$URL" --start 1:12:00            # from 1h12m to end

Other knobs (passed to scripts/watch.py):

• --detail transcript|efficient|balanced|token-burner — fidelity/speed dial. transcript skips frames (transcript only); efficient uses fast keyframes (cap 50); balanced uses scene-aware frames (cap 100); token-burner is scene-aware and uncapped.
• --timestamps T1,T2,… — grab a frame at each absolute timestamp (SS/MM:SS/HH:MM:SS). Claude reads the transcript first, then targets the moments the presenter flags ("look here", "as you can see"). Added on top of the detail frames (reserved against the cap); out-of-window cues are dropped in focus mode; with --detail transcript these become the only frames.
• --max-frames N — lower the frame cap for a tighter token budget.
• --resolution W — bump frame width to 1024 px when Claude needs to read on-screen text (slides, terminals, code).
• --fps F — override the auto-fps calculation (still capped at 2 fps).
• --whisper groq|openai — force a specific Whisper backend.
• --no-whisper — disable transcription entirely; frames only.
• --no-dedup — keep near-duplicate frames. By default a frame-delta pass drops frames that are visually near-identical to the one before them (held slides, static screen recordings, paused video), so the frame budget is spent on distinct content; this flag turns that off.
• --out-dir DIR — keep working files somewhere specific (default: auto-generated tmp dir).

Limits

• Long-video accuracy depends on the detail mode. On the capped modes (efficient, default balanced) coverage thins out past ~10 minutes — the frame cap spreads across the whole clip, so the script prints a "sparse scan" warning and you're better off re-running focused with --start/--end. token-burner lifts the cap and keeps every scene-change frame across the full video, so it stays complete on longer clips at the cost of more image tokens. The 10-minute mark is guidance for the capped modes, not a hard ceiling.
• Detail is one dial. Defaults are balanced: scene-aware frames, 2 fps max, 100-frame cap. Use --detail efficient for a fast 50-frame keyframe pass, or --detail token-burner for uncapped scene candidates. Set WATCH_DETAIL in ~/.config/watch/.env to change the default.

Structure

.
├── skills/watch/                 # self-contained skill — copied as a unit by every installer
│   ├── SKILL.md                  # skill contract — the source of truth across all surfaces
│   └── scripts/
│       ├── watch.py              # entry point — orchestrates download → frames → transcript
│       ├── download.py           # yt-dlp wrapper
│       ├── frames.py             # ffmpeg frame extraction + auto-fps logic
│       ├── transcribe.py         # VTT parsing + dedupe + Whisper orchestration
│       ├── whisper.py            # Groq / OpenAI clients (pure stdlib)
│       ├── config.py             # shared config (~/.config/watch/.env)
│       ├── setup.py              # preflight + installer
│       └── build-skill.sh        # build dist/watch.skill for claude.ai upload (dev-only)
├── hooks/                        # SessionStart status hook (Claude Code only)
├── .claude-plugin/               # plugin.json + marketplace.json (Claude Code)
├── .codex-plugin/                # plugin.json — Codex/agents manifest ("skills": "./skills/")
├── .agents/plugins/              # marketplace.json — Agent Skills marketplace listing
├── AGENTS.md → CLAUDE.md         # generic-agent entry point
├── tests/                        # pytest suite (ffmpeg-synthesized clips, no network)
└── .github/workflows/            # release.yml — auto-builds watch.skill on tag push

Develop

# Run the test suite (stdlib + pytest; ffmpeg required for frame tests):
python3 -m pytest -q

# Build the claude.ai upload bundle:
bash skills/watch/scripts/build-skill.sh      # → dist/watch.skill

Releasing: tag vX.Y.Z, push the tag. The workflow builds dist/watch.skill and attaches it to the GitHub release. Keep the version in sync across skills/watch/SKILL.md, .claude-plugin/plugin.json, and .codex-plugin/plugin.json.

See CHANGELOG.md for version history.

Open source

MIT license.

Built on yt-dlp, ffmpeg, and Claude's multimodal Read tool. Whisper transcription via Groq or OpenAI.

Built by Brad Bonanno — I make content about building with AI on YouTube (@bradbonanno), and build AI operating systems for businesses at Solaris Automation. If /watch saves you from scrubbing through a video, come say hi on the channel.

Star History

---

github.com/bradautomates/claude-video · @bradbonanno · Solaris Automation · LICENSE