01 — What this isA film is only half picture
Turn off the sound on any film you love and watch two minutes of it. It goes flat. The punch stops hurting. The room stops being a room. About half of what you felt was never in the picture at all, it was in your ears, and you never noticed because good sound is invisible.
Sound design is the job of deciding what a picture should sound like. Not recording it, not mixing it, just deciding. That decision comes first and everything else follows it.
The examples here all come from one short film, The Brain Brake. Two minutes, fifty frames, mostly drawn in pencil. Fifty frames is a lot of deciding, so we built a helper.
02 — The trapYour brain fills in things that aren't there
Most people look at that runner and hear a crowd. A stadium. Feet on a track. Wind.
Now look again. There is no stadium. There is no track, no crowd, no ground, no sky. There is a man and there is blank cream paper. Everything else you heard, you added yourself, because you have seen a thousand sports films and your brain helpfully supplied the rest.
That is the single most important idea in this guide, so here it is in one line:
Blank paper is not an empty background waiting to be filled. Blank paper is a decision the artist made, and the sound has to respect it. So the real answer to frame 1.1 is almost nothing. One breath. A man gasping, close and dry, with no room around him at all. Then quiet.
That takes nerve. Silence feels like you forgot to do your job. It isn't. It is the job.
03 — The helperShow it a picture, it writes a list
Our tool is called soundscape. You give it one frame. It looks at the picture with an AI that can actually see, and it hands back a list of sounds that belong there.
The clever part is the middle box. Before the AI is allowed to look at the picture, we tell it the rules of this particular film. Blank paper means silence. Name what is missing, not just what is there. Never invent an object you cannot see. Without those rules it hands back a stadium. With them, it hands back a breath.
04 — Four kinds of soundEverything fits in one of these
Every sound in every film you have ever seen is one of four things. Once you know them you cannot un-know them, and you will start sorting sounds automatically for the rest of your life. Sorry.
| Layer | What it is | In this film |
|---|---|---|
| Bed | The constant hum under a scene. Air, a room, a distant motorway. | Rare. Most frames are on blank paper, so there is no room to hum. |
| Spot | One thing happening at one moment. A door, a gasp, a switch. | The workhorse. Almost every cue in this film is a spot. |
| Move | A whoosh that carries you from one shot into the next. | Used on jumps and cuts. Easy to overdo. |
| Music | The score. Also the decision to have none. | Sherlock Holmes, Star Wars, Mission Impossible. Adventurous. |
05 — The one rule to keepThe machine suggests. You decide.
The tool never writes into the real cue sheet. It writes into a separate file marked suggestions. A human being copies the good ones across by hand.
This is not caution for its own sake. On frame 1.1 the AI called that runner's sound "dry fabric rustle". There is fabric in the frame, so it is not wrong. It is just the wrong choice, because the first sound of a film about a human being under strain has to be the human being.
The machine is a fast assistant with no taste. You are slow and you have taste. That is a good trade, and it is why the two files stay separate.
01 — OrientationWhat you are operating
soundscape.py takes a frame id, fetches that frame's picture from the film repository, sends it to a vision model wrapped in a prompt that encodes this film's rules, and stores the structured answer. It generates no audio. It produces a brief: bed, spot, move and music, each cue carrying both a library search term and a text-to-sound prompt.
That split matters. Every cue arrives with two routes: search a stock library for it, or generate it. You choose per cue.
Set up once per session
export VISION_API_KEY=your-gemini-key
export VISION_ENDPOINT=https://generativelanguage.googleapis.com/v1beta/openai/chat/completions
export VISION_MODEL=gemini-3.7-flash
Then
python3 tools/soundscape.py --frame 1.1 # one frame python3 tools/soundscape.py --scene 3 # a whole scene python3 tools/soundscape.py --frame 1.1 --dry-run # print the prompt, spend nothing python3 tools/soundscape.py --report # rebuild the readable page python3 tools/soundscape.py --reconcile # check briefs against the current cut
--dry-run is the one to learn first. It shows you exactly what will be sent and costs nothing. Read that prompt once, all the way through. Everything the tool believes about the film is in it.
02 — Where the intelligence livesNot in the model. In the prompt.
A general vision model is trained on photographs. Point it at a pencil drawing of a runner and it recognises "runner", reaches for everything that usually accompanies runners, and returns a soundscape for a stadium that does not exist.
So the prompt states the film's law before the model is allowed to look:
1. Describe ONLY what is in the picture. If the drawing stops and the paper is blank, the world stops there too. 2. Name what is ABSENT as explicitly as what is present. 3. This frame is DRAWN / LIVE / STRIP / BOOTH. [world description] 4. The running time is fixed. Sound may never ask for another frame. 5. The mix is judged on a phone speaker.
It works. Tested on 18 August against Gemini and Groq, both models correctly reported that frame 1.1 has no stadium, no crowd and no track, and that the absence means silence. That was not the predicted result. The rules are load-bearing, so do not trim them to save tokens.
03 — The four worldsThe same object sounds different depending on what it is made of
Each frame is tagged with which world it belongs to, and the prompt swaps in a different paragraph accordingly. This is the highest-leverage part of the design.
| Layer | Sound world |
|---|---|
| DRAWN | Pencil on cream paper. Nothing was ever photographed. Dry, small, sitting under the paper texture. Blank paper is silence, not room tone. |
| LIVE | A photographed human composited into the drawn world. The only place real air, real breath and real reverb belong. The contrast is the point. |
| STRIP | Comic panel, Coach Brain. A voice inside the skull. Close, enclosed, never shouted. |
| BOOTH | The endcard. A sheet of paper writing itself. Paper and graphite, nothing else. |
04 — Reading a briefAnd knowing when to argue with it
Each cue comes back with a name, a reason, a library search term, a generator prompt, and a level in dB. Spots also carry at_s, seconds from the start of that frame.
Here is the real machine output for 1.1, unedited, next to what a human wrote by eye before any model saw the frame:
| Proposed | Verdict | |
|---|---|---|
| Gemini | Dry ragged open-mouth panting, 0.2 s, −14 dB | Matches the human note almost exactly |
| Groq | Dry fabric rustle, 0.5 s, −18 dB | Not wrong. Wrong choice. |
| Human | One exhausted breath, close, dry, no reverb, 0:00 | — |
Groq saw a man with his mouth open gasping and proposed the sound of his shirt. Both observations are true. Only one is the first sound of the film.
05 — Nothing is overwritten, ever
Suggestions go to assets/spotting/suggestions.json. Decisions live in assets/spotting/spotting.csv. The tool writes only to the first and cannot touch the second.
Re-briefing a frame does not replace the old brief either. The new one becomes current and the old one is pushed onto superseded, in order, with its original timestamp. You can always read what the machine said last week and why you disagreed.
And when the picture changes, --reconcile compares every stored brief against the current cut. A frame that no longer exists is reported as orphaned, never deleted. A cue that now lands past its frame's out point is reported as overrun. This matters more than it sounds: between v3 and v4 of this film, forty frames got shorter. A cue written at 0.6 seconds into a frame that used to run 0.72 and now runs 0.40 did not move. The frame moved out from under it. Nothing raises an error. Only reconciling finds it.
06 — Things that will bite you
- 403 with "error code: 1010" is not a dead key. Groq and Hume sit behind Cloudflare, which rejects the default Python user agent. Ten perfectly valid keys once looked dead for this reason. Check the header before you touch the keys.
- Groq returns HTTP 413 on images for eight of the ten keys in the ring, regardless of size. Two work. Downscale frames to roughly 180 KB.
- Qwen thinks out loud. Set
reasoning_effort: "none"or the reasoning eats the token budget and the JSON never arrives. Groq accepts onlynoneordefaultfor that field. - Two frames have no picture. 4.4b and 4.8 were shot on 18 August. Until they are in the repo the tool cannot read them, and the test suite names them deliberately so that it goes red when they land.
01 — PositionWhy this is not a video-to-audio model
The obvious architecture for this problem is end-to-end video-to-audio: encode frames, detect visual onsets, force audio transients onto those frames, generate with a latent diffusion model conditioned on the visual embedding. That is roughly what the current generation of tools does, and it is the wrong instrument here.
Those models are trained on audio-visual corpora of photographed reality. Their learned prior maps "runner" to the acoustic environment that statistically accompanies runners. On a film whose defining formal choice is that the environment is not drawn, that prior is not merely unhelpful, it is precisely inverted. The model's confidence is highest exactly where the film is most deliberate.
So the system splits the problem. The vision model does perception and is constrained by an explicit normative frame. Selection and generation stay with the human and with tools the human drives. The artefact passed between them is a brief, in a schema, with both a retrieval key and a generation key per cue. The machine reads. It does not choose.
02 — The prompt as specification
The law block is not prompt decoration, it is the specification, and it is ordered adversarially. The film's own text, dialogue lines and camera notes, reaches the prompt as data and could in principle contain instructions. Therefore the law is stated before any film-derived text appears, film text is confined below a THE FRAME marker, and the final sentence of the prompt is ours, not the film's. Test 3 asserts all four properties by index comparison, and asserts that the prompt's last token is "not in the picture."
There is also a duration-conditional branch. Below 0.5 s a frame cannot hold its own sound event, so the prompt swaps in language telling the model that returning empty lists is a correct answer. Roughly a third of this film's frames are under 0.5 s. Both sides of that boundary are tested at 0.49, 0.50 and 0.51.
03 — Failure modes found by running it for real
Two defects surfaced only against live models, and neither would have appeared against hand-written fixtures.
The reasoning-block mine
Qwen emits <think> before answering. On a truncated response the parser's first-brace-to-last-brace scan lifted a JSON fragment out of the model's own deliberation and returned a syntactically valid, semantically empty brief. It reported success. A confidently wrong answer is worse than no answer, and this one was invisible: the frame appeared briefed. Fixed by stripping terminated <think> blocks and refusing unterminated ones outright.
The empty brief
A response with no reading and no cues in any layer now raises. Silence is a legitimate and frequent decision on this film, but it must be stated, not merely absent, because absence is indistinguishable from truncation. This was a deliberate behaviour change; prior expectations asserting {} was valid were updated and annotated in place rather than deleted, so the change is visible in the diff.
error code: 1010 to a default Python user agent, on every endpoint, with valid credentials. This presents as total key failure. The client now sends a browser agent, and test 3 stands up a local socket to confirm the header is on the wire rather than merely in the source.04 — Test discipline
Four tests, and the property that matters is that each can fail while the other three pass. Verified by targeted sabotage, each producing exactly one red column.
| Sabotage | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| _num stops stripping units from "−30 dB" | RED | · | · | · |
| picture base points at the wrong folder | · | RED | · | · |
| read deadline removed from the vision call | · | · | RED | · |
| merge overwrites instead of superseding | · | · | · | RED |
| reconcile silently drops orphans | · | · | · | RED |
Test 1 is the mechanism alone. Test 2 drives the real CLI against the real fifty-frame film over the real network, and takes its expectations from measurement: the strongest assertion in it compares bytes received against GitHub's own Content-Length header, a number this process does not control.
Test 3 is the ugly cases, and its centrepiece is a socket that accepts the connection and then says nothing for ever. A call that neither resolves nor rejects reaches no catch handler, because nothing is ever raised. Every wait on anything outside the process carries a deadline and the deadline has its own test, including the crueller variant where headers arrive and the body never does.
Test 4 is the upgrade, run against the film's real v3 to v4 transition: 45 frames to 50, four retired, nine new, forty shortened, the whole cut 20 frames tighter. Suggestions written legally against v3 are checked for orphaning and overrun against v4. Expectations are measured from the v3 file, never assumed.
tools/spotting.py 56 / 32 / 34 / 38 passed, 0 failed tools/soundscape.py 102 / 197 / 70 / 61 passed, 0 failed
05 — A test that skipped its own broken cases
Worth recording, because it passed for the wrong reason for two sessions.
The picture fetcher assumed all fifty frames lived in assets/V7/. Measured against the repository tree, they do not: 39 are in assets/V7, 5 in boards/v5, 3 in assets/V3A, 1 in boards/v4. Nine frames were unreadable.
Test 2 did not catch it, because the test contained if not name.startswith("V7_"): continue. It skipped precisely the frames that were broken and reported green. The fetcher now tries each known folder, treating 404 as "keep looking" and any other status as a network fault worth raising at once, and the test checks all fifty with no exemptions.
06 — What remains unproven
- Only four of fifty frames have been briefed by a live model. Nothing above is a claim about the other forty-six.
- Frame 6.9 was never completed on Groq. Rate limited on every key. Its comparison column is empty.
- The STRIP world was tested on 6.1 only, and Groq failed that frame, so the comic-panel comparison is Gemini against the human alone, with no third reading.
- Hume was exercised for one line, one direction, one generation. Nothing about consistency across takes has been measured.
- Nothing has been mixed, heard, or checked on a phone speaker. Every judgement in this guide concerns text describing sound, not sound. That gap is the entire remaining project.