Blog
Borrowed Eyes
“Passed” does not mean “verified”. A test has to be able to fail before its passing counts.
A loop with one link missing
I wanted to build a “product design” capability on DeepSeek Harness, the everything-is-a-plugin agent runtime from my last post, DSH for short. The capability is a loop: start from a design mock, write the implementation, judge how close it looks, fix what’s off, judge again, until it converges.
The deadliest link in that loop is judgment. Without trustworthy judgment, the loop either never stops or stops in the wrong place.
And DeepSeek is a text-only model. It can write the code, but it can see neither the design mock nor the page it just built. The judging link is one it simply cannot do.
So I built a tool: images never enter the model’s context. When the model needs to look, it calls deepseek_vision, and a real vision model looks on its behalf and brings back words.
Wiring on the eye took a fraction of the time. The rest went into one question: is the borrowed eye fit to be a judge? Along the way I crashed three times: each time I thought I had verified something, and had verified nothing.
The pipe is laid, only the valve is shut
Before touching anything I read DSH’s source, and found something interesting: the image pipeline is already there. Images make it in, get recognized, get converted; the whole chain exists. All that’s missing is one admission check: when an image arrives, DSH looks at whether the currently selected model declares image support, and rejects the image outright if it doesn’t. The pipe is laid, only the valve is shut.
That creates a pick-two-of-three dilemma, and the dilemma is dictated by DSH’s own architecture: whether an image gets in depends on one thing only, the declaration of the currently selected model. The check consults that declaration and nothing else. So for “paste an image” to work there are exactly two doors: change the check itself, which means touching the core; or register a new model option that declares image support, which means one more entry in the model picker. Refuse both, and all that’s left is a tool that takes file paths, giving up paste entirely. No core changes, paste works, no extra model option. You get at most two of the three:
| Approach | No core changes | Paste works | No extra model option |
|---|---|---|---|
| Patch the core’s admission check | No | Yes | Yes |
| Register a new model option | Yes | Yes | No |
| Tool only, pass paths or URLs | Yes | No | Yes |
The community has already taken door two: register a model option that declares image support, have a vision model describe the image into text, hand the text to DeepSeek. Zero core changes, very clean. I still took door one.
Why I chose to patch the core
First, a thing that’s easy to conflate: letting the model decide what to ask is not something you buy by patching the core. Door two can have it too: instead of describing the image, it could inject a one-line text pointer and let a plugin register the tool.
What patching the core actually buys is two product judgments.
One, users shouldn’t have to switch models. DSH’s model picker lists which accounts and keys you have; which capabilities you have is a different list. Wedging a “DeepSeek + Vision” entry into it smuggles a capability item into an account list. The user’s intuition is “one key gets me one model”, not “installing a plugin grows a new model option”. That’s a category error.
Two, this core will be changed anyway. Today DSH flat-out rejects images for DeepSeek even though the chain is fully written, which at the product level is hard to defend. Since it has to change sooner or later, my three patches add nothing new; they connect capability that is already written but never opened up.
There was an unexpected bonus: every configured text-only model gets covered at once. I tested with a model that shares zero code with DeepSeek and is likewise text-only: all 7 reading probes came back correct.
One more thing that should be said out loud: I know this is a short-term solution. The capability is lying right there in the core’s code; official support is a matter of time, and the day it ships, this patch set deserves to be replaced. I also accept the cost: upgrades will wipe the patches, upstream edits to those three sites will break the anchors, uninstalling must restore them explicitly. I hedge with backups, one-click rollback and a daily automated check, but it’s a maintenance tax that never stops. Why do it knowing it will be replaced? That answer comes at the end.
The real question: is this eye fit to be a judge
Attaching the eye only makes the model able to see, and able to see and usable for judgment are two different things. A model may read every word on an image and still:
- see but not look: it misses hard defects under open-ended questions
- see but confabulate: it reports “differences” between a mock and itself
- see but wobble: three verdicts from three runs of the same image, so the loop oscillates
- see but mumble: “the colors are inconsistent” gives you nothing to fix in code
So I built an eval: four fixture sets (landing page, dashboard, list, form), each a triple of design mock, faithful implementation, and defect-injected variant, 23 known defects in all: 9 hard, 8 medium, 6 easy. Four pass lines, one per failure mode above. Only a clean sweep qualifies the eye for the production judging loop.
Then I started running it, and all three crashes came out of these runs.
Nothing verified, round one: two rounds on blank images
The first two rounds produced conclusions that were beautiful, and wrong.
The record at the time: three vision models failing in unison on full-page composites, claiming they couldn’t see the image while producing complete analyses anyway, inventing blues precise to the hex code, “#3498db”. The actual design is coral orange on warm white; there isn’t one blue pixel in it. From that I concluded: full-page composites don’t work, sections must be reviewed separately.
Round three added assertions, and the root cause surfaced:
The composite script loaded local images in a way that carries a hidden trap. For security reasons the browser silently blocks local images loaded that way, while the “page finished loading” signal returns success as usual, because blocked requests never count as pending. The “comparison image” being produced was two blank boxes and two lines of English labels, with zero pixels of page content.
Which meant every record from those rounds had to be reread:
| What the record said | What actually happened |
|---|---|
| ”Claims it can’t see, analyses anyway” | The image really was blank — the model was reporting the truth |
| ”Invents nonexistent colors” | Pressed for answers on a blank image, all it could do was guess |
| ”Full page 0/6, must review by section” | The full page was never actually tested |
One model answered identically across all three rounds: “Both panels appear blank with only broken-image icons; no judgment can be made.”
It had diagnosed the problem. My eval records had logged its honesty as hallucination.
That episode changed two habits: composites now embed images directly into the page, with an assertion after generation that the image really has width; and the eval set permanently includes one known-blank image as a negative control: the day it stops being reported blank, something has silently broken again.
I left the two voided rounds in the repo with a line on top: conclusions void. Because the process of misattribution is itself a finding: the “fixtures can silently break” trap comes straight from there.
Nothing verified, round two: one hit logged as a capability
In round two I logged a hit. A hard defect, heading font weight 800 versus 500, and the model answered “the left one is bolder”. Right direction. Scored as a hit.
Round three extended it to 15 runs; 8 were correct.
53% is a coin flip.
The error distribution was the insidious part: errors clustered by round, one whole round correct, the next whole round inverted. Look at any single round and it looks stable.
Into the spec it went: a two-way directional judgment scores only after 3 full runs.
The same batch held a cousin of that trap. My original injection changed font weight 800 to 600. Then I found that in common font stacks, 600, 700, 800 and 900 all render at the same visible weight. An injection with zero pixel effect, which the model naturally “missed”. It became 800 versus 500, plus a rule: after injection, a pixel-level assertion must confirm the defect actually exists.
Nothing verified, round three: testing the installer where it’s already installed
The first two falls were mine. This time I was the one doing the catching.
I’d had Claude build this plugin. When the install script was done, it ran its own end-to-end check and reported back all green, ready to call it a day.
One look and I stopped it: “This test is wrong. The environment has to be a pristine, freshly downloaded DSH.”
The environment it ran in had the patches applied long ago and the config already written. From start to finish the script printed “already patched, skipping” and “already exists, skipping”. Green all the way, and not one line of install logic actually ran.
Rerun in a genuinely clean environment, it caught two real bugs on the spot.
That rule later went into the daily automated check: the installer dry-runs first, and if every target says “already patched”, it counts as a failure, because that kind of green verifies nothing.
Three incidents, one lesson:
“Passed” does not mean “verified”. First prove the test was able to fail.
On blank images, the test couldn’t discover what the models could really do; on a single directional question, it couldn’t tell skill from luck; in an already-installed environment, it never touched a single code path.
The most useful finding: how you ask beats which model you pick
This is the most counterintuitive and most practical conclusion of the whole run.
Two metrics first. “Zero-diff control” pairs the design mock with its faithful implementation: the ideal answer is “no differences”, and any difference reported is a hallucination. “Hard recall” asks whether the hardest defects get caught.
Same model, same images, only the phrasing changes:
| Question form | Zero-diff hallucinations | Hard recall |
|---|---|---|
| ”Find the differences yourself” | 0/30 | 0/2 |
| ”Do they differ in this aspect?“ | 0/30 | 0/2 — a yes/no question, it defaults to no |
| ”Which one is larger?“ | 0/36 | 1/2 — whitespace answered backwards 3 of 3 |
| ”What is the value in each?“ | 0/12 | 2/2 |
Recall walks from 0/2 to 2/2 with hallucinations at zero throughout, and the only thing that changed was the phrasing.
Note the pretty zero in row one: part of it is bought with laziness: lots of “they look consistent”, and saying less means erring less. Recall and hallucination have to be read together; either one alone leads to the opposite conclusion.
The finding has a direct architectural consequence: it closes off the auto-description route.
Door two describes the image the moment it arrives. But the description happens before the question exists. It cannot know what downstream wants verified, so the only question it can ask is row one, “find the differences yourself”. Row one scores 0/2.
For a vision model, how much detail an image gets seen at depends on what you ask it, and every vision-based judgment downstream is built on that level of detail. For a base that will host a pile of downstream capabilities, that matters a great deal.
Precisely because phrasing changes the outcome, I shipped this as tool plus questioning discipline. The model holds the right to ask: it sees a one-line text pointer and composes its own questions for whatever it needs. The discipline of what makes a trustworthy question lives in the skill text, not hard-coded in the tool. The eval forced this design, and in testing it beats fixed templates.
A word on caching. The obvious move is to describe each image once, cache it, save money. But be precise about what gets cached. Caching what was seen is a liability to downstream understanding, and at worst a source of hallucination: a description generated for question A gets reused for question B, drifting further with every reuse. Caching what was asked and what came back is an asset. The two get conflated all the time.
A few more field readings
Direction is trustworthy, magnitude is not. True weights 800 and 500 read back as 800 and 700; true spacing 80 and 25 reads back as 72 and 38. The measured side always gets pulled toward the reference side. Use the eye for “is there a difference, and which way”, never as a ruler. Exact values on the implementation side come straight from the code; the one place a vision model cannot be replaced is the design side, which is pixels with no code behind it.
Image tokens are per image, not per size. On one provider, a large and a small image cost identical input tokens: the image occupies a fixed budget, and the large one simply gets sampled coarser. Corollary: a bigger context window does not fix “can’t see clearly”. Another provider bills by dimensions. Both schemes exist; assume neither.
Side by side is itself information. Splitting a comparison into two separate submissions doubles the cost and performs worse. One fixture went from recognizable side by side to 0/3 split apart: shown a single cropped image, the model read a house icon as “a book”. The pairing provides the anchor for comparison; split it and the anchor is gone.
Thinking off, 113x less output. One reading with chain-of-thought on produced 1,582 tokens of reasoning; with it off, 14, and the identical reading.
Models distrust tools, then take the long way around. In an early version the model would receive the vision result, decline to believe it, and go count pixels with its own scripts. After the “don’t go around the tool” constraint moved from an error message into the tool description and the skill text, the same replayed question went from 14 steps, 240K tokens and 181 seconds to 2 steps, 23K tokens and 16 seconds, with workaround scripting down from 9 calls to zero. For the same constraint, where it lives turned out to matter more than what it said.
What this eval still lacks
If I don’t write this section, everything above about “nothing verified” becomes a performance.
- Ground truth exists for all 23 defects, but the scorer currently wires up only one set’s 6; the other three sets’ descriptors are unfinished and will throw.
- One model row in the comparison table left no raw outputs — aggregates only, nothing to recheck cell by cell.
- The zero-diff hallucination rate ran on one fixture set only, because the other set’s “faithful implementation” isn’t faithful — the differences reported there are mostly real, so hallucination can’t be measured on it.
- The four-row phrasing table above is a record of real runs, but its raw outputs were not preserved; those denominators can’t be rechecked from the repo. What is on record is an independent control — same model, same defects, directed questions 12/12, open-ended missing them — so the direction stands; discount the exact numbers accordingly.
Beyond those gaps, the unedited raw model output behind every number is in the repo. The reason is simple: this project’s claim is that it seriously studied when the judge can be trusted, and a claim nobody can recheck is an empty one.
When it should retire
The day DeepSeek officially opens up image input.
Nothing needs changing that day: the tool will find that its caller can see for itself, refuse its own service, and step aside; images take the native path. The three patches restore with one command. That is half the answer to the account from earlier: the patch layer was born to be replaced.
The other half is that only the patch layer retires. DSH is an open base; anyone’s model can plug in. And models walking toward AGI on pure text, strong reasoning with no eyes, have been around all along and are not going away. An open gate solves “can the image get in”; it does not solve “can the model see”. A text-only model handed image bytes is still blind. As long as such models exist, the job of borrowing an eye, and knowing when that eye can be trusted, does not expire.
Patches get used up and swapped out. The tool and the eval are what I want to keep.
The project is open source at dsh-design-qa; fixtures, the scorer and raw model outputs live under eval/.