refine(live): skip screenshot upload when no annotations present

Without comments or strokes the screenshot is pure visual anchoring,
biasing the model toward the existing rendering and fighting the
three-distinct-directions brief. Local blob still drives the shader
overlay; upload and screenshotPath are gated on annotation presence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Paul Bakaus
2026-04-23 08:19:00 -07:00
co-authored by Claude Opus 4.7
parent d60de0fb97
commit 6846a135ce
24 changed files with 108 additions and 24 deletions
+3 -1
View File
@@ -58,7 +58,9 @@ Speed matters — the user is watching a spinner. Minimize tool calls by using t
### 1. Read the screenshot (if present)
When the browser captured the element successfully, `event.screenshotPath` is an absolute path to a PNG showing the element as rendered, including any comment pins and drawn strokes the user placed before Go. **Read it before planning.** Annotations encode user intent not recoverable from `element.outerHTML` alone.
`event.screenshotPath` is **only sent when the user placed at least one comment or stroke before Go.** When present, it's an absolute path to a PNG of the element as rendered with the annotations baked in. **Read it before planning** — annotations encode user intent not recoverable from `element.outerHTML` alone.
When `screenshotPath` is absent, don't ask for one and don't go looking for the current rendering. The omission is deliberate: without annotations, a screenshot would anchor the model on the existing design and fight the three-distinct-directions brief. Work from `element.outerHTML`, the computed styles in `event.element`, and the freeform prompt if present.
`event.comments` and `event.strokes` carry structured metadata alongside the visual. Treat the screenshot as primary; use the structured data for specifics worth quoting (e.g. the exact text of a comment).
@@ -2548,7 +2548,12 @@
if (blob && state === 'GENERATING') {
showShaderOverlay(el, blob, rect);
}
if (blob) {
// Only upload + forward the screenshot when annotations (comments/strokes)
// are present. Without annotations the image is pure visual anchoring —
// it biases the model toward the current rendering and works against the
// three-distinct-directions brief.
const hasAnnotations = snapshot && (snapshot.comments.length > 0 || snapshot.strokes.length > 0);
if (blob && hasAnnotations) {
try {
const uploadRes = await fetch(
'http://localhost:' + PORT + '/annotation?token=' + encodeURIComponent(TOKEN) +