try to fix gemma messing with other AIs

This commit is contained in:
Frederik Spirgi
2026-09-23 14:40:32 +02:00
parent 67f7d47128
commit 85e6d0c957
5 changed files with 125 additions and 38 deletions
+19 -15
View File
@@ -107,7 +107,7 @@ Ollama client sends `model: VISION_MODEL`, `think: false`, a JSON Schema in `for
and the existing cancellation/timeouts. Its stream is consumed privately; only the
final selected chat model's answer and opted-in reasoning reach the user.
Gemma is the visual perception stage. The user's question guides which visible
The configured vision model is the visual perception stage. The user's question guides which visible
details it inspects; it does not answer that question. Qwen combines the report
with the question, conversation, its own knowledge, and optional Web Search to
reason and produce the final answer. The bounded perception record is:
@@ -116,30 +116,33 @@ reason and produce the final answer. The bounded perception record is:
{
"scene": "Detailed factual overview of what is visible",
"subjects": [{"type": "bird", "description": "Fluffy gray down", "location": "center"}],
"visualDetails": ["Black head with large white facial patches"],
"visualDetails": ["The crown is black", "A narrow dark line runs directly beneath the chin"],
"visibleText": ["Clearly readable text"],
"spatialRelationships": ["The bird faces the camera"],
"colorsAndAppearance": ["Gray body and black head"],
"recognizedElements": ["Juvenile penguin"],
"notableFeatures": ["Large white facial patches"],
"notableFeatures": ["Narrow dark line beneath the chin"],
"uncertainties": []
}
```
Gemma reports direct observations and visually recognizable elements, while avoiding
semantic conclusions such as a species inferred from markings or an unseen backend
cause. Each field has a length/item limit that allows detailed screenshots. The
The vision model reports direct observations and visually recognizable elements,
while avoiding semantic conclusions such as a species inferred from markings or an
unseen backend cause. The prompt asks for precise location, geometry, boundaries,
and named regions where visible, with separate entries for distinguishing markings.
Uncertainty concerns what can be seen, not the final identification. Each field has
a length/item limit that allows detailed screenshots. The
generation cap is 2,400 tokens; simple images can use much less.
Malformed JSON falls back to a bounded scene plus `rawAnalysis` (up to 6,000
characters) as unstructured visual evidence. Valid JSON is parsed before
individual fields are bounded, so truncation cannot corrupt its structure.
An empty response or unavailable Gemma fails the turn explicitly, without asking the
An empty response or unavailable vision model fails the turn explicitly, without asking the
chat model to invent image details. The cache key is attachment, configured vision
model, `VISION_PROMPT_VERSION` (currently 3), and trimmed user question; case and
model, `VISION_PROMPT_VERSION` (currently 4), and trimmed user question; case and
internal whitespace are preserved. Records include the analysis and timestamp.
Only an exact matching key is reused. Version 2 answer-oriented reports remain in
storage but cannot be cache hits for version 3. Legacy model-only snapshots are
Only an exact matching key is reused. Earlier prompt-version reports remain in
storage but cannot be cache hits for version 4. Legacy model-only snapshots are
retained for compatibility but never used as cache hits. Bump the prompt version
when changing the perception contract. Cache records cascade away when the
attachment is deleted.
@@ -149,13 +152,14 @@ attachment order. Earlier images are selected only for explicit historical refer
they do not compete with an unrelated current image. Regeneration uses the pinned
original prompt and its attachments. Repeated questions and regeneration reuse
matching results. Different question results coexist, including across shared
branches, so a new question can require extra Gemma calls.
branches, so a new question can require extra vision-model calls.
The final model receives intact structured evidence in a separate untrusted user
context message, not a truncated JSON string in a system prompt. Trusted guidance
tells Qwen to weigh observed features and visual recognition, perform the final
interpretation itself, retain real ambiguity, and distinguish visible UI state from
unseen backend causes. It should not mention internal tools/models unless explicitly asked.
tells Qwen to weigh discriminating observed features, compare plausible candidates,
and keep observed details distinct from inferred traits. It must not sharpen vague
descriptions into specific markings or invent a backend cause. It should not mention
internal tools/models unless explicitly asked.
OCR and raw fallback text are data, never instructions. Analysis remains separate
from visible messages; the final model never receives image bytes or filesystem paths.
These are model instructions, not a deterministic guarantee of identification accuracy
@@ -163,7 +167,7 @@ or final-answer wording. Automated tests cover the contract and data flow; seman
quality requires evaluation with real images and the deployed models.
Web Search and Reasoning remain independent. For requested searches, concise visual
evidence from recognized elements, visible text, notable features, and scene helps
evidence from notable features, visual details, visible text, and scene helps
form the query without overriding the user's search decision. Current images take
priority over historical ones. Search finishes before final generation;
source ordering/rendering is unchanged. The final