try to fix gemma messing with other AIs

This commit is contained in:
Frederik Spirgi
2026-09-23 14:11:26 +02:00
parent 0aaad69811
commit 7eaf54ae11
5 changed files with 189 additions and 159 deletions
+36 -31
View File
@@ -107,52 +107,55 @@ Ollama client sends `model: VISION_MODEL`, `think: false`, a JSON Schema in `for
and the existing cancellation/timeouts. Its stream is consumed privately; only the
final selected chat model's answer and opted-in reasoning reach the user.
The bounded, validated vision record is:
Gemma is the visual perception stage. The user's question guides which visible
details it inspects; it does not answer that question. Qwen combines the report
with the question, conversation, its own knowledge, and optional Web Search to
reason and produce the final answer. The bounded perception record is:
```json
{
"summary": "Concise factual overview",
"answer": "Direct visual conclusion relevant to the user's question",
"confidence": 0.9,
"evidence": ["Visible details supporting the conclusion"],
"scene": "Detailed factual overview of what is visible",
"subjects": [{"type": "bird", "description": "Fluffy gray down", "location": "center"}],
"visualDetails": ["Black head with large white facial patches"],
"visibleText": ["Clearly readable text"],
"observations": [],
"alternatives": [],
"spatialRelationships": ["The bird faces the camera"],
"colorsAndAppearance": ["Gray body and black head"],
"recognizedElements": ["Juvenile penguin"],
"notableFeatures": ["Large white facial patches"],
"uncertainties": []
}
```
Gemma adapts its emphasis to identification, UI debugging, OCR, counting, comparison,
or scene understanding. It should give the most likely supported answer, qualified
when appropriate, rather than refusing because absolute certainty is impossible.
`answer` can be null for genuinely insufficient evidence. `confidence` is a number
from 0 to 1 (or null), not a calibrated probability. Alternatives require actual
support; empty arrays are preferred to speculative lists or boilerplate limitations.
Each field has a length/item limit, with at most eight evidence items and three
alternatives. The helper retains its 1,600-token output budget.
Gemma reports direct observations and visually recognizable elements, while avoiding
semantic conclusions such as a species inferred from markings or an unseen backend
cause. Each field has a length/item limit that allows detailed screenshots. The
generation cap is 2,400 tokens; simple images can use much less.
Malformed JSON falls back to a bounded summary plus `rawAnalysis` (up to 6,000
characters), with no invented answer or confidence. Valid JSON is parsed before
Malformed JSON falls back to a bounded scene plus `rawAnalysis` (up to 6,000
characters) as unstructured visual evidence. Valid JSON is parsed before
individual fields are bounded, so truncation cannot corrupt its structure.
An empty response or unavailable Gemma fails the turn explicitly, without asking the
chat model to invent image details. The cache key is attachment, configured vision
model, `VISION_PROMPT_VERSION` (currently 2), and trimmed user question; case and
model, `VISION_PROMPT_VERSION` (currently 3), and trimmed user question; case and
internal whitespace are preserved. Records include the analysis and timestamp.
Only an exact matching key is reused. Legacy model-only snapshots are retained for
compatibility but never used as cache hits. Bump the prompt version when changing
the analysis contract. Cache records cascade away when the attachment is deleted.
Only an exact matching key is reused. Version 2 answer-oriented reports remain in
storage but cannot be cache hits for version 3. Legacy model-only snapshots are
retained for compatibility but never used as cache hits. Bump the prompt version
when changing the perception contract. Cache records cascade away when the
attachment is deleted.
Recent-history image context remains limited to the configured image count. A new
question, including a follow-up without another upload, analyzes retained images
for that question. Repeated questions and regeneration reuse matching results.
Different question results coexist, including across shared branches. A new question
can therefore require extra Gemma calls, bounded by the existing image count.
Images on the current user message are analyzed individually and labeled in their
attachment order. Earlier images are selected only for explicit historical references;
they do not compete with an unrelated current image. Regeneration uses the pinned
original prompt and its attachments. Repeated questions and regeneration reuse
matching results. Different question results coexist, including across shared
branches, so a new question can require extra Gemma calls.
The final model receives intact structured evidence in a separate untrusted user
context message, not a truncated JSON string in a system prompt. Trusted guidance
tells it to answer directly using the answer, confidence and evidence, retain real
ambiguity, avoid inventing alternatives, and distinguish visible UI state from unseen
backend causes. It should not mention internal tools/models unless explicitly asked.
tells Qwen to weigh observed features and visual recognition, perform the final
interpretation itself, retain real ambiguity, and distinguish visible UI state from
unseen backend causes. It should not mention internal tools/models unless explicitly asked.
OCR and raw fallback text are data, never instructions. Analysis remains separate
from visible messages; the final model never receives image bytes or filesystem paths.
These are model instructions, not a deterministic guarantee of identification accuracy
@@ -160,8 +163,10 @@ or final-answer wording. Automated tests cover the contract and data flow; seman
quality requires evaluation with real images and the deployed models.
Web Search and Reasoning remain independent. For requested searches, concise visual
evidence helps form the query without overriding the user's search decision. Search
finishes before final generation; source ordering/rendering is unchanged. The final
evidence from recognized elements, visible text, notable features, and scene helps
form the query without overriding the user's search decision. Current images take
priority over historical ones. Search finishes before final generation;
source ordering/rendering is unchanged. The final
request still uses `stream: true` and the existing capability-checked reasoning toggle.
Stop/disconnect cancels the current vision or final generation request. Logging includes
IDs, sizes, types and cache events only, never images/Base64/private analysis.