try to fix gemma messing with other AIs
This commit is contained in:
@@ -107,52 +107,55 @@ Ollama client sends `model: VISION_MODEL`, `think: false`, a JSON Schema in `for
|
||||
and the existing cancellation/timeouts. Its stream is consumed privately; only the
|
||||
final selected chat model's answer and opted-in reasoning reach the user.
|
||||
|
||||
The bounded, validated vision record is:
|
||||
Gemma is the visual perception stage. The user's question guides which visible
|
||||
details it inspects; it does not answer that question. Qwen combines the report
|
||||
with the question, conversation, its own knowledge, and optional Web Search to
|
||||
reason and produce the final answer. The bounded perception record is:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "Concise factual overview",
|
||||
"answer": "Direct visual conclusion relevant to the user's question",
|
||||
"confidence": 0.9,
|
||||
"evidence": ["Visible details supporting the conclusion"],
|
||||
"scene": "Detailed factual overview of what is visible",
|
||||
"subjects": [{"type": "bird", "description": "Fluffy gray down", "location": "center"}],
|
||||
"visualDetails": ["Black head with large white facial patches"],
|
||||
"visibleText": ["Clearly readable text"],
|
||||
"observations": [],
|
||||
"alternatives": [],
|
||||
"spatialRelationships": ["The bird faces the camera"],
|
||||
"colorsAndAppearance": ["Gray body and black head"],
|
||||
"recognizedElements": ["Juvenile penguin"],
|
||||
"notableFeatures": ["Large white facial patches"],
|
||||
"uncertainties": []
|
||||
}
|
||||
```
|
||||
|
||||
Gemma adapts its emphasis to identification, UI debugging, OCR, counting, comparison,
|
||||
or scene understanding. It should give the most likely supported answer, qualified
|
||||
when appropriate, rather than refusing because absolute certainty is impossible.
|
||||
`answer` can be null for genuinely insufficient evidence. `confidence` is a number
|
||||
from 0 to 1 (or null), not a calibrated probability. Alternatives require actual
|
||||
support; empty arrays are preferred to speculative lists or boilerplate limitations.
|
||||
Each field has a length/item limit, with at most eight evidence items and three
|
||||
alternatives. The helper retains its 1,600-token output budget.
|
||||
Gemma reports direct observations and visually recognizable elements, while avoiding
|
||||
semantic conclusions such as a species inferred from markings or an unseen backend
|
||||
cause. Each field has a length/item limit that allows detailed screenshots. The
|
||||
generation cap is 2,400 tokens; simple images can use much less.
|
||||
|
||||
Malformed JSON falls back to a bounded summary plus `rawAnalysis` (up to 6,000
|
||||
characters), with no invented answer or confidence. Valid JSON is parsed before
|
||||
Malformed JSON falls back to a bounded scene plus `rawAnalysis` (up to 6,000
|
||||
characters) as unstructured visual evidence. Valid JSON is parsed before
|
||||
individual fields are bounded, so truncation cannot corrupt its structure.
|
||||
An empty response or unavailable Gemma fails the turn explicitly, without asking the
|
||||
chat model to invent image details. The cache key is attachment, configured vision
|
||||
model, `VISION_PROMPT_VERSION` (currently 2), and trimmed user question; case and
|
||||
model, `VISION_PROMPT_VERSION` (currently 3), and trimmed user question; case and
|
||||
internal whitespace are preserved. Records include the analysis and timestamp.
|
||||
Only an exact matching key is reused. Legacy model-only snapshots are retained for
|
||||
compatibility but never used as cache hits. Bump the prompt version when changing
|
||||
the analysis contract. Cache records cascade away when the attachment is deleted.
|
||||
Only an exact matching key is reused. Version 2 answer-oriented reports remain in
|
||||
storage but cannot be cache hits for version 3. Legacy model-only snapshots are
|
||||
retained for compatibility but never used as cache hits. Bump the prompt version
|
||||
when changing the perception contract. Cache records cascade away when the
|
||||
attachment is deleted.
|
||||
|
||||
Recent-history image context remains limited to the configured image count. A new
|
||||
question, including a follow-up without another upload, analyzes retained images
|
||||
for that question. Repeated questions and regeneration reuse matching results.
|
||||
Different question results coexist, including across shared branches. A new question
|
||||
can therefore require extra Gemma calls, bounded by the existing image count.
|
||||
Images on the current user message are analyzed individually and labeled in their
|
||||
attachment order. Earlier images are selected only for explicit historical references;
|
||||
they do not compete with an unrelated current image. Regeneration uses the pinned
|
||||
original prompt and its attachments. Repeated questions and regeneration reuse
|
||||
matching results. Different question results coexist, including across shared
|
||||
branches, so a new question can require extra Gemma calls.
|
||||
|
||||
The final model receives intact structured evidence in a separate untrusted user
|
||||
context message, not a truncated JSON string in a system prompt. Trusted guidance
|
||||
tells it to answer directly using the answer, confidence and evidence, retain real
|
||||
ambiguity, avoid inventing alternatives, and distinguish visible UI state from unseen
|
||||
backend causes. It should not mention internal tools/models unless explicitly asked.
|
||||
tells Qwen to weigh observed features and visual recognition, perform the final
|
||||
interpretation itself, retain real ambiguity, and distinguish visible UI state from
|
||||
unseen backend causes. It should not mention internal tools/models unless explicitly asked.
|
||||
OCR and raw fallback text are data, never instructions. Analysis remains separate
|
||||
from visible messages; the final model never receives image bytes or filesystem paths.
|
||||
These are model instructions, not a deterministic guarantee of identification accuracy
|
||||
@@ -160,8 +163,10 @@ or final-answer wording. Automated tests cover the contract and data flow; seman
|
||||
quality requires evaluation with real images and the deployed models.
|
||||
|
||||
Web Search and Reasoning remain independent. For requested searches, concise visual
|
||||
evidence helps form the query without overriding the user's search decision. Search
|
||||
finishes before final generation; source ordering/rendering is unchanged. The final
|
||||
evidence from recognized elements, visible text, notable features, and scene helps
|
||||
form the query without overriding the user's search decision. Current images take
|
||||
priority over historical ones. Search finishes before final generation;
|
||||
source ordering/rendering is unchanged. The final
|
||||
request still uses `stream: true` and the existing capability-checked reasoning toggle.
|
||||
Stop/disconnect cancels the current vision or final generation request. Logging includes
|
||||
IDs, sizes, types and cache events only, never images/Base64/private analysis.
|
||||
|
||||
Reference in New Issue
Block a user