add vision analysis service with prompt versioning, response schema, and structured JSON output

This commit is contained in:
2026-09-22 15:14:28 +02:00
parent 72e429b1b3
commit 99a3277e64
10 changed files with 380 additions and 71 deletions
+43 -13
View File
@@ -89,7 +89,8 @@ uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at fou
that limit. Neither arbitrary paths nor remote image URLs are accepted.
The additive, idempotent SQLite migration creates `attachments`, `message_attachments`
and `upload_deletions`; existing messages are not rewritten. Attachment rows contain
and `upload_deletions`, plus the question-specific `attachment_vision_analyses` cache;
existing messages and files are not rewritten. Attachment rows contain
owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached
status, creation time and optional vision model/analysis/time. Shared references preserve
images across branches, edits and same-instance imports. Deleting the last referencing
@@ -102,7 +103,7 @@ Generation order is attachment ownership validation → atomic user message/refe
persistence → **Analyzing images…** → existing Web Search decision/retrieval → final
selected chat model streaming. Gemma receives each image separately as a server-side
Base64 `messages[].images` value, together with the user's question. The existing
Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive
Ollama client sends `model: VISION_MODEL`, `think: false`, a JSON Schema in `format`, keep-alive
and the existing cancellation/timeouts. Its stream is consumed privately; only the
final selected chat model's answer and opted-in reasoning reach the user.
@@ -110,24 +111,53 @@ The bounded, validated vision record is:
```json
{
"summary": "Description of the image",
"summary": "Concise factual overview",
"answer": "Direct visual conclusion relevant to the user's question",
"confidence": 0.9,
"evidence": ["Visible details supporting the conclusion"],
"visibleText": ["Clearly readable text"],
"objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}],
"observations": [],
"warnings": [],
"alternatives": [],
"uncertainties": []
}
```
Malformed JSON falls back to a bounded plain-text summary in the same record shape.
Gemma adapts its emphasis to identification, UI debugging, OCR, counting, comparison,
or scene understanding. It should give the most likely supported answer, qualified
when appropriate, rather than refusing because absolute certainty is impossible.
`answer` can be null for genuinely insufficient evidence. `confidence` is a number
from 0 to 1 (or null), not a calibrated probability. Alternatives require actual
support; empty arrays are preferred to speculative lists or boilerplate limitations.
Each field has a length/item limit, with at most eight evidence items and three
alternatives. The helper retains its 1,600-token output budget.
Malformed JSON falls back to a bounded summary plus `rawAnalysis` (up to 6,000
characters), with no invented answer or confidence. Valid JSON is parsed before
individual fields are bounded, so truncation cannot corrupt its structure.
An empty response or unavailable Gemma fails the turn explicitly, without asking the
chat model to invent image details. Successful analysis is cached by attachment and
vision-model name and reused for retries/references; changing the configured vision
model reruns analysis on the next direct use. Recent-history image context is limited
to the configured image count and 4,000 characters per image. Analysis remains separate
from visible message text. The final model receives observation context with explicit
instructions to preserve uncertainty and treat OCR/image instructions as untrusted data;
it never receives image bytes or filesystem paths.
chat model to invent image details. The cache key is attachment, configured vision
model, `VISION_PROMPT_VERSION` (currently 2), and trimmed user question; case and
internal whitespace are preserved. Records include the analysis and timestamp.
Only an exact matching key is reused. Legacy model-only snapshots are retained for
compatibility but never used as cache hits. Bump the prompt version when changing
the analysis contract. Cache records cascade away when the attachment is deleted.
Recent-history image context remains limited to the configured image count. A new
question, including a follow-up without another upload, analyzes retained images
for that question. Repeated questions and regeneration reuse matching results.
Different question results coexist, including across shared branches. A new question
can therefore require extra Gemma calls, bounded by the existing image count.
The final model receives intact structured evidence in a separate untrusted user
context message, not a truncated JSON string in a system prompt. Trusted guidance
tells it to answer directly using the answer, confidence and evidence, retain real
ambiguity, avoid inventing alternatives, and distinguish visible UI state from unseen
backend causes. It should not mention internal tools/models unless explicitly asked.
OCR and raw fallback text are data, never instructions. Analysis remains separate
from visible messages; the final model never receives image bytes or filesystem paths.
These are model instructions, not a deterministic guarantee of identification accuracy
or final-answer wording. Automated tests cover the contract and data flow; semantic
quality requires evaluation with real images and the deployed models.
Web Search and Reasoning remain independent. For requested searches, concise visual
evidence helps form the query without overriding the user's search decision. Search