add vision analysis service with prompt versioning, response schema, and structured JSON output
This commit is contained in:
@@ -89,7 +89,8 @@ uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at fou
|
||||
that limit. Neither arbitrary paths nor remote image URLs are accepted.
|
||||
|
||||
The additive, idempotent SQLite migration creates `attachments`, `message_attachments`
|
||||
and `upload_deletions`; existing messages are not rewritten. Attachment rows contain
|
||||
and `upload_deletions`, plus the question-specific `attachment_vision_analyses` cache;
|
||||
existing messages and files are not rewritten. Attachment rows contain
|
||||
owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached
|
||||
status, creation time and optional vision model/analysis/time. Shared references preserve
|
||||
images across branches, edits and same-instance imports. Deleting the last referencing
|
||||
@@ -102,7 +103,7 @@ Generation order is attachment ownership validation → atomic user message/refe
|
||||
persistence → **Analyzing images…** → existing Web Search decision/retrieval → final
|
||||
selected chat model streaming. Gemma receives each image separately as a server-side
|
||||
Base64 `messages[].images` value, together with the user's question. The existing
|
||||
Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive
|
||||
Ollama client sends `model: VISION_MODEL`, `think: false`, a JSON Schema in `format`, keep-alive
|
||||
and the existing cancellation/timeouts. Its stream is consumed privately; only the
|
||||
final selected chat model's answer and opted-in reasoning reach the user.
|
||||
|
||||
@@ -110,24 +111,53 @@ The bounded, validated vision record is:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "Description of the image",
|
||||
"summary": "Concise factual overview",
|
||||
"answer": "Direct visual conclusion relevant to the user's question",
|
||||
"confidence": 0.9,
|
||||
"evidence": ["Visible details supporting the conclusion"],
|
||||
"visibleText": ["Clearly readable text"],
|
||||
"objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}],
|
||||
"observations": [],
|
||||
"warnings": [],
|
||||
"alternatives": [],
|
||||
"uncertainties": []
|
||||
}
|
||||
```
|
||||
|
||||
Malformed JSON falls back to a bounded plain-text summary in the same record shape.
|
||||
Gemma adapts its emphasis to identification, UI debugging, OCR, counting, comparison,
|
||||
or scene understanding. It should give the most likely supported answer, qualified
|
||||
when appropriate, rather than refusing because absolute certainty is impossible.
|
||||
`answer` can be null for genuinely insufficient evidence. `confidence` is a number
|
||||
from 0 to 1 (or null), not a calibrated probability. Alternatives require actual
|
||||
support; empty arrays are preferred to speculative lists or boilerplate limitations.
|
||||
Each field has a length/item limit, with at most eight evidence items and three
|
||||
alternatives. The helper retains its 1,600-token output budget.
|
||||
|
||||
Malformed JSON falls back to a bounded summary plus `rawAnalysis` (up to 6,000
|
||||
characters), with no invented answer or confidence. Valid JSON is parsed before
|
||||
individual fields are bounded, so truncation cannot corrupt its structure.
|
||||
An empty response or unavailable Gemma fails the turn explicitly, without asking the
|
||||
chat model to invent image details. Successful analysis is cached by attachment and
|
||||
vision-model name and reused for retries/references; changing the configured vision
|
||||
model reruns analysis on the next direct use. Recent-history image context is limited
|
||||
to the configured image count and 4,000 characters per image. Analysis remains separate
|
||||
from visible message text. The final model receives observation context with explicit
|
||||
instructions to preserve uncertainty and treat OCR/image instructions as untrusted data;
|
||||
it never receives image bytes or filesystem paths.
|
||||
chat model to invent image details. The cache key is attachment, configured vision
|
||||
model, `VISION_PROMPT_VERSION` (currently 2), and trimmed user question; case and
|
||||
internal whitespace are preserved. Records include the analysis and timestamp.
|
||||
Only an exact matching key is reused. Legacy model-only snapshots are retained for
|
||||
compatibility but never used as cache hits. Bump the prompt version when changing
|
||||
the analysis contract. Cache records cascade away when the attachment is deleted.
|
||||
|
||||
Recent-history image context remains limited to the configured image count. A new
|
||||
question, including a follow-up without another upload, analyzes retained images
|
||||
for that question. Repeated questions and regeneration reuse matching results.
|
||||
Different question results coexist, including across shared branches. A new question
|
||||
can therefore require extra Gemma calls, bounded by the existing image count.
|
||||
|
||||
The final model receives intact structured evidence in a separate untrusted user
|
||||
context message, not a truncated JSON string in a system prompt. Trusted guidance
|
||||
tells it to answer directly using the answer, confidence and evidence, retain real
|
||||
ambiguity, avoid inventing alternatives, and distinguish visible UI state from unseen
|
||||
backend causes. It should not mention internal tools/models unless explicitly asked.
|
||||
OCR and raw fallback text are data, never instructions. Analysis remains separate
|
||||
from visible messages; the final model never receives image bytes or filesystem paths.
|
||||
These are model instructions, not a deterministic guarantee of identification accuracy
|
||||
or final-answer wording. Automated tests cover the contract and data flow; semantic
|
||||
quality requires evaluation with real images and the deployed models.
|
||||
|
||||
Web Search and Reasoning remain independent. For requested searches, concise visual
|
||||
evidence helps form the query without overriding the user's search decision. Search
|
||||
|
||||
Reference in New Issue
Block a user