improve reasoning toggle tests and refactor model capability checks

2026-09-22 10:30:03 +02:00
parent a8e348e0c7
commit 3b6c8969cc
15 changed files with 201 additions and 23 deletions
@@ -46,6 +46,101 @@ Node.js 22 or newer is required.
 - `RATE_LIMIT_CLEANUP_INTERVAL_MS`: interval for pruning stale in-memory rate-limit buckets.
 - `DROP_LEGACY_CHATS`: when `true`, migrate any legacy `chats` table into `chat_sessions` / `chat_messages` and remove the old table.

+## Image attachments and vision
+
+The composer supports PNG, JPEG and WebP through the image button, dropping files onto
+the composer, or pasting a screenshot. Text paste is unchanged. Previews show upload
+status and support retry/removal; sending waits until every remaining image is ready.
+Images also render in saved messages and open at full size in a protected browser tab.
+
+Production configuration (the existing `/app/data` volume remains unchanged):
+
+```dotenv
+UPLOAD_DIR=/app/data/uploads
+UPLOAD_MAX_SIZE_MB=10
+UPLOAD_MAX_FILES=5
+UPLOAD_MAX_PIXELS=30000000
+VISION_MODEL=gemma3:4b
+```
+
+Outside production, the default directory is `data/uploads`; `UPLOAD_DIR` overrides
+it. Startup creates the directory and checks write permission, failing clearly rather
+than falling back to another location. Files use `YYYY/MM/DD/<uuid>.<format>` keys.
+Do not publish this directory as a static directory. Back up **both the SQLite database
+and uploads directory**. JSON workspace exports contain attachment references only;
+imports reconnect existing images owned by the same account, not portable image bytes.
+
+Authenticated API endpoints:
+
+- `GET /uploads/config`: safe client limits, without storage paths.
+- `POST /uploads`: multipart field `image`, one file per request; returns
+  `{id,type:"image",originalName,mimeType,sizeBytes,width,height}`.
+- `GET /attachments/:id`: ownership-checked image bytes with private/no-store caching
+  and `nosniff`; attached images also require access to a referencing conversation.
+- `DELETE /attachments/:id`: removes an owned temporary upload only.
+- `/chat` and `/chat/stream`: optional `attachments: [id, ...]`; image-only messages
+  receive a default description prompt. Existing text-only payloads remain valid.
+
+Multer bounds multipart input in memory; Sharp detects the actual format, checks the
+pixel limit and fully decodes it. Browser MIME types and filenames are not trusted.
+SVG, GIF, animated/multipage images, empty, malformed, oversized and fake images are
+rejected. File counts are checked again during atomic message association. Concurrent
+uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at four times
+that limit. Neither arbitrary paths nor remote image URLs are accepted.
+
+The additive, idempotent SQLite migration creates `attachments`, `message_attachments`
+and `upload_deletions`; existing messages are not rewritten. Attachment rows contain
+owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached
+status, creation time and optional vision model/analysis/time. Shared references preserve
+images across branches, edits and same-instance imports. Deleting the last referencing
+message/conversation (or the account) queues the file for deletion. Cleanup drains this
+durable queue at startup, after modifying requests, before uploads and hourly; failed
+file deletion is retried. Temporary uploads expire after 24 hours. Attached images do
+not expire. No binary or Base64 data is stored in SQLite.
+
+Generation order is attachment ownership validation → atomic user message/reference
+persistence → **Analyzing images…** → existing Web Search decision/retrieval → final
+selected chat model streaming. Gemma receives each image separately as a server-side
+Base64 `messages[].images` value, together with the user's question. The existing
+Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive
+and the existing cancellation/timeouts. Its stream is consumed privately; only the
+final selected chat model's answer and opted-in reasoning reach the user.
+
+The bounded, validated vision record is:
+
+```json
+{
+  "summary": "Description of the image",
+  "visibleText": ["Clearly readable text"],
+  "objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}],
+  "observations": [],
+  "warnings": [],
+  "uncertainties": []
+}
+```
+
+Malformed JSON falls back to a bounded plain-text summary in the same record shape.
+An empty response or unavailable Gemma fails the turn explicitly, without asking the
+chat model to invent image details. Successful analysis is cached by attachment and
+vision-model name and reused for retries/references; changing the configured vision
+model reruns analysis on the next direct use. Recent-history image context is limited
+to the configured image count and 4,000 characters per image. Analysis remains separate
+from visible message text. The final model receives observation context with explicit
+instructions to preserve uncertainty and treat OCR/image instructions as untrusted data;
+it never receives image bytes or filesystem paths.
+
+Web Search and Reasoning remain independent. For requested searches, concise visual
+evidence helps form the query without overriding the user's search decision. Search
+finishes before final generation; source ordering/rendering is unchanged. The final
+request still uses `stream: true` and the existing capability-checked reasoning toggle.
+Stop/disconnect cancels the current vision or final generation request. Logging includes
+IDs, sizes, types and cache events only, never images/Base64/private analysis.
+
+Dependencies added: `multer` (bounded multipart handling), `sharp` (content detection,
+dimensions and full image decoding). Install through the normal `npm install`/`npm ci`
+workflow, including Sharp's platform-specific optional dependencies in Docker. The
+application does not pull Ollama models; provision `gemma3:4b` separately.
+
 ## Model Backend Expectations

 The server is configured around an Ollama-compatible API surface: