improve reasoning toggle tests and refactor model capability checks

This commit is contained in:
2026-09-22 10:30:03 +02:00
parent a8e348e0c7
commit 3b6c8969cc
15 changed files with 201 additions and 23 deletions
+95
View File
@@ -46,6 +46,101 @@ Node.js 22 or newer is required.
- `RATE_LIMIT_CLEANUP_INTERVAL_MS`: interval for pruning stale in-memory rate-limit buckets. - `RATE_LIMIT_CLEANUP_INTERVAL_MS`: interval for pruning stale in-memory rate-limit buckets.
- `DROP_LEGACY_CHATS`: when `true`, migrate any legacy `chats` table into `chat_sessions` / `chat_messages` and remove the old table. - `DROP_LEGACY_CHATS`: when `true`, migrate any legacy `chats` table into `chat_sessions` / `chat_messages` and remove the old table.
## Image attachments and vision
The composer supports PNG, JPEG and WebP through the image button, dropping files onto
the composer, or pasting a screenshot. Text paste is unchanged. Previews show upload
status and support retry/removal; sending waits until every remaining image is ready.
Images also render in saved messages and open at full size in a protected browser tab.
Production configuration (the existing `/app/data` volume remains unchanged):
```dotenv
UPLOAD_DIR=/app/data/uploads
UPLOAD_MAX_SIZE_MB=10
UPLOAD_MAX_FILES=5
UPLOAD_MAX_PIXELS=30000000
VISION_MODEL=gemma3:4b
```
Outside production, the default directory is `data/uploads`; `UPLOAD_DIR` overrides
it. Startup creates the directory and checks write permission, failing clearly rather
than falling back to another location. Files use `YYYY/MM/DD/<uuid>.<format>` keys.
Do not publish this directory as a static directory. Back up **both the SQLite database
and uploads directory**. JSON workspace exports contain attachment references only;
imports reconnect existing images owned by the same account, not portable image bytes.
Authenticated API endpoints:
- `GET /uploads/config`: safe client limits, without storage paths.
- `POST /uploads`: multipart field `image`, one file per request; returns
`{id,type:"image",originalName,mimeType,sizeBytes,width,height}`.
- `GET /attachments/:id`: ownership-checked image bytes with private/no-store caching
and `nosniff`; attached images also require access to a referencing conversation.
- `DELETE /attachments/:id`: removes an owned temporary upload only.
- `/chat` and `/chat/stream`: optional `attachments: [id, ...]`; image-only messages
receive a default description prompt. Existing text-only payloads remain valid.
Multer bounds multipart input in memory; Sharp detects the actual format, checks the
pixel limit and fully decodes it. Browser MIME types and filenames are not trusted.
SVG, GIF, animated/multipage images, empty, malformed, oversized and fake images are
rejected. File counts are checked again during atomic message association. Concurrent
uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at four times
that limit. Neither arbitrary paths nor remote image URLs are accepted.
The additive, idempotent SQLite migration creates `attachments`, `message_attachments`
and `upload_deletions`; existing messages are not rewritten. Attachment rows contain
owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached
status, creation time and optional vision model/analysis/time. Shared references preserve
images across branches, edits and same-instance imports. Deleting the last referencing
message/conversation (or the account) queues the file for deletion. Cleanup drains this
durable queue at startup, after modifying requests, before uploads and hourly; failed
file deletion is retried. Temporary uploads expire after 24 hours. Attached images do
not expire. No binary or Base64 data is stored in SQLite.
Generation order is attachment ownership validation → atomic user message/reference
persistence → **Analyzing images…** → existing Web Search decision/retrieval → final
selected chat model streaming. Gemma receives each image separately as a server-side
Base64 `messages[].images` value, together with the user's question. The existing
Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive
and the existing cancellation/timeouts. Its stream is consumed privately; only the
final selected chat model's answer and opted-in reasoning reach the user.
The bounded, validated vision record is:
```json
{
"summary": "Description of the image",
"visibleText": ["Clearly readable text"],
"objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}],
"observations": [],
"warnings": [],
"uncertainties": []
}
```
Malformed JSON falls back to a bounded plain-text summary in the same record shape.
An empty response or unavailable Gemma fails the turn explicitly, without asking the
chat model to invent image details. Successful analysis is cached by attachment and
vision-model name and reused for retries/references; changing the configured vision
model reruns analysis on the next direct use. Recent-history image context is limited
to the configured image count and 4,000 characters per image. Analysis remains separate
from visible message text. The final model receives observation context with explicit
instructions to preserve uncertainty and treat OCR/image instructions as untrusted data;
it never receives image bytes or filesystem paths.
Web Search and Reasoning remain independent. For requested searches, concise visual
evidence helps form the query without overriding the user's search decision. Search
finishes before final generation; source ordering/rendering is unchanged. The final
request still uses `stream: true` and the existing capability-checked reasoning toggle.
Stop/disconnect cancels the current vision or final generation request. Logging includes
IDs, sizes, types and cache events only, never images/Base64/private analysis.
Dependencies added: `multer` (bounded multipart handling), `sharp` (content detection,
dimensions and full image decoding). Install through the normal `npm install`/`npm ci`
workflow, including Sharp's platform-specific optional dependencies in Docker. The
application does not pull Ollama models; provision `gemma3:4b` separately.
## Model Backend Expectations ## Model Backend Expectations
The server is configured around an Ollama-compatible API surface: The server is configured around an Ollama-compatible API surface:
+3 -1
View File
@@ -18,7 +18,9 @@
"body-parser": "^1.20.2", "body-parser": "^1.20.2",
"express": "^4.19.2", "express": "^4.19.2",
"express-session": "^1.17.3", "express-session": "^1.17.3",
"multer": "^2.4.0",
"qs": "^6.15.0", "qs": "^6.15.0",
"session-file-store": "^1.5.0" "session-file-store": "^1.5.0",
"sharp": "^0.35.4"
} }
} }
+5
View File
@@ -196,7 +196,12 @@
<div class="input-meta"> <div class="input-meta">
<p class="composer-hint">Enter to send · Shift+Enter for new line</p> <p class="composer-hint">Enter to send · Shift+Enter for new line</p>
</div> </div>
<div id="uploadPreviews" class="upload-previews" aria-live="polite" hidden></div>
<div class="input-container"> <div class="input-container">
<input id="imagePicker" type="file" accept="image/png,image/jpeg,image/webp" multiple hidden>
<button id="attachImage" class="icon-button" type="button" aria-label="Attach image" title="Attach image">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><rect x="3" y="3" width="18" height="18" rx="3"/><circle cx="8" cy="8" r="1.5"/><path d="m3 17 5-5 4 4 4-6 5 7"/></svg>
</button>
<button id="webSearch" class="icon-button" type="button" aria-label="Search the web" title="Search the web" aria-pressed="false"> <button id="webSearch" class="icon-button" type="button" aria-label="Search the web" title="Search the web" aria-pressed="false">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><circle cx="12" cy="12" r="9"/><ellipse cx="12" cy="12" rx="4" ry="9"/><path d="M3 12h18"/></svg> <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><circle cx="12" cy="12" r="9"/><ellipse cx="12" cy="12" rx="4" ry="9"/><path d="M3 12h18"/></svg>
</button> </button>
+15 -6
View File
@@ -1,5 +1,6 @@
import {del, get, post, put, stream} from "./app/api.js?v=20260909-account-controls"; import {del, get, post, put, stream} from "./app/api.js?v=20260909-account-controls";
import {appendSources, createWebSearchToggle} from "./app/webSearch.js"; import {appendSources, createWebSearchToggle} from "./app/webSearch.js";
import {appendAttachments, createAttachmentComposer} from "./app/attachments.js";
import {createReasoningToggle, finishReasoning, updateReasoning} from "./app/reasoning.js?v=20260921-reasoning-capability"; import {createReasoningToggle, finishReasoning, updateReasoning} from "./app/reasoning.js?v=20260921-reasoning-capability";
import {closeChatDrawer, openChatDrawer} from "./app/chatDrawer.js?v=20260909-account-controls"; import {closeChatDrawer, openChatDrawer} from "./app/chatDrawer.js?v=20260909-account-controls";
import {onboardingPrompts, requestedChatId} from "./app/constants.js?v=20260909-account-controls"; import {onboardingPrompts, requestedChatId} from "./app/constants.js?v=20260909-account-controls";
@@ -109,6 +110,7 @@ const webSearchToggle = createWebSearchToggle(webSearchButton);
const reasoningToggle = createReasoningToggle(document.getElementById("reasoning")); const reasoningToggle = createReasoningToggle(document.getElementById("reasoning"));
let isProcessing = false, activeChatId = null, editingPersonaId = null, editingPersonaType = "assistant", currentUsername = ""; let isProcessing = false, activeChatId = null, editingPersonaId = null, editingPersonaType = "assistant", currentUsername = "";
const attachmentComposer = createAttachmentComposer({onChange: updateSendState, onError: message => setNotice(message, "error")});
let chatSessions = [], currentMessages = [], assistantPersonas = [], userPersonas = []; let chatSessions = [], currentMessages = [], assistantPersonas = [], userPersonas = [];
let activeUserPersonaId = null, currentSummary = null, publishedPersonaIds = new Set(); let activeUserPersonaId = null, currentSummary = null, publishedPersonaIds = new Set();
let popupResolver = null, popupMode = null, popupLastFocus = null; let popupResolver = null, popupMode = null, popupLastFocus = null;
@@ -329,7 +331,7 @@ function updateComposerPlaceholder() {
function updateSendState() { function updateSendState() {
const hasText = Boolean(msgInput.value.trim()); const hasText = Boolean(msgInput.value.trim());
sendBtn.disabled = isProcessing ? false : !hasText; sendBtn.disabled = isProcessing ? false : attachmentComposer.pending || (!hasText && !attachmentComposer.attachments.length);
sendBtn.setAttribute("aria-label", isProcessing ? "Stop generation" : "Send message"); sendBtn.setAttribute("aria-label", isProcessing ? "Stop generation" : "Send message");
} }
@@ -742,7 +744,9 @@ function addMessage(contentOrMessage, isUser = false, isLoading = false, options
actions.append(prevBtn, indexBadge, nextBtn); actions.append(prevBtn, indexBadge, nextBtn);
} }
msgDiv.append(header, body); msgDiv.append(header);
appendAttachments(msgDiv, message.attachments);
msgDiv.append(body);
if (!isMessageUser) updateReasoning(msgDiv, message.thinking); if (!isMessageUser) updateReasoning(msgDiv, message.thinking);
if (!isMessageUser) appendSources(msgDiv, message.sources); if (!isMessageUser) appendSources(msgDiv, message.sources);
if (message.deliveryStatus === "failed" && message.errorMessage) { if (message.deliveryStatus === "failed" && message.errorMessage) {
@@ -1364,6 +1368,7 @@ function setLoadingState(loading, overlayOptions = null) {
msgInput.disabled = loading; msgInput.disabled = loading;
webSearchButton.disabled = loading; webSearchButton.disabled = loading;
reasoningToggle.setBusy(loading); reasoningToggle.setBusy(loading);
attachmentComposer.setBusy(loading);
sendBtn.classList.toggle("loading", loading); sendBtn.classList.toggle("loading", loading);
messagesDiv.querySelectorAll('[data-message-action="regenerate"]').forEach((button) => { messagesDiv.querySelectorAll('[data-message-action="regenerate"]').forEach((button) => {
button.disabled = loading || button.dataset.retryLimitReached === "true"; button.disabled = loading || button.dataset.retryLimitReached === "true";
@@ -1811,7 +1816,9 @@ async function checkSession() {
async function sendMessage() { async function sendMessage() {
if (isProcessing) return; if (isProcessing) return;
const message = msgInput.value.trim(); if (attachmentComposer.pending) return;
const attachments = attachmentComposer.attachments;
const message = msgInput.value.trim() || (attachments.length ? "Describe the attached images." : "");
if (!message) return; if (!message) return;
const webSearch = webSearchToggle.enabled; const webSearch = webSearchToggle.enabled;
const reasoning = reasoningToggle.enabled; const reasoning = reasoningToggle.enabled;
@@ -1821,7 +1828,8 @@ async function sendMessage() {
const autoTitle = message.length > 40 ? `${message.slice(0, 40)}...` : message; const autoTitle = message.length > 40 ? `${message.slice(0, 40)}...` : message;
currentChat.title = autoTitle; updateWorkspaceCopy(); renderChatList(); await put(`/chats/${activeChatId}`, {title: autoTitle}); currentChat.title = autoTitle; updateWorkspaceCopy(); renderChatList(); await put(`/chats/${activeChatId}`, {title: autoTitle});
} }
currentMessages.push({role: "user", content: message}); addMessage(message, true); updateChatActionState(); const userMessage = {role: "user", content: message, attachments};
currentMessages.push(userMessage); addMessage(userMessage, true); updateChatActionState();
msgInput.value = ""; msgInput.value = "";
resizeComposerInput(); resizeComposerInput();
setLoadingState(true, false); setNotice("Generating reply..."); setLoadingState(true, false); setNotice("Generating reply...");
@@ -1832,14 +1840,15 @@ async function sendMessage() {
loadingLabel.textContent = "Preparing response..."; loadingLabel.textContent = "Preparing response...";
try { try {
let streamError = null; let streamError = null;
await stream("/chat/stream", {message, model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, { await stream("/chat/stream", {message, attachments: attachments.map(item => item.id), model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, {
onStatus: ({status, sourceCount, searchState}) => { onStatus: ({status, sourceCount, searchState}) => {
if (status === "accepted") { if (status === "accepted") {
attachmentComposer.clear();
webSearchToggle.reset(); webSearchToggle.reset();
reasoningToggle.reset(); reasoningToggle.reset();
return; return;
} }
loadingLabel.textContent = status === "searching" ? "Searching the web..." loadingLabel.textContent = status === "analyzing" ? "Analyzing images..." : status === "searching" ? "Searching the web..."
: searchState === "no_relevant_results" ? "No sufficiently relevant results. Generating response..." : searchState === "no_relevant_results" ? "No sufficiently relevant results. Generating response..."
: sourceCount === 0 ? "No web results found. Generating response..." : "Generating response..."; : sourceCount === 0 ? "No web results found. Generating response..." : "Generating response...";
}, },
+12 -1
View File
@@ -456,7 +456,18 @@ input:focus, textarea:focus, select:focus { border-color: var(--brand); }
.input-meta, .input-container { width: min(940px, 100%); margin-inline: auto; } .input-meta, .input-container { width: min(940px, 100%); margin-inline: auto; }
.input-meta { margin-bottom: 5px; padding-left: 10px; } .input-meta { margin-bottom: 5px; padding-left: 10px; }
.composer-hint { margin: 0; color: var(--text-muted); font-size: .72rem; } .composer-hint { margin: 0; color: var(--text-muted); font-size: .72rem; }
.input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); } .input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "attach search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); }
#attachImage { grid-area: attach; }
.upload-previews { width: min(940px, 100%); margin: 0 auto 8px; display: flex; gap: 8px; flex-wrap: wrap; }
.upload-previews[hidden] { display: none; }
.upload-preview { display: flex; align-items: center; gap: 8px; padding: 6px; border: 1px solid var(--border-strong); border-radius: 12px; max-width: 100%; }
.upload-preview span { overflow-wrap: anywhere; font-size: 12px; }
.upload-preview img { width: 48px; height: 48px; object-fit: contain; border-radius: 6px; }
.message-images { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 10px; }
.message-images img { max-width: min(360px, 100%); max-height: 300px; object-fit: contain; border-radius: 12px; }
.message-images a { max-width: 100%; }
.image-drag-target { outline: 2px solid currentColor; }
@media (max-width: 480px) { .input-container { grid-template-columns: 36px 36px 36px minmax(0, 1fr) 36px; gap: 3px; } }
#webSearch, #reasoning { width: 44px; height: 44px; display: grid; place-items: center; border-radius: 17px; } #webSearch, #reasoning { width: 44px; height: 44px; display: grid; place-items: center; border-radius: 17px; }
#webSearch { grid-area: search; } #webSearch { grid-area: search; }
#reasoning { grid-area: reasoning; } #reasoning { grid-area: reasoning; }
+5
View File
@@ -1,6 +1,11 @@
import {execFileSync} from "node:child_process"; import {execFileSync} from "node:child_process";
const files = [ const files = [
"src/db/attachments.js",
"src/services/localFileStorage.js",
"src/services/attachmentService.js",
"src/routes/uploads.js",
"public/app/attachments.js",
"server.js", "server.js",
"src/app.js", "src/app.js",
"src/pageLayout.js", "src/pageLayout.js",
+11 -1
View File
@@ -16,6 +16,8 @@ import {createPagesRouter} from "./routes/pages.js";
import {createPersonasRouter} from "./routes/personas.js"; import {createPersonasRouter} from "./routes/personas.js";
import {createSettingsRouter} from "./routes/settings.js"; import {createSettingsRouter} from "./routes/settings.js";
import {createSystemRouter} from "./routes/system.js"; import {createSystemRouter} from "./routes/system.js";
import {createAttachmentService} from "./services/attachmentService.js";
import {createUploadsRouter} from "./routes/uploads.js";
export function createCloseHandler(resources) { export function createCloseHandler(resources) {
let closed = false; let closed = false;
@@ -245,13 +247,15 @@ export function createApp(options = {}) {
const db = createDatabase(config); const db = createDatabase(config);
const repositories = createRepositories(db, config); const repositories = createRepositories(db, config);
const modelService = options.modelService || createModelService(config, options.modelOverrides); const modelService = options.modelService || createModelService(config, options.modelOverrides);
const chatService = createChatService(repositories, modelService, config); const attachmentService = createAttachmentService(db, config, modelService);
const chatService = createChatService(repositories, modelService, config, undefined, attachmentService);
const FileStore = FileStoreFactory(session); const FileStore = FileStoreFactory(session);
const authRateLimiters = createAuthRateLimiters(config); const authRateLimiters = createAuthRateLimiters(config);
const chatRateLimiters = createChatRateLimiters(config); const chatRateLimiters = createChatRateLimiters(config);
closeHandlers.push( closeHandlers.push(
...authRateLimiters.map((limiter) => limiter.close), ...authRateLimiters.map((limiter) => limiter.close),
...chatRateLimiters.map((limiter) => limiter.close), ...chatRateLimiters.map((limiter) => limiter.close),
() => attachmentService.close(),
() => db.close() () => db.close()
); );
@@ -271,11 +275,16 @@ export function createApp(options = {}) {
})); }));
app.use(sameOriginProtection(config)); app.use(sameOriginProtection(config));
app.use(enforceSessionVersion(repositories)); app.use(enforceSessionVersion(repositories));
app.use((req, res, next) => {
if (["DELETE", "POST"].includes(req.method)) res.on("finish", () => { void attachmentService.cleanup(); });
next();
});
app.locals.config = config; app.locals.config = config;
app.locals.db = db; app.locals.db = db;
app.locals.repositories = repositories; app.locals.repositories = repositories;
app.locals.modelService = modelService; app.locals.modelService = modelService;
app.locals.attachmentService = attachmentService;
app.locals.observability = observability; app.locals.observability = observability;
app.locals.close = createCloseHandler(closeHandlers); app.locals.close = createCloseHandler(closeHandlers);
@@ -290,6 +299,7 @@ export function createApp(options = {}) {
fallthrough: config.staticFallthrough fallthrough: config.staticFallthrough
})); }));
app.use(createAuthRouter({repositories, authRateLimiters, config})); app.use(createAuthRouter({repositories, authRateLimiters, config}));
app.use(createUploadsRouter(attachmentService, config));
app.use(createSettingsRouter({repositories})); app.use(createSettingsRouter({repositories}));
app.use(createChatsRouter({repositories, chatService, modelService, config, chatRateLimiters})); app.use(createChatsRouter({repositories, chatService, modelService, config, chatRateLimiters}));
app.use(createPersonasRouter({repositories, chatService, modelService, config})); app.use(createPersonasRouter({repositories, chatService, modelService, config}));
+5
View File
@@ -29,6 +29,11 @@ export function createConfig(overrides = {}) {
rootDir, rootDir,
publicDir: path.resolve(rootDir, "public"), publicDir: path.resolve(rootDir, "public"),
dataDir, dataDir,
uploadDir: process.env.UPLOAD_DIR || (process.env.NODE_ENV === "production" ? "/app/data/uploads" : path.resolve(dataDir, "uploads")),
uploadMaxSizeMb: Math.max(1, int(process.env.UPLOAD_MAX_SIZE_MB, 10)),
uploadMaxFiles: Math.max(1, int(process.env.UPLOAD_MAX_FILES, 5)),
uploadMaxPixels: Math.max(1, int(process.env.UPLOAD_MAX_PIXELS, 30000000)),
visionModel: process.env.VISION_MODEL || "gemma3:4b",
sessionsDir, sessionsDir,
port: int(process.env.PORT, 3000), port: int(process.env.PORT, 3000),
dbPath: process.env.DB_PATH || path.resolve(dataDir, "app.db"), dbPath: process.env.DB_PATH || path.resolve(dataDir, "app.db"),
+2
View File
@@ -1,6 +1,7 @@
import fs from "fs"; import fs from "fs";
import Database from "better-sqlite3"; import Database from "better-sqlite3";
import {initializeSchema, runMigrations} from "./schema.js"; import {initializeSchema, runMigrations} from "./schema.js";
import {initializeAttachments} from "./attachments.js";
export function createDatabase(config) { export function createDatabase(config) {
if (config.dbPath !== ":memory:") { if (config.dbPath !== ":memory:") {
@@ -10,5 +11,6 @@ export function createDatabase(config) {
const db = new Database(config.dbPath); const db = new Database(config.dbPath);
initializeSchema(db); initializeSchema(db);
runMigrations(db, config); runMigrations(db, config);
initializeAttachments(db);
return db; return db;
} }
+8
View File
@@ -1,3 +1,5 @@
import {attachmentMetadata} from "./attachments.js";
const parseJsonArray = (value) => { const parseJsonArray = (value) => {
if (!value) return []; if (!value) return [];
try { try {
@@ -869,6 +871,11 @@ export function createRepositories(db, config) {
if (message.sourceId != null) importedMessageIds.set(message.sourceId, Number(result.lastInsertRowid)); if (message.sourceId != null) importedMessageIds.set(message.sourceId, Number(result.lastInsertRowid));
if (message.sources) statements.updateChatMessageSources.run(JSON.stringify(message.sources), result.lastInsertRowid, chatId); if (message.sources) statements.updateChatMessageSources.run(JSON.stringify(message.sources), result.lastInsertRowid, chatId);
if (message.thinking) statements.updateChatMessageThinking.run(message.thinking, result.lastInsertRowid, chatId); if (message.thinking) statements.updateChatMessageThinking.run(message.thinking, result.lastInsertRowid, chatId);
for (const [position, attachment] of (message.attachments || []).slice(0, config.uploadMaxFiles || 5).entries()) {
// JSON backups contain references, not files. Only restore existing, owned images.
db.prepare("INSERT OR IGNORE INTO message_attachments(message_id,attachment_id,position) SELECT ?,id,? FROM attachments WHERE id=? AND username=? AND status='attached'")
.run(result.lastInsertRowid, position, attachment.id, username);
}
} }
statements.updateChatGenerationSettings.run( statements.updateChatGenerationSettings.run(
chat.preferred_model ?? null, chat.preferred_model ?? null,
@@ -1271,6 +1278,7 @@ export function createRepositories(db, config) {
persona_state: this.getPersonaChatState(chat.id, username), persona_state: this.getPersonaChatState(chat.id, username),
messages: this.listChatMessages(chat.id).map((message) => ({ messages: this.listChatMessages(chat.id).map((message) => ({
...message, ...message,
attachments: db.prepare("SELECT a.* FROM attachments a JOIN message_attachments m ON a.id=m.attachment_id WHERE m.message_id=? ORDER BY m.position").all(message.id).map(attachmentMetadata),
retryVariants: parseJsonArray(message.retry_variants) retryVariants: parseJsonArray(message.retry_variants)
})) }))
})), })),
+6 -2
View File
@@ -17,13 +17,15 @@ const messageValidator = (body) => {
}; };
const sendMessageValidator = (body) => { const sendMessageValidator = (body) => {
const message = String(body?.message || "").trim(); const attachments = body?.attachments ?? [];
if (!Array.isArray(attachments) || attachments.some(id => typeof id !== "string" || !/^[a-f0-9-]{36}$/.test(id))) return {error: "Invalid attachment IDs"};
const message = String(body?.message || "").trim() || (attachments.length ? "Describe the attached images." : "");
const chatId = Number(body?.chatId); const chatId = Number(body?.chatId);
if (!chatId) return {error: "chatId is required"}; if (!chatId) return {error: "chatId is required"};
if (!message) return {error: "message is required"}; if (!message) return {error: "message is required"};
if (body.webSearch !== undefined && typeof body.webSearch !== "boolean") return {error: "webSearch must be a boolean"}; if (body.webSearch !== undefined && typeof body.webSearch !== "boolean") return {error: "webSearch must be a boolean"};
if (body.reasoning !== undefined && typeof body.reasoning !== "boolean") return {error: "reasoning must be a boolean"}; if (body.reasoning !== undefined && typeof body.reasoning !== "boolean") return {error: "reasoning must be a boolean"};
return {value: {message, chatId, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}}; return {value: {message, chatId, attachments, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}};
}; };
const organizationValidator = (body) => ({ const organizationValidator = (body) => ({
@@ -413,6 +415,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf
user: req.session.user, user: req.session.user,
chatId: req.validatedBody.chatId, chatId: req.validatedBody.chatId,
message: req.validatedBody.message, message: req.validatedBody.message,
attachments: req.validatedBody.attachments,
model: req.validatedBody.model, model: req.validatedBody.model,
webSearch: req.validatedBody.webSearch, webSearch: req.validatedBody.webSearch,
reasoning: req.validatedBody.reasoning reasoning: req.validatedBody.reasoning
@@ -449,6 +452,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf
user: req.session.user, user: req.session.user,
chatId: req.validatedBody.chatId, chatId: req.validatedBody.chatId,
message: req.validatedBody.message, message: req.validatedBody.message,
attachments: req.validatedBody.attachments,
model: req.validatedBody.model, model: req.validatedBody.model,
onChunk: (chunk, fullReply) => sendEvent("chunk", {chunk, fullReply}), onChunk: (chunk, fullReply) => sendEvent("chunk", {chunk, fullReply}),
onThinking: (chunk, fullThinking) => sendEvent("thinking", {chunk, fullThinking}), onThinking: (chunk, fullThinking) => sendEvent("thinking", {chunk, fullThinking}),
+25 -8
View File
@@ -12,7 +12,7 @@ import {
buildMemoryPrompt, buildMemoryPrompt,
buildPersonaStatePrompt, buildPersonaStatePrompt,
clampRetryActiveIndex, clampRetryActiveIndex,
formatChatMessage, formatChatMessage as formatBaseChatMessage,
formatMessagesForSummary, formatMessagesForSummary,
parseSources, parseSources,
retryStylePrompts retryStylePrompts
@@ -21,7 +21,12 @@ import {
import {SearXNGSearchProvider, WebSearchError, buildWebSearchContext, retrieveWebSearch} from "./webSearchService.js"; import {SearXNGSearchProvider, WebSearchError, buildWebSearchContext, retrieveWebSearch} from "./webSearchService.js";
import {supportsReasoning} from "../../public/app/modelCapabilities.js"; import {supportsReasoning} from "../../public/app/modelCapabilities.js";
export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config)) { export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config), attachmentService = null) {
const formatChatMessage = row => {
const message = formatBaseChatMessage(row);
const attachments = row && attachmentService?.list(row.id);
return message && attachments?.length ? {...message, attachments} : message;
};
const chatGenerationQueues = new Map(); const chatGenerationQueues = new Map();
const summaryQueues = new Map(); const summaryQueues = new Map();
const activeGenerations = new Map(); const activeGenerations = new Map();
@@ -272,6 +277,8 @@ export function createChatService(repositories, modelService, config, webSearchP
content: entry.content content: entry.content
})); }));
messagesPayload.push(...conversationMessages.slice(0, -1)); messagesPayload.push(...conversationMessages.slice(0, -1));
const visionContext = attachmentService?.context(history);
if (visionContext) messagesPayload.push(visionContext);
if (webSearchContext) messagesPayload.push(webSearchContext[0]); if (webSearchContext) messagesPayload.push(webSearchContext[0]);
if (webSearchContext?.[1]) messagesPayload.push(webSearchContext[1]); if (webSearchContext?.[1]) messagesPayload.push(webSearchContext[1]);
if (conversationMessages.length) messagesPayload.push(conversationMessages.at(-1)); if (conversationMessages.length) messagesPayload.push(conversationMessages.at(-1));
@@ -307,6 +314,7 @@ export function createChatService(repositories, modelService, config, webSearchP
return {fullReply: opener, thinking, promptMessageId: null}; return {fullReply: opener, thinking, promptMessageId: null};
} }
await attachmentService?.analyze(promptMessage.id, promptMessage.content, signal);
const {messagesPayload} = buildConversationContext({ const {messagesPayload} = buildConversationContext({
user, user,
chatId, chatId,
@@ -327,7 +335,7 @@ export function createChatService(repositories, modelService, config, webSearchP
return {fullReply, thinking, promptMessageId: promptMessage.id}; return {fullReply, thinking, promptMessageId: promptMessage.id};
}; };
const generateChatTurn = ({user, chatId, message, model, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) => const generateChatTurn = ({user, chatId, message, model, attachments = [], reuseChatId = null, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) =>
withChatGenerationLock(chatId, async () => { withChatGenerationLock(chatId, async () => {
await (summaryQueues.get(chatId) || Promise.resolve()); await (summaryQueues.get(chatId) || Promise.resolve());
const session = repositories.getChat(chatId, user); const session = repositories.getChat(chatId, user);
@@ -346,7 +354,8 @@ export function createChatService(repositories, modelService, config, webSearchP
const {model: effectiveModel, generation} = getGenerationSettings(session, model); const {model: effectiveModel, generation} = getGenerationSettings(session, model);
const useReasoning = reasoning === true && supportsReasoning(effectiveModel); const useReasoning = reasoning === true && supportsReasoning(effectiveModel);
signal?.throwIfAborted(); signal?.throwIfAborted();
const inserted = repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId); const insert = () => repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId);
const inserted = attachmentService ? attachmentService.createMessage(attachments, user, chatId, insert, reuseChatId) : insert();
const userMessageId = Number(inserted?.lastInsertRowid); const userMessageId = Number(inserted?.lastInsertRowid);
try { try {
let sources = null; let sources = null;
@@ -361,8 +370,13 @@ export function createChatService(repositories, modelService, config, webSearchP
const fullReply = await withActiveGeneration(chatId, signal, async (activeSignal) => { const fullReply = await withActiveGeneration(chatId, signal, async (activeSignal) => {
activeSignal.throwIfAborted(); activeSignal.throwIfAborted();
onStatus?.({status: "accepted"}); onStatus?.({status: "accepted"});
if (attachments.length) {
onStatus?.({status: "analyzing"});
await attachmentService.analyze(userMessageId, message, activeSignal);
}
if (webSearch) { if (webSearch) {
const search = await retrieveWebSearch(message, webSearchProvider, { const search = await retrieveWebSearch(message, webSearchProvider, {
referenceText: attachments.length ? attachmentService.searchContext(userMessageId) : "",
signal: activeSignal, signal: activeSignal,
onSearching: () => onStatus?.({status: "searching"}) onSearching: () => onStatus?.({status: "searching"})
}); });
@@ -489,6 +503,7 @@ export function createChatService(repositories, modelService, config, webSearchP
parentMessageId parentMessageId
); );
parentMessageId = Number(inserted.lastInsertRowid); parentMessageId = Number(inserted.lastInsertRowid);
attachmentService?.copy(sourceMessage.id, parentMessageId);
if (sourceMessage.sources) repositories.updateChatMessageSources(clonedChat.id, parentMessageId, parseSources(sourceMessage.sources)); if (sourceMessage.sources) repositories.updateChatMessageSources(clonedChat.id, parentMessageId, parseSources(sourceMessage.sources));
if (sourceMessage.thinking) repositories.updateChatMessageThinking(clonedChat.id, parentMessageId, sourceMessage.thinking); if (sourceMessage.thinking) repositories.updateChatMessageThinking(clonedChat.id, parentMessageId, sourceMessage.thinking);
} }
@@ -515,11 +530,11 @@ export function createChatService(repositories, modelService, config, webSearchP
containsUserVoiceInRoleplayOpener, containsUserVoiceInRoleplayOpener,
persistChatMessageRetryState, persistChatMessageRetryState,
buildMessagesPayload: buildConversationContext, buildMessagesPayload: buildConversationContext,
async sendChatMessage({user, chatId, message, model, webSearch = false, reasoning = false}) { async sendChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false}) {
return generateChatTurn({user, chatId, message, model, webSearch, reasoning}); return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning});
}, },
async streamChatMessage({user, chatId, message, model, webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) { async streamChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) {
return generateChatTurn({user, chatId, message, model, webSearch, reasoning, onStatus, onChunk, onThinking, signal}); return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning, onStatus, onChunk, onThinking, signal});
}, },
async retryMessage({user, chatId, selectedModel, targetMessage, retryStyle = null}) { async retryMessage({user, chatId, selectedModel, targetMessage, retryStyle = null}) {
return withChatGenerationLock(chatId, async () => { return withChatGenerationLock(chatId, async () => {
@@ -574,6 +589,8 @@ export function createChatService(repositories, modelService, config, webSearchP
const generated = await generateChatTurn({ const generated = await generateChatTurn({
user, user,
chatId: cloned.chat.id, chatId: cloned.chat.id,
attachments: attachmentService?.list(targetMessage.id).map(item => item.id) || [],
reuseChatId: chatId,
message: content, message: content,
model, model,
webSearch, webSearch,
+3 -1
View File
@@ -1,4 +1,5 @@
import {supportsReasoning} from "../../public/app/modelCapabilities.js"; import {supportsReasoning} from "../../public/app/modelCapabilities.js";
import {AttachmentError} from "./attachmentService.js";
class ModelServiceError extends Error { class ModelServiceError extends Error {
constructor(code, message, status = 500) { constructor(code, message, status = 500) {
@@ -196,7 +197,7 @@ export function createModelService(config, overrides = {}) {
const response = await fetchImpl(`${config.modelApiBaseUrl}/chat`, { const response = await fetchImpl(`${config.modelApiBaseUrl}/chat`, {
method: "POST", method: "POST",
headers: {"Content-Type": "application/json"}, headers: {"Content-Type": "application/json"},
body: JSON.stringify(buildRequestBody(model, messagesPayload, options.generation, options.reasoning)), body: JSON.stringify({...buildRequestBody(model, messagesPayload, options.generation, options.reasoning), ...(options.format ? {format: options.format} : {})}),
signal signal
}); });
@@ -332,6 +333,7 @@ export function createModelService(config, overrides = {}) {
}; };
}, },
mapError(error) { mapError(error) {
if (error instanceof AttachmentError) return {status: error.status, body: {error: error.message}};
if (error instanceof ModelServiceError) { if (error instanceof ModelServiceError) {
return {status: error.status, body: {error: error.message, code: error.code}}; return {status: error.status, body: {error: error.message, code: error.code}};
} }
+5 -3
View File
@@ -106,14 +106,16 @@ export function buildWebSearchQueries(message, decision = decideWebSearch(messag
} }
/** Retrieve before generation; only try a second date query when the first has no usable evidence. */ /** Retrieve before generation; only try a second date query when the first has no usable evidence. */
export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}} = {}) { export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}, referenceText = ""} = {}) {
const decision = decideWebSearch(message); const decision = decideWebSearch(message);
if (!decision.shouldSearch) return {decision, state: "skipped", queries: [], candidates: [], sources: null}; if (!decision.shouldSearch) return {decision, state: "skipped", queries: [], candidates: [], sources: null};
onSearching(); onSearching();
const queries = []; const queries = [];
let candidates = []; let candidates = [];
let sources = []; let sources = [];
for (const query of buildWebSearchQueries(message, decision)) { // Image evidence may clarify "is this accurate?"; it must never alter the user's tool decision.
const searchMessage = referenceText ? `${message} ${referenceText}` : message;
for (const query of buildWebSearchQueries(searchMessage, decision)) {
queries.push(query); queries.push(query);
try { try {
const results = await provider.search(query, { const results = await provider.search(query, {
@@ -125,7 +127,7 @@ export async function retrieveWebSearch(message, provider, {signal, onSearching
const merged = [...results, ...candidates].sort((a, b) => const merged = [...results, ...candidates].sort((a, b) =>
Number(Boolean(b?.snippet || b?.content)) - Number(Boolean(a?.snippet || a?.content))); Number(Boolean(b?.snippet || b?.content)) - Number(Boolean(a?.snippet || a?.content)));
candidates = normalizeSearchCandidates(merged, MAX_CANDIDATES * 2); candidates = normalizeSearchCandidates(merged, MAX_CANDIDATES * 2);
sources = selectSearchResults(message, candidates, decision); sources = selectSearchResults(searchMessage, candidates, decision);
if (sources.length) break; if (sources.length) break;
} catch (error) { } catch (error) {
if (signal?.aborted || queries.length === 1) throw error; if (signal?.aborted || queries.length === 1) throw error;
+1
View File
@@ -86,6 +86,7 @@ function validateMessage(message, chatIndex, messageIndex, errors) {
content, content,
model_name: optionalText(message.model_name || message.modelName), model_name: optionalText(message.model_name || message.modelName),
thinking: optionalText(message.thinking), thinking: optionalText(message.thinking),
attachments: asArray(message.attachments).filter(item => item && typeof item.id === "string" && /^[a-f0-9-]{36}$/.test(item.id)).slice(0, 100).map(item => ({id: item.id})),
...(message.sources ? {sources: parseSources(message.sources)} : {}), ...(message.sources ? {sources: parseSources(message.sources)} : {}),
retryVariants, retryVariants,
retryActiveIndex: Number.isInteger(Number(message.retryActiveIndex ?? message.retry_active_index)) retryActiveIndex: Number.isInteger(Number(message.retryActiveIndex ?? message.retry_active_index))