improve reasoning toggle tests and refactor model capability checks
This commit is contained in:
@@ -46,6 +46,101 @@ Node.js 22 or newer is required.
|
||||
- `RATE_LIMIT_CLEANUP_INTERVAL_MS`: interval for pruning stale in-memory rate-limit buckets.
|
||||
- `DROP_LEGACY_CHATS`: when `true`, migrate any legacy `chats` table into `chat_sessions` / `chat_messages` and remove the old table.
|
||||
|
||||
## Image attachments and vision
|
||||
|
||||
The composer supports PNG, JPEG and WebP through the image button, dropping files onto
|
||||
the composer, or pasting a screenshot. Text paste is unchanged. Previews show upload
|
||||
status and support retry/removal; sending waits until every remaining image is ready.
|
||||
Images also render in saved messages and open at full size in a protected browser tab.
|
||||
|
||||
Production configuration (the existing `/app/data` volume remains unchanged):
|
||||
|
||||
```dotenv
|
||||
UPLOAD_DIR=/app/data/uploads
|
||||
UPLOAD_MAX_SIZE_MB=10
|
||||
UPLOAD_MAX_FILES=5
|
||||
UPLOAD_MAX_PIXELS=30000000
|
||||
VISION_MODEL=gemma3:4b
|
||||
```
|
||||
|
||||
Outside production, the default directory is `data/uploads`; `UPLOAD_DIR` overrides
|
||||
it. Startup creates the directory and checks write permission, failing clearly rather
|
||||
than falling back to another location. Files use `YYYY/MM/DD/<uuid>.<format>` keys.
|
||||
Do not publish this directory as a static directory. Back up **both the SQLite database
|
||||
and uploads directory**. JSON workspace exports contain attachment references only;
|
||||
imports reconnect existing images owned by the same account, not portable image bytes.
|
||||
|
||||
Authenticated API endpoints:
|
||||
|
||||
- `GET /uploads/config`: safe client limits, without storage paths.
|
||||
- `POST /uploads`: multipart field `image`, one file per request; returns
|
||||
`{id,type:"image",originalName,mimeType,sizeBytes,width,height}`.
|
||||
- `GET /attachments/:id`: ownership-checked image bytes with private/no-store caching
|
||||
and `nosniff`; attached images also require access to a referencing conversation.
|
||||
- `DELETE /attachments/:id`: removes an owned temporary upload only.
|
||||
- `/chat` and `/chat/stream`: optional `attachments: [id, ...]`; image-only messages
|
||||
receive a default description prompt. Existing text-only payloads remain valid.
|
||||
|
||||
Multer bounds multipart input in memory; Sharp detects the actual format, checks the
|
||||
pixel limit and fully decodes it. Browser MIME types and filenames are not trusted.
|
||||
SVG, GIF, animated/multipage images, empty, malformed, oversized and fake images are
|
||||
rejected. File counts are checked again during atomic message association. Concurrent
|
||||
uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at four times
|
||||
that limit. Neither arbitrary paths nor remote image URLs are accepted.
|
||||
|
||||
The additive, idempotent SQLite migration creates `attachments`, `message_attachments`
|
||||
and `upload_deletions`; existing messages are not rewritten. Attachment rows contain
|
||||
owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached
|
||||
status, creation time and optional vision model/analysis/time. Shared references preserve
|
||||
images across branches, edits and same-instance imports. Deleting the last referencing
|
||||
message/conversation (or the account) queues the file for deletion. Cleanup drains this
|
||||
durable queue at startup, after modifying requests, before uploads and hourly; failed
|
||||
file deletion is retried. Temporary uploads expire after 24 hours. Attached images do
|
||||
not expire. No binary or Base64 data is stored in SQLite.
|
||||
|
||||
Generation order is attachment ownership validation → atomic user message/reference
|
||||
persistence → **Analyzing images…** → existing Web Search decision/retrieval → final
|
||||
selected chat model streaming. Gemma receives each image separately as a server-side
|
||||
Base64 `messages[].images` value, together with the user's question. The existing
|
||||
Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive
|
||||
and the existing cancellation/timeouts. Its stream is consumed privately; only the
|
||||
final selected chat model's answer and opted-in reasoning reach the user.
|
||||
|
||||
The bounded, validated vision record is:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "Description of the image",
|
||||
"visibleText": ["Clearly readable text"],
|
||||
"objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}],
|
||||
"observations": [],
|
||||
"warnings": [],
|
||||
"uncertainties": []
|
||||
}
|
||||
```
|
||||
|
||||
Malformed JSON falls back to a bounded plain-text summary in the same record shape.
|
||||
An empty response or unavailable Gemma fails the turn explicitly, without asking the
|
||||
chat model to invent image details. Successful analysis is cached by attachment and
|
||||
vision-model name and reused for retries/references; changing the configured vision
|
||||
model reruns analysis on the next direct use. Recent-history image context is limited
|
||||
to the configured image count and 4,000 characters per image. Analysis remains separate
|
||||
from visible message text. The final model receives observation context with explicit
|
||||
instructions to preserve uncertainty and treat OCR/image instructions as untrusted data;
|
||||
it never receives image bytes or filesystem paths.
|
||||
|
||||
Web Search and Reasoning remain independent. For requested searches, concise visual
|
||||
evidence helps form the query without overriding the user's search decision. Search
|
||||
finishes before final generation; source ordering/rendering is unchanged. The final
|
||||
request still uses `stream: true` and the existing capability-checked reasoning toggle.
|
||||
Stop/disconnect cancels the current vision or final generation request. Logging includes
|
||||
IDs, sizes, types and cache events only, never images/Base64/private analysis.
|
||||
|
||||
Dependencies added: `multer` (bounded multipart handling), `sharp` (content detection,
|
||||
dimensions and full image decoding). Install through the normal `npm install`/`npm ci`
|
||||
workflow, including Sharp's platform-specific optional dependencies in Docker. The
|
||||
application does not pull Ollama models; provision `gemma3:4b` separately.
|
||||
|
||||
## Model Backend Expectations
|
||||
|
||||
The server is configured around an Ollama-compatible API surface:
|
||||
|
||||
+3
-1
@@ -18,7 +18,9 @@
|
||||
"body-parser": "^1.20.2",
|
||||
"express": "^4.19.2",
|
||||
"express-session": "^1.17.3",
|
||||
"multer": "^2.4.0",
|
||||
"qs": "^6.15.0",
|
||||
"session-file-store": "^1.5.0"
|
||||
"session-file-store": "^1.5.0",
|
||||
"sharp": "^0.35.4"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -196,7 +196,12 @@
|
||||
<div class="input-meta">
|
||||
<p class="composer-hint">Enter to send · Shift+Enter for new line</p>
|
||||
</div>
|
||||
<div id="uploadPreviews" class="upload-previews" aria-live="polite" hidden></div>
|
||||
<div class="input-container">
|
||||
<input id="imagePicker" type="file" accept="image/png,image/jpeg,image/webp" multiple hidden>
|
||||
<button id="attachImage" class="icon-button" type="button" aria-label="Attach image" title="Attach image">
|
||||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><rect x="3" y="3" width="18" height="18" rx="3"/><circle cx="8" cy="8" r="1.5"/><path d="m3 17 5-5 4 4 4-6 5 7"/></svg>
|
||||
</button>
|
||||
<button id="webSearch" class="icon-button" type="button" aria-label="Search the web" title="Search the web" aria-pressed="false">
|
||||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><circle cx="12" cy="12" r="9"/><ellipse cx="12" cy="12" rx="4" ry="9"/><path d="M3 12h18"/></svg>
|
||||
</button>
|
||||
|
||||
+15
-6
@@ -1,5 +1,6 @@
|
||||
import {del, get, post, put, stream} from "./app/api.js?v=20260909-account-controls";
|
||||
import {appendSources, createWebSearchToggle} from "./app/webSearch.js";
|
||||
import {appendAttachments, createAttachmentComposer} from "./app/attachments.js";
|
||||
import {createReasoningToggle, finishReasoning, updateReasoning} from "./app/reasoning.js?v=20260921-reasoning-capability";
|
||||
import {closeChatDrawer, openChatDrawer} from "./app/chatDrawer.js?v=20260909-account-controls";
|
||||
import {onboardingPrompts, requestedChatId} from "./app/constants.js?v=20260909-account-controls";
|
||||
@@ -109,6 +110,7 @@ const webSearchToggle = createWebSearchToggle(webSearchButton);
|
||||
const reasoningToggle = createReasoningToggle(document.getElementById("reasoning"));
|
||||
|
||||
let isProcessing = false, activeChatId = null, editingPersonaId = null, editingPersonaType = "assistant", currentUsername = "";
|
||||
const attachmentComposer = createAttachmentComposer({onChange: updateSendState, onError: message => setNotice(message, "error")});
|
||||
let chatSessions = [], currentMessages = [], assistantPersonas = [], userPersonas = [];
|
||||
let activeUserPersonaId = null, currentSummary = null, publishedPersonaIds = new Set();
|
||||
let popupResolver = null, popupMode = null, popupLastFocus = null;
|
||||
@@ -329,7 +331,7 @@ function updateComposerPlaceholder() {
|
||||
|
||||
function updateSendState() {
|
||||
const hasText = Boolean(msgInput.value.trim());
|
||||
sendBtn.disabled = isProcessing ? false : !hasText;
|
||||
sendBtn.disabled = isProcessing ? false : attachmentComposer.pending || (!hasText && !attachmentComposer.attachments.length);
|
||||
sendBtn.setAttribute("aria-label", isProcessing ? "Stop generation" : "Send message");
|
||||
}
|
||||
|
||||
@@ -742,7 +744,9 @@ function addMessage(contentOrMessage, isUser = false, isLoading = false, options
|
||||
actions.append(prevBtn, indexBadge, nextBtn);
|
||||
}
|
||||
|
||||
msgDiv.append(header, body);
|
||||
msgDiv.append(header);
|
||||
appendAttachments(msgDiv, message.attachments);
|
||||
msgDiv.append(body);
|
||||
if (!isMessageUser) updateReasoning(msgDiv, message.thinking);
|
||||
if (!isMessageUser) appendSources(msgDiv, message.sources);
|
||||
if (message.deliveryStatus === "failed" && message.errorMessage) {
|
||||
@@ -1364,6 +1368,7 @@ function setLoadingState(loading, overlayOptions = null) {
|
||||
msgInput.disabled = loading;
|
||||
webSearchButton.disabled = loading;
|
||||
reasoningToggle.setBusy(loading);
|
||||
attachmentComposer.setBusy(loading);
|
||||
sendBtn.classList.toggle("loading", loading);
|
||||
messagesDiv.querySelectorAll('[data-message-action="regenerate"]').forEach((button) => {
|
||||
button.disabled = loading || button.dataset.retryLimitReached === "true";
|
||||
@@ -1811,7 +1816,9 @@ async function checkSession() {
|
||||
|
||||
async function sendMessage() {
|
||||
if (isProcessing) return;
|
||||
const message = msgInput.value.trim();
|
||||
if (attachmentComposer.pending) return;
|
||||
const attachments = attachmentComposer.attachments;
|
||||
const message = msgInput.value.trim() || (attachments.length ? "Describe the attached images." : "");
|
||||
if (!message) return;
|
||||
const webSearch = webSearchToggle.enabled;
|
||||
const reasoning = reasoningToggle.enabled;
|
||||
@@ -1821,7 +1828,8 @@ async function sendMessage() {
|
||||
const autoTitle = message.length > 40 ? `${message.slice(0, 40)}...` : message;
|
||||
currentChat.title = autoTitle; updateWorkspaceCopy(); renderChatList(); await put(`/chats/${activeChatId}`, {title: autoTitle});
|
||||
}
|
||||
currentMessages.push({role: "user", content: message}); addMessage(message, true); updateChatActionState();
|
||||
const userMessage = {role: "user", content: message, attachments};
|
||||
currentMessages.push(userMessage); addMessage(userMessage, true); updateChatActionState();
|
||||
msgInput.value = "";
|
||||
resizeComposerInput();
|
||||
setLoadingState(true, false); setNotice("Generating reply...");
|
||||
@@ -1832,14 +1840,15 @@ async function sendMessage() {
|
||||
loadingLabel.textContent = "Preparing response...";
|
||||
try {
|
||||
let streamError = null;
|
||||
await stream("/chat/stream", {message, model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, {
|
||||
await stream("/chat/stream", {message, attachments: attachments.map(item => item.id), model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, {
|
||||
onStatus: ({status, sourceCount, searchState}) => {
|
||||
if (status === "accepted") {
|
||||
attachmentComposer.clear();
|
||||
webSearchToggle.reset();
|
||||
reasoningToggle.reset();
|
||||
return;
|
||||
}
|
||||
loadingLabel.textContent = status === "searching" ? "Searching the web..."
|
||||
loadingLabel.textContent = status === "analyzing" ? "Analyzing images..." : status === "searching" ? "Searching the web..."
|
||||
: searchState === "no_relevant_results" ? "No sufficiently relevant results. Generating response..."
|
||||
: sourceCount === 0 ? "No web results found. Generating response..." : "Generating response...";
|
||||
},
|
||||
|
||||
+12
-1
@@ -456,7 +456,18 @@ input:focus, textarea:focus, select:focus { border-color: var(--brand); }
|
||||
.input-meta, .input-container { width: min(940px, 100%); margin-inline: auto; }
|
||||
.input-meta { margin-bottom: 5px; padding-left: 10px; }
|
||||
.composer-hint { margin: 0; color: var(--text-muted); font-size: .72rem; }
|
||||
.input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); }
|
||||
.input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "attach search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); }
|
||||
#attachImage { grid-area: attach; }
|
||||
.upload-previews { width: min(940px, 100%); margin: 0 auto 8px; display: flex; gap: 8px; flex-wrap: wrap; }
|
||||
.upload-previews[hidden] { display: none; }
|
||||
.upload-preview { display: flex; align-items: center; gap: 8px; padding: 6px; border: 1px solid var(--border-strong); border-radius: 12px; max-width: 100%; }
|
||||
.upload-preview span { overflow-wrap: anywhere; font-size: 12px; }
|
||||
.upload-preview img { width: 48px; height: 48px; object-fit: contain; border-radius: 6px; }
|
||||
.message-images { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 10px; }
|
||||
.message-images img { max-width: min(360px, 100%); max-height: 300px; object-fit: contain; border-radius: 12px; }
|
||||
.message-images a { max-width: 100%; }
|
||||
.image-drag-target { outline: 2px solid currentColor; }
|
||||
@media (max-width: 480px) { .input-container { grid-template-columns: 36px 36px 36px minmax(0, 1fr) 36px; gap: 3px; } }
|
||||
#webSearch, #reasoning { width: 44px; height: 44px; display: grid; place-items: center; border-radius: 17px; }
|
||||
#webSearch { grid-area: search; }
|
||||
#reasoning { grid-area: reasoning; }
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
import {execFileSync} from "node:child_process";
|
||||
|
||||
const files = [
|
||||
"src/db/attachments.js",
|
||||
"src/services/localFileStorage.js",
|
||||
"src/services/attachmentService.js",
|
||||
"src/routes/uploads.js",
|
||||
"public/app/attachments.js",
|
||||
"server.js",
|
||||
"src/app.js",
|
||||
"src/pageLayout.js",
|
||||
|
||||
+11
-1
@@ -16,6 +16,8 @@ import {createPagesRouter} from "./routes/pages.js";
|
||||
import {createPersonasRouter} from "./routes/personas.js";
|
||||
import {createSettingsRouter} from "./routes/settings.js";
|
||||
import {createSystemRouter} from "./routes/system.js";
|
||||
import {createAttachmentService} from "./services/attachmentService.js";
|
||||
import {createUploadsRouter} from "./routes/uploads.js";
|
||||
|
||||
export function createCloseHandler(resources) {
|
||||
let closed = false;
|
||||
@@ -245,13 +247,15 @@ export function createApp(options = {}) {
|
||||
const db = createDatabase(config);
|
||||
const repositories = createRepositories(db, config);
|
||||
const modelService = options.modelService || createModelService(config, options.modelOverrides);
|
||||
const chatService = createChatService(repositories, modelService, config);
|
||||
const attachmentService = createAttachmentService(db, config, modelService);
|
||||
const chatService = createChatService(repositories, modelService, config, undefined, attachmentService);
|
||||
const FileStore = FileStoreFactory(session);
|
||||
const authRateLimiters = createAuthRateLimiters(config);
|
||||
const chatRateLimiters = createChatRateLimiters(config);
|
||||
closeHandlers.push(
|
||||
...authRateLimiters.map((limiter) => limiter.close),
|
||||
...chatRateLimiters.map((limiter) => limiter.close),
|
||||
() => attachmentService.close(),
|
||||
() => db.close()
|
||||
);
|
||||
|
||||
@@ -271,11 +275,16 @@ export function createApp(options = {}) {
|
||||
}));
|
||||
app.use(sameOriginProtection(config));
|
||||
app.use(enforceSessionVersion(repositories));
|
||||
app.use((req, res, next) => {
|
||||
if (["DELETE", "POST"].includes(req.method)) res.on("finish", () => { void attachmentService.cleanup(); });
|
||||
next();
|
||||
});
|
||||
|
||||
app.locals.config = config;
|
||||
app.locals.db = db;
|
||||
app.locals.repositories = repositories;
|
||||
app.locals.modelService = modelService;
|
||||
app.locals.attachmentService = attachmentService;
|
||||
app.locals.observability = observability;
|
||||
app.locals.close = createCloseHandler(closeHandlers);
|
||||
|
||||
@@ -290,6 +299,7 @@ export function createApp(options = {}) {
|
||||
fallthrough: config.staticFallthrough
|
||||
}));
|
||||
app.use(createAuthRouter({repositories, authRateLimiters, config}));
|
||||
app.use(createUploadsRouter(attachmentService, config));
|
||||
app.use(createSettingsRouter({repositories}));
|
||||
app.use(createChatsRouter({repositories, chatService, modelService, config, chatRateLimiters}));
|
||||
app.use(createPersonasRouter({repositories, chatService, modelService, config}));
|
||||
|
||||
@@ -29,6 +29,11 @@ export function createConfig(overrides = {}) {
|
||||
rootDir,
|
||||
publicDir: path.resolve(rootDir, "public"),
|
||||
dataDir,
|
||||
uploadDir: process.env.UPLOAD_DIR || (process.env.NODE_ENV === "production" ? "/app/data/uploads" : path.resolve(dataDir, "uploads")),
|
||||
uploadMaxSizeMb: Math.max(1, int(process.env.UPLOAD_MAX_SIZE_MB, 10)),
|
||||
uploadMaxFiles: Math.max(1, int(process.env.UPLOAD_MAX_FILES, 5)),
|
||||
uploadMaxPixels: Math.max(1, int(process.env.UPLOAD_MAX_PIXELS, 30000000)),
|
||||
visionModel: process.env.VISION_MODEL || "gemma3:4b",
|
||||
sessionsDir,
|
||||
port: int(process.env.PORT, 3000),
|
||||
dbPath: process.env.DB_PATH || path.resolve(dataDir, "app.db"),
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import fs from "fs";
|
||||
import Database from "better-sqlite3";
|
||||
import {initializeSchema, runMigrations} from "./schema.js";
|
||||
import {initializeAttachments} from "./attachments.js";
|
||||
|
||||
export function createDatabase(config) {
|
||||
if (config.dbPath !== ":memory:") {
|
||||
@@ -10,5 +11,6 @@ export function createDatabase(config) {
|
||||
const db = new Database(config.dbPath);
|
||||
initializeSchema(db);
|
||||
runMigrations(db, config);
|
||||
initializeAttachments(db);
|
||||
return db;
|
||||
}
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import {attachmentMetadata} from "./attachments.js";
|
||||
|
||||
const parseJsonArray = (value) => {
|
||||
if (!value) return [];
|
||||
try {
|
||||
@@ -869,6 +871,11 @@ export function createRepositories(db, config) {
|
||||
if (message.sourceId != null) importedMessageIds.set(message.sourceId, Number(result.lastInsertRowid));
|
||||
if (message.sources) statements.updateChatMessageSources.run(JSON.stringify(message.sources), result.lastInsertRowid, chatId);
|
||||
if (message.thinking) statements.updateChatMessageThinking.run(message.thinking, result.lastInsertRowid, chatId);
|
||||
for (const [position, attachment] of (message.attachments || []).slice(0, config.uploadMaxFiles || 5).entries()) {
|
||||
// JSON backups contain references, not files. Only restore existing, owned images.
|
||||
db.prepare("INSERT OR IGNORE INTO message_attachments(message_id,attachment_id,position) SELECT ?,id,? FROM attachments WHERE id=? AND username=? AND status='attached'")
|
||||
.run(result.lastInsertRowid, position, attachment.id, username);
|
||||
}
|
||||
}
|
||||
statements.updateChatGenerationSettings.run(
|
||||
chat.preferred_model ?? null,
|
||||
@@ -1271,6 +1278,7 @@ export function createRepositories(db, config) {
|
||||
persona_state: this.getPersonaChatState(chat.id, username),
|
||||
messages: this.listChatMessages(chat.id).map((message) => ({
|
||||
...message,
|
||||
attachments: db.prepare("SELECT a.* FROM attachments a JOIN message_attachments m ON a.id=m.attachment_id WHERE m.message_id=? ORDER BY m.position").all(message.id).map(attachmentMetadata),
|
||||
retryVariants: parseJsonArray(message.retry_variants)
|
||||
}))
|
||||
})),
|
||||
|
||||
+6
-2
@@ -17,13 +17,15 @@ const messageValidator = (body) => {
|
||||
};
|
||||
|
||||
const sendMessageValidator = (body) => {
|
||||
const message = String(body?.message || "").trim();
|
||||
const attachments = body?.attachments ?? [];
|
||||
if (!Array.isArray(attachments) || attachments.some(id => typeof id !== "string" || !/^[a-f0-9-]{36}$/.test(id))) return {error: "Invalid attachment IDs"};
|
||||
const message = String(body?.message || "").trim() || (attachments.length ? "Describe the attached images." : "");
|
||||
const chatId = Number(body?.chatId);
|
||||
if (!chatId) return {error: "chatId is required"};
|
||||
if (!message) return {error: "message is required"};
|
||||
if (body.webSearch !== undefined && typeof body.webSearch !== "boolean") return {error: "webSearch must be a boolean"};
|
||||
if (body.reasoning !== undefined && typeof body.reasoning !== "boolean") return {error: "reasoning must be a boolean"};
|
||||
return {value: {message, chatId, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}};
|
||||
return {value: {message, chatId, attachments, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}};
|
||||
};
|
||||
|
||||
const organizationValidator = (body) => ({
|
||||
@@ -413,6 +415,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf
|
||||
user: req.session.user,
|
||||
chatId: req.validatedBody.chatId,
|
||||
message: req.validatedBody.message,
|
||||
attachments: req.validatedBody.attachments,
|
||||
model: req.validatedBody.model,
|
||||
webSearch: req.validatedBody.webSearch,
|
||||
reasoning: req.validatedBody.reasoning
|
||||
@@ -449,6 +452,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf
|
||||
user: req.session.user,
|
||||
chatId: req.validatedBody.chatId,
|
||||
message: req.validatedBody.message,
|
||||
attachments: req.validatedBody.attachments,
|
||||
model: req.validatedBody.model,
|
||||
onChunk: (chunk, fullReply) => sendEvent("chunk", {chunk, fullReply}),
|
||||
onThinking: (chunk, fullThinking) => sendEvent("thinking", {chunk, fullThinking}),
|
||||
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
buildMemoryPrompt,
|
||||
buildPersonaStatePrompt,
|
||||
clampRetryActiveIndex,
|
||||
formatChatMessage,
|
||||
formatChatMessage as formatBaseChatMessage,
|
||||
formatMessagesForSummary,
|
||||
parseSources,
|
||||
retryStylePrompts
|
||||
@@ -21,7 +21,12 @@ import {
|
||||
import {SearXNGSearchProvider, WebSearchError, buildWebSearchContext, retrieveWebSearch} from "./webSearchService.js";
|
||||
import {supportsReasoning} from "../../public/app/modelCapabilities.js";
|
||||
|
||||
export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config)) {
|
||||
export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config), attachmentService = null) {
|
||||
const formatChatMessage = row => {
|
||||
const message = formatBaseChatMessage(row);
|
||||
const attachments = row && attachmentService?.list(row.id);
|
||||
return message && attachments?.length ? {...message, attachments} : message;
|
||||
};
|
||||
const chatGenerationQueues = new Map();
|
||||
const summaryQueues = new Map();
|
||||
const activeGenerations = new Map();
|
||||
@@ -272,6 +277,8 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
content: entry.content
|
||||
}));
|
||||
messagesPayload.push(...conversationMessages.slice(0, -1));
|
||||
const visionContext = attachmentService?.context(history);
|
||||
if (visionContext) messagesPayload.push(visionContext);
|
||||
if (webSearchContext) messagesPayload.push(webSearchContext[0]);
|
||||
if (webSearchContext?.[1]) messagesPayload.push(webSearchContext[1]);
|
||||
if (conversationMessages.length) messagesPayload.push(conversationMessages.at(-1));
|
||||
@@ -307,6 +314,7 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
return {fullReply: opener, thinking, promptMessageId: null};
|
||||
}
|
||||
|
||||
await attachmentService?.analyze(promptMessage.id, promptMessage.content, signal);
|
||||
const {messagesPayload} = buildConversationContext({
|
||||
user,
|
||||
chatId,
|
||||
@@ -327,7 +335,7 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
return {fullReply, thinking, promptMessageId: promptMessage.id};
|
||||
};
|
||||
|
||||
const generateChatTurn = ({user, chatId, message, model, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) =>
|
||||
const generateChatTurn = ({user, chatId, message, model, attachments = [], reuseChatId = null, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) =>
|
||||
withChatGenerationLock(chatId, async () => {
|
||||
await (summaryQueues.get(chatId) || Promise.resolve());
|
||||
const session = repositories.getChat(chatId, user);
|
||||
@@ -346,7 +354,8 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
const {model: effectiveModel, generation} = getGenerationSettings(session, model);
|
||||
const useReasoning = reasoning === true && supportsReasoning(effectiveModel);
|
||||
signal?.throwIfAborted();
|
||||
const inserted = repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId);
|
||||
const insert = () => repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId);
|
||||
const inserted = attachmentService ? attachmentService.createMessage(attachments, user, chatId, insert, reuseChatId) : insert();
|
||||
const userMessageId = Number(inserted?.lastInsertRowid);
|
||||
try {
|
||||
let sources = null;
|
||||
@@ -361,8 +370,13 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
const fullReply = await withActiveGeneration(chatId, signal, async (activeSignal) => {
|
||||
activeSignal.throwIfAborted();
|
||||
onStatus?.({status: "accepted"});
|
||||
if (attachments.length) {
|
||||
onStatus?.({status: "analyzing"});
|
||||
await attachmentService.analyze(userMessageId, message, activeSignal);
|
||||
}
|
||||
if (webSearch) {
|
||||
const search = await retrieveWebSearch(message, webSearchProvider, {
|
||||
referenceText: attachments.length ? attachmentService.searchContext(userMessageId) : "",
|
||||
signal: activeSignal,
|
||||
onSearching: () => onStatus?.({status: "searching"})
|
||||
});
|
||||
@@ -489,6 +503,7 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
parentMessageId
|
||||
);
|
||||
parentMessageId = Number(inserted.lastInsertRowid);
|
||||
attachmentService?.copy(sourceMessage.id, parentMessageId);
|
||||
if (sourceMessage.sources) repositories.updateChatMessageSources(clonedChat.id, parentMessageId, parseSources(sourceMessage.sources));
|
||||
if (sourceMessage.thinking) repositories.updateChatMessageThinking(clonedChat.id, parentMessageId, sourceMessage.thinking);
|
||||
}
|
||||
@@ -515,11 +530,11 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
containsUserVoiceInRoleplayOpener,
|
||||
persistChatMessageRetryState,
|
||||
buildMessagesPayload: buildConversationContext,
|
||||
async sendChatMessage({user, chatId, message, model, webSearch = false, reasoning = false}) {
|
||||
return generateChatTurn({user, chatId, message, model, webSearch, reasoning});
|
||||
async sendChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false}) {
|
||||
return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning});
|
||||
},
|
||||
async streamChatMessage({user, chatId, message, model, webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) {
|
||||
return generateChatTurn({user, chatId, message, model, webSearch, reasoning, onStatus, onChunk, onThinking, signal});
|
||||
async streamChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) {
|
||||
return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning, onStatus, onChunk, onThinking, signal});
|
||||
},
|
||||
async retryMessage({user, chatId, selectedModel, targetMessage, retryStyle = null}) {
|
||||
return withChatGenerationLock(chatId, async () => {
|
||||
@@ -574,6 +589,8 @@ export function createChatService(repositories, modelService, config, webSearchP
|
||||
const generated = await generateChatTurn({
|
||||
user,
|
||||
chatId: cloned.chat.id,
|
||||
attachments: attachmentService?.list(targetMessage.id).map(item => item.id) || [],
|
||||
reuseChatId: chatId,
|
||||
message: content,
|
||||
model,
|
||||
webSearch,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import {supportsReasoning} from "../../public/app/modelCapabilities.js";
|
||||
import {AttachmentError} from "./attachmentService.js";
|
||||
|
||||
class ModelServiceError extends Error {
|
||||
constructor(code, message, status = 500) {
|
||||
@@ -196,7 +197,7 @@ export function createModelService(config, overrides = {}) {
|
||||
const response = await fetchImpl(`${config.modelApiBaseUrl}/chat`, {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify(buildRequestBody(model, messagesPayload, options.generation, options.reasoning)),
|
||||
body: JSON.stringify({...buildRequestBody(model, messagesPayload, options.generation, options.reasoning), ...(options.format ? {format: options.format} : {})}),
|
||||
signal
|
||||
});
|
||||
|
||||
@@ -332,6 +333,7 @@ export function createModelService(config, overrides = {}) {
|
||||
};
|
||||
},
|
||||
mapError(error) {
|
||||
if (error instanceof AttachmentError) return {status: error.status, body: {error: error.message}};
|
||||
if (error instanceof ModelServiceError) {
|
||||
return {status: error.status, body: {error: error.message, code: error.code}};
|
||||
}
|
||||
|
||||
@@ -106,14 +106,16 @@ export function buildWebSearchQueries(message, decision = decideWebSearch(messag
|
||||
}
|
||||
|
||||
/** Retrieve before generation; only try a second date query when the first has no usable evidence. */
|
||||
export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}} = {}) {
|
||||
export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}, referenceText = ""} = {}) {
|
||||
const decision = decideWebSearch(message);
|
||||
if (!decision.shouldSearch) return {decision, state: "skipped", queries: [], candidates: [], sources: null};
|
||||
onSearching();
|
||||
const queries = [];
|
||||
let candidates = [];
|
||||
let sources = [];
|
||||
for (const query of buildWebSearchQueries(message, decision)) {
|
||||
// Image evidence may clarify "is this accurate?"; it must never alter the user's tool decision.
|
||||
const searchMessage = referenceText ? `${message} ${referenceText}` : message;
|
||||
for (const query of buildWebSearchQueries(searchMessage, decision)) {
|
||||
queries.push(query);
|
||||
try {
|
||||
const results = await provider.search(query, {
|
||||
@@ -125,7 +127,7 @@ export async function retrieveWebSearch(message, provider, {signal, onSearching
|
||||
const merged = [...results, ...candidates].sort((a, b) =>
|
||||
Number(Boolean(b?.snippet || b?.content)) - Number(Boolean(a?.snippet || a?.content)));
|
||||
candidates = normalizeSearchCandidates(merged, MAX_CANDIDATES * 2);
|
||||
sources = selectSearchResults(message, candidates, decision);
|
||||
sources = selectSearchResults(searchMessage, candidates, decision);
|
||||
if (sources.length) break;
|
||||
} catch (error) {
|
||||
if (signal?.aborted || queries.length === 1) throw error;
|
||||
|
||||
@@ -86,6 +86,7 @@ function validateMessage(message, chatIndex, messageIndex, errors) {
|
||||
content,
|
||||
model_name: optionalText(message.model_name || message.modelName),
|
||||
thinking: optionalText(message.thinking),
|
||||
attachments: asArray(message.attachments).filter(item => item && typeof item.id === "string" && /^[a-f0-9-]{36}$/.test(item.id)).slice(0, 100).map(item => ({id: item.id})),
|
||||
...(message.sources ? {sources: parseSources(message.sources)} : {}),
|
||||
retryVariants,
|
||||
retryActiveIndex: Number.isInteger(Number(message.retryActiveIndex ?? message.retry_active_index))
|
||||
|
||||
Reference in New Issue
Block a user