diff --git a/README.md b/README.md index 7f59d67..c1a36ed 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,101 @@ Node.js 22 or newer is required. - `RATE_LIMIT_CLEANUP_INTERVAL_MS`: interval for pruning stale in-memory rate-limit buckets. - `DROP_LEGACY_CHATS`: when `true`, migrate any legacy `chats` table into `chat_sessions` / `chat_messages` and remove the old table. +## Image attachments and vision + +The composer supports PNG, JPEG and WebP through the image button, dropping files onto +the composer, or pasting a screenshot. Text paste is unchanged. Previews show upload +status and support retry/removal; sending waits until every remaining image is ready. +Images also render in saved messages and open at full size in a protected browser tab. + +Production configuration (the existing `/app/data` volume remains unchanged): + +```dotenv +UPLOAD_DIR=/app/data/uploads +UPLOAD_MAX_SIZE_MB=10 +UPLOAD_MAX_FILES=5 +UPLOAD_MAX_PIXELS=30000000 +VISION_MODEL=gemma3:4b +``` + +Outside production, the default directory is `data/uploads`; `UPLOAD_DIR` overrides +it. Startup creates the directory and checks write permission, failing clearly rather +than falling back to another location. Files use `YYYY/MM/DD/.` keys. +Do not publish this directory as a static directory. Back up **both the SQLite database +and uploads directory**. JSON workspace exports contain attachment references only; +imports reconnect existing images owned by the same account, not portable image bytes. + +Authenticated API endpoints: + +- `GET /uploads/config`: safe client limits, without storage paths. +- `POST /uploads`: multipart field `image`, one file per request; returns + `{id,type:"image",originalName,mimeType,sizeBytes,width,height}`. +- `GET /attachments/:id`: ownership-checked image bytes with private/no-store caching + and `nosniff`; attached images also require access to a referencing conversation. +- `DELETE /attachments/:id`: removes an owned temporary upload only. +- `/chat` and `/chat/stream`: optional `attachments: [id, ...]`; image-only messages + receive a default description prompt. Existing text-only payloads remain valid. + +Multer bounds multipart input in memory; Sharp detects the actual format, checks the +pixel limit and fully decodes it. Browser MIME types and filenames are not trusted. +SVG, GIF, animated/multipage images, empty, malformed, oversized and fake images are +rejected. File counts are checked again during atomic message association. Concurrent +uploads per account are capped at `UPLOAD_MAX_FILES`, and pending uploads at four times +that limit. Neither arbitrary paths nor remote image URLs are accepted. + +The additive, idempotent SQLite migration creates `attachments`, `message_attachments` +and `upload_deletions`; existing messages are not rewritten. Attachment rows contain +owner, original filename, relative storage key, MIME/size/dimensions, temporary/attached +status, creation time and optional vision model/analysis/time. Shared references preserve +images across branches, edits and same-instance imports. Deleting the last referencing +message/conversation (or the account) queues the file for deletion. Cleanup drains this +durable queue at startup, after modifying requests, before uploads and hourly; failed +file deletion is retried. Temporary uploads expire after 24 hours. Attached images do +not expire. No binary or Base64 data is stored in SQLite. + +Generation order is attachment ownership validation → atomic user message/reference +persistence → **Analyzing images…** → existing Web Search decision/retrieval → final +selected chat model streaming. Gemma receives each image separately as a server-side +Base64 `messages[].images` value, together with the user's question. The existing +Ollama client sends `model: VISION_MODEL`, `think: false`, `format: "json"`, keep-alive +and the existing cancellation/timeouts. Its stream is consumed privately; only the +final selected chat model's answer and opted-in reasoning reach the user. + +The bounded, validated vision record is: + +```json +{ + "summary": "Description of the image", + "visibleText": ["Clearly readable text"], + "objects": [{"type": "button", "description": "Disabled control", "location": "bottom left"}], + "observations": [], + "warnings": [], + "uncertainties": [] +} +``` + +Malformed JSON falls back to a bounded plain-text summary in the same record shape. +An empty response or unavailable Gemma fails the turn explicitly, without asking the +chat model to invent image details. Successful analysis is cached by attachment and +vision-model name and reused for retries/references; changing the configured vision +model reruns analysis on the next direct use. Recent-history image context is limited +to the configured image count and 4,000 characters per image. Analysis remains separate +from visible message text. The final model receives observation context with explicit +instructions to preserve uncertainty and treat OCR/image instructions as untrusted data; +it never receives image bytes or filesystem paths. + +Web Search and Reasoning remain independent. For requested searches, concise visual +evidence helps form the query without overriding the user's search decision. Search +finishes before final generation; source ordering/rendering is unchanged. The final +request still uses `stream: true` and the existing capability-checked reasoning toggle. +Stop/disconnect cancels the current vision or final generation request. Logging includes +IDs, sizes, types and cache events only, never images/Base64/private analysis. + +Dependencies added: `multer` (bounded multipart handling), `sharp` (content detection, +dimensions and full image decoding). Install through the normal `npm install`/`npm ci` +workflow, including Sharp's platform-specific optional dependencies in Docker. The +application does not pull Ollama models; provision `gemma3:4b` separately. + ## Model Backend Expectations The server is configured around an Ollama-compatible API surface: diff --git a/package.json b/package.json index f520c67..66bf6fe 100644 --- a/package.json +++ b/package.json @@ -18,7 +18,9 @@ "body-parser": "^1.20.2", "express": "^4.19.2", "express-session": "^1.17.3", + "multer": "^2.4.0", "qs": "^6.15.0", - "session-file-store": "^1.5.0" + "session-file-store": "^1.5.0", + "sharp": "^0.35.4" } } diff --git a/public/chat.html b/public/chat.html index 60f0abf..42a4250 100644 --- a/public/chat.html +++ b/public/chat.html @@ -196,7 +196,12 @@

Enter to send · Shift+Enter for new line

+
+ + diff --git a/public/script.js b/public/script.js index bb18c98..5620e23 100644 --- a/public/script.js +++ b/public/script.js @@ -1,5 +1,6 @@ import {del, get, post, put, stream} from "./app/api.js?v=20260909-account-controls"; import {appendSources, createWebSearchToggle} from "./app/webSearch.js"; +import {appendAttachments, createAttachmentComposer} from "./app/attachments.js"; import {createReasoningToggle, finishReasoning, updateReasoning} from "./app/reasoning.js?v=20260921-reasoning-capability"; import {closeChatDrawer, openChatDrawer} from "./app/chatDrawer.js?v=20260909-account-controls"; import {onboardingPrompts, requestedChatId} from "./app/constants.js?v=20260909-account-controls"; @@ -109,6 +110,7 @@ const webSearchToggle = createWebSearchToggle(webSearchButton); const reasoningToggle = createReasoningToggle(document.getElementById("reasoning")); let isProcessing = false, activeChatId = null, editingPersonaId = null, editingPersonaType = "assistant", currentUsername = ""; +const attachmentComposer = createAttachmentComposer({onChange: updateSendState, onError: message => setNotice(message, "error")}); let chatSessions = [], currentMessages = [], assistantPersonas = [], userPersonas = []; let activeUserPersonaId = null, currentSummary = null, publishedPersonaIds = new Set(); let popupResolver = null, popupMode = null, popupLastFocus = null; @@ -329,7 +331,7 @@ function updateComposerPlaceholder() { function updateSendState() { const hasText = Boolean(msgInput.value.trim()); - sendBtn.disabled = isProcessing ? false : !hasText; + sendBtn.disabled = isProcessing ? false : attachmentComposer.pending || (!hasText && !attachmentComposer.attachments.length); sendBtn.setAttribute("aria-label", isProcessing ? "Stop generation" : "Send message"); } @@ -742,7 +744,9 @@ function addMessage(contentOrMessage, isUser = false, isLoading = false, options actions.append(prevBtn, indexBadge, nextBtn); } - msgDiv.append(header, body); + msgDiv.append(header); + appendAttachments(msgDiv, message.attachments); + msgDiv.append(body); if (!isMessageUser) updateReasoning(msgDiv, message.thinking); if (!isMessageUser) appendSources(msgDiv, message.sources); if (message.deliveryStatus === "failed" && message.errorMessage) { @@ -1364,6 +1368,7 @@ function setLoadingState(loading, overlayOptions = null) { msgInput.disabled = loading; webSearchButton.disabled = loading; reasoningToggle.setBusy(loading); + attachmentComposer.setBusy(loading); sendBtn.classList.toggle("loading", loading); messagesDiv.querySelectorAll('[data-message-action="regenerate"]').forEach((button) => { button.disabled = loading || button.dataset.retryLimitReached === "true"; @@ -1811,7 +1816,9 @@ async function checkSession() { async function sendMessage() { if (isProcessing) return; - const message = msgInput.value.trim(); + if (attachmentComposer.pending) return; + const attachments = attachmentComposer.attachments; + const message = msgInput.value.trim() || (attachments.length ? "Describe the attached images." : ""); if (!message) return; const webSearch = webSearchToggle.enabled; const reasoning = reasoningToggle.enabled; @@ -1821,7 +1828,8 @@ async function sendMessage() { const autoTitle = message.length > 40 ? `${message.slice(0, 40)}...` : message; currentChat.title = autoTitle; updateWorkspaceCopy(); renderChatList(); await put(`/chats/${activeChatId}`, {title: autoTitle}); } - currentMessages.push({role: "user", content: message}); addMessage(message, true); updateChatActionState(); + const userMessage = {role: "user", content: message, attachments}; + currentMessages.push(userMessage); addMessage(userMessage, true); updateChatActionState(); msgInput.value = ""; resizeComposerInput(); setLoadingState(true, false); setNotice("Generating reply..."); @@ -1832,14 +1840,15 @@ async function sendMessage() { loadingLabel.textContent = "Preparing response..."; try { let streamError = null; - await stream("/chat/stream", {message, model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, { + await stream("/chat/stream", {message, attachments: attachments.map(item => item.id), model: modelSelect.value, chatId: activeChatId, webSearch, reasoning}, { onStatus: ({status, sourceCount, searchState}) => { if (status === "accepted") { + attachmentComposer.clear(); webSearchToggle.reset(); reasoningToggle.reset(); return; } - loadingLabel.textContent = status === "searching" ? "Searching the web..." + loadingLabel.textContent = status === "analyzing" ? "Analyzing images..." : status === "searching" ? "Searching the web..." : searchState === "no_relevant_results" ? "No sufficiently relevant results. Generating response..." : sourceCount === 0 ? "No web results found. Generating response..." : "Generating response..."; }, diff --git a/public/style.css b/public/style.css index e7523b9..5bd75d0 100644 --- a/public/style.css +++ b/public/style.css @@ -456,7 +456,18 @@ input:focus, textarea:focus, select:focus { border-color: var(--brand); } .input-meta, .input-container { width: min(940px, 100%); margin-inline: auto; } .input-meta { margin-bottom: 5px; padding-left: 10px; } .composer-hint { margin: 0; color: var(--text-muted); font-size: .72rem; } -.input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); } +.input-container { padding: 7px; display: grid; grid-template-columns: 44px 44px 44px minmax(0, 1fr) 44px; grid-template-areas: "attach search reasoning message send"; gap: 7px; align-items: end; background: var(--input-bg); border: 1px solid var(--border-strong); border-radius: 24px; box-shadow: var(--shadow-sm); } +#attachImage { grid-area: attach; } +.upload-previews { width: min(940px, 100%); margin: 0 auto 8px; display: flex; gap: 8px; flex-wrap: wrap; } +.upload-previews[hidden] { display: none; } +.upload-preview { display: flex; align-items: center; gap: 8px; padding: 6px; border: 1px solid var(--border-strong); border-radius: 12px; max-width: 100%; } +.upload-preview span { overflow-wrap: anywhere; font-size: 12px; } +.upload-preview img { width: 48px; height: 48px; object-fit: contain; border-radius: 6px; } +.message-images { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 10px; } +.message-images img { max-width: min(360px, 100%); max-height: 300px; object-fit: contain; border-radius: 12px; } +.message-images a { max-width: 100%; } +.image-drag-target { outline: 2px solid currentColor; } +@media (max-width: 480px) { .input-container { grid-template-columns: 36px 36px 36px minmax(0, 1fr) 36px; gap: 3px; } } #webSearch, #reasoning { width: 44px; height: 44px; display: grid; place-items: center; border-radius: 17px; } #webSearch { grid-area: search; } #reasoning { grid-area: reasoning; } diff --git a/scripts/lint.mjs b/scripts/lint.mjs index 20b531a..04ee0e9 100644 --- a/scripts/lint.mjs +++ b/scripts/lint.mjs @@ -1,6 +1,11 @@ import {execFileSync} from "node:child_process"; const files = [ + "src/db/attachments.js", + "src/services/localFileStorage.js", + "src/services/attachmentService.js", + "src/routes/uploads.js", + "public/app/attachments.js", "server.js", "src/app.js", "src/pageLayout.js", diff --git a/src/app.js b/src/app.js index 7d3c189..1fffe4b 100644 --- a/src/app.js +++ b/src/app.js @@ -16,6 +16,8 @@ import {createPagesRouter} from "./routes/pages.js"; import {createPersonasRouter} from "./routes/personas.js"; import {createSettingsRouter} from "./routes/settings.js"; import {createSystemRouter} from "./routes/system.js"; +import {createAttachmentService} from "./services/attachmentService.js"; +import {createUploadsRouter} from "./routes/uploads.js"; export function createCloseHandler(resources) { let closed = false; @@ -245,13 +247,15 @@ export function createApp(options = {}) { const db = createDatabase(config); const repositories = createRepositories(db, config); const modelService = options.modelService || createModelService(config, options.modelOverrides); - const chatService = createChatService(repositories, modelService, config); + const attachmentService = createAttachmentService(db, config, modelService); + const chatService = createChatService(repositories, modelService, config, undefined, attachmentService); const FileStore = FileStoreFactory(session); const authRateLimiters = createAuthRateLimiters(config); const chatRateLimiters = createChatRateLimiters(config); closeHandlers.push( ...authRateLimiters.map((limiter) => limiter.close), ...chatRateLimiters.map((limiter) => limiter.close), + () => attachmentService.close(), () => db.close() ); @@ -271,11 +275,16 @@ export function createApp(options = {}) { })); app.use(sameOriginProtection(config)); app.use(enforceSessionVersion(repositories)); + app.use((req, res, next) => { + if (["DELETE", "POST"].includes(req.method)) res.on("finish", () => { void attachmentService.cleanup(); }); + next(); + }); app.locals.config = config; app.locals.db = db; app.locals.repositories = repositories; app.locals.modelService = modelService; + app.locals.attachmentService = attachmentService; app.locals.observability = observability; app.locals.close = createCloseHandler(closeHandlers); @@ -290,6 +299,7 @@ export function createApp(options = {}) { fallthrough: config.staticFallthrough })); app.use(createAuthRouter({repositories, authRateLimiters, config})); + app.use(createUploadsRouter(attachmentService, config)); app.use(createSettingsRouter({repositories})); app.use(createChatsRouter({repositories, chatService, modelService, config, chatRateLimiters})); app.use(createPersonasRouter({repositories, chatService, modelService, config})); diff --git a/src/config.js b/src/config.js index 46a3664..4212556 100644 --- a/src/config.js +++ b/src/config.js @@ -29,6 +29,11 @@ export function createConfig(overrides = {}) { rootDir, publicDir: path.resolve(rootDir, "public"), dataDir, + uploadDir: process.env.UPLOAD_DIR || (process.env.NODE_ENV === "production" ? "/app/data/uploads" : path.resolve(dataDir, "uploads")), + uploadMaxSizeMb: Math.max(1, int(process.env.UPLOAD_MAX_SIZE_MB, 10)), + uploadMaxFiles: Math.max(1, int(process.env.UPLOAD_MAX_FILES, 5)), + uploadMaxPixels: Math.max(1, int(process.env.UPLOAD_MAX_PIXELS, 30000000)), + visionModel: process.env.VISION_MODEL || "gemma3:4b", sessionsDir, port: int(process.env.PORT, 3000), dbPath: process.env.DB_PATH || path.resolve(dataDir, "app.db"), diff --git a/src/db/index.js b/src/db/index.js index d57b94d..8d6f91a 100644 --- a/src/db/index.js +++ b/src/db/index.js @@ -1,6 +1,7 @@ import fs from "fs"; import Database from "better-sqlite3"; import {initializeSchema, runMigrations} from "./schema.js"; +import {initializeAttachments} from "./attachments.js"; export function createDatabase(config) { if (config.dbPath !== ":memory:") { @@ -10,5 +11,6 @@ export function createDatabase(config) { const db = new Database(config.dbPath); initializeSchema(db); runMigrations(db, config); + initializeAttachments(db); return db; } diff --git a/src/db/repositories.js b/src/db/repositories.js index 20a0bfd..40d8272 100644 --- a/src/db/repositories.js +++ b/src/db/repositories.js @@ -1,3 +1,5 @@ +import {attachmentMetadata} from "./attachments.js"; + const parseJsonArray = (value) => { if (!value) return []; try { @@ -869,6 +871,11 @@ export function createRepositories(db, config) { if (message.sourceId != null) importedMessageIds.set(message.sourceId, Number(result.lastInsertRowid)); if (message.sources) statements.updateChatMessageSources.run(JSON.stringify(message.sources), result.lastInsertRowid, chatId); if (message.thinking) statements.updateChatMessageThinking.run(message.thinking, result.lastInsertRowid, chatId); + for (const [position, attachment] of (message.attachments || []).slice(0, config.uploadMaxFiles || 5).entries()) { + // JSON backups contain references, not files. Only restore existing, owned images. + db.prepare("INSERT OR IGNORE INTO message_attachments(message_id,attachment_id,position) SELECT ?,id,? FROM attachments WHERE id=? AND username=? AND status='attached'") + .run(result.lastInsertRowid, position, attachment.id, username); + } } statements.updateChatGenerationSettings.run( chat.preferred_model ?? null, @@ -1271,6 +1278,7 @@ export function createRepositories(db, config) { persona_state: this.getPersonaChatState(chat.id, username), messages: this.listChatMessages(chat.id).map((message) => ({ ...message, + attachments: db.prepare("SELECT a.* FROM attachments a JOIN message_attachments m ON a.id=m.attachment_id WHERE m.message_id=? ORDER BY m.position").all(message.id).map(attachmentMetadata), retryVariants: parseJsonArray(message.retry_variants) })) })), diff --git a/src/routes/chats.js b/src/routes/chats.js index dc8f172..4af41cc 100644 --- a/src/routes/chats.js +++ b/src/routes/chats.js @@ -17,13 +17,15 @@ const messageValidator = (body) => { }; const sendMessageValidator = (body) => { - const message = String(body?.message || "").trim(); + const attachments = body?.attachments ?? []; + if (!Array.isArray(attachments) || attachments.some(id => typeof id !== "string" || !/^[a-f0-9-]{36}$/.test(id))) return {error: "Invalid attachment IDs"}; + const message = String(body?.message || "").trim() || (attachments.length ? "Describe the attached images." : ""); const chatId = Number(body?.chatId); if (!chatId) return {error: "chatId is required"}; if (!message) return {error: "message is required"}; if (body.webSearch !== undefined && typeof body.webSearch !== "boolean") return {error: "webSearch must be a boolean"}; if (body.reasoning !== undefined && typeof body.reasoning !== "boolean") return {error: "reasoning must be a boolean"}; - return {value: {message, chatId, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}}; + return {value: {message, chatId, attachments, model: String(body?.model || "qwen3:14b"), webSearch: body.webSearch === true, reasoning: body.reasoning === true}}; }; const organizationValidator = (body) => ({ @@ -413,6 +415,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf user: req.session.user, chatId: req.validatedBody.chatId, message: req.validatedBody.message, + attachments: req.validatedBody.attachments, model: req.validatedBody.model, webSearch: req.validatedBody.webSearch, reasoning: req.validatedBody.reasoning @@ -449,6 +452,7 @@ export function createChatsRouter({repositories, chatService, modelService, conf user: req.session.user, chatId: req.validatedBody.chatId, message: req.validatedBody.message, + attachments: req.validatedBody.attachments, model: req.validatedBody.model, onChunk: (chunk, fullReply) => sendEvent("chunk", {chunk, fullReply}), onThinking: (chunk, fullThinking) => sendEvent("thinking", {chunk, fullThinking}), diff --git a/src/services/chatService.js b/src/services/chatService.js index 45c8a74..d4b58d7 100644 --- a/src/services/chatService.js +++ b/src/services/chatService.js @@ -12,7 +12,7 @@ import { buildMemoryPrompt, buildPersonaStatePrompt, clampRetryActiveIndex, - formatChatMessage, + formatChatMessage as formatBaseChatMessage, formatMessagesForSummary, parseSources, retryStylePrompts @@ -21,7 +21,12 @@ import { import {SearXNGSearchProvider, WebSearchError, buildWebSearchContext, retrieveWebSearch} from "./webSearchService.js"; import {supportsReasoning} from "../../public/app/modelCapabilities.js"; -export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config)) { +export function createChatService(repositories, modelService, config, webSearchProvider = new SearXNGSearchProvider(config), attachmentService = null) { + const formatChatMessage = row => { + const message = formatBaseChatMessage(row); + const attachments = row && attachmentService?.list(row.id); + return message && attachments?.length ? {...message, attachments} : message; + }; const chatGenerationQueues = new Map(); const summaryQueues = new Map(); const activeGenerations = new Map(); @@ -272,6 +277,8 @@ export function createChatService(repositories, modelService, config, webSearchP content: entry.content })); messagesPayload.push(...conversationMessages.slice(0, -1)); + const visionContext = attachmentService?.context(history); + if (visionContext) messagesPayload.push(visionContext); if (webSearchContext) messagesPayload.push(webSearchContext[0]); if (webSearchContext?.[1]) messagesPayload.push(webSearchContext[1]); if (conversationMessages.length) messagesPayload.push(conversationMessages.at(-1)); @@ -307,6 +314,7 @@ export function createChatService(repositories, modelService, config, webSearchP return {fullReply: opener, thinking, promptMessageId: null}; } + await attachmentService?.analyze(promptMessage.id, promptMessage.content, signal); const {messagesPayload} = buildConversationContext({ user, chatId, @@ -327,7 +335,7 @@ export function createChatService(repositories, modelService, config, webSearchP return {fullReply, thinking, promptMessageId: promptMessage.id}; }; - const generateChatTurn = ({user, chatId, message, model, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) => + const generateChatTurn = ({user, chatId, message, model, attachments = [], reuseChatId = null, webSearch = false, reasoning = false, onStatus = null, onChunk = null, onThinking = null, signal = null, parentMessageId = undefined}) => withChatGenerationLock(chatId, async () => { await (summaryQueues.get(chatId) || Promise.resolve()); const session = repositories.getChat(chatId, user); @@ -346,7 +354,8 @@ export function createChatService(repositories, modelService, config, webSearchP const {model: effectiveModel, generation} = getGenerationSettings(session, model); const useReasoning = reasoning === true && supportsReasoning(effectiveModel); signal?.throwIfAborted(); - const inserted = repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId); + const insert = () => repositories.insertBranchChatMessage(chatId, "user", message, null, null, "pending", null, effectiveParentId); + const inserted = attachmentService ? attachmentService.createMessage(attachments, user, chatId, insert, reuseChatId) : insert(); const userMessageId = Number(inserted?.lastInsertRowid); try { let sources = null; @@ -361,8 +370,13 @@ export function createChatService(repositories, modelService, config, webSearchP const fullReply = await withActiveGeneration(chatId, signal, async (activeSignal) => { activeSignal.throwIfAborted(); onStatus?.({status: "accepted"}); + if (attachments.length) { + onStatus?.({status: "analyzing"}); + await attachmentService.analyze(userMessageId, message, activeSignal); + } if (webSearch) { const search = await retrieveWebSearch(message, webSearchProvider, { + referenceText: attachments.length ? attachmentService.searchContext(userMessageId) : "", signal: activeSignal, onSearching: () => onStatus?.({status: "searching"}) }); @@ -489,6 +503,7 @@ export function createChatService(repositories, modelService, config, webSearchP parentMessageId ); parentMessageId = Number(inserted.lastInsertRowid); + attachmentService?.copy(sourceMessage.id, parentMessageId); if (sourceMessage.sources) repositories.updateChatMessageSources(clonedChat.id, parentMessageId, parseSources(sourceMessage.sources)); if (sourceMessage.thinking) repositories.updateChatMessageThinking(clonedChat.id, parentMessageId, sourceMessage.thinking); } @@ -515,11 +530,11 @@ export function createChatService(repositories, modelService, config, webSearchP containsUserVoiceInRoleplayOpener, persistChatMessageRetryState, buildMessagesPayload: buildConversationContext, - async sendChatMessage({user, chatId, message, model, webSearch = false, reasoning = false}) { - return generateChatTurn({user, chatId, message, model, webSearch, reasoning}); + async sendChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false}) { + return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning}); }, - async streamChatMessage({user, chatId, message, model, webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) { - return generateChatTurn({user, chatId, message, model, webSearch, reasoning, onStatus, onChunk, onThinking, signal}); + async streamChatMessage({user, chatId, message, model, attachments = [], webSearch = false, reasoning = false, onStatus, onChunk, onThinking, signal = null}) { + return generateChatTurn({user, chatId, message, model, attachments, webSearch, reasoning, onStatus, onChunk, onThinking, signal}); }, async retryMessage({user, chatId, selectedModel, targetMessage, retryStyle = null}) { return withChatGenerationLock(chatId, async () => { @@ -574,6 +589,8 @@ export function createChatService(repositories, modelService, config, webSearchP const generated = await generateChatTurn({ user, chatId: cloned.chat.id, + attachments: attachmentService?.list(targetMessage.id).map(item => item.id) || [], + reuseChatId: chatId, message: content, model, webSearch, diff --git a/src/services/modelService.js b/src/services/modelService.js index fb71886..4875210 100644 --- a/src/services/modelService.js +++ b/src/services/modelService.js @@ -1,4 +1,5 @@ import {supportsReasoning} from "../../public/app/modelCapabilities.js"; +import {AttachmentError} from "./attachmentService.js"; class ModelServiceError extends Error { constructor(code, message, status = 500) { @@ -196,7 +197,7 @@ export function createModelService(config, overrides = {}) { const response = await fetchImpl(`${config.modelApiBaseUrl}/chat`, { method: "POST", headers: {"Content-Type": "application/json"}, - body: JSON.stringify(buildRequestBody(model, messagesPayload, options.generation, options.reasoning)), + body: JSON.stringify({...buildRequestBody(model, messagesPayload, options.generation, options.reasoning), ...(options.format ? {format: options.format} : {})}), signal }); @@ -332,6 +333,7 @@ export function createModelService(config, overrides = {}) { }; }, mapError(error) { + if (error instanceof AttachmentError) return {status: error.status, body: {error: error.message}}; if (error instanceof ModelServiceError) { return {status: error.status, body: {error: error.message, code: error.code}}; } diff --git a/src/services/webSearchService.js b/src/services/webSearchService.js index 485c149..a715c6a 100644 --- a/src/services/webSearchService.js +++ b/src/services/webSearchService.js @@ -106,14 +106,16 @@ export function buildWebSearchQueries(message, decision = decideWebSearch(messag } /** Retrieve before generation; only try a second date query when the first has no usable evidence. */ -export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}} = {}) { +export async function retrieveWebSearch(message, provider, {signal, onSearching = () => {}, referenceText = ""} = {}) { const decision = decideWebSearch(message); if (!decision.shouldSearch) return {decision, state: "skipped", queries: [], candidates: [], sources: null}; onSearching(); const queries = []; let candidates = []; let sources = []; - for (const query of buildWebSearchQueries(message, decision)) { + // Image evidence may clarify "is this accurate?"; it must never alter the user's tool decision. + const searchMessage = referenceText ? `${message} ${referenceText}` : message; + for (const query of buildWebSearchQueries(searchMessage, decision)) { queries.push(query); try { const results = await provider.search(query, { @@ -125,7 +127,7 @@ export async function retrieveWebSearch(message, provider, {signal, onSearching const merged = [...results, ...candidates].sort((a, b) => Number(Boolean(b?.snippet || b?.content)) - Number(Boolean(a?.snippet || a?.content))); candidates = normalizeSearchCandidates(merged, MAX_CANDIDATES * 2); - sources = selectSearchResults(message, candidates, decision); + sources = selectSearchResults(searchMessage, candidates, decision); if (sources.length) break; } catch (error) { if (signal?.aborted || queries.length === 1) throw error; diff --git a/src/services/workspaceImportService.js b/src/services/workspaceImportService.js index a0f25e6..857c384 100644 --- a/src/services/workspaceImportService.js +++ b/src/services/workspaceImportService.js @@ -86,6 +86,7 @@ function validateMessage(message, chatIndex, messageIndex, errors) { content, model_name: optionalText(message.model_name || message.modelName), thinking: optionalText(message.thinking), + attachments: asArray(message.attachments).filter(item => item && typeof item.id === "string" && /^[a-f0-9-]{36}$/.test(item.id)).slice(0, 100).map(item => ({id: item.id})), ...(message.sources ? {sources: parseSources(message.sources)} : {}), retryVariants, retryActiveIndex: Number.isInteger(Number(message.retryActiveIndex ?? message.retry_active_index))