From e2877b458fedbf9a75be16439d18b9de9f5ad924 Mon Sep 17 00:00:00 2001 From: Mohamed Bassem Date: Fri, 28 Nov 2025 10:29:49 +0000 Subject: fix: lazy load js-tiktoken in prompts module (#2176) * feat: lazy load tiktoken to reduce memory footprint The js-tiktoken module loads a large encoding dictionary into memory immediately on import. This change defers the loading of the encoding until it's actually needed by using a lazy getter pattern. This reduces memory usage for processes that import this module but don't actually use the token encoding functions. * fix: use createRequire for lazy tiktoken import in ES module The previous implementation used bare require() which fails at runtime in ES modules (ReferenceError: require is not defined). This fixes it by using createRequire from Node's 'module' package, which creates a require function that works in ES module contexts. * refactor: convert tiktoken lazy loading to async dynamic imports Changed from createRequire to async import() for lazy loading tiktoken, making buildTextPrompt and buildSummaryPrompt async. This is cleaner for ES modules and properly defers the large tiktoken encoding data until it's actually needed. Updated all callers to await these async functions: - packages/trpc/routers/bookmarks.ts - apps/workers/workers/inference/tagging.ts - apps/workers/workers/inference/summarize.ts - apps/web/components/settings/AISettings.tsx (converted to useEffect) * feat: add untruncated prompt builders for UI previews Added buildTextPromptUntruncated and buildSummaryPromptUntruncated functions that don't require token counting or truncation. These are synchronous and don't load tiktoken, making them perfect for UI previews where exact token limits aren't needed. Updated AISettings.tsx to use these untruncated versions, eliminating the need for useEffect/useState and avoiding unnecessary tiktoken loading in the browser. * fix * fix --------- Co-authored-by: Claude --- packages/trpc/routers/bookmarks.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'packages/trpc/routers') diff --git a/packages/trpc/routers/bookmarks.ts b/packages/trpc/routers/bookmarks.ts index 05ffa240..f68d5ada 100644 --- a/packages/trpc/routers/bookmarks.ts +++ b/packages/trpc/routers/bookmarks.ts @@ -954,7 +954,7 @@ Author: ${bookmark.author ?? ""} }, }); - const summaryPrompt = buildSummaryPrompt( + const summaryPrompt = await buildSummaryPrompt( serverConfig.inference.inferredTagLang, prompts.map((p) => p.text), bookmarkDetails, -- cgit v1.2.3-70-g09d2