Introduces a dedicated `checkUrl` tRPC endpoint that directly queries the
database for URL existence instead of going through the full search pipeline.
The endpoint uses the same LIKE matching semantics as the `url:` search
qualifier, then normalizes URLs (stripping hash fragments and trailing slashes)
for exact comparison server-side. This replaces the browser extension's
previous approach of calling searchBookmarks with `url:` and filtering
client-side.
* feat: add synchronized reading progress for bookmarks
Track and restore reading position in reader mode across web and mobile.
Uses text character offsets (matching the highlights pattern) to find
the first visible paragraph and scroll back to it on return.
- Add readingProgressOffset field to bookmarkLinks table
- Add updateReadingProgress tRPC mutation
- Create shared useReadingProgress hook with auto-save on unload
- Integrate in web full-screen reader (/reader/[bookmarkId])
- Integrate in mobile reader mode via WebView injectedJavaScript
- Only tracks progress for bookmark owners
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* it works
* align mobile functions
* fmt
* use generator for webview js fns, remove empty index.ts
* revert comments
* move shared fns into core
* address pr review
* watch core too
* address some pr comments
* fix race condition and jump when scrolling
* fix dead code and error handling, simplify some duplicate logic
* move reading progress to separate table
* remove arbitrary >100 char threshold for updates to progress
* only allow reading progress for link bookmarks, error on others
* add tests
* add tests for shared access reading progress
* simplify reading-progress hooks, visibility checks, remove char threshold on mobile
* remove unnecessary restoration lock
* remove unnecessary exports from reading-progress-core
* move position restoration to onload, more deterministic
* cleanup effects
* Update packages/shared-react/hooks/reading-progress.ts
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* fix function signatures in generated js comment
* remove verbose comments
* add percentage progress fallback
* address comments
* some code dedupe and simplification
* missed one
* Remove lastKnownPositionRef
* remove unnecessary reading-progress-core test
* update savePosition comment
* drop the esbuild generated code
* re-add the db migrations
* wire the mobile dom with the reading progress
* banner for reading progress instead of hiding content
* dedup code from reader view and preview page
* more simplifications
* add a sticky progress bar tracker at the top
* more code simplification
* rename test
* extract reading progress into its own handler
* more polish
* fix scroll behavior of full page bookmark preview
* more polish
* fix pnpm-lock
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Mohamed Bassem <me@mbassem.com>
* feat: add OAuth auto-redirect functionality
Add OAUTH_AUTO_REDIRECT environment variable to automatically redirect
users to the OAuth provider when both password authentication is disabled
and an OAuth provider is configured.
Changes:
- Add OAUTH_AUTO_REDIRECT config in packages/shared/config.ts
- Create OAuthAutoRedirect component for client-side redirect logic
- Update SignInForm to include auto-redirect functionality
- Add oauthAutoRedirect to client config context
- Document new environment variable in configuration docs
This improves user experience by eliminating unnecessary clicks when
OAuth is the only available authentication method.
Fixes #1189
Co-authored-by: Mohamed Bassem <MohamedBassem@users.noreply.github.com>
* review comments
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Mohamed Bassem <MohamedBassem@users.noreply.github.com>
* feat(mobile): make the bookmark edit UIs look more native
* fix: dont invalidate getBookmark query on bookmark deletion
* feat: add list definitions and memberships to JSON export and backup
Include list definitions (name, icon, type, hierarchy) and per-bookmark
list memberships in both the JSON export endpoint and backup worker output.
The parser is updated to convert list data back into folder paths on import.
Streaming memory bounds are preserved by fetching memberships per batch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* more fixes
* more fixes
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(mobile): make the bookmark edit UIs look more native
* fix: dont invalidate getBookmark query on bookmark deletion
* feat: run metascraper & readability parsing in a subprocess
Extract HTML parsing (metascraper metadata extraction + readability
content extraction) into a separate Node.js subprocess. This prevents
OOM crashes in the parser from killing the entire worker process.
The subprocess communicates via JSON over stdin/stdout, and all log
output is redirected to stderr. Two new config vars control the
subprocess heap size (CRAWLER_PARSE_MAX_OLD_SPACE_SIZE_MB, default
512MB) and timeout (CRAWLER_PARSE_TIMEOUT_SEC, default 60s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix format
* some fixes
* some fixes
* some fixes
* last fix
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add source filter to query language
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* autocomplete source
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When a bookmark is deleted before the rule engine worker processes its
event, the worker would throw an error, triggering failure metrics,
error logging, and retries. This changes both the worker and
RuleEngine.forBookmark to gracefully skip processing with an info log
instead.
Co-authored-by: Claude <noreply@anthropic.com>
Instruments the better-sqlite3 driver so that every prepared statement
execution (run/get/all) produces an OTel span with db.system,
db.statement, and db.operation attributes. The instrumentation is a
no-op when no TracerProvider is registered (i.e. tracing is disabled).
https://claude.ai/code/session_01JZut7LqeHPUKAFbFLfVP8F
* feat(ocr): add LLM-based OCR support alongside Tesseract
Add support for using configured LLM inference providers (OpenAI or Ollama)
for OCR text extraction from images as an alternative to Tesseract.
Changes:
- Add OCR_USE_LLM environment variable flag (default: false)
- Add buildOCRPrompt function for LLM-based text extraction
- Add readImageTextWithLLM function in asset preprocessing worker
- Update extractAndSaveImageText to route between Tesseract and LLM OCR
- Update documentation with the new configuration option
When OCR_USE_LLM is enabled, the system uses the configured inference model
to extract text from images. If no inference provider is configured, it
falls back to Tesseract.
https://claude.ai/code/session_01Y7h7kDAmqXKXEWDmWbVkDs
* format
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add `tag:` as an alternative syntax to `#` for tag search queries,
and `!` as an alternative to `-` for negating qualifiers. This provides
more intuitive syntax options for users who prefer text-based qualifiers
over special characters.
Co-authored-by: Claude <noreply@anthropic.com>
* feat: Add attachedBy field to updateTags endpoint
This change allows callers to specify the attachedBy field when updating
tags on a bookmark. The field defaults to "human" if not provided,
maintaining backward compatibility with existing code.
Changes:
- Added attachedBy field to zManipulatedTagSchema with default "human"
- Updated updateTags endpoint to use the specified attachedBy value
- Created mapping logic to correctly assign attachedBy to each tag
* fix(cli): migrate bookmark source in migration command
* fix
* reduce queries
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix: parallelize queue enqueues in bookmark routes
* fix: guard meilisearch client init with mutex
* fix: fix propagation of last error in restate
* fix: don't fail invocations when the job fails
* fix: add a timeout around the worker runner logic
* fix: add leases to handle dangling semaphores
* feat: separate dispatchers and runners
* add a test
* fix silent promise failure
* Use the Ollama generate endpoint instead of chat
Ollama has two API endpoints for text generation. There is a chat endpoint for interactive and interactive chat like generation of text and there is a generate endpoint that is used one one-shot prompts, such as summarization tasks and similar things.
Karakeep used the chat endpoint that resulted in odd summaries. This commit makes karakeep use the generate endpoint instead, which results in better and more compact summaries.
* format
* feat: add "URL Does Not Contain" condition to rule engine
Add a new condition type `urlDoesNotContain` that allows users to create
rules based on URLs that do NOT contain specific strings. This enables
more flexible rule configurations, such as:
- Automatically adding bookmarks to a "Read Later" list if the URL
does not contain "reddit.com" or "youtube.com"
Changes:
- Added `urlDoesNotContain` condition type to Zod schema
- Implemented evaluation logic in RuleEngine
- Added UI support in ConditionBuilder component
- Added translation key for new condition type
- Added test coverage for the new condition
Fixes #2259
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Mohamed Bassem <MohamedBassem@users.noreply.github.com>
* fix type link
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Mohamed Bassem <MohamedBassem@users.noreply.github.com>
* feat: add OpenTelemetry tracing infrastructure
Introduce distributed tracing capabilities using OpenTelemetry:
- Add @opentelemetry packages to shared-server for tracing
- Create tracing utility module with span helpers (withSpan, addSpanEvent, etc.)
- Add tRPC middleware for automatic span creation on API calls
- Initialize tracing in API and workers entry points
- Add demo instrumentation to bookmark creation and crawler worker
- Add configuration options (OTEL_TRACING_ENABLED, OTEL_EXPORTER_OTLP_ENDPOINT, etc.)
- Document tracing configuration in environment variables docs
When enabled, traces are collected for tRPC calls, bookmark creation flow,
and crawler operations, with support for any OTLP-compatible backend (Jaeger, Tempo, etc.)
* refactor: remove tracing from workers for now
Keep tracing infrastructure but remove worker instrumentation:
- Remove tracing initialization from workers entry point
- Remove tracing instrumentation from crawler worker
- Fix formatting in tracing files
The tracing infrastructure remains available for future use.
* add hono and next tracing
* remove extra span logging
* more fixes
* update config
* some fixes
* upgrade packages
* remove unneeded packages
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add OPENAI_PROXY_URL configuration and support for proxy in OpenAIInferenceClient
* docs: add OPENAI_PROXY_URL configuration for proxy support in OpenAI API requests
* format
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
* fix: preserve retry count when rate-limited jobs are rescheduled
Previously, when a domain was rate-limited in the crawler worker,
the job would be re-enqueued as a new job, which reset the failure
count. This meant rate-limited jobs could retry indefinitely without
respecting the max retry limit.
This commit introduces a RateLimitRetryError exception that signals
the queue system to retry the job after a delay without counting it
as a failed attempt. The job is retried within the same invocation,
preserving the original retry count.
Changes:
- Add RateLimitRetryError class to shared/queueing.ts
- Update crawler worker to throw RateLimitRetryError instead of re-enqueuing
- Update Restate queue service to handle RateLimitRetryError with delay
- Update Liteque queue wrapper to handle RateLimitRetryError with delay
This ensures that rate-limited jobs respect the configured retry limits
while still allowing for delayed retries when domains are rate-limited.
* refactor: use liteque's native RetryAfterError for rate limiting
Instead of manually handling retries in a while loop, translate
RateLimitRetryError to liteque's native RetryAfterError. This is
cleaner and lets liteque handle the retry logic using its built-in
mechanism.
* test: add tests for RateLimitRetryError handling in restate queue
Added comprehensive tests to verify that:
1. RateLimitRetryError delays retry appropriately
2. Rate-limited retries don't count against the retry limit
3. Jobs can be rate-limited more times than the retry limit
4. Regular errors still respect the retry limit
These tests ensure the queue correctly handles rate limiting
without exhausting retry attempts.
* lint & format
* fix: prevent onError callback for RateLimitRetryError
Fixed two issues with RateLimitRetryError handling in restate queue:
1. RateLimitRetryError now doesn't trigger the onError callback since
it's not a real error - it's an expected rate limiting behavior
2. Check for RateLimitRetryError in runWorkerLogic before calling onError,
ensuring the instanceof check works correctly before the error gets
further wrapped by restate
Updated tests to verify onError is not called for rate limit retries.
* fix: catch RateLimitRetryError before ctx.run wraps it
Changed approach to use a discriminated union instead of throwing
and catching RateLimitRetryError. Now we catch the error inside the
ctx.run callback before it gets wrapped by restate's TerminalError,
and return a RunResult type that indicates success, rate limit, or error.
This fixes the issue where instanceof checks would fail because
ctx.run wraps all errors in TerminalError.
* more fixes
* rename error name
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: Add owner icon to bookmarks in shared lists
Display a small icon showing the bookmark owner's name and email on hover when viewing bookmarks from other users in shared lists. The icon appears in the top-right corner of bookmark cards across all layout types (grid, list, compact).
Changes:
- Add user field to ZBookmark type to include owner name and email
- Update bookmark queries to fetch user information via join
- Create BookmarkOwnerIcon component with tooltip showing owner details
- Integrate owner indicator into BookmarkLayoutAdaptingCard for all layouts
- Only show icon for bookmarks not owned by current user
* use icons in more places
* remove tooltip providers
* fix non list context
---------
Co-authored-by: Claude <noreply@anthropic.com>
When importing bookmarks from an HTML file, empty H3 tags (folder names)
are now replaced with "Unnamed" to prevent import failures.
Fixes #2299
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Mohamed Bassem <MohamedBassem@users.noreply.github.com>
* feat: Add per-user settings to disable auto-tagging and auto-summarization
This commit adds user-level controls for AI features when they are enabled
on the server. Users can now toggle auto-tagging and auto-summarization
on/off from the AI Settings page.
Changes:
- Added autoTaggingEnabled and autoSummarizationEnabled fields to user table
- Updated user settings schemas and API endpoints to handle new fields
- Modified inference workers to check user preferences before processing
- Added toggle switches to AI Settings page (only visible when server has features enabled)
- Generated database migration for new fields
- Exposed enableAutoTagging and enableAutoSummarization in client config
The settings default to null (use server default). When explicitly set to false,
the user's bookmarks will skip the respective AI processing.
* revert migration
* i18n
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add initial impl
* fix some format inconsistencies, add indicator in user settings when local is out of sync
* Fix sliders in user settings, unify constants and formatting
* address CodeRabbit suggestions
* add mobile implementation
* address coderabbit nitpicks
* fix responsiveness of the reader settings popover
* Move more of the web UI strings to i18n
* update translations for more coverage
* remove duplicate logic/definitions
* fix android font family
* add shared reading setting hook between web and mobile
* unify reader settings context for both web and mobile
* remove unused export
* address coderabbit suggestions
* fix tests
- Added ASSET_PREPROCESSING_JOB_TIMEOUT_SEC environment variable with default of 60 seconds (increased from hardcoded 30 seconds)
- Updated worker to use the configurable timeout from serverConfig
- Added documentation for the new configuration option