aboutsummaryrefslogtreecommitdiffstats
path: root/apps/workers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* feat: Store large html content in the asset dbMohamed Bassem2025-07-065-9/+135
|
* feat: Add per user storage quotaMohamed Bassem2025-07-064-75/+183
|
* feat(workers): Allow custmoizing max parallelism for a bunch of workers. ↵Mohamed Bassem2025-07-055-5/+7
| | | | Fixes #724
* fix(workers): A more lenient JSON parsing for LLM responses. Fixes #1267Mohamed Bassem2025-07-041-1/+39
|
* fix(workers): Disable the metascraper readability as it's causing slowness ↵Mohamed Bassem2025-06-221-2/+0
| | | | in worker
* fix(workers): Fix jsdom console logs leaking into worker logsMohamed Bassem2025-06-221-2/+3
|
* feat(workers): adding a local metascraper plugin for Reddit posts (#1302)David Woods2025-06-223-13/+115
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * chore: metascraper 5.x comes with its own types, including @types/metascraper is now redundant; also updating to latest versions of metascraper libraries * feat (workers): creating a local metascraper plugin for Reddit posts In the past, the preview images for bookmarks from Reddit links were poorly chosen. Reddit does not use opengraph tags, so metascraper-images simply looked for all images on the page and returned the first. This tended to be the profile picture for the poster for the Reddit link. This new plugin, using the existing metascraper framework, provides a better selection of image for the bookmark when the URL domain is 'reddit'. In addition, recent changes (I believe this was a side effect of adding the metascraper-author and/or the metascaper-publisher plugins, but it could also be related to the metascraper-readibility plugin) broke what used to be a good choice of bookmark title. Previously, titles looked like 'Tinyauth just reached 1000 stars! : r/selfhosted' with both thread title and subreddit mentioned. After this update, all Reddit posts now have the same title: 'The heart of the internet'. To return to the better format, this new metascraper-reddit plugin now attempts to retrieve the better title from reddit URLs. Note that in order to gain precendence in title selection, the 'metascraperReddit()' inclusion in the crawlerWorkers.ts metascraper instantiation list had to be moved above metascraperReadability(). * chore: updated Hoarder in text to Karakeep * chore: update metascraper versions fix for metascraper types has been merged; the expect-error comment can be removed * chore: merge with master --------- Co-authored-by: Mohamed Bassem <me@mbassem.com>
* feat(workers): migrate from puppeteer to playwright (#1296)Mael2025-06-222-34/+39
| | | | | | | | | | | | | | | | | | | | | | | * feat: convert to playwright Convert crawling to use Playwright instead of Chrome. - Update Dockerfile to include Playwright - Update crawler worker to use Playwright API - Update dependencies * feat: convert from Puppeteer to Playwright for crawling * feat: update docker-compose * use separate browser context for better isolation * skip chrome download in linux script * readd the stealth plugin --------- Co-authored-by: Mohamed Bassem <me@mbassem.com>
* chore: More oxlint changesMohamed Bassem2025-06-223-7/+4
|
* chore: migrate away from eslint to oxlint (#1642)xuatz2025-06-225-12/+27
| | | | | | | * chore: migrate away from eslint to oxlint * revert turbo task name lint * it seems like we can remove the seemingly default globals
* fix: Fix webhook not firing on deletion. Fixes #1613Mohamed Bassem2025-06-211-18/+19
|
* fix(workers): video downloader should log yt-dlp errors (#1624)irobot2025-06-211-3/+6
| | | | In the event that yt-dlp errors out, the error details should be logged. yt-dlp prints out the error message to stderr.
* feat: Allow specifying the overwrite mode for singlefile archives. Fixes #1125Mohamed Bassem2025-06-011-3/+3
|
* feat: Generate RSS feeds from lists (#1507)Mohamed Bassem2025-05-311-31/+3
| | | | | | | | | | | * refactor: Move bookmark utils from shared-react to shared * Expose RSS feeds for lists * Add e2e tests * Slightly improve the look of the share dialog * allow specifying a limit in the rss endpoint
* feat: Add AI auto summarization. Fixes #1163Mohamed Bassem2025-05-1812-100/+264
|
* feat: Allow enabling/disabling RSS feedsMohamed Bassem2025-05-171-0/+1
|
* feat: Implement generic rule engine (#1318)Mohamed Bassem2025-04-275-16/+133
| | | | | | | | | | | | | | | | | * Add schema for the new rule engine * Add rule engine backend logic * Implement the worker logic and event firing * Implement the UI changesfor the rule engine * Ensure that when a referenced list or tag are deleted, the corresponding event/action is * Dont show smart lists in rule engine events * Add privacy validations for attached tag and list ids * Move the rules logic into a models
* chore: rename missing files/conf from Hoarder to Karakeep (#1280)adripo2025-04-212-3/+3
| | | | | | | | | * refactor: Rename remaining project configuration from Hoarder to Karakeep * some fixes --------- Co-authored-by: Mohamed Bassem <me@mbassem.com>
* fix(workers): Fix dompurify to run on readability's input not outputMohamed Bassem2025-04-211-4/+12
|
* deps: Upgrade readability to 0.6 & adblocker to 2.5.1Mohamed Bassem2025-04-211-2/+2
|
* fix(workers): Close browser if connect on demand (#1151)Chang-Yen Tseng2025-04-161-0/+3
|
* feat: Add an MCP server for karakeepMohamed Bassem2025-04-131-1/+1
|
* chore: Rename hoarder packages to karakeepMohamedBassem2025-04-1214-73/+73
|
* feat(workers): Add CRAWLER_SCREENSHOT_TIMEOUT_SEC (#1155)Chang-Yen Tseng2025-03-271-10/+18
|
* feat(workers): Adds publisher and author og:meta tags to Bookmark (#1141)erik-nilcoast2025-03-223-1/+32
|
* deps: Upgrade pdfjs and dompurifyMohamed Bassem2025-03-221-4/+3
|
* feat(workers): allows videoWorker to use ytdlp command line arguments ↵erik-nilcoast2025-03-161-1/+2
| | | | specified in the config. Fixes #775 #792 (#1117)
* fix: Revert the accidental upgrade of deps. #1107Mohamed Bassem2025-03-101-1/+1
|
* build(deps): bump dompurify from 3.0.9 to 3.2.4 (#1102)dependabot[bot]2025-03-091-1/+1
| | | | | | | | | | | | | | Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.0.9 to 3.2.4. - [Release notes](https://github.com/cure53/DOMPurify/releases) - [Commits](https://github.com/cure53/DOMPurify/compare/3.0.9...3.2.4) --- updated-dependencies: - dependency-name: dompurify dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(workers): Small typo fix in assetPreprocessingWorker.tsChris2025-03-081-2/+2
|
* feat: Add support for 'ai tagged' as a webhook trigger. Fixes #1092Mohamed Bassem2025-03-071-0/+4
|
* fix(workers): Fix webhook events being sent regardless of selected user ↵Mohamed Bassem2025-03-071-46/+50
| | | | operation. Fixes #1092
* fix(workers): Fix pdf screenshotting not working when a text is extracted. ↵Mohamed Bassem2025-03-071-5/+9
| | | | Fixes #1096
* fix: Validate the output of the feed xml parser. Fixes #1085Mohamed Bassem2025-03-021-8/+19
|
* fix: Fix fixMode preventing images from getting taggedMohamed Bassem2025-03-021-1/+1
|
* fix: Move away from JSON outputs to structured outputs. Fixes #1047Mohamed Bassem2025-03-021-3/+6
|
* feat: Add PDF screenshot generation and display (#995)Ahmad Mujahid2025-02-173-40/+179
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Updated pdf2json to 3.1.5 * Extract and store a screenshot from PDF files using pdf2pic * Installing graphicsmagick and ghostscript * Generate Missing PDF screenshot with tidyAssets worker for backward support * Display PDF screenshot instead of the PDF in web if it exists. * Display PDF screenshot in mobile app if exists. * Updated pnpm-lock.yaml * Removed console.log * Revert the unnecessary changes in package.json * Revert pnpm-lock changes * Prevent rendering PDF files if the screenshot is not generated * refactor: replace useEffect with useMemo for section initialization * feat: show PDF file download button and handle large PDFs by defaulting to screenshot view * feat: add file size to openapi spec * feature: Add Assets preprocessing in fix mode to admin actions * i18n: add reprocess_assets_fix_mode translation * i18n: Add missing ar translations * A bunch of fixes * Fix openspec schema --------- Co-authored-by: Mohamed Bassem <me@mbassem.com>
* fix: Fix node22 error in worker container. Fixes #962Mohamed Bassem2025-02-021-1/+1
|
* fix: Dont rearchive singlefile uploads and consider them as archivesMohamed Bassem2025-02-021-2/+6
|
* fix: Abort all IO when workers timeout instead of detaching. Fixes #742Mohamed Bassem2025-02-014-22/+92
|
* deps: Upgrade typescript to 5.7Mohamed Bassem2025-02-012-4/+3
|
* build: Fix CI error because of mismatched cmdk versionMohamed Bassem2025-01-191-1/+2
|
* feat: Change webhooks to be configurable by usersMohamed Bassem2025-01-192-11/+14
|
* chore: set firefox user-agent for rss reader (#908)Meysam2025-01-191-1/+1
| | | fixes #906
* feat(webhook): Implement webhook functionality for bookmark events (#852)玄猫2025-01-193-11/+163
| | | | | | | | | | | | | | | | | | | | | | | | | * feat(webhook): Implement webhook functionality for bookmark events - Added WebhookWorker to handle webhook requests. - Integrated webhook triggering in crawlerWorker after video processing. - Updated main worker initialization to include WebhookWorker. - Enhanced configuration to support webhook URLs, token, and timeout. - Documented webhook configuration options in the documentation. - Introduced zWebhookRequestSchema for validating webhook requests. * feat(webhook): Update webhook handling and configuration - Changed webhook operation type from "create" to "crawled" in crawlerWorker and documentation. - Enhanced webhook retry logic in WebhookWorker to support multiple attempts. - Updated Docker configuration to include new webhook environment variables. - Improved validation for webhook configuration in shared config. - Adjusted zWebhookRequestSchema to reflect the new operation type. - Updated documentation to clarify webhook configuration options and usage. * minor modifications --------- Co-authored-by: Mohamed Bassem <me@mbassem.com>
* feat: Support customizing the summarization prompt. Fixes #731Mohamed Bassem2025-01-121-1/+1
|
* feat: Add support for singlefile extension uploads. #172Mohamed Bassem2025-01-112-6/+33
|
* chore: add format:fix and lint:fix scripts to all packagesMohamed Bassem (aider)2024-12-311-0/+2
|
* deps: Upgrade drizzle-orm to 0.38.3Mohamed Bassem2024-12-291-1/+1
|
* refactor: Move asset preprocessing to its own worker out of the inference workerMohamed Bassem2024-12-265-118/+231
|
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2026-01-29 18:06:07 +0000