| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* feat: Add automated bookmark backup system
Implements a comprehensive automated backup feature for user bookmarks with the following capabilities:
Database Schema:
- Add backupSettings table to store user backup preferences (enabled, frequency, retention)
- Add backups table to track backup records with status and metadata
- Add BACKUP asset type for storing compressed backup files
- Add migration 0066_add_backup_tables.sql
Background Workers:
- Implement BackupSchedulingWorker cron job (runs daily at midnight UTC)
- Create BackupWorker to process individual backup jobs
- Deterministic scheduling spreads backup jobs across 24 hours based on user ID hash
- Support for daily and weekly backup frequencies
- Automated retention cleanup to delete old backups based on user settings
Export & Compression:
- Reuse existing export functionality for bookmark data
- Compress exports using Node.js built-in zlib (gzip level 9)
- Store compressed backups as assets with proper metadata
- Track backup size and bookmark count for statistics
tRPC API:
- backups.getSettings - Retrieve user backup configuration
- backups.updateSettings - Update backup preferences
- backups.list - List all user backups with metadata
- backups.get - Get specific backup details
- backups.delete - Delete a backup
- backups.download - Download backup file (base64 encoded)
- backups.triggerBackup - Manually trigger backup creation
UI Components:
- BackupSettings component with configuration form
- Enable/disable automatic backups toggle
- Frequency selection (daily/weekly)
- Retention period configuration (1-365 days)
- Backup list table with download and delete actions
- Manual backup trigger button
- Display backup stats (size, bookmark count, status)
- Added backups page to settings navigation
Technical Details:
- Uses Restate queue system for distributed job processing
- Implements idempotency keys to prevent duplicate backups
- Background worker concurrency: 2 jobs at a time
- 10-minute timeout for large backup exports
- Proper error handling and logging throughout
- Type-safe implementation with Zod schemas
* refactor: simplify backup settings and asset handling
- Move backup settings from separate table to user table columns
- Update BackupSettings model to use static methods with users table
- Remove download mutation in favor of direct asset links
- Implement proper quota checks using QuotaService.checkStorageQuota
- Update UI to use new property names and direct asset downloads
- Update shared types to match new schema
Key changes:
- backupSettingsTable removed, settings now in users table
- Backup downloads use direct /api/assets/{id} links
- Quota properly validated before creating backup assets
- Cleaner separation of concerns in tRPC models
* migration
* use zip instead of gzip
* fix drizzle
* fix settings
* streaming json
* remove more dead code
* add e2e tests
* return backup
* poll for backups
* more fixes
* more fixes
* fix test
* fix UI
* fix delete asset
* fix ui
* redirect for backup download
* cleanups
* fix idempotency
* fix tests
* add ratelimit
* add error handling for background backups
* i18n
* model changes
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
| |
* feat: add gif asset type support
* skip inference for gis
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
|
| |
|
|
|
|
|
|
| |
This commit allows the following mime types to be uploaded and attached
as video assets on bookmarks.
- video/mp4
- video/webm
- video/x-matroska
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* feat: Add support for S3 as an asset storage layer. Fixes #305
* some minor fixes
* use bulk deletion api
* stream the file to s3
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Updated pdf2json to 3.1.5
* Extract and store a screenshot from PDF files using pdf2pic
* Installing graphicsmagick and ghostscript
* Generate Missing PDF screenshot with tidyAssets worker for backward support
* Display PDF screenshot instead of the PDF in web if it exists.
* Display PDF screenshot in mobile app if exists.
* Updated pnpm-lock.yaml
* Removed console.log
* Revert the unnecessary changes in package.json
* Revert pnpm-lock changes
* Prevent rendering PDF files if the screenshot is not generated
* refactor: replace useEffect with useMemo for section initialization
* feat: show PDF file download button and handle large PDFs by defaulting to screenshot view
* feat: add file size to openapi spec
* feature: Add Assets preprocessing in fix mode to admin actions
* i18n: add reprocess_assets_fix_mode translation
* i18n: Add missing ar translations
* A bunch of fixes
* Fix openspec schema
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Allow downloading more content from a webpage and index it #215
Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
* Some tweaks
* Drop the dependency on the yt-dlp wrapper
* Update openapi specs
* Dont log an error when the url is not supported
* Better handle supported websites that dont download anything
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
|
| | |
|
| | |
|
| |
|
|
| |
#273
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* feature request: pdf support #28
Added a new sourceUrl column to the asset bookmarks
Added transforming a link bookmark pointing at a pdf to an asset bookmark
made sure the "View Original" link is also shown for asset bookmarks that have a sourceURL
updated gitignore for IDEA
* remove pdf parsing from the crawler
* extract the http logic into its own function to avoid duplicating the post-processing actions (openai/index)
* Add 5s timeout to the content type fetch
---------
Co-authored-by: MohamedBassem <me@mbassem.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* feature: Add PDF support
* fix: PDF feature enhancements
* fix: Freeze expo-share-intent version to prevent breaking changes
* fix: set endOfLine to auto for cross-platform development
* fix: Upgrading eslint/parser and eslint-plugin to 7.6.0 to solve the linting issues
* fix: enhancing PDF feature
* fix: Allowing null in fiename for backward compatibility
* fix: update pnpm file with pnpm 9.0.0-alpha-8
* fix:(web): PDF Preview for web
|
| | |
|
| | |
|
| |
|
