| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* feat: Add automated bookmark backup system
Implements a comprehensive automated backup feature for user bookmarks with the following capabilities:
Database Schema:
- Add backupSettings table to store user backup preferences (enabled, frequency, retention)
- Add backups table to track backup records with status and metadata
- Add BACKUP asset type for storing compressed backup files
- Add migration 0066_add_backup_tables.sql
Background Workers:
- Implement BackupSchedulingWorker cron job (runs daily at midnight UTC)
- Create BackupWorker to process individual backup jobs
- Deterministic scheduling spreads backup jobs across 24 hours based on user ID hash
- Support for daily and weekly backup frequencies
- Automated retention cleanup to delete old backups based on user settings
Export & Compression:
- Reuse existing export functionality for bookmark data
- Compress exports using Node.js built-in zlib (gzip level 9)
- Store compressed backups as assets with proper metadata
- Track backup size and bookmark count for statistics
tRPC API:
- backups.getSettings - Retrieve user backup configuration
- backups.updateSettings - Update backup preferences
- backups.list - List all user backups with metadata
- backups.get - Get specific backup details
- backups.delete - Delete a backup
- backups.download - Download backup file (base64 encoded)
- backups.triggerBackup - Manually trigger backup creation
UI Components:
- BackupSettings component with configuration form
- Enable/disable automatic backups toggle
- Frequency selection (daily/weekly)
- Retention period configuration (1-365 days)
- Backup list table with download and delete actions
- Manual backup trigger button
- Display backup stats (size, bookmark count, status)
- Added backups page to settings navigation
Technical Details:
- Uses Restate queue system for distributed job processing
- Implements idempotency keys to prevent duplicate backups
- Background worker concurrency: 2 jobs at a time
- 10-minute timeout for large backup exports
- Proper error handling and logging throughout
- Type-safe implementation with Zod schemas
* refactor: simplify backup settings and asset handling
- Move backup settings from separate table to user table columns
- Update BackupSettings model to use static methods with users table
- Remove download mutation in favor of direct asset links
- Implement proper quota checks using QuotaService.checkStorageQuota
- Update UI to use new property names and direct asset downloads
- Update shared types to match new schema
Key changes:
- backupSettingsTable removed, settings now in users table
- Backup downloads use direct /api/assets/{id} links
- Quota properly validated before creating backup assets
- Cleaner separation of concerns in tRPC models
* migration
* use zip instead of gzip
* fix drizzle
* fix settings
* streaming json
* remove more dead code
* add e2e tests
* return backup
* poll for backups
* more fixes
* more fixes
* fix test
* fix UI
* fix delete asset
* fix ui
* redirect for backup download
* cleanups
* fix idempotency
* fix tests
* add ratelimit
* add error handling for background backups
* i18n
* model changes
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* refactor(trpc): extract rate limiter into dedicated plugin
Move the rate limiting middleware from the trpc package to the
centralized plugins package. This improves code organization by
consolidating all plugins in a single location.
Changes:
- Created packages/plugins/trpc-ratelimit/ plugin
- Moved rate limiter from packages/trpc/rateLimit.ts to
packages/plugins/trpc-ratelimit/src/index.ts
- Added trpc-ratelimit export to plugins package.json
- Added @trpc/server dependency to plugins package
- Updated trpc package to import from @karakeep/plugins/trpc-ratelimit
- Added @karakeep/plugins dependency to trpc package
- Removed packages/trpc/plugins/ directory
* refactor(plugins): decouple rate limiter from tRPC
Refactor the rate limiting plugin to be framework-agnostic, allowing
it to be used outside of tRPC contexts. The plugin now has a generic
core with a tRPC-specific adapter.
Changes:
- Renamed trpc-ratelimit to ratelimit plugin
- Created generic RateLimiter class with framework-agnostic API
- Added checkRateLimit() method that returns allow/deny results
- Created separate tRPC adapter (src/trpc.ts) that uses the generic core
- Exported both generic (RateLimiter, globalRateLimiter) and
tRPC-specific (createRateLimitMiddleware) APIs
- Updated trpc package to import from @karakeep/plugins/ratelimit
- Updated plugins package.json exports
Benefits:
- Rate limiter can now be used in any context (HTTP handlers, WebSocket, etc.)
- Cleaner separation of concerns
- Easy to create adapters for other frameworks
- Generic API allows for custom error handling
* refactor(plugins): integrate rate limiter with plugin registry
Refactor the rate limiting plugin to use the centralized plugin
system with PluginManager, making it consistent with other plugins
like queue and search providers.
Changes:
- Added RateLimit plugin type to PluginType enum
- Created RateLimitClient interface in packages/shared/ratelimiting.ts
- Created RateLimitProvider class implementing PluginProvider
- Updated plugin to auto-register with PluginManager on import
- Updated tRPC adapter to use getRateLimitClient() from PluginManager
- Added ratelimit plugin to loadAllPlugins() in shared-server
- Updated shared/plugins.ts with RateLimit type mapping
Benefits:
- Consistent plugin architecture across the codebase
- Rate limiter can be swapped with alternative implementations
- Centralized plugin management and logging
- Better separation of concerns
- Framework-agnostic core with tRPC adapter pattern
* refactor(trpc): move rate limit middleware to trpc package
Move the tRPC-specific rate limiting middleware from the plugins
package to the trpc package, making the plugins package
framework-agnostic.
Changes:
- Moved packages/plugins/ratelimit/src/trpc.ts to
packages/trpc/lib/rateLimit.ts
- Updated packages/trpc/index.ts to import from local lib/rateLimit
- Removed tRPC export from packages/plugins/ratelimit/index.ts
- Removed @trpc/server dependency from packages/plugins/package.json
Benefits:
- plugins package is now framework-agnostic
- tRPC-specific code lives in the trpc package where it belongs
- Cleaner separation of concerns
- Rate limiter plugin can be used in any context without tRPC
* refactor(plugins): rename to ratelimit-memory and add tests
Rename the rate limiting plugin from "ratelimit" to "ratelimit-memory"
to better indicate it's an in-memory implementation. This naming leaves
room for future implementations like ratelimit-redis. Also added
comprehensive test coverage.
Changes:
- Renamed packages/plugins/ratelimit to ratelimit-memory
- Updated package.json export from ./ratelimit to ./ratelimit-memory
- Updated shared-server to import @karakeep/plugins/ratelimit-memory
- Added comprehensive unit tests (index.test.ts):
- Rate limit enforcement tests
- Window expiration tests
- Identifier and path isolation tests
- Reset functionality tests
- Cleanup mechanism tests
- Added provider integration tests (provider.test.ts):
- PluginProvider interface compliance
- Client singleton behavior
- End-to-end rate limiting functionality
Benefits:
- More descriptive plugin name indicating the storage mechanism
- Better test coverage ensuring reliability
- Easier to add alternative implementations (Redis, etc.)
* change the api to only take the key
* move the serverConfig check to the trpc
* fix lockfile
* get rid of the timer
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* refactor: consolidate plugin packages into single plugins directory
- Create new `packages/plugins` directory with consolidated package.json
- Move queue-liteque, queue-restate, and search-meilisearch to subdirectories
- Update imports in packages/shared-server/src/plugins.ts
- Remove individual plugin package directories
- Update shared-server dependency to use @karakeep/plugins
This reduces overhead of maintaining multiple separate packages for plugins.
* refactor: consolidate plugin config files to root level
- Move .oxlintrc.json to packages/plugins root
- Move vitest.config.ts to packages/plugins root
- Update vitest config paths to work from root
- Remove individual config files from plugin subdirectories
This reduces configuration duplication across plugin subdirectories.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
| |
|
|
|
|
|
| |
* Add admin maintenance job to migrate large inline HTML
* add cursor
* more fixes
|
| |
|
|
|
| |
* refactor: generalize admin maintenance queue
* more fixes
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WIP: Initial restate integration
* add retry
* add delay + idempotency
* implement concurrency limits
* add admin stats
* add todos
* add id provider
* handle onComplete failures
* add tests
* add pub key and fix logging
* add priorities
* fail call after retries
* more fixes
* fix retries left
* some refactoring
* fix package.json
* upgrade sdk
* some test cleanups
|
| | |
|
| | |
|
| | |
|
| |
|
