From d3be75deb8784afdb123c4a1f3a7db4bd6f3c87d Mon Sep 17 00:00:00 2001 From: MohamedBassem Date: Fri, 22 Mar 2024 17:50:30 +0000 Subject: docs: Add screenshots, and update other docs --- README.md | 22 +++++--- docs/docs/01-intro.md | 2 +- docs/docs/02-installation.md | 25 +++------ docs/docs/03-configuration.md | 1 - docs/docs/04-quick-sharing.md | 17 ------ docs/docs/04-screenshots.md | 26 +++++++++ docs/docs/05-openai.md | 11 ---- docs/docs/05-quick-sharing.md | 17 ++++++ docs/docs/06-Development/01-setup.md | 68 ----------------------- docs/docs/06-Development/02-directories.md | 28 ---------- docs/docs/06-Development/03-database.md | 11 ---- docs/docs/06-openai.md | 11 ++++ docs/docs/07-Development/01-setup.md | 68 +++++++++++++++++++++++ docs/docs/07-Development/02-directories.md | 28 ++++++++++ docs/docs/07-Development/03-database.md | 11 ++++ docs/docs/07-Development/04-architecture.md | 10 ++++ docs/docs/07-security-considerations.md | 14 ----- docs/docs/08-security-considerations.md | 14 +++++ docs/static/img/architecture/arch.png | Bin 0 -> 106144 bytes docs/static/img/screenshots/admin.png | Bin 0 -> 237460 bytes docs/static/img/screenshots/all-lists.png | Bin 0 -> 188073 bytes docs/static/img/screenshots/all-tags.png | Bin 0 -> 278520 bytes docs/static/img/screenshots/bookmark-preview.png | Bin 0 -> 2138263 bytes docs/static/img/screenshots/homepage.png | Bin 0 -> 3196208 bytes docs/static/img/screenshots/settings.png | Bin 0 -> 259583 bytes docs/static/img/screenshots/share-sheet.png | Bin 0 -> 987806 bytes 26 files changed, 208 insertions(+), 176 deletions(-) delete mode 100644 docs/docs/04-quick-sharing.md create mode 100644 docs/docs/04-screenshots.md delete mode 100644 docs/docs/05-openai.md create mode 100644 docs/docs/05-quick-sharing.md delete mode 100644 docs/docs/06-Development/01-setup.md delete mode 100644 docs/docs/06-Development/02-directories.md delete mode 100644 docs/docs/06-Development/03-database.md create mode 100644 docs/docs/06-openai.md create mode 100644 docs/docs/07-Development/01-setup.md create mode 100644 docs/docs/07-Development/02-directories.md create mode 100644 docs/docs/07-Development/03-database.md create mode 100644 docs/docs/07-Development/04-architecture.md delete mode 100644 docs/docs/07-security-considerations.md create mode 100644 docs/docs/08-security-considerations.md create mode 100644 docs/static/img/architecture/arch.png create mode 100644 docs/static/img/screenshots/admin.png create mode 100644 docs/static/img/screenshots/all-lists.png create mode 100644 docs/static/img/screenshots/all-tags.png create mode 100644 docs/static/img/screenshots/bookmark-preview.png create mode 100644 docs/static/img/screenshots/homepage.png create mode 100644 docs/static/img/screenshots/settings.png create mode 100644 docs/static/img/screenshots/share-sheet.png diff --git a/README.md b/README.md index 91822414..f2582843 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ A self-hostable bookmark-everything app with a touch of AI for the data hoarders - 🔎 Full text search of all the content stored. - ✨ AI-based (aka chatgpt) automatic tagging. - 🔖 [Chrome plugin](https://chromewebstore.google.com/detail/hoarder/kgcjekpmcjjogibpjebkhaanilehneje) for quick bookmarking. -- 📱 [iOS shortcut](https://www.icloud.com/shortcuts/78734b46624c4a3297187c85eb50d800) for bookmarking content from the phone. A minimal mobile app is in the works. +- 📱 An iOS app that's pending apple's review. - 💾 Self-hosting first. - [Planned] Archiving the content for offline reading. @@ -22,6 +22,7 @@ A self-hostable bookmark-everything app with a touch of AI for the data hoarders - [Installation](https://docs.hoarder.app/installation) - [Configuration](https://docs.hoarder.app/configuration) +- [Screenshots](https://docs.hoarder.app/screenshots) - [Security Considerations](https://docs.hoarder.app/security-considerations) - [Development](https://docs.hoarder.app/Development/setup) @@ -53,11 +54,14 @@ I browse reddit, twitter and hackernews a lot from my phone. I frequently find i I'm a systems engineer in my day job (and have been for the past 7 years). I didn't want to get too detached from the web development world. I decided to build this app as a way to keep my hand dirty with web development, and at the same time, build something that I care about and will use everyday. -## Why not X? - -- [Pocket](getpocket.com): Pocket is what hooked me into the whole idea of read-it-later apps. I used it [a lot](https://blog.mbassem.com/2019/01/27/favorite-articles-2018/). However, I recently got into home-labbing and became obsessed with the idea of running my services in my home server. Hoarder is meant to be a self-hosting first app. -- [Omnivore](https://omnivore.app/): Omnivore is pretty cool open source read-it-later app. Unfortunately, it's heavily dependent on google cloud infra which makes self-hosting it quite hard. They published a [blog post](https://docs.omnivore.app/self-hosting/self-hosting.html) on how to run a minimal omnivore but it was lacking a lot of stuff. Self-hosting doesn't really seem to be a high priority for them, and that's something I care about, so I decided to build an alternative. -- [Instapaper](https://www.instapaper.com/): Not open source and not self-hostable. -- [memos](https://github.com/usememos/memos): I love memos. I have it running on my home server and it's one of my most used self-hosted apps. I, however, don't like the fact that it doesn't preview the content of the links I dump there and to be honest, it doesn't have to because that's not what it was designed for. It's just that I dump a lot of links there and I'd have loved if I'd be able to figure which link is that by just looking at my timeline. Also, given the variety of things I dump there, I'd have loved if it does some sort of automatic tagging for what I save there. This is exactly the usecase that I'm trying to tackle with Hoarder. -- [Wallabag](https://wallabag.it): Wallabag is a well-established open source read-it-later app written in php and I think it's the common recommendation on reddit for such apps. To be honest, I didn't give it a real shot, and the UI just felt a bit dated for my liking. Honestly, it's probably much more stable and feature complete than this app, but where's the fun in that? -- [Shiori](https://github.com/go-shiori/shiori): Shiori is meant to be an open source pocket clone written in Go. It ticks all the marks but doesn't have my super sophisticated AI-based tagging. (JK, I only found about it after I decided to build my own app, so here we are 🤷). +## Alternatives + +- [memos](https://github.com/usememos/memos): I love memos. I have it running on my home server and it's one of my most used self-hosted apps. It doesn't, however, archive or preview the links shared in it. It's just that I dump a lot of links there and I'd have loved if I'd be able to figure which link is that by just looking at my timeline. Also, given the variety of things I dump there, I'd have loved if it does some sort of automatic tagging for what I save there. This is exactly the usecase that I'm trying to tackle with Hoarder. +- [mymind](https://mymind.com/): Mymind is the closest alternative to this project and from where I drew a lot of inspirations. It's a commercial product though. +- [raindrop](https://raindrop.io): A polished open source bookmark manager that supports links, images and files. It's not self-hostable though. +- Bookmark managers (mostly focused on bookmarking links): + - [Pocket](getpocket.com): Pocket is what hooked me into the whole idea of read-it-later apps. I used it [a lot](https://blog.mbassem.com/2019/01/27/favorite-articles-2018/). However, I recently got into home-labbing and became obsessed with the idea of running my services in my home server. Hoarder is meant to be a self-hosting first app. + - [Linkwarden](https://linkwarden.app/): An open-source self-hostable bookmark manager that I ran for a bit in my homelab. It's focused mostly on links and supports collaborative collections. + - [Omnivore](https://omnivore.app/): Omnivore is pretty cool open source read-it-later app. Unfortunately, it's heavily dependent on google cloud infra which makes self-hosting it quite hard. They published a [blog post](https://docs.omnivore.app/self-hosting/self-hosting.html) on how to run a minimal omnivore but it was lacking a lot of stuff. Self-hosting doesn't really seem to be a high priority for them, and that's something I care about, so I decided to build an alternative. + - [Wallabag](https://wallabag.it): Wallabag is a well-established open source read-it-later app written in php and I think it's the common recommendation on reddit for such apps. To be honest, I didn't give it a real shot, and the UI just felt a bit dated for my liking. Honestly, it's probably much more stable and feature complete than this app, but where's the fun in that? + - [Shiori](https://github.com/go-shiori/shiori): Shiori is meant to be an open source pocket clone written in Go. It ticks all the marks but doesn't have my super sophisticated AI-based tagging. (JK, I only found about it after I decided to build my own app, so here we are 🤷). diff --git a/docs/docs/01-intro.md b/docs/docs/01-intro.md index 7418bbb6..4aae4b79 100644 --- a/docs/docs/01-intro.md +++ b/docs/docs/01-intro.md @@ -17,7 +17,7 @@ Hoarder is an open source "Bookmark Everything" app that uses AI for automatical - 🔎 Full text search of all the content stored. - ✨ AI-based (aka chatgpt) automatic tagging. - 🔖 [Chrome plugin](https://chromewebstore.google.com/detail/hoarder/kgcjekpmcjjogibpjebkhaanilehneje) for quick bookmarking. -- 📱 [iOS shortcut](https://www.icloud.com/shortcuts/78734b46624c4a3297187c85eb50d800) for bookmarking content from the phone. A minimal mobile app is in the works. +- 📱 An iOS app that's pending apple's review. - 💾 Self-hosting first. - [Planned] Archiving the content for offline reading. diff --git a/docs/docs/02-installation.md b/docs/docs/02-installation.md index 1ee01d79..0a25c7bf 100644 --- a/docs/docs/02-installation.md +++ b/docs/docs/02-installation.md @@ -16,37 +16,30 @@ Create a new directory to host the compose file and env variables. Download the docker compose file provided [here](https://github.com/MohamedBassem/hoarder-app/blob/main/docker/docker-compose.yml). ``` -$ wget https://raw.githubusercontent.com/MohamedBassem/hoarder-app/main/docker/docker-compose.yml +wget https://raw.githubusercontent.com/MohamedBassem/hoarder-app/main/docker/docker-compose.yml ``` ### 3. Populate the environment variables -You can use the env file template provided [here](https://github.com/MohamedBassem/hoarder-app/blob/main/.env.sample) and fill it manually using the documentation [here](/configuration). - -``` -$ wget https://raw.githubusercontent.com/MohamedBassem/hoarder-app/main/.env.sample -$ mv .env.sample .env -``` - -Alternatively, here is a minimal `.env` file to use: +To configure the app, create a `.env` file in the directory and add this minimal env file: ``` NEXTAUTH_SECRET=super_random_string -NEXTAUTH_URL= - HOARDER_VERSION=release - -MEILI_ADDR=http://meilisearch:7700 MEILI_MASTER_KEY=another_random_string ``` -You can use `openssl rand -base64 36` to generate the random strings. +You **should** change the random strings. You can use `openssl rand -base64 36` to generate the random strings. Persistent storage and the wiring between the different services is already taken care of in the docker compose file. +Keep in mind that every time you change the `.env` file, you'll need to re-run `docker compose up`. + +If you want more config params, check the config documentation [here](/configuration). + ### 4. Setup OpenAI -To enable automatic tagging, you'll need to configure open ai. This is optional though but hightly recommended. +To enable automatic tagging, you'll need to configure OpenAI. This is optional though but hightly recommended. - Follow [OpenAI's help](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key) to get an API key. - Add `OPENAI_API_KEY=` to the env file. @@ -60,5 +53,5 @@ Learn more about the costs of using openai [here](/openai). Start the service by running: ``` -$ docker compose up -d +docker compose up -d ``` diff --git a/docs/docs/03-configuration.md b/docs/docs/03-configuration.md index a9c02611..585d25b5 100644 --- a/docs/docs/03-configuration.md +++ b/docs/docs/03-configuration.md @@ -6,7 +6,6 @@ The app is mainly configured by environment variables. All the used environment | ---------------- | ------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | DATA_DIR | Yes | Not set | The path for the persistent data directory. This is where the db and the uploaded assets live. | | NEXTAUTH_SECRET | Yes | Not set | Random string used to sign the JWT tokens. Generate one with `openssl rand -base64 36`. | -| NEXTAUTH_URL | Yes | Not set | The url on which the service will be running on. E.g. (`https://demo.hoarder.app`). | | REDIS_HOST | Yes | localhost | The address of redis used by background jobs | | REDIS_POST | Yes | 6379 | The port of redis used by background jobs | | OPENAI_API_KEY | No | Not set | The OpenAI key used for automatic tagging. If not set, automatic tagging won't be enabled. More on that in [here](/openai). | diff --git a/docs/docs/04-quick-sharing.md b/docs/docs/04-quick-sharing.md deleted file mode 100644 index 05ff5448..00000000 --- a/docs/docs/04-quick-sharing.md +++ /dev/null @@ -1,17 +0,0 @@ -# Quick Sharing Extensions - -The whole point of Hoarder is making it easy to hoard the content. That's why there are a couple of - -## Mobile Apps - -mobile screenshot - - -- iOS app: TODO -- Android App: The app is built in using a cross-platform framework (react native). So technically, the android app should just work, but I didn't test it. If there's enough demand, I'll publish it to the google play store. - -## Chrome Extensions - -mobile screenshot - -- To quickly bookmark links, you can also use the chrome extension [here](https://chromewebstore.google.com/detail/hoarder/kgcjekpmcjjogibpjebkhaanilehneje). diff --git a/docs/docs/04-screenshots.md b/docs/docs/04-screenshots.md new file mode 100644 index 00000000..03137367 --- /dev/null +++ b/docs/docs/04-screenshots.md @@ -0,0 +1,26 @@ +# Screenshots + +## Homepage + +![Homepage](/img/screenshots/homepage.png) + +## Tags + +![All Tags](/img/screenshots/all-tags.png) + +## Lists + +![All Lists](/img/screenshots/all-lists.png) + +## Settings + +![Settings](/img/screenshots/settings.png) + +## Admin Panel + +![Ammin](/img/screenshots/admin.png) + + +## iOS Sharing + + diff --git a/docs/docs/05-openai.md b/docs/docs/05-openai.md deleted file mode 100644 index 91e37c07..00000000 --- a/docs/docs/05-openai.md +++ /dev/null @@ -1,11 +0,0 @@ -# OpenAI Costs - -This service uses OpenAI for automatic tagging. This means that you'll incur some costs if automatic tagging is enabled. There are two type of inferences that we do: - -## Text Tagging - -For text tagging, we use the `gpt-3.5-turbo-0125` model. This model is [extremely cheap](https://openai.com/pricing). Cost per inference varies depending on the content size per article. Though, roughly, You'll be able to generate tags for almost 1000+ bookmarks for less than $1. - -## Image Tagging - -For image uploads, we use the `gpt-4-vision-preview` model for extracting tags from the image. You can learn more about the costs of using this model [here](https://platform.openai.com/docs/guides/vision/calculating-costs). To lower the costs, we're using the low resolution mode (fixed number of tokens regardless of image size). The gpt-4 model, however, is much more expensive than the `gpt-3.5-turbo`. Currently, we're using around 350 token per image inference which ends up costing around $0.01 per inference. So around 10x more expensive than the text tagging. diff --git a/docs/docs/05-quick-sharing.md b/docs/docs/05-quick-sharing.md new file mode 100644 index 00000000..05ff5448 --- /dev/null +++ b/docs/docs/05-quick-sharing.md @@ -0,0 +1,17 @@ +# Quick Sharing Extensions + +The whole point of Hoarder is making it easy to hoard the content. That's why there are a couple of + +## Mobile Apps + +mobile screenshot + + +- iOS app: TODO +- Android App: The app is built in using a cross-platform framework (react native). So technically, the android app should just work, but I didn't test it. If there's enough demand, I'll publish it to the google play store. + +## Chrome Extensions + +mobile screenshot + +- To quickly bookmark links, you can also use the chrome extension [here](https://chromewebstore.google.com/detail/hoarder/kgcjekpmcjjogibpjebkhaanilehneje). diff --git a/docs/docs/06-Development/01-setup.md b/docs/docs/06-Development/01-setup.md deleted file mode 100644 index 775a5806..00000000 --- a/docs/docs/06-Development/01-setup.md +++ /dev/null @@ -1,68 +0,0 @@ -# Setup - -## Manual Setup -### First Setup - -- You'll need to prepare the environment variables for the dev env. -- Easiest would be to set it up once in the root of the repo and then symlink it in each app directory. -- Start by copying the template by `cp .env.sample .env`. -- The most important env variables to set are: - - `DATA_DIR`: Where the database and assets will be stored. This is the only required env variable. You can use an absolute path so that all apps point to the same dir. - - `REDIS_HOST` and `REDIS_PORT` default to `localhost` and `6379` change them if redis is running on a different address. - - `MEILI_ADDR`: If not set, search will be disabled. You can set it to `http://127.0.0.1:7700` if you run meilisearch using the command below. - - `OPENAI_API_KEY`: If you want to enable auto tag inference in the dev env. - -### Dependencies - -#### Redis - -Redis is used as the background job queue. The easiest way to get it running is with docker `docker run -p 6379:6379 redis:alpine`. - -#### Meilisearch - -Meilisearch is the provider for the full text search. You can get it running with `docker run -p 7700:7700 getmeili/meilisearch:v1.6`. - -Mount persistent volume if you want to keep index data across restarts. You can trigger a re-index for the entire items collection in the admin panel in the web app. - -#### Chrome - -The worker app will automatically start headless chrome on startup for crawling pages. You don't need to do anything there. - -### Web App - -- Run `pnpm web` in the root of the repo. -- Go to `http://localhost:3000`. - -> NOTE: The web app kinda works without any dependencies. However, search won't work unless meilisearch is running. Also, new items added won't get crawled/indexed unless redis is running. - -### Workers - -- Run `pnpm workers` in the root of the repo. - -> NOTE: The workers package requires having redis working as it's the queue provider. - -### iOS Mobile App - -- `cd apps/mobile` -- `pnpm exec expo prebuild --no-install` to build the app. -- Start the ios simulator. -- `pnpm exec expo run:ios` -- The app will be installed and started in the simulator. - -Changing the code will hot reload the app. However, installing new packages requires restarting the expo server. - -### Browser Extension - -- `cd apps/browser-extension` -- `pnpm dev` -- This will generate a `dist` package -- Go to extension settings in chrome and enable developer mode. -- Press `Load unpacked` and point it to the `dist` directory. -- The plugin will pop up in the plugin list. - -In dev mode, opening and closing the plugin menu should reload the code. - - -## Docker Dev Env - -If the manual setup is too much hassle for you. You can use a docker based dev environment by running `docker compose -f docker/docker-compose.dev.yml up` in the root of the repo. This setup wasn't super reliable for me though. diff --git a/docs/docs/06-Development/02-directories.md b/docs/docs/06-Development/02-directories.md deleted file mode 100644 index 54552402..00000000 --- a/docs/docs/06-Development/02-directories.md +++ /dev/null @@ -1,28 +0,0 @@ -# Directory Structure - -## Apps - -| Directory | Description | -| ------------------------ | ------------------------------------------------------ | -| `apps/web` | The main web app | -| `apps/workers` | The background workers logic | -| `apps/mobile` | The react native based mobile app | -| `apps/browser-extension` | The browser extension | -| `apps/landing` | The landing page of [hoarder.app](https://hoarder.app) | - -## Shared Packages - -| Directory | Description | -| ----------------- | ---------------------------------------------------------------------------- | -| `packages/db` | The database schema and migrations | -| `packages/trpc` | Where most of the business logic lies built as TRPC routes | -| `packages/shared` | Some shared code between the different apps (e.g. loggers, configs, assetdb) | - -## Toolings - -| Directory | Description | -| -------------------- | ----------------------- | -| `tooling/typescript` | The shared tsconfigs | -| `tooling/eslint` | ESlint configs | -| `tooling/prettier` | Prettier configs | -| `tooling/tailwind` | Shared tailwind configs | diff --git a/docs/docs/06-Development/03-database.md b/docs/docs/06-Development/03-database.md deleted file mode 100644 index 40e2d164..00000000 --- a/docs/docs/06-Development/03-database.md +++ /dev/null @@ -1,11 +0,0 @@ -# Database Migrations - -- The database schema lives in `packages/db/schema.ts`. -- Changing the schema, requires a migration. -- You can generate the migration by running `pnpm drizzle-kit generate:sqlite` in the `packages/db` dir. -- You can then apply the migration by running `pnpm run migrate`. - - -## Drizzle Studio - -You can start the drizzle studio by running `pnpm db:studio` in the root of the repo. diff --git a/docs/docs/06-openai.md b/docs/docs/06-openai.md new file mode 100644 index 00000000..91e37c07 --- /dev/null +++ b/docs/docs/06-openai.md @@ -0,0 +1,11 @@ +# OpenAI Costs + +This service uses OpenAI for automatic tagging. This means that you'll incur some costs if automatic tagging is enabled. There are two type of inferences that we do: + +## Text Tagging + +For text tagging, we use the `gpt-3.5-turbo-0125` model. This model is [extremely cheap](https://openai.com/pricing). Cost per inference varies depending on the content size per article. Though, roughly, You'll be able to generate tags for almost 1000+ bookmarks for less than $1. + +## Image Tagging + +For image uploads, we use the `gpt-4-vision-preview` model for extracting tags from the image. You can learn more about the costs of using this model [here](https://platform.openai.com/docs/guides/vision/calculating-costs). To lower the costs, we're using the low resolution mode (fixed number of tokens regardless of image size). The gpt-4 model, however, is much more expensive than the `gpt-3.5-turbo`. Currently, we're using around 350 token per image inference which ends up costing around $0.01 per inference. So around 10x more expensive than the text tagging. diff --git a/docs/docs/07-Development/01-setup.md b/docs/docs/07-Development/01-setup.md new file mode 100644 index 00000000..775a5806 --- /dev/null +++ b/docs/docs/07-Development/01-setup.md @@ -0,0 +1,68 @@ +# Setup + +## Manual Setup +### First Setup + +- You'll need to prepare the environment variables for the dev env. +- Easiest would be to set it up once in the root of the repo and then symlink it in each app directory. +- Start by copying the template by `cp .env.sample .env`. +- The most important env variables to set are: + - `DATA_DIR`: Where the database and assets will be stored. This is the only required env variable. You can use an absolute path so that all apps point to the same dir. + - `REDIS_HOST` and `REDIS_PORT` default to `localhost` and `6379` change them if redis is running on a different address. + - `MEILI_ADDR`: If not set, search will be disabled. You can set it to `http://127.0.0.1:7700` if you run meilisearch using the command below. + - `OPENAI_API_KEY`: If you want to enable auto tag inference in the dev env. + +### Dependencies + +#### Redis + +Redis is used as the background job queue. The easiest way to get it running is with docker `docker run -p 6379:6379 redis:alpine`. + +#### Meilisearch + +Meilisearch is the provider for the full text search. You can get it running with `docker run -p 7700:7700 getmeili/meilisearch:v1.6`. + +Mount persistent volume if you want to keep index data across restarts. You can trigger a re-index for the entire items collection in the admin panel in the web app. + +#### Chrome + +The worker app will automatically start headless chrome on startup for crawling pages. You don't need to do anything there. + +### Web App + +- Run `pnpm web` in the root of the repo. +- Go to `http://localhost:3000`. + +> NOTE: The web app kinda works without any dependencies. However, search won't work unless meilisearch is running. Also, new items added won't get crawled/indexed unless redis is running. + +### Workers + +- Run `pnpm workers` in the root of the repo. + +> NOTE: The workers package requires having redis working as it's the queue provider. + +### iOS Mobile App + +- `cd apps/mobile` +- `pnpm exec expo prebuild --no-install` to build the app. +- Start the ios simulator. +- `pnpm exec expo run:ios` +- The app will be installed and started in the simulator. + +Changing the code will hot reload the app. However, installing new packages requires restarting the expo server. + +### Browser Extension + +- `cd apps/browser-extension` +- `pnpm dev` +- This will generate a `dist` package +- Go to extension settings in chrome and enable developer mode. +- Press `Load unpacked` and point it to the `dist` directory. +- The plugin will pop up in the plugin list. + +In dev mode, opening and closing the plugin menu should reload the code. + + +## Docker Dev Env + +If the manual setup is too much hassle for you. You can use a docker based dev environment by running `docker compose -f docker/docker-compose.dev.yml up` in the root of the repo. This setup wasn't super reliable for me though. diff --git a/docs/docs/07-Development/02-directories.md b/docs/docs/07-Development/02-directories.md new file mode 100644 index 00000000..54552402 --- /dev/null +++ b/docs/docs/07-Development/02-directories.md @@ -0,0 +1,28 @@ +# Directory Structure + +## Apps + +| Directory | Description | +| ------------------------ | ------------------------------------------------------ | +| `apps/web` | The main web app | +| `apps/workers` | The background workers logic | +| `apps/mobile` | The react native based mobile app | +| `apps/browser-extension` | The browser extension | +| `apps/landing` | The landing page of [hoarder.app](https://hoarder.app) | + +## Shared Packages + +| Directory | Description | +| ----------------- | ---------------------------------------------------------------------------- | +| `packages/db` | The database schema and migrations | +| `packages/trpc` | Where most of the business logic lies built as TRPC routes | +| `packages/shared` | Some shared code between the different apps (e.g. loggers, configs, assetdb) | + +## Toolings + +| Directory | Description | +| -------------------- | ----------------------- | +| `tooling/typescript` | The shared tsconfigs | +| `tooling/eslint` | ESlint configs | +| `tooling/prettier` | Prettier configs | +| `tooling/tailwind` | Shared tailwind configs | diff --git a/docs/docs/07-Development/03-database.md b/docs/docs/07-Development/03-database.md new file mode 100644 index 00000000..40e2d164 --- /dev/null +++ b/docs/docs/07-Development/03-database.md @@ -0,0 +1,11 @@ +# Database Migrations + +- The database schema lives in `packages/db/schema.ts`. +- Changing the schema, requires a migration. +- You can generate the migration by running `pnpm drizzle-kit generate:sqlite` in the `packages/db` dir. +- You can then apply the migration by running `pnpm run migrate`. + + +## Drizzle Studio + +You can start the drizzle studio by running `pnpm db:studio` in the root of the repo. diff --git a/docs/docs/07-Development/04-architecture.md b/docs/docs/07-Development/04-architecture.md new file mode 100644 index 00000000..df69376a --- /dev/null +++ b/docs/docs/07-Development/04-architecture.md @@ -0,0 +1,10 @@ +# Architecture + +![Architecture Diagram](/img/architecture/arch.png) + +- Webapp: NextJS based using sqlite for data storage. +- Redis: Used with BullMQ for scheduling background jobs for the workers. +- Workers: Consume the jobs from redis and executes them, there are three job types: + 1. Crawling: Fetches the content of links using a headless chrome browser running in the workers container. + 2. OpenAI: Uses OpenAI APIs to infer the tags of the content. + 3. Indexing: Indexes the content in meilisearch for faster retrieval during search. diff --git a/docs/docs/07-security-considerations.md b/docs/docs/07-security-considerations.md deleted file mode 100644 index 7cab2e07..00000000 --- a/docs/docs/07-security-considerations.md +++ /dev/null @@ -1,14 +0,0 @@ -# Security Considerations - -If you're going to give app access to untrusted users, there's some security considerations that you'll need to be aware of given how the crawler works. The crawler is basically running a browser to fetch the content of the bookmarks. Any untrusted user can submit bookmarks to be crawled from your server and they'll be able to see the crawling result. This can be abused in multiple ways: - -1. Untrused users can submit crawl requests to websites that you don't want to be coming out of your IPs. -2. Crawling user controlled websites can expose your origin IP (and location) even if your service is hosted behind cloudflare for example. -3. The crawling requests will be coming out from your own network, which untrusted users can leverage to crawl internal non-internet exposed endpoints. - -To mitigate those risks, you can do one of the following: - -1. Limit access to trusted users -2. Let the browser traffic go through some VPN with restricted network policies. -3. Host the browser container outside of your network. -4. Use a hosted browser as a service (e.g. [browserless](https://browserless.io)). Note: I've never used them before. diff --git a/docs/docs/08-security-considerations.md b/docs/docs/08-security-considerations.md new file mode 100644 index 00000000..7cab2e07 --- /dev/null +++ b/docs/docs/08-security-considerations.md @@ -0,0 +1,14 @@ +# Security Considerations + +If you're going to give app access to untrusted users, there's some security considerations that you'll need to be aware of given how the crawler works. The crawler is basically running a browser to fetch the content of the bookmarks. Any untrusted user can submit bookmarks to be crawled from your server and they'll be able to see the crawling result. This can be abused in multiple ways: + +1. Untrused users can submit crawl requests to websites that you don't want to be coming out of your IPs. +2. Crawling user controlled websites can expose your origin IP (and location) even if your service is hosted behind cloudflare for example. +3. The crawling requests will be coming out from your own network, which untrusted users can leverage to crawl internal non-internet exposed endpoints. + +To mitigate those risks, you can do one of the following: + +1. Limit access to trusted users +2. Let the browser traffic go through some VPN with restricted network policies. +3. Host the browser container outside of your network. +4. Use a hosted browser as a service (e.g. [browserless](https://browserless.io)). Note: I've never used them before. diff --git a/docs/static/img/architecture/arch.png b/docs/static/img/architecture/arch.png new file mode 100644 index 00000000..7583970e Binary files /dev/null and b/docs/static/img/architecture/arch.png differ diff --git a/docs/static/img/screenshots/admin.png b/docs/static/img/screenshots/admin.png new file mode 100644 index 00000000..cbc0cd86 Binary files /dev/null and b/docs/static/img/screenshots/admin.png differ diff --git a/docs/static/img/screenshots/all-lists.png b/docs/static/img/screenshots/all-lists.png new file mode 100644 index 00000000..befecb4b Binary files /dev/null and b/docs/static/img/screenshots/all-lists.png differ diff --git a/docs/static/img/screenshots/all-tags.png b/docs/static/img/screenshots/all-tags.png new file mode 100644 index 00000000..669ba5a4 Binary files /dev/null and b/docs/static/img/screenshots/all-tags.png differ diff --git a/docs/static/img/screenshots/bookmark-preview.png b/docs/static/img/screenshots/bookmark-preview.png new file mode 100644 index 00000000..200c598a Binary files /dev/null and b/docs/static/img/screenshots/bookmark-preview.png differ diff --git a/docs/static/img/screenshots/homepage.png b/docs/static/img/screenshots/homepage.png new file mode 100644 index 00000000..d4403659 Binary files /dev/null and b/docs/static/img/screenshots/homepage.png differ diff --git a/docs/static/img/screenshots/settings.png b/docs/static/img/screenshots/settings.png new file mode 100644 index 00000000..ed59ae33 Binary files /dev/null and b/docs/static/img/screenshots/settings.png differ diff --git a/docs/static/img/screenshots/share-sheet.png b/docs/static/img/screenshots/share-sheet.png new file mode 100644 index 00000000..538b9540 Binary files /dev/null and b/docs/static/img/screenshots/share-sheet.png differ -- cgit v1.2.3-70-g09d2