blob: df69376ab0d9c20f369dd99ccc5ae89e637373f2 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
|
# Architecture
- Webapp: NextJS based using sqlite for data storage.
- Redis: Used with BullMQ for scheduling background jobs for the workers.
- Workers: Consume the jobs from redis and executes them, there are three job types:
1. Crawling: Fetches the content of links using a headless chrome browser running in the workers container.
2. OpenAI: Uses OpenAI APIs to infer the tags of the content.
3. Indexing: Indexes the content in meilisearch for faster retrieval during search.
|
