diff options
Diffstat (limited to '')
| -rw-r--r-- | docs/docs/03-configuration/02-different-ai-providers.md (renamed from docs/versioned_docs/version-v0.25.0/14-Guides/05-different-ai-providers.md) | 48 |
1 files changed, 48 insertions, 0 deletions
diff --git a/docs/versioned_docs/version-v0.25.0/14-Guides/05-different-ai-providers.md b/docs/docs/03-configuration/02-different-ai-providers.md index 33408c00..7d1a4589 100644 --- a/docs/versioned_docs/version-v0.25.0/14-Guides/05-different-ai-providers.md +++ b/docs/docs/03-configuration/02-different-ai-providers.md @@ -18,6 +18,28 @@ OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Ollama is a local LLM provider that you can use to run your own LLM server. You'll need to pass ollama's address to karakeep and you need to ensure that it's accessible from within the karakeep container (e.g. no localhost addresses). +Ollama provides two API endpoints: + +1. **OpenAI-compatible API (Recommended)** - Uses the `/v1` chat endpoint which handles message formatting automatically +2. **Native Ollama API** - Requires manual formatting for some models + +### Option 1: OpenAI-compatible API (Recommended) + +This approach uses Ollama's OpenAI-compatible endpoint and is more reliable with various models: + +``` +OPENAI_API_KEY=ollama +OPENAI_BASE_URL=http://ollama.mylab.com:11434/v1 + +# Make sure to pull the models in ollama first. Example models: +INFERENCE_TEXT_MODEL=gemma3 +INFERENCE_IMAGE_MODEL=llava +``` + +### Option 2: Native Ollama API + +Alternatively, you can use the native Ollama API: + ``` # MAKE SURE YOU DON'T HAVE OPENAI_API_KEY set, otherwise it takes precedence. @@ -31,6 +53,10 @@ INFERENCE_IMAGE_MODEL=llava # INFERENCE_OUTPUT_SCHEMA=plain ``` +:::tip +If you experience issues with certain models (especially OpenAI's gpt-oss models or other models requiring specific chat formats), try using the OpenAI-compatible API endpoint instead. +::: + ## Gemini Gemini has an OpenAI-compatible API. You need to get an api key from Google AI Studio. @@ -64,3 +90,25 @@ OPENAI_API_KEY: Your Perplexity API Key INFERENCE_TEXT_MODEL: sonar-pro INFERENCE_IMAGE_MODEL: sonar-pro ``` + +## Azure + +Azure has an OpenAI-compatible API. + +You can get your API key from the Overview page of the Azure AI Foundry Portal or via "Keys + Endpoints" on the resource in the Azure Portal. + +:::warning +The [model name is the deployment name](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/switching-endpoints#keyword-argument-for-model) you specified when deploying the model, which may differ from the base model name. +::: + +``` +# Deployed via Azure AI Foundry: +OPENAI_BASE_URL=https://{your-azure-ai-foundry-resource-name}.cognitiveservices.azure.com/openai/v1/ + +# Deployed via Azure OpenAI Service: +OPENAI_BASE_URL=https://{your-azure-openai-resource-name}.openai.azure.com/openai/v1/ + +OPENAI_API_KEY=YOUR_API_KEY +INFERENCE_TEXT_MODEL=YOUR_DEPLOYMENT_NAME +INFERENCE_IMAGE_MODEL=YOUR_DEPLOYMENT_NAME +``` |