Local models and custom endpoints

Point Haarness at any model server or gateway: a model running on your own computer, a company gateway, or your own Azure, Vertex or Bedrock deployments. You add a provider definition with an id of its own; the key, if one is needed, stays in your environment or credential store.

Ollama and LM Studio

Start the app before Haarness and there is nothing to set up. Haarness looks for Ollama on 127.0.0.1:11434 and LM Studio on 127.0.0.1:1234 when its server starts, and lists their models as ollama/… and lmstudio/….

haarness models ollama
haarness run -m ollama/qwen3-coder:30b "Explain src/main.rs"

Started Ollama after Haarness? Stop the Haarness server for the project with haarness serve --stop and open Haarness again.

To hide a detected runtime, set "enabled": false for ollama or lmstudio in the configuration file.

Add a local model server

For vLLM, llama.cpp's server, LocalAI or anything else that speaks the OpenAI Chat Completions API:

haarness provider add local-vllm \
  --url http://127.0.0.1:8000/v1 \
  --no-auth \
  --model my-org/my-model

You see Added custom provider 'local-vllm' to … with the file it wrote. The model is now in the picker as local-vllm/my-org/my-model:

haarness run -m local-vllm/my-org/my-model "Hello"

Leave out --model if the server lists its models at GET /models; Haarness then discovers them. With --model, only the models you list are offered.

Add a gateway that needs a key

Name the environment variable that holds the key. The key itself is never written to a file.

haarness provider add company \
  --url https://models.example.com/v1 \
  --api-key-env COMPANY_MODELS_KEY

Or store the key instead: haarness auth set company.

If the gateway wants a different header, send it with --header. Quote it so your shell does not expand the variable, and make ${VAR} the whole value. Haarness reads the variable when you add the provider and whenever its server starts, so it must be set both times:

haarness provider add company \
  --url https://models.example.com/v1 \
  --header 'X-Api-Key=${COMPANY_MODELS_KEY}'

Endpoints with another API format

Add --protocol when the endpoint does not speak OpenAI Chat Completions:

haarness provider add company-claude \
  --url https://models.example.com/anthropic/v1 \
  --protocol anthropic-messages \
  --api-key-env COMPANY_MODELS_KEY \
  --model company-sonnet

Use your own Azure deployments or Bedrock models

The built-in Azure, Vertex and Bedrock providers come with two example models each. To use your own deployment names or model ids, add a provider with that protocol and list them:

haarness provider add my-azure \
  --url https://my-resource.openai.azure.com/openai/v1 \
  --protocol azure-openai \
  --api-key-env AZURE_OPENAI_API_KEY \
  --model my-gpt-deployment

haarness provider add bedrock-eu \
  --url https://bedrock-runtime.eu-west-1.amazonaws.com \
  --protocol amazon-bedrock \
  --model eu.anthropic.claude-sonnet-4-5-20250929-v1:0

Without --api-key-env, store a key with haarness auth set my-azure. Bedrock also uses your AWS keys or profile; set AWS_REGION to the region in the URL.

Add it from the app

• Desktop: Settings → Providers → Add custom provider. Fill in Provider ID, Base URL and API format, choose Authentication (API key, Environment variable or No authentication), and under Models choose Discover from provider or Choose model IDs. The button appears while Desktop uses a Haarness server on this computer.
• Terminal UI: /providers, then Add a custom endpoint, or /connect, then Custom endpoint….

Available in (Desktop and the terminal UI) and --project (CLI) choose whether the provider is for every project or only the current one.

Add it in the configuration file

haarness provider add writes a providers entry to your configuration file, haarness.json on a new install. You can write it yourself, and add what Haarness cannot discover about a model:

{
  "providers": {
    "local-vllm": {
      "name": "Local vLLM",
      "base_url": "http://127.0.0.1:8000/v1",
      "requires_auth": false,
      "models": ["my-org/my-model"],
      "context_window": 131072,
      "max_output_tokens": 8192,
      "images": true,
      "tools": true,
      "reasoning": false
    }
  }
}

These settings apply to every model of that provider. The same keys work under [providers.ID] in an older config.toml. See Configuration for where the files live.

Why does a slow model stop answering?

Haarness gives up on an answer that sends nothing for five minutes, and retries it. A model on your own computer (localhost, 127.0.0.1, ::1) is waited for as long as it takes, since it may say nothing while it loads. Bedrock answers in one piece and has 15 minutes in all.

To change it, set stream_idle_timeout (or request_timeout for Bedrock) in seconds. An entry with only these keys tunes a built-in provider without redefining it:

{ "providers": { "anthropic": { "stream_idle_timeout": 600 } } }

Rules

• Give each endpoint its own id. haarness provider add, Desktop and the terminal UI refuse the id of a built-in provider, such as openai, anthropic or ollama: openai is a built-in provider. Give your endpoint its own ID, for example openai-proxy. So a key you stored for a built-in provider is never sent to an endpoint you add.
• A project cannot redirect a provider. In a project's haarness.json, an entry that gives a built-in provider, or one from your global configuration, another base_url is ignored, and haarness providers says so. Define the project's endpoint under its own id instead.
• HTTPS is required except for localhost, 127.0.0.1 and [::1], which may use plain HTTP. A URL with a user name or password in it is refused.
• Changes apply on their own. A running Haarness server picks up an added, changed or removed provider once no turn is running.
• Removing a provider keeps its stored key. haarness provider remove ID deletes the definition; haarness auth logout ID deletes the key.
• Always name the provider on the command line: -m local-vllm/my-org/my-model. With -m my-org/my-model, my-org would be read as the provider.

Troubleshooting

The provider does not appear. Haarness skips a definition it cannot use, for example an http:// URL that is not on loopback. Run haarness providers in the project; it prints a line such as Provider lan-box: base_url must use https, or http on an exact loopback host; skipped.

"does not advertise image input". Set "images": true for that provider if the model accepts images.

Related

• Sign-ins and API keys
• Haarness Router — many providers behind one endpoint