Proxy OpenAI-compatible models through Sentinel

If your architecture uses a self-hosted or alternative model provider (such as local LLMs run via Ollama, vLLM, GPUStack, or LM Studio), Sentinel can seamlessly proxy and protect these streams. The gateway supports any provider that exposes an OpenAI-compatible API format.

When a custom provider is configured, Sentinel listens for that specific provider's name in the incoming request path. It automatically assumes the upstream endpoint adheres to the standard OpenAI routing schema (intercepting /v1/chat/completions) and forwards the sanitized payload straight to your designated backend base_url.

1. Register the provider on the gateway

Pass the CUSTOM_PROVIDERS environment variable during container initialization, formatted as a JSON array of objects. Each object takes a provider_name, a base_url, and optionally target_routes — the path(s) to intercept, which default to v1/chat/completions (the OpenAI-compatible completion route).

docker run --platform linux/amd64 -p 8000:8000 \
  -e SENTINEL_ID=<sentinel-id> \
  -e SW_CLIENT_ID=<client-id> \
  -e SW_CLIENT_SECRET=<client-secret> \
  -e CUSTOM_PROVIDERS='[{"provider_name":"my-custom-provider","base_url":"https://gpustack.internal.example.com"}]' \
  us-central1-docker.pkg.dev/admina33d6818/sentinel-public/sentinel:latest

SW_CLIENT_ID and SW_CLIENT_SECRET are your SUPERWISE® API credentials — see how to generate credentials.

Via the Sentinel CLI

For local gateways, register the same providers with the --custom-providers flag instead of setting the environment variable yourself:

sentinel gateway start --custom-providers '[{"provider_name":"my-custom-provider","base_url":"https://gpustack.internal.example.com"}]'

The CLI saves them to ~/.config/sentinel/config.json and passes them to the container as CUSTOM_PROVIDERS on every start.

2. Point your application at the custom path

Because custom providers use the OpenAI-compatible formatting structure, you simply point the standard OPENAI_BASE_URL variable (or however the environment variable is defined for your app) to your custom provider's path exposed by the Sentinel gateway.

Using the "my-custom-provider" example above:

# Point OpenAI framework traffic to the custom proxy path on the gateway
export OPENAI_BASE_URL="http://localhost:8000/my-custom-provider/v1"

Example: running an application (Python)

When your application initializes a standard OpenAI client instance, it streams through Sentinel to your custom backend target without requiring any custom code overrides:

from openai import OpenAI

# The client automatically picks up OPENAI_BASE_URL from the environment
client = OpenAI(api_key="mock-key-if-required-by-your-backend")

response = client.chat.completions.create(
    model="your-custom-model-name",
    messages=[{"role": "user", "content": "Hello world!"}]
)

print(response.choices[0].message.content)

How it routes in practice

Pointing your client application to http://localhost:8000/my-custom-provider/v1 tells Sentinel to apply your active security policies locally before transparently proxying the data directly to https://gpustack.internal.example.com/v1/chat/completions.

Keep the /v1. The OpenAI SDKs add only /chat/completions to the base URL, so a base URL without /v1 asks the gateway for /my-custom-provider/chat/completions. That path is not in target_routes, so the gateway passes the request to https://gpustack.internal.example.com/chat/completions without running the guardrails. The same applies to any path you have not listed in target_routes.

Azure OpenAI

Azure OpenAI works as a custom provider. Set base_url to the resource, and target_routes to the path your client calls. The gateway forwards the api-key or Authorization header and the api-version query string unchanged.

The v1 endpoint, called with the standard OpenAI client:

CUSTOM_PROVIDERS='[{"provider_name":"azure","base_url":"https://<resource>.openai.azure.com","target_routes":["openai/v1/chat/completions"]}]'

export OPENAI_BASE_URL="http://localhost:8000/azure/openai/v1"

The deployment-based endpoint, called with the AzureOpenAI client. List one route per deployment:

CUSTOM_PROVIDERS='[{"provider_name":"azure","base_url":"https://<resource>.openai.azure.com","target_routes":["openai/deployments/<deployment>/chat/completions"]}]'

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="http://localhost:8000/azure",
    api_version="2024-10-21",
    api_key="<your Azure OpenAI key>",
)

Leave the trailing slash off base_url. The gateway adds one when it builds the upstream address.

Get Sentinel: Create a free SUPERWISE account.

Custom providers