Operations guide · version 2.1
Run it in
production.
How to deploy, scale, monitor and troubleshoot OAUTH2Relay, and how to rotate its credentials. Every setting mentioned here is described in the documentation.
Deploy
Production checklist
| Setting | Recommendation |
|---|---|
tls | Set it, or run behind a TLS-terminating proxy that is the only way to reach the relay port. |
apiKey | Set it unless the network already limits who can reach the relay. Always set it in parameter mode, where the caller chooses where the token is sent. |
admin_port | Set it, on a private network, so metrics and health checks are separate from relayed traffic. |
log_level | info. debug logs request and response content and is for short investigations only. |
log_format | json if a log collector reads the logs. |
| Credentials | ${env:NAME} injected by your platform, or a secrets manager with secrets_refresh_interval if they rotate. Never in the file. |
login.grant_type | client_credentials or jwt. Avoid password. |
relay.allowed_hosts | parameter mode only: as narrow as possible, with allowed_networks empty unless needed. |
server.shutdown_delay | Behind a load balancer or in Kubernetes, a few seconds (e.g. 5) so traffic stops before the listener closes. |
Check every configuration change before deploying it with oauth2relay --check-config --config config.json. It exits with status 1 and names the problem if the file is invalid, and prints warnings for discouraged settings. It contacts nothing, so it is safe in CI.
The relay needs its configuration file, outbound HTTPS to the identity provider, the API and any secrets managers, and nothing else: no database, and no disk apart from an optional logfile.
systemd
Each Linux archive includes deploy/systemd/oauth2relay.service, which runs the relay as an unprivileged oauth2relay user with a restrictive sandbox (systemd-analyze security rates it 1.2, "OK"). To install it:
sudo install -m 0755 oauth2relay /usr/local/bin/oauth2relay
sudo useradd --system --no-create-home --shell /usr/sbin/nologin oauth2relay
sudo install -d -m 0750 -g oauth2relay /etc/oauth2relay
sudo install -m 0640 -g oauth2relay config.json /etc/oauth2relay/config.json
sudo install -m 0600 /dev/null /etc/oauth2relay/oauth2relay.env # then add VAR=value lines
sudo install -m 0644 deploy/systemd/oauth2relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now oauth2relay
journalctl -u oauth2relay -f
- Environment variables go in
/etc/oauth2relay/oauth2relay.env, mode0600, owned by root: credentials referenced as${env:NAME}, and secrets manager settings such asAWS_REGION,VAULT_ADDRorVAULT_SECRET_ID. The configuration file then holds no secrets. - The configuration is checked before each start, so a restart with an invalid configuration fails at once with the reason in the journal.
- Logs go to the journal. If you set
logfile, use/var/log/oauth2relay/, the only directory the service can write to. - The relay port must be 1024 or higher unless you uncomment the
CAP_NET_BIND_SERVICElines in the unit. - Keep the unit's
TimeoutStopSec(45 seconds) aboveshutdown_delay+shutdown_timeout.
To apply a configuration change, check it, then systemctl restart oauth2relay. In-flight requests are drained, but new connections are refused until the relay is ready again, so restart one instance at a time behind a load balancer.
Containers
Build an image from the Linux binary with the Dockerfile in the installation guide. It is distroless, has no shell, and runs as UID 65532.
docker build -t oauth2relay:2.1.0 .
docker run -d --name oauth2relay -p 8080:8080 -p 9090:9090 \
-v /etc/oauth2relay:/etc/oauth2relay:ro \
--env-file /etc/oauth2relay/oauth2relay.env \
--read-only --cap-drop ALL --security-opt no-new-privileges \
oauth2relay:2.1.0
- Set
"host": "0.0.0.0", otherwise the relay only listens inside the container. - The configuration file must be readable by UID 65532.
- The image has no
HEALTHCHECK, because it has no shell or HTTP client; use the orchestrator's HTTP checks on/healthzand/readyz.
Kubernetes
The essential parts of a Deployment, with "host": "0.0.0.0", "admin_port": 9090 and "server": {"shutdown_delay": 5} in the configuration:
spec:
template:
spec:
terminationGracePeriodSeconds: 30
securityContext:
runAsNonRoot: true
seccompProfile: {type: RuntimeDefault}
containers:
- name: oauth2relay
image: registry.example.com/oauth2relay:2.1.0
ports:
- {name: relay, containerPort: 8080}
- {name: admin, containerPort: 9090}
envFrom:
- secretRef: {name: oauth2relay-env} # API_CLIENT_SECRET=...
livenessProbe:
httpGet: {path: /healthz, port: admin}
readinessProbe:
httpGet: {path: /readyz, port: admin}
periodSeconds: 5
startupProbe:
httpGet: {path: /healthz, port: admin}
failureThreshold: 30
periodSeconds: 2
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
volumeMounts:
- {name: config, mountPath: /etc/oauth2relay, readOnly: true}
volumes:
- name: config
configMap:
name: oauth2relay-config
With the credentials in a Secret exposed as environment variables, the configuration has no secrets and can live in a ConfigMap. For Vault, use the kubernetes auth method with a dedicated service account; for AWS and Azure, use workload identity (IRSA, EKS Pod Identity or Azure Workload Identity) rather than static keys.
Liveness passes as soon as the admin port is listening, so a slow first login does not get the pod restarted; readiness passes once the first token is obtained. TLS certificates mounted from a Secret, e.g. by cert-manager, are reloaded without a restart.
Shutdown
On SIGTERM or SIGINT the relay:
- reports
drainingon/readyzand keeps serving forserver.shutdown_delayseconds (default 0), so load balancers stop sending traffic; - stops accepting connections and waits up to
server.shutdown_timeoutseconds (default 15) for in-flight requests; - exits with status 0.
Run
Scaling
The relay is stateless apart from its current token, so it scales horizontally behind any load balancer without session affinity.
Each replica obtains and renews its own token: with background refresh, N replicas make about N token requests per token lifetime, whether or not they serve traffic. Check that this stays within the identity provider's rate limits and any limit on active tokens per client. Login retries are jittered so replicas do not retry in step after an outage.
Form and JSON bodies are read into memory to merge parameters, up to max_request_body_bytes (10 MiB) each; other bodies are streamed. There are no published benchmarks yet: size replicas by load testing with your own traffic, watching process_resident_memory_bytes and oauth2relay_request_duration_seconds.
Monitoring
Scrape /metrics with Prometheus (all metrics are in the documentation). Suggested alerts:
| Alert | Expression | Meaning |
|---|---|---|
| Token renewal failing | increase(oauth2relay_token_renewals_total{result="failure"}[10m]) > 0 | The identity provider is failing or rejecting the credentials. The current token is used until it expires. |
| Token expired | oauth2relay_token_expires_in_seconds < 0 | Requests are failing with 503 or 500. |
| Relay errors | sum(rate(oauth2relay_requests_total{code=~"5.."}[5m])) / sum(rate(oauth2relay_requests_total[5m])) > 0.05 | More than 5% of requests fail, from the relay or the API; the request log lines tell them apart. |
| Certificate expiring | oauth2relay_tls_certificate_expiry_timestamp_seconds - time() < 14 * 86400 | The relay's certificate expires within 14 days and has not been renewed. |
| Secrets stale | time() - oauth2relay_secrets_last_refresh_timestamp_seconds > 3 * <interval> | Secret re-fetches are failing; rotated credentials will not be picked up. |
| Relay down | up{job="oauth2relay"} == 0 | Prometheus cannot scrape the relay. |
Readiness does not fail when token renewal fails, so these alerts, not the load balancer, are how an identity provider problem is noticed. Every response carries an X-Request-ID header that matches the relay's log line and is sent to the API; ask callers for it when they report a failed request.
Startup failures
Startup failures are logged at ERROR with the cause in error, and the process exits with status 1.
| Log message | Cause and fix |
|---|---|
Failed to load config | The file is missing, unreadable or invalid. Run --check-config for the exact problem. |
Failed to prepare the configuration | An environment variable in ${env:...} is not set, or a secrets manager could not be reached, refused the credentials, or has no such secret or key. The error names the variable or the alias and secret. |
Failed to load ca_file | The ca_file is missing or has no PEM certificates. |
Failed to get initial valid token | The first login failed. not retrying because the error is permanent: the provider rejected the request (e.g. invalid_client), so check the client ID, secret, key and token URL. no retries remaining: it was unreachable or failing throughout. |
Failed to set up TLS | tls.cert_file or tls.key_file is missing or unreadable, or the certificate does not match the key. |
Server failed | Usually the port is in use, or below 1024 without the capability to bind it. |
Request failures
The relay's own errors are JSON with an error code, and the request log line with the same request_id has the cause in detail. Responses without that body come from the API.
| Status and error | Likely cause | What to do |
|---|---|---|
401 unauthorized | Missing or wrong X-API-Key. | Check the caller's key; inbound_auth_failures_total counts these. |
403 forbidden | Target host not in allowed_hosts. | Add the host if it should be allowed. |
403 destination_blocked | The target resolves to an internal address. | If it is meant to be internal, add it to allowed_networks; otherwise the protection is working. |
503 token_unavailable | The identity provider is unreachable, failing, or slower than token_wait_timeout. | Check the provider and the Login failed; retrying log lines. Requests succeed again once a token is obtained; no restart needed. |
500 token_error | The provider rejected the credentials, e.g. after they were revoked. | Fix the credentials (see Rotating credentials). |
502 upstream_error | DNS, connection or TLS failure to the API. | Check detail; for a private CA, set ca_file. |
502 redirect_blocked | The API redirected to a host that is not allowed. | Point the relay at the final URL, or check follow_redirects. |
504 upstream_timeout | The API took longer than relay.timeout. | Raise relay.timeout (and server.write_timeout) if it is legitimately slow. |
413 request_too_large | Body over max_request_body_bytes. | Raise the limit if needed. |
401/403 from the API | The token was issued but the API refuses it, usually a missing scope or audience. | Check the scope and audience requested and the API's logs. |
To investigate, restart one instance with --log-level debug. Debug logs include token and relayed requests and responses with secrets masked, but may contain personal data from request bodies; switch back to info afterwards. If a renewed certificate is not picked up, look for Could not reload the TLS certificate, usually a key that does not match.
Rotating credentials
Client secret or private key
With the credential in a secrets manager and secrets_refresh_interval set, no restart is needed:
- Create the new secret, or register the new public key, at the identity provider, keeping the old one valid.
- Store the new value in the secrets manager under the same name (and update
jwt_key_idif the provider selects keys bykid). - Wait one interval (check
oauth2relay_secrets_last_refresh_timestamp_secondson every replica) plus one token lifetime, so the next login uses the new credential. - Revoke the old credential.
With ${env:NAME} or a value in the file, update it and restart the replicas one at a time, then revoke the old credential.
Relay API key
The relay accepts one apiKey, so callers and relay switch together: make the new key available to callers, change it on the relay (at the next re-fetch if it comes from a secrets manager, otherwise by restarting), then switch the callers. Requests with the old key fail with 401 in between, so keep the window short.
Secrets manager credentials
- AWS: use an IAM role (instance profile, IRSA or EKS Pod Identity); the SDK renews it itself.
- Vault AppRole: update the SecretID variable and restart. With
kubernetesauth there is nothing to rotate. - Azure: use a managed or workload identity; with a service principal, update the secret's variable and restart.
TLS certificate
Replace the files at tls.cert_file and tls.key_file; the relay picks them up within 10 seconds of the next connection. Write the key first or rename both into place atomically, since the old certificate stays in use while the pair does not match.
Upgrading
Read the upgrade notes for the versions you are crossing, run --check-config with the new binary, then upgrade one replica at a time and check readiness and oauth2relay_token_renewals_total before continuing. The relay keeps no state apart from its token, so rolling back means starting the old binary, as long as the configuration uses no setting the older version lacks, such as ${env:...} before 2.1.