Configuration Reference

This page documents the current runtime knobs exposed by the codebase.

Precedence

Configuration sources differ by component:

• gateway: CLI flags override environment variables, which override code defaults
• transformers backend: CLI flags are primary; many container-oriented behaviors come from the boot loader and environment
• boot loader: environment variables plus positional/backend-specific arguments
• Docker: run-docker.sh host env drives compose, and compose env drives the container

Gateway CLI And Environment

Source: flexserv/gateway/src/config.rs

CLI flag	Environment	Default	Effect
--port	GATEWAY_PORT	8000	Public gateway listen port.
--backend	BACKEND_ADDR	http://127.0.0.1:8001	Upstream backend base URL.
--backend-kind	BACKEND_KIND	transformers	Label used by the landing page and OpenAPI metadata.
--cert	TLS_CERT	unset	Enable TLS when set together with --key.
--key	TLS_KEY	unset	TLS private key path. Must be paired with --cert.
--timeout	TIMEOUT_SECS	300	Reqwest client timeout for proxied/backend requests.
--http-redirect-port	HTTP_REDIRECT_PORT	unset	Starts a second plain HTTP listener that redirects to HTTPS. Ignored when TLS is disabled.
--workers	WORKERS	unset	Overrides Actix worker count.
--trace-access-details	GATEWAY_TRACE_ACCESS_DETAILS	false	Enables verbose request tracing into the trace log file.
--trace-log-file	GATEWAY_TRACE_LOG_FILE	/tmp/flexserv-gateway-trace.log	Detailed trace log destination.
--log-file	GATEWAY_LOG_FILE	/tmp/flexserv-gateway.log	Access/general log destination.
--log-rotation	GATEWAY_LOG_ROTATION	daily	Rotation for the main gateway log. Accepted values: never, minutely, hourly, daily.
--trace-log-rotation	GATEWAY_TRACE_LOG_ROTATION	daily	Rotation for the trace log.
--startup-lock-file	GATEWAY_STARTUP_LOCK_FILE	/tmp/flexserv/start.lock	File whose presence makes /healthz/startup return STARTING.
--readiness-lock-file	GATEWAY_READINESS_LOCK_FILE	/tmp/flexserv/ready.lock	File whose presence makes /healthz/ready return NOT_READY.
--backend-health-path	GATEWAY_BACKEND_HEALTH_PATH	/v1/flexserv/health	Path appended to backend base URL for readiness probes.
--backend-probe-timeout-secs	GATEWAY_BACKEND_PROBE_TIMEOUT_SECS	2	Timeout for readiness probe requests.
--max-payload-bytes	GATEWAY_MAX_PAYLOAD_BYTES	67108864	Max incoming request payload size accepted by Actix.
--flexserv-token	FLEXSERV_TOKEN	unset	Enables auth on all non-public routes. Accepts either x-flexserv-secret or Bearer auth.
--admin-secret	FLEXSERV_ADMIN_SECRET	unset	Stable admin secret for x-flexserv-admin-secret. If unset, a fallback token is generated at startup.
--pub-model-repo	PUB_MODEL_REPO	/app/models/public	Public/shared model root.
--pri-model-repo	PRI_MODEL_REPO	/app/models/private	Private model root.
--fetch-state-dir	GATEWAY_MODEL_FETCH_STATE_DIR	/tmp/flexserv-model-fetch	Directory for persisted async fetch/copy/unpack state.
--model-rewrite-paths	GATEWAY_MODEL_REWRITE_PATHS	/load_model, inference, unload, and YOLO paths	Comma-separated list of HTTP request paths where gateway-side model rewriting is applied. YOLO websocket model rewriting is always handled by the gateway proxy.
--fetch-header-passthrough	GATEWAY_MODEL_FETCH_HEADER_PASSTHROUGH	authorization,x-hf-token,x-tapis-token	Headers forwarded into fetch workflows.
--fetch-parallelism	GATEWAY_MODEL_FETCH_PARALLELISM	4	Maximum concurrent fetch operations per request.
--hf-model-list-limit	GATEWAY_HF_MODEL_LIST_LIMIT	100	Gateway-side cap for Hugging Face model listing helpers.
--index-rebuild-interval-secs	GATEWAY_MODEL_INDEX_REBUILD_INTERVAL_SECS	60	Periodic background index rebuild interval.

Notes:

• /healthz/ready succeeds only when the readiness lock is absent and the backend health probe succeeds.
• The gateway starts a periodic model-index rebuild task using index_rebuild_interval_secs.
• If TLS is enabled and http_redirect_port is set, the redirect listener sends 301 redirects to the HTTPS port.
• FLEXSERV_ADMIN_SECRET is the exact runtime value of x-flexserv-admin-secret when explicitly set.
• If FLEXSERV_ADMIN_SECRET is unset, the gateway generates a fallback random UUID and derives the actual admin token as a base64url string.
• With explicit FLEXSERV_ADMIN_SECRET, the secret value is not printed to console logs.
• With the fallback path, the gateway logs both the fallback UUID and the actual base64url admin token.

Transformers Backend CLI

Source: flexserv/backend/hf_transformers_backend/backend_server.py

Argument	Default	Effect
--default-model	Qwen/Qwen3-0.6B	Default text-generation model loaded by the backend.
--log-level	INFO	Wrapper logging verbosity; also maps to internal transformers serve logging (TRACE maps to debug).
--host	0.0.0.0	Backend listen address.
--port	8001	Backend listen port.
--device	auto	Target runtime device. auto prefers CUDA, then MPS, then CPU.
--dtype	auto	Requested model dtype.
--default-embedding-model	sentence-transformers/all-mpnet-base-v2	Embedding encoder model used by /v1/embeddings.
--enable-cors	enabled	Passes CORS enablement through to the wrapped transformers server.
--continuous-batching	disabled	Enables transformers continuous batching when supported upstream.
--trust-remote-code	disabled	Allows loading models that require remote code execution.
--attn-implementation	unset	Attention backend hint such as sdpa or flash_attention_2.
--quantization	none	Quantization mode. Accepted values: none, bnb-4bit, bnb-8bit.
--model-timeout	86400	Backend inactivity timeout before model unload.
--default-seed	unset	Default generation seed for reproducible sampling.
--input-validation	disabled	Enables stricter transformers-side request validation.
--flexserv-token	unset	Accepted by the backend, but backend auth middleware is intentionally disabled when the gateway is the intended front door.
--default-max-tokens	1024	Fallback max tokens for completions/chat/responses when omitted.
--force-default-model	disabled	Forces requests to use --default-model rather than the user-supplied model.
--force-default-embedding-model	disabled	Accepted by the wrapper, but the current embeddings handler does not actively switch behavior based on this flag.
--block-on-unexpected-fields	disabled	Keeps upstream validation behavior instead of trimming unexpected request keys before validation.

Boot Loader Environment

Source: flexserv/boot_loader.sh

Variable	Default	Effect
FLEXSERV_BACKEND_TYPE	transformers	Chooses transformers, vllm, or sglang.
ENABLE_GATEWAY	false	Starts the Rust gateway in front of the backend when true.
GATEWAY_PORT	8000	Public port used by the gateway in gateway mode.
GATEWAY_BACKEND_PORT	8001	Internal backend port used in gateway mode. Any user --port is replaced with this value in gateway mode.
FLEXSERV_TOKEN	unset	Passed through to the gateway for auth enforcement.
PUB_MODEL_REPO	/app/models/public	Public model root.
PRI_MODEL_REPO	/app/models/private	Private model root.
HF_HOME	derived from model mount	Preferred Hugging Face cache root. Falls back to /tmp/flexserv-hf-cache if not writable.
FLEXSERV_VENV	autodetected	Preferred Python interpreter root.
BACKEND_HEALTH_ENDPOINT	derived by backend type	Overrides the readiness URL the boot loader probes.
BACKEND_READY_TIMEOUT_SECS	600	Backend readiness timeout.
BACKEND_SMOKE_TEST_ENABLED	true	Enables startup smoke tests.
BACKEND_SMOKE_TEST_STRICT	false	Fails startup when smoke tests fail.
BACKEND_SMOKE_TEST_TIMEOUT_SECS	20	Timeout per smoke-test request.
GATEWAY_STARTUP_LOCK_FILE	/tmp/flexserv/start.lock	Startup lock file path.
GATEWAY_READINESS_LOCK_FILE	/tmp/flexserv/ready.lock	Readiness lock file path.
GATEWAY_TLS_CERT / GATEWAY_TLS_KEY	unset	Enables HTTPS gateway mode when both are set.
GATEWAY_TRACE_ACCESS_DETAILS	false	Enables detailed gateway trace logging.
GATEWAY_TRACE_LOG_FILE	unset	Optional path for detailed trace logs.
GATEWAY_LANDING_TEMPLATE	/app/flexserv/gateway/landing.html	Primary landing page template path.
FLEXSERV_OWNER	resolved from runtime user	Returned by /v1/flexserv/owner and used by the landing UI permission logic.
USER / USERNAME	set to flexserv when missing	Prevents username lookup failures in container/HPC contexts.
TORCHINDUCTOR_CACHE_DIR	/tmp/torch_inductor_cache	Stabilizes PyTorch cache behavior in constrained environments.
NUMBA_DISABLE_JIT	1	Avoids fragile numba/librosa cache paths in some container runtimes.
NUMBA_CACHE_DIR	/tmp/numba_cache	Explicit numba cache root.

Notes:

• In direct backend mode, the boot loader starts only the backend and probes its health endpoint directly.
• In gateway mode, the boot loader creates startup/readiness lock files before starting the gateway and clears them as startup progresses.
• GATEWAY_LANDING_TEMPLATE works with the gateway loader in landing.rs; if the file is unreadable, the gateway falls back to built-in search paths.

Boot Loader Arguments

Source: flexserv/boot_loader.sh

Usage shape:

/app/boot_loader.sh [MODEL] [BACKEND_ARGS...]

Behavior:

Input	Effect
positional MODEL	Convenience model argument. The boot loader rewrites it to --default-model, --model, or --model-path depending on backend type.
backend-specific args	Passed through to the selected backend command.
--disable-smoke-test	Removes startup smoke tests even when BACKEND_SMOKE_TEST_ENABLED would normally allow them.

Backend mapping:

FLEXSERV_BACKEND_TYPE	Command built by boot loader
transformers	python backend_server.py ...
vllm	python -m vllm.entrypoints.openai.api_server ...
sglang	python -m sglang.launch_server ...

Important behavior:

• in gateway mode, any user-provided backend --port is removed and replaced with GATEWAY_BACKEND_PORT
• in direct mode, the backend receives the supplied port directly
• gateway token enforcement is configured on the gateway side, not by backend auth middleware

Docker And Compose Variables

Primary user-facing variables:

Variable	Consumed by	Effect
FLEXSERV_IMAGE	run-docker.sh, compose	Image tag to run.
FLEXSERV_BACKEND_TYPE	compose -> boot loader	Selects transformers, vllm, or sglang.
FLEXSERV_MODEL	compose command	Initial model argument passed to /app/boot_loader.sh.
FLEXSERV_PORT	compose, run-docker.sh	Public service port.
FLEXSERV_CONTAINER_NAME	compose	Container name.
FLEXSERV_ENABLE_GATEWAY	compose -> boot loader	Enables/disables gateway mode in the container.
GATEWAY_BACKEND_PORT	compose -> boot loader	Internal backend port behind the gateway.
FLEXSERV_DTYPE	compose command	Backend dtype argument.
FLEXSERV_ATTN_IMPL	compose command	Backend attention implementation argument.
FLEXSERV_MODEL_TIMEOUT	compose command	Backend model timeout argument.
FLEXSERV_PUBLIC_MODEL_DIR	run-docker.sh, compose volumes	Host path mounted at /app/models/public.
FLEXSERV_PRIVATE_MODEL_DIR	run-docker.sh, compose volumes	Host path mounted at /app/models/private.
FLEXSERV_TLS_DIR	run-docker.sh, compose volumes	Host path mounted at /app/certs.
FLEXSERV_TOKEN	compose env -> gateway	Gateway auth token.
FLEXSERV_ADMIN_SECRET	compose env -> gateway	Stable admin secret for x-flexserv-admin-secret. If unset, the gateway generates and logs a fallback token at startup.
HF_TOKEN	optional downstream use	Token available to model downloads and backend runtime when exported into the environment.
FLEXSERV_UID / FLEXSERV_GID	compose, run-docker.sh	Runtime user mapping. run-docker.sh defaults these to the invoking host UID/GID unless explicitly set.
FLEXSERV_OWNER	compose -> boot loader	Explicit owner label for /v1/flexserv/owner.

Host-side run-docker.sh only variables:

Variable	Default	Effect
HEALTH_TIMEOUT	180	Timeout for compose readiness polling.
FLEXSERV_FIX_PERMS	0	Attempts sudo ownership repair on the model directory.
FLEXSERV_FORCE_PULL	0	Forces an image pull even when a local image exists.
FLEXSERV_TLS_CN	localhost	CN for generated self-signed cert.
FLEXSERV_TLS_ORG	MyDevOrg	O value for generated cert.
FLEXSERV_TLS_COUNTRY	US	C value for generated cert.
FLEXSERV_TLS_IP2	129.114.35.21	Secondary IP SAN entry baked into generated certs.

Practical Examples

Run gateway with explicit logs and tracing:

./flexserv/gateway/target/debug/flexserv-gateway \
  --backend http://127.0.0.1:8001 \
  --backend-kind transformers \
  --port 8443 \
  --cert /tmp/cert.pem \
  --key /tmp/key.pem \
  --trace-access-details \
  --trace-log-file /tmp/flexserv-trace.log \
  --workers 4

Run backend with safer local defaults:

./venvs/flexserv/bin/python flexserv/backend/hf_transformers_backend/backend_server.py \
  --default-model hf-internal-testing/tiny-random-gpt2 \
  --host 127.0.0.1 \
  --port 8001 \
  --device cpu \
  --dtype float32 \
  --default-max-tokens 256

Run Docker with custom public/private model mounts and a gateway token:

FLEXSERV_PUBLIC_MODEL_DIR=/data/flexserv-models/public \
FLEXSERV_PRIVATE_MODEL_DIR=/data/flexserv-models/private \
FLEXSERV_TOKEN=my-secret \
./run-docker.sh cpu

Run Docker with a stable admin secret:

FLEXSERV_ADMIN_SECRET=my-stable-admin-secret \
./run-docker.sh cpu

Get the fallback admin token from logs when FLEXSERV_ADMIN_SECRET is unset:

docker logs <container> 2>&1 | awk -F': ' '/FlexServ admin secret token \(base64url\)/{print $2}' | tail -n1

Run Docker as a specific UID/GID and make the model directories match:

sudo chown -R $(id -u):$(id -g) /data/flexserv-models/public /data/flexserv-models/private
sudo chmod -R 755 /data/flexserv-models/public /data/flexserv-models/private

export FLEXSERV_UID=$(id -u)
export FLEXSERV_GID=$(id -g)
FLEXSERV_PUBLIC_MODEL_DIR=/data/flexserv-models/public
FLEXSERV_PRIVATE_MODEL_DIR=/data/flexserv-models/private
./run-docker.sh cuda