Skip to content

Docker Guide

Supported Images

The repository builds four Docker variants:

  • base
  • transformers
  • vllm
  • sglang

The main local runtime entrypoint is run-docker.sh.

Running with Compose

./run-docker.sh

Modes:

  • ./run-docker.sh or ./run-docker.sh auto
  • ./run-docker.sh cpu
  • ./run-docker.sh cuda

What the script does:

  • selects docker-compose.cpu.yml or docker-compose.cuda.yml
  • ensures the model directories are writable
  • generates a self-signed gateway certificate under .flexserv_tls/
  • pulls the selected image when needed
  • starts the container
  • waits for /healthz/ready

Default Runtime Shape

  • gateway enabled
  • TLS enabled
  • public port 8080
  • public model bind mount ./models/public:/app/models/public
  • private model bind mount ./models/private:/app/models/private
  • image default zhangwei217245/flexserv-transformers:1.4.0

Important Environment Variables

  • FLEXSERV_IMAGE
  • FLEXSERV_MODEL
  • FLEXSERV_PORT
  • FLEXSERV_PUBLIC_MODEL_DIR
  • FLEXSERV_PRIVATE_MODEL_DIR
  • FLEXSERV_UID
  • FLEXSERV_GID
  • FLEXSERV_TOKEN
  • FLEXSERV_ADMIN_SECRET
  • FLEXSERV_BACKEND_TYPE
  • FLEXSERV_ENABLE_GATEWAY
  • GATEWAY_BACKEND_PORT
  • FLEXSERV_DTYPE
  • FLEXSERV_ATTN_IMPL
  • FLEXSERV_FORCE_PULL
  • FLEXSERV_FIX_PERMS

Model Directory Permissions

Before running Docker Compose with bind mounts, create the host model directories yourself and make them writable by the container user.

If you run docker compose directly, the compose files default to UID/GID 1000:1000.

If you run through ./run-docker.sh, the script exports FLEXSERV_UID and FLEXSERV_GID from the invoking host user by default, so the container runs as your current host UID/GID unless you override them explicitly.

If you mount a host directory owned by root:root with mode 755, the container can read it but cannot create subdirectories inside it unless the container is also running as root. This commonly breaks model downloads when the gateway tries to create a staging directory.

Recommended setup when running as 1000:1000:

mkdir -p ./models/public ./models/private
sudo chown -R 1000:1000 ./models/public ./models/private
chmod 755 ./models/public ./models/private

If you want to run the container as a specific UID/GID, set the environment variables and make sure the mounted model directories are owned by that same UID/GID:

mkdir -p /data/flexserv/public /data/flexserv/private
sudo chown -R 903279:815499 /data/flexserv/public /data/flexserv/private
chmod 755 /data/flexserv/public /data/flexserv/private

FLEXSERV_UID=903279 \
FLEXSERV_GID=815499 \
FLEXSERV_PUBLIC_MODEL_DIR=/data/flexserv/public \
FLEXSERV_PRIVATE_MODEL_DIR=/data/flexserv/private \
./run-docker.sh cuda

If multiple users on the host need write access, use at least 775 with an appropriate shared group instead.

Admin Secret

FLEXSERV_ADMIN_SECRET controls the x-flexserv-admin-secret header used by the gateway's admin-only surfaces.

Current admin-secret protected or admin-augmented APIs:

  • POST /v1/flexserv/config
  • GET /v1/flexserv/config for admin-only fields

Behavior:

  • If FLEXSERV_ADMIN_SECRET is set, that exact value is used and it is not printed to the console.
  • If FLEXSERV_ADMIN_SECRET is unset, the gateway generates a fallback random UUID and logs the actual admin token as a base64url string.

Example with explicit secret:

FLEXSERV_ADMIN_SECRET=my-stable-admin-secret ./run-docker.sh cpu

To get the fallback admin token from logs:

docker logs <container> 2>&1 | awk -F': ' '/FlexServ admin secret token \(base64url\)/{print $2}' | tail -n1

When FLEXSERV_ADMIN_SECRET is explicitly set, there is no need to use a base64 tool because the configured value is already the final header value.

TLS

Compose runs use HTTPS when cert and key files are present, and run-docker.sh creates them automatically.

Default paths:

  • ${FLEXSERV_TLS_DIR:-./.flexserv_tls}/tls.crt
  • ${FLEXSERV_TLS_DIR:-./.flexserv_tls}/tls.key

Verify:

curl -k https://localhost:8080/healthz/live

Auth

To require auth on non-public endpoints:

FLEXSERV_TOKEN=my-secret ./run-docker.sh cpu

Then call protected routes with:

curl -k https://localhost:8080/v1/models \
  -H "Authorization: Bearer my-secret"

Building Images

Examples:

./build-docker.sh build auto base
./build-docker.sh build auto transformers
./build-docker.sh build auto vllm
./build-docker.sh build auto sglang
./build-docker.sh build multiarch transformers myorg/flexserv

Boot Loader Behavior

The container entrypoint is /app/boot_loader.sh.

It:

  • resolves a Python interpreter
  • prepares HF cache and model directories
  • launches the selected backend
  • optionally launches the gateway
  • waits for backend readiness
  • runs startup smoke tests

Backend mapping:

  • transformers -> in-repo backend_server.py
  • vllm -> python -m vllm.entrypoints.openai.api_server
  • sglang -> python -m sglang.launch_server