Skip to content

Build & Deploy

Local Gateway Build

The Rust gateway links against Python through pyo3, so build it with a real backend Python interpreter.

python3 -m venv venvs/flexserv
source venvs/flexserv/bin/activate
pip install huggingface_hub
deactivate

PYO3_PYTHON="$(pwd)/venvs/flexserv/bin/python" \
  cargo build --manifest-path flexserv/gateway/Cargo.toml

Release build:

PYO3_PYTHON="$(pwd)/venvs/flexserv/bin/python" \
  cargo build --release --manifest-path flexserv/gateway/Cargo.toml

Local Backend Setup

The maintained in-repo backend is the transformers backend:

python3 -m venv venvs/flexserv
source venvs/flexserv/bin/activate
pip install -r flexserv/backend/hf_transformers_backend/requirements-transformers.txt
pip install huggingface_hub soundfile

Docker Image Builds

The supported build entrypoint is build-docker.sh.

Usage:

./build-docker.sh build [auto|multiarch|amd64|arm64] [base|transformers|vllm|sglang] [IMAGE_BASE]

Examples:

./build-docker.sh build auto transformers
./build-docker.sh build auto vllm
./build-docker.sh build auto sglang
./build-docker.sh build multiarch transformers myorg/flexserv

Image naming rules:

  • no IMAGE_BASE: use project.toml image base plus backend and version
  • base name without tag: append -<backend>:<version>
  • explicit name:tag: use as-is

Docker Run Path

Use run-docker.sh for normal local container runs:

FLEXSERV_FORCE_PULL=1 ./run-docker.sh
FLEXSERV_FORCE_PULL=1 ./run-docker.sh cpu
FLEXSERV_FORCE_PULL=1 ./run-docker.sh cuda

It handles:

  • compose file selection
  • TLS certificate generation
  • model directory writability checks
  • pull-or-use-local-image logic
  • readiness polling

FLEXSERV_FORCE_PULL=1 ensures a forcible docker pull if there is a new image update to the same version. This is very helpful if you are in developing process.

Version Metadata

project.toml is the source of truth for the project version.

Check sync:

python scripts/sync_versions.py --check

Rewrite synced targets:

python scripts/sync_versions.py --write