Skip to content

Development Guide

Repository Areas

  • flexserv/gateway: Rust gateway
  • flexserv/backend/hf_transformers_backend: transformers backend
  • flexserv/backend/utils: model helper scripts
  • deploy/src: deployment helper API
  • spec: protocol and architecture reference assets
  1. Create a backend Python environment in venvs/flexserv.
  2. Build the gateway with PYO3_PYTHON pointed at that interpreter.
  3. Use ./run-local.sh for end-to-end local iteration.
  4. Use ./test_local_https_e2e.sh for broader validation.

Key Runtime Scripts

  • run-local.sh: local backend + HTTPS gateway launcher
  • test-api.sh: starts the stack and keeps it running
  • test_local_https_e2e.sh: end-to-end checks against the local HTTPS stack
  • run-docker.sh: host-side Docker orchestration
  • flexserv/boot_loader.sh: in-container backend/gateway startup

Gateway Development Notes

The gateway owns more than reverse proxying. If you change gateway behavior, check:

  • auth bypass list in auth.rs
  • readiness behavior in health.rs
  • OpenAPI merging in openapi.rs
  • model fetch/copy/index/unpack logic in model_fetch.rs
  • model request rewriting in model_fetch.rs

Transformers Backend Notes

Transformers 5.3.0 introduced a breaking change that causes issue for transformers serve, where even /v1/chat/completions and /v1/audio/transcriptions are not working properly.

The backend intentionally patches upstream transformers serve behavior:

  • route rebinding for request-body compatibility
  • optional trimming of unexpected fields
  • cache continuation guard for multimodal turns
  • transcription decoding fallback that avoids fragile librosa-only paths
  • retry for known response-stream continuation bug

Validation

Common checks:

python scripts/sync_versions.py --check
./test_local_https_e2e.sh

For focused local work:

curl -k https://127.0.0.1:8443/healthz/ready
curl -k https://127.0.0.1:8443/openapi.json | jq '.paths | keys'

Documentation Rule

The repository previously contained many docs for a removed flexserv CLI and YAML config flow. When updating docs, treat the gateway binary, backend server, and boot loader as the source of truth.