FlexServ Documentation

FlexServ is an OpenAI-compatible inference stack with a Rust gateway in front of Python-serving backends.

This documentation set tracks the code that is in the repository today:

• flexserv/gateway: the public gateway
• flexserv/backend/hf_transformers_backend: the in-repo transformers backend
• flexserv/boot_loader.sh: the container entrypoint for transformers, vllm, and sglang
• deploy/src: the separate Tapis deployment helper API

Start Here

• Overview: architecture, supported components, and current scope
• Quick Start: fastest local and Docker workflows
• API Documentation: gateway and backend endpoints
• Configuration Reference: CLI flags, environment variables, and runtime effects
• Docker Guide: compose, images, and runtime environment variables
• Build & Deploy: local builds and Docker builds
• Landing UI: operator UI behavior, storage, and live patching
• HPC Deployment Notes: Tapis app/job files and HPC launch scripts

Current Runtime Surface

Gateway-owned endpoints:

• GET /
• GET /healthz/live
• GET /healthz/startup
• GET /healthz/ready
• GET /openapi.json
• GET /openapi.yaml
• GET /v1/models
• POST /v1/flexserv/models/fetch
• POST /v1/flexserv/models/copy
• POST /v1/flexserv/models/index/rebuild
• POST /v1/flexserv/models/unpack
• GET /v1/flexserv/owner

Proxied inference surface:

• POST /v1/chat/completions
• POST /v1/completions
• POST /v1/embeddings
• POST /v1/responses
• POST /v1/audio/transcriptions

Important Scope Notes

• The maintained local backend in this repo is the transformers backend.
• vllm and sglang are supported through boot_loader.sh and the Docker images, not through separate in-repo backend wrappers.
• The older flexserv CLI, YAML runtime config, and console/grpc interfaces described in legacy docs are no longer part of the implementation.
• Ollama spec files remain in spec/ as reference artifacts only; the running server exposes OpenAI-compatible routes.