Skip to content

Overview

What FlexServ Is

FlexServ is a gateway-first inference service:

  • a Rust gateway handles public HTTP traffic, health checks, auth, OpenAPI, and model-management APIs
  • a backend serves inference requests
  • the gateway proxies inference traffic and enriches the model list with gateway-managed metadata

Core Components

Gateway

Location: flexserv/gateway

Responsibilities:

  • optional auth enforcement with x-flexserv-secret or Authorization: Bearer
  • landing page and documentation passthrough
  • healthz/live, healthz/startup, healthz/ready
  • openapi.json and openapi.yaml
  • GET /v1/models enrichment
  • model fetch, copy, index rebuild, unpack, and owner APIs
  • request-body model rewriting for supported inference routes

Transformers Backend

Location: flexserv/backend/hf_transformers_backend

Responsibilities:

  • wraps transformers serve
  • patches route compatibility issues across transformer releases
  • adds /v1/completions and /v1/embeddings
  • adds /v1/flexserv/health, /v1/flexserv/info, /v1/flexserv/resources
  • serves Swagger/ReDoc and OpenAPI JSON/YAML

Container Boot Loader

Location: flexserv/boot_loader.sh

Responsibilities:

  • starts transformers, vllm, or sglang
  • optionally starts the gateway in front of the backend
  • prepares HF cache and model directories
  • manages readiness locks
  • runs lightweight smoke tests after startup

Supported Backends

Fully Documented in-repo backend

  • transformers

Container-launched backends

  • vllm
  • sglang

These container-launched backends are started via Python modules from boot_loader.sh:

  • python -m vllm.entrypoints.openai.api_server
  • python -m sglang.launch_server

Model Storage

The gateway manages two model repositories:

  • PUB_MODEL_REPO: shared/public models
  • PRI_MODEL_REPO: private models

The gateway also understands models that are only visible from:

  • HF cache
  • backend-owned storage

GET /v1/models?complete=true exposes storage metadata such as:

  • dir_name
  • source
  • size_on_disk
  • total_size
  • pipeline_tag
  • labeled_task
  • is_shared
  • storage_tier
  • estimated_vram_gb_by_dtype

Auth Model

If FLEXSERV_TOKEN is not set, the gateway is open.

If FLEXSERV_TOKEN is set, every non-public endpoint requires either:

  • x-flexserv-secret: <token>
  • Authorization: Bearer <token>

Gateway admin configuration uses a separate header:

  • x-flexserv-admin-secret: <admin-secret>

Current admin-secret protected or admin-augmented APIs:

  • POST /v1/flexserv/config
  • GET /v1/flexserv/config for admin-only fields

Public paths:

  • /
  • /healthz/*
  • /docs
  • /redoc
  • /openapi.json
  • /openapi.yaml

Reference Material

  • spec/openai.yaml: primary protocol reference
  • spec/Architecture.md: current architecture summary
  • spec/Overall.md: current scope and compatibility notes