Skip to content

Quick Start

Fastest Docker Path

cp .env.example .env
./run-docker.sh

This starts FlexServ with:

  • gateway enabled
  • TLS enabled with a generated self-signed cert
  • public URL https://localhost:8080

Verify:

curl -k https://localhost:8080/healthz/live
curl -k https://localhost:8080/healthz/ready
curl -k https://localhost:8080/openapi.json | jq '.info'

Enable auth:

FLEXSERV_TOKEN=my-secret ./run-docker.sh cpu

Fastest Local Dev Path

Create the backend environment:

python3 -m venv venvs/flexserv
source venvs/flexserv/bin/activate
pip install -r flexserv/backend/hf_transformers_backend/requirements-transformers.txt
pip install huggingface_hub soundfile
deactivate

Build the gateway:

PYO3_PYTHON="$(pwd)/venvs/flexserv/bin/python" \
  cargo build --manifest-path flexserv/gateway/Cargo.toml

Run the local stack:

./run-local.sh

That launches:

  • backend at http://127.0.0.1:8001
  • gateway at https://127.0.0.1:8443

Verify:

curl -k https://127.0.0.1:8443/healthz/ready
curl -k https://127.0.0.1:8443/v1/models

Manual Startup

Backend:

./venvs/flexserv/bin/python flexserv/backend/hf_transformers_backend/backend_server.py \
  --default-model hf-internal-testing/tiny-random-gpt2 \
  --host 127.0.0.1 \
  --port 8001 \
  --device cpu \
  --dtype float32

Gateway:

./flexserv/gateway/target/debug/flexserv-gateway \
  --backend http://127.0.0.1:8001 \
  --backend-kind transformers \
  --port 8000

First API Calls

Chat completion:

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hf-internal-testing/tiny-random-gpt2",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": false,
    "max_tokens": 32
  }'

Embeddings:

curl -X POST http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sentence-transformers/all-mpnet-base-v2",
    "input": "hello"
  }'

Complete model list:

curl "http://127.0.0.1:8000/v1/models?complete=true"