Skip to content

Transformers Backend

What It Is

The transformers backend is the maintained in-repo backend implementation under:

flexserv/backend/hf_transformers_backend

It wraps transformers serve, exposes compatible OpenAI endpoints, and adds FlexServ-specific utility routes.

Added Endpoints

In addition to upstream routes, the backend registers:

  • POST /v1/completions
  • POST /v1/embeddings
  • GET /v1/flexserv/models/loaded
  • POST /v1/flexserv/models/unload
  • GET /v1/flexserv/health
  • GET /v1/flexserv/info
  • GET /v1/flexserv/resources
  • GET /openapi.yaml

Compatibility Patches

The backend currently applies a few important patches around upstream behavior:

  • rebinds chat/responses/audio routes when FastAPI infers the wrong request shape
  • optionally trims unexpected request keys before validation
  • disables continuation cache reuse for multimodal requests with images
  • replaces fragile transcription audio loading with a soundfile/torchaudio-first path
  • retries once when the known transformers response-stream continuation indexing bug appears

Running It Directly

./venvs/flexserv/bin/python flexserv/backend/hf_transformers_backend/backend_server.py \
  --default-model hf-internal-testing/tiny-random-gpt2 \
  --default-embedding-model sentence-transformers/all-mpnet-base-v2 \
  --host 127.0.0.1 \
  --port 8001 \
  --device cpu \
  --dtype float32

Useful flags:

  • --default-model
  • --default-embedding-model
  • --device
  • --dtype
  • --model-timeout
  • --continuous-batching
  • --trust-remote-code
  • --attn-implementation
  • --quantization
  • --default-max-tokens
  • --block-on-unexpected-fields
  • --oom-monitor
  • --oom-ram-threshold
  • --oom-vram-threshold
  • --oom-check-interval

Health and Resource Endpoints

GET /v1/flexserv/health

Returns:

  • backend liveness
  • currently selected text model
  • embedding model status
  • device
  • dtype

GET /v1/flexserv/info

Returns high-level backend configuration information.

GET /v1/flexserv/resources

Returns a real-time hardware resource snapshot. The backend probes:

  • NVIDIA GPUs — NVML (pynvml) reports per-device used/free/total bytes, GPU core utilisation %, temperature (°C), and power draw (W).
  • Apple Silicon — torch.mps reports current and driver-allocated memory when NVML is unavailable. Utilisation, temperature, and power are omitted (not exposed by MPS).
  • CPU — psutil reports per-logical-core usage over a 0.2 s measurement window.
  • System RAM — psutil reports total, used, available, and used %.

Example response:

{
    "gpu": [
        {
            "gpu_id": 0,
            "used_percentage": 52.4,
            "used_gb": 21.0,
            "free_gb": 19.1,
            "total_gb": 40.0,
            "gpu_utilization_percentage": 87.0,
            "temperature_celsius": 68.5,
            "power_watts": 312.1
        }
    ],
    "cpu": [
        { "cpu_id": 0, "used_percentage": 14.2 },
        { "cpu_id": 1, "used_percentage": 9.8 }
    ],
    "memory": {
        "used_percentage": 61.0,
        "used_gb": 49.2,
        "free_gb": 13.8,
        "total_gb": 64.0
    }
}

All measurement logic lives in flexserv/backend/hf_transformers_backend/resource_metrics.py, which is the single source of truth shared by the endpoint handler and the OOM monitor. The public API:

Function Returns Purpose
get_ram_info() RamInfo System RAM snapshot via psutil
get_cpu_info(interval) List[CpuCoreInfo] Per-core CPU % over interval seconds
get_gpu_info() List[GpuMemoryInfo] NVML probe, falls back to MPS
get_max_vram_used_pct() float Max VRAM used % across all GPUs, 0.0 if none
get_system_resources(cpu_interval) SystemResourceSnapshot Full snapshot (RAM + CPU + GPU)

Gateway Integration

When deployed behind the gateway:

  • the gateway uses /v1/flexserv/health as the default backend readiness probe
  • backend auth middleware is intentionally not enabled
  • gateway auth is the expected enforcement layer

Memory Management

How upstream transformers serve tracks models

Serve stores every loaded model in a dict:

self.loaded_models: dict[str, TimedModel] = {}

The key is "model-id@revision" (or just "model-id" when no revision is pinned). TimedModel wraps the PreTrainedModel + its tokenizer/processor and owns a threading.Timer that fires after model_timeout seconds of inactivity.

Inactivity timeout (passive eviction)

When load_model_and_processor is called for a model that is already loaded, it resets the inactivity timer. When no request touches the model for --model-timeout seconds the timer fires:

timeout_reached() → delete_model()
  del self.model / del self.processor   # remove attribute references
  gc.collect()                          # Python GC cycle
  torch.cuda.empty_cache()              # release unallocated CUDA cache blocks
  self._timer.cancel()

The loaded_models dict entry is not removed — the slot remains with model = None and is_deleted() returns True. On the next request for the same model, the slot is detected as deleted and the model is reloaded from scratch.

Default --model-timeout in the FlexServ wrapper is 86 400 s (24 h). Set it to 0 to pin the model in memory indefinitely.

Active-request tracking

FlexServ registers an outermost FastAPI middleware (_track_active_requests) that increments a counter when an HTTP request enters the stack and decrements it when the response leaves. The counter is read by unload_models() and the OOM monitor to produce accurate warnings.

Manual eviction — POST /v1/flexserv/models/unload

// evict a specific model
{ "model_id": "meta-llama/Llama-2-7b-chat-hf" }

// evict all loaded models
{}           // or { "model_id": "*" }

Response:

{
    "unloaded": [
        {
            "model_id": "meta-llama/Llama-2-7b-chat-hf",
            "was_loaded": true,
            "had_active_requests": false
        }
    ],
    "warnings": [],
    "active_requests_at_eviction": 0,
    "oom_triggered": false
}

Safety guarantee: Python's reference counting protects any in-flight generation. When a request calls load_model_and_processor() it receives local references to the model and tokenizer objects. delete_model() removes the TimedModel attribute (making is_deleted() true) and calls gc.collect() + cuda.empty_cache(), but the underlying tensor memory is only freed after every Python reference drops. A request already mid-generation will complete normally before the model tensors are freed.

When had_active_requests: true appears in the response, a human-readable explanation is included in the warnings list. After eviction the KV-cache pointer (last_kv_cache) is cleared so the next request for the same model does not try to continue from a stale cache.

Behavior when unload is called during an in-progress load

Model loading is a blocking synchronous call inside from_pretrained(). TimedModel is only written to loaded_models after from_pretrained() returns. Therefore:

  • While the model is being loaded, its key is absent from loaded_models (or marked is_deleted from a prior eviction).
  • An unload_models() call issued at that moment sees was_loaded: false and is a silent no-op.
  • Loading completes normally and the model is stored in VRAM as if the unload never happened.
  • A second unload_models() call after loading finishes will successfully evict the model.

There is no locking around the load-then-store sequence inside the upstream library. The OOM monitor and the unload API are both subject to this race window.

Loaded inventory — GET /v1/flexserv/models/loaded

This route returns the set of models currently resident in backend runtime state.

Current response shape:

{
    "models": [
        { "model_id": "Qwen/Qwen3-0.6B", "kind": "serve" },
        { "model_id": "sentence-transformers/all-mpnet-base-v2", "kind": "embedding" }
    ]
}

Notes:

  • kind: "serve" corresponds to entries in upstream Serve.loaded_models
  • kind: "embedding" corresponds to the sentence-transformers encoder stored separately on the backend wrapper
  • the endpoint is intentionally binary inventory only; it does not attempt exact per-model RAM or VRAM attribution

OOM monitor (proactive eviction)

Enable with --oom-monitor:

python backend_server.py \
  --oom-monitor \
  --oom-ram-threshold 88 \
  --oom-vram-threshold 90 \
  --oom-check-interval 20

A daemon thread wakes every --oom-check-interval seconds and calls get_ram_info() and get_max_vram_used_pct() from resource_metrics:

  • RAM — psutil.virtual_memory().percent
  • VRAM — NVML (pynvml) on NVIDIA GPUs; torch.mps allocated/driver ratio on Apple Silicon; 0.0 when no GPU metrics are available (pure CPU host)

When either metric meets or exceeds its threshold, all loaded models are evicted and a WARNING is logged. If active requests are in flight at that moment, an additional warning names the active count so operators can correlate with request logs.

Flag Default Description
--oom-monitor off Enable the background OOM probe thread
--oom-ram-threshold 90.0 System RAM % that triggers eviction
--oom-vram-threshold 90.0 GPU VRAM % that triggers eviction
--oom-check-interval 30 Seconds between probe cycles

OOM-triggered evictions are not currently reflected in the oom_triggered field of the manual unload response; that field is reserved for callers that POST the unload endpoint in response to an OOM event they detect externally.

Process-exit and kill risk analysis

Quick-reference table

Shutdown type VRAM System RAM HuggingFace disk cache
Graceful (SIGTERM / Ctrl-C) delete_model() called for all loaded models; CUDA cache flushed; process exit completes deallocation Released Intact
Hard kill (SIGKILL / OS OOM killer) CUDA driver reclaims all device memory immediately on process exit — no leaked VRAM OS reclaims all address space Possible orphaned temp shard files in ~/.cache/huggingface/hub/; recovered automatically on next startup
Killed mid-load Same as hard kill; from_pretrained interrupted before loaded_models is written — no cleanup needed Same Incomplete shard downloads are healed on next from_pretrained call
Killed mid-generation Same as hard kill; tensors freed by CUDA driver Same Intact

Why there is no persistent VRAM or RAM leak

Both CUDA device memory and host RAM are OS-managed resources scoped to the process lifetime. When a process exits — regardless of how — the operating system and CUDA driver reclaim all associated memory immediately. CUDA does not maintain device allocations across process boundaries, so:

  • No zombie allocations remain on the GPU after process exit.
  • No RAM pages remain mapped to a dead process.

This is true whether the process exits cleanly via sys.exit(), crashes on an exception, or is killed with SIGKILL.

Graceful shutdown path

A graceful shutdown (SIGTERM, Ctrl-C) triggers FastAPI's lifespan cleanup and uvicorn's shutdown handler. FlexServ registers an OOM monitor stop event and the monitor thread joins before process exit. However, FlexServ does not explicitly call delete_model() at shutdown — the models are freed implicitly when Python tears down module globals and the process exits. The CUDA driver flushes all remaining allocations at that point.

The practical consequence is that torch.cuda.empty_cache() and gc.collect() are not guaranteed to run during graceful shutdown. This is harmless because the process is exiting anyway, but it means the GPU counter in NVML may stay elevated for a brief window between the last Python decref and the driver's final cleanup.

Killed mid-load risk

When the process is killed while from_pretrained() is in progress:

  1. VRAM — any CUDA allocations made so far by the model loader are recovered by the driver on process exit. No permanent GPU memory leak.
  2. RAM — same; the OS reclaims pages.
  3. HuggingFace shard files — from_pretrained downloads each shard to a temp path like .incomplete_download_<hash> and renames it atomically on completion. A kill interrupts the rename, leaving the temp file on disk. The Hub library detects this on next startup by checking the cache index and re-downloads the affected shard. The cache index itself (a JSON file) is never corrupted by an interrupted download.
  4. loaded_models dict — the entry is only written after from_pretrained returns, so the dict state is never partially updated. A subsequent startup begins with an empty loaded_models dict.

Recommendation: if disk space is a concern, periodically run huggingface-cli scan-cache to find and remove incomplete shard files left by prior interrupted downloads.

Killed mid-generation risk

When the process is killed while a generation loop is running (tokens streamed to the client):

  1. VRAM — freed by CUDA driver on exit.
  2. RAM — freed by OS.
  3. KV cache (last_kv_cache) — the in-memory past_key_values tensor is freed with the process. The last_kv_cache pointer in Serve is a Python object reference; it does not represent a file or external resource. There is no stale pointer after restart.
  4. Client-visible effect — the streaming response is interrupted; the client receives a truncated SSE stream. FlexServ does not persist generation state across restarts, so the client must re-submit the request.

Killed during OOM eviction

If a kill arrives while unload_models() is executing:

  • delete_model() calls del self.model before calling gc.collect() and cuda.empty_cache(). If the kill arrives between these steps, the model Python object may not have been collected yet, but the process exit will free everything at the OS level.
  • The loaded_models dict may be in a partially updated state in memory. This does not matter because the state is not persisted to disk.

Summary of actual risks

Risk category Severity Notes
Persistent VRAM leak None CUDA driver owns device memory; reclaimed on exit
Persistent RAM leak None OS reclaims all address space
Corrupted model weights None Model files are never modified by inference
Corrupted HF disk cache Low Temp shard files may remain; auto-healed on next start
Truncated client response Expected Mid-stream kill always truncates the SSE stream
Stale KV cache on restart None last_kv_cache is in-process memory only
OOM mis-count after restart None OOM monitor re-reads live psutil/NVML data each cycle

The HuggingFace Hub download path writes shards to temp files and renames them atomically on completion. An interrupted download leaves an orphaned temp file but does not corrupt the cache index, so the next startup recovers automatically.

Chat Template Variables (chat_template_kwargs)

Some models use their Jinja chat template to expose optional generation-time flags. The canonical example is Qwen3, which gates <think>...</think> blocks with an enable_thinking variable in its template.

FlexServ forwards arbitrary Jinja variables to apply_chat_template through the chat_template_kwargs request field.

How it works

transformers' apply_chat_template(**kwargs) merges extra keyword arguments into the Jinja template context (via template_kwargs = {**self.special_tokens_map, **kwargs}). The upstream Serve.generate_chat_completion does not forward unknown request fields, so FlexServ monkey-patches apply_chat_template at the PreTrainedTokenizerBase and ProcessorMixin level. A thread-local variable holds the kwargs for the duration of each request and is cleared in a finally block, making the injection safe under concurrent requests.

Usage

Generic dict form

{
    "model": "Qwen/Qwen3-8B",
    "messages": [
        { "role": "user", "content": "Solve step by step: 9.11 vs 9.9?" }
    ],
    "chat_template_kwargs": {
        "enable_thinking": true
    }
}

Any key–value pair in chat_template_kwargs is forwarded verbatim to the Jinja context. Unknown or model-unsupported keys are ignored by the template.

Top-level shorthand

{
    "model": "Qwen/Qwen3-8B",
    "messages": [{ "role": "user", "content": "hi" }],
    "enable_thinking": false
}

enable_thinking can be supplied as a top-level field for convenience. It is treated as an alias for chat_template_kwargs.enable_thinking.

Combining both

{
    "chat_template_kwargs": { "enable_thinking": false, "custom_key": "val" },
    "enable_thinking": true
}

Top-level keys win: when the same key appears both at the top level and inside chat_template_kwargs, the top-level value takes precedence. In the example above the effective value is enable_thinking: true.

Precedence rule (implementation detail)

template_kwargs = dict(req.pop("chat_template_kwargs", None) or {})
enable_thinking = req.pop("enable_thinking", None)
if enable_thinking is not None:
    # Top-level overrides whatever was in chat_template_kwargs
    template_kwargs["enable_thinking"] = bool(enable_thinking)

The dict is populated first; then each top-level shorthand unconditionally overwrites the matching key.

Validation bypass

The OpenAI schema validator (validate_chat_completion_request) strips unknown fields before they reach the generation layer. FlexServ pops chat_template_kwargs and enable_thinking from the request body before validation and re-injects them after, so the validator never sees them and they are preserved intact for _patched_generate_chat_completion.

Model support

Chat template kwargs only have effect when the model's Jinja template references the same key name. Sending enable_thinking: true to a model whose template does not define that variable is silently ignored — the template renders as if the key were absent.

Model family Supported template variable Effect
Qwen3 enable_thinking Wraps reasoning in <think>…</think> blocks when true
Other models any key Passed through; silently ignored if not in template