Skip to content

UI And Live Patching

What The UI Is

The root route GET / serves the FlexServ landing UI when the client looks like a browser. Non-browser clients receive a JSON link index instead.

Source files:

  • HTML template: flexserv/gateway/src/landing_assets/landing.html
  • CSS and JavaScript assets: flexserv/gateway/src/landing_assets/
  • Browser-vs-JSON dispatch and template lookup: flexserv/gateway/src/landing.rs

Where To Access It

  • local dev with run-local.sh: https://127.0.0.1:8443/
  • Docker with run-docker.sh: https://localhost:8080/

We currently have an instance available on our DGX Spark node hosted at TACC. To experience it, you can visit:

Also, you can go to This Tutorial

The UI is intended as an integrated smoke-test and operator console for the gateway and backend APIs.

Main UI Areas

The current landing page includes three primary groups:

Gateway And Platform APIs

  • gateway readiness check
  • OpenAPI summary
  • resource reporter view

Inference APIs

  • chat completions
  • responses
  • completions
  • embeddings
  • audio transcription

Model Management APIs

  • model pool browser
  • drag-and-drop model copy between public and private pools
  • private model sticker actions:
    • edit metadata
    • fix metadata from Hugging Face
    • refresh size metadata
    • unload from memory
    • delete
  • loaded-state star badges on private stickers
  • loaded private model size distribution summary
  • model fetch batch builder
  • fetch status polling
  • archive unpack trigger and status

Model Selector Behavior

The UI populates model selectors by calling:

/v1/models?complete=true

It classifies models into UI buckets by looking at:

  • labeled_task
  • pipeline_tag
  • selected storage metadata

The landing page currently restricts inference selectors to private models only.

This is why model catalog quality improves when the gateway index is up to date.

Stored Browser State

The landing page keeps local interactive state in the browser:

  • FlexServ token
  • Hugging Face token
  • conversation context for chat and responses
  • long prompts
  • reasoning/activity history
  • media attachments

Implementation details:

  • local token values are obfuscated before being placed into localStorage
  • structured UI state is stored in IndexedDB / OPFS-backed browser storage

This is browser convenience only, not a security boundary.

Conversation And Prompt Behavior

The chat and responses panes support:

  • multi-turn conversation state
  • long-prompt capture once the word threshold is crossed
  • markdown editing and preview
  • cURL generation for the current form state
  • optional context summarization once the history grows large

Current thresholds and defaults are hardcoded in landing.html.

Permission Rules In The UI

The model-pool UI includes client-side rules for promotion/demotion affordances:

  • ALLOW_POOL_DEMOTION

The page checks /v1/flexserv/config and enables private-to-public promotion or overwrite-related affordances according to the runtime permission response.

This is UI behavior, not the full security model. The gateway APIs still make the final decision.

Private Sticker Management

Private model stickers are now the primary management surface.

Behavior:

  • a solid star means the backend currently reports that model as loaded
  • an empty star means it is currently not loaded
  • right-click on desktop or use the overflow button on touch/mobile to open the same action menu
  • the metadata editor uses dropdown-only task and pipeline choices to match gateway-supported values
  • when a metadata sidecar exists, task remains derived from the metadata sidecar and is not overridden from the UI

Live Patching In Docker

run-docker.sh automatically includes docker-compose.override.yml when that file exists.

That makes it the primary development hook for patching the landing page or backend code without rebuilding the image.

Example override:

services:
    flexserv:
        volumes:
            - ./gateway_dir:/app/flexserv/gateway
            - ./flexserv/backend/hf_transformers_backend:/app/flexserv/backend

What those mounts do:

  • /app/flexserv/gateway: lets you replace landing.html or add landing.patch.html
  • /app/flexserv/backend: lets you replace backend_server.py and the backend helper modules

Important path note:

  • the runtime image keeps the backend at /app/flexserv/backend
  • for override mounts, mount the repo folder flexserv/backend/hf_transformers_backend directly onto /app/flexserv/backend

Landing Template Lookup Order

The gateway resolves the landing page in this order:

  1. GATEWAY_LANDING_TEMPLATE if set
  2. /app/flexserv/gateway/landing.patch.html
  3. /app/flexserv/gateway/landing.html
  4. local source-tree fallbacks
  5. the compiled-in template

That means there are two easy patching styles:

Full replacement

Set:

GATEWAY_LANDING_TEMPLATE=/app/flexserv/gateway/my-custom-landing.html

Conventional patch file

Place:

/app/flexserv/gateway/landing.patch.html

If present, it wins over the shipped landing.html.

Developer Notes

  • The page is large and intentionally self-contained. Most UI behavior lives in one file.
  • Root-route HTML vs JSON selection happens in landing.rs, not in the template.
  • Generated cURL snippets include gateway auth headers and, for fetch operations, optional HF token forwarding.
  • Any backend API surface change that affects request shapes, task labels, or model metadata should be validated against the landing page selectors and form builders.
  • If you patch backend Python modules in a container with bind mounts, restart the container or relaunch the backend process; there is no hot reload in the production image path.