Skip to main content
Version: Latest

Installation

This guide will help you install and run the vLLM Semantic Router. The router runs entirely on CPU and does not require GPU for inference.

System Requirements​

note

No GPU required - the router runs efficiently on CPU using optimized BERT models.

Requirements:

  • Python: 3.10 or higher
  • Container Runtime: Docker or Podman (required for running the router container)

Quick Start​

1. Install vLLM Semantic Router​

# Create a virtual environment (recommended)
python -m venv vsr
source vsr/bin/activate  # On Windows: vsr\Scripts\activate

# Install from PyPI
pip install vllm-sr

Verify installation:

vllm-sr --version

2. Start vllm-sr​

vllm-sr serve

If config.yaml does not exist yet, vllm-sr serve bootstraps a minimal workspace and starts the dashboard in setup mode.

The router will:

  • Automatically download required ML models (~1.5GB, one-time)
  • Start the dashboard on port 8700
  • Start Envoy proxy on port 8888 after activation
  • Start the semantic router service after activation
  • Enable metrics on port 9190

3. Open the Dashboard​

Open http://localhost:8700 in your browser.

For first-run setup:

  1. Configure one or more models.
  2. Choose a routing preset or keep the single-model baseline.
  3. Activate the generated config.

After activation, config.yaml is written to the current directory and the router exits setup mode.

4. Test the Router​

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MoM",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

5. Optional: open the dashboard from the CLI​

vllm-sr dashboard

Common Commands​

# View logs
vllm-sr logs router        # Router logs
vllm-sr logs envoy         # Envoy logs
vllm-sr logs router -f     # Follow logs

# Check status
vllm-sr status

# Stop the router
vllm-sr stop

Advanced Configuration​

YAML-first workflow​

If you prefer to edit YAML directly instead of using the dashboard setup flow:

# Generate a lean advanced sample in the current directory
vllm-sr init

# Validate it before serving
vllm-sr validate config.yaml

vllm-sr init is optional. It generates an advanced sample and .vllm-sr/router-defaults.yaml for YAML-first users. router-defaults.yaml contains advanced runtime defaults and is not required for first-run dashboard setup.

HuggingFace Settings​

Set environment variables before starting:

export HF_ENDPOINT=https://huggingface.co  # Or mirror: https://hf-mirror.com
export HF_TOKEN=your_token_here            # Only for gated models
export HF_HOME=/path/to/cache              # Custom cache directory

vllm-sr serve

Custom Options​

# Use custom config file
vllm-sr serve --config my-config.yaml

# Use custom Docker image
vllm-sr serve --image ghcr.io/vllm-project/semantic-router/vllm-sr:latest

# Control image pull policy
vllm-sr serve --image-pull-policy always

Kubernetes Deployment​

For production deployments on Kubernetes or OpenShift, use the Kubernetes Operator:

Quick Start with Operator​

# Clone repository
git clone https://github.com/vllm-project/semantic-router
cd semantic-router/deploy/operator

# Install CRDs and operator
make install
make deploy IMG=ghcr.io/vllm-project/semantic-router-operator:latest

# Deploy a semantic router instance
kubectl apply -f config/samples/vllm_v1alpha1_semanticrouter.yaml

Benefits:

  • ✅ Declarative configuration using Kubernetes CRDs
  • ✅ Automatic platform detection (OpenShift/Kubernetes)
  • ✅ Built-in high availability and scaling
  • ✅ Integrated monitoring and observability
  • ✅ Lifecycle management and upgrades

See the Kubernetes Operator Guide for complete documentation.

Other Kubernetes Deployment Options​

Docker Compose​

For local development and testing:

Next Steps​

Getting Help​