SetuprollSetuproll
S
Aider logoAiderServe one open model as an internal API

Team Self-Hosted LLM Gateway

Setuproll editorial@setuproll
90.0Overall score

Hosts an open model behind an OpenAI-compatible endpoint with high throughput batching so every internal app works unchanged. For teams with an NVIDIA GPU box that want a private URL instead of a metered third party.

90.0Score
4Components

Install this build

Export
terminal
vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2

Components

Model

  • Llama 3.3 70B Instruct
  • Qwen3 32B for a single GPU

Stack

  • vLLM
  • OpenAI-compatible /v1 endpoint
  • Caddy reverse proxy

Hardware

  • 2x RTX 4090 or 2x A100 80GB
  • CUDA 12.x

How it works

  • Launch vLLM with tensor parallel across the GPUs
  • Point any OpenAI SDK at the base URL with a shared token
  • Caddy adds TLS and auth in front of the endpoint

Summary

Hosts an open model behind an OpenAI-compatible endpoint with high throughput batching so every internal app works unchanged. For teams with an NVIDIA GPU box that want a private URL instead of a metered third party.

90.0 score

0 Reviews

Your rating
Sign in to post

Loading discussion...

← All builds