Design a Multi-Tenant LLM API Gateway — System Design Practice | AmanAI Lab

Sheet›Design a Multi-Tenant LLM API Gateway

AmazonMSFTGoogle

45:00

Hard

The Problem

Problem

Design an LLM API gateway that provides a unified API over multiple LLM providers (OpenAI, Anthropic, Google Gemini, self-hosted models) for 1,000 enterprise tenants.

Requirements:

•Each tenant has their own API keys, rate limits, and spending caps
•Route to optimal provider based on cost/latency/capability
•Complete audit log of every request per tenant (for compliance)
•99.99% availability (the gateway itself cannot be a single point of failure)

What you'll be assessed on

Multi-tenancy isolation, model routing strategy, quota enforcement, observability, and failover.

Scale & Constraints

▸1,000 enterprise tenants, each with unique rate limits
▸100M requests/day total across all tenants
▸P99 gateway overhead ≤ 50ms (added on top of model latency)
▸Tenant spending cap enforcement within 1-minute accuracy
▸99.99% gateway availability (≤ 52 minutes downtime/year)
▸Full audit log retained for 2 years (compliance)

Must Cover0/10

Hints (if stuck)

💡 Token bucket rate limiting in Redis: decrement on request, refill on schedule.

💡 Circuit breaker pattern: trip after N failures, half-open after timeout, close after success.

💡 Route cheaply by default (GPT-4o-mini, Haiku), escalate to expensive models only when needed.

💡 Async audit logging via Kafka → S3/BigQuery to avoid adding latency to the critical path.

💡 Per-tenant Redis namespace prevents cross-tenant data leakage.

0 words · auto-saved

Back to sheet

Mock interviewTake quizFlashcards

Problem

Design an LLM API gateway that provides a unified API over multiple LLM providers (OpenAI, Anthropic, Google Gemini, self-hosted models) for 1,000 enterprise tenants.

Requirements:

•Each tenant has their own API keys, rate limits, and spending caps

•Route to optimal provider based on cost/latency/capability

•Complete audit log of every request per tenant (for compliance)

•99.99% availability (the gateway itself cannot be a single point of failure)

What you'll be assessed on