Sage Inference: The Universal LLM Gateway Powering 16 Systems

When you run 16 different AI-powered systems, every one of them needs to talk to a language model. You could give each system its own API keys, let them figure out provider routing, error handling, and retry logic independently. Or you could build one gateway that does it all.

We built Sage Inference — a universal LLM gateway that every DevBlock product routes through. One endpoint, one credit system, seven provider models behind it, automatic failover when something goes down.

Why a Single Gateway?

Before Sage Inference, every DevBlock system had its own direct integration with AI providers. Console talked directly to DeepSeek. ConverseIQ had its own fallback chain. Bigglesworth managed its own provider keys. This meant every system duplicated error handling, rate limiting, and fallback logic. When a provider went down, each system handled it differently — some gracefully, some not.

Sage Inference centralizes all of that. One gateway understands the state of every provider. One credit system tracks usage across all products. One set of circuit breakers prevents cascading failures.

"One gateway, seven providers, 16 systems. Every AI call in the ecosystem routes through a single point of control."

How It Works

Sage Inference runs as a Cloudflare Worker with D1-backed circuit breakers. When a provider returns errors, the circuit trips open — subsequent requests automatically route to the next healthy provider. Provider health is checked every 30 seconds. When a provider recovers, the circuit closes and traffic resumes.

The system supports 7 provider models: DeepSeek V4 Flash and Pro, Groq Llama, OpenAI GPT, Anthropic Claude, and Workers AI. Each model has configurable rate limits, credit costs, and quality tiers. The gateway handles streaming, non-streaming, TTS, STT, image generation, and embeddings through separate endpoints.

Credit tracking happens at the inference layer. Every response deducts credits based on token count. Failed responses get refunded. The credit system prevents abuse with rate limiting per API key and per-model caps.

What We Learned

Building a reliable inference gateway taught us that provider reliability varies wildly. Some providers have 99.9% uptime but slow response times. Others are fast but have frequent outages. The circuit breaker pattern — track failures, trip open, probe periodically, close when healthy — is the right abstraction for this. It handles both transient blips and extended outages without manual intervention.

The biggest surprise was how much complexity the gateway absorbed from downstream systems. Teams building new features on Console, MesageAI, or Bigglesworth don't think about which AI provider they're using. They just call Sage Inference. That abstraction alone has been worth the investment.