overview
What is Respan Gateway?
Respan Gateway is an LLM engineering platform developed by Respan that enables LLM developers, founders, engineers, and product teams to unify observability, evaluations, prompt optimization, and LLM gateway functions. It provides a single OpenAI-compatible endpoint for routing requests to over 500 LLMs. The platform acts as a unified control plane, integrating an AI gateway with comprehensive observability, automated evaluation pipelines, and prompt optimization capabilities. This architecture simplifies multi-model deployments, mitigates vendor lock-in, and allows for seamless switching between models without requiring code changes. Key functionalities include ensuring application uptime through ordered model fallback, load balancing across deployments and providers, and configurable auto-retries with backoff to address issues such as model errors or rate limits. Respan Gateway traces every LLM call end-to-end, capturing critical metrics like latency, errors, tokens, and cost, providing detailed visibility into agent behavior and performance in production environments. It also facilitates prompt management with version control, A/B testing, and automated evaluation pipelines that combine code-based rule checks, LLM judge graders, and human-in-the-loop review to measure output quality and detect regressions.