What Evidently does
Evidently (evidentlyai.com) is an open-source framework for testing AI systems. The Evidently Python library runs evaluations with 100+ built-in metrics and produces Reports and Test Suites you can view in Python, export to JSON or HTML, or run inside a CI pipeline. For LLM products it checks things like adherence to guidelines and format, hallucinations and factuality, PII, retrieval quality and context relevance, sentiment, toxicity and tone, plus custom evals that combine rules, classifiers and LLM judges. For predictive ML models it tracks data drift, data quality and model performance. Newer releases add synthetic test data generation and automated prompt optimization for LLM judges.
You install it with pip install evidently (or evidently[llm] for LLM evals) or conda. The sister library Tracely, based on OpenTelemetry, captures inputs and outputs from a live LLM app. The open-source Evidently Platform is a self-hosted UI service (evidently ui) that stores evaluation snapshots, traces and datasets on a file system, a SQL database such as SQLite or Postgres, or S3-compatible storage, and shows them in a monitoring dashboard. Evidently's documentation now states that Evidently Cloud is no longer available as a SaaS product; the website's pricing page redirects to the homepage.
