Arize Phoenix

Phoenix is Arize AI's open platform for seeing and testing what an LLM app or agent does. It records each run as a trace (model calls, retrieval, tool use, custom logic), lets you score outputs with evaluators, and lets you replay and compare prompt or model changes on the same inputs before shipping them.

It is a single containerized app (web UI, trace collector and a SQL database) that receives traces over OpenTelemetry's OTLP protocol. Instrumentation comes from Arize's OpenInference libraries, which auto-instrument providers such as OpenAI, Anthropic and Bedrock and frameworks such as LangChain, LlamaIndex, DSPy, Mastra and the Vercel AI SDK, with packages for Python, TypeScript and Java. You can start it locally with one command and open it on localhost.

Beyond tracing it includes LLM-as-a-judge and code evaluators, human annotation, versioned prompts with a playground and span replay, and datasets with experiments. Storage is SQLite by default and Postgres for production; Arize documents Docker, Kubernetes/Helm, AWS CloudFormation, Cloud Run, Railway and Render deployments, and every feature is available when self-hosted.

Each instance is a single tenant, so isolating teams means running more instances. You operate, scale and secure it yourself with community support; managed hosting, SLAs, monitors and SOC 2/HIPAA certifications belong to Arize's commercial AX platform instead.