Stop Guessing. Start Shipping Agents.
An open-source toolkit for building, testing, and monitoring AI agents in production.
Agent development lifecycle
Where the Toolkit fits in the agent lifecycle.
Arthur splits agent work into three phases. The Toolkit owns the middle one, and Agent Security & Governance owns the last.
Phase 1
Plan and build
Scoping the agent, choosing models and frameworks, and getting a first implementation working. Your stack, your choice.
Phase 2
You are hereThe Agent Development Flywheel
Iterative evaluation and improvement. Observe where the agent underperforms, feed those failures into your eval suite, experiment, and ship without regressions.
Agent Development Toolkit
Phase 3
Agent Security and Governance
Discovery, inventory, ownership, and policy evaluation at runtime across every agent in the enterprise.
Learn more
Ship Reliable AI Agents. Fast.
Manage
1/5Prompts that behave like code.
Most teams treat prompts like config files — unversioned, untracked, and painful to roll back. One bad change can quietly break production.
- Version and promote prompts across environments without redeploying your agent
- Roll back in seconds when performance drops — no firefighting, no guesswork
- Template prompts to control structure and variables at runtime, across teams or tenants
Experiment
2/5Test changes before they reach users.
Swapping a model or tweaking a prompt is a gamble without structured tests. Most teams ship first and find out what broke second.
- A/B test prompts, models, and RAG configs against real production data — not synthetic examples
- Test full agent workflows — tool use, reasoning paths, and output formatting, not just single completions
- Score results automatically or with human review — and see exactly what changed and why
Trace
3/5See exactly what your agent did.
When an agent fails, you shouldn't have to piece together logs and hope for the best.
- Inspect every step — inputs, tool calls, reasoning paths, and outputs across every run
- Filter by prompt version, user, outcome, or cost to find the source of a failure fast
- Built on OpenTelemetry — works with LangChain, LlamaIndex, OpenAI, Anthropic, and anything else in the OpenInference ecosystem
Monitor
4/5Know before your users do.
Quality problems in production are invisible until someone complains. By then it's too late.
- Run evals on live traffic — hallucination, PII, prompt injection, toxicity, and correctness, continuously
- Set alerts the moment quality drifts — not after a user escalation
- Validate before you ship with curated datasets and pre-deployment test runs
Integrate
5/5Works with what you already have.
You shouldn't have to rebuild your stack to get observability.
- Use any model — OpenAI, Anthropic, Cohere, or open-source
- Bring any framework — LangChain, LangGraph, LlamaIndex, Vercel AI SDK, and more
- Deploy your way — Docker, CloudFormation, or Helm. Your environment, your data.
Guardrails
Guardrails that run on every call.
Arthur scores every prompt and response against your policies and returns a pass or fail with the reasoning behind it. Your application acts on that verdict inline.
Sensitive data leakage
Protect user data and proprietary company data from being unintentionally exposed.
Hallucinations
Detect likely incorrect or unsubstantiated responses before they reach the end user.
Toxicity
Flag responses that are not value-aligned with your organization.
Prompt injection
Identify attempts to override an agent's intended behavior.
- Fits your architectureYour application calls Arthur to validate user prompts and model responses, then acts on the verdict inline.
- Works with any modelOpenAI, Anthropic, or anything else you run.
- Runs in your environmentPart of the open-source Arthur Engine, so evaluation happens where your data already is.
Continuous evaluation
Evaluation at every stage.
Evals are not a pre-launch gate. The same suite follows an agent from its first curated dataset through to live traffic.
- 01Pre-productionDefine KPIs, squash inconsistent behavior, and validate against curated datasets before anything ships.
- 02RuntimeEvaluate every inference against your acceptable use policies as it happens.
- 03Always-on productionContinuously score live traffic, alert when quality drifts, and adapt as user behavior changes.
| Agent Framework | Eval Platform | Arthur Engine + Toolkit | |
|---|---|---|---|
| Build and run agents | |||
| Prompt versioning & management | Basic | ||
| Structured A/B experiments | |||
| Real-time guardrail evaluation (hallucination, PII, injection) | |||
| End-to-end trace debugging | |||
| Traditional ML model eval | |||
| Self-hosted / open-source | Varies | SaaS | MIT licensed |
Perfect fit
How It Fits Into the Arthur Engine
The Agent Toolkit is part of the Arthur Engine — Arthur's free, open-source AI evaluation and monitoring platform. The Engine provides the foundation: real-time guardrails, LLM eval infrastructure, and flexible deployment. The Toolkit builds on top of that with the full agent development workflow. Pick a time and we'll walk you through how it fits your stack.
Works with every model and framework
Get Started Today
Ready to turn your AI into real-world impact?
We'll help you move from pilots and prototypes to production-grade applications, with evaluation every step of the way.