Open-Source LLM Monitoring Tools: Top 13 in 2026

Updated September 1, 2026.

TL;DR: The strongest open-source choices in 2026 are Langfuse and Opik for integrated observability, MLflow for teams that also operate conventional ML, LangWatch for agent simulations, AgentOps for session replay, and OpenLIT for AI-plus-infrastructure telemetry. OpenLLMetry and OpenTelemetry provide portable instrumentation; Evidently adds quality and drift evaluation; and Laminar combines OTel-native tracing, evaluations, datasets, and SQL access in one self-hostable platform.

Monitoring one model call is no longer enough. A production AI system may retrieve documents, call APIs, hand work to sub-agents, write to memory, retry failed steps, and use several models before it returns an answer. If a run fails, teams need to reconstruct that entire trajectory – not just inspect the final prompt and response.

This guide focuses on open-source tools that can trace, evaluate, and monitor LLM applications and AI agents. For the underlying metrics and deployment patterns, see our guide on how to monitor LLMs in production.

AI agent observability is the ability to reconstruct and evaluate an agent’s complete execution: model calls, tool calls, retrieval, memory, handoffs, retries, errors, latency, token use, cost, and final outcome. Unlike basic LLM logging, it connects these events into traces and adds evaluations so teams can determine not only where a workflow failed, but whether it completed the task correctly and safely.

A useful production view usually combines four layers:

OpenTelemetry is increasingly the common transport layer. Its GenAI conventions describe model and agent telemetry, while the Collector can batch, filter, redact, and route traces, metrics, and logs to different backends. The conventions are still evolving, and prompt or tool content can contain sensitive data, so content capture should be deliberate rather than automatic.

Important license note: Arize Phoenix remains a capable, actively maintained observability and evaluation platform, but its server is now licensed under Elastic License 2.0. ELv2 permits free internal self-hosting but restricts offering the software as a managed service. Because ELv2 is not on the OSI-approved license list, Phoenix is source-available rather than open source in the strict licensing sense, so it is not counted among the thirteen above.

This is not a ranking by GitHub stars. We included projects that were active as of August 27, 2026; provide a meaningful tracing, monitoring, evaluation, or telemetry function; publish relevant code under an OSI-approved license; and solve a real production problem. We also separated complete platforms from instrumentation and infrastructure, because those categories are complementary rather than interchangeable.

The table’s license column refers to the named open-source component. Cloud plans, enterprise modules, hosted retention, SSO, audit logs, and support can use different commercial terms. Teams should verify the exact repository and deployment path with legal and security reviewers before standardizing on a tool.

Langfuse is an integrated LLM engineering platform for traces, evaluations, prompt management, datasets, experiments, cost, latency, and user feedback. Its current Python and JavaScript SDKs are based on OpenTelemetry, which makes it easier to combine native Langfuse features with third-party instrumentation.

Best fit: teams that want one self-hostable product for LLM and agent observability instead of assembling separate tracing, prompt, and evaluation services. The core is MIT-licensed; enterprise-only modules such as advanced RBAC, audit logs, and server-side masking are licensed separately.

Opik, built by Comet, combines multi-step agent tracing, datasets, experiments, LLM-as-a-judge evaluation, production dashboards, prompt management, online evaluation rules, guardrails, and agent optimization. It records model, retrieval, tool, and delegation steps as a trace tree and can attach automated or human scores to those traces.

Best fit: teams that want a permissively licensed, self-hostable platform with a strong evaluation and optimization loop. Unlike several open-core alternatives, Opik’s full platform is published under Apache-2.0.

MLflow Tracing now provides first-class observability for LLM applications and agents. It captures nested model, tool, retriever, and application spans; renders inputs and outputs; tracks latency, tokens, cost, errors, assessments, and expectations; and connects production traces to evaluation datasets. It is compatible with OpenTelemetry and its GenAI conventions.

Best fit: organizations that already use MLflow for experiments, models, and deployment and want one open-source control plane for conventional ML plus GenAI. Its automatic integrations cover major model SDKs and agent frameworks, but the breadth of MLflow can feel heavy for a small agent-only project.

LangWatch connects production observability with pre-release testing. Its distinctive feature is agent simulation: a simulated user and judge can exercise a real multi-turn agent, including tools and state, and identify the decision where the run failed. The platform is OpenTelemetry/OTLP-native and also includes evaluations, prompts, annotations, and an AI gateway.

Best fit: teams for which a successful single response is not enough and multi-turn behavior must be regression-tested. The core is Apache-2.0; enterprise modules such as SCIM and audit logs use a commercial license.

AgentOps is purpose-built for agent debugging. It records sessions, LLM calls, tool use, errors, latency, and cost, then presents step-by-step execution graphs and session replay. Its integrations target agent frameworks such as CrewAI, AG2, Agno, and LangGraph.

Best fit: developers who need a fast path from an opaque agent run to a replayable execution history. It is less comprehensive than an evaluation-first platform, but the agent-specific workflow is straightforward. The application is MIT-licensed and includes a self-hosting path.

Helicone combines an AI gateway with LLM observability. Routing requests through the gateway provides request logs, sessions and agent traces, latency and cost analytics, prompt experiments, fallbacks, and model routing with minimal application changes.

Best fit: teams that want gateway-level visibility and cost control across several model providers. The trade-off is architectural: proxy-based capture is convenient, but internal tool, retrieval, and business-logic spans still require application instrumentation. Helicone is Apache-2.0 and supports self-hosting.

OpenLIT is an OpenTelemetry-native AI engineering platform that spans model calls, vector databases, agent frameworks, evaluations, guardrails, prompt management, and infrastructure. Its GPU collector exports host metrics, hardware telemetry, process attribution, and CUDA activity through OTLP, connecting application behavior with the compute underneath it.

Best fit: self-hosted or GPU-heavy AI systems where model quality, cost, errors, and hardware behavior need to be investigated together. OpenLIT is Apache-2.0.

OpenLLMetry is an Apache-2.0 instrumentation project built on OpenTelemetry. It automatically creates spans for model providers, vector databases, agent frameworks, and MCP, then exports standard OTLP data to a backend such as Grafana, Datadog, Honeycomb, Langfuse, MLflow, or Traceloop.

Best fit: teams that already operate an observability backend and need GenAI-aware instrumentation rather than another dashboard. OpenLLMetry can capture prompts, completions, and embeddings, so production deployments should disable or filter content that could contain secrets, personal data, or customer documents.

Evidently is an Apache-2.0 framework for evaluating, testing, and monitoring ML and LLM systems. It supports offline reports, test suites, and live dashboards with more than 100 built-in metrics covering text properties, LLM outputs, retrieval and ranking, data quality, drift, and predictive performance.

Best fit: teams that need measurable quality and drift checks across both generative and conventional ML. Evidently is an evaluation and monitoring layer, not a deep agent trace viewer, so it often complements Langfuse, Opik, MLflow, or an OpenTelemetry stack.

Weave is the Apache-2.0 GenAI toolkit from Weights & Biases. It can trace arbitrary Python functions, model calls, tools, and data transformations, then run repeatable evaluations over those operations. This makes the connection between an observed failure and a reproducible experiment particularly clear.

Best fit: teams already using Weights & Biases or teams prioritizing evaluation workflows over a standalone self-hosted observability platform. The standard quickstart uses a W&B account, so deployment requirements should be checked before selecting it for a fully isolated environment.

OpenTelemetry Collector is not an LLM dashboard. It is the vendor-neutral pipeline that receives, processes, and exports traces, metrics, and logs. In production it can handle retries, batching, sampling, encryption, redaction, and routing to more than one backend without coupling every application to a vendor SDK.

Best fit: organizations standardizing telemetry across AI and non-AI services. Pair it with OpenLLMetry, OpenInference, native framework instrumentation, or the OTel GenAI conventions, then send data to the observability backend that fits each team.

Grafana correlates metrics, logs, traces, and profiles. A common open-source deployment combines Grafana with Tempo for traces, Loki for logs, and Prometheus or Mimir-compatible metrics. It can display OTLP telemetry from LLM applications alongside service latency, queues, databases, GPUs, and downstream APIs.

Best fit: platform teams that already run cloud-native observability and want AI telemetry inside the same operational system. Grafana itself does not provide LLM-specific evaluation datasets or judges; those must come from instrumentation and evaluation tools. Grafana is AGPL-3.0, so teams distributing modifications or offering a network service should review the license obligations.

Laminar is an Apache-2.0 observability and evaluation platform for AI agents. It uses OpenTelemetry for tracing and combines traces with evaluations, datasets, dashboards, annotations, and direct SQL access to captured data. The platform is self-hostable, and its backend is written in Rust.

Best fit: teams that want an OTel-native platform but also need evaluation and data-analysis workflows in the same product. Laminar is a full observability backend rather than only an instrumentation library.

Record and replay means capturing an agent run so a team can inspect it later or reproduce the conditions that caused a failure. The phrase covers two different capabilities: trace replay, which plays back the recorded sequence of model, tool, retrieval, and state events; and execution replay, which reruns the workflow with reconstructed or stubbed dependencies.

Most observability platforms provide trace or session replay. That is enough to follow the path an agent took, compare runs, and turn a failure into an evaluation case. Deterministic execution replay is harder because model outputs, external APIs, databases, clocks, and environment state can all change between runs.

Two newer projects focus more directly on that debugging layer. AgentDebugX can diagnose, attribute, recover, and rerun failures across several agent frameworks. AgentReplay defines a language-neutral trace format and replay harness for turning failed runs into tests. Both are useful projects to watch, but they are earlier-stage debugging tools rather than replacements for a production observability backend.

A mature stack connects all four. A trace should let an engineer move from a failed task to the responsible agent step, the model or tool call inside that step, and the infrastructure event underneath it.

Tracing shows what happened; it does not prove that an output is fair, safe, correct, or useful. Bias monitoring requires defined groups, slices, expected behavior, representative datasets, and metrics tied to a real decision. Evidently can compare performance and quality across data slices, while AI Fairness 360 provides bias metrics and mitigation algorithms for structured ML workflows.

AIF360 is valuable, but it is not an LLM observability platform, which is why it no longer appears in the numbered list. For free-form agent behavior, teams usually need a combination of rule-based checks, human review, task-specific datasets, model-based judges that have themselves been validated, and outcome monitoring. No generic “bias score” replaces that design work.

Open-source components can support serious production systems, but running the software is only part of the job. A managed or enterprise layer may be justified when a team needs:

The practical choice is often hybrid: open instrumentation and portable telemetry, plus a managed backend or enterprise module where operating requirements demand it. Portability matters because trace data becomes expensive to move after every application and framework has adopted one proprietary format.

AI agent observability is the practice of tracing and evaluating an agent’s complete execution, including model calls, retrieval, tools, memory, sub-agent handoffs, retries, errors, latency, cost, and final outcome. It helps teams reconstruct why the agent took a particular path and whether that path produced a correct, safe, and useful result.

Langfuse and Opik are strong integrated choices; MLflow fits teams that also operate conventional ML; LangWatch emphasizes simulations and multi-turn testing; AgentOps specializes in session replay; Laminar combines OTel-native tracing with evaluations and SQL access; and OpenLIT connects AI traces with infrastructure and GPU telemetry. The best choice depends on licensing, self-hosting, evaluation needs, existing observability systems, and expected trace volume.

LLM monitoring focuses on individual model interactions: prompts, responses, latency, tokens, cost, and errors. AI agent observability follows the full workflow around those calls, including planning, retrieval, tools, memory, handoffs, retries, state changes, and task outcomes. Agent observability therefore uses nested, end-to-end traces rather than isolated request logs.

Monitor end-to-end traces, model and prompt versions, tool arguments and results, retrieval quality, memory operations, handoffs, retries, exceptions, latency, token usage, cost, task completion, safety and policy checks, and user feedback. Also connect agent traces to service, database, queue, and GPU telemetry so application and infrastructure failures can be distinguished.

OpenTelemetry is a vendor-neutral framework for generating, collecting, processing, and exporting traces, metrics, and logs. For AI agents, GenAI semantic conventions provide shared names for model, token, tool, conversation, and agent attributes. An OpenTelemetry Collector can then filter, redact, sample, and route that telemetry to one or more monitoring backends.