15+ Open-Source Tools to Monitor Your Large Language Models (LLMs)

We have curated a list of open-source tools that solve some of the most pressing problems with LLM monitoring and observability:

1. AllenNLP Interpret: A library for interpreting and visualizing predictions from Large Language Models (LLMs), suitable for model explanation and debugging across various models.

2. LangKit: An open-source toolkit for monitoring LLMs, offering tools for text quality assessment, hallucination detection, and analysis of sentiment and toxicity.

3. BERTViz: Designed for visualizing attention mechanisms in BERT-based LLMs and other NLP models such as BERT, GPT-2, and BART.

4. SHAP (SHapley Additive exPlanations): Applies a game-theoretic approach to explain the outputs of machine learning models, including those from the transformers library by HuggingFace.

5. AI Fairness 360: A toolkit for identifying, documenting, and mitigating bias and discrimination throughout the machine learning model lifecycle.

6. Prometheus: An open-source system for collecting and analyzing real-time metrics from LLMs.

7. Grafana: Works with monitoring systems like Prometheus and Elasticsearch to analyze and visualize metrics and logs from LLMs.

8. Evidently: A Python library aimed at evaluating, testing, and monitoring NLP and LLM-powered systems, supporting various data types including tabular data, text, and embeddings.

9. Deepchecks: Offers a solution for validating AI & ML models and data, enabling thorough testing from research to production.

10. Giskard: Automatically identifies and manages risks in ML models and LLMs, providing coverage for performance and security metrics.

11. whylogs: Logs any type of data, allowing users to create summaries of their datasets.

12. lunary: Focuses on observability and prompt management for LLMs, aiding in collaborative debugging and development of LLM applications.

13. Arize Phoenix: Offers insights for MLOps and LLMOps with a focus on monitoring models and LLM applications through a notebook-first approach.

14. Pezzo: A cloud-native platform for LLMOps, designed for monitoring AI operations, troubleshooting, reducing costs and latency, managing prompts, and delivering AI updates.

15. Langfuse: Aids teams in debugging, analyzing, and iterating on LLM applications through collaborative efforts.

16. Fiddler Auditor: Allows testing of LLMs and NLP models to identify and address potential weaknesses and prevent adversarial outcomes before production deployment.

17. OpenLLMetry: Built on OpenTelemetry, this extension provides comprehensive observability for LLM applications, compatible with existing observability solutions like Datadog and Honeycomb.

We post helpful lists and bite-sized explanations daily on our X (Twitter). Let’s connect!