Monitoring

promptfoo is a tool for testing and evaluating LLM output quality.

With promptfoo, you can:

Systematically test prompts & models against predefined test cases

Evaluate quality and catch regressions by comparing LLM outputs side-by-side

Speed up evaluations with caching and concurrency

Score outputs automatically by defining test cases

Use as a

... See more

Dynamically route every prompt to the best LLM. Highest performance, lowest costs, incredibly easy to use.

There are over 250,000 LLMs today. Some are good at coding. Some are good at holding conversations. Some are up to 300x cheaper than others. You could hire an ML engineering team to test every single one — or you can switch to the best one fo

... See more

baserun.ai💪💪💪

Testing & Observability Platform for LLM Apps

From prompt playground to end-to-end tests, baserun helps you ship your LLM apps with confidence and speed.

Capture all of your AI product data

Get the full picture of your model's performance. Log inputs and outputs and seamlessly enrich them with metadata and user feedback.

02

/

05

Analyze model performance

Figure out how your model is really working, and where you can improve. Monitor for errors and discover underperforming cohorts and use cases.

03

/

05

Impr... See more

🌳 Galileo LLM Studio

Algorithm-powered LLMOps Platform

Find the best prompt, inspect data errors while fine-tuning, monitor LLM outputs in real-time. All in one powerful, collaborative platform.

Langfuse is an open source observability & analytics solution for LLM-based applications. It is mostly geared towards production usage but some users also use it for local development of their LLM applications.

Langfuse is focused on applications built on top of LLMs. Many new abstractions and common best practices evolved recently, e.g. agents,... See more