Nicolay Gerold

promptfoo is a tool for testing and evaluating LLM output quality.

With promptfoo, you can:

Systematically test prompts & models against predefined test cases

Evaluate quality and catch regressions by comparing LLM outputs side-by-side

Speed up evaluations with caching and concurrency

Score outputs automatically by defining test cases

Use as a

• Container Service : Most often Kubernetes to run LLM Serving solutions like Hugging Face Text Generation Inference or vLLM.


• Compute Resources : GPUs for running models, CPUs for management services


• Networking and DNS : Routing traffic to the appropriate

• Self-contained
  
  We ship a single dependency-free binary that runs on all Linux distributions


• Fast to deploy, safe to operate
  
  We are sysadmins, we know the value of operator-friendly software


• Deploy everywhere on every machine
  
  We do not have a dedicated backbone, and neither do you,
  
  so

Second, TA teaches us to never overlook the simplest solutions. There are a lot of complicated methods to automatically find augmentation policies, but the simplest method was so-far overlooked , even though it performs comparably or better

• “LLMs can write SQL, but they are often prone to making up tables, making up field”


• “LLMs have some context window which limits the amount of text they can operate over”


• “The SQL it writes may be incorrect for whatever reason, or it could be correct but just return an unexpected

• [2024/01] dstack 0.14.0: OpenAI-compatible endpoints preview (Release)


• [2023/12] dst