Question: I'm looking for a solution that enables my team to systematically test new AI prompts and measure output quality, do you know of any?

If you need a heavy-duty system to systematically try out new AI prompts and score output quality, Athina is a great option. This end-to-end platform for corporate GenAI teams spans the full stack for experimentation, measurement and optimization. Features include real-time monitoring, customizable alerting and multiple workspaces, so it's a good option for speeding up your development cycle.

Another option is HoneyHive, a more general-purpose AI evaluation, testing and observability system. It's designed as a collaborative environment for testing and evaluating applications, with automated CI testing, production pipeline monitoring and prompt management. Its automated evaluators and human feedback collection abilities can help you iterate on your AI models more quickly.

If you're a developer who likes a command-line interface, Promptfoo offers a simple YAML-based configuration system for evaluating the quality of large language model (LLM) output. It supports multiple LLM providers, lets you customize evaluation metrics and offers a web viewer for inspecting results. This open-source tool is good for optimizing model quality and monitoring for regressions.

Last, PROMPTMETHEUS is geared for writing, testing, optimizing and deploying one-shot prompts for more than 80 LLMs. It includes a prompt toolbox, performance testing and deployment options to custom endpoints. The service supports collaboration, cost estimation and insights, so it's a good all-in-one option for AI teams.