Deepchecks

Deepchecks is a tool to help you get your LLM (Large Language Model) apps to market faster without sacrificing testing. That's especially crucial for LLM apps, where generative AI can produce subjective results and manual evaluation can be slow and laborious.

Deepchecks gets around that problem by automating the evaluation process so you can spot and fix problems like hallucinations, wrong answers, bias and toxic content. The tool uses a "Golden Set" approach similar to a traditional test set for machine learning, but it's designed to be more comprehensive. It combines automated annotation with manual overrides so you can create a ground truth for your LLM app quickly.

Among other abilities, Deepchecks offers:

• Automated Evaluation: Estimated annotations you can override as needed to minimize the manual effort required for evaluation.
• LLM Monitoring: Ongoing monitoring of your model to ensure it's performing as desired.
• Debugging: Identification of weak spots in your LLM app and drill-down into specific steps for root-cause analysis.
• Version Comparison: Testing of different components and versions of your LLM app to find the best combination.
• Properties: Use of custom and off-the-shelf properties to thoroughly test every aspect of your LLM app.

Deepchecks pricing tiers include:

• Startup: $250 per month, with evaluation in production, properties and golden set management.
• Scale: Starts with a free trial, with support channels and custom auto annotation.
• Dedicated: Custom pricing, with dedicated support and private hosting.
• Open-Source: Free, but with model and data retention limits.

Deepchecks is geared for developers and teams building LLM apps who need a fast and reliable way to test and monitor their models. By automating evaluation and providing in-depth monitoring, Deepchecks ensures LLM-based applications are reliable and of high quality throughout their entire lifecycle.

Published on June 14, 2024