Spellforge

Spellforge is an AI quality gatekeeper that simulates and tests Large Language Models (LLMs) and Custom GPTs in your existing release pipeline. The platform helps you account for the uncertainty of how real users will interact with your AI by simulating real-world user personas to test and optimize AI agent responses. This means your AI systems are more likely to be ready and reliable for real-world user interactions.

Some of the key features of Spellforge include:

• Synthetic User Personas: Test your LLM or GPT with synthetic users before real ones.
• Automatic Quality Evaluation: Evaluate the quality of conversations between synthetic users and your AI.
• Easy Integration: Connect to your app or REST API in less than five minutes.
• Support for Multiple LLM Providers: Supports multiple popular LLMs and allows for custom LLM integration.

The integration is simple, requiring just a few lines of code. For example, in Python:

from langchain import OpenAI
from langchain.chains import LLMChain
from spellforge_tracing import SpellforgeClient, PromptTemplate, SpellforgeTracer

tracing_client = SpellforgeClient()  # Initialize the tracing client
llm = OpenAI()
prompt = PromptTemplate(
    template="2+{a}=", 
    input_variables=['a'], 
    alias='first-prompt'  # Add alias for tracking
)
chain = LLMChain(llm=llm, prompt=prompt)
chain.run(a="2", callbacks=[
    SpellforgeTracer(prompt=chain.prompt)  # Add tracing layer
])

Spellforge is designed to cover the entire development and production lifecycle, providing critical services for startups and companies that rely on prompt-based requests. It can help optimize costs by intelligently controlling LLM usage, which can save significant resources over time.

The platform is designed to be flexible and support a wide range of use cases, including:

• Custom GPT: Clone, modify, and test custom GPT models, simulating real-world interactions and evaluating performance.
• LLM-based Applications: Supports a range of popular LLMs and custom LLMs, giving you the flexibility and customization you need.
• ML Models, RAGs, and Custom Cases: Supports a wide range of Machine Learning models and Retrieval-Augmented Generation systems.

Spellforge is working on pre-configured solutions for seamless integration with a variety of Continuous Integration (CI) systems. You can easily add Spellforge to your release pipeline to ensure high-quality AI interactions and gain insights from real user interactions.

To learn more or get early access, check out the Spellforge website.

Published on June 13, 2024