LMSYS Org

LMSYS Org, short for Large Model Systems Organization, is a research organization focused on democratizing large model technology and the system infrastructure that supports it. The organization was founded by UC Berkeley students and faculty members and other collaborating institutions to make large models more accessible and scalable through open-source development.

LMSYS Org has several projects focused on large language models (LLMs) and their uses. Among them:

• Vicuna: An open-source chatbot available in 7B, 13B and 33B sizes with 90% of ChatGPT quality.
• Chatbot Arena: A game-based evaluation tool for LLMs using crowdsourcing and Elo ratings.
• SGLang: A lightweight interface and runtime for sophisticated LLM programs.
• FastChat: An open-source platform for training, serving and evaluating LLM-based chatbots.
• MT-Bench: A collection of challenging multi-turn and open-ended questions for evaluating chatbots.
• Arena Hard Auto: An automated pipeline to convert live data into high-quality benchmarks for evaluating chat assistants.

The organization also provides a number of large-scale datasets to evaluate and develop LLMs, including LMSYS-Chat-1M and Chatbot Arena Conversations.

LMSYS Org is a useful resource for anyone developing or researching large language models. By providing open models, datasets, systems and evaluation tools, it helps advance large model technology and make it useful in the real world. You can learn more at lmsys.org.

Published on July 2, 2024