Salesforce Announces the World’s First LLM Benchmark for CRM

Salesforce just announced the world’s first LLM benchmark for CRM to help businesses evaluate the rapidly growing number of large language models (LLMs) for use in their customer relationship management (CRM) systems. Let’s find out with us!

The new benchmark is a comprehensive evaluation framework that measures the performance of LLMs against four key measures: accuracy, cost, speed, and trust and safety. It’s been specifically designed to evaluate common sales and service use cases, including prospecting, lead nurturing, as well as sales opportunity and service case summaries. The benchmark also includes a public leaderboard to help professionals decide which LLM is best for their CRM needs. Salesforce will continue to incorporate new use case scenarios into the benchmark and enhance its evaluation of LLMs, which will soon include fine-tuned LLMs.

Developed by Salesforce AI Research, the benchmark uniquely uses real-world CRM data, and also uniquely makes use of expert human evaluations by practitioners. This enables businesses to use the benchmark to make more strategic decisions about how to incorporate generative AI into their CRM systems, with specific attention to: 

1. Accuracy: This metric comprises four subcategories: factuality, completeness, conciseness, and instruction-following. The more accurate the predictions or recommendations, the more valuable the results are to teams across the organization. And the more valuable the results, the better the actions they can take to improve customer experience. If a model is accurate enough for a use case, it’s also important to consider the other metrics. Even if the model isn’t accurate enough, techniques like prompt engineering and fine-tuning can improve it.

2. Cost: The cost metric is categorized as high, medium, and low, based on percentiles. It’s the estimated operational cost that varies by CRM use case. Customers can evaluate the cost-effectiveness of different LLMs to ensure they align with their budget and resource allocation strategies.

3. Speed: This metric assesses the LLM’s responsiveness and efficiency in processing and delivering information. Faster response times enhance the user experience, reduce wait times for customers, and enable sales and service teams to address inquiries and issues promptly.

4. Trust and Safety: This metric measures the LLM’s capability to shield sensitive customer data, adhere to data privacy regulations, secure information, and refrain from bias and toxicity for CRM use cases. By assessing the reliability of LLMs for CRM, this benchmark gives organizations a sense of transparency regarding trust and safety.

Conclusion:

Organizations can use this benchmark to compare LLMs, identify the best solution, and make more informed decisions that will deliver customer success and propel their business forward.

With Salesforce’s Einstein 1 Platform, customers can choose from existing LLMs or bring their own models to meet their unique business needs. By selecting models for their CRM use cases using the benchmark, businesses can deploy more effective and efficient generative AI solutions.

If want to learn more about Salesforce or need more information, you can contact us at:

• • Email: support@keizu.vn
  • Phone: (+84) 2862903301