In the rapidly evolving landscape of artificial intelligence, the question of trust remains paramount. For enterprises deploying AI agents to interact with customers, the central challenge is not merely technical capability, but the integrity of the information provided. At Salesforce, the development of the Salesforce Help Agent has demonstrated that the secret to reliable, trustworthy AI starts long before the model is deployed—it begins with the quality, organization, and accessibility of the data behind it.
The architecture powering this reliability is known as "grounding." By connecting an AI agent directly to an organization’s proprietary knowledge base, companies can ensure that responses are not hallucinations, but rather verified insights derived from trusted, internal content. Salesforce leverages its Data 360 platform to curate and organize this product knowledge, which is then operationalized through Agentforce. When a customer poses a question, the agent employs Retrieval Augmented Generation (RAG)—a process that retrieves relevant documents and synthesizes an answer. However, this system follows a fundamental law of computing: the output is only as robust as the content retrieved.
To master this process, Salesforce has adopted a "Customer Zero" philosophy, acting as the primary tester of its own technology. Through this journey, the company has distilled several critical lessons on how to effectively prepare content for an AI-driven environment.
Assessing Content Before Indexing: The Fit, Gap, and Clash Framework
A common pitfall in AI implementation is the assumption that more data equates to better performance. In reality, an expansive, disorganized index can overwhelm an agent, leading to increased latency and decreased accuracy. Before any configuration work begins, content must undergo a rigorous assessment.
The approach centers on a "Fit, Gap, and Clash" analysis. Every article within the knowledge base is scrutinized against four core performance signals: an overall health score, the "Self-Help Failure Rate"—which tracks instances where a user reads an article but still feels compelled to open a support case—the "Problem Resolution Rate," and total session volume. By isolating high-traffic areas with low resolution rates, teams can pinpoint exactly where their knowledge base is failing to serve the user.
When these metrics are paired with top case drivers—the specific issues customers contact support about most frequently—the gaps in documentation become clear. This methodical approach allows for strategic pruning and refinement. For instance, during a recent integration, Salesforce opted to build a standalone agent for the Informatica help portal rather than merging that content into the primary Help Agent. The decision was a tactical success, resulting in an agent that resolves 92% of queries without ever requiring human intervention.
Success in this area requires a collaborative effort between two distinct functions. The content team focuses on the art of authoring, ensuring that tone, structure, and best practices are consistent. Simultaneously, the AI and Data team manages the technical readiness of the information, overseeing metadata, the "chunking" of articles for search indexing, and ongoing retrieval tuning. Bridging these two groups are "Agent Managers," typically former support engineers who analyze production conversation data to identify and patch knowledge gaps in real-time. This dual-team structure is essential because even the most elegantly written content can fail if it is improperly chunked or lacks the metadata required for the agent to route it effectively.
Building by Hypothesis Rather Than Specification
Traditional software development often relies on strict, upfront specifications. However, the nature of generative AI requires a more fluid, experimental approach. Because agent performance is influenced by a multitude of variables—including the embedding models used, the search index configuration, and the source content itself—the ultimate outcome is often unpredictable until the system is built and tested.
Consequently, development at Salesforce has shifted to a cycle of hypothesis-driven iteration. Rather than adhering to a static blueprint, the team develops a hypothesis, builds a specific "recipe" to test that assumption, and evaluates the results. A notable example of this process involved the use of metadata. The team hypothesized that explicitly tagging articles with product names would improve answer accuracy. After building the configuration and running the test, the metrics remained flat. A qualitative review confirmed the lack of improvement, and the team made the disciplined decision not to ship the update.
This experiment provided a valuable insight: metadata is only effective when the agent is explicitly instructed on how to utilize it. By pairing metadata with specific, refined prompt instructions, subsequent tests yielded significantly better results. This iterative loop ensures that resources are not wasted on features that do not demonstrably improve the user experience.
Defining "Good" and Testing with Rigor
To accurately evaluate the performance of an agent, organizations must first define what constitutes a "good" answer. This begins with the creation of a "golden data set"—a collection of approximately 200 representative user questions, each paired with a validated, expected answer and links to the source documentation. While AI tools can assist in drafting these, human validation remains non-negotiable. Subject-matter experts must review every entry to ensure that the expected answer is not only accurate but fully supported by the source text.
For qualitative evaluations, criteria are established around three pillars: correctness, completeness, and relevance. Previously, subject-matter experts spent hundreds of hours manually crafting these criteria for every utterance. Today, that process has been streamlined; experts now identify the best content source, while an AI prompt generates the initial criteria for human review.
This testing philosophy extends to the "four-tier" quantitative approach, where each tier addresses a different facet of performance. This includes measuring technical retrieval metrics, evaluating response accuracy, and assessing overall service quality. It is a vital distinction, as an answer can be factually perfect while remaining unhelpful if it is delivered in a poor format or provides an overwhelming list of links.
The efficacy of this testing is best illustrated by the team’s work on "section-aware" chunking. The team hypothesized that by splitting articles at natural section breaks rather than using fixed token counts, they could provide the agent with more coherent context. When tested against the golden data set, the section-aware method outperformed the old standard on both retrieval accuracy and human-reviewed helpfulness. It was this evidence that ultimately authorized the change for production.
Charting a Path for Your Own AI Agent
For organizations looking to deploy their own agents, the starting point is determined by their current stage of development. For those not yet live, the priority is to open the knowledge base and assess the top 100 articles using the Fit, Gap, and Clash framework. This assessment should take precedence over any model tuning.
For teams with agents already in production, the focus must shift to identifying the most frequent failure points. Whether it is missing documentation, an inappropriate tone, or the right documents producing the wrong answers, every failure point points to a specific lever in the Data 360 process.
The journey toward a high-performing AI agent is one of continuous, small-scale iteration. By prioritizing the quality of the content and maintaining a clear, measurable way to evaluate success, companies can build the kind of reliable AI infrastructure that customers trust. As the experience at Salesforce has shown, the most powerful AI is the one that has been grounded in the most accurate, well-organized, and carefully curated information available.
