Haystack Evaluators 完全指南用 9 大评估器量化 RAG 与 Agent 管线质量【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack本篇技术指南聚焦开源 AI 编排框架 Haystack 2.19 的haystack.components.evaluators评估器组件集系统讲解精确匹配、召回率、MRR、MAP、NDCG、SAS 语义相似度以及基于 LLM 的上下文相关性、忠实度与通用评估器的原理、参数、输出结构与实战用法。读完本文你将掌握在 RAG 问答、检索排序与生成式评估场景中如何为每个评估器构造输入、解读分数并接入 Pipeline 进行离线质量评测。一、评估器概览从零到一的量化评估体系Haystack 将评估器设计为标准的 Pipeline 组件统一通过component装饰器注册全部集中在 haystack/components/evaluators 目录下并在init.py 中对外导出。从源码结构看这 9 个评估器可分为两大类类别评估器依赖适用场景无监督确定性指标AnswerExactMatchEvaluator、DocumentRecallEvaluator、DocumentMRREvaluator、DocumentMAPEvaluator、DocumentNDCGEvaluator无需外部服务检索质量、答案字面匹配语义/生成式指标SASEvaluator、ContextRelevanceEvaluator、FaithfulnessEvaluator、LLMEvaluatorSentence-Transformers 模型 / LLM答案语义相似度、上下文相关性、答案忠实度、自定义标准其中ContextRelevanceEvaluator与FaithfulnessEvaluator都继承自LLMEvaluator见 context_relevance.py 与 faithfulness.py三者共享同一套指令 少样本示例 JSON 输出的提示词模板机制因此底层参数高度一致。所有评估器的通用约定ground_truth_*真实答案/文档与predicted_*/retrieved_*预测/检索结果列表必须等长输出字典通常同时包含score全体平均分与individual_scores逐条分数。二、确定性检索与答案指标2.1 AnswerExactMatchEvaluator答案精确匹配功能逐条判断预测答案是否与任一真实答案完全相等区分大小写的字符串精确比较。结果在 0.0 到 1.0 之间表示匹配比例。支持多条真实答案与多条预测答案输入。用法示例源自 answer_exact_match.pyfrom haystack.components.evaluators import AnswerExactMatchEvaluator evaluator AnswerExactMatchEvaluator() result evaluator.run( ground_truth_answers[Berlin, Paris], predicted_answers[Berlin, Lyon], ) print(result[individual_scores]) # [1, 0] print(result[score]) # 0.5源码级行为说明answer_exact_match.py输入长度不一致时抛出ValueError(The length of ground_truth_answers and predicted_answers must be the same.)使用zip(..., strictTrue)逐对比较命中记 1、未命中记 0score sum(matches) / len(predicted_answers)即精确匹配比例。局限性该指标不做任何归一化、同义词处理或语义判断两个意思相同但措辞不同的答案会被判为不匹配。它最适合答案高度标准化如命名实体、专有名词的场景。2.2 DocumentRecallEvaluator文档召回率支持两种模式功能计算检索文档对真实文档的召回率。每个问题可有多条真实文档和多条检索文档。构造函数__init__(modeRecallMode.SINGLE_HIT, document_comparison_fieldcontent)document_recall.py支持两个关键参数mode召回计算模式由枚举RecallMode定义document_recall.pyRecallMode.SINGLE_HITsingle_hit只要任一真实文档被检索到该问题即得 1 分否则 0 分——判断是否召回单条分数恒为 0 或 1RecallMode.MULTI_HITmulti_hit按命中真实文档的比例计分即命中去重真实文档数 / 真实文档总数传入字符串时由RecallMode.from_str()转换未知模式抛出ValueError。document_comparison_field文档比对字段可选content默认比较doc.content、id比较doc.id、或meta.key比较元数据支持嵌套如meta.source.url这是四个文档类评估器共享的参数。用法示例源自 document_recall.pyfrom haystack import Document from haystack.components.evaluators import DocumentRecallEvaluator evaluator DocumentRecallEvaluator() # 默认 SINGLE_HIT result evaluator.run( ground_truth_documents[ [Document(contentFrance)], [Document(content9th century), Document(content9th)], ], retrieved_documents[ [Document(contentFrance)], [Document(content9th century), Document(content10th century), Document(content9th)], ], ) print(result[individual_scores]) # [1.0, 1.0] print(result[score]) # 1.0第二问的真实文档是9th century与9th检索结果中二者均出现故multi_hit模式下得 1.0若真实文档含三条而只召回两条则multi_hit得 0.67、single_hit仍为 1.0。源码级行为说明document_recall.py比对基于去重后的比较值集合_unique_comparison_values会忽略空字符串与None若真实或检索一侧没有可比的比较值会记录logger.warning并将该条分数置 0.0。2.3 DocumentMRREvaluator均值倒数排名功能MRRMean Reciprocal Rank衡量首个相关文档在检索结果中的排名。对每个问题取第一个命中真实文档的位置的倒数1 / (rank 1)若完全未命中则为 0。用法示例源自 document_mrr.pyfrom haystack import Document from haystack.components.evaluators import DocumentMRREvaluator evaluator DocumentMRREvaluator() result evaluator.run( ground_truth_documents[ [Document(contentFrance)], [Document(content9th century), Document(content9th)], ], retrieved_documents[ [Document(contentFrance)], [Document(content9th century), Document(content10th century), Document(content9th)], ], ) print(result[individual_scores]) # [1.0, 1.0] print(result[score]) # 1.0第二问中9th century位于第 1 位rank 0倒数排名为1/(01)1.0若首条命中在位置 3rank 2则该项为1/3 ≈ 0.33。源码逻辑见 document_mrr.py命中后立即break只计算首个命中位置的倒数最终score为所有问题的平均。适用提示MRR 只关心第一个相关文档适合找到一条答案即可的问答检索若关心所有相关文档的排序质量应改用 MAP 或 NDCG。2.4 DocumentMAPEvaluator平均精度均值功能MAPMean Average Precision衡量全部相关文档在检索结果中的排名质量。对每个问题先算 Average PrecisionAP遍历检索结果每命中一条相关文档就累加当前命中数 / 当前位置最后除以相关文档总数score为所有问题 AP 的均值。用法示例源自 document_map.pyfrom haystack import Document from haystack.components.evaluators import DocumentMAPEvaluator evaluator DocumentMAPEvaluator() result evaluator.run( ground_truth_documents[ [Document(contentFrance)], [Document(content9th century), Document(content9th)], ], retrieved_documents[ [Document(contentFrance)], [Document(content9th century), Document(content10th century), Document(content9th)], ], ) print(result[individual_scores]) # [1.0, 0.8333333333333333] print(result[score]) # 0.9166666666666666第二问的手工验算检索结果第 1 位9th century命中 → 累加1/1第 3 位9th命中 → 累加2/3AP (1 2/3) / 2 ≈ 0.8333。实现细节见 document_map.py源码用uncredited_ground_truth_values列表做去重与先到先得扣减避免重复文档重复计分。2.5 DocumentNDCGEvaluator归一化折损累计增益功能NDCGNormalized Discounted Cumulative Gain同时考虑相关性与排名位置是信息检索中最常用的排序质量指标之一。若真实文档带有score相关性分数NDCG 使用这些分数否则假定所有真实文档的二元相关性为 1.0。用法示例源自 document_ndcg.pyfrom haystack import Document from haystack.components.evaluators import DocumentNDCGEvaluator evaluator DocumentNDCGEvaluator() result evaluator.run( ground_truth_documents[[Document(contentFrance, score1.0), Document(contentParis, score0.5)]], retrieved_documents[[Document(contentFrance), Document(contentGermany), Document(contentParis)]], ) print(result[individual_scores]) # [0.8869] print(result[score]) # 0.8869核心公式与源码实现document_ndcg.pyDCGcalculate_dcgDCG Σ relevance_i / log2(i 2)其中i为 0 起始的排名源码用relevant_value_to_score.pop(value)保证每个相关值最多计分一次防止重复检索虚增分数IDCGcalculate_idcg将真实文档的相关性分数降序排列后套用同一折损公式得到理想排序下的最高可能 DCGNDCG DCG / IDCG当idcg 0时有效否则为 0score为所有问题的 NDCG 均值。输入校验validate_inputsdocument_ndcg.py会抛ValueError的情形ground_truth_documents或retrieved_documents为空列表、两侧长度不一致、或某一问的真实文档中同时混有带分数与不带分数的文档要求要么全带、要么全不带。重要注意MAP、MRR、NDCG 三个评估器均不主动归一化输入文档官方建议在送入评估器前先用DocumentCleaner组件清洗并归一化文档内容避免因空白、换行、大小写等差异导致比较失真。三、语义与生成式评估3.1 SASEvaluator语义答案相似度功能SASSemantic Answer Similarity基于 Hugging Face 预训练模型计算预测答案与真实答案的语义相似度常用于 RAG 管线中评估生成答案质量。模型可以是Bi-Encoder分别编码两段文本后计算余弦相似度或Cross-Encoder成对输入直接输出相似度自动依据模型架构判断加载方式sas_evaluator.py。构造参数sas_evaluator.py参数默认值说明modelsentence-transformers/paraphrase-multilingual-mpnet-base-v2SentenceTransformers 语义相似度模型可为模型名或本地路径batch_size32每次编码的预测-标签对数量deviceNone模型加载设备None时自动选择tokenSecret.from_env_var([HF_API_TOKEN, HF_TOKEN], strictFalse)Hugging Face 访问令牌读取环境变量非必填依赖说明SASEvaluator 使用LazyImport惰性加载sentence-transformers若未安装会提示Run pip install sentence-transformers5.0.0sas_evaluator.py。用法示例源自 sas_evaluator.pyfrom haystack.components.evaluators.sas_evaluator import SASEvaluator evaluator SASEvaluator(modelcross-encoder/ms-marco-MiniLM-L-6-v2) evaluator.warm_up() # 首次使用时必须预热以加载模型 ground_truths [ A construction budget of US $2.3 billion, The Eiffel Tower, completed in 1889, symbolizes Pariss cultural magnificence., The Meiji Restoration in 1868 transformed Japan into a modernized world power., ] predictions [ A construction budget of US $2.3 billion, The Eiffel Tower, completed in 1889, symbolizes Pariss cultural magnificence., The Meiji Restoration in 1868 transformed Japan into a modernized world power., ] result evaluator.run( ground_truths_answersground_truths, predicted_answerspredictions ) print(result[score]) # 0.9999673763910929 print(result[individual_scores]) # [0.9999765157699585, 0.999968409538269, 0.9999572038650513]源码级行为说明sas_evaluator.py预测数与标签数不一致、或预测答案含None时抛ValueError空预测列表返回{score: 0.0, individual_scores: [0.0]}Cross-Encoder 路径将预测-标签组成句子对用model.predict(..., convert_to_numpyTrue)打分若原始分数大于 1非归一化 logits自动用expitsigmoid归一化到 0~1Bi-Encoder 路径分别编码两侧文本用util.cos_sim逐对计算余弦相似度最终score np_mean(similarity_scores)。3.2 LLMEvaluator通用 LLM 评估器功能基于提示词含指令与少样本示例让 LLM 对输入做自定义评估。输入列表由用户按需定义对应期望指标输出为字典results键下是逐条评估结果每条是用户自定义键 0FALSE/ 1TRUE的字典。默认使用 OpenAI需要环境变量OPENAI_API_KEY也可通过chat_generator参数替换为任意实现了ChatGenerator协议的生成器。用法示例源自 llm_evaluator.pyfrom haystack.components.evaluators import LLMEvaluator evaluator LLMEvaluator( instructionsIs this answer problematic for children?, inputs[(predicted_answers, list[str])], outputs[score], examples[ {inputs: {predicted_answers: Damn, this is straight outta hell!!!}, outputs: {score: 1}}, {inputs: {predicted_answers: Football is the most popular sport.}, outputs: {score: 0}}, ], ) predicted_answers [ Football is the most popular sport with around 4 billion followers worldwide, Python language was created by Guido van Rossum., ] results evaluator.run(predicted_answerspredicted_answers) print(results) # {results: [{score: 0}, {score: 0}]}构造参数详解llm_evaluator.py参数必填说明instructions是评估指令应为可用 yes/no 回答的关于输入的提问inputs是期望接收的输入即 Pipeline 输入连接每个为(输入名, 类型)元组类型必须是 listoutputs是评估结果输出名对应输出字典中的键examples是少样本示例每条是含inputs与outputs两个字典键的字典progress_bar否是否显示评估进度条默认Trueraise_on_failure否API 调用失败时是否抛异常默认TrueFalse时记录 warning 并把该条结果置为Nonechat_generator否自定义 LLM不传时使用OpenAIChatGenerator(generation_kwargs{response_format: {type: json_object}, seed: 42})JSON 输出要求无论使用哪个 LLM都必须配置为返回 JSON 对象。以OpenAIChatGenerator为例需在generation_kwargs中传入{response_format: {type: json_object}}。提示词模板机制prepare_templatellm_evaluator.py自动拼装为固定格式Instructions: instructions Generate the response in JSON format with the following keys: list of output keys Consider the instructions and the examples below to determine those values. Examples: Inputs: example inputs JSON Outputs: example outputs JSON Inputs: {input_name: {{ input_name }}} Outputs:该模板与PromptBuilder配合self.builder PromptBuilder(templatetemplate)运行时逐条渲染输入。run方法的执行链路llm_evaluator.py校验输入 → 将多个输入列表按位置 zip 成逐条字典 → 循环调用生成器 → 用_parse_dict_from_json解析 JSON 并核对输出键 → 汇总results与metaOpenAI 响应含meta键时会附带元数据。同步run与异步run_async双接口均已实现。序列化支持to_dict会将inputs中的类型序列化为字符串、chat_generator序列化为子组件字典from_dict反序列化时通过deserialize_type还原类型、deserialize_chatgenerator_inplace还原生成器llm_evaluator.py因此 LLMEvaluator 可无缝嵌入 YAML Pipeline 描述文件。3.3 ContextRelevanceEvaluator上下文相关性功能判断给定上下文contexts对回答问题是否相关。LLM 先把上下文拆分为多条陈述再逐条判断该陈述是否有助于回答问题每条上下文得二元分数 1 或 0并输出相关陈述列表与全部输入对的平均分。用法示例源自 context_relevance.pyfrom haystack.components.evaluators import ContextRelevanceEvaluator questions [Who created the Python language?, Why does Java needs a JVM?, Is C better than Python?] contexts [ [(Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.)], [(Java is a high-level, class-based, object-oriented programming language that is designed to have as few implementation dependencies as possible. The JVM has two primary functions: to allow Java programs to run on any device or operating system (known as the write once, run anywhere principle), and to manage and optimize program memory.)], [(C is a general-purpose programming language created by Bjarne Stroustrup as an extension of the C programming language.)], ] evaluator ContextRelevanceEvaluator() result evaluator.run(questionsquestions, contextscontexts) print(result[score]) # 0.67 print(result[individual_scores]) # [1, 1, 0] print(result[results]) # [{relevant_statements: [Python, created by Guido van Rossum in the late 1980s.], score: 1.0}, ...]构造参数context_relevance.py与FaithfulnessEvaluator、LLMEvaluator一致examples可选少样本示例格式为{inputs: {questions: ..., contexts: ...}, outputs: {relevant_statements: [...]}}不传则使用内置默认示例progress_bar是否显示进度条默认Trueraise_on_failureAPI 调用失败时是否抛异常默认Truechat_generator自定义 LLM需配置为返回 JSON默认使用 OpenAI JSON 模式。输入输出runquestions为问题列表contexts为每个问题对应一组上下文的嵌套列表。返回score所有问题的平均上下文相关分与results每条上下文含relevant_statements与score的字典列表。该方法内部使用component.set_input_types动态声明输入类型questions: list[str]与contexts: list[list[str]]并通过validate_input_parameters校验输入长度一致性。3.4 FaithfulnessEvaluator答案忠实度功能判断生成答案中的每条陈述能否从给定上下文中推断出来用于检测 LLM 幻觉。LLM 先将答案拆分为多条陈述逐条判断是否被上下文支持最终分数为该答案中可推断陈述的比例0.0~1.0。用法示例源自 faithfulness.pyfrom haystack.components.evaluators import FaithfulnessEvaluator questions [Who created the Python language?] contexts [ [(Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.)], ] predicted_answers [ Python is a high-level general-purpose programming language that was created by George Lucas. ] evaluator FaithfulnessEvaluator() result evaluator.run(questionsquestions, contextscontexts, predicted_answerspredicted_answers) print(result[individual_scores]) # [0.5] print(result[score]) # 0.5 print(result[results]) # [{statements: [Python is a high-level general-purpose programming language., # Python was created by George Lucas.], # statement_scores: [1, 0], # score: 0.5}]示例中答案被拆成两句Python is a high-level general-purpose programming language.可推断1 分与 Python was created by George Lucas.不可推断0 分——上下文说的是 Guido van Rossum忠实度 0.5有效暴露了模型把创始人替换成 George Lucas 的幻觉。构造参数与输出__init__参数与ContextRelevanceEvaluator完全一致examples格式为{inputs: {questions, contexts, predicted_answers}, outputs: {statements, statement_scores}}。run接收questions、contexts嵌套列表、predicted_answers返回score所有答案的平均忠实度individual_scores每条答案的忠实度列表results每条答案的statements陈述列表、statement_scores逐句支持情况与score。典型用法将 RAG 检索到的上下文与生成答案同时送入该评估器即可量化答案是否忠于检索证据是 RAG 幻觉检测的核心组件。四、评估器校验机制与序列化设计输入校验四类校验贯穿所有评估器——validate_init_parametersLLMEvaluator系列校验inputs必须是(名称, list 类型)元组列表、outputs必须是字符串列表、examples必须是含inputs/outputs字符串键字典的列表违规即抛ValueErrorvalidate_input_parameters校验所有期望输入均已提供、且均为等长列表is_valid_json_and_has_expected_keys/_parse_dict_from_json校验 LLM 输出是合法 JSON 且含期望键——raise_on_failureTrue时抛ValueErrorFalse时发出 warning 并返回False对应条目记为None文档类评估器的长度一致性校验如 answer_exact_match.py、document_ndcg.py。序列化AnswerExactMatchEvaluator与SASEvaluator通过default_to_dict/default_from_dict实现轻量序列化LLMEvaluator家族额外处理了 tuple 类型转成[name, serialized_type]列表存储与内嵌chat_generator的递归序列化。这意味着整个评估器集都能与 Pipeline YAML 编组 体系协同评估流程可持久化为配置文件。五、在 Pipeline 中编排评估流程评估器本质是标准 Haystack 组件可直接连接进 Pipeline。以 RAG 离线评估为例可将检索器输出接到文档类评估器、生成器输出接到语义/生成类评估器from haystack import Document, Pipeline from haystack.components.evaluators import DocumentRecallEvaluator, SASEvaluator pipeline Pipeline() pipeline.add_component(recall, DocumentRecallEvaluator(modemulti_hit)) pipeline.add_component(sas, SASEvaluator(modelsentence-transformers/paraphrase-multilingual-mpnet-base-v2)) pipeline.add_component(sas_warm_up, ...) # 注意 SASEvaluator 需在 run 前 warm_up result pipeline.run({ recall: { ground_truth_documents: [[Document(contentFrance)]], retrieved_documents: [[Document(contentFrance), Document(contentGermany)]], }, sas: { ground_truth_answers: [The capital of France is Paris.], predicted_answers: [Paris is the capital of France.], }, }) print(result[recall][score]) # 1.0 print(result[sas][score]) # 0.0x语义相似但措辞不同显著高于字面匹配实践建议检索排序评估优先组合DocumentRecallEvaluator覆盖DocumentMRREvaluator首命位置DocumentNDCGEvaluator排序质量答案质量评估组合AnswerExactMatchEvaluator字面SASEvaluator语义FaithfulnessEvaluator幻觉检测ContextRelevanceEvaluator检索上下文相关性LLM 类评估器务必先完成warm_up()并为自定义chat_generator配置 JSON 输出格式送入文档类评估器前先经DocumentCleaner归一化避免格式差异导致误判。六、深入阅读评估器源码haystack/components/evaluators含answer_exact_match.py、document_recall.py、document_mrr.py、document_map.py、document_ndcg.py、sas_evaluator.py、llm_evaluator.py、context_relevance.py、faithfulness.py组件导出入口haystack/components/evaluators/init.py评估结果数据类haystack/evaluation/eval_run_result.py文档归一化组件haystack/components/preprocessors 下的DocumentCleaner官方 API 参考docs-website 中的 evaluators_api.md本文档即其 2.19 版本其余版本位于 reference_versioned_docs 与 reference/haystack-api【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考