agno 实战用 LLM as Judge 构建结构化输出评测管线附 three 个可运行示例【免费下载链接】agnoBuild, run, and manage agent platforms.项目地址: https://gitcode.com/GitHub_Trending/ag/agno导读本文基于 agno 仓库中 cookbook/data_labeling/_17_llm_as_judge/ 目录展开讲解如何让大模型充当评审员LLM as Judge把(prompt, response)输入对喂给 Agent用output_schema强制返回 1–5 分的结构化评分。读完本文你将掌握三种评测模式——单维度打分、多维 Rubric 打分、带理由rationale的打分理解其与数据标注同源的底层机制并能直接运行、验证这些示例将其接入测试框架或生产 Agent 的评估看板。文末还附上了基于gemini-3.5-flash的真实测试记录见 TEST_LOG.md。一、什么是 LLM as Judge评测与标注的同源机制在 README.md 中agno 对这一模式给出了一句精辟的概括Score a generated output against criteria. The same machinery as labeling - input is the (prompt, response) pair, output is a structured score - but applied to evaluating models rather than producing training labels.即LLM as Judge 与数据标注共享同一套机制——输入是(prompt, response)对输出是结构化评分区别仅在于用途标注面向生产训练标签而评测面向评估模型输出质量。这一设计意味着评测 Agent 与标注 Agent 在代码结构上完全一致只是 Schema 与指令不同你可以复用 agno 的结构化输出能力output_schema获得机器可读的评分结果结果天然适合接入评测仪表盘、CI 门禁或作为奖励模型reward model训练数据。适用场景按 README.md 的 When to use 章节典型用途有三类在测试框架中评估模型输出对同一 prompt 的不同响应批量打分量化模型质量为生产环境的 Agent 工作负载构建评估看板持续监控线上 Agent 输出质量构建奖励模型训练数据可与 cookbook/data_labeling/_05_text_pairwise_preference/ 中的成对偏好数据结合使用评分结果 成对比较是经典 RLHF/RM 数据组合。二、环境准备与运行方式运行前提安装 agno本示例基于 agno 2.7.4 验证见 TEST_LOG.md三个示例均使用 Google Gemini 模型google:gemini-3.5-flash需要配置GOOGLE_API_KEY环境变量依赖pydantic用于定义评分 Schema与rich用于美化打印结果。运行命令按 README.md 提供的方式在仓库根目录下分别执行python cookbook/data_labeling/_17_llm_as_judge/basic.py python cookbook/data_labeling/_17_llm_as_judge/single_rubric.py python cookbook/data_labeling/_17_llm_as_judge/with_rationale.py每个脚本都包含独立的if __name__ __main__入口无需额外配置即可独立运行。三、示例一基础打分basic.py——单维 1–5 分完整代码basic.py 是最简单的评测原语只输出一个 1–5 分的总体质量评分。from agno.agent import Agent, RunOutput from pydantic import BaseModel, Field from rich.pretty import pprint # --------------------------------------------------------------------------- # Schema # --------------------------------------------------------------------------- class Score(BaseModel): overall: int Field( ..., ge1, le5, descriptionOverall quality on a 1-5 scale where 5 is excellent, ) # --------------------------------------------------------------------------- # Agent Instructions # --------------------------------------------------------------------------- instructions \ Score the response on overall quality: 1 - unusable 2 - poor 3 - acceptable 4 - good 5 - excellent Use the full scale. Reserve 5 for genuinely excellent responses. # --------------------------------------------------------------------------- # Create Agent # --------------------------------------------------------------------------- agent Agent( modelgoogle:gemini-3.5-flash, instructionsinstructions, output_schemaScore, ) # --------------------------------------------------------------------------- # Run Agent # --------------------------------------------------------------------------- def build_input(prompt: str, response: str) - str: return fPrompt:\n{prompt}\n\nResponse:\n{response} if __name__ __main__: prompt Explain why the sky is blue, in one sentence. samples [ ( Sunlight scatters off air molecules; shorter (blue) wavelengths scatter more, so blue dominates what we see. ), It just is., ] for response in samples: run: RunOutput agent.run(build_input(prompt, response)) pprint({response: response, score: run.content})关键点拆解评分 Schema 用 Pydantic 约束overall: int Field(..., ge1, le5)通过ge大于等于与le小于等于把分数严格限定在 1–5 整数区间从 Schema 层面杜绝越界输出。...表示该字段必填。指令定义评分标尺指令中明确写出 1–5 各档含义unusable / poor / acceptable / good / excellent并强调用满整个标尺只有真正优秀的响应才给 5 分——这是防止评测模型打分区间的关键提示工程技巧。输入模板build_input将(prompt, response)拼装为固定格式文本保证评测输入的一致性Prompt: prompt Response: responserun.content直接是结构化对象由于配置了output_schemaScoreagent.run()返回的 RunOutput 的content字段已解析为Score实例而非 JSON 字符串可直接读取.overall。测试结果来自 TEST_LOG.md在 TEST_LOG.md 的basic.py记录中测试时间 2026-07-18agno 2.7.4gemini-3.5-flash对同一 prompt用一句话解释天空为什么是蓝色的正确回答瑞利散射解释评分为Score(overall5)对非回答It just is.评分为Score(overall1)两次运行均在第一次尝试时就返回了合法的结构化输出valid structured output on the first attempt。这验证了评测 Agent 能有效区分高质量答案与敷衍答案且结构化输出具有良好的一次成功率。四、示例二多维度 Rubric 打分single_rubric.py——逐项评分 总体分完整代码basic.py 只能给出一个总分当需要跨运行、跨评审者保持一致的评测时single_rubric.py 给出了显式、命名的多维评分卡每个维度一个分数外加一个总体分。from agno.agent import Agent, RunOutput from pydantic import BaseModel, Field from rich.pretty import pprint # --------------------------------------------------------------------------- # Schema # --------------------------------------------------------------------------- class RubricScore(BaseModel): correctness: int Field(..., ge1, le5, descriptionIs everything stated true) completeness: int Field( ..., ge1, le5, descriptionDoes it answer the prompt fully ) clarity: int Field( ..., ge1, le5, descriptionEasy to understand for the target reader ) concision: int Field(..., ge1, le5, descriptionFree of wasted words) overall: int Field(..., ge1, le5, descriptionHolistic quality) # --------------------------------------------------------------------------- # Agent Instructions # --------------------------------------------------------------------------- instructions \ Score the response on each rubric criterion using a 1-5 scale: 1 - unusable, 2 - poor, 3 - acceptable, 4 - good, 5 - excellent. The overall score should reflect the holistic quality, not a simple average. A response that fails correctness can still score well on clarity, but the overall should reflect the worst dimension. # --------------------------------------------------------------------------- # Create Agent # --------------------------------------------------------------------------- agent Agent( modelgoogle:gemini-3.5-flash, instructionsinstructions, output_schemaRubricScore, ) # --------------------------------------------------------------------------- # Run Agent # --------------------------------------------------------------------------- def build_input(prompt: str, response: str) - str: return fPrompt:\n{prompt}\n\nResponse:\n{response} if __name__ __main__: prompt How do I cancel my subscription? response ( Go to Settings Subscription and click Cancel. The cancellation takes effect at the end of your current billing period. ) run: RunOutput agent.run(build_input(prompt, response)) pprint({response: response, scores: run.content})关键点拆解五维评分 SchemaRubricScore包含四个可解释维度 一个总体分correctness正确性是否所有陈述都为真completeness完整性是否完整回答了 promptclarity清晰度目标读者是否容易理解concision简洁性是否没有冗余废话overall总体质量整体质量。每个维度都是ge1, le5的整数Pydantic 的description同时充当了给模型看的字段语义说明。总体分不是平均值指令中特别强调overall应反映整体质量而非简单平均——一个正确性失败的响应可能在清晰度上拿高分但总体分应反映最差的那个维度。这一约束避免了评测模型用算术平均掩盖短板。评测一致性显式命名 Rubric 的价值在于跨运行、跨评审者的一致consistency across runs and graders——同一评分卡反复使用结果可比较、可回归。测试结果来自 TEST_LOG.md针对如何取消订阅的客服问答示例提供的回答是清晰完整的取消步骤Settings Subscription Cancel账单周期结束时生效。测试结果为RubricScore(correctness5, completeness5, clarity5, concision5, overall5)五个字段在一次结构化响应中全部独立填充且对这份清晰完整的回答给出了满分。五、示例三评分 理由with_rationale.py——可审计、可作训练数据完整代码分数本身难以审计with_rationale.py 在评分之外强制要求一句自由文本的理由rationale让人类可以快速抽查模型的推理依据正如文件 docstring 所说这段理由本身也可用作奖励模型的训练数据。from agno.agent import Agent, RunOutput from pydantic import BaseModel, Field from rich.pretty import pprint # --------------------------------------------------------------------------- # Schema # --------------------------------------------------------------------------- class Score(BaseModel): overall: int Field(..., ge1, le5, descriptionOverall quality 1-5) rationale: str Field(..., descriptionOne sentence explaining the score) # --------------------------------------------------------------------------- # Agent Instructions # --------------------------------------------------------------------------- instructions \ Score the response on a 1-5 scale where 5 is excellent. In the rationale, name the specific quality (or absence) that drove your score - quote a phrase from the response when possible. # --------------------------------------------------------------------------- # Create Agent # --------------------------------------------------------------------------- agent Agent( modelgoogle:gemini-3.5-flash, instructionsinstructions, output_schemaScore, ) # --------------------------------------------------------------------------- # Run Agent # --------------------------------------------------------------------------- def build_input(prompt: str, response: str) - str: return fPrompt:\n{prompt}\n\nResponse:\n{response} if __name__ __main__: prompt Suggest a name for a side project that helps people sleep better. response ( How about Drift? Short, evocative, and the .app domain is probably free. ) run: RunOutput agent.run(build_input(prompt, response)) pprint({response: response, score: run.content})关键点拆解混合类型 SchemaScore同时包含整数评分overall与自由文本rationale证明output_schema不只支持纯数字/纯枚举结构也支持文本字段这为可解释评测打开了空间。指令要求引用原文指令要求理由指出驱动分数的具体质量或缺陷尽可能引用响应中的原话。引用原文能让理由可验证避免模型给出与响应无关的泛泛而谈。双重价值可审计性人类评审者只需扫一眼 rationale 即可判断评分是否合理实现快速 spot-check训练数据价值(response, score, rationale)三元组本身就是优质的奖励模型reward model训练语料——理由解释了为什么得这个分。测试结果来自 TEST_LOG.md针对为助眠副项目起名的 prompt响应建议 Drift简短、有感染力、.app 域名可能可用。测试结果评分overall5理由The suggested name Drift is highly relevant and evocative for a sleep project, and the response provides helpful context regarding its domain availability.模型按指令要求引用了具体名称 Drift并给出了与该名称直接相关的推理理由与评分自洽。六、底层机制output_schema 如何驱动结构化评测三个示例都只依赖一个核心参数——Agent(output_schema...)。理解它的底层行为有助于你设计更复杂的评测 Schema。Agent 参数定义在 libs/agno/agno/agent/agent.py 的 Agent Response Model Settings 段落中与输出结构化相关的关键参数包括参数类型作用output_schemaOptional[Union[Type[BaseModel], Dict[str, Any]]]提供响应模型让输出符合指定格式可传 Pydantic 模型或符合 provider 期望的 JSON 结构input_schemaOptional[Type[BaseModel]]用于校验输入的 Schemaparse_responsebool True为 True 时模型响应被转换为output_schema对象否则返回 JSON 字符串structured_outputsOptional[bool]若 provider 支持如 OpenAIChat使用模型强制的结构化输出use_json_modebool False若为 True将输出 Schema 的 JSON 描述注入 system message 而不是传 Pydantic 对象save_response_to_fileOptional[str]将响应保存到文件在评测场景中output_schemaScorePydantic 模型会同时产生两个效果驱动模型按 Schema 生成符合字段约束如ge1, le5的 JSON驱动解析器把模型原始输出解析为Score实例当parse_responseTrue时。RunOutput 的 content 字段agent.run()返回 RunOutput其content字段类型为Optional[Any]。当配置了output_schema且parse_responseTrue时run.content就是解析后的 Pydantic 对象可直接访问字段如run.content.overall、run.content.rationale无需手动解析 JSON。这正是三个示例中pprint({score: run.content})能直接打印结构化对象的原因。从源码结构看你可以通过把parse_responseFalse改为拿到原始 JSON 字符串或通过structured_outputs/use_json_mode切换不同 provider 的结构化输出实现这为兼容不同模型供应商提供了灵活性。七、组合进阶评测结果如何流向下游评测不是终点。基于本目录示例可将结构化评分接入更完整的评测体系接入测试框架 / 评估看板把(prompt, response, RubricScore, rationale)记录到存储如 agno 的 session storage按维度聚合分数、跟踪回归构建奖励模型训练数据本目录的with_rationale.py产出的(score, rationale)可直接作为 RM 训练语料若需要成对偏好数据哪个响应更好可参考 cookbook/data_labeling/_05_text_pairwise_preference/ 目录——README 明确建议两者结合使用多人/多模型交叉评审同一(prompt, response)交给多个评测 Agent 打分取一致性统计如同分率来评估评测器本身的稳定性。八、经验总结与注意事项Schema 即约束用Field(ge1, le5)等 Pydantic 约束把评分边界写死在类型系统里比靠提示词软约束更可靠指令定标尺显式给出 1–5 各档含义、要求用满标尺能有效抑制评测模型的分数坍缩倾向打中间分Rubric 要独立且整体逐项维度 独立总体分并明确总体分反映最差维度而非平均值理由引用原文with_rationale.py证明指令可以强制模型引用响应原文从而让评分可审计一次成功率从 TEST_LOG.md 看gemini-3.5-flash agno 2.7.4 下三个示例均一次返回合法结构化输出——但这是特定模型与版本下的观测结果切换模型供应商时应重新验证。参考文件示例目录 README测试日志 TEST_LOG.mdbasic.pysingle_rubric.pywith_rationale.pyAgent 输出 Schema 参数定义RunOutput 定义成对偏好标注可与评分结合【免费下载链接】agnoBuild, run, and manage agent platforms.项目地址: https://gitcode.com/GitHub_Trending/ag/agno创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考