传统代码有单元测试：输入固定，输出对就 PASS。但 Agent 输出的是"自然语言 + 大模型推理结果"——同一个问题两次回答可能措辞都不一样，没法简单 assert 相等。

可如果不评测，你改一句 prompt、换个模型，根本不知道 Agent 变好了还是变差了。所以框架把评测做成一等公民：准备一套"标准考卷"(golden set)，给 Agent 跑一遍，从多个维度打分，不及格就拦住不让合并。代码在 packages/ai-trust-toolkit/src/ai_trust_toolkit/eval/。

"标准考卷"就是一批 EvalSample，存成 JSONL 文件（每行一个 JSON 样本），放在每个 agent 的 data/eval_samples.jsonl。先看它的真实 dataclass 定义（eval/schemas.py:61）：

# eval/schemas.py:61
@dataclass
class EvalSample:
    id: str
    description: str
    input: dict[str, Any]              # 初始 state 的输入(trace_id / service_name / user_question)
    ground_truth: GroundTruth          # 标准答案(人工标注)
    mock_responses: dict[str, list[str]] | None = None  # 喂给 FakeLLM 的固定回复

真实样本长这样（来自 apps/sre-rca-agent/data/eval_samples.jsonl 第 1 行，已折叠 mock_responses）：

{
  "id": "db-pool-001",
  "description": "service-A 连接池打满 + 慢查询",
  "input": {
    "trace_id": "trace-db-pool-001",
    "service_name": "service-A",
    "user_question": "service-A 14:23 突然变慢，DB 连接池打满"
  },
  "mock_responses": {
    "Trace 专家":   ["{\"evidence_ids\":[\"trace-1\"], \"hint\":\"SELECT span 8.2s\", ...}"],
    "Metric 专家":  ["{\"evidence_ids\":[\"metric-1\"], \"hint\":\"connection_pool_usage 98%\", ...}"],
    "最终综合节点": ["{\"hypotheses\":[{\"category\":\"DB_POOL_EXHAUSTED\", ...}], \"confidence\":\"高\"}"],
    "质检员":      ["{\"critique_passed\": true, \"rule_results\": {...}}"]
  },
  "ground_truth": {
    "primary_category": "DB_POOL_EXHAUSTED",
    "candidate_categories": ["DB_POOL_EXHAUSTED", "DB_SLOW_QUERY"],
    "keywords_in_hypothesis": ["service-A", "连接池", "98%"],
    "min_supporting_specialists": 3,
    "overall_confidence_at_least": "高"
  }
}

再看"参考答案" GroundTruth 的字段（schemas.py:21）——每个字段就是评测的一把尺子：

# eval/schemas.py:21
@dataclass
class GroundTruth:
    primary_category: str = ""                       # 期望的第一根因 category
    candidate_categories: list[str] = field(default_factory=list)   # 期望进 Top-3 的集合
    keywords_in_hypothesis: list[str] = field(default_factory=list) # 结论文本必须含的关键词
    min_supporting_specialists: int = 2              # 至少几位专家印证
    overall_confidence_at_least: str = "中"          # 置信度至少几级(高>中>低)
    detail: dict[str, Any] = field(default_factory=dict)  # v0.6+:扩展字段桶

load_jsonl()（schemas.py:84）逐行读、跳过空行和 # 注释行，每行 json.loads 后交给 from_dict。而 GroundTruth.from_dict（schemas.py:44）里藏着关键一手：

# eval/schemas.py:44
@classmethod
def from_dict(cls, d: dict) -> GroundTruth:
    detail = dict(d.get("detail") or {})
    for k, v in d.items():
        if k not in _KNOWN_GT_FIELDS:      # 不认识的 key
            detail.setdefault(k, v)        # 自动收进 detail 桶
    return cls(
        primary_category=d.get("primary_category") or d.get("top1_category", ""),  # 别名兼容
        ...
    )

像 sre-rca 这种"给结论"的 agent，用 FiveDimScorer（scorers.py:48）打 5 个布尔维度——每一维过没过：

现在钻进 FiveDimScorer.score()（scorers.py:54）看它逐维怎么算。第一步是从图的最终 state 里把结论抠出来：

# eval/scorers.py:60
score = EvalScore(sample_id=sample_id)
conclusion = final_state.get(self.conclusion_field) or {}          # 默认取 state["conclusion"]
hypotheses = conclusion.get("hypotheses") or conclusion.get("top_root_causes") or []

然后 5 个维度一个个填。挑最关键的三维看：

# ① top1_hit —— 第一条 hypothesis 的 category 对不对（scorers.py:65）
top1_cat = _get_hypothesis_field(hypotheses[0] if hypotheses else None, "category")
score.top1_hit = top1_cat is not None and _category_match(top1_cat, ground_truth.primary_category)

# ③ confidence —— 用 {低:1,中:2,高:3} 把中文置信度变成可比的数（scorers.py:76）
actual_conf   = conclusion.get("confidence") or conclusion.get("overall_confidence") or "低"
actual_rank   = CONFIDENCE_RANK.get(str(actual_conf), 1)
expected_rank = CONFIDENCE_RANK.get(ground_truth.overall_confidence_at_least, 2)
score.confidence_meets = actual_rank >= expected_rank

# ⑤ keywords —— 半数以上关键词命中算过（scorers.py:89）
text_blob = (_flatten_text(hypotheses) + _flatten_text([conclusion])).lower()
if not ground_truth.keywords_in_hypothesis:
    score.keywords_meets = True                       # 没要求关键词 → 直接算过
else:
    hits = sum(1 for kw in ground_truth.keywords_in_hypothesis if kw.lower() in text_blob)
    ratio = hits / len(ground_truth.keywords_in_hypothesis)
    score.keywords_meets = ratio >= 0.5

像 spec-executor 这种"改代码"的 agent，评价标准不同——用 CodingAgentScorer（scorers.py:107）打 6 个加权浮点维度（权重之和=1.0）：

权重是写死的常量（scorers.py:22），build_pass（能不能通过构建）占最重的 0.30——因为对 coding agent 来说"改完能跑"是硬指标：

# eval/scorers.py:22
CODING_DIM_WEIGHTS: dict[str, float] = {
    "tool_success": 0.20, "build_pass": 0.30, "patch_count": 0.15,
    "iterations": 0.15, "risk_distribution": 0.10, "change_minimality": 0.10,
}   # sum == 1.0

# score() 里加权求和 → total（scorers.py:153）
total = round(sum(dims[k] * self.weights[k] for k in self.weights), 4)
return EvalScore(sample_id=sample_id, dim_scores=dims, total=total,
                 total_passed=total >= self.slo)   # 默认 SLO=0.6

change_minimality（改动是否最小化）有两种裁判模式（scorers.py:166）：默认 heuristic 纯算数（实际 hunk 数 vs golden hunk 数的超出比例），或 haiku 让便宜 LLM 当裁判。而 haiku 模式最能体现工程化的谨慎——它有三层回退：

# eval/scorers.py:177  _haiku_minimality_judge
def _haiku_minimality_judge(self, state, gt) -> float:
    try:
        from ai_trust_toolkit.llm import get_llm
        from langchain_core.messages import HumanMessage, SystemMessage
    except ImportError:
        return self._score_minimality_heuristic_fallback(state, gt)   # 回退①:没装 langchain
    if not self._minimality_prompt:
        return self._score_minimality_heuristic_fallback(state, gt)   # 回退②:prompt 文件缺失
    llm = get_llm("haiku")
    try:
        resp = llm.invoke([SystemMessage(...), HumanMessage(...)])
        return _extract_score_from_text(str(resp.content))
    except Exception:
        return self._score_minimality_heuristic_fallback(state, gt)   # 回退③:LLM 调用失败

把样本、图、scorer 交给 run_eval()（runner.py:17）就能跑批：

samples = EvalSample.load_jsonl("data/eval_samples.jsonl")
report = await run_eval(graph, samples, scorer=FiveDimScorer())
report.print_summary()              # 漂亮打印各维通过率

# CI 里用阈值判定，不达标就退出码非 0
if not report.meets_thresholds(EvalThresholds(top1=0.5, top3=0.7)):
    raise SystemExit(1)

现在看 run_eval 的真实实现（runner.py:17）——它跟具体 agent 完全解耦，你只要把编译好的 graph 传进来。核心是这段并发跑批：

# eval/runner.py:42
semaphore = asyncio.Semaphore(concurrency)          # 控制同时跑几条(默认 1=串行)

async def _run_one(idx: int, sample: EvalSample) -> EvalScore:
    async with semaphore:                           # 拿到令牌才开跑
        try:
            initial = initial_state_builder(sample)         # sample.input → 初始 state
            config  = config_builder(sample, idx)           # 每条一个独立 thread_id
            final   = await graph.ainvoke(initial, config=config)   # 真跑图
            return scorer.score(sample_id=sample.id, final_state=final,
                                ground_truth=sample.ground_truth)   # 打分
        except Exception as e:
            err_score = EvalScore(sample_id=sample.id)
            err_score.error = f"{type(e).__name__}: {e}"    # 挂了只记这条的 error
            return err_score

scores = await asyncio.gather(*[_run_one(i, s) for i, s in enumerate(samples)])

EvalThresholds（schemas.py:98）定义每一维的及格线，默认：top1=0.5、top3=0.7、confidence=0.7、specialists=0.7、keywords=0.6。跑完得到的 EvalReport 会先把每条样本的布尔汇总成"通过率"，再逐维对照阈值：

# eval/schemas.py:172  某一维的通过率 = 这一维为 True 的样本数 / 总样本数
def rate(self, dim: str) -> float:
    attr_name = {"top1":"top1_hit", "top3":"top3_hit", "confidence":"confidence_meets",
                 "specialists":"specialists_meets", "keywords":"keywords_meets"}[dim]
    return sum(1 for s in self.scores if getattr(s, attr_name)) / len(self.scores)

# eval/schemas.py:222  任何一维通过率低于阈值 → 整体不达标
def meets_thresholds(self, thresholds: EvalThresholds) -> bool:
    s = self.summary()
    return all(s[dim] >= getattr(thresholds, dim)
               for dim in ("top1","top3","confidence","specialists","keywords"))

这个能力被接进了 CI 流水线（Day 18 细讲）——有一道专门的闸门 ci-gate-eval：

除了 CI 里用 golden set 的"离线评测"，框架还有两个进阶能力：

ObservableScorer — 在线评测（scorers.py:381）

golden set 需要人工标答案，成本高、覆盖有限。ObservableScorer 不依赖标准答案（也 不调 LLM，纯统计），从线上真实调用的 envelope 信号算 5 个 SLO 维度。默认阈值写在 OBSERVABLE_SLO_DEFAULTS（scorers.py:372），看它怎么判"违约"：

# eval/scorers.py:372  在线 SLO 默认阈值
OBSERVABLE_SLO_DEFAULTS = {
    "success_rate": 0.85,        # status="success" 占比 · 跌破触红
    "critic_end_rate": 0.10,     # Critic 拦下(不敢下结论)占比 · 涨过触红
    "p99_latency_ms": 30000.0,   # 延迟 p99 · 涨过触红
    "p99_cost_usd": 0.50,        # 成本 p99 · 涨过触红
    "failure_no_LLM_rate": 0.02, # 预算/限流类硬失败占比
}

# eval/scorers.py:441  score_aggregate 里判一维是否违约(以成功率为例)
success_rate = status_distribution.get("success", 0.0)
if success_rate < self.slos["success_rate"]:
    severity = "red" if success_rate < self.slos["success_rate"] - 0.1 else "yellow"
    breaches.append(SloBreach(dim="success_rate", actual=success_rate,
                              threshold=self.slos["success_rate"], severity=severity))

RegressionDetector — 回归检测

对比"这次评测"和"历史基线"，自动找出哪一维退步了。配合仓库里的 eval-replay skill（Day 19）：给一个线上出问题的 case_id，能拉出历史输入重放、对比 5 维评分、定位是哪一维掉了。

✋ 动手

# 1. 看 eval 模块清单
sed -n '1,61p' packages/ai-trust-toolkit/src/ai_trust_toolkit/eval/__init__.py

# 2. 看一个真实的 golden 样本文件
head -3 apps/sre-rca-agent/data/eval_samples.jsonl

# 3. 读 5 维评分器
sed -n '48,110p' packages/ai-trust-toolkit/src/ai_trust_toolkit/eval/scorers.py

# 4. 亲自跑一遍 sre-rca 的评测
uv run python apps/sre-rca-agent/scripts/run_eval.py

# 5. 跑 eval 单测
uv run pytest packages/ai-trust-toolkit/tests/test_eval.py -q