Make the governed RAG evidence path executable end to end
Some checks failed
verify / verify (push) Has been cancelled
Some checks failed
verify / verify (push) Has been cancelled
Separate local parsing from model indexing, bind review decisions to immutable manifests, persist vectors behind active profiles, and expose retrieval, chat, evaluation, and document workflows through the React workbench. Constraint: Live Bailian authentication currently fails for all three configured capabilities Rejected: Direct upload-to-embedding flow | bypasses local review and manifest binding Confidence: high Scope-risk: broad Directive: Keep private-data deployment blocked until authentication, RBAC, and separate database roles land Tested: make verify; fresh and replay Docker document smoke; worker recovery smoke; frozen synthetic evaluation; migration 0003-0004 roundtrip Not-tested: Successful live Bailian calls, OCR, real multi-user authorization
This commit is contained in:
112
docs/07-retrieval-worker-evaluation-runbook.md
Normal file
112
docs/07-retrieval-worker-evaluation-runbook.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# 正式检索、Worker 与评测运行手册
|
||||
|
||||
本文验证已落地的可运行切片:正式 pgvector 检索与重排接口、PostgreSQL 租约栅栏,
|
||||
以及可复现的 synthetic 检索评测。它不把 synthetic 指标等同于真实地质语料效果,
|
||||
也不表示当前百炼凭据已经通过线上验收。
|
||||
|
||||
## 1. 启动并准备合成知识库
|
||||
|
||||
```bash
|
||||
bash scripts/init-local-secrets.sh
|
||||
docker compose up -d --build web
|
||||
docker compose --profile tools run --rm seed-demo-offline
|
||||
docker compose ps --all
|
||||
```
|
||||
|
||||
`migrate` 是一次性任务,显示 `Exited (0)` 表示 Alembic 已成功升级后正常退出。持续运行的
|
||||
`db/model-gateway/api/gateway/web` 应为 `healthy`,且只有 Web 发布
|
||||
`127.0.0.1:8000`。
|
||||
|
||||
## 2. 验证正式 Retrieval API
|
||||
|
||||
```bash
|
||||
curl -sS -X POST http://127.0.0.1:8000/api/v1/retrieval/search \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"knowledge_base_id":"3acd3785-970b-55f7-a669-9eb4695e27eb",
|
||||
"query":"花岗斑岩铜矿化特征",
|
||||
"vector_top_k":50,
|
||||
"rerank_top_n":10
|
||||
}'
|
||||
```
|
||||
|
||||
成功响应应包含:
|
||||
|
||||
- 服务端生效的 embedding profile hash、模型和 1024 维契约;
|
||||
- `vector_rank/vector_score` 与 `rank/rerank_score`;
|
||||
- 稳定且不透明的 `citation_id`、文档 ID、章节、页码和安全片段;
|
||||
- embedding、数据库、rerank 和总耗时,以及请求 trace ID;
|
||||
- `rerank_status=applied`;重排不可用时则保留初召回并明确标记 `degraded`。
|
||||
|
||||
请求体不能提交 access scope。当前 synthetic 身份由服务器固定映射到 synthetic 知识库;候选
|
||||
SQL 在 `LIMIT` 前同时过滤知识库、授权 scope、当前激活版本、云审批、READY assignment、
|
||||
profile 和 searchable 状态。
|
||||
|
||||
浏览器访问 <http://127.0.0.1:8000/retrieval> 可查看同一正式接口的 React 检索实验室。
|
||||
|
||||
## 3. 运行冻结配置评测
|
||||
|
||||
```bash
|
||||
docker compose --profile tools run --rm seed-demo-offline \
|
||||
python -m app.tools.evaluate_demo \
|
||||
/demo/demo_documents.jsonl /demo/demo_queries.jsonl
|
||||
```
|
||||
|
||||
输出是单行 JSON 工件,至少包含:
|
||||
|
||||
- corpus/query set SHA-256;
|
||||
- active embedding profile hash;
|
||||
- vector/rerank/cutoff 参数和 bootstrap seed;
|
||||
- 完整冻结配置 SHA-256;
|
||||
- 每题排序及 Hit、Recall、MRR、nDCG、CompleteHit、EvidenceGroupRecall;
|
||||
- 聚合指标和固定 seed 的 95% bootstrap 区间。
|
||||
|
||||
公开样例的 9 个可回答问题当前应达到 `Hit@3=1.0`。这是为了验证流水线和指标实现的
|
||||
synthetic 基线,不是论文最终质量结论;正式结论必须使用冻结的开发集/盲测集和真实合法语料。
|
||||
任何 top-k 未判定候选都会使正式评分失败,不能静默按 0 处理。
|
||||
|
||||
## 4. 验证 Worker 并发租约与 fencing
|
||||
|
||||
```bash
|
||||
docker compose --profile tools run --rm worker-smoke
|
||||
```
|
||||
|
||||
预期输出:
|
||||
|
||||
```json
|
||||
{
|
||||
"claim_winners": 1,
|
||||
"expired_lease_rejected": true,
|
||||
"recovery_claim_winners": 1,
|
||||
"recovery_token_rotated": true,
|
||||
"stale_fence_rejected": true,
|
||||
"status": "ok",
|
||||
"terminal_status": "SUCCEEDED"
|
||||
}
|
||||
```
|
||||
|
||||
工具在真实 PostgreSQL 中创建一条随机 capability 的 synthetic 任务,让两个领取者并发竞争,
|
||||
验证只有一个领取成功、伪造 token 和已过期租约均无法心跳、过期任务只能由一个新 Worker
|
||||
使用旋转后的 token 重领并完成,最后删除该验证行。Worker
|
||||
心跳配置被强制限制为不超过租约的三分之一。该工具证明队列运行时和数据库 fence;具体文档
|
||||
解析/向量化 handler 仍需按各自业务验收,不可据此推断完整入库链已经完成。
|
||||
|
||||
## 5. 真实百炼边界
|
||||
|
||||
正式非 synthetic profile 会由 API/Worker 通过内部 token 调用 `model-gateway`,只有后者持有
|
||||
百炼 Key 和公网出口。当前本机凭据对 `text-embedding-v4`、`qwen3-rerank` 和
|
||||
`deepseek-v4-flash` 均返回供应商鉴权失败;离线检索成功不代表线上三模型成功。更换为有效、
|
||||
同一北京工作空间且具备模型权限的新 Key 后,先按
|
||||
[Stage 1 运行手册](05-stage1-runbook.md)执行 `provider-smoke`,三项均成功后才运行真实 seed。
|
||||
|
||||
## 6. 提交前门禁
|
||||
|
||||
```bash
|
||||
make verify
|
||||
docker compose --profile tools run --rm worker-smoke
|
||||
docker compose --profile tools run --rm seed-demo-offline \
|
||||
python -m app.tools.evaluate_demo \
|
||||
/demo/demo_documents.jsonl /demo/demo_queries.jsonl
|
||||
```
|
||||
|
||||
任何输出都不得包含 API Key、内部 token、数据库密码、DSN 或私有文档正文。
|
||||
Reference in New Issue
Block a user