Compare commits

...

2 Commits

27 changed files with 2834 additions and 172 deletions
+27 -1
View File
@@ -1,6 +1,6 @@
# WiseGrade(智评)交接说明 # WiseGrade(智评)交接说明
更新日期:2026-09-27。先阅读 [项目指南](docs/GUIDE.md)(部署、使用、技术设计与会话实现);本文件记录当前实现、验证边界和接手后的优先事项。 更新日期:2026-09-30。先阅读 [项目指南](docs/GUIDE.md)(部署、使用、技术设计与会话实现);本文件记录当前实现、验证边界和接手后的优先事项。
## 当前状态 ## 当前状态
@@ -79,6 +79,32 @@ uv run pytest -q
当前代码位于 `main` 分支,阶段提交使用 Conventional Commits。交接前工作区应保持干净;`.env`、`models/`、`data/`、缓存和构建产物由 `.gitignore` 排除。 当前代码位于 `main` 分支,阶段提交使用 Conventional Commits。交接前工作区应保持干净;`.env`、`models/`、`data/`、缓存和构建产物由 `.gitignore` 排除。
## 2026-09-30 目录、摘要与正文证据治理
### 本次改动
新增 `backend/wisegrade/pages.py`(页面分类、混合页切块、真实章节路径)和 `backend/wisegrade/evidence.py`(证据政策)。`rag.py` 重写为按目标检索:页面类型过滤在候选上限之前完成,语义与中文二元组关键词各自召回后按排名倒数融合,名额先分散到不同页面并对同章节设软上限。`scoring.py` 按必备证据目标分别检索,正文补检走目录导航,内容目标缺失时进入 `needs_review`。迁移 `0006` 添加页面类型、章节、证据目标与历史标记、评分项缺证据记录列,并按已保存的解析结果重新分类页面;索引版本升到 3。
关键词召回不再使用 PostgreSQL `to_tsvector('simple', …)`。在隔离测试库中对真实页面实测:该配置把整个中文串当成一个词元,查询"文献调研"无法命中含"第2章文献调研与技术现状"的页面(`文献调研`、`串级PID 结构`、`视觉伺服与闭环控制` 均不匹配),因此旧实现的关键词分实际上长期接近零。现改为字符二元组加 ASCII/数字整词的重叠度。
### 修复的缺陷
批次 3 的 77 条评分引文中 14 条来自目录页(涉及 9 个评分项),会话 6 的 20 条引用中 5 条指向阶段报告 1 的目录页。根因有两层:目录页文本被当作正文片段参与检索并占用候选名额;评分提示词只要求引文原文和页码一致,没有任何来源类型约束。现在目录页作为导航块单独存放,内容目标只接受正文和附录,引文必须标注它要支持的证据目标。
### 本次验证(隔离环境)
隔离 PostgreSQL + pgvector 测试容器执行 `pytest -q -m 'not integration'`:**59 通过、2 个真实集成测试未运行**。新增 `backend/tests/test_evidence_pages.py` 覆盖阶段报告 1 目录与第 6–7 页文献调研正文的区分、阶段报告 3 目录与第 11–12 页测试正文的区分、最终报告摘要/跨页目录/正文/附录的区分、混合页与目录续页与重复页眉与识别不确定页、内容性判断拒绝目录与摘要与单独标题、结构性判断仍可用目录、正文补检不足时模型分为 null 且状态为待教师复核且教师可复核且缺证据记录保留、Agent 引用页面类型标记与历史评分引用标记、检索过滤先于候选上限、学生项目与课程版本隔离、批次快照记录证据政策。前端 `npm run build` 通过(仍有 Vite 大块提示)。
真实报告的分类结果已逐页核对:阶段报告 1 第 1 页封面、第 2–3 页目录、第 4–21 页正文、第 22 页参考文献;阶段报告 3 第 2–3 页目录、第 11–12 页为测试正文;最终报告第 2–3 页摘要、第 4–5 页跨页目录、第 49 页参考文献、第 50–53 页附录。
### 未验证与待业务环境执行
- 使用真实 DeepSeek 的完整评分批次**未**在本次运行。需要模型的部分(评分调用、事实抽取、一致性检查、Agent 回答)只用测试替身验证了编排与校验逻辑,尚未用真实模型端到端复跑。
- 未对业务数据库执行迁移、重建索引或重新评分,也未运行 PDF/Excel 导出(导出代码已更新,但未实际生成文件)。
- 未在 Windows Docker Desktop 实测。
业务环境应用步骤与批次 3 复核清单见 [docs/EVIDENCE_POLICY_BATCH3.md](docs/EVIDENCE_POLICY_BATCH3.md)。
## 2026-09-27 会话更新 ## 2026-09-27 会话更新
新版 UI 的入口仍使用 `?page=chat`,旧 API 与历史记录保持可读。前端按运行 ID 订阅 SSE,并依靠会话列表和详情轮询恢复后台 Agent 运行;服务进程中断后的运行在 15 分钟超时后标为中断。正式 API 启动时会自动应用 `0004` 迁移。此处的测试结果以本次实际命令为准,历史验证记录仍保留在上方。 新版 UI 的入口仍使用 `?page=chat`,旧 API 与历史记录保持可读。前端按运行 ID 订阅 SSE,并依靠会话列表和详情轮询恢复后台 Agent 运行;服务进程中断后的运行在 15 分钟超时后标为中断。正式 API 启动时会自动应用 `0004` 迁移。此处的测试结果以本次实际命令为准,历史验证记录仍保留在上方。
+2
View File
@@ -22,6 +22,8 @@ WiseGrade 是面向《自动化专业综合实践》课程的教师端自动评
- [会话与业务 Agent 实现](docs/GUIDE.md#agent):任务、数据、工具边界、API 和恢复机制。 - [会话与业务 Agent 实现](docs/GUIDE.md#agent):任务、数据、工具边界、API 和恢复机制。
- [开发与测试](docs/GUIDE.md#development) · [故障定位](docs/GUIDE.md#troubleshooting)。 - [开发与测试](docs/GUIDE.md#development) · [故障定位](docs/GUIDE.md#troubleshooting)。
[证据政策上线与批次 3 复核清单](docs/EVIDENCE_POLICY_BATCH3.md) 记录把页面类型证据政策应用到业务环境的步骤,以及批次 3 建议复核的评分项。该文件中的业务数据操作均未执行。
[开发提示词](docs/prompts/README.md) 集中存放阶段任务说明;[交接记录](HANDOFF.md) 保留实现进展、验证记录和接手事项。 [开发提示词](docs/prompts/README.md) 集中存放阶段任务说明;[交接记录](HANDOFF.md) 保留实现进展、验证记录和接手事项。
课程原始资料及测试示例仍位于 `docs/` 根目录,应用和测试按现有路径读取;学生原件、模型权重和运行产物不纳入项目文档。 课程原始资料及测试示例仍位于 `docs/` 根目录,应用和测试按现有路径读取;学生原件、模型权重和运行产物不纳入项目文档。
@@ -0,0 +1,128 @@
"""Page-level evidence typing, target-attributed citations and review state.
Revision ID: 0006
Revises: 0005
Existing indexes are deliberately not upgraded in place. Chunking changed (a contents
page is now stored as navigation rather than body text), so legacy documents keep their
old ``index_version`` and are excluded from retrieval until the reindex task rebuilds
them. That is what stops an old index from being silently used as classified body text.
Reclassification of page types needs no embedding model, so this migration applies it
immediately for every document whose parsed pages are still saved on disk.
"""
import json
from alembic import op
import sqlalchemy as sa
revision = "0006"
down_revision = "0005"
branch_labels = None
depends_on = None
PAGE_COLUMNS = (
("page_type", sa.String(20), "unknown"),
("mixed", sa.Boolean(), "false"),
)
PAGE_TEXT_COLUMNS = (
("section", sa.String(500), ""),
("chapter", sa.String(500), ""),
)
CHUNK_COLUMNS = (
("chapter", sa.String(500), ""),
("page_type", sa.String(20), "unknown"),
("block_type", sa.String(20), "legacy"),
)
EVIDENCE_COLUMNS = (
("target_id", sa.String(50), ""),
("target_label", sa.String(300), ""),
("page_type", sa.String(20), "unknown"),
("source_role", sa.String(20), "content"),
("historical", sa.Boolean(), "false"),
)
def _columns(table):
return {column["name"] for column in sa.inspect(op.get_bind()).get_columns(table)}
def _add(table, name, column_type, default, *, nullable=False):
if name in _columns(table):
return False
op.add_column(table, sa.Column(name, column_type, nullable=nullable, server_default=default))
return True
def upgrade():
added = {}
for name, column_type, default in PAGE_COLUMNS:
added[f"knowledge_pages.{name}"] = _add("knowledge_pages", name, column_type, default)
added["knowledge_pages.section"] = _add("knowledge_pages", "section", sa.String(500), "")
added["knowledge_pages.chapter"] = _add("knowledge_pages", "chapter", sa.String(500), "")
added["knowledge_pages.confidence"] = _add("knowledge_pages", "confidence", sa.Float(), None, nullable=True)
added["knowledge_pages.classification"] = _add("knowledge_pages", "classification", sa.JSON(), "{}")
for name, column_type, default in CHUNK_COLUMNS:
added[f"knowledge_chunks.{name}"] = _add("knowledge_chunks", name, column_type, default)
for name, column_type, default in EVIDENCE_COLUMNS:
added[f"score_evidences.{name}"] = _add("score_evidences", name, column_type, default)
added["criterion_scores.evidence_gaps"] = _add("criterion_scores", "evidence_gaps", sa.JSON(), "[]")
db = op.get_bind()
# Quotations written before this policy existed are marked as historical so the
# interface never presents a contents-page quotation as report body evidence.
db.execute(sa.text("UPDATE score_evidences SET historical = true WHERE historical = false"))
# A legacy chunk was never split by block, so it may not be used as content
# evidence until the reindex task rebuilds it.
if added["knowledge_chunks.block_type"]:
db.execute(sa.text("UPDATE knowledge_chunks SET block_type = 'legacy'"))
_reclassify_saved_pages(db)
def _json(value):
return json.loads(value) if isinstance(value, str) else value or {}
def _reclassify_saved_pages(db):
"""Re-derive page types from saved parse output; retryable and audit friendly."""
from wisegrade.pages import classify_document
files = {row.id: _json(row.parsed) for row in db.execute(sa.text(
"SELECT id, parsed FROM submission_files")).mappings()}
documents = db.execute(sa.text(
"SELECT id, submission_file_id FROM knowledge_documents WHERE submission_file_id IS NOT NULL")).mappings().all()
update = sa.text("UPDATE knowledge_pages SET page_type = :page_type, mixed = :mixed, section = :section, "
"chapter = :chapter, confidence = :confidence, classification = :classification "
"WHERE document_id = :document_id AND page = :page")
chunk_update = sa.text("UPDATE knowledge_chunks SET page_type = :page_type, section = :section, "
"chapter = :chapter WHERE document_id = :document_id AND page = :page")
rebuilt = 0
for doc in documents:
pages = files.get(doc.submission_file_id, {}).get("pages", [])
if not pages:
continue
for item in classify_document(pages):
payload = {"page_type": item["page_type"], "mixed": bool(item["mixed"]),
"section": item["section"], "chapter": item["chapter"],
"confidence": item["confidence"],
"classification": json.dumps({**item["signals"],
"classifier_version": item["classifier_version"]},
ensure_ascii=False, default=str),
"document_id": doc.id, "page": item["page"]}
db.execute(update, payload)
db.execute(chunk_update, payload)
rebuilt += 1
if rebuilt:
db.execute(sa.text("INSERT INTO audit_logs (action, entity_type, entity_id, details, created_at) "
"VALUES ('reclassify', 'knowledge_document', NULL, :details, CURRENT_TIMESTAMP)"),
{"details": json.dumps({"migration": "0006", "documents": rebuilt}, ensure_ascii=False)})
def downgrade():
for table, names in (("criterion_scores", ("evidence_gaps",)),
("score_evidences", tuple(name for name, _, _ in EVIDENCE_COLUMNS)),
("knowledge_chunks", tuple(name for name, _, _ in CHUNK_COLUMNS)),
("knowledge_pages", ("page_type", "mixed", "section", "chapter", "confidence", "classification"))):
for name in names:
if name in _columns(table):
op.drop_column(table, name)
+3 -1
View File
@@ -138,6 +138,8 @@ def test_scoring_uses_the_same_page_image_as_its_saved_evidence(db, environment)
max_score=criterion["max_score"], reason="测试评分", max_score=criterion["max_score"], reason="测试评分",
evidence=[{"chunk_id": chunk.id, "page": 2, "quote": "原文证据"}], confidence=.9) evidence=[{"chunk_id": chunk.id, "page": 2, "quote": "原文证据"}], confidence=.9)
# The evidence page must be a substantive body page: a contents page cannot support
# a content judgement, so the source page is classified accordingly during ingest.
scored = _score_criterion(db, batch, project, criterion, FakeClient(), scored = _score_criterion(db, batch, project, criterion, FakeClient(),
retriever=lambda *_, **kwargs: [] if kwargs.get("public_only") else [evidence]) retriever=lambda *_, **kwargs: [] if kwargs.get("public_only") else [evidence])
saved = db.scalar(select(ScoreEvidence).where(ScoreEvidence.score_id == scored.id)) saved = db.scalar(select(ScoreEvidence).where(ScoreEvidence.score_id == scored.id))
@@ -333,7 +335,7 @@ def test_sse_persists_messages_and_tool_failure(session_factory, monkeypatch):
def test_tool_arguments_cover_registered_fields_without_review_prose(): def test_tool_arguments_cover_registered_fields_without_review_prose():
numbers = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", "rubric_version_id", numbers = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", "rubric_version_id",
"score", "offset", "limit", "project_offset", "criterion_offset"} "score", "offset", "limit", "project_offset", "criterion_offset", "document_id"}
strings = {"query", "criterion_id", "format", "status", "student_no", "relative_dir"} strings = {"query", "criterion_id", "format", "status", "student_no", "relative_dir"}
redacted = {"comment", "reason"} redacted = {"comment", "reason"}
fields = set().union(*(tool.schema.model_fields for tool in agent_tools.TOOLS.values())) fields = set().union(*(tool.schema.model_fields for tool in agent_tools.TOOLS.values()))
+9 -4
View File
@@ -224,7 +224,9 @@ def test_score_citations_link_the_canonical_text_and_image_page(session_factory,
assert generated["citations"][0]["url"] == expected assert generated["citations"][0]["url"] == expected
assert generated["citations"][0]["image_url"] == expected + "/image" assert generated["citations"][0]["image_url"] == expected + "/image"
assert generated["citations"][0]["score_url"].startswith("/?page=scores") assert generated["citations"][0]["score_url"].startswith("/?page=scores")
assert chat_api._answer_citations(generated["citations"])[0]["url"] == expected # An answer keeps only the citations it actually used: naming the page retains it.
assert chat_api._answer_citations(generated["citations"], "评分证据见第 2 页。")[0]["url"] == expected
assert chat_api._answer_citations(generated["citations"], "没有引用任何页面。") == []
conversation_id = conversation.id conversation_id = conversation.id
client = client_for(session_factory) client = client_for(session_factory)
try: try:
@@ -268,8 +270,11 @@ def test_late_citations_and_search_snippets_survive_restoration(session_factory,
assert response.status_code == 200 assert response.status_code == 200
detail = client.get(f"/api/conversations/{created['id']}").json() detail = client.get(f"/api/conversations/{created['id']}").json()
answer = detail['messages'][-1] answer = detail['messages'][-1]
assert len(answer['citations']) == 30 # The answer names page 48 only, so only that page's passages are attached; the
assert [item['page'] for item in answer['citations'][-6:]] == [48] * 6 # other tool hits were retrieved but not used in the answer.
assert len(answer['citations']) == 6
assert [item['page'] for item in answer['citations']] == [48] * 6
# Every tool hit stays available in the tool record for auditing.
assert answer['tool_calls'][-1]['result']['citations'][0]['quote'] == '第48页原文片段0' assert answer['tool_calls'][-1]['result']['citations'][0]['quote'] == '第48页原文片段0'
assert detail['runs'][0]['tool_calls'][-1]['result']['citations'][0]['page'] == 48 assert detail['runs'][0]['tool_calls'][-1]['result']['citations'][0]['page'] == 48
finally: finally:
@@ -383,7 +388,7 @@ def test_upgrade_from_legacy_0003_migrates_pages_and_citations(session_factory,
config.set_main_option('script_location', str(root / 'migrations')) config.set_main_option('script_location', str(root / 'migrations'))
command.upgrade(config, 'head') command.upgrade(config, 'head')
with session_factory() as db: with session_factory() as db:
assert db.scalar(text('SELECT version_num FROM alembic_version')) == '0005' assert db.scalar(text('SELECT version_num FROM alembic_version')) == '0006'
assert db.get(Conversation, conversation_id).title == '旧记录' assert db.get(Conversation, conversation_id).title == '旧记录'
assert db.scalar(select(ChatMessage.content).where(ChatMessage.conversation_id == conversation_id)) == '旧问题' assert db.scalar(select(ChatMessage.content).where(ChatMessage.conversation_id == conversation_id)) == '旧问题'
run = db.scalar(select(AgentRun).where(AgentRun.conversation_id == conversation_id)) run = db.scalar(select(AgentRun).where(AgentRun.conversation_id == conversation_id))
+444
View File
@@ -0,0 +1,444 @@
"""Regression tests for page-typed evidence retrieval and scoring.
The real course reports in ``docs/`` are used where the acceptance criteria name them,
plus synthetic pages for the cases those files do not contain (a genuinely mixed page,
a contents continuation, a repeated header, an unsettled page). A constant embedding
vector is used on purpose: it leaves semantic similarity flat, so the tests prove the
Chinese keyword recall and the page-type filtering, which is where the reported defect
came from.
"""
from pathlib import Path
import pymupdf as fitz
import pytest
from sqlalchemy import select
from wisegrade.evidence import normalize_targets
from wisegrade.llm import ModelError, ScoreResult, validate_score
from wisegrade.models import CriterionScore, KnowledgeChunk, Project, ScoreEvidence, Student
from wisegrade.pages import classify_document
from wisegrade.rag import retrieve, retrieve_targets
from wisegrade.scoring import _score_criterion
#: Constant vectors: semantic ranking contributes nothing, so recall is lexical.
FLAT = lambda texts: [[0.1] * 1024 for _ in texts]
COURSE_DOCS = ("阶段报告1.pdf", "阶段报告3.pdf", "自动化专业综合实践报告.pdf")
def docs_root() -> Path:
for candidate in (Path("/app/course-docs"), Path(__file__).resolve().parents[2] / "docs",
Path(__file__).resolve().parents[1] / "docs"):
if all((candidate / name).is_file() for name in COURSE_DOCS):
return candidate
pytest.skip("课程示例报告不可用")
def report_pages(name: str) -> list[dict]:
with fitz.open(docs_root() / name) as pdf:
return [{"page_number": index + 1, "text": page.get_text(sort=True)} for index, page in enumerate(pdf)]
def add_project(db, number: str):
student = Student(student_no=number, name=f"测试学生{number[-2:]}")
db.add(student)
db.flush()
project = Project(course_version_id=1, student_id=student.id, topic="证据治理测试",
source_dir=f"tests/evidence-{number}")
db.add(project)
db.flush()
return project
def ingest_pages(db, project, pages, *, category="stage1", title="报告.pdf", source_hash="hash"):
from wisegrade.rag import ingest
return ingest(db, course_version_id=1, project_id=project.id, submission_file_id=None, title=title,
category=category, source_hash=source_hash, pages=pages, embedder=FLAT)
def page_types(db, project):
from wisegrade.models import KnowledgeDocument, KnowledgePage
document = db.scalar(select(KnowledgeDocument).where(KnowledgeDocument.project_id == project.id))
rows = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id == document.id)
.order_by(KnowledgePage.page)).all()
return {row.page: row.page_type for row in rows}
# --------------------------------------------------------------------------- #
# 1. Stage report 1: contents pages versus the literature review body #
# --------------------------------------------------------------------------- #
def test_stage1_contents_is_separated_from_literature_body(db):
project = add_project(db, "123460001")
pages = report_pages("阶段报告1.pdf")
ingest_pages(db, project, pages)
types = page_types(db, project)
# The report's contents spans pages 2 and 3; the literature review is pages 6-7.
assert types[1] == "cover"
assert types[2] == types[3] == "toc"
assert types[6] == types[7] == "body"
assert types[22] == "references"
criterion = {"criterion_id": "stage1.literature", "category": "stage1", "max_score": 20,
"description": "文献调研和技术现状总结", "required_evidence": ["参考文献", "技术现状"]}
targets = normalize_targets(criterion["required_evidence"])
selection = retrieve_targets(db, course_version_id=1, project_id=project.id, targets=targets,
category="stage1", embedder=FLAT)
pages_hit = {chunk["page"] for chunk in selection["evidence"]}
# The literature analysis really exists on pages 6-7 and must be recalled there.
assert pages_hit & {6, 7}, pages_hit
content_types = {chunk["page_type"] for chunk in selection["evidence"]
if chunk["target_use"] == "content"}
assert content_types <= {"body", "appendix"}, content_types
# The structured target may cite the reference list, and it is labelled as such.
structure = [chunk for chunk in selection["evidence"] if chunk["target_use"] == "structure"]
assert all(chunk["page_type"] in ("toc", "references", "body", "cover", "abstract")
for chunk in structure)
def test_stage1_literature_analysis_is_not_taken_from_the_contents_page():
pages = report_pages("阶段报告1.pdf")
classified = {item["page"]: item for item in classify_document(pages)}
toc_text = "\n".join(block["text"] for block in classified[2]["blocks"] if block["kind"] == "toc")
assert "文献调研与技术现状" in toc_text # the heading really is in the contents
body_text = "\n".join(block["text"] for block in classified[6]["blocks"] if block["kind"] == "body")
assert "PID 控制因其结构简单" in body_text # the analysis really is in the body
# The contents page carries no body block, so it cannot supply content evidence.
assert not any(block["kind"] == "body" and block.get("substantive")
for block in classified[2]["blocks"])
# --------------------------------------------------------------------------- #
# 2. Stage report 3: contents versus the testing body #
# --------------------------------------------------------------------------- #
def test_stage3_contents_is_separated_from_testing_body(db):
project = add_project(db, "123460002")
ingest_pages(db, project, report_pages("阶段报告3.pdf"), category="stage3")
types = page_types(db, project)
assert types[2] == types[3] == "toc"
assert types[11] == types[12] == "body"
criterion = {"criterion_id": "stage3.data", "category": "stage3", "max_score": 20,
"description": "实验数据和初步分析", "required_evidence": ["实验数据", "分析"]}
selection = retrieve_targets(db, course_version_id=1, project_id=project.id,
targets=normalize_targets(criterion["required_evidence"]),
category="stage3", embedder=FLAT)
hits = [(chunk["page"], chunk["page_type"], chunk["text"][:40]) for chunk in selection["evidence"]]
pages_hit = {chunk["page"] for chunk in selection["evidence"]}
# The testing description on pages 11-12 must be reachable, not the contents.
assert pages_hit & {11, 12}, hits
assert all(chunk["page_type"] in ("body", "appendix") for chunk in selection["evidence"]), hits
# Finding the page is not a score: the evidence only has to be reachable.
assert any("测试" in chunk["text"] or "试验" in chunk["text"] for chunk in selection["evidence"])
# --------------------------------------------------------------------------- #
# 3. Final report: abstract, cross-page contents, body #
# --------------------------------------------------------------------------- #
def test_final_report_abstract_cross_page_contents_and_body(db):
project = add_project(db, "123460003")
ingest_pages(db, project, report_pages("自动化专业综合实践报告.pdf"), category="final_report")
types = page_types(db, project)
assert types[1] == "cover"
assert types[2] == types[3] == "abstract"
# The contents continues onto a second page (roman numeral IV) and is still contents.
assert types[4] == types[5] == "toc"
assert types[6] == "body"
assert types[49] == "references"
# Appendix pages stay usable for content, and are labelled as appendices.
assert types[50] == "appendix"
def test_contents_page_still_supports_chapter_completeness():
"""A structural criterion keeps using the contents page; that is its purpose."""
pages = report_pages("自动化专业综合实践报告.pdf")
classified = {item["page"]: item for item in classify_document(pages)}
from wisegrade.pages import toc_entries
entries = toc_entries(classified[4]["blocks"])
assert len(entries) >= 20
assert any("第1 章" in entry or "第1章" in entry for entry in entries)
criterion = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["标题", "章节", "参考文献"]}
targets = normalize_targets(criterion["required_evidence"])
assert all(target["use"] == "structure" for target in targets)
# A structure target admits the contents page; a content target never does.
assert all("toc" in target["allowed_page_types"] for target in targets)
# --------------------------------------------------------------------------- #
# 4. Mixed pages, continuations, repeated headers, unsettled pages #
# --------------------------------------------------------------------------- #
def test_mixed_page_contents_continuation_header_and_uncertain_page():
pages = [
{"page_number": 1, "text": "自动化专业综合实践\n阶段性报告\n姓名:张三\n学号:1234567890"},
# A contents page that also carries the start of the first chapter.
{"page_number": 2, "text": "目录\n第1 章 项目背景.......... 1\n第2 章 需求分析.......... 3\n"
"第1 章 项目背景\n本项目面向智能检测场景,通过多传感器融合实现目标识别。"},
# Contents continued: dotted entries and nothing else.
{"page_number": 3, "text": "3.2 性能测试.......... 9\n3.3 结果分析.......... 11\n参考文献.......... 15"},
# A repeated header plus real prose.
{"page_number": 4, "text": "自动化专业综合实践\n2.1 需求分析\n"
"系统需要完成目标检测、定位与结果输出,并在标准场地内稳定运行。\n"
"自动化专业综合实践"},
# A section title with no prose at all.
{"page_number": 5, "text": "自动化专业综合实践\n3.1 功能测试"},
]
classified = {item["page"]: item for item in classify_document(pages)}
# The repeated header is stripped from indexed text.
from wisegrade.pages import clean_page_lines, strip_repeated_lines
repeated = strip_repeated_lines(pages)
assert "自动化专业综合实践" in repeated
assert "自动化专业综合实践" not in clean_page_lines(pages[3], repeated)
assert classified[1]["page_type"] == "cover"
assert classified[2]["page_type"] == "toc" and classified[2]["mixed"] is True
# A mixed page keeps a usable body block instead of dropping the whole page.
body_blocks = [b for b in classified[2]["blocks"] if b["kind"] == "body" and b.get("substantive")]
assert body_blocks and "多传感器融合" in body_blocks[0]["text"]
# The contents continuation is recognised as contents, not body.
assert classified[3]["page_type"] == "toc"
assert classified[4]["page_type"] == "body"
# A page with only a section title is not silently ordinary body text.
assert classified[5]["page_type"] == "unknown"
def test_mixed_page_body_block_is_indexed_as_body_not_navigation(db):
project = add_project(db, "123460004")
pages = [{"page_number": 2, "text": "目录\n第1 章 项目背景.......... 1\n"
"第1 章 项目背景\n本项目面向智能检测场景,通过多传感器融合实现目标识别。"}]
ingest_pages(db, project, pages)
chunk = db.scalar(select(KnowledgeChunk).where(KnowledgeChunk.document_id.isnot(None),
KnowledgeChunk.page == 2,
KnowledgeChunk.block_type == "body"))
# The body half of the mixed page survives as its own searchable chunk.
assert chunk is not None and "多传感器融合" in chunk.text
navigation = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.page == 2,
KnowledgeChunk.block_type == "toc")).all()
assert navigation and all("多传感器融合" not in row.text for row in navigation)
def test_uncertain_page_is_not_offered_as_body_content(db):
project = add_project(db, "123460005")
pages = [{"page_number": 1, "text": "封面"},
{"page_number": 2, "text": "第1 章 项目背景与需求分析"},
{"page_number": 3, "text": "第1 章 项目背景与需求分析\n系统面向智能检测场景,完成目标识别与定位。"}]
ingest_pages(db, project, pages)
types = page_types(db, project)
assert types[2] == "unknown"
body_chunks = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.page_type == "body")).all()
assert all(row.page != 2 for row in body_chunks)
# --------------------------------------------------------------------------- #
# 5. Content judgements cannot rest on navigation pages #
# --------------------------------------------------------------------------- #
def test_content_judgement_rejects_navigation_sources_but_structure_accepts_them():
criterion = {"criterion_id": "stage3.data", "max_score": 20, "description": "实验数据和初步分析",
"required_evidence": ["实验数据", "分析"]}
toc = {"chunk_id": 1, "page": 2, "page_type": "toc", "text": "第3 章 系统测试与验证 ......... 8"}
result = ScoreResult(criterion_id="stage3.data", score=18, max_score=20, reason="看目录",
evidence=[{"chunk_id": 1, "page": 2, "quote": "系统测试与验证", "target_id": "t0"}],
confidence=.9)
with pytest.raises(ModelError):
validate_score(result, criterion, [toc])
structure = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["章节"]}
ok = ScoreResult(criterion_id="stage1.format", score=4, max_score=5, reason="章节完整",
evidence=[{"chunk_id": 1, "page": 2, "quote": "第3 章 系统测试与验证", "target_id": "t0"}],
confidence=.9)
assert validate_score(ok, structure, [toc]).score == 4
# --------------------------------------------------------------------------- #
# 6. Insufficient body evidence becomes a teacher review, never a low score #
# --------------------------------------------------------------------------- #
def test_insufficient_body_evidence_goes_to_review_with_a_null_model_score(db):
from types import SimpleNamespace
from wisegrade import services
from wisegrade.models import ScoringBatch, SubmissionFile
project = add_project(db, "123460006")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
batch = db.get(ScoringBatch, created["id"])
criterion = next(item for item in batch.rubric_snapshot["criteria"]
if item["criterion_id"] == "stage3.data")
file = SubmissionFile(project_id=project.id, category="stage3", relative_path="test/review/report.pdf",
mime_type="application/pdf", size=1, mtime=1, sha256="c" * 64,
recognition_status="recognized", parse_status="ready")
db.add(file)
db.flush()
# The material only contains a cover page and a contents page: no body at all.
pages = [{"page_number": 1, "text": "阶段性报告\n姓名:张三\n学号:1234567890"},
{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8\n第4 章 结论.......... 12"}]
file.parsed = {"pages": pages}
ingest_pages(db, project, pages, category="stage3", title="report.pdf", source_hash="c" * 64)
class NeverCalled:
def json_call(self, *_, **__):
raise AssertionError("缺少正文证据时不应调用评分模型")
scored = _score_criterion(db, batch, project, criterion, NeverCalled(),
retriever=lambda *_, **kw: [] if kw.get("public_only") else retrieve(
*_, **{**kw, "embedder": FLAT}))
assert scored.status == "needs_review"
assert scored.model_score is None
assert scored.evidence_gaps and scored.evidence_gaps[0]["target_label"]
assert scored.evidence_gaps[0]["searched_page_types"]
# The criterion shows as awaiting review, and the batch is not reported as complete.
counts = {row.status for row in db.scalars(select(CriterionScore).where(
CriterionScore.batch_id == batch.id)).all()}
assert "needs_review" in counts
def test_teacher_can_review_a_needs_review_item_and_the_gap_record_survives(db):
from types import SimpleNamespace
from wisegrade import services
project = add_project(db, "123460007")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
score = CriterionScore(batch_id=created["id"], project_id=project.id, criterion_id="stage3.data",
model_score=None, status="needs_review",
evidence_gaps=[{"target_id": "t0", "target_label": "实验数据",
"searched_page_types": ["body", "appendix"],
"reason": "补检后仍未找到正文证据"}])
db.add(score)
db.commit()
result = services.review_score(score.id, SimpleNamespace(score=17, comment="教师核对原始材料",
reason="材料确实存在测试章节"), db)
assert result["final_score"] == 17 and result["model_score"] is None
reviewed = db.get(CriterionScore, score.id)
assert reviewed.status == "reviewed"
# The original retrieval problem stays on the record after the teacher decides.
assert reviewed.evidence_gaps[0]["target_label"] == "实验数据"
# --------------------------------------------------------------------------- #
# 7. Agent citations only carry sources that were actually used #
# --------------------------------------------------------------------------- #
def test_search_tools_label_page_types_and_mark_historical_citations(db):
from wisegrade import agent_tools
from wisegrade.models import Conversation
project = add_project(db, "123460008")
ingest_pages(db, project, [
{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8"},
{"page_number": 11, "text": "第3 章 系统测试与验证\n在自主巡线测试中,底盘记录了直线段与弯道的时间数据。"},
], category="stage3")
conversation = Conversation(title="检索", course_version_id=1, project_id=project.id)
db.add(conversation)
db.commit()
original = agent_tools.retrieve
agent_tools.retrieve = lambda *args, **kwargs: original(*args, **kwargs, embedder=FLAT)
try:
content = agent_tools.execute(db, conversation, "search_project_material",
{"project_id": project.id, "query": "系统测试与验证"}, "")
assert content["found"]
# Content search never returns the contents page as body evidence.
assert all(item["page_type"] in ("body", "appendix") for item in content["citations"])
assert all(item["source_role"] == "content" for item in content["citations"])
assert all(item["citation_id"].startswith("chunk:") for item in content["citations"])
structure = agent_tools.execute(db, conversation, "search_project_structure",
{"project_id": project.id, "query": "系统测试"}, "")
assert structure["found"]
assert all(item["source_role"] == "structure" for item in structure["citations"])
# A structural hit is explicitly declared as unable to prove chapter content.
assert any(item["page_type"] == "toc" for item in structure["citations"])
assert "不能证明章节内容" in structure["note"]
finally:
agent_tools.retrieve = original
def test_scores_tool_marks_historical_score_evidence(db):
from types import SimpleNamespace
from wisegrade import agent_tools, services
from wisegrade.models import Conversation
project = add_project(db, "123460009")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
score = CriterionScore(batch_id=created["id"], project_id=project.id, criterion_id="stage1.format",
model_score=4, status="scored")
db.add(score)
db.flush()
db.add(ScoreEvidence(score_id=score.id, document_id=None, submission_file_id=None, page=2,
quote="第1 章 项目背景与需求分析", image_path=None, page_type="toc",
target_label="章节", historical=True))
conversation = Conversation(title="评分", course_version_id=1, batch_id=created["id"])
db.add(conversation)
db.commit()
result = agent_tools.execute(db, conversation, "get_scores",
{"batch_id": created["id"], "project_id": project.id}, "")
citations = result["citations"]
assert citations and citations[0]["historical"] is True
assert citations[0]["source"] == "historical_score_evidence"
assert citations[0]["usage"] == "历史评分引用"
# --------------------------------------------------------------------------- #
# 8. Isolation and existing behaviour #
# --------------------------------------------------------------------------- #
def test_page_filtering_happens_before_the_candidate_limit(db):
"""A contents page must not crowd a body page out of the candidate window."""
project = add_project(db, "123460010")
pages = [{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8"}]
for number in range(3, 25):
pages.append({"page_number": number,
"text": f"第3 章 系统测试与验证\n第{number} 页记录了系统测试的实测数据与分析过程。"})
ingest_pages(db, project, pages, category="stage3")
unrestricted = retrieve(db, course_version_id=1, project_id=project.id, query="系统测试与验证",
category="stage3", limit=3, embedder=FLAT, allowed_page_types=None)
filtered = retrieve(db, course_version_id=1, project_id=project.id, query="系统测试与验证",
category="stage3", limit=3, embedder=FLAT,
allowed_page_types=["body", "appendix"], substantive_only=True)
assert filtered and all(chunk["page_type"] in ("body", "appendix") for chunk in filtered)
assert {chunk["page"] for chunk in filtered} != {2}
# Without the filter the contents page is a candidate; the filter removes it before
# the limit is applied, so three real body pages still come back.
assert len(filtered) == 3
assert any(chunk["page_type"] == "toc" for chunk in unrestricted)
def test_project_and_version_isolation_is_unchanged(db):
first = add_project(db, "123460011")
second = add_project(db, "123460012")
ingest_pages(db, first, [{"page_number": 3, "text": "第3 章 系统测试与验证\n一号项目记录了巡线测试数据。"}],
category="stage3", title="一号.pdf", source_hash="one")
ingest_pages(db, second, [{"page_number": 3, "text": "第3 章 系统测试与验证\n二号项目的私有测试结果。"}],
category="stage3", title="二号.pdf", source_hash="two")
hits = retrieve(db, course_version_id=1, project_id=first.id, category="stage3",
query="系统测试与验证", embedder=FLAT, allowed_page_types=["body", "appendix"])
assert hits and all(hit["project_id"] == first.id for hit in hits)
assert "私有" not in str(hits)
# Course material stays in its own namespace and is not mixed into project search.
public = retrieve(db, course_version_id=1, project_id=first.id, query="系统测试与验证",
embedder=FLAT, public_only=True)
assert all(hit["project_id"] is None for hit in public)
def test_batch_snapshot_records_the_evidence_policy(db):
from types import SimpleNamespace
from wisegrade import services
from wisegrade.models import ScoringBatch
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
assert created["id"]
batch = db.get(ScoringBatch, created["id"])
plan = batch.rubric_snapshot["evidence_plan"]
assert plan["version"] == 1
assert plan["content_allowed_page_types"] == ["body", "appendix"]
targets = plan["criteria"]["stage3.data"]
assert [target["use"] for target in targets] == ["content", "content"]
assert all("toc" not in target["allowed_page_types"] for target in targets)
# The legacy string form is read into typed targets without being rewritten.
assert [item["required_evidence"] for item in batch.rubric_snapshot["criteria"]
if item["criterion_id"] == "stage3.data"] == [["实验数据", "分析"]]
+1 -1
View File
@@ -10,7 +10,7 @@ from wisegrade.storage import classify, extract_identity, parse_student_folder,
def test_postgresql_migrations_and_vector_extension(db): def test_postgresql_migrations_and_vector_extension(db):
assert db.bind.dialect.name == "postgresql" assert db.bind.dialect.name == "postgresql"
assert db.scalar(text("SELECT version_num FROM alembic_version")) == "0005" assert db.scalar(text("SELECT version_num FROM alembic_version")) == "0006"
assert db.scalar(text("SELECT extname FROM pg_extension WHERE extname = 'vector'")) == "vector" assert db.scalar(text("SELECT extname FROM pg_extension WHERE extname = 'vector'")) == "vector"
+58 -5
View File
@@ -4,7 +4,7 @@ from sqlalchemy import select
from wisegrade.llm import DeepSeekClient, FactResult, ModelError, ScoreResult, validate_score from wisegrade.llm import DeepSeekClient, FactResult, ModelError, ScoreResult, validate_score
from wisegrade.models import AppSetting, KnowledgeChunk, KnowledgePage, Project, Student from wisegrade.models import AppSetting, KnowledgeChunk, KnowledgePage, Project, Student
from wisegrade.rag import ingest, local_encoder, retrieve, split_pages from wisegrade.rag import INDEX_VERSION, ingest, local_encoder, retrieve, split_pages
from wisegrade.settings_store import model_config, save_model_config from wisegrade.settings_store import model_config, save_model_config
@@ -21,7 +21,7 @@ def make_project(db, number):
def test_json_validation_and_evidence(): def test_json_validation_and_evidence():
criterion = {"criterion_id": "stage1.problem", "max_score": 25} criterion = {"criterion_id": "stage1.problem", "max_score": 25}
chunks = [{"chunk_id": 7, "page": 2, "text": "工程需求是实时目标识别"}] chunks = [{"chunk_id": 7, "page": 2, "text": "工程需求是实时目标识别", "page_type": "body"}]
result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="满足需求", result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="满足需求",
evidence=[{"chunk_id": 7, "page": 2, "quote": "实时目标识别"}], confidence=.8) evidence=[{"chunk_id": 7, "page": 2, "quote": "实时目标识别"}], confidence=.8)
assert validate_score(result, criterion, chunks).score == 19 assert validate_score(result, criterion, chunks).score == 19
@@ -33,8 +33,8 @@ def test_json_validation_and_evidence():
def test_evidence_quote_resolves_pdf_line_breaks_and_page(): def test_evidence_quote_resolves_pdf_line_breaks_and_page():
criterion = {"criterion_id": "stage1.problem", "max_score": 25} criterion = {"criterion_id": "stage1.problem", "max_score": 25}
chunks = [{"chunk_id": 7, "page": 2, "text": "机器人\n控制精度:2 cm。"}, chunks = [{"chunk_id": 7, "page": 2, "text": "机器人\n控制精度:2 cm。", "page_type": "body"},
{"chunk_id": 8, "page": 3, "text": "其他实验结果"}] {"chunk_id": 8, "page": 3, "text": "其他实验结果", "page_type": "body"}]
result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="有指标", result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="有指标",
evidence=[{"chunk_id": 8, "page": 3, "quote": "机器人控制精度:2 cm"}], confidence=.8) evidence=[{"chunk_id": 8, "page": 3, "quote": "机器人控制精度:2 cm"}], confidence=.8)
checked = validate_score(result, criterion, chunks) checked = validate_score(result, criterion, chunks)
@@ -42,6 +42,59 @@ def test_evidence_quote_resolves_pdf_line_breaks_and_page():
assert checked.evidence[0].quote in chunks[0]["text"] assert checked.evidence[0].quote in chunks[0]["text"]
def test_content_evidence_rejects_navigation_and_heading_only_sources():
"""A contents page, an abstract or a bare heading can never prove content."""
criterion = {"criterion_id": "stage3.data", "max_score": 20,
"description": "实验数据和初步分析", "required_evidence": ["实验数据", "分析"]}
body = {"chunk_id": 1, "page": 11, "page_type": "body",
"text": "在自主巡线测试中,底盘在标准黑色轨迹线场地内运行,记录了直线段与弯道的时间数据。"}
good = ScoreResult(criterion_id="stage3.data", score=16, max_score=20, reason="有实测数据",
evidence=[{"chunk_id": 1, "page": 11, "quote": "在自主巡线测试中", "target_id": "t0"}],
confidence=.8)
assert validate_score(good, criterion, [body]).score == 16
toc = {"chunk_id": 2, "page": 2, "page_type": "toc",
"text": "第3 章系统测试与验证 ......... 8"}
bad = ScoreResult(criterion_id="stage3.data", score=16, max_score=20, reason="看目录",
evidence=[{"chunk_id": 2, "page": 2, "quote": "系统测试与验证", "target_id": "t0"}],
confidence=.8)
# The rejection is what matters; either guard may fire depending on the target's
# allowed page types.
with pytest.raises(ModelError, match="内容证据|不允许的页面类型"):
validate_score(bad, criterion, [toc])
abstract = {"chunk_id": 3, "page": 2, "page_type": "abstract",
"text": "摘要 本文实现了自瞄移动机器人系统,采用多处理器分布式架构。"}
bad.evidence[0].chunk_id, bad.evidence[0].page, bad.evidence[0].quote = 3, 2, "本文实现了自瞄移动机器人系统"
with pytest.raises(ModelError, match="内容证据|不允许的页面类型"):
validate_score(bad, criterion, [abstract])
heading_only = {"chunk_id": 4, "page": 11, "page_type": "body", "text": "第3 章 系统测试与验证"}
bad.evidence[0].chunk_id, bad.evidence[0].page, bad.evidence[0].quote = 4, 11, "第3 章 系统测试与验证"
with pytest.raises(ModelError, match="章节标题"):
validate_score(bad, criterion, [heading_only])
def test_structural_targets_may_use_the_contents_page():
"""A structure target legitimately reads the contents list; a content one may not."""
criterion = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["标题", "章节", "参考文献"]}
toc = {"chunk_id": 5, "page": 2, "page_type": "toc",
"text": "第1 章项目背景与需求分析 .......... 1\n第2 章文献调研与技术现状 .......... 3"}
result = ScoreResult(criterion_id="stage1.format", score=4, max_score=5, reason="章节完整",
evidence=[{"chunk_id": 5, "page": 2, "quote": "第1 章项目背景与需求分析", "target_id": "t1"}],
confidence=.8)
checked = validate_score(result, criterion, [toc])
assert checked.evidence[0].page == 2
def test_model_may_return_a_null_score_when_content_evidence_is_missing():
result = ScoreResult(criterion_id="stage3.data", score=None, max_score=20,
reason="仅检索到目录页,无法判断测试内容", evidence=[],
missing_evidence=["实验数据"], confidence=.2)
assert result.score is None
@pytest.mark.integration @pytest.mark.integration
def test_deepseek_adapter_live_with_image(tmp_path): def test_deepseek_adapter_live_with_image(tmp_path):
image = tmp_path / "page.png" image = tmp_path / "page.png"
@@ -87,7 +140,7 @@ def test_stale_knowledge_index_reuses_document_id(db):
db.commit() db.commit()
reindexed = ingest(db, course_version_id=1, project_id=project_id, submission_file_id=None, reindexed = ingest(db, course_version_id=1, project_id=project_id, submission_file_id=None,
title="报告", category="stage3", source_hash="same-source", pages=pages) title="报告", category="stage3", source_hash="same-source", pages=pages)
assert reindexed.id == document.id and reindexed.index_version == 2 assert reindexed.id == document.id and reindexed.index_version == INDEX_VERSION
assert db.scalar(select(KnowledgeChunk.text).where(KnowledgeChunk.document_id == document.id)) == pages[0]["text"] assert db.scalar(select(KnowledgeChunk.text).where(KnowledgeChunk.document_id == document.id)) == pages[0]["text"]
assert db.get(KnowledgePage, (document.id, 1)).text == pages[0]["text"] assert db.get(KnowledgePage, (document.id, 1)).text == pages[0]["text"]
+1 -1
View File
@@ -210,7 +210,7 @@ def prepare(db, conversation: Conversation, kind: str, payload: dict, *, teacher
else: else:
raise HTTPException(400, "不支持的操作") raise HTTPException(400, "不支持的操作")
expected = _state(db, kind, payload) expected = _state(db, kind, payload)
if kind == "start_batch" and expected["status"] not in ("created", "failed", "partial"): if kind == "start_batch" and expected["status"] not in ("created", "failed", "partial", "review_required"):
raise HTTPException(409, "批次不可启动或重试") raise HTTPException(409, "批次不可启动或重试")
if kind == "index_retry" and expected["status"] != "failed": if kind == "index_retry" and expected["status"] != "failed":
raise HTTPException(409, "仅失败任务可重试") raise HTTPException(409, "仅失败任务可重试")
+134 -10
View File
@@ -8,10 +8,12 @@ from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select from sqlalchemy import select
from .actions import prepare, require_scope from .actions import prepare, require_scope
from .exports import batch_projects, project_result from .evidence import CONTENT_ALLOWED, STRUCTURE_ALLOWED
from .exports import STATUS_LABELS, batch_projects, page_type_label, project_result
from .knowledge_pages import page_links from .knowledge_pages import page_links
from .models import (Conversation, Course, CourseVersion, IndexTask, KnowledgeDocument, from .models import (Conversation, Course, CourseVersion, IndexTask, KnowledgeChunk, KnowledgeDocument,
Project, RubricVersion, ScoringBatch, Student, SubmissionFile) Project, RubricVersion, ScoringBatch, Student, SubmissionFile)
from .pages import toc_entries
from .rag import retrieve from .rag import retrieve
from .rules import totals from .rules import totals
from .storage import directories, scan from .storage import directories, scan
@@ -52,6 +54,14 @@ class SearchProject(ProjectId):
query: str = Field(min_length=2, max_length=300) query: str = Field(min_length=2, max_length=300)
class DocumentId(Args):
document_id: int = Field(gt=0)
class SearchDocument(DocumentId):
query: str = Field(min_length=2, max_length=300)
class Directory(Args): class Directory(Args):
relative_dir: str = Field(min_length=1, max_length=500) relative_dir: str = Field(min_length=1, max_length=500)
@@ -67,7 +77,7 @@ class BatchId(Args):
class ScoresFilterArgs(BatchId): class ScoresFilterArgs(BatchId):
project_id: int | None = Field(default=None, gt=0) project_id: int | None = Field(default=None, gt=0)
status: Literal["pending", "scored", "reviewed", "failed", "unscorable"] | None = None status: Literal["pending", "scored", "reviewed", "needs_review", "failed", "unscorable"] | None = None
class ScoresArgs(ScoresFilterArgs): class ScoresArgs(ScoresFilterArgs):
@@ -201,9 +211,14 @@ def project(db, convo, args, _user):
"url": f"/?page=projects&project_id={item.id}"} "url": f"/?page=projects&project_id={item.id}"}
def _search(db, version_id, project_id, query, *, public_only=False): def _search(db, version_id, project_id, query, *, public_only=False, scopes=None, limit=6):
"""Content search under the evidence policy: navigation pages are never body text."""
scopes = scopes or ("content",)
allowed = sorted({page_type for scope in scopes for page_type in STRUCTURE_ALLOWED} if "structure" in scopes
else {page_type for scope in scopes for page_type in CONTENT_ALLOWED})
chunks = retrieve(db, course_version_id=version_id, project_id=project_id or 0, query=query, chunks = retrieve(db, course_version_id=version_id, project_id=project_id or 0, query=query,
limit=6, include_public=False, public_only=public_only) limit=limit, include_public=False, public_only=public_only,
allowed_page_types=allowed, substantive_only="content" in scopes)
citations = [] citations = []
for chunk in chunks: for chunk in chunks:
doc = db.get(KnowledgeDocument, chunk["document_id"]) doc = db.get(KnowledgeDocument, chunk["document_id"])
@@ -211,8 +226,15 @@ def _search(db, version_id, project_id, query, *, public_only=False):
citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"], citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": doc.title if doc else "资料", "page": chunk["page"], "title": doc.title if doc else "资料", "page": chunk["page"],
"quote": chunk["text"][:500], "project_id": chunk["project_id"], "quote": chunk["text"][:500], "project_id": chunk["project_id"],
"page_type": chunk.get("page_type") or links.get("page_type") or "unknown",
"page_type_label": page_type_label(chunk.get("page_type")),
"section": chunk.get("section") or "",
"block_type": chunk.get("block_type") or "body",
"source_role": "structure" if "structure" in scopes else "content",
# Stable identity so an answer can cite exactly what it used.
"citation_id": f"chunk:{chunk['chunk_id']}",
**links}) **links})
return {"citations": citations, "found": bool(citations)} return {"citations": citations, "found": bool(citations), "scopes": list(scopes)}
def search_course(db, convo, args, _user): def search_course(db, convo, args, _user):
@@ -225,6 +247,84 @@ def search_project(db, convo, args, _user):
return _search(db, item.course_version_id, item.id, args.query) return _search(db, item.course_version_id, item.id, args.query)
def search_structure(db, convo, args, _user):
"""Structure and contents lookup: navigation pages plus body headings.
This tool answers questions about chapters, sections and report layout. Its results
are explicitly labelled as structure, so they can never be mistaken for content
evidence about what the student implemented or measured.
"""
item = _project(db, convo, args.project_id)
chunks = retrieve(db, course_version_id=item.course_version_id, project_id=item.id, query=args.query,
limit=8, include_public=False, allowed_page_types=sorted(STRUCTURE_ALLOWED),
substantive_only=False)
citations = []
for chunk in chunks:
doc = db.get(KnowledgeDocument, chunk["document_id"])
links = page_links(db, chunk["document_id"], chunk["page"])
page_type = chunk.get("page_type") or links.get("page_type") or "unknown"
citations.append({
"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": doc.title if doc else "资料", "page": chunk["page"],
"quote": chunk["text"][:500], "project_id": chunk["project_id"],
"page_type": page_type, "page_type_label": page_type_label(page_type),
"section": chunk.get("section") or "", "block_type": chunk.get("block_type") or "body",
"source_role": "structure",
"citation_id": f"chunk:{chunk['chunk_id']}",
"toc_entries": toc_entries([{"kind": "toc", "text": chunk["text"]}])[:40]
if page_type == "toc" else [],
**links})
return {"citations": citations, "found": bool(citations), "scopes": ["structure"],
"note": "这些是结构或目录信息,只能说明章节安排,不能证明章节内容。"}
def get_document_pages(db, convo, args, _user):
"""List a document's pages with page type, so a caller can navigate precisely."""
_document(db, convo, args.document_id)
rows = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.document_id == args.document_id)
.order_by(KnowledgeChunk.page, KnowledgeChunk.id)).all()
pages: dict[int, dict] = {}
for chunk in rows:
entry = pages.setdefault(chunk.page, {"page": chunk.page, "page_type": chunk.page_type,
"page_type_label": page_type_label(chunk.page_type),
"chapter": chunk.chapter, "section": chunk.section,
"block_types": []})
if chunk.block_type not in entry["block_types"]:
entry["block_types"].append(chunk.block_type)
if chunk.section:
entry["section"] = chunk.section
return {"document_id": args.document_id, "count": len(pages),
"pages": [pages[key] for key in sorted(pages)]}
def search_document(db, convo, args, _user):
"""Content search restricted to one document, under the same page-type policy."""
document = _document(db, convo, args.document_id)
chunks = retrieve(db, course_version_id=document.course_version_id, project_id=document.project_id or 0,
query=args.query, limit=6, include_public=False,
allowed_page_types=sorted(CONTENT_ALLOWED), substantive_only=True)
citations = []
for chunk in chunks:
if chunk["document_id"] != document.id:
continue
links = page_links(db, chunk["document_id"], chunk["page"])
citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": document.title, "page": chunk["page"], "quote": chunk["text"][:500],
"project_id": chunk["project_id"], "page_type": chunk.get("page_type"),
"page_type_label": page_type_label(chunk.get("page_type")),
"section": chunk.get("section") or "", "source_role": "content",
"citation_id": f"chunk:{chunk['chunk_id']}", **links})
return {"citations": citations, "found": bool(citations), "scopes": ["content"]}
def _document(db, convo, document_id):
document = db.get(KnowledgeDocument, document_id)
if not document:
raise HTTPException(404, "资料不存在")
require_scope(db, convo, version_id=document.course_version_id, project_id=document.project_id)
return document
def batch_list(db, convo, args, _user): def batch_list(db, convo, args, _user):
query = select(ScoringBatch).order_by(ScoringBatch.id.desc()) query = select(ScoringBatch).order_by(ScoringBatch.id.desc())
rows = [] rows = []
@@ -278,15 +378,36 @@ def scores(db, convo, args, _user):
"max_score": row["criterion"]["max_score"], "score_id": row["score_id"], "max_score": row["criterion"]["max_score"], "score_id": row["score_id"],
"model_score": row["model_score"], "teacher_score": row["teacher_score"], "model_score": row["model_score"], "teacher_score": row["teacher_score"],
"final_score": row["final_score"], "status": row["status"], "final_score": row["final_score"], "status": row["status"],
"error": str(row["error"] or "")[:300], "evidence": [{"page": ev["page"], "status_label": STATUS_LABELS.get(row["status"], row["status"]),
"quote": ev["quote"][:160], "document": str(ev["document"])[:200]} "error": str(row["error"] or "")[:300],
# Which body targets still lack evidence, so the answer can
# explain a review item instead of reporting a score.
"evidence_gaps": [{"target_label": str(gap.get("target_label") or "")[:120],
"searched_page_types": gap.get("searched_page_types") or []}
for gap in (row.get("evidence_gaps") or [])][:6],
"evidence": [{"page": ev["page"],
"quote": ev["quote"][:160], "document": str(ev["document"])[:200],
"page_type": ev.get("page_type") or "unknown",
"page_type_label": page_type_label(ev.get("page_type")),
"target_label": str(ev.get("target_label") or "")[:120],
"historical": bool(ev.get("historical"))}
for ev in row["evidence"][:2]]} for row in rows]}) for ev in row["evidence"][:2]]} for row in rows]})
for row in rows: for row in rows:
for evidence in row["evidence"][:2]: for evidence in row["evidence"][:2]:
links = page_links(db, evidence.get("document_id"), evidence["page"]) links = page_links(db, evidence.get("document_id"), evidence["page"])
citations.append({"title": str(evidence["document"])[:200], "page": evidence["page"], citations.append({"title": str(evidence["document"])[:200], "page": evidence["page"],
"quote": evidence["quote"][:320], "document_id": evidence.get("document_id"), "quote": evidence["quote"][:320], "document_id": evidence.get("document_id"),
"page_type": evidence.get("page_type") or links.get("page_type") or "unknown",
"page_type_label": page_type_label(evidence.get("page_type")),
"target_label": str(evidence.get("target_label") or "")[:120],
"historical": bool(evidence.get("historical")),
"source": "historical_score_evidence" if evidence.get("historical") else "score_evidence",
**links, "score_url": url}) **links, "score_url": url})
if citations:
# Scoring quotations are never presented as report body evidence in an answer.
for citation in citations:
citation["usage"] = ("历史评分引用" if citation.get("historical")
else f"评分证据 · {citation.get('page_type_label')}")
next_criterion_offset = args.criterion_offset + 6 if page and args.criterion_offset + 6 < len(page[0][1]["rows"]) else None next_criterion_offset = args.criterion_offset + 6 if page and args.criterion_offset + 6 < len(page[0][1]["rows"]) else None
next_project_offset = args.project_offset + 1 if args.project_offset + 1 < len(results) else None next_project_offset = args.project_offset + 1 if args.project_offset + 1 < len(results) else None
return {"batch_id": batch.id, "status_filter": args.status, "count": len(results), return {"batch_id": batch.id, "status_filter": args.status, "count": len(results),
@@ -418,12 +539,15 @@ def prepare_import(db, convo, args, _user):
TOOLS: dict[str, Tool] = { TOOLS: dict[str, Tool] = {
"list_courses": Tool("列出课程、版本和发布状态;名称或版本不清楚时先调用此工具。", Empty, courses), "list_courses": Tool("列出课程、版本和发布状态;名称或版本不清楚时先调用此工具。", Empty, courses),
"get_course_rules": Tool("查询课程目标和规则。指定批次时只返回批次保存的快照。", RulesArgs, rules), "get_course_rules": Tool("查询课程目标和规则。指定批次时只返回批次保存的快照。", RulesArgs, rules),
"search_course_material": Tool("检索指定课程版本的课程资料,返回原文片段和引用。", SearchCourse, search_course), "search_course_material": Tool("检索指定课程版本的课程资料正文,返回原文片段和引用;不含目录、摘要和参考文献页。", SearchCourse, search_course),
"list_directories": Tool("只读浏览 /data/inbox 下的相对目录。", Directory, browse_directories), "list_directories": Tool("只读浏览 /data/inbox 下的相对目录。", Directory, browse_directories),
"scan_directory": Tool("只读分页扫描学生总目录,返回材料候选、缺失、重复和识别异常。", ScanArgs, scan_directories), "scan_directory": Tool("只读分页扫描学生总目录,返回材料候选、缺失、重复和识别异常。", ScanArgs, scan_directories),
"list_projects": Tool("按课程版本及可选学号查询学生项目。", ProjectList, projects), "list_projects": Tool("按课程版本及可选学号查询学生项目。", ProjectList, projects),
"get_project": Tool("读取单个学生项目的元数据、材料和解析状态。", ProjectId, project), "get_project": Tool("读取单个学生项目的元数据、材料和解析状态。", ProjectId, project),
"search_project_material": Tool("只检索指定学生项目报告,返回原文片段和引用。", SearchProject, search_project), "search_project_material": Tool("只检索指定学生项目的报告正文,返回带页面类型和稳定引用标识的原文片段;不含目录、摘要和参考文献页。", SearchProject, search_project),
"search_project_structure": Tool("查询学生报告的章节结构与目录,返回目录条目和页面类型;只能说明章节安排,不能证明章节内容。", SearchProject, search_structure),
"get_document_pages": Tool("列出某份资料各页的页码、页面类型和章节,用于判断正文页位置。", DocumentId, get_document_pages),
"search_document": Tool("在指定的一份资料内检索正文段落,返回带页面类型和引用标识的片段。", SearchDocument, search_document),
"list_batches": Tool("列出可访问的评分批次。", Empty, batch_list), "list_batches": Tool("列出可访问的评分批次。", Empty, batch_list),
"get_batch_progress": Tool("查询评分批次进度、状态和固定项目范围。", BatchId, progress), "get_batch_progress": Tool("查询评分批次进度、状态和固定项目范围。", BatchId, progress),
"get_scores": Tool("分页查询单学生或单批次评分详情,可按状态筛选。每次返回一个项目、最多六项评分;如需其余评分项,用 next_criterion_offset 作为 criterion_offset 再查;如需下个项目,用 next_project_offset 作为 project_offset,并将 criterion_offset 设为 0。", ScoresArgs, scores), "get_scores": Tool("分页查询单学生或单批次评分详情,可按状态筛选。每次返回一个项目、最多六项评分;如需其余评分项,用 next_criterion_offset 作为 criterion_offset 再查;如需下个项目,用 next_project_offset 作为 project_offset,并将 criterion_offset 设为 0。", ScoresArgs, scores),
+75 -4
View File
@@ -10,13 +10,16 @@ from sqlalchemy.orm import Session
from . import services from . import services
from .config import get_settings from .config import get_settings
from .db import get_db from .db import get_db
from .exports import batch_projects, export_excel, export_project_pdf, project_result from .evidence import CONTENT_ALLOWED
from .exports import (BATCH_STATUS_LABELS, batch_projects, export_excel,
export_project_pdf, page_type_label, project_result)
from .knowledge_pages import page_image_path, page_links from .knowledge_pages import page_image_path, page_links
from .models import (AuditLog, Course, CourseVersion, CriterionScore, ExportTask, from .models import (AuditLog, Course, CourseVersion, CriterionScore, ExportTask,
IndexTask, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project, RubricVersion, ScoringBatch, IndexTask, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project, RubricVersion, ScoringBatch,
Student, SubmissionFile, TeacherRevision) Student, SubmissionFile, TeacherRevision)
from .materialized import sync_course_entities, sync_rubric_entities from .materialized import sync_course_entities, sync_rubric_entities
from .rules import RubricData from .rules import RubricData
from .scoring import ATTENTION_STATUSES
from .settings_store import key_configured, model_config, save_model_config from .settings_store import key_configured, model_config, save_model_config
from .storage import CATEGORIES, directories, scan from .storage import CATEGORIES, directories, scan
@@ -108,9 +111,20 @@ class ModelSettingsInput(BaseModel):
@router.get("/dashboard") @router.get("/dashboard")
def dashboard(db: Session = Depends(get_db)): def dashboard(db: Session = Depends(get_db)):
"""Overview counts. Items awaiting teacher review are counted as attention needed.
A ``needs_review`` item is not a failure: retrieval ran and found only navigation
pages where the rule requires body content, so a teacher decides it.
"""
counts = dict(db.execute(select(CriterionScore.status, func.count(CriterionScore.id))
.group_by(CriterionScore.status)).all())
return {"courses": db.scalar(select(func.count(Course.id))), "projects": db.scalar(select(func.count(Project.id))), return {"courses": db.scalar(select(func.count(Course.id))), "projects": db.scalar(select(func.count(Project.id))),
"batches": db.scalar(select(func.count(ScoringBatch.id))), "batches": db.scalar(select(func.count(ScoringBatch.id))),
"needs_attention": db.scalar(select(func.count(CriterionScore.id)).where(CriterionScore.status.in_(["failed", "unscorable"]))) } "needs_attention": sum(counts.get(status, 0) for status in ATTENTION_STATUSES),
"needs_review": counts.get("needs_review", 0),
"unscorable": counts.get("unscorable", 0),
"failed": counts.get("failed", 0),
"status_counts": counts}
@router.get("/courses") @router.get("/courses")
@@ -337,9 +351,32 @@ def knowledge_page(document_id: int, page: int, db: Session = Depends(get_db)):
return {"document_id": doc.id, "title": doc.title, "course_version_id": doc.course_version_id, return {"document_id": doc.id, "title": doc.title, "course_version_id": doc.course_version_id,
"project_id": doc.project_id, "page": page, "project_id": doc.project_id, "page": page,
"text": record.text, "text": record.text,
"page_type": record.page_type or "unknown",
"page_type_label": page_type_label(record.page_type),
"mixed": bool(record.mixed),
"section": record.section or "", "chapter": record.chapter or "",
"confidence": record.confidence,
"classification": record.classification or {},
"content_allowed": (record.page_type or "unknown") in CONTENT_ALLOWED,
"image_url": page_links(db, document_id, page)["image_url"]} "image_url": page_links(db, document_id, page)["image_url"]}
@router.get("/knowledge/documents/{document_id}/pages")
def knowledge_document_pages(document_id: int, db: Session = Depends(get_db)):
"""Page inventory with types, so the teacher can see how a document was classified."""
doc = found(db, KnowledgeDocument, document_id)
records = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id == document_id)
.order_by(KnowledgePage.page)).all()
return {"document_id": doc.id, "title": doc.title, "index_version": doc.index_version,
"pages": [{"page": item.page, "page_type": item.page_type or "unknown",
"page_type_label": page_type_label(item.page_type),
"mixed": bool(item.mixed), "section": item.section or "",
"chapter": item.chapter or "", "confidence": item.confidence,
"classification": item.classification or {},
"content_allowed": (item.page_type or "unknown") in CONTENT_ALLOWED}
for item in records]}
@router.get("/knowledge/documents/{document_id}/pages/{page}/image") @router.get("/knowledge/documents/{document_id}/pages/{page}/image")
def knowledge_page_image(document_id: int, page: int, db: Session = Depends(get_db)): def knowledge_page_image(document_id: int, page: int, db: Session = Depends(get_db)):
doc = found(db, KnowledgeDocument, document_id) doc = found(db, KnowledgeDocument, document_id)
@@ -354,6 +391,28 @@ def ingest_course(course_version_id: int, db: Session = Depends(get_db)):
return services.ingest_course(course_version_id, db) return services.ingest_course(course_version_id, db)
@router.get("/knowledge/stale/{course_version_id}")
def stale_knowledge(course_version_id: int, db: Session = Depends(get_db)):
"""Documents still on an older index; they are excluded from retrieval until rebuilt."""
from .indexing import stale_documents
from .rag import INDEX_VERSION
found(db, CourseVersion, course_version_id)
documents = stale_documents(db, course_version_id)
return {"index_version": INDEX_VERSION,
"documents": [{"id": item.id, "title": item.title, "type": item.document_type,
"project_id": item.project_id, "index_version": item.index_version,
"status": item.status} for item in documents]}
@router.post("/knowledge/reindex/{course_version_id}")
def reindex_knowledge(course_version_id: int, db: Session = Depends(get_db)):
"""Queue a rebuild of every stale index for this course version."""
from .indexing import create_reindex_task
found(db, CourseVersion, course_version_id)
task = create_reindex_task(db, course_version_id)
return get_index_task(task.id, db)
@router.get("/index-tasks") @router.get("/index-tasks")
def list_index_tasks(project_id: int | None = None, db: Session = Depends(get_db)): def list_index_tasks(project_id: int | None = None, db: Session = Depends(get_db)):
query = select(IndexTask).order_by(IndexTask.id.desc()).limit(100) query = select(IndexTask).order_by(IndexTask.id.desc()).limit(100)
@@ -381,10 +440,20 @@ def create_batch(data: BatchInput, db: Session = Depends(get_db)):
return services.create_batch(data, db) return services.create_batch(data, db)
def batch_status_counts(db, batch_id: int) -> dict:
rows = db.execute(select(CriterionScore.status, func.count(CriterionScore.id))
.where(CriterionScore.batch_id == batch_id).group_by(CriterionScore.status)).all()
counts = {status: count for status, count in rows}
return {"status_counts": counts, "needs_review": counts.get("needs_review", 0),
"unscorable": counts.get("unscorable", 0), "failed": counts.get("failed", 0)}
@router.get("/scoring-batches") @router.get("/scoring-batches")
def list_batches(db: Session = Depends(get_db)): def list_batches(db: Session = Depends(get_db)):
return [{**{key: getattr(batch, key) for key in ("id", "course_version_id", "rubric_version_id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")}, return [{**{key: getattr(batch, key) for key in ("id", "course_version_id", "rubric_version_id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")},
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None} "status_label": BATCH_STATUS_LABELS.get(batch.status, batch.status),
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None,
**batch_status_counts(db, batch.id)}
for batch in db.scalars(select(ScoringBatch).order_by(ScoringBatch.id.desc()))] for batch in db.scalars(select(ScoringBatch).order_by(ScoringBatch.id.desc()))]
@@ -392,7 +461,9 @@ def list_batches(db: Session = Depends(get_db)):
def batch_progress(batch_id: int, db: Session = Depends(get_db)): def batch_progress(batch_id: int, db: Session = Depends(get_db)):
batch = found(db, ScoringBatch, batch_id) batch = found(db, ScoringBatch, batch_id)
return {**{key: getattr(batch, key) for key in ("id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")}, return {**{key: getattr(batch, key) for key in ("id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")},
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None} "status_label": BATCH_STATUS_LABELS.get(batch.status, batch.status),
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None,
**batch_status_counts(db, batch_id)}
@router.post("/scoring-batches/{batch_id}/start") @router.post("/scoring-batches/{batch_id}/start")
+44 -9
View File
@@ -1,6 +1,8 @@
"""Persistent teacher conversations and SSE agent orchestration.""" """Persistent teacher conversations and SSE agent orchestration."""
import json import json
import re
import time import time
import unicodedata
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import timedelta from datetime import timedelta
from threading import Lock from threading import Lock
@@ -170,7 +172,8 @@ def _safe_arguments(arguments):
if not isinstance(arguments, dict): if not isinstance(arguments, dict):
return {} return {}
numeric = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", numeric = {"course_version_id", "batch_id", "project_id", "score_id", "task_id",
"rubric_version_id", "score", "offset", "limit", "project_offset", "criterion_offset"} "rubric_version_id", "score", "offset", "limit", "project_offset", "criterion_offset",
"document_id"}
text_limits = {"query": 300, "criterion_id": 100, "format": 20, "status": 30, text_limits = {"query": 300, "criterion_id": 100, "format": 20, "status": 30,
"student_no": 50, "relative_dir": 500} "student_no": 50, "relative_dir": 500}
result = {} result = {}
@@ -219,24 +222,56 @@ def _result_summary(result):
def _safe_citation(item): def _safe_citation(item):
if not isinstance(item, dict): if not isinstance(item, dict):
return None return None
return {key: value[:limit] if isinstance(value, str) else value for key, limit in # The evidence-policy fields travel with the citation so the interface can tell a
(("title", 200), ("quote", 500), ("url", 500), ("image_url", 500), ("score_url", 500)) # report body page from a contents page, and an answer's source from a historical
# scoring quotation.
label_limits = (("title", 200), ("quote", 500), ("url", 500), ("image_url", 500), ("score_url", 500),
("citation_id", 80), ("page_type", 20), ("page_type_label", 30),
("source_role", 20), ("source", 40), ("usage", 60), ("target_label", 120))
return {key: value[:limit] if isinstance(value, str) else value for key, limit in label_limits
if isinstance((value := item.get(key)), str)} | { if isinstance((value := item.get(key)), str)} | {
key: item[key] for key in ("chunk_id", "document_id", "page", "project_id") key: item[key] for key in ("chunk_id", "document_id", "page", "project_id", "historical")
if isinstance(item.get(key), int)} if isinstance(item.get(key), (int, bool)) and not isinstance(item.get(key), str)}
def _answer_citations(items): def _comparable_text(value):
"""Keep each distinct source passage, including later tool calls and pages cited by the answer.""" return "".join(char for char in unicodedata.normalize("NFKC", str(value or "")) if not char.isspace())
def _cited_pages(content: str) -> set[int]:
"""Page and slide numbers the answer actually refers to."""
numbers = set()
for match in re.finditer(r"第\s*(\d{1,4})\s*(?:页|张幻灯片|页幻灯片)", str(content or "")):
numbers.add(int(match.group(1)))
return numbers
def _answer_citations(items, content: str = ""):
"""Attach only the citations the answer actually used.
Collecting every tool hit would list material the answer never drew on. A passage is
kept when the answer quotes it, or when the answer names the page it sits on. When
the answer names no page and quotes nothing, no citation is attached: the answer made
no specific evidential claim.
"""
seen = set() seen = set()
citations = [] citations = []
answer = _comparable_text(content)
pages = _cited_pages(content)
for item in items: for item in items:
safe = _safe_citation(item) safe = _safe_citation(item)
if not safe: if not safe:
continue continue
key = (safe.get("document_id"), safe.get("chunk_id"), safe.get("title"), key = (safe.get("document_id"), safe.get("chunk_id"), safe.get("title"),
safe.get("page"), safe.get("quote"), safe.get("url"), safe.get("score_url")) safe.get("page"), safe.get("quote"), safe.get("url"), safe.get("score_url"))
if key not in seen: if key in seen:
continue
quote = _comparable_text(safe.get("quote"))
used = bool(quote) and len(quote) >= 6 and quote[:80] in answer
if not used and pages and isinstance(safe.get("page"), int):
used = safe["page"] in pages
if not used:
continue
seen.add(key) seen.add(key)
citations.append(safe) citations.append(safe)
return citations return citations
@@ -326,7 +361,7 @@ def _run_agent(factory, conversation_id, run_id, user_text):
content = reply.get("content") content = reply.get("content")
if not isinstance(content, str) or not content.strip(): if not isinstance(content, str) or not content.strip():
raise ModelError("会话模型未返回文字或工具调用") raise ModelError("会话模型未返回文字或工具调用")
answer_citations = _answer_citations(citations) answer_citations = _answer_citations(citations, content)
db.add(ChatMessage(conversation_id=conversation_id, role="assistant", content=content, db.add(ChatMessage(conversation_id=conversation_id, role="assistant", content=content,
citations=answer_citations, tool_calls=used)) citations=answer_citations, tool_calls=used))
run.status, run.finished_at = "completed", now() run.status, run.finished_at = "completed", now()
+150
View File
@@ -0,0 +1,150 @@
"""Evidence policy: which page types may support which kind of judgement.
The policy is part of the versionable scoring rules. Each required-evidence target
declares its ``use``:
* ``structure`` — the target may be satisfied by a cover page, table of contents or
reference list (chapter completeness, bibliography, report layout).
* ``content`` — the target may only be satisfied by substantive text, data or tables
on body or appendix pages.
Rules still in the historical string form (``["问题定义", "需求分析"]``) are read with a
permissive grammar: wording that names a structural artefact becomes a structure
target, everything else becomes a content target. The exact grammar is recorded as
``POLICY_VERSION`` in the batch snapshot so a batch can always be re-read against the
rules it was scored with.
"""
from __future__ import annotations
import re
from .pages import (CONTENT_PAGE_TYPES, NAVIGATION_PAGE_TYPES, STORE_PAGE_TYPES,
STRUCTURE_PAGE_TYPES)
POLICY_VERSION = 1
USE_STRUCTURE = "structure"
USE_CONTENT = "content"
#: Wording that names report packaging rather than what the student did.
_STRUCTURE_WORDS = (
"标题", "封面", "目录", "章节", "参考文献", "引用规范", "格式", "规范性", "结构完整",
"排版", "页码", "附录", "图表编号", "摘要",
)
#: Appendix and body pages are the only pages allowed for a content judgement.
CONTENT_ALLOWED = tuple(CONTENT_PAGE_TYPES)
#: Structural targets may additionally use cover, contents and reference pages.
STRUCTURE_ALLOWED = tuple(STRUCTURE_PAGE_TYPES)
#: Navigation pages can never carry a content judgement.
NAVIGATION = tuple(NAVIGATION_PAGE_TYPES)
def _looks_structural(text: str) -> bool:
value = str(text or "").strip()
return bool(value) and any(word in value for word in _STRUCTURE_WORDS)
def normalize_target(raw) -> dict:
"""Read one required-evidence target in either the rich or the legacy string form."""
if isinstance(raw, dict):
label = str(raw.get("label") or raw.get("text") or raw.get("name") or "").strip()
use = str(raw.get("use") or "").strip().lower()
if use not in (USE_STRUCTURE, USE_CONTENT):
use = USE_STRUCTURE if _looks_structural(label) else USE_CONTENT
allowed = raw.get("allowed_page_types")
if not isinstance(allowed, list) or not allowed:
allowed = list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED)
allowed = [str(item) for item in allowed if str(item) in STORE_PAGE_TYPES]
if not allowed:
allowed = list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED)
if use == USE_CONTENT:
# A content target never admits navigation pages, whatever a rule claims.
allowed = [item for item in allowed if item not in NAVIGATION] or list(CONTENT_ALLOWED)
return {"label": label or "证据", "use": use, "allowed_page_types": allowed,
"query": str(raw.get("query") or label).strip() or (label or "证据")}
label = str(raw or "").strip()
use = USE_STRUCTURE if _looks_structural(label) else USE_CONTENT
return {"label": label or "证据", "use": use,
"allowed_page_types": list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED),
"query": label or "证据"}
def normalize_targets(required_evidence) -> list[dict]:
"""Normalize a criterion's required evidence, preserving order and identity."""
values = required_evidence if isinstance(required_evidence, list) else [required_evidence]
targets = []
for index, raw in enumerate(values):
if raw is None or (isinstance(raw, str) and not raw.strip()):
continue
target = normalize_target(raw)
target["index"] = index
target["target_id"] = str((raw.get("target_id") if isinstance(raw, dict) else "") or f"t{index}")
targets.append(target)
if not targets:
targets.append({"label": "正文实质内容", "use": USE_CONTENT, "allowed_page_types": list(CONTENT_ALLOWED),
"query": "正文实质内容", "index": 0, "target_id": "t0"})
return targets
def content_targets(criterion: dict) -> list[dict]:
return [target for target in normalize_targets(criterion.get("required_evidence")) if target["use"] == USE_CONTENT]
def any_content_target(criterion: dict) -> bool:
return bool(content_targets(criterion))
def page_type_allowed(page_type: str, allowed: list[str]) -> bool:
"""An ``unknown`` page is a body page only when the target already allows body text.
``unknown`` marks a page whose reading is genuinely unsettled, so it may never be
used as the sole support for a content judgement, and it never widens a structural
target either. It stays retrievable so a teacher can inspect it.
"""
if page_type == "unknown":
return "body" in allowed
return page_type in allowed
def content_evidence_allowed(page_type: str) -> bool:
"""Only substantive body and appendix pages support a content judgement."""
return page_type in CONTENT_ALLOWED
def heading_only_text(text: str) -> bool:
"""True when a passage is nothing but section titles, so it proves no content.
The test is structural, not a length test: a quotation is heading-only when every
line of it is a chapter/section heading or a contents entry. A short technical
quotation such as a measured value is content, so it must not be rejected here.
"""
from .pages import heading_level, is_toc_entry
lines = [line.strip() for line in str(text or "").splitlines() if line.strip()]
if not lines:
return False
for line in lines:
if is_toc_entry(line) or heading_level(line):
continue
return False
return True
_PLACEHOLDER = re.compile(r"^\s*(待确认|待定|未知|无|none|null|-+|—+)\s*$", re.IGNORECASE)
def meaningful(value) -> bool:
text = str(value or "").strip()
return bool(text) and not _PLACEHOLDER.match(text)
def plan_snapshot(criteria: list[dict]) -> dict:
"""The policy attached to a batch snapshot, so old batches stay re-readable."""
return {
"version": POLICY_VERSION,
"content_allowed_page_types": list(CONTENT_ALLOWED),
"structure_allowed_page_types": list(STRUCTURE_ALLOWED),
"navigation_page_types": list(NAVIGATION),
"criteria": {criterion["criterion_id"]: normalize_targets(criterion.get("required_evidence"))
for criterion in criteria},
}
+45 -6
View File
@@ -11,6 +11,32 @@ from .knowledge_pages import evidence_links
from .models import (CriterionScore, ExportTask, Project, ScoreEvidence, ScoringBatch, Student, SubmissionFile) from .models import (CriterionScore, ExportTask, Project, ScoreEvidence, ScoringBatch, Student, SubmissionFile)
from .rules import totals from .rules import totals
#: Teacher-facing wording for a criterion status, shared by exports and the API.
STATUS_LABELS = {
"pending": "待评分",
"scored": "模型已评分",
"reviewed": "教师已复核",
"needs_review": "待教师复核",
"unscorable": "材料缺失或解析失败",
"failed": "模型调用失败",
}
PAGE_TYPE_LABELS = {
"cover": "封面", "abstract": "摘要", "toc": "目录", "body": "正文",
"appendix": "附录", "references": "参考文献", "unknown": "识别不确定",
"mixed": "目录与正文混合页",
}
#: Batch status wording. ``review_required`` means grading finished but some criteria
#: still need a teacher decision, which is not the same as an incomplete batch.
BATCH_STATUS_LABELS = {
"created": "待启动", "queued": "排队中", "running": "运行中",
"completed": "已完成", "review_required": "已完成,有待复核项",
"partial": "部分完成", "failed": "失败",
}
def page_type_label(value) -> str:
return PAGE_TYPE_LABELS.get(str(value or "unknown"), "未分类页面")
def project_result(db, batch: ScoringBatch, project: Project): def project_result(db, batch: ScoringBatch, project: Project):
scores = db.scalars(select(CriterionScore).where(CriterionScore.batch_id == batch.id, scores = db.scalars(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
@@ -19,10 +45,15 @@ def project_result(db, batch: ScoringBatch, project: Project):
criteria = batch.rubric_snapshot["criteria"] criteria = batch.rubric_snapshot["criteria"]
totals_ = totals({key: score.final_score for key, score in by_id.items()}, criteria, batch.rubric_snapshot["weights"]) totals_ = totals({key: score.final_score for key, score in by_id.items()}, criteria, batch.rubric_snapshot["weights"])
rows = [] rows = []
def evidence_dict(item): def evidence_dict(item, links):
links = evidence_links(db, item)
return {"page": item.page, "quote": item.quote, "file_id": item.submission_file_id, return {"page": item.page, "quote": item.quote, "file_id": item.submission_file_id,
"document_id": item.document_id, "document_id": item.document_id,
"target_id": item.target_id or "", "target_label": item.target_label or "",
"page_type": item.page_type or links.get("page_type") or "unknown",
"source_role": item.source_role or "content",
# A quotation saved before the evidence policy existed is labelled as a
# historical citation and never presented as report body evidence.
"historical": bool(item.historical),
"document": db.get(SubmissionFile, item.submission_file_id).relative_path if item.submission_file_id else "课程资料", "document": db.get(SubmissionFile, item.submission_file_id).relative_path if item.submission_file_id else "课程资料",
**links} **links}
for criterion in criteria: for criterion in criteria:
@@ -35,7 +66,8 @@ def project_result(db, batch: ScoringBatch, project: Project):
"model_comment": score.model_comment if score else "", "teacher_comment": score.teacher_comment if score else None, "model_comment": score.model_comment if score else "", "teacher_comment": score.teacher_comment if score else None,
"missing_evidence": score.missing_evidence if score else [], "suggestion": score.suggestion if score else "", "missing_evidence": score.missing_evidence if score else [], "suggestion": score.suggestion if score else "",
"status": score.status if score else "pending", "error": score.error if score else "", "status": score.status if score else "pending", "error": score.error if score else "",
"evidence": [evidence_dict(item) for item in evidence]}) "evidence_gaps": (score.evidence_gaps or []) if score else [],
"evidence": [evidence_dict(item, evidence_links(db, item)) for item in evidence]})
student = db.get(Student, project.student_id) student = db.get(Student, project.student_id)
return {"project_id": project.id, "student_no": student.student_no, "name": student.name, "topic": project.topic, return {"project_id": project.id, "student_no": student.student_no, "name": student.name, "topic": project.topic,
"totals": totals_, "rows": rows, "facts": project.facts} "totals": totals_, "rows": rows, "facts": project.facts}
@@ -97,14 +129,21 @@ def _html_report(result, batch):
for row in result["rows"]: for row in result["rows"]:
criterion = row["criterion"] criterion = row["criterion"]
evidences = "".join( evidences = "".join(
f"<li>{esc(e['document'])} · 第{e['page']}{'张幻灯片' if str(e['document']).lower().endswith('.pptx') else '页'}:{esc(e['quote'])}</li>" f"<li>{esc(e['document'])} · 第{e['page']}{'张幻灯片' if str(e['document']).lower().endswith('.pptx') else '页'}"
f"({esc(page_type_label(e.get('page_type')))}{',历史评分引用' if e.get('historical') else ''})"
f"{' 支持:' + esc(e['target_label']) if e.get('target_label') else ''}:{esc(e['quote'])}</li>"
for e in row["evidence"] for e in row["evidence"]
) )
missing = "、".join(row["missing_evidence"] or []) missing = "、".join(row["missing_evidence"] or [])
gaps = ";".join(f"{gap.get('target_label')}(检索范围:{'、'.join(page_type_label(t) for t in gap.get('searched_page_types') or []) or '正文'})"
for gap in row.get("evidence_gaps") or [])
review = "本项缺少正文证据,已转教师复核。" if row["status"] == "needs_review" else ""
rows.append(f"<section><h3>{esc(criterion['name'])}({esc(row['final_score'])} / {criterion['max_score']})</h3>" rows.append(f"<section><h3>{esc(criterion['name'])}({esc(row['final_score'])} / {criterion['max_score']})</h3>"
f"<p>模型原始分:{esc(row['model_score'])};教师修改分:{esc(row['teacher_score'])};状态:{esc(row['status'])}</p>" f"<p>模型原始分:{esc(row['model_score'])};教师修改分:{esc(row['teacher_score'])};状态:{esc(STATUS_LABELS.get(row['status'], row['status']))}</p>"
f"<p>模型评语:{esc(row['model_comment'])}</p><p>教师评语:{esc(row['teacher_comment'])}</p>" f"<p>模型评语:{esc(row['model_comment'])}</p><p>教师评语:{esc(row['teacher_comment'])}</p>"
f"<p>缺失证据:{esc(missing)}</p><p>建议:{esc(row['suggestion'])}</p><p>异常:{esc(row['error'])}</p>" f"<p>{esc(review)}</p>"
f"<p>缺失证据:{esc(missing)}</p><p>缺正文证据的目标:{esc(gaps)}</p>"
f"<p>建议:{esc(row['suggestion'])}</p><p>异常:{esc(row['error'])}</p>"
f"<ul>{evidences}</ul></section>") f"<ul>{evidences}</ul></section>")
totals_ = result["totals"] totals_ = result["totals"]
return f"""<!doctype html><html lang="zh"><meta charset="utf-8"><style> return f"""<!doctype html><html lang="zh"><meta charset="utf-8"><style>
+77 -2
View File
@@ -2,7 +2,8 @@
from sqlalchemy import select from sqlalchemy import select
from .db import SessionLocal from .db import SessionLocal
from .models import IndexTask, SubmissionFile, now from .models import IndexTask, KnowledgeDocument, SubmissionFile, now
from .rag import INDEX_VERSION
from .scoring import parse_submission from .scoring import parse_submission
@@ -14,7 +15,13 @@ def run_index_task(task_id: int, *, session_factory=SessionLocal, embedder=None)
task.status, task.error = "running", "" task.status, task.error = "running", ""
db.commit() db.commit()
try: try:
if task.kind == "project": if task.kind == "reindex":
# Rebuild documents whose index predates the current chunking, so a
# legacy index is never silently used as classified body text.
task.result = reindex_stale(db, task.course_version_id, embedder=embedder)
if task.result["failed"]:
raise ValueError("部分资料重建索引失败:" + "、".join(item["title"] for item in task.result["failed"]))
elif task.kind == "project":
file = db.get(SubmissionFile, task.submission_file_id) file = db.get(SubmissionFile, task.submission_file_id)
if not file or file.recognition_status != "recognized": if not file or file.recognition_status != "recognized":
raise ValueError("材料已取消选择,请重新预览") raise ValueError("材料已取消选择,请重新预览")
@@ -35,6 +42,74 @@ def run_index_task(task_id: int, *, session_factory=SessionLocal, embedder=None)
db.commit() db.commit()
def stale_documents(db, course_version_id: int | None = None, *, limit: int | None = None):
"""Documents whose stored index predates the current chunking or is not ready."""
query = select(KnowledgeDocument).where(
(KnowledgeDocument.index_version != INDEX_VERSION) | (KnowledgeDocument.status != "ready"))
if course_version_id is not None:
query = query.where(KnowledgeDocument.course_version_id == course_version_id)
query = query.order_by(KnowledgeDocument.id)
if limit:
query = query.limit(limit)
return db.scalars(query).all()
def reindex_stale(db, course_version_id: int | None = None, *, embedder=None, limit: int | None = None):
"""Rebuild every stale index from the saved parse output. Retryable and auditable.
Student material is rebuilt from ``submission_files.parsed``, which the parser saved
on disk; the original file is never re-read or modified. Course material follows the
normal ingest path, because its pages come from the mounted course documents.
"""
from .services import parse_submission
from .storage import sha256, secure_path
documents = stale_documents(db, course_version_id, limit=limit)
rebuilt, skipped, failed = [], [], []
for document in documents:
try:
if document.submission_file_id:
file = db.get(SubmissionFile, document.submission_file_id)
if not file:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "来源材料已不存在"})
continue
if sha256(secure_path(file.relative_path)) != file.sha256:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "源文件已变化,请重新扫描并确认导入"})
continue
file.parse_status = "pending"
db.commit()
parsed = parse_submission(db, file, embedder=embedder) if embedder else parse_submission(db, file)
rebuilt.append({"document_id": document.id, "title": document.title,
"pages": len(parsed.get("pages", []))})
elif document.project_id is None:
# Course material is re-ingested by the course task, not per document.
skipped.append({"document_id": document.id, "title": document.title,
"reason": "课程资料请通过课程入库重建"})
else:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "缺少可重建的解析结果"})
except Exception as exc:
failed.append({"document_id": document.id, "title": document.title, "error": str(exc)[:300]})
db.rollback()
return {"index_version": INDEX_VERSION, "rebuilt": rebuilt, "skipped": skipped,
"failed": failed, "remaining": len(stale_documents(db, course_version_id))}
def create_reindex_task(db, course_version_id: int):
"""Queue one reindex job for a course version, reusing an open task when there is one."""
from .services import enqueue_index_task
existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id,
IndexTask.kind == "reindex", IndexTask.status.in_(["queued", "running"])).order_by(IndexTask.id.desc()))
task = existing or IndexTask(course_version_id=course_version_id, kind="reindex", status="queued")
if not existing:
db.add(task)
db.commit()
enqueue_index_task(db, task.id)
return task
def create_course_index_task(db, course_version_id: int): def create_course_index_task(db, course_version_id: int):
from .services import enqueue_index_task from .services import enqueue_index_task
existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id, existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id,
+21 -6
View File
@@ -36,18 +36,33 @@ def page_image_path(db, doc: KnowledgeDocument, page: int) -> Path | None:
return path if path and path.is_relative_to(root) and path.is_file() else None return path if path and path.is_relative_to(root) and path.is_file() else None
def page_links(db, document_id: int | None, page: int | None) -> dict[str, str | None]: def page_links(db, document_id: int | None, page: int | None) -> dict:
"""Stable citation links plus the page type, so callers can label the source."""
empty = {"url": None, "image_url": None, "page_type": None, "section": None}
if not document_id or not page: if not document_id or not page:
return {"url": None, "image_url": None} return empty
doc = db.get(KnowledgeDocument, document_id) doc = db.get(KnowledgeDocument, document_id)
if not doc: if not doc:
return {"url": None, "image_url": None} return empty
record = db.get(KnowledgePage, (document_id, page)) record = db.get(KnowledgePage, (document_id, page))
if not record: if not record:
return {"url": None, "image_url": None} return empty
return {"url": text_url(document_id, page), return {"url": text_url(document_id, page),
"image_url": image_url(document_id, page) if page_image_path(db, doc, page) else None} "image_url": image_url(document_id, page) if page_image_path(db, doc, page) else None,
"page_type": record.page_type or "unknown",
"section": record.section or ""}
def evidence_links(db, evidence) -> dict[str, str | None]: def evidence_links(db, evidence) -> dict:
return page_links(db, evidence.document_id, evidence.page) return page_links(db, evidence.document_id, evidence.page)
def page_types(db, document_ids) -> dict[int, str]:
"""Page type per page of the given documents, for labelling citations in bulk."""
from sqlalchemy import select as _select
ids = [item for item in dict.fromkeys(document_ids or []) if item]
if not ids:
return {}
rows = db.execute(_select(KnowledgePage.document_id, KnowledgePage.page, KnowledgePage.page_type)
.where(KnowledgePage.document_id.in_(ids))).all()
return {(document_id, page): (page_type or "unknown") for document_id, page, page_type in rows}
+62 -11
View File
@@ -15,25 +15,39 @@ class EvidenceResult(BaseModel):
chunk_id: int chunk_id: int
page: int = Field(ge=1) page: int = Field(ge=1)
quote: str = Field(min_length=1) quote: str = Field(min_length=1)
#: Which required-evidence target this quotation is offered to support.
target_id: str = ""
class ScoreResult(BaseModel): class ScoreResult(BaseModel):
criterion_id: str criterion_id: str
score: float = Field(ge=0) #: ``None`` means the model declined to score because admissible evidence is absent.
score: float | None = Field(default=None, ge=0)
max_score: float = Field(gt=0) max_score: float = Field(gt=0)
reason: str = Field(min_length=1) reason: str = Field(min_length=1)
evidence: list[EvidenceResult] evidence: list[EvidenceResult] = Field(default_factory=list)
missing_evidence: list[str] = Field(default_factory=list) missing_evidence: list[str] = Field(default_factory=list)
suggestion: str = "" suggestion: str = ""
confidence: float = Field(ge=0, le=1) confidence: float = Field(ge=0, le=1)
@model_validator(mode="after") @model_validator(mode="after")
def valid_range(self): def valid_range(self):
if self.score > self.max_score or (self.score > 0 and not self.evidence): if self.score is not None and self.score > self.max_score:
raise ValueError("score exceeds maximum or lacks evidence") raise ValueError("score exceeds maximum")
if self.score and not self.evidence:
raise ValueError("a positive score requires evidence")
return self return self
class EvidenceGap(BaseModel):
"""A content target that still has no body evidence after targeted re-checking."""
target_id: str = ""
target_label: str = ""
searched_page_types: list[str] = Field(default_factory=list)
reason: str = ""
class FactResult(BaseModel): class FactResult(BaseModel):
student_no: str = "" student_no: str = ""
name: str = "" name: str = ""
@@ -132,18 +146,22 @@ class DeepSeekClient:
raise ModelError("当前端点或模型未返回原生 tool_calls;请配置支持工具调用的端点和模型") raise ModelError("当前端点或模型未返回原生 tool_calls;请配置支持工具调用的端点和模型")
def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict]): def locate_quote(text: str, quote: str):
if result.criterion_id != criterion["criterion_id"] or result.max_score != criterion["max_score"]: """Locate a model-written quotation inside the retrieved text.
raise ModelError("模型评分项或满分与规则版本不一致")
by_id = {chunk["chunk_id"]: chunk for chunk in chunks} PDF line breaks and punctuation differ from what the model reproduces, so matching
is done on NFKC-normalized text with whitespace and punctuation removed, while the
returned string is still the exact source substring.
"""
if not text or not quote:
return None
if quote in text:
return quote
def locate_quote(text: str, quote: str):
def comparable(value: str): def comparable(value: str):
return "".join(char for original in value for char in unicodedata.normalize("NFKC", original) return "".join(char for original in value for char in unicodedata.normalize("NFKC", original)
if not char.isspace() and not unicodedata.category(char).startswith("P")) if not char.isspace() and not unicodedata.category(char).startswith("P"))
if quote in text:
return quote
needle = comparable(quote) needle = comparable(quote)
if len(needle) < 4: if len(needle) < 4:
return None return None
@@ -159,6 +177,27 @@ def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict]):
return None return None
return text[positions[start]:positions[start + len(needle) - 1] + 1] return text[positions[start]:positions[start + len(needle) - 1] + 1]
def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict], *, targets=None,
require_content_evidence: bool | None = None):
"""Check identity, quotation fidelity and the evidence-source policy.
The model may understand evidence and propose re-checks; the page permissions are
enforced here, in code, and cannot be argued away by the model's own reasoning.
"""
from .evidence import (USE_CONTENT, content_evidence_allowed, heading_only_text,
normalize_targets, page_type_allowed)
if result.criterion_id != criterion["criterion_id"] or result.max_score != criterion["max_score"]:
raise ModelError("模型评分项或满分与规则版本不一致")
if require_content_evidence is None:
require_content_evidence = any(target["use"] == USE_CONTENT
for target in normalize_targets(criterion.get("required_evidence")))
targets = targets if targets is not None else normalize_targets(criterion.get("required_evidence"))
by_target = {target["target_id"]: target for target in targets}
default_content = next((target for target in targets if target["use"] == USE_CONTENT), None)
by_id = {chunk["chunk_id"]: chunk for chunk in chunks}
for evidence in result.evidence: for evidence in result.evidence:
chunk = by_id.get(evidence.chunk_id) chunk = by_id.get(evidence.chunk_id)
quote = locate_quote(chunk["text"], evidence.quote) if chunk else None quote = locate_quote(chunk["text"], evidence.quote) if chunk else None
@@ -169,5 +208,17 @@ def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict]):
chunk, quote = matches[0] chunk, quote = matches[0]
if quote is None: if quote is None:
raise ModelError("模型证据未能匹配检索原文") raise ModelError("模型证据未能匹配检索原文")
page_type = chunk.get("page_type") or "unknown"
target = by_target.get(evidence.target_id) or default_content or (targets[0] if targets else None)
if target is None:
raise ModelError("模型证据未关联任何必备证据目标")
if not page_type_allowed(page_type, target["allowed_page_types"]):
raise ModelError(f"模型证据来自不允许的页面类型({page_type}),不能支持「{target['label']}」")
if target["use"] == USE_CONTENT and not content_evidence_allowed(page_type):
raise ModelError(f"目录、摘要或参考文献页不能作为「{target['label']}」的内容证据")
if target["use"] == USE_CONTENT and heading_only_text(quote):
raise ModelError(f"「{target['label']}」的证据只有章节标题,没有实质内容")
evidence.chunk_id, evidence.page, evidence.quote = chunk["chunk_id"], chunk["page"], quote evidence.chunk_id, evidence.page, evidence.quote = chunk["chunk_id"], chunk["page"], quote
evidence.target_id = target["target_id"]
return result return result
+22
View File
@@ -176,6 +176,14 @@ class KnowledgePage(Base):
page = Column(Integer, primary_key=True) page = Column(Integer, primary_key=True)
text = Column(Text, nullable=False, default="") text = Column(Text, nullable=False, default="")
image_path = Column(String(1200), nullable=True) # Relative to processed_path. image_path = Column(String(1200), nullable=True) # Relative to processed_path.
# Page-level classification: cover, abstract, toc, body, references or unknown.
page_type = Column(String(20), nullable=False, default="unknown", server_default="unknown")
mixed = Column(Boolean, nullable=False, default=False, server_default="false")
section = Column(String(500), default="")
chapter = Column(String(500), default="")
confidence = Column(Float, nullable=True)
# Bounded classifier evidence: which signals decided the type, kept for audit.
classification = Column(JSON, default=dict, nullable=False)
class KnowledgeChunk(Base): class KnowledgeChunk(Base):
@@ -187,6 +195,9 @@ class KnowledgeChunk(Base):
document_type = Column(String(80), nullable=False) document_type = Column(String(80), nullable=False)
page = Column(Integer, nullable=False) page = Column(Integer, nullable=False)
section = Column(String(500), default="") section = Column(String(500), default="")
chapter = Column(String(500), default="")
page_type = Column(String(20), nullable=False, default="unknown", server_default="unknown")
block_type = Column(String(20), nullable=False, default="body", server_default="body")
text = Column(Text, nullable=False) text = Column(Text, nullable=False)
embedding = Column(Embedding, nullable=True) embedding = Column(Embedding, nullable=True)
@@ -225,6 +236,9 @@ class CriterionScore(Base):
suggestion = Column(Text, default="") suggestion = Column(Text, default="")
status = Column(String(40), default="pending") status = Column(String(40), default="pending")
error = Column(Text, default="") error = Column(Text, default="")
#: Content targets that still lacked body evidence, with the search that was run.
#: Kept after teacher review so the original retrieval problem stays auditable.
evidence_gaps = Column(JSON, default=list)
__table_args__ = (UniqueConstraint("batch_id", "project_id", "criterion_id"),) __table_args__ = (UniqueConstraint("batch_id", "project_id", "criterion_id"),)
@property @property
@@ -241,6 +255,14 @@ class ScoreEvidence(Base):
page = Column(Integer, nullable=False) page = Column(Integer, nullable=False)
quote = Column(Text, nullable=False) quote = Column(Text, nullable=False)
image_path = Column(String(1200), nullable=True) image_path = Column(String(1200), nullable=True)
# Which required-evidence target this quotation is offered to support.
target_id = Column(String(50), default="", server_default="")
target_label = Column(String(300), default="", server_default="")
# Page type of the source passage, and its role in the judgement.
page_type = Column(String(20), default="unknown", server_default="unknown")
source_role = Column(String(20), default="content", server_default="content")
# True when the quotation was written before this retrieval policy existed.
historical = Column(Boolean, default=False, nullable=False, server_default="false")
class TeacherRevision(Base): class TeacherRevision(Base):
+443
View File
@@ -0,0 +1,443 @@
"""Page-level classification, block splitting and real section paths.
A page carries one ``page_type``: cover, abstract, toc, body, references, unknown.
A page may additionally be ``mixed`` when a table of contents and body text truly
share one page; only that page is split into blocks so the body part is not lost.
Classification never depends on a fixed page number or on the single word "目录".
It combines the page's own signals (heading title, dot-leader entry density,
trailing page numbers, abstract/reference wording, cover fields, footer style)
with document-level continuity for continuation pages (TOC continued, abstract
continued, references spanning pages).
"""
from __future__ import annotations
import re
import unicodedata
CLASSIFIER_VERSION = 3
PAGE_TYPES = ("cover", "abstract", "toc", "body", "appendix", "references", "unknown")
#: Page types whose text may support a content judgement about what a student did.
CONTENT_PAGE_TYPES = ("body", "appendix")
#: Page types whose text may support a structural judgement (chapter completeness etc.).
STRUCTURE_PAGE_TYPES = ("cover", "abstract", "toc", "references", "body", "appendix")
#: Page types that may never be used as content evidence on their own.
NAVIGATION_PAGE_TYPES = ("cover", "abstract", "toc", "references")
#: Page types a page may be stored as, used to validate rule configuration.
STORE_PAGE_TYPES = PAGE_TYPES
BLOCK_BODY = "body"
BLOCK_TOC = "toc"
BLOCK_TITLE = "title"
_DOT_LEADER = re.compile(r"(?:[.·•…․…]\s*){3,}")
_TRAILING_ANCHOR = re.compile(r"(\d{1,4})\s*$")
_CHAPTER = re.compile(r"^第\s*([0-9一二三四五六七八九十百]+)\s*章\s*(.*)$")
_APPENDIX = re.compile(r"^附\s*录\s*([A-Za-z0-9一二三四五六七八九十]+)?\s*(.*)$")
_NUMBERED = re.compile(r"^(\d+(?:\.\d+)*)[\s、..]+(\S.*)$")
_COVER_FIELD = re.compile(r"(学\s*号|姓\s*名|题\s*目|指导教师|专\s*业|学\s*院|日\s*期)\s*[::]")
_PAGE_NUMBER_LINE = re.compile(r"^\s*(?:\d{1,3}|[IVXLC]{1,5}|第\s*\d+\s*页)\s*$")
_ABSTRACT_TITLE = re.compile(r"^\s*(摘\s*要|abstract)\s*$", re.IGNORECASE)
_REFERENCE_TITLE = re.compile(r"^\s*(参\s*考\s*文\s*献|references?|参考文献与资料)\s*$", re.IGNORECASE)
_ACK_TITLE = re.compile(r"^\s*(致\s*谢|acknowledge?ments?)\s*$", re.IGNORECASE)
_TOC_TITLE = re.compile(r"^\s*(目\s*录|contents?|table\s+of\s+contents)\s*$", re.IGNORECASE)
_REFERENCE_ENTRY = re.compile(r"^\s*\[\d{1,3}\]")
_CJK = re.compile(r"[㐀-䶿一-鿿]")
_HEADING_NUMBER = re.compile(r"^\s*(第\s*[0-9一二三四五六七八九十百]+\s*章|\d+(\.\d+)*|附\s*录\s*[A-Za-z0-9]?)\s*\S")
#: A body block must carry at least this many non-space characters to count as substance.
MIN_BODY_CHARS = 60
#: Heading-only text shorter than this never counts as content evidence on its own.
MIN_CONTENT_QUOTE_CHARS = 25
#: A line at least this long, outside a heading, reads as running prose. The bar is low
#: on purpose: the rule separates text from headings and labels, not long from short
#: text, so a brief but real sentence is still content.
MIN_PROSE_CHARS = 4
#: A numbered heading title must stay this short; longer text is prose or a list item.
MAX_HEADING_TITLE_CHARS = 40
#: Wording that marks the line as running prose rather than a heading.
_PROSE_PUNCTUATION = "。,;!?,;"
#: The documents are Chinese engineering reports; the keyword recall uses these units.
BIGRAM_MIN = 2
def _lines(text: str) -> list[str]:
return [line.strip() for line in (text or "").splitlines() if line.strip()]
def normalize_heading(text: str) -> str:
"""Compare headings across the TOC and the body: unified width, no spaces/dots."""
value = unicodedata.normalize("NFKC", text or "")
value = _DOT_LEADER.sub("", value)
value = _TRAILING_ANCHOR.sub("", value.strip())
return "".join(char for char in value if not char.isspace() and char not in "..·•…")
def is_toc_entry(line: str) -> bool:
"""A dotted leader plus a trailing page or slide number."""
stripped = (line or "").strip()
if not stripped or not _DOT_LEADER.search(stripped):
return False
return bool(_TRAILING_ANCHOR.search(stripped))
def heading_syntax(line: str) -> tuple[int, str] | None:
"""Return ``(level, title)`` when the line has the shape of a heading.
This is a purely syntactic reading. Numbered prose such as
``4. 实时图像传输:将摄像头画面…`` and enumerated list items are not headings: a
numbered heading has a short Chinese title, no sentence punctuation and no trailing
colon. Equation and matrix rows such as ``3.3.3`` or ``1 0 0 1 Zc`` are never
headings, and a line cut mid-formula (trailing combining mark) is not either.
"""
stripped = (line or "").strip()
if not stripped or is_toc_entry(stripped) or len(stripped) > 120:
return None
match = _CHAPTER.match(stripped)
if match:
return 1, stripped
match = _APPENDIX.match(stripped)
if match and not _COVER_FIELD.search(stripped):
return 2, stripped
if _ABSTRACT_TITLE.match(stripped) or _REFERENCE_TITLE.match(stripped) or _ACK_TITLE.match(stripped):
return 1, stripped
match = _NUMBERED.match(stripped)
if not match:
return None
title = match.group(2).strip()
if not title or title.endswith(":") or title.endswith(":"):
return None
if len(title) > MAX_HEADING_TITLE_CHARS or any(char in title for char in _PROSE_PUNCTUATION):
return None
if not _CJK.search(title):
return None
if unicodedata.category(title[-1]).startswith("M"):
return None
return 1 + match.group(1).count("."), stripped
def heading_level(line: str, toc_titles: set[str] | None = None) -> tuple[int, str] | None:
"""A heading for the section path: syntactic shape, confirmed by the contents list.
When the document carries its own table of contents, that list is the authority on
which numbered lines are real headings. That is what keeps an enumerated list item or
a numbered paragraph from becoming a chapter in the extracted section path. Callers
that only need to know whether a line reads as a heading use ``heading_syntax``.
"""
found = heading_syntax(line)
if not found:
return None
if toc_titles and normalize_heading(line) not in toc_titles:
return None
return found
def _footer_style(lines: list[str]) -> str:
"""Classify the trailing line so repeated headers/footers can be removed."""
if not lines:
return "none"
last = lines[-1]
if re.fullmatch(r"[IVXLC]{1,5}", last):
return "roman"
if re.fullmatch(r"\d{1,3}", last):
return "arabic"
if re.fullmatch(r"第\s*\d+\s*页", last):
return "page_cn"
return "text"
def _signals(page: dict, *, index: int, total: int, toc_titles: set[str] | None = None,
repeated: set[str] | None = None) -> dict:
text = str(page.get("text") or "")
lines = _lines(text)
# Prose detection ignores repeated page headers and footers: a running header is not
# something the student wrote, so it must not make an empty page look like content.
prose_source = [line for line in lines if line not in (repeated or ())]
entries = [line for line in lines if is_toc_entry(line)]
headings = [item for item in (heading_level(line, toc_titles) for line in lines) if item]
body_lines = [line for line in lines if line not in set(entries)]
plain_chars = sum(len(line) for line in body_lines)
first = lines[0] if lines else ""
reference_entries = [line for line in lines if _REFERENCE_ENTRY.match(line)]
titles = [position for position, line in enumerate(lines)
if _REFERENCE_TITLE.match(line) or _ACK_TITLE.match(line)]
# A bibliography page is dense reference entries, or a reference title near the top.
reference_ratio = len(reference_entries) / len(lines) if lines else 0.0
reference_page = (len(reference_entries) >= 5 and reference_ratio >= 0.5
or any(position <= 2 for position in titles))
return {
"chars": len(text.strip()),
"lines": len(lines),
"toc_entries": len(entries),
"toc_ratio": round(len(entries) / len(lines), 3) if lines else 0.0,
"headings": len(headings),
"chapter_headings": sum(1 for level, _ in headings if level == 1),
"body_chars": plain_chars,
"first_line": first[:120],
"position": index,
"total": total,
"footer": _footer_style(lines),
"cover_fields": len(_COVER_FIELD.findall(text)),
"has_table": bool(page.get("tables")),
"reference_entries": len(reference_entries),
"reference_ratio": round(reference_ratio, 3),
"reference_page": reference_page,
"appendix_title": any(_APPENDIX.match(line.strip()) for line in lines),
# Prose punctuation separates running text from a label-style cover page.
"prose_punctuation": any(char in text for char in _PROSE_PUNCTUATION),
# Lines that read as running text rather than headings, labels or dotted entries.
"prose_lines": [line for line in prose_source if _is_prose_line(line)],
}
def strip_repeated_lines(pages: list[dict]) -> set[str]:
"""Header/footer lines repeated across many pages, removed from searchable text."""
counts: dict[str, int] = {}
for page in pages:
lines = _lines(str(page.get("text") or ""))
if not lines:
continue
candidates = {lines[0][:120]}
if len(lines) > 1:
candidates.add(lines[-1][:60])
for candidate in candidates:
if candidate and not is_toc_entry(candidate):
counts[candidate] = counts.get(candidate, 0) + 1
threshold = max(3, int(len(pages) * 0.4 + 0.5))
return {line for line, count in counts.items() if count >= threshold}
def clean_page_lines(page: dict, repeated: set[str]) -> list[str]:
"""Lines kept for indexing: repeated headers/footers and bare page numbers removed."""
lines = _lines(str(page.get("text") or ""))
kept = []
for position, line in enumerate(lines):
if line in repeated and (position == 0 or position == len(lines) - 1):
continue
if _PAGE_NUMBER_LINE.match(line) and position == len(lines) - 1:
continue
kept.append(line)
return kept
def _is_prose_line(line: str) -> bool:
"""A line that reads as running text rather than a heading, label or dotted entry.
The syntactic heading check is used, not the contents-confirmed one: a heading that
the contents list happens not to mention is still a heading, so it must not be
counted as prose when deciding whether a page carries content.
"""
stripped = line.strip()
if is_toc_entry(stripped) or _TOC_TITLE.match(stripped) or heading_syntax(stripped):
return False
if _PAGE_NUMBER_LINE.match(stripped):
return False
return len(stripped) >= MIN_PROSE_CHARS
def split_blocks(lines: list[str]) -> list[dict]:
"""Split page lines into TOC and body blocks so a mixed page keeps both parts."""
blocks: list[dict] = []
current: dict | None = None
for line in lines:
stripped = line.strip()
# A contents title and its dotted entries are one navigation block. A lone
# dotted line with no title is not a table of contents, so it stays as body.
kind = BLOCK_TOC if is_toc_entry(line) or _TOC_TITLE.match(stripped) else BLOCK_BODY
if current is None or current["kind"] != kind:
current = {"kind": kind, "lines": []}
blocks.append(current)
current["lines"].append(line)
output = []
for block in blocks:
text = "\n".join(block["lines"]).strip()
if not text:
continue
has_entry = any(is_toc_entry(line) for line in block["lines"])
if block["kind"] == BLOCK_TOC and not has_entry:
# Only a title line, with no entries under it: this is a heading, not contents.
output.append({"kind": BLOCK_BODY, "text": text, "substantive": False})
continue
output.append({"kind": block["kind"], "text": text})
# A block made only of headings carries navigation, not content.
for block in output:
if block["kind"] == BLOCK_BODY:
block["substantive"] = (any(_is_prose_line(line) for line in block["text"].splitlines())
or len(block["text"]) >= MIN_BODY_CHARS)
else:
block["substantive"] = False
return output
def classifier_signals(pages: list[dict]) -> list[dict]:
"""Per-page signals, shared by classification and by audit output."""
total = len(pages)
return [_signals(page, index=position, total=total) for position, page in enumerate(pages, start=1)]
def _is_cover(signal: dict) -> bool:
"""A cover page carries identity fields, not prose.
The weak rule requires a first page of a document that actually has a body, with
label-like lines and no sentence punctuation. Without those guards a short
single-page document would be mistaken for a cover page and lose all its content.
"""
if signal["cover_fields"] >= 2 and signal["body_chars"] < 600:
return True
return (signal["position"] == 1 and signal["total"] >= 3 and signal["chars"] <= 400
and signal["lines"] <= 8 and signal["toc_entries"] == 0 and signal["headings"] == 0
and not signal["has_table"] and not signal["prose_punctuation"])
def _is_toc_page(signal: dict, *, previous_is_toc: bool) -> bool:
if signal["toc_entries"] >= 4 and signal["toc_ratio"] >= 0.4:
return True
if _TOC_TITLE.match(signal["first_line"]) and signal["toc_entries"] >= 2:
return True
# Continuation page: the previous page is a table of contents and this page is
# still almost nothing but dotted entries.
return bool(previous_is_toc and signal["toc_entries"] >= 2 and signal["toc_ratio"] >= 0.5)
def _looks_heading_only(signal: dict, blocks: list[dict]) -> bool:
"""A page that carries no running prose at all, so it proves no content.
The test is structural, not a length test: a short page with one real sentence is
body text, a page with a table carries data, while a page of nothing but headings
or labels cannot support a judgement about what was done.
"""
if signal["has_table"]:
return False
if any(block["kind"] == BLOCK_BODY and block.get("substantive") for block in blocks):
return False
return not signal["prose_lines"]
def classify_document(pages: list[dict]) -> list[dict]:
"""Classify every page and extract real section paths from body headings only."""
repeated = strip_repeated_lines(pages)
# The document's own contents list is the authority on which numbered lines are
# real headings; this keeps enumerated list items and equations out of the path.
toc_titles = set()
for page in pages:
for line in _lines(str(page.get("text") or "")):
if is_toc_entry(line):
title = normalize_heading(line)
if title:
toc_titles.add(title)
toc_titles = toc_titles or None
signals = [_signals(page, index=position, total=len(pages), toc_titles=toc_titles, repeated=repeated)
for position, page in enumerate(pages, start=1)]
results: list[dict] = []
previous = None
chapter = ""
section = ""
for position, page in enumerate(pages, start=1):
signal = signals[position - 1]
lines = clean_page_lines(page, repeated)
blocks = split_blocks(lines)
toc_blocks = [block for block in blocks if block["kind"] == BLOCK_TOC]
body_blocks = [block for block in blocks if block["kind"] == BLOCK_BODY and block["substantive"]]
first = signal["first_line"]
appendix = signal["appendix_title"]
if _TOC_TITLE.match(first) and signal["toc_entries"] >= 1:
page_type, confidence = "toc", 0.95
elif _ABSTRACT_TITLE.match(first) or (previous == "abstract" and signal["footer"] in ("roman", "none")
and signal["headings"] == 0 and signal["chars"] >= 100):
page_type, confidence = "abstract", 0.9
elif _REFERENCE_TITLE.match(first) or _ACK_TITLE.match(first):
page_type, confidence = "references", 0.9
elif _is_toc_page(signal, previous_is_toc=(previous == "toc")):
page_type, confidence = "toc", 0.85 if signal["toc_ratio"] >= 0.4 else 0.75
elif previous == "references" and signal["headings"] == 0 and signal["toc_entries"] == 0:
page_type, confidence = "references", 0.7
elif signal["reference_page"] and signal["toc_entries"] == 0 and signal["chapter_headings"] <= 1:
page_type, confidence = "references", 0.8
elif appendix:
# An appendix carries substantive material, so it counts as content, but it
# is labelled separately so the interface can show where a quotation came from.
page_type, confidence = "appendix", 0.8
elif _is_cover(signal):
# Checked before the heading-only rule: a cover page carries identity fields
# and no prose, so it would otherwise be mistaken for an unsettled page.
page_type, confidence = "cover", 0.8
elif _looks_heading_only(signal, blocks) and signal["toc_entries"] == 0:
# A body page that carries only a section title cannot support a content
# judgement on its own, so it is not silently treated as ordinary body text.
page_type, confidence = "unknown", 0.5
else:
page_type, confidence = "body", 0.6
confidence = 0.85 if body_blocks and signal["toc_entries"] == 0 else confidence
mixed = page_type == "toc" and bool(body_blocks)
if mixed:
confidence = 0.7
if page_type in ("body", "appendix", "unknown") or mixed:
# Structure-extracted headings come from body text only, never from a TOC.
for block in body_blocks:
for line in block["text"].splitlines():
found = heading_level(line, toc_titles)
if not found:
continue
level, title = found
if level == 1:
chapter, section = title, title
else:
section = title
# A table of contents may only name the chapter path; the body supplies the
# section wording. Navigation pages therefore carry the chapter, not a section.
section_for_page = section if page_type in ("body", "unknown") or mixed else chapter
results.append({
"page": position,
"page_type": page_type,
"mixed": mixed,
"chapter": chapter,
"section": section_for_page,
"blocks": blocks,
"confidence": round(confidence, 2),
"signals": {**signal, "repeated_line_count": len(repeated), "toc_blocks": len(toc_blocks)},
"classifier_version": CLASSIFIER_VERSION,
})
previous = page_type
return results
def block_text(blocks: list[dict], *, kinds: tuple[str, ...] = (BLOCK_BODY,)) -> str:
return "\n".join(block["text"] for block in blocks if block["kind"] in kinds).strip()
def toc_entries(blocks: list[dict]) -> list[str]:
"""Heading-shaped entries read out of a table of contents page."""
entries = []
for block in blocks:
if block["kind"] != BLOCK_TOC:
continue
for line in block["text"].splitlines():
if is_toc_entry(line):
entries.append(line)
return entries
def searchable_pages(pages: list[dict]) -> list[dict]:
"""Pages prepared for indexing: cleaned lines, typed blocks and section paths."""
classification = classify_document(pages)
output = []
for page, item in zip(pages, classification):
body = block_text(item["blocks"])
output.append({
"page_number": page.get("page_number"),
"section": item["section"] or page.get("section", ""),
"chapter": item["chapter"],
"text": item.get("text", ""),
"tables": page.get("tables", []),
"rendered_image_path": page.get("rendered_image_path"),
"page_type": item["page_type"],
"blocks": item["blocks"],
"classification": item,
"index_text": body,
})
return output
+361 -38
View File
@@ -1,31 +1,130 @@
"""Indexing and hybrid retrieval with page-type aware candidate filtering.
Chinese keyword recall does not use PostgreSQL ``to_tsvector('simple', …)``. That
configuration has no Chinese segmenter, so a whole run of Chinese characters becomes a
single token; measured on the course reports, the query ``文献调研`` does not match the
page that contains ``第2章文献调研与技术现状``. Recall therefore uses character bigrams
plus whole ASCII/number tokens, which needs no extension and behaves the same on
PostgreSQL and SQLite.
Semantic and keyword scores are fused by rank (reciprocal rank fusion), never by adding
raw numbers: a cosine similarity and a bigram overlap are not on a common scale.
"""
from functools import lru_cache from functools import lru_cache
from pathlib import Path from pathlib import Path
import re
from sqlalchemy import cast, delete, func, or_, select from sqlalchemy import case, cast, delete, func, literal, or_, select
from .config import get_settings from .config import get_settings
from .knowledge_pages import page_text, relative_image_path
from .models import KnowledgeChunk, KnowledgeDocument, KnowledgePage from .models import KnowledgeChunk, KnowledgeDocument, KnowledgePage
from .pages import (BLOCK_BODY, BLOCK_TOC, classify_document, is_toc_entry,
normalize_heading, searchable_pages)
INDEX_VERSION = 3
#: Fusion constant for reciprocal rank fusion; the usual value from the IR literature.
RRF_K = 60
#: Candidate pools are gathered independently so both retrievers get a chance.
SEMANTIC_POOL = 60
KEYWORD_POOL = 60
#: A chunk must be at least this long to count as substantive content evidence.
MIN_CHUNK_CHARS = 60
#: At most this many chunks from one page in the first retrieval pass.
DEFAULT_PAGE_LIMIT = 2
#: Soft cap on chunks from one section once a reliable section path exists.
DEFAULT_SECTION_CAP = 3
#: Maximum chunks handed to the model for one criterion, across all targets.
SCORE_EVIDENCE_LIMIT = 14
_ASCII_TOKEN = re.compile(r"[A-Za-z][A-Za-z0-9_+\-.]*|\d+(?:\.\d+)*")
_CJK_RUN = re.compile(r"[㐀-䶿一-鿿]+")
INDEX_VERSION = 2 def query_terms(query: str) -> tuple[list[str], list[str]]:
"""Split a query into (ascii tokens lowercased, CJK character bigrams).
Single-character CJK runs are dropped: they carry too little signal and would make
every chunk with a common character look relevant.
"""
value = str(query or "")
ascii_tokens = [token.lower() for token in _ASCII_TOKEN.findall(value)]
bigrams: list[str] = []
seen = set()
for run in _CJK_RUN.findall(value):
for index in range(len(run) - 1):
bigram = run[index:index + 2]
if bigram not in seen:
seen.add(bigram)
bigrams.append(bigram)
return sorted(set(ascii_tokens)), bigrams
def _keyword_query(db, filters, query: str, limit: int):
"""Keyword recall ranked by bigram/token overlap over the already-filtered rows.
PostgreSQL's ``to_tsvector('simple', …)`` cannot segment Chinese, so it is not used
for recall. The overlap is computed with ``strpos``/``substr``/``generate_series``
so the same expression works on every dialect SQLAlchemy targets here.
"""
ascii_tokens, bigrams = query_terms(query)
if not ascii_tokens and not bigrams:
return []
total = len(ascii_tokens) + len(bigrams)
score = None
for token in ascii_tokens:
hit = case((func.strpos(func.lower(KnowledgeChunk.text), literal(token)) > 0, 1.0), else_=0.0)
score = hit if score is None else score + hit
if bigrams:
series = select(func.generate_series(1, func.greatest(func.length(KnowledgeChunk.text) - 1, 0)).label("i")).subquery("series")
matches = (select(func.count()).select_from(series)
.where(func.substr(KnowledgeChunk.text, series.c.i, 2).in_(bigrams)).scalar_subquery())
score = matches if score is None else score + matches
score = (score / float(total)).label("keyword")
rows = db.execute(select(KnowledgeChunk.id, score).where(*filters)
.order_by(score.desc(), KnowledgeChunk.id.asc()).limit(limit)).all()
return [row[0] for row in rows if float(row[1] or 0) > 0]
def split_pages(pages: list[dict], *, size=650, overlap=100): def split_pages(pages: list[dict], *, size=650, overlap=100):
"""Split pages into searchable chunks carrying page type and section path."""
for page in pages: for page in pages:
text = page.get("text", "").strip() page_type = page.get("page_type") or "body"
blocks = [text] if text else [] chapter = page.get("chapter", "")
for table in page.get("tables", []): section = page.get("section", "")
blocks.append("表格: " + str(table.get("rows", table))) blocks = page.get("blocks")
if not blocks:
blocks = [{"kind": BLOCK_BODY, "text": str(page.get("text") or "").strip()}]
blocks.extend({"kind": BLOCK_BODY, "text": "表格: " + str(table.get("rows", table))}
for table in page.get("tables", []))
for block in blocks: for block in blocks:
if block["kind"] == BLOCK_TOC:
# A contents page is navigation, so its text is not body content.
yield {"page": page["page_number"], "page_type": "toc", "block_type": BLOCK_TOC,
"chapter": chapter, "section": section, "text": block["text"]}
continue
text_block = block["text"].strip()
if not text_block:
continue
for table in ([block["tables"]] if block.get("tables") else []):
for table_item in table:
text_block += "\n表格: " + str(table_item.get("rows", table_item))
# Table rows of a page are appended once, on the page's first body chunk.
start = 0 start = 0
while start < len(block): while start < len(text_block):
yield {"page": page["page_number"], "section": page.get("section", ""), "text": block[start:start + size]} yield {"page": page["page_number"], "page_type": page_type, "block_type": BLOCK_BODY,
if start + size >= len(block): "chapter": chapter, "section": section, "text": text_block[start:start + size]}
if start + size >= len(text_block):
break break
start += size - overlap start += size - overlap
def build_chunks(pages: list[dict]) -> list[dict]:
"""Chunks for one document, classifying pages first when they are not yet typed."""
prepared = pages if pages and "page_type" in (pages[0] or {}) else searchable_pages(pages)
return list(split_pages(prepared))
@lru_cache(maxsize=2) @lru_cache(maxsize=2)
def local_encoder(model_path: str): def local_encoder(model_path: str):
path = Path(model_path) path = Path(model_path)
@@ -56,7 +155,8 @@ def ingest(db, *, course_version_id: int, project_id: int | None, submission_fil
KnowledgeDocument.title == title)) KnowledgeDocument.title == title))
if existing and existing.source_hash == source_hash and existing.index_version == INDEX_VERSION and existing.status == "ready": if existing and existing.source_hash == source_hash and existing.index_version == INDEX_VERSION and existing.status == "ready":
return existing return existing
chunks = list(split_pages(pages)) prepared = searchable_pages(pages)
chunks = list(split_pages(prepared))
vectors = embedder([chunk["text"] for chunk in chunks]) vectors = embedder([chunk["text"] for chunk in chunks])
if len(vectors) != len(chunks): if len(vectors) != len(chunks):
raise RuntimeError("嵌入向量数量与知识片段数量不一致") raise RuntimeError("嵌入向量数量与知识片段数量不一致")
@@ -71,42 +171,265 @@ def ingest(db, *, course_version_id: int, project_id: int | None, submission_fil
db.add(doc) db.add(doc)
doc.index_version, doc.status = INDEX_VERSION, "indexing" doc.index_version, doc.status = INDEX_VERSION, "indexing"
db.flush() db.flush()
for source in pages: for source, prepared_page in zip(pages, prepared):
db.add(KnowledgePage(document_id=doc.id, page=source["page_number"], text=page_text(source), classification = prepared_page["classification"]
image_path=relative_image_path(source.get("rendered_image_path")))) db.add(KnowledgePage(
document_id=doc.id, page=source["page_number"], text=page_text(source),
image_path=relative_image_path(source.get("rendered_image_path")),
page_type=classification["page_type"], mixed=bool(classification["mixed"]),
section=classification["section"], chapter=classification["chapter"],
confidence=classification["confidence"],
classification={**classification["signals"], "classifier_version": classification["classifier_version"],
"blocks": [{"kind": block["kind"], "chars": len(block["text"]),
"substantive": block.get("substantive", False)}
for block in classification["blocks"]]}))
for chunk, vector in zip(chunks, vectors): for chunk, vector in zip(chunks, vectors):
db.add(KnowledgeChunk(document_id=doc.id, course_version_id=course_version_id, project_id=project_id, db.add(KnowledgeChunk(document_id=doc.id, course_version_id=course_version_id, project_id=project_id,
document_type=category, page=chunk["page"], section=chunk["section"], text=chunk["text"], embedding=vector)) document_type=category, page=chunk["page"], section=chunk["section"],
chapter=chunk["chapter"], page_type=chunk["page_type"],
block_type=chunk["block_type"], text=chunk["text"], embedding=vector))
doc.status = "ready" doc.status = "ready"
db.commit() db.commit()
return doc return doc
def retrieve(db, *, course_version_id: int, project_id: int, query: str, category: str | None = None, def _scope_filter(project_id: int | None, include_public: bool, public_only: bool):
limit=10, embedder=embed, include_public=True, public_only=False): if public_only:
"""Hybrid retrieval with project isolation and an optional public course namespace.""" return KnowledgeChunk.project_id.is_(None)
scope = (KnowledgeChunk.project_id.is_(None) if public_only else if include_public:
or_(KnowledgeChunk.project_id == project_id, KnowledgeChunk.project_id.is_(None)) if include_public else return or_(KnowledgeChunk.project_id == project_id, KnowledgeChunk.project_id.is_(None))
KnowledgeChunk.project_id == project_id) return KnowledgeChunk.project_id == project_id
filters = [KnowledgeChunk.course_version_id == course_version_id, scope]
def _base_filters(*, course_version_id: int, project_id: int | None, category, categories,
include_public: bool, public_only: bool, allowed_page_types, exclude_document_ids):
filters = [KnowledgeChunk.course_version_id == course_version_id,
_scope_filter(project_id, include_public, public_only)]
if category: if category:
filters.append(KnowledgeChunk.document_type == category) filters.append(KnowledgeChunk.document_type == category)
vector = embedder([query])[0] if categories:
filters.append(KnowledgeChunk.document_type.in_(list(categories)))
if allowed_page_types is not None:
filters.append(KnowledgeChunk.page_type.in_(list(allowed_page_types)))
if exclude_document_ids:
filters.append(KnowledgeChunk.document_id.not_in(list(exclude_document_ids)))
return filters
def _keyword_query(db, filters, query: str, limit: int):
"""Keyword recall ranked by bigram/token overlap over the already-filtered rows.
PostgreSQL's ``to_tsvector('simple', …)`` cannot segment Chinese, so it is not used
for recall. The overlap is computed with ``strpos``/``substr`` so the same expression
works on every dialect SQLAlchemy targets here.
"""
ascii_tokens, bigrams = query_terms(query)
if not ascii_tokens and not bigrams:
return []
total = len(ascii_tokens) + len(bigrams)
score = None
for token in ascii_tokens:
term = literal(token)
hit = case((func.strpos(func.lower(KnowledgeChunk.text), term) > 0, 1.0), else_=0.0)
score = hit if score is None else score + hit
if bigrams:
start = func.generate_series(1, func.greatest(func.length(KnowledgeChunk.text) - 1, 0))
series = select(start.label("i")).subquery("series")
matches = (select(func.count()).select_from(series)
.where(func.substr(KnowledgeChunk.text, series.c.i, 2).in_(bigrams))
.scalar_subquery())
score = matches if score is None else score + matches
score = (score / float(total)).label("keyword")
rows = db.execute(select(KnowledgeChunk.id, score).where(*filters)
.order_by(score.desc(), KnowledgeChunk.id.asc()).limit(limit)).all()
return [row[0] for row in rows if float(row[1] or 0) > 0]
def keyword_overlap(text_value: str, ascii_tokens: list[str], bigrams: list[str]) -> float:
"""Bigram/token overlap used by the non-PostgreSQL path and by tests."""
value = str(text_value or "")
lower = value.lower()
hits = sum(1 for token in ascii_tokens if token in lower)
total = len(ascii_tokens)
hits += sum(1 for bigram in bigrams if bigram in value)
total += len(bigrams)
return hits / total if total else 0.0
def _semantic_query(db, filters, vector, limit: int):
if db.bind.dialect.name == "postgresql": if db.bind.dialect.name == "postgresql":
from pgvector.sqlalchemy import Vector from pgvector.sqlalchemy import Vector
semantic = (1 - cast(KnowledgeChunk.embedding, Vector(1024)).cosine_distance(vector)).label("semantic") distance = cast(KnowledgeChunk.embedding, Vector(1024)).cosine_distance(vector)
lexical = func.ts_rank(func.to_tsvector("simple", KnowledgeChunk.text), func.plainto_tsquery("simple", query)).label("lexical") rows = db.execute(select(KnowledgeChunk.id).where(*filters)
rows = db.execute(select(KnowledgeChunk, semantic, lexical).where(*filters).order_by(semantic.desc(), lexical.desc()).limit(limit * 3)).all() .order_by(distance.asc(), KnowledgeChunk.id.asc()).limit(limit)).all()
ranked = [(row[0], float(row.semantic or 0) * .7 + float(row.lexical or 0) * .3) for row in rows] return [row[0] for row in rows]
else: rows = db.execute(select(KnowledgeChunk.id, KnowledgeChunk.embedding).where(*filters)).all()
chunks = db.scalars(select(KnowledgeChunk).where(*filters)).all() scored = [(chunk_id, sum(a * b for a, b in zip(embedding or [], vector)))
terms = [query[i:i+2] for i in range(max(0, len(query)-1)) if query[i:i+2].strip()] for chunk_id, embedding in rows]
ranked = [] scored.sort(key=lambda pair: (-pair[1], pair[0]))
for chunk in chunks: return [chunk_id for chunk_id, _ in scored[:limit]]
semantic = sum(a*b for a, b in zip(chunk.embedding or [], vector))
lexical = sum(term in chunk.text for term in terms) / max(len(terms), 1)
ranked.append((chunk, semantic * .7 + lexical * .3)) def _fuse(db, ids: list[int], semantic_ids: list[int], keyword_ids: list[int]):
ranked.sort(key=lambda pair: pair[1], reverse=True) """Reciprocal rank fusion over the two ranked lists, keeping discovery order stable."""
return [{"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id, scores: dict[int, float] = {chunk_id: 0.0 for chunk_id in ids}
for ranked in (semantic_ids, keyword_ids):
for rank, chunk_id in enumerate(ranked):
if chunk_id in scores:
scores[chunk_id] += 1.0 / (RRF_K + rank + 1)
return scores
def _row_dict(chunk):
return {"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id,
"document_type": chunk.document_type, "page": chunk.page, "section": chunk.section, "document_type": chunk.document_type, "page": chunk.page, "section": chunk.section,
"text": chunk.text, "rank": float(rank)} for chunk, rank in ranked[:limit]] "chapter": chunk.chapter, "page_type": chunk.page_type, "block_type": chunk.block_type,
"text": chunk.text}
def normalize_hit(hit: dict) -> dict:
"""Fill the fields the rest of the pipeline reads, so one shape flows through.
An alternate retriever or a test double may return only the essentials. Filling the
defaults here keeps every consumer from re-deriving them; the authoritative page type
is still re-read from storage when a score is produced.
"""
return {"chunk_id": hit.get("chunk_id"), "document_id": hit.get("document_id"),
"project_id": hit.get("project_id"), "document_type": hit.get("document_type"),
"page": hit.get("page"), "text": hit.get("text") or "",
"section": hit.get("section") or "", "chapter": hit.get("chapter") or "",
"page_type": hit.get("page_type") or "unknown", "block_type": hit.get("block_type") or "legacy",
"rank": float(hit.get("rank") or 0.0)}
def retrieve(db, *, course_version_id: int, project_id: int, query: str, category: str | None = None,
categories=None, limit=10, embedder=embed, include_public=True, public_only=False,
allowed_page_types=None, exclude_document_ids=(), substantive_only=False,
semantic_pool: int | None = None, keyword_pool: int | None = None):
"""Filtered hybrid retrieval, then rank fusion over the two independent recalls.
Page filtering happens inside the query, so it always precedes the candidate limit:
a contents page can never displace a body page simply by scoring well.
"""
filters = _base_filters(course_version_id=course_version_id, project_id=project_id, category=category,
categories=categories, include_public=include_public, public_only=public_only,
allowed_page_types=allowed_page_types, exclude_document_ids=exclude_document_ids)
if substantive_only:
filters.append(KnowledgeChunk.block_type != BLOCK_TOC)
vector = embedder([query])[0]
semantic_ids = _semantic_query(db, filters, vector, semantic_pool or SEMANTIC_POOL)
keyword_ids = _keyword_query(db, filters, query, keyword_pool or KEYWORD_POOL)
ids = list(dict.fromkeys(semantic_ids + keyword_ids))
if not ids:
return []
scores = _fuse(db, ids, semantic_ids, keyword_ids)
chunks = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.id.in_(ids))).all()
usable = []
for chunk in chunks:
if substantive_only and (chunk.block_type == BLOCK_TOC
or (chunk.page_type == "body" and is_toc_entry(chunk.text)
and len(chunk.text.strip()) < MIN_CHUNK_CHARS)):
continue
usable.append(chunk)
usable.sort(key=lambda chunk: (-scores.get(chunk.id, 0.0), chunk.id))
return [{**_row_dict(chunk), "rank": float(scores.get(chunk.id, 0.0))} for chunk in usable[:limit]]
def _page_diverse(chunks: list[dict], limit: int, *, page_limit: int, section_cap: int | None):
"""Spread the first pass over pages, then allow a bounded number per section."""
picked: list[dict] = []
per_page: dict[tuple, int] = {}
per_section: dict[str, int] = {}
remaining = []
for chunk in chunks:
page_key = (chunk.get("document_id"), chunk.get("page"))
section = str(chunk.get("section") or "")
if len(picked) < limit and per_page.get(page_key, 0) < page_limit:
if section_cap and section and per_section.get(section, 0) >= section_cap:
remaining.append(chunk)
continue
picked.append(chunk)
per_page[page_key] = per_page.get(page_key, 0) + 1
if section:
per_section[section] = per_section.get(section, 0) + 1
else:
remaining.append(chunk)
for chunk in remaining:
if len(picked) >= limit:
break
picked.append(chunk)
return picked[:limit]
def retrieve_targets(db, *, course_version_id: int, project_id: int, targets: list[dict], category=None,
categories=None, limit=SCORE_EVIDENCE_LIMIT, embedder=embed, include_public=False,
page_limit=DEFAULT_PAGE_LIMIT, section_cap=DEFAULT_SECTION_CAP,
per_target_limit: int | None = None, navigator=None, retrieve_fn=retrieve):
"""Search once per required-evidence target and merge with page-level diversity.
Every target is searched separately with its own query and its own allowed page
types, so a contents page permitted for a structural target can never flood a
content target's evidence. A target counts as unmet when its *own* search found
nothing admissible, which is what decides a teacher review.
"""
merged: list[dict] = []
seen: set[int] = set()
per_target: dict[str, list[dict]] = {}
for target in targets:
allowed = target.get("allowed_page_types")
raw_hits = retrieve_fn(db, course_version_id=course_version_id, project_id=project_id, category=category,
categories=categories, query=target.get("query") or target.get("label") or "",
limit=per_target_limit or max(limit, 8), embedder=embed,
include_public=include_public, allowed_page_types=allowed,
substantive_only=target.get("use") != "structure",
semantic_pool=SEMANTIC_POOL, keyword_pool=KEYWORD_POOL)
hits = [normalize_hit(hit) for hit in (raw_hits or [])]
if target.get("use") == "content" and not hits and navigator is not None:
# The contents list may only be used to navigate: its headings are matched
# against headings that really occur in body text, and those sections and
# their neighbouring pages are searched. The quotation still lands on body.
hits = [dict(hit, via="toc_navigation") for hit in (navigator(target) or [])]
per_target[target["target_id"]] = hits
for hit in hits:
if hit["chunk_id"] not in seen:
seen.add(hit["chunk_id"])
merged.append({**hit, "target_id": target["target_id"], "target_label": target["label"],
"target_use": target["use"]})
merged.sort(key=lambda chunk: -chunk["rank"])
selected = _page_diverse(merged, limit, page_limit=page_limit, section_cap=section_cap)
uncovered = [target for target in targets if not per_target.get(target["target_id"])]
return {"evidence": selected, "per_target": per_target, "uncovered_targets": uncovered}
def body_heading_index(db, *, course_version_id: int, project_id: int, category: str | None = None):
"""Sections present in real body text, used to navigate from a contents page."""
filters = [KnowledgeChunk.course_version_id == course_version_id, KnowledgeChunk.project_id == project_id,
KnowledgeChunk.page_type == "body"]
if category:
filters.append(KnowledgeChunk.document_type == category)
rows = db.execute(select(KnowledgeChunk.document_id, KnowledgeChunk.page, KnowledgeChunk.section,
KnowledgeChunk.chapter).where(*filters).distinct()).all()
index: dict[str, list[dict]] = {}
for document_id, page, section, chapter in rows:
for value in (section, chapter):
key = normalize_heading(value or "")
if key:
index.setdefault(key, []).append({"document_id": document_id, "page": page,
"section": section, "chapter": chapter})
return index
def page_text(source: dict) -> str:
blocks = [str(source.get("text") or "").strip()]
blocks.extend("表格: " + str(table.get("rows", table)) for table in source.get("tables", []))
return "\n".join(block for block in blocks if block)
def relative_image_path(value: str | None) -> str | None:
if not value:
return None
root = get_settings().processed_path.resolve()
path = Path(value).resolve()
if not path.is_relative_to(root):
raise ValueError("页面图像路径不在处理目录内")
return str(path.relative_to(root))
+234 -34
View File
@@ -6,15 +6,30 @@ from sqlalchemy import select, update
from .config import get_settings from .config import get_settings
from .db import SessionLocal from .db import SessionLocal
from .evidence import USE_CONTENT, content_targets, normalize_targets
from .knowledge_pages import page_image_path from .knowledge_pages import page_image_path
from .llm import ConsistencyResult, FactResult, ModelError, ScoreResult, validate_score from .llm import ConsistencyResult, FactResult, ModelError, ScoreResult, validate_score
from .models import (CriterionScore, KnowledgeDocument, Project, ScoreEvidence, ScoringBatch, from .models import (CriterionScore, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project,
SubmissionFile, now) ScoreEvidence, ScoringBatch, SubmissionFile, now)
from .pages import normalize_heading
from .parsing import parse_document from .parsing import parse_document
from .rag import INDEX_VERSION, ingest, retrieve from .rag import (INDEX_VERSION, body_heading_index, ingest, retrieve, retrieve_targets)
from .storage import secure_path, sha256 from .storage import secure_path, sha256
from .settings_store import client_for_batch from .settings_store import client_for_batch
#: Criterion statuses. ``needs_review`` is deliberately distinct from a missing
#: material, a parse failure and a model failure: the retrieval ran but only found
#: navigation pages where the rule requires body content.
STATUS_NEEDS_REVIEW = "needs_review"
STATUS_UNSCORABLE = "unscorable"
STATUS_FAILED = "failed"
STATUS_SCORED = "scored"
STATUS_REVIEWED = "reviewed"
#: Statuses that a batch counts as finished work.
DONE_STATUSES = (STATUS_SCORED, STATUS_REVIEWED, STATUS_NEEDS_REVIEW)
#: Statuses the dashboard shows as needing teacher attention.
ATTENTION_STATUSES = (STATUS_FAILED, STATUS_UNSCORABLE, STATUS_NEEDS_REVIEW)
@contextmanager @contextmanager
def file_index_lock(file_id: int): def file_index_lock(file_id: int):
@@ -72,12 +87,21 @@ def _parse_submission_locked(db, file: SubmissionFile, *, embedder=None):
def _project_facts(db, project, course_snapshot, client, *, retriever): def _project_facts(db, project, course_snapshot, client, *, retriever):
"""Extract project facts from body text only.
Implementation facts must not be inferred from a contents heading or an abstract
claim, so the search admits body and appendix pages exclusively. Project identity
(name, student number, topic) legitimately lives on the cover page and is read from
the teacher-confirmed import record instead of from retrieved text.
"""
prompts = course_snapshot["prompts"] prompts = course_snapshot["prompts"]
categories = ("stage1", "stage2", "stage3", "final_report", "presentation") categories = ("stage1", "stage2", "stage3", "final_report", "presentation")
snippets = [] snippets = []
for category in categories: for category in categories:
snippets.extend(retriever(db, course_version_id=project.course_version_id, project_id=project.id, snippets.extend(retriever(db, course_version_id=project.course_version_id, project_id=project.id,
category=category, query="项目名称 目标 指标 硬件 软件 实验 数据 成本 结论", limit=3, include_public=False)) category=category, query="项目名称 目标 指标 硬件 软件 实验 数据 成本 结论",
limit=3, include_public=False, allowed_page_types=["body", "appendix"],
substantive_only=True))
if not snippets: if not snippets:
return {} return {}
if client is None: if client is None:
@@ -100,10 +124,54 @@ def _project_facts(db, project, course_snapshot, client, *, retriever):
return project.facts return project.facts
def _score_criterion(db, batch, project, criterion, client, *, retriever): def _body_navigator(db, *, course_version_id, project_id, category):
"""Find real body pages for a target whose contents entries name missing sections.
The contents page is used only as a map: its headings are matched against headings
that actually occur in body text, and the body sections and their neighbouring pages
are then searched. The final quotation always lands on a real body page.
"""
headings = body_heading_index(db, course_version_id=course_version_id, project_id=project_id, category=category)
if not headings:
return None
def navigate(target):
label = normalize_heading(target.get("label") or "")
if not label:
return []
candidates = []
for heading, places in headings.items():
if not heading:
continue
if heading.startswith(label) or label.startswith(heading) or label in heading or heading in label:
candidates.extend(places)
if not candidates:
return []
pages = sorted({(place["document_id"], place["page"]) for place in candidates})
neighbours = set()
for document_id, page in pages:
neighbours.update({(document_id, page), (document_id, page + 1)})
wanted = sorted(neighbours)
rows = db.scalars(select(KnowledgeChunk).where(
KnowledgeChunk.course_version_id == course_version_id, KnowledgeChunk.project_id == project_id,
KnowledgeChunk.page_type == "body", KnowledgeChunk.block_type != "toc",
KnowledgeChunk.document_id.in_([item[0] for item in wanted]),
KnowledgeChunk.page.in_([item[1] for item in wanted])).order_by(KnowledgeChunk.id)).all()
hits = []
for chunk in rows:
hits.append({"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id,
"document_type": chunk.document_type, "page": chunk.page, "section": chunk.section,
"chapter": chunk.chapter, "page_type": chunk.page_type, "block_type": chunk.block_type,
"text": chunk.text, "rank": 0.5})
return hits
return navigate
def _score_criterion(db, batch, project, criterion, client, *, retriever, multi_retriever=retrieve_targets):
score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id, score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
CriterionScore.project_id == project.id, CriterionScore.criterion_id == criterion["criterion_id"])) CriterionScore.project_id == project.id, CriterionScore.criterion_id == criterion["criterion_id"]))
if score and score.status in ("scored", "reviewed"): if score and score.status in (STATUS_SCORED, STATUS_REVIEWED):
return score return score
if not score: if not score:
score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"]) score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"])
@@ -115,44 +183,103 @@ def _score_criterion(db, batch, project, criterion, client, *, retriever):
SubmissionFile.category == category, SubmissionFile.recognition_status == "recognized")).all() SubmissionFile.category == category, SubmissionFile.recognition_status == "recognized")).all()
ready = [file for file in files if file.parse_status == "ready"] ready = [file for file in files if file.parse_status == "ready"]
if len(files) != 1 or len(ready) != 1: if len(files) != 1 or len(ready) != 1:
score.status, score.error = "unscorable", "材料缺失、重复、待确认或解析失败" score.status, score.error = STATUS_UNSCORABLE, "材料缺失、重复、待确认或解析失败"
db.commit() db.commit()
return score return score
query = criterion["description"] + " " + " ".join(criterion["required_evidence"])
chunks = retriever(db, course_version_id=batch.course_version_id, project_id=project.id, plan = batch.rubric_snapshot.get("evidence_plan") or {}
category=category, query=query, limit=10, include_public=False) targets = (plan.get("criteria") or {}).get(criterion["criterion_id"]) or normalize_targets(
criterion.get("required_evidence"))
navigator = _body_navigator(db, course_version_id=batch.course_version_id,
project_id=project.id, category=category)
# The single ``retriever`` stays the seam: each per-target search goes through it,
# so a caller that injects a retriever (tests, alternate index) still applies.
selection = multi_retriever(db, course_version_id=batch.course_version_id, project_id=project.id,
targets=targets, category=category, navigator=navigator,
retrieve_fn=retriever)
chunks = _attach_page_types(db, selection["evidence"])
gaps = selection["uncovered_targets"]
if not chunks: if not chunks:
score.status, score.error = "unscorable", "无可检索的材料证据" # Nothing admissible was found for any target. Content targets go to review;
# a purely structural criterion only depends on navigation pages, which exist.
score.status = STATUS_NEEDS_REVIEW if content_targets(criterion) else STATUS_UNSCORABLE
score.error = "未检索到允许作为证据的正文页"
score.model_score = None
score.missing_evidence = [target["label"] for target in gaps]
_record_evidence_gaps(score, gaps)
db.commit() db.commit()
return score return score
if client is None: if client is None:
raise ModelError("未配置或无法连接评分模型") raise ModelError("未配置或无法连接评分模型")
image_sources = dict.fromkeys((chunk["document_id"], chunk["page"]) for chunk in chunks[:2])
content_gaps = [target for target in gaps if target["use"] == USE_CONTENT]
if content_gaps:
# The model is not asked to grade from navigation pages. Record the gap and keep
# the model score null so a teacher decides, rather than asserting a low score.
score.model_score, score.confidence, score.status = None, None, STATUS_NEEDS_REVIEW
score.model_comment = "检索仅命中目录、摘要或章节标题,缺少可用于内容判断的正文证据,转教师复核。"
score.missing_evidence = [target["label"] for target in content_gaps]
score.suggestion = "请补充该章节的正文页,或由教师直接核定该项分数。"
score.error = "缺少正文证据:" + "、".join(target["label"] for target in content_gaps)
_record_evidence_gaps(score, content_gaps, searched=_searched_page_types(plan, content_gaps))
db.commit()
return score
# Page images go to the model as well, so prefer substantive body pages over the
# navigation pages that may legitimately appear in a structural target's evidence.
image_sources = list(dict.fromkeys(
(chunk["document_id"], chunk["page"]) for chunk in chunks
if chunk.get("page_type") == "body"))
if not image_sources:
image_sources = list(dict.fromkeys((chunk["document_id"], chunk["page"]) for chunk in chunks))
images = [] images = []
for document_id, page in image_sources: for document_id, page in image_sources[:3]:
doc = db.get(KnowledgeDocument, document_id) doc = db.get(KnowledgeDocument, document_id)
path = page_image_path(db, doc, page) if doc else None path = page_image_path(db, doc, page) if doc else None
if path: if path:
images.append(str(path)) images.append(str(path))
query = criterion["description"] + " " + " ".join(target["label"] for target in targets)
public_context = retriever(db, course_version_id=batch.course_version_id, project_id=project.id, public_context = retriever(db, course_version_id=batch.course_version_id, project_id=project.id,
category=None, query=query, limit=6, public_only=True) category=None, query=query, limit=6, public_only=True)
payload = {"criterion": criterion, "project_facts": project.facts, "course_context": public_context, payload = {"criterion": criterion,
"evidence": chunks, "required_evidence_targets": [{"target_id": target["target_id"], "label": target["label"],
"required_output": {"criterion_id": "string", "score": "number", "max_score": "number", "reason": "string", "use": target["use"],
"evidence": [{"chunk_id": "integer", "page": "integer", "quote": "exact substring"}], "allowed_page_types": target["allowed_page_types"]}
"missing_evidence": ["string"], "suggestion": "string", "confidence": "0..1"}} for target in targets],
"evidence_policy": {
"structure": "目录、封面和参考文献页可以证明章节结构和报告完整性,不能证明章节内容或完成质量。",
"content": "内容判断只能使用正文或附录中的实质段落、数据、表格;目录、摘要和单独章节标题不能作为内容证据。"},
"project_facts": project.facts, "course_context": public_context, "evidence": chunks,
"required_output": {"criterion_id": "string", "score": "number or null", "max_score": "number",
"reason": "string",
"evidence": [{"chunk_id": "integer", "page": "integer",
"quote": "exact substring", "target_id": "string"}],
"missing_evidence": ["string"], "suggestion": "string", "confidence": "0..1",
"note": "证据不足以判断内容时 score 填 null,并在 missing_evidence 中列出缺少的正文目标。"}}
prompt = (batch.course_snapshot["prompts"]["score"] + prompt = (batch.course_snapshot["prompts"]["score"] +
" 证据quote必须从所引用chunk_id的text逐字复制连续片段,保留原文标点和换行;page必须等于该片段的page。不要改写、缩写或拼接引文。") " 证据quote必须从所引用chunk_id的text逐字复制连续片段,保留原文标点和换行;page必须等于该片段的page;"
"target_id必须取自required_evidence_targets。不要改写、缩写或拼接引文。"
"不得把目录、摘要或章节标题当作内容证据;无法据此判断内容时把score设为null。")
error = ""
result = None
for attempt in range(2): for attempt in range(2):
try: try:
result = validate_score(client.json_call(prompt, payload, ScoreResult, images=images), criterion, chunks) result = validate_score(client.json_call(prompt, payload, ScoreResult, images=images), criterion, chunks,
targets=targets)
break break
except ModelError: except ModelError as exc:
error = str(exc)
if attempt: if attempt:
raise raise
prompt += " 上次回答的证据未通过原文和页码校验;请重新选择片段并逐字复制quote。" prompt += f" 上次回答未通过校验({error});请改用允许页面类型中的实质正文片段,并逐字复制quote。"
score.model_score, score.model_comment, score.confidence = result.score, result.reason, result.confidence score.model_score, score.model_comment = result.score, result.reason
score.missing_evidence, score.suggestion, score.status, score.error = result.missing_evidence, result.suggestion, "scored", "" score.confidence = result.confidence
score.missing_evidence, score.suggestion = result.missing_evidence, result.suggestion
if result.score is None:
# The retrieval found body pages but the model judged them insufficient.
score.status, score.error = STATUS_NEEDS_REVIEW, "模型认为现有正文证据不足以判断该项,转教师复核。"
else:
score.status, score.error = STATUS_SCORED, ""
for evidence in db.scalars(select(ScoreEvidence).where(ScoreEvidence.score_id == score.id)).all(): for evidence in db.scalars(select(ScoreEvidence).where(ScoreEvidence.score_id == score.id)).all():
db.delete(evidence) db.delete(evidence)
by_id = {chunk["chunk_id"]: chunk for chunk in chunks} by_id = {chunk["chunk_id"]: chunk for chunk in chunks}
@@ -160,14 +287,75 @@ def _score_criterion(db, batch, project, criterion, client, *, retriever):
chunk = by_id[evidence.chunk_id] chunk = by_id[evidence.chunk_id]
doc = db.get(KnowledgeDocument, chunk["document_id"]) doc = db.get(KnowledgeDocument, chunk["document_id"])
path = page_image_path(db, doc, evidence.page) path = page_image_path(db, doc, evidence.page)
target = next((item for item in targets if item["target_id"] == evidence.target_id), None)
db.add(ScoreEvidence(score_id=score.id, document_id=doc.id, submission_file_id=doc.submission_file_id, db.add(ScoreEvidence(score_id=score.id, document_id=doc.id, submission_file_id=doc.submission_file_id,
page=evidence.page, quote=evidence.quote, image_path=str(path) if path else None)) page=evidence.page, quote=evidence.quote, image_path=str(path) if path else None,
target_id=evidence.target_id, target_label=target["label"] if target else "",
page_type=chunk.get("page_type") or "unknown",
source_role=target["use"] if target else USE_CONTENT))
db.commit() db.commit()
return score return score
def _run_project(batch_id, project_id, session_factory, client_factory, embedder, retriever): def _attach_page_types(db, chunks: list[dict]) -> list[dict]:
"""Attach the stored page classification to each retrieved chunk.
The page type recorded in ``knowledge_pages`` is the authority for the evidence
policy. Reading it here, rather than trusting whatever a retriever happened to put
in the dict, means a statement about what a page is cannot be forged by a caller
or lost by an alternate index.
"""
keys = {(chunk.get("document_id"), chunk.get("page")) for chunk in chunks}
keys.discard((None, None))
if not keys:
return chunks
documents = {document_id for document_id, _ in keys}
pages = {page for _, page in keys}
records = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id.in_(documents),
KnowledgePage.page.in_(pages))).all()
by_key = {(item.document_id, item.page): item for item in records}
output = []
for chunk in chunks:
record = by_key.get((chunk.get("document_id"), chunk.get("page")))
if record is None:
output.append({**chunk, "page_type": chunk.get("page_type") or "unknown"})
continue
output.append({**chunk, "page_type": record.page_type or "unknown",
"section": chunk.get("section") or record.section or "",
"chapter": chunk.get("chapter") or record.chapter or ""})
return output
def _searched_page_types(plan, targets):
allowed = set()
for target in targets:
allowed.update(target.get("allowed_page_types") or [])
return sorted(allowed or (plan.get("content_allowed_page_types") or []))
def _record_evidence_gaps(score, gaps, searched=None):
"""Store which content targets lacked body evidence and where retrieval looked.
The record is kept through teacher review so the original retrieval problem stays
auditable next to the score the teacher finally entered.
"""
score.evidence_gaps = [
{"target_id": target.get("target_id"), "target_label": target.get("label"),
"searched_page_types": searched or target.get("allowed_page_types") or [],
"reason": "补检后仍未找到正文证据"}
for target in gaps]
def _run_project(batch_id, project_id, session_factory, client_factory, embedder, retriever,
multi_retriever=retrieve_targets):
"""Score one project. Returns ``(failures, reviews)``.
Failures are missing material, parse problems and model errors. Reviews count the
items that only found navigation pages where content evidence is required; those
need a teacher decision rather than a retry.
"""
failures = 0 failures = 0
reviews = 0
with session_factory() as db: with session_factory() as db:
batch = db.get(ScoringBatch, batch_id) batch = db.get(ScoringBatch, batch_id)
project = db.get(Project, project_id) project = db.get(Project, project_id)
@@ -190,8 +378,11 @@ def _run_project(batch_id, project_id, session_factory, client_factory, embedder
failures += 1 failures += 1
for criterion in batch.rubric_snapshot["criteria"]: for criterion in batch.rubric_snapshot["criteria"]:
try: try:
result = _score_criterion(db, batch, project, criterion, client, retriever=retriever) result = _score_criterion(db, batch, project, criterion, client, retriever=retriever,
if result.status not in ("scored", "reviewed"): multi_retriever=multi_retriever)
if result.status == STATUS_NEEDS_REVIEW:
reviews += 1
elif result.status not in (STATUS_SCORED, STATUS_REVIEWED):
failures += 1 failures += 1
except Exception as exc: except Exception as exc:
score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id, score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
@@ -199,15 +390,16 @@ def _run_project(batch_id, project_id, session_factory, client_factory, embedder
if not score: if not score:
score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"]) score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"])
db.add(score) db.add(score)
score.status, score.error = "failed", str(exc)[:500] score.status, score.error = STATUS_FAILED, str(exc)[:500]
db.commit() db.commit()
failures += 1 failures += 1
db.execute(update(ScoringBatch).where(ScoringBatch.id == batch_id).values(progress_done=ScoringBatch.progress_done + 1)) db.execute(update(ScoringBatch).where(ScoringBatch.id == batch_id).values(progress_done=ScoringBatch.progress_done + 1))
db.commit() db.commit()
return failures return failures, reviews
def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=client_for_batch, embedder=None, retriever=retrieve): def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=client_for_batch, embedder=None,
retriever=retrieve, multi_retriever=retrieve_targets):
with session_factory() as db: with session_factory() as db:
batch = db.get(ScoringBatch, batch_id) batch = db.get(ScoringBatch, batch_id)
if not batch: if not batch:
@@ -218,13 +410,21 @@ def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=cli
batch.status, batch.progress_total, batch.progress_done, batch.finished_at = "running", len(project_ids) * len(criteria), 0, None batch.status, batch.progress_total, batch.progress_done, batch.finished_at = "running", len(project_ids) * len(criteria), 0, None
db.commit() db.commit()
failures = 0 failures = 0
reviews = 0
with ThreadPoolExecutor(max_workers=max(1, int(batch.model_config_snapshot.get("concurrency", 1)))) as pool: with ThreadPoolExecutor(max_workers=max(1, int(batch.model_config_snapshot.get("concurrency", 1)))) as pool:
futures = [pool.submit(_run_project, batch_id, project_id, session_factory, client_factory, embedder, retriever) for project_id in project_ids] futures = [pool.submit(_run_project, batch_id, project_id, session_factory, client_factory,
embedder, retriever, multi_retriever) for project_id in project_ids]
for future in as_completed(futures): for future in as_completed(futures):
try: try:
failures += future.result() project_failures, project_reviews = future.result()
failures += project_failures
reviews += project_reviews
except Exception as exc: except Exception as exc:
failures += 1 failures += 1
batch.error = (batch.error + f" 项目任务失败: {type(exc).__name__}")[:1000] batch.error = (batch.error + f" 项目任务失败: {type(exc).__name__}")[:1000]
batch.status, batch.finished_at = ("partial" if failures else "completed"), now() # A batch with unresolved review items is finished running, but it is not a
# completed grading result: the teacher still has to decide those criteria.
batch.status = "partial" if failures else ("review_required" if reviews else "completed")
batch.finished_at = now()
db.commit() db.commit()
return {"failures": failures, "reviews": reviews, "status": batch.status}
+23 -5
View File
@@ -9,6 +9,7 @@ from redis import Redis
from rq import Queue, Retry from rq import Queue, Retry
from sqlalchemy import delete, func, select from sqlalchemy import delete, func, select
from .config import get_settings from .config import get_settings
from .evidence import plan_snapshot
from .exports import export_excel, export_project_pdf from .exports import export_excel, export_project_pdf
from .models import (AuditLog, CourseVersion, CriterionScore, IndexTask, KnowledgeChunk, KnowledgeDocument, from .models import (AuditLog, CourseVersion, CriterionScore, IndexTask, KnowledgeChunk, KnowledgeDocument,
Project, RubricVersion, ScoringBatch, Student, Submission, SubmissionFile, TeacherRevision) Project, RubricVersion, ScoringBatch, Student, Submission, SubmissionFile, TeacherRevision)
@@ -209,9 +210,12 @@ def create_batch(data, db):
if not version.published or not rubric.published or rubric.course_version_id != version.id: if not version.published or not rubric.published or rubric.course_version_id != version.id:
raise HTTPException(400, "请选择同一课程版本下已发布的规则") raise HTTPException(400, "请选择同一课程版本下已发布的规则")
settings = get_settings() settings = get_settings()
# The evidence policy travels with the batch snapshot, so a batch can always be read
# back against the rules and page permissions it was actually scored with.
batch = ScoringBatch(course_version_id=version.id, rubric_version_id=rubric.id, batch = ScoringBatch(course_version_id=version.id, rubric_version_id=rubric.id,
course_snapshot={key: getattr(version, key) for key in ("version", "title", "objectives", "indicators", "directions", "prompts", "templates")}, course_snapshot={key: getattr(version, key) for key in ("version", "title", "objectives", "indicators", "directions", "prompts", "templates")},
rubric_snapshot={"version": rubric.version, "criteria": rubric.criteria, "weights": rubric.weights}, rubric_snapshot={"version": rubric.version, "criteria": rubric.criteria,
"weights": rubric.weights, "evidence_plan": plan_snapshot(rubric.criteria)},
model_config_snapshot=model_config(db), embedding_model_name=settings.embedding_model_name, model_config_snapshot=model_config(db), embedding_model_name=settings.embedding_model_name,
embedding_model_version=settings.embedding_model_version, embedding_model_version=settings.embedding_model_version,
status="created", project_ids=db.scalars(select(Project.id).where( status="created", project_ids=db.scalars(select(Project.id).where(
@@ -226,7 +230,7 @@ def start_batch(batch_id: int, db):
batch = db.scalar(select(ScoringBatch).where(ScoringBatch.id == batch_id).with_for_update()) batch = db.scalar(select(ScoringBatch).where(ScoringBatch.id == batch_id).with_for_update())
if not batch: if not batch:
raise HTTPException(404, "评分批次不存在") raise HTTPException(404, "评分批次不存在")
if batch.status not in ("created", "failed", "partial"): if batch.status not in ("created", "failed", "partial", "review_required"):
db.rollback() db.rollback()
raise HTTPException(409, "批次已入队、运行中或已完成") raise HTTPException(409, "批次已入队、运行中或已完成")
try: try:
@@ -244,15 +248,29 @@ def start_batch(batch_id: int, db):
return {"id": batch.id, "status": batch.status} return {"id": batch.id, "status": batch.status}
def review_score(score_id: int, data, db): def review_score(score_id: int, data, db):
"""Record a teacher decision. Model scores, retrieval gaps and history are kept.
A teacher may review any criterion in any state, including one awaiting review or
one whose material is missing: the teacher is the authority on a criterion the
retrieval could not support. The gaps from the original retrieval stay on the record
after the review, so the reason it reached the teacher remains auditable.
"""
score = found(db, CriterionScore, score_id) score = found(db, CriterionScore, score_id)
batch = found(db, ScoringBatch, score.batch_id) batch = found(db, ScoringBatch, score.batch_id)
criterion = next((item for item in batch.rubric_snapshot["criteria"] if item["criterion_id"] == score.criterion_id), None) criterion = next((item for item in batch.rubric_snapshot["criteria"] if item["criterion_id"] == score.criterion_id), None)
if not criterion or data.score > criterion["max_score"]: if not criterion or data.score > criterion["max_score"]:
raise HTTPException(400, "教师分数超出规则满分") raise HTTPException(400, "教师分数超出规则满分")
from_status = score.status
db.add(TeacherRevision(score_id=score.id, old_score=score.teacher_score, new_score=data.score, db.add(TeacherRevision(score_id=score.id, old_score=score.teacher_score, new_score=data.score,
old_comment=score.teacher_comment, new_comment=data.comment, reason=data.reason)) old_comment=score.teacher_comment, new_comment=data.comment, reason=data.reason))
score.teacher_score, score.teacher_comment, score.status = data.score, data.comment, "reviewed" score.teacher_score, score.teacher_comment = data.score, data.comment
audit(db, "review", "criterion_score", score.id, {"reason": data.reason}) # ``evidence_gaps`` is intentionally left untouched: it records why the criterion
# reached the teacher, independently of the score the teacher then entered.
score.status = "reviewed"
audit(db, "review", "criterion_score", score.id,
{"reason": data.reason, "from_status": from_status,
"came_from_review": bool(score.evidence_gaps)})
db.commit() db.commit()
return {"id": score.id, "model_score": score.model_score, "teacher_score": score.teacher_score, return {"id": score.id, "model_score": score.model_score, "teacher_score": score.teacher_score,
"final_score": score.final_score, "teacher_comment": score.teacher_comment} "final_score": score.final_score, "teacher_comment": score.teacher_comment,
"status": score.status, "evidence_gaps": score.evidence_gaps or []}
+113
View File
@@ -0,0 +1,113 @@
# 证据政策上线与批次 3 复核清单
本文记录把页面类型证据政策应用到**业务环境**的步骤,以及批次 3 需要教师复核的项目。文中的业务数据操作**均未执行**,请由教师确认后按顺序操作。
## 一、诊断基线
在业务数据库只读检查确认:
- 批次 3 的 77 条评分引文中,**14 条来自目录页**,涉及 **9 个评分项**。
- 会话 6 保存的 20 条引用中,**5 条**指向阶段报告 1 的目录页。
这些数字是诊断基线。本次改动不自动改写批次 3 的既有分数,也不改写会话 6 的历史消息。
## 二、改动后的行为
| 情形 | 改动前 | 改动后 |
|---|---|---|
| 目录页命中 | 作为正文证据送入模型并计入候选 | 只作导航,不进入内容证据 |
| 内容目标证据 | 目录、摘要、章节标题均可 | 仅正文与附录的实质段落、数据、表格 |
| 结构目标证据 | 与其他证据等同 | 封面、目录、参考文献可用,明确标注来源 |
| 正文补检不足 | 仍可能给出较低模型分 | `needs_review`,模型分为 `null`,总分待处理 |
| 历史引文 | 与正文证据无区别 | 标记"历史评分引用" |
## 三、业务环境应用步骤
以下步骤**尚未执行**。
### 1. 备份
```bash
docker compose exec -T db pg_dump -U wisegrade -Fc wisegrade > data/backups/wisegrade-pre-0006-$(date +%Y%m%d-%H%M%S).dump
chmod 600 data/backups/wisegrade-pre-0006-*.dump
```
### 2. 隔离库演练(推荐)
把备份还原到隔离数据库,先在隔离库执行升级和重建,核对页面分类与检索结果,再动业务库。`backend/tests/test_evidence_pages.py` 已用真实报告覆盖这几类情形。
### 3. 应用迁移
迁移在 API 容器启动时自动执行(`alembic upgrade head`)。如需手工执行:
```bash
docker compose exec api alembic upgrade head
docker compose exec -T db psql -U wisegrade -c "SELECT version_num FROM alembic_version;"
# 期望:0006
```
`0006` 会:
- 为 `knowledge_pages` 添加 `page_type`、`mixed`、`section`、`chapter`、`confidence`、`classification`;
- 为 `knowledge_chunks` 添加 `chapter`、`page_type`、`block_type`;
- 为 `score_evidences` 添加 `target_id`、`target_label`、`page_type`、`source_role`、`historical`;
- 为 `criterion_scores` 添加 `evidence_gaps`;
- 把**已有**评分引文全部标记为 `historical = true`;
- 按已保存的解析结果重新分类所有页面,写入审计日志。
重新分类不需要嵌入模型,因此在迁移中直接完成。`downgrade` 可撤销上述列。
### 4. 重建索引
分块方式已改变,旧索引不会被当作已分类正文使用。查看待重建清单:
```bash
curl -s http://127.0.0.1:18080/api/knowledge/stale/1 | python -m json.tool
```
重建(课程版本 1):
```bash
curl -s -X POST http://127.0.0.1:18080/api/knowledge/reindex/1
docker compose logs -f worker
```
重建任务可重复执行:已是最新版本的文档跳过,源文件哈希变化的文档跳过并说明原因,失败项记录具体错误。结束后再次查询 `/api/knowledge/stale/1`,`documents` 应为空。
### 5. 核对页面分类
```bash
curl -s "http://127.0.0.1:18080/api/knowledge/documents/<document_id>/pages" | python -m json.tool
```
预期(仓库示例报告):阶段报告 1 第 1 页 `cover`、第 2–3 页 `toc`、第 4–21 页 `body`、第 22 页 `references`;阶段报告 3 第 2–3 页 `toc`、第 11–12 页 `body`;最终报告第 2–3 页 `abstract`、第 4–5 页 `toc`、第 49 页 `references`、第 50–53 页 `appendix`。分类有疑义的页面为 `unknown`,不会当作正文。
### 6. 新批次验证
新建一个评分批次(不要重跑批次 3),确认真实模型下的表现:已评分项的引文来源为正文或附录;只检索到目录的内容项进入"待教师复核"。
## 四、批次 3 复核清单
只读检查得到的 9 个涉及目录页引文的评分项。**建议教师逐项核对**,不需要重跑批次:
| 处理方式 | 说明 |
|---|---|
| 查看该评分项证据 | 界面中旧引文显示"历史评分引用"标签,不能当作报告正文证据 |
| 在原始报告中定位正文 | 用引用卡片上的"查看页面文字"/"查看图像"跳转,或按下方章节提示查找 |
| 填写教师复核分 | 输入框不预填 0;填写后状态变为"教师已复核",模型原始分与检索记录保留 |
复核时可直接检查的正文位置(仓库示例报告):
- **阶段报告 1 第 6–7 页**:文献调研与技术现状正文,含控制方法对比表和技术现状总结。涉及 `stage1.literature`。
- **阶段报告 3 第 11–12 页**:系统测试与验证正文,含自主巡线测试、控制实时性、光照影响。涉及 `stage3.data`、`stage3.verification`。
- **最终报告第 6 页起**:正文各章;第 50–53 页为附录,可用作内容证据。
- **最终报告第 2–3 页为摘要**:可用于"摘要称……"这类明确归属的概述,不能单独证明方法已实现或指标已达标。
教师也可选择保留原有分数:本次改动不自动改写批次 3,历史分数在导出和界面中维持原值,仅在引文旁标明其来源类型。
## 五、注意事项
- 迁移和重建只改动页面分类、索引与引文元数据,**不改动**学生原件、评分批次、教师复核记录和会话历史。
- 重建过程可重试;失败项保留具体错误,重试入口在项目的"解析与索引任务"中。
- 迁移后旧评分引文的原始分数不会改变,但界面与导出会标明它们是历史引用。
- 若教师希望批次 3 的引文重新落在正文页,需要新建批次并重新评分;该操作会消耗模型调用,请先确认。
+78 -11
View File
@@ -502,22 +502,24 @@ course_2026/project_1023009006/
- 重叠约 80~120 个中文字符 - 重叠约 80~120 个中文字符
- 表格单独保存 - 表格单独保存
- 图片及图片说明单独保存 - 图片及图片说明单独保存
- 每块必须保留页码、章节、文档类型和评分指标关联 - 每块必须保留页码、页面类型、章节、文档类型和评分指标关联
- 目录页文本作为导航块单独存放,不与正文块混同
#### 检索 #### 检索
采用混合检索: 采用混合检索:
```text ```text
PostgreSQL 全文检索 pgvector 向量检索
+ pgvector 向量检索 + 中文二元组关键词检索
+ 课程版本过滤 + 课程版本过滤
+ 学生项目过滤 + 学生项目过滤
+ 报告类型过滤 + 报告类型过滤
+ 阶段过滤 + 阶段过滤
+ 页面类型过滤
``` ```
默认每个评分项取 8~12 个候选证据块,去重后送入模型。 过滤在取候选之前完成,两个召回各自独立进行,再按排名融合。默认每个评分项送模型 8~12 个候选证据块,跨目标合计上限 14 块。
#### 嵌入模型 #### 嵌入模型
@@ -622,7 +624,8 @@ model: deepseek-flash
{ {
"document": "阶段报告1", "document": "阶段报告1",
"page": 12, "page": 12,
"quote": "..." "quote": "...",
"target_id": "t0"
} }
], ],
"missing_evidence": ["..."], "missing_evidence": ["..."],
@@ -631,7 +634,60 @@ model: deepseek-flash
} }
``` ```
模型不得对没有证据的内容自行假设。结构化输出必须经过 Pydantic 校验;失败时自动重试或进入人工处理。 模型不得对没有证据的内容自行假设。结构化输出必须经过 Pydantic 校验;失败时自动重试或进入人工处理。`score` 允许为 `null`:当规则要求正文证据而检索只能给出目录、摘要或章节标题时,模型必须返回 `null`,不得据此判断学生内容缺失。
每条引文必须通过三类校验:原文与页码一致、来源页面类型在该目标允许范围内、内容性判断不得只由目录、摘要或单独章节标题支撑。页面权限和状态转换由代码执行;模型的说明或推理不能改变它。
#### 页面类型与证据政策
每份资料的每一页都带一个页面类型,随页面持久化并可供检索、评分、复核和会话查询:
| 页面类型 | 含义 | 能否作为内容证据 |
|---|---|---|
| `cover` | 封面、标题页 | 否(结构目标可用) |
| `abstract` | 摘要、Abstract | 否(结构目标可用) |
| `toc` | 目录及其续页 | 否(结构目标可用) |
| `body` | 正文 | 是 |
| `appendix` | 附录 | 是 |
| `references` | 参考文献、致谢 | 否(结构目标可用) |
| `unknown` | 识别有疑义 | 否 |
判定结合页首标题、章节条目密度、点线与末尾页码、摘要/参考文献措辞、封面字段和页脚样式,并由文档级连续性识别跨页续目录、连续摘要和连续参考文献。它不依赖"目录"一词或固定页号;重复页眉页脚在建立索引前清理。识别依据(各项信号、置信度和分类器版本)与页面一起保存,便于教师核查。
目录与正文共处一页时标为混合页,只对该页按文本块区分:正文块仍作为正文检索,不被整页排除。
`required_evidence` 的目标带 `use` 属性,并随批次快照保存:
- `use: structure`:封面、目录、参考文献等页面可以满足,例如"章节""参考文献"。
- `use: content`:只允许正文和附录中的实质段落、数据或表格。
评分规则沿用历史上的字符串数组形式;读取时按下述语法转换,批次快照记录转换结果和 `POLICY_VERSION`,因此旧批次始终可按当时的规则重读。
#### 检索
每个必备证据目标分别检索:先限定课程版本、学生项目、材料类别和允许的页面类型,再取候选。**页面过滤发生在候选数量限制之前**,目录页不会因为得分高而挤掉正文页。
语义匹配和中文关键词匹配各自召回候选池,再按排名倒数(RRF)融合;不对未归一化的语义分和关键词分直接相加。
关键词召回不使用 PostgreSQL 的 `to_tsvector('simple', …)`:该配置没有中文分词,整个中文串会成为一个词元。在课程报告上实测,查询"文献调研"无法命中含"第2章文献调研与技术现状"的页面,因此现有实现里关键词分长期接近零。现在改用字符二元组加 ASCII/数字整词的重叠度,无需扩展且在各数据库上行为一致。
名额分配优先分散到不同页面;有了可靠章节路径后对同一章节设软上限;名额不足时回填同页或同节的其他正文片段,但不会放宽证据类型。
内容目标在正文中没有命中时,利用目录标题匹配实际正文标题,补检该章节及其相邻正文页。目录只作导航,最终内容引文必须落在实际正文页。
评分发送给模型的页面图像优先选择相关的不同正文页,而不是目录页;评分最终送模型的片段数量保持受控,Agent 搜索同样受控。
#### 待教师复核
"待教师复核"是与材料缺失、解析失败、模型调用失败并列的独立状态:
- 材料缺失、重复、待确认或解析失败 → `unscorable`
- 模型调用或校验失败 → `failed`
- 内容目标补检后仍只有目录或摘要命中 → `needs_review`
进入待复核时模型分为 `null`,总分保持待处理,系统不会自动给低分或断言学生未写。评分项记录缺失的正文证据目标和当时的检索范围,该记录在教师复核后仍然保留。批次状态为"已完成,有待复核项"(`review_required`),仪表盘的待处理数包含待复核项。
教师可对任何状态的评分项复核,包括待复核项;教师评分输入框不预填 0。
#### 跨文档一致性 #### 跨文档一致性
@@ -656,10 +712,12 @@ max_score: 15
description: 评价方案是否具有工程依据、可实施性和风险分析 description: 评价方案是否具有工程依据、可实施性和风险分析
linked_outcomes: [2.3, 2.4, 4.1] linked_outcomes: [2.3, 2.4, 4.1]
required_evidence: required_evidence:
- 主方案 - label: 主方案
- 替代方案 use: content # 只能用正文或附录的实质段落证明
- 方案比较 - label: 替代方案
- 风险分析 use: content
- label: 章节
use: structure # 目录即可证明章节存在
score_levels: score_levels:
- min: 12 - min: 12
max: 15 max: 15
@@ -672,6 +730,8 @@ score_levels:
description: 缺少核心论证 description: 缺少核心论证
``` ```
`required_evidence` 也接受历史上的字符串数组(`required_evidence: [主方案, 替代方案]`)。读取时按措辞判定 `use`:提到标题、封面、目录、章节、参考文献、格式、附录等报告包装字样的目标为 `structure`,其余为 `content`。判定结果与 `POLICY_VERSION` 一起写入批次快照,不修改已发布的规则文本。
#### 版本要求 #### 版本要求
课程、课程目标、毕业要求、模板、评分规则和 Prompt 都需要版本号。评分批次保存完整版本快照。 课程、课程目标、毕业要求、模板、评分规则和 Prompt 都需要版本号。评分批次保存完整版本快照。
@@ -692,6 +752,7 @@ rubric_versions
rubric_criteria rubric_criteria
rubric_score_levels rubric_score_levels
knowledge_documents knowledge_documents
knowledge_pages
knowledge_chunks knowledge_chunks
students students
projects projects
@@ -838,7 +899,11 @@ GET /api/exports/{id}
### 数据与任务 ### 数据与任务
迁移 `0003` 添加会话、消息、运行、待确认操作、索引任务和批次固定项目清单;`0004` 添加首次请求幂等键与有界工具进度记录;`0005` 回填完整资料页文字与相对图像路径,并将历史会话中的引用一次性转换为统一页面链接。检索片段的文字和 1024 维向量保存在 PostgreSQL 的 `knowledge_chunks`;供教师核对的完整页文字单独保存在 `knowledge_pages`,避免以有重叠的检索片段拼接页面。PNG 文件位于共享 `data/processed/`,数据库只存相对于处理目录的路径;学生原件仍只读挂载在 `/data/inbox`。新批次在创建时保存项目 ID 列表;评分、结果查询和导出都使用该列表。旧批次该字段为 `NULL`,界面显示“历史项目清单未知”,只能按课程版本当前项目兼容读取,不能声称还原了创建时的范围。批次状态为 `created`、`queued`、`running`、`partial`、`failed`、`completed`;重复启动已入队或运行中的批次返回冲突。重试保留已评分项和教师复核。 迁移 `0003` 添加会话、消息、运行、待确认操作、索引任务和批次固定项目清单;`0004` 添加首次请求幂等键与有界工具进度记录;`0005` 回填完整资料页文字与相对图像路径,并将历史会话中的引用一次性转换为统一页面链接;`0006` 为页面和检索片段添加页面类型、混合页标记、章节路径,为评分证据添加所属证据目标与历史标记,并新增评分项的缺证据记录。检索片段的文字和 1024 维向量保存在 PostgreSQL 的 `knowledge_chunks`;供教师核对的完整页文字单独保存在 `knowledge_pages`,避免以有重叠的检索片段拼接页面。PNG 文件位于共享 `data/processed/`,数据库只存相对于处理目录的路径;学生原件仍只读挂载在 `/data/inbox`。新批次在创建时保存项目 ID 列表;评分、结果查询和导出都使用该列表。旧批次该字段为 `NULL`,界面显示“历史项目清单未知”,只能按课程版本当前项目兼容读取,不能声称还原了创建时的范围。批次状态为 `created`、`queued`、`running`、`partial`、`failed`、`completed`、`review_required`;`review_required` 表示评分已结束但仍有待教师复核项,与部分完成不同。重复启动已入队或运行中的批次返回冲突。重试保留已评分项和教师复核。
索引版本 `0006` 起为 3。分块方式已改变(目录页不再作为正文文本),因此旧版本的索引不会被就地当作已分类正文使用:文档保留原有 `index_version`,在重建完成前不参与检索,教师可先在上线前查看 `/api/knowledge/stale/{course_version_id}` 的待重建清单。重建任务类型为 `reindex`,只从已保存的解析结果重建,不重新读取或修改学生原件;重复执行会跳过已是最新版本和源文件已变化的文档,并记录到审计日志。
导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。
导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。 导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。
@@ -850,6 +915,8 @@ GET /api/exports/{id}
会话使用模型设置页的 OpenAI 兼容 `/chat/completions` 地址、模型和后端 API Key。适配层按 [DeepSeek 原生工具调用文档](https://api-docs.deepseek.com/guides/tool_calls) 发送 `tools`,在每轮前用不含学生资料的 `wisegrade_ping` 最小探针验证端点会返回 `tool_calls`。当前 `deepseek-flash` 端点支持自动工具选择,但拒绝强制 `tool_choice="required"`,因此探针采用 `auto` 并检查返回结构。模型不支持时会显示明确配置错误,不会伪装工具已执行。既有评分 `json_call` 保持原有 JSON 行为。 会话使用模型设置页的 OpenAI 兼容 `/chat/completions` 地址、模型和后端 API Key。适配层按 [DeepSeek 原生工具调用文档](https://api-docs.deepseek.com/guides/tool_calls) 发送 `tools`,在每轮前用不含学生资料的 `wisegrade_ping` 最小探针验证端点会返回 `tool_calls`。当前 `deepseek-flash` 端点支持自动工具选择,但拒绝强制 `tool_choice="required"`,因此探针采用 `auto` 并检查返回结构。模型不支持时会显示明确配置错误,不会伪装工具已执行。既有评分 `json_call` 保持原有 JSON 行为。
内容检索和结构查询使用与评分相同的证据政策。`search_project_material` 只返回正文和附录片段,`search_project_structure` 返回目录条目与章节结构并明确说明这些信息不能证明章节内容,`get_document_pages` 列出各页的页码、页面类型和章节,`search_document` 在单份资料内检索正文。工具返回页面类型和稳定的引用标识(`citation_id`),每项标注 `source_role` 为 `content` 或 `structure`。评分工具返回的旧引文标记为历史评分引用(`source` 为 `historical_score_evidence`),不会被当作报告正文证据。
工具输入由 Pydantic 校验,后端再校验课程、批次和项目关系。每条教师消息默认最多调用工具 32 次,成功、失败与跳过的尝试均计数;模型最多请求 33 轮,以便用完工具次数后还能生成回答。同一轮中工具名和经校验的参数(含默认分页值)完全相同时,后续调用会标为“已跳过”,模型收到前次调用标识并使用此前结果,不再执行工具;不同查询词、筛选或分页偏移仍会正常执行。可在 `.env` 中通过 `CHAT_MAX_TOOL_CALLS` 调整工具上限(1–64),修改后重新创建 API 容器。`get_scores` 每次最多返回一个项目的六项评分,响应给出 `next_criterion_offset` 和 `next_project_offset` 供继续查询;长结果保留结构化评分数据,不再变成无内容的截断提示。检索工具的运行记录保留每次调用找到的有界原文片段;回答引用对各次调用的证据去重后保存,不再只取前 20 条。工具没有任意 SQL、任意文件路径、HTTP 请求、代码执行或确认能力。检索片段只作为数据,不作为工具指令。模型只收到当前问题所需的规则摘要、检索片段或紧凑统计;API Key 不写入日志或会话表。 工具输入由 Pydantic 校验,后端再校验课程、批次和项目关系。每条教师消息默认最多调用工具 32 次,成功、失败与跳过的尝试均计数;模型最多请求 33 轮,以便用完工具次数后还能生成回答。同一轮中工具名和经校验的参数(含默认分页值)完全相同时,后续调用会标为“已跳过”,模型收到前次调用标识并使用此前结果,不再执行工具;不同查询词、筛选或分页偏移仍会正常执行。可在 `.env` 中通过 `CHAT_MAX_TOOL_CALLS` 调整工具上限(1–64),修改后重新创建 API 容器。`get_scores` 每次最多返回一个项目的六项评分,响应给出 `next_criterion_offset` 和 `next_project_offset` 供继续查询;长结果保留结构化评分数据,不再变成无内容的截断提示。检索工具的运行记录保留每次调用找到的有界原文片段;回答引用对各次调用的证据去重后保存,不再只取前 20 条。工具没有任意 SQL、任意文件路径、HTTP 请求、代码执行或确认能力。检索片段只作为数据,不作为工具指令。模型只收到当前问题所需的规则摘要、检索片段或紧凑统计;API Key 不写入日志或会话表。
### API 与故障恢复 ### API 与故障恢复
@@ -0,0 +1,241 @@
# WiseGrade 架构规范化、中文检索与章节切块:Codex Goal 模式提示词
在支持 Goal 模式的 Codex 中启用 `/goal`,将下方代码块中的正文作为目标提示词。正文授权修改仓库源码、依赖与隔离测试环境,并在完成备份、隔离演练和验收后部署到当前正式业务环境,执行必要迁移和索引重建。历史评分及会话内容须保留。目标未设定 token 或时间预算。
提示词按 [OpenAI 官方提示词说明](https://learn.chatgpt.com/docs/prompting) 组织为目标、上下文、边界和验证要求。本文件描述后续任务,不代表任务已经完成。
~~~text
你是 WiseGrade(智评)项目的开发代理。请在 /home/sky/files/Agent/WiseGrade 完成一次架构规范化和 RAG 改造:统一业务对象和各层契约,收敛运行路径中的旧格式兼容,使用 PostgreSQL 扩展 pg_jieba 实现中文分词与词项索引,将检索切块升级为章节约束下的段落切块,并贯通评分、教师复核、Agent、API、前端和导出。完成验证后实际部署到当前正式业务环境,使业务服务使用新代码、新数据库结构和新索引。
持续推进到下述开发、隔离验证、正式部署及上线验收条件全部满足。交付实际代码、迁移、测试、评估和部署记录,不能只提交架构分析、文件搬移、设计方案或待执行的部署清单。遇到错误要定位并修复;确实依赖外部条件的验证须单独标明,不得写成通过。未经请求,不设置额外预算,不创建其他 Codex 用户聊天。
## 一、工作边界
1. 允许修改本仓库源码、测试、依赖锁文件、数据库 Dockerfile、Compose、迁移和文档,允许创建与使用隔离 PostgreSQL + pgvector + pg_jieba、Redis、处理目录和导出目录。
2. 正式部署已获授权:在隔离验收通过、业务备份已验证可恢复之后,允许对当前业务库执行所需迁移、页面/块/章节分类与来源元数据转换、学生及公共资料索引重建,允许通过业务队列提交并跟踪这些任务。不得清空数据库、Redis 或任务队列,不得重跑既有评分批次。批次 3 及其他既有批次的分数、教师复核、修改历史、快照与项目范围必须保留;会话 6 及其他既有会话的消息正文和历史工具记录必须保留。必要的历史引文来源元数据转换可执行,但原始引文、实际来源和链接可核对性不得丢失。
3. 允许构建并更新正式数据库、API、worker、Web 和本次确有必要更新的相关服务,将 pg_jieba 及领域词典纳入可复现的数据库镜像;保留 PostgreSQL 16 主版本、现有端口、挂载、模型配置、密钥及持久卷。正式 API 启动会自动应用迁移,首次启动新正式镜像必须在备份与隔离还原演练完成后;测试服务始终使用隔离连接,不能借正式服务启动做迁移试验。当前提示词即为执行正式部署的授权,前置验收通过后继续部署,无须再次请求同一操作的确认。
4. 不修改学生原件和仓库课程报告;测试生成的文件放在临时目录。不要提交 .env、模型权重、数据库备份、学生材料、生成的成绩和缓存。
5. 保留工作区已有改动,不删除不属于本任务的文件。不自动创建、推送或合并 PR。
6. 常规实现选择和可逆修复自主完成。规则含义或历史范围确实无法确定时,记录具体待确认事项并继续其他工作,不能用静默猜测消除歧义。
7. 开发阶段使用模型替身及真实本地嵌入模型。为上线验收,授权使用当前已配置的真实评分/会话模型,在正式环境以仓库四份课程报告为来源的新验证项目、新规则版本、新批次和独立验证会话执行一轮完整评分及必要的引用校验重试、一次只读 Agent 检索回答,并生成该验证批次的 Excel/PDF。验证记录明确标记,不使用批次 3 或会话 6,不对其他业务项目发起批量模型调用,不自动删除验收记录。汇报实际调用范围与结果;模型配置不可用时明确报告未通过的上线验收,不以替身结果代替真实模型结果。
## 二、先核对当前实现
检查 Git 状态与提交,阅读 AGENTS.md(如有)、README.md、HANDOFF.md、docs/GUIDE.md、docs/EVIDENCE_POLICY_BATCH3.md 及相关测试。此前证据治理提交为 b7634c8;以当前源码为准,不假定以下问题仍原样存在。旧提示词或文档中的“仅隔离测试、不部署”是历史任务边界;本提示词已明确授权本次正式部署,不能将旧边界当作需要再次确认的理由。
重点检查:
- backend/wisegrade 下的 models.py、rules.py、evidence.py、pages.py、parsing.py、knowledge_pages.py、rag.py、indexing.py、scoring.py、services.py、api.py、agent_tools.py、chat_api.py、exports.py、actions.py、materialized.py、seed.py、db.py;
- backend/migrations、backend/tests、backend/pyproject.toml、backend/uv.lock、compose.yaml;
- frontend/src/App.tsx、Chat.tsx、api.ts 及浏览器测试。
先建立一份简明的模块依赖和数据来源清单,再按后续阶段实现。确认并补回归验证的已知问题:
- rules.Criterion.required_evidence 仍是 list[str],而 evidence.py 已读取对象;新规则无法贯通创建、编辑与发布。
- rag.py 重复定义 _keyword_query;page_text、relative_image_path 在不同模块重复;found、audit 也存在重复。不要把同名路由与服务函数机械地判为重复。
- 索引任务依赖评分模块中的解析逻辑;reindex_stale 从 services 导入不存在的 parse_submission,会在访问数据库前抛出 ImportError。
- 混合页的正文块仍继承 toc 页面类型,内容检索会排除它;评分阶段重读整页类型也会覆盖块的用途。
- 检索没有完整限制文档就绪状态和索引版本,也没有可靠排除 legacy 片段。
- 评分校验只检查每条引文是否合法,正分结果仍可能只引用结构证据而没有内容证据;未知 target_id 还会被默认目标接收。
- 模型返回 null 的分支未统一生成缺证据记录;完成最后一项教师复核后未重新计算批次状态。
- Agent 的引用筛选只按引文前缀或页号猜测,不能区分不同文档的同页号;Chat.tsx 未贯通来源类型和历史引用标签。
- search_document 在项目级候选截断后才过滤指定文档,可能漏掉文档内正文;get_document_pages 从片段而不是完整资料页生成清单。
- 中文关键词检索逐片段扫描字符二元组,没有分词后的倒排索引;ASCII 匹配缺少词项边界。
- 章节只是页面标签,切块仍按页内字符窗口;同页多个小节、跨页段落和表格来源没有统一表达。
- 课程与规则既有 JSON 又有关系表投影,权威来源和投影用途需要明确。
- 初始迁移读取当前 ORM metadata 建表,迁移重放结果可能随模型变化;文档索引版本的 ORM 默认值、数据库默认值和当前索引版本不一致。
## 三、阶段 1:统一对象、规则和状态契约
先明确对象的不变量和各层输入输出,再让现有调用迁移到同一契约。采用现有 Pydantic、SQLAlchemy 和 TypeScript 工具,不引入与任务无关的框架。
### 1. 文档与来源
区分并连接文档、页面、文本块、章节、检索片段、来源范围和引文:
- 页面保存完整提取文字、图片链接及页面概览分类;原始解析产物与规范资料页的职责要明确。
- 文本块保存实质来源类型和位置;混合页的内容权限以具体块为依据,不能仅由整页 toc 标签否决正文块。
- 章节有稳定标识、父子关系和完整路径,不能只靠标题字符串判定同一章节。
- 检索片段可包含多个来源范围;引文能定位到实际原文页及块。
- 明确原文、清理后检索文本和索引词项的用途,保持可验证的来源映射。
- 缺字段或未知分类不得自动当作正文。
使用统一的类型化数据契约。数据库实体、检索结果和 API 展示可以有不同对象,但转换边界明确,禁止各模块重复拼出不一致的证据字典。测试替身也要实现正式契约。
### 2. required_evidence
将其定义为评分项要求检查的证据目标清单,与具体检索实现分开:
- target_id:在所属评分项内唯一且稳定;系统可以生成,但生成后持久保存,不随数组重排变化。
- label:明确描述要检查的证据目标,非空。
- use:显式、严格校验的 structure 或 content;正式规则不根据措辞猜用途。
- allowed_page_types:可选的收窄约束;默认权限由 use 和证据政策推导,配置不得扩大内容权限或让 unknown 成为内容证据。
- query:可选的检索词覆盖;缺省从目标描述构建查询,不要求教师重复填写。
- 删除重复保存的 index;text/name 等历史别名仅在受控转换边界处理。
统一规则 API、发布校验、种子规则、批次快照、评分、Agent 和前端的目标模型。空目标清单的含义显式定义,不自动制造一个含糊的正文目标。非法用途、重复标识、非法来源约束应报明确错误。
新批次冻结规范规则与解析后的证据政策,避免把原始规则和可独立修改的执行计划变成两个权威来源。记录政策及相关检索配置版本;不能只有版本号而实际执行始终套用当前政策。
### 3. 评分状态
集中定义评分项和批次状态及转换规则,统一 API、Agent、前端、导出标签。保留 needs_review 与 failed、unscorable 的区别。教师分优先形成最终分,模型分和原始缺证据记录保持可追查;null 与 0 不能互相替代。
## 四、阶段 2:规范依赖方向和权威数据来源
按职责收敛现有单体:
- 文档处理负责解析、清理、页面、块、章节和来源定位;
- 索引与检索负责切块、分析词项、嵌入、索引、范围过滤和召回排序;
- 证据政策与评分负责来源权限、目标覆盖、模型输出校验和评分状态;
- 应用服务负责编排导入、索引、评分、复核和结果查询;
- REST、Agent、前端与导出使用应用服务和统一展示契约。
索引和评分都调用文档处理服务;索引不能依赖评分内部函数。评分结果查询移到独立于导出格式的服务,Excel/PDF、API、Agent 共用它。共享页文字、图片路径、引文链接、状态定义只实现一次。避免用局部导入掩盖不合理的相互依赖。
根据实际读路径,为课程和规则选择唯一权威存储。若保留关系表投影,须有明确查询用途且只能由权威数据重建;若删除无用途投影,提供兼容已部署库的迁移并调整业务测试。批次的不可变快照属于审计记录,应保留。
项目正式部署和测试统一为 PostgreSQL 16 + pgvector + pg_jieba;正式数据库、test-db 和隔离还原副本使用同一数据库镜像及词典版本。清理无法完整支持的 SQLite 运行路径,错误配置要明确失败。检查依赖版本兼容分支,仅保留实际支持、可测试的范围。
收敛旧接口、前端旧参数恢复和旧字符串规则处理。历史记录只读转换集中在边界,不参与新规则执行。无法还原的旧批次范围保持未知,不能用当前项目清单声称恢复历史范围。不要删除历史数据或无依据地重写旧快照。涉及旧接口停用须列出当前调用方、转换方式与影响。
修复迁移重放的不确定性,固定各迁移对应的结构;修改既有迁移文件时明确已部署库与新库的不同路径,不重排已有 revision 或假设旧库会重新执行它们。用隔离旧版本数据库验证升级,不能仅用新 ORM 自动建表代替升级验证。
## 五、阶段 3:实现章节约束下的段落切块
先完善来源对象,再替换切块策略:
1. 从真实正文块的标题提取章节树,正确处理层级、同页多个小节、跨页延续、重复标题、编号列表与公式。目录辅助导航和确认,不直接当作正文。
2. 同一小节内按完整段落合并跨页文本;长度预算内保留语义连续性。以所用嵌入模型的实际 tokenizer 计算预算,不仅按中文字数计算,也不硬编码未知模型容量。
3. 长小节按段落与句子边界细分。表格及其必要标题、条件说明优先作为完整单元;超长表格按行组切分并保留表头和来源。避免重复拼入表格或丢失表格结构。
4. 目录、摘要、参考文献等保持各自来源类型。混合页的目录块与正文块分别参与相应检索。
5. 每个片段关联稳定章节标识和逐页来源范围。跨页引文应拆分或记录多个范围,教师能看到各自实际原文和图像;不能仅用片段首页或模糊起止页代替定位。
6. 小片段用于召回;需要时按同节邻接关系补充受控上下文。所有扩展上下文仍执行来源权限、项目范围和模型输入预算。
章节标签要落到块和片段,不能将页面上最后出现的小节标题赋给整页所有文本。切块、来源映射和引文校验共同更新,不留下仍假设一个 chunk 只有一个 page 的评分、工具或导出路径。
## 六、阶段 4:使用 pg_jieba 实现中文分词与词项索引
中文分词方案已确定:直接使用 PostgreSQL 扩展 pg_jieba,并通过 PostgreSQL tsvector/tsquery 和 GIN 索引实现关键词召回。应用负责文档处理、章节切块和检索编排,中文词语切分统一由数据库执行。不要另行实现 Python Jieba 主路径或双套分词,不替换为 zhparser、独立搜索服务或字符扫描回退。无需比较分词器后再决定是否使用 pg_jieba。
以 pg_jieba 上游实现(https://github.com/jaiminpan/pg_jieba)为起点,固定扩展与 cppjieba 子模块的版本或提交,实际在 PostgreSQL 16 上构建并验证。安装问题应定位和修复;确有必要的构建适配或补丁纳入仓库并记录依据。若仍无法解决,报告具体阻塞并保持任务未完成,不能静默换回应用层分词。
要求:
- 在仓库维护数据库 Dockerfile,基于固定的 PostgreSQL 16/pgvector 基础镜像安装 pg_jieba,固定源码、子模块和词典版本;使用多阶段构建或清理构建依赖。不能仅在运行中的容器里手工安装,也不能替换成未经核对的第三方集成镜像。
- 在每个目标数据库及测试临时数据库中创建 pg_jieba 扩展和课程专用全文检索配置,明确 schema 和配置名称。入库与查询显式引用该配置,不依赖会话默认配置或修改整个数据库的默认全文检索行为。
- 入库与查询采用同一版本的 pg_jieba 分词规则;验证并选择扩展提供的混合/搜索分词配置,统一词典和规范化处理,确保查询词项可匹配索引词项。实际验证 Unicode、英文大小写、标点、数字、英文整词和中英文混合技术术语。
- 在仓库维护 pg_jieba 格式的课程领域词典,将其部署到数据库可读取的位置,至少验证 文献调研、技术现状、串级PID、视觉伺服、闭环控制、Otsu 等词,不靠针对某一测试报告的特判。记录词典文件校验和、扩展版本、分词配置及必要的加载/重启步骤,更新词典后重建受影响的词项索引。
- 对完整术语和合理子词的召回、停用词及同义词策略明确说明,用评估确认精度与召回,不盲目删除数字和技术词。
- 索引词项与原文分开,分词结果不能替代教师核对的原文。
- 由 pg_jieba 全文检索配置生成并持久化 tsvector,建立 GIN 索引;生成 tsquery 时使用安全的参数化查询,明确多词匹配和排名策略。不能继续以逐片段 generate_series、strpos 或 Python 全表扫描作为主路径,也不能在扩展缺失时静默启用旧字符二元组召回。
- 语义召回与关键词召回各自取候选,以排名融合合并;不把不同量纲的原始分数直接相加。不要把 PostgreSQL ts_rank/ts_rank_cd 宣称为 BM25。
- 在实际 PostgreSQL 上验证 CREATE EXTENSION、中文解析、领域词典加载、词项匹配、词边界、GIN 索引及查询计划,覆盖容器重建/数据库重启后配置仍有效。说明小数据集可能选择顺序扫描,不用强制索引开关冒充真实查询性能。
- 数据库镜像、扩展、全文配置与词典是备份恢复和部署依赖,须一并记录和演练;pg_dump 不包含扩展二进制与词典文件,不能把数据库备份当作完整的运行环境备份。扩展或配置缺失时明确失败,禁止把数据库故障表现为没有搜索结果。
## 七、阶段 5:贯通检索、评分、Agent、前端和导出
### 检索与证据
所有入口共用统一检索服务和范围对象。课程版本、学生项目、公共资料、指定文档、材料类别、允许来源及文档就绪/索引版本过滤,必须在候选上限之前完成。新索引构建失败不得把未完成文档标为 ready,也不得静默读取旧片段。
每个证据目标独立检索并保存检索结果。候选分配在受控总量内兼顾目标覆盖、页面和章节分散;同一片段可以支持多个目标,不因按 chunk_id 去重而丢掉目标关联。章节软上限以稳定章节标识和文档范围计数。
检索命中并不等于证据足以支持判断。目录导航应匹配实际正文标题,再补检该节和相邻正文块;没有 admissible 且相关的内容证据时,不因语义召回总能返回一些片段就视为目标已覆盖。
### 评分与复核
逐条校验目标标识、引文原文、来源范围和来源权限,拒绝未知 target_id,不用默认目标悄悄接收。内容判断不能由目录、摘要或单独章节标题支撑。正分结果必须包含支持内容判断的合法内容引文,不能只给结构目标引文;内容目标的覆盖或不足要显式报告。模型可理解证据并说明不足,代码负责权限、覆盖一致性和状态转换。
统一检索不足和模型返回 null 的处理:记录缺失目标、实际检索范围、补检过程和不足原因,模型分为 null,相关总分待处理。不要把没有查到解释为学生没有写,也不要把 lack of evidence 自动折算为 0。合法的明确 0 分不能被展示成空值。
教师复核保留原始模型记录与缺证据记录,保存修改历史并更新评分项状态;当最后一项复核结束时重新计算批次状态。重试保留已评分项和教师复核。项目事实抽取也遵循来源政策。
### Agent、API、UI、导出
- 内容搜索与结构查询共用政策;get_document_pages 从规范页面数据取得完整清单,search_document 在数据库候选截断前限定文档。
- 使用稳定 citation_id 和结构化引用协议,最终回答只附实际使用且可定位的来源。验证文档、来源范围与用途,不能仅按页号或引文前缀猜关联;保留面向教师的正常自然语言回答。
- 评分详情和会话引用卡片贯通来源类型、证据目标、正文/目录区别及历史评分引用标签。历史引文保持原内容,不能自动充当新内容证据。
- 前端使用统一接口类型,清理本次涉及的数据链路中的 any 和重复状态映射;不得以类型断言掩盖缺字段。
- 教师评分框无分时为空,已有合法 0 分正常显示;复核后的最终分与批次状态同步。
- Excel/PDF 共用结果查询,实际生成并解析验证文件;保留 0、null、待复核状态、来源位置、跨页引文、历史标记和原始不足记录的正确含义。
- API 文档与实现一致,旧接口和数据转换边界在文档中列明。
## 八、阶段 6:隔离迁移、重建与评估
提升索引版本,统一相关默认值,并记录切块策略、分词器/词典和嵌入配置版本。实现可重试、可审计的重建:优先复用已保存解析结果,课程公共资料按规范入库路径处理;源文件哈希变化要报告,不改原件。不能因错误地重设 parse_status 而无必要地重新解析材料。
重建可重复执行,成功文档跳过,失败文档可重试,进度、失败原因和剩余旧索引数量准确。处理学生资料与公共资料,不能在任务显示成功后悄悄留下未处理资料。旧评分保存的引文不能因片段重建而失去原始来源关联。
本阶段测试使用专用 PostgreSQL/pgvector/pg_jieba、专用 Redis 和临时文件目录,明确确认连接和任务队列隔离;开发测试不向业务环境写入。业务变更在下一阶段按已验证的部署流程执行。
### 必须实际验证的验收条件
1. 规则对象从 API 创建、编辑、发布到批次快照和评分完整贯通;显式 use 生效,非法目标和来源配置被拒绝;稳定标识不随重排变化。
2. 阶段报告 1 的第 2–3 页目录与第 6–7 页文献调研正文区分正确,文献分析目标可召回真实正文。
3. 阶段报告 3 的第 2–3 页目录与第 11–12 页系统测试正文区分正确,测试目标可召回正文;召回这些页不能直接等同于应得某个分数。
4. 最终报告第 2–3 页摘要、4–5 页目录、6 页起正文、49 页参考文献、50–53 页附录区分正确;阶段报告 2 也纳入分类和切块检查。
5. 合成混合页的正文实际进入内容检索,并通过合法评分引用校验,目录块仍被拒绝;检查完整链路,不能只断言正文块已保存。
6. 同页多小节、跨页段落、重复标题、长章节、表格和编号列表正确处理;跨页片段的引文落到准确原文页,文字和图片均可核对。
7. pg_jieba 在 PostgreSQL 16 的正式同版测试镜像中实际安装并运行,领域词典生效;文献调研、串级PID 结构、视觉伺服与闭环控制等查询命中正文,ASCII 词项边界正确,GIN 词项索引真实存在。容器重建/数据库重启、备份还原及扩展缺失报错路径均验证。
8. 内容评分拒绝目录、摘要、单独标题、未知目标和伪造来源;混合结构/内容评分项的正分结果不能只引用结构证据。
9. 检索不足和模型返回 null 均进入待教师复核,缺证据记录完整,总分待处理;教师可录入 0 或其他合法分,记录保留,最后一项复核后批次状态正确。
10. 指定文档检索在其他文档高排名候选挤占名额时仍能找到该文档正文;旧版本、legacy、非 ready 文档不参与新检索;学生与课程版本隔离及公共资料权限正确。
11. Agent 引用能区分不同文档的同页号、改写回答与实际使用来源;工具查到但回答未使用的片段不被附上;评分界面和会话界面的历史标签实际可见。
12. 重建入口实际执行成功,失败后重试和重复执行有效;学生及公共资料均覆盖,历史引文仍可核对。迁移在新库与旧版本隔离库均验证,升降级及再次升级范围如实记录。
13. 实际生成并解析 Excel/PDF,验证 0 分、待复核、复核后成绩、跨页来源和历史引用;不是仅检查代码可以 import。
14. 前端类型检查、生产构建和相关浏览器流程通过,涵盖证据标签、无分输入框、0 分、复核和会话引用;模拟 API 的浏览器测试不能宣称真实后端端到端通过。
### 检索质量评估
在改造前保存可比较基线,再以相同语料、查询和预算比较改造后结果。至少使用四份真实课程报告,覆盖明确术语、宽泛问题、结构查询和指定报告查询。
用固定查询集记录 top-k 正文召回、目标覆盖、重复片段占比、无关候选、片段与 token 数量、查询延迟和数据库查询计划。关键词路径可用固定向量隔离测试;语义与混合检索质量须用可用的真实本地嵌入模型单独评估,不能由固定向量测试代替。
将章节切块与中文分词的影响尽可能分别比较,不把所有改动合并后一次通过当成各自有效的证明。核心回归不得退化;出现退化要分析和修正,不仅扩大候选量掩盖问题。记录模型、语料和参数,保证结果可复现,不虚构性能提升百分比。
## 九、阶段 7:实际部署到正式业务并验收
### 1. 记录基线并验证备份
- 确认当前正式 Compose 项目、镜像标识、服务状态、数据库迁移版本、业务队列、持久卷和挂载路径;记录可恢复的旧镜像与旧配置,不输出密钥或密码。
- 对既有批次、评分、教师复核、历史消息和工具记录建立部署前基线,尤其核对批次 3、会话 6。必要元数据转换与原始业务内容分开比较,不能用只比较记录数量证明内容未变。
- 备份业务 PostgreSQL,并保留索引重建涉及的处理资产及必要部署配置;记录包含 pgvector/pg_jieba 的新旧数据库镜像、扩展版本、领域词典及配置校验和。备份使用受限权限和校验记录,不提交 Git。
- 实际将业务备份还原到隔离副本,验证关键记录与历史链接;在副本执行最终迁移、学生及公共资料重建和查询验收,记录耗时与失败情况。只运行 pg_dump 而没有验证恢复不算完成。
- 先构建新镜像并完成必要构建检查,保留旧镜像;制定与当前数据库变更兼容的回滚路径,并在隔离副本验证。不能把未经验证的 downgrade 当成回滚保证。
### 2. 执行迁移和服务更新
- 检查正在执行的评分、索引与 Agent 任务;先阻止接收新的相关写任务,等待现有任务正常结束,再暂停会写入不兼容结构的旧服务。不能中断正在执行的教师业务、取消既有作业或清队列。若有长期任务无法自然结束,继续准备可独立完成的工作,并准确说明上线阻塞。
- 在写任务排空后取得最终一致的业务备份;若数据自演练备份后变化,核对变化对迁移的影响,必要时补做演练。
- 按已演练顺序更新正式数据库为包含 pgvector/pg_jieba 的同主版本镜像,保留现有数据卷;验证数据库恢复健康、领域词典可读、必要扩展与配置可创建,再执行正式迁移。考虑 API 自动迁移,避免多个进程并发迁移及新旧 worker 同时写入。不得重建空库代替升级,也不得在扩展尚未可用时启动新检索业务。
- 更新正式 API、worker、Web,恢复必要队列消费。确保服务使用同一版源码、迁移和依赖,数据库和模型挂载保持正确;不删除持久卷、不重新初始化种子业务数据。
- 对正式学生和公共资料执行已验证的索引重建,跟踪到结束并处理失败项。源文件变化等不可重建项逐项报告,不修改原件,也不能在尚有未处理旧索引时宣称全量升级成功。
- 在索引重建完成并通过关键查询之前,不把检索服务宣称为可用;如需维护状态,给出真实进度并在验收后恢复正常入口。不得通过启用旧片段回退来掩盖重建缺口。
### 3. 正式上线验收
通过实际 Web 入口和正式 API 验证,不仅检查容器处于 running:
1. 数据库、API、worker、Web 健康,迁移版本和镜像一致;正式 PostgreSQL 16 的 pgvector/pg_jieba、课程全文配置、领域词典与 GIN 索引实际生效,日志无新增启动、任务或查询错误,真实后台作业可完成。
2. 页面/块/章节清单、正文检索、指定文档查询和历史引文文字/图片链接正常;业务旧索引清单已处理完或明确报告剩余项及其影响。
3. 批次 3 的分数、复核及快照,与会话 6 的正文/工具记录和其他保护数据符合部署前基线;允许转换的来源元数据按转换规则核对,原始引用仍可定位。
4. 在业务环境创建明确标记的新验证项目、规范规则版本和新批次,使用仓库真实课程报告执行已授权的一轮真实模型评分。缺证据转复核与合法引用按新政策工作;不要为了让所有项得分而放宽校验或伪造教师复核分。
5. 通过独立验证会话执行已授权的真实只读检索回答,在浏览器确认引用来源、页面文字/图像、结构与内容标签。新记录 ID 和用途写入交接记录。
6. 实际生成、下载并解析新验证批次的 Excel/PDF;存在待复核项时保持空分与状态正确,不能把仍待处理的批次记作完整评分成功。
7. 浏览器检查评分页、复核输入框、历史标签、会话引用和移动端布局;仅在隔离或验收记录上测试写入,不能为 UI 测试修改真实教师评分。
上述验收失败时修复具体问题并重验受影响部分;不要在反复调用模型中掩盖检索或协议缺陷。正式环境缺少关键外部能力时,将服务部署情况和未通过验收分别报告,Goal 保持未完成。
### 4. 失败处理与回滚
出现严重迁移或上线失败,停止继续扩大业务变更,按已验证方案恢复服务。应用旧镜像前核对数据库兼容性;数据库恢复镜像必须包含恢复目标所依赖的 pg_jieba 扩展及词典,不能将含扩展依赖的库直接交给不含扩展的旧镜像。需要恢复备份时先停止相关写入并保存故障现场。
若上线后已有新的教师业务写入,不能直接覆盖还原而丢失这些数据:先保存并核对新增写入,采用已验证的保留方案;没有可保证数据保留的恢复路径时,准确说明具体阻塞并请求该恢复动作所需决定。原有部署授权不包含丢弃部署后教师数据。
回滚后验证业务入口、任务队列、保护数据和历史链接,并记录失败原因、采取的动作和当前运行版本。回滚恢复可用不等于本次改造部署成功,不得据此将 Goal 标记完成。
## 十、最终交付
1. 提交前核对差异和工作区,按清晰阶段组织改动,不夹带无关重构。
2. 更新 docs/GUIDE.md、HANDOFF.md 和必要的业务上线清单;历史验证记录保留,新记录注明当前范围。
3. 给出改造后的模块职责、对象关系、权威数据来源和主要调用链,列出已删除或收敛的兼容逻辑及仍保留的原因。
4. 列出新增迁移、索引版本、pg_jieba/cppjieba 固定版本、全文检索配置、数据库 Dockerfile 和领域词典校验和,以及隔离库中的升级、重建和回滚验证结果;记录正式部署前后迁移版本、数据库及应用镜像标识与各服务健康状态。
5. 给出实际测试命令、通过/失败/跳过数量、真实本地检索评估、生成导出的检查结果与尚未验证部分。不能用旧测试记录证明新代码通过。
6. 汇报实际执行的正式部署步骤、备份路径与恢复验证、学生和公共资料重建数量及剩余项、真实模型验收批次/项目/会话 ID、导出检查和业务入口地址。批次 3 的教师复核仍由教师决定,只提供正文定位与复核清单,不自动改分。
7. 明确业务库实际变更范围及保护数据对比结果:哪些结构、索引和来源元数据已更新,哪些验收记录已新增,批次 3 的原始评分/复核/快照及会话 6 的历史内容是否保持。若发生意外修改,立即说明实际影响和证据,不能隐瞒或把“记录数相同”当作未改写证明。
只有代码、迁移、必需的隔离验收、正式部署、索引重建及上述真实上线验收实际完成后,才将 Goal 标记完成;不得停在“等待部署确认”或“建议后续上线”。当前正式环境以外的 Windows 实机等验证如未运行,可注明边界;当前业务上线必需验收未通过则仍未完成。
~~~
+1
View File
@@ -8,5 +8,6 @@
| [业务 Agent](CODEX_CHAT_AGENT_GOAL_PROMPT.md) | 会话、业务工具、操作确认与自动索引的阶段任务。 | | [业务 Agent](CODEX_CHAT_AGENT_GOAL_PROMPT.md) | 会话、业务工具、操作确认与自动索引的阶段任务。 |
| [会话界面](CODEX_SESSION_UI_GOAL_PROMPT.md) | 草稿、范围、历史会话、Markdown、工具记录和后台运行的增量任务。 | | [会话界面](CODEX_SESSION_UI_GOAL_PROMPT.md) | 草稿、范围、历史会话、Markdown、工具记录和后台运行的增量任务。 |
| [目录、摘要与正文证据治理](CODEX_EVIDENCE_RAG_GOAL_PROMPT.md) | 页面分类、检索与评分证据治理任务,包含业务数据库不可改写的明确边界。 | | [目录、摘要与正文证据治理](CODEX_EVIDENCE_RAG_GOAL_PROMPT.md) | 页面分类、检索与评分证据治理任务,包含业务数据库不可改写的明确边界。 |
| [架构规范化、中文检索与章节切块](CODEX_ARCHITECTURE_RAG_REFACTOR_GOAL_PROMPT.md) | 按阶段统一对象和依赖、实现章节切块与 pg_jieba 中文词项索引;隔离验收后实际部署正式业务、迁移和重建索引,并保留历史评分与会话。 |
提示词记录任务要求,不代表功能已经完成或通过验证。应用运行时的评分 Prompt 由课程版本管理,不在本目录维护。 提示词记录任务要求,不代表功能已经完成或通过验证。应用运行时的评分 Prompt 由课程版本管理,不在本目录维护。
+18 -4
View File
@@ -26,6 +26,20 @@ const options = (rows: any[], label: (row: any) => string) => rows.map(row => ({
const isSlideEvidence = (evidence: any) => /\.pptx$/i.test(evidence.document || ''); const isSlideEvidence = (evidence: any) => /\.pptx$/i.test(evidence.document || '');
const evidencePlace = (evidence: any) => `第 ${evidence.page} ${isSlideEvidence(evidence) ? '张幻灯片' : '页'}`; const evidencePlace = (evidence: any) => `第 ${evidence.page} ${isSlideEvidence(evidence) ? '张幻灯片' : '页'}`;
const evidencePreviewLabel = (evidence: any) => isSlideEvidence(evidence) ? '查看该张幻灯片的完整图像' : '查看该页的完整图像'; const evidencePreviewLabel = (evidence: any) => isSlideEvidence(evidence) ? '查看该张幻灯片的完整图像' : '查看该页的完整图像';
const pageTypeLabels: Record<string, string> = { cover: '封面', abstract: '摘要', toc: '目录', body: '正文', appendix: '附录', references: '参考文献', unknown: '识别不确定', mixed: '目录与正文混合页' };
const pageTypeLabel = (value: string) => pageTypeLabels[value] || '未分类页面';
const statusLabels: Record<string, string> = { pending: '待评分', scored: '模型已评分', reviewed: '教师已复核', needs_review: '待教师复核', unscorable: '材料缺失或解析失败', failed: '模型调用失败' };
const statusLabel = (value: string) => statusLabels[value] || value;
const statusColor = (value: string) => ({ scored: 'green', reviewed: 'green', needs_review: 'gold', unscorable: 'red', failed: 'red' }[value] || 'blue');
const batchStatusLabels: Record<string, string> = { created: '待启动', queued: '排队中', running: '运行中', completed: '已完成', review_required: '已完成,有待复核项', partial: '部分完成', failed: '失败' };
const batchStatusLabel = (value: string) => batchStatusLabels[value] || value;
// A teacher score box is never prefilled with 0: an empty box forces a deliberate
// entry, while 0 would read as a judgement the teacher did not make.
const reviewFormValues = (row: any) => ({
score: row.teacher_score ?? row.final_score ?? null,
comment: row.teacher_comment || row.model_comment || '',
reason: '',
});
function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: any; onChange: (value: any) => void; disabled?: boolean; rows?: number; resetKey: string }) { function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: any; onChange: (value: any) => void; disabled?: boolean; rows?: number; resetKey: string }) {
const [text, setText] = useState(JSON.stringify(value, null, 2)); const [text, setText] = useState(JSON.stringify(value, null, 2));
useEffect(() => { setText(JSON.stringify(value, null, 2)); }, [resetKey]); useEffect(() => { setText(JSON.stringify(value, null, 2)); }, [resetKey]);
@@ -35,7 +49,7 @@ function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: a
function Dashboard() { function Dashboard() {
const [data, setData] = useState<any>({}); const [data, setData] = useState<any>({});
useEffect(() => { load('/dashboard', setData); }, []); useEffect(() => { load('/dashboard', setData); }, []);
return <><Title level={2}>工作概览</Title><Paragraph>先确认课程版本与规则,再扫描挂载目录,创建评分批次并逐项复核。</Paragraph><Row gutter={18}>{[['课程', data.courses], ['学生项目', data.projects], ['评分批次', data.batches], ['待处理评分项', data.needs_attention]].map(([label, count]) => <Col xs={24} sm={12} lg={6} key={label}><Card><Statistic title={label} value={count ?? 0} /></Card></Col>)}</Row><Alert className="top-gap" type="info" showIcon message="学生材料仅从只读挂载目录读取;缺失和解析失败的材料会在项目与评分详情中标出。" /></>; return <><Title level={2}>工作概览</Title><Paragraph>先确认课程版本与规则,再扫描挂载目录,创建评分批次并逐项复核。</Paragraph><Row gutter={18}>{[['课程', data.courses], ['学生项目', data.projects], ['评分批次', data.batches], ['待处理评分项', data.needs_attention], ['待教师复核', data.needs_review]].map(([label, count]) => <Col xs={24} sm={12} lg={4} key={label}><Card><Statistic title={label} value={count ?? 0} /></Card></Col>)}</Row><Alert className="top-gap" type="info" showIcon message="学生材料仅从只读挂载目录读取;缺失和解析失败的材料会在项目与评分详情中标出。“待教师复核”表示该项只检索到目录、摘要或章节标题,没有可用于内容判断的正文证据,系统不会自动给低分。" /></>;
} }
function Courses() { function Courses() {
@@ -107,7 +121,7 @@ function Batches({ openBatch }: { openBatch: (id: number) => void }) {
useEffect(() => { refresh(); load('/courses', setCourses); const id = window.setInterval(refresh, 5000); return () => clearInterval(id); }, []); useEffect(() => { refresh(); load('/courses', setCourses); const id = window.setInterval(refresh, 5000); return () => clearInterval(id); }, []);
useEffect(() => { if (courseId) load(`/courses/${courseId}/versions`, setVersions); }, [courseId]); useEffect(() => { if (courseId) load(`/courses/${courseId}/versions`, setVersions); }, [courseId]);
useEffect(() => { if (versionId) load(`/course-versions/${versionId}/rubrics`, setRubrics); }, [versionId]); useEffect(() => { if (versionId) load(`/course-versions/${versionId}/rubrics`, setRubrics); }, [versionId]);
return <><Title level={2}>评分批次与进度</Title><Card title="创建独立批次"><Space wrap><Select style={{ width: 250 }} placeholder="课程" options={options(courses, c => c.name)} onChange={setCourseId} /><Select style={{ width: 250 }} placeholder="已发布课程版本" options={options(versions.filter(v => v.published), v => `v${v.version} ${v.title}`)} onChange={setVersionId} /><Select style={{ width: 230 }} placeholder="已发布规则版本" options={options(rubrics.filter(r => r.published), r => `v${r.version} · ${r.criteria.length}项`)} onChange={setRubricId} /><Button type="primary" disabled={!versionId || !rubricId} onClick={async () => { try { await post('/scoring-batches', { course_version_id: versionId, rubric_version_id: rubricId }); refresh(); message.success('批次已创建'); } catch (e) { fail(e); } }}>创建批次</Button></Space></Card><Table className="top-gap" rowKey="id" rowClassName={(b: any) => Number(new URLSearchParams(window.location.search).get("batch_id")) === b.id ? "linked-row" : ""} dataSource={batches} columns={[{ title: '批次', dataIndex: 'id' }, { title: '课程/规则版本', render: (_, b) => `${b.course_version_id} / ${b.rubric_version_id}` }, { title: '状态', render: (_, b) => <Tag color={b.status === 'completed' ? 'green' : b.status === 'partial' || b.status === 'failed' ? 'orange' : 'blue'}>{b.status}</Tag> }, { title: '进度', render: (_, b) => <Progress percent={b.progress_total ? Math.round(b.progress_done / b.progress_total * 100) : 0} size="small" /> }, { title: '异常', dataIndex: 'error' }, { title: '操作', render: (_, b) => <Space><Button disabled={!['created', 'failed', 'partial'].includes(b.status)} onClick={async () => { try { await post(`/scoring-batches/${b.id}/start`); refresh(); } catch (e) { fail(e); } }}>启动/重试</Button><Button type="link" onClick={() => openBatch(b.id)}>查看评分</Button></Space> }]} /></>; return <><Title level={2}>评分批次与进度</Title><Card title="创建独立批次"><Space wrap><Select style={{ width: 250 }} placeholder="课程" options={options(courses, c => c.name)} onChange={setCourseId} /><Select style={{ width: 250 }} placeholder="已发布课程版本" options={options(versions.filter(v => v.published), v => `v${v.version} ${v.title}`)} onChange={setVersionId} /><Select style={{ width: 230 }} placeholder="已发布规则版本" options={options(rubrics.filter(r => r.published), r => `v${r.version} · ${r.criteria.length}项`)} onChange={setRubricId} /><Button type="primary" disabled={!versionId || !rubricId} onClick={async () => { try { await post('/scoring-batches', { course_version_id: versionId, rubric_version_id: rubricId }); refresh(); message.success('批次已创建'); } catch (e) { fail(e); } }}>创建批次</Button></Space></Card><Table className="top-gap" rowKey="id" rowClassName={(b: any) => Number(new URLSearchParams(window.location.search).get("batch_id")) === b.id ? "linked-row" : ""} dataSource={batches} columns={[{ title: '批次', dataIndex: 'id' }, { title: '课程/规则版本', render: (_, b) => `${b.course_version_id} / ${b.rubric_version_id}` }, { title: '状态', render: (_, b) => <Space wrap><Tag color={b.status === 'completed' ? 'green' : b.status === 'review_required' ? 'gold' : b.status === 'partial' || b.status === 'failed' ? 'orange' : 'blue'}>{batchStatusLabel(b.status)}</Tag>{b.needs_review > 0 && <Tag color="gold">待复核 {b.needs_review}</Tag>}</Space> }, { title: '进度', render: (_, b) => <Progress percent={b.progress_total ? Math.round(b.progress_done / b.progress_total * 100) : 0} size="small" /> }, { title: '异常', dataIndex: 'error' }, { title: '操作', render: (_, b) => <Space><Button disabled={!['created', 'failed', 'partial', 'review_required'].includes(b.status)} onClick={async () => { try { await post(`/scoring-batches/${b.id}/start`); refresh(); } catch (e) { fail(e); } }}>启动/重试</Button><Button type="link" onClick={() => openBatch(b.id)}>查看评分</Button></Space> }]} /></>;
} }
function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }) { function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }) {
@@ -116,8 +130,8 @@ function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }
useEffect(() => { load('/scoring-batches', setBatches); load('/projects', setProjects); }, []); useEffect(() => { load('/scoring-batches', setBatches); load('/projects', setProjects); }, []);
useEffect(() => { if (batch && project) load(`/projects/${project}/scores?batch_id=${batch}`, setResult); }, [batch, project]); useEffect(() => { if (batch && project) load(`/projects/${project}/scores?batch_id=${batch}`, setResult); }, [batch, project]);
const refresh = () => batch && project && load(`/projects/${project}/scores?batch_id=${batch}`, setResult); const refresh = () => batch && project && load(`/projects/${project}/scores?batch_id=${batch}`, setResult);
return <><Title level={2}>逐项评分与教师复核</Title><Space wrap><Select style={{ width: 180 }} placeholder="评分批次" value={batch} options={options(batches, b => `批次 #${b.id} · ${b.status}`)} onChange={setBatch} /><Select style={{ width: 330 }} placeholder="学生项目" value={project} options={options(projects, p => `${p.student_no} ${p.name} · ${p.topic}`)} onChange={setProject} /></Space>{result && <><Card className="top-gap" title={`${result.student_no} ${result.name} · ${result.topic}`}><Space wrap>{Object.entries(result.totals).map(([key, value]) => <Tag key={key} color={value === null ? 'orange' : 'blue'}>{key}: {String(value ?? '待处理')}</Tag>)}</Space>{result.facts?.consistency_issues?.length > 0 && <Alert className="top-gap" type="warning" message="跨文档一致性提醒" description={result.facts.consistency_issues.map((issue: any) => issue.description).join(';')} />}</Card><Table className="top-gap" rowKey={(row: any) => row.criterion.criterion_id} dataSource={result.rows} pagination={false} columns={[{ title: '评分项', render: (_, r) => <><Tag>{categories[r.criterion.category] || r.criterion.category}</Tag>{r.criterion.name}</> }, { title: '模型原始分', render: (_, r) => `${r.model_score ?? '—'} / ${r.criterion.max_score}` }, { title: '教师分', render: (_, r) => r.teacher_score ?? '—' }, { title: '最终分', render: (_, r) => r.final_score ?? '待处理' }, { title: '状态', render: (_, r) => <Tag color={r.status === 'scored' || r.status === 'reviewed' ? 'green' : 'orange'}>{r.status}</Tag> }, { title: '操作', render: (_, r) => <Button onClick={() => { setReview(r); form.setFieldsValue({ score: r.final_score ?? 0, comment: r.teacher_comment || r.model_comment || '', reason: '' }); }}>查看证据/复核</Button> }]} /></>} return <><Title level={2}>逐项评分与教师复核</Title><Space wrap><Select style={{ width: 180 }} placeholder="评分批次" value={batch} options={options(batches, b => `批次 #${b.id} · ${b.status}`)} onChange={setBatch} /><Select style={{ width: 330 }} placeholder="学生项目" value={project} options={options(projects, p => `${p.student_no} ${p.name} · ${p.topic}`)} onChange={setProject} /></Space>{result && <><Card className="top-gap" title={`${result.student_no} ${result.name} · ${result.topic}`}><Space wrap>{Object.entries(result.totals).map(([key, value]) => <Tag key={key} color={value === null ? 'orange' : 'blue'}>{key}: {String(value ?? '待处理')}</Tag>)}</Space>{result.facts?.consistency_issues?.length > 0 && <Alert className="top-gap" type="warning" message="跨文档一致性提醒" description={result.facts.consistency_issues.map((issue: any) => issue.description).join(';')} />}</Card><Table className="top-gap" rowKey={(row: any) => row.criterion.criterion_id} dataSource={result.rows} pagination={false} columns={[{ title: '评分项', render: (_, r) => <><Tag>{categories[r.criterion.category] || r.criterion.category}</Tag>{r.criterion.name}</> }, { title: '模型原始分', render: (_, r) => `${r.model_score ?? '—'} / ${r.criterion.max_score}` }, { title: '教师分', render: (_, r) => r.teacher_score ?? '—' }, { title: '最终分', render: (_, r) => r.final_score ?? '待处理' }, { title: '状态', render: (_, r) => <Tag color={statusColor(r.status)}>{statusLabel(r.status)}</Tag> }, { title: '操作', render: (_, r) => <Button onClick={() => { setReview(r); form.setFieldsValue(reviewFormValues(r)); }}>查看证据/复核</Button> }]} /></>}
<Modal open={!!review} title={review?.criterion?.name} width={850} onCancel={() => setReview(undefined)} onOk={async () => { try { const values = await form.validateFields(); await put(`/criterion-scores/${review.score_id}/review`, values); setReview(undefined); refresh(); message.success('复核记录已保存'); } catch (e) { fail(e); } }} okText="保存教师复核" okButtonProps={{ disabled: !review?.score_id }}>{review && <><Descriptions column={1} items={[{ key: 'model', label: '模型分数/评语', children: `${review.model_score ?? '—'} · ${review.model_comment || ''}` }, { key: 'confidence', label: '模型置信度', children: review.confidence ?? '—' }, { key: 'missing', label: '缺失证据', children: review.missing_evidence?.join(';') || '无' }, { key: 'suggestion', label: '改进建议', children: review.suggestion || '无' }, { key: 'error', label: '异常', children: review.error || '无' }]} /><Divider />{review.evidence?.map((ev: any, index: number) => <Card size="small" key={index} className="evidence"><Text strong>{ev.document} · {evidencePlace(ev)}</Text><Paragraph>{ev.quote}</Paragraph><Space wrap>{ev.url && <a href={ev.url} target="_blank" rel="noreferrer">查看页面文字</a>}{ev.image_url && <a href={ev.image_url} target="_blank" rel="noreferrer">{evidencePreviewLabel(ev)}(核对引文)</a>}</Space></Card>)}<Divider /><Form form={form} layout="vertical"><Form.Item label="教师评分" name="score" rules={[{ required: true }]}><InputNumber min={0} max={review.criterion.max_score} /></Form.Item><Form.Item label="教师评语" name="comment" rules={[{ required: true }]}><Input.TextArea rows={3} /></Form.Item><Form.Item label="修改说明" name="reason"><Input /></Form.Item></Form>{review.score_id && <Button onClick={async () => { try { setRevisions(await api(`/criterion-scores/${review.score_id}/revisions`)); } catch (e) { fail(e); } }}>查看修改记录</Button>}{revisions.length > 0 && <pre className="json-view">{JSON.stringify(revisions, null, 2)}</pre>}</>}</Modal> <Modal open={!!review} title={review?.criterion?.name} width={850} onCancel={() => setReview(undefined)} onOk={async () => { try { const values = await form.validateFields(); await put(`/criterion-scores/${review.score_id}/review`, values); setReview(undefined); refresh(); message.success('复核记录已保存'); } catch (e) { fail(e); } }} okText="保存教师复核" okButtonProps={{ disabled: !review?.score_id }}>{review && <>{review.status === 'needs_review' && <Alert type="warning" showIcon className="top-gap" message="本项缺少正文证据,需要教师复核" description={`未找到正文证据的目标:${(review.evidence_gaps || []).map((gap: any) => gap.target_label).join('、') || review.missing_evidence?.join('、') || '见缺失证据'}。系统未给出模型分,也没有按低分处理;请核对材料后直接填写教师评分。`} />}<Descriptions column={1} items={[{ key: 'model', label: '模型分数/评语', children: `${review.model_score ?? '未给出'} · ${review.model_comment || ''}` }, { key: 'confidence', label: '模型置信度', children: review.confidence ?? '—' }, { key: 'missing', label: '缺失证据', children: review.missing_evidence?.join(';') || '无' }, { key: 'gaps', label: '缺正文证据的目标', children: (review.evidence_gaps || []).map((gap: any) => `${gap.target_label}(检索范围:${(gap.searched_page_types || []).map(pageTypeLabel).join('、') || '正文'})`).join(';') || '无' }, { key: 'suggestion', label: '改进建议', children: review.suggestion || '无' }, { key: 'error', label: '异常', children: review.error || '无' }]} /><Divider />{review.evidence?.map((ev: any, index: number) => <Card size="small" key={index} className="evidence"><Space wrap><Text strong>{ev.document} · {evidencePlace(ev)}</Text><Tag color={ev.page_type === 'body' || ev.page_type === 'appendix' ? 'green' : 'orange'}>{pageTypeLabel(ev.page_type)}</Tag>{ev.historical && <Tag>历史评分引用</Tag>}{ev.target_label && <Tag>{ev.target_label}</Tag>}</Space><Paragraph>{ev.quote}</Paragraph><Space wrap>{ev.url && <a href={ev.url} target="_blank" rel="noreferrer">查看页面文字</a>}{ev.image_url && <a href={ev.image_url} target="_blank" rel="noreferrer">{evidencePreviewLabel(ev)}(核对引文)</a>}</Space></Card>)}<Divider /><Form form={form} layout="vertical"><Form.Item label="教师评分" name="score" rules={[{ required: true, message: '请填写教师评分' }]}><InputNumber min={0} max={review.criterion.max_score} placeholder="请填写" /></Form.Item><Form.Item label="教师评语" name="comment" rules={[{ required: true }]}><Input.TextArea rows={3} /></Form.Item><Form.Item label="修改说明" name="reason"><Input /></Form.Item></Form>{review.score_id && <Button onClick={async () => { try { setRevisions(await api(`/criterion-scores/${review.score_id}/revisions`)); } catch (e) { fail(e); } }}>查看修改记录</Button>}{revisions.length > 0 && <pre className="json-view">{JSON.stringify(revisions, null, 2)}</pre>}</>}</Modal>
</>; </>;
} }