feat: separate contents, abstract and body evidence

This commit is contained in:
2026-10-01 00:07:40 +08:00
parent 540d0e2918
commit b7634c8374
25 changed files with 2592 additions and 172 deletions
+27 -1
View File
@@ -1,6 +1,6 @@
# WiseGrade(智评)交接说明 # WiseGrade(智评)交接说明
更新日期:2026-09-27。先阅读 [项目指南](docs/GUIDE.md)(部署、使用、技术设计与会话实现);本文件记录当前实现、验证边界和接手后的优先事项。 更新日期:2026-09-30。先阅读 [项目指南](docs/GUIDE.md)(部署、使用、技术设计与会话实现);本文件记录当前实现、验证边界和接手后的优先事项。
## 当前状态 ## 当前状态
@@ -79,6 +79,32 @@ uv run pytest -q
当前代码位于 `main` 分支,阶段提交使用 Conventional Commits。交接前工作区应保持干净;`.env`、`models/`、`data/`、缓存和构建产物由 `.gitignore` 排除。 当前代码位于 `main` 分支,阶段提交使用 Conventional Commits。交接前工作区应保持干净;`.env`、`models/`、`data/`、缓存和构建产物由 `.gitignore` 排除。
## 2026-09-30 目录、摘要与正文证据治理
### 本次改动
新增 `backend/wisegrade/pages.py`(页面分类、混合页切块、真实章节路径)和 `backend/wisegrade/evidence.py`(证据政策)。`rag.py` 重写为按目标检索:页面类型过滤在候选上限之前完成,语义与中文二元组关键词各自召回后按排名倒数融合,名额先分散到不同页面并对同章节设软上限。`scoring.py` 按必备证据目标分别检索,正文补检走目录导航,内容目标缺失时进入 `needs_review`。迁移 `0006` 添加页面类型、章节、证据目标与历史标记、评分项缺证据记录列,并按已保存的解析结果重新分类页面;索引版本升到 3。
关键词召回不再使用 PostgreSQL `to_tsvector('simple', …)`。在隔离测试库中对真实页面实测:该配置把整个中文串当成一个词元,查询"文献调研"无法命中含"第2章文献调研与技术现状"的页面(`文献调研`、`串级PID 结构`、`视觉伺服与闭环控制` 均不匹配),因此旧实现的关键词分实际上长期接近零。现改为字符二元组加 ASCII/数字整词的重叠度。
### 修复的缺陷
批次 3 的 77 条评分引文中 14 条来自目录页(涉及 9 个评分项),会话 6 的 20 条引用中 5 条指向阶段报告 1 的目录页。根因有两层:目录页文本被当作正文片段参与检索并占用候选名额;评分提示词只要求引文原文和页码一致,没有任何来源类型约束。现在目录页作为导航块单独存放,内容目标只接受正文和附录,引文必须标注它要支持的证据目标。
### 本次验证(隔离环境)
隔离 PostgreSQL + pgvector 测试容器执行 `pytest -q -m 'not integration'`:**59 通过、2 个真实集成测试未运行**。新增 `backend/tests/test_evidence_pages.py` 覆盖阶段报告 1 目录与第 6–7 页文献调研正文的区分、阶段报告 3 目录与第 11–12 页测试正文的区分、最终报告摘要/跨页目录/正文/附录的区分、混合页与目录续页与重复页眉与识别不确定页、内容性判断拒绝目录与摘要与单独标题、结构性判断仍可用目录、正文补检不足时模型分为 null 且状态为待教师复核且教师可复核且缺证据记录保留、Agent 引用页面类型标记与历史评分引用标记、检索过滤先于候选上限、学生项目与课程版本隔离、批次快照记录证据政策。前端 `npm run build` 通过(仍有 Vite 大块提示)。
真实报告的分类结果已逐页核对:阶段报告 1 第 1 页封面、第 2–3 页目录、第 4–21 页正文、第 22 页参考文献;阶段报告 3 第 2–3 页目录、第 11–12 页为测试正文;最终报告第 2–3 页摘要、第 4–5 页跨页目录、第 49 页参考文献、第 50–53 页附录。
### 未验证与待业务环境执行
- 使用真实 DeepSeek 的完整评分批次**未**在本次运行。需要模型的部分(评分调用、事实抽取、一致性检查、Agent 回答)只用测试替身验证了编排与校验逻辑,尚未用真实模型端到端复跑。
- 未对业务数据库执行迁移、重建索引或重新评分,也未运行 PDF/Excel 导出(导出代码已更新,但未实际生成文件)。
- 未在 Windows Docker Desktop 实测。
业务环境应用步骤与批次 3 复核清单见 [docs/EVIDENCE_POLICY_BATCH3.md](docs/EVIDENCE_POLICY_BATCH3.md)。
## 2026-09-27 会话更新 ## 2026-09-27 会话更新
新版 UI 的入口仍使用 `?page=chat`,旧 API 与历史记录保持可读。前端按运行 ID 订阅 SSE,并依靠会话列表和详情轮询恢复后台 Agent 运行;服务进程中断后的运行在 15 分钟超时后标为中断。正式 API 启动时会自动应用 `0004` 迁移。此处的测试结果以本次实际命令为准,历史验证记录仍保留在上方。 新版 UI 的入口仍使用 `?page=chat`,旧 API 与历史记录保持可读。前端按运行 ID 订阅 SSE,并依靠会话列表和详情轮询恢复后台 Agent 运行;服务进程中断后的运行在 15 分钟超时后标为中断。正式 API 启动时会自动应用 `0004` 迁移。此处的测试结果以本次实际命令为准,历史验证记录仍保留在上方。
+2
View File
@@ -22,6 +22,8 @@ WiseGrade 是面向《自动化专业综合实践》课程的教师端自动评
- [会话与业务 Agent 实现](docs/GUIDE.md#agent):任务、数据、工具边界、API 和恢复机制。 - [会话与业务 Agent 实现](docs/GUIDE.md#agent):任务、数据、工具边界、API 和恢复机制。
- [开发与测试](docs/GUIDE.md#development) · [故障定位](docs/GUIDE.md#troubleshooting)。 - [开发与测试](docs/GUIDE.md#development) · [故障定位](docs/GUIDE.md#troubleshooting)。
[证据政策上线与批次 3 复核清单](docs/EVIDENCE_POLICY_BATCH3.md) 记录把页面类型证据政策应用到业务环境的步骤,以及批次 3 建议复核的评分项。该文件中的业务数据操作均未执行。
[开发提示词](docs/prompts/README.md) 集中存放阶段任务说明;[交接记录](HANDOFF.md) 保留实现进展、验证记录和接手事项。 [开发提示词](docs/prompts/README.md) 集中存放阶段任务说明;[交接记录](HANDOFF.md) 保留实现进展、验证记录和接手事项。
课程原始资料及测试示例仍位于 `docs/` 根目录,应用和测试按现有路径读取;学生原件、模型权重和运行产物不纳入项目文档。 课程原始资料及测试示例仍位于 `docs/` 根目录,应用和测试按现有路径读取;学生原件、模型权重和运行产物不纳入项目文档。
@@ -0,0 +1,128 @@
"""Page-level evidence typing, target-attributed citations and review state.
Revision ID: 0006
Revises: 0005
Existing indexes are deliberately not upgraded in place. Chunking changed (a contents
page is now stored as navigation rather than body text), so legacy documents keep their
old ``index_version`` and are excluded from retrieval until the reindex task rebuilds
them. That is what stops an old index from being silently used as classified body text.
Reclassification of page types needs no embedding model, so this migration applies it
immediately for every document whose parsed pages are still saved on disk.
"""
import json
from alembic import op
import sqlalchemy as sa
revision = "0006"
down_revision = "0005"
branch_labels = None
depends_on = None
PAGE_COLUMNS = (
("page_type", sa.String(20), "unknown"),
("mixed", sa.Boolean(), "false"),
)
PAGE_TEXT_COLUMNS = (
("section", sa.String(500), ""),
("chapter", sa.String(500), ""),
)
CHUNK_COLUMNS = (
("chapter", sa.String(500), ""),
("page_type", sa.String(20), "unknown"),
("block_type", sa.String(20), "legacy"),
)
EVIDENCE_COLUMNS = (
("target_id", sa.String(50), ""),
("target_label", sa.String(300), ""),
("page_type", sa.String(20), "unknown"),
("source_role", sa.String(20), "content"),
("historical", sa.Boolean(), "false"),
)
def _columns(table):
return {column["name"] for column in sa.inspect(op.get_bind()).get_columns(table)}
def _add(table, name, column_type, default, *, nullable=False):
if name in _columns(table):
return False
op.add_column(table, sa.Column(name, column_type, nullable=nullable, server_default=default))
return True
def upgrade():
added = {}
for name, column_type, default in PAGE_COLUMNS:
added[f"knowledge_pages.{name}"] = _add("knowledge_pages", name, column_type, default)
added["knowledge_pages.section"] = _add("knowledge_pages", "section", sa.String(500), "")
added["knowledge_pages.chapter"] = _add("knowledge_pages", "chapter", sa.String(500), "")
added["knowledge_pages.confidence"] = _add("knowledge_pages", "confidence", sa.Float(), None, nullable=True)
added["knowledge_pages.classification"] = _add("knowledge_pages", "classification", sa.JSON(), "{}")
for name, column_type, default in CHUNK_COLUMNS:
added[f"knowledge_chunks.{name}"] = _add("knowledge_chunks", name, column_type, default)
for name, column_type, default in EVIDENCE_COLUMNS:
added[f"score_evidences.{name}"] = _add("score_evidences", name, column_type, default)
added["criterion_scores.evidence_gaps"] = _add("criterion_scores", "evidence_gaps", sa.JSON(), "[]")
db = op.get_bind()
# Quotations written before this policy existed are marked as historical so the
# interface never presents a contents-page quotation as report body evidence.
db.execute(sa.text("UPDATE score_evidences SET historical = true WHERE historical = false"))
# A legacy chunk was never split by block, so it may not be used as content
# evidence until the reindex task rebuilds it.
if added["knowledge_chunks.block_type"]:
db.execute(sa.text("UPDATE knowledge_chunks SET block_type = 'legacy'"))
_reclassify_saved_pages(db)
def _json(value):
return json.loads(value) if isinstance(value, str) else value or {}
def _reclassify_saved_pages(db):
"""Re-derive page types from saved parse output; retryable and audit friendly."""
from wisegrade.pages import classify_document
files = {row.id: _json(row.parsed) for row in db.execute(sa.text(
"SELECT id, parsed FROM submission_files")).mappings()}
documents = db.execute(sa.text(
"SELECT id, submission_file_id FROM knowledge_documents WHERE submission_file_id IS NOT NULL")).mappings().all()
update = sa.text("UPDATE knowledge_pages SET page_type = :page_type, mixed = :mixed, section = :section, "
"chapter = :chapter, confidence = :confidence, classification = :classification "
"WHERE document_id = :document_id AND page = :page")
chunk_update = sa.text("UPDATE knowledge_chunks SET page_type = :page_type, section = :section, "
"chapter = :chapter WHERE document_id = :document_id AND page = :page")
rebuilt = 0
for doc in documents:
pages = files.get(doc.submission_file_id, {}).get("pages", [])
if not pages:
continue
for item in classify_document(pages):
payload = {"page_type": item["page_type"], "mixed": bool(item["mixed"]),
"section": item["section"], "chapter": item["chapter"],
"confidence": item["confidence"],
"classification": json.dumps({**item["signals"],
"classifier_version": item["classifier_version"]},
ensure_ascii=False, default=str),
"document_id": doc.id, "page": item["page"]}
db.execute(update, payload)
db.execute(chunk_update, payload)
rebuilt += 1
if rebuilt:
db.execute(sa.text("INSERT INTO audit_logs (action, entity_type, entity_id, details, created_at) "
"VALUES ('reclassify', 'knowledge_document', NULL, :details, CURRENT_TIMESTAMP)"),
{"details": json.dumps({"migration": "0006", "documents": rebuilt}, ensure_ascii=False)})
def downgrade():
for table, names in (("criterion_scores", ("evidence_gaps",)),
("score_evidences", tuple(name for name, _, _ in EVIDENCE_COLUMNS)),
("knowledge_chunks", tuple(name for name, _, _ in CHUNK_COLUMNS)),
("knowledge_pages", ("page_type", "mixed", "section", "chapter", "confidence", "classification"))):
for name in names:
if name in _columns(table):
op.drop_column(table, name)
+3 -1
View File
@@ -138,6 +138,8 @@ def test_scoring_uses_the_same_page_image_as_its_saved_evidence(db, environment)
max_score=criterion["max_score"], reason="测试评分", max_score=criterion["max_score"], reason="测试评分",
evidence=[{"chunk_id": chunk.id, "page": 2, "quote": "原文证据"}], confidence=.9) evidence=[{"chunk_id": chunk.id, "page": 2, "quote": "原文证据"}], confidence=.9)
# The evidence page must be a substantive body page: a contents page cannot support
# a content judgement, so the source page is classified accordingly during ingest.
scored = _score_criterion(db, batch, project, criterion, FakeClient(), scored = _score_criterion(db, batch, project, criterion, FakeClient(),
retriever=lambda *_, **kwargs: [] if kwargs.get("public_only") else [evidence]) retriever=lambda *_, **kwargs: [] if kwargs.get("public_only") else [evidence])
saved = db.scalar(select(ScoreEvidence).where(ScoreEvidence.score_id == scored.id)) saved = db.scalar(select(ScoreEvidence).where(ScoreEvidence.score_id == scored.id))
@@ -333,7 +335,7 @@ def test_sse_persists_messages_and_tool_failure(session_factory, monkeypatch):
def test_tool_arguments_cover_registered_fields_without_review_prose(): def test_tool_arguments_cover_registered_fields_without_review_prose():
numbers = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", "rubric_version_id", numbers = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", "rubric_version_id",
"score", "offset", "limit", "project_offset", "criterion_offset"} "score", "offset", "limit", "project_offset", "criterion_offset", "document_id"}
strings = {"query", "criterion_id", "format", "status", "student_no", "relative_dir"} strings = {"query", "criterion_id", "format", "status", "student_no", "relative_dir"}
redacted = {"comment", "reason"} redacted = {"comment", "reason"}
fields = set().union(*(tool.schema.model_fields for tool in agent_tools.TOOLS.values())) fields = set().union(*(tool.schema.model_fields for tool in agent_tools.TOOLS.values()))
+9 -4
View File
@@ -224,7 +224,9 @@ def test_score_citations_link_the_canonical_text_and_image_page(session_factory,
assert generated["citations"][0]["url"] == expected assert generated["citations"][0]["url"] == expected
assert generated["citations"][0]["image_url"] == expected + "/image" assert generated["citations"][0]["image_url"] == expected + "/image"
assert generated["citations"][0]["score_url"].startswith("/?page=scores") assert generated["citations"][0]["score_url"].startswith("/?page=scores")
assert chat_api._answer_citations(generated["citations"])[0]["url"] == expected # An answer keeps only the citations it actually used: naming the page retains it.
assert chat_api._answer_citations(generated["citations"], "评分证据见第 2 页。")[0]["url"] == expected
assert chat_api._answer_citations(generated["citations"], "没有引用任何页面。") == []
conversation_id = conversation.id conversation_id = conversation.id
client = client_for(session_factory) client = client_for(session_factory)
try: try:
@@ -268,8 +270,11 @@ def test_late_citations_and_search_snippets_survive_restoration(session_factory,
assert response.status_code == 200 assert response.status_code == 200
detail = client.get(f"/api/conversations/{created['id']}").json() detail = client.get(f"/api/conversations/{created['id']}").json()
answer = detail['messages'][-1] answer = detail['messages'][-1]
assert len(answer['citations']) == 30 # The answer names page 48 only, so only that page's passages are attached; the
assert [item['page'] for item in answer['citations'][-6:]] == [48] * 6 # other tool hits were retrieved but not used in the answer.
assert len(answer['citations']) == 6
assert [item['page'] for item in answer['citations']] == [48] * 6
# Every tool hit stays available in the tool record for auditing.
assert answer['tool_calls'][-1]['result']['citations'][0]['quote'] == '第48页原文片段0' assert answer['tool_calls'][-1]['result']['citations'][0]['quote'] == '第48页原文片段0'
assert detail['runs'][0]['tool_calls'][-1]['result']['citations'][0]['page'] == 48 assert detail['runs'][0]['tool_calls'][-1]['result']['citations'][0]['page'] == 48
finally: finally:
@@ -383,7 +388,7 @@ def test_upgrade_from_legacy_0003_migrates_pages_and_citations(session_factory,
config.set_main_option('script_location', str(root / 'migrations')) config.set_main_option('script_location', str(root / 'migrations'))
command.upgrade(config, 'head') command.upgrade(config, 'head')
with session_factory() as db: with session_factory() as db:
assert db.scalar(text('SELECT version_num FROM alembic_version')) == '0005' assert db.scalar(text('SELECT version_num FROM alembic_version')) == '0006'
assert db.get(Conversation, conversation_id).title == '旧记录' assert db.get(Conversation, conversation_id).title == '旧记录'
assert db.scalar(select(ChatMessage.content).where(ChatMessage.conversation_id == conversation_id)) == '旧问题' assert db.scalar(select(ChatMessage.content).where(ChatMessage.conversation_id == conversation_id)) == '旧问题'
run = db.scalar(select(AgentRun).where(AgentRun.conversation_id == conversation_id)) run = db.scalar(select(AgentRun).where(AgentRun.conversation_id == conversation_id))
+444
View File
@@ -0,0 +1,444 @@
"""Regression tests for page-typed evidence retrieval and scoring.
The real course reports in ``docs/`` are used where the acceptance criteria name them,
plus synthetic pages for the cases those files do not contain (a genuinely mixed page,
a contents continuation, a repeated header, an unsettled page). A constant embedding
vector is used on purpose: it leaves semantic similarity flat, so the tests prove the
Chinese keyword recall and the page-type filtering, which is where the reported defect
came from.
"""
from pathlib import Path
import pymupdf as fitz
import pytest
from sqlalchemy import select
from wisegrade.evidence import normalize_targets
from wisegrade.llm import ModelError, ScoreResult, validate_score
from wisegrade.models import CriterionScore, KnowledgeChunk, Project, ScoreEvidence, Student
from wisegrade.pages import classify_document
from wisegrade.rag import retrieve, retrieve_targets
from wisegrade.scoring import _score_criterion
#: Constant vectors: semantic ranking contributes nothing, so recall is lexical.
FLAT = lambda texts: [[0.1] * 1024 for _ in texts]
COURSE_DOCS = ("阶段报告1.pdf", "阶段报告3.pdf", "自动化专业综合实践报告.pdf")
def docs_root() -> Path:
for candidate in (Path("/app/course-docs"), Path(__file__).resolve().parents[2] / "docs",
Path(__file__).resolve().parents[1] / "docs"):
if all((candidate / name).is_file() for name in COURSE_DOCS):
return candidate
pytest.skip("课程示例报告不可用")
def report_pages(name: str) -> list[dict]:
with fitz.open(docs_root() / name) as pdf:
return [{"page_number": index + 1, "text": page.get_text(sort=True)} for index, page in enumerate(pdf)]
def add_project(db, number: str):
student = Student(student_no=number, name=f"测试学生{number[-2:]}")
db.add(student)
db.flush()
project = Project(course_version_id=1, student_id=student.id, topic="证据治理测试",
source_dir=f"tests/evidence-{number}")
db.add(project)
db.flush()
return project
def ingest_pages(db, project, pages, *, category="stage1", title="报告.pdf", source_hash="hash"):
from wisegrade.rag import ingest
return ingest(db, course_version_id=1, project_id=project.id, submission_file_id=None, title=title,
category=category, source_hash=source_hash, pages=pages, embedder=FLAT)
def page_types(db, project):
from wisegrade.models import KnowledgeDocument, KnowledgePage
document = db.scalar(select(KnowledgeDocument).where(KnowledgeDocument.project_id == project.id))
rows = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id == document.id)
.order_by(KnowledgePage.page)).all()
return {row.page: row.page_type for row in rows}
# --------------------------------------------------------------------------- #
# 1. Stage report 1: contents pages versus the literature review body #
# --------------------------------------------------------------------------- #
def test_stage1_contents_is_separated_from_literature_body(db):
project = add_project(db, "123460001")
pages = report_pages("阶段报告1.pdf")
ingest_pages(db, project, pages)
types = page_types(db, project)
# The report's contents spans pages 2 and 3; the literature review is pages 6-7.
assert types[1] == "cover"
assert types[2] == types[3] == "toc"
assert types[6] == types[7] == "body"
assert types[22] == "references"
criterion = {"criterion_id": "stage1.literature", "category": "stage1", "max_score": 20,
"description": "文献调研和技术现状总结", "required_evidence": ["参考文献", "技术现状"]}
targets = normalize_targets(criterion["required_evidence"])
selection = retrieve_targets(db, course_version_id=1, project_id=project.id, targets=targets,
category="stage1", embedder=FLAT)
pages_hit = {chunk["page"] for chunk in selection["evidence"]}
# The literature analysis really exists on pages 6-7 and must be recalled there.
assert pages_hit & {6, 7}, pages_hit
content_types = {chunk["page_type"] for chunk in selection["evidence"]
if chunk["target_use"] == "content"}
assert content_types <= {"body", "appendix"}, content_types
# The structured target may cite the reference list, and it is labelled as such.
structure = [chunk for chunk in selection["evidence"] if chunk["target_use"] == "structure"]
assert all(chunk["page_type"] in ("toc", "references", "body", "cover", "abstract")
for chunk in structure)
def test_stage1_literature_analysis_is_not_taken_from_the_contents_page():
pages = report_pages("阶段报告1.pdf")
classified = {item["page"]: item for item in classify_document(pages)}
toc_text = "\n".join(block["text"] for block in classified[2]["blocks"] if block["kind"] == "toc")
assert "文献调研与技术现状" in toc_text # the heading really is in the contents
body_text = "\n".join(block["text"] for block in classified[6]["blocks"] if block["kind"] == "body")
assert "PID 控制因其结构简单" in body_text # the analysis really is in the body
# The contents page carries no body block, so it cannot supply content evidence.
assert not any(block["kind"] == "body" and block.get("substantive")
for block in classified[2]["blocks"])
# --------------------------------------------------------------------------- #
# 2. Stage report 3: contents versus the testing body #
# --------------------------------------------------------------------------- #
def test_stage3_contents_is_separated_from_testing_body(db):
project = add_project(db, "123460002")
ingest_pages(db, project, report_pages("阶段报告3.pdf"), category="stage3")
types = page_types(db, project)
assert types[2] == types[3] == "toc"
assert types[11] == types[12] == "body"
criterion = {"criterion_id": "stage3.data", "category": "stage3", "max_score": 20,
"description": "实验数据和初步分析", "required_evidence": ["实验数据", "分析"]}
selection = retrieve_targets(db, course_version_id=1, project_id=project.id,
targets=normalize_targets(criterion["required_evidence"]),
category="stage3", embedder=FLAT)
hits = [(chunk["page"], chunk["page_type"], chunk["text"][:40]) for chunk in selection["evidence"]]
pages_hit = {chunk["page"] for chunk in selection["evidence"]}
# The testing description on pages 11-12 must be reachable, not the contents.
assert pages_hit & {11, 12}, hits
assert all(chunk["page_type"] in ("body", "appendix") for chunk in selection["evidence"]), hits
# Finding the page is not a score: the evidence only has to be reachable.
assert any("测试" in chunk["text"] or "试验" in chunk["text"] for chunk in selection["evidence"])
# --------------------------------------------------------------------------- #
# 3. Final report: abstract, cross-page contents, body #
# --------------------------------------------------------------------------- #
def test_final_report_abstract_cross_page_contents_and_body(db):
project = add_project(db, "123460003")
ingest_pages(db, project, report_pages("自动化专业综合实践报告.pdf"), category="final_report")
types = page_types(db, project)
assert types[1] == "cover"
assert types[2] == types[3] == "abstract"
# The contents continues onto a second page (roman numeral IV) and is still contents.
assert types[4] == types[5] == "toc"
assert types[6] == "body"
assert types[49] == "references"
# Appendix pages stay usable for content, and are labelled as appendices.
assert types[50] == "appendix"
def test_contents_page_still_supports_chapter_completeness():
"""A structural criterion keeps using the contents page; that is its purpose."""
pages = report_pages("自动化专业综合实践报告.pdf")
classified = {item["page"]: item for item in classify_document(pages)}
from wisegrade.pages import toc_entries
entries = toc_entries(classified[4]["blocks"])
assert len(entries) >= 20
assert any("第1 章" in entry or "第1章" in entry for entry in entries)
criterion = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["标题", "章节", "参考文献"]}
targets = normalize_targets(criterion["required_evidence"])
assert all(target["use"] == "structure" for target in targets)
# A structure target admits the contents page; a content target never does.
assert all("toc" in target["allowed_page_types"] for target in targets)
# --------------------------------------------------------------------------- #
# 4. Mixed pages, continuations, repeated headers, unsettled pages #
# --------------------------------------------------------------------------- #
def test_mixed_page_contents_continuation_header_and_uncertain_page():
pages = [
{"page_number": 1, "text": "自动化专业综合实践\n阶段性报告\n姓名:张三\n学号:1234567890"},
# A contents page that also carries the start of the first chapter.
{"page_number": 2, "text": "目录\n第1 章 项目背景.......... 1\n第2 章 需求分析.......... 3\n"
"第1 章 项目背景\n本项目面向智能检测场景,通过多传感器融合实现目标识别。"},
# Contents continued: dotted entries and nothing else.
{"page_number": 3, "text": "3.2 性能测试.......... 9\n3.3 结果分析.......... 11\n参考文献.......... 15"},
# A repeated header plus real prose.
{"page_number": 4, "text": "自动化专业综合实践\n2.1 需求分析\n"
"系统需要完成目标检测、定位与结果输出,并在标准场地内稳定运行。\n"
"自动化专业综合实践"},
# A section title with no prose at all.
{"page_number": 5, "text": "自动化专业综合实践\n3.1 功能测试"},
]
classified = {item["page"]: item for item in classify_document(pages)}
# The repeated header is stripped from indexed text.
from wisegrade.pages import clean_page_lines, strip_repeated_lines
repeated = strip_repeated_lines(pages)
assert "自动化专业综合实践" in repeated
assert "自动化专业综合实践" not in clean_page_lines(pages[3], repeated)
assert classified[1]["page_type"] == "cover"
assert classified[2]["page_type"] == "toc" and classified[2]["mixed"] is True
# A mixed page keeps a usable body block instead of dropping the whole page.
body_blocks = [b for b in classified[2]["blocks"] if b["kind"] == "body" and b.get("substantive")]
assert body_blocks and "多传感器融合" in body_blocks[0]["text"]
# The contents continuation is recognised as contents, not body.
assert classified[3]["page_type"] == "toc"
assert classified[4]["page_type"] == "body"
# A page with only a section title is not silently ordinary body text.
assert classified[5]["page_type"] == "unknown"
def test_mixed_page_body_block_is_indexed_as_body_not_navigation(db):
project = add_project(db, "123460004")
pages = [{"page_number": 2, "text": "目录\n第1 章 项目背景.......... 1\n"
"第1 章 项目背景\n本项目面向智能检测场景,通过多传感器融合实现目标识别。"}]
ingest_pages(db, project, pages)
chunk = db.scalar(select(KnowledgeChunk).where(KnowledgeChunk.document_id.isnot(None),
KnowledgeChunk.page == 2,
KnowledgeChunk.block_type == "body"))
# The body half of the mixed page survives as its own searchable chunk.
assert chunk is not None and "多传感器融合" in chunk.text
navigation = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.page == 2,
KnowledgeChunk.block_type == "toc")).all()
assert navigation and all("多传感器融合" not in row.text for row in navigation)
def test_uncertain_page_is_not_offered_as_body_content(db):
project = add_project(db, "123460005")
pages = [{"page_number": 1, "text": "封面"},
{"page_number": 2, "text": "第1 章 项目背景与需求分析"},
{"page_number": 3, "text": "第1 章 项目背景与需求分析\n系统面向智能检测场景,完成目标识别与定位。"}]
ingest_pages(db, project, pages)
types = page_types(db, project)
assert types[2] == "unknown"
body_chunks = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.page_type == "body")).all()
assert all(row.page != 2 for row in body_chunks)
# --------------------------------------------------------------------------- #
# 5. Content judgements cannot rest on navigation pages #
# --------------------------------------------------------------------------- #
def test_content_judgement_rejects_navigation_sources_but_structure_accepts_them():
criterion = {"criterion_id": "stage3.data", "max_score": 20, "description": "实验数据和初步分析",
"required_evidence": ["实验数据", "分析"]}
toc = {"chunk_id": 1, "page": 2, "page_type": "toc", "text": "第3 章 系统测试与验证 ......... 8"}
result = ScoreResult(criterion_id="stage3.data", score=18, max_score=20, reason="看目录",
evidence=[{"chunk_id": 1, "page": 2, "quote": "系统测试与验证", "target_id": "t0"}],
confidence=.9)
with pytest.raises(ModelError):
validate_score(result, criterion, [toc])
structure = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["章节"]}
ok = ScoreResult(criterion_id="stage1.format", score=4, max_score=5, reason="章节完整",
evidence=[{"chunk_id": 1, "page": 2, "quote": "第3 章 系统测试与验证", "target_id": "t0"}],
confidence=.9)
assert validate_score(ok, structure, [toc]).score == 4
# --------------------------------------------------------------------------- #
# 6. Insufficient body evidence becomes a teacher review, never a low score #
# --------------------------------------------------------------------------- #
def test_insufficient_body_evidence_goes_to_review_with_a_null_model_score(db):
from types import SimpleNamespace
from wisegrade import services
from wisegrade.models import ScoringBatch, SubmissionFile
project = add_project(db, "123460006")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
batch = db.get(ScoringBatch, created["id"])
criterion = next(item for item in batch.rubric_snapshot["criteria"]
if item["criterion_id"] == "stage3.data")
file = SubmissionFile(project_id=project.id, category="stage3", relative_path="test/review/report.pdf",
mime_type="application/pdf", size=1, mtime=1, sha256="c" * 64,
recognition_status="recognized", parse_status="ready")
db.add(file)
db.flush()
# The material only contains a cover page and a contents page: no body at all.
pages = [{"page_number": 1, "text": "阶段性报告\n姓名:张三\n学号:1234567890"},
{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8\n第4 章 结论.......... 12"}]
file.parsed = {"pages": pages}
ingest_pages(db, project, pages, category="stage3", title="report.pdf", source_hash="c" * 64)
class NeverCalled:
def json_call(self, *_, **__):
raise AssertionError("缺少正文证据时不应调用评分模型")
scored = _score_criterion(db, batch, project, criterion, NeverCalled(),
retriever=lambda *_, **kw: [] if kw.get("public_only") else retrieve(
*_, **{**kw, "embedder": FLAT}))
assert scored.status == "needs_review"
assert scored.model_score is None
assert scored.evidence_gaps and scored.evidence_gaps[0]["target_label"]
assert scored.evidence_gaps[0]["searched_page_types"]
# The criterion shows as awaiting review, and the batch is not reported as complete.
counts = {row.status for row in db.scalars(select(CriterionScore).where(
CriterionScore.batch_id == batch.id)).all()}
assert "needs_review" in counts
def test_teacher_can_review_a_needs_review_item_and_the_gap_record_survives(db):
from types import SimpleNamespace
from wisegrade import services
project = add_project(db, "123460007")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
score = CriterionScore(batch_id=created["id"], project_id=project.id, criterion_id="stage3.data",
model_score=None, status="needs_review",
evidence_gaps=[{"target_id": "t0", "target_label": "实验数据",
"searched_page_types": ["body", "appendix"],
"reason": "补检后仍未找到正文证据"}])
db.add(score)
db.commit()
result = services.review_score(score.id, SimpleNamespace(score=17, comment="教师核对原始材料",
reason="材料确实存在测试章节"), db)
assert result["final_score"] == 17 and result["model_score"] is None
reviewed = db.get(CriterionScore, score.id)
assert reviewed.status == "reviewed"
# The original retrieval problem stays on the record after the teacher decides.
assert reviewed.evidence_gaps[0]["target_label"] == "实验数据"
# --------------------------------------------------------------------------- #
# 7. Agent citations only carry sources that were actually used #
# --------------------------------------------------------------------------- #
def test_search_tools_label_page_types_and_mark_historical_citations(db):
from wisegrade import agent_tools
from wisegrade.models import Conversation
project = add_project(db, "123460008")
ingest_pages(db, project, [
{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8"},
{"page_number": 11, "text": "第3 章 系统测试与验证\n在自主巡线测试中,底盘记录了直线段与弯道的时间数据。"},
], category="stage3")
conversation = Conversation(title="检索", course_version_id=1, project_id=project.id)
db.add(conversation)
db.commit()
original = agent_tools.retrieve
agent_tools.retrieve = lambda *args, **kwargs: original(*args, **kwargs, embedder=FLAT)
try:
content = agent_tools.execute(db, conversation, "search_project_material",
{"project_id": project.id, "query": "系统测试与验证"}, "")
assert content["found"]
# Content search never returns the contents page as body evidence.
assert all(item["page_type"] in ("body", "appendix") for item in content["citations"])
assert all(item["source_role"] == "content" for item in content["citations"])
assert all(item["citation_id"].startswith("chunk:") for item in content["citations"])
structure = agent_tools.execute(db, conversation, "search_project_structure",
{"project_id": project.id, "query": "系统测试"}, "")
assert structure["found"]
assert all(item["source_role"] == "structure" for item in structure["citations"])
# A structural hit is explicitly declared as unable to prove chapter content.
assert any(item["page_type"] == "toc" for item in structure["citations"])
assert "不能证明章节内容" in structure["note"]
finally:
agent_tools.retrieve = original
def test_scores_tool_marks_historical_score_evidence(db):
from types import SimpleNamespace
from wisegrade import agent_tools, services
from wisegrade.models import Conversation
project = add_project(db, "123460009")
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
score = CriterionScore(batch_id=created["id"], project_id=project.id, criterion_id="stage1.format",
model_score=4, status="scored")
db.add(score)
db.flush()
db.add(ScoreEvidence(score_id=score.id, document_id=None, submission_file_id=None, page=2,
quote="第1 章 项目背景与需求分析", image_path=None, page_type="toc",
target_label="章节", historical=True))
conversation = Conversation(title="评分", course_version_id=1, batch_id=created["id"])
db.add(conversation)
db.commit()
result = agent_tools.execute(db, conversation, "get_scores",
{"batch_id": created["id"], "project_id": project.id}, "")
citations = result["citations"]
assert citations and citations[0]["historical"] is True
assert citations[0]["source"] == "historical_score_evidence"
assert citations[0]["usage"] == "历史评分引用"
# --------------------------------------------------------------------------- #
# 8. Isolation and existing behaviour #
# --------------------------------------------------------------------------- #
def test_page_filtering_happens_before_the_candidate_limit(db):
"""A contents page must not crowd a body page out of the candidate window."""
project = add_project(db, "123460010")
pages = [{"page_number": 2, "text": "目录\n第3 章 系统测试与验证.......... 8"}]
for number in range(3, 25):
pages.append({"page_number": number,
"text": f"第3 章 系统测试与验证\n第{number} 页记录了系统测试的实测数据与分析过程。"})
ingest_pages(db, project, pages, category="stage3")
unrestricted = retrieve(db, course_version_id=1, project_id=project.id, query="系统测试与验证",
category="stage3", limit=3, embedder=FLAT, allowed_page_types=None)
filtered = retrieve(db, course_version_id=1, project_id=project.id, query="系统测试与验证",
category="stage3", limit=3, embedder=FLAT,
allowed_page_types=["body", "appendix"], substantive_only=True)
assert filtered and all(chunk["page_type"] in ("body", "appendix") for chunk in filtered)
assert {chunk["page"] for chunk in filtered} != {2}
# Without the filter the contents page is a candidate; the filter removes it before
# the limit is applied, so three real body pages still come back.
assert len(filtered) == 3
assert any(chunk["page_type"] == "toc" for chunk in unrestricted)
def test_project_and_version_isolation_is_unchanged(db):
first = add_project(db, "123460011")
second = add_project(db, "123460012")
ingest_pages(db, first, [{"page_number": 3, "text": "第3 章 系统测试与验证\n一号项目记录了巡线测试数据。"}],
category="stage3", title="一号.pdf", source_hash="one")
ingest_pages(db, second, [{"page_number": 3, "text": "第3 章 系统测试与验证\n二号项目的私有测试结果。"}],
category="stage3", title="二号.pdf", source_hash="two")
hits = retrieve(db, course_version_id=1, project_id=first.id, category="stage3",
query="系统测试与验证", embedder=FLAT, allowed_page_types=["body", "appendix"])
assert hits and all(hit["project_id"] == first.id for hit in hits)
assert "私有" not in str(hits)
# Course material stays in its own namespace and is not mixed into project search.
public = retrieve(db, course_version_id=1, project_id=first.id, query="系统测试与验证",
embedder=FLAT, public_only=True)
assert all(hit["project_id"] is None for hit in public)
def test_batch_snapshot_records_the_evidence_policy(db):
from types import SimpleNamespace
from wisegrade import services
from wisegrade.models import ScoringBatch
created = services.create_batch(SimpleNamespace(course_version_id=1, rubric_version_id=1), db)
assert created["id"]
batch = db.get(ScoringBatch, created["id"])
plan = batch.rubric_snapshot["evidence_plan"]
assert plan["version"] == 1
assert plan["content_allowed_page_types"] == ["body", "appendix"]
targets = plan["criteria"]["stage3.data"]
assert [target["use"] for target in targets] == ["content", "content"]
assert all("toc" not in target["allowed_page_types"] for target in targets)
# The legacy string form is read into typed targets without being rewritten.
assert [item["required_evidence"] for item in batch.rubric_snapshot["criteria"]
if item["criterion_id"] == "stage3.data"] == [["实验数据", "分析"]]
+1 -1
View File
@@ -10,7 +10,7 @@ from wisegrade.storage import classify, extract_identity, parse_student_folder,
def test_postgresql_migrations_and_vector_extension(db): def test_postgresql_migrations_and_vector_extension(db):
assert db.bind.dialect.name == "postgresql" assert db.bind.dialect.name == "postgresql"
assert db.scalar(text("SELECT version_num FROM alembic_version")) == "0005" assert db.scalar(text("SELECT version_num FROM alembic_version")) == "0006"
assert db.scalar(text("SELECT extname FROM pg_extension WHERE extname = 'vector'")) == "vector" assert db.scalar(text("SELECT extname FROM pg_extension WHERE extname = 'vector'")) == "vector"
+58 -5
View File
@@ -4,7 +4,7 @@ from sqlalchemy import select
from wisegrade.llm import DeepSeekClient, FactResult, ModelError, ScoreResult, validate_score from wisegrade.llm import DeepSeekClient, FactResult, ModelError, ScoreResult, validate_score
from wisegrade.models import AppSetting, KnowledgeChunk, KnowledgePage, Project, Student from wisegrade.models import AppSetting, KnowledgeChunk, KnowledgePage, Project, Student
from wisegrade.rag import ingest, local_encoder, retrieve, split_pages from wisegrade.rag import INDEX_VERSION, ingest, local_encoder, retrieve, split_pages
from wisegrade.settings_store import model_config, save_model_config from wisegrade.settings_store import model_config, save_model_config
@@ -21,7 +21,7 @@ def make_project(db, number):
def test_json_validation_and_evidence(): def test_json_validation_and_evidence():
criterion = {"criterion_id": "stage1.problem", "max_score": 25} criterion = {"criterion_id": "stage1.problem", "max_score": 25}
chunks = [{"chunk_id": 7, "page": 2, "text": "工程需求是实时目标识别"}] chunks = [{"chunk_id": 7, "page": 2, "text": "工程需求是实时目标识别", "page_type": "body"}]
result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="满足需求", result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="满足需求",
evidence=[{"chunk_id": 7, "page": 2, "quote": "实时目标识别"}], confidence=.8) evidence=[{"chunk_id": 7, "page": 2, "quote": "实时目标识别"}], confidence=.8)
assert validate_score(result, criterion, chunks).score == 19 assert validate_score(result, criterion, chunks).score == 19
@@ -33,8 +33,8 @@ def test_json_validation_and_evidence():
def test_evidence_quote_resolves_pdf_line_breaks_and_page(): def test_evidence_quote_resolves_pdf_line_breaks_and_page():
criterion = {"criterion_id": "stage1.problem", "max_score": 25} criterion = {"criterion_id": "stage1.problem", "max_score": 25}
chunks = [{"chunk_id": 7, "page": 2, "text": "机器人\n控制精度:2 cm。"}, chunks = [{"chunk_id": 7, "page": 2, "text": "机器人\n控制精度:2 cm。", "page_type": "body"},
{"chunk_id": 8, "page": 3, "text": "其他实验结果"}] {"chunk_id": 8, "page": 3, "text": "其他实验结果", "page_type": "body"}]
result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="有指标", result = ScoreResult(criterion_id="stage1.problem", score=19, max_score=25, reason="有指标",
evidence=[{"chunk_id": 8, "page": 3, "quote": "机器人控制精度:2 cm"}], confidence=.8) evidence=[{"chunk_id": 8, "page": 3, "quote": "机器人控制精度:2 cm"}], confidence=.8)
checked = validate_score(result, criterion, chunks) checked = validate_score(result, criterion, chunks)
@@ -42,6 +42,59 @@ def test_evidence_quote_resolves_pdf_line_breaks_and_page():
assert checked.evidence[0].quote in chunks[0]["text"] assert checked.evidence[0].quote in chunks[0]["text"]
def test_content_evidence_rejects_navigation_and_heading_only_sources():
"""A contents page, an abstract or a bare heading can never prove content."""
criterion = {"criterion_id": "stage3.data", "max_score": 20,
"description": "实验数据和初步分析", "required_evidence": ["实验数据", "分析"]}
body = {"chunk_id": 1, "page": 11, "page_type": "body",
"text": "在自主巡线测试中,底盘在标准黑色轨迹线场地内运行,记录了直线段与弯道的时间数据。"}
good = ScoreResult(criterion_id="stage3.data", score=16, max_score=20, reason="有实测数据",
evidence=[{"chunk_id": 1, "page": 11, "quote": "在自主巡线测试中", "target_id": "t0"}],
confidence=.8)
assert validate_score(good, criterion, [body]).score == 16
toc = {"chunk_id": 2, "page": 2, "page_type": "toc",
"text": "第3 章系统测试与验证 ......... 8"}
bad = ScoreResult(criterion_id="stage3.data", score=16, max_score=20, reason="看目录",
evidence=[{"chunk_id": 2, "page": 2, "quote": "系统测试与验证", "target_id": "t0"}],
confidence=.8)
# The rejection is what matters; either guard may fire depending on the target's
# allowed page types.
with pytest.raises(ModelError, match="内容证据|不允许的页面类型"):
validate_score(bad, criterion, [toc])
abstract = {"chunk_id": 3, "page": 2, "page_type": "abstract",
"text": "摘要 本文实现了自瞄移动机器人系统,采用多处理器分布式架构。"}
bad.evidence[0].chunk_id, bad.evidence[0].page, bad.evidence[0].quote = 3, 2, "本文实现了自瞄移动机器人系统"
with pytest.raises(ModelError, match="内容证据|不允许的页面类型"):
validate_score(bad, criterion, [abstract])
heading_only = {"chunk_id": 4, "page": 11, "page_type": "body", "text": "第3 章 系统测试与验证"}
bad.evidence[0].chunk_id, bad.evidence[0].page, bad.evidence[0].quote = 4, 11, "第3 章 系统测试与验证"
with pytest.raises(ModelError, match="章节标题"):
validate_score(bad, criterion, [heading_only])
def test_structural_targets_may_use_the_contents_page():
"""A structure target legitimately reads the contents list; a content one may not."""
criterion = {"criterion_id": "stage1.format", "max_score": 5, "description": "报告完整性和规范性",
"required_evidence": ["标题", "章节", "参考文献"]}
toc = {"chunk_id": 5, "page": 2, "page_type": "toc",
"text": "第1 章项目背景与需求分析 .......... 1\n第2 章文献调研与技术现状 .......... 3"}
result = ScoreResult(criterion_id="stage1.format", score=4, max_score=5, reason="章节完整",
evidence=[{"chunk_id": 5, "page": 2, "quote": "第1 章项目背景与需求分析", "target_id": "t1"}],
confidence=.8)
checked = validate_score(result, criterion, [toc])
assert checked.evidence[0].page == 2
def test_model_may_return_a_null_score_when_content_evidence_is_missing():
result = ScoreResult(criterion_id="stage3.data", score=None, max_score=20,
reason="仅检索到目录页,无法判断测试内容", evidence=[],
missing_evidence=["实验数据"], confidence=.2)
assert result.score is None
@pytest.mark.integration @pytest.mark.integration
def test_deepseek_adapter_live_with_image(tmp_path): def test_deepseek_adapter_live_with_image(tmp_path):
image = tmp_path / "page.png" image = tmp_path / "page.png"
@@ -87,7 +140,7 @@ def test_stale_knowledge_index_reuses_document_id(db):
db.commit() db.commit()
reindexed = ingest(db, course_version_id=1, project_id=project_id, submission_file_id=None, reindexed = ingest(db, course_version_id=1, project_id=project_id, submission_file_id=None,
title="报告", category="stage3", source_hash="same-source", pages=pages) title="报告", category="stage3", source_hash="same-source", pages=pages)
assert reindexed.id == document.id and reindexed.index_version == 2 assert reindexed.id == document.id and reindexed.index_version == INDEX_VERSION
assert db.scalar(select(KnowledgeChunk.text).where(KnowledgeChunk.document_id == document.id)) == pages[0]["text"] assert db.scalar(select(KnowledgeChunk.text).where(KnowledgeChunk.document_id == document.id)) == pages[0]["text"]
assert db.get(KnowledgePage, (document.id, 1)).text == pages[0]["text"] assert db.get(KnowledgePage, (document.id, 1)).text == pages[0]["text"]
+1 -1
View File
@@ -210,7 +210,7 @@ def prepare(db, conversation: Conversation, kind: str, payload: dict, *, teacher
else: else:
raise HTTPException(400, "不支持的操作") raise HTTPException(400, "不支持的操作")
expected = _state(db, kind, payload) expected = _state(db, kind, payload)
if kind == "start_batch" and expected["status"] not in ("created", "failed", "partial"): if kind == "start_batch" and expected["status"] not in ("created", "failed", "partial", "review_required"):
raise HTTPException(409, "批次不可启动或重试") raise HTTPException(409, "批次不可启动或重试")
if kind == "index_retry" and expected["status"] != "failed": if kind == "index_retry" and expected["status"] != "failed":
raise HTTPException(409, "仅失败任务可重试") raise HTTPException(409, "仅失败任务可重试")
+134 -10
View File
@@ -8,10 +8,12 @@ from pydantic import BaseModel, ConfigDict, Field
from sqlalchemy import select from sqlalchemy import select
from .actions import prepare, require_scope from .actions import prepare, require_scope
from .exports import batch_projects, project_result from .evidence import CONTENT_ALLOWED, STRUCTURE_ALLOWED
from .exports import STATUS_LABELS, batch_projects, page_type_label, project_result
from .knowledge_pages import page_links from .knowledge_pages import page_links
from .models import (Conversation, Course, CourseVersion, IndexTask, KnowledgeDocument, from .models import (Conversation, Course, CourseVersion, IndexTask, KnowledgeChunk, KnowledgeDocument,
Project, RubricVersion, ScoringBatch, Student, SubmissionFile) Project, RubricVersion, ScoringBatch, Student, SubmissionFile)
from .pages import toc_entries
from .rag import retrieve from .rag import retrieve
from .rules import totals from .rules import totals
from .storage import directories, scan from .storage import directories, scan
@@ -52,6 +54,14 @@ class SearchProject(ProjectId):
query: str = Field(min_length=2, max_length=300) query: str = Field(min_length=2, max_length=300)
class DocumentId(Args):
document_id: int = Field(gt=0)
class SearchDocument(DocumentId):
query: str = Field(min_length=2, max_length=300)
class Directory(Args): class Directory(Args):
relative_dir: str = Field(min_length=1, max_length=500) relative_dir: str = Field(min_length=1, max_length=500)
@@ -67,7 +77,7 @@ class BatchId(Args):
class ScoresFilterArgs(BatchId): class ScoresFilterArgs(BatchId):
project_id: int | None = Field(default=None, gt=0) project_id: int | None = Field(default=None, gt=0)
status: Literal["pending", "scored", "reviewed", "failed", "unscorable"] | None = None status: Literal["pending", "scored", "reviewed", "needs_review", "failed", "unscorable"] | None = None
class ScoresArgs(ScoresFilterArgs): class ScoresArgs(ScoresFilterArgs):
@@ -201,9 +211,14 @@ def project(db, convo, args, _user):
"url": f"/?page=projects&project_id={item.id}"} "url": f"/?page=projects&project_id={item.id}"}
def _search(db, version_id, project_id, query, *, public_only=False): def _search(db, version_id, project_id, query, *, public_only=False, scopes=None, limit=6):
"""Content search under the evidence policy: navigation pages are never body text."""
scopes = scopes or ("content",)
allowed = sorted({page_type for scope in scopes for page_type in STRUCTURE_ALLOWED} if "structure" in scopes
else {page_type for scope in scopes for page_type in CONTENT_ALLOWED})
chunks = retrieve(db, course_version_id=version_id, project_id=project_id or 0, query=query, chunks = retrieve(db, course_version_id=version_id, project_id=project_id or 0, query=query,
limit=6, include_public=False, public_only=public_only) limit=limit, include_public=False, public_only=public_only,
allowed_page_types=allowed, substantive_only="content" in scopes)
citations = [] citations = []
for chunk in chunks: for chunk in chunks:
doc = db.get(KnowledgeDocument, chunk["document_id"]) doc = db.get(KnowledgeDocument, chunk["document_id"])
@@ -211,8 +226,15 @@ def _search(db, version_id, project_id, query, *, public_only=False):
citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"], citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": doc.title if doc else "资料", "page": chunk["page"], "title": doc.title if doc else "资料", "page": chunk["page"],
"quote": chunk["text"][:500], "project_id": chunk["project_id"], "quote": chunk["text"][:500], "project_id": chunk["project_id"],
"page_type": chunk.get("page_type") or links.get("page_type") or "unknown",
"page_type_label": page_type_label(chunk.get("page_type")),
"section": chunk.get("section") or "",
"block_type": chunk.get("block_type") or "body",
"source_role": "structure" if "structure" in scopes else "content",
# Stable identity so an answer can cite exactly what it used.
"citation_id": f"chunk:{chunk['chunk_id']}",
**links}) **links})
return {"citations": citations, "found": bool(citations)} return {"citations": citations, "found": bool(citations), "scopes": list(scopes)}
def search_course(db, convo, args, _user): def search_course(db, convo, args, _user):
@@ -225,6 +247,84 @@ def search_project(db, convo, args, _user):
return _search(db, item.course_version_id, item.id, args.query) return _search(db, item.course_version_id, item.id, args.query)
def search_structure(db, convo, args, _user):
"""Structure and contents lookup: navigation pages plus body headings.
This tool answers questions about chapters, sections and report layout. Its results
are explicitly labelled as structure, so they can never be mistaken for content
evidence about what the student implemented or measured.
"""
item = _project(db, convo, args.project_id)
chunks = retrieve(db, course_version_id=item.course_version_id, project_id=item.id, query=args.query,
limit=8, include_public=False, allowed_page_types=sorted(STRUCTURE_ALLOWED),
substantive_only=False)
citations = []
for chunk in chunks:
doc = db.get(KnowledgeDocument, chunk["document_id"])
links = page_links(db, chunk["document_id"], chunk["page"])
page_type = chunk.get("page_type") or links.get("page_type") or "unknown"
citations.append({
"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": doc.title if doc else "资料", "page": chunk["page"],
"quote": chunk["text"][:500], "project_id": chunk["project_id"],
"page_type": page_type, "page_type_label": page_type_label(page_type),
"section": chunk.get("section") or "", "block_type": chunk.get("block_type") or "body",
"source_role": "structure",
"citation_id": f"chunk:{chunk['chunk_id']}",
"toc_entries": toc_entries([{"kind": "toc", "text": chunk["text"]}])[:40]
if page_type == "toc" else [],
**links})
return {"citations": citations, "found": bool(citations), "scopes": ["structure"],
"note": "这些是结构或目录信息,只能说明章节安排,不能证明章节内容。"}
def get_document_pages(db, convo, args, _user):
"""List a document's pages with page type, so a caller can navigate precisely."""
_document(db, convo, args.document_id)
rows = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.document_id == args.document_id)
.order_by(KnowledgeChunk.page, KnowledgeChunk.id)).all()
pages: dict[int, dict] = {}
for chunk in rows:
entry = pages.setdefault(chunk.page, {"page": chunk.page, "page_type": chunk.page_type,
"page_type_label": page_type_label(chunk.page_type),
"chapter": chunk.chapter, "section": chunk.section,
"block_types": []})
if chunk.block_type not in entry["block_types"]:
entry["block_types"].append(chunk.block_type)
if chunk.section:
entry["section"] = chunk.section
return {"document_id": args.document_id, "count": len(pages),
"pages": [pages[key] for key in sorted(pages)]}
def search_document(db, convo, args, _user):
"""Content search restricted to one document, under the same page-type policy."""
document = _document(db, convo, args.document_id)
chunks = retrieve(db, course_version_id=document.course_version_id, project_id=document.project_id or 0,
query=args.query, limit=6, include_public=False,
allowed_page_types=sorted(CONTENT_ALLOWED), substantive_only=True)
citations = []
for chunk in chunks:
if chunk["document_id"] != document.id:
continue
links = page_links(db, chunk["document_id"], chunk["page"])
citations.append({"chunk_id": chunk["chunk_id"], "document_id": chunk["document_id"],
"title": document.title, "page": chunk["page"], "quote": chunk["text"][:500],
"project_id": chunk["project_id"], "page_type": chunk.get("page_type"),
"page_type_label": page_type_label(chunk.get("page_type")),
"section": chunk.get("section") or "", "source_role": "content",
"citation_id": f"chunk:{chunk['chunk_id']}", **links})
return {"citations": citations, "found": bool(citations), "scopes": ["content"]}
def _document(db, convo, document_id):
document = db.get(KnowledgeDocument, document_id)
if not document:
raise HTTPException(404, "资料不存在")
require_scope(db, convo, version_id=document.course_version_id, project_id=document.project_id)
return document
def batch_list(db, convo, args, _user): def batch_list(db, convo, args, _user):
query = select(ScoringBatch).order_by(ScoringBatch.id.desc()) query = select(ScoringBatch).order_by(ScoringBatch.id.desc())
rows = [] rows = []
@@ -278,15 +378,36 @@ def scores(db, convo, args, _user):
"max_score": row["criterion"]["max_score"], "score_id": row["score_id"], "max_score": row["criterion"]["max_score"], "score_id": row["score_id"],
"model_score": row["model_score"], "teacher_score": row["teacher_score"], "model_score": row["model_score"], "teacher_score": row["teacher_score"],
"final_score": row["final_score"], "status": row["status"], "final_score": row["final_score"], "status": row["status"],
"error": str(row["error"] or "")[:300], "evidence": [{"page": ev["page"], "status_label": STATUS_LABELS.get(row["status"], row["status"]),
"quote": ev["quote"][:160], "document": str(ev["document"])[:200]} "error": str(row["error"] or "")[:300],
# Which body targets still lack evidence, so the answer can
# explain a review item instead of reporting a score.
"evidence_gaps": [{"target_label": str(gap.get("target_label") or "")[:120],
"searched_page_types": gap.get("searched_page_types") or []}
for gap in (row.get("evidence_gaps") or [])][:6],
"evidence": [{"page": ev["page"],
"quote": ev["quote"][:160], "document": str(ev["document"])[:200],
"page_type": ev.get("page_type") or "unknown",
"page_type_label": page_type_label(ev.get("page_type")),
"target_label": str(ev.get("target_label") or "")[:120],
"historical": bool(ev.get("historical"))}
for ev in row["evidence"][:2]]} for row in rows]}) for ev in row["evidence"][:2]]} for row in rows]})
for row in rows: for row in rows:
for evidence in row["evidence"][:2]: for evidence in row["evidence"][:2]:
links = page_links(db, evidence.get("document_id"), evidence["page"]) links = page_links(db, evidence.get("document_id"), evidence["page"])
citations.append({"title": str(evidence["document"])[:200], "page": evidence["page"], citations.append({"title": str(evidence["document"])[:200], "page": evidence["page"],
"quote": evidence["quote"][:320], "document_id": evidence.get("document_id"), "quote": evidence["quote"][:320], "document_id": evidence.get("document_id"),
"page_type": evidence.get("page_type") or links.get("page_type") or "unknown",
"page_type_label": page_type_label(evidence.get("page_type")),
"target_label": str(evidence.get("target_label") or "")[:120],
"historical": bool(evidence.get("historical")),
"source": "historical_score_evidence" if evidence.get("historical") else "score_evidence",
**links, "score_url": url}) **links, "score_url": url})
if citations:
# Scoring quotations are never presented as report body evidence in an answer.
for citation in citations:
citation["usage"] = ("历史评分引用" if citation.get("historical")
else f"评分证据 · {citation.get('page_type_label')}")
next_criterion_offset = args.criterion_offset + 6 if page and args.criterion_offset + 6 < len(page[0][1]["rows"]) else None next_criterion_offset = args.criterion_offset + 6 if page and args.criterion_offset + 6 < len(page[0][1]["rows"]) else None
next_project_offset = args.project_offset + 1 if args.project_offset + 1 < len(results) else None next_project_offset = args.project_offset + 1 if args.project_offset + 1 < len(results) else None
return {"batch_id": batch.id, "status_filter": args.status, "count": len(results), return {"batch_id": batch.id, "status_filter": args.status, "count": len(results),
@@ -418,12 +539,15 @@ def prepare_import(db, convo, args, _user):
TOOLS: dict[str, Tool] = { TOOLS: dict[str, Tool] = {
"list_courses": Tool("列出课程、版本和发布状态;名称或版本不清楚时先调用此工具。", Empty, courses), "list_courses": Tool("列出课程、版本和发布状态;名称或版本不清楚时先调用此工具。", Empty, courses),
"get_course_rules": Tool("查询课程目标和规则。指定批次时只返回批次保存的快照。", RulesArgs, rules), "get_course_rules": Tool("查询课程目标和规则。指定批次时只返回批次保存的快照。", RulesArgs, rules),
"search_course_material": Tool("检索指定课程版本的课程资料,返回原文片段和引用。", SearchCourse, search_course), "search_course_material": Tool("检索指定课程版本的课程资料正文,返回原文片段和引用;不含目录、摘要和参考文献页。", SearchCourse, search_course),
"list_directories": Tool("只读浏览 /data/inbox 下的相对目录。", Directory, browse_directories), "list_directories": Tool("只读浏览 /data/inbox 下的相对目录。", Directory, browse_directories),
"scan_directory": Tool("只读分页扫描学生总目录,返回材料候选、缺失、重复和识别异常。", ScanArgs, scan_directories), "scan_directory": Tool("只读分页扫描学生总目录,返回材料候选、缺失、重复和识别异常。", ScanArgs, scan_directories),
"list_projects": Tool("按课程版本及可选学号查询学生项目。", ProjectList, projects), "list_projects": Tool("按课程版本及可选学号查询学生项目。", ProjectList, projects),
"get_project": Tool("读取单个学生项目的元数据、材料和解析状态。", ProjectId, project), "get_project": Tool("读取单个学生项目的元数据、材料和解析状态。", ProjectId, project),
"search_project_material": Tool("只检索指定学生项目报告,返回原文片段和引用。", SearchProject, search_project), "search_project_material": Tool("只检索指定学生项目的报告正文,返回带页面类型和稳定引用标识的原文片段;不含目录、摘要和参考文献页。", SearchProject, search_project),
"search_project_structure": Tool("查询学生报告的章节结构与目录,返回目录条目和页面类型;只能说明章节安排,不能证明章节内容。", SearchProject, search_structure),
"get_document_pages": Tool("列出某份资料各页的页码、页面类型和章节,用于判断正文页位置。", DocumentId, get_document_pages),
"search_document": Tool("在指定的一份资料内检索正文段落,返回带页面类型和引用标识的片段。", SearchDocument, search_document),
"list_batches": Tool("列出可访问的评分批次。", Empty, batch_list), "list_batches": Tool("列出可访问的评分批次。", Empty, batch_list),
"get_batch_progress": Tool("查询评分批次进度、状态和固定项目范围。", BatchId, progress), "get_batch_progress": Tool("查询评分批次进度、状态和固定项目范围。", BatchId, progress),
"get_scores": Tool("分页查询单学生或单批次评分详情,可按状态筛选。每次返回一个项目、最多六项评分;如需其余评分项,用 next_criterion_offset 作为 criterion_offset 再查;如需下个项目,用 next_project_offset 作为 project_offset,并将 criterion_offset 设为 0。", ScoresArgs, scores), "get_scores": Tool("分页查询单学生或单批次评分详情,可按状态筛选。每次返回一个项目、最多六项评分;如需其余评分项,用 next_criterion_offset 作为 criterion_offset 再查;如需下个项目,用 next_project_offset 作为 project_offset,并将 criterion_offset 设为 0。", ScoresArgs, scores),
+75 -4
View File
@@ -10,13 +10,16 @@ from sqlalchemy.orm import Session
from . import services from . import services
from .config import get_settings from .config import get_settings
from .db import get_db from .db import get_db
from .exports import batch_projects, export_excel, export_project_pdf, project_result from .evidence import CONTENT_ALLOWED
from .exports import (BATCH_STATUS_LABELS, batch_projects, export_excel,
export_project_pdf, page_type_label, project_result)
from .knowledge_pages import page_image_path, page_links from .knowledge_pages import page_image_path, page_links
from .models import (AuditLog, Course, CourseVersion, CriterionScore, ExportTask, from .models import (AuditLog, Course, CourseVersion, CriterionScore, ExportTask,
IndexTask, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project, RubricVersion, ScoringBatch, IndexTask, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project, RubricVersion, ScoringBatch,
Student, SubmissionFile, TeacherRevision) Student, SubmissionFile, TeacherRevision)
from .materialized import sync_course_entities, sync_rubric_entities from .materialized import sync_course_entities, sync_rubric_entities
from .rules import RubricData from .rules import RubricData
from .scoring import ATTENTION_STATUSES
from .settings_store import key_configured, model_config, save_model_config from .settings_store import key_configured, model_config, save_model_config
from .storage import CATEGORIES, directories, scan from .storage import CATEGORIES, directories, scan
@@ -108,9 +111,20 @@ class ModelSettingsInput(BaseModel):
@router.get("/dashboard") @router.get("/dashboard")
def dashboard(db: Session = Depends(get_db)): def dashboard(db: Session = Depends(get_db)):
"""Overview counts. Items awaiting teacher review are counted as attention needed.
A ``needs_review`` item is not a failure: retrieval ran and found only navigation
pages where the rule requires body content, so a teacher decides it.
"""
counts = dict(db.execute(select(CriterionScore.status, func.count(CriterionScore.id))
.group_by(CriterionScore.status)).all())
return {"courses": db.scalar(select(func.count(Course.id))), "projects": db.scalar(select(func.count(Project.id))), return {"courses": db.scalar(select(func.count(Course.id))), "projects": db.scalar(select(func.count(Project.id))),
"batches": db.scalar(select(func.count(ScoringBatch.id))), "batches": db.scalar(select(func.count(ScoringBatch.id))),
"needs_attention": db.scalar(select(func.count(CriterionScore.id)).where(CriterionScore.status.in_(["failed", "unscorable"]))) } "needs_attention": sum(counts.get(status, 0) for status in ATTENTION_STATUSES),
"needs_review": counts.get("needs_review", 0),
"unscorable": counts.get("unscorable", 0),
"failed": counts.get("failed", 0),
"status_counts": counts}
@router.get("/courses") @router.get("/courses")
@@ -337,9 +351,32 @@ def knowledge_page(document_id: int, page: int, db: Session = Depends(get_db)):
return {"document_id": doc.id, "title": doc.title, "course_version_id": doc.course_version_id, return {"document_id": doc.id, "title": doc.title, "course_version_id": doc.course_version_id,
"project_id": doc.project_id, "page": page, "project_id": doc.project_id, "page": page,
"text": record.text, "text": record.text,
"page_type": record.page_type or "unknown",
"page_type_label": page_type_label(record.page_type),
"mixed": bool(record.mixed),
"section": record.section or "", "chapter": record.chapter or "",
"confidence": record.confidence,
"classification": record.classification or {},
"content_allowed": (record.page_type or "unknown") in CONTENT_ALLOWED,
"image_url": page_links(db, document_id, page)["image_url"]} "image_url": page_links(db, document_id, page)["image_url"]}
@router.get("/knowledge/documents/{document_id}/pages")
def knowledge_document_pages(document_id: int, db: Session = Depends(get_db)):
"""Page inventory with types, so the teacher can see how a document was classified."""
doc = found(db, KnowledgeDocument, document_id)
records = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id == document_id)
.order_by(KnowledgePage.page)).all()
return {"document_id": doc.id, "title": doc.title, "index_version": doc.index_version,
"pages": [{"page": item.page, "page_type": item.page_type or "unknown",
"page_type_label": page_type_label(item.page_type),
"mixed": bool(item.mixed), "section": item.section or "",
"chapter": item.chapter or "", "confidence": item.confidence,
"classification": item.classification or {},
"content_allowed": (item.page_type or "unknown") in CONTENT_ALLOWED}
for item in records]}
@router.get("/knowledge/documents/{document_id}/pages/{page}/image") @router.get("/knowledge/documents/{document_id}/pages/{page}/image")
def knowledge_page_image(document_id: int, page: int, db: Session = Depends(get_db)): def knowledge_page_image(document_id: int, page: int, db: Session = Depends(get_db)):
doc = found(db, KnowledgeDocument, document_id) doc = found(db, KnowledgeDocument, document_id)
@@ -354,6 +391,28 @@ def ingest_course(course_version_id: int, db: Session = Depends(get_db)):
return services.ingest_course(course_version_id, db) return services.ingest_course(course_version_id, db)
@router.get("/knowledge/stale/{course_version_id}")
def stale_knowledge(course_version_id: int, db: Session = Depends(get_db)):
"""Documents still on an older index; they are excluded from retrieval until rebuilt."""
from .indexing import stale_documents
from .rag import INDEX_VERSION
found(db, CourseVersion, course_version_id)
documents = stale_documents(db, course_version_id)
return {"index_version": INDEX_VERSION,
"documents": [{"id": item.id, "title": item.title, "type": item.document_type,
"project_id": item.project_id, "index_version": item.index_version,
"status": item.status} for item in documents]}
@router.post("/knowledge/reindex/{course_version_id}")
def reindex_knowledge(course_version_id: int, db: Session = Depends(get_db)):
"""Queue a rebuild of every stale index for this course version."""
from .indexing import create_reindex_task
found(db, CourseVersion, course_version_id)
task = create_reindex_task(db, course_version_id)
return get_index_task(task.id, db)
@router.get("/index-tasks") @router.get("/index-tasks")
def list_index_tasks(project_id: int | None = None, db: Session = Depends(get_db)): def list_index_tasks(project_id: int | None = None, db: Session = Depends(get_db)):
query = select(IndexTask).order_by(IndexTask.id.desc()).limit(100) query = select(IndexTask).order_by(IndexTask.id.desc()).limit(100)
@@ -381,10 +440,20 @@ def create_batch(data: BatchInput, db: Session = Depends(get_db)):
return services.create_batch(data, db) return services.create_batch(data, db)
def batch_status_counts(db, batch_id: int) -> dict:
rows = db.execute(select(CriterionScore.status, func.count(CriterionScore.id))
.where(CriterionScore.batch_id == batch_id).group_by(CriterionScore.status)).all()
counts = {status: count for status, count in rows}
return {"status_counts": counts, "needs_review": counts.get("needs_review", 0),
"unscorable": counts.get("unscorable", 0), "failed": counts.get("failed", 0)}
@router.get("/scoring-batches") @router.get("/scoring-batches")
def list_batches(db: Session = Depends(get_db)): def list_batches(db: Session = Depends(get_db)):
return [{**{key: getattr(batch, key) for key in ("id", "course_version_id", "rubric_version_id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")}, return [{**{key: getattr(batch, key) for key in ("id", "course_version_id", "rubric_version_id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")},
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None} "status_label": BATCH_STATUS_LABELS.get(batch.status, batch.status),
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None,
**batch_status_counts(db, batch.id)}
for batch in db.scalars(select(ScoringBatch).order_by(ScoringBatch.id.desc()))] for batch in db.scalars(select(ScoringBatch).order_by(ScoringBatch.id.desc()))]
@@ -392,7 +461,9 @@ def list_batches(db: Session = Depends(get_db)):
def batch_progress(batch_id: int, db: Session = Depends(get_db)): def batch_progress(batch_id: int, db: Session = Depends(get_db)):
batch = found(db, ScoringBatch, batch_id) batch = found(db, ScoringBatch, batch_id)
return {**{key: getattr(batch, key) for key in ("id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")}, return {**{key: getattr(batch, key) for key in ("id", "status", "progress_done", "progress_total", "error", "created_at", "finished_at")},
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None} "status_label": BATCH_STATUS_LABELS.get(batch.status, batch.status),
"project_ids": batch.project_ids, "legacy_membership": batch.project_ids is None,
**batch_status_counts(db, batch_id)}
@router.post("/scoring-batches/{batch_id}/start") @router.post("/scoring-batches/{batch_id}/start")
+46 -11
View File
@@ -1,6 +1,8 @@
"""Persistent teacher conversations and SSE agent orchestration.""" """Persistent teacher conversations and SSE agent orchestration."""
import json import json
import re
import time import time
import unicodedata
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import timedelta from datetime import timedelta
from threading import Lock from threading import Lock
@@ -170,7 +172,8 @@ def _safe_arguments(arguments):
if not isinstance(arguments, dict): if not isinstance(arguments, dict):
return {} return {}
numeric = {"course_version_id", "batch_id", "project_id", "score_id", "task_id", numeric = {"course_version_id", "batch_id", "project_id", "score_id", "task_id",
"rubric_version_id", "score", "offset", "limit", "project_offset", "criterion_offset"} "rubric_version_id", "score", "offset", "limit", "project_offset", "criterion_offset",
"document_id"}
text_limits = {"query": 300, "criterion_id": 100, "format": 20, "status": 30, text_limits = {"query": 300, "criterion_id": 100, "format": 20, "status": 30,
"student_no": 50, "relative_dir": 500} "student_no": 50, "relative_dir": 500}
result = {} result = {}
@@ -219,26 +222,58 @@ def _result_summary(result):
def _safe_citation(item): def _safe_citation(item):
if not isinstance(item, dict): if not isinstance(item, dict):
return None return None
return {key: value[:limit] if isinstance(value, str) else value for key, limit in # The evidence-policy fields travel with the citation so the interface can tell a
(("title", 200), ("quote", 500), ("url", 500), ("image_url", 500), ("score_url", 500)) # report body page from a contents page, and an answer's source from a historical
# scoring quotation.
label_limits = (("title", 200), ("quote", 500), ("url", 500), ("image_url", 500), ("score_url", 500),
("citation_id", 80), ("page_type", 20), ("page_type_label", 30),
("source_role", 20), ("source", 40), ("usage", 60), ("target_label", 120))
return {key: value[:limit] if isinstance(value, str) else value for key, limit in label_limits
if isinstance((value := item.get(key)), str)} | { if isinstance((value := item.get(key)), str)} | {
key: item[key] for key in ("chunk_id", "document_id", "page", "project_id") key: item[key] for key in ("chunk_id", "document_id", "page", "project_id", "historical")
if isinstance(item.get(key), int)} if isinstance(item.get(key), (int, bool)) and not isinstance(item.get(key), str)}
def _answer_citations(items): def _comparable_text(value):
"""Keep each distinct source passage, including later tool calls and pages cited by the answer.""" return "".join(char for char in unicodedata.normalize("NFKC", str(value or "")) if not char.isspace())
def _cited_pages(content: str) -> set[int]:
"""Page and slide numbers the answer actually refers to."""
numbers = set()
for match in re.finditer(r"第\s*(\d{1,4})\s*(?:页|张幻灯片|页幻灯片)", str(content or "")):
numbers.add(int(match.group(1)))
return numbers
def _answer_citations(items, content: str = ""):
"""Attach only the citations the answer actually used.
Collecting every tool hit would list material the answer never drew on. A passage is
kept when the answer quotes it, or when the answer names the page it sits on. When
the answer names no page and quotes nothing, no citation is attached: the answer made
no specific evidential claim.
"""
seen = set() seen = set()
citations = [] citations = []
answer = _comparable_text(content)
pages = _cited_pages(content)
for item in items: for item in items:
safe = _safe_citation(item) safe = _safe_citation(item)
if not safe: if not safe:
continue continue
key = (safe.get("document_id"), safe.get("chunk_id"), safe.get("title"), key = (safe.get("document_id"), safe.get("chunk_id"), safe.get("title"),
safe.get("page"), safe.get("quote"), safe.get("url"), safe.get("score_url")) safe.get("page"), safe.get("quote"), safe.get("url"), safe.get("score_url"))
if key not in seen: if key in seen:
seen.add(key) continue
citations.append(safe) quote = _comparable_text(safe.get("quote"))
used = bool(quote) and len(quote) >= 6 and quote[:80] in answer
if not used and pages and isinstance(safe.get("page"), int):
used = safe["page"] in pages
if not used:
continue
seen.add(key)
citations.append(safe)
return citations return citations
@@ -326,7 +361,7 @@ def _run_agent(factory, conversation_id, run_id, user_text):
content = reply.get("content") content = reply.get("content")
if not isinstance(content, str) or not content.strip(): if not isinstance(content, str) or not content.strip():
raise ModelError("会话模型未返回文字或工具调用") raise ModelError("会话模型未返回文字或工具调用")
answer_citations = _answer_citations(citations) answer_citations = _answer_citations(citations, content)
db.add(ChatMessage(conversation_id=conversation_id, role="assistant", content=content, db.add(ChatMessage(conversation_id=conversation_id, role="assistant", content=content,
citations=answer_citations, tool_calls=used)) citations=answer_citations, tool_calls=used))
run.status, run.finished_at = "completed", now() run.status, run.finished_at = "completed", now()
+150
View File
@@ -0,0 +1,150 @@
"""Evidence policy: which page types may support which kind of judgement.
The policy is part of the versionable scoring rules. Each required-evidence target
declares its ``use``:
* ``structure`` — the target may be satisfied by a cover page, table of contents or
reference list (chapter completeness, bibliography, report layout).
* ``content`` — the target may only be satisfied by substantive text, data or tables
on body or appendix pages.
Rules still in the historical string form (``["问题定义", "需求分析"]``) are read with a
permissive grammar: wording that names a structural artefact becomes a structure
target, everything else becomes a content target. The exact grammar is recorded as
``POLICY_VERSION`` in the batch snapshot so a batch can always be re-read against the
rules it was scored with.
"""
from __future__ import annotations
import re
from .pages import (CONTENT_PAGE_TYPES, NAVIGATION_PAGE_TYPES, STORE_PAGE_TYPES,
STRUCTURE_PAGE_TYPES)
POLICY_VERSION = 1
USE_STRUCTURE = "structure"
USE_CONTENT = "content"
#: Wording that names report packaging rather than what the student did.
_STRUCTURE_WORDS = (
"标题", "封面", "目录", "章节", "参考文献", "引用规范", "格式", "规范性", "结构完整",
"排版", "页码", "附录", "图表编号", "摘要",
)
#: Appendix and body pages are the only pages allowed for a content judgement.
CONTENT_ALLOWED = tuple(CONTENT_PAGE_TYPES)
#: Structural targets may additionally use cover, contents and reference pages.
STRUCTURE_ALLOWED = tuple(STRUCTURE_PAGE_TYPES)
#: Navigation pages can never carry a content judgement.
NAVIGATION = tuple(NAVIGATION_PAGE_TYPES)
def _looks_structural(text: str) -> bool:
value = str(text or "").strip()
return bool(value) and any(word in value for word in _STRUCTURE_WORDS)
def normalize_target(raw) -> dict:
"""Read one required-evidence target in either the rich or the legacy string form."""
if isinstance(raw, dict):
label = str(raw.get("label") or raw.get("text") or raw.get("name") or "").strip()
use = str(raw.get("use") or "").strip().lower()
if use not in (USE_STRUCTURE, USE_CONTENT):
use = USE_STRUCTURE if _looks_structural(label) else USE_CONTENT
allowed = raw.get("allowed_page_types")
if not isinstance(allowed, list) or not allowed:
allowed = list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED)
allowed = [str(item) for item in allowed if str(item) in STORE_PAGE_TYPES]
if not allowed:
allowed = list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED)
if use == USE_CONTENT:
# A content target never admits navigation pages, whatever a rule claims.
allowed = [item for item in allowed if item not in NAVIGATION] or list(CONTENT_ALLOWED)
return {"label": label or "证据", "use": use, "allowed_page_types": allowed,
"query": str(raw.get("query") or label).strip() or (label or "证据")}
label = str(raw or "").strip()
use = USE_STRUCTURE if _looks_structural(label) else USE_CONTENT
return {"label": label or "证据", "use": use,
"allowed_page_types": list(STRUCTURE_ALLOWED if use == USE_STRUCTURE else CONTENT_ALLOWED),
"query": label or "证据"}
def normalize_targets(required_evidence) -> list[dict]:
"""Normalize a criterion's required evidence, preserving order and identity."""
values = required_evidence if isinstance(required_evidence, list) else [required_evidence]
targets = []
for index, raw in enumerate(values):
if raw is None or (isinstance(raw, str) and not raw.strip()):
continue
target = normalize_target(raw)
target["index"] = index
target["target_id"] = str((raw.get("target_id") if isinstance(raw, dict) else "") or f"t{index}")
targets.append(target)
if not targets:
targets.append({"label": "正文实质内容", "use": USE_CONTENT, "allowed_page_types": list(CONTENT_ALLOWED),
"query": "正文实质内容", "index": 0, "target_id": "t0"})
return targets
def content_targets(criterion: dict) -> list[dict]:
return [target for target in normalize_targets(criterion.get("required_evidence")) if target["use"] == USE_CONTENT]
def any_content_target(criterion: dict) -> bool:
return bool(content_targets(criterion))
def page_type_allowed(page_type: str, allowed: list[str]) -> bool:
"""An ``unknown`` page is a body page only when the target already allows body text.
``unknown`` marks a page whose reading is genuinely unsettled, so it may never be
used as the sole support for a content judgement, and it never widens a structural
target either. It stays retrievable so a teacher can inspect it.
"""
if page_type == "unknown":
return "body" in allowed
return page_type in allowed
def content_evidence_allowed(page_type: str) -> bool:
"""Only substantive body and appendix pages support a content judgement."""
return page_type in CONTENT_ALLOWED
def heading_only_text(text: str) -> bool:
"""True when a passage is nothing but section titles, so it proves no content.
The test is structural, not a length test: a quotation is heading-only when every
line of it is a chapter/section heading or a contents entry. A short technical
quotation such as a measured value is content, so it must not be rejected here.
"""
from .pages import heading_level, is_toc_entry
lines = [line.strip() for line in str(text or "").splitlines() if line.strip()]
if not lines:
return False
for line in lines:
if is_toc_entry(line) or heading_level(line):
continue
return False
return True
_PLACEHOLDER = re.compile(r"^\s*(待确认|待定|未知|无|none|null|-+|—+)\s*$", re.IGNORECASE)
def meaningful(value) -> bool:
text = str(value or "").strip()
return bool(text) and not _PLACEHOLDER.match(text)
def plan_snapshot(criteria: list[dict]) -> dict:
"""The policy attached to a batch snapshot, so old batches stay re-readable."""
return {
"version": POLICY_VERSION,
"content_allowed_page_types": list(CONTENT_ALLOWED),
"structure_allowed_page_types": list(STRUCTURE_ALLOWED),
"navigation_page_types": list(NAVIGATION),
"criteria": {criterion["criterion_id"]: normalize_targets(criterion.get("required_evidence"))
for criterion in criteria},
}
+45 -6
View File
@@ -11,6 +11,32 @@ from .knowledge_pages import evidence_links
from .models import (CriterionScore, ExportTask, Project, ScoreEvidence, ScoringBatch, Student, SubmissionFile) from .models import (CriterionScore, ExportTask, Project, ScoreEvidence, ScoringBatch, Student, SubmissionFile)
from .rules import totals from .rules import totals
#: Teacher-facing wording for a criterion status, shared by exports and the API.
STATUS_LABELS = {
"pending": "待评分",
"scored": "模型已评分",
"reviewed": "教师已复核",
"needs_review": "待教师复核",
"unscorable": "材料缺失或解析失败",
"failed": "模型调用失败",
}
PAGE_TYPE_LABELS = {
"cover": "封面", "abstract": "摘要", "toc": "目录", "body": "正文",
"appendix": "附录", "references": "参考文献", "unknown": "识别不确定",
"mixed": "目录与正文混合页",
}
#: Batch status wording. ``review_required`` means grading finished but some criteria
#: still need a teacher decision, which is not the same as an incomplete batch.
BATCH_STATUS_LABELS = {
"created": "待启动", "queued": "排队中", "running": "运行中",
"completed": "已完成", "review_required": "已完成,有待复核项",
"partial": "部分完成", "failed": "失败",
}
def page_type_label(value) -> str:
return PAGE_TYPE_LABELS.get(str(value or "unknown"), "未分类页面")
def project_result(db, batch: ScoringBatch, project: Project): def project_result(db, batch: ScoringBatch, project: Project):
scores = db.scalars(select(CriterionScore).where(CriterionScore.batch_id == batch.id, scores = db.scalars(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
@@ -19,10 +45,15 @@ def project_result(db, batch: ScoringBatch, project: Project):
criteria = batch.rubric_snapshot["criteria"] criteria = batch.rubric_snapshot["criteria"]
totals_ = totals({key: score.final_score for key, score in by_id.items()}, criteria, batch.rubric_snapshot["weights"]) totals_ = totals({key: score.final_score for key, score in by_id.items()}, criteria, batch.rubric_snapshot["weights"])
rows = [] rows = []
def evidence_dict(item): def evidence_dict(item, links):
links = evidence_links(db, item)
return {"page": item.page, "quote": item.quote, "file_id": item.submission_file_id, return {"page": item.page, "quote": item.quote, "file_id": item.submission_file_id,
"document_id": item.document_id, "document_id": item.document_id,
"target_id": item.target_id or "", "target_label": item.target_label or "",
"page_type": item.page_type or links.get("page_type") or "unknown",
"source_role": item.source_role or "content",
# A quotation saved before the evidence policy existed is labelled as a
# historical citation and never presented as report body evidence.
"historical": bool(item.historical),
"document": db.get(SubmissionFile, item.submission_file_id).relative_path if item.submission_file_id else "课程资料", "document": db.get(SubmissionFile, item.submission_file_id).relative_path if item.submission_file_id else "课程资料",
**links} **links}
for criterion in criteria: for criterion in criteria:
@@ -35,7 +66,8 @@ def project_result(db, batch: ScoringBatch, project: Project):
"model_comment": score.model_comment if score else "", "teacher_comment": score.teacher_comment if score else None, "model_comment": score.model_comment if score else "", "teacher_comment": score.teacher_comment if score else None,
"missing_evidence": score.missing_evidence if score else [], "suggestion": score.suggestion if score else "", "missing_evidence": score.missing_evidence if score else [], "suggestion": score.suggestion if score else "",
"status": score.status if score else "pending", "error": score.error if score else "", "status": score.status if score else "pending", "error": score.error if score else "",
"evidence": [evidence_dict(item) for item in evidence]}) "evidence_gaps": (score.evidence_gaps or []) if score else [],
"evidence": [evidence_dict(item, evidence_links(db, item)) for item in evidence]})
student = db.get(Student, project.student_id) student = db.get(Student, project.student_id)
return {"project_id": project.id, "student_no": student.student_no, "name": student.name, "topic": project.topic, return {"project_id": project.id, "student_no": student.student_no, "name": student.name, "topic": project.topic,
"totals": totals_, "rows": rows, "facts": project.facts} "totals": totals_, "rows": rows, "facts": project.facts}
@@ -97,14 +129,21 @@ def _html_report(result, batch):
for row in result["rows"]: for row in result["rows"]:
criterion = row["criterion"] criterion = row["criterion"]
evidences = "".join( evidences = "".join(
f"<li>{esc(e['document'])} · 第{e['page']}{'张幻灯片' if str(e['document']).lower().endswith('.pptx') else '页'}:{esc(e['quote'])}</li>" f"<li>{esc(e['document'])} · 第{e['page']}{'张幻灯片' if str(e['document']).lower().endswith('.pptx') else '页'}"
f"({esc(page_type_label(e.get('page_type')))}{',历史评分引用' if e.get('historical') else ''})"
f"{' 支持:' + esc(e['target_label']) if e.get('target_label') else ''}:{esc(e['quote'])}</li>"
for e in row["evidence"] for e in row["evidence"]
) )
missing = "、".join(row["missing_evidence"] or []) missing = "、".join(row["missing_evidence"] or [])
gaps = ";".join(f"{gap.get('target_label')}(检索范围:{'、'.join(page_type_label(t) for t in gap.get('searched_page_types') or []) or '正文'})"
for gap in row.get("evidence_gaps") or [])
review = "本项缺少正文证据,已转教师复核。" if row["status"] == "needs_review" else ""
rows.append(f"<section><h3>{esc(criterion['name'])}({esc(row['final_score'])} / {criterion['max_score']})</h3>" rows.append(f"<section><h3>{esc(criterion['name'])}({esc(row['final_score'])} / {criterion['max_score']})</h3>"
f"<p>模型原始分:{esc(row['model_score'])};教师修改分:{esc(row['teacher_score'])};状态:{esc(row['status'])}</p>" f"<p>模型原始分:{esc(row['model_score'])};教师修改分:{esc(row['teacher_score'])};状态:{esc(STATUS_LABELS.get(row['status'], row['status']))}</p>"
f"<p>模型评语:{esc(row['model_comment'])}</p><p>教师评语:{esc(row['teacher_comment'])}</p>" f"<p>模型评语:{esc(row['model_comment'])}</p><p>教师评语:{esc(row['teacher_comment'])}</p>"
f"<p>缺失证据:{esc(missing)}</p><p>建议:{esc(row['suggestion'])}</p><p>异常:{esc(row['error'])}</p>" f"<p>{esc(review)}</p>"
f"<p>缺失证据:{esc(missing)}</p><p>缺正文证据的目标:{esc(gaps)}</p>"
f"<p>建议:{esc(row['suggestion'])}</p><p>异常:{esc(row['error'])}</p>"
f"<ul>{evidences}</ul></section>") f"<ul>{evidences}</ul></section>")
totals_ = result["totals"] totals_ = result["totals"]
return f"""<!doctype html><html lang="zh"><meta charset="utf-8"><style> return f"""<!doctype html><html lang="zh"><meta charset="utf-8"><style>
+77 -2
View File
@@ -2,7 +2,8 @@
from sqlalchemy import select from sqlalchemy import select
from .db import SessionLocal from .db import SessionLocal
from .models import IndexTask, SubmissionFile, now from .models import IndexTask, KnowledgeDocument, SubmissionFile, now
from .rag import INDEX_VERSION
from .scoring import parse_submission from .scoring import parse_submission
@@ -14,7 +15,13 @@ def run_index_task(task_id: int, *, session_factory=SessionLocal, embedder=None)
task.status, task.error = "running", "" task.status, task.error = "running", ""
db.commit() db.commit()
try: try:
if task.kind == "project": if task.kind == "reindex":
# Rebuild documents whose index predates the current chunking, so a
# legacy index is never silently used as classified body text.
task.result = reindex_stale(db, task.course_version_id, embedder=embedder)
if task.result["failed"]:
raise ValueError("部分资料重建索引失败:" + "、".join(item["title"] for item in task.result["failed"]))
elif task.kind == "project":
file = db.get(SubmissionFile, task.submission_file_id) file = db.get(SubmissionFile, task.submission_file_id)
if not file or file.recognition_status != "recognized": if not file or file.recognition_status != "recognized":
raise ValueError("材料已取消选择,请重新预览") raise ValueError("材料已取消选择,请重新预览")
@@ -35,6 +42,74 @@ def run_index_task(task_id: int, *, session_factory=SessionLocal, embedder=None)
db.commit() db.commit()
def stale_documents(db, course_version_id: int | None = None, *, limit: int | None = None):
"""Documents whose stored index predates the current chunking or is not ready."""
query = select(KnowledgeDocument).where(
(KnowledgeDocument.index_version != INDEX_VERSION) | (KnowledgeDocument.status != "ready"))
if course_version_id is not None:
query = query.where(KnowledgeDocument.course_version_id == course_version_id)
query = query.order_by(KnowledgeDocument.id)
if limit:
query = query.limit(limit)
return db.scalars(query).all()
def reindex_stale(db, course_version_id: int | None = None, *, embedder=None, limit: int | None = None):
"""Rebuild every stale index from the saved parse output. Retryable and auditable.
Student material is rebuilt from ``submission_files.parsed``, which the parser saved
on disk; the original file is never re-read or modified. Course material follows the
normal ingest path, because its pages come from the mounted course documents.
"""
from .services import parse_submission
from .storage import sha256, secure_path
documents = stale_documents(db, course_version_id, limit=limit)
rebuilt, skipped, failed = [], [], []
for document in documents:
try:
if document.submission_file_id:
file = db.get(SubmissionFile, document.submission_file_id)
if not file:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "来源材料已不存在"})
continue
if sha256(secure_path(file.relative_path)) != file.sha256:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "源文件已变化,请重新扫描并确认导入"})
continue
file.parse_status = "pending"
db.commit()
parsed = parse_submission(db, file, embedder=embedder) if embedder else parse_submission(db, file)
rebuilt.append({"document_id": document.id, "title": document.title,
"pages": len(parsed.get("pages", []))})
elif document.project_id is None:
# Course material is re-ingested by the course task, not per document.
skipped.append({"document_id": document.id, "title": document.title,
"reason": "课程资料请通过课程入库重建"})
else:
skipped.append({"document_id": document.id, "title": document.title,
"reason": "缺少可重建的解析结果"})
except Exception as exc:
failed.append({"document_id": document.id, "title": document.title, "error": str(exc)[:300]})
db.rollback()
return {"index_version": INDEX_VERSION, "rebuilt": rebuilt, "skipped": skipped,
"failed": failed, "remaining": len(stale_documents(db, course_version_id))}
def create_reindex_task(db, course_version_id: int):
"""Queue one reindex job for a course version, reusing an open task when there is one."""
from .services import enqueue_index_task
existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id,
IndexTask.kind == "reindex", IndexTask.status.in_(["queued", "running"])).order_by(IndexTask.id.desc()))
task = existing or IndexTask(course_version_id=course_version_id, kind="reindex", status="queued")
if not existing:
db.add(task)
db.commit()
enqueue_index_task(db, task.id)
return task
def create_course_index_task(db, course_version_id: int): def create_course_index_task(db, course_version_id: int):
from .services import enqueue_index_task from .services import enqueue_index_task
existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id, existing = db.scalar(select(IndexTask).where(IndexTask.course_version_id == course_version_id,
+21 -6
View File
@@ -36,18 +36,33 @@ def page_image_path(db, doc: KnowledgeDocument, page: int) -> Path | None:
return path if path and path.is_relative_to(root) and path.is_file() else None return path if path and path.is_relative_to(root) and path.is_file() else None
def page_links(db, document_id: int | None, page: int | None) -> dict[str, str | None]: def page_links(db, document_id: int | None, page: int | None) -> dict:
"""Stable citation links plus the page type, so callers can label the source."""
empty = {"url": None, "image_url": None, "page_type": None, "section": None}
if not document_id or not page: if not document_id or not page:
return {"url": None, "image_url": None} return empty
doc = db.get(KnowledgeDocument, document_id) doc = db.get(KnowledgeDocument, document_id)
if not doc: if not doc:
return {"url": None, "image_url": None} return empty
record = db.get(KnowledgePage, (document_id, page)) record = db.get(KnowledgePage, (document_id, page))
if not record: if not record:
return {"url": None, "image_url": None} return empty
return {"url": text_url(document_id, page), return {"url": text_url(document_id, page),
"image_url": image_url(document_id, page) if page_image_path(db, doc, page) else None} "image_url": image_url(document_id, page) if page_image_path(db, doc, page) else None,
"page_type": record.page_type or "unknown",
"section": record.section or ""}
def evidence_links(db, evidence) -> dict[str, str | None]: def evidence_links(db, evidence) -> dict:
return page_links(db, evidence.document_id, evidence.page) return page_links(db, evidence.document_id, evidence.page)
def page_types(db, document_ids) -> dict[int, str]:
"""Page type per page of the given documents, for labelling citations in bulk."""
from sqlalchemy import select as _select
ids = [item for item in dict.fromkeys(document_ids or []) if item]
if not ids:
return {}
rows = db.execute(_select(KnowledgePage.document_id, KnowledgePage.page, KnowledgePage.page_type)
.where(KnowledgePage.document_id.in_(ids))).all()
return {(document_id, page): (page_type or "unknown") for document_id, page, page_type in rows}
+78 -27
View File
@@ -15,25 +15,39 @@ class EvidenceResult(BaseModel):
chunk_id: int chunk_id: int
page: int = Field(ge=1) page: int = Field(ge=1)
quote: str = Field(min_length=1) quote: str = Field(min_length=1)
#: Which required-evidence target this quotation is offered to support.
target_id: str = ""
class ScoreResult(BaseModel): class ScoreResult(BaseModel):
criterion_id: str criterion_id: str
score: float = Field(ge=0) #: ``None`` means the model declined to score because admissible evidence is absent.
score: float | None = Field(default=None, ge=0)
max_score: float = Field(gt=0) max_score: float = Field(gt=0)
reason: str = Field(min_length=1) reason: str = Field(min_length=1)
evidence: list[EvidenceResult] evidence: list[EvidenceResult] = Field(default_factory=list)
missing_evidence: list[str] = Field(default_factory=list) missing_evidence: list[str] = Field(default_factory=list)
suggestion: str = "" suggestion: str = ""
confidence: float = Field(ge=0, le=1) confidence: float = Field(ge=0, le=1)
@model_validator(mode="after") @model_validator(mode="after")
def valid_range(self): def valid_range(self):
if self.score > self.max_score or (self.score > 0 and not self.evidence): if self.score is not None and self.score > self.max_score:
raise ValueError("score exceeds maximum or lacks evidence") raise ValueError("score exceeds maximum")
if self.score and not self.evidence:
raise ValueError("a positive score requires evidence")
return self return self
class EvidenceGap(BaseModel):
"""A content target that still has no body evidence after targeted re-checking."""
target_id: str = ""
target_label: str = ""
searched_page_types: list[str] = Field(default_factory=list)
reason: str = ""
class FactResult(BaseModel): class FactResult(BaseModel):
student_no: str = "" student_no: str = ""
name: str = "" name: str = ""
@@ -132,33 +146,58 @@ class DeepSeekClient:
raise ModelError("当前端点或模型未返回原生 tool_calls;请配置支持工具调用的端点和模型") raise ModelError("当前端点或模型未返回原生 tool_calls;请配置支持工具调用的端点和模型")
def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict]): def locate_quote(text: str, quote: str):
"""Locate a model-written quotation inside the retrieved text.
PDF line breaks and punctuation differ from what the model reproduces, so matching
is done on NFKC-normalized text with whitespace and punctuation removed, while the
returned string is still the exact source substring.
"""
if not text or not quote:
return None
if quote in text:
return quote
def comparable(value: str):
return "".join(char for original in value for char in unicodedata.normalize("NFKC", original)
if not char.isspace() and not unicodedata.category(char).startswith("P"))
needle = comparable(quote)
if len(needle) < 4:
return None
positions = []
normalized = []
for index, original in enumerate(text):
for char in unicodedata.normalize("NFKC", original):
if not char.isspace() and not unicodedata.category(char).startswith("P"):
normalized.append(char)
positions.append(index)
start = "".join(normalized).find(needle)
if start < 0:
return None
return text[positions[start]:positions[start + len(needle) - 1] + 1]
def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict], *, targets=None,
require_content_evidence: bool | None = None):
"""Check identity, quotation fidelity and the evidence-source policy.
The model may understand evidence and propose re-checks; the page permissions are
enforced here, in code, and cannot be argued away by the model's own reasoning.
"""
from .evidence import (USE_CONTENT, content_evidence_allowed, heading_only_text,
normalize_targets, page_type_allowed)
if result.criterion_id != criterion["criterion_id"] or result.max_score != criterion["max_score"]: if result.criterion_id != criterion["criterion_id"] or result.max_score != criterion["max_score"]:
raise ModelError("模型评分项或满分与规则版本不一致") raise ModelError("模型评分项或满分与规则版本不一致")
if require_content_evidence is None:
require_content_evidence = any(target["use"] == USE_CONTENT
for target in normalize_targets(criterion.get("required_evidence")))
targets = targets if targets is not None else normalize_targets(criterion.get("required_evidence"))
by_target = {target["target_id"]: target for target in targets}
default_content = next((target for target in targets if target["use"] == USE_CONTENT), None)
by_id = {chunk["chunk_id"]: chunk for chunk in chunks} by_id = {chunk["chunk_id"]: chunk for chunk in chunks}
def locate_quote(text: str, quote: str):
def comparable(value: str):
return "".join(char for original in value for char in unicodedata.normalize("NFKC", original)
if not char.isspace() and not unicodedata.category(char).startswith("P"))
if quote in text:
return quote
needle = comparable(quote)
if len(needle) < 4:
return None
positions = []
normalized = []
for index, original in enumerate(text):
for char in unicodedata.normalize("NFKC", original):
if not char.isspace() and not unicodedata.category(char).startswith("P"):
normalized.append(char)
positions.append(index)
start = "".join(normalized).find(needle)
if start < 0:
return None
return text[positions[start]:positions[start + len(needle) - 1] + 1]
for evidence in result.evidence: for evidence in result.evidence:
chunk = by_id.get(evidence.chunk_id) chunk = by_id.get(evidence.chunk_id)
quote = locate_quote(chunk["text"], evidence.quote) if chunk else None quote = locate_quote(chunk["text"], evidence.quote) if chunk else None
@@ -169,5 +208,17 @@ def validate_score(result: ScoreResult, criterion: dict, chunks: list[dict]):
chunk, quote = matches[0] chunk, quote = matches[0]
if quote is None: if quote is None:
raise ModelError("模型证据未能匹配检索原文") raise ModelError("模型证据未能匹配检索原文")
page_type = chunk.get("page_type") or "unknown"
target = by_target.get(evidence.target_id) or default_content or (targets[0] if targets else None)
if target is None:
raise ModelError("模型证据未关联任何必备证据目标")
if not page_type_allowed(page_type, target["allowed_page_types"]):
raise ModelError(f"模型证据来自不允许的页面类型({page_type}),不能支持「{target['label']}」")
if target["use"] == USE_CONTENT and not content_evidence_allowed(page_type):
raise ModelError(f"目录、摘要或参考文献页不能作为「{target['label']}」的内容证据")
if target["use"] == USE_CONTENT and heading_only_text(quote):
raise ModelError(f"「{target['label']}」的证据只有章节标题,没有实质内容")
evidence.chunk_id, evidence.page, evidence.quote = chunk["chunk_id"], chunk["page"], quote evidence.chunk_id, evidence.page, evidence.quote = chunk["chunk_id"], chunk["page"], quote
evidence.target_id = target["target_id"]
return result return result
+22
View File
@@ -176,6 +176,14 @@ class KnowledgePage(Base):
page = Column(Integer, primary_key=True) page = Column(Integer, primary_key=True)
text = Column(Text, nullable=False, default="") text = Column(Text, nullable=False, default="")
image_path = Column(String(1200), nullable=True) # Relative to processed_path. image_path = Column(String(1200), nullable=True) # Relative to processed_path.
# Page-level classification: cover, abstract, toc, body, references or unknown.
page_type = Column(String(20), nullable=False, default="unknown", server_default="unknown")
mixed = Column(Boolean, nullable=False, default=False, server_default="false")
section = Column(String(500), default="")
chapter = Column(String(500), default="")
confidence = Column(Float, nullable=True)
# Bounded classifier evidence: which signals decided the type, kept for audit.
classification = Column(JSON, default=dict, nullable=False)
class KnowledgeChunk(Base): class KnowledgeChunk(Base):
@@ -187,6 +195,9 @@ class KnowledgeChunk(Base):
document_type = Column(String(80), nullable=False) document_type = Column(String(80), nullable=False)
page = Column(Integer, nullable=False) page = Column(Integer, nullable=False)
section = Column(String(500), default="") section = Column(String(500), default="")
chapter = Column(String(500), default="")
page_type = Column(String(20), nullable=False, default="unknown", server_default="unknown")
block_type = Column(String(20), nullable=False, default="body", server_default="body")
text = Column(Text, nullable=False) text = Column(Text, nullable=False)
embedding = Column(Embedding, nullable=True) embedding = Column(Embedding, nullable=True)
@@ -225,6 +236,9 @@ class CriterionScore(Base):
suggestion = Column(Text, default="") suggestion = Column(Text, default="")
status = Column(String(40), default="pending") status = Column(String(40), default="pending")
error = Column(Text, default="") error = Column(Text, default="")
#: Content targets that still lacked body evidence, with the search that was run.
#: Kept after teacher review so the original retrieval problem stays auditable.
evidence_gaps = Column(JSON, default=list)
__table_args__ = (UniqueConstraint("batch_id", "project_id", "criterion_id"),) __table_args__ = (UniqueConstraint("batch_id", "project_id", "criterion_id"),)
@property @property
@@ -241,6 +255,14 @@ class ScoreEvidence(Base):
page = Column(Integer, nullable=False) page = Column(Integer, nullable=False)
quote = Column(Text, nullable=False) quote = Column(Text, nullable=False)
image_path = Column(String(1200), nullable=True) image_path = Column(String(1200), nullable=True)
# Which required-evidence target this quotation is offered to support.
target_id = Column(String(50), default="", server_default="")
target_label = Column(String(300), default="", server_default="")
# Page type of the source passage, and its role in the judgement.
page_type = Column(String(20), default="unknown", server_default="unknown")
source_role = Column(String(20), default="content", server_default="content")
# True when the quotation was written before this retrieval policy existed.
historical = Column(Boolean, default=False, nullable=False, server_default="false")
class TeacherRevision(Base): class TeacherRevision(Base):
+443
View File
@@ -0,0 +1,443 @@
"""Page-level classification, block splitting and real section paths.
A page carries one ``page_type``: cover, abstract, toc, body, references, unknown.
A page may additionally be ``mixed`` when a table of contents and body text truly
share one page; only that page is split into blocks so the body part is not lost.
Classification never depends on a fixed page number or on the single word "目录".
It combines the page's own signals (heading title, dot-leader entry density,
trailing page numbers, abstract/reference wording, cover fields, footer style)
with document-level continuity for continuation pages (TOC continued, abstract
continued, references spanning pages).
"""
from __future__ import annotations
import re
import unicodedata
CLASSIFIER_VERSION = 3
PAGE_TYPES = ("cover", "abstract", "toc", "body", "appendix", "references", "unknown")
#: Page types whose text may support a content judgement about what a student did.
CONTENT_PAGE_TYPES = ("body", "appendix")
#: Page types whose text may support a structural judgement (chapter completeness etc.).
STRUCTURE_PAGE_TYPES = ("cover", "abstract", "toc", "references", "body", "appendix")
#: Page types that may never be used as content evidence on their own.
NAVIGATION_PAGE_TYPES = ("cover", "abstract", "toc", "references")
#: Page types a page may be stored as, used to validate rule configuration.
STORE_PAGE_TYPES = PAGE_TYPES
BLOCK_BODY = "body"
BLOCK_TOC = "toc"
BLOCK_TITLE = "title"
_DOT_LEADER = re.compile(r"(?:[.·•…․…]\s*){3,}")
_TRAILING_ANCHOR = re.compile(r"(\d{1,4})\s*$")
_CHAPTER = re.compile(r"^第\s*([0-9一二三四五六七八九十百]+)\s*章\s*(.*)$")
_APPENDIX = re.compile(r"^附\s*录\s*([A-Za-z0-9一二三四五六七八九十]+)?\s*(.*)$")
_NUMBERED = re.compile(r"^(\d+(?:\.\d+)*)[\s、..]+(\S.*)$")
_COVER_FIELD = re.compile(r"(学\s*号|姓\s*名|题\s*目|指导教师|专\s*业|学\s*院|日\s*期)\s*[::]")
_PAGE_NUMBER_LINE = re.compile(r"^\s*(?:\d{1,3}|[IVXLC]{1,5}|第\s*\d+\s*页)\s*$")
_ABSTRACT_TITLE = re.compile(r"^\s*(摘\s*要|abstract)\s*$", re.IGNORECASE)
_REFERENCE_TITLE = re.compile(r"^\s*(参\s*考\s*文\s*献|references?|参考文献与资料)\s*$", re.IGNORECASE)
_ACK_TITLE = re.compile(r"^\s*(致\s*谢|acknowledge?ments?)\s*$", re.IGNORECASE)
_TOC_TITLE = re.compile(r"^\s*(目\s*录|contents?|table\s+of\s+contents)\s*$", re.IGNORECASE)
_REFERENCE_ENTRY = re.compile(r"^\s*\[\d{1,3}\]")
_CJK = re.compile(r"[㐀-䶿一-鿿]")
_HEADING_NUMBER = re.compile(r"^\s*(第\s*[0-9一二三四五六七八九十百]+\s*章|\d+(\.\d+)*|附\s*录\s*[A-Za-z0-9]?)\s*\S")
#: A body block must carry at least this many non-space characters to count as substance.
MIN_BODY_CHARS = 60
#: Heading-only text shorter than this never counts as content evidence on its own.
MIN_CONTENT_QUOTE_CHARS = 25
#: A line at least this long, outside a heading, reads as running prose. The bar is low
#: on purpose: the rule separates text from headings and labels, not long from short
#: text, so a brief but real sentence is still content.
MIN_PROSE_CHARS = 4
#: A numbered heading title must stay this short; longer text is prose or a list item.
MAX_HEADING_TITLE_CHARS = 40
#: Wording that marks the line as running prose rather than a heading.
_PROSE_PUNCTUATION = "。,;!?,;"
#: The documents are Chinese engineering reports; the keyword recall uses these units.
BIGRAM_MIN = 2
def _lines(text: str) -> list[str]:
return [line.strip() for line in (text or "").splitlines() if line.strip()]
def normalize_heading(text: str) -> str:
"""Compare headings across the TOC and the body: unified width, no spaces/dots."""
value = unicodedata.normalize("NFKC", text or "")
value = _DOT_LEADER.sub("", value)
value = _TRAILING_ANCHOR.sub("", value.strip())
return "".join(char for char in value if not char.isspace() and char not in "..·•…")
def is_toc_entry(line: str) -> bool:
"""A dotted leader plus a trailing page or slide number."""
stripped = (line or "").strip()
if not stripped or not _DOT_LEADER.search(stripped):
return False
return bool(_TRAILING_ANCHOR.search(stripped))
def heading_syntax(line: str) -> tuple[int, str] | None:
"""Return ``(level, title)`` when the line has the shape of a heading.
This is a purely syntactic reading. Numbered prose such as
``4. 实时图像传输:将摄像头画面…`` and enumerated list items are not headings: a
numbered heading has a short Chinese title, no sentence punctuation and no trailing
colon. Equation and matrix rows such as ``3.3.3`` or ``1 0 0 1 Zc`` are never
headings, and a line cut mid-formula (trailing combining mark) is not either.
"""
stripped = (line or "").strip()
if not stripped or is_toc_entry(stripped) or len(stripped) > 120:
return None
match = _CHAPTER.match(stripped)
if match:
return 1, stripped
match = _APPENDIX.match(stripped)
if match and not _COVER_FIELD.search(stripped):
return 2, stripped
if _ABSTRACT_TITLE.match(stripped) or _REFERENCE_TITLE.match(stripped) or _ACK_TITLE.match(stripped):
return 1, stripped
match = _NUMBERED.match(stripped)
if not match:
return None
title = match.group(2).strip()
if not title or title.endswith(":") or title.endswith(":"):
return None
if len(title) > MAX_HEADING_TITLE_CHARS or any(char in title for char in _PROSE_PUNCTUATION):
return None
if not _CJK.search(title):
return None
if unicodedata.category(title[-1]).startswith("M"):
return None
return 1 + match.group(1).count("."), stripped
def heading_level(line: str, toc_titles: set[str] | None = None) -> tuple[int, str] | None:
"""A heading for the section path: syntactic shape, confirmed by the contents list.
When the document carries its own table of contents, that list is the authority on
which numbered lines are real headings. That is what keeps an enumerated list item or
a numbered paragraph from becoming a chapter in the extracted section path. Callers
that only need to know whether a line reads as a heading use ``heading_syntax``.
"""
found = heading_syntax(line)
if not found:
return None
if toc_titles and normalize_heading(line) not in toc_titles:
return None
return found
def _footer_style(lines: list[str]) -> str:
"""Classify the trailing line so repeated headers/footers can be removed."""
if not lines:
return "none"
last = lines[-1]
if re.fullmatch(r"[IVXLC]{1,5}", last):
return "roman"
if re.fullmatch(r"\d{1,3}", last):
return "arabic"
if re.fullmatch(r"第\s*\d+\s*页", last):
return "page_cn"
return "text"
def _signals(page: dict, *, index: int, total: int, toc_titles: set[str] | None = None,
repeated: set[str] | None = None) -> dict:
text = str(page.get("text") or "")
lines = _lines(text)
# Prose detection ignores repeated page headers and footers: a running header is not
# something the student wrote, so it must not make an empty page look like content.
prose_source = [line for line in lines if line not in (repeated or ())]
entries = [line for line in lines if is_toc_entry(line)]
headings = [item for item in (heading_level(line, toc_titles) for line in lines) if item]
body_lines = [line for line in lines if line not in set(entries)]
plain_chars = sum(len(line) for line in body_lines)
first = lines[0] if lines else ""
reference_entries = [line for line in lines if _REFERENCE_ENTRY.match(line)]
titles = [position for position, line in enumerate(lines)
if _REFERENCE_TITLE.match(line) or _ACK_TITLE.match(line)]
# A bibliography page is dense reference entries, or a reference title near the top.
reference_ratio = len(reference_entries) / len(lines) if lines else 0.0
reference_page = (len(reference_entries) >= 5 and reference_ratio >= 0.5
or any(position <= 2 for position in titles))
return {
"chars": len(text.strip()),
"lines": len(lines),
"toc_entries": len(entries),
"toc_ratio": round(len(entries) / len(lines), 3) if lines else 0.0,
"headings": len(headings),
"chapter_headings": sum(1 for level, _ in headings if level == 1),
"body_chars": plain_chars,
"first_line": first[:120],
"position": index,
"total": total,
"footer": _footer_style(lines),
"cover_fields": len(_COVER_FIELD.findall(text)),
"has_table": bool(page.get("tables")),
"reference_entries": len(reference_entries),
"reference_ratio": round(reference_ratio, 3),
"reference_page": reference_page,
"appendix_title": any(_APPENDIX.match(line.strip()) for line in lines),
# Prose punctuation separates running text from a label-style cover page.
"prose_punctuation": any(char in text for char in _PROSE_PUNCTUATION),
# Lines that read as running text rather than headings, labels or dotted entries.
"prose_lines": [line for line in prose_source if _is_prose_line(line)],
}
def strip_repeated_lines(pages: list[dict]) -> set[str]:
"""Header/footer lines repeated across many pages, removed from searchable text."""
counts: dict[str, int] = {}
for page in pages:
lines = _lines(str(page.get("text") or ""))
if not lines:
continue
candidates = {lines[0][:120]}
if len(lines) > 1:
candidates.add(lines[-1][:60])
for candidate in candidates:
if candidate and not is_toc_entry(candidate):
counts[candidate] = counts.get(candidate, 0) + 1
threshold = max(3, int(len(pages) * 0.4 + 0.5))
return {line for line, count in counts.items() if count >= threshold}
def clean_page_lines(page: dict, repeated: set[str]) -> list[str]:
"""Lines kept for indexing: repeated headers/footers and bare page numbers removed."""
lines = _lines(str(page.get("text") or ""))
kept = []
for position, line in enumerate(lines):
if line in repeated and (position == 0 or position == len(lines) - 1):
continue
if _PAGE_NUMBER_LINE.match(line) and position == len(lines) - 1:
continue
kept.append(line)
return kept
def _is_prose_line(line: str) -> bool:
"""A line that reads as running text rather than a heading, label or dotted entry.
The syntactic heading check is used, not the contents-confirmed one: a heading that
the contents list happens not to mention is still a heading, so it must not be
counted as prose when deciding whether a page carries content.
"""
stripped = line.strip()
if is_toc_entry(stripped) or _TOC_TITLE.match(stripped) or heading_syntax(stripped):
return False
if _PAGE_NUMBER_LINE.match(stripped):
return False
return len(stripped) >= MIN_PROSE_CHARS
def split_blocks(lines: list[str]) -> list[dict]:
"""Split page lines into TOC and body blocks so a mixed page keeps both parts."""
blocks: list[dict] = []
current: dict | None = None
for line in lines:
stripped = line.strip()
# A contents title and its dotted entries are one navigation block. A lone
# dotted line with no title is not a table of contents, so it stays as body.
kind = BLOCK_TOC if is_toc_entry(line) or _TOC_TITLE.match(stripped) else BLOCK_BODY
if current is None or current["kind"] != kind:
current = {"kind": kind, "lines": []}
blocks.append(current)
current["lines"].append(line)
output = []
for block in blocks:
text = "\n".join(block["lines"]).strip()
if not text:
continue
has_entry = any(is_toc_entry(line) for line in block["lines"])
if block["kind"] == BLOCK_TOC and not has_entry:
# Only a title line, with no entries under it: this is a heading, not contents.
output.append({"kind": BLOCK_BODY, "text": text, "substantive": False})
continue
output.append({"kind": block["kind"], "text": text})
# A block made only of headings carries navigation, not content.
for block in output:
if block["kind"] == BLOCK_BODY:
block["substantive"] = (any(_is_prose_line(line) for line in block["text"].splitlines())
or len(block["text"]) >= MIN_BODY_CHARS)
else:
block["substantive"] = False
return output
def classifier_signals(pages: list[dict]) -> list[dict]:
"""Per-page signals, shared by classification and by audit output."""
total = len(pages)
return [_signals(page, index=position, total=total) for position, page in enumerate(pages, start=1)]
def _is_cover(signal: dict) -> bool:
"""A cover page carries identity fields, not prose.
The weak rule requires a first page of a document that actually has a body, with
label-like lines and no sentence punctuation. Without those guards a short
single-page document would be mistaken for a cover page and lose all its content.
"""
if signal["cover_fields"] >= 2 and signal["body_chars"] < 600:
return True
return (signal["position"] == 1 and signal["total"] >= 3 and signal["chars"] <= 400
and signal["lines"] <= 8 and signal["toc_entries"] == 0 and signal["headings"] == 0
and not signal["has_table"] and not signal["prose_punctuation"])
def _is_toc_page(signal: dict, *, previous_is_toc: bool) -> bool:
if signal["toc_entries"] >= 4 and signal["toc_ratio"] >= 0.4:
return True
if _TOC_TITLE.match(signal["first_line"]) and signal["toc_entries"] >= 2:
return True
# Continuation page: the previous page is a table of contents and this page is
# still almost nothing but dotted entries.
return bool(previous_is_toc and signal["toc_entries"] >= 2 and signal["toc_ratio"] >= 0.5)
def _looks_heading_only(signal: dict, blocks: list[dict]) -> bool:
"""A page that carries no running prose at all, so it proves no content.
The test is structural, not a length test: a short page with one real sentence is
body text, a page with a table carries data, while a page of nothing but headings
or labels cannot support a judgement about what was done.
"""
if signal["has_table"]:
return False
if any(block["kind"] == BLOCK_BODY and block.get("substantive") for block in blocks):
return False
return not signal["prose_lines"]
def classify_document(pages: list[dict]) -> list[dict]:
"""Classify every page and extract real section paths from body headings only."""
repeated = strip_repeated_lines(pages)
# The document's own contents list is the authority on which numbered lines are
# real headings; this keeps enumerated list items and equations out of the path.
toc_titles = set()
for page in pages:
for line in _lines(str(page.get("text") or "")):
if is_toc_entry(line):
title = normalize_heading(line)
if title:
toc_titles.add(title)
toc_titles = toc_titles or None
signals = [_signals(page, index=position, total=len(pages), toc_titles=toc_titles, repeated=repeated)
for position, page in enumerate(pages, start=1)]
results: list[dict] = []
previous = None
chapter = ""
section = ""
for position, page in enumerate(pages, start=1):
signal = signals[position - 1]
lines = clean_page_lines(page, repeated)
blocks = split_blocks(lines)
toc_blocks = [block for block in blocks if block["kind"] == BLOCK_TOC]
body_blocks = [block for block in blocks if block["kind"] == BLOCK_BODY and block["substantive"]]
first = signal["first_line"]
appendix = signal["appendix_title"]
if _TOC_TITLE.match(first) and signal["toc_entries"] >= 1:
page_type, confidence = "toc", 0.95
elif _ABSTRACT_TITLE.match(first) or (previous == "abstract" and signal["footer"] in ("roman", "none")
and signal["headings"] == 0 and signal["chars"] >= 100):
page_type, confidence = "abstract", 0.9
elif _REFERENCE_TITLE.match(first) or _ACK_TITLE.match(first):
page_type, confidence = "references", 0.9
elif _is_toc_page(signal, previous_is_toc=(previous == "toc")):
page_type, confidence = "toc", 0.85 if signal["toc_ratio"] >= 0.4 else 0.75
elif previous == "references" and signal["headings"] == 0 and signal["toc_entries"] == 0:
page_type, confidence = "references", 0.7
elif signal["reference_page"] and signal["toc_entries"] == 0 and signal["chapter_headings"] <= 1:
page_type, confidence = "references", 0.8
elif appendix:
# An appendix carries substantive material, so it counts as content, but it
# is labelled separately so the interface can show where a quotation came from.
page_type, confidence = "appendix", 0.8
elif _is_cover(signal):
# Checked before the heading-only rule: a cover page carries identity fields
# and no prose, so it would otherwise be mistaken for an unsettled page.
page_type, confidence = "cover", 0.8
elif _looks_heading_only(signal, blocks) and signal["toc_entries"] == 0:
# A body page that carries only a section title cannot support a content
# judgement on its own, so it is not silently treated as ordinary body text.
page_type, confidence = "unknown", 0.5
else:
page_type, confidence = "body", 0.6
confidence = 0.85 if body_blocks and signal["toc_entries"] == 0 else confidence
mixed = page_type == "toc" and bool(body_blocks)
if mixed:
confidence = 0.7
if page_type in ("body", "appendix", "unknown") or mixed:
# Structure-extracted headings come from body text only, never from a TOC.
for block in body_blocks:
for line in block["text"].splitlines():
found = heading_level(line, toc_titles)
if not found:
continue
level, title = found
if level == 1:
chapter, section = title, title
else:
section = title
# A table of contents may only name the chapter path; the body supplies the
# section wording. Navigation pages therefore carry the chapter, not a section.
section_for_page = section if page_type in ("body", "unknown") or mixed else chapter
results.append({
"page": position,
"page_type": page_type,
"mixed": mixed,
"chapter": chapter,
"section": section_for_page,
"blocks": blocks,
"confidence": round(confidence, 2),
"signals": {**signal, "repeated_line_count": len(repeated), "toc_blocks": len(toc_blocks)},
"classifier_version": CLASSIFIER_VERSION,
})
previous = page_type
return results
def block_text(blocks: list[dict], *, kinds: tuple[str, ...] = (BLOCK_BODY,)) -> str:
return "\n".join(block["text"] for block in blocks if block["kind"] in kinds).strip()
def toc_entries(blocks: list[dict]) -> list[str]:
"""Heading-shaped entries read out of a table of contents page."""
entries = []
for block in blocks:
if block["kind"] != BLOCK_TOC:
continue
for line in block["text"].splitlines():
if is_toc_entry(line):
entries.append(line)
return entries
def searchable_pages(pages: list[dict]) -> list[dict]:
"""Pages prepared for indexing: cleaned lines, typed blocks and section paths."""
classification = classify_document(pages)
output = []
for page, item in zip(pages, classification):
body = block_text(item["blocks"])
output.append({
"page_number": page.get("page_number"),
"section": item["section"] or page.get("section", ""),
"chapter": item["chapter"],
"text": item.get("text", ""),
"tables": page.get("tables", []),
"rendered_image_path": page.get("rendered_image_path"),
"page_type": item["page_type"],
"blocks": item["blocks"],
"classification": item,
"index_text": body,
})
return output
+362 -39
View File
@@ -1,31 +1,130 @@
"""Indexing and hybrid retrieval with page-type aware candidate filtering.
Chinese keyword recall does not use PostgreSQL ``to_tsvector('simple', …)``. That
configuration has no Chinese segmenter, so a whole run of Chinese characters becomes a
single token; measured on the course reports, the query ``文献调研`` does not match the
page that contains ``第2章文献调研与技术现状``. Recall therefore uses character bigrams
plus whole ASCII/number tokens, which needs no extension and behaves the same on
PostgreSQL and SQLite.
Semantic and keyword scores are fused by rank (reciprocal rank fusion), never by adding
raw numbers: a cosine similarity and a bigram overlap are not on a common scale.
"""
from functools import lru_cache from functools import lru_cache
from pathlib import Path from pathlib import Path
import re
from sqlalchemy import cast, delete, func, or_, select from sqlalchemy import case, cast, delete, func, literal, or_, select
from .config import get_settings from .config import get_settings
from .knowledge_pages import page_text, relative_image_path
from .models import KnowledgeChunk, KnowledgeDocument, KnowledgePage from .models import KnowledgeChunk, KnowledgeDocument, KnowledgePage
from .pages import (BLOCK_BODY, BLOCK_TOC, classify_document, is_toc_entry,
normalize_heading, searchable_pages)
INDEX_VERSION = 3
#: Fusion constant for reciprocal rank fusion; the usual value from the IR literature.
RRF_K = 60
#: Candidate pools are gathered independently so both retrievers get a chance.
SEMANTIC_POOL = 60
KEYWORD_POOL = 60
#: A chunk must be at least this long to count as substantive content evidence.
MIN_CHUNK_CHARS = 60
#: At most this many chunks from one page in the first retrieval pass.
DEFAULT_PAGE_LIMIT = 2
#: Soft cap on chunks from one section once a reliable section path exists.
DEFAULT_SECTION_CAP = 3
#: Maximum chunks handed to the model for one criterion, across all targets.
SCORE_EVIDENCE_LIMIT = 14
_ASCII_TOKEN = re.compile(r"[A-Za-z][A-Za-z0-9_+\-.]*|\d+(?:\.\d+)*")
_CJK_RUN = re.compile(r"[㐀-䶿一-鿿]+")
INDEX_VERSION = 2 def query_terms(query: str) -> tuple[list[str], list[str]]:
"""Split a query into (ascii tokens lowercased, CJK character bigrams).
Single-character CJK runs are dropped: they carry too little signal and would make
every chunk with a common character look relevant.
"""
value = str(query or "")
ascii_tokens = [token.lower() for token in _ASCII_TOKEN.findall(value)]
bigrams: list[str] = []
seen = set()
for run in _CJK_RUN.findall(value):
for index in range(len(run) - 1):
bigram = run[index:index + 2]
if bigram not in seen:
seen.add(bigram)
bigrams.append(bigram)
return sorted(set(ascii_tokens)), bigrams
def _keyword_query(db, filters, query: str, limit: int):
"""Keyword recall ranked by bigram/token overlap over the already-filtered rows.
PostgreSQL's ``to_tsvector('simple', …)`` cannot segment Chinese, so it is not used
for recall. The overlap is computed with ``strpos``/``substr``/``generate_series``
so the same expression works on every dialect SQLAlchemy targets here.
"""
ascii_tokens, bigrams = query_terms(query)
if not ascii_tokens and not bigrams:
return []
total = len(ascii_tokens) + len(bigrams)
score = None
for token in ascii_tokens:
hit = case((func.strpos(func.lower(KnowledgeChunk.text), literal(token)) > 0, 1.0), else_=0.0)
score = hit if score is None else score + hit
if bigrams:
series = select(func.generate_series(1, func.greatest(func.length(KnowledgeChunk.text) - 1, 0)).label("i")).subquery("series")
matches = (select(func.count()).select_from(series)
.where(func.substr(KnowledgeChunk.text, series.c.i, 2).in_(bigrams)).scalar_subquery())
score = matches if score is None else score + matches
score = (score / float(total)).label("keyword")
rows = db.execute(select(KnowledgeChunk.id, score).where(*filters)
.order_by(score.desc(), KnowledgeChunk.id.asc()).limit(limit)).all()
return [row[0] for row in rows if float(row[1] or 0) > 0]
def split_pages(pages: list[dict], *, size=650, overlap=100): def split_pages(pages: list[dict], *, size=650, overlap=100):
"""Split pages into searchable chunks carrying page type and section path."""
for page in pages: for page in pages:
text = page.get("text", "").strip() page_type = page.get("page_type") or "body"
blocks = [text] if text else [] chapter = page.get("chapter", "")
for table in page.get("tables", []): section = page.get("section", "")
blocks.append("表格: " + str(table.get("rows", table))) blocks = page.get("blocks")
if not blocks:
blocks = [{"kind": BLOCK_BODY, "text": str(page.get("text") or "").strip()}]
blocks.extend({"kind": BLOCK_BODY, "text": "表格: " + str(table.get("rows", table))}
for table in page.get("tables", []))
for block in blocks: for block in blocks:
if block["kind"] == BLOCK_TOC:
# A contents page is navigation, so its text is not body content.
yield {"page": page["page_number"], "page_type": "toc", "block_type": BLOCK_TOC,
"chapter": chapter, "section": section, "text": block["text"]}
continue
text_block = block["text"].strip()
if not text_block:
continue
for table in ([block["tables"]] if block.get("tables") else []):
for table_item in table:
text_block += "\n表格: " + str(table_item.get("rows", table_item))
# Table rows of a page are appended once, on the page's first body chunk.
start = 0 start = 0
while start < len(block): while start < len(text_block):
yield {"page": page["page_number"], "section": page.get("section", ""), "text": block[start:start + size]} yield {"page": page["page_number"], "page_type": page_type, "block_type": BLOCK_BODY,
if start + size >= len(block): "chapter": chapter, "section": section, "text": text_block[start:start + size]}
if start + size >= len(text_block):
break break
start += size - overlap start += size - overlap
def build_chunks(pages: list[dict]) -> list[dict]:
"""Chunks for one document, classifying pages first when they are not yet typed."""
prepared = pages if pages and "page_type" in (pages[0] or {}) else searchable_pages(pages)
return list(split_pages(prepared))
@lru_cache(maxsize=2) @lru_cache(maxsize=2)
def local_encoder(model_path: str): def local_encoder(model_path: str):
path = Path(model_path) path = Path(model_path)
@@ -56,7 +155,8 @@ def ingest(db, *, course_version_id: int, project_id: int | None, submission_fil
KnowledgeDocument.title == title)) KnowledgeDocument.title == title))
if existing and existing.source_hash == source_hash and existing.index_version == INDEX_VERSION and existing.status == "ready": if existing and existing.source_hash == source_hash and existing.index_version == INDEX_VERSION and existing.status == "ready":
return existing return existing
chunks = list(split_pages(pages)) prepared = searchable_pages(pages)
chunks = list(split_pages(prepared))
vectors = embedder([chunk["text"] for chunk in chunks]) vectors = embedder([chunk["text"] for chunk in chunks])
if len(vectors) != len(chunks): if len(vectors) != len(chunks):
raise RuntimeError("嵌入向量数量与知识片段数量不一致") raise RuntimeError("嵌入向量数量与知识片段数量不一致")
@@ -71,42 +171,265 @@ def ingest(db, *, course_version_id: int, project_id: int | None, submission_fil
db.add(doc) db.add(doc)
doc.index_version, doc.status = INDEX_VERSION, "indexing" doc.index_version, doc.status = INDEX_VERSION, "indexing"
db.flush() db.flush()
for source in pages: for source, prepared_page in zip(pages, prepared):
db.add(KnowledgePage(document_id=doc.id, page=source["page_number"], text=page_text(source), classification = prepared_page["classification"]
image_path=relative_image_path(source.get("rendered_image_path")))) db.add(KnowledgePage(
document_id=doc.id, page=source["page_number"], text=page_text(source),
image_path=relative_image_path(source.get("rendered_image_path")),
page_type=classification["page_type"], mixed=bool(classification["mixed"]),
section=classification["section"], chapter=classification["chapter"],
confidence=classification["confidence"],
classification={**classification["signals"], "classifier_version": classification["classifier_version"],
"blocks": [{"kind": block["kind"], "chars": len(block["text"]),
"substantive": block.get("substantive", False)}
for block in classification["blocks"]]}))
for chunk, vector in zip(chunks, vectors): for chunk, vector in zip(chunks, vectors):
db.add(KnowledgeChunk(document_id=doc.id, course_version_id=course_version_id, project_id=project_id, db.add(KnowledgeChunk(document_id=doc.id, course_version_id=course_version_id, project_id=project_id,
document_type=category, page=chunk["page"], section=chunk["section"], text=chunk["text"], embedding=vector)) document_type=category, page=chunk["page"], section=chunk["section"],
chapter=chunk["chapter"], page_type=chunk["page_type"],
block_type=chunk["block_type"], text=chunk["text"], embedding=vector))
doc.status = "ready" doc.status = "ready"
db.commit() db.commit()
return doc return doc
def retrieve(db, *, course_version_id: int, project_id: int, query: str, category: str | None = None, def _scope_filter(project_id: int | None, include_public: bool, public_only: bool):
limit=10, embedder=embed, include_public=True, public_only=False): if public_only:
"""Hybrid retrieval with project isolation and an optional public course namespace.""" return KnowledgeChunk.project_id.is_(None)
scope = (KnowledgeChunk.project_id.is_(None) if public_only else if include_public:
or_(KnowledgeChunk.project_id == project_id, KnowledgeChunk.project_id.is_(None)) if include_public else return or_(KnowledgeChunk.project_id == project_id, KnowledgeChunk.project_id.is_(None))
KnowledgeChunk.project_id == project_id) return KnowledgeChunk.project_id == project_id
filters = [KnowledgeChunk.course_version_id == course_version_id, scope]
def _base_filters(*, course_version_id: int, project_id: int | None, category, categories,
include_public: bool, public_only: bool, allowed_page_types, exclude_document_ids):
filters = [KnowledgeChunk.course_version_id == course_version_id,
_scope_filter(project_id, include_public, public_only)]
if category: if category:
filters.append(KnowledgeChunk.document_type == category) filters.append(KnowledgeChunk.document_type == category)
vector = embedder([query])[0] if categories:
filters.append(KnowledgeChunk.document_type.in_(list(categories)))
if allowed_page_types is not None:
filters.append(KnowledgeChunk.page_type.in_(list(allowed_page_types)))
if exclude_document_ids:
filters.append(KnowledgeChunk.document_id.not_in(list(exclude_document_ids)))
return filters
def _keyword_query(db, filters, query: str, limit: int):
"""Keyword recall ranked by bigram/token overlap over the already-filtered rows.
PostgreSQL's ``to_tsvector('simple', …)`` cannot segment Chinese, so it is not used
for recall. The overlap is computed with ``strpos``/``substr`` so the same expression
works on every dialect SQLAlchemy targets here.
"""
ascii_tokens, bigrams = query_terms(query)
if not ascii_tokens and not bigrams:
return []
total = len(ascii_tokens) + len(bigrams)
score = None
for token in ascii_tokens:
term = literal(token)
hit = case((func.strpos(func.lower(KnowledgeChunk.text), term) > 0, 1.0), else_=0.0)
score = hit if score is None else score + hit
if bigrams:
start = func.generate_series(1, func.greatest(func.length(KnowledgeChunk.text) - 1, 0))
series = select(start.label("i")).subquery("series")
matches = (select(func.count()).select_from(series)
.where(func.substr(KnowledgeChunk.text, series.c.i, 2).in_(bigrams))
.scalar_subquery())
score = matches if score is None else score + matches
score = (score / float(total)).label("keyword")
rows = db.execute(select(KnowledgeChunk.id, score).where(*filters)
.order_by(score.desc(), KnowledgeChunk.id.asc()).limit(limit)).all()
return [row[0] for row in rows if float(row[1] or 0) > 0]
def keyword_overlap(text_value: str, ascii_tokens: list[str], bigrams: list[str]) -> float:
"""Bigram/token overlap used by the non-PostgreSQL path and by tests."""
value = str(text_value or "")
lower = value.lower()
hits = sum(1 for token in ascii_tokens if token in lower)
total = len(ascii_tokens)
hits += sum(1 for bigram in bigrams if bigram in value)
total += len(bigrams)
return hits / total if total else 0.0
def _semantic_query(db, filters, vector, limit: int):
if db.bind.dialect.name == "postgresql": if db.bind.dialect.name == "postgresql":
from pgvector.sqlalchemy import Vector from pgvector.sqlalchemy import Vector
semantic = (1 - cast(KnowledgeChunk.embedding, Vector(1024)).cosine_distance(vector)).label("semantic") distance = cast(KnowledgeChunk.embedding, Vector(1024)).cosine_distance(vector)
lexical = func.ts_rank(func.to_tsvector("simple", KnowledgeChunk.text), func.plainto_tsquery("simple", query)).label("lexical") rows = db.execute(select(KnowledgeChunk.id).where(*filters)
rows = db.execute(select(KnowledgeChunk, semantic, lexical).where(*filters).order_by(semantic.desc(), lexical.desc()).limit(limit * 3)).all() .order_by(distance.asc(), KnowledgeChunk.id.asc()).limit(limit)).all()
ranked = [(row[0], float(row.semantic or 0) * .7 + float(row.lexical or 0) * .3) for row in rows] return [row[0] for row in rows]
else: rows = db.execute(select(KnowledgeChunk.id, KnowledgeChunk.embedding).where(*filters)).all()
chunks = db.scalars(select(KnowledgeChunk).where(*filters)).all() scored = [(chunk_id, sum(a * b for a, b in zip(embedding or [], vector)))
terms = [query[i:i+2] for i in range(max(0, len(query)-1)) if query[i:i+2].strip()] for chunk_id, embedding in rows]
ranked = [] scored.sort(key=lambda pair: (-pair[1], pair[0]))
for chunk in chunks: return [chunk_id for chunk_id, _ in scored[:limit]]
semantic = sum(a*b for a, b in zip(chunk.embedding or [], vector))
lexical = sum(term in chunk.text for term in terms) / max(len(terms), 1)
ranked.append((chunk, semantic * .7 + lexical * .3)) def _fuse(db, ids: list[int], semantic_ids: list[int], keyword_ids: list[int]):
ranked.sort(key=lambda pair: pair[1], reverse=True) """Reciprocal rank fusion over the two ranked lists, keeping discovery order stable."""
return [{"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id, scores: dict[int, float] = {chunk_id: 0.0 for chunk_id in ids}
"document_type": chunk.document_type, "page": chunk.page, "section": chunk.section, for ranked in (semantic_ids, keyword_ids):
"text": chunk.text, "rank": float(rank)} for chunk, rank in ranked[:limit]] for rank, chunk_id in enumerate(ranked):
if chunk_id in scores:
scores[chunk_id] += 1.0 / (RRF_K + rank + 1)
return scores
def _row_dict(chunk):
return {"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id,
"document_type": chunk.document_type, "page": chunk.page, "section": chunk.section,
"chapter": chunk.chapter, "page_type": chunk.page_type, "block_type": chunk.block_type,
"text": chunk.text}
def normalize_hit(hit: dict) -> dict:
"""Fill the fields the rest of the pipeline reads, so one shape flows through.
An alternate retriever or a test double may return only the essentials. Filling the
defaults here keeps every consumer from re-deriving them; the authoritative page type
is still re-read from storage when a score is produced.
"""
return {"chunk_id": hit.get("chunk_id"), "document_id": hit.get("document_id"),
"project_id": hit.get("project_id"), "document_type": hit.get("document_type"),
"page": hit.get("page"), "text": hit.get("text") or "",
"section": hit.get("section") or "", "chapter": hit.get("chapter") or "",
"page_type": hit.get("page_type") or "unknown", "block_type": hit.get("block_type") or "legacy",
"rank": float(hit.get("rank") or 0.0)}
def retrieve(db, *, course_version_id: int, project_id: int, query: str, category: str | None = None,
categories=None, limit=10, embedder=embed, include_public=True, public_only=False,
allowed_page_types=None, exclude_document_ids=(), substantive_only=False,
semantic_pool: int | None = None, keyword_pool: int | None = None):
"""Filtered hybrid retrieval, then rank fusion over the two independent recalls.
Page filtering happens inside the query, so it always precedes the candidate limit:
a contents page can never displace a body page simply by scoring well.
"""
filters = _base_filters(course_version_id=course_version_id, project_id=project_id, category=category,
categories=categories, include_public=include_public, public_only=public_only,
allowed_page_types=allowed_page_types, exclude_document_ids=exclude_document_ids)
if substantive_only:
filters.append(KnowledgeChunk.block_type != BLOCK_TOC)
vector = embedder([query])[0]
semantic_ids = _semantic_query(db, filters, vector, semantic_pool or SEMANTIC_POOL)
keyword_ids = _keyword_query(db, filters, query, keyword_pool or KEYWORD_POOL)
ids = list(dict.fromkeys(semantic_ids + keyword_ids))
if not ids:
return []
scores = _fuse(db, ids, semantic_ids, keyword_ids)
chunks = db.scalars(select(KnowledgeChunk).where(KnowledgeChunk.id.in_(ids))).all()
usable = []
for chunk in chunks:
if substantive_only and (chunk.block_type == BLOCK_TOC
or (chunk.page_type == "body" and is_toc_entry(chunk.text)
and len(chunk.text.strip()) < MIN_CHUNK_CHARS)):
continue
usable.append(chunk)
usable.sort(key=lambda chunk: (-scores.get(chunk.id, 0.0), chunk.id))
return [{**_row_dict(chunk), "rank": float(scores.get(chunk.id, 0.0))} for chunk in usable[:limit]]
def _page_diverse(chunks: list[dict], limit: int, *, page_limit: int, section_cap: int | None):
"""Spread the first pass over pages, then allow a bounded number per section."""
picked: list[dict] = []
per_page: dict[tuple, int] = {}
per_section: dict[str, int] = {}
remaining = []
for chunk in chunks:
page_key = (chunk.get("document_id"), chunk.get("page"))
section = str(chunk.get("section") or "")
if len(picked) < limit and per_page.get(page_key, 0) < page_limit:
if section_cap and section and per_section.get(section, 0) >= section_cap:
remaining.append(chunk)
continue
picked.append(chunk)
per_page[page_key] = per_page.get(page_key, 0) + 1
if section:
per_section[section] = per_section.get(section, 0) + 1
else:
remaining.append(chunk)
for chunk in remaining:
if len(picked) >= limit:
break
picked.append(chunk)
return picked[:limit]
def retrieve_targets(db, *, course_version_id: int, project_id: int, targets: list[dict], category=None,
categories=None, limit=SCORE_EVIDENCE_LIMIT, embedder=embed, include_public=False,
page_limit=DEFAULT_PAGE_LIMIT, section_cap=DEFAULT_SECTION_CAP,
per_target_limit: int | None = None, navigator=None, retrieve_fn=retrieve):
"""Search once per required-evidence target and merge with page-level diversity.
Every target is searched separately with its own query and its own allowed page
types, so a contents page permitted for a structural target can never flood a
content target's evidence. A target counts as unmet when its *own* search found
nothing admissible, which is what decides a teacher review.
"""
merged: list[dict] = []
seen: set[int] = set()
per_target: dict[str, list[dict]] = {}
for target in targets:
allowed = target.get("allowed_page_types")
raw_hits = retrieve_fn(db, course_version_id=course_version_id, project_id=project_id, category=category,
categories=categories, query=target.get("query") or target.get("label") or "",
limit=per_target_limit or max(limit, 8), embedder=embed,
include_public=include_public, allowed_page_types=allowed,
substantive_only=target.get("use") != "structure",
semantic_pool=SEMANTIC_POOL, keyword_pool=KEYWORD_POOL)
hits = [normalize_hit(hit) for hit in (raw_hits or [])]
if target.get("use") == "content" and not hits and navigator is not None:
# The contents list may only be used to navigate: its headings are matched
# against headings that really occur in body text, and those sections and
# their neighbouring pages are searched. The quotation still lands on body.
hits = [dict(hit, via="toc_navigation") for hit in (navigator(target) or [])]
per_target[target["target_id"]] = hits
for hit in hits:
if hit["chunk_id"] not in seen:
seen.add(hit["chunk_id"])
merged.append({**hit, "target_id": target["target_id"], "target_label": target["label"],
"target_use": target["use"]})
merged.sort(key=lambda chunk: -chunk["rank"])
selected = _page_diverse(merged, limit, page_limit=page_limit, section_cap=section_cap)
uncovered = [target for target in targets if not per_target.get(target["target_id"])]
return {"evidence": selected, "per_target": per_target, "uncovered_targets": uncovered}
def body_heading_index(db, *, course_version_id: int, project_id: int, category: str | None = None):
"""Sections present in real body text, used to navigate from a contents page."""
filters = [KnowledgeChunk.course_version_id == course_version_id, KnowledgeChunk.project_id == project_id,
KnowledgeChunk.page_type == "body"]
if category:
filters.append(KnowledgeChunk.document_type == category)
rows = db.execute(select(KnowledgeChunk.document_id, KnowledgeChunk.page, KnowledgeChunk.section,
KnowledgeChunk.chapter).where(*filters).distinct()).all()
index: dict[str, list[dict]] = {}
for document_id, page, section, chapter in rows:
for value in (section, chapter):
key = normalize_heading(value or "")
if key:
index.setdefault(key, []).append({"document_id": document_id, "page": page,
"section": section, "chapter": chapter})
return index
def page_text(source: dict) -> str:
blocks = [str(source.get("text") or "").strip()]
blocks.extend("表格: " + str(table.get("rows", table)) for table in source.get("tables", []))
return "\n".join(block for block in blocks if block)
def relative_image_path(value: str | None) -> str | None:
if not value:
return None
root = get_settings().processed_path.resolve()
path = Path(value).resolve()
if not path.is_relative_to(root):
raise ValueError("页面图像路径不在处理目录内")
return str(path.relative_to(root))
+234 -34
View File
@@ -6,15 +6,30 @@ from sqlalchemy import select, update
from .config import get_settings from .config import get_settings
from .db import SessionLocal from .db import SessionLocal
from .evidence import USE_CONTENT, content_targets, normalize_targets
from .knowledge_pages import page_image_path from .knowledge_pages import page_image_path
from .llm import ConsistencyResult, FactResult, ModelError, ScoreResult, validate_score from .llm import ConsistencyResult, FactResult, ModelError, ScoreResult, validate_score
from .models import (CriterionScore, KnowledgeDocument, Project, ScoreEvidence, ScoringBatch, from .models import (CriterionScore, KnowledgeChunk, KnowledgeDocument, KnowledgePage, Project,
SubmissionFile, now) ScoreEvidence, ScoringBatch, SubmissionFile, now)
from .pages import normalize_heading
from .parsing import parse_document from .parsing import parse_document
from .rag import INDEX_VERSION, ingest, retrieve from .rag import (INDEX_VERSION, body_heading_index, ingest, retrieve, retrieve_targets)
from .storage import secure_path, sha256 from .storage import secure_path, sha256
from .settings_store import client_for_batch from .settings_store import client_for_batch
#: Criterion statuses. ``needs_review`` is deliberately distinct from a missing
#: material, a parse failure and a model failure: the retrieval ran but only found
#: navigation pages where the rule requires body content.
STATUS_NEEDS_REVIEW = "needs_review"
STATUS_UNSCORABLE = "unscorable"
STATUS_FAILED = "failed"
STATUS_SCORED = "scored"
STATUS_REVIEWED = "reviewed"
#: Statuses that a batch counts as finished work.
DONE_STATUSES = (STATUS_SCORED, STATUS_REVIEWED, STATUS_NEEDS_REVIEW)
#: Statuses the dashboard shows as needing teacher attention.
ATTENTION_STATUSES = (STATUS_FAILED, STATUS_UNSCORABLE, STATUS_NEEDS_REVIEW)
@contextmanager @contextmanager
def file_index_lock(file_id: int): def file_index_lock(file_id: int):
@@ -72,12 +87,21 @@ def _parse_submission_locked(db, file: SubmissionFile, *, embedder=None):
def _project_facts(db, project, course_snapshot, client, *, retriever): def _project_facts(db, project, course_snapshot, client, *, retriever):
"""Extract project facts from body text only.
Implementation facts must not be inferred from a contents heading or an abstract
claim, so the search admits body and appendix pages exclusively. Project identity
(name, student number, topic) legitimately lives on the cover page and is read from
the teacher-confirmed import record instead of from retrieved text.
"""
prompts = course_snapshot["prompts"] prompts = course_snapshot["prompts"]
categories = ("stage1", "stage2", "stage3", "final_report", "presentation") categories = ("stage1", "stage2", "stage3", "final_report", "presentation")
snippets = [] snippets = []
for category in categories: for category in categories:
snippets.extend(retriever(db, course_version_id=project.course_version_id, project_id=project.id, snippets.extend(retriever(db, course_version_id=project.course_version_id, project_id=project.id,
category=category, query="项目名称 目标 指标 硬件 软件 实验 数据 成本 结论", limit=3, include_public=False)) category=category, query="项目名称 目标 指标 硬件 软件 实验 数据 成本 结论",
limit=3, include_public=False, allowed_page_types=["body", "appendix"],
substantive_only=True))
if not snippets: if not snippets:
return {} return {}
if client is None: if client is None:
@@ -100,10 +124,54 @@ def _project_facts(db, project, course_snapshot, client, *, retriever):
return project.facts return project.facts
def _score_criterion(db, batch, project, criterion, client, *, retriever): def _body_navigator(db, *, course_version_id, project_id, category):
"""Find real body pages for a target whose contents entries name missing sections.
The contents page is used only as a map: its headings are matched against headings
that actually occur in body text, and the body sections and their neighbouring pages
are then searched. The final quotation always lands on a real body page.
"""
headings = body_heading_index(db, course_version_id=course_version_id, project_id=project_id, category=category)
if not headings:
return None
def navigate(target):
label = normalize_heading(target.get("label") or "")
if not label:
return []
candidates = []
for heading, places in headings.items():
if not heading:
continue
if heading.startswith(label) or label.startswith(heading) or label in heading or heading in label:
candidates.extend(places)
if not candidates:
return []
pages = sorted({(place["document_id"], place["page"]) for place in candidates})
neighbours = set()
for document_id, page in pages:
neighbours.update({(document_id, page), (document_id, page + 1)})
wanted = sorted(neighbours)
rows = db.scalars(select(KnowledgeChunk).where(
KnowledgeChunk.course_version_id == course_version_id, KnowledgeChunk.project_id == project_id,
KnowledgeChunk.page_type == "body", KnowledgeChunk.block_type != "toc",
KnowledgeChunk.document_id.in_([item[0] for item in wanted]),
KnowledgeChunk.page.in_([item[1] for item in wanted])).order_by(KnowledgeChunk.id)).all()
hits = []
for chunk in rows:
hits.append({"chunk_id": chunk.id, "document_id": chunk.document_id, "project_id": chunk.project_id,
"document_type": chunk.document_type, "page": chunk.page, "section": chunk.section,
"chapter": chunk.chapter, "page_type": chunk.page_type, "block_type": chunk.block_type,
"text": chunk.text, "rank": 0.5})
return hits
return navigate
def _score_criterion(db, batch, project, criterion, client, *, retriever, multi_retriever=retrieve_targets):
score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id, score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
CriterionScore.project_id == project.id, CriterionScore.criterion_id == criterion["criterion_id"])) CriterionScore.project_id == project.id, CriterionScore.criterion_id == criterion["criterion_id"]))
if score and score.status in ("scored", "reviewed"): if score and score.status in (STATUS_SCORED, STATUS_REVIEWED):
return score return score
if not score: if not score:
score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"]) score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"])
@@ -115,44 +183,103 @@ def _score_criterion(db, batch, project, criterion, client, *, retriever):
SubmissionFile.category == category, SubmissionFile.recognition_status == "recognized")).all() SubmissionFile.category == category, SubmissionFile.recognition_status == "recognized")).all()
ready = [file for file in files if file.parse_status == "ready"] ready = [file for file in files if file.parse_status == "ready"]
if len(files) != 1 or len(ready) != 1: if len(files) != 1 or len(ready) != 1:
score.status, score.error = "unscorable", "材料缺失、重复、待确认或解析失败" score.status, score.error = STATUS_UNSCORABLE, "材料缺失、重复、待确认或解析失败"
db.commit() db.commit()
return score return score
query = criterion["description"] + " " + " ".join(criterion["required_evidence"])
chunks = retriever(db, course_version_id=batch.course_version_id, project_id=project.id, plan = batch.rubric_snapshot.get("evidence_plan") or {}
category=category, query=query, limit=10, include_public=False) targets = (plan.get("criteria") or {}).get(criterion["criterion_id"]) or normalize_targets(
criterion.get("required_evidence"))
navigator = _body_navigator(db, course_version_id=batch.course_version_id,
project_id=project.id, category=category)
# The single ``retriever`` stays the seam: each per-target search goes through it,
# so a caller that injects a retriever (tests, alternate index) still applies.
selection = multi_retriever(db, course_version_id=batch.course_version_id, project_id=project.id,
targets=targets, category=category, navigator=navigator,
retrieve_fn=retriever)
chunks = _attach_page_types(db, selection["evidence"])
gaps = selection["uncovered_targets"]
if not chunks: if not chunks:
score.status, score.error = "unscorable", "无可检索的材料证据" # Nothing admissible was found for any target. Content targets go to review;
# a purely structural criterion only depends on navigation pages, which exist.
score.status = STATUS_NEEDS_REVIEW if content_targets(criterion) else STATUS_UNSCORABLE
score.error = "未检索到允许作为证据的正文页"
score.model_score = None
score.missing_evidence = [target["label"] for target in gaps]
_record_evidence_gaps(score, gaps)
db.commit() db.commit()
return score return score
if client is None: if client is None:
raise ModelError("未配置或无法连接评分模型") raise ModelError("未配置或无法连接评分模型")
image_sources = dict.fromkeys((chunk["document_id"], chunk["page"]) for chunk in chunks[:2])
content_gaps = [target for target in gaps if target["use"] == USE_CONTENT]
if content_gaps:
# The model is not asked to grade from navigation pages. Record the gap and keep
# the model score null so a teacher decides, rather than asserting a low score.
score.model_score, score.confidence, score.status = None, None, STATUS_NEEDS_REVIEW
score.model_comment = "检索仅命中目录、摘要或章节标题,缺少可用于内容判断的正文证据,转教师复核。"
score.missing_evidence = [target["label"] for target in content_gaps]
score.suggestion = "请补充该章节的正文页,或由教师直接核定该项分数。"
score.error = "缺少正文证据:" + "、".join(target["label"] for target in content_gaps)
_record_evidence_gaps(score, content_gaps, searched=_searched_page_types(plan, content_gaps))
db.commit()
return score
# Page images go to the model as well, so prefer substantive body pages over the
# navigation pages that may legitimately appear in a structural target's evidence.
image_sources = list(dict.fromkeys(
(chunk["document_id"], chunk["page"]) for chunk in chunks
if chunk.get("page_type") == "body"))
if not image_sources:
image_sources = list(dict.fromkeys((chunk["document_id"], chunk["page"]) for chunk in chunks))
images = [] images = []
for document_id, page in image_sources: for document_id, page in image_sources[:3]:
doc = db.get(KnowledgeDocument, document_id) doc = db.get(KnowledgeDocument, document_id)
path = page_image_path(db, doc, page) if doc else None path = page_image_path(db, doc, page) if doc else None
if path: if path:
images.append(str(path)) images.append(str(path))
query = criterion["description"] + " " + " ".join(target["label"] for target in targets)
public_context = retriever(db, course_version_id=batch.course_version_id, project_id=project.id, public_context = retriever(db, course_version_id=batch.course_version_id, project_id=project.id,
category=None, query=query, limit=6, public_only=True) category=None, query=query, limit=6, public_only=True)
payload = {"criterion": criterion, "project_facts": project.facts, "course_context": public_context, payload = {"criterion": criterion,
"evidence": chunks, "required_evidence_targets": [{"target_id": target["target_id"], "label": target["label"],
"required_output": {"criterion_id": "string", "score": "number", "max_score": "number", "reason": "string", "use": target["use"],
"evidence": [{"chunk_id": "integer", "page": "integer", "quote": "exact substring"}], "allowed_page_types": target["allowed_page_types"]}
"missing_evidence": ["string"], "suggestion": "string", "confidence": "0..1"}} for target in targets],
"evidence_policy": {
"structure": "目录、封面和参考文献页可以证明章节结构和报告完整性,不能证明章节内容或完成质量。",
"content": "内容判断只能使用正文或附录中的实质段落、数据、表格;目录、摘要和单独章节标题不能作为内容证据。"},
"project_facts": project.facts, "course_context": public_context, "evidence": chunks,
"required_output": {"criterion_id": "string", "score": "number or null", "max_score": "number",
"reason": "string",
"evidence": [{"chunk_id": "integer", "page": "integer",
"quote": "exact substring", "target_id": "string"}],
"missing_evidence": ["string"], "suggestion": "string", "confidence": "0..1",
"note": "证据不足以判断内容时 score 填 null,并在 missing_evidence 中列出缺少的正文目标。"}}
prompt = (batch.course_snapshot["prompts"]["score"] + prompt = (batch.course_snapshot["prompts"]["score"] +
" 证据quote必须从所引用chunk_id的text逐字复制连续片段,保留原文标点和换行;page必须等于该片段的page。不要改写、缩写或拼接引文。") " 证据quote必须从所引用chunk_id的text逐字复制连续片段,保留原文标点和换行;page必须等于该片段的page;"
"target_id必须取自required_evidence_targets。不要改写、缩写或拼接引文。"
"不得把目录、摘要或章节标题当作内容证据;无法据此判断内容时把score设为null。")
error = ""
result = None
for attempt in range(2): for attempt in range(2):
try: try:
result = validate_score(client.json_call(prompt, payload, ScoreResult, images=images), criterion, chunks) result = validate_score(client.json_call(prompt, payload, ScoreResult, images=images), criterion, chunks,
targets=targets)
break break
except ModelError: except ModelError as exc:
error = str(exc)
if attempt: if attempt:
raise raise
prompt += " 上次回答的证据未通过原文和页码校验;请重新选择片段并逐字复制quote。" prompt += f" 上次回答未通过校验({error});请改用允许页面类型中的实质正文片段,并逐字复制quote。"
score.model_score, score.model_comment, score.confidence = result.score, result.reason, result.confidence score.model_score, score.model_comment = result.score, result.reason
score.missing_evidence, score.suggestion, score.status, score.error = result.missing_evidence, result.suggestion, "scored", "" score.confidence = result.confidence
score.missing_evidence, score.suggestion = result.missing_evidence, result.suggestion
if result.score is None:
# The retrieval found body pages but the model judged them insufficient.
score.status, score.error = STATUS_NEEDS_REVIEW, "模型认为现有正文证据不足以判断该项,转教师复核。"
else:
score.status, score.error = STATUS_SCORED, ""
for evidence in db.scalars(select(ScoreEvidence).where(ScoreEvidence.score_id == score.id)).all(): for evidence in db.scalars(select(ScoreEvidence).where(ScoreEvidence.score_id == score.id)).all():
db.delete(evidence) db.delete(evidence)
by_id = {chunk["chunk_id"]: chunk for chunk in chunks} by_id = {chunk["chunk_id"]: chunk for chunk in chunks}
@@ -160,14 +287,75 @@ def _score_criterion(db, batch, project, criterion, client, *, retriever):
chunk = by_id[evidence.chunk_id] chunk = by_id[evidence.chunk_id]
doc = db.get(KnowledgeDocument, chunk["document_id"]) doc = db.get(KnowledgeDocument, chunk["document_id"])
path = page_image_path(db, doc, evidence.page) path = page_image_path(db, doc, evidence.page)
target = next((item for item in targets if item["target_id"] == evidence.target_id), None)
db.add(ScoreEvidence(score_id=score.id, document_id=doc.id, submission_file_id=doc.submission_file_id, db.add(ScoreEvidence(score_id=score.id, document_id=doc.id, submission_file_id=doc.submission_file_id,
page=evidence.page, quote=evidence.quote, image_path=str(path) if path else None)) page=evidence.page, quote=evidence.quote, image_path=str(path) if path else None,
target_id=evidence.target_id, target_label=target["label"] if target else "",
page_type=chunk.get("page_type") or "unknown",
source_role=target["use"] if target else USE_CONTENT))
db.commit() db.commit()
return score return score
def _run_project(batch_id, project_id, session_factory, client_factory, embedder, retriever): def _attach_page_types(db, chunks: list[dict]) -> list[dict]:
"""Attach the stored page classification to each retrieved chunk.
The page type recorded in ``knowledge_pages`` is the authority for the evidence
policy. Reading it here, rather than trusting whatever a retriever happened to put
in the dict, means a statement about what a page is cannot be forged by a caller
or lost by an alternate index.
"""
keys = {(chunk.get("document_id"), chunk.get("page")) for chunk in chunks}
keys.discard((None, None))
if not keys:
return chunks
documents = {document_id for document_id, _ in keys}
pages = {page for _, page in keys}
records = db.scalars(select(KnowledgePage).where(KnowledgePage.document_id.in_(documents),
KnowledgePage.page.in_(pages))).all()
by_key = {(item.document_id, item.page): item for item in records}
output = []
for chunk in chunks:
record = by_key.get((chunk.get("document_id"), chunk.get("page")))
if record is None:
output.append({**chunk, "page_type": chunk.get("page_type") or "unknown"})
continue
output.append({**chunk, "page_type": record.page_type or "unknown",
"section": chunk.get("section") or record.section or "",
"chapter": chunk.get("chapter") or record.chapter or ""})
return output
def _searched_page_types(plan, targets):
allowed = set()
for target in targets:
allowed.update(target.get("allowed_page_types") or [])
return sorted(allowed or (plan.get("content_allowed_page_types") or []))
def _record_evidence_gaps(score, gaps, searched=None):
"""Store which content targets lacked body evidence and where retrieval looked.
The record is kept through teacher review so the original retrieval problem stays
auditable next to the score the teacher finally entered.
"""
score.evidence_gaps = [
{"target_id": target.get("target_id"), "target_label": target.get("label"),
"searched_page_types": searched or target.get("allowed_page_types") or [],
"reason": "补检后仍未找到正文证据"}
for target in gaps]
def _run_project(batch_id, project_id, session_factory, client_factory, embedder, retriever,
multi_retriever=retrieve_targets):
"""Score one project. Returns ``(failures, reviews)``.
Failures are missing material, parse problems and model errors. Reviews count the
items that only found navigation pages where content evidence is required; those
need a teacher decision rather than a retry.
"""
failures = 0 failures = 0
reviews = 0
with session_factory() as db: with session_factory() as db:
batch = db.get(ScoringBatch, batch_id) batch = db.get(ScoringBatch, batch_id)
project = db.get(Project, project_id) project = db.get(Project, project_id)
@@ -190,8 +378,11 @@ def _run_project(batch_id, project_id, session_factory, client_factory, embedder
failures += 1 failures += 1
for criterion in batch.rubric_snapshot["criteria"]: for criterion in batch.rubric_snapshot["criteria"]:
try: try:
result = _score_criterion(db, batch, project, criterion, client, retriever=retriever) result = _score_criterion(db, batch, project, criterion, client, retriever=retriever,
if result.status not in ("scored", "reviewed"): multi_retriever=multi_retriever)
if result.status == STATUS_NEEDS_REVIEW:
reviews += 1
elif result.status not in (STATUS_SCORED, STATUS_REVIEWED):
failures += 1 failures += 1
except Exception as exc: except Exception as exc:
score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id, score = db.scalar(select(CriterionScore).where(CriterionScore.batch_id == batch.id,
@@ -199,15 +390,16 @@ def _run_project(batch_id, project_id, session_factory, client_factory, embedder
if not score: if not score:
score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"]) score = CriterionScore(batch_id=batch.id, project_id=project.id, criterion_id=criterion["criterion_id"])
db.add(score) db.add(score)
score.status, score.error = "failed", str(exc)[:500] score.status, score.error = STATUS_FAILED, str(exc)[:500]
db.commit() db.commit()
failures += 1 failures += 1
db.execute(update(ScoringBatch).where(ScoringBatch.id == batch_id).values(progress_done=ScoringBatch.progress_done + 1)) db.execute(update(ScoringBatch).where(ScoringBatch.id == batch_id).values(progress_done=ScoringBatch.progress_done + 1))
db.commit() db.commit()
return failures return failures, reviews
def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=client_for_batch, embedder=None, retriever=retrieve): def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=client_for_batch, embedder=None,
retriever=retrieve, multi_retriever=retrieve_targets):
with session_factory() as db: with session_factory() as db:
batch = db.get(ScoringBatch, batch_id) batch = db.get(ScoringBatch, batch_id)
if not batch: if not batch:
@@ -218,13 +410,21 @@ def run_batch(batch_id: int, *, session_factory=SessionLocal, client_factory=cli
batch.status, batch.progress_total, batch.progress_done, batch.finished_at = "running", len(project_ids) * len(criteria), 0, None batch.status, batch.progress_total, batch.progress_done, batch.finished_at = "running", len(project_ids) * len(criteria), 0, None
db.commit() db.commit()
failures = 0 failures = 0
reviews = 0
with ThreadPoolExecutor(max_workers=max(1, int(batch.model_config_snapshot.get("concurrency", 1)))) as pool: with ThreadPoolExecutor(max_workers=max(1, int(batch.model_config_snapshot.get("concurrency", 1)))) as pool:
futures = [pool.submit(_run_project, batch_id, project_id, session_factory, client_factory, embedder, retriever) for project_id in project_ids] futures = [pool.submit(_run_project, batch_id, project_id, session_factory, client_factory,
embedder, retriever, multi_retriever) for project_id in project_ids]
for future in as_completed(futures): for future in as_completed(futures):
try: try:
failures += future.result() project_failures, project_reviews = future.result()
failures += project_failures
reviews += project_reviews
except Exception as exc: except Exception as exc:
failures += 1 failures += 1
batch.error = (batch.error + f" 项目任务失败: {type(exc).__name__}")[:1000] batch.error = (batch.error + f" 项目任务失败: {type(exc).__name__}")[:1000]
batch.status, batch.finished_at = ("partial" if failures else "completed"), now() # A batch with unresolved review items is finished running, but it is not a
# completed grading result: the teacher still has to decide those criteria.
batch.status = "partial" if failures else ("review_required" if reviews else "completed")
batch.finished_at = now()
db.commit() db.commit()
return {"failures": failures, "reviews": reviews, "status": batch.status}
+23 -5
View File
@@ -9,6 +9,7 @@ from redis import Redis
from rq import Queue, Retry from rq import Queue, Retry
from sqlalchemy import delete, func, select from sqlalchemy import delete, func, select
from .config import get_settings from .config import get_settings
from .evidence import plan_snapshot
from .exports import export_excel, export_project_pdf from .exports import export_excel, export_project_pdf
from .models import (AuditLog, CourseVersion, CriterionScore, IndexTask, KnowledgeChunk, KnowledgeDocument, from .models import (AuditLog, CourseVersion, CriterionScore, IndexTask, KnowledgeChunk, KnowledgeDocument,
Project, RubricVersion, ScoringBatch, Student, Submission, SubmissionFile, TeacherRevision) Project, RubricVersion, ScoringBatch, Student, Submission, SubmissionFile, TeacherRevision)
@@ -209,9 +210,12 @@ def create_batch(data, db):
if not version.published or not rubric.published or rubric.course_version_id != version.id: if not version.published or not rubric.published or rubric.course_version_id != version.id:
raise HTTPException(400, "请选择同一课程版本下已发布的规则") raise HTTPException(400, "请选择同一课程版本下已发布的规则")
settings = get_settings() settings = get_settings()
# The evidence policy travels with the batch snapshot, so a batch can always be read
# back against the rules and page permissions it was actually scored with.
batch = ScoringBatch(course_version_id=version.id, rubric_version_id=rubric.id, batch = ScoringBatch(course_version_id=version.id, rubric_version_id=rubric.id,
course_snapshot={key: getattr(version, key) for key in ("version", "title", "objectives", "indicators", "directions", "prompts", "templates")}, course_snapshot={key: getattr(version, key) for key in ("version", "title", "objectives", "indicators", "directions", "prompts", "templates")},
rubric_snapshot={"version": rubric.version, "criteria": rubric.criteria, "weights": rubric.weights}, rubric_snapshot={"version": rubric.version, "criteria": rubric.criteria,
"weights": rubric.weights, "evidence_plan": plan_snapshot(rubric.criteria)},
model_config_snapshot=model_config(db), embedding_model_name=settings.embedding_model_name, model_config_snapshot=model_config(db), embedding_model_name=settings.embedding_model_name,
embedding_model_version=settings.embedding_model_version, embedding_model_version=settings.embedding_model_version,
status="created", project_ids=db.scalars(select(Project.id).where( status="created", project_ids=db.scalars(select(Project.id).where(
@@ -226,7 +230,7 @@ def start_batch(batch_id: int, db):
batch = db.scalar(select(ScoringBatch).where(ScoringBatch.id == batch_id).with_for_update()) batch = db.scalar(select(ScoringBatch).where(ScoringBatch.id == batch_id).with_for_update())
if not batch: if not batch:
raise HTTPException(404, "评分批次不存在") raise HTTPException(404, "评分批次不存在")
if batch.status not in ("created", "failed", "partial"): if batch.status not in ("created", "failed", "partial", "review_required"):
db.rollback() db.rollback()
raise HTTPException(409, "批次已入队、运行中或已完成") raise HTTPException(409, "批次已入队、运行中或已完成")
try: try:
@@ -244,15 +248,29 @@ def start_batch(batch_id: int, db):
return {"id": batch.id, "status": batch.status} return {"id": batch.id, "status": batch.status}
def review_score(score_id: int, data, db): def review_score(score_id: int, data, db):
"""Record a teacher decision. Model scores, retrieval gaps and history are kept.
A teacher may review any criterion in any state, including one awaiting review or
one whose material is missing: the teacher is the authority on a criterion the
retrieval could not support. The gaps from the original retrieval stay on the record
after the review, so the reason it reached the teacher remains auditable.
"""
score = found(db, CriterionScore, score_id) score = found(db, CriterionScore, score_id)
batch = found(db, ScoringBatch, score.batch_id) batch = found(db, ScoringBatch, score.batch_id)
criterion = next((item for item in batch.rubric_snapshot["criteria"] if item["criterion_id"] == score.criterion_id), None) criterion = next((item for item in batch.rubric_snapshot["criteria"] if item["criterion_id"] == score.criterion_id), None)
if not criterion or data.score > criterion["max_score"]: if not criterion or data.score > criterion["max_score"]:
raise HTTPException(400, "教师分数超出规则满分") raise HTTPException(400, "教师分数超出规则满分")
from_status = score.status
db.add(TeacherRevision(score_id=score.id, old_score=score.teacher_score, new_score=data.score, db.add(TeacherRevision(score_id=score.id, old_score=score.teacher_score, new_score=data.score,
old_comment=score.teacher_comment, new_comment=data.comment, reason=data.reason)) old_comment=score.teacher_comment, new_comment=data.comment, reason=data.reason))
score.teacher_score, score.teacher_comment, score.status = data.score, data.comment, "reviewed" score.teacher_score, score.teacher_comment = data.score, data.comment
audit(db, "review", "criterion_score", score.id, {"reason": data.reason}) # ``evidence_gaps`` is intentionally left untouched: it records why the criterion
# reached the teacher, independently of the score the teacher then entered.
score.status = "reviewed"
audit(db, "review", "criterion_score", score.id,
{"reason": data.reason, "from_status": from_status,
"came_from_review": bool(score.evidence_gaps)})
db.commit() db.commit()
return {"id": score.id, "model_score": score.model_score, "teacher_score": score.teacher_score, return {"id": score.id, "model_score": score.model_score, "teacher_score": score.teacher_score,
"final_score": score.final_score, "teacher_comment": score.teacher_comment} "final_score": score.final_score, "teacher_comment": score.teacher_comment,
"status": score.status, "evidence_gaps": score.evidence_gaps or []}
+113
View File
@@ -0,0 +1,113 @@
# 证据政策上线与批次 3 复核清单
本文记录把页面类型证据政策应用到**业务环境**的步骤,以及批次 3 需要教师复核的项目。文中的业务数据操作**均未执行**,请由教师确认后按顺序操作。
## 一、诊断基线
在业务数据库只读检查确认:
- 批次 3 的 77 条评分引文中,**14 条来自目录页**,涉及 **9 个评分项**。
- 会话 6 保存的 20 条引用中,**5 条**指向阶段报告 1 的目录页。
这些数字是诊断基线。本次改动不自动改写批次 3 的既有分数,也不改写会话 6 的历史消息。
## 二、改动后的行为
| 情形 | 改动前 | 改动后 |
|---|---|---|
| 目录页命中 | 作为正文证据送入模型并计入候选 | 只作导航,不进入内容证据 |
| 内容目标证据 | 目录、摘要、章节标题均可 | 仅正文与附录的实质段落、数据、表格 |
| 结构目标证据 | 与其他证据等同 | 封面、目录、参考文献可用,明确标注来源 |
| 正文补检不足 | 仍可能给出较低模型分 | `needs_review`,模型分为 `null`,总分待处理 |
| 历史引文 | 与正文证据无区别 | 标记"历史评分引用" |
## 三、业务环境应用步骤
以下步骤**尚未执行**。
### 1. 备份
```bash
docker compose exec -T db pg_dump -U wisegrade -Fc wisegrade > data/backups/wisegrade-pre-0006-$(date +%Y%m%d-%H%M%S).dump
chmod 600 data/backups/wisegrade-pre-0006-*.dump
```
### 2. 隔离库演练(推荐)
把备份还原到隔离数据库,先在隔离库执行升级和重建,核对页面分类与检索结果,再动业务库。`backend/tests/test_evidence_pages.py` 已用真实报告覆盖这几类情形。
### 3. 应用迁移
迁移在 API 容器启动时自动执行(`alembic upgrade head`)。如需手工执行:
```bash
docker compose exec api alembic upgrade head
docker compose exec -T db psql -U wisegrade -c "SELECT version_num FROM alembic_version;"
# 期望:0006
```
`0006` 会:
- 为 `knowledge_pages` 添加 `page_type`、`mixed`、`section`、`chapter`、`confidence`、`classification`;
- 为 `knowledge_chunks` 添加 `chapter`、`page_type`、`block_type`;
- 为 `score_evidences` 添加 `target_id`、`target_label`、`page_type`、`source_role`、`historical`;
- 为 `criterion_scores` 添加 `evidence_gaps`;
- 把**已有**评分引文全部标记为 `historical = true`;
- 按已保存的解析结果重新分类所有页面,写入审计日志。
重新分类不需要嵌入模型,因此在迁移中直接完成。`downgrade` 可撤销上述列。
### 4. 重建索引
分块方式已改变,旧索引不会被当作已分类正文使用。查看待重建清单:
```bash
curl -s http://127.0.0.1:18080/api/knowledge/stale/1 | python -m json.tool
```
重建(课程版本 1):
```bash
curl -s -X POST http://127.0.0.1:18080/api/knowledge/reindex/1
docker compose logs -f worker
```
重建任务可重复执行:已是最新版本的文档跳过,源文件哈希变化的文档跳过并说明原因,失败项记录具体错误。结束后再次查询 `/api/knowledge/stale/1`,`documents` 应为空。
### 5. 核对页面分类
```bash
curl -s "http://127.0.0.1:18080/api/knowledge/documents/<document_id>/pages" | python -m json.tool
```
预期(仓库示例报告):阶段报告 1 第 1 页 `cover`、第 2–3 页 `toc`、第 4–21 页 `body`、第 22 页 `references`;阶段报告 3 第 2–3 页 `toc`、第 11–12 页 `body`;最终报告第 2–3 页 `abstract`、第 4–5 页 `toc`、第 49 页 `references`、第 50–53 页 `appendix`。分类有疑义的页面为 `unknown`,不会当作正文。
### 6. 新批次验证
新建一个评分批次(不要重跑批次 3),确认真实模型下的表现:已评分项的引文来源为正文或附录;只检索到目录的内容项进入"待教师复核"。
## 四、批次 3 复核清单
只读检查得到的 9 个涉及目录页引文的评分项。**建议教师逐项核对**,不需要重跑批次:
| 处理方式 | 说明 |
|---|---|
| 查看该评分项证据 | 界面中旧引文显示"历史评分引用"标签,不能当作报告正文证据 |
| 在原始报告中定位正文 | 用引用卡片上的"查看页面文字"/"查看图像"跳转,或按下方章节提示查找 |
| 填写教师复核分 | 输入框不预填 0;填写后状态变为"教师已复核",模型原始分与检索记录保留 |
复核时可直接检查的正文位置(仓库示例报告):
- **阶段报告 1 第 6–7 页**:文献调研与技术现状正文,含控制方法对比表和技术现状总结。涉及 `stage1.literature`。
- **阶段报告 3 第 11–12 页**:系统测试与验证正文,含自主巡线测试、控制实时性、光照影响。涉及 `stage3.data`、`stage3.verification`。
- **最终报告第 6 页起**:正文各章;第 50–53 页为附录,可用作内容证据。
- **最终报告第 2–3 页为摘要**:可用于"摘要称……"这类明确归属的概述,不能单独证明方法已实现或指标已达标。
教师也可选择保留原有分数:本次改动不自动改写批次 3,历史分数在导出和界面中维持原值,仅在引文旁标明其来源类型。
## 五、注意事项
- 迁移和重建只改动页面分类、索引与引文元数据,**不改动**学生原件、评分批次、教师复核记录和会话历史。
- 重建过程可重试;失败项保留具体错误,重试入口在项目的"解析与索引任务"中。
- 迁移后旧评分引文的原始分数不会改变,但界面与导出会标明它们是历史引用。
- 若教师希望批次 3 的引文重新落在正文页,需要新建批次并重新评分;该操作会消耗模型调用,请先确认。
+78 -11
View File
@@ -502,22 +502,24 @@ course_2026/project_1023009006/
- 重叠约 80~120 个中文字符 - 重叠约 80~120 个中文字符
- 表格单独保存 - 表格单独保存
- 图片及图片说明单独保存 - 图片及图片说明单独保存
- 每块必须保留页码、章节、文档类型和评分指标关联 - 每块必须保留页码、页面类型、章节、文档类型和评分指标关联
- 目录页文本作为导航块单独存放,不与正文块混同
#### 检索 #### 检索
采用混合检索: 采用混合检索:
```text ```text
PostgreSQL 全文检索 pgvector 向量检索
+ pgvector 向量检索 + 中文二元组关键词检索
+ 课程版本过滤 + 课程版本过滤
+ 学生项目过滤 + 学生项目过滤
+ 报告类型过滤 + 报告类型过滤
+ 阶段过滤 + 阶段过滤
+ 页面类型过滤
``` ```
默认每个评分项取 8~12 个候选证据块,去重后送入模型。 过滤在取候选之前完成,两个召回各自独立进行,再按排名融合。默认每个评分项送模型 8~12 个候选证据块,跨目标合计上限 14 块。
#### 嵌入模型 #### 嵌入模型
@@ -622,7 +624,8 @@ model: deepseek-flash
{ {
"document": "阶段报告1", "document": "阶段报告1",
"page": 12, "page": 12,
"quote": "..." "quote": "...",
"target_id": "t0"
} }
], ],
"missing_evidence": ["..."], "missing_evidence": ["..."],
@@ -631,7 +634,60 @@ model: deepseek-flash
} }
``` ```
模型不得对没有证据的内容自行假设。结构化输出必须经过 Pydantic 校验;失败时自动重试或进入人工处理。 模型不得对没有证据的内容自行假设。结构化输出必须经过 Pydantic 校验;失败时自动重试或进入人工处理。`score` 允许为 `null`:当规则要求正文证据而检索只能给出目录、摘要或章节标题时,模型必须返回 `null`,不得据此判断学生内容缺失。
每条引文必须通过三类校验:原文与页码一致、来源页面类型在该目标允许范围内、内容性判断不得只由目录、摘要或单独章节标题支撑。页面权限和状态转换由代码执行;模型的说明或推理不能改变它。
#### 页面类型与证据政策
每份资料的每一页都带一个页面类型,随页面持久化并可供检索、评分、复核和会话查询:
| 页面类型 | 含义 | 能否作为内容证据 |
|---|---|---|
| `cover` | 封面、标题页 | 否(结构目标可用) |
| `abstract` | 摘要、Abstract | 否(结构目标可用) |
| `toc` | 目录及其续页 | 否(结构目标可用) |
| `body` | 正文 | 是 |
| `appendix` | 附录 | 是 |
| `references` | 参考文献、致谢 | 否(结构目标可用) |
| `unknown` | 识别有疑义 | 否 |
判定结合页首标题、章节条目密度、点线与末尾页码、摘要/参考文献措辞、封面字段和页脚样式,并由文档级连续性识别跨页续目录、连续摘要和连续参考文献。它不依赖"目录"一词或固定页号;重复页眉页脚在建立索引前清理。识别依据(各项信号、置信度和分类器版本)与页面一起保存,便于教师核查。
目录与正文共处一页时标为混合页,只对该页按文本块区分:正文块仍作为正文检索,不被整页排除。
`required_evidence` 的目标带 `use` 属性,并随批次快照保存:
- `use: structure`:封面、目录、参考文献等页面可以满足,例如"章节""参考文献"。
- `use: content`:只允许正文和附录中的实质段落、数据或表格。
评分规则沿用历史上的字符串数组形式;读取时按下述语法转换,批次快照记录转换结果和 `POLICY_VERSION`,因此旧批次始终可按当时的规则重读。
#### 检索
每个必备证据目标分别检索:先限定课程版本、学生项目、材料类别和允许的页面类型,再取候选。**页面过滤发生在候选数量限制之前**,目录页不会因为得分高而挤掉正文页。
语义匹配和中文关键词匹配各自召回候选池,再按排名倒数(RRF)融合;不对未归一化的语义分和关键词分直接相加。
关键词召回不使用 PostgreSQL 的 `to_tsvector('simple', …)`:该配置没有中文分词,整个中文串会成为一个词元。在课程报告上实测,查询"文献调研"无法命中含"第2章文献调研与技术现状"的页面,因此现有实现里关键词分长期接近零。现在改用字符二元组加 ASCII/数字整词的重叠度,无需扩展且在各数据库上行为一致。
名额分配优先分散到不同页面;有了可靠章节路径后对同一章节设软上限;名额不足时回填同页或同节的其他正文片段,但不会放宽证据类型。
内容目标在正文中没有命中时,利用目录标题匹配实际正文标题,补检该章节及其相邻正文页。目录只作导航,最终内容引文必须落在实际正文页。
评分发送给模型的页面图像优先选择相关的不同正文页,而不是目录页;评分最终送模型的片段数量保持受控,Agent 搜索同样受控。
#### 待教师复核
"待教师复核"是与材料缺失、解析失败、模型调用失败并列的独立状态:
- 材料缺失、重复、待确认或解析失败 → `unscorable`
- 模型调用或校验失败 → `failed`
- 内容目标补检后仍只有目录或摘要命中 → `needs_review`
进入待复核时模型分为 `null`,总分保持待处理,系统不会自动给低分或断言学生未写。评分项记录缺失的正文证据目标和当时的检索范围,该记录在教师复核后仍然保留。批次状态为"已完成,有待复核项"(`review_required`),仪表盘的待处理数包含待复核项。
教师可对任何状态的评分项复核,包括待复核项;教师评分输入框不预填 0。
#### 跨文档一致性 #### 跨文档一致性
@@ -656,10 +712,12 @@ max_score: 15
description: 评价方案是否具有工程依据、可实施性和风险分析 description: 评价方案是否具有工程依据、可实施性和风险分析
linked_outcomes: [2.3, 2.4, 4.1] linked_outcomes: [2.3, 2.4, 4.1]
required_evidence: required_evidence:
- 主方案 - label: 主方案
- 替代方案 use: content # 只能用正文或附录的实质段落证明
- 方案比较 - label: 替代方案
- 风险分析 use: content
- label: 章节
use: structure # 目录即可证明章节存在
score_levels: score_levels:
- min: 12 - min: 12
max: 15 max: 15
@@ -672,6 +730,8 @@ score_levels:
description: 缺少核心论证 description: 缺少核心论证
``` ```
`required_evidence` 也接受历史上的字符串数组(`required_evidence: [主方案, 替代方案]`)。读取时按措辞判定 `use`:提到标题、封面、目录、章节、参考文献、格式、附录等报告包装字样的目标为 `structure`,其余为 `content`。判定结果与 `POLICY_VERSION` 一起写入批次快照,不修改已发布的规则文本。
#### 版本要求 #### 版本要求
课程、课程目标、毕业要求、模板、评分规则和 Prompt 都需要版本号。评分批次保存完整版本快照。 课程、课程目标、毕业要求、模板、评分规则和 Prompt 都需要版本号。评分批次保存完整版本快照。
@@ -692,6 +752,7 @@ rubric_versions
rubric_criteria rubric_criteria
rubric_score_levels rubric_score_levels
knowledge_documents knowledge_documents
knowledge_pages
knowledge_chunks knowledge_chunks
students students
projects projects
@@ -838,7 +899,11 @@ GET /api/exports/{id}
### 数据与任务 ### 数据与任务
迁移 `0003` 添加会话、消息、运行、待确认操作、索引任务和批次固定项目清单;`0004` 添加首次请求幂等键与有界工具进度记录;`0005` 回填完整资料页文字与相对图像路径,并将历史会话中的引用一次性转换为统一页面链接。检索片段的文字和 1024 维向量保存在 PostgreSQL 的 `knowledge_chunks`;供教师核对的完整页文字单独保存在 `knowledge_pages`,避免以有重叠的检索片段拼接页面。PNG 文件位于共享 `data/processed/`,数据库只存相对于处理目录的路径;学生原件仍只读挂载在 `/data/inbox`。新批次在创建时保存项目 ID 列表;评分、结果查询和导出都使用该列表。旧批次该字段为 `NULL`,界面显示“历史项目清单未知”,只能按课程版本当前项目兼容读取,不能声称还原了创建时的范围。批次状态为 `created`、`queued`、`running`、`partial`、`failed`、`completed`;重复启动已入队或运行中的批次返回冲突。重试保留已评分项和教师复核。 迁移 `0003` 添加会话、消息、运行、待确认操作、索引任务和批次固定项目清单;`0004` 添加首次请求幂等键与有界工具进度记录;`0005` 回填完整资料页文字与相对图像路径,并将历史会话中的引用一次性转换为统一页面链接;`0006` 为页面和检索片段添加页面类型、混合页标记、章节路径,为评分证据添加所属证据目标与历史标记,并新增评分项的缺证据记录。检索片段的文字和 1024 维向量保存在 PostgreSQL 的 `knowledge_chunks`;供教师核对的完整页文字单独保存在 `knowledge_pages`,避免以有重叠的检索片段拼接页面。PNG 文件位于共享 `data/processed/`,数据库只存相对于处理目录的路径;学生原件仍只读挂载在 `/data/inbox`。新批次在创建时保存项目 ID 列表;评分、结果查询和导出都使用该列表。旧批次该字段为 `NULL`,界面显示“历史项目清单未知”,只能按课程版本当前项目兼容读取,不能声称还原了创建时的范围。批次状态为 `created`、`queued`、`running`、`partial`、`failed`、`completed`、`review_required`;`review_required` 表示评分已结束但仍有待教师复核项,与部分完成不同。重复启动已入队或运行中的批次返回冲突。重试保留已评分项和教师复核。
索引版本 `0006` 起为 3。分块方式已改变(目录页不再作为正文文本),因此旧版本的索引不会被就地当作已分类正文使用:文档保留原有 `index_version`,在重建完成前不参与检索,教师可先在上线前查看 `/api/knowledge/stale/{course_version_id}` 的待重建清单。重建任务类型为 `reindex`,只从已保存的解析结果重建,不重新读取或修改学生原件;重复执行会跳过已是最新版本和源文件已变化的文档,并记录到审计日志。
导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。
导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。 导入后的每个待解析文件对应持久 `IndexTask`。worker 复用评分的解析和 RAG 入库代码;同一文件用共享处理目录中的文件锁串行化,避免自动索引与评分竞争写入。任务有 queued/running/completed/failed 状态、错误和重试入口。课程资料入库在会话中也作为索引任务执行。若 Redis 不可用,导入数据仍保存,索引任务标记失败并可重试。原始学生文件只从只读 `/data/inbox` 读取,文件 SHA-256 在预览和确认时复查。
@@ -850,6 +915,8 @@ GET /api/exports/{id}
会话使用模型设置页的 OpenAI 兼容 `/chat/completions` 地址、模型和后端 API Key。适配层按 [DeepSeek 原生工具调用文档](https://api-docs.deepseek.com/guides/tool_calls) 发送 `tools`,在每轮前用不含学生资料的 `wisegrade_ping` 最小探针验证端点会返回 `tool_calls`。当前 `deepseek-flash` 端点支持自动工具选择,但拒绝强制 `tool_choice="required"`,因此探针采用 `auto` 并检查返回结构。模型不支持时会显示明确配置错误,不会伪装工具已执行。既有评分 `json_call` 保持原有 JSON 行为。 会话使用模型设置页的 OpenAI 兼容 `/chat/completions` 地址、模型和后端 API Key。适配层按 [DeepSeek 原生工具调用文档](https://api-docs.deepseek.com/guides/tool_calls) 发送 `tools`,在每轮前用不含学生资料的 `wisegrade_ping` 最小探针验证端点会返回 `tool_calls`。当前 `deepseek-flash` 端点支持自动工具选择,但拒绝强制 `tool_choice="required"`,因此探针采用 `auto` 并检查返回结构。模型不支持时会显示明确配置错误,不会伪装工具已执行。既有评分 `json_call` 保持原有 JSON 行为。
内容检索和结构查询使用与评分相同的证据政策。`search_project_material` 只返回正文和附录片段,`search_project_structure` 返回目录条目与章节结构并明确说明这些信息不能证明章节内容,`get_document_pages` 列出各页的页码、页面类型和章节,`search_document` 在单份资料内检索正文。工具返回页面类型和稳定的引用标识(`citation_id`),每项标注 `source_role` 为 `content` 或 `structure`。评分工具返回的旧引文标记为历史评分引用(`source` 为 `historical_score_evidence`),不会被当作报告正文证据。
工具输入由 Pydantic 校验,后端再校验课程、批次和项目关系。每条教师消息默认最多调用工具 32 次,成功、失败与跳过的尝试均计数;模型最多请求 33 轮,以便用完工具次数后还能生成回答。同一轮中工具名和经校验的参数(含默认分页值)完全相同时,后续调用会标为“已跳过”,模型收到前次调用标识并使用此前结果,不再执行工具;不同查询词、筛选或分页偏移仍会正常执行。可在 `.env` 中通过 `CHAT_MAX_TOOL_CALLS` 调整工具上限(1–64),修改后重新创建 API 容器。`get_scores` 每次最多返回一个项目的六项评分,响应给出 `next_criterion_offset` 和 `next_project_offset` 供继续查询;长结果保留结构化评分数据,不再变成无内容的截断提示。检索工具的运行记录保留每次调用找到的有界原文片段;回答引用对各次调用的证据去重后保存,不再只取前 20 条。工具没有任意 SQL、任意文件路径、HTTP 请求、代码执行或确认能力。检索片段只作为数据,不作为工具指令。模型只收到当前问题所需的规则摘要、检索片段或紧凑统计;API Key 不写入日志或会话表。 工具输入由 Pydantic 校验,后端再校验课程、批次和项目关系。每条教师消息默认最多调用工具 32 次,成功、失败与跳过的尝试均计数;模型最多请求 33 轮,以便用完工具次数后还能生成回答。同一轮中工具名和经校验的参数(含默认分页值)完全相同时,后续调用会标为“已跳过”,模型收到前次调用标识并使用此前结果,不再执行工具;不同查询词、筛选或分页偏移仍会正常执行。可在 `.env` 中通过 `CHAT_MAX_TOOL_CALLS` 调整工具上限(1–64),修改后重新创建 API 容器。`get_scores` 每次最多返回一个项目的六项评分,响应给出 `next_criterion_offset` 和 `next_project_offset` 供继续查询;长结果保留结构化评分数据,不再变成无内容的截断提示。检索工具的运行记录保留每次调用找到的有界原文片段;回答引用对各次调用的证据去重后保存,不再只取前 20 条。工具没有任意 SQL、任意文件路径、HTTP 请求、代码执行或确认能力。检索片段只作为数据,不作为工具指令。模型只收到当前问题所需的规则摘要、检索片段或紧凑统计;API Key 不写入日志或会话表。
### API 与故障恢复 ### API 与故障恢复
+18 -4
View File
@@ -26,6 +26,20 @@ const options = (rows: any[], label: (row: any) => string) => rows.map(row => ({
const isSlideEvidence = (evidence: any) => /\.pptx$/i.test(evidence.document || ''); const isSlideEvidence = (evidence: any) => /\.pptx$/i.test(evidence.document || '');
const evidencePlace = (evidence: any) => `第 ${evidence.page} ${isSlideEvidence(evidence) ? '张幻灯片' : '页'}`; const evidencePlace = (evidence: any) => `第 ${evidence.page} ${isSlideEvidence(evidence) ? '张幻灯片' : '页'}`;
const evidencePreviewLabel = (evidence: any) => isSlideEvidence(evidence) ? '查看该张幻灯片的完整图像' : '查看该页的完整图像'; const evidencePreviewLabel = (evidence: any) => isSlideEvidence(evidence) ? '查看该张幻灯片的完整图像' : '查看该页的完整图像';
const pageTypeLabels: Record<string, string> = { cover: '封面', abstract: '摘要', toc: '目录', body: '正文', appendix: '附录', references: '参考文献', unknown: '识别不确定', mixed: '目录与正文混合页' };
const pageTypeLabel = (value: string) => pageTypeLabels[value] || '未分类页面';
const statusLabels: Record<string, string> = { pending: '待评分', scored: '模型已评分', reviewed: '教师已复核', needs_review: '待教师复核', unscorable: '材料缺失或解析失败', failed: '模型调用失败' };
const statusLabel = (value: string) => statusLabels[value] || value;
const statusColor = (value: string) => ({ scored: 'green', reviewed: 'green', needs_review: 'gold', unscorable: 'red', failed: 'red' }[value] || 'blue');
const batchStatusLabels: Record<string, string> = { created: '待启动', queued: '排队中', running: '运行中', completed: '已完成', review_required: '已完成,有待复核项', partial: '部分完成', failed: '失败' };
const batchStatusLabel = (value: string) => batchStatusLabels[value] || value;
// A teacher score box is never prefilled with 0: an empty box forces a deliberate
// entry, while 0 would read as a judgement the teacher did not make.
const reviewFormValues = (row: any) => ({
score: row.teacher_score ?? row.final_score ?? null,
comment: row.teacher_comment || row.model_comment || '',
reason: '',
});
function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: any; onChange: (value: any) => void; disabled?: boolean; rows?: number; resetKey: string }) { function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: any; onChange: (value: any) => void; disabled?: boolean; rows?: number; resetKey: string }) {
const [text, setText] = useState(JSON.stringify(value, null, 2)); const [text, setText] = useState(JSON.stringify(value, null, 2));
useEffect(() => { setText(JSON.stringify(value, null, 2)); }, [resetKey]); useEffect(() => { setText(JSON.stringify(value, null, 2)); }, [resetKey]);
@@ -35,7 +49,7 @@ function JsonField({ value, onChange, disabled, rows = 4, resetKey }: { value: a
function Dashboard() { function Dashboard() {
const [data, setData] = useState<any>({}); const [data, setData] = useState<any>({});
useEffect(() => { load('/dashboard', setData); }, []); useEffect(() => { load('/dashboard', setData); }, []);
return <><Title level={2}>工作概览</Title><Paragraph>先确认课程版本与规则,再扫描挂载目录,创建评分批次并逐项复核。</Paragraph><Row gutter={18}>{[['课程', data.courses], ['学生项目', data.projects], ['评分批次', data.batches], ['待处理评分项', data.needs_attention]].map(([label, count]) => <Col xs={24} sm={12} lg={6} key={label}><Card><Statistic title={label} value={count ?? 0} /></Card></Col>)}</Row><Alert className="top-gap" type="info" showIcon message="学生材料仅从只读挂载目录读取;缺失和解析失败的材料会在项目与评分详情中标出。" /></>; return <><Title level={2}>工作概览</Title><Paragraph>先确认课程版本与规则,再扫描挂载目录,创建评分批次并逐项复核。</Paragraph><Row gutter={18}>{[['课程', data.courses], ['学生项目', data.projects], ['评分批次', data.batches], ['待处理评分项', data.needs_attention], ['待教师复核', data.needs_review]].map(([label, count]) => <Col xs={24} sm={12} lg={4} key={label}><Card><Statistic title={label} value={count ?? 0} /></Card></Col>)}</Row><Alert className="top-gap" type="info" showIcon message="学生材料仅从只读挂载目录读取;缺失和解析失败的材料会在项目与评分详情中标出。“待教师复核”表示该项只检索到目录、摘要或章节标题,没有可用于内容判断的正文证据,系统不会自动给低分。" /></>;
} }
function Courses() { function Courses() {
@@ -107,7 +121,7 @@ function Batches({ openBatch }: { openBatch: (id: number) => void }) {
useEffect(() => { refresh(); load('/courses', setCourses); const id = window.setInterval(refresh, 5000); return () => clearInterval(id); }, []); useEffect(() => { refresh(); load('/courses', setCourses); const id = window.setInterval(refresh, 5000); return () => clearInterval(id); }, []);
useEffect(() => { if (courseId) load(`/courses/${courseId}/versions`, setVersions); }, [courseId]); useEffect(() => { if (courseId) load(`/courses/${courseId}/versions`, setVersions); }, [courseId]);
useEffect(() => { if (versionId) load(`/course-versions/${versionId}/rubrics`, setRubrics); }, [versionId]); useEffect(() => { if (versionId) load(`/course-versions/${versionId}/rubrics`, setRubrics); }, [versionId]);
return <><Title level={2}>评分批次与进度</Title><Card title="创建独立批次"><Space wrap><Select style={{ width: 250 }} placeholder="课程" options={options(courses, c => c.name)} onChange={setCourseId} /><Select style={{ width: 250 }} placeholder="已发布课程版本" options={options(versions.filter(v => v.published), v => `v${v.version} ${v.title}`)} onChange={setVersionId} /><Select style={{ width: 230 }} placeholder="已发布规则版本" options={options(rubrics.filter(r => r.published), r => `v${r.version} · ${r.criteria.length}项`)} onChange={setRubricId} /><Button type="primary" disabled={!versionId || !rubricId} onClick={async () => { try { await post('/scoring-batches', { course_version_id: versionId, rubric_version_id: rubricId }); refresh(); message.success('批次已创建'); } catch (e) { fail(e); } }}>创建批次</Button></Space></Card><Table className="top-gap" rowKey="id" rowClassName={(b: any) => Number(new URLSearchParams(window.location.search).get("batch_id")) === b.id ? "linked-row" : ""} dataSource={batches} columns={[{ title: '批次', dataIndex: 'id' }, { title: '课程/规则版本', render: (_, b) => `${b.course_version_id} / ${b.rubric_version_id}` }, { title: '状态', render: (_, b) => <Tag color={b.status === 'completed' ? 'green' : b.status === 'partial' || b.status === 'failed' ? 'orange' : 'blue'}>{b.status}</Tag> }, { title: '进度', render: (_, b) => <Progress percent={b.progress_total ? Math.round(b.progress_done / b.progress_total * 100) : 0} size="small" /> }, { title: '异常', dataIndex: 'error' }, { title: '操作', render: (_, b) => <Space><Button disabled={!['created', 'failed', 'partial'].includes(b.status)} onClick={async () => { try { await post(`/scoring-batches/${b.id}/start`); refresh(); } catch (e) { fail(e); } }}>启动/重试</Button><Button type="link" onClick={() => openBatch(b.id)}>查看评分</Button></Space> }]} /></>; return <><Title level={2}>评分批次与进度</Title><Card title="创建独立批次"><Space wrap><Select style={{ width: 250 }} placeholder="课程" options={options(courses, c => c.name)} onChange={setCourseId} /><Select style={{ width: 250 }} placeholder="已发布课程版本" options={options(versions.filter(v => v.published), v => `v${v.version} ${v.title}`)} onChange={setVersionId} /><Select style={{ width: 230 }} placeholder="已发布规则版本" options={options(rubrics.filter(r => r.published), r => `v${r.version} · ${r.criteria.length}项`)} onChange={setRubricId} /><Button type="primary" disabled={!versionId || !rubricId} onClick={async () => { try { await post('/scoring-batches', { course_version_id: versionId, rubric_version_id: rubricId }); refresh(); message.success('批次已创建'); } catch (e) { fail(e); } }}>创建批次</Button></Space></Card><Table className="top-gap" rowKey="id" rowClassName={(b: any) => Number(new URLSearchParams(window.location.search).get("batch_id")) === b.id ? "linked-row" : ""} dataSource={batches} columns={[{ title: '批次', dataIndex: 'id' }, { title: '课程/规则版本', render: (_, b) => `${b.course_version_id} / ${b.rubric_version_id}` }, { title: '状态', render: (_, b) => <Space wrap><Tag color={b.status === 'completed' ? 'green' : b.status === 'review_required' ? 'gold' : b.status === 'partial' || b.status === 'failed' ? 'orange' : 'blue'}>{batchStatusLabel(b.status)}</Tag>{b.needs_review > 0 && <Tag color="gold">待复核 {b.needs_review}</Tag>}</Space> }, { title: '进度', render: (_, b) => <Progress percent={b.progress_total ? Math.round(b.progress_done / b.progress_total * 100) : 0} size="small" /> }, { title: '异常', dataIndex: 'error' }, { title: '操作', render: (_, b) => <Space><Button disabled={!['created', 'failed', 'partial', 'review_required'].includes(b.status)} onClick={async () => { try { await post(`/scoring-batches/${b.id}/start`); refresh(); } catch (e) { fail(e); } }}>启动/重试</Button><Button type="link" onClick={() => openBatch(b.id)}>查看评分</Button></Space> }]} /></>;
} }
function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }) { function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }) {
@@ -116,8 +130,8 @@ function Scores({ batchId, projectId }: { batchId?: number; projectId?: number }
useEffect(() => { load('/scoring-batches', setBatches); load('/projects', setProjects); }, []); useEffect(() => { load('/scoring-batches', setBatches); load('/projects', setProjects); }, []);
useEffect(() => { if (batch && project) load(`/projects/${project}/scores?batch_id=${batch}`, setResult); }, [batch, project]); useEffect(() => { if (batch && project) load(`/projects/${project}/scores?batch_id=${batch}`, setResult); }, [batch, project]);
const refresh = () => batch && project && load(`/projects/${project}/scores?batch_id=${batch}`, setResult); const refresh = () => batch && project && load(`/projects/${project}/scores?batch_id=${batch}`, setResult);
return <><Title level={2}>逐项评分与教师复核</Title><Space wrap><Select style={{ width: 180 }} placeholder="评分批次" value={batch} options={options(batches, b => `批次 #${b.id} · ${b.status}`)} onChange={setBatch} /><Select style={{ width: 330 }} placeholder="学生项目" value={project} options={options(projects, p => `${p.student_no} ${p.name} · ${p.topic}`)} onChange={setProject} /></Space>{result && <><Card className="top-gap" title={`${result.student_no} ${result.name} · ${result.topic}`}><Space wrap>{Object.entries(result.totals).map(([key, value]) => <Tag key={key} color={value === null ? 'orange' : 'blue'}>{key}: {String(value ?? '待处理')}</Tag>)}</Space>{result.facts?.consistency_issues?.length > 0 && <Alert className="top-gap" type="warning" message="跨文档一致性提醒" description={result.facts.consistency_issues.map((issue: any) => issue.description).join(';')} />}</Card><Table className="top-gap" rowKey={(row: any) => row.criterion.criterion_id} dataSource={result.rows} pagination={false} columns={[{ title: '评分项', render: (_, r) => <><Tag>{categories[r.criterion.category] || r.criterion.category}</Tag>{r.criterion.name}</> }, { title: '模型原始分', render: (_, r) => `${r.model_score ?? '—'} / ${r.criterion.max_score}` }, { title: '教师分', render: (_, r) => r.teacher_score ?? '—' }, { title: '最终分', render: (_, r) => r.final_score ?? '待处理' }, { title: '状态', render: (_, r) => <Tag color={r.status === 'scored' || r.status === 'reviewed' ? 'green' : 'orange'}>{r.status}</Tag> }, { title: '操作', render: (_, r) => <Button onClick={() => { setReview(r); form.setFieldsValue({ score: r.final_score ?? 0, comment: r.teacher_comment || r.model_comment || '', reason: '' }); }}>查看证据/复核</Button> }]} /></>} return <><Title level={2}>逐项评分与教师复核</Title><Space wrap><Select style={{ width: 180 }} placeholder="评分批次" value={batch} options={options(batches, b => `批次 #${b.id} · ${b.status}`)} onChange={setBatch} /><Select style={{ width: 330 }} placeholder="学生项目" value={project} options={options(projects, p => `${p.student_no} ${p.name} · ${p.topic}`)} onChange={setProject} /></Space>{result && <><Card className="top-gap" title={`${result.student_no} ${result.name} · ${result.topic}`}><Space wrap>{Object.entries(result.totals).map(([key, value]) => <Tag key={key} color={value === null ? 'orange' : 'blue'}>{key}: {String(value ?? '待处理')}</Tag>)}</Space>{result.facts?.consistency_issues?.length > 0 && <Alert className="top-gap" type="warning" message="跨文档一致性提醒" description={result.facts.consistency_issues.map((issue: any) => issue.description).join(';')} />}</Card><Table className="top-gap" rowKey={(row: any) => row.criterion.criterion_id} dataSource={result.rows} pagination={false} columns={[{ title: '评分项', render: (_, r) => <><Tag>{categories[r.criterion.category] || r.criterion.category}</Tag>{r.criterion.name}</> }, { title: '模型原始分', render: (_, r) => `${r.model_score ?? '—'} / ${r.criterion.max_score}` }, { title: '教师分', render: (_, r) => r.teacher_score ?? '—' }, { title: '最终分', render: (_, r) => r.final_score ?? '待处理' }, { title: '状态', render: (_, r) => <Tag color={statusColor(r.status)}>{statusLabel(r.status)}</Tag> }, { title: '操作', render: (_, r) => <Button onClick={() => { setReview(r); form.setFieldsValue(reviewFormValues(r)); }}>查看证据/复核</Button> }]} /></>}
<Modal open={!!review} title={review?.criterion?.name} width={850} onCancel={() => setReview(undefined)} onOk={async () => { try { const values = await form.validateFields(); await put(`/criterion-scores/${review.score_id}/review`, values); setReview(undefined); refresh(); message.success('复核记录已保存'); } catch (e) { fail(e); } }} okText="保存教师复核" okButtonProps={{ disabled: !review?.score_id }}>{review && <><Descriptions column={1} items={[{ key: 'model', label: '模型分数/评语', children: `${review.model_score ?? '—'} · ${review.model_comment || ''}` }, { key: 'confidence', label: '模型置信度', children: review.confidence ?? '—' }, { key: 'missing', label: '缺失证据', children: review.missing_evidence?.join(';') || '无' }, { key: 'suggestion', label: '改进建议', children: review.suggestion || '无' }, { key: 'error', label: '异常', children: review.error || '无' }]} /><Divider />{review.evidence?.map((ev: any, index: number) => <Card size="small" key={index} className="evidence"><Text strong>{ev.document} · {evidencePlace(ev)}</Text><Paragraph>{ev.quote}</Paragraph><Space wrap>{ev.url && <a href={ev.url} target="_blank" rel="noreferrer">查看页面文字</a>}{ev.image_url && <a href={ev.image_url} target="_blank" rel="noreferrer">{evidencePreviewLabel(ev)}(核对引文)</a>}</Space></Card>)}<Divider /><Form form={form} layout="vertical"><Form.Item label="教师评分" name="score" rules={[{ required: true }]}><InputNumber min={0} max={review.criterion.max_score} /></Form.Item><Form.Item label="教师评语" name="comment" rules={[{ required: true }]}><Input.TextArea rows={3} /></Form.Item><Form.Item label="修改说明" name="reason"><Input /></Form.Item></Form>{review.score_id && <Button onClick={async () => { try { setRevisions(await api(`/criterion-scores/${review.score_id}/revisions`)); } catch (e) { fail(e); } }}>查看修改记录</Button>}{revisions.length > 0 && <pre className="json-view">{JSON.stringify(revisions, null, 2)}</pre>}</>}</Modal> <Modal open={!!review} title={review?.criterion?.name} width={850} onCancel={() => setReview(undefined)} onOk={async () => { try { const values = await form.validateFields(); await put(`/criterion-scores/${review.score_id}/review`, values); setReview(undefined); refresh(); message.success('复核记录已保存'); } catch (e) { fail(e); } }} okText="保存教师复核" okButtonProps={{ disabled: !review?.score_id }}>{review && <>{review.status === 'needs_review' && <Alert type="warning" showIcon className="top-gap" message="本项缺少正文证据,需要教师复核" description={`未找到正文证据的目标:${(review.evidence_gaps || []).map((gap: any) => gap.target_label).join('、') || review.missing_evidence?.join('、') || '见缺失证据'}。系统未给出模型分,也没有按低分处理;请核对材料后直接填写教师评分。`} />}<Descriptions column={1} items={[{ key: 'model', label: '模型分数/评语', children: `${review.model_score ?? '未给出'} · ${review.model_comment || ''}` }, { key: 'confidence', label: '模型置信度', children: review.confidence ?? '—' }, { key: 'missing', label: '缺失证据', children: review.missing_evidence?.join(';') || '无' }, { key: 'gaps', label: '缺正文证据的目标', children: (review.evidence_gaps || []).map((gap: any) => `${gap.target_label}(检索范围:${(gap.searched_page_types || []).map(pageTypeLabel).join('、') || '正文'})`).join(';') || '无' }, { key: 'suggestion', label: '改进建议', children: review.suggestion || '无' }, { key: 'error', label: '异常', children: review.error || '无' }]} /><Divider />{review.evidence?.map((ev: any, index: number) => <Card size="small" key={index} className="evidence"><Space wrap><Text strong>{ev.document} · {evidencePlace(ev)}</Text><Tag color={ev.page_type === 'body' || ev.page_type === 'appendix' ? 'green' : 'orange'}>{pageTypeLabel(ev.page_type)}</Tag>{ev.historical && <Tag>历史评分引用</Tag>}{ev.target_label && <Tag>{ev.target_label}</Tag>}</Space><Paragraph>{ev.quote}</Paragraph><Space wrap>{ev.url && <a href={ev.url} target="_blank" rel="noreferrer">查看页面文字</a>}{ev.image_url && <a href={ev.image_url} target="_blank" rel="noreferrer">{evidencePreviewLabel(ev)}(核对引文)</a>}</Space></Card>)}<Divider /><Form form={form} layout="vertical"><Form.Item label="教师评分" name="score" rules={[{ required: true, message: '请填写教师评分' }]}><InputNumber min={0} max={review.criterion.max_score} placeholder="请填写" /></Form.Item><Form.Item label="教师评语" name="comment" rules={[{ required: true }]}><Input.TextArea rows={3} /></Form.Item><Form.Item label="修改说明" name="reason"><Input /></Form.Item></Form>{review.score_id && <Button onClick={async () => { try { setRevisions(await api(`/criterion-scores/${review.score_id}/revisions`)); } catch (e) { fail(e); } }}>查看修改记录</Button>}{revisions.length > 0 && <pre className="json-view">{JSON.stringify(revisions, null, 2)}</pre>}</>}</Modal>
</>; </>;
} }