Expose a runnable backend without giving the ingress layer secrets
Some checks failed
verify / verify (push) Has been cancelled
Some checks failed
verify / verify (push) Has been cancelled
The backend can now be inspected through a loopback-only gateway while the database-aware API remains on the internal data network. A governed synthetic demo proves readiness, pgvector retrieval, reranking, and citation output through real HTTP without invoking cloud models. Constraint: The previously exposed Bailian key is compromised and cannot be used for live validation Constraint: The API must be locally reachable while retaining no internet egress Rejected: Attach the API directly to the ingress network | a real socket test proved that configuration still had egress Rejected: Publish a port from the internal-only network | Docker Desktop did not expose the host port Confidence: high Scope-risk: moderate Reversibility: clean Directive: Keep model and database credentials out of the gateway; do not relax the fixed demo identity/profile filters Tested: make verify; 63 pytest tests; strict mypy; Ruff; Secret scan; Compose config; three backend image builds; API/DB/gateway healthy; migration exit 0; Swagger browser check; live/ready/meta/status/search HTTP; 20/20/20 index; API egress ENETUNREACH; empty gateway mounts and business environment Not-tested: Live Bailian calls require a newly rotated key; full generated-answer flow and React UI are not implemented
This commit is contained in:
@@ -52,9 +52,11 @@
|
||||
make setup-hooks
|
||||
make backend-sync
|
||||
bash scripts/init-local-secrets.sh
|
||||
docker compose up -d --build db migrate
|
||||
docker compose up -d --build gateway
|
||||
docker compose --profile tools run --rm seed-demo-offline
|
||||
curl http://127.0.0.1:8000/health/ready
|
||||
curl http://127.0.0.1:8000/api/v1/demo/status
|
||||
make verify
|
||||
```
|
||||
|
||||
这组命令不需要百炼 Key，只使用 20 条虚构数据验证 pgvector 写入、检索、重排和幂等。真实模型运行必须先轮换聊天中已暴露的旧 Key，再按 [Stage 1 运行手册](docs/05-stage1-runbook.md) 操作。
|
||||
这组命令不需要百炼 Key，只使用 20 条虚构数据验证 pgvector 写入、检索、重排和幂等。`gateway` 会依次启动数据库、迁移和内部 API；启动后可在 <http://127.0.0.1:8000/docs> 查看 FastAPI Swagger，并通过 `POST /api/v1/demo/search` 体验只读离线检索。真实模型运行必须先轮换聊天中已暴露的旧 Key，再按 [Stage 1 运行手册](docs/05-stage1-runbook.md) 操作。
|
||||
|
||||
@@ -24,10 +24,9 @@ COPY migrations ./migrations
|
||||
COPY app ./app
|
||||
COPY README.md ./
|
||||
|
||||
RUN uv sync --frozen --no-dev \
|
||||
&& chown -R app:app /app
|
||||
RUN uv sync --frozen --no-dev
|
||||
|
||||
USER app
|
||||
USER 10001:10001
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["python", "-m", "app.main"]
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
||||
1
backend/app/api/__init__.py
Normal file
1
backend/app/api/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""HTTP API routers for the geological RAG backend."""
|
||||
5
backend/app/api/v1/__init__.py
Normal file
5
backend/app/api/v1/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
||||
"""Version 1 HTTP API routers."""
|
||||
|
||||
from app.api.v1.demo import router as demo_router
|
||||
|
||||
__all__ = ["demo_router"]
|
||||
356
backend/app/api/v1/demo.py
Normal file
356
backend/app/api/v1/demo.py
Normal file
@@ -0,0 +1,356 @@
|
||||
"""Read-only offline RAG demo endpoints backed by the synthetic dataset."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Annotated, Any, Literal, Protocol, cast
|
||||
|
||||
import psycopg
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from pgvector.psycopg import register_vector
|
||||
from pgvector.vector import Vector
|
||||
from psycopg.rows import dict_row
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from app.adapters.fake import FakeEmbeddingProvider, FakeReranker
|
||||
from app.core.config import Settings, get_settings
|
||||
from app.core.demo_identity import (
|
||||
ACCESS_SCOPE_ID,
|
||||
DEMO_EXPECTED_CHUNKS,
|
||||
DEMO_FAKE_EMBEDDING_MODEL,
|
||||
DEMO_SCOPE_NAME,
|
||||
KNOWLEDGE_BASE_ID,
|
||||
offline_embedding_profile_hash,
|
||||
)
|
||||
from app.core.secrets import SecretFileError
|
||||
|
||||
QUERY_MAX_LENGTH = 500
|
||||
SNIPPET_MAX_LENGTH = 600
|
||||
TITLE_MAX_LENGTH = 120
|
||||
_SPACE_PATTERN = re.compile(r"\s+")
|
||||
|
||||
|
||||
class DemoCounts(BaseModel):
|
||||
"""Public aggregate counts for the synthetic dataset."""
|
||||
|
||||
chunks: int = Field(ge=0)
|
||||
vectors: int = Field(ge=0)
|
||||
searchable: int = Field(ge=0)
|
||||
|
||||
|
||||
class DemoStatusResponse(BaseModel):
|
||||
"""Safe readiness summary without database or approval internals."""
|
||||
|
||||
status: Literal["ready", "empty_dataset", "incomplete_dataset"]
|
||||
dataset: Literal["synthetic-demo"] = "synthetic-demo"
|
||||
counts: DemoCounts
|
||||
|
||||
|
||||
class DemoSearchRequest(BaseModel):
|
||||
"""Bounded request accepted by the offline demo search."""
|
||||
|
||||
query: str = Field(min_length=1, max_length=QUERY_MAX_LENGTH)
|
||||
top_k: int = Field(default=5, ge=1, le=10)
|
||||
|
||||
@field_validator("query")
|
||||
@classmethod
|
||||
def normalize_query(cls, value: str) -> str:
|
||||
normalized = _SPACE_PATTERN.sub(" ", value).strip()
|
||||
if not normalized:
|
||||
raise ValueError("query must contain non-whitespace text")
|
||||
return normalized
|
||||
|
||||
|
||||
class DemoSearchItem(BaseModel):
|
||||
"""Public synthetic result; internal identifiers and hashes are excluded."""
|
||||
|
||||
title: str
|
||||
snippet: str
|
||||
page_label: str
|
||||
score: float = Field(ge=0.0, le=1.0)
|
||||
citation_id: str
|
||||
|
||||
|
||||
class DemoSearchResponse(BaseModel):
|
||||
"""Offline retrieval response with an explicit empty-dataset state."""
|
||||
|
||||
status: Literal["ok", "empty_dataset"]
|
||||
dataset: Literal["synthetic-demo"] = "synthetic-demo"
|
||||
results: list[DemoSearchItem]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DemoCandidate:
|
||||
"""Private retrieval projection used only while constructing safe results."""
|
||||
|
||||
source_key: str
|
||||
title: str
|
||||
text: str
|
||||
page_start: int | None
|
||||
page_end: int | None
|
||||
|
||||
@property
|
||||
def rerank_text(self) -> str:
|
||||
return f"{self.title}\n{self.text}"
|
||||
|
||||
|
||||
class DemoRepository(Protocol):
|
||||
"""Read-only persistence boundary used by the demo router."""
|
||||
|
||||
def counts(self) -> DemoCounts: ...
|
||||
|
||||
def search(self, query_vector: tuple[float, ...], *, limit: int) -> list[DemoCandidate]: ...
|
||||
|
||||
|
||||
class PostgresDemoRepository:
|
||||
"""Read-only PostgreSQL/pgvector projection for approved synthetic chunks."""
|
||||
|
||||
def __init__(self, settings: Settings) -> None:
|
||||
self._settings = settings
|
||||
|
||||
def _dsn(self) -> str:
|
||||
return (
|
||||
self._settings.database_url()
|
||||
.set(drivername="postgresql")
|
||||
.render_as_string(hide_password=False)
|
||||
)
|
||||
|
||||
def counts(self) -> DemoCounts:
|
||||
profile_hash = offline_embedding_profile_hash(self._settings.embedding_dimension)
|
||||
with psycopg.connect(
|
||||
self._dsn(),
|
||||
connect_timeout=2,
|
||||
row_factory=dict_row,
|
||||
) as connection:
|
||||
row = connection.execute(
|
||||
"""
|
||||
SELECT
|
||||
count(*)::integer AS chunks,
|
||||
count(*) FILTER (WHERE chunk.embedding IS NOT NULL)::integer AS vectors,
|
||||
count(*) FILTER (WHERE chunk.searchable)::integer AS searchable
|
||||
FROM rag.chunks AS chunk
|
||||
JOIN rag.access_scopes AS scope
|
||||
ON scope.id = chunk.access_scope_id
|
||||
AND scope.knowledge_base_id = chunk.knowledge_base_id
|
||||
JOIN rag.documents AS document
|
||||
ON document.id = chunk.document_id
|
||||
AND document.knowledge_base_id = chunk.knowledge_base_id
|
||||
AND document.access_scope_id = chunk.access_scope_id
|
||||
JOIN rag.document_versions AS version
|
||||
ON version.id = chunk.document_version_id
|
||||
AND version.document_id = chunk.document_id
|
||||
WHERE chunk.knowledge_base_id = %s
|
||||
AND chunk.access_scope_id = %s
|
||||
AND scope.name = %s
|
||||
AND chunk.metadata ->> 'source_type' = 'synthetic'
|
||||
AND chunk.index_status = 'READY'
|
||||
AND chunk.approval_status = 'CLOUD_APPROVED'
|
||||
AND chunk.deleted_at IS NULL
|
||||
AND chunk.embedding_model = %s
|
||||
AND chunk.embedding_profile_hash = %s
|
||||
AND document.status = 'READY'
|
||||
AND document.deleted_at IS NULL
|
||||
AND document.active_version_id = chunk.document_version_id
|
||||
AND version.status = 'READY'
|
||||
AND version.review_state = 'CLOUD_APPROVED'
|
||||
AND version.outbound_manifest_sha256 = chunk.outbound_manifest_sha256
|
||||
AND version.embedding_profile_hash = chunk.embedding_profile_hash
|
||||
""",
|
||||
(
|
||||
KNOWLEDGE_BASE_ID,
|
||||
ACCESS_SCOPE_ID,
|
||||
DEMO_SCOPE_NAME,
|
||||
DEMO_FAKE_EMBEDDING_MODEL,
|
||||
profile_hash,
|
||||
),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return DemoCounts(chunks=0, vectors=0, searchable=0)
|
||||
return DemoCounts(
|
||||
chunks=int(row["chunks"]),
|
||||
vectors=int(row["vectors"]),
|
||||
searchable=int(row["searchable"]),
|
||||
)
|
||||
|
||||
def search(self, query_vector: tuple[float, ...], *, limit: int) -> list[DemoCandidate]:
|
||||
if len(query_vector) != self._settings.embedding_dimension:
|
||||
raise ValueError("query vector dimension does not match the demo index")
|
||||
|
||||
vector = Vector(list(query_vector))
|
||||
profile_hash = offline_embedding_profile_hash(self._settings.embedding_dimension)
|
||||
with psycopg.connect(
|
||||
self._dsn(),
|
||||
connect_timeout=2,
|
||||
row_factory=dict_row,
|
||||
) as connection:
|
||||
register_vector(connection)
|
||||
connection.execute("SET LOCAL statement_timeout = '3000ms'")
|
||||
connection.execute("SET LOCAL hnsw.iterative_scan = strict_order")
|
||||
connection.execute("SET LOCAL hnsw.ef_search = 100")
|
||||
rows = connection.execute(
|
||||
"""
|
||||
SELECT
|
||||
chunk.id::text AS source_key,
|
||||
COALESCE(NULLIF(chunk.section_path ->> 0, ''), '合成地质资料') AS title,
|
||||
chunk.cloud_text AS text,
|
||||
chunk.page_start,
|
||||
chunk.page_end
|
||||
FROM rag.chunks AS chunk
|
||||
JOIN rag.access_scopes AS scope
|
||||
ON scope.id = chunk.access_scope_id
|
||||
AND scope.knowledge_base_id = chunk.knowledge_base_id
|
||||
JOIN rag.documents AS document
|
||||
ON document.id = chunk.document_id
|
||||
AND document.knowledge_base_id = chunk.knowledge_base_id
|
||||
AND document.access_scope_id = chunk.access_scope_id
|
||||
JOIN rag.document_versions AS version
|
||||
ON version.id = chunk.document_version_id
|
||||
AND version.document_id = chunk.document_id
|
||||
WHERE chunk.knowledge_base_id = %s
|
||||
AND chunk.access_scope_id = %s
|
||||
AND scope.name = %s
|
||||
AND chunk.metadata ->> 'source_type' = 'synthetic'
|
||||
AND chunk.searchable IS TRUE
|
||||
AND chunk.embedding IS NOT NULL
|
||||
AND chunk.index_status = 'READY'
|
||||
AND chunk.approval_status = 'CLOUD_APPROVED'
|
||||
AND chunk.deleted_at IS NULL
|
||||
AND chunk.embedding_model = %s
|
||||
AND chunk.embedding_profile_hash = %s
|
||||
AND document.status = 'READY'
|
||||
AND document.deleted_at IS NULL
|
||||
AND document.active_version_id = chunk.document_version_id
|
||||
AND version.status = 'READY'
|
||||
AND version.review_state = 'CLOUD_APPROVED'
|
||||
AND version.outbound_manifest_sha256 = chunk.outbound_manifest_sha256
|
||||
AND version.embedding_profile_hash = chunk.embedding_profile_hash
|
||||
ORDER BY chunk.embedding <=> %s
|
||||
LIMIT %s
|
||||
""",
|
||||
(
|
||||
KNOWLEDGE_BASE_ID,
|
||||
ACCESS_SCOPE_ID,
|
||||
DEMO_SCOPE_NAME,
|
||||
DEMO_FAKE_EMBEDDING_MODEL,
|
||||
profile_hash,
|
||||
vector,
|
||||
limit,
|
||||
),
|
||||
).fetchall()
|
||||
|
||||
return [
|
||||
DemoCandidate(
|
||||
source_key=cast(str, row["source_key"]),
|
||||
title=cast(str, row["title"]),
|
||||
text=cast(str, row["text"]),
|
||||
page_start=cast(int | None, row["page_start"]),
|
||||
page_end=cast(int | None, row["page_end"]),
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
|
||||
def get_demo_repository(
|
||||
settings: Annotated[Settings, Depends(get_settings)],
|
||||
) -> DemoRepository:
|
||||
"""Build the default repository without loading any model credential."""
|
||||
|
||||
return PostgresDemoRepository(settings)
|
||||
|
||||
|
||||
def _bounded_text(value: str, max_length: int) -> str:
|
||||
normalized = _SPACE_PATTERN.sub(" ", value).strip()
|
||||
if len(normalized) <= max_length:
|
||||
return normalized
|
||||
return f"{normalized[: max_length - 1]}…"
|
||||
|
||||
|
||||
def make_citation_id(source_key: str) -> str:
|
||||
"""Derive a stable opaque citation without returning an internal UUID."""
|
||||
|
||||
digest = hashlib.sha256(f"{DEMO_SCOPE_NAME}:{source_key}".encode()).hexdigest()
|
||||
return f"demo-{digest[:16]}"
|
||||
|
||||
|
||||
def make_page_label(page_start: int | None, page_end: int | None) -> str:
|
||||
if page_start is None or page_end is None:
|
||||
return "页码未知"
|
||||
if page_start == page_end:
|
||||
return f"第 {page_start} 页"
|
||||
return f"第 {page_start}-{page_end} 页"
|
||||
|
||||
|
||||
def _safe_result(candidate: DemoCandidate, score: float) -> DemoSearchItem:
|
||||
return DemoSearchItem(
|
||||
title=f"合成资料｜{_bounded_text(candidate.title, TITLE_MAX_LENGTH)}",
|
||||
snippet=_bounded_text(candidate.text, SNIPPET_MAX_LENGTH),
|
||||
page_label=make_page_label(candidate.page_start, candidate.page_end),
|
||||
score=round(max(0.0, min(1.0, score)), 6),
|
||||
citation_id=make_citation_id(candidate.source_key),
|
||||
)
|
||||
|
||||
|
||||
def _database_unavailable(exc: BaseException) -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="database unavailable",
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/api/v1/demo", tags=["offline-demo"])
|
||||
|
||||
|
||||
@router.get("/status", response_model=DemoStatusResponse)
|
||||
def demo_status(repository: Annotated[DemoRepository, Depends(get_demo_repository)]) -> Any:
|
||||
try:
|
||||
counts = repository.counts()
|
||||
except (OSError, SecretFileError, psycopg.Error) as exc:
|
||||
raise _database_unavailable(exc) from exc
|
||||
state: Literal["ready", "empty_dataset", "incomplete_dataset"]
|
||||
if counts.chunks == 0:
|
||||
state = "empty_dataset"
|
||||
elif (
|
||||
counts.chunks == DEMO_EXPECTED_CHUNKS
|
||||
and counts.vectors == DEMO_EXPECTED_CHUNKS
|
||||
and counts.searchable == DEMO_EXPECTED_CHUNKS
|
||||
):
|
||||
state = "ready"
|
||||
else:
|
||||
state = "incomplete_dataset"
|
||||
return DemoStatusResponse(status=state, counts=counts)
|
||||
|
||||
|
||||
@router.post("/search", response_model=DemoSearchResponse)
|
||||
async def demo_search(
|
||||
request: DemoSearchRequest,
|
||||
repository: Annotated[DemoRepository, Depends(get_demo_repository)],
|
||||
settings: Annotated[Settings, Depends(get_settings)],
|
||||
) -> Any:
|
||||
embedder = FakeEmbeddingProvider(settings.embedding_dimension)
|
||||
reranker = FakeReranker()
|
||||
query_result = await embedder.embed_query(request.query)
|
||||
candidate_limit = min(max(request.top_k * 3, 10), 20)
|
||||
try:
|
||||
candidates = await asyncio.to_thread(
|
||||
repository.search,
|
||||
query_result.vectors[0],
|
||||
limit=candidate_limit,
|
||||
)
|
||||
except (OSError, SecretFileError, psycopg.Error) as exc:
|
||||
raise _database_unavailable(exc) from exc
|
||||
|
||||
if not candidates:
|
||||
return DemoSearchResponse(status="empty_dataset", results=[])
|
||||
|
||||
reranked = await reranker.rerank(
|
||||
request.query,
|
||||
[candidate.rerank_text for candidate in candidates],
|
||||
top_n=min(request.top_k, len(candidates)),
|
||||
)
|
||||
results = [
|
||||
_safe_result(candidates[item.index], item.relevance_score) for item in reranked.items
|
||||
]
|
||||
return DemoSearchResponse(status="ok", results=results)
|
||||
35
backend/app/core/demo_identity.py
Normal file
35
backend/app/core/demo_identity.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""Stable identity and embedding profile for the public synthetic demo corpus."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import uuid
|
||||
|
||||
DEMO_SCOPE_NAME = "synthetic-demo"
|
||||
DEMO_EXPECTED_CHUNKS = 20
|
||||
DEMO_FAKE_EMBEDDING_MODEL = "fake-feature-hash-v1"
|
||||
|
||||
IDENTITY_NAMESPACE = uuid.UUID("eef85571-1f64-4a09-86d7-53fd329c3eb2")
|
||||
KNOWLEDGE_BASE_ID = uuid.uuid5(IDENTITY_NAMESPACE, "synthetic-demo-knowledge-base")
|
||||
ACCESS_SCOPE_ID = uuid.uuid5(IDENTITY_NAMESPACE, "synthetic-demo-public-scope")
|
||||
|
||||
|
||||
def offline_embedding_profile_hash(dimension: int) -> str:
|
||||
"""Return the exact profile bound to vectors produced by the offline embedder."""
|
||||
|
||||
profile = {
|
||||
"api_mode": "deterministic-offline",
|
||||
"dimension": dimension,
|
||||
"endpoint_identity_hash": "local-fake",
|
||||
"model": DEMO_FAKE_EMBEDDING_MODEL,
|
||||
"normalization": "provider-default",
|
||||
"profile_version": 1,
|
||||
}
|
||||
serialized = json.dumps(
|
||||
profile,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
return hashlib.sha256(serialized.encode("utf-8")).hexdigest()
|
||||
161
backend/app/gateway.py
Normal file
161
backend/app/gateway.py
Normal file
@@ -0,0 +1,161 @@
|
||||
"""Small fixed-origin ingress gateway with explicit proxy boundaries."""
|
||||
|
||||
from collections.abc import AsyncIterator, Mapping
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI, Request, Response, status
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
UPSTREAM_ORIGIN = httpx.URL("http://api:8000")
|
||||
MAX_REQUEST_BODY_BYTES = 1024 * 1024
|
||||
SUPPORTED_METHODS = ["GET", "POST", "HEAD", "OPTIONS"]
|
||||
|
||||
REQUEST_HEADER_ALLOWLIST = frozenset(
|
||||
{
|
||||
"accept",
|
||||
"accept-language",
|
||||
"access-control-request-headers",
|
||||
"access-control-request-method",
|
||||
"authorization",
|
||||
"content-type",
|
||||
"if-match",
|
||||
"if-none-match",
|
||||
"origin",
|
||||
"range",
|
||||
"traceparent",
|
||||
"tracestate",
|
||||
"x-request-id",
|
||||
}
|
||||
)
|
||||
|
||||
RESPONSE_HEADER_ALLOWLIST = frozenset(
|
||||
{
|
||||
"accept-ranges",
|
||||
"access-control-allow-credentials",
|
||||
"access-control-allow-headers",
|
||||
"access-control-allow-methods",
|
||||
"access-control-allow-origin",
|
||||
"access-control-max-age",
|
||||
"allow",
|
||||
"cache-control",
|
||||
"content-disposition",
|
||||
"content-language",
|
||||
"content-range",
|
||||
"content-type",
|
||||
"etag",
|
||||
"last-modified",
|
||||
"retry-after",
|
||||
"vary",
|
||||
"www-authenticate",
|
||||
"x-request-id",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _allowlisted_headers(headers: Mapping[str, str], allowlist: frozenset[str]) -> dict[str, str]:
|
||||
return {name: value for name, value in headers.items() if name.lower() in allowlist}
|
||||
|
||||
|
||||
def _upstream_url(request: Request) -> httpx.URL:
|
||||
raw_path_value: object = request.scope.get("raw_path")
|
||||
if isinstance(raw_path_value, bytes):
|
||||
raw_path = raw_path_value
|
||||
else:
|
||||
raw_path = request.url.path.encode("utf-8")
|
||||
if not raw_path.startswith(b"/"):
|
||||
raw_path = b"/" + raw_path
|
||||
|
||||
query_value: object = request.scope.get("query_string")
|
||||
query = query_value if isinstance(query_value, bytes) else b""
|
||||
raw_target = raw_path + (b"?" + query if query else b"")
|
||||
# copy_with preserves the fixed scheme, host, and port even for //host-like paths.
|
||||
return UPSTREAM_ORIGIN.copy_with(raw_path=raw_target)
|
||||
|
||||
|
||||
async def _bounded_body(request: Request) -> bytes | None:
|
||||
declared_length = request.headers.get("content-length")
|
||||
if declared_length is not None:
|
||||
try:
|
||||
parsed_length = int(declared_length)
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed_length < 0 or parsed_length > MAX_REQUEST_BODY_BYTES:
|
||||
return None
|
||||
|
||||
body = bytearray()
|
||||
async for chunk in request.stream():
|
||||
if len(body) + len(chunk) > MAX_REQUEST_BODY_BYTES:
|
||||
return None
|
||||
body.extend(chunk)
|
||||
return bytes(body)
|
||||
|
||||
|
||||
def create_gateway_app(transport: httpx.AsyncBaseTransport | None = None) -> FastAPI:
|
||||
"""Create a no-secret gateway; an injected transport enables hermetic tests."""
|
||||
|
||||
upstream_client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(10.0, connect=2.0),
|
||||
limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
|
||||
follow_redirects=False,
|
||||
transport=transport,
|
||||
trust_env=False,
|
||||
)
|
||||
# Drop httpx convenience defaults; Host and body framing are added by the protocol layer.
|
||||
upstream_client.headers.clear()
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await upstream_client.aclose()
|
||||
|
||||
gateway = FastAPI(
|
||||
title="Geological RAG Gateway",
|
||||
docs_url=None,
|
||||
redoc_url=None,
|
||||
openapi_url=None,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
@gateway.get("/gateway/live", include_in_schema=False)
|
||||
async def gateway_live() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
@gateway.api_route("/{path:path}", methods=SUPPORTED_METHODS, include_in_schema=False)
|
||||
async def proxy(request: Request, path: str) -> Response: # noqa: ARG001
|
||||
body = await _bounded_body(request)
|
||||
if body is None:
|
||||
return JSONResponse(
|
||||
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
|
||||
content={"detail": "request body too large"},
|
||||
)
|
||||
|
||||
upstream_request = upstream_client.build_request(
|
||||
request.method,
|
||||
_upstream_url(request),
|
||||
headers=_allowlisted_headers(request.headers, REQUEST_HEADER_ALLOWLIST),
|
||||
content=body,
|
||||
)
|
||||
try:
|
||||
upstream_response = await upstream_client.send(upstream_request)
|
||||
except httpx.RequestError:
|
||||
return JSONResponse(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
content={"detail": "upstream unavailable"},
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=upstream_response.content,
|
||||
status_code=upstream_response.status_code,
|
||||
headers=_allowlisted_headers(
|
||||
upstream_response.headers,
|
||||
RESPONSE_HEADER_ALLOWLIST,
|
||||
),
|
||||
)
|
||||
|
||||
return gateway
|
||||
|
||||
|
||||
app = create_gateway_app()
|
||||
@@ -1,39 +1,51 @@
|
||||
"""Minimal FastAPI entrypoint; product endpoints are added in stage 2."""
|
||||
"""FastAPI entrypoint with dependency-free liveness and database readiness probes."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
import psycopg
|
||||
import uvicorn
|
||||
from fastapi import FastAPI, HTTPException, status
|
||||
from fastapi import FastAPI, Response, status
|
||||
|
||||
from app import __version__
|
||||
from app.api.v1 import demo_router
|
||||
from app.core.config import get_settings
|
||||
from app.core.secrets import SecretFileError
|
||||
|
||||
app = FastAPI(title="Geological RAG API", version=__version__)
|
||||
app.include_router(demo_router)
|
||||
|
||||
type HealthPayload = dict[str, str | dict[str, str]]
|
||||
|
||||
|
||||
@app.get("/health/live", tags=["health"])
|
||||
@app.get("/api/v1/health/live", tags=["health"])
|
||||
def live() -> dict[str, str]:
|
||||
return {"status": "ok", "version": __version__}
|
||||
|
||||
|
||||
@app.get(
|
||||
"/health/ready",
|
||||
tags=["health"],
|
||||
responses={status.HTTP_503_SERVICE_UNAVAILABLE: {"description": "Database unavailable"}},
|
||||
)
|
||||
@app.get("/api/v1/health/ready", tags=["health"])
|
||||
def ready() -> dict[str, str]:
|
||||
def ready(response: Response) -> HealthPayload:
|
||||
settings = get_settings()
|
||||
try:
|
||||
dsn = settings.database_url().set(drivername="postgresql")
|
||||
with psycopg.connect(
|
||||
dsn.render_as_string(hide_password=False),
|
||||
connect_timeout=2,
|
||||
autocommit=True,
|
||||
) as connection:
|
||||
connection.execute("SELECT 1")
|
||||
except (OSError, SecretFileError, psycopg.Error) as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="database unavailable",
|
||||
) from exc
|
||||
return {"status": "ready"}
|
||||
result = connection.execute("SELECT 1").fetchone()
|
||||
if result != (1,):
|
||||
raise psycopg.DatabaseError("readiness query returned an unexpected result")
|
||||
except (OSError, SecretFileError, psycopg.Error):
|
||||
# Do not expose connection strings, secret paths, hostnames, or driver errors.
|
||||
response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
|
||||
return {"status": "not_ready", "checks": {"database": "unavailable"}}
|
||||
return {"status": "ready", "checks": {"database": "ok"}}
|
||||
|
||||
|
||||
@app.get("/api/v1/meta", tags=["meta"])
|
||||
@@ -52,5 +64,5 @@ def meta() -> dict[str, Any]:
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Container ingress is controlled by Compose; only the edge proxy publishes a port.
|
||||
# Compose publishes this listener only on the host loopback interface.
|
||||
uvicorn.run("app.main:app", host="0.0.0.0", port=8000) # noqa: S104
|
||||
|
||||
@@ -23,14 +23,17 @@ from psycopg.types.json import Jsonb
|
||||
from app.adapters.bailian import BailianEmbeddingAdapter, BailianRerankerAdapter
|
||||
from app.adapters.fake import FakeEmbeddingProvider, FakeReranker, lexical_features
|
||||
from app.core.config import Settings
|
||||
from app.core.demo_identity import (
|
||||
ACCESS_SCOPE_ID,
|
||||
IDENTITY_NAMESPACE,
|
||||
KNOWLEDGE_BASE_ID,
|
||||
offline_embedding_profile_hash,
|
||||
)
|
||||
from app.core.secrets import SecretFileError
|
||||
from app.ports.model_providers import EmbeddingProvider, ModelProviderError, Reranker
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
DEFAULT_SAMPLE_ROOT = PROJECT_ROOT / "data" / "samples" / "public"
|
||||
IDENTITY_NAMESPACE = uuid.UUID("eef85571-1f64-4a09-86d7-53fd329c3eb2")
|
||||
KNOWLEDGE_BASE_ID = uuid.uuid5(IDENTITY_NAMESPACE, "synthetic-demo-knowledge-base")
|
||||
ACCESS_SCOPE_ID = uuid.uuid5(IDENTITY_NAMESPACE, "synthetic-demo-public-scope")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -139,10 +142,9 @@ def load_queries(path: Path) -> list[DemoQuery]:
|
||||
|
||||
|
||||
def embedding_profile_hash(settings: Settings, mode: str) -> str:
|
||||
endpoint_identity = "local-fake"
|
||||
model = "fake-feature-hash-v1"
|
||||
api_mode = "deterministic-offline"
|
||||
if mode == "bailian":
|
||||
if mode != "bailian":
|
||||
return offline_embedding_profile_hash(settings.embedding_dimension)
|
||||
|
||||
endpoint_identity = sha256_text(urlsplit(settings.bailian_openai_base_url).hostname or "")
|
||||
model = settings.embedding_model
|
||||
api_mode = "openai-compatible"
|
||||
|
||||
@@ -17,6 +17,8 @@ from app.persistence.job_queue_sql import ( # noqa: E402
|
||||
)
|
||||
|
||||
COMPOSE = (ROOT / "compose.yaml").read_text(encoding="utf-8")
|
||||
DOCKERFILE = (BACKEND / "Dockerfile").read_text(encoding="utf-8")
|
||||
DEMO_API = (BACKEND / "app/api/v1/demo.py").read_text(encoding="utf-8")
|
||||
BOOTSTRAP = (ROOT / "ops/postgres/init/10-bootstrap-rag.sh").read_text(encoding="utf-8")
|
||||
MIGRATION = (BACKEND / "migrations/versions/0001_initial_schema.py").read_text(encoding="utf-8")
|
||||
|
||||
@@ -31,6 +33,8 @@ def _service_block(name: str) -> str:
|
||||
def test_compose_isolates_database_credentials_and_networks() -> None:
|
||||
db = _service_block("db")
|
||||
migrate = _service_block("migrate")
|
||||
api = _service_block("api")
|
||||
gateway = _service_block("gateway")
|
||||
provider_smoke = _service_block("provider-smoke")
|
||||
seed_demo = _service_block("seed-demo")
|
||||
seed_demo_offline = _service_block("seed-demo-offline")
|
||||
@@ -43,6 +47,27 @@ def test_compose_isolates_database_credentials_and_networks() -> None:
|
||||
assert "postgres_bootstrap_password" not in migrate
|
||||
assert "postgres_app_password" not in migrate
|
||||
|
||||
assert "postgres_app_password" in api
|
||||
assert "postgres_bootstrap_password" not in api
|
||||
assert "postgres_migrator_password" not in api
|
||||
assert "bailian_api_key" not in api
|
||||
assert '"127.0.0.1:8000:8000"' not in api
|
||||
assert " - data" in api
|
||||
assert " - edge" not in api
|
||||
assert " - egress" not in api
|
||||
assert "read_only: true" in api
|
||||
assert "no-new-privileges:true" in api
|
||||
assert "cap_drop:" in api and " - ALL" in api
|
||||
|
||||
assert '"127.0.0.1:8000:8000"' in gateway
|
||||
assert " - edge" in gateway
|
||||
assert " - data" in gateway
|
||||
assert "secrets:" not in gateway
|
||||
assert "POSTGRES_" not in gateway
|
||||
assert "BAILIAN_" not in gateway
|
||||
assert "read_only: true" in gateway
|
||||
assert "no-new-privileges:true" in gateway
|
||||
|
||||
assert "bailian_api_key" in provider_smoke
|
||||
assert "postgres_" not in provider_smoke
|
||||
|
||||
@@ -61,6 +86,34 @@ def test_compose_isolates_database_credentials_and_networks() -> None:
|
||||
assert re.search(r"(?m)^ edge:$", COMPOSE)
|
||||
assert re.search(r"(?m)^ egress:$", COMPOSE)
|
||||
|
||||
assert "USER 10001:10001" in DOCKERFILE
|
||||
assert 'CMD ["uvicorn", "app.main:app"' in DOCKERFILE
|
||||
|
||||
|
||||
def test_demo_queries_pin_governed_offline_identity() -> None:
|
||||
normalized = " ".join(DEMO_API.split())
|
||||
|
||||
for required_filter in (
|
||||
"chunk.knowledge_base_id = %s",
|
||||
"chunk.access_scope_id = %s",
|
||||
"chunk.metadata ->> 'source_type' = 'synthetic'",
|
||||
"chunk.searchable IS TRUE",
|
||||
"chunk.index_status = 'READY'",
|
||||
"chunk.approval_status = 'CLOUD_APPROVED'",
|
||||
"document.active_version_id = chunk.document_version_id",
|
||||
"version.review_state = 'CLOUD_APPROVED'",
|
||||
"version.outbound_manifest_sha256 = chunk.outbound_manifest_sha256",
|
||||
"version.embedding_profile_hash = chunk.embedding_profile_hash",
|
||||
"chunk.embedding_model = %s",
|
||||
"chunk.embedding_profile_hash = %s",
|
||||
):
|
||||
assert required_filter in normalized
|
||||
|
||||
assert "KNOWLEDGE_BASE_ID" in DEMO_API
|
||||
assert "ACCESS_SCOPE_ID" in DEMO_API
|
||||
assert "DEMO_FAKE_EMBEDDING_MODEL" in DEMO_API
|
||||
assert "offline_embedding_profile_hash" in DEMO_API
|
||||
|
||||
|
||||
def test_database_health_requires_tcp_and_atomic_bootstrap_sentinel() -> None:
|
||||
db = _service_block("db")
|
||||
|
||||
224
backend/tests/unit/test_demo_api.py
Normal file
224
backend/tests/unit/test_demo_api.py
Normal file
@@ -0,0 +1,224 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import httpx
|
||||
import psycopg
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.api.v1.demo import (
|
||||
DemoCandidate,
|
||||
DemoCounts,
|
||||
DemoRepository,
|
||||
get_demo_repository,
|
||||
make_citation_id,
|
||||
make_page_label,
|
||||
router,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class StubRepository:
|
||||
count_result: DemoCounts
|
||||
candidates: list[DemoCandidate]
|
||||
failure: psycopg.Error | None = None
|
||||
|
||||
def counts(self) -> DemoCounts:
|
||||
if self.failure is not None:
|
||||
raise self.failure
|
||||
return self.count_result
|
||||
|
||||
def search(self, query_vector: tuple[float, ...], *, limit: int) -> list[DemoCandidate]:
|
||||
if self.failure is not None:
|
||||
raise self.failure
|
||||
assert len(query_vector) == 1024
|
||||
return self.candidates[:limit]
|
||||
|
||||
|
||||
def create_app(repository: DemoRepository) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.dependency_overrides[get_demo_repository] = lambda: repository
|
||||
return app
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_status_reports_only_public_synthetic_counts() -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=20, vectors=20, searchable=20),
|
||||
candidates=[],
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.get("/api/v1/demo/status")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {
|
||||
"status": "ready",
|
||||
"dataset": "synthetic-demo",
|
||||
"counts": {"chunks": 20, "vectors": 20, "searchable": 20},
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_status_explicitly_reports_empty_dataset() -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=0, vectors=0, searchable=0),
|
||||
candidates=[],
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.get("/api/v1/demo/status")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "empty_dataset"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_status_does_not_mark_a_partial_demo_index_ready() -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=20, vectors=19, searchable=18),
|
||||
candidates=[],
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.get("/api/v1/demo/status")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "incomplete_dataset"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_embeds_reranks_and_returns_only_safe_fields() -> None:
|
||||
repository = StubRepository(
|
||||
count_result=DemoCounts(chunks=2, vectors=2, searchable=2),
|
||||
candidates=[
|
||||
DemoCandidate(
|
||||
source_key="internal-chunk-one",
|
||||
title="西岭铜矿化合成记录",
|
||||
text="该合成记录描述花岗斑岩接触带，未包含真实矿权或人员信息。",
|
||||
page_start=3,
|
||||
page_end=3,
|
||||
),
|
||||
DemoCandidate(
|
||||
source_key="internal-chunk-two",
|
||||
title="北谷金异常合成记录",
|
||||
text="北谷合成样点出现低强度金异常，仅用于离线验证。",
|
||||
page_start=7,
|
||||
page_end=8,
|
||||
),
|
||||
],
|
||||
)
|
||||
app = create_app(repository)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.post(
|
||||
"/api/v1/demo/search",
|
||||
json={"query": "花岗斑岩 铜矿化", "top_k": 1},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["status"] == "ok"
|
||||
assert payload["dataset"] == "synthetic-demo"
|
||||
assert len(payload["results"]) == 1
|
||||
result = payload["results"][0]
|
||||
assert set(result) == {"title", "snippet", "page_label", "score", "citation_id"}
|
||||
assert result["title"].startswith("合成资料｜")
|
||||
assert result["page_label"] == "第 3 页"
|
||||
assert result["citation_id"] == make_citation_id("internal-chunk-one")
|
||||
assert "internal-chunk-one" not in str(payload)
|
||||
assert "manifest" not in str(payload).lower()
|
||||
assert "sha256" not in str(payload).lower()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_returns_explicit_empty_dataset_state() -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=0, vectors=0, searchable=0),
|
||||
candidates=[],
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.post(
|
||||
"/api/v1/demo/search",
|
||||
json={"query": "不存在的合成资料", "top_k": 3},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {
|
||||
"status": "empty_dataset",
|
||||
"dataset": "synthetic-demo",
|
||||
"results": [],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
{"query": " ", "top_k": 3},
|
||||
{"query": "x" * 501, "top_k": 3},
|
||||
{"query": "铜矿", "top_k": 0},
|
||||
{"query": "铜矿", "top_k": 11},
|
||||
],
|
||||
)
|
||||
async def test_search_rejects_invalid_bounded_input(body: dict[str, object]) -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=0, vectors=0, searchable=0),
|
||||
candidates=[],
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
response = await client.post("/api/v1/demo/search", json=body)
|
||||
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("path", ["/api/v1/demo/status", "/api/v1/demo/search"])
|
||||
async def test_database_failure_is_sanitized_as_503(path: str) -> None:
|
||||
app = create_app(
|
||||
StubRepository(
|
||||
count_result=DemoCounts(chunks=0, vectors=0, searchable=0),
|
||||
candidates=[],
|
||||
failure=psycopg.OperationalError("host=db password=must-not-leak"),
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
if path.endswith("search"):
|
||||
response = await client.post(path, json={"query": "铜矿", "top_k": 3})
|
||||
else:
|
||||
response = await client.get(path)
|
||||
|
||||
assert response.status_code == 503
|
||||
assert response.json() == {"detail": "database unavailable"}
|
||||
assert "must-not-leak" not in response.text
|
||||
|
||||
|
||||
def test_citation_and_page_labels_are_stable_and_opaque() -> None:
|
||||
first = make_citation_id("private-database-uuid")
|
||||
assert first == make_citation_id("private-database-uuid")
|
||||
assert first != make_citation_id("another-private-database-uuid")
|
||||
assert "private" not in first
|
||||
assert make_page_label(None, None) == "页码未知"
|
||||
assert make_page_label(2, 4) == "第 2-4 页"
|
||||
163
backend/tests/unit/test_gateway.py
Normal file
163
backend/tests/unit/test_gateway.py
Normal file
@@ -0,0 +1,163 @@
|
||||
import json
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.gateway import MAX_REQUEST_BODY_BYTES, create_gateway_app
|
||||
|
||||
type Handler = Callable[[httpx.Request], httpx.Response]
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _gateway_client(handler: Handler) -> AsyncIterator[httpx.AsyncClient]:
|
||||
gateway = create_gateway_app(httpx.MockTransport(handler))
|
||||
async with gateway.router.lifespan_context(gateway):
|
||||
transport = httpx.ASGITransport(app=gateway)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://gateway") as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gateway_live_and_docs_are_disabled() -> None:
|
||||
gateway = create_gateway_app(httpx.MockTransport(lambda _: httpx.Response(500)))
|
||||
try:
|
||||
assert gateway.docs_url is None
|
||||
assert gateway.redoc_url is None
|
||||
assert gateway.openapi_url is None
|
||||
|
||||
transport = httpx.ASGITransport(app=gateway)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://gateway") as client:
|
||||
response = await client.get("/gateway/live")
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"status": "ok"}
|
||||
finally:
|
||||
await _close_gateway(gateway)
|
||||
|
||||
|
||||
async def _close_gateway(gateway: FastAPI) -> None:
|
||||
async with gateway.router.lifespan_context(gateway):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_successful_get_preserves_path_query_and_allowed_header() -> None:
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return httpx.Response(200, json={"result": "ok"}, headers={"x-request-id": "req-1"})
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.get(
|
||||
"/api/v1/search?q=granite%20deposit&limit=5",
|
||||
headers={"x-request-id": "req-1", "x-forwarded-host": "evil.invalid"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"result": "ok"}
|
||||
assert response.headers["x-request-id"] == "req-1"
|
||||
assert seen[0].url == "http://api:8000/api/v1/search?q=granite%20deposit&limit=5"
|
||||
assert seen[0].headers["host"] == "api:8000"
|
||||
assert "x-forwarded-host" not in seen[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_post_json_body_and_content_type_are_forwarded() -> None:
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return httpx.Response(201, json={"accepted": True})
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.post("/api/v1/query", json={"question": "Where is gold?"})
|
||||
|
||||
assert response.status_code == 201
|
||||
assert response.json() == {"accepted": True}
|
||||
assert seen[0].method == "POST"
|
||||
assert seen[0].headers["content-type"] == "application/json"
|
||||
assert json.loads(seen[0].content) == {"question": "Where is gold?"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upstream_origin_is_fixed_against_host_and_path_ssrf_attempts() -> None:
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return httpx.Response(200)
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.get(
|
||||
"//evil.invalid/http://metadata.internal/latest?target=http://attacker.invalid",
|
||||
headers={"host": "attacker.invalid", "forwarded": "host=attacker.invalid"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert seen[0].url.scheme == "http"
|
||||
assert seen[0].url.host == "api"
|
||||
assert seen[0].url.port == 8000
|
||||
assert seen[0].headers["host"] == "api:8000"
|
||||
assert "forwarded" not in seen[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_larger_than_one_mib_is_rejected_before_upstream() -> None:
|
||||
upstream_calls = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal upstream_calls
|
||||
upstream_calls += 1
|
||||
return httpx.Response(200)
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.post("/upload", content=b"x" * (MAX_REQUEST_BODY_BYTES + 1))
|
||||
|
||||
assert response.status_code == 413
|
||||
assert response.json() == {"detail": "request body too large"}
|
||||
assert upstream_calls == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upstream_transport_error_returns_redacted_502() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ConnectError(
|
||||
"api:8000 refused connection with internal-token-value",
|
||||
request=request,
|
||||
)
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.get("/api/v1/health/ready")
|
||||
|
||||
assert response.status_code == 502
|
||||
assert response.json() == {"detail": "upstream unavailable"}
|
||||
assert "api:8000" not in response.text
|
||||
assert "internal-token-value" not in response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sensitive_upstream_response_headers_are_not_forwarded() -> None:
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
200,
|
||||
content=b"ok",
|
||||
headers={
|
||||
"content-type": "text/plain",
|
||||
"cache-control": "no-store",
|
||||
"set-cookie": "session=private",
|
||||
"server": "internal-api",
|
||||
"x-internal-token": "private",
|
||||
},
|
||||
)
|
||||
|
||||
async with _gateway_client(handler) as client:
|
||||
response = await client.get("/status")
|
||||
|
||||
assert response.headers["content-type"] == "text/plain"
|
||||
assert response.headers["cache-control"] == "no-store"
|
||||
assert "set-cookie" not in response.headers
|
||||
assert "server" not in response.headers
|
||||
assert "x-internal-token" not in response.headers
|
||||
@@ -1,22 +1,82 @@
|
||||
import httpx
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from app.main import app
|
||||
import httpx
|
||||
import psycopg
|
||||
import pytest
|
||||
from sqlalchemy import URL
|
||||
|
||||
from app import main
|
||||
|
||||
|
||||
class _SettingsStub:
|
||||
def database_url(self) -> URL:
|
||||
return URL.create(
|
||||
"postgresql+psycopg",
|
||||
username="app",
|
||||
password="database-test-password",
|
||||
host="db.internal",
|
||||
port=5432,
|
||||
database="rag",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_liveness_does_not_require_database_or_model_credentials() -> None:
|
||||
transport = httpx.ASGITransport(app=app)
|
||||
transport = httpx.ASGITransport(app=main.app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/health/live")
|
||||
response = await client.get("/health/live")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "ok"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_readiness_executes_select_one_and_returns_redacted_status() -> None:
|
||||
connection = MagicMock()
|
||||
connection.__enter__.return_value = connection
|
||||
connection.execute.return_value.fetchone.return_value = (1,)
|
||||
|
||||
with (
|
||||
patch.object(main, "get_settings", return_value=_SettingsStub()),
|
||||
patch.object(psycopg, "connect", return_value=connection) as connect,
|
||||
):
|
||||
transport = httpx.ASGITransport(app=main.app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health/ready")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"status": "ready", "checks": {"database": "ok"}}
|
||||
connection.execute.assert_called_once_with("SELECT 1")
|
||||
assert connect.call_args.kwargs == {"connect_timeout": 2, "autocommit": True}
|
||||
assert "database-test-password" not in response.text
|
||||
assert "db.internal" not in response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_readiness_returns_503_without_exposing_database_error() -> None:
|
||||
private_error = psycopg.OperationalError(
|
||||
"could not connect to db.internal with database-test-password"
|
||||
)
|
||||
with (
|
||||
patch.object(main, "get_settings", return_value=_SettingsStub()),
|
||||
patch.object(psycopg, "connect", side_effect=private_error),
|
||||
):
|
||||
transport = httpx.ASGITransport(app=main.app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health/ready")
|
||||
|
||||
assert response.status_code == 503
|
||||
assert response.json() == {
|
||||
"status": "not_ready",
|
||||
"checks": {"database": "unavailable"},
|
||||
}
|
||||
assert "database-test-password" not in response.text
|
||||
assert "db.internal" not in response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_meta_exposes_model_names_but_no_credentials() -> None:
|
||||
transport = httpx.ASGITransport(app=app)
|
||||
transport = httpx.ASGITransport(app=main.app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/meta")
|
||||
|
||||
|
||||
70
compose.yaml
70
compose.yaml
@@ -81,6 +81,76 @@ services:
|
||||
- data
|
||||
restart: "no"
|
||||
|
||||
api:
|
||||
build:
|
||||
context: ./backend
|
||||
command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
environment: *runtime-config
|
||||
secrets:
|
||||
- postgres_app_password
|
||||
networks:
|
||||
- data
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- python
|
||||
- -c
|
||||
- >-
|
||||
import urllib.request;
|
||||
urllib.request.urlopen('http://127.0.0.1:8000/health/ready', timeout=2)
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
start_period: 5s
|
||||
init: true
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
restart: unless-stopped
|
||||
|
||||
gateway:
|
||||
build:
|
||||
context: ./backend
|
||||
command: ["uvicorn", "app.gateway:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "127.0.0.1:8000:8000"
|
||||
networks:
|
||||
- edge
|
||||
- data
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- python
|
||||
- -c
|
||||
- >-
|
||||
import urllib.request;
|
||||
urllib.request.urlopen('http://127.0.0.1:8000/health/ready', timeout=2)
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
start_period: 5s
|
||||
init: true
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
restart: unless-stopped
|
||||
|
||||
provider-smoke:
|
||||
build:
|
||||
context: ./backend
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
| 已完成阶段 | Stage 0：仓库、安全和设计基线 |
|
||||
| 整体完成度 | 约 12%，合理区间 10%–15% |
|
||||
| 设计完成度 | 设计基线已完成；实现中发现新约束时继续通过 ADR 和文档修订维护 |
|
||||
| 业务代码完成度 | 约 8%；Stage 1 的离线/数据库 PoC 已验收，真实百炼 smoke 仍待新 Key，前端产品功能尚未实现 |
|
||||
| 业务代码完成度 | 约 10%；Stage 1 的 Docker API、离线检索和数据库 PoC 已验收，真实百炼 smoke 仍待新 Key，前端产品功能尚未实现 |
|
||||
| 当前预计剩余工期 | 约 9–13 周，含 300 题正式标注、盲测、论文和答辩缓冲 |
|
||||
| 进度权威来源 | 本文的阶段状态、验收证据和已推送提交 |
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
| Stage 9 答辩、发布与归档 | 3% | `TODO` | 0% |
|
||||
| **合计** | **100%** | — | **12%** |
|
||||
|
||||
Stage 1 已完成可离线验收的数据库、适配器契约和 synthetic seed 子闭环，阶段内部约完成 85%；但真实三模型 smoke 仍依赖轮换后的新 Key，因此 Stage 1 不提前计入整体里程碑。本文复选框表示“已通过验收”，不表示“文件是否已经出现”。当前可诚实表述为：**设计基线完成，整体里程碑约 12%，Stage 1 离线部分可运行，真实百炼验证尚未完成。**
|
||||
Stage 1 已完成可离线验收的数据库、适配器契约、synthetic seed 和只读 API 子闭环，阶段内部约完成 90%；但真实三模型 smoke 仍依赖轮换后的新 Key，因此 Stage 1 不提前计入整体里程碑。本文复选框表示“已通过验收”，不表示“文件是否已经出现”。当前可诚实表述为：**设计基线完成，整体里程碑约 12%，Stage 1 离线后端可运行，真实百炼验证尚未完成。**
|
||||
|
||||
## 2. 阶段依赖与关键路径
|
||||
|
||||
@@ -134,6 +134,7 @@ Stage 0 DONE
|
||||
- [x] 创建 `backend/pyproject.toml` 和依赖锁文件，不引入未说明依赖。
|
||||
- [x] 创建最小 `backend/Dockerfile`，使用非 root 用户和固定基础镜像 digest。
|
||||
- [x] 创建最小 `compose.yaml`，包含 `db + migrate + provider-smoke + seed-demo/offline`。
|
||||
- [x] 启动最小 FastAPI `api + gateway`，内部 API 无 egress，gateway 无 Secret 并仅发布回环端口；提供真实数据库 readiness、Swagger 和只读 synthetic demo 检索。
|
||||
- [x] 创建 PostgreSQL bootstrap、migrator 和 app 分权角色初始化脚本；备份只读角色在备份功能落地时单独创建。
|
||||
- [x] 创建 Alembic 基线迁移，启用 pgvector 并建立 1024 维向量表/HNSW 基线。
|
||||
- [x] 确保运行期服务不挂载 bootstrap 或 migrator Secret。
|
||||
@@ -162,8 +163,9 @@ Stage 0 DONE
|
||||
- [x] `docker compose up db migrate` 在空 volume 成功建立扩展、角色和表。
|
||||
- [ ] `docker compose run --rm provider-smoke` 三模型探测成功，日志无 Secret。
|
||||
- [x] `docker compose run --rm seed-demo-offline` 完成 20 条虚构数据写入、检索和重排；真实模式待新 Key。
|
||||
- [x] `api + gateway` 镜像构建并达到 healthy；live/ready/meta/demo status/demo search 均通过真实 HTTP 验收。
|
||||
- [x] 第二次运行 seed 后数据库计数与第一次相同，均为 chunks/vectors/searchable = 20/20/20。
|
||||
- [x] 41 项测试证明 Embedding 维度/下标、Rerank 下标、Chat 流式响应和失败路径正确。
|
||||
- [x] 63 项测试证明模型契约、健康检查、固定上游 gateway、离线 demo、数据治理和失败路径正确。
|
||||
- [x] `make verify`、Secret 扫描、固定镜像构建和 `git diff --check` 通过。
|
||||
|
||||
### 2026-07-12 已验证运行证据
|
||||
@@ -171,6 +173,8 @@ Stage 0 DONE
|
||||
- 全新 volume 上 `db` 达到 healthy，`migrate` 以非超级用户退出码 0，版本为 `0001_initial_schema`。
|
||||
- `vector` 扩展与 HNSW 基线存在；app/migrator 均非超级用户，app 无 `rag` schema DDL 权限但具有所需 DML。
|
||||
- 两次离线 seed 均输出 20/20/20，9 个可回答虚构问题 Hit@3 = 9/9。
|
||||
- FastAPI 容器使用 app 最小权限角色，根文件系统只读且无 egress；无 Secret gateway 仅提供回环入口；Swagger 可见，demo search 返回合成片段与不透明 citation ID。
|
||||
- API 容器对公网 TCP 探测返回 `ENETUNREACH (101)`；gateway 的 mounts 为空且不含数据库或百炼业务环境变量。
|
||||
- PostgreSQL 重启后计数仍为 20/20/20。
|
||||
- app 尝试修改已 `CLOUD_APPROVED` 的 `cloud_text` 被审批不可变触发器拒绝；尝试建表被权限拒绝。
|
||||
- 唯一未验收项是真实 `text-embedding-v4` / `qwen3-rerank` / `deepseek-v4-flash` smoke 与真实模式 seed。
|
||||
@@ -180,6 +184,7 @@ Stage 0 DONE
|
||||
- [x] `S1-A`：数据库初始化、分权角色、迁移和最小 Compose 已随 `f4ba5d5` 验证并推送。
|
||||
- [ ] `S1-B`：三模型适配器和 fake 契约测试已随 `f4ba5d5` 推送；新 Key 下的 live smoke 未完成，因此节点保持未完成。
|
||||
- [x] `S1-C`：20 条虚构 seed、幂等验证、运行证据和文档已随 `f4ba5d5` 推送。
|
||||
- [ ] `S1-D`：内部 API、无 Secret gateway、只读 demo 检索和真实 HTTP/Docker 运行证据；验证后提交并推送。
|
||||
|
||||
`f4ba5d5` 是 Stage 1 离线可执行闭环提交，不代表 Stage 1 已完成；阶段完成仍以真实三模型 smoke 和真实模式 seed 通过为准。
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ bash scripts/init-local-secrets.sh
|
||||
离线模式使用确定性的 1024 维 feature-hash 向量和词法重排器，只验证数据治理、批处理、迁移、向量写入、权限过滤、检索、重排和幂等性，不把它当成真实模型质量。
|
||||
|
||||
```bash
|
||||
docker compose up -d --build db migrate
|
||||
docker compose up -d --build gateway
|
||||
docker compose --profile tools run --rm seed-demo-offline
|
||||
docker compose --profile tools run --rm seed-demo-offline
|
||||
```
|
||||
@@ -44,6 +44,28 @@ Seed 的实际流程是：
|
||||
7. 对 Top K 候选重排并输出虚构问题的 Hit@3；
|
||||
8. 相同输入再次执行使用相同 ID 和幂等 upsert，不增加行数。
|
||||
|
||||
### 2.1 查看后端运行效果
|
||||
|
||||
内部 API 只连接 `internal` data 网络，只挂载 app 数据库 Secret，不连接外网，也不挂载百炼 Key。无 Secret、无数据库凭证的 gateway 同时连接 ingress 与 data 网络，并且仅发布到本机回环地址 `127.0.0.1:8000`。两个容器都以数值非 root 用户运行，根文件系统只读，并移除 Linux capabilities。
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8000/health/live
|
||||
curl http://127.0.0.1:8000/health/ready
|
||||
curl http://127.0.0.1:8000/api/v1/demo/status
|
||||
curl -X POST http://127.0.0.1:8000/api/v1/demo/search \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data '{"query":"花岗斑岩铜矿化特征","top_k":3}'
|
||||
```
|
||||
|
||||
预期结果：
|
||||
|
||||
- readiness 返回 `database=ok`；
|
||||
- demo status 返回 chunks/vectors/searchable = 20/20/20；
|
||||
- search 只返回 `synthetic-demo` 数据集中的合成标题、批准后的片段、页码、分数和不透明 citation ID；
|
||||
- Swagger 位于 <http://127.0.0.1:8000/docs>。
|
||||
|
||||
该接口是 Stage 1 的离线可见效果，不调用百炼，也不代表完整问答生成链或 React 前端已经完成。
|
||||
|
||||
## 3. 真实百炼能力探测
|
||||
|
||||
完成旧 Key 轮换后：
|
||||
|
||||
Reference in New Issue
Block a user