Skip to content

feat(store): model consumers, model keys and ctx_model_auth (Migration 161, W01-1) - #85

Merged
GottZ merged 2 commits into
rootfrom
feat/oai-w01-1
Nov 3, 2026
Merged

GottZ merged 2 commits into
rootfrom
feat/oai-w01-1

Conversation

@GottZ

@GottZ GottZ commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

W01-1 (RC-B, first migration of the credential class): Migration 161 — context_model_consumers (tenant-bound caller identity + policy: tenant_scope, all_models, prompt_sensitivity, fair_weight, admission_lane, allowed_cidrs, allow_plaintext, peer binding), context_model_keys (credential of exactly one consumer, stored as SHA-256 + hint), ctx_model_auth(hash, ip, secure) STABLE with six gates and one byte-identical sentinel row, deactivation trigger (revokes live keys), identity pins (tenant_id / consumer_id immutable, 23514), whitelist audit of both tables through audit_actor_meta() with metadata.tenant_id/consumer_id, model_cidrs_private, context_tenants.max_model_consumers, and the tenant expression index idx_settings_audit_entity_tenant on context_settings_audit (W06-7 review F1). Nothing writes these tables yet (store primitives W01-3, admin actions W01-5, the /v1 door W01-6).

  • Second commit (review 2 F-1): a BEFORE INSERT guard on context_model_keys locks the consumer FOR SHARE and refuses an inactive consumer with 23514, so a mint and a deactivation serialize on the consumer row under READ COMMITTED (the FK alone takes FOR KEY SHARE, which the deactivating UPDATE does not wait for — a mint overlapping a deactivation committed a live key at an inactive consumer). Known limit documented: a deactivation under explicit REPEATABLE READ does not revoke a key committed after its snapshot.
  • Manifest regenerated (extension floors left as on root, their raise is a separate decision), test.sh T07 59 → 61 tables, docs (operations.md Migration 161, security.md audit trail, multi-tenancy.md).

Gates (red against 6773d5b / migration 160, then green)

  • Store gates against 160: 42P01 (tables), ctx_model_auth 42883, classification pin 36 deviations, plan test 2 400 / 1 407 rows removed by filter → 0. Gate matrix 13/13 + reads_only (n_tup_upd unchanged, provolatile = s).
  • Overlays CTX_M161_OVERLAY=gate1…gate6 make exactly their rows red (3/1/2/1/2/1); no-deactivate, no-identity, no-of-list, blacklist, extra-fk, key-policy, cidr-null, no-insert-guard each red. Main context re-ran gate3 (2 rows) and no-insert-guard (3/3 subtests red).
  • Index at 1 M audit rows: tenant without rows 1.47 s → 331 µs per request (3.00 s → 346 µs under load); index build 4.4–9.1 s on a loaded host (writes wait, reads do not; live ≈ 500 rows ⇒ ms).
  • Review 1 (lead-own, Opus): 3 minor hardened (deactivation with a key committed after tx start, stamped actor that no longer exists). Review 2 (fresh Opus, main context): merge-able after F-1 (above); F-2 Low sentinel timing 14.5 µs gate 5/6 vs gate 1 (unknown vs revoked stay indistinguishable) documented; 14 mutations all caught; scope of manifest floors and _migrations way-back confirmed.
  • Suites: build, vet (integration tag), lint ×2, deadcode ×2, -short -race, integration store (full) / handler Audit / auth / schemacontract / ctxd — green on lead and main context. TestClusterAnnotationCostAtCeiling (p95 latency budget) red once under host load ≈ 60, isolated 10.3 ms PASS — outside this wave.

Hand-offs

  • W01-3: prune step 7b (keys first, two statements), revoke with GREATEST, refuse mint into an inactive consumer on the Go side too (lock duty), set revoked_by_principal. W01-5: peer binding visibility (design/05 vs design/04 disagree). W01-6: normalize client IP with Unmap(). W01-8: NOTIFY also on admission_lane/fair_weight. W01-9: grant table with its own whitelist. Offboarding wave: pin context_api_keys.tenant_id incl. fixture rebuild. Open: F4 timing index, extension-floor decision.

Wave report .project/plan-openai-api-2026-11-03/reports/bau/W01-1.md, review reports/review/W01-1-fremdreview-2.md.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added model consumer and model key management, including tenant limits, key lifecycle controls, and authentication checks.
    • Added audit history for model consumers and keys, with tenant-specific audit reads.
  • Documentation
    • Updated operational, security, onboarding, and audit documentation to describe the new capabilities, restrictions, and audit behavior.

GottZ and others added 2 commits November 3, 2026 22:43
…n 161)

Wave W01-1 of the OpenAI-compatible API layer (RC-B, first migration of the
chain; design/01 §3.3-§3.5 with the masterplan amendments K16, K20, K24,
design/03 §3.1, design/04 M-F3, W06-7 F1). Schema only: nothing in ctx
writes the new tables yet, no behaviour changes outside the database.

- context_model_consumers (identity + policy: tenant, tenant_scope, kind,
  all_models, prompt_sensitivity, fair_weight, admission_lane, CIDR
  allowlist, allow_plaintext, active, peer binding) and context_model_keys
  (SHA-256 + hint only, no policy column); context_tenants.max_model_consumers
  = 64 for every tenant; model_cidrs_private() for the plaintext CHECK.
- ctx_model_auth(key_hash, client_ip, secure): STABLE, read-only, six gates,
  one sentinel row for every refusal; result columns fixed, admission_lane
  and peer_instance_id last.
- Deactivating a consumer revokes its live keys on every path; a
  consumer's tenant_id and a key's consumer_id are immutable (23514).
- Whitelist audit of both tables into context_settings_audit
  (model_consumer / model_key) via audit_actor_meta(), metadata.tenant_id
  and metadata.consumer_id on every row; one trigger set per table.
- idx_settings_audit_entity_tenant serves the tenant shapes of
  GET /api/audit (1 000 000 rows, tenant without rows: 1.47 s -> 0.33 ms).

Gates, red first, then green:
- TestModelAuthGateMatrix (13 rows): red against 160 (42P01 at the
  fixture; ctx_model_auth 42883 in TestModelAuthContract); overlays
  gate1..gate6 turn exactly their rows red (3/1/2/1/2/1); green 13/13 +
  reads_only (n_tup_upd unchanged, provolatile s).
- I-K8 red with no-deactivate, identity pins red with no-identity, audit
  bookkeeping red with no-of-list (2 subtests), whitelist red with
  blacklist, FK/policy census red with extra-fk / key-policy.
- Classification pin + per-column probe: red against 160 (36 mismatches),
  red with the migration but without the entries (new completeness
  check: 2 mismatches); green.
- TestAuditListPlan: tenant shapes on the new index under custom and
  generic plans; tenant without rows removed 2400 rows by filter (160),
  0 now.
- Handler: tenant-admin reads its own model_consumer/model_key rows only;
  red against 160 (42P01) and against 161 without its audit triggers
  (rows: [] for both readers).
- T07 gates red with test.sh at 59 tables, green at 61; idempotent second
  run with identical catalog; documented way back verified.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…61, W01-1 review 2 F-1)

Main-context review 2 of W01-1, F-1 (Medium, executed): a key INSERT and a
consumer deactivation did not wait for each other. The foreign key takes
FOR KEY SHARE on the consumer row, the deactivating UPDATE FOR NO KEY
UPDATE; the revoking trigger did not see the uncommitted key. Under READ
COMMITTED a key minted during a deactivation stayed live at the inactive
consumer and authenticated again after active = true; a key inserted for
an already inactive consumer did the same.

161 is not landed yet, so the migration itself changes (no 162):
- BEFORE INSERT trigger trg_model_keys_consumer_active /
  model_keys_consumer_active(): locks the consumer row FOR SHARE and
  refuses a missing or inactive consumer with 23514. A mint holding the lock
  makes the deactivation wait (its revoke then sees the key); a mint behind
  a running deactivation waits and fails.
- Fixture TestModelAuthGateMatrix/consumer_inactive: key minted while the
  consumer is active, then deactivated with the revoking trigger disabled
  (gate 2 stays testable as defence in depth).
- Header, docs/operations.md and docs/security.md: "revokes and refuses";
  known limit named (a deactivation under explicit REPEATABLE READ does not
  revoke a key committed after its snapshot; ctx writes under READ
  COMMITTED). Way back and manifest carry the new function and trigger.

Gates:
- TestModelKeyInsertRacesDeactivation (READ COMMITTED, insert_first,
  deactivate_first, sequential insert into an inactive consumer): red
  against 161 without the guard (3/3: "consumer active=false with 1 live
  key(s)", "valid after reactivation", "err <nil>, want 23514"), red again
  with CTX_M161_OVERLAY=no-insert-guard, green with the guard
  (deactivate_first: insert fails 23514).
- All 161 tests green; every overlay still red as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 3, 2026 22:20
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Migration 161 adds model-consumer and model-key tables, authentication and lifecycle rules, audit writing, and a tenant-scoped audit index. Integration tests cover the migration, authentication gates, lifecycle constraints, audit records, and tenant-filtered audit reads.

Changes: Model Consumer Data and Audit

Layer / File(s) Summary
Consumer and key schema
go/migrations/161_model_consumers.sql, go/internal/schemacontract/manifest.json, docs/multi-tenancy.md, test.sh
Migration 161 adds consumer and key tables, a tenant consumer limit defaulting to 64, and related constraints and indexes. The schema manifest and documentation record these additions, and the schema-count fallback advances to migration 161.
Lifecycle and authentication
go/migrations/161_model_consumers.sql, go/internal/store/model_consumers_migration_integration_test.go, docs/operations.md
Triggers enforce key revocation on consumer deactivation, active-consumer checks on key insertion, and immutable consumer and key ownership. ctx_model_auth checks key, consumer, tenant, scope, network, and transport conditions. Integration tests cover these behaviors and migration rollback and reapplication.
Audit writing and classification
go/migrations/161_model_consumers.sql, go/internal/store/audit_classification_test.go, go/internal/store/api_keys_audit_integration_test.go, go/internal/store/model_consumers_migration_integration_test.go, go/internal/schemacontract/manifest.json, docs/security.md
Audit projections and triggers record consumer and key lifecycle and policy changes while excluding key hashes and other unlisted fields. Tests check column classifications, trigger coverage, audit events, and attribution.
Audit reads and tenant filtering
go/internal/store/audit.go, go/internal/store/audit_integration_test.go, go/internal/handler/audit.go, go/internal/handler/audit_integration_test.go, docs/api.md, docs/security.md
Audit handling and documentation include model consumers and model keys. Integration tests check tenant filtering, foreign-ID responses, actor visibility, and index plans for tenant and server-admin queries.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Possibly related PRs

  • GottZ/ctx#78: Adds the audit reader whose tenant-filtered queries use the index and entity handling covered here.
  • GottZ/ctx#76: Adds audit infrastructure and classification patterns used by migration 161.
  • GottZ/ctx#82: Defines model-key hash and hint values stored by the new model-key table.
Merge Risk: Minimal · up to 31d50

Migration 161 adds model-consumer and model-key storage, authentication gates, lifecycle rules and audit history. No application code uses these tables yet, so there is no user-facing impact at merge. No merge-blocking issue was found in the supplied changes.

Security architecture risk: Low · up to 31d50

The new credential storage has explicit ownership, network, transport and revocation controls. No production model-authentication endpoint is connected yet, limiting immediate exposure. Lifecycle guarantees depend on documented transaction and cleanup ordering, and deployed database permissions remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The immediate authority surface is the database credential and audit state, not a newly exposed model-serving endpoint. Incorrect future enforcement could affect the models and policy available to an affected consumer, but the SQL function itself does not grant corpus access or return backend credentials. Effective direct-database access cannot be bounded without deployed role permissions.

Trust Boundaries and Controls

  • observed — Authentication requires a live key, active consumer, active tenant, continued tenant ownership of any configured scope, a matching configured network allowlist, and permitted transport. Unknown client IP fails a configured allowlist, unknown transport security is treated as insecure, and plaintext permission requires private-network CIDRs. Client IP and transport truth remain inputs supplied by the eventual trusted caller.

Resilience and Maintainability Implications

  • observed — Under READ COMMITTED, minting locks the consumer FOR SHARE and serializes with deactivation: the new key is either revoked or its insertion is refused. Tests cover both race directions and reactivation without reviving revoked keys. Explicit REPEATABLE READ has a documented exception: a key committed after the deactivator's snapshot can remain unrevoked and pass authentication after reactivation. This is a caller-isolation requirement, not an established remote attack path at this head.

Hardening Proposals

  • proposed — When production writers arrive, make READ COMMITTED isolation and separate keys-first cleanup explicit store-level requirements, including retry and recovery paths. Verify effective database privileges and trusted derivation of client IP and transport security before connecting the public model endpoint.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 51.72% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and specifically summarizes the main changes: model consumers, model keys, and ctx_model_auth in Migration 161. The W01-1 reference adds useful scope without making the title vague.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 51.72% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 29 functions across 8 files. (6 skipped: 6 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 40a056c0-5fa9-47bb-8c82-e198c0f9da8e
Commits

Reviewing files that changed from the base of the PR and between b7809eb and 31d50fa.

Files selected for processing: 14
  • docs/api.md
  • docs/multi-tenancy.md
  • docs/operations.md
  • docs/security.md
  • go/internal/handler/audit.go
  • go/internal/handler/audit_integration_test.go
  • go/internal/schemacontract/manifest.json
  • go/internal/store/api_keys_audit_integration_test.go
  • go/internal/store/audit.go
  • go/internal/store/audit_classification_test.go
  • go/internal/store/audit_integration_test.go
  • go/internal/store/model_consumers_migration_integration_test.go
  • go/migrations/161_model_consumers.sql
  • test.sh

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

The change cannot be approved because the audit integration test’s EXPLAIN (ANALYZE) index/plan verification currently ignores a potential auCheckPlan error, allowing performance regressions (e.g., falling back to a sequential scan) to pass undetected.

1 open issue
What changed in this PR

This pull request introduces Migration 161, adding database primitives for authenticating tenant-scoped “model consumers” and their per-consumer “model keys,” alongside a stable ctx_model_auth(hash, ip, secure) function, lifecycle/immutability constraints, and audited change history. It also updates schema contract artifacts, integration tests, and documentation (security/ops/multi-tenancy/API) to reflect the new surfaces and expected behavior.

Changes:

  • Add Migration 161 schema: context_model_consumers, context_model_keys, ctx_model_auth(...), related triggers/guards, and tenant audit performance indexing.
  • Expand store/handler integration tests and audit classification/whitelist coverage for the new entities and invariants.
  • Regenerate schema manifest and update operational + security documentation for the new auth/audit model.
File Summary
test.sh Updates schema integrity fallback expectations for migration 161 table counts.
go/​migrations/​161_model_consumers.sql Defines model consumer/key tables, ctx_model_auth(...), triggers/guards, audit projections, and a tenant-audit index.
go/​internal/​store/​model_consumers_migration_integration_test.go Adds integration gates for migration 161 behavior (auth semantics, triggers, audit whitelist, idempotence).
go/​internal/​store/​audit.go Aligns/document audit read-path behavior for tenant vs server-admin readers (including tenant index usage).
go/​internal/​store/​audit_integration_test.go Updates audit plan checks for the new tenant index and adds an EXPLAIN ANALYZE-based guard; moderate: plan check currently discards auCheckPlan error—test should fail on plan-check error to prevent silent index regressions.
go/​internal/​store/​audit_classification_test.go Extends audit classification/pinning to the new model tables and enforces audited-table classification completeness.
go/​internal/​store/​api_keys_audit_integration_test.go Expands the “one audit trigger set per table” gate to include model consumer/key audit triggers.
go/​internal/​schemacontract/​manifest.json Regenerates schema contract manifest to include migration 161 objects and updated counters.
go/​internal/​handler/​audit.go Updates/notes handler-level audit visibility/redaction considerations for new model entities.
go/​internal/​handler/​audit_integration_test.go Adds handler-level integration coverage for tenant filtering/redaction of model_consumer/model_key audit rows.
docs/​security.md Documents audit trail semantics for model consumers/keys (actions, whitelist contents, filtering/redaction behavior).
docs/​operations.md Adds migration 161 operational notes (locks/behavior, downgrade/rollback considerations).
docs/​multi-tenancy.md Documents max_model_consumers and immutable tenant binding for consumers/keys.
docs/​api.md Updates audit/API documentation to reflect model_consumer/model_key audit behavior introduced by the migration.

Review effort: Balanced

Comment on lines +515 to +516
summary, _ := auCheckPlan(raw, "")
t.Logf("tenant %s: %d row(s) returned, Rows Removed by Filter %d — %s", tenant, len(rows), removed, summary)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants