Commit 0003e99
feat(detector): enforce structural eligibility for threat verdicts (#916)
* feat(detector): enforce structural eligibility for threat verdicts
Add three-part defense against the false-positive class where the detection
model reports a structurally impossible verdict (e.g. malicious_patch=true
with zero patch files, or prompt_injection=true with zero untrusted input in
the workflow prompt):
1. Structural eligibility check in the threat_detection_result tool.
pkg/detector/eligibility.go computes per-category eligibility from the
loaded artifacts and prompt analysis; the detector transports it to the
report-result subprocess via THREAT_DETECTION_ELIGIBLE_* env vars, and
report-result rejects any threat=true claim against an ineligible
category as a normal correctable error. Missing/unparseable transport
defaults to permissive so pre-existing callers are not tightened.
2. Prompt taxonomy update. Adds a "Not a Threat" section to the default
detection prompt clarifying that instruction non-compliance is a
quality signal (not a security verdict), framework-rejected safe-output
validation errors are guardrails working, prompt_injection requires an
untrusted origin, and malicious_patch requires an actual patch. Includes
a self-check the model runs before setting any flag true.
3. Retry budget raised from 1 to 3 (env: THREAT_DETECTION_RETRIES). With
eligibility rejections now flowing through the correction loop, one
retry burns the budget on taxonomy noise rather than genuine malformed
output. Three keeps the safety net cheap.
Spec: adds TD-10g normative statement of the eligibility invariants and
the correction-loop enforcement contract. README documents the new
Structural eligibility section and the retries=3 default.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* fix(detector): make eligibility enforcement authoritative and honest
Addresses review feedback on the eligibility change:
- Add comment memory as an eligible prompt_injection origin (the agent
reads it back into its own prompt) and as a secret_leak channel.
- Fail open when prompt provenance is degraded: an absent optional
prompt-template.txt leaves UntrustedInputs empty for reasons unrelated
to untrusted content, so prompt_injection stays eligible there.
- Re-validate every sink result in the detector process. The
THREAT_DETECTION_ELIGIBLE_* environment check runs in a subprocess
whose command line the model composes, so it can be overridden or
stripped; it is now documented as advisory, and the binding check runs
against the eligibility the detector computed from artifacts itself.
An ineligible result is discarded and self-corrected, never rewritten.
- Scope the prompt self-check per category: only prompt_injection needs
an untrusted cause, and only it maps to "attack". Secret leaks are
often accidental and patches are malicious on their own merits. The
contradiction check ("does my reason argue against my verdict") stays
universal, since that is what run 32239956539 failed.
- Drop the "prefer a false negative on marginal calls" tie-breaker,
which contradicted two existing err-toward-reporting rules.
- Document the mid-run blind spot: MCP tool results, fetched content and
the engine transcript are never staged, so eligibility neither sees nor
claims to see injections delivered only through them.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* revert(detector): restore default retries to 1
Each retry is a full engine restart, not another turn in a warm
session: the sink is cleared, a new subprocess is spawned, and the
entire artifact set is re-analyzed with the original prompt plus a
bounded correction line. A default of 3 therefore meant up to four
complete detection passes per run, and a systematically confused model
would fail slowly and expensively rather than fast.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* docs(detector): clarify that safe-output bodies are the agent-output channel
"Non-empty agent output" reads as free-form prose, which invites the
question of whether create_issue, add_comment and create_pull_request
bodies are separately gated for secret_leak. They are not, because they
are not separate artifacts: agent_output.json is gh-aw's safe-output
file and those bodies are its contents. No behavior change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* fix(detector): stop truncating detector-authored correction feedback
The 512-byte cap on correction feedback exists to bound text originating
outside the detector — parser errors and model-written previews. The
eligibility explanation is composed from the detector's own fixed
strings, and applying the cap to it had a real cost: rejecting two or
three categories at once produces up to 999 bytes, so the trailing
explanations were cut mid-sentence and the model was told its verdict
was rejected without being told why for every category it had to
re-answer.
Add BuildTrustedCorrectionPrompt for detector-composed feedback, which
is bounded by construction, and use it on the eligibility path.
BuildCorrectionPrompt keeps truncating and now delegates to it.
Also name safe outputs in the secret_leak rejection message, so a model
that found a secret in an issue or comment body can tell that body would
have appeared in agent_output.json and re-check where it actually read
the value.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* docs(detector): document detection as a downstream gate, not a session audit
The in-session blind spots (MCP tool results and fetched pages never
staged; secrets exfiltrated mid-run never entering the bundle) were
being read as gaps to close. They are not: detection runs after the
agent finishes and gates whether its requested outputs may be applied,
so anything the agent already did is by construction outside what it can
gate. Network egress restriction and MCP tool constraints are the
controls that cover in-session risk.
Records this in the spec (new section 1.3), the README, and the prompt
itself, which now frames the model's task as "should this be allowed to
proceed?" and tells it not to infer threats from evidence it was never
given.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* fix(detector): fail open on indeterminate provenance in eligibility
A GPT-5.6 review pass found three paths where a missing or unreadable
artifact made a category ineligible, so the binding parent-side check
would discard a genuine finding. Suppressing real threats is the worst
outcome this gate can produce, and each was a fail-closed bug.
- Untrusted-input extraction needs both the prompt template and the
rendered prompt, and matches the template's static segments against the
rendered text; unmatched segments are skipped silently. An empty
result therefore had two indistinguishable causes, and only the absent-
template one was treated as degraded. Record indeterminacy explicitly
in PromptAnalysis and grant eligibility on it, so an unreadable
prompt.txt or a template/rendered divergence can no longer rule out
injection.
- An unreadable comment-memory directory was collapsed to "no files",
making it indistinguishable from an absent one and able to make both
prompt_injection and secret_leak ineligible. An uninspectable channel
is a channel: add Artifacts.HasWarningForField and count it.
- The prompt told the model to report unattributable suspicious content,
then told it a prompt_injection verdict is false unless it can name the
untrusted region, while the reason format offers "ORIGIN: unknown".
The conflict landed exactly where eligibility now fails open. Resolve
it on the axis that matters: the check requires concrete suspicious
content, not a resolved origin, and degraded provenance is reported
with ORIGIN: unknown rather than suppressed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* refactor(detector): derive eligibility from declared channels
Eligibility was three hand-rolled boolean expressions with the
comment-memory condition duplicated across two of them, and rejection
messages that restated the channel list in prose. Adding a source meant
finding every scattered condition, remembering the fail-open pattern,
and updating strings by hand — and the two fail-closed bugs already
found came from exactly that.
Declare each source as a channel with separate `present` and
`uninspectable` signals, either of which grants eligibility, and derive
both the verdict and the rejection text from the same declarations.
Adding a source is now one struct literal in inputChannels,
outputChannels or patchChannel. Tests enforce the contract: every
channel must be eligible on either signal, and every channel must be
named in its rejection message.
Uniform treatment also closes a fourth fail-closed path the earlier
review did not reach: gh-aw sets HAS_PATCH when the agent job reported a
patch, so a patch the host failed to stage is uninspectable rather than
absent, and must not make malicious_patch ineligible.
Behavior is otherwise unchanged; the existing eligibility tests pass
untouched apart from one assertion on reworded message text.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
* fix(artifacts): stop describing uninspectable channels as empty
An artifact channel that exists but could not be read was handed to the
detection model as "No comment-memory files found" / "No patch or bundle
file found" -- the same text as a channel that genuinely holds nothing.
The model then reported clean about content nobody looked at and the run
exited 0, a fail-open outcome one layer above the eligibility check added
earlier in this PR.
Uninspectable channels now carry an explicit notice that they were not
analyzed, plus a statement that the inspection failure is not itself
evidence of a threat. That second half matters: without it a staging
fault becomes a false-positive source, which is the failure mode this
work exists to reduce.
Also warns on the two silent refusal paths in comment-memory loading: a
comment-memory entry that is not a directory, and a .md entry that is not
a regular file. Both refusals are deliberate security guards against
resolving outside the artifacts tree, which makes them agent-reachable --
returning silently let a run hide its comment memory while still reading
as fully inspected.
Refs #954, #955
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>
---------
Co-authored-by: GitHub Ace <githubnext@users.noreply.github.com>
Co-authored-by: David Slater <12449447+davidslater@users.noreply.github.com>1 parent b298231 commit 0003e99
14 files changed
Lines changed: 1535 additions & 39 deletions
File tree
- cmd/threat-detect
- pkg
- artifacts
- detector
- prompts
- engine
- specs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
48 | 77 | | |
49 | 78 | | |
50 | 79 | | |
| |||
63 | 92 | | |
64 | 93 | | |
65 | 94 | | |
66 | | - | |
| 95 | + | |
67 | 96 | | |
68 | 97 | | |
69 | 98 | | |
| |||
73 | 102 | | |
74 | 103 | | |
75 | 104 | | |
76 | | - | |
77 | | - | |
78 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
79 | 110 | | |
80 | 111 | | |
81 | 112 | | |
| |||
120 | 151 | | |
121 | 152 | | |
122 | 153 | | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
123 | 246 | | |
124 | 247 | | |
125 | 248 | | |
| |||
167 | 290 | | |
168 | 291 | | |
169 | 292 | | |
170 | | - | |
| 293 | + | |
171 | 294 | | |
172 | 295 | | |
173 | 296 | | |
| |||
0 commit comments