[BUG] Subdirectory .gitignore patterns with leading "/" not applied, causing token overflow on large repos

[jamclell-aws]

Describe the bug

Subdirectory .gitignore patterns with leading / are not applied, causing token overflow on large repos

Description

When the working directory contains subdirectories with their own .gitignore files, anchored patterns (those starting with /, e.g. /node_modules) are not correctly matched. This causes the file listing tool to return tens of thousands of files that should be excluded, which in turn causes agents to exceed the model's context window.

Root Cause

Two issues in threat_composer_list_workdir_files_gitignore_filtered.py:

1. Broken pattern adjustment for subdirectory .gitignore files

In load_gitignore_patterns(), when a .gitignore is in a subdirectory, patterns are prefixed with the relative directory path. However, anchored patterns with a leading / are not stripped before prefixing:

# Before (broken): produces "subdir//node_modules" — double slash breaks matching
adjusted_patterns.append(f"{relative_dir}/{pattern}")

# After (fixed): strip leading '/' since relative_dir already provides anchoring
clean = pattern.lstrip("/")
adjusted_patterns.append(f"{relative_dir}/{clean}")

A .gitignore at GenericServiceCDK/.gitignore containing /node_modules would produce the pattern GenericServiceCDK//node_modules, which pathspec cannot match against file paths like GenericServiceCDK/node_modules/aws-cdk/....

2. No directory pruning during os.walk (performance issue)

Separately from the pattern matching bug, the tool collects all files first via os.walk and filters afterward. Even with correct gitignore matching, this means os.walk still descends into every directory to enumerate files before the filter discards them. The fix passes the exclusion specs into collect_files() and prunes matching directories in-place during the walk, preventing descent into ignored subtrees entirely. This is a performance optimization — it does not affect what files are returned, only how quickly the tool completes on large repos.

Impact

• Repos with node_modules, vendor, target, or other large ignored directories in subdirectories will fail with token overflow errors
• The file listing tool unnecessarily walks tens of thousands of files that will be discarded
• In the test case: 30,730 of 31,215 files (98.4%) were in node_modules and should have been excluded

Proposed Fix

1. Strip leading / from anchored patterns before prepending the relative directory prefix
2. Pass gitignore and hardcoded exclusion specs to collect_files() and prune directories in-place during os.walk to avoid traversing ignored subtrees. Pruning is negation-aware: directories referenced by ! patterns (e.g. !node_modules/special-package) are not pruned, so the negation can be evaluated file-by-file during the filter pass.

Affected Code

• packages/threat-composer-ai/src/threat_composer_ai/tools/threat_composer_list_workdir_files_gitignore_filtered.py
  • load_gitignore_patterns() — pattern adjustment logic
  • collect_files() — directory walk without pruning

Expected Behavior

.gitignore files in subdirectories should be fully honored, including anchored patterns like /node_modules. The file listing tool should exclude the same files that git would, regardless of whether the .gitignore is at the working directory root or in a nested subdirectory.

Current Behavior

Surfaces as "prompt is too long" ValidationException

Reproduction Steps

Steps to Reproduce

1. Point threat-composer-ai-cli at a directory where:

   • The .gitignore is in a subdirectory (not at the working directory root)
   • That .gitignore contains anchored patterns like /node_modules
   • The node_modules directory contains a large number of files (e.g. 30K+)

2. Run:

uv run threat-composer-ai-cli /path/to/repo/src \
  --aws-profile my-profile \
  --aws-model-id us.anthropic.claude-opus-4-6-v1

Error Message

ERROR    🔧 SYSTEM | ❌ Error (Run with --verbose to see full details):
         An error occurred (ValidationException) when calling the
         ConverseStream operation: The model returned the following
         errors: prompt is too long: 1092667 tokens > 1000000 maximum

Possible Solution

No response

Additional Information/Context

No response