[Joel Stidley (joelst)]

Fixes #1128.

Companion to #1130 (Minja macro scope), which fixes the related extra-<|turn>model symptom. The PRs are independent and can merge in either order.

Problem

NormalizeTools() unwraps OpenAI function tools into a flat Phi-style shape. Gemma 4's format_function_declaration(tool_data) macro reads tool_data['function']['name'], so rendering a tool currently fails with Trying to access property 'name' on null!. Once the nested object is preserved, Gemma also uses Minja's missing upper filter (including through map('upper')) to format JSON Schema types.

Change scope

• shared/api/chat_template.cc and shared/api/chat_template_utils.h: preserve raw OpenAI tools when the exact Gemma function-declaration macro has tool_data as a formal parameter. Default expressions are scanned with nested delimiter and quote handling, so references to tool_data inside defaults do not count. Comments and Jinja raw-block contents are ignored. Existing tool.function and tojson detection is unchanged.
• shared/api/minja.hpp: add an upper filter next to lower. It uppercases ASCII a-z and preserves non-ASCII bytes, so UTF-8 text is not corrupted. This is not Unicode-aware case conversion.
• test/pp_api_test/test_tokenizer_chat.cc: cover direct Gemma rendering, macro signature/detector boundaries, raw-block lookalikes, direct and mapped upper, false-positive dispatch from tool-call history/comments/strings, and continued normalization when the Gemma macro is absent.

No changes are made to tool normalization itself, model templates, Foundry Local, or image decoding. Templates without an existing raw-tool marker or the Gemma macro keep the prior normalized-tool behavior. A downstream package must include a build of onnxruntime-extensions containing this change for the fix to take effect.

Reviewer questions before merge

1. Is the exact macro name a reasonable compatibility boundary for Gemma? The detector now requires tool_data to be a formal parameter, but a future template that renames the macro may need an update.
2. Is ASCII-only upper the intended Minja behavior for this change? It also uses Minja string conversion for non-string values (including Gemma's array-valued schema types); it does not implement full Unicode casing.
3. Are there supported Gemma template variants or downstream integration tests we should add? Current rendering coverage uses the repository fixture. The detector's raw-block handling is unit-tested directly; Minja itself does not currently render {% raw %} tags.

Validation

• Real-runtime check: I built ONNX Runtime GenAI rel-0.17.1 (the version Foundry Local pins) with ORT 1.30.0, pointing FETCHCONTENT_SOURCE_DIR_ONNXRUNTIME_EXTENSIONS at this branch. The base build reproduces Gemma 4 tool calls fail because tool definitions are normalized to the wrong shape #1128 exactly (property 'name' on null). With this PR plus Give each Minja macro call its own variable scope #1130, the published Gemma 4 E2B template renders byte-identical to Jinja2 for user+tools, no-tools, and tool-call history.

• All OrtxTokenizerTest.* tests pass (92/92) on macOS arm64; with Give each Minja macro call its own variable scope #1130 merged in, 94/94.

• Built pp_api_test successfully.

• Latest full tokenizer suite: OrtxTokenizerTest.*, 92/92 passed, including all Gemma, Phi-4, Qwen, and GPT-OSS tests.

• Full pp_api_test on the preceding detector revision passed 180/181; ImageDecoderTest.TestJpegEncoderDecoder failed on a macOS pixel expectation in unchanged image-decoder code (tracked separately in Make JPEG decoder test reliable across supported developer platforms #1127).

• Security review found no vulnerabilities in the change.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🟡 Changes recommended

The detector mishandles raw tags using the supported ~ control marker.

Review effort: Balanced
Findings: 1

Open (1)

• Handle ~ as a raw-tag control marker · New

What changed in this PR

Preserves Gemma 4’s nested tool schema and adds Minja uppercase filtering.

Changes:

• Detects Gemma’s tool-definition macro to bypass lossy normalization.
• Adds ASCII-only upper, including mapped usage.
• Adds detector, rendering, and UTF-8 regression tests.

File	Description
shared/​api/​chat_template.cc	Selects raw tools for Gemma templates.
shared/​api/​chat_template_utils.h	Implements Gemma macro detection.
shared/​api/​minja.hpp	Adds the upper filter.
test/​pp_api_test/​test_tokenizer_chat.cc	Adds regression coverage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation is scoped, preserves existing dispatch behavior, and includes strong boundary and integration coverage.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Handle ~ as a raw-tag control marker

[Copilot]

Copilot review overview

🔵 Needs a closer look

The new custom template scanner affects tool-shape dispatch and warrants final human review despite strong regression coverage.

Review effort: Balanced
Findings: None

[Sayan Shaw (sayanshaw24)]

/azp run onnxruntime-extensions.ci

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

🟢 Approval recommended

No blocking issues remain in the scoped detector, filter implementation, or regression coverage.

0 open findings

🧠 Review effort: Balanced

[Joel Stidley (joelst)]

/azp run onnxruntime-extensions.ci

[azure-pipelines]

Commenter does not have sufficient privileges for PR 1129 in repo microsoft/onnxruntime-extensions