[Mustapha Jaber (mustjab)]

Summary

• Translate tokenizer JSON exceptions into explicit error codes and error text instead of letting them escape extern-C creation APIs.
• Share the default/options path and initialize outputs to null on failure; valid loading and tokenization remain unchanged.
• Add direct blob/file regressions for malformed syntax, invalid JSON types, and output ownership.

Validation

The direct malformed-vocabulary C API test fails before the fix because a JSON exception escapes. All 15 targeted native creation, parity, option, lookup, and stream tests pass afterward.

Relevant to microsoft/onnxruntime-genai#2800: MSVC /EHsc assumes extern-C calls do not throw. This fixes a reproduced production boundary defect without changing compiler flags, workflows, or GenAI negative assertions. Actual Windows confirmation of the observed crash remains pending.

[Copilot]

🔵 Needs a closer look

Native Windows execution of the C API regressions remains pending, so the exception translation has not yet been confirmed to resolve the production crash under MSVC /EHsc.

No open issues 🎉

What changed in this PR

This PR makes tokenizer creation report JSON exceptions through the C API’s error-code and error-message contract, rather than allowing them to escape to callers.

Changes:

• Share the default and options-based creation path and catch JSON exceptions for file and blob loading.
• Validate output pointers and initialize valid outputs to null before loading.
• Document the failure contract and add regressions for malformed JSON, invalid types, and output ownership.

File	Description
test/​pp_api_test/​test_tokenizer_blob.cc	Adds file/blob JSON failure and output-pointer regression tests.
shared/​api/​c_api_tokenizer.cc	Unifies creation paths and translates JSON exceptions into C API errors.
include/​ortx_tokenizer.h	Documents JSON error reporting and null outputs on failure.

_{Review effort: Balanced}