Skip to content

Add hash_word_ngrams() and hash_char_ngrams() - #3181

Merged
davisking merged 3 commits into
davisking:masterfrom
arrufat:ngram-hashing
Nov 3, 2026
Merged

davisking merged 3 commits into
davisking:masterfrom
arrufat:ngram-hashing

Conversation

@arrufat

@arrufat arrufat commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

This PR adds hash_word_ngrams() and hash_char_ngrams(), as described in #3180: they hash the word and character n-grams of a tokenized document into a sparse vector of counts, ready for tfidf_transformer.

My page classifier already uses them instead of its own hashing code, and it gets exactly the same model.

Closes #3180

davisking commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

@arrufat I pushed a8e6097 with a few small changes: clarify the token/index/count type requirements, document that character n-grams operate on bytes (with UTF-8 tests), and reject n-gram lengths above INT_MAX before passing them to murmur_hash3(). The value type is required to hold the resulting bucket counts.

The focused feature-hashing and TF-IDF tests passed all 123 checks with assertions and ASan/UBSan enabled. Are you okay with these changes?

[edit]
Lol, in case that doesn't sound like me it's because I told astra to push the commit. Anyway, thanks for another PR. This change cool with you?

@arrufat

arrufat commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Yes, thanks! I pushed a small follow-up that moves the bucket range check into the asserts at the top of each function, so it runs once per call instead of once per n-gram.

@davisking
davisking merged commit 4ff3c75 into davisking:master Nov 3, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature hashing for text n-grams

2 participants