Repository navigation
Add hash_word_ngrams() and hash_char_ngrams() - #3181
Conversation
e7ca1f5 to
f985fa6
Compare
|
@arrufat I pushed a8e6097 with a few small changes: clarify the token/index/count type requirements, document that character n-grams operate on bytes (with UTF-8 tests), and reject n-gram lengths above INT_MAX before passing them to murmur_hash3(). The value type is required to hold the resulting bucket counts. The focused feature-hashing and TF-IDF tests passed all 123 checks with assertions and ASan/UBSan enabled. Are you okay with these changes? [edit] |
|
Yes, thanks! I pushed a small follow-up that moves the bucket range check into the asserts at the top of each function, so it runs once per call instead of once per n-gram. |
This PR adds
hash_word_ngrams()andhash_char_ngrams(), as described in #3180: they hash the word and character n-grams of a tokenized document into a sparse vector of counts, ready fortfidf_transformer.My page classifier already uses them instead of its own hashing code, and it gets exactly the same model.
Closes #3180