Skip to content

Add text_classification_ex.cpp - #3182

Merged
davisking merged 14 commits into
davisking:masterfrom
arrufat:text-classification-example
Nov 14, 2026
Merged

davisking merged 14 commits into
davisking:masterfrom
arrufat:text-classification-example

Conversation

@arrufat

@arrufat arrufat commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

dlib has the pieces to classify text, but no example that puts them together, and for my page classifier I had to find the right order and the pitfalls by trial and error. This example is that pipeline on the SMS Spam Collection.

To explain how a model works, you want to show what's inside it, so word_ngrams() and char_ngrams() list the n-grams the hash functions count, and the example uses them to show which n-grams the SVM weighs most.

auto_train_linear_classifier() is there because the example and my page classifier repeated the same recipe around the SVM, and the easy mistakes in it cost accuracy without any error: forgetting to balance C on unbalanced classes, or fitting Platt scaling on folds that follow the sample order. #3183 documents the second one for train_probabilistic_decision_function().

@arrufat

arrufat commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

A question about text_features, the class in the example that turns a message into the SVM's sparse vector: would you want something like it in dlib, or should it stay in user code?

As written it's mostly policy for this dataset: how to tokenize, which n-gram ranges, the bucket count, and the TF-IDF settings for each family. My page classifier couldn't use it as is, since it tokenizes differently, caps each vocabulary and appends some extra features after the TF-IDF columns. So my inclination is to leave it in the example.

If you'd like it in dlib, it could be a hashed_tfidf_vectorizer (name to be decided) that takes the delimiters or a tokenizer, the word and char n-gram ranges, the number of buckets and a tfidf_transformer per family. It would serialize those settings along with the learned vocabularies, so a saved model can't be loaded with different ones, which the example only warns about in a comment. That would cover the example and most of what the page classifier does. The smaller generic piece is appending one sparse vector after another with an index offset, which both of them do by hand.

@davisking

Copy link
Copy Markdown
Owner

Yeah I would leave the feature extractor in the example. Since invariably someone doing that kind of thing wants to mess around with it themselves. Or very often anyway.

The trainer class had a solver, a list of C candidates, a fold count and
two train() overloads for what is used as a single call. It is now a
function next to auto_train_rbf_classifier() that chooses C from a fixed
grid by 5-fold cross-validation and returns a Platt-scaled classifier.
The function returns probabilities, but chose C by the sign of the held out
scores, which can favor strong regularization.  text_classification_ex.cpp
picked C = 0.1 that way.  It now picks C = 10, and its test log loss goes from
0.040 to 0.036.
@arrufat

arrufat commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I changed how auto_train_linear_classifier() picks C. Balanced accuracy only looks at the sign of the held out scores, but the function returns probabilities. Now each C gets its own sigmoid from learn_platt_scaling(), and the C with the lowest held out log loss wins.

You can see the difference in the example. Balanced accuracy picks C = 0.1 and log loss picks C = 10, which takes the test log loss from 0.040 to 0.036. Over 20 random splits, mean test accuracy goes from 98.5% to 99.1%, and the mean log loss from 0.034 to 0.030.

The gap grows as the classes get more unbalanced. Keeping 10% of the spam, balanced accuracy drops to C = 0.1 or below, and accuracy goes from 98.9% to 99.8%.

Choosing C by log loss also beat balanced accuracy in our internal experiments on our own data.

arrufat and others added 4 commits November 9, 2026 09:58
C was chosen from a fixed list between 0.001 and 1000, so a best C outside
it went unnoticed.  The search now starts at 0.1, 1 and 10 and moves to the
next power of 10 while the lowest held out log loss is at either end, within
1e-6 and 1e6.
It took std::vectors, so a caller training on some of its samples had to copy
them.  It now takes anything mat() accepts, such as a rowm() view, as
linear_classifier_trainer::train() did.
Once the best of 0.1, 1 and 10 is known, the search can only move one way,
so it keeps the best fit instead of a map of every fit.  Also drop two
sentences from the n-gram contracts that point at other tools, fold a
duplicated training into test_sparse, and include <iomanip> in the example.
@davisking

Copy link
Copy Markdown
Owner

Yeah nice, that's a good way to do it. Thanks for another PR :D

@davisking
davisking merged commit f912c88 into davisking:master Nov 14, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants