Repository navigation
Add text_classification_ex.cpp - #3182
Conversation
|
A question about As written it's mostly policy for this dataset: how to tokenize, which n-gram ranges, the bucket count, and the TF-IDF settings for each family. My page classifier couldn't use it as is, since it tokenizes differently, caps each vocabulary and appends some extra features after the TF-IDF columns. So my inclination is to leave it in the example. If you'd like it in dlib, it could be a |
|
Yeah I would leave the feature extractor in the example. Since invariably someone doing that kind of thing wants to mess around with it themselves. Or very often anyway. |
The trainer class had a solver, a list of C candidates, a fold count and two train() overloads for what is used as a single call. It is now a function next to auto_train_rbf_classifier() that chooses C from a fixed grid by 5-fold cross-validation and returns a Platt-scaled classifier.
The function returns probabilities, but chose C by the sign of the held out scores, which can favor strong regularization. text_classification_ex.cpp picked C = 0.1 that way. It now picks C = 10, and its test log loss goes from 0.040 to 0.036.
|
I changed how You can see the difference in the example. Balanced accuracy picks C = 0.1 and log loss picks C = 10, which takes the test log loss from 0.040 to 0.036. Over 20 random splits, mean test accuracy goes from 98.5% to 99.1%, and the mean log loss from 0.034 to 0.030. The gap grows as the classes get more unbalanced. Keeping 10% of the spam, balanced accuracy drops to C = 0.1 or below, and accuracy goes from 98.9% to 99.8%. Choosing C by log loss also beat balanced accuracy in our internal experiments on our own data. |
C was chosen from a fixed list between 0.001 and 1000, so a best C outside it went unnoticed. The search now starts at 0.1, 1 and 10 and moves to the next power of 10 while the lowest held out log loss is at either end, within 1e-6 and 1e6.
It took std::vectors, so a caller training on some of its samples had to copy them. It now takes anything mat() accepts, such as a rowm() view, as linear_classifier_trainer::train() did.
Once the best of 0.1, 1 and 10 is known, the search can only move one way, so it keeps the best fit instead of a map of every fit. Also drop two sentences from the n-gram contracts that point at other tools, fold a duplicated training into test_sparse, and include <iomanip> in the example.
|
Yeah nice, that's a good way to do it. Thanks for another PR :D |
dlib has the pieces to classify text, but no example that puts them together, and for my page classifier I had to find the right order and the pitfalls by trial and error. This example is that pipeline on the SMS Spam Collection.
To explain how a model works, you want to show what's inside it, so
word_ngrams()andchar_ngrams()list the n-grams the hash functions count, and the example uses them to show which n-grams the SVM weighs most.auto_train_linear_classifier()is there because the example and my page classifier repeated the same recipe around the SVM, and the easy mistakes in it cost accuracy without any error: forgetting to balance C on unbalanced classes, or fitting Platt scaling on folds that follow the sample order. #3183 documents the second one fortrain_probabilistic_decision_function().