I perform prequential evaluation like this: start with a training set, classify a number of examples, then add the correctly classified examples in the training set and continue to classifying the next number of unseen examples. Is this supposed to increase performance as examples are added to the training set or this doesn't apply to every case? By increasing performance I mean if the average F1, not only accuracy, of the second piece of unseen examples must be higher than that of the first or is it possible for a latter piece to have worse average F1 than a former? And if it has worse, what does this could be possibly mean? Could it mean a problem with training data?
This paper Sentiment Knowledge Discovery in Twitter Streaming Data describes prequential evaluation. It experiments with Naive Bayes and not SVMs, probably for the reasons mentioned in comments below.
Thanks a lot in advance!
Best Answer
I would say that "in theory" the quality should increase until it reaches a saturation point, as it usually happens when you apply Active Learning. This should be true in general.
If adding more information you consistently get worse results, that would suggest that the new data is wrongly labelled, noisy or that it contradicts the previously seen one for some reason (e.g., some information might have changed over time).
I hope this helps,
Similar Posts:
- Solved – training and validation accuracy increasing – XGBoost
- Solved – CV for model parameter tuning AND then model evaluation
- Solved – CV for model parameter tuning AND then model evaluation
- Solved – Can overfitting be a good thing in some cases
- Solved – Do I need to remove duplicated image when building a CNN classification model