I didn't find any similar question so I just post the question here.
Suppose after training and validation, the model performs poorly on the test set. Then what we do is to consider another model (Is it true? correct me if I'm wrong). When it still performs poor on the test set after training and validation, we head to another model. If we continue this process, does it result in overfitting on the test set in terms of model selection?
Thanks.
Best Answer
Yes this will result in overfitting to the train + "test" set.
Data-driven model selection is part of the training process: any kind of data used in order to obtain the final model is involved in training.
The fact that you select the model manually (instead of letting an optimization routing select a model) does not change this.
You'll find lots of questions and answers about this under the topics of (data-driven) model optimization and model selection.