Solved – dumthe vs one-hot encoding – ML for prediction

Firstly, you certainly do not need to add explanatory vectors that are linear combinations of existing explanatory vectors in the model. This leads to identifiability problems, and —at best— these will be handled by the algorithm ignoring one of your inputs. Thus, when working with a factor variable with $k$ categories, you would either use $k$ indicators and no intercept term, or use $k-1$ indicators with an intercept term. There are some advantages and disadvantages to both methods, depending on what you want to do.

Using $k$ indicators and no intercept term: With this method the coefficients corresponding to each of the indicators in your model are interpreted as absolute effects, and are not relative to any base category. This can be useful if you want to plot all of the estimated coefficients for the categories, and you don't want any of the estimated effects to be forced to a baseline of zero. Sometimes you want to see the estimated "total effect" for a particular category with its associated confidence interval, and this is easiest to obtain if you fit the model with this method. (You can still get it from the other method, but it requires some mucking around.)

The down-side of this method is that you have to be very careful when looking at ANOVA outputs and other outputs that compare your model to a null model. Since your specified model has no intercept term, these outputs will generally compare your model to a null model with no intercept term, so that the null model is effectively just white noise. This means that your ANOVA outputs and other similar outputs give comparisons to a really shitty model and the apparent success of your model will appear overstated.

As an example, suppose you build a regression model of height using sex as the explanatory variable, and you code it so that there is no intercept, but there are two indicators, for males and females. In this case, plotting your estimated coefficients for the two categories has a simple and natural interpretation — they each represent estimates of the mean height of that sex. This is a nice aspect of the method. However, if you look at the ANOVA output for the model, it will be made against a null model that postulates heights of all people as white noise (with zero mean) rather than as having a non-zero mean but with no sex difference. This means that your ANOVA outputs will be very misleading.

Using $k-1$ indicators plus an intercept term: This is the opposite case, so the advantages and disadvantages are reversed. Under this method, the coefficients for categories of your factor variable represent relative effects, which are differences in effect size between the present category and the baseline category. This can be useful if you want to look at relative effects, but it means that you do not have direct access to the absolute effects, and these take a bit of mucking around to obtain.

The main upside of this method is that ANOVA output and other model-comparisons will compare your model to a null model with an intercept term, which is usually the comparison you want to make in these cases. This means that the outputs of these model comparisons will show the success or failure of your model against a baseline model that does not assume a zero mean for the response variable.

Continuing the above example, suppose you now build a regression model of height using sex as the explanatory variable, and you code it so that there is an intercept, and then an indicator for females (with males as the baseline category). In this case, plotting your estimated coefficients has a less simple and natural interpretation — you get an estimate of the average height of all people, and an estimate of the mean difference in the height of males and females. This is probably not the ideal presentation of that information, so this is a sub-optimal aspect of this method. On the other hand, if you look at the ANOVA output for the model, it will be made against a null model that allows a non-zero mean height but with no sex difference. This means that your ANOVA outputs will give a useful comparison to a simple base model.