Recently, two different co-workers have used a kind of argument about differences between conditions that seems incorrect to me. Both of these co-workers use statistics, but they are not statisticians. I am a novice in statistics.
In both cases, I argued that, because there was no significant difference between two conditions in an experiment, it was incorrect to make a general claim about these groups with regard to the manipulation. Note that "making a general claim" means something like writing: "Group A used X more often than group B".
My co-workers retorted with: "even though there is no significant difference, the trend is still there" and "even though there is no significant difference, there is still a difference". To me, both of these sound like an equivocation, i.e., they changed the meaning of "difference" from: "a difference that is likely to be the result of something other than chance" (i.e., statistical significance), to "any non-zero difference in measurement between groups".
Was the response of my co-workers correct? I did not take it up with them because they outrank me.
Best Answer
This is a great question; the answer depends a lot on context.
In general I would say you are right: making an unqualified general claim like "group A used X more often than group B" is misleading. It would be better to say something like
in our experiment group A used X more often than group B, but we're very uncertain how this will play out in the general population
or
although group A used X 13% more often than group B in our experiment, our estimate of the difference in the general population is not clear: the plausible values range from A using X 5% less often than group B to A using X 21% more often than group B
or
group A used X 13% more often than group B, but the difference was not statistically significant (95% CI -5% to 21%; p=0.75)
On the other hand: your co-workers are right that in this particular experiment, group A used X more often than group B. However, people rarely care about the participants in a particular experiment; they want to know how your results will generalize to a larger population, and in this case the general answer is that you can't say with confidence whether a randomly selected group A will use X more or less often than a randomly selected group B.
If you needed to make a choice today about whether to use treatment A or treatment B to increase the usage of X, in the absence of any other information or differences in costs etc., then choosing A would be your best bet. But if you wanted be comfortable that you were probably making the right choice, you would need more information.
Note that you should not say "there is no difference between group A and group B in their usage of X", or "group A and group B use X the same amount". This is true neither of the participants in your experiment (where A used X 13% more) or in the general population; in most real-world contexts, you know that there must really be some effect (no matter how slight) of A vs. B; you just don't know which direction it goes.