Solved – How to compare results of a questionnaire consisting of 40 Likert items, before and after an intervention

In my research (medical education field), I had 32 participants answering a questionnaire consisting of 40 Likert items rating their level of confidence in different aspects of treating a certain disease (1= Not at all confident, 6= Completely confident). The participants then undertook a course, and rated their confidence again after the course, using the same questionnaire (exactly same 40 Likert items). My hypothesis is that their self-rated confidence after the course is significantly higher than before the course.

So my question, as a total beginner in statistics, is what is the most appropriate method on comparing my sample's confidence, before and after this course. Should I calculate the mean of each participant, and then the mean of the means before and after and compare those using a paired t-test, or am I thinking completely wrong here?

Your approach seems reasonable. One difficulty is that interpreting the mean as a level of confidence implicitly relies on the fact that confidence varies in the same way across the board (as opposed to being specific to some items/aspects of the disease). Your analysis provides no way to check that or look at more fine-grained variation but it is certainly not completely wrong.

Apart from that, there is a whole lot of things you could do but whether it is appropriate or makes sense to spend time doing anything fancy with such a small sample size is somewhat doubtful. A common way to treat this type of data would be to use exploratory factor analysis to understand the structure of the questionnaire or confirmatory factor analysis/latent variable modeling to see if all 40 items really do reflect a single underlying construct (“confidence”) and to extract a cleaner measure of this construct. However, all this would typically require a (much) larger sample size. Still, you could at least look at the correlations between items (at each time point) and hope that they are not ridiculously low but with 30 participants you would in any case expect huge sampling variability. MANOVA might also be interesting, especially if you want to include other explanatory variables (say student characteristics).

Similar Posts:

Rate this post

Leave a Comment