Solved – How to treat illogical survey responses

I have submitted a survey to a sample of artists. One of the question was to indicate the percentage of income derived by: artistic activity, government support, private pension, activities not related with arts. About 65% of the individuals have replied such that the sum of the percentage is 100. The others don’t: for example, there are who answers that 70% of their income derived by his/her artistic activities and 60% by income government, and so on.
My question is: how should I treat these observations? Should I delete, modify or keep them?
Thank you!

This is a good situation for a sensitivity analysis. Analyze your data in each of three ways —

  1. As they are
  2. After excluding "the illogicals", i.e., people whose percentages don't add up to 100 (or 100 +/- 10)
  3. After adjusting where necessary so that each person's percentages add up to 100

Then compare results, sharing any rationale you can develop as to which results might be more accurate, or more accurate in certain respects.

You can also investigate the range of ways in which the logicals and illogicals differ, if any. Do the illogicals tend to report higher incomes? To show greater support for certain ideas or programs? To skip more questions? To evince more bias in the sense of straightlining or disproportionately choosing middle responses or extreme responses?

With about 400 or these illogicals, you have enough data even to assess the relationship between degree of illogicality and degree of a given type of bias. Something like a dose-response relationship.

What you learn from these investigations might be fed back into your plan for dealing with the illogicals when it comes to the main analyses of interest.

Similar Posts:

Rate this post

Leave a Comment