Lesson 8 of 15 · free to read
When evaluating studies, consider non-response bias, confounding variables, and the distinction between correlation and causation to judge whether conclusions are valid.
A study of 10,000 part-time students found that two different samples gave wildly different book-spending averages—one around $150, the other around $40. Neither reflected the true population. How can you tell when a study's numbers are trustworthy?
The idea
To judge a study, check for non-response bias, confounding, and whether results show causation or mere correlation.
When you read a study, you need to evaluate whether its conclusions are valid. Several problems can undermine a study's results.
Definition
confounding
When the effects of multiple factors on a response cannot be separated, making it difficult or impossible to draw valid conclusions about the effect of each factor.
When several possible causes are tangled together, you can't tell which one really produced the result.
Another problem is non-response: when people refuse to participate, the collected responses may no longer represent the population. Often, people with strong opinions are the ones who answer surveys, skewing the results.
Causality is also tricky: a relationship between two variables does not mean one causes the other. They may be correlated because both relate to a third variable. Also watch for self-funded studies (done to support a claim) and misleading data displays.
A sample can be biased or unbiased. A biased sample is not representative of the population because some members had little or no chance of being chosen. An unbiased sample is representative because every member had a fair chance.
Vocabulary· 2 terms
Worked example
ABC College has 10,000 part-time students. Two samples were taken to estimate average book spending.
First sample: convenience sample of ten students from an organic chemistry class.
These students are likely science-oriented and taking expensive courses, so they probably spend more than the average part-time student.
Second sample: every fifth senior citizen from a list of P.E. class students.
Senior citizens taking P.E. for health or interest likely spend much less on books than the average part-time student.
Check if either sample is representative of all 10,000 part-time students.
Both samples are biased because not all students had a chance to be included, and the groups are not typical of the whole population.
Neither sample is representative; both are biased.
Which of these samples is biased?
A third sample was then taken: ten part-time students, one from each of ten disciplines (chemistry, math, English, psychology, sociology, history, nursing, physical education, art, early childhood development), each chosen by simple random sampling. This sample is unbiased, though a larger sample would be recommended.
Easily confused
A biased sample is just one that's small or has a wide spread of values.
Bias is about who gets a chance to be chosen, not about size or spread. In a biased sample, some population members have little or no chance of being included.
Even a large sample can be biased if it systematically excludes part of the population, like only surveying chemistry students.
Why can't we conclude that ice cream sales cause drownings?
Recap
Say each point to yourself, then tap it to check.
Non-response bias: strong-opinion people answer, skewing results.
Confounding: multiple factors' effects can't be separated.
Advertisement
Clear the page · $6.67/moSource
Adapted from Introductory Statistics 2e, © 2023 OpenStax, Rice University
Barbara Illowsky and Susan Dean
Licensed under CC BY 4.0. This course is an adaptation, not the original work.
Access for free at openstax.org/details/books/introductory-statistics-2e