Research · Reference
Research vocabulary
Names for the moves and measures of a real study: design, sampling, validity, statistics, sources, reasoning, applied methods.
How a study is structured, which shapes how much you can trust its conclusions.
Description
How a study is structured, which shapes how much you can trust its conclusions.
Terms
, testable prediction
A specific, testable prediction the study sets out to confirm or refute.
The default "there is no effect/difference" that you try to disprove.
The group that gets no intervention vs the one that does, compared to isolate the effect.
→ gold standard
Participants randomly assigned to groups, the strongest design for causal claims.
Researchers observe without intervening. Can show correlation, rarely causation.
vs one snapshot ▮
Longitudinal tracks the same subjects over time; cross-sectional is one point in time.
vs numbers & counts
Qual explores meaning (interviews, themes); quant measures (numbers, stats).
Hiding who's in which group from subjects (single) or also researchers (double) to cut bias.
Who was studied, and whether they represent who you care about.
Description
Who was studied, and whether they represent who you care about.
Terms
The whole group you want conclusions about (e.g. all adults in the US).
The subset actually studied. Bigger, well-chosen samples give more reliable estimates.
chance of selection
Selecting subjects by chance so the sample mirrors the population.
population's mix
A sample whose makeup matches the population on key traits.
got included
When the sample systematically over/under-represents some group, distorting results.
beyond the sample?
How well findings extend from the sample to the wider world.
What's measured, what's manipulated, and what might be quietly distorting things.
Description
What's measured, what's manipulated, and what might be quietly distorting things.
Terms
you change
The factor the researcher manipulates to see its effect.
you measure
The outcome measured to detect the effect of the independent variable.
factor
An unaccounted variable that affects both cause and effect, faking a relationship.
Two variables move together. Says nothing about cause on its own.
One variable actually produces the change in another. Needs strong design to claim.
Whether the study measures what it claims to measure (the right target).
if repeated
Whether the measurement is consistent and repeatable.
The numbers behind the claim. These let you judge whether a result is real or noise.
Description
The numbers behind the claim. These let you judge whether a result is real or noise.
Terms
most frequent
Three "centers." Median resists outliers; mean doesn't.
The shape of how values spread, the bell curve is the normal distribution.
around the mean
How spread out values are. Small = clustered; large = dispersed.
Roughly: the chance of seeing this result if the null were true. Low = unlikely to be flukey. Not "importance."
to chance alone
A result unlikely to be random, not the same as large or meaningful.
How big the difference actually is, the practical magnitude, beyond significance.
A range the true value likely falls within. Wide = uncertain; narrow = precise.
The plus/minus around a poll or estimate from sampling.
A model fitting a line/curve to data to estimate relationships and predict.
Where knowledge comes from and how vetted it is.
Description
Where knowledge comes from and how vetted it is.
Terms
vs reporting on it
Primary = the original work; secondary = summaries/coverage. Cite primary for claims.
Expert vetting before publication, the quality bar for scientific work.
peer review
Early, unvetted release. Useful but flag it as not yet reviewed.
A study that statistically combines many studies for a pooled, stronger estimate.
of all evidence
A rigorous, transparent survey of all studies on a question.
References to a work; high citation counts hint at influence (not always quality).
The thinking traps and the tools to avoid them.
Description
The thinking traps and the tools to avoid them.
Terms
vs specific→general
Deductive applies a rule to a case; inductive infers a pattern from cases.
proven wrong?
A claim is scientific only if there's evidence that could refute it.
frequency
The underlying rate a result should be judged against. Ignoring it = base-rate fallacy.
supporting evidence
The tendency to notice evidence that fits what you already believe.
more likely published
The literature skews toward exciting/positive findings; null results vanish.
when repeated?
Independent repetition getting the same result, the real test of a finding.
of the other side
Engaging the best form of an opposing argument, not a strawman.
favorable data
Highlighting supportive data while ignoring the rest.
Research techniques you'll meet in product, design, and market work.
Description
Research techniques you'll meet in product, design, and market work.
Terms
Showing two versions to split traffic to see which performs better. An online RCT.
Watching real users attempt tasks to find friction in a design.
at scale
Standardized questions to many people. Watch wording and sampling bias.
In-depth one-on-one conversation. Rich qualitative insight, small n.
Tracking a group who share a start point (e.g. signup month) over time.
Combining multiple methods/sources so they corroborate each other.