Research vocabulary
Vocabulary
Names for the moves and measures of a real study.
Juan M. Cornejo
Notes, no. 02
Names for the moves and measures of a real study: design, sampling, validity, statistics, sources, reasoning, applied methods.
1Research vocabulary
Part 01
How a study is structured, which shapes how much you can trust its conclusions.
How a study is structured, which shapes how much you can trust its conclusions.
A specific, testable prediction the study sets out to confirm or refute.
The default "there is no effect/difference" that you try to disprove.
The group that gets no intervention vs the one that does, compared to isolate the effect.
Participants randomly assigned to groups, the strongest design for causal claims.
Researchers observe without intervening. Can show correlation, rarely causation.
Longitudinal tracks the same subjects over time; cross-sectional is one point in time.
Qual explores meaning (interviews, themes); quant measures (numbers, stats).
Hiding who's in which group from subjects (single) or also researchers (double) to cut bias.
Part 02
Who was studied, and whether they represent who you care about.
Who was studied, and whether they represent who you care about.
The whole group you want conclusions about (e.g. all adults in the US).
The subset actually studied. Bigger, well-chosen samples give more reliable estimates.
Selecting subjects by chance so the sample mirrors the population.
A sample whose makeup matches the population on key traits.
When the sample systematically over/under-represents some group, distorting results.
How well findings extend from the sample to the wider world.
Part 03
What's measured, what's manipulated, and what might be quietly distorting things.
What's measured, what's manipulated, and what might be quietly distorting things.
The factor the researcher manipulates to see its effect.
The outcome measured to detect the effect of the independent variable.
An unaccounted variable that affects both cause and effect, faking a relationship.
Two variables move together. Says nothing about cause on its own.
One variable actually produces the change in another. Needs strong design to claim.
Whether the study measures what it claims to measure (the right target).
Whether the measurement is consistent and repeatable.
Part 04
The numbers behind the claim. These let you judge whether a result is real or noise.
The tones of one photograph as a histogram, in a photo developer. Most values bunch in the dark half, with a long tail to the light: the spread is skewed.
The numbers behind the claim. These let you judge whether a result is real or noise.
Three "centers." Median resists outliers; mean doesn't.
The shape of how values spread, the bell curve is the normal distribution.
How spread out values are. Small = clustered; large = dispersed.
Roughly: the chance of seeing this result if the null were true. Low = unlikely to be flukey. Not "importance."
A result unlikely to be random, not the same as large or meaningful.
How big the difference actually is, the practical magnitude, beyond significance.
A range the true value likely falls within. Wide = uncertain; narrow = precise.
The plus/minus around a poll or estimate from sampling.
A model fitting a line/curve to data to estimate relationships and predict.
Part 05
Where knowledge comes from and how vetted it is.
Where knowledge comes from and how vetted it is.
Primary = the original work; secondary = summaries/coverage. Cite primary for claims.
Expert vetting before publication, the quality bar for scientific work.
Early, unvetted release. Useful but flag it as not yet reviewed.
A study that statistically combines many studies for a pooled, stronger estimate.
A rigorous, transparent survey of all studies on a question.
References to a work; high citation counts hint at influence (not always quality).
Part 06
The thinking traps and the tools to avoid them.
The thinking traps and the tools to avoid them.
Deductive applies a rule to a case; inductive infers a pattern from cases.
A claim is scientific only if there's evidence that could refute it.
The underlying rate a result should be judged against. Ignoring it = base-rate fallacy.
The tendency to notice evidence that fits what you already believe.
The literature skews toward exciting/positive findings; null results vanish.
Independent repetition getting the same result, the real test of a finding.
Engaging the best form of an opposing argument, not a strawman.
Highlighting supportive data while ignoring the rest.
Part 07
Research techniques you'll meet in product, design, and market work.
Research techniques you'll meet in product, design, and market work.
Showing two versions to split traffic to see which performs better. An online RCT.
Watching real users attempt tasks to find friction in a design.
Standardized questions to many people. Watch wording and sampling bias.
In-depth one-on-one conversation. Rich qualitative insight, small n.
Tracking a group who share a start point (e.g. signup month) over time.
Combining multiple methods/sources so they corroborate each other.
Elsewhere
another notebook Brand vocabulary
AI vocabulary
Brand vocabulary