Reading public evidence · Survey claims

What does margin of error mean in a survey?

“Most residents want longer library hours: 52% support the proposal.” The headline looks straightforward. Below the chart, smaller print gives a margin of error of plus or minus three percentage points. Can you still write “most residents”?

A margin of sampling error describes uncertainty associated with estimating a population result from a sample, under the survey’s method and assumptions. It is not an allowance for every possible mistake. Before using it, check which estimate, population and confidence level it belongs to. Then ask whether the interval supports the particular claim you want to make.

Work on the headline before opening a calculator

Invented teaching example: a local library report supplies an estimate of 52%, a margin of three percentage points at 95% confidence, and a probability sample of 1,000 adult residents. For this first exercise, assume the producer’s stated margin is appropriate for this specific estimate. These are fictional inputs, not the results of a survey conducted by OSINT Jet.

The reported interval runs from 49% to 55%: subtract three from 52, then add three. It includes values below and above 50%. This interval therefore does not establish a population majority at the stated confidence level. That is a limit on the inference, not proof that a majority is absent.

Compare three possible edits:

• Too strong: “Residents clearly back the proposal.” The supplied interval does not establish the implied majority.
• Too dismissive: “The survey tells us nothing.” It supplies an estimate and uncertainty; that is useful information.
• Supported by the exercise: “The survey estimates support at 52%, with a reported 95% interval of 49%–55%; a population majority is not established by this interval.”

Keep the estimate in the sentence. Reporting only the lower endpoint would be just as selective as reporting only the upper endpoint. If the source rounds the percentage or margin, retain that rounding; do not manufacture extra decimal places.

What the confidence level adds

Confidence describes the procedure: under its assumptions, intervals from repeated samples contain the population value at the stated rate. It does not certify the whole study. See the BLS explanation of confidence intervals.

For the library example, keep “95%” attached to the interval statement. Do not transfer it to unrelated conclusions such as “95% certain the extra hours will improve attendance.” The survey question concerns stated support; the effect on future attendance would require different evidence.

Does this survey support a sampling margin at all?

Probability sampling uses known selection probabilities. AAPOR’s guide to understanding polls warns against ordinary sampling margins for non-probability samples. A model-based uncertainty measure needs its own method and assumptions identified.

Now change the fictional library packet: the 1,000 responses came from a button on its social page. Keep the 52% but remove the probability-sample description. You cannot repair the inference by feeding 1,000 into a generic margin-of-error calculator. The missing fact is how participants relate to the target population.

A narrower sentence might be: “Of the responses collected through the library’s social-page poll, 52% supported the proposal.” Add the collection dates and any known response controls. Whether those responses represent all adult residents remains unresolved.

“Online” alone is not the diagnosis. AAPOR’s survey research practices explain that online surveys can use either probability or non-probability recruitment. Read how people were selected, not just where they clicked.

The margin does not cover the whole investigation

Statistics Canada separates sampling uncertainty from other sources of survey error, including coverage, nonresponse and question-related problems. A small sampling margin does not account for all those risks.

Apply that distinction to the library claim. If the sampling list excludes residents without library cards, the intended population and the people reachable through the list may differ. If the question says “Do you support longer hours to help struggling families?”, its wording also needs examination. If the published chart omits a “depends on the cost” option, the chart may not preserve what respondents were offered.

These are questions to investigate, not automatic reasons to declare the survey false. Save the questionnaire and method note. Record exactly which relevant detail is missing. “No recruitment description found in the published report” is a reproducible observation; “bad survey” is not a substitute for one.

A subgroup needs its own uncertainty

Suppose the report also says that 58% of respondents under 25 support the proposal. Only 100 people in the sample belong to that group, and no subgroup margin is supplied. You cannot simply copy the full-sample three-point margin onto the 58%. AAPOR notes that subgroup margins generally increase as subgroup samples get smaller. Ask for the margin or standard error appropriate to that estimate and design.

In your notes, keep the full-sample result and subgroup result on separate lines. Mark the second line “uncertainty not supplied.” Do not silently convert the subgroup result into a finding about all younger residents, and do not invent a subgroup margin from the headcount alone when weighting or design details are unknown.

Overlapping intervals alone do not establish that a difference is statistically insignificant. BLS calls for a formal comparison; use a method appropriate to the estimates and any dependence between them.

A short exercise you can reproduce

Use a second fictional estimate: 61% with a valid, supplied margin of four percentage points at 95% confidence. The interval is 57%–65%. It lies entirely above 50%, so the interval supports a population-majority statement under the stated survey assumptions. It still does not establish that the proposed service change will work.

Now replace “adult residents” with “current library members.” The arithmetic stays the same, but the population in the conclusion must change. A correct interval can still accompany the wrong headline if you broaden the group it describes.

Save enough detail to make the claim reviewable

Before quoting a survey, fill in this compact note. Leave a visible blank where the source does not provide an answer:

Claim and intended population:
Original question and response options:
Fieldwork dates / report version / source link:
Recruitment method and sample size for this estimate:
Estimate / margin / confidence level:
Weighting or design information supplied:
Missing detail that could change the conclusion:
Sentence the evidence supports:

AAPOR’s transparency guidance, linked above, calls for disclosure of the method, questions, population, sample and weighting. Your note turns those details into something useful: a reader can see whether the reported margin belongs to the number being quoted.

For a larger investigation, use OSINT Jet’s report template to keep this numerical finding beside its sources and open questions. The guide to reviewing a finished report also helps distinguish an observed result from the action someone recommends. Neither a precise-looking percentage nor a polished chart should do that reasoning for you.