Statistical Tests & Research Approaches

THE LOGIC OF A SIGNIFICANCE TEST

A.1 Describing a group, or generalising from it

ch03 teaches descriptive statistics, which describe only the group actually measured: the mean, SD and correlation of this class. Inferential statistics go one step further and use a sample to draw conclusions about the population it came from. Every test in this note (z, t, F, χ², U, H) is inferential.

Inference estimates parameters from statistics. A sample can mislead only by chance, and a significance test is a rule for deciding how much chance to tolerate.

Trap: the Greek ρ in this table is the population correlation. Spearman's ρ (Part G) is a different quantity that happens to share the letter.

A.2 The sampling distribution and the standard error

Draw a random sample of 100 pupils and compute its mean. Draw another 100 and the mean will differ a little. Repeat many times and the means themselves form a distribution, the sampling distribution of the mean. Its standard deviation has its own name, the standard error (SE).

SE of the mean = σ / √N          (σ unknown in practice → estimated from s)

Two consequences that examiners use:

• The SE shrinks as N grows, but only as √N, so quadrupling N halves the SE.
• The SD of the scores does not shrink as N grows. A bigger sample estimates the spread more accurately; it does not reduce it.

Worked example A2 — what a bigger sample buys
Population SD σ = 10.
N = 100:  SE = 10 / √100 = 10 / 10 = 1
N = 400:  SE = 10 / √400 = 10 / 20 = 0.5    (four times the sample, half the SE)
The SD of the pupils' scores is still 10. Only the SE has fallen.

Central limit theorem. For large random samples, conventionally N > 30, sample means are normally distributed about μ with SD equal to the SE, whatever the shape of the population. That is why a z-test can be used on a large sample even when the scores themselves are skewed. The figure 30 is a convention, not part of the theorem.

Small sample means N ≤ 30 by the same convention. A small sample with σ unknown uses the t distribution, not z (Part C).

Trap: "a larger sample reduces the SD of the scores" is false. Only the SE falls.

A.3 Two hypotheses

A directional H₁ ("higher", "lower") takes a one-tailed test. A non-directional H₁ ("there will be a difference") takes a two-tailed test. The choice is made before the data are seen.

A.4 Level of significance, critical region and the decision

• Level of significance (α) is the probability of a Type I error (rejecting a true H₀). It is fixed in advance, conventionally at .05 or .01 (sometimes .001).
• The critical value marks off the critical (rejection) region, the tail area whose total is α. In a two-tailed test at .05 that is 2.5% in each tail. A one-tailed test puts all 5% in one tail, which is why its critical value is smaller.
• The p-value is the probability of a result at least as extreme as the one obtained if H₀ were true. H₀ is rejected when p is at or below α.
• Not rejecting H₀ does not prove it true. The correct wording is "H₀ is retained" or "not rejected". Older texts say "H₀ is accepted". Prefer "retained", but accept "accepted" when it is the only option offered.
• Confidence level = 1 − α. The 95% limits for a mean are M ± 1.96 SE; the 99% limits are M ± 2.58 SE.

The critical-value card (z, normal curve)

One-tailed values are read from the .10 and .02 columns of a two-tailed table. Decision rule, two-tailed: reject H₀ at .05 if the absolute value of z is 1.96 or more, and at .01 if it is 2.58 or more.

Trap: "p = .03 means there is a 3% chance that H₀ is true." False. p is computed assuming H₀ is true; it is the probability of data this extreme, not the probability of the hypothesis. Equally false: "a non-significant result proves H₀."

A.5 The steps, in order

1. State H₀ and H₁; the direction of H₁ decides one- or two-tailed.
2. Fix the level of significance (α).
3. Choose the test by scale, design and assumptions (Part I), and note its df.
4. Compute the test statistic from the data.
5. Compare it with the critical value, or compare p with α.
6. Decide: reject or retain H₀, and interpret the result in words.

Worked example A1 — a z test on one mean
A school's Class X pupils (N = 100) score a mean of 52 on a standardised test
whose population mean is 50 and population SD is 10. Does the school differ?

H₀: μ = 50        H₁: μ ≠ 50   (non-directional → two-tailed)

Step 1  SE = σ / √N = 10 / √100 = 10 / 10 = 1
Step 2  z  = (M − μ) / SE = (52 − 50) / 1 = 2.00
Step 3  Two-tailed:  2.00 ≥ 1.96  → reject H₀ at .05
                     2.00 < 2.58  → retain H₀ at .01
Step 4  One-tailed (only if H₁ "higher" had been fixed in advance):
                     2.00 ≥ 1.645 → reject at .05
                     2.00 < 2.33  → retain at .01
Check   95% limits: 52 ± 1.96 × 1 = 50.04 to 53.96 — 50 lies outside → reject at .05
        99% limits: 52 ± 2.58 × 1 = 49.42 to 54.58 — 50 lies inside  → retain at .01

The check line is worth remembering: for a test on a mean, a two-tailed test at α and the (1 − α) confidence interval always agree.