Practice Set 5 — Measurement I: Foundations & Test Construction

Statements 2 and 3 are correct. The quantity is defined at the level of the individual: it estimates how much one person's observed score is likely to fluctuate around that person's true score, which is why it is used to place a band round a single candidate's mark rather than to describe a group (2). And it has two inputs, not one. Since SEM = SD × √(1 − r), a test with SD = 20 and r = 0.75 has a SEM of 10 while a test with SD = 8 and the same r = 0.75 has a SEM of 4 — equal reliability, unequal error bands (3).

Statement 1 is false and rules out (a), (c) and (d). It is the reliability coefficient that is a dimensionless proportion, being a ratio of variances. The SEM is not: √(1 − r) is a pure number, so the product SD × √(1 − r) carries the units of the standard deviation — the marks of the test. A SEM of 9 means nine marks, and quoting it without naming the test whose marks they are says nothing.

(a) "1 and 2 only" accepts the units claim and drops the point that the standard deviation has as much say in the SEM as the reliability does.

(c) "1 and 3 only" accepts the units claim and denies that the SEM speaks to an individual score — which would leave the formula with nothing to be used for.

(d) "1, 2 and 3" accepts all three, and so treats a quantity measured in marks as though it were a unitless ratio like r.

Trap: ch01 — the SEM is in marks and belongs to an individual score; the reliability coefficient is a unitless group statistic. Equal reliability does not mean equal SEM.

Both statements are correct, and the second explains the first. A reliability coefficient obtained by test-retest, parallel forms or split-half is a correlation, and a correlation needs variation to work on: the more heterogeneous the group, the higher the coefficient, while a homogeneous group compresses the range and depresses it. A criterion-referenced mastery test is designed so that everyone who has learned the material reaches the standard, so its scores bunch near the top by intention -- and the classical coefficient falls, without the test having become any less trustworthy for the decision it exists to make. This is why criterion-referenced testing is judged by different evidence, and why a low classical coefficient on a mastery test is not by itself a criticism of the test. (b) accepts both statements but denies the link, when the compression of range is the whole mechanism. (c) rejects Statement-II and so rejects the heterogeneity rule -- the most frequently mis-answered statement in the whole reliability topic, where intuition says a mixed group must be harder to measure consistently and the opposite is true. (d) rejects Statement-I, expecting a well-built mastery test to show a high classical coefficient; it usually will not, because there is little variance left for the correlation to explain. Trap: ch02 with ch01 -- spread is essential to a norm-referenced test and irrelevant to a criterion-referenced one, and the classical reliability coefficient rewards spread.

Both statements are correct, but the second does not explain the first. Statement-I checks out. Rulon: r = 1 − σ²d/σ²x = 1 − 8/32 = 1 − 0.25 = 0.75. Flanagan: r = 2 × [1 − (9 + 11)/32] = 2 × [1 − 0.625] = 2 × 0.375 = 0.75. Statement-II is also true and is stated in the chapter's table: Rulon, Flanagan, Hoyt, KR-20/21 and alpha all report whole-test reliability, and only the raw odd-even split-half correlation needs stepping up. But the agreement in Statement-I does not follow from Statement-II. Plenty of methods report whole-test reliability directly without agreeing numerically — a test-retest coefficient and a KR-20 coefficient are both whole-test figures and routinely differ. Rulon and Flanagan agree because they are algebraically the same expression. Writing c for the covariance of the halves, σ²x = σ²a + σ²b + 2c and σ²d = σ²a + σ²b − 2c, so Rulon reduces to 4c/σ²x and Flanagan reduces to 4c/σ²x as well. Here c = (32 − 20)/2 = 6, and 4 × 6/32 = 0.75 by both routes. The identity, not the absence of a Spearman-Brown step, is the reason.

(a) makes Statement-II the explanation, which is the error the item is built on: needing no correction is a shared property, not a cause of numerical identity.

(c) and (d) each call one of the two statements false, and both statements verify — one arithmetically, one against the chapter's table.

Trap: ch01 — Rulon and Flanagan are two spellings of one formula. They agree by algebra, not because neither needs Spearman-Brown.