Test-retest reliability, measurement error, and reliable change
Source:R/nomo_retest.R
nomo_retest.Rdnomo_retest() estimates how stable a composite's scores are when the same
people answer the same measure on two or more occasions, how large a
person's measurement error is, and which people changed by more than that
error.
Arguments
- data
A data frame with one row per person.
- scores
The columns holding the same composite on successive occasions, in order: a character vector of two or more column names, or a named list of such vectors, one per composite.
- interval
Optional text describing the time between occasions, such as
"two weeks". It is recorded with the results, because a test-retest reliability depends on it.
Value
A nomo_retest object. The fields to read are:
icc: one row per composite, with the number of complete cases and of occasions, ICC(A,1) with its 95% interval and Koo and Li's description of that interval (koo_li), ICC(C,1) with its interval, the mean change from the first to the last occasion with its interval, the pooled standard deviation (sd, the standard deviation of the scores pooled over occasions, not of the change), the standard error of measurement (sem, the square root of the residual mean square), and the smallest detectable change (sdc).reliable_change: one row per person and composite, with the first and last scores, the change, the reliable change index (rci), and whether the change is a reliable increase, a reliable decrease, or neither. A person whose score did not change has anrciof 0, even when there is no measurement error at all.intervalanddecision_log.
A case without a score on every occasion of a composite is left out of
that composite, and the decision log says how many were. Scores must be
finite numbers or NA.
Other fields record the call and the columns used (scores). They may
change between releases and are not part of the stable interface (see
?nomologR).
print() shows each composite's ICC(A,1) with its interval and Koo and
Li's description, the measurement error, and how many people changed
reliably; summary() adds the consistency form ICC(C,1) and the mean
change with its interval, and the full reason for each flag.
Details
Which intraclass correlation. Shrout and Fleiss (1979) and McGraw and Wong (1996) define several intraclass correlations, which differ in their model, type, and definition. For test-retest reliability, Koo and Li (2016) recommend a two-way mixed-effects model with absolute agreement: the occasions are not a random sample of occasions, and scores that do not agree across occasions are not reliable however well they rank people. That is ICC(A,1) in McGraw and Wong's notation, for a single measurement. The consistency form, ICC(C,1), ignores a shift in the mean between occasions and is reported beside it, so the gap between them shows how much of the disagreement is a systematic change. The mean change from the first to the last occasion, with its interval, is reported too.
Reading the interval. Koo and Li (2016) describe reliability as poor
below .50, moderate from .50 to .75, good from .75 to .90, and excellent
above .90, and ask that the description come from the 95% confidence
interval rather than the estimate. koo_li gives the description of the
interval, such as "good to excellent". These are reference values, not a pass
or a fail, and a reliability depends on the interval between occasions and on
whether the construct itself changed.
Measurement error and change. The standard error of measurement is \(\sqrt{MS_E}\), the square root of the residual mean square of the two-way model (Weir, 2005): how far a person's scores spread across occasions once the shift in the mean between occasions is removed. It does not depend on which ICC is chosen. When the occasions do not differ in mean, it is close to \(SD \sqrt{1 - ICC}\) (Nunnally & Bernstein, 1994), with ICC(A,1) and the standard deviation pooled over occasions; when they do, that form counts the shift as measurement error and overstates it. The smallest detectable change is \(1.96 \sqrt{2} \, SEM\): a change in a person's score smaller than that is within measurement error at 95% (Weir, 2005). Jacobson and Truax's (1991) reliable change index divides a person's change by \(\sqrt{2} \, SEM\), so it exceeds 1.96 exactly when the change exceeds the smallest detectable change. The change compared includes any shift in the mean, so a shift that everyone shares can make many changes reliable. A reliable change is not necessarily a meaningful one.
References
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12-19. doi:10.1037/0022-006X.59.1.12
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. doi:10.1016/j.jcm.2016.02.012
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi:10.1037/1082-989X.1.1.30
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. doi:10.1037/0033-2909.86.2.420
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231-240. doi:10.1519/15184.1
Examples
# Agency scores on two occasions, two weeks apart (simulated).
set.seed(2026)
true <- stats::rnorm(150)
panel <- data.frame(
agency_t1 = 3 + true + stats::rnorm(150, sd = .45),
agency_t2 = 3.1 + true + stats::rnorm(150, sd = .45)
)
rt <- nomo_retest(panel, scores = c("agency_t1", "agency_t2"),
interval = "two weeks")
rt
#> <nomo_retest> Test-retest reliability
#> McGraw and Wong (1996); Koo and Li (2016).
#> Composites: 1 | Cases: 150 | Interval: two weeks
#>
#> Reliability across occasions
#> Composite n ICC(A,1) 95% CI Koo and Li Pooled SD SEM SDC
#> composite 150 .85 [.80, .89] good 1.17 0.45 1.25
#>
#> Reliable change, first to last occasion
#> - composite: 6 up, 2 down, and 142 within measurement error, of 150 people.
#>
#> What these columns mean
#> n -- Cases with a score on every occasion.
#> ICC(A,1) -- Intraclass correlation from a two-way mixed-effects model,
#> absolute agreement, single measurement.
#> CI -- Confidence interval.
#> Koo and Li -- Their description of the ICC(A,1) interval: poor below .50,
#> moderate to .75, good to .90, excellent above.
#> Pooled SD -- Standard deviation of the scores pooled over occasions, not of
#> the change.
#> SEM -- Standard error of measurement, the square root of the residual mean
#> square (Weir, 2005).
#> SDC -- Smallest detectable change, 1.96 x sqrt(2) x SEM (Weir, 2005).
#>
#> Reference ranges describe the interval; they are not a pass or a fail. A
#> reliable change is larger than measurement error, which does not make it a
#> meaningful one.
#>
#> See summary(x) for the consistency form and the mean change and
#> nomo_table(x, "reliable_change") for each person's change.
nomo_table(rt, "reliable_change")
#> # A tibble: 150 × 7
#> composite row first_score last_score change rci status
#> <chr> <int> <dbl> <dbl> <dbl> <dbl> <chr>
#> 1 composite 1 3.53 3.55 0.0222 0.0349 no_reliable_change
#> 2 composite 2 2.07 1.94 -0.133 -0.209 no_reliable_change
#> 3 composite 3 3.58 2.85 -0.731 -1.15 no_reliable_change
#> 4 composite 4 3.14 3.45 0.306 0.482 no_reliable_change
#> 5 composite 5 2.59 2.06 -0.537 -0.845 no_reliable_change
#> 6 composite 6 0.998 0.242 -0.756 -1.19 no_reliable_change
#> 7 composite 7 1.95 1.97 0.0284 0.0447 no_reliable_change
#> 8 composite 8 1.70 2.83 1.12 1.77 no_reliable_change
#> 9 composite 9 3.11 3.46 0.354 0.558 no_reliable_change
#> 10 composite 10 2.68 2.90 0.218 0.343 no_reliable_change
#> # ℹ 140 more rows