Choosing a method
Use the design that matches the question posed to judges rather than selecting an index after data collection.
-
Item sort: judges assign each item to the construct
definition that best represents it. Use
sort_validity()for Psa/Csv, exact target-count screening, strongest-competitor diagnostics, and scale-level Colquitt norms. -
Construct rating: the same judges rate each item
against every focal and orbiting definition. Use
rating_validity()for HTC/HTD, repeated-measures inference, planned target-versus-orbiting contrasts, and scale-level norms. -
Expert panel: experts answer a relevance,
essentiality, or congruence question. Use
expert_validity()with the corresponding explicitmode. -
Delphi rounds: the same panel rates the items over
several rounds, with feedback in between. Use
delphi_validity()for consensus against a threshold fixed before the study, and stability between rounds.
The designs can complement one another during scale development, but
their statistics are not interchangeable. Two further questions apply to
any of them: whether the conclusions depend on the particular judges who
served (judge_validity()), and whether the items cover the
blueprint of the domain (domain_validity()).
Sample-size planning
For item sorts, plan judge N in relation to the exact retention rule
and a plausible true target-assignment probability.
sort_power() provides exact planning probabilities; avoid a
universal judge-count rule of thumb.
For construct-rating studies, power depends on the number of judges, number of construct definitions, within-judge target-orbiting separation, and missing profiles. Report the effective complete-judge N itemwise.
For expert panels, panel-size sensitivity is part of the statistic. CVR exact critical counts and common CVI review guidelines therefore need to be interpreted with the actual effective N, not a nominal panel size that ignores missingness.
Recommended reporting sequence
- Design: judge population, recruitment, construct definitions, item pool, instructions, and response format.
- A priori rules: alpha, relevance threshold, target mapping, multiplicity adjustment if used, and any planned norms.
- Item-level evidence: correspondence, distinctiveness, exact/paired inference, strongest competitor, and review status as appropriate.
- Scale-level evidence: target-scale averages or S-CVI summaries where the method defines them.
- Substantive decisions: revisions, removals, retained domain-coverage items, and how qualitative comments informed those decisions.
- Reproducibility: package version, analysis settings, anonymized data when permitted, and the script used to reproduce tables/figures.
The reporting-examples vignette expands this sequence
into reusable methods and results scaffolds. Treat those examples as
reporting patterns rather than fixed language that must be copied
verbatim.
Deterministic bundled examples
sort_dat <- read_example("sort_example.csv")
rating_dat <- read_example("rating_example.csv")
expert_rel <- read_example("expert_relevance_example.csv")
sort_fit <- sort_validity(sort_dat)
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
expert_fit <- expert_validity(
as.matrix(expert_rel[setdiff(names(expert_rel), "expert")]),
mode = "relevance", lo = 1, hi = 4
)These files are synthetic and generated by
data-raw/build-example-data.R in the source repository.
They deliberately include both supported and review-worthy items so
documentation exercises realistic output paths without depending on
random-number generation.
Table templates
content_report() turns a fit into a manuscript table,
with the columns a reader needs and numbers in APA form;
format = "markdown" gives the same table to paste. The
unrounded values stay in each fit’s results.
Item sort
content_report(sort_fit)
#> <contentvalid_report> Results table in APA style
#>
#> Item Target Judges Competitor Psa 95% CI Csv p Decision
#> A1 A 18/20 B; C .90 [.70, .97] .85 < .001 Retain
#> A2 A 15/20 B .75 [.53, .89] .60 .021 Retain
#> B1 B 17/20 A .85 [.64, .95] .75 .001 Retain
#> B2 B 13/20 A .65 [.43, .82] .40 .132 Review
#> C1 C 18/20 A; B .90 [.70, .97] .85 < .001 Retain
#> C2 C 14/20 B .70 [.48, .85] .50 .058 Review
#>
#> Note. Psa = proportion of substantive agreement; CI = confidence interval;
#> Csv = coefficient of substantive validity. Judges = target assignments, out of
#> the judges who sorted the item. 95% CI = Wilson score confidence interval.
#> Retain = at least the number of target assignments the exact one-sided
#> binomial test needs at alpha = .05 with p0 = .50 (Howard & Melloy, 2016).
#>
#> See content_report(fit, format = "markdown") for the table as Markdown, ready
#> for a manuscript.At the target-scale level, report mean Psa/Csv and the benchmark set actually used. Do not convert Colquitt’s scale-level norms into individual-item cutoffs.
Construct rating
content_report(rating_fit)
#> <contentvalid_report> Results table in APA style
#>
#> Item Target Judges HTC HTD F test p Contrast p Decision
#> A1 A 24 .93 .67 F(1.53, 35.18) = 237.08 < .001 < .001 Retain
#> A2 A 24 .73 .21 F(1.39, 32.00) = 37.38 < .001 .052 Review
#> B1 B 24 .93 .67 F(1.53, 35.18) = 237.08 < .001 < .001 Retain
#> B2 B 24 .73 .21 F(1.39, 32.00) = 37.38 < .001 .052 Review
#> C1 C 24 .93 .67 F(1.53, 35.18) = 237.08 < .001 < .001 Retain
#> C2 C 24 .73 .21 F(1.39, 32.00) = 37.38 < .001 .052 Review
#>
#> Note. HTC = Hinkin-Tracey correspondence; HTD = Hinkin-Tracey distinctiveness.
#> Judges = judges who rated the item against every construct.
#> F test = within-judge omnibus test, with Greenhouse-Geisser corrected degrees
#> of freedom where the correction applied. Contrast p = the largest one-sided p
#> among the planned contrasts of the intended construct with each other
#> construct. Retain = omnibus p and every one-sided contrast p at or below
#> alpha = .05 (MacKenzie et al., 2011).
#>
#> See content_report(fit, format = "markdown") for the table as Markdown, ready
#> for a manuscript.Report the repeated-measures design and target-versus-orbiting
contrasts. For a review item, naming the strongest competitor, which
rating_fit$results holds as
strongest_competitor, is often more informative than a
standalone p value.
Expert relevance
content_report(expert_fit)
#> <contentvalid_report> Results table in APA style
#>
#> Item Experts V 95% CI I-CVI 95% CI Kappa Decision
#> Item1 8 1.00 [.86, 1.00] 1.00 [.68, 1.00] 1.00 Strong support
#> Item2 8 .92 [.74, .98] 1.00 [.68, 1.00] 1.00 Strong support
#> Item3 8 .83 [.64, .93] 1.00 [.68, 1.00] 1.00 Strong support
#> Item4 8 .58 [.39, .76] .75 [.41, .93] .72 Review
#> Item5 8 .42 [.24, .61] .25 [.07, .59] .16 Review
#>
#> Note. V = Aiken's content validity coefficient; CI = confidence interval;
#> I-CVI = item-level content validity index. Kappa = modified kappa (Polit et
#> al., 2007). 95% CI = Penfield-Giacobbi score confidence interval for V and
#> Wilson score confidence interval for I-CVI. Strong support = at least the
#> number of experts rating the item relevant that Lynn's (1986) criterion
#> requires for the panel size.
#>
#> See content_report(fit, format = "markdown") for the table as Markdown, ready
#> for a manuscript.
expert_fit$scale_summary[c("S_CVI_Ave", "S_CVI_UA", "agreement")]
#> S_CVI_Ave S_CVI_UA agreement
#> 1 0.8 0.6 0.6934253Essentiality and congruence require different expert tasks. Do not place CVR, CVI, Aiken V, and IOC in one generic threshold table as if they answer the same question.
Diagnostics and later validation
signal_detection() and
reproducibility_phi() remain auxiliary helpers. They can be
useful when researchers later compare pretest decisions with CFA/IRT
retention or an independent replication pretest, but they are not
required parts of the flagship workflows.
pretest_supported <- sort_fit$results$status == "Supported"
later_retained <- c(TRUE, TRUE, TRUE, FALSE, TRUE, FALSE)
signal_detection(pretest_supported, later_retained)
#> <contentvalid_signal> Retention decisions compared with the actual outcome
#>
#> Predicted Actual: Retain Actual: Not retained
#> Retain 4 0
#> Not retained 0 2
#>
#> accuracy = 1.00, sensitivity = 1.00, specificity = 1.00, phi = 1.00, Fisher's
#> exact p = .067.
#> An expected count is below 5, so the exact test is reported in place of the
#> chi-square approximation (chi-square(1, N = 6) = 6.00, p = .014).
#>
#> See as.data.frame(x) for the counts and the test as one row.
replication_supported <- c(TRUE, TRUE, TRUE, FALSE, TRUE, TRUE)
reproducibility_phi(pretest_supported, replication_supported)
#> <contentvalid_reproducibility> Retention decisions in two pretests
#>
#> Pretest1 Pretest2: Retain Pretest2: Not retained
#> Retain 4 0
#> Not retained 1 1
#>
#> phi = .63, Fisher's exact p = .333.
#> An expected count is below 5, so the exact test is reported in place of the
#> chi-square approximation (chi-square(1, N = 6) = 2.40, p = .121).
#>
#> See as.data.frame(x) for the counts and the test as one row.With six items every expected count is below 5, so both helpers report Fisher’s exact p and show the chi-square approximation beside it for reference. With enough items for the approximation they report the chi-square test, with its degrees of freedom and N.
Planning how many judges to recruit
sort_power() and expert_power() answer the
question exactly for one design, from the decision rule the analysis
will apply, rather than from a rule of thumb.
gtheory_content() estimates it from a panel’s ratings.
For an item sort, sort_power() gives the probability
that an item reaches the exact retention count, for each panel size and
each rate at which you expect judges to choose the target:
sort_power(N = c(20, 30, 40), true_p = c(.60, .70, .80))
#> <contentvalid_sort_power> Item-sort planning
#> Retention rule: Howard-Melloy exact test (p0 = .50, alpha = .05)
#>
#> Judges Required Minimum Psa Power at .60 Power at .70 Power at .80
#> 20 15/20 .75 .13 .42 .80
#> 30 20/30 .67 .29 .73 .97
#> 40 26/40 .65 .32 .81 .99
#>
#> Required: target assignments an item needs to be retained. Minimum Psa: the
#> same as a proportion (Psa = proportion of substantive agreement). Power at a
#> value: the exact probability of reaching the required count if each judge
#> assigns the item to its target with that probability.
#>
#> See plot(x) for the power curve.For an expert panel, expert_power() gives the
probability that an item clears the I-CVI criterion, for each panel size
and an assumed probability that one expert endorses it:
expert_power(n_experts = 3:8, prob = 0.9)
#> <contentvalid_expert_power> Expert-panel planning
#> Criterion: Exact binomial planning against the panel-size I-CVI guideline
#>
#> Probability that an item clears the criterion
#> Experts Required Prob = .90
#> 3 3/3 .73
#> 4 4/4 .66
#> 5 5/5 .59
#> 6 5/6 .89
#> 7 6/7 .85
#> 8 7/8 .81
#>
#> Required: endorsements the criterion needs from the panel. Prob: the
#> probability you assume that one expert endorses the item.
#>
#> Note the step at six experts. Lynn's criterion requires unanimity with three
#> to five experts and allows one disagreement from six, so a sixth expert
#> relaxes the criterion while a fourth or fifth makes unanimity harder. That
#> is a property of the guideline, not of the items.
#>
#> This is a contentvalidR planning tool, not a published power method: it treats
#> the experts as endorsing the item independently, each with probability `prob`,
#> while the criterion it applies is Lynn's (1986). This table reports the
#> consequences of the panel sizes you asked about. It does not recommend one.
#> `prob` is an assumption you supply, so treat the result as conditional on it
#> and report the value you assumed.
#>
#> See plot(x) for the probabilities by panel size.Note the step. Lynn’s (1986) criterion requires every expert to agree on panels of up to five, and allows one disagreement from six, so a fourth or fifth expert lowers the probability of clearing while a sixth raises it sharply. That is a property of the guideline, not of the items, and the package reports it rather than smoothing it away.
When a panel has already rated the items, generalizability theory (Brennan, 2001; Crocker et al., 1988) estimates how many judges the same design would need to reach a given dependability:
judge_ratings <- rbind(
c(4, 4, 4, 3, 2, 2), c(4, 4, 3, 4, 2, 1), c(4, 3, 4, 4, 1, 2),
c(3, 4, 4, 4, 2, 2), c(4, 4, 4, 4, 2, 1), c(4, 3, 4, 3, 1, 2),
c(4, 4, 3, 4, 2, 2), c(2, 2, 2, 2, 1, 1)
)
dimnames(judge_ratings) <- list(paste0("Judge", 1:8), paste0("Item", 1:6))
gtheory_content(judge_ratings)
#> <contentvalid_gtheory> Generalizability analysis
#> Design: items x judges, crossed, one rating per cell
#> Items: 6 | Judges: 8
#>
#> Observed design
#> Generalizability coefficient (relative, rank ordering): .97
#> Dependability coefficient (absolute, fixed standard): .94
#> Status: Supported (criterion: Phi >= .80, a contentvalidR convention)
#> With 8 judges, absolute decisions about these items would generalize to
#> another panel of the same size at Phi = .94, at or above the .80 criterion.
#> Judge differences account for 15.6% of total variance.
#>
#> Variance components
#> Source df MS Estimate Used % of total
#> item 5 7.77 0.94 0.94 66.8
#> judge 7 1.57 0.22 0.22 15.6
#> residual 35 0.25 0.25 0.25 17.5
#>
#> MS: mean square. Estimate: the ANOVA estimate of each variance component.
#> Used: the same with a negative estimate set to 0, which the coefficients use
#> (Brennan, 2001).
#>
#> Judges needed to reach each coefficient
#> Target Relative (G) Absolute (Phi)
#> .70 1 2
#> .80 2 2
#> .90 3 5
#> These panel sizes are estimates from one panel's variance components, with
#> no interval: read them as planning figures.
#>
#> A dependability coefficient describes generalization over judges only. It is
#> not evidence that the items cover the intended content domain.
#>
#> See x$dstudy for the coefficients projected by panel size.The dependability coefficient (Phi) concerns the absolute level of
the ratings and is lowered by differences in judge severity, which is
usually what a content-validity decision rests on. The status beside it
compares Phi with phi_cut (.80 by default), a convention of
this package, not a published standard. Here every target is reached
with five judges or fewer, because the items account for most of the
variance. Had the judges barely distinguished the items, a target no
realistic panel reaches would print as “unreachable” (NA in
the data). The projected panel sizes come from one panel’s variance
components and carry no interval, so read them as planning figures.
Across pretest rounds, compare_rounds() sets each item’s
status side by side and checks whether the analysis settings changed
between rounds, so a relaxed criterion cannot read as an improved
item.
Good practices
Pre-register quantitative screening criteria when feasible, preserve qualitative judge feedback, report effective N after missingness, and archive the construct definitions and item wording used in the pretest. A content-validation statistic is evidence from a designed judgment task; it is not a substitute for defining and sampling the construct domain.
References
Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
Crocker, L., Llabre, M., & Miller, M. D. (1988). The generalizability of content validity ratings. Journal of Educational Measurement, 25(4), 287–299. https://doi.org/10.1111/j.1745-3984.1988.tb00309.x
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017