Choosing a method
Use the design that matches the question posed to judges rather than selecting an index after data collection.
-
Item sort: judges assign each item to the construct
definition that best represents it. Use
sort_validity()for Psa/Csv, exact target-count screening, strongest-competitor diagnostics, and scale-level Colquitt norms. -
Construct rating: the same judges rate each item
against every focal and orbiting definition. Use
rating_validity()for HTC/HTD, repeated-measures inference, planned target-versus-orbiting contrasts, and scale-level norms. -
Expert panel: experts answer a relevance,
essentiality, or congruence question. Use
expert_validity()with the corresponding explicitmode.
The three designs can complement one another during scale development, but their statistics are not interchangeable.
Sample-size planning
For item sorts, plan judge N in relation to the exact retention rule
and a plausible true target-assignment probability.
sort_power() provides exact planning probabilities; avoid a
universal judge-count rule of thumb.
For construct-rating studies, power depends on the number of judges, number of construct definitions, within-judge target-orbiting separation, and missing profiles. Report the effective complete-judge N itemwise.
For expert panels, panel-size sensitivity is part of the statistic. CVR exact critical counts and common CVI review guidelines therefore need to be interpreted with the actual effective N, not a nominal panel size that ignores missingness.
Recommended reporting sequence
- Design: judge population, recruitment, construct definitions, item pool, instructions, and response format.
- A priori rules: alpha, relevance threshold, target mapping, multiplicity adjustment if used, and any planned norms.
- Item-level evidence: correspondence, distinctiveness, exact/paired inference, strongest competitor, and review status as appropriate.
- Scale-level evidence: target-scale averages or S-CVI summaries where the method defines them.
- Substantive decisions: revisions, removals, retained domain-coverage items, and how qualitative comments informed those decisions.
- Reproducibility: package version, analysis settings, anonymized data when permitted, and the script used to reproduce tables/figures.
The reporting-examples vignette expands this sequence
into reusable methods and results scaffolds. Treat those examples as
reporting patterns rather than fixed language that must be copied
verbatim.
Deterministic bundled examples
sort_dat <- read_example("sort_example.csv")
rating_dat <- read_example("rating_example.csv")
expert_rel <- read_example("expert_relevance_example.csv")
sort_fit <- sort_validity(sort_dat)
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
expert_fit <- expert_validity(
as.matrix(expert_rel[setdiff(names(expert_rel), "expert")]),
mode = "relevance", lo = 1, hi = 4
)These files are synthetic and generated by
data-raw/build-example-data.R in the source repository.
They deliberately include both supported and review-worthy items so
documentation exercises realistic output paths without depending on
random-number generation.
Table templates
Item sort
sort_fit$results[c(
"item", "target", "n", "n_target", "competitor",
"psa", "csv", "p_value", "status", "recommendation"
)]
#> item target n n_target competitor psa csv p_value status
#> 1 A1 A 20 18 B; C 0.90 0.85 0.0002012253 Supported
#> 2 A2 A 20 15 B 0.75 0.60 0.0206947327 Supported
#> 3 B1 B 20 17 A 0.85 0.75 0.0012884140 Supported
#> 4 B2 B 20 13 A 0.65 0.40 0.1315879822 Review
#> 5 C1 C 20 18 A; B 0.90 0.85 0.0002012253 Supported
#> 6 C2 C 20 14 B 0.70 0.50 0.0576591492 Review
#> recommendation
#> 1 Retain
#> 2 Retain
#> 3 Retain
#> 4 Review
#> 5 Retain
#> 6 ReviewAt the target-scale level, report mean Psa/Csv and the benchmark set actually used. Do not convert Colquitt’s scale-level norms into individual-item cutoffs.
Construct rating
rating_fit$results[c(
"item", "target", "n_complete", "strongest_competitor",
"htc", "htd", "p_value", "max_contrast_p", "status", "recommendation"
)]
#> item target n_complete strongest_competitor htc htd p_value
#> 1 A1 A 24 C 0.9333333 0.6666667 1.274299e-19
#> 2 A2 A 24 B 0.7333333 0.2083333 7.375102e-08
#> 3 B1 B 24 C 0.9333333 0.6666667 1.274299e-19
#> 4 B2 B 24 A 0.7333333 0.2083333 7.375102e-08
#> 5 C1 C 24 B 0.9333333 0.6666667 1.274299e-19
#> 6 C2 C 24 B 0.7333333 0.2083333 7.375102e-08
#> max_contrast_p status recommendation
#> 1 1.122784e-13 Supported Retain
#> 2 5.173102e-02 Review Review
#> 3 1.122784e-13 Supported Retain
#> 4 5.173102e-02 Review Review
#> 5 1.122784e-13 Supported Retain
#> 6 5.173102e-02 Review Review
rating_fit$scale_summary
#> target n_items n_htc n_htd n_retain n_review n_insufficient mean_htc
#> 1 A 2 2 2 1 1 0 0.8333333
#> 2 B 2 2 2 1 1 0 0.8333333
#> 3 C 2 2 2 1 1 0 0.8333333
#> htc_strength mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1 Weak 0.4375 Very Strong Weak NA overall
#> 2 Weak 0.4375 Very Strong Weak NA overall
#> 3 Weak 0.4375 Very Strong Weak NA overall
#> evidence
#> 1 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.
#> 2 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.
#> 3 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.Report the repeated-measures design and target-versus-orbiting contrasts. For a review item, naming the strongest competitor is often more informative than a standalone p value.
Expert relevance
expert_fit$results[c(
"item", "N", "V", "ci_low", "ci_high", "I_CVI",
"kappa_mod", "status", "recommendation"
)]
#> item N V ci_low ci_high I_CVI kappa_mod status
#> 1 Item1 8 1.0000000 0.8620238 1.0000000 1.00 1.0000000 Supported
#> 2 Item2 8 0.9166667 0.7415120 0.9768412 1.00 1.0000000 Supported
#> 3 Item3 8 0.8333333 0.6414693 0.9332132 1.00 1.0000000 Supported
#> 4 Item4 8 0.5833333 0.3883467 0.7553240 0.75 0.7192982 Review
#> 5 Item5 8 0.4166667 0.2446760 0.6116533 0.25 0.1578947 Review
#> recommendation
#> 1 Strong support
#> 2 Strong support
#> 3 Strong support
#> 4 Review
#> 5 Review
expert_fit$scale_summary
#> n_items n_experts_min n_experts_max mean_Aiken_V S_CVI_Ave S_CVI_UA
#> 1 5 8 8 0.75 0.8 0.6
#> n_strong_support n_support n_review n_insufficient
#> 1 3 0 2 0Essentiality and congruence require different expert tasks. Do not place CVR, CVI, Aiken V, and IOC in one generic threshold table as if they answer the same question.
Diagnostics and later validation
signal_detection() and
reproducibility_phi() remain auxiliary helpers. They can be
useful when researchers later compare pretest decisions with CFA/IRT
retention or an independent replication pretest, but they are not
required parts of the flagship workflows.
pretest_supported <- sort_fit$results$status == "Supported"
later_retained <- c(TRUE, TRUE, TRUE, FALSE, TRUE, FALSE)
signal_detection(pretest_supported, later_retained)
#> $confusion
#> Actual
#> Predicted Retain Not retained
#> Retain 4 0
#> Not retained 0 2
#>
#> $accuracy
#> [1] 1
#>
#> $sensitivity
#> [1] 1
#>
#> $specificity
#> [1] 1
#>
#> $phi
#> [1] 1
#>
#> $chisq
#> [1] 6
#>
#> $p
#> [1] 0.01430588
replication_supported <- c(TRUE, TRUE, TRUE, FALSE, TRUE, TRUE)
reproducibility_phi(pretest_supported, replication_supported)
#> $table
#> Pretest2
#> Pretest1 Retain Not retained
#> Retain 4 0
#> Not retained 1 1
#>
#> $phi
#> [1] 0.6324555
#>
#> $chisq
#> [1] 2.4
#>
#> $p
#> [1] 0.1213353Power quick check
sort_power(N = c(20, 30, 40), true_p = c(.60, .70, .80))
#> Exact item-sort planning analysis
#> ---------------------------------
#> Retention rule: p0 = 0.50, alpha = 0.050
#>
#> N true_p critical_n_target minimum_observed_psa power
#> 20 0.6 15 0.750 0.126
#> 30 0.6 20 0.667 0.291
#> 40 0.6 26 0.650 0.317
#> 20 0.7 15 0.750 0.416
#> 30 0.7 20 0.667 0.730
#> 40 0.7 26 0.650 0.807
#> 20 0.8 15 0.750 0.804
#> 30 0.8 20 0.667 0.974
#> 40 0.8 26 0.650 0.992
#>
#> Power is the exact probability of reaching the required target-assignment count
#> under the assumed true target-assignment probability.Good practices
Pre-register quantitative screening criteria when feasible, preserve qualitative judge feedback, report effective N after missingness, and archive the construct definitions and item wording used in the pretest. A content-validation statistic is evidence from a designed judgment task; it is not a substitute for defining and sampling the construct domain.