Analyze an item-sort content-validity pretest — sort

Provides the recommended user-facing workflow for item-sort studies. At the item level, sort_validity() combines Anderson and Gerbing's (1991) Psa and Csv statistics with the exact target-count significance test recommended by Howard and Melloy (2016). Items meeting the exact criterion are labeled "Retain"; items that do not meet it are labeled "Review", not automatically "Delete". An item meets the criterion when its exact p is at or below alpha, so a p equal to alpha counts as meeting it; see csv_binom_test(). An item sorted by so few judges that no count could meet the criterion (four or fewer at the defaults) is labeled "Insufficient panel", with status "Insufficient data".

Construct and item labels are compared as text after leading and trailing spaces are removed, so constructs may be coded as numbers, text or factors. Results list the items in the order they first appear in the data, or in the order of the levels when the item column is a factor.

At the target-scale level, Psa and Csv are averaged across items and interpreted using the empirical percentile norms from Colquitt et al. (2019). This mirrors how those norms were constructed. The Colquitt categories are descriptive benchmarks rather than pass/fail rules. Each index is read against its own band; the advice in scale_summary$evidence, keyed to the lower of the two bands, is labeled as this package's suggestion, not Colquitt et al.'s.

Usage

sort_validity(
  assignments,
  item_col = "item",
  rater_col = "rater",
  assigned_col = "assigned_construct",
  target_col = "target_construct",
  p0 = 0.5,
  alpha = 0.05,
  orbiting_r = NULL,
  judge_type = c("naive", "expert"),
  proportion_ci = c("wilson", "agresti_coull", "exact", "none"),
  legacy = FALSE,
  n_constructs = NULL
)

Arguments

assignments: A data.frame containing item-sort responses.
item_col, rater_col, assigned_col, target_col: Column names for the item, rater, assigned construct, and intended target construct.
p0: Null target-assignment probability for the exact binomial test. Default 0.5, following Howard and Melloy (2016). It is not the rate expected from random assignment, which is 1 divided by the number of constructs. Howard and Melloy describe .5 as arbitrary and lenient, and suggest a higher value such as .6 or .75, chosen before data collection, when the alternative constructs are clearly different from the target or the judges are subject-matter experts.
alpha: Significance level. Default 0.05. A p equal to alpha meets the criterion.
orbiting_r: Optional average correlation between each focal/target scale and its orbiting scales. For one target, supply one correlation. For multiple targets, supply a named numeric vector keyed by target construct. If omitted, the overall Colquitt et al. norms are used.
judge_type: Either "naive" (the Anderson-Gerbing/Colquitt design) or "expert". Colquitt benchmark labels are not applied to expert judges.
proportion_ci: Interval method for Psa: "wilson" (default), "agresti_coull", "exact", or "none". The interval is two-sided at level 1 - alpha, while the exact test is one-sided, so the interval of an item that just meets the criterion can still include p0. The decision comes from the test, not from the interval, and the printout says so when a retained item's interval includes p0. See ci in cvi() for the methods and the evidence for each.
legacy: Print the earlier published rules beside the decision, for comparison. Default FALSE. They are computed either way, stored in details$earlier_methods, and never change the decision; print(fit, legacy = TRUE) shows them for any fit.
n_constructs: Optional number of constructs judges could choose among. It is used by the comparison block's chance-based extension and by the caution added to the Colquitt et al. (2019) bands when it is not three. By default it is the number of constructs that appear in the data, which is too few when judges were offered a construct none of them chose.

Value

An object of class contentvalid_sort and contentvalid_workflow. All flagship workflow objects expose the common components results, scale_summary, settings, design, and details. Item-level results include a standardized status field while retaining the method-specific recommendation field. print(), summary(), and plot() provide user-facing interpretation; see contentvalid-methods.

Results columns. results has one row per item:

item: The item.
target: The construct the item was written for.
n_total: Rows for the item, missing assignments included.
n: Judges who sorted the item: its assignments that are not missing.
n_missing: Missing assignments, n_total - n.
n_target: Assignments to the target construct.
competitor: The other construct judges chose most often, with ties joined by "; "; NA when no judge chose another construct.
n_other_max: Assignments to the competitor.
psa: Psa, n_target / n.
psa_low, psa_high: The interval for Psa at level 1 - alpha, by the method in proportion_ci; NA with "none".
csv: Csv, (n_target - n_other_max) / n.
p_value: The one-sided exact binomial p of the target count against p0.
critical_n_target: The fewest target assignments that meet the criterion with n judges; NA when no count can.
passes_chance: Whether n_target reaches critical_n_target.
recommendation: "Retain", "Review", "Insufficient panel" (no count could meet the test), or "Insufficient data" (no judge sorted the item).
issue: The reason in a few words, such as "Competing construct favored".
interpretation: The decision explained in a sentence.
status: The shared status: "Supported" for "Retain", "Review", or "Insufficient data".

Earlier methods, for comparison

The decision uses the exact test of Howard and Melloy (2016). Two earlier published rules, and one labeled package extension, are reported beside it for teaching, the way a methods text reports eta-squared beside omega-squared. None of them changes the decision:

Anderson and Gerbing (1991) judged Csv against a critical value. With N judges, m is the fewest target assignments whose one-tailed binomial probability at .5 falls below alpha (their Equation 5), and the critical Csv is (2m - N) / N (Equation 6): .50 for 20 judges at .05. Equation 6 assumes every judge who misses the target picks the same rival. When those judges spread across several constructs, the leading rival's count falls, so Csv can reach the critical value with fewer target assignments than the exact test requires. That is why it is not used for the decision.
Yao et al. (2008) required Psa and Csv both to reach .30, which they chose for a four-domain sort, where an item assigned at random lands in its domain with probability .25 (p. 486). They give no rule for other numbers of domains.
A contentvalidR extension, not a published rule. Yao et al.'s reasoning carried to k constructs as chance plus .05: Psa and Csv both at least 1/k + .05, which is their .30 when k = 4. It is shown in its own column, marked as an extension, whenever k is not 4. k is n_constructs when given, and otherwise the number of constructs in the data.

Csv counts only the single most-chosen rival construct (Anderson & Gerbing, 1991, p. 734). Pooling every other construct into that count instead gives 2 * Psa - 1, a different index; the printout notes this.

References

Anderson, J. C., & Gerbing, D. W. (1991). Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities. Journal of Applied Psychology, 76(5), 732–740. doi:10.1037/0021-9010.76.5.732

Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. Journal of Applied Psychology, 104(10), 1243–1265. doi:10.1037/apl0000406

Howard, M. C., & Melloy, R. C. (2016). Evaluating item-sort task methods: The presentation of a new statistical significance formula and methodological best practices. Journal of Business and Psychology, 31(1), 173–186. doi:10.1007/s10869-015-9404-y

Yao, G., Wu, C.-H., & Yang, C.-T. (2008). Examining the content validity of the WHOQOL-BREF from respondents' perspective by quantitative methods. Social Indicators Research, 85(3), 483–498. doi:10.1007/s11205-007-9112-8

Examples

sort_dat <- data.frame(
  item = rep(c("A1", "A2", "A3"), each = 20),
  rater = rep(1:20, 3),
  target_construct = "A",
  assigned_construct = c(
    rep("A", 18), rep("B", 2),
    rep("A", 16), rep("B", 4),
    rep("A", 12), rep("B", 8)
  )
)
fit <- sort_validity(sort_dat)
fit
#> <contentvalid_sort> Item-sort analysis
#> Items: 3 | Judges: 20 | Target constructs: 1
#> Test: Howard-Melloy exact target-count test (p0 = .50, alpha = .05)
#> Judges: naive, meaning drawn from the kind of people who will answer the
#> items.
#> 
#> 2 of 3 items meet the exact target-assignment criterion.
#> Flagged for review: A3
#> 
#> Item-level evidence
#>   Item  Target  Decision  Judges  Psa      95% CI  Csv  Competitor       p
#>   A1    A       Retain     18/20  .90  [.70, .97]  .80  B           < .001
#>   A2    A       Retain     16/20  .80  [.58, .92]  .60  B             .006
#>   A3    A       Review     12/20  .60  [.39, .78]  .20  B             .252
#> 
#>   Judges: assignments to the target construct, out of the judges who sorted
#>   the item.
#>   95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#>   compared seven methods and recommends score intervals over the Wald
#>   interval. An interval reflects how few ratings an item received, not whether
#>   the right people rated it.
#> 
#> Scale-level Colquitt benchmarks
#>   Target  Items  Mean Psa  Psa level  Mean Csv  Csv level
#>   A           3       .77  Moderate        .53  Moderate
#>   Benchmark set: Overall (not correlation-normed)
#>   These bands come from tasks with three definitions (one focal, two
#>   orbiting); judges here used 2 (set `n_constructs` if more were offered), so
#>   the comparison is approximate. This is a contentvalidR caution: Colquitt et
#>   al. do not discuss other numbers.
#> 
#>   Colquitt labels are empirical percentile norms derived from scale-level
#>   averages, not universal cutoffs or automatic scale-retention rules. They
#>   place a scale against published scales. Psa and Csv sit on different scales,
#>   so their values cannot be compared with each other; their labels can,
#>   because each is a percentile position among published scales.
#> 
#> What these columns mean
#>   Psa -- Proportion of Substantive Agreement. Share of judges who put the item
#>       in the construct it was written for (0 to 1; higher is stronger).
#>   95% CI -- Interval for Psa. Wider when fewer judges sorted the item; the
#>       method is named above.
#>   Csv -- Coefficient of Substantive Validity. How much more often judges chose
#>       the intended construct than its closest rival (-1 to 1; 0 is a tie).
#>   Competitor -- Strongest competing construct. The construct other than the
#>       intended one that judges chose most often.
#>   p -- Howard-Melloy exact test. Probability of at least this many target
#>       assignments if each judge picked the target at rate p0; compare with
#>       alpha.
#> 
#> What the decisions mean
#>   Retain -- met the exact target-assignment criterion.
#>   Review -- did not meet the exact target-assignment criterion; the competitor
#>       column shows where judges put it instead.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> A flag for review is not an automatic deletion decision. Use theory,
#> construct-domain coverage, item wording, and qualitative judge feedback
#> alongside these statistics.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.
summary(fit)
#> <contentvalid_sort summary> Item-sort analysis
#> Retain: 2 of 3 | Review: 1 of 3
#> 
#> Scale-level evidence
#>   Target  Items  Retain  Review  Mean Psa  Psa level  Mean Csv  Csv level
#>   A           3       2       1       .77  Moderate        .53  Moderate
#> 
#>   Psa = proportion of substantive agreement; Csv = coefficient of substantive
#>   validity (Anderson & Gerbing, 1991).
#>   A: Mean Psa and mean Csv both fall in the Moderate band of published scales
#>     (Colquitt et al., 2019). A contentvalidR suggestion for that band: review
#>     the weaker items before finalizing. These bands come from tasks with three
#>     definitions (one focal, two orbiting); judges here used 2 (set
#>     `n_constructs` if more were offered), so the comparison is approximate.
#>     This is a contentvalidR caution: Colquitt et al. do not discuss other
#>     numbers.
#> 
#> Flagged
#>   Item  Target  Decision  Psa  Csv  Competitor     p
#>   A3    A       Review    .60  .20  B           .252
#> 
#>   p = Howard-Melloy exact test of the target assignments.
#>   - A3 (Review): The target was at least as common as any competitor but did
#>     not meet the exact retention criterion (strongest competitor: B); review
#>     before deciding whether to revise or remove the item.
#> 
#> Interpret scale norms and item flags alongside theory, domain coverage, and
#> qualitative feedback. This analysis does not by itself establish
#> comprehensiveness or the full content-validity argument.
#> 
#> See summary(x)$reviewed_items for the flagged items as a data frame.

# The same result beside the earlier published rules.
print(fit, legacy = TRUE)
#> <contentvalid_sort> Item-sort analysis
#> Items: 3 | Judges: 20 | Target constructs: 1
#> Test: Howard-Melloy exact target-count test (p0 = .50, alpha = .05)
#> Judges: naive, meaning drawn from the kind of people who will answer the
#> items.
#> 
#> 2 of 3 items meet the exact target-assignment criterion.
#> Flagged for review: A3
#> 
#> Item-level evidence
#>   Item  Target  Decision  Judges  Psa      95% CI  Csv  Competitor       p
#>   A1    A       Retain     18/20  .90  [.70, .97]  .80  B           < .001
#>   A2    A       Retain     16/20  .80  [.58, .92]  .60  B             .006
#>   A3    A       Review     12/20  .60  [.39, .78]  .20  B             .252
#> 
#>   Judges: assignments to the target construct, out of the judges who sorted
#>   the item.
#>   95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#>   compared seven methods and recommends score intervals over the Wald
#>   interval. An interval reflects how few ratings an item received, not whether
#>   the right people rated it.
#> 
#> Scale-level Colquitt benchmarks
#>   Target  Items  Mean Psa  Psa level  Mean Csv  Csv level
#>   A           3       .77  Moderate        .53  Moderate
#>   Benchmark set: Overall (not correlation-normed)
#>   These bands come from tasks with three definitions (one focal, two
#>   orbiting); judges here used 2 (set `n_constructs` if more were offered), so
#>   the comparison is approximate. This is a contentvalidR caution: Colquitt et
#>   al. do not discuss other numbers.
#> 
#>   Colquitt labels are empirical percentile norms derived from scale-level
#>   averages, not universal cutoffs or automatic scale-retention rules. They
#>   place a scale against published scales. Psa and Csv sit on different scales,
#>   so their values cannot be compared with each other; their labels can,
#>   because each is a percentile position among published scales.
#> 
#> Earlier methods, for comparison (not used for the decision)
#>   Item  Decision  Psa  Csv  A&G (1991)  Yao et al. (2008)  Extension*
#>   A1    Retain    .90  .80  Meets       Meets              Meets
#>   A2    Retain    .80  .60  Meets       Meets              Meets
#>   A3    Review    .60  .20  Below       Below              Below
#> 
#>   Anderson and Gerbing (1991): Csv of at least .50, their critical value for
#>   20 judges at alpha = .05 (Equations 5 and 6). Their critical value assumes
#>   every judge who misses the target picks the same rival. When those judges
#>   spread across several constructs, Csv can reach it with fewer target
#>   assignments than the exact test needs.
#>   Yao et al. (2008): Psa and Csv both at least .30, set for a four-domain sort
#>   where chance assignment is .25.
#>   Extension*: a contentvalidR extension, not a published rule. It carries Yao
#>   et al.'s reasoning to this sort's 2 constructs as chance plus .05, so Psa
#>   and Csv both at least .55 (1/2 + .05).
#>   The 2 constructs are the ones judges used. If more were offered, set
#>   `n_constructs`, since chance depends on the number offered.
#>   Agreement with the decision above: Anderson and Gerbing on 3 of 3 items, Yao
#>   et al. on 3 of 3, the extension on 3 of 3.
#>   Csv counts only the single most-chosen rival construct (Anderson & Gerbing,
#>   1991, p. 734). Pooling every other construct into it instead gives twice Psa
#>   minus one, a different index.
#> 
#> What these columns mean
#>   Psa -- Proportion of Substantive Agreement. Share of judges who put the item
#>       in the construct it was written for (0 to 1; higher is stronger).
#>   95% CI -- Interval for Psa. Wider when fewer judges sorted the item; the
#>       method is named above.
#>   Csv -- Coefficient of Substantive Validity. How much more often judges chose
#>       the intended construct than its closest rival (-1 to 1; 0 is a tie).
#>   Competitor -- Strongest competing construct. The construct other than the
#>       intended one that judges chose most often.
#>   p -- Howard-Melloy exact test. Probability of at least this many target
#>       assignments if each judge picked the target at rate p0; compare with
#>       alpha.
#> 
#> What the decisions mean
#>   Retain -- met the exact target-assignment criterion.
#>   Review -- did not meet the exact target-assignment criterion; the competitor
#>       column shows where judges put it instead.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> A flag for review is not an automatic deletion decision. Use theory,
#> construct-domain coverage, item wording, and qualitative judge feedback
#> alongside these statistics.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.