Expert-Panel Content Validation: Relevance, Essentiality, and Congruence

Why expert-panel methods need separate modes

Expert-panel content validation is not one statistical task. A relevance rating, an essential/not-essential judgment, and an item-objective congruence judgment ask experts different questions and therefore support different indices. expert_validity() uses an explicit mode so those designs are not treated as interchangeable.

The three modes are:

• relevance: Aiken’s (1980) V plus CVI and the modified kappa of Polit et al. (2007), with a panel-level agreement coefficient;
• essentiality: Lawshe’s CVR with exact binomial inference; and
• congruence: the index of item-objective congruence (IOC) of Rovinelli and Hambleton (1977).

All three are quantitative complements to qualitative expert comments, construct coverage, comprehensibility review, and other parts of the content-validity argument.

Relevance: Aiken V, score intervals, and CVI

Suppose six experts rate item relevance from 1 (not relevant) to 4 (highly relevant):

R <- matrix(
  c(4,4,4,4,4,4,
    4,4,4,3,4,4,
    4,3,4,4,3,4,
    3,3,4,3,2,3),
  nrow = 6,
  dimnames = list(NULL, paste0("Item", 1:4))
)

fit <- expert_validity(R, mode = "relevance", lo = 1, hi = 4, seed = 1)
fit
#> <contentvalid_expert> Expert-panel analysis
#> Mode: relevance
#> Items: 4 | Experts per item: 6
#> Scale: 1 to 4 | Relevant: a rating of 3 or higher
#> Mean Aiken's V: .88 | S-CVI/Ave: .96 | S-CVI/UA: .75
#> Panel agreement, Krippendorff's alpha (ordinal): .37, 95% CI [-.12, .63].
#>   Identical rating pairs: 63%.
#> 
#> 4 of 4 items meet the I-CVI criterion.
#> 
#>   Item   Decision        Experts     V       95% CI  I-CVI       95% CI  Kappa
#>   Item1  Strong support        6  1.00  [.82, 1.00]   1.00  [.61, 1.00]   1.00
#>   Item2  Strong support        6   .94   [.74, .99]   1.00  [.61, 1.00]   1.00
#>   Item3  Strong support        6   .89   [.67, .97]   1.00  [.61, 1.00]   1.00
#>   Item4  Strong support        6   .67   [.44, .84]    .83   [.44, .97]    .82
#> 
#> Each 95% CI follows its estimate: Aiken's V has a Penfield-Giacobbi score
#> interval, and I-CVI the proportion interval named below.
#> I-CVI criterion for 6 experts: 5 agreeing (.83), following Lynn (1986). Kappa
#> is modified kappa, read as excellent above .74 (Polit et al., 2007).
#> 95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#> compared seven methods and recommends score intervals over the Wald interval.
#> An interval reflects how few ratings an item received, not whether the right
#> people rated it.
#> 
#> Panel agreement is one coefficient for the whole panel, whereas modified kappa
#> (the kappa column) describes each item. Alpha can be low when nearly every
#> rating is the same value, even on a panel that agrees closely, so read it
#> beside the share of identical rating pairs. A low alpha with many identical
#> pairs is not by itself evidence of a poor panel. Print x$details$agreement for
#> the full explanation and interval details.
#> 
#> CVI criteria are published panel-size guidelines, not universal validity
#> cutoffs.
#> 
#> What these columns mean
#>   S-CVI/Ave -- Scale-level CVI, averaging method. Mean of the items' I-CVIs;
#>       Polit and Beck (2006) recommend .90 or higher.
#>   S-CVI/UA -- Scale-level CVI, universal agreement. Share of items every
#>       expert rated relevant; it falls as experts are added.
#>   V -- Aiken's V. Mean relevance rating rescaled to run from 0 (lowest
#>       possible) to 1 (highest).
#>   I-CVI -- Item-level Content Validity Index. Share of experts rating the item
#>       relevant, against Lynn's criterion for the panel size (beyond ten, this
#>       package's).
#>   95% CI after I-CVI -- Interval for I-CVI. Wide because expert panels are
#>       small; the method is named above.
#>   Kappa -- Modified kappa. I-CVI corrected for chance agreement (at most 1;
#>       below 0 only when no expert, or one of three, rated it relevant).
#>   Panel agreement -- Panel-level agreement. One coefficient for the whole
#>       panel (1 is perfect, 0 is chance); it can be low when nearly every
#>       rating is the same.
#> 
#> What the decisions mean
#>   Strong support -- met the I-CVI criterion, which also puts modified kappa
#>       above .74.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> Use quantitative indices alongside expert comments, construct coverage, and
#> comprehensibility review.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.
summary(fit)
#> <contentvalid_expert summary> Expert-panel analysis
#> Mode: relevance
#> Scale: 1 to 4 | Relevant: a rating of 3 or higher
#> Strong support: 4 of 4 | Review: 0 of 4
#> I-CVI criterion for 6 experts: 5 agreeing (.83), following Lynn (1986).
#> Panel agreement, Krippendorff's alpha (ordinal): .37, 95% CI [-.12, .63].
#>   Identical rating pairs: 63%.
#> 
#> No items were flagged by the workflow's quantitative review rules.
#> 
#> These summaries support, but do not replace, qualitative content review.
#> 
#> See summary(x)$reviewed_items for the flagged items as a data frame.

Aiken’s V rescales the bounded expert ratings to the 0-1 interval. The default confidence interval is the score interval proposed by Penfield and Giacobbi (2004), rather than a simulation-dependent bootstrap interval. Bootstrap intervals remain available through the low-level function:

aikens_v(R, lo = 1, hi = 4, ci = "bootstrap", B = 200, seed = 1)
#> <contentvalid_aiken> Aiken's V
#> Aiken (1980).
#> 
#>   Item   Experts     V        95% CI
#>   Item1        6  1.00  [1.00, 1.00]
#>   Item2        6   .94   [.83, 1.00]
#>   Item3        6   .89   [.78, 1.00]
#>   Item4        6   .67    [.50, .83]
#> 
#> Scale: 1 to 4.
#> Interval: percentile bootstrap.
#> 
#> See as.data.frame(x) for the unrounded values.

For CVI, the workflow dichotomizes ratings at relevance_cut. On a 1-4 scale the default is 3, so ratings of 3 or 4 count as relevant. Declare a different threshold if the study protocol used one.

The workflow reports Lynn’s (1986) panel-size criterion for the I-CVI as a review aid: with three to five experts all must agree, from six one may disagree, and from nine two may (7 of 9, the .78 usually quoted). Lynn’s table stops at ten experts; beyond that the package holds her lowest proportion, 7 of 9, an extension of her rule that the printout and the handoff label as this package’s own. Polit and Beck (2006) restate her rule as no lower than .78 for six or more experts, though their own recommendation covers six to ten. The criterion is not presented as universal proof that an item is or is not content valid. Modified kappa provides a chance-corrected complement to I-CVI.

At the scale level, S-CVI/Ave and S-CVI/UA are reported together. S-CVI/Ave is generally less brittle than universal agreement, but both should be interpreted alongside the distribution of item-level evidence.

Panel-level agreement

I-CVI and modified kappa describe one item at a time. Relevance mode also reports how consistently the panel rated the whole item set, as one coefficient with a bootstrap interval:

fit$scale_summary[, c("agreement", "agreement_low", "agreement_high")]
#>   agreement agreement_low agreement_high
#> 1 0.3743873    -0.1210084      0.6340909
fit$details$agreement
#> <contentvalid_agreement> Panel agreement
#> Items rated by two or more raters: 4 | Raters: 6
#> Krippendorff's alpha (ordinal) = .37, 95% CI [-.12, .63]
#> Identical rating pairs: 63%
#> 
#> Alpha compares the disagreement observed within items with the disagreement
#> expected if these same ratings were assigned to items at random: 1 means
#> perfect agreement and 0 means agreement no better than chance. Alpha falls
#> when ratings cluster on a few values, because little disagreement is then
#> expected by chance. A high share of identical rating pairs alongside a low
#> alpha reflects that clustering, which is common when nearly every item is
#> rated relevant, and is not by itself evidence of a poor panel.
#> 
#> Krippendorff's alpha is the default because it handles ordinal ratings and
#> missing ratings (Zapf et al., 2016). It is a general reliability coefficient;
#> no publication applying it specifically to content-validity panels was found.
#> 
#> The interval resamples items with all of their ratings, following Zapf et al.
#> (2016), and varies slightly between runs unless `seed` is set. In 5 of 1000
#> resamples the coefficient could not be computed, usually because every
#> resampled rating was identical; the interval uses the rest. With few items
#> this interval is imprecise and can be misleading.
#> 
#> Panel agreement describes how consistently raters rated these items. It does
#> not show that the items are relevant or that the domain is covered.
#> 
#> See as.data.frame(x) for the estimate as one row.

The default coefficient is Krippendorff’s alpha (Krippendorff, 2011). It accepts any number of experts and missing ratings, and Zapf et al. (2016) recommend it when ratings are ordinal or incomplete, which describes most expert panels. It is a general reliability coefficient (Hayes & Krippendorff, 2007) rather than one developed for content validity; no publication applying it specifically to content-validity panels was found.

Choose the measurement level that matches the rating scale. Relevance ratings are treated as ordinal by default. agreement_level = "interval" treats the distances between scale points as equal, and "nominal" treats every disagreement as equally serious:

expert_validity(R, mode = "relevance", lo = 1, hi = 4,
                agreement_level = "interval", agreement_B = 0)$scale_summary$agreement
#> [1] 0.3715847

Why alpha can be low when experts agree

Alpha compares the disagreement within items with the disagreement expected if the same ratings were scattered across items at random. When a panel rates nearly every item 4, very little disagreement is expected by chance, so a few 3s pull alpha down even though most rating pairs are identical. Feinstein and Cicchetti (1990) described the same pattern for kappa. The output reports the share of identical rating pairs next to alpha so the two can be read together. A low alpha alongside a high share of identical pairs is not by itself evidence of a poor panel.

Gwet’s AC1

Gwet’s (2008) AC1 was designed to stay high in that situation, and it is available with agreement = "ac1". It is never the default. Vach and Gerke (2023) show that AC1 rises as ratings concentrate in one category even when agreement does not change, and that it can be above zero when experts rate independently. Its output always repeats that critique. In relevance mode, AC1 is computed on the relevant/not-relevant decision at relevance_cut:

ac1_fit <- expert_validity(R, mode = "relevance", lo = 1, hi = 4,
                           agreement = "ac1", agreement_B = 0)
ac1_fit$details$agreement
#> <contentvalid_agreement> Panel agreement
#> Items rated by two or more raters: 4 | Raters: 6
#> Gwet's AC1 = .91
#> Identical rating pairs: 92%
#> 
#> AC1 compares observed agreement with the agreement expected by chance,
#> estimated so that it stays high when nearly every rating falls in one category
#> (Gwet, 2008).
#> 
#> Gwet's AC1 is available but is not the default. Vach and Gerke (2023) show
#> that it rises as ratings concentrate in one category even when agreement is
#> unchanged, that it can be non-zero when raters are independent, and that
#> benchmark labels developed for kappa, such as Landis and Koch's, must not be
#> applied to it.
#> 
#> Panel agreement describes how consistently raters rated these items. It does
#> not show that the items are relevant or that the domain is covered.
#> 
#> See as.data.frame(x) for the estimate as one row.

The interval

The interval resamples items with all of their ratings intact, the procedure Zapf et al. (2016) evaluated; they found that Krippendorff’s original bootstrap, which ignores dependence between raters, reached only about 60% coverage. The interval varies slightly between runs, so set seed to make it reproducible, or set agreement_B = 0 to skip it. panel_agreement() runs the same analysis on any rater-by-item matrix.

Essentiality: Lawshe CVR with exact critical values

Lawshe’s task asks experts whether an item is essential. With twelve experts:

expert_validity(c(10, 8, 6), mode = "essentiality", N = 12)
#> <contentvalid_expert> Expert-panel analysis
#> Mode: essentiality
#> Items: 3 | Experts per item: 12
#> Method: Lawshe CVR with exact binomial critical values
#> 
#> 1 of 3 items meets the exact essentiality criterion.
#> Flagged for review: Item2, Item3
#> 
#>   Item   Decision   Essential  CVR     p
#>   Item1  Supported      10/12  .67  .019
#>   Item2  Review          8/12  .33  .194
#>   Item3  Review          6/12  .00  .613
#> 
#> Essential: experts rating the item essential, out of those who rated it.
#> With 12 experts, an item needs at least 10 rating it essential for the exact
#> one-sided binomial test at alpha = .05 (Ayre & Scally, 2014).
#> 
#> What these columns mean
#>   CVR -- Lawshe's Content Validity Ratio. Lean of the panel toward calling the
#>       item essential (-1 to 1; above 0 means more than half did).
#> 
#> What the decisions mean
#>   Supported -- enough experts rated it essential to pass the exact test.
#>   Review -- too few experts rated it essential to pass the exact test.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> Use quantitative indices alongside expert comments, construct coverage, and
#> comprehensibility review.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.

cvr() derives the smallest essential count whose one-sided binomial upper-tail probability is no greater than alpha. This makes the panel-size dependency explicit and follows the exact-probability logic revisited by Ayre and Scally (2014).

Judge-by-item binary data can be supplied directly:

E <- cbind(
  Item1 = c(1,1,1,1,1,1,1,1),
  Item2 = c(1,1,1,1,1,0,0,0)
)
expert_validity(E, mode = "essentiality")
#> <contentvalid_expert> Expert-panel analysis
#> Mode: essentiality
#> Items: 2 | Experts per item: 8
#> Method: Lawshe CVR with exact binomial critical values
#> 
#> 1 of 2 items meets the exact essentiality criterion.
#> Flagged for review: Item2
#> 
#>   Item   Decision   Essential   CVR     p
#>   Item1  Supported        8/8  1.00  .004
#>   Item2  Review           5/8   .25  .363
#> 
#> Essential: experts rating the item essential, out of those who rated it.
#> With 8 experts, an item needs at least 7 rating it essential for the exact
#> one-sided binomial test at alpha = .05 (Ayre & Scally, 2014).
#> 
#> What these columns mean
#>   CVR -- Lawshe's Content Validity Ratio. Lean of the panel toward calling the
#>       item essential (-1 to 1; above 0 means more than half did).
#> 
#> What the decisions mean
#>   Supported -- enough experts rated it essential to pass the exact test.
#>   Review -- too few experts rated it essential to pass the exact test.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> Use quantitative indices alongside expert comments, construct coverage, and
#> comprehensibility review.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.

A failure to clear the exact criterion is labeled Review, not automatic deletion. Expert rationales and domain coverage matter when deciding whether an item should be rewritten, retained for breadth, or removed.

With four or fewer experts no count can clear the criterion at the default alpha: all four calling an item essential gives p = .0625. Such an item is labeled Insufficient panel, not Review, because the panel is too small to decide, whatever the experts said.

Lawshe’s table and its recalculation, for comparison

Lawshe (1975) published a table of minimum CVR values, and Wilson et al. (2012) recalculated it. legacy = TRUE prints both beside the exact decision, without changing it:

old <- options(contentvalidR.show_key = FALSE)
expert_validity(c(9, 8, 7), mode = "essentiality", N = 9, legacy = TRUE)
#> <contentvalid_expert> Expert-panel analysis
#> Mode: essentiality
#> Items: 3 | Experts per item: 9
#> Method: Lawshe CVR with exact binomial critical values
#> 
#> 2 of 3 items meet the exact essentiality criterion.
#> Flagged for review: Item3
#> 
#>   Item   Decision   Essential   CVR     p
#>   Item1  Supported        9/9  1.00  .002
#>   Item2  Supported        8/9   .78  .020
#>   Item3  Review           7/9   .56  .090
#> 
#> Essential: experts rating the item essential, out of those who rated it.
#> With 9 experts, an item needs at least 8 rating it essential for the exact
#> one-sided binomial test at alpha = .05 (Ayre & Scally, 2014).
#> 
#> Earlier methods, for comparison (not used for the decision)
#>   Item   Decision   Essential   CVR  Lawshe (1975)  Wilson et al. (2012)
#>   Item1  Supported        9/9  1.00  Meets          Meets
#>   Item2  Supported        8/9   .78  Meets          Meets
#>   Item3  Review           7/9   .56  Below          Meets
#> 
#>   Lawshe (1975, Table 1): minimum CVR .78 for 9 panelists (8 of 9), which he
#>   labeled a one-tailed test at .05. Wilson et al. (2012) found the table
#>   closer to a two-tailed test.
#>   Wilson et al. (2012, Table 2): minimum CVR .55, the normal approximation
#>   z/sqrt(N) at one-tailed alpha = .05.
#>   The decision above uses the exact binomial test (Ayre & Scally, 2014).
#>   Lawshe's content validity index, the mean CVR of the items his table
#>   retains: .89 (2 items).
#> 
#> Use quantitative indices alongside expert comments, construct coverage, and
#> comprehensibility review.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.
options(old)

Nine experts show why the table was questioned. Its minimum rises steadily from 40 experts down to nine, then drops from .78 at nine to .75 at eight, which Wilson et al. (2012) treated as an anomaly. Ayre and Scally (2014) showed it is not one. With so few experts only a few CVR values are possible: .75 is 7 of 8, and .78 is 8 of 9 (.778) printed to two decimals, the same counts the exact test requires.

In relevance mode, legacy = TRUE adds Fleiss’ (1971) kappa on the relevant/not-relevant decision and the Polit and Beck (2006) benchmarks for S-CVI/Ave and S-CVI/UA. It also adds the content validity coefficient (Ccv) of Hernández-Nieto (2002), which is widely cited, beside Aiken’s V. Read it as a cautionary example rather than evidence. Ccv is the mean rating divided by the scale maximum, minus a “chance” term, (1/J)^J, that depends only on the number of judges and is .00032 for five of them. So on a scale starting at 0 it is Aiken’s V, and on a scale starting at 1 it cannot fall below one over the scale maximum. It uses only the mean, so despite the book’s claim it cannot reflect agreement: in the book’s own Table 7, ratings of 1, 3, 4, 5, 2 and of 3, 3, 3, 3, 3 both score .60. Its .80 cutoff is stated without derivation, and the book’s own examples do not keep to it. ?expert_validity sets out each of these with page references.

Congruence: item-objective alignment

Each expert rates each item against every objective: +1 if the item clearly measures it, 0 if unclear, -1 if it clearly does not. The index of Rovinelli and Hambleton (1977) for an objective is half the gap between the experts’ mean rating on that objective and their mean rating on the item’s other objectives, so it reaches 1 only when every expert gives +1 to the objective and -1 to every other. Turner and Carlson (2003) extend the index to items written for more than one objective, which this package does not do. A target mapping says which objective each item was written for:

d <- expand.grid(
  item = c("I1", "I2"),
  judge = 1:4,
  objective = c("A", "B")
)
d$target_objective <- ifelse(d$item == "I1", "A", "B")
d$score <- ifelse(d$objective == d$target_objective, 1, -1)

expert_validity(d, mode = "congruence")
#> <contentvalid_expert> Expert-panel analysis
#> Mode: congruence
#> Items: 2 | Experts per cell: 4 | Objectives: 2
#> Method: Index of item-objective congruence (Rovinelli & Hambleton, 1977)
#> Criterion: IOC at or above .70, the criterion Rovinelli and Hambleton applied.
#> 
#> 2 of 2 items meet the congruence criterion for the target objective.
#> 
#>   Item Target Decision  Experts  IOC Mean Competitor Competitor mean Margin
#>   I1   A      Congruent       4 1.00 1.00 B                    -1.00   2.00
#>   I2   B      Congruent       4 1.00 1.00 A                    -1.00   2.00
#> 
#> IOC: the index for the target objective. Mean: the experts' mean rating on the
#> target objective (-1 to 1). Competitor mean: the highest mean on another
#> objective. Margin: mean less competitor mean, a description beside the index,
#> not part of its criterion.
#> 
#> I1, I2: The index of item-objective congruence for the intended objective is
#>   at or above .70: the experts matched the item to it and not to the other
#>   objectives.
#> 
#> What these columns mean
#>   IOC -- Index of item-objective congruence. Whether experts matched the item
#>       to this objective and not to the others (-1 to 1; Rovinelli and
#>       Hambleton used .70).
#> 
#> What the decisions mean
#>   Congruent -- its index of item-objective congruence met the criterion.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> Use quantitative indices alongside expert comments, construct coverage, and
#> comprehensibility review.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.

An item is "Congruent" when its index for the target objective reaches ioc_cut, by default the .70 Rovinelli and Hambleton applied. The printout also shows the experts’ mean rating on the target and on the other objective with the highest mean rating, and the margin between those means. Applied work often reports the mean rating on the target as “the IOC”; it is an ingredient of the index, not the index, because it reaches 1 whatever the experts say about the other objectives. The margin is a description beside the index, not part of its criterion. Without a target mapping, each item’s index on every objective is described, with no decision.

Missing ratings

Missing data are never silently ignored by default. Set na.rm = TRUE only when itemwise/cellwise deletion matches the study protocol. Effective expert counts and missing counts are then reported so downstream interpretation uses the actual panel size.

Plotting expert evidence

Each mode uses a plot matched to the expert task rather than forcing unlike indices into one generic chart.

plot(expert_validity(R, mode = "relevance", lo = 1, hi = 4))

plot(expert_validity(c(10, 8, 6), mode = "essentiality", N = 12))

plot(expert_validity(d, mode = "congruence"))

Relevance mode displays Aiken’s V with its score interval and overlays I-CVI as a separate marker. Essentiality mode displays observed CVR against the exact panel-specific critical CVR, and marks with a cross an item whose panel was too small to test. Congruence mode places each item’s index against the criterion, with the two mean ratings behind it in gray. These displays are diagnostic summaries; they do not create new validity thresholds.

An index reduces each item’s ratings to one number, and the relevance plot shows that number. The distribution view shows the ratings behind it, as diverging stacked bars (Heiberger & Robbins, 2014). Each bar splits at the relevance cut: ratings below it extend left and ratings at or above it extend right, so the right-hand length is the I-CVI, read against the dashed criterion line.

plot(fit, type = "distribution",
     labels = c("Not relevant", "Somewhat", "Quite", "Highly relevant"))

Item1 and Item3 have the same I-CVI, but every expert gave Item1 a 4, while two gave Item3 a 3. The index cannot tell them apart; the bars can, and so can Aiken’s V, which uses the whole scale. The figure is gray by default, as an APA figure is printed. For slides or a poster, apa = FALSE draws it in color: brown for ratings below the cut and teal for ratings at or above it. The symbol beside each bar still carries the decision, so no reading depends on color.

Reporting

A concise methods/results description should identify:

1. who the experts were and why they were qualified;
2. the exact task and response scale;
3. the index and inference/CI procedure used, and for relevance ratings the agreement coefficient and its measurement level;
4. the panel size, including item-specific missingness;
5. quantitative item and scale evidence; and
6. how expert comments, construct coverage, and comprehensibility informed the final item decisions.

A content-validity coefficient is evidence about a defined expert task. It is not, by itself, a complete validity argument.

References

Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. https://doi.org/10.1348/000711006X126600

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. https://doi.org/10.1080/19312450709336664

Heiberger, R. M., & Robbins, N. B. (2014). Design of diverging stacked bar charts for Likert scales and other applications. Journal of Statistical Software, 57(5), 1–32. https://doi.org/10.18637/jss.v057.i05

Hernández-Nieto, R. (2002). Contributions to statistical analysis: The coefficients of proportional variance, content validity and kappa. BookSurge.

Krippendorff, K. (2011). Computing Krippendorff’s alpha-reliability. Annenberg School for Communication, University of Pennsylvania. https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf

Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017

Penfield, R. D., & Giacobbi, P. R., Jr. (2004). Applying a score confidence interval to Aiken’s item content-relevance index. Measurement in Physical Education and Exercise Science, 8(4), 213–225. https://doi.org/10.1207/S15327841MPEE0804_3

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199

Rovinelli, R. J., & Hambleton, R. K. (1977). On the use of content specialists in the assessment of criterion-referenced test item validity. Dutch Journal of Educational Research, 2, 49–60.

Turner, R. C., & Carlson, L. (2003). Indexes of item-objective congruence for multidimensional items. International Journal of Testing, 3(2), 163–171. https://doi.org/10.1207/S15327574IJT0302_5

Vach, W., & Gerke, O. (2023). Gwet’s AC1 is not a substitute for Cohen’s kappa: A comparison of basic properties. MethodsX, 10, Article 102212. https://doi.org/10.1016/j.mex.2023.102212

Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45(3), 197–210. https://doi.org/10.1177/0748175612440286

Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater reliability for nominal data: Which coefficients and confidence intervals are appropriate? BMC Medical Research Methodology, 16, Article 93. https://doi.org/10.1186/s12874-016-0200-9