Plan an expert panel against an explicit decision criterion

Reports the probability that an item will clear its expert-panel criterion, given a panel size and an assumed probability that a single expert endorses the item.

This replaces advice of the form "use six experts" with a question that has an answer: if an expert endorses this item with probability prob, how often will a panel of this size actually clear the criterion? Nothing here recommends a panel size. It reports the consequences of the sizes you ask about, so the choice stays yours and stays documented.

This is a contentvalidR planning tool, not a published method: the probability is binomial, assuming that experts endorse the item independently and with the same probability. The criteria it applies are published (Lynn, 1986; Ayre & Scally, 2014), except Lynn's beyond ten experts, which is this package's extension (see criterion).

Usage

expert_power(
  n_experts = 3:12,
  prob = c(0.7, 0.8, 0.9),
  criterion = c("cvi", "cvr"),
  alpha = 0.05,
  response_rate = 1
)

Arguments

n_experts: Panel sizes to evaluate.
prob: Probability that one expert endorses the item, as relevant (criterion = "cvi") or essential (criterion = "cvr"). Values well below 0.5 describe items the panel largely rejects.
criterion: "cvi" uses Lynn's (1986) panel-size criterion for the I-CVI: every expert must agree with three to five, one may disagree from six, and two from nine (7 of 9, the .78 usually quoted). Beyond ten experts, where Lynn's table stops, the package holds her lowest proportion, 7 of 9: a contentvalidR extension of her rule, not part of it. Polit and Beck (2006, p. 491) restate her rule as no lower than .78 for six or more experts, though their own recommendation covers 6 to 10 experts (p. 496). "cvr" uses the exact Lawshe critical count at level alpha.
alpha: Significance level for the CVR criterion. Ignored for CVI.
response_rate: Expected proportion of invited experts who return usable ratings. When below 1, n_experts is treated as the number invited and the realized panel size is averaged over, so the reported probability accounts for both a smaller panel and the criterion that a smaller panel triggers.

Value

An object of class contentvalid_expert_power: a list whose results data frame holds one row per panel size and probability, with the required endorsement count and the probability of clearing.

Why the curve is not always smooth

The I-CVI criterion is a step function of panel size: every expert must agree up to five, and from six one may disagree. Adding a sixth expert relaxes the criterion and can raise the clearing probability sharply, while adding a fourth or fifth expert under unanimity makes clearing harder. A planning curve that rose smoothly with panel size would be hiding this, so it is reported as it is.

References

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe's content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. doi:10.1177/0748175613513808

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. doi:10.1097/00006199-198611000-00017

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. doi:10.1002/nur.20147

Examples

expert_power(n_experts = 3:10, prob = c(0.8, 0.9))
#> <contentvalid_expert_power> Expert-panel planning
#> Criterion: Exact binomial planning against the panel-size I-CVI guideline
#> 
#> Probability that an item clears the criterion
#>   Experts  Required  Prob = .80  Prob = .90
#>         3       3/3         .51         .73
#>         4       4/4         .41         .66
#>         5       5/5         .33         .59
#>         6       5/6         .66         .89
#>         7       6/7         .58         .85
#>         8       7/8         .50         .81
#>         9       7/9         .74         .95
#>        10      8/10         .68         .93
#> 
#>   Required: endorsements the criterion needs from the panel. Prob: the
#>   probability you assume that one expert endorses the item.
#> 
#>   Note the step at six experts. Lynn's criterion requires unanimity with three
#>   to five experts and allows one disagreement from six, so a sixth expert
#>   relaxes the criterion while a fourth or fifth makes unanimity harder. That
#>   is a property of the guideline, not of the items.
#> 
#> This is a contentvalidR planning tool, not a published power method: it treats
#> the experts as endorsing the item independently, each with probability `prob`,
#> while the criterion it applies is Lynn's (1986). This table reports the
#> consequences of the panel sizes you asked about. It does not recommend one.
#> `prob` is an assumption you supply, so treat the result as conditional on it
#> and report the value you assumed.
#> 
#> See plot(x) for the probabilities by panel size.
expert_power(n_experts = c(5, 10, 15), prob = 0.75, criterion = "cvr")
#> <contentvalid_expert_power> Expert-panel planning
#> Criterion: Exact binomial planning against the Lawshe CVR critical count
#> Alpha: .05
#> 
#> Probability that an item clears the criterion
#>   Experts  Required  Prob = .75
#>         5       5/5         .24
#>        10      9/10         .24
#>        15     12/15         .46
#> 
#>   Required: endorsements the criterion needs from the panel. Prob: the
#>   probability you assume that one expert endorses the item.
#> 
#> This is a contentvalidR planning tool, not a published power method: it treats
#> the experts as endorsing the item independently, each with probability `prob`,
#> while the criterion it applies is the exact test of Ayre and Scally (2014).
#> This table reports the consequences of the panel sizes you asked about. It
#> does not recommend one. `prob` is an assumption you supply, so treat the
#> result as conditional on it and report the value you assumed.
#> 
#> See plot(x) for the probabilities by panel size.