Skip to contents

Content Validity Index (CVI)

Source: R/cvi.R
cvi.Rd

Computes item-level Content Validity Index (I-CVI), scale-level average CVI (S-CVI/Ave), universal-agreement CVI (S-CVI/UA), and the modified kappa described by Polit et al. (2007).

The I-CVI is the proportion of experts rating an item 3 or 4 on a 4-point relevance scale (Lynn, 1986). Polit and Beck (2006) named the two scale-level versions after finding that published studies used both without saying which: S-CVI/UA is the share of items every expert rated relevant, and S-CVI/Ave is the average of the I-CVIs, the same quantity as the average congruency percentage. The two can differ sharply, since S-CVI/UA falls as experts are added, so Polit and Beck recommend reporting both.

For each item, modified kappa adjusts I-CVI for chance agreement using the probability of observing exactly A agreements among N judges:

$$P_c = {N \choose A}(0.5)^N$$

and

$$k^* = \frac{\textrm{I-CVI} - P_c}{1 - P_c}.$$

I-CVI is a proportion of what is usually a small panel, so an interval is reported alongside it. The interval method is selectable; see ci.

Usage

cvi(
  binary,
  na.rm = FALSE,
  ci = c("wilson", "agresti_coull", "exact", "none"),
  alpha = 0.05
)

Arguments

binary

Matrix/data.frame with judges in rows and items in columns, coded 1 = relevant and 0 = not relevant. Every column is an item; a column whose name looks like a rater ID (such as expert or rater_id) stops the function, so remove it, or rename an item that has such a name.

na.rm

Logical. If FALSE (default), missing ratings are an error. If TRUE, missing ratings are removed itemwise and each item's effective judge count is reported in N.

ci

Interval method for I-CVI:

  • "wilson" (default): the Wilson (1927) score interval. Newcombe (1998) compared seven methods and recommends score intervals over the Wald interval.

  • "agresti_coull": the adjusted Wald interval of Agresti and Coull (1998), which they show performs well even in small samples. Limits are clipped so they stay between 0 and 1.

  • "exact": the Clopper and Pearson (1934) interval. It is conservative: Agresti and Coull (1998) show its coverage runs above the nominal level.

  • "none": no interval is computed, and the interval columns are NA.

alpha

Two-sided alpha level for the interval; 0.05 gives a 95% interval.

Value

A classed list with:

  • item_level: item, A, N, I_CVI, I_CVI_low, I_CVI_high, Pc, kappa_mod

  • scale_level: S_CVI_Ave and S_CVI_UA

  • ci and alpha: the interval settings used

References

Agresti, A., & Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126. doi:10.1080/00031305.1998.10480550

Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413. doi:10.1093/biomet/26.4.404

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. doi:10.1097/00006199-198611000-00017

Newcombe, R. G. (1998). Two-sided confidence intervals for the single proportion: Comparison of seven methods. Statistics in Medicine, 17(8), 857–872. doi:10.1002/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. doi:10.1002/nur.20147

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. doi:10.1002/nur.20199

Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. doi:10.1080/01621459.1927.10502953

Examples

M <- matrix(
  c(1,1,1,1, 1,1,1,0, 1,1,0,0),
  nrow = 4,
  dimnames = list(NULL, c("Item1", "Item2", "Item3"))
)
cvi(M)
#> <contentvalid_cvi> Content validity index (CVI)
#> Items: 3 | Experts per item: 4
#> S-CVI/Ave: .75 | S-CVI/UA: .33
#> I-CVI = item-level content validity index; S-CVI/Ave = scale-level CVI, the
#> mean I-CVI; S-CVI/UA = scale-level CVI, the share of items every expert rated
#> relevant (Polit & Beck, 2006).
#> 
#> Item-level results
#>   Item   Agree  I-CVI       95% CI    Pc  Kappa
#>   Item1    4/4   1.00  [.51, 1.00]  .063   1.00
#>   Item2    3/4    .75   [.30, .95]  .250    .67
#>   Item3    2/4    .50   [.15, .85]  .375    .20
#> 
#>   Agree: experts rating the item relevant, out of those who rated it. Pc: the
#>   probability that this many experts would agree by chance. Kappa: the
#>   modified kappa of Polit et al. (2007), the I-CVI chance-corrected by Pc.
#> 
#>   95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#>   compared seven methods and recommends score intervals over the Wald
#>   interval. An interval reflects how few ratings an item received, not whether
#>   the right people rated it.
#> 
#> Polit and Beck (2006) recommend reporting both S-CVI/Ave and S-CVI/UA.
#> Interpretation should consider panel size, item purpose, and qualitative
#> expert feedback; CVI statistics alone do not establish comprehensive content
#> validity.
#> 
#> See as.data.frame(x) for the item-level values.
cvi(M, ci = "exact")
#> <contentvalid_cvi> Content validity index (CVI)
#> Items: 3 | Experts per item: 4
#> S-CVI/Ave: .75 | S-CVI/UA: .33
#> I-CVI = item-level content validity index; S-CVI/Ave = scale-level CVI, the
#> mean I-CVI; S-CVI/UA = scale-level CVI, the share of items every expert rated
#> relevant (Polit & Beck, 2006).
#> 
#> Item-level results
#>   Item   Agree  I-CVI       95% CI    Pc  Kappa
#>   Item1    4/4   1.00  [.40, 1.00]  .063   1.00
#>   Item2    3/4    .75   [.19, .99]  .250    .67
#>   Item3    2/4    .50   [.07, .93]  .375    .20
#> 
#>   Agree: experts rating the item relevant, out of those who rated it. Pc: the
#>   probability that this many experts would agree by chance. Kappa: the
#>   modified kappa of Polit et al. (2007), the I-CVI chance-corrected by Pc.
#> 
#>   95% intervals for proportions: Clopper-Pearson exact. This is conservative:
#>   Agresti and Coull (1998) show its coverage runs above the nominal level, so
#>   intervals are wider than they need to be. An interval reflects how few
#>   ratings an item received, not whether the right people rated it.
#> 
#> Polit and Beck (2006) recommend reporting both S-CVI/Ave and S-CVI/UA.
#> Interpretation should consider panel size, item purpose, and qualitative
#> expert feedback; CVI statistics alone do not establish comprehensive content
#> validity.
#> 
#> See as.data.frame(x) for the item-level values.