Content review and an empirical screen answer different questions. An expert panel judges whether an item represents the construct as it was defined. Response data show how the item behaves: whether it varies, what it shares with the other items, and whether it works the same way in different groups. Neither answer contains the other, and an item can pass one and fail the other.
This article takes one item set across that boundary. A content
review sorted twelve items and carried ten of them forward. Here those
ten are screened on 400 simulated responses. The content-review half,
meaning how the panel’s sort was analyzed and how the handoff was made,
is the joint walkthrough in contentvalidR, One
item set, both stages. Both articles use the same data. The data are
simulated from a model that is documented in
?nomo_demo_walkthrough, so every claim below can be checked
against the truth.
What content review decided
contentvalidR records its decisions in a
handoff. This package stores the handoff for these items, so
the article runs without contentvalidR installed:
handoff <- readRDS(
system.file("extdata", "content-handoff-walkthrough.rds", package = "nomologR")
)
handoff$item_evidence[, c("item", "scale", "status", "carried", "keying")]
#> item scale status carried keying
#> 1 EF1 EF Supported TRUE 1
#> 2 EF2 EF Supported TRUE -1
#> 3 EF3 EF Supported TRUE 1
#> 4 EF4 EF Supported TRUE 1
#> 5 EF5 EF Review FALSE 1
#> 6 EF6 EF Supported TRUE 1
#> 7 TF1 TF Supported TRUE 1
#> 8 TF2 TF Supported TRUE -1
#> 9 TF3 TF Supported TRUE 1
#> 10 TF4 TF Supported TRUE 1
#> 11 TF5 TF Review FALSE 1
#> 12 TF6 TF Supported TRUE 1The panel carried ten items. It held back EF5 and
TF5, whose sorts did not meet the criterion. It placed the
carried items in two facets:
- Effort Regulation (
EF), keeping going when the work is dull or hard; - Task Focus (
TF), staying on the task rather than switching away.
Keying -1 marks EF2 and TF2 as
reverse-worded: a high answer means less persistence.
The responses, and what each item was built to do
nomo_demo_walkthrough_items[, c("item", "role")]
#> item role
#> 1 EF1 ordinary: the panel places it and it behaves as intended
#> 2 EF2 reverse-worded: behaves as intended once recoded
#> 3 EF3 flagged by an empirical screen and worth keeping anyway
#> 4 EF4 passes content review, then carries almost no common variance
#> 5 EF5 fails content review: the wording pulls judges toward test anxiety
#> 6 EF6 meets the content criterion by one judge, then behaves well
#> 7 TF1 ordinary: the panel places it and it behaves as intended
#> 8 TF2 reverse-worded: behaves as intended once recoded
#> 9 TF3 ordinary: the panel places it and it behaves as intended
#> 10 TF4 passes content review, then loads on both facets
#> 11 TF5 fails content review: a competing facet takes more assignments
#> 12 TF6 behaves differently in the two cohortsA real study does not come with a role column. Here it
is known because the data were simulated to contain specific problems,
and it is the answer key for what follows.
Screening the data as they arrive
nomo_run() takes the handoff whole: its carried items
and facets become the run’s scales, and each facet is screened on its
own.
run <- nomo_run(nomo_demo_walkthrough, scales = handoff)
run
#> <nomo_run> Guided workflow
#> Status: Paused | Mode: teaching | Sample design: same sample
#> Exploratory cases: 400 | Confirmatory cases: 400 | Scales: 2
#> Completed: screen -> factors | Next: EFA
#>
#> Key evidence
#> - Item audit: 10 items; flags: 10 review, 0 concern
#> - Parallel analysis suggests: EF 1, TF 1
#>
#> Researcher decision required on the factor counts
#> Reason: The EFA factor count changes the fitted model. Retention evidence
#> can inform that choice, but it does not authorize the pipeline to choose for
#> the researcher.
#> Options: Inspect the full `nomo_factors` result, compare plausible
#> neighboring solutions when appropriate, and supply a positive integer for
#> each scale.
#> Consequence: No EFA is fitted until an explicit researcher factor-count
#> decision is supplied.
#> - EF: Parallel analysis currently suggests 1 factor; the retained plausible
#> set is 1. The pipeline has not adopted a factor count.
#> - TF: Parallel analysis currently suggests 1 factor; the retained plausible
#> set is 1. The pipeline has not adopted a factor count.
#> Example: decisions = list(factor_count = c(EF = <integer>, TF = <integer>))
#>
#> No later stage has been run automatically while this consequential decision is
#> unresolved.
#>
#> EFA = exploratory factor analysis.
#>
#> See summary(x) for the stages and recorded decisions and
#> nomo_table(x, "requests") for the decision requests.Every item is flagged. The Effort Regulation audit shows why:
summary(run$results$screen$EF)
#> <nomo_screen summary> Item and data audit
#> Cases: 400 | Items: 5 | Item flags: 5 review, 0 concern
#> Items with missing responses: 0 | Constant: 0 | All missing: 0
#> Items in correlation diagnostics: 5
#>
#> Item review
#> Item Type Missing Top share Item-rest r Flag
#> EF1 discrete 0.0% 34.8% .17 Review
#> EF2 discrete 0.0% 28.5% -.50 Review
#> EF3 discrete 0.0% 78.5% .21 Review
#> EF4 discrete 0.0% 33.8% .17 Review
#> EF6 discrete 0.0% 32.5% .18 Review
#> Top share is the share of observed responses in the most common category.
#> Item-rest r is the correlation of an item with the sum of the other items.
#>
#> Flagged
#> - EF1 (Review): `EF1` has a corrected item-rest correlation of r = .17
#> (n = 400), below the teaching reference of .30. Its rest score includes
#> EF2, declared reverse-keyed and not yet recoded in the data; with EF2
#> recoded as declared, its item-rest correlation is r = .56. Its correlation
#> with EF2 is negative.
#> - EF2 (Review): `EF2` has a negative corrected item-rest correlation
#> (r = -.50, n = 400). It is declared reverse-keyed, and this is the sign
#> such an item shows before it is recoded: recoded on the declared 1 to 5
#> scale, its item-rest correlation is r = .50. The data were not recoded.
#> Its correlations with EF1, EF3, EF4, and EF6 are negative.
#> - EF3 (Review): `EF3` has a corrected item-rest correlation of r = .21
#> (n = 400), below the teaching reference of .30. Its rest score includes
#> EF2, declared reverse-keyed and not yet recoded in the data; with EF2
#> recoded as declared, its item-rest correlation is r = .34. Its correlation
#> with EF2 is negative.
#> - EF4 (Review): `EF4` has a corrected item-rest correlation of r = .17
#> (n = 400), below the teaching reference of .30. Its rest score includes
#> EF2, declared reverse-keyed and not yet recoded in the data; with EF2
#> recoded as declared, its item-rest correlation is r = .23. Its correlation
#> with EF2 is negative.
#> - EF6 (Review): `EF6` has a corrected item-rest correlation of r = .18
#> (n = 400), below the teaching reference of .30. Its rest score includes
#> EF2, declared reverse-keyed and not yet recoded in the data; with EF2
#> recoded as declared, its item-rest correlation is r = .47. Its correlation
#> with EF2 is negative.
#> - Inter-item correlations (Review): 4 estimable inter-item correlations are
#> negative. All involve EF2, declared reverse-keyed and not yet recoded in
#> the data.
#>
#> Flags are review aids, not decisions to keep or delete an item.
#>
#> See nomo_table(x, "decision_log") for every log entry and plot(x) for the item
#> evidence map.EF2 correlates negatively with the rest of its facet.
The audit says what that most likely means: the item is declared
reverse-keyed, and recoded as declared its correlation would be
positive. The negative sign is a coding matter, not evidence against the
item.
Left as answered, the item also makes the rest of the facet look bad.
Every other EF item’s correlation with the rest of the scale includes
EF2 running the wrong way, so all of them fall below the
teaching reference. A screen of data that are not yet recoded cannot say
much about any item in the facet.
nomologR never recodes data. Recoding is the
researcher’s step, and it belongs in the analysis code where it can be
seen.
Recoding as content review declared
walk <- nomo_demo_walkthrough
reversed <- handoff$item_evidence$item[handoff$item_evidence$keying %in% -1]
walk[reversed] <- 6L - walk[reversed]The responses are now recoded, so the run should not treat any item
as still needing it. reverse = character(0) says so, in
place of the reverse-keyed items the handoff declared. It replaces only
that part of the handoff’s keying: the response scale still comes from
the handoff, and the run’s decision log records the change.
run <- nomo_run(
walk,
scales = handoff,
settings = list(screen = list(reverse = character(0)))
)
run$decision_log$observation[run$decision_log$id == "keying"]
#> [1] "`settings$screen$reverse` (none) was used in place of the reverse-keyed item(s) the content-review handoff declared (EF2, TF2)."
ef <- run$results$screen$EF
summary(ef)
#> <nomo_screen summary> Item and data audit
#> Cases: 400 | Items: 5 | Item flags: 1 review, 0 concern
#> Items with missing responses: 0 | Constant: 0 | All missing: 0
#> Items in correlation diagnostics: 5
#>
#> Item review
#> Item Type Missing Top share Item-rest r Flag
#> EF1 discrete 0.0% 34.8% .56
#> EF2 discrete 0.0% 28.5% .50
#> EF3 discrete 0.0% 78.5% .34
#> EF4 discrete 0.0% 33.8% .23 Review
#> EF6 discrete 0.0% 32.5% .47
#> Top share is the share of observed responses in the most common category.
#> Item-rest r is the correlation of an item with the sum of the other items.
#>
#> Flagged
#> - EF4 (Review): `EF4` has a corrected item-rest correlation of r = .23
#> (n = 400), below the teaching reference of .30.
#>
#> Flags are review aids, not decisions to keep or delete an item.
#>
#> See nomo_table(x, "decision_log") for every log entry and plot(x) for the item
#> evidence map.Recoded, EF2 correlates .50 with the rest of its facet,
and one EF item remains flagged: EF4, at .23. Eighteen of
twenty judges placed EF4 in Effort Regulation, and they
were right about its wording. Keeping to a self-set study schedule reads
as effort. In the responses, though, it shares almost nothing with the
other items. This is what content review cannot see.
EF3 is not flagged here (.34), but its distribution
stands out: the “Top share” column shows most answers in a single
category. Almost everyone finishes the assignments that count toward
their grade. The next stage shows what that costs.
The measurement model
An exploratory factor analysis of the ten recoded items, with the two facets the panel defined:
efa <- nomo_efa(walk[handoff$items], factors = 2)
summary(efa)
#> <nomo_efa summary> Exploratory factor analysis
#> Cases: 400 | Items: 10 | Factors: 2 (researcher specified)
#> Correlation: Pearson | Extraction: minres | Rotation: oblimin (oblique)
#> KMO: .84 | Bartlett's test: chi-square(45) = 810.20, p < .001
#>
#> Item structure
#> Item Factor Loading Next factor Loading Communality Flag
#> EF1 F1 0.78 F2 -0.07 .56
#> EF2 F1 0.65 F2 0.01 .44
#> EF3 F1 0.37 F2 0.08 .17 Concern
#> EF4 F1 0.28 F2 0.02 .08 Concern
#> EF6 F1 0.57 F2 0.05 .35 Review
#> TF1 F2 0.69 F1 0.02 .50
#> TF2 F2 0.56 F1 0.09 .37 Review
#> TF3 F2 0.60 F1 -0.01 .35 Review
#> TF4 F1 0.44 F2 0.32 .42 Review
#> TF6 F2 0.64 F1 -0.08 .37 Review
#>
#> Flagged
#> - EF3 (Concern): The primary loading, 0.37 in absolute value, is below the
#> 0.40 teaching reference. The communality, .17, is below the .40 teaching
#> reference.
#> - EF4 (Concern): The primary loading, 0.28 in absolute value, is below the
#> 0.40 teaching reference. The communality, .08, is below the .40 teaching
#> reference.
#> - EF6, TF3 (Review): The communality, .35, is below the .40 teaching
#> reference.
#> - TF2, TF6 (Review): The communality, .37, is below the .40 teaching
#> reference.
#> - TF4 (Review): The secondary loading, 0.32 in absolute value, is at or
#> above the 0.30 cross-loading reference.
#>
#> Factor correlations
#> Factor 1 Factor 2 r
#> F1 F2 .44
#>
#> Largest residual correlations
#> RMSR: 0.021
#> Item 1 Item 2 Residual
#> EF4 TF6 -.063
#> EF2 TF2 -.046
#> EF4 TF4 .040
#> TF3 TF4 .037
#> EF2 TF6 .035
#>
#> Abbreviations
#> KMO -- Kaiser-Meyer-Olkin measure of sampling adequacy.
#> RMSR -- Root mean square of the off-diagonal residual correlations.
#>
#> Numerical references trigger inspection, not automatic deletion or hidden
#> refitting.
#>
#> See nomo_table(x, "pattern") for the full pattern matrix and
#> nomo_table(x, "decision_log") for every recorded decision.Three items stand apart:
-
EF4has almost no common variance, as the screen suggested. -
EF3loads weakly as well. Its answers are piled at the top of the scale, and a variable that barely varies cannot correlate strongly with anything. -
TF4loads on both factors. Working through a task without taking breaks is effort as much as focus. The panel saw one facet, and the data see two.
A confirmatory model with the panel’s structure locates the same strain:
cfa <- nomo_cfa(nomo_model(handoff$scales), data = walk)
cfa
#> <nomo_cfa> Confirmatory factor analysis
#> Cases: 400 | Estimator: ML | Converged: yes
#> Fit: CFI .941 | TLI 0.922 | RMSEA 0.058 | SRMR 0.056
#> Loadings: 10 | Flags: 4 review, 0 concern
#> No parameter was freed and no model was refit automatically.
#>
#> ML = maximum likelihood; CFI = comparative fit index; TLI = Tucker-Lewis
#> index; RMSEA = root mean square error of approximation; SRMR = standardized
#> root mean square residual.
#>
#> See summary(x) for the evidence and each flag and nomo_table(x, "fit") for
#> every fit index.
head(nomo_table(cfa, "modification_indices"), 3)
#> # A tibble: 3 × 8
#> lhs op rhs mi epc sepc.lv sepc.all sepc.nox
#> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 EF =~ TF4 53.6 0.786 0.640 0.529 0.529
#> 2 EF1 ~~ TF4 11.4 0.164 0.164 0.214 0.214
#> 3 EF =~ TF6 8.35 -0.321 -0.262 -0.210 -0.210The largest modification index asks for TF4 to load on
Effort Regulation too. It is the same item the exploratory analysis
flagged, found from the other direction. As always in this package, the
index locates strain; it does not free the parameter.
Two cohorts
The respondents came from two cohorts. Before comparing their means, check that the items measure the same thing in both:
inv <- nomo_invariance(
nomo_model(handoff$scales),
data = walk,
group = "cohort",
levels = c("configural", "metric", "scalar")
)
summary(inv)
#> <nomo_invariance summary> Measurement invariance across groups
#> Cases: 400 (A n = 200, B n = 200) | Estimator: ML
#> Grouping variable: cohort | Indicators: continuous
#> Levels completed: configural -> metric -> scalar
#>
#> Identification and sequence
#> Continuous indicators use the conventional configural, metric, scalar, and
#> strict sequence.
#>
#> Fit by level
#> Level Chi-square df p CFI RMSEA SRMR
#> configural 114.09 68 < .001 .941 0.058 0.057
#> metric 121.68 76 < .001 .941 0.055 0.064
#> scalar 153.60 84 < .001 .911 0.064 0.071
#> Held equal: loadings from metric; intercepts from scalar.
#>
#> Changes from the preceding level
#> Level CFI change RMSEA change SRMR change Delta chi-square df p
#> metric +.001 -0.003 +0.007 7.58 8 .475
#> scalar -.031 +0.010 +0.008 31.92 8 < .001
#>
#> Latent means relative to A, in its latent standard deviations
#> Level Group Factor Difference 95% CI p
#> scalar B EF -0.14 [-0.36, 0.08] .218
#> scalar B TF 0.22 [0.00, 0.44] .054
#> Comparable only with invariant intercepts, full or partial.
#>
#> Largest score diagnostics for equality constraints
#> Level Constraint Score chi-square df p
#> scalar Intercept: TF6 (A vs. B) 24.30 1 < .001
#> scalar Intercept: TF4 (A vs. B) 6.40 1 .011
#> metric Loading: TF -> TF6 (A vs. B) 2.71 1 .100
#> scalar Intercept: EF1 (A vs. B) 2.55 1 .110
#> metric Loading: EF -> EF3 (A vs. B) 2.32 1 .127
#> scalar Intercept: TF2 (A vs. B) 2.24 1 .135
#> scalar Loading: EF -> EF3 (A vs. B) 2.23 1 .136
#> metric Loading: TF -> TF1 (A vs. B) 2.07 1 .150
#> scalar Loading: TF -> TF1 (A vs. B) 1.69 1 .194
#> scalar Intercept: EF2 (A vs. B) 1.49 1 .222
#>
#> Flagged
#> - Score diagnostics (Review): 30 univariate equality-constraint score
#> diagnostics were retained. Use these diagnostics to localize strain, not
#> to authorize automatic constraint release. Partial invariance requires an
#> explicit researcher specification and rationale.
#>
#> What these columns mean
#> CFI -- Comparative fit index.
#> RMSEA -- Root mean square error of approximation.
#> SRMR -- Standardized root mean square residual.
#> df -- Degrees of freedom.
#> CI -- Confidence interval.
#> ML -- Maximum likelihood.
#>
#> No single CFI change, RMSEA change, SRMR change, chi-square difference, or
#> score diagnostic is treated as a universal invariance rule.
#>
#> See nomo_table(x, "local_strain") for all 30 score diagnostics and
#> nomo_table(x, "decision_log") for every recorded decision.Equal loadings hold; equal intercepts cost fit. The largest score
diagnostic is TF6’s intercept. Putting a phone away while
studying is a specific behavior, and the two cohorts faced different
classroom rules about phones. An item can be content-valid and still not
comparable across groups. Whether to release that intercept (see
nomo_partial()) is a researcher decision, and it needs a
rationale such as that one.
Where the two stages disagree
| Item | Content review | Empirical evidence | What it teaches |
|---|---|---|---|
EF4 |
Carried: 18 of 20 judges | Almost no common variance | Evidence against the item that no panel could see. Revise it, or drop it with a rationale. |
EF3 |
Carried: 18 of 20 judges | Weak loading, from answers piled at the ceiling | The screen is not the final word. EF3 is the only item
about finishing required work; dropping it narrows the domain the panel
defined. |
TF4 |
Carried: 16 of 20 judges | Loads on both facets | A question about content. Revise the wording, or model the cross-loading with a rationale. |
TF6 |
Carried: 18 of 20 judges | Intercept differs by cohort | Explain the difference before comparing cohort means. |
EF2, TF2
|
Carried; reverse-worded | Negative until recoded | Coding, not evidence. |
EF5, TF5
|
Held back | Not screened | They remain in nomo_demo_walkthrough, so you can see
what keeping them would have done. |
Neither stage overrules the other. EF4 is the case for
not trusting a panel alone, and EF3 is the case for not
trusting a screen alone.
What each stage establishes
Content review establishes whether items represent the construct as it was defined: whether the domain is covered, and whether each item belongs where it was written to belong. It cannot see how an item varies, what it shares with the others, or how it behaves in different groups.
An empirical screen establishes how the responses behave. It cannot see whether the domain is covered. A screen would happily keep a scale of ten well-behaved items that all measured one corner of the construct.
nomologR keeps both in the record. A report from a run
that started from a handoff opens with the content review, quoting the
panel’s decisions in contentvalidR’s words, and each
empirical flag carries its own explanation. Neither is presented as a
verdict on the other:
nomo_report(run, file = file.path(tempdir(), "walkthrough-report.html"))