Method variance from a marker variable: the comprehensive CFA marker technique
Source:R/nomo_method_variance.R
nomo_method_variance.Rdnomo_method_variance() tests whether a marker variable carries method
variance into a measurement model's indicators, whether that variance biases
the correlations among the substantive factors, and how much of each factor's
reliability it accounts for. It follows the comprehensive CFA marker
technique of Williams, Hartman, and Cavazotte (2010).
Usage
nomo_method_variance(
model,
data,
marker,
alpha = 0.05,
estimator = NULL,
missing = NULL,
marker_name = "Marker"
)Arguments
- model
A lavaan measurement model for the substantive factors, as a string or a
nomo_model()object. Indicators load on one factor each.- data
A data frame holding the substantive and marker indicators.
- marker
The marker variable's indicators: two or more numeric column names that
modeldoes not use.- alpha
Significance level for the model comparisons. Default
.05.- estimator, missing
Optional lavaan
estimatorandmissingoptions. With a robust estimator, the comparisons use lavaan's scaled difference tests.- marker_name
Name given to the marker factor. Default
"Marker". It names a factor in lavaan syntax, so it must be a syntactic name, such as"SocialDesirability", without spaces.
Value
A nomo_method_variance object. The fields to read are:
models: each model's chi-square, degrees of freedom, p value (pvalue), CFI, TLI, RMSEA, and SRMR. With a robust estimator, the chi-square is scaled and the indices are robust, as innomo_cfa().comparisons: the three model comparisons, with their chi-square differences (chisq_diff), degrees of freedom (df_diff), andp_value.retained:"Method-C"or"Method-U".method_loadings: each substantive indicator's standardized substantive and method loadings in the retained model, the share of its variance the marker accounts for, and the method loading'sp_value.reliability: each substantive factor's Baseline reliability, its substantive and method parts, andmethod_share.correlations: each pair of substantive factors (factor1,factor2) and their correlation in the CFA, Baseline, retained, Method-S(.05), and Method-S(.01) models, with p values in the retained and sensitivity models (retained_p_value,method_s_05_p_value,method_s_01_p_value).marker_correlations: the marker's correlation with each substantive factor in the CFA model;factor1is the substantive factor andfactor2the marker.engine_warnings: the warnings lavaan raised, by model and for the comparisons.fitsanddecision_log.
A table with one p value for a test or an estimate names it p_value, as
the rest of the package does. correlations has one for each model, so
each is named <model>_p_value. In models, pvalue is the p value of
each model's chi-square test, named as in the fit tables of nomo_esem()
and nomo_invariance().
Other fields record the call, the settings used, and the syntax fitted.
They may change between releases and are not part of the stable interface
(see ?nomologR).
print() shows the model comparisons, the reliability decomposition, the
substantive correlations across the models, and any flag. summary() adds
each model's fit, the loadings in the retained model, and each flag's
recommendation.
Details
Experimental. This function is experimental and remains so after
nomologR 1.0.0. Its output may be reorganized during 1.x, without a
deprecation period, as the marker technique is extended beyond continuous
indicators. Any change will be described in NEWS; see the package help page,
?nomologR, for the stability policy.
The marker. A marker variable is theoretically unrelated to the substantive variables, so what it shares with them is taken to be method variance (Lindell & Whitney, 2001). Williams et al. (2010) expand the definition: a marker should also tap one or more of the biases present in the measurement context, such as social desirability, transient mood, or common scale anchors (Podsakoff et al., 2003). A marker chosen only because it is unrelated, such as a demographic, may capture no method variance at all. The marker is measured by two or more indicators, three or more by the authors' recommendation.
Phase I: model comparisons. Five models are fitted with
lavaan::cfa(std.lv = TRUE):
CFA: the substantive factors and the marker factor, all correlated, with no method loadings. It supplies the marker indicators' loadings and error variances.
Baseline: those loadings and error variances fixed at their CFA values, so the marker factor keeps its meaning, and the marker made orthogonal to the substantive factors.
Method-C: Baseline plus a loading from the marker on every substantive indicator, constrained equal.
Method-U: the same loadings, free to differ.
Method-R: the retained method model with the substantive factor correlations fixed at their Baseline values.
Baseline versus Method-C tests whether marker-based method variance is
present. Method-C versus Method-U tests whether its effects are equal, and
decides which method model is retained. The retained model versus Method-R
tests whether the method variance biases the substantive correlations. Each
comparison is a likelihood-ratio test at alpha.
Phase II: reliability decomposition. From the completely standardized
estimates, each substantive factor's reliability in the Baseline model is
decomposed, using the retained method model, into a substantive part and a
method part (Williams et al., 2010, equations 1-3). method_share is the
method part as a proportion of the Baseline reliability.
Phase III: sensitivity. Because the method loadings are estimates, the retained model is refitted with them fixed at the ends of their 95% and 99% confidence intervals farther from zero (Method-S(.05) and Method-S(.01)), so a negative loading, as from a marker keyed opposite to the substantive items, becomes more negative. The substantive correlations are then compared across the models.
Checks. lavaan's warnings are kept for each model and comparison, in
engine_warnings and as review rows of the decision log, and a model with
an improper solution, such as a negative variance, is a concern. The marker's
loadings and error variances from the CFA anchor every later model, so a CFA
that does not converge, or gives the marker a negative error variance or no
standard errors, stops the analysis, as does any model that does not
converge. With two marker indicators, the marker's loadings are identified
only through its correlations with the substantive factors, which the
technique assumes are zero, so the decision log asks for review.
What the technique cannot do. It requires the marker to be orthogonal to the substantive factors. In simulations, with an ideal marker it did not find method variance that was absent. With a nonideal marker it sometimes did, and with either kind it did not recover the substantive correlations accurately (Richardson, Simmering, & Sturman, 2009). The results are evidence about the method variance the chosen marker captures, not corrected estimates, and other sources of method variance may remain (Podsakoff, MacKenzie, & Podsakoff, 2012).
References
Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86(1), 114-121. doi:10.1037/0021-9010.86.1.114
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. doi:10.1037/0021-9010.88.5.879
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539-569. doi:10.1146/annurev-psych-120710-100452
Richardson, H. A., Simmering, M. J., & Sturman, M. C. (2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12(4), 762-800. doi:10.1177/1094428109332834
Williams, L. J., Hartman, N., & Cavazotte, F. (2010). Method variance and marker variables: A review and comprehensive CFA marker technique. Organizational Research Methods, 13(3), 477-514. doi:10.1177/1094428110366036
Examples
# Social desirability as the marker for the Agency and Persistence items of
# the simulated validation study. In its population the marker shares
# nothing with those items, so no method variance should be detected.
model <- nomo_model(list(
Agency = paste0("ag", 1:4),
Persistence = paste0("pe", 1:4)
))
mv <- nomo_method_variance(model, data = nomo_demo_network,
marker = c("sd1", "sd2", "sd3"))
mv
#> <nomo_method_variance> Marker-based method variance
#> Williams, Hartman, and Cavazotte (2010).
#> Marker: sd1, sd2, sd3 | Cases: 800 | Estimator: ML
#> Retained: Method-C | Comparisons at alpha = .05
#>
#> Model comparisons
#> Comparison Question Delta chi-square df p
#> Baseline vs. Method-C Method variance present? 0.00 1 .954
#> Method-C vs. Method-U Method effects equal? 2.37 7 .937
#> Method-C vs. Method-R Correlations biased? 0.00 1 > .999
#>
#> Reliability decomposition
#> Factor Total Substantive Method Method share
#> Agency .85 .85 .00 0.0%
#> Persistence .82 .82 .00 0.0%
#>
#> Substantive correlations
#> Factors CFA Baseline Method-C Method-S(.05) Method-S(.01)
#> Agency with Persistence .457 .457 .457 .459 .460
#>
#> Models
#> CFA -- Confirmatory factor analysis of the substantive factors and the
#> marker, all correlated, with no method loadings; it gives the marker's
#> loadings and error variances.
#> Baseline -- The marker uncorrelated with the substantive factors, its
#> loadings and error variances fixed at their CFA values.
#> Method-C -- Baseline plus equal marker loadings on every substantive item.
#> Method-U -- Baseline plus marker loadings free to differ.
#> Method-R -- The retained model with the substantive correlations fixed at
#> their Baseline values.
#> Method-S(.05), Method-S(.01) -- The retained method loadings fixed at the
#> ends of their 95% and 99% intervals farther from zero.
#>
#> df = degrees of freedom; ML = maximum likelihood.
#>
#> The results describe the method variance this marker captures; they are not
#> corrected estimates, and other sources of method variance may remain.
#>
#> See summary(x) for each model's fit and the method loadings and
#> nomo_table(x, "decision_log") for every recorded decision.
nomo_table(mv, "reliability")
#> # A tibble: 2 × 5
#> factor reliability_total reliability_substantive reliability_method
#> <chr> <dbl> <dbl> <dbl>
#> 1 Agency 0.849 0.849 0.00000374
#> 2 Persistence 0.816 0.816 0.00000413
#> # ℹ 1 more variable: method_share <dbl>