Design & Reporting Guide

Choosing a method

Use the design that matches the question posed to judges rather than selecting an index after data collection.

Item sort: judges assign each item to the construct definition that best represents it. Use sort_validity() for Psa/Csv, exact target-count screening, strongest-competitor diagnostics, and scale-level Colquitt norms.
Construct rating: the same judges rate each item against every focal and orbiting definition. Use rating_validity() for HTC/HTD, repeated-measures inference, planned target-versus-orbiting contrasts, and scale-level norms.
Expert panel: experts answer a relevance, essentiality, or congruence question. Use expert_validity() with the corresponding explicit mode.

The three designs can complement one another during scale development, but their statistics are not interchangeable.

Sample-size planning

For item sorts, plan judge N in relation to the exact retention rule and a plausible true target-assignment probability. sort_power() provides exact planning probabilities; avoid a universal judge-count rule of thumb.

For construct-rating studies, power depends on the number of judges, number of construct definitions, within-judge target-orbiting separation, and missing profiles. Report the effective complete-judge N itemwise.

For expert panels, panel-size sensitivity is part of the statistic. CVR exact critical counts and common CVI review guidelines therefore need to be interpreted with the actual effective N, not a nominal panel size that ignores missingness.

Recommended reporting sequence

Design: judge population, recruitment, construct definitions, item pool, instructions, and response format.
A priori rules: alpha, relevance threshold, target mapping, multiplicity adjustment if used, and any planned norms.
Item-level evidence: correspondence, distinctiveness, exact/paired inference, strongest competitor, and review status as appropriate.
Scale-level evidence: target-scale averages or S-CVI summaries where the method defines them.
Substantive decisions: revisions, removals, retained domain-coverage items, and how qualitative comments informed those decisions.
Reproducibility: package version, analysis settings, anonymized data when permitted, and the script used to reproduce tables/figures.

The reporting-examples vignette expands this sequence into reusable methods and results scaffolds. Treat those examples as reporting patterns rather than fixed language that must be copied verbatim.

Deterministic bundled examples

sort_dat <- read_example("sort_example.csv")
rating_dat <- read_example("rating_example.csv")
expert_rel <- read_example("expert_relevance_example.csv")

sort_fit <- sort_validity(sort_dat)
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
expert_fit <- expert_validity(
  as.matrix(expert_rel[setdiff(names(expert_rel), "expert")]),
  mode = "relevance", lo = 1, hi = 4
)

These files are synthetic and generated by data-raw/build-example-data.R in the source repository. They deliberately include both supported and review-worthy items so documentation exercises realistic output paths without depending on random-number generation.

Table templates

Item sort

sort_fit$results[c(
  "item", "target", "n", "n_target", "competitor",
  "psa", "csv", "p_value", "status", "recommendation"
)]
#>   item target  n n_target competitor  psa  csv      p_value    status
#> 1   A1      A 20       18       B; C 0.90 0.85 0.0002012253 Supported
#> 2   A2      A 20       15          B 0.75 0.60 0.0206947327 Supported
#> 3   B1      B 20       17          A 0.85 0.75 0.0012884140 Supported
#> 4   B2      B 20       13          A 0.65 0.40 0.1315879822    Review
#> 5   C1      C 20       18       A; B 0.90 0.85 0.0002012253 Supported
#> 6   C2      C 20       14          B 0.70 0.50 0.0576591492    Review
#>   recommendation
#> 1         Retain
#> 2         Retain
#> 3         Retain
#> 4         Review
#> 5         Retain
#> 6         Review

At the target-scale level, report mean Psa/Csv and the benchmark set actually used. Do not convert Colquitt’s scale-level norms into individual-item cutoffs.

Construct rating

rating_fit$results[c(
  "item", "target", "n_complete", "strongest_competitor",
  "htc", "htd", "p_value", "max_contrast_p", "status", "recommendation"
)]
#>   item target n_complete strongest_competitor       htc       htd      p_value
#> 1   A1      A         24                    C 0.9333333 0.6666667 1.274299e-19
#> 2   A2      A         24                    B 0.7333333 0.2083333 7.375102e-08
#> 3   B1      B         24                    C 0.9333333 0.6666667 1.274299e-19
#> 4   B2      B         24                    A 0.7333333 0.2083333 7.375102e-08
#> 5   C1      C         24                    B 0.9333333 0.6666667 1.274299e-19
#> 6   C2      C         24                    B 0.7333333 0.2083333 7.375102e-08
#>   max_contrast_p    status recommendation
#> 1   1.122784e-13 Supported         Retain
#> 2   5.173102e-02    Review         Review
#> 3   1.122784e-13 Supported         Retain
#> 4   5.173102e-02    Review         Review
#> 5   1.122784e-13 Supported         Retain
#> 6   5.173102e-02    Review         Review
rating_fit$scale_summary
#>   target n_items n_htc n_htd n_retain n_review n_insufficient  mean_htc
#> 1      A       2     2     2        1        1              0 0.8333333
#> 2      B       2     2     2        1        1              0 0.8333333
#> 3      C       2     2     2        1        1              0 0.8333333
#>   htc_strength mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1         Weak   0.4375  Very Strong             Weak         NA       overall
#> 2         Weak   0.4375  Very Strong             Weak         NA       overall
#> 3         Weak   0.4375  Very Strong             Weak         NA       overall
#>                                                                                                                                                            evidence
#> 1 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.
#> 2 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.
#> 3 Weak normative standing on at least one dimension; review item wording, construct boundaries, and orbiting-construct choice and consider re-pretesting revisions.

Report the repeated-measures design and target-versus-orbiting contrasts. For a review item, naming the strongest competitor is often more informative than a standalone p value.

Expert relevance

expert_fit$results[c(
  "item", "N", "V", "ci_low", "ci_high", "I_CVI",
  "kappa_mod", "status", "recommendation"
)]
#>    item N         V    ci_low   ci_high I_CVI kappa_mod    status
#> 1 Item1 8 1.0000000 0.8620238 1.0000000  1.00 1.0000000 Supported
#> 2 Item2 8 0.9166667 0.7415120 0.9768412  1.00 1.0000000 Supported
#> 3 Item3 8 0.8333333 0.6414693 0.9332132  1.00 1.0000000 Supported
#> 4 Item4 8 0.5833333 0.3883467 0.7553240  0.75 0.7192982    Review
#> 5 Item5 8 0.4166667 0.2446760 0.6116533  0.25 0.1578947    Review
#>   recommendation
#> 1 Strong support
#> 2 Strong support
#> 3 Strong support
#> 4         Review
#> 5         Review
expert_fit$scale_summary
#>   n_items n_experts_min n_experts_max mean_Aiken_V S_CVI_Ave S_CVI_UA agreement
#> 1       5             8             8         0.75       0.8      0.6 0.6934253
#>   agreement_low agreement_high n_strong_support n_support n_review
#> 1    0.07142857      0.8457143                3         0        2
#>   n_insufficient
#> 1              0

Essentiality and congruence require different expert tasks. Do not place CVR, CVI, Aiken V, and IOC in one generic threshold table as if they answer the same question.

Diagnostics and later validation

signal_detection() and reproducibility_phi() remain auxiliary helpers. They can be useful when researchers later compare pretest decisions with CFA/IRT retention or an independent replication pretest, but they are not required parts of the flagship workflows.

pretest_supported <- sort_fit$results$status == "Supported"
later_retained <- c(TRUE, TRUE, TRUE, FALSE, TRUE, FALSE)
signal_detection(pretest_supported, later_retained)
#> $confusion
#>               Actual
#> Predicted      Retain Not retained
#>   Retain            4            0
#>   Not retained      0            2
#> 
#> $accuracy
#> [1] 1
#> 
#> $sensitivity
#> [1] 1
#> 
#> $specificity
#> [1] 1
#> 
#> $phi
#> [1] 1
#> 
#> $chisq
#> [1] 6
#> 
#> $p
#> [1] 0.01430588

replication_supported <- c(TRUE, TRUE, TRUE, FALSE, TRUE, TRUE)
reproducibility_phi(pretest_supported, replication_supported)
#> $table
#>               Pretest2
#> Pretest1       Retain Not retained
#>   Retain            4            0
#>   Not retained      1            1
#> 
#> $phi
#> [1] 0.6324555
#> 
#> $chisq
#> [1] 2.4
#> 
#> $p
#> [1] 0.1213353

Power quick check

sort_power(N = c(20, 30, 40), true_p = c(.60, .70, .80))
#> Exact item-sort planning analysis
#> ---------------------------------
#> Retention rule: p0 = 0.50, alpha = 0.050
#> 
#>   N true_p critical_n_target minimum_observed_psa power
#>  20    0.6                15                0.750 0.126
#>  30    0.6                20                0.667 0.291
#>  40    0.6                26                0.650 0.317
#>  20    0.7                15                0.750 0.416
#>  30    0.7                20                0.667 0.730
#>  40    0.7                26                0.650 0.807
#>  20    0.8                15                0.750 0.804
#>  30    0.8                20                0.667 0.974
#>  40    0.8                26                0.650 0.992
#> 
#> Power is the exact probability of reaching the required target-assignment count
#> under the assumed true target-assignment probability.

Good practices

Pre-register quantitative screening criteria when feasible, preserve qualitative judge feedback, report effective N after missingness, and archive the construct definitions and item wording used in the pretest. A content-validation statistic is evidence from a designed judgment task; it is not a substitute for defining and sampling the construct domain.