Where the data comes from
The plugin exports its data as a small family of CSV files. The first
column of every file says what kind of export it is and which version of
the exchange format it follows. qdaR checks both against
the contract file and stops with an error when it meets a version it has
never seen – guessing would be worse than failing here, since a misread
column ends up in somebody’s findings without anyone noticing.
qda_formats()[, c("format", "file", "grain")]
#> format file
#> 1 fragments zotqda-fragments.csv
#> 2 uncoded zotqda-uncoded.csv
#> 3 codebook zotqda-codebook.csv
#> 4 history zotqda-history.csv
#> 5 abbrevs zotqda-abbrevs.csv
#> 6 consensus-mapping zotqda-konsens-abbildung.csv
#> 7 consensus-protocol zotqda-konsens-codesystem.csv
#> 8 consensus-metrics zotqda-konsens-kennzahlen.csv
#> 9 multi-coded zotqda-mehrfachkodierungen.csv
#> grain
#> 1 one row per annotation x code
#> 2 one row per annotation without any code (column code is empty)
#> 3 one row per code
#> 4 one row per coding-log event (add and remove), oldest first
#> 5 one row per code abbreviation
#> 6 one row per coder code; the bridge from phase-2 codings to the consensus system
#> 7 one row per consensus decision; the first row is a settings header
#> 8 one row per coder pair, plus mean and round comparison
#> 9 one row per segment carrying more than one genuinely different code
This vignette runs entirely on reference files installed with the
package, so you can follow along without a Zotero installation:
frag <- qda_read_fragments(qda_example("zotqda-fragments.csv"))
attr(frag, "qda_format")
#> [1] "fragments"
attr(frag, "qda_grain")
#> [1] "one row per annotation x code"
Codes carry an identity, not just a name
Codes get renamed, moved and merged while a project matures. Every
export therefore names each code twice: code holds the path
a person reads, and codeId holds a stable identifier that
stays put through all of that housekeeping. The distinction matters for
any analysis that runs more than once. If it groups by the path, a code
vanishes from the results as soon as somebody renames it in Zotero;
grouped by codeId, it simply follows along.
frag[, c("code", "codeId")]
#> code codeId
#> 1 Belastung cX1y2z3a4
#> 2 Belastung/beruflich, akut cQ9w8e7r6
What the plugins deliberately leave out
qdaZ sticks to description and never runs a significance test. That
is a considered position, not a gap: an inferential statistic invites
claims that many qualitative designs cannot carry. If your design does
support one, this is where you run it – and you pick the test
yourself.
res <- suppressWarnings(qda_chisq(frag, group = "citekey"))
res$table
#> group
#> code Beispiel
#> Belastung 1
#> Belastung/beruflich, akut 0
#> group
#> code Er sagte: "Ja, wirklich"\nund ging; danach nichts.
#> Belastung 0
#> Belastung/beruflich, akut 1
res$cramers_v
#> [1] 0
res$expected_ok
#> [1] FALSE
Note expected_ok. With the tiny reference table the
chi-squared approximation does not hold, so the function reports
Fisher’s exact test and says so rather than printing a p-value that
looks respectable and is not.
Codes can also be arranged by the segments they share:
qda_code_distance(frag, min_n = 1)
#> Belastung
#> Belastung/beruflich, akut 1
cl <- qda_cluster(frag, min_n = 1)
cl$cophenetic
#> [1] NA
The cophenetic correlation is NA here because two codes
give a single distance, which has no variance to correlate. A dendrogram
always looks convincing; this number tells you whether it deserves
to.
Consensus data
A code-system consensus run in zotQDA ends with a mapping: this
coder’s code corresponds to that consensus code. The mapping travels as
ordinary data, and qda_apply_mapping() does nothing more
than add a consensusCode column next to the original
coding. Nothing gets rewritten. Had the codings themselves been
rewritten to the consensus system, any agreement you later computed on
it would come out inflated – the disagreements would have been edited
away first.
map <- qda_read_mapping(qda_example("zotqda-konsens-abbildung.csv"))
head(qda_apply_mapping(frag, map)[, c("code", "consensusCode")])
#> code consensusCode
#> 1 Belastung Belastung
#> 2 Belastung/beruflich, akut Belastung/beruflich, akut
How well did the coders agree?
Reliability is the one place where an independent reimplementation
earns its keep. qdaR recomputes the coefficients the plugin reports, in
a different language, from the exported file alone – so a figure that
appears in a methods section has been produced twice, by two code bases
that share nothing but the contract. The package’s test suite checks
this against frozen plugin results on randomly generated coder
matrices.
The fragments export is long; the measures need a unit-by-coder
matrix.
frag2 <- data.frame(
annotationKey = rep(paste0("s", 1:6), each = 2),
codedBy = rep(c("ann", "bob"), 6),
code = c("A", "A", "A", "A", "A", "A",
"A", "A", "A", "A", "B", "A")
)
u <- qda_units(frag2)
qda_agreement(u)
#> units coders categories multi_set_aside percent cohen brennan fleiss
#> 1 6 2 2 0 0.8333333 0 0.6666667 -0.09090909
#> alpha ac1
#> 1 0 0.8032787
Note what happened there: the coders disagreed once in six segments,
and Cohen’s kappa still came out at zero. That is not a defect of the
coding but the marginals – almost everything is A, so
chance alone would produce this much agreement. Gwet’s AC1 is the
coefficient that stays interpretable in that situation, which is why
both are reported side by side. A single coefficient never settles the
question.
Building the matrix forces two decisions. Both are easy to get
silently wrong in a hand-rolled script, so qda_units()
makes you take them consciously:
- A segment that no coder coded still says something: both coders
judged it irrelevant, and that is agreement too. Such segments only
enter the matrix when you pass the
uncoded export. Without
it, your figures describe just the material somebody marked – a
different and usually friendlier question.
- When one coder gave a segment several codes, there is no single
value to compare. The segment is set aside and counted in
multi_set_aside. Quote that count next to the coefficient;
a kappa that quietly dropped a tenth of the material is not the kappa of
the study. The honest way to include such material is the per-code
binary view, qda_units_binary().
Where in the code system the agreement is lost
A hierarchical code system can be read at several resolutions. Coders
who split over Belastung/beruflich against
Belastung/privat still agree that the segment is about
Belastung. Flattening the paths level by level shows where
the agreement breaks down – a statement about the code system rather
than about the coders.
u2 <- cbind(ann = c("A/x", "A/y", "B/x", "B/y"),
bob = c("A/y", "A/y", "B/x", "B/x"))
qda_level_agreement(u2)
#> level units coders categories multi_set_aside percent cohen brennan
#> 1 1 4 2 2 0 1.0 1.0000000 1.0000000
#> 2 2 4 2 4 0 0.5 0.3333333 0.3333333
#> fleiss alpha ac1
#> 1 1.0000000 1.0000000 1.0000000
#> 2 0.2727273 0.3636364 0.3513514
qda_plot_level_agreement(u2)
#> Warning: Use of `long$level` is discouraged.
#> ℹ Use `level` instead.
#> Warning: Use of `long$value` is discouraged.
#> ℹ Use `value` instead.
#> Warning: Use of `long$measure` is discouraged.
#> ℹ Use `measure` instead.
#> Warning: Use of `long$level` is discouraged.
#> ℹ Use `level` instead.
#> Warning: Use of `long$value` is discouraged.
#> ℹ Use `value` instead.
#> Warning: Use of `long$measure` is discouraged.
#> ℹ Use `measure` instead.
And when the number is disappointing, the confusion table says which
pairs of categories cost it – usually a handful, and usually the ones
whose definitions need work.
qda_confusion(u2, only_disagreements = TRUE)
#> ann bob n
#> 1 A/x A/y 1
#> 2 B/y B/x 1
