Changelog • craft

craft 0.1.0

First public release of craft — an R package implementing the five-step C-R-A-F-T framework (Construct, Report, Assess, Field-audit, Translate) for evaluating LLM-generated data in political and social science research.

12 exported functions

C — role()
R — reliab(), reliab_pairs(), valid(), dual()
A — stab()
F — audit(), disagree(), tau_sens()
T — dsl_fit(), dsl_cmp()
Report — report() emits a reproducibility supplement (HTML/PDF) capturing role-task, LLM version, prompt, metrics, thresholds, audit summary, and DSL coefficients.

Reliability + validity

reliab() dispatches to six methods: Cohen’s $\kappa$ , weighted $\kappa$ , Fleiss’ $\kappa$ , Krippendorff’s $\alpha$ , ICC, and percent agreement. valid() reports precision, recall, F1 (macro and weighted), accuracy, balanced accuracy, and MCC. dual() bundles both in a single call and accepts a named list of predictions for multi-annotator validity tables.

Stability + audit

stab() compares any number of prompt/model variants on the same gold set. audit(), disagree(), and tau_sens() surface low-confidence agreements, high-confidence disagreements, and sensitivity to the confidence threshold.

Translate to inference

dsl_fit() and dsl_cmp() wrap design-based supervised learning (Egami et al. 2023) so misclassification uncertainty propagates into downstream regression coefficients.

Reproducibility

report() writes a single Rmd that renders to either HTML or PDF (requires tinytex for PDF). Output includes a publication-quality DSL coefficient table grouped by hypothesis.

Data shipped

inst/extdata/craft_demo.csv — 50-row synthetic stance-labeling example with per-LLM confidences and rationales.
inst/extdata/data.rda — 200-row synthetic legislator-level dataset for the dsl_cmp() example.
inst/extdata/prompt_dev4.txt — example codebook prompt.

Installation

remotes::install_github("casstai/craft-r")