--- title: "Rasch analysis of comparative judgements" author: "Josh McGrane" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Rasch analysis of comparative judgements} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 4.2) options(digits = 4) ``` ```{r library} library(rasch) ``` ## Fit the Bradley--Terry--Luce model Paired-comparison data record two objects and an observed preference. In the Bradley--Terry--Luce model (Bradley and Terry 1952; Luce 1959), the log odds of choosing object A over object B are their location difference: $$ P(A\succ B)=\frac{\exp(\beta_A)} {\exp(\beta_A)+\exp(\beta_B)}. $$ This is the conditional form of the dichotomous Rasch model (Rasch 1960; Andrich 1978). The comparison graph must connect all objects; otherwise their relative locations are not identified. For ordered comparisons, `btl` fits the adjacent-category extension $$ \log\frac{P(Y=r)}{P(Y=r-1)}=\beta_A-\beta_B-\tau_r, $$ with thresholds symmetric under reversal of presentation order (Tutz 1986). ```{r fit} d <- simulate_btl(n_objects = 7, n_judges = 12, reps_per_pair = 20, seed = 5) fit <- btl(d, object_a = "object_a", object_b = "object_b", winner = "winner", judge = "judge") fit fit$objects ``` Judge residuals describe agreement with the common object scale; they are not person measures. Object fit, judge fit, targeting, and comparison information address different parts of the design and should be considered together. ```{r judges} fit$judges judge_surprise(fit, "J1") btl_information(fit) ``` ## Check transitivity and residual structure Circular triads (Kendall and Babington Smith 1940) identify local contradictions in the observed ordering. Residual dimensionality asks whether comparisons contain a structured second attribute after the primary scale is fitted. ```{r structure} tr <- btl_transitivity(fit) tr dimensions <- btl_dimensionality(fit, reps = 20) dimensions ``` ```{r plot-transitivity, fig.alt = "Per-judge consistency of the paired comparisons."} plot_btl_transitivity(tr) ``` ```{r plot-scree, fig.alt = "Residual bimension strengths against the simulated noise reference."} plot_btl_scree(dimensions) ``` The simulation reference for dimensionality uses twenty replicates here to keep the vignette quick. A final analysis should use enough replicates to stabilise the reference distribution. ## Examine DIF across judge groups `btl_dif` tests whether object locations differ across nominated judge factors. The omnibus analysis uses judges as the independent units. A significant term is resolved into factor-specific object locations and pairwise logit differences. HC3 covariance allows the precision of judge means to vary with their comparison workloads. Omnibus and pairwise inference require at least eight judges and eight effective judges in each factor level; estimates remain descriptive below that boundary. The tables report both counts. ```{r btl-dif-example, eval = FALSE} judge_group <- setNames(panel_data$discipline, panel_data$judge) bd <- btl_dif(fit, judge_group) bd$summary bd$sizes ``` ## Equate panels through common objects `btl_equate` aligns two calibrations that share at least three objects. For two fitted calibrations, drift inference is withheld until independent judges and comparisons are stated explicitly. ```{r equating-example, eval = FALSE} eq <- btl_equate(current_panel, reference_panel, independent = TRUE) eq$table eq$equated # reference panel on the current panel's origin ``` A bank table may be used in place of `reference_panel`. Marginal object standard errors are not enough for drift tests because they omit the covariance created by the bank's fitted origin. Attach the joint matrix as `attr(bank, "cov_location")`, ordered like the bank rows or named by object; otherwise the alignment is descriptive. A bank treated as fixed may instead carry zero standard errors. Dependent panels require a joint or paired bootstrap outside this function. ## Linked frames for paired comparisons `btl_efrm` combines the comparative judgement model with Humphry's extended frame of reference structure (Humphry and Andrich 2008). Judges belong to panels, and objects belong to linked sets. For object $k$ in set $s$, $$ v_k=\alpha_s\beta_k+\kappa_s. $$ A same-set comparison in panel $g$ has logit $\phi_g(\beta_A-\beta_B)$; a cross-set comparison has logit $\phi_g(v_A-v_B)$. Cross-set comparisons identify the set units and origins. The cross-set likelihood holds the within-set locations and panel units fixed. It estimates the set transformations directly from the comparison outcomes and does not use the finite-grid person-distribution link in `rasch_efrm()`. ```{r efrm} de <- simulate_btl_efrm( n_objects_per_set = 5, n_sets = 2, n_judges_per_panel = 6, n_panels = 2, reps_within = 15, reps_cross = 15, set_units = c(1, 1.3), set_origins = c(0, 0.6), seed = 9 ) ef <- btl_efrm( de, "object_a", "object_b", winner = "winner", judge = "judge", panels = "panel", object_sets = attr(de, "truth")$object_sets, se_method = "conditional" ) ef$phi_table ef$alpha_table ef$kappa_table ``` The default judge bootstrap resamples judges within panels and refits both stages. The parametric bootstrap (`se_method = "bootstrap"`) draws independent outcomes from the fitted model. The conditional option used above reports estimates and conditional standard errors but withholds probabilities because it does not propagate stage-one uncertainty into the set link. With either bootstrap, omnibus tests cover the unit families and individual contrasts are Holm-adjusted follow-ups. Judge-bootstrap tests require at least six judges and 5.5 effective judges in every panel, and eight of each on a set link. Judge resamples are distributed over four workers by default, or fewer where the system imposes a lower limit. Set `seed` to reproduce the resamples. The parametric bootstrap remains serial because its refits are inexpensive. In the application, frame estimation runs in the background and may be cancelled. With 12 judges and 20 repetitions per pair, null rejection for the three unit families was 3.3--5.3 per cent under the judge bootstrap and 3.0--6.7 per cent under the independent-outcome bootstrap. The staged set-unit estimate has small finite-sample attenuation when the within-set locations are imprecise: log-unit bias was -0.041 at 20 repetitions per pair, -0.016 at 50 and -0.007 at 100. The bootstrap intervals retained nominal coverage at the 20-repetition design. ## A worked analysis on real data The [party blocs case study](https://drjoshmcgrane.github.io/rasch/articles/articles/party-blocs-crisis.html) fits the comparative judgement frame model to real paired comparisons between political parties, with judge panels and ideological blocs as frames. Its script ships with the package under `casestudies`. ## References Andrich, D. (1978). Relationships between the Thurstone and Rasch approaches to item scaling. *Applied Psychological Measurement*, 2, 451--462. Bradley, R. A., and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. *Biometrika*, 39, 324--345. Humphry, S. M., and Andrich, D. (2008). Understanding the unit in the Rasch model. *Journal of Applied Measurement*, 9(3), 249--264. Kendall, M. G., and Babington Smith, B. (1940). On the method of paired comparisons. *Biometrika*, 31(3/4), 324--345. Tutz, G. (1986). Bradley-Terry-Luce models with an ordered response. *Journal of Mathematical Psychology*, 30(3), 306--316. Luce, R. D. (1959). *Individual Choice Behavior: A Theoretical Analysis*. Wiley. Rasch, G. (1960). *Probabilistic Models for Some Intelligence and Attainment Tests*. Copenhagen: Danish Institute for Educational Research. (Expanded edition, 1980, Chicago: University of Chicago Press.)