What we compared against, and where it differed

Summary

Record: docs/parity/spss_diff.md · docs/parity/amos_diff.md

Text & decision — what we compared against

SPSS and Amos do not produce these numbers, so instead of “do the values agree?” we record what each engine was compared against — an external reference, a same-library reproduction, an accuracy measured on a labelled public sample, or nothing comparable yet.

Record: backend/tests/parity/family_reference.json

Values not counted as agreeing — and why

These values were not counted as agreeing. "Convention differs" means the values are supposed to differ (the tools compute them differently); "not comparable" means the reference tool does not report that value, or we could not extract it.

By reference tool

Engine grades

Grades are derived automatically from golden tests (fixed inputs with frozen expected outputs) and invariance checks (the same data changed in ways that should not matter — does the answer move?). No one assigns them by hand.

Corrections — engines we fixed, and results to run again

These engines once produced wrong numbers in the places described, and were fixed. If you kept a result from one of them that you ran before the date shown, please run it again. The same note is in the change log.

• Comparison Analysis2026-10-20 · v1.6.45Comparison analysis read every group’s indicator means as 0 when the group column held number codes (1, 2, 3) — the table, best/worst and overall rank had nothing to do with the data. Groups are now matched as text and the real means are counted (blank group cells are left out, text cells in an indicator are missing). When one group has no countable value for an indicator, it stops and says why.If you ran comparison analysis over MCP (the Claude or ChatGPT connection) with a number-coded group column before this date, please run it again.
• IV Regression · Centrality · Difference-in-Differences2026-10-19 · v1.6.44The standard errors of IV regression (2SLS) were computed from residuals built with the first-stage fitted values, so they came out too small — p-values too small and confidence intervals too narrow. They now use the structural residuals built with the original endogenous variable (the same values as statsmodels IV2SLS), and the confidence interval uses the t distribution. A footnote marks a weak instrument when the first-stage F is below 10. Closeness in centrality ignored the strength of the links — distance is now 1/|r| (the same distance as betweenness). Difference-in-differences gave numbers such as 0 on data where the estimate does not exist (an empty treated/control × before/after cell, a constant outcome and so on) — it now stops and says why. It also read time and group levels alphabetically, so with pre/post or before/after the before and after periods swapped and the sign of the effect flipped — they are now read by meaning (number-like text in number order).If you ran IV regression, centrality (closeness) or difference-in-differences (above all with text time levels such as pre/post) before this date, please run it again. An IV result that was significant may no longer be.
• IV Regression · Centrality · Hierarchical Regression2026-10-19 · v1.6.43The "first-stage F" of IV regression was the F of the whole first-stage regression, so it included what the covariates explain — a weak instrument could look strong and "Weak instrument?" could read "No". It is now the partial F of the instrument alone, controlling for the covariates (the same value when there are no covariates). Betweenness in centrality used the correlation r itself as path length, so a strong link became a long path and a negative correlation a negative length — distance is now 1/|r|. Hierarchical regression counted number-coded variables whose measurement level is "nominal" by the size of their codes — they now enter their block as dummy columns (the smallest code is the reference).If you ran IV regression with covariates, centrality with three or more variables, or hierarchical regression with a number-coded nominal variable before this date, please run it again.
• Multiple Regression · Logistic Regression2026-10-18 · v1.6.42A nominal variable coded as numbers (sex or region as 1/2/3) used as a predictor was counted by the size of its codes (3 as three times 1). A variable whose measurement level is "nominal" in variable settings is now entered as dummy columns (the smallest code is the reference), the coefficient table shows coefficients against the reference such as "region=2", and a line under the result table says what was changed. A variable with no measurement level set is still counted as a number, and the result says so.If you ran it before this date with a number-coded variable whose measurement level is nominal as a predictor, please run it again.
• ML Classification · Gradient Boosting · AutoML · Decision Tree · ML Regression2026-10-17 · v1.6.39A nominal variable coded as numbers (sex or region as 1/2/3) used as a predictor was counted by the size of its codes (3 as three times 1). A variable whose measurement level is "nominal" in variable settings is now entered as dummy columns (the smallest code is the reference), and a line under the result table says what was changed. A variable with no measurement level set is still counted as a number, and the result says so.If you ran it before this date with a number-coded variable whose measurement level is nominal as a predictor, please run it again.
• ML Regression2026-10-17 · v1.6.39Choosing the target variable as a predictor as well let the target predict itself, so R² came out inflated. Rows with full-width digits were dropped without a word, and nothing said that non-numeric cells had been left out. The same variable in two roles now stops the run, full-width digits are read as numbers, and the result says how many non-numeric cells were treated as missing.If you ran it before this date with the target also chosen as a predictor, or on data with full-width digits, please run it again.
• ML Classification · Gradient Boosting · AutoML · Decision Tree2026-10-17 · v1.6.39When the target column held the same value written in different ways (1, "1", " 1 ", "1.0", full-width digits), the run stopped with a server error, or, when every cell was text, the same value was split into several classes. Regression runs stopped on text in the target. The notation is now merged into one value, a genuine mix of text and numeric values (a word and 1) stops with the data-shape notice, and text in a regression target is treated as missing and noted in the result.If you ran it before this date on data whose target values were written in more than one way, please run it again.
• ML Classification · Gradient Boosting · AutoML · Decision Tree2026-10-17 · v1.6.38When a predictor cell held text that is not a number (for example a word or a score written with a unit), the cell was calculated as 0 and the missing-data line counted the row as valid. Such cells are now treated as missing, the row is left out, and the result says how many cells were dropped. A variable made only of text is refused with advice to recode it in Prep. Full-width digits are read as numbers.If you ran it before this date on data with text in a predictor column, please run it again.
• One-Way ANOVA2026-10-16 · v1.6.37A blank group cell that arrived as an empty string entered the calculation as a "blank group" (F and post hoc pairs), and the missing-data line said 0. Rows with a blank group cell are now left out.If you ran it before this date on data with blank group cells, please run it again.
• ML Classification · Gradient Boosting · AutoML · Decision Tree2026-10-16 · v1.6.37When a blank cell arrived as an empty string, a predictor cell was filled with 0, a blank outcome became a class of its own, and the missing-data line counted those rows as valid. Rows with blanks are now left out.If you ran it before this date on data with blank cells, please run it again.
• Two-Way ANOVA2026-10-16 · v1.6.37A blank factor cell that arrived as an empty string entered the calculation as a "blank level", and the missing-data line said 0. Rows with blanks are now left out (the same rule as N-way ANOVA).If you ran it before this date on data with blank factor cells, please run it again.
• N-Way ANOVA2026-10-15 · v1.6.36The Type III sums of squares on the main-effect rows used reference-category coding, so they meant something different from SPSS (in a model with an interaction), and a blank cell could enter the calculation as a level of its own. It now uses effect (sum-to-zero) coding and leaves out rows with blanks.If you ran N-way ANOVA before this date, please run it again.

How each engine was verified

Every engine has at least one form of evidence. Counts by kind:

How to cite Statory

Copy the line below into your paper or report.

Statory. (2026). Statory (Version 1.6.64) [Statistical analysis software]. https://www.statory.org

Reproducibility — will the same result come back

• Each analysis records “which engine” ran it and “with which options”, alongside the result.
• Norms are stored under a “version name” (e.g. 20260906), so you can trace which norm a past result used.
• The comparison records themselves are frozen in the repository and re-run on every release — if a value changes, the release is blocked.