# Reconstruction study protocol (proposed)

Status: **proposed**. Nothing here has been run. This file backs claim C-12 only as a design; the claim stays `proposed` until the study is run and reported.

## Question

Can claims professionals reconstruct an automated decision faster and more accurately from a warrant receipt than from ordinary system logs?

## Why it matters

Insurers cite opacity as a reason to exclude AI-related losses. If receipts measurably improve reconstruction, "we make AI decisions reconstructable" becomes an evidenced claim. If they do not, that result is published and the claim is withdrawn.

## Design

- **Participants:** claims handlers, adjusters or claims counsel. Target at least 12; report the actual number and roles.
- **Cases:** the three synthetic insurance crossings on this site (INS-001, INS-002, INS-003).
- **Conditions:**
  - **Log condition:** the plain log packet for the case (`packets/INS-00x-log.txt`).
  - **Receipt condition:** the determination and appraisal JSON for the same case, downloaded from the Study page.
- **Assignment:** each participant sees every case once, in one condition, counterbalanced so each case appears equally often in each condition.
- **Task:** answer the eight reconstruction questions (below) for each case.

## Reconstruction questions

1. What action was taken or proposed, on what object?
2. Under whose authority, and does that authority trace to someone other than the system itself?
3. Which evidence was relied on, and was it about this claim?
4. Was the evidence current when the decision took effect?
5. Did review complete before the decision was committed?
6. Which claimant routes (review, correction, appeal) remained available afterwards?
7. Did the decision take effect before it was warranted?
8. What would have had to change for the decision to be warranted?

## Measures (fixed before any run)

- **Accuracy:** answers scored against `answer-key.v0.json`, which is derived from the gate's checks. Partial credit: each question scores the share of its `must_include` facts the participant named (paraphrase allowed), from 0 to 1, so a case scores 0–8. The scorer records which facts were named, not just the score. Scorers are blind to condition.
- **Time:** minutes from opening the packet to submitting answers.
- **Confidence:** self-rated 1–5 per answer, reported separately and never combined with accuracy.

## Analysis (fixed before any run)

- Primary: difference in mean accuracy between conditions, with a 95% interval.
- Secondary: difference in median time.
- No subgroup analysis unless declared here before the run.
- All results are reported, including a null or negative result.

## Threats to validity

- Receipts were designed by the same people who wrote the answer key. Mitigation: publish the answer key before the run; have one independent reviewer check it.
- Synthetic cases may be easier than real ones. The result applies only to these cases.
- Log packets were written by hand for the study. Publish them, so readers can judge whether they are a fair comparison.

## Files

- `preregistration.json` — the fields to freeze before recruitment.
- `answer-key.v0.json` — derived from the gate's checks by `src/lib/study.ts`; its SHA-256 is recorded in `preregistration.json`.
- `scoring-sheet.csv` — one row per participant, case and question.
- `packets/INS-001-log.txt`, `packets/INS-002-log.txt`, `packets/INS-003-log.txt` — log condition.