Accuracy Metrics

This appendix provides a comprehensive breakdown of every verification metric across the included dataset. These numbers are the empirical basis for the trust claims made throughout this documentation.

All verification metrics are deterministic — computed by code against the source bill text, with zero LLM involvement. TAS resolution metrics include both deterministic matching and LLM-verified results.

Aggregate Summary

The single most important number: 99.995% of dollar amounts verified across 34,568 provisions from 32 bills. Every extracted dollar amount was confirmed to exist in the source bill text.

Per-Bill Breakdown

H.R. 4366 — Consolidated Appropriations Act, 2024 (Omnibus)

Notes on H.R. 4366 metrics:

• The 723 ambiguous dollar amounts reflect the high frequency of round numbers in a 1,500-page bill. The most common: $5,000,000 appears 50 times, $1,000,000 appears 45 times, and $10,000,000 appears 38 times in the source text.
• The 20 “no match” raw text provisions are all non-dollar provisions — statutory amendments (riders and mandatory spending extensions) where the LLM slightly reformatted section references. No provision with a dollar amount has a raw text mismatch.
• Coverage of 94.2% means 5.8% of dollar strings in the source text were not matched to a provision. These are primarily statutory cross-references, loan guarantee ceilings, struck amounts in amendments, and proviso sub-references that are correctly excluded from extraction. See What Coverage Means (and Doesn’t).

H.R. 5860 — Continuing Appropriations Act, 2024 (CR)

Notes on H.R. 5860 metrics:

• The CR has a much higher proportion of non-spending provisions (riders and mandatory spending extensions) compared to an omnibus. Only 5 provisions are standalone appropriations — principally the $16 billion FEMA Disaster Relief Fund.
• All 13 CR substitution pairs are fully verified: both the new amount ($X) and old amount ($Y) were found in the source text.
• The 16 “no match” raw text provisions are riders and mandatory spending extensions that amend existing statutes. The LLM sometimes reformats section numbering in these provisions (e.g., adding a space after a closing parenthesis).
• Coverage of 61.1% is expected for a continuing resolution. CRs reference prior-year appropriations acts extensively — those references contain dollar amounts that appear in the CR’s text but are contextual citations, not new provisions.

H.R. 9468 — Veterans Benefits Supplemental (Supplemental)

Notes on H.R. 9468 metrics:

• This is the simplest bill in the example data — only 2 dollar amounts in the entire source text, both uniquely verifiable.
• Perfect coverage: every dollar string in the source is accounted for.
• The 2 “no match” raw text provisions are the SEC. 103 directives (reporting requirements), where the LLM’s raw text excerpt was truncated and doesn’t appear as-is in the source. The content is correct; only the excerpt boundary is slightly off.
• Both appropriations ($2,285,513,000 for Compensation and Pensions + $596,969,000 for Readjustment Benefits) are verified at unique positions — the strongest possible verification result.

Amount Verification Detail

The verification pipeline searches for each provision’s text_as_written dollar string (e.g., "$2,285,513,000") verbatim in the source bill text.

Three outcomes

Why ambiguous is so common

Round numbers appear frequently in appropriations bills. In H.R. 4366:

When the tool finds $5,000,000 at 50 positions, it confirms the amount is real but can’t determine which of the 50 occurrences corresponds to this specific provision. That’s “ambiguous” — correct amount, uncertain location.

The 797 “verified” provisions have dollar amounts unique enough to appear exactly once in the entire bill — amounts like $10,643,713,000 (FBI Salaries and Expenses) or $33,266,226,000 (Child Nutrition Programs).

Internal consistency check

Beyond source text verification, the pipeline also checks that the parsed integer in amount.value.dollars is consistent with the text_as_written string. For example:

Across all 1,522 provisions with dollar amounts: 0 internal consistency mismatches.

Raw Text Verification Detail

Each provision’s raw_text excerpt (~first 150 characters of the bill language) is checked as a substring of the source text using four-tier matching.

Tier results across all example data

Analysis of the 38 no-match provisions

All 38 “no match” provisions share a critical property: none of them carry dollar amounts. They are all non-dollar provisions — riders and mandatory spending extensions that amend existing statutes.

The typical pattern:

• Source text: Section 1886(d)(5)(G) of the Social Security Act (42 U.S.C. 1395ww(d)(5)(G)) is amended—
• LLM raw_text: Section 1886(d)(5)(G) of the Social Security Act (42 U.S.C. 1395ww(d)(5)(G)) is amended— (1) clause...

The LLM included text from the next line, creating a raw_text that doesn’t appear as a contiguous substring in the source. The statutory reference and substance are correct; the excerpt boundary is slightly off.

Implication: The 38 no-match provisions don’t undermine the tool’s financial accuracy — they affect only the provenance trail for non-dollar legislative provisions. Dollar amounts are verified independently through the amount checks, which show 0 not-found across all data.

Per-bill breakdown

Note: The detailed per-bill breakdown above covers the original three FY2024 example bills (H.R. 4366, H.R. 5860, H.R. 9468). The aggregate metrics at the top of this page reflect all 32 bills across FY2019–FY2026.

The omnibus has the highest exact match rate (96.7%), which makes sense — it’s the most straightforward appropriations text. The CR and supplemental have more statutory amendments (which are harder to quote exactly), contributing to their higher no-match rates.

Completeness (Coverage) Detail

Coverage measures what percentage of dollar-sign patterns in the source text were matched to at least one extracted provision’s text_as_written field.

Per-bill coverage

Why coverage varies

H.R. 9468 (100%): The simplest bill — only 2 dollar amounts in the entire source text, both captured.

H.R. 4366 (94.2%): The ~100 unaccounted dollar strings are primarily:

• Statutory cross-references to other laws (dollar amounts cited for context, not new provisions)
• Loan guarantee face values (not budget authority)
• Old amounts being struck by amendments (“striking ‘$50,000’ and inserting ‘$75,000’”)
• Proviso sub-amounts that are part of a parent provision’s context

H.R. 5860 (61.1%): Continuing resolutions reference prior-year appropriations acts extensively. Those referenced acts contain many dollar amounts that appear in the CR’s text but are citations of prior-year levels, not new provisions. Only the 13 CR substitutions, 5 standalone appropriations, and a few limitations represent genuine new provisions with dollar amounts.

Why coverage < 100% doesn’t mean errors

Coverage below 100% means there are dollar strings in the source text that weren’t captured as provisions. For most of these, non-capture is the correct behavior:

• A statutory reference like “section 1241(a) ($500,000,000 for each fiscal year)” contains a dollar amount from another law — it’s not a new appropriation in this bill.
• A loan guarantee ceiling like “$3,500,000,000 for guaranteed farm ownership loans” is a loan volume limit, not budget authority.
• An amendment language like “striking ‘$50,000’” contains an old amount that’s being replaced — the replacement amount is the one that matters.

See What Coverage Means (and Doesn’t) for a comprehensive explanation with examples.

CR Substitution Verification

All 13 CR substitutions in H.R. 5860 are fully verified — both the new amount ($X in “substituting $X for $Y”) and the old amount ($Y) were found in the source bill text:

26 of 26 dollar amounts verified (13 new + 13 old). This is the strongest verification possible for CR substitutions — both sides of every “substituting X for Y” pair are confirmed in the source text.

Budget Authority Verification

Budget authority is computed deterministically from provisions — never from LLM-generated summaries.

The formula

Budget Authority = sum of amount.value.dollars
    WHERE provision_type = "appropriation"
    AND   amount.semantics = "new_budget_authority"
    AND   detail_level NOT IN ("sub_allocation", "proviso_amount")

Detail level filtering

In H.R. 4366, the detail level distribution for appropriation-type provisions is:

Without the detail level filter, the budget authority sum would be $846,159,099,554 — approximately $22 million higher than the correct total of $846,137,099,554. The $22 million represents sub-allocations and proviso amounts correctly excluded from the total.

Regression testing

The exact budget authority totals are hardcoded in the integration test suite:

#![allow(unused)]
fn main() {
let expected: Vec<(&str, i64, i64)> = vec![
    ("H.R. 4366", 846_137_099_554, 24_659_349_709),
    ("H.R. 5860", 16_000_000_000, 0),
    ("H.R. 9468", 2_882_482_000, 0),
];
}

Any change to the extraction data, provision parsing, or budget authority calculation that would alter these numbers is caught immediately by the budget_authority_totals_match_expected test. This is the tool’s primary financial integrity guard.

Independent reproducibility

The budget authority calculation can be independently reproduced in Python:

import json

with open("data/118-hr4366/extraction.json") as f:
    data = json.load(f)

ba = 0
for p in data["provisions"]:
    if p["provision_type"] != "appropriation":
        continue
    amt = p.get("amount")
    if not amt or amt.get("semantics") != "new_budget_authority":
        continue
    val = amt.get("value", {})
    if val.get("kind") != "specific":
        continue
    dl = p.get("detail_level", "")
    if dl in ("sub_allocation", "proviso_amount"):
        continue
    ba += val["dollars"]

print(f"Budget Authority: ${ba:,.0f}")
# Output: Budget Authority: $846,137,099,554

This produces exactly the same number as the CLI. If the Python and Rust calculations ever disagree, something is wrong.

What These Metrics Do and Don’t Prove

What the metrics prove

What the metrics don’t prove

The 95.6% exact match rate as attribution evidence

While verification cannot mathematically prove attribution (that a dollar amount is assigned to the correct account), the 95.6% exact raw text match rate provides strong indirect evidence:

• If the raw text excerpt is byte-identical to a passage in the source, and that passage mentions an account name and a dollar amount, the provision is almost certainly attributed correctly.
• The 38 provisions without text matches are all non-dollar provisions, so attribution is a non-issue for them.
• For the 725 ambiguous dollar amounts, the combination of a verified dollar amount and an exact raw text match narrows the attribution to the specific passage the raw text came from.

For high-stakes analysis, supplement the automated verification with manual spot-checks of critical provisions. See Verify Extraction Accuracy for the procedure.

Reproducing These Metrics

You can reproduce every metric in this appendix using the included example data:

# The full audit table
congress-approp audit --dir data

# Budget authority totals
congress-approp summary --dir data --format json

# Provision type counts
congress-approp search --dir data --format json | \
  jq 'group_by(.provision_type) | map({type: .[0].provision_type, count: length}) | sort_by(-.count)'

# CR substitution verification
congress-approp search --dir data/118-hr5860 --type cr_substitution --format json | jq length

# Detailed verification data
cat data/118-hr9468/verification.json | python3 -m json.tool | head -50

All of these commands work with no API keys against the included data/ directory.

How Metrics Change with Re-Extraction

Due to LLM non-determinism, re-extracting the same bill may produce slightly different metrics:

The verification pipeline ensures that dollar amount accuracy is invariant across re-extractions — even if provision counts or classifications change, the verified amounts are always correct because they’re checked against the source text, not against the LLM’s internal state.

Next Steps

• Verify Extraction Accuracy — practical guide for running your own audit
• How Verification Works — technical details of the verification pipeline
• What Coverage Means (and Doesn’t) — understanding the completeness metric
• Included Example Bills — detailed profiles of each example bill