AVM quality control

AVM quality control

The federal banking and housing regulators’ interagency AVM Quality Control Standards (implementing the Dodd-Frank Act’s automated-valuation-model provision; the rule text lives at 12 CFR Part 34 Subpart H for national banks and parallel provisions for other regulators) name five factors an automated valuation model should be designed to achieve. 12 CFR Part 34 governs federally regulated mortgage lenders and the appraisal management companies they use — it does not apply to this system today, and this system is deliberately kept out of lender-facing use (see AVM disclosure: “no output may be used for tenant screening, credit, employment, or insurance decisions”). The five factors are still the clearest public standard for “is an AVM’s quality control trustworthy,” so this page states, plainly and without overclaiming compliance, where current practice stands on each one.

This page is descriptive, not a compliance certification. It is operator-written and not lawyer-reviewed — like AVM disclosure and fair housing, which it extends, counsel review is an open item before any statement here is treated as a compliance position.

1. Confidence in the estimates produced

Current practice:

• Every valuation ships with a method + confidence + range, never a bare number (INV-5: no displayed valuation without {method, confidence, as_of}; a Tier-3/suppressed result renders no number at all).
• A suppression ladder is enforced in exactly one place in the engine: enough verified renovated comps clears a point estimate; a thinner comp set degrades to a range or an area-level estimate labeled as such; too few comps suppresses the number entirely with a machine-readable reason (valuation).
• Per-segment accuracy is published, not a single blended number: South Side SFH, North Side 2–4 unit, suburban Cook SFH, and the $1M+ luxury tail each carry their own MdAPE, within-±20% rate, and bias, measured leave-one-out with no future leakage (accuracy). A newer, stricter spatial-block walk-forward backtest (held out by time, reported per township block rather than randomly split, so it can’t inherit the accuracy inflation that random splits produce on spatially autocorrelated data) is published on the same page as it becomes available.
• A golden reference set of locked hard addresses (deep-value SFH, luxury, thin-comp, Lake County) runs offline against the real engine on every change; a champion/challenger promotion rule blocks any model update whose MdAPE regresses on even one segment by more than half a point (Model Ops).

2. Protection against data manipulation

Current practice:

• Versioned, not hand-edited, artifacts. Every pricing constant the engine uses resolves to a versioned constants registry — no literals buried in code (INV-7). Every output carries model_version; a change that moves numbers is logged in the changelog with what changed and why. Nothing is quietly re-tuned per property or per request.
• Input drift tripwires. A monitor compares recent sale-price medians and the live ARV/ask band against a trailing baseline and fires warn/alert on anomalous shifts, rather than silently absorbing a data shock (Model Ops).
• No poisoned batches reach serving. A pipeline run that fails its data-quality expectations is quarantined, never written to the serving tables (INV-6).
• No PII in the serving layer. County deed records carry buyer/seller names; the loaders that build the corpus deliberately omit those columns, and the serving schema structurally lacks a name column — enforcement is “the column doesn’t exist,” not a filter that could be forgotten. Party names never reach any consumer surface, API response, or MCP tool output. Full policy: see the deed-party PII section referenced from privacy.
• Deterministic, reproducible tests. The golden reference set runs against frozen fixtures and a frozen clock — the same inputs produce the same outputs forever, so a regression is a code change, not a data coincidence.

3. Avoidance of conflicts of interest

Current practice, stated plainly: this is a solo-operator tool (about) — the same person builds the valuation model and uses it to underwrite real deals. That is not the separation-of-duties model traditional AVM QC guidance assumes (an independent model-risk function, distinct from loan production). There is no institutional wall here, and this page will not pretend there is one.

What actually mitigates the conflict:

• The model is deterministic and not manually overridable per property. Every valuation runs the same published formula from the same versioned constants regardless of who is asking or what they’d like the number to be; there is no admin control that sets an individual address’s ARV by hand.
• The full methodology is published, not held back as a trade secret — comp policy, weights, penalties, and every assumption behind the underwriting math are on this docs site, so the model is falsifiable by anyone, not self-certified. The tone is deliberately blunt about limitations rather than promotional (see limitations).
• The operator’s own deals are graded on the same public scorecard as everyone else’s. The standing flip-pair validation feed grows from realized Cook County resales generally, and the ledger is being seeded with the operator’s own real deals so they are graded, not exempted — the same MdAPE and bias numbers apply.
• No paid tiers, no accounts, no advertiser relationship exist that would create an incentive to inflate a score for a customer (about: “no company, no sales team”). Removing the commercial incentive doesn’t remove the personal one — the operator can still transact on properties the model scores — which is exactly why this section exists instead of being silent about it.

This factor is the one where “in progress” is the honest word for the underlying structural question (this system will not have an independent model-governance function until it has more than one person); the mitigations above are what’s real today.

4. Random sample testing and reviews

Current practice:

• Nightly synthetic canaries fire real underwrite calls against the live production API across 20 geographically dispersed tracts (North/West/South Chicago, suburban Cook, and Lake), checking latency, honest-suppression-or-value behavior, and invariant compliance — not a manual spot check, a scheduled one (Model Ops).
• Nightly drift monitoring on input distributions and the output ARV/ask band, described above.
• Standing flip-pair validation grades every trailing-24-month arms-length flip pair using only pre-purchase-date data, growing nightly — a continuously refreshed random(-ish, market-determined) sample of realized outcomes, not a one-time backtest (accuracy).
• Spatial-block walk-forward backtesting, sampled by quarter and property type/geography strata with per-quarter caps to avoid a single hot segment dominating the sample, with results broken out per township block so no single geography can carry the pooled number (accuracy — “Accuracy by segment”).
• CI-blocking golden and invariant suites run on every change before it ships, and the champion/challenger shadow-mode comparison runs any candidate model against a fixed, stratified 100-subject set before promotion.

Not yet built: independent (non-operator) review of test results, and IAAO-style ratio studies (COD/PRD/PRB) per segment — both are Phase 2 items, not yet shipped (DEVELOPMENT_PLAN_V3 §3, task 2.6).

5. Compliance with applicable nondiscrimination laws

Current practice:

• Protected-class composition is used only in offline fairness testing, never in valuation, opportunity scoring, property risk, recommendation, or serving. Applicant/household income and crime-derived proxy fields are restricted offline diagnostics and are stripped from the current public model and payloads (fair housing).
• A per-lens fair-housing audit exists and is public (specs/FAIR_HOUSING_CHECKLIST.md): every risk/opportunity signal carries a verdict (OK / needs reworded copy / needs review), crime-type breakdowns and school-quality rankings are excluded from steering-adjacent public display, and a lens without a verdict does not ship.
• Programmatic content (including the Chicago markets pages) draws only from checklist-OK, non-demographic, non-crime, non-school signals — enforced as a generator allow-list, not a post-hoc filter.

Built, and currently failing one gate — said plainly: pipeline/out/fairness.json compares absolute percentage error and interval coverage across majority-minority and other tracts with bootstrap intervals. The current artifact reports pass:false: the error-level differential exceeds its bound and its confidence interval excludes zero. The per-stratum 80% interval-coverage parity check passes, but that does not cancel the error-level failure. The result is report-only and not a legal conclusion; it blocks fairness sign-off, score naming/promotion, and relaxation of the proxy stop-loss pending model review and counsel.

Summary table

Factor	Status
Confidence in estimates	Substantially in place — published accuracy + confidence + suppression ladder + golden/invariant gating
Data-manipulation protection	Substantially in place — versioned artifacts, no PII in serving layer, drift tripwires, quarantined bad batches
Conflict-of-interest avoidance	Disclosed, not structurally solved — solo-operator conflict named openly; mitigated by determinism, published methodology, and grading the operator’s own deals publicly, not by an independent review function
Random sample testing and reviews	Live and nightly — canary, drift, flip-pair grading, spatial-block backtest; independent review and ratio studies not yet built
Nondiscrimination	Failing/open — proxy stop-loss is enforced and interval-coverage parity passes, but the current fairness regression fails its error-level differential gate; counsel review remains open

Counsel review of this page, alongside the fair-housing checklist verdicts, is an open item before any claim on this page is treated as a compliance position rather than an engineering practices statement.