UQO Working Paper No. 16· v1.2· November 2026· 104 pages
Does the Sales Comparison Approach Actually Work? A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation
Evidence from 390,996 Single-Family Transactions in Quebec, 2021–2026
A transparent comparables rule frozen in 2024 reaches an 11.1% median absolute error on 2025–2026 sales, 0.2–0.4 points behind gradient boosting given the same information and 2.2 points behind one that also sees the assessed value.
Adjusting, choosing similar rather than merely nearby sales, and using twenty to fifty comparables instead of three are the rules the evidence supports, on 124,559 out-of-sample sales and again on the 2022–2023 price decline. The paper evaluates simulated rules applied to assessment-roll characteristics, not the judgement of practising appraisers, and every level is a level on sales screened by their price-to-assessment ratio.
Simon-Pierre Boucher — contact@spboucher.ai
17.6% → 13.0%
median absolute percentage error of the five nearest sales, time adjustment only versus hedonic grid, reconciliation held at the median (paired difference 4.6 points, 95% interval 4.43–4.82); a stylised conventional grid recovers all but 0.28 points of the gain
4.6 pt
gain from selecting the five most similar rather than the five nearest sales with no physical adjustment at all: the same magnitude as the grid
20–50
comparables that are equally good once a grid is applied (paired difference between twenty and fifty −0.00 points, [−0.07; 0.05]); without a grid the optimum falls with density, from fifty in the sparsest markets to seven to ten in the densest
11.1%
out-of-sample error on 2025–2026 of the best transparent rule chosen on 2024 (compromise selection, 30 comparables, 25 km, 24 months, hedonic grid); it ranks 1 of the 1,298 eligible configurations on the test period as well
0.2–0.4 pt
cost of transparency against a quarterly-retrained gradient-boosting model given the same characteristics; 2.2 points against the same model with the assessed value as an input
10.2% vs 25.3%
error of the hedonic-grid method where two hundred or more sales occurred within 2 km in the past year versus where fewer than five did; the gap to the opaque model shrinks from 4.4 to 1.2 points from the least to the most liquid tercile
2.42 vs 2.19 pt
gap to the opaque model when the whole chain is re-run on the July 2022–December 2023 window, which contains a −13.1% peak-to-trough decline of the provincial index: the same configuration is chosen and the prescriptions hold
Abstract
The sales comparison approach rests on operating rules—how many comparables, how far, how far back, which adjustments, which weights—that are conventions rather than evidence. We backtest 2,256 simulated configurations of these rules, point in time, on 189,354 masked single-family sales in Quebec from 390,996 transactions matched to the assessment roll; rules are chosen on 2024 and evaluated on 2025–2026 with paired cluster-bootstrap inference. Holding reconciliation fixed, a hedonic grid lowers the median absolute error of five nearby sales from 17.6% to 13.0%; selecting similar rather than nearby sales is worth as much. With a grid, twenty to fifty comparables are equally good, wide windows dominate, and the radius matters only in sparse markets. The best transparent rule reaches 11.1% out of sample, 0.2–0.4 points behind a gradient-boosting model with the same information and 2.2 points behind one that also uses the assessed value. The prescriptions survive the 2022–2023 price decline.
JEL: R31, R32, C52, C53, G12
Key Results
The rules themselves as the object of study
How many comparables, how far, how far back, which adjustments and which weights are treated as explicit, simulated professional rules and backtested at the information set of each valuation date, with rules chosen on 2024 and evaluated on 2025–2026.
Adjustments dominate, and a crude grid is nearly enough
What matters is that one adjusts, far more than the sophistication of the grid: a regional grid is indistinguishable from the global one, a locally re-estimated grid improves on it by 0.16 points and cross-fitted gradient-boosting adjustments by 0.38 points, both in dense markets only.
Similarity is worth as much as adjustment
Selecting physically similar sales without any physical adjustment delivers the same gain as the grid; with the grid the two criteria converge, and a compromise criterion that uses both is best.
Three comparables is too few; weights barely matter
With a grid the error falls to about twenty comparables and is then flat; three-month windows are dominated everywhere; the radius matters little beyond five kilometres but a half-kilometre radius costs 0.43 points overall. The ten weighting–reconciliation combinations span 1.0 points and the median beats the dollar-weighted mean by 0.5 points.
The price of transparency depends on what the opaque model sees
Most of what the opaque model adds in these data comes from the assessed value and from location, not from non-linearity. When the assessed value is allowed as a transparent adjustment, a fifty-comparable assessment-ratio rule reaches 9.94% with 97.5% coverage.
Accuracy is strongly associated with submarket liquidity
The hedonic-grid method is two and a half times more accurate in the most liquid markets than in the least; a wide-net rule that retains the most similar sales adapts to liquidity on its own, and rules tuned stratum by stratum are not more accurate on the targets they cover.
Quality control
This paper went through a multi-stage quality control: 6 quality-control reports and 84 points checked. The version presented on this page is the revised manuscript. Scores are on a 0–100 scale and are a diagnostic of the manuscript, not an acceptance probability.
Score after revision
76.5/100
66.9 before quality control · +9.6
Quality control dated November 4, 2026
- Quality-control decision
- major revision
- Scientific integrity
- PASS
- Reproducibility
- R1reproduced from documented analysis data
- Maturity
- Bone revision cycle from submission
- Quality-control reports
- 6
- Points checked
- 84
Scores by dimension
Before quality controlAfter revision
- Originality & contribution74.2 → 74.0
- Scientific rigor65.0 → 76.0
- Identification & methodology67.5 → 76.0
- Data & reproducibility54.0 → 78.0
- Robustness & validation69.0 → 80.0
- Writing75.0 → 75.0
- Figures & tables68.0 → 76.0
- Literature positioning67.8 → 80.0
Revised manuscript104 pages7 figures78 tables9 equations78 references40,169 words
What quality control changed
Quality control left the headline accuracy figures standing but changed how they are read. The effects of adjustment and selection are now measured one factor at a time at fixed reconciliation, the cost of transparency is split by what the opaque model is allowed to see, a clustering defect was fixed, and the prescriptions were re-tested on a falling market.
- Adjustment effect: 18.1% → 13.0% (a comparison that also changed the reconciliation rule) → 17.6% → 13.0% with reconciliation held at the median, a paired difference of 4.6 points [4.43; 4.82].
- Cost of transparency: a single figure of 2.2 points → 0.2–0.4 points against a gradient-boosting model with the same information, and 2.2 points against one that also uses the assessed value.
- Non-linear adjustments: the in-sample gradient-boosting scheme is set aside; cross-fitted, it adds 0.38 points and a locally re-estimated grid 0.16 points, both in dense markets only, so the gain of the opaque model is attributed to the assessed value and location rather than to non-linearity.
- Cluster sensitivity: the regional clustering key was undefined for 37.9% of the targets, which collapsed them into one cluster; with the key fixed the interval width of the reference contrast goes from 1.04 to 0.42, and two-way and wild-cluster inference are added with no verdict changed.
- Selection stability: chosen rule ‘second among 1,370 configurations’ out of sample → rank 1 of the 1,298 eligible configurations of a ranked universe of 2,148.
- Added: a re-run on the 2022–2023 price decline (same configuration chosen, gap 2.42 instead of 2.19 points), a thirty-day embargo on registration dates, and point-in-time attribution of the roll variables.
Limitations stated in the paper
- The paper simulates rules and does not observe appraisers, whose files may contain information (condition, renovations, listing history) that would improve on any rule; the levels are neither an upper nor a lower bound on practice.
- Only assessment-roll characteristics are observed: adjustments for garages, pools, bedrooms and quality are untested, and for the training years a small share of dwellings carries characteristics that post-date their sale.
- Sales are screened by their price-to-assessment ratio, so an unscreened user faces levels higher by three to six tenths of a point; the sensitivity of the comparable pool to a looser screen was not re-run.
- The opaque benchmark is one untuned boosting model, so the gaps are lower bounds on the gap to the best opaque model, and its advantage in sparse markets was not tested with municipalities held out of training; the study covers single-family dwellings in one province and the registry snapshot is not redistributable.
Data
Vrai Prix transaction database
Research database linking the registered residential transactions of the Registre foncier du Québec to municipal assessment-roll extracts; snapshot of 10 October 2026 with 766,596 transactions. From 456,950 transactions coded as single-family, sequential screening rules leave 390,996 arm's-length sales from January 2021 to September 2026, with a median price of $400,000.
Chronological partition
201,642 sales from January 2021 to December 2023 develop the adjustment grids and benchmark models; 64,795 sales of 2024 serve to choose the rules; 124,559 sales of 2025–2026 are the out-of-sample test. In total 189,354 target sales are valued with their price masked.
Assessment-roll variables at the sale date
The roll variables that enter a valuation (assessed value, roll vintage, land share) are those of the roll in force at the sale, not of the current roll; a control run shows that this choice leaves the headline gap unchanged within noise (−0.02 points, [−0.07; 0.04]).
Methodology
Point-in-time backtest
Only transactions registered strictly before the valuation date are admissible as comparables; the price index is re-estimated every month and the adjustment grids and machine-learning benchmarks every quarter, from prior sales only.
Error surface and factorial design
A first experiment maps the error over the number of comparables (1 to 50), the search radius (0.5 to 25 km) and the look-back window (3 to 24 months) for three selection rules and two adjustment regimes; a second crosses four selection criteria, eight adjustment schemes, five weighting schemes and two reconciliation rules at three anchor configurations.
Horse race against benchmarks
Named models, from the median of the three nearest sales to hedonic models, a quarterly-retrained gradient-boosting model with and without the assessed value, and a hybrid, are compared out of sample and by liquidity, heterogeneity, region and price strata; five hypotheses are tested with paired intervals within strata.
Paired cluster-robust inference
Paired differences of forecast losses with a 999-replication cluster bootstrap over municipality–quarter cells, checked against coarser spatial clusters, two-way clustering by municipality and quarter, and a wild cluster bootstrap over quarters.
Selection audit and robustness
Validation-to-test rank correlation of 0.996 over all ranked configurations, a falling-market re-run on 2022–2023, a thirty-day embargo on registration dates, sales excluded by the price-to-assessment screen, alternative accuracy measures and benchmark seeds.
Reproducibility
- The repository contains the full pipeline in Python with pinned dependencies: sample construction, index, quarterly models, distributed backtest, analysis, quality-control analyses and the generation of every table, figure and number of the paper.
- A single entry point regenerates the exhibits from the stored per-target estimates and checks them against the committed versions; a documented dry-run path executes the full chain on a random subsample of targets.
- No number in the text is typed: every figure quoted is a macro generated from a stored result file. The boosting fits are seeded and bit-reproducible across the nodes of the cluster.
- The linked registry snapshot is not redistributed because it identifies individual properties; access for replication is granted to researchers on request under a data-use agreement that excludes redistribution.
- The per-target estimates of every configuration and model contain no price or address and are available from the author to any researcher; file sizes and checksums are recorded in the paper and the replication package.
Cite this paper
Boucher, S.-P. (2026). Does the Sales Comparison Approach Actually Work? A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation. UQO Working Paper No. 16, Université du Québec en Outaouais.
@techreport{boucher2026wp16,
author = {Simon-Pierre Boucher},
title = {Does the Sales Comparison Approach Actually Work? A Large-Scale Point-in-Time Validation of Comparable Selection, Price Adjustments, and Value Reconciliation},
institution = {Université du Québec en Outaouais, Département des sciences administratives},
number = {16},
type = {UQO Working Paper},
year = {2026},
month = {nov},
url = {https://www.spboucher.ai/research/wp16},
}More in Real Estate & Urban Economics