Analyzing and Predicting Used Car Prices, Take Two — The Technical Write-Up

project · august 2026 · python, statsmodels · back to the short version · source on github · the 2025 version

this is the full, assumption-by-assumption version of the write-up; the short version tells the same story in a five-minute read. in march 2025 i built my first linear regression project: a model that predicts what a used car should sell for, trained on craigslist listings. it worked, and i was proud of it. it also cut every corner i didn't yet know existed. this is the same project done again with the full toolkit from my mcmaster stats courses (stats 3a03, math 3mb3) — the transformation chosen by box-cox instead of by truncating the data, every assumption checked instead of assumed, every claim carrying a confidence interval, and every headline number measured on listings the model never saw.

contents

1. why do it again
2. the data, cleaned honestly
3. what the data looks like
4. letting box-cox choose the transformation
5. three models, two nested tests
6. checking the assumptions
7. how well it predicts
8. what moves the price
9. the depreciation curve
10. six cars, including my dream car
11. what i'd still improve
12. reproduce it
13. references

why do it again

the 2025 version got the big things right: real data, a sensible set of predictors, a train/test split, and honest writing about where it failed. but rereading it after finishing my degree, three things stood out.

• it counted the same cars many times. craigslist sellers cross-post one car in several cities. 40% of the raw rows are exact duplicates on year, make, model, price, and odometer. duplicate rows aren't independent observations, and independence is the assumption every standard error in a regression rests on.
• it solved skewness by deleting the right tail. capping prices at the 95th percentile ($44,500) made the histogram look nicer — by removing every luxury car from the data. that's why the old model missed my dream car, a lexus is f, by 31%: it had never been allowed to see a car like it. the textbook fix for right-skew isn't truncation, it's a transformation, and there's a formal tool for choosing one.
• it stopped at r². no residual plots, no leverage, no multicollinearity check, no confidence intervals, no prediction intervals. the model's error was quoted as a single number (±$3,781) when the honest answer is a distribution.

the data, cleaned honestly

the data is the same kaggle snapshot of craigslist listings (426,880 rows, april–may 2021) scraped by austin reese. instead of one big filter, every cleaning decision is applied one step at a time and logged, so the whole funnel is auditable — the table below is generated straight from the pipeline. rows missing any of the duplicate-key fields are left alone at the first step: a duplicate you can't verify isn't a duplicate.

the biggest judgment call is complete cases: requiring cylinders, condition and drive costs 60k rows. the trade is deliberate — those variables earn their keep in the model, and the alternative (imputing or "not stated" levels) is on the future-work list.

the final sample: 72,120 listings, 34 brands, model years 1996–2022, prices $1,000 to $125,000. about 2.4% of these cars sit above the old $44,500 cap — the segment the 2025 model was never allowed to learn.

what the data looks like

asking prices are strongly right-skewed (skewness 2.46): median $9,000, mean $12,857, and a long tail of trucks and luxury cars. on a log scale the same distribution is nearly symmetric (skewness −0.03). that's the first hint that the model should think in percentages, not dollars.

price against age and against mileage, with price on a log scale. the median line falls roughly straight for the first decade — which on a log axis means a constant percentage drop per year — then flattens: old cars stop losing value once they're cheap.

brands sort themselves before any model is fit — porsche's median asking price is six times saturn's. (ram and gmc rank high not because trucks are luxurious but because their listings are trucks; the regression later separates the truck effect from the brand effect.)

the 2025 version's correlation analysis, redone on the log scale it should have used: log(price) correlates with age at −0.63 and odometer at −0.53 (stronger than the −0.55 and −0.48 in dollar space), with cylinders a mild +0.24. age and odometer are themselves correlated (+0.55) — cars age and accumulate miles together — which is exactly why the regression, not the correlation table, gets the last word on who does what.

letting box-cox choose the transformation

the 2025 write-up ended with "i would like to apply a natural logarithm to the price variable". this version starts there, but doesn't take the log on faith. the box-cox procedure fits the full covariate model at every power transformation of the response — λ = 1 is untransformed dollars, λ = 0 is the log — and profiles the likelihood.

the mle is λ̂ = 0.18 with a razor-thin 95% interval of (0.175, 0.180) — with 57,715 observations, the likelihood can tell apart values of λ that are practically identical, so the interval excludes both 0 and 1. the honest reading follows the textbook advice: λ = 1 (dollars) is emphatically rejected; λ̂ sits near zero; and among nearby values, λ = 0 is the one with an interpretation — coefficients become percentage effects on price. i use the log, and let the diagnostics section judge whether it was enough.

three models, two nested tests

three specifications, each an answer to "what did the change buy?". m1 is the 2025 model refit on the improved sample; m2 changes the response to log(price) and adds drive wheels (a variable the 2025 version threw away); m3 adds a quadratic age term, because the depreciation curve visibly flattens. all three are fit by ols on the same 80% training split (seed 138, same as 2025), with the same 20% held out.

log(price) = β₀ + β₁·age + β₂·age² + β₃·odometer + β₄·cylinders
           + brand + condition + fuel + title + transmission + body type + drive + ε

r² is not comparable between the dollar and log rows (different response scales) — the out-of-sample comparison below is the fair one. aic is comparable only between m2 and m3.

the additions are tested formally, the way nested models should be. because breusch-pagan (next section) rejects constant error variance, both tests are run as wald tests on the heteroskedasticity-robust (hc1) covariance; the classical partial f-tests, which assume the rejected constant variance, agree on both counts.

• curvature. adding age² : robust wald f(1, 57,655) = 986.0, p < 2.2×10⁻¹⁶ (classical partial f = 1,515.7). the depreciation curve genuinely bends — the constant-percentage-per-year story is too simple.
• brand. dropping all 33 brand dummies from m3: robust wald f(33, 57,655) = 223.6, p < 2.2×10⁻¹⁶ (classical f = 210.4). who made the car matters even after age, mileage, size and body type are held fixed.

multicollinearity, the thing correlation tables can't adjudicate, is checked with variance inflation factors on m2: the largest is 2.87 (4wd), and no term exceeds the usual concern threshold of 5 (full table in vif.csv). age and odometer are correlated, but not enough to destabilize the coefficients.

checking the assumptions

this is the section the 2025 version didn't have. the residual plots below are the before-and-after argument for the log transformation: in dollar space (m1, top row) the residuals fan out as fitted values grow — textbook heteroskedasticity — and the q-q plot bends hard away from normal in both tails. in log space (m3, bottom row) the residual band is level and the q-q plot hugs the line through the central 99%+ of the data, with modest heavy tails.

the formal version of the same story:

• heteroskedasticity. breusch-pagan rejects constant variance for both models (with n = 57,715 it detects even trivial patterns; the lm statistic falls by more than half from m1 to m3, 3,400 → 1,593). the remedy is not to pretend otherwise: every standard error, confidence interval, p-value and nested-model test quoted in this report uses heteroskedasticity-robust (hc1) covariance, so the inference doesn't lean on the rejected assumption.
• influence. the largest cook's distance in the training fit is 0.003 — far below even the most conservative rule of thumb. with 57,715 observations, no single craigslist ad moves the model.
• normality. mild heavy tails remain in the log model. with this sample size the coefficient inference is protected by the central limit theorem; where normality actually matters is the prediction intervals, so instead of trusting it i measure their coverage below.

how well it predicts

every number in this section is computed on the 14,405 held-out listings, with 5-fold cross-validation on the training split as a second opinion. predictions from the log models are converted back to dollars with duan's smearing estimator (a naive exp() would be biased low; the smearing factor here is 1.071).

cross-validation agrees: mae $4,236 / $3,215 / $3,250, mape 58.3% / 34.1% / 33.7% for m1 / m2 / m3.

the log transformation, not extra complexity, does the heavy lifting: on identical data the 2025 specification's typical (median) error is $3,032, and thinking in percentages cuts it to $1,835 — 39% smaller — while mape drops from 58% to 33%. between m2 and m3 the choice depends on the loss you care about: m3 is better in log space (aic 52,129 vs 53,625), on median error and on mape; m2 edges it on the mean-type dollar metrics (mae, rmse, dollar r²), which are dominated by the priciest cars. i take m3 as the final model — its errors are smaller for the typical car, and its quadratic term is what makes the depreciation curve below believable — and report the comparison so the trade-off is visible.

the part i care most about: uncertainty that means what it says. the final model's 95% prediction intervals, built in log space and exponentiated, cover the true price for 95.3% of the 14,405 held-out cars — within a third of a point of nominal. coverage holds across the price range, from 93.4% in the cheapest quartile of predictions to 97.2% in the priciest. the intervals are honest and they are wide (the upper bound is typically ~4.4× the lower): a single regression on ad-listed features genuinely cannot pin a used car's price much tighter, and pretending otherwise is what a point forecast does.

what moves the price

because the response is log(price), each coefficient converts to a percentage effect on price, holding everything else fixed. all intervals below are 95%, from robust standard errors. these are associations in 2021 craigslist asking prices, not causal effects — a diesel badge can't be bolted onto a corolla to raise its value 82%.

• every year of age costs about 11.8% of the price (ci 11.5–12.1) when the car is new, easing to roughly 8% per year at age ten and under 5% by age twenty — that's the age² term flattening the curve.
• every 10,000 miles costs a further 3.3% (ci 3.2–3.4), independent of age.
• diesel is the biggest premium outside the brand effects: +82% (ci 79–85). in 2021, used diesel pickups were their own seller's market.
• the manual transmission premium the 2025 version spotted survives the rebuild: +15.8% (ci 14.0–17.5). automatics took over the commodity market; the manuals that remain are enthusiasts' cars.
• a branded history is expensive: a rebuilt title costs 16.5%, a salvage title 17.6%, and cars whose condition is salvage lose 40%.
• honesty check: not every coefficient earns a headline. the hatchback effect (+1.4%, ci −0.2 to +2.9, p = 0.08) is not statistically distinguishable from zero; the hybrid premium is real but small (+2.8%, ci 0.5–5.2); and self-reported "new" condition adds a noisy +6.6% (ci 0.5–13.1) — less than "excellent" does. a report that only prints impressive coefficients is advertising, not analysis.

brand effects, now measured against an otherwise identical ford rather than as raw medians: porsche commands +110%, lexus +55%, toyota +39% — while fiat (−24%) and mercury (−22%) sit at the bottom. the 2025 version's "price boosters" table mixed brand prestige with body type; holding body type fixed is what separates "porsches are expensive" from "trucks are expensive".

the depreciation curve

the model's age terms, drawn as a curve for a concrete car (a 4-cylinder automatic toyota sedan, clean title, good condition, carrying the median mileage for its age): it sheds half its nearly-new value by year five, and by year twelve it's a $6,000 car. this is the shape the quadratic term buys — the 2025 model's straight line in dollar space couldn't fall fast early and flatten late at the same time.

six cars, including my dream car

the 2025 write-up tested the model on three cars i found on autotrader: a corolla, a rav4, and the lexus is f i still daydream about. those three are re-scored below — but scoring 2025 canadian listings with a model of the april–may 2021 us market mixes two questions: is the model good, and has the market moved? (it has.) so the fair test comes first: the same three archetypes scored against 2021 listings the model didn't fit. the corolla and rav4 come straight from the held-out test split. the dataset's only real is f landed on the training side of the split, so it's scored leave-one-out: the model is refit without that listing before predicting it — out of sample by construction (and with the largest cook's distance anywhere at 0.003, removing one row barely moves the fit anyway).

the 2025 version fed the corolla's odometer in kilometres into a model expecting miles; the re-scores above use the converted 82k miles.

the toyotas behave: both corollas and both rav4s fall inside their intervals, even across four years of market drift. the is f does not behave, twice — four of the six actual prices land inside their intervals, and both misses are the same car. the 2021 is f's $34,999 asking price sits ninety-two dollars above its interval's upper edge; the 2025 listing sits about $4,900 above its. and this time the model can't blame the data. the reason is visible in the model's own structure: to a manufacturer-level regression, an is f is "a 13-year-old 8-cylinder lexus sedan", i.e. a used ls 460. the 500-horsepower badge lives in the model column, which carries 9,650 distinct values and stayed out of this regression. the honest conclusion isn't that the car is mispriced — it's that a limited-production performance car's value is its model name, and a brand-level model has a measurable ceiling there. (the 2025 version "did better" on the is f — 31% off instead of 58% on the same autotrader listing — but by accident: its straight-line depreciation overvalued all old cars, which happened to flatter the one old car that actually holds its value.)

what i'd still improve

• model-level effects. the is f problem. random effects or partial pooling over the ~9,600-value model column (or even a curated "enthusiast car" flag) would price special cars without exploding the design matrix.
• asking vs selling. craigslist prices are asks, not transactions. every effect here is an effect on what sellers hope, measured precisely.
• a time index. the model is a snapshot of april–may 2021 — the start of the wildest used-car market in memory. a multi-period dataset with a month index would separate the car from the market and make the autotrader comparison clean.
• missingness. complete-case filtering cost 60k rows; condition, cylinders and drive missingness is unlikely to be random. explicit "not stated" levels or imputation would test how much that choice matters.
• region. the state column sat unused; rust-belt and sun-belt cars price differently, and a 4wd truck is worth more in montana than in florida (an interaction, which is the next tool in the box).

reproduce it

everything on this page — every number, table and figure, in both color schemes — is generated by one pipeline in the repository:

data/raw/vehicles.csv        # kaggle export (1.4 gb, see readme for link)
analysis/clean.py            # the attrition table above
analysis/model.py            # fits, box-cox, nested tests, vif, robust inference
analysis/diagnostics.py      # residual and influence figures (run model.py first)
analysis/validate.py         # held-out metrics, cv, interval coverage, the six cars
analysis/eda.py, results_figs.py, style.py
python analysis/run_all.py   # raw csv in, report assets out

the cleaned modeling table ships with the repo as data/cars_clean.parquet, so everything after the cleaning stage can be reproduced without the 1.4 gb download.

references

• austin reese, used cars dataset (craigslist listings, kaggle)
• course material from stats 3a03 (applied regression) and math 3mb3 (mathematical modelling), mcmaster university
• box & cox (1964), "an analysis of transformations"; duan (1983), "smearing estimate: a nonparametric retransformation method"
• seabold & perktold, statsmodels; hunter, matplotlib; mckinney, pandas
• the 2025 version of this project, kept as-is: it's the "before" picture