The problem with averages
Lenient judges inflate every project they touch; harsh judges sink theirs. A raw average confuses generosity with quality.
We calibrate per judge, per criterion: z = (s − μ_judge) / σ_judge, remapped into global scale as z × σ_global + μ_global. Constant scorers (σ = 0) fall back to raw and are flagged constant:true, excludable via /ranking?exclude_judge=.
Three more views, one verdict
Over the same scores the organizer endpoint computes Bradley-Terry latent strengths from per-judge pairwise comparisons (ties skipped, disconnected-graph warning) and Borda points with fractional ties.
The prize boundary (top-k, default 3) is ROBUST only when raw, calibrated, Bradley-Terry and Borda agree on the set above the line — otherwise FRAGILE with the disagreeing methods named. Both seeded demos flip: Food Rescue Route raw #2 → calibrated #5, Mural Mile raw #2 → calibrated #3.
Why organizers trust it
Math alone does not convince anyone. The progress dashboard, raw-vs-calibrated CSV columns, low_confidence flags on thinly reviewed projects and the duplicates list (second Solar Kiosk counted once) make the calibration inspectable.
Details and formulas: JUDGING.md. Reproduce: python demo-5min/seed_judging_demo.py and seed_grand_jury.py.
Run it yourself
Every post maps to a real route or file in this repo. Start with the gallery, then the docs.
Keep reading
View allOct 3, 2026 · Engineering
Run the whole platform with one command: setup guide (main branch)
Clone main, docker compose up, seed fixtures, test accounts, and the 7/7 acceptance checker — everything to go from zero to judging in minutes.
Sep 25, 2026 · Engineering
Judge isolation belongs in the backend, not the frontend
The #1 checker failure point across teams: judges must only see their own scores. How require_role deny-by-default plus 403s passes 7/7.
Oct 3, 2026 · Company
Build the platform that will judge you: how we built DOGFOOD 2026
FastAPI + Next.js + Postgres with backend-enforced judge isolation, z-score calibration and one-command Docker — our Write-Up Quest build log.