Dubai Land Department · machine learning platform

Dubimator

Price estimation, duplicate & fraud detection, property search and price forecasting over 1,047,965 real Dubai Land Department transactions, served through one API and a Streamlit demo. Eleven phases, each with its own spec, plan, tests and review — all eleven shipped.

as of 17 Sep 2026 174 commits 1007 tests estimates as of Q1 2023 (data ends 2023-03-17) runs on Docker + one RTX 3060

Build status

All eleven phases shipped.

shipped in progress planned

How data moves

One Postgres database holds everything. Real transactions feed the price model; the listings corpus is generated on top of those same sales, because DLD publishes no photos or descriptions.

Source
Transactions.csv
637 MB · 1,047,965 rows · 1995–2023
Houses-dataset photos
2,140 images · 535 × 4 rooms
Ingest
python -m ingestion
validate · normalise · classify · 29 s
schema dld
707,655 market sales · 253 areas
Model
Price model
XGBoost on GPU · Optuna × 60 · 6.1 min
MLflow registry
dubimator-price v2 @champion
Listings
listings build
20,000 synthetic listings · labelled · 56 s
listings embed
CLIP 512-d · MiniLM 384-d · GPU · 227 s
listings detect
pgvector HNSW · 3 retrieval channels · 12 signals
Fraud flags
bait price · photo reuse · inconsistent relist
Serve
python -m search
parse · pgvector + full text · 25 features · LightGBM ranker
python -m models.forecast
3m served · 1y / 3y not deployed · GPU · 5 min
python -m api serve
keys · rate limits · /metrics · warm 3–60 ms
python -m demo
Streamlit · 5 pages · key held server-side
Cloudflare Tunnel · Prometheus · Grafana
demo only · API 3.41 GB CPU image · verified live

The eleven phases

Order matters here: each phase consumes what the one before it produced.

01
Foundation14 Sep 2026
Docker Compose stack: Postgres 16 + pgvector, MLflow, Airflow. uv, pytest, smoke tests.
shipped
02
DLD ingestion15 Sep 2026
Every row loaded and kept, each non-sale carrying exactly one exclusion reason. Area alias table maps “JVC” and “Dubai Marina” to official names. Runs from the CLI or an Airflow trigger.
shipped
03
Price model16 Sep 2026
Homes sold 2015 on, priced relative to a leak-free market index with building → project → area → city fallbacks. Beat a comps baseline by 27% to earn registration.
shipped
04
Duplicate & fraud16 Sep 2026
Synthetic listings over real sales, with planted reposts and two kinds of listing that must not be flagged. Candidates come from pgvector, decisions from twelve signals, fraud flags from the price model and the photo graph.
shipped
05
Search ranking16 Sep 2026
Natural-language property search — “2BR in Dubai Marina under 1.5M”. A rule parser, semantic and full-text retrieval fused by rank, one listing per duplicate cluster, and a LightGBM ranker that had to beat a hand-written rule scorer to earn registration.
shipped
06
Price forecasting17 Sep 2026
Separate 3-month, 1-year and 3-year growth models, walk-forward validated with a purge gap, conformal ranges and 15 sourced infrastructure projects. Only the 3-month horizon passed its gate; 1-year failed on the 2022–23 surge and 3-year lacks data — both are served as “not deployed”, with the reason.
shipped
07
API17 Sep 2026
One FastAPI service: price, forecast, search, stored listing flags, on-the-fly checks of new listings and area statistics. API keys, per-key rate limits, JSON logs, request IDs, Prometheus metrics, lock-safe reload and database reconnects.
shipped
08
Demo UI17 Sep 2026
Streamlit, talking only to the API: price & forecast, plain-English search, listing checks and an area explorer. No map — DLD has no coordinates.
shipped
09
CI/CD17 Sep 2026
GitHub Actions for lint, database tests and image builds (ready for the first push), a local runner with the same steps, and python -m pipelines retrain for scheduled, gated retraining.
shipped
10
Deploy & monitor17 Sep 2026
CPU-only API and demo images, a public link through a free Cloudflare Tunnel (demo only, on an isolated network), Prometheus + Grafana, and a PSI/KS drift report. A landing page ships to Vercel from git, and deploy/ holds the kit that moves the stack to one server behind HTTPS.
shipped
11
Documentation17 Sep 2026
README front page with results at a glance, quickstart and honest limitations; this page; and the design-decisions casebook.
shipped

What the numbers say

The data phase 02

Transactions loaded1,047,965
Clean market sales707,655
Mortgages set aside224,183
Gifts set aside35,890
Non-market procedures57,241
Price outliers flagged18,936
Sizes likely in sq ft2,808

Nothing is deleted. Every excluded row keeps its reason, so the filter can be audited or reversed.

The price model phase 03

ModelMdAPE±10%±20%
Area comps rule17.3%32.3%55.1%
LightGBM13.0%39.6%67.3%
XGBoost (champion)12.6%41.2%69.1%

Measured on sales from Nov 2022 – Mar 2023, never used for fitting or tuning. Median error is 12.6%: half of estimates land within that of the real sale price.

Listings corpus phase 04

Listings generated20,000
Exact reposts planted1,200
Reworded copies900
Edited-photo copies900
Bait-priced listings600
Pairs if compared naively200M
Candidate pairs scored213,244
Planted duplicates retrieved (pooled)97.1%
Text index vs exact scan7.9 s vs 156 s

Locations, sizes and prices are real DLD sales. Text, agents, dates and every duplicate are invented and labelled — stated plainly wherever results appear.

Duplicate & fraud results phase 04

Held-out listingsModelPhoto rule
Precision (target 98%)98.3%0.6%
Recall21.1%91.0%
Same-building pairs flagged0.0%48.9%
Stock-photo pairs flagged0.0%99.7%
Bait price · precision / recall45.2% / 96.5%
Inconsistent relist · precision / recall90.8% / 56.2%

The model almost never flags an innocent listing but finds only one planted duplicate in five (20.4% end to end). Bait-price results are in-sample for the price model, so they are optimistic.

Search ranking results phase 05

Held-out queries · orderingNDCG@1095% CITop-10 fraud share
LightGBM ranker (registered v2)0.9970.995–0.9990.96%
Hand-written rules on the parsed query0.9720.968–0.9763.95%
Retrieval order (semantic + full text)0.5660.549–0.5833.13%
Learned lift over the rules (paired CI)+0.025 (0.021–0.029)
Perfect matches among the 200 candidates52.5%
Parser accuracy, weakest slot (building)99.6%
Engine start-up · per search12–13 s · 19–94 ms

6,000 synthetic queries, graded 0–3 by rule. The ranker mostly relearns those rules, so the honest comparison is against the rule scorer, not retrieval order. NDCG re-ranks within the retrieved candidates.

API smoke run phase 07

EndpointStatusFirst callWarm
POST /v1/price200141 ms18 ms
POST /v1/forecast20042 ms35 ms
GET /v1/search200533 ms60 ms
GET /v1/listings/{id}/flags200160 ms7 ms
POST /v1/listings/check2003.3 s61 ms
GET /v1/areas · history2009 ms4–5 ms
Components loadedprice · forecast-3m · search · listings · areas
Duplicate pair modelregistered v1 · threshold 0.9995
Demo pages, live against the API6 of 6 · no errors
Deployment checkcontainers healthy · all endpoints 200 · tunnel reached

Measured in-process on 17 Sep 2026 with every champion loaded. The first listing check pays for loading the text model; after that each check takes about 60 ms.

Price forecasting results phase 06

Horizon · test periodModel MAPENo changeArea trend80% coverageGate
3 months · Sep–Nov 2022 (13,848 sales)8.22%8.54%12.72%79.9%passed · v1
1 year · Sep 2021–Feb 2022 (19,728 sales)18.51%12.70%26.28%55.9%failed
3 years————not enough data
3m primary segments · resale / top-5 areas / older buildings9.82% / 7.21% / 10.41%
Sales kept after cleaning336,587 of 366,132
Walk-forward folds · 3m19 score · 4 tune · 2 gap · 1 test
Training time, all horizons (RTX 3060)5 min 23 s

The 3-month champion stopped after 3 trees: in every tuning fold the trees overfit from the first round, so it forecasts roughly the average growth (about +1.4% a quarter) and beats “no change” only because prices were rising. A trailing-average baseline in the gate is the obvious next test. Every number comes from MLflow run af509861.

What lives where

One module per job, the same shape in every phase: pure functions first, database and model code at the edges.

ingestion/read the DLD file, normalise it, classify every row, load Postgres models/price/features, market index, location priors, training, the DB-free predictor listings/photo pool, corpus generator, embeddings, pgvector retrieval, detection, fraud flags, evaluation, python -m listings models/forecast/rows, exact-window targets, as-of features, infrastructure table, walk-forward folds, gates, the Forecaster, python -m models.forecast search/query parser, synthetic graded queries, retrieval, features, rankers, evaluation, the search engine, python -m search dags/Airflow: the ingestion pipeline, triggered by hand api/FastAPI service: auth, rate limits, logs, metrics, every model's routes, python -m api demo/Streamlit demo over the API, python -m demo pipelines/gated retraining, stage by stage, python -m pipelines retrain monitoring/drift report, Prometheus and Grafana config .github/CI (lint, tests, image builds), release and Dependabot tests/1007 tests, including one per non-negotiable edge case docs/a spec and an implementation plan for every phase, plus this page

Stack and cost

Everything runs locally on free, open tools. Hosting stays free too: the public link in phase 10 is a Cloudflare Tunnel to this machine.

ComponentVersionRoleCost
PostgreSQL + pgvector16 · 0.8.6transactions, listings, vector search$0
MLflow2.17.2experiments, metrics, model registry$0
Airflow2.10.3pipeline orchestration$0
XGBoost / LightGBM3.2 · 4.7price model, search rankers$0
PyTorch + sentence-transformers2.9.1+cu128CLIP and text embeddings on the GPU$0
Polars1.44all data wrangling$0
FastAPI + uvicorn0.141the unified API$0
Streamlit1.64the demo$0
Prometheus + Grafana—metrics and dashboards$0
Cloudflare Tunnelquick tunnelpublic link while the PC is on$0
Vercelhobbylanding page + this page, built from git$0
Server (optional)4 vCPU / 8 GBalways-on API, demo, Postgres, MLflow~$9/mo

Build log

Phases 7–11 shipped overnight. Built in parallel, with one review per phase. The API review found connections that a reload could close mid-request and no recovery after a database restart; both are fixed. The demo review caught two forms that sent invalid requests, which the tests missed because the fake client ignored request bodies. Postgres restarted twice under the combined load of image builds and database tests, so heavy work now runs one job at a time. The containerised API first failed to load any model, because models saved on Windows record their file paths with backslashes; a one-line path fix repaired it. After that, every endpoint returned 200 from the containers, Prometheus scraped all five components, and a quick Cloudflare tunnel served the demo publicly before being stopped again.

Phase 6 shipped. Three forecasting horizons trained in about five minutes on the GPU. The whole-phase review found no leakage but four real problems, all fixed before the final run: a failed retrain could leave an old model live, tuning could see the test period's future prices, plot-priced villas (42% of villa sales) were mixed with built-up ones, and the test window could shrink to a sliver. Only the 3-month horizon passed, and the README says plainly that it behaves like an average-growth rule. Work now runs in parallel, with one review per phase.

Phase 5 shipped. Search over the 20,000 listings, trained and scored on 6,000 rule-graded queries in about 30 minutes. The final review showed a 20-line rule scorer already reaches 0.972 NDCG, so the ranker now has to beat that to be registered (it does, by 0.025). It also fixed an engine that stayed mid-transaction, ordinary words read as building names, and missing community names like “Arabian Ranches 2”. Price forecasting was added to the roadmap as Phase 6.

Phase 4 shipped. 20,000 listings embedded on the GPU and checked in about 21 minutes end to end. The final review found a relist flag that could never fire and same-building controls easier than the spec asked for; both were fixed and the corpus rebuilt. Precision now meets the 98% target; recall fell to 21% and the README says so. Hosting switched from Cloud Run to a free Cloudflare Tunnel.

Phase 4 under way. Photo pool, corpus generator and the listings schema are in. Reviews caught four ground-truth flaws before they could flatter the results — bait prices computed off the wrong base, control groups collapsing into one district, stock photos indistinguishable from ordinary ones, and two columns that gave the answer away for free.

Phase 3 shipped. Price model registered as v2. Beats the comps rule by 27% and LightGBM by 0.4 points. Its 80% price ranges are calibrated on the test period, and the README says plainly where they under-cover.

Phase 2 shipped. The full DLD file loads in 29 seconds, atomically. MLflow's store moved off the OneDrive-synced folder after intermittent 15–20 second stalls.

Phase 1 shipped. Docker Compose stack up, with smoke tests that fail loudly rather than skip when a service is misconfigured.