Japan Real Estate Clustering Project (8th Copy)

Research Question

Can property transaction clusters identify real estate market segments in Japan?

Business Framing

Real estate investment strategy -- using unsupervised clustering to discover distinct market segments within Japan's property transaction data spanning 2005 to 2024 Q3.

Dataset

Source: MLIT Japan Real Estate Price 2005-2024 Q3
Size: 5.9 million rows, 30 columns (~2.05 GB)
Content: Japanese property transaction records including price, area, building details, location, and zoning information

Algorithms

Algorithm	Framework	Type
KMeans	MLlib (PySpark)	Distributed
BisectingKMeans	MLlib (PySpark)	Distributed
GaussianMixture	MLlib (PySpark)	Distributed
HDBSCAN	cuML (GPU)	Density-based
AgglomerativeClustering	cuML (GPU)	Hierarchical
KMeans	sklearn (CPU)	Baseline

Custom Transformer

PricePerAreaCalc -- computes derived property metrics:

price_per_sqm: Total transaction value / Area (sqm)
age_of_building: Current year - Construction year
distance_to_station_num: Numeric station distance
floor_ratio: Floor area ratio (cleaned)

Project Structure

8th_copy/
  notebooks/
    1_data_ingestion.ipynb       -- Load from HuggingFace, validate, partition
    2_feature_engineering.ipynb   -- Clean, encode, transform, PCA
    3_model_training.ipynb        -- Train all 6 algorithms, CrossValidator
    4_evaluation.ipynb            -- Metrics, bootstrap CI, stability, Tableau export
  scripts/
    setup_environment.sh          -- Install all dependencies
    run_pipeline.py               -- Orchestrate full pipeline
    performance_profiler.py       -- CPU vs GPU benchmarks
  config/
    spark_config.yaml             -- Spark and pipeline configuration
    tableau_config.json           -- Tableau dashboard specifications
  data/
    schemas/                      -- Data schema definitions
    samples/                      -- Small data samples
  tests/
    test_pipeline.py              -- Unit and integration tests
  tableau/
    README_tableau.md             -- Tableau integration guide
  .gitignore
  environment.yml                 -- Conda environment specification
  Dockerfile                      -- Container with GPU support
  README.md                       -- This file

Environment

Runtime: Google Colab Pro with A100 GPU
PySpark: 3.5.8
cuML: 26.02
cuDF: 26.02
Python: 3.10

Quick Start (Google Colab)

Upload the notebooks to Google Colab
Select A100 GPU runtime
Run notebooks in order: 1 -> 2 -> 3 -> 4

Quick Start (Local with Docker)

docker build -t japan-real-estate-clustering .
docker run --gpus all -p 8888:8888 japan-real-estate-clustering

Quick Start (Pipeline Script)

bash scripts/setup_environment.sh
python scripts/run_pipeline.py --stage all

Key Design Decisions

500K sample for GPU operations: Full 5.9M rows for MLlib (distributed), 500K sample for cuML GPU algorithms (single-node GPU memory constraint)
50K sample for AgglomerativeClustering: O(n^2) memory requirement
PCA on cuML for speed comparison: Demonstrates GPU acceleration for dimensionality reduction
k range 3-8: Broad sweep to find optimal market segmentation granularity
HDBSCAN min_cluster_size [50, 100, 250]: Tests sensitivity to density threshold
Bootstrap 100 iterations: Sufficient for stable 95% confidence intervals

Evaluation Metrics

Silhouette Score (primary)
Calinski-Harabasz Index
Davies-Bouldin Index
Adjusted Rand Index (stability)
Bootstrap 95% Confidence Intervals

Sirikondau/7006SCN-Code

About

Resources

Stars

Watchers

Forks

Releases

Languages