7006SCN2627SEPNOV - Machine Learning and Big Data

Student Information

Student Name: Coventry Student / Academic User

Student ID: 7006SCN_Regression_Week3_Repo

Module: 7006SCN - Machine Learning and Big Data

Module Leader: Dr Katerina Stamou

Assessment: Applied Core Assessment -- Task 2 Sub-Task Commit Architecture

Dataset: Kaggle NYC Taxi Trip Duration Regression

Dataset Source: Kaggle / OpenML Dataset Repository

Direct URL: https://www.kaggle.com/c/nyc-taxi-trip-duration

Notes / Access Caveats

Over 14,586,442 records and 11 continuous/categorical columns. Continuous target variable: trip_duration (trip duration prediction in seconds).

GitHub repository URL: https://github.coventry.ac.uk/7006SCN-2627SEPNOV/7006SCN_Regression_Week3_Repo/tree/Task2

Tableau Public URL: https://public.tableau.com/views/7006SCN_Taxi_Regression_Analytics/Dashboard1_DataQuality

Task 2 Sub-Tasks & Commit History (Days 05-06 | AnswerSheets Aligned)

The Task2 branch implements strictly Task 2 sub-tasks across Days 05-06, with 3 commits per day mapped to the official AnswerSheet specification, considering Task1-metadata as input.

Day	Sub-Task Reference	Commit Hash	Commit Message / Sub-Task Objective	AnswerSheet Section
Day 06	Sub-Task 2.6	98bca01	Day06 - Sub-Task 2.6: Model Selection, Composite Evidence Pack & Airflow Export -- Final Model Recommendation & Task2 Metadata (AnswerSheet 2.3 & EP2)	AnswerSheet 2.6
Day 06	Sub-Task 2.5	94df4dc	Day06 - Sub-Task 2.5: Multi-Metric Model Comparison & Residual Evaluation -- RMSE, MAE, R2 & Residual Error (AnswerSheet 2.3)	AnswerSheet 2.5
Day 06	Sub-Task 2.4	4f7379e	Day06 - Sub-Task 2.4: Scaled Hyperparameter Tuning & Model Performance Benchmarking -- ParamGrid Search & Training Timings (AnswerSheet 2.2)	AnswerSheet 2.4
Day 05	Sub-Task 2.3	e12dcc5	Day05 - Sub-Task 2.3: Distributed Model Cross-Validation -- 5-Fold CrossValidator & Metric Evaluation (AnswerSheet 2.2)	AnswerSheet 2.3
Day 05	Sub-Task 2.2	45ad94b	Day05 - Sub-Task 2.2: Ensemble & Advanced Regressors -- RandomForestRegressor & GBTRegressor Implementation (AnswerSheet 2.1)	AnswerSheet 2.2
Day 05	Sub-Task 2.1	a3cbbd0	Day05 - Sub-Task 2.1: Baseline Regression Models -- LinearRegression & DecisionTreeRegressor ParamGrid Setup (AnswerSheet 2.1)	AnswerSheet 2.1

Task 2 Deliverable Details

Task 2 -- Distributed Modeling, Evaluation & Deployment (Week 3 | 30% | Days 05-06)

Notebook: /notebooks/Task2.ipynb | Results Output: /results/task2.json
Deliverables:
1. Task1 Metadata Handoff Consumption: Loads task1_metadata.json via subprocess verification (verify_exists()) and configures SparkSession with driver memory (4g) and shuffle partitions (200).
2. ParamGrid GridSearch & 5-Fold CV: Evaluated Linear Regression, Decision Tree, Random Forest, and GBT Regressors.
3. Residual Analysis & Error Metrics: Computed RMSE, MAE, R2, and analyzed residual error homoscedasticity.
4. Production Export & Airflow MLOps: Exported champion GBT Regressor model artifacts (task2_metadata.json) and Airflow DAG pipeline specifications.

Directory Structure

7006SCN_Regression_Week3_Repo/
â”œâ”€â”€ Day05/
â”‚   â”œâ”€â”€ Day05_1_Baseline_Regression_Models.ipynb
â”‚   â”œâ”€â”€ Day05_2_Ensemble_Regression_Models.ipynb
â”‚   â”œâ”€â”€ Day05_3_Cross_Validation_Setup.ipynb
â”‚   â””â”€â”€ README.md
â”œâ”€â”€ Day06/
â”‚   â”œâ”€â”€ Day06_1_Hyperparameter_Tuning_Benchmarking.ipynb
â”‚   â”œâ”€â”€ Day06_2_MultiMetric_Model_Evaluation.ipynb
â”‚   â”œâ”€â”€ Day06_3_Composite_Evidence_Pack_Export.ipynb
â”‚   â””â”€â”€ README.md
â”œâ”€â”€ metadata/
â”‚   â”œâ”€â”€ task1_metadata.json
â”‚   â””â”€â”€ task2_metadata.json
â”œâ”€â”€ notebooks/
â”‚   â””â”€â”€ Task2.ipynb
â”œâ”€â”€ results/
â”‚   â”œâ”€â”€ task1.json
â”‚   â””â”€â”€ task2.json
â”œâ”€â”€ README.md
â””â”€â”€ requirements.txt

7006SCN-2627SEPNOV/7006SCN_Regression_Week3_Repo

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Languages