README

IE7374-MachineLearningOperations · Dec 9, 2024 · 37f8975 · 37f8975
1 parent 257dc95
commit 37f8975
Show file tree

Hide file tree

Showing 10 changed files with 717 additions and 269 deletions.
diff --git a/Assignments_Submissions/Model Deployment/README.md b/Assignments_Submissions/Model Deployment/README.md
@@ -18,7 +18,7 @@ Steps for Replication
     - A GitHub repository with access to GitHub Actions for automation.
     - Required IAM roles for deploying models to Vertex AI and managing Cloud Build resources.
 
-![GCP Billing Dashboard](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/GCP%20billing%20dashboard.png)
+![GCP Billing Dashboard](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/GCP%20billing%20dashboard.png)
 
 #### 2. **Running Deployment Automation**
    - Push changes to the main branch of the GitHub repository.
@@ -31,7 +31,7 @@ Steps for Replication
    - Test the deployed model endpoint to confirm successful deployment and validate model predictions.
    - Review monitoring dashboards to ensure no issues with prediction outputs or feature drift.
 
-![Drift Detection Logging](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Drift%20Detection%20logging.png)
+![Drift Detection Logging](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Drift%20Detection%20logging.png)
 
 ---
 
@@ -57,25 +57,25 @@ Workflows and setups for managing machine learning pipelines on Vertex AI in Goo
 ### Steps for Deployment of Trained Models
 1. **Model Registration**: Once a model is trained, register it in Vertex AI's Model Registry. Specify the model name, version, and any relevant metadata.
 
-![Vertex AI Jupyter Notebooks](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Vertex%20Ai%20jupyter%20notebooks.png)
+![Vertex AI Jupyter Notebooks](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Vertex%20Ai%20jupyter%20notebooks.png)
 
-![Model Serving](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Model%20serving.png)
+![Model Serving](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Model%20serving.png)
 
-![Vertex AI Model Registry](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Vertex%20Ai%20model%20registry.png)
+![Vertex AI Model Registry](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Vertex%20Ai%20model%20registry.png)
 
 2. **Create an Endpoint**: 
    - In Vertex AI, create an endpoint. This endpoint will act as the interface for serving predictions.
    - Navigate to Vertex AI > Online prediction > Endpoints > Create.
    - Assign a name and select the appropriate region.
 
-![Vertex AI Endpoints](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Vertex%20Ai%20endpoints.png)
+![Vertex AI Endpoints](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Vertex%20Ai%20endpoints.png)
 
 3. **Deploy the Model to an Endpoint**:
    - Select the registered model and choose "Deploy to Endpoint".
    - Configure the deployment settings such as machine type, traffic splitting among model versions, and whether to enable logging or monitoring.
    - Confirm deployment which will make the model ready to serve predictions.
 
-![Vertex AI Model Development Training](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Vertex%20Ai%20model%20development%20training.png)
+![Vertex AI Model Development Training](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Vertex%20Ai%20model%20development%20training.png)
 
 ### Model Versioning
 - **Manage Versions**: In Vertex AI, each model can have multiple versions allowing easy rollback and version comparison.
@@ -88,7 +88,7 @@ Workflows and setups for managing machine learning pipelines on Vertex AI in Goo
    - **GitHub Actions**: Configure workflows in `.github/workflows/` directory to automate testing, building, and deploying models.
    - **Cloud Build**: Create a `cloudbuild.yaml` file specifying steps to build, test, and deploy models based on changes in the repository.
 
-![GitHub Actions CI/CD](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Github%20Actions%20CICD.png)
+![GitHub Actions CI/CD](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Github%20Actions%20CICD.png)
 
 ---
 
@@ -147,7 +147,7 @@ Workflows and setups for managing machine learning pipelines on Vertex AI in Goo
   - **Verification**: Lists uploaded files to confirm the sync.
 ---
 
-![GitHub Actions](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Github%20Workflows.png)
+![GitHub Actions](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Github%20Workflows.png)
 
 
 #### Summary
@@ -167,25 +167,24 @@ Each workflow is tailored for a specific task in CI/CD for ML operations, levera
    - Vertex AI provides dashboards to monitor model performance and data drift.
    - Alerts are configured to notify stakeholders when anomalies, such as feature attribution drift, are detected.
 
-![Model Monitoring Notification](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Model%20Monitoring%20notification.png)
+![Model Monitoring Notification](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Model%20Monitoring%20notification.png)
 
 
-![Model Monitoring Anomalies](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Model%20monitoring%20Anomolies.png)
+![Model Monitoring Anomalies](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Model%20monitoring%20Anomolies.png)
 
 The provided images highlight the active setup and management of a Vertex AI model monitoring system. Files like `anomalies.json` and `anomalies.textproto` document identified issues in the input data. The structure also includes folders such as `baseline`, `logs`, and `metrics`, which organize monitoring data effectively for future analysis. A notification email confirming the creation of a model monitoring job for a specific Vertex AI endpoint. This email provides essential details, such as the endpoint name, monitoring job link, and the GCS bucket path where statistics and anomalies will be saved. 
 
 2. **Maintenance**:
    - Pre-configured thresholds for model performance trigger retraining or redeployment of updated models.
    - Logs and alerts from Vertex AI and Cloud Build ensure the system remains reliable and scalable.
 
-![Monitor Details](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Monitor%20details.png)
+![Monitor Details](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Monitor%20details.png)
 
-![Logging Dashboard](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Logging%20Dashboard.png)
+![Logging Dashboard](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Logging%20Dashboard.png)
 
-![Monitor Feature Detection](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Monitor%20feature%20detection.png)
-
-![Monitor Drift Detection](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/104a48ddf826520ccc31374002d8df92f2015796/assets/Monitor%20drift%20detection.png)
+![Monitor Feature Detection](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Monitor%20feature%20detection.png)
 
+![Monitor Drift Detection](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Monitor%20drift%20detection.png)
 
 ---
 

diff --git a/Assignments_Submissions/Model development pipeline phase/README.md b/Assignments_Submissions/Model development pipeline phase/README.md
@@ -227,9 +227,46 @@ Model validation is a crucial step to evaluate how well the selected model perfo
 ### 4. Model Bias Detection (Using Slicing Techniques)
 Bias detection ensures that the model behaves equitably across different subgroups of data.
 
+> Please refer to [Model Bias Notebook](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/c50d70f57592fa1ca139141ce09fe82099e7ea1b/src/FeatureEng_and_ModelBiasDetn.ipynb)
+
 - **Tools Used**: `Fairlearn` for data slicing and detecting model bias.
 - **Purpose**: To evaluate model fairness across various demographic or sensitive features.
 
+  - **Metrics by Slice**:
+    - High VIX Slice: Mean Squared Error (MSE): 2.04 , Mean Absolute Error (MAE): 1.14
+    - Low VIX Slice: Mean Squared Error (MSE): 1.34 , Mean Absolute Error (MAE): 0.87
+  - **Disparity Analysis**: Overall MSE Disparity: 0.70 , Overall MAE Disparity: 0.27
+
+- **ElasticNet Model Performance Summary**:
+
+  - **ElasticNet Test MSE with Resampling**: 1.37
+  - **ElasticNet Test MAE with Resampling**: 0.91
+
+  - **1. Dataset Slicing and Bias Detection**:
+    - **Slices Defined**: We defined slices based on SP500 and VIX values (high and low conditions) to evaluate model performance under different market scenarios.
+    - **Metrics Tracked**: For each slice, we tracked Mean Squared Error (MSE) and Mean Absolute Error (MAE) to assess prediction accuracy.
+    - **Results**: Initial analysis showed disparities in MSE and MAE between slices, with higher errors in some slices (e.g., high SP500), suggesting potential bias.
+
+  - **2. Bias Mitigation Technique Applied**:
+    - **Resampling**: To mitigate bias, we applied resampling to balance the training dataset, focusing on slices with less representation or higher error.
+    - **Model Re-evaluation**: After re-training the ElasticNet model on the resampled data, we saw improvements in MSE and MAE across slices, which indicated reduced disparity.
+
+  - **3. Performance Improvement After Mitigation**:
+    - **Metrics After Resampling**:
+      - **Test MSE**: Improved to 1.37.
+      - **Test MAE**: Improved to 0.91.
+    - **Disparity Reduction**: The disparity between slices decreased, showing that the model predictions were more consistent across different market conditions.
+
+  - **4. Considerations and Trade-offs**:
+    - **Trade-offs**: Resampling helped improve fairness across slices, but it may have slightly affected the model's predictive power for over-represented groups. However, our main goal was to ensure balanced fairness across conditions to minimize bias.
+    - **Future Improvements**: We could further enhance model fairness by applying additional techniques like re-weighting or fine-tuning model hyperparameters.
+
+- **Observations**:
+  - These metrics show how well the model is doing on the test set after rebalancing the training data to ensure fairness across slices defined by VIX values.
+  - The low MSE and MAE values suggest that the model's predictions are pretty close to the actual values, which means we've likely improved both fairness and overall accuracy.
+  - The high VIX slice has a higher error (MSE of 2.04) compared to the low VIX slice (MSE of 1.34). This suggests that the model struggles a bit more with accuracy when market volatility (VIX) is high.
+  - The disparity in both MSE and MAE points to a noticeable performance gap between these slices, with the model performing better under stable (low VIX) conditions.
+
 **Airflow Graph Depicting Bias Detection Task**:
 
 ![Airflow Graph](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v1.0/assets/airflow_graph.png)
@@ -275,7 +312,7 @@ Sensitivity analysis helps understand how changes in input features and hyperpar
 
 - **Hyperparameter Sensitivity Analysis**:
 
-  ![Hyperparameter Sensitivity](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v1.0/assets/results_linear_regression .png)
+  ![Hyperparameter Sensitivity](https://github.com/IE7374-MachineLearningOperations/StockPricePrediction/blob/v2.1/assets/Linear%20Regression%20-%20Hyperparameter%20Sensitivity_%20model__alpha.png.png)
 
 ### 8. Experiment Tracking and Results with Weights & Biases