|
| 1 | +--- |
| 2 | +challenge: |
| 3 | + module: Convert a notebook to production code |
| 4 | + challenge: '0: Convert a notebook to production code' |
| 5 | +--- |
| 6 | + |
| 7 | +<style> |
| 8 | +.button { |
| 9 | + border: none; |
| 10 | + color: white; |
| 11 | + padding: 12px 28px; |
| 12 | + background-color: #4285F4; |
| 13 | + float: right; |
| 14 | +} |
| 15 | +</style> |
| 16 | + |
| 17 | +# Challenge 0: Convert a notebook to production code |
| 18 | + |
| 19 | +<button class="button" onclick="window.location.href='https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform';">Back to overview</button> |
| 20 | + |
| 21 | +## Challenge scenario |
| 22 | + |
| 23 | +The first step to automate machine learning workflows is to convert a Jupyter notebook to production-ready code. When you store your code as scripts, it's easier to automate the code execution. You can parameterize scripts to easily reuse the code for retraining with Google Cloud Vertex AI. |
| 24 | + |
| 25 | +## Prerequisites |
| 26 | + |
| 27 | +To complete this challenge, you'll need: |
| 28 | + |
| 29 | +- Access to a Google Cloud Platform (GCP) account with appropriate permissions. |
| 30 | +- A GitHub account. |
| 31 | +- Basic familiarity with Vertex AI and Google Cloud services. |
| 32 | +- Google Cloud SDK (gcloud) installed and configured. |
| 33 | + |
| 34 | +## Objectives |
| 35 | + |
| 36 | +By completing this challenge, you'll learn how to: |
| 37 | + |
| 38 | +- Clean nonessential code. |
| 39 | +- Convert your code to Python scripts compatible with Vertex AI. |
| 40 | +- Use functions in your scripts. |
| 41 | +- Use parameters in your scripts. |
| 42 | +- Implement experiment tracking with Vertex AI Experiments. |
| 43 | + |
| 44 | +> **Important!** |
| 45 | +> Each challenge is designed to allow you to explore how to implement DevOps principles when working with machine learning models on Google Cloud Platform. Some instructions may be intentionally vague, inviting you to think about your own preferred approach. If for example, the instructions ask you to create a Vertex AI Workbench instance or enable Vertex AI APIs, it's up to you to explore and decide how you want to create it. To make it the best learning experience for you, it's up to you to make it as simple or as challenging as you want. |
| 46 | +
|
| 47 | +## Challenge Duration |
| 48 | + |
| 49 | +- **Estimated Time**: 30 minutes |
| 50 | + |
| 51 | +## Instructions |
| 52 | + |
| 53 | +To work through the challenges, you need **your own public repo** which includes the challenge files. Create a new public repo by navigating to [https://github.com/GoogleCloudPlatform/vertex-ai-samples](https://github.com/GoogleCloudPlatform/vertex-ai-samples) and fork or use as a template to create your own repo. |
| 54 | + |
| 55 | +In the **experimentation** folder, you'll find a Jupyter notebook that trains a classification model. The data used by the notebook is in the **experimentation/data** folder and contains a CSV file. |
| 56 | + |
| 57 | +In the **src/model** folder you'll find a `train.py` script which already includes code converted from part of the notebook. It's up to you to complete it for Vertex AI compatibility. |
| 58 | + |
| 59 | +- Go through the notebook to understand what the code does. |
| 60 | +- Convert the code under the **Split data** header and include it in the `train.py` script as a `split_data` function. Remember to: |
| 61 | + - Remove nonessential code. |
| 62 | + - Include the necessary code as a function. |
| 63 | + - Include any necessary libraries at the top of the script. |
| 64 | + - Ensure compatibility with Vertex AI training environment. |
| 65 | + |
| 66 | +<details> |
| 67 | +<summary>Hint</summary> |
| 68 | +<br/> |
| 69 | +The <code>split_data</code> function is already included in the main function. You only need to add the function itself with the required inputs and outputs underneath the comment <code>TO DO: add function to split data</code>. Make sure to handle Vertex AI's expected input/output paths using environment variables like <code>AIP_MODEL_DIR</code> and <code>AIP_TRAINING_DATA_URI</code>. |
| 70 | +</details> |
| 71 | + |
| 72 | +- Add experiment tracking so that every time you run the script, all parameters and metrics are tracked. Use Vertex AI Experiments to track your training runs, or alternatively, integrate with Vertex AI's managed MLflow to ensure the necessary model files are stored with the job run for easy deployment. |
| 73 | + |
| 74 | +<details> |
| 75 | +<summary>Hint</summary> |
| 76 | +<br/> |
| 77 | +Vertex AI provides native experiment tracking capabilities through Vertex AI Experiments. You can also use the managed MLflow service on Vertex AI for experiment tracking. For Vertex AI Experiments, use the Vertex AI SDK to create and track experiments. For MLflow integration, you can use <code>mlflow.autolog()</code> with Vertex AI's managed MLflow tracking server. Enable experiment tracking in the main function under <code>TO DO: enable experiment tracking</code>. |
| 78 | +</details> |
| 79 | + |
| 80 | +- Ensure your script is compatible with Vertex AI custom training jobs by: |
| 81 | + - Using Vertex AI environment variables for input/output paths |
| 82 | + - Saving the model to the correct output directory (using `AIP_MODEL_DIR`) |
| 83 | + - Adding proper argument parsing for hyperparameters |
| 84 | + - Integrating with Google Cloud Storage for data access |
| 85 | + |
| 86 | +<details> |
| 87 | +<summary>Hint</summary> |
| 88 | +<br/> |
| 89 | +Vertex AI provides specific environment variables like <code>AIP_MODEL_DIR</code>, <code>AIP_TRAINING_DATA_URI</code>, and <code>AIP_VALIDATION_DATA_URI</code>. Use these to make your script portable across different Vertex AI environments. Also, implement argument parsing using <code>argparse</code> to handle hyperparameters passed from Vertex AI training jobs. Use the Google Cloud Storage client library to read training data from GCS buckets. |
| 90 | +</details> |
| 91 | + |
| 92 | +- Integrate with Google Cloud services: |
| 93 | + - Use Google Cloud Storage for data storage and model artifacts |
| 94 | + - Implement proper logging with Google Cloud Logging |
| 95 | + - Consider using Vertex AI Pipelines for workflow orchestration |
| 96 | + |
| 97 | +<details> |
| 98 | +<summary>Hint</summary> |
| 99 | +<br/> |
| 100 | +Import the necessary Google Cloud libraries: <code>google-cloud-storage</code> for GCS operations, <code>google-cloud-logging</code> for structured logging, and <code>google-cloud-aiplatform</code> for Vertex AI integration. Set up proper authentication using Application Default Credentials (ADC) or service account keys. |
| 101 | +</details> |
| 102 | + |
| 103 | +## Success criteria |
| 104 | + |
| 105 | +To complete this challenge successfully, you should be able to show: |
| 106 | + |
| 107 | +- A training script which includes a function to split the data and experiment tracking using Vertex AI Experiments or managed MLflow. |
| 108 | +- The script is compatible with Vertex AI custom training jobs (uses appropriate environment variables and paths). |
| 109 | +- Proper model serialization and saving to Google Cloud Storage. |
| 110 | +- Integration with Google Cloud services for logging and data management. |
| 111 | + |
| 112 | +> **Note:** |
| 113 | +> If you've used a Vertex AI Workbench instance or Colab Enterprise for experimentation, remember to stop the instance when you're done to avoid unnecessary charges. Also, clean up any Google Cloud Storage buckets and Vertex AI resources you created during testing. |
| 114 | +
|
| 115 | +## Useful resources |
| 116 | + |
| 117 | +- [Vertex AI Custom Training Documentation](https://cloud.google.com/vertex-ai/docs/training/custom-training) |
| 118 | +- [Vertex AI Experiments for Experiment Tracking](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments) |
| 119 | +- [Using MLflow with Vertex AI](https://cloud.google.com/vertex-ai/docs/experiments/vertex-ai-mlflow) |
| 120 | +- [Vertex AI Workbench User Guide](https://cloud.google.com/vertex-ai/docs/workbench) |
| 121 | +- [Google Cloud Storage Client Libraries](https://cloud.google.com/storage/docs/reference/libraries) |
| 122 | +- [Vertex AI Python SDK Documentation](https://cloud.google.com/python/docs/reference/aiplatform/latest) |
| 123 | +- [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) |
| 124 | +- [Google Cloud ML Engineering Best Practices](https://cloud.google.com/architecture/ml-on-gcp-best-practices) |
| 125 | + |
| 126 | +<button class="button" onclick="window.location.href='01-vertex-ai-job';">Continue with challenge 1</button> |
0 commit comments