Skip to content

Commit f8dfccc

Browse files
praneet22praneet22
committed
initial commit
0 parents  commit f8dfccc

38 files changed

+1976
-0
lines changed

‎Readme.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# DevOps For AI
2+
3+
DevOps for AI template will help you to understand how to build the Continuous Integration and Continuous Delivery pipeline for a ML/AI project. We will be using the Azure DevOps Project for build and release pipelines along with Azure ML services for ML/AI model management and operationalization.
4+
5+
This template contains code and pipeline definition for a machine learning project demonstrating how to automate the end to end ML/AI project. The build pipelines include DevOps tasks for data sanity test, unit test, model training on different compute targets, model version management, model evaluation/model selection, model deployment as realtime web service, staged deployment to QA/prod, integration testing and functional testing.
6+
7+
## Prerequisite
8+
- Active Azure subscription
9+
- Minimum contributor access to Azure subscription
10+
11+
## Getting Started:
12+
Once the template is imported for personal Azure DevOps account using DevOps demo generator, you need to follow below steps to get the pipeline running:
13+
14+
### Update Pipeline Config:
15+
16+
#### Build Pipeline
17+
1. Go to the **Pipelines -> Builds** on the newly created project and click **Edit** on top right
18+
![EditPipeline1](/docs/images/EditPipeline1.png)
19+
2. Click on **Create or Get Workspace** task, select the Azure subscription where you want to deploy and run the solution, and click **Authorize**
20+
![EditPipeline2](/docs/images/EditPipeline2.png)
21+
3. Click all other tasks below it and select the same subscription (no need to authorize again)
22+
4. Once the tasks are updated with subscription, click on **Save & queue** and select **Save**
23+
![EditPipeline3](/docs/images/EditPipeline3.png)
24+
25+
#### Release Pipeline
26+
1. Go to the **Pipelines -> Releases** and click **Edit** on top
27+
![EditPipeline4](/docs/images/EditPipeline4.png)
28+
2. Click on **1 job, 4 tasks** to open the tasks in **QA stage**
29+
![EditPipeline5](/docs/images/EditPipeline5.png)
30+
3. Update the subscription details in two tasks
31+
![EditPipeline6](/docs/images/EditPipeline6.png)
32+
4. Click on **Tasks** on the top to switch to the Prod stage, update the subscription details for the two tasks in prod
33+
![EditPipeline7](/docs/images/EditPipeline7.png)
34+
5. Once you fix all the missing subscription, the **Save** is no longer grayed, click on save to save the changes in release pepeline
35+
![EditPipeline8](/docs/images/EditPipeline8.png)
36+
37+
### Update Repo config:
38+
1. Go to the **Repos** on the newly created Azure DevOps project
39+
2. Open the config file [/aml_config/config.json](/aml_config/config.json) and edit it
40+
3. Put your Azure subscription ID in place of <>
41+
4. Change resource group and AML workspace name if you want
42+
5. Put the location where you want to deploy your Azure ML service workspace
43+
6. Save the changes and commit these changes to master branch
44+
7. The commit will trigger the build pipeline to run deploying AML end to end solution
45+
8. Go to **Pipelines -> Builds** to see the pipeline run
46+
47+
## Steps Performed in the Build Pipeline:
48+
49+
1. Prepare the python environment
50+
2. Get or Create the workspace
51+
3. Submit Training job on the remote DSVM / Local Python Env
52+
4. Register model to workspace
53+
5. Create Docker Image for Scoring Webservice
54+
6. Copy and Publish the Artifacts to Release Pipeline
55+
56+
## Steps Performed in the Release Pipeline
57+
In Release pipeline we deploy the image created from the build pipeline to Azure Container Instance and Azure Kubernetes Services
58+
59+
### Deploy on ACI - QA Stage
60+
1. Prepare the python environment
61+
2. Create ACI and Deploy webservice image created in Build Pipeline
62+
3. Test the scoring image
63+
64+
### Deploy on AKS - PreProd/Prod Stage
65+
1. Prepare the python environment
66+
2. Deploy on AKS
67+
- Create AKS and create a new webservice on AKS with the scoring docker image
68+
69+
OR
70+
71+
- Get the existing AKS and update the webservice with new image created in Build Pipeline
72+
3. Test the scoring image
73+
74+
### Repo Details
75+
76+
You can find the details of the code ans scripts in the repository [here](/docs/code_description.md)
77+
78+
### References
79+
80+
- [Azure Machine Learning(Azure ML) Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/overview-what-is-azure-ml)
81+
82+
- [Azure ML Samples](https://docs.microsoft.com/en-us/azure/machine-learning/service/samples-notebooks)
83+
- [Azure ML Python SDK Quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)
84+
- [Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/?view=vsts)

‎aml_config/conda_dependencies.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Conda environment specification. The dependencies defined in this file will
2+
3+
# be automatically provisioned for managed runs. These include runs against
4+
5+
# the localdocker, remotedocker, and cluster compute targets.
6+
7+
8+
# Note that this file is NOT used to automatically manage dependencies for the
9+
10+
# local compute target. To provision these dependencies locally, run:
11+
12+
# conda env update --file conda_dependencies.yml
13+
14+
15+
# Details about the Conda environment file format:
16+
17+
# https://conda.io/docs/using/envs.html#create-environment-file-by-hand
18+
19+
20+
# For managing Spark packages and configuration, see spark_dependencies.yml.
21+
22+
23+
# Version of this configuration file's structure and semantics in AzureML.
24+
25+
# This directive is stored in a comment to preserve the Conda file structure.
26+
27+
# [AzureMlVersion] = 2
28+
29+
30+
name: project_environment
31+
dependencies:
32+
# The python interpreter version.
33+
34+
# Currently Azure ML Workbench only supports 3.5.2 and later.
35+
36+
- python=3.6.2
37+
# Required by azureml-defaults, installed separately through Conda to
38+
39+
# get a prebuilt version and not require build tools for the install.
40+
41+
- psutil=5.3
42+
43+
- pip:
44+
# Required packages for AzureML execution, history, and data preparation.
45+
- azureml-sdk[notebooks]
46+
- pynacl==1.2.1
47+
- scipy==1.0.0
48+
- scikit-learn==0.19.1
49+
- pandas==0.23.1
50+
- numpy==1.14.5

‎aml_config/config.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"subscription_id": "<>",
3+
"resource_group": "DevOps_AzureML_Demo",
4+
"workspace_name": "AzureML_Demo_ws",
5+
"location": "southcentralus"
6+
}

‎aml_config/security_config.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"sp_user" : "<>",
3+
"sp_password" : "<>",
4+
"sp_tenant_id" : "<>",
5+
"remote_vm_name" : "<>",
6+
"remote_vm_username" : "<>",
7+
"remote_vm_password" : "<>",
8+
"remote_vm_ip" : "<>"
9+
}

‎aml_service/00-WorkSpace.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
"""
2+
Copyright (C) Microsoft Corporation. All rights reserved.​
3+
​
4+
Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
5+
royalty-free right to use, copy, and modify the software code provided by us
6+
("Software Code"). You may not sublicense the Software Code or any use of it
7+
(except to your affiliates and to vendors to perform work on your behalf)
8+
through distribution, network access, service agreement, lease, rental, or
9+
otherwise. This license does not purport to express any claim of ownership over
10+
data you may have shared with Microsoft in the creation of the Software Code.
11+
Unless applicable law gives you more rights, Microsoft reserves all other
12+
rights not expressly granted herein, whether by implication, estoppel or
13+
otherwise. ​
14+
​
15+
THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
16+
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+
MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
19+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
20+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
21+
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22+
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23+
ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
24+
POSSIBILITY OF SUCH DAMAGE.
25+
"""
26+
from azureml.core import Workspace
27+
import os, json
28+
import azureml.core
29+
print("SDK Version:", azureml.core.VERSION)
30+
#print('current dir is ' +os.curdir)
31+
with open("aml_config/config.json") as f:
32+
config = json.load(f)
33+
34+
workspace_name = config['workspace_name']
35+
resource_group = config['resource_group']
36+
subscription_id = config['subscription_id']
37+
location = config['location']
38+
#location = 'southcentralus'
39+
try:
40+
ws = Workspace.get(name = workspace_name,
41+
subscription_id = subscription_id,
42+
resource_group = resource_group)
43+
44+
except:
45+
# this call might take a minute or two.
46+
print('Creating new workspace')
47+
ws = Workspace.create(name = workspace_name,
48+
subscription_id = subscription_id,
49+
resource_group = resource_group,
50+
#create_resource_group=True,
51+
location=location)
52+
53+
# print Workspace details
54+
print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\n')

‎aml_service/01-Experiment.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
"""
2+
Copyright (C) Microsoft Corporation. All rights reserved.​
3+
​
4+
Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
5+
royalty-free right to use, copy, and modify the software code provided by us
6+
("Software Code"). You may not sublicense the Software Code or any use of it
7+
(except to your affiliates and to vendors to perform work on your behalf)
8+
through distribution, network access, service agreement, lease, rental, or
9+
otherwise. This license does not purport to express any claim of ownership over
10+
data you may have shared with Microsoft in the creation of the Software Code.
11+
Unless applicable law gives you more rights, Microsoft reserves all other
12+
rights not expressly granted herein, whether by implication, estoppel or
13+
otherwise. ​
14+
​
15+
THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
16+
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+
MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
19+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
20+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
21+
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22+
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23+
ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
24+
POSSIBILITY OF SUCH DAMAGE.
25+
"""
26+
import os
27+
from azureml.core import Experiment
28+
from azureml.core import Workspace
29+
30+
def getExperiment():
31+
ws = Workspace.from_config()
32+
script_folder = "."
33+
experiment_name = 'devops-ai-demo'
34+
exp = Experiment(workspace = ws, name = experiment_name)
35+
print(exp.name, exp.workspace.name, sep = '\n')
36+
return exp
37+
38+
if __name__ == "__main__":
39+
exp = getExperiment()

‎aml_service/02-AttachTrainingVM.py‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
"""
2+
Copyright (C) Microsoft Corporation. All rights reserved.​
3+
​
4+
Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
5+
royalty-free right to use, copy, and modify the software code provided by us
6+
("Software Code"). You may not sublicense the Software Code or any use of it
7+
(except to your affiliates and to vendors to perform work on your behalf)
8+
through distribution, network access, service agreement, lease, rental, or
9+
otherwise. This license does not purport to express any claim of ownership over
10+
data you may have shared with Microsoft in the creation of the Software Code.
11+
Unless applicable law gives you more rights, Microsoft reserves all other
12+
rights not expressly granted herein, whether by implication, estoppel or
13+
otherwise. ​
14+
​
15+
THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
16+
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+
MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
19+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
20+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
21+
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22+
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23+
ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
24+
POSSIBILITY OF SUCH DAMAGE.
25+
"""
26+
27+
from azureml.core import Workspace
28+
from azureml.core import Run
29+
from azureml.core import Experiment
30+
from azureml.core.conda_dependencies import CondaDependencies
31+
from azureml.core.runconfig import RunConfiguration
32+
import os, json
33+
from azureml.core.compute import RemoteCompute
34+
from azureml.core.compute import DsvmCompute
35+
from azureml.core.compute_target import ComputeTargetException
36+
37+
38+
# Get workspace
39+
ws = Workspace.from_config()
40+
41+
# Read the New VM Config
42+
with open("aml_config/security_config.json") as f:
43+
config = json.load(f)
44+
45+
remote_vm_name = config['remote_vm_name']
46+
remote_vm_username = config['remote_vm_username']
47+
remote_vm_password = config['remote_vm_password']
48+
remote_vm_ip = config['remote_vm_ip']
49+
50+
try:
51+
dsvm_compute = RemoteCompute.attach(ws, name=remote_vm_name,
52+
username=remote_vm_username,
53+
address=remote_vm_ip,
54+
ssh_port=22,
55+
password=remote_vm_password)
56+
dsvm_compute.wait_for_completion(show_output =True)
57+
58+
except Exception as e:
59+
print("Caught = {}".format(e.message))
60+
print("Compute config already attached.")
61+
62+
63+
## Create VM if not available
64+
# compute_target_name = remote_vm_name
65+
66+
# try:
67+
# dsvm_compute = DsvmCompute(workspace=ws, name=compute_target_name)
68+
# print('found existing:', dsvm_compute.name)
69+
# except ComputeTargetException:
70+
# print('creating new.')
71+
# dsvm_config = DsvmCompute.provisioning_configuration(vm_size="Standard_D2_v2")
72+
# dsvm_compute = DsvmCompute.create(ws, name=compute_target_name, provisioning_configuration=dsvm_config)
73+
# dsvm_compute.wait_for_completion(show_output=True)

‎aml_service/10-TrainOnLocal.py‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
"""
2+
Copyright (C) Microsoft Corporation. All rights reserved.​
3+
​
4+
Microsoft Corporation (“Microsoft”) grants you a nonexclusive, perpetual,
5+
royalty-free right to use, copy, and modify the software code provided by us
6+
("Software Code"). You may not sublicense the Software Code or any use of it
7+
(except to your affiliates and to vendors to perform work on your behalf)
8+
through distribution, network access, service agreement, lease, rental, or
9+
otherwise. This license does not purport to express any claim of ownership over
10+
data you may have shared with Microsoft in the creation of the Software Code.
11+
Unless applicable law gives you more rights, Microsoft reserves all other
12+
rights not expressly granted herein, whether by implication, estoppel or
13+
otherwise. ​
14+
​
15+
THE SOFTWARE CODE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
16+
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+
MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
19+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
20+
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
21+
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22+
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23+
ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE
24+
POSSIBILITY OF SUCH DAMAGE.
25+
"""
26+
27+
from azureml.core.runconfig import RunConfiguration
28+
from azureml.core import Workspace
29+
from azureml.core import Experiment
30+
from azureml.core import ScriptRunConfig
31+
import os, json
32+
33+
# Get workspace
34+
ws = Workspace.from_config()
35+
36+
# Attach Experiment
37+
experiment_name = 'devops-ai-demo'
38+
exp = Experiment(workspace = ws, name = experiment_name)
39+
print(exp.name, exp.workspace.name, sep = '\n')
40+
41+
# Editing a run configuration property on-fly.
42+
run_config_user_managed = RunConfiguration()
43+
run_config_user_managed.environment.python.user_managed_dependencies = True
44+
45+
print("Submitting an experiment.")
46+
src = ScriptRunConfig(source_directory = './code', script = 'training/train.py', run_config = run_config_user_managed)
47+
run = exp.submit(src)
48+
49+
# Shows output of the run on stdout.
50+
run.wait_for_completion(show_output = True, wait_post_processing = True)
51+
52+
# Raise exception if run fails
53+
if run.get_status() == 'Failed':
54+
raise Exception('Training on local failed with following run status: {} and logs: \n {}'.format(run.get_status(),run.get_details_with_logs()))
55+
56+
# Writing the run id to /aml_config/run_id.json
57+
58+
run_id={}
59+
run_id['run_id'] = run.id
60+
run_id['experiment_name'] = run.experiment.name
61+
with open('aml_config/run_id.json', 'w') as outfile:
62+
json.dump(run_id,outfile)

0 commit comments

Comments
 (0)