Skip to content

Update install docs with SP comments and backend state comments #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

Update install docs with SP comments and backend state comments #284

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3c31881
update docs with test detection comment
TechnicallyWilliams Sep 6, 2019
fc4a2b1
updating docs with backendstate and SP comments
TechnicallyWilliams Sep 6, 2019
7b10b33
Merge branch 'master' into dexterw/features/update-install-docs
TechnicallyWilliams Sep 12, 2019
f41f1ae
split add_pat_owner into separate ui and cli docs
TechnicallyWilliams Sep 12, 2019
996f30d
added more detail to add pat owner cli doc
TechnicallyWilliams Sep 12, 2019
6ce7078
divided add pat owner cli doc into more sections
TechnicallyWilliams Sep 14, 2019
e69871f
add more details to add pat owner cli doc
TechnicallyWilliams Sep 14, 2019
7d36f6a
update style and punctuation
TechnicallyWilliams Sep 14, 2019
441ce05
clarify final steps of add pat owner cli
TechnicallyWilliams Sep 14, 2019
da78311
link to azure portal steps from add pat owner cli docs
TechnicallyWilliams Sep 14, 2019
8ac899c
update instructions for both add pat owner docs
TechnicallyWilliams Sep 24, 2019
024845e
merging conflicts
TechnicallyWilliams Sep 24, 2019
e928c4e
Updates the formating of the AdPat owner doc (#293)
negupt Sep 24, 2019
45044f5
address merge conflicts
TechnicallyWilliams Sep 24, 2019
813eec4
update getting started add pat owner docs
TechnicallyWilliams Sep 25, 2019
42bb4de
update readability of add pat owner docs
TechnicallyWilliams Oct 1, 2019
fcfc36c
update readability of add pat owner docs
TechnicallyWilliams Oct 1, 2019
82d87f2
add clarity to several add pat owner docs
TechnicallyWilliams Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 14 additions & 147 deletions docs/GETTING_STARTED_ADD_PAT_OWNER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,53 +1,22 @@
# Getting Started - Advocated Pattern Owner

## Cobalt Enterprise Integration - Overview
*Prefer using cli? Follow the [cli-based walkthrough](./GETTING_STARTED_APP_PAT_OWNER_CLI.md).*

## Overview

Completion of the steps from this document results in an Azure Devops Repo initialized with carefully selected Cobalt Advocated Pattern Templates (Infrastructure as code) along with a CI/CD pipeline ready for multi-stage deployments.

This document provides Cobalt users instructions for initializing and integrating Cobalt into their existing AzureDevops organization using an Azure subscription. These steps assume some basic familiarity with the Azure DevOps portal and a desire to automate the creation of infrastructure. For more information on Cobalt, visit the following link: [READ ME](../README.md)
This document provides Cobalt users instructions for initializing and integrating Cobalt into their existing AzureDevops organization using an Azure subscription. These steps assume some basic familiarity with the Azure DevOps portal and Azure cloud portal. For more information on Cobalt, visit the following link: [READ ME](../README.md)

### Prerequisites
## Prerequisites

* An Azure Subscription
* Azure Devops Organization
* Permissions to your Organization's Azure Devops account
* *Global administrator role* permissions in your Organization's Azure Active Directory tenant to setup service principals
* If this is not allowed by your organization, step two and the Service Connection creation in step three will need to be completed by someone within your organization with this permission.

> If using the CLI commands please set the following variables:

| Variable | Sample Value | Description |
|----------|--------------|-------------|
| `TEMPLATE_DEVOPS_PROJECT_NAME` | `My Application` | The name of the project representing your Cobalt template application that serves as your organization's advocated pattern for a specific template. |
| `TEMPLATE_DEVOPS_INFRA_REPO_NAME` | `az-hello-world` | The name of the repo that will be created in the application Azure DevOps project to host the Cobalt template. |
| `TEMPLATE_DEVOPS_INFRA_YML_PATH` | `devops/providers/azure-devops/templates/azure-pipelines.yml` | The path relative to the `TEMPLATE_DEVOPS_INFRA_REPO_NAME` root that contains the Cobalt template pipeline to be created for testing and provisioning resources. |
| `DEFAULT_ORGANIZATION` | `https://dev.azure.com/MyOrganization/` | The full URL path of the organization in which your `TEMPLATE_DEVOPS_PROJECT_NAME` resides or will be created. |
| `COBALT_SOURCE_URL` | `https://github.com/microsoft/cobalt.git` | The Git clone URL for Cobalt (containing all templates including the one to be targeted by this template repository) from which this Cobalt template repository will be sourced. |

Update these values for your environment and application based on the guidance in the table above.
```bash
export TEMPLATE_DEVOPS_PROJECT_NAME=""
export TEMPLATE_DEVOPS_INFRA_REPO_NAME=""
export TEMPLATE_DEVOPS_INFRA_YML_PATH=""
export DEFAULT_ORGANIZATION=""
export COBALT_SOURCE_URL=""
export SUBSCRIPTION_ID=""
export SUBSCRIPTION_NAME=""
export TENANT_ID=""
export SERVICE_PRIN_ID=""
export SERVICE_CONN_NAME=""
export AGENT_POOL_NAME=""
```

The following values are used like constants and should not need to change (unless the build pipeline definition is modified).

```bash
export COBALT_VAR_GROUP_INFRA="Infrastructure Pipeline Variables"
export COBALT_VAR_GROUP_ENV_SUFFIX="Environment Variables"
export COBALT_PIPELINE_NAME="Cobalt CICD Pipeline"
```

### STEPS
### Steps

1. **Initialize Azure Repo Subscription with Cobalt**

Expand All @@ -61,15 +30,6 @@ export COBALT_PIPELINE_NAME="Cobalt CICD Pipeline"

* Select Create

> The following CLI command(s) can be run as an alternative to using the portal-based instructions:


```bash
az devops configure --defaults organization="$DEFAULT_ORGANIZATION"
az devops project create --name "$TEMPLATE_DEVOPS_PROJECT_NAME" --source-control git --visibility private
az devops configure -d project="$TEMPLATE_DEVOPS_PROJECT_NAME"
```

* Create new repository by fetching source code from the master branch of Cobalt's open-source github project
* Select Repos tab within side-navigation menu
* Select 'Import a repository' from the Repos tab sub-menu and click [Import]
Expand Down Expand Up @@ -97,13 +57,6 @@ az devops configure -d project="$TEMPLATE_DEVOPS_PROJECT_NAME"
| Cobalt-Contoso | If the aim is to use a single repository as ground truth for all potential patterns across your organization, effectively having to manage a combination of Cobalt patterns from a single repo, it's recommended to stick with a name that matches the project name. |


> The following CLI command(s) can be run as an alternative to using the portal-based instructions:

```bash
az repos create --name "$TEMPLATE_DEVOPS_INFRA_REPO_NAME"
az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME"
```

* Initialize new Azure Devops pipeline
* Select Pipelines tab from within side-navigation menu
* Select Create Pipeline and then choose 'Azure Repos Git [YAML]'
Expand Down Expand Up @@ -131,14 +84,7 @@ az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVO

![Fail Screenshot](https://user-images.githubusercontent.com/10041279/63546484-8ccd3f80-c4ef-11e9-8d9f-2f06dc725fc7.png)

> NOTE: Azure Devops forces a run so expect this to fail. Future steps will resolve this problem.


> The following CLI command(s) can be run as an alternative to using the portal-based instructions:

```
az pipelines create --name "$COBALT_PIPELINE_NAME" --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" --branch master --repository-type tfsgit --yml-path $TEMPLATE_DEVOPS_INFRA_YML_PATH --skip-run true
```
> NOTE: Azure Devops forces a run so expect this to fail. Later steps will resolve this problem.

2. **Provision Azure resources needed for Azure Devops pipeline**

Expand All @@ -160,7 +106,7 @@ az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVO
![Request Permissions menu](https://user-images.githubusercontent.com/10041279/63549279-b6896500-c4f5-11e9-9c92-40ac2a4295c9.png)

* Click [Add permissions] to save this configuration
* Click [Grant admin consent for *Your Directory*] to grant consent on behalf of users in this directory for this permission
* Click [Grant admin consent for *Your Directory*] to grant consent on behalf of users in this directory for this permission

* Configure the new AAD app as a Cobalt admin service-principal/service-endpoint
* From the App registrations service blade, click the [Certificates & secrets] tab
Expand Down Expand Up @@ -199,12 +145,6 @@ az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVO
* Once deployment for storage account is completed, go to the resource and visit the Blobs sub-menu
* Click [+Container] then create a container name (ex. az-hw-remote-state-container or az-iso-remote-state-container) with private access

> The following CLI command(s) can be run as an alternative to using the portal-based instructions. Note this creates the Service Connection within Azure DevOps, but you will still need to setup the Service Principal:

```bash
az devops service-endpoint azurerm create --azure-rm-subscription-id $SUBSCRIPTION_ID --azure-rm-subscription-name "$SUBSCRIPTION_NAME" --azure-rm-tenant-id $TENANT_ID --azure-rm-service-principal-id $SERVICE_PRIN_ID --name "$SERVICE_CONN_NAME"
```

3. **Configure Azure Devops pipeline using Azure resource values**

This step is about making sure Azure Devops references all the values and resources you took the time to create in the Azure portal.
Expand Down Expand Up @@ -237,11 +177,11 @@ az devops service-endpoint azurerm create --azure-rm-subscription-id $SUBSCRIPTI
| Name | Value | Var Description |
|-------------|-----------|-----------|
| `AGENT_POOL` | Hosted Ubuntu 1604 | The type of build agent used for your deployment. |
| `ARM_PROVIDER_STRICT` | false | Terraform ARM provider modification |
| `ARM_PROVIDER_STRICT` | true | Terraform ARM provider modification |
| `BUILD_ARTIFACT_NAME` | drop | Name to identity the folder containing artifacts output by a build. |
| `GO_VERSION`| 1.12.5 | The version of Go terraform deployments are bound to. |
| `PIPELINE_ROOT_DIR` | devops/providers/azure-devops/templates/ | A path for finding Cobalt templates. |
| `REMOTE_STATE_CONTAINER` | `<CONTAINER_NAME>`| The remote blob storage container name for managing the state of a Cobalt Template's deployed infrastructure. Also is used as a naming convention for partitioning state into multiple workspaces. This name was created in an earlier step from within the azure portal. |
| `REMOTE_STATE_CONTAINER` | `<BACKEND_STATE_CONTAINER_NAME>`| The remote blob storage container name for managing the state of a Cobalt Template's deployed infrastructure. Also is used as a naming convention for partitioning state into multiple workspaces. This name was created in an earlier step from within the azure portal. |
| `SCRIPTS_DIR` | infrastructure/scripts | Path to scripts used at runtime for composing build and release jobs at various pipeline stages. |
| `TEST_HARNESS_DIR` | test-harness/ | A path to the cobalt test harness for running integration and unit tests written in Docker and Golang. |
| `TF_ROOT_DIR`| infra | The primary path for all Cobalt templates and the modules they are composed of. |
Expand All @@ -251,27 +191,6 @@ az devops service-endpoint azurerm create --azure-rm-subscription-id $SUBSCRIPTI
> Important: Every targeted environment specified within the build pipeline expects a
> variable group specified with the naming convention `<ENVIRONMENT_NAME> Environment Variables`

> The following CLI command(s) can be run as an alternative to using the portal-based instructions:

```bash
# IMPORTANT: Replace these values as necessary to fit your environment.
az pipelines variable-group create --authorize true --name "$COBALT_VAR_GROUP_INFRA" --variables \
AGENT_POOL="$AGENT_POOL_NAME" \
ARM_PROVIDER_STRICT=true \
BUILD_ARTIFACT_NAME='drop' \
BUILD_ARTIFACT_PATH_ALIAS='artifact' \
GO_VERSION='1.12.5' \
PIPELINE_ROOT_DIR='devops/providers/azure-devops/templates/infrastructure' \
REMOTE_STATE_CONTAINER='BACKENDSTATECONTAINERNAME' \
SCRIPTS_DIR='scripts' \
TEST_HARNESS_DIR='test-harness/' \
TF_DEPLOYMENT_TEMPLATE_ROOT='infra/templates/$TEMPLATE_DEVOPS_INFRA_REPO_NAME' \
TF_DEPLOYMENT_WORKSPACE_PREFIX='PROJECTDEPLOYMENTWORKSPACEPREFIX' \
TF_ROOT_DIR='infra' \
TF_VERSION='0.12.4' \
TF_WARN_OUTPUT_ERRORS=1
```

* Configure *DevInt Environment Variables* as the final variable group
* Environment-specific variables have no default values and must be assigned
* Return to the Library tab
Expand All @@ -285,30 +204,6 @@ az pipelines variable-group create --authorize true --name "$COBALT_VAR_GROUP_IN
| `SERVICE_CONNECTION_NAME` | ex. Cobalt Deployment Administrator-`<TenantName>` | The custom name of the service connection configured in a previous Azure Devops step that establishes a connection between the Service Principal and the Azure subscription that it's permissioned for. |
| `TF_CLI_ARGS` | "-refresh=false" | specify additional arguments to the command-line. This allows easier automation in CI environments as well as modifying default behavior of Terraform |

> The following CLI command(s) can be run as an alternative to using the portal-based instructions:

```bash
# IMPORTANT: Replace these values as necessary to fit your environment.
DEVINT_VAR_GROUP="DevInt $COBALT_VAR_GROUP_ENV_SUFFIX"
az pipelines variable-group create --authorize true --name "$DEVINT_VAR_GROUP" --variables \
ARM_SUBSCRIPTION_ID='TARGETSUBSCRIPTIONID' \
REMOTE_STATE_ACCOUNT='BACKENDSTATESTORAGEACCOUNTNAME' \
SERVICE_CONNECTION_NAME='SERVICECONNECTIONNAME' \
TF_CLI_ARGS='-refresh=false'
```

* Additional Setup Instructions per Template

Select Cobalt templates require additional pipeline setup. Please complete extended steps if chosen template resides in the below list.

* az-isolated-service-single-region
1. Create ASE w/ VNET
2. Add additional env vars to *Infrastructure Pipeline Variables* group

| Name | Value | Var Description |
|-------|-------|-----------------|
| `TF_DEPLOYMENT_TEMPLATE_ROOT` | infra/templates/az-isolated-service-single-region | Pipeline reference for relative location of this template |

* Link Variable Groups for DevInt and Infrastructure to the Build Pipeline
* Select Pipelines tab from within side-navigation menu
* Select existing pipeline and then click [Edit]
Expand All @@ -323,32 +218,6 @@ az pipelines variable-group create --authorize true --name "$DEVINT_VAR_GROUP" -

* Save the build pipeline

> The following CLI command(s) can be run as an alternative to using the portal-based instructions:

```bash
az pipelines show --name "$COBALT_PIPELINE_NAME" -o json > builddef.json
PIPELINE_ID=$(az pipelines show --name "$COBALT_PIPELINE_NAME" --query id)
```

Execute the list command to find the Variable Group IDs created earlier. Make note of the IDs as they will need to be added to the build pipeline definition.

```bash
az pipelines variable-group list
```

For the workaround, you'll be manually editing the builddef.json file to add the variable group references. At the end of the file, you should see the line `"variableGroups" : null`. Replace the value with the following, replacing the variable group ID placeholders (`0`) with those from the above command for the Infrastructure Pipeline Variables group and DevInt Environment Variables group:
```bash
"variableGroups": [
{ "id": 0 },
{ "id": 0 }
],
```

```bash
az devops invoke --http-method PUT --area build --resource definitions --route-parameters project="$TEMPLATE_DEVOPS_PROJECT_NAME" definitionId=$PIPELINE_ID --in-file builddef.json

```

4. **Clone newly created Azure DevOps Repo from your organization**

With this step, the goal is to pull down the repo into a local environment so that one can begin making code changes.
Expand All @@ -367,17 +236,15 @@ az devops invoke --http-method PUT --area build --resource definitions --route-p

* Open the project from your favorite IDE and navigate to infrastructure templates `./infra/templates` directory.
* Manually delete template directories not needed for your enterprise.
* * The CI/CD pipeline needs to detect a code change to run the template-specific build and release jobs (in their respective stages). To force the template build and release to run, you may add a `FORCE_RUN` environment variable with a value of `true` to your *devint Environment Variables* variable group. You may also add a comment or extra line to a TF or Go file within the template in order for the pipeline script to detect a change without adding any additional override flags.

> NOTE: Do not delete 'backend-state-setup' template! We also recommended keeping the 'az-hello-world' template as a starter template.
* Commit the newly pruned project to your newly forked repo.
```bash
$ git commit -m "Removed unrelated templates." && git push
```
> NOTE: Integration tests running in the release stage of the pipeline may have resource group level naming conflicts if other tests of the same templates are also running or have been persisted in the Azure portal.

```bash
az pipelines run --name "$COBALT_PIPELINE_NAME"
```

## Conclusion
## Additional Recommendations

Recommended next step is to either reference containerized applications by their image name from within a Cobalt template in order to run a deployment or to employ this repo as ground truth for acceptable patterns and versioning across an organization.
Recommended next step is to target containerized applications via their image names from within a Cobalt Template.
Loading