Skip to content

Commit ec27b55

Browse files
v-regandownerv-regandowner
authored
Merge pull request #211889 from austinmccollum/austinmc-repo-custom
split repo how to custom
2 parents 95678aa + 4151e17 commit ec27b55

File tree

5 files changed

+502
-146
lines changed

5 files changed

+502
-146
lines changed

‎articles/sentinel/ci-cd-custom-content.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ The following Microsoft Sentinel content types can be deployed through a reposit
4242
> This article does *not* describe how to create these types of content from scratch. For more information, see the relevant [Microsoft Sentinel GitHub wiki](https://github.com/Azure/Azure-Sentinel/wiki#get-started) for each content type.
4343
>
4444
45-
Repositories content needs to be stored as [ARM templates](/azure/azure-resource-manager/templates/overview). The repositories deployment pipeline doesn't validate the content except to confirm it's in the correct JSON format.
45+
Repositories content needs to be stored as [ARM templates](/azure/azure-resource-manager/templates/overview). The repositories deployment doesn't validate the content except to confirm it's in the correct JSON format.
4646

4747
The first step to validate your content is to test it within Microsoft Sentinel. You can also apply the [Microsoft Sentinel GitHub validation process](https://github.com/Azure/Azure-Sentinel/wiki#test-your-contribution) and tools to complement your validation process.
4848

@@ -64,7 +64,7 @@ A sample repository is available with ARM templates for each of the content type
6464

6565
The **smart deployments** feature is a back-end capability that improves performance by actively tracking modifications made to the content files of a connected repository. It uses a CSV file within the '.sentinel' folder in your repository to audit each commit. The workflow avoids redeploying content that hasn't been modified since the last deployment. This process improves your deployment performance and prevents tampering with unchanged content in your workspace, such as resetting dynamic schedules of your analytics rules.
6666

67-
Smart deployments are enabled by default on newly created connections. If you prefer all source control content to be deployed every time a deployment is triggered, regardless of whether that content was modified or not, you can modify your workflow to disable smart deployments. For more information, see [Customize the deployment workflow](ci-cd.md#customize-the-deployment-workflow).
67+
Smart deployments are enabled by default on newly created connections. If you prefer all source control content to be deployed every time a deployment is triggered, regardless of whether that content was modified or not, you can modify your workflow to disable smart deployments. For more information, see [Customize the workflow or pipeline](ci-cd-custom-deploy.md#customize-the-workflow-or-pipeline).
6868

6969
> [!NOTE]
7070
> This capability was launched in public preview on April 20th, 2022. Connections created prior to launch would need to be updated or recreated for smart deployments to be turned on.
@@ -73,23 +73,33 @@ Smart deployments are enabled by default on newly created connections. If you pr
7373

7474
## Consider deployment customization options
7575

76-
Even with smart deployments enabled, the default behavior is to push all the updated content from the connected repository branch. If the default configuration for your content deployment from GitHub or Azure DevOps doesn't meet all your requirements, you can modify the experience to fit your needs.
76+
A number of customization options are available to consider when deploying content with Microsoft Sentinel repositories.
7777

78-
For example, you may want to:
79-
- turn off smart deployments
78+
#### Customize the workflow or pipeline
79+
80+
You may want to customize the workflow or pipeline in one of the following ways:
8081
- configure different deployment triggers
8182
- deploy content only from a specific root folder for a given workspace
8283
- schedule the workflow to run periodically
8384
- combine different workflow events together
84-
- prioritize content to be evaluated before the entire repo is enumerated for valid ARM templates
85+
- turn off smart deployments
86+
87+
These customizations are defined in a .yml file specific to your workflow or pipeline. For more details on how to implement, see [Customize repository deployments](ci-cd-custom-deploy.md#customize-the-workflow-or-pipeline)
8588

86-
For more details on how to implement these customizations, see [Customize the deployment workflow](ci-cd.md#customize-the-deployment-workflow).
89+
#### Customize the deployment
90+
91+
Once the workflow or pipeline is triggered, the deployment supports the following scenarios:
92+
- prioritize content to be deployed before the rest of the repo content
93+
- exclude content from deployment
94+
- specify ARM template parameter files
95+
96+
These options are available through a feature of the PowerShell deployment script called from the workflow or pipeline. For more details on how to implement these customizations, see [Customize repository deployments](ci-cd-custom-deploy.md#customize-your-connection-configuration).
8797

8898

8999
## Next steps
90100

91101
Get more examples and step by step instructions on deploying Microsoft Sentinel repositories.
92102

93-
- [Sentinel CICD sample repository](https://github.com/SentinelCICD/RepositoriesSampleContent)
94103
- [Deploy custom content from your repository](ci-cd.md)
104+
- [Sentinel CICD sample repository](https://github.com/SentinelCICD/RepositoriesSampleContent)
95105
- [Automate Sentinel integration with DevOps](/azure/architecture/example-scenario/devops/automate-sentinel-integration#microsoft-sentinel-repositories)

‎articles/sentinel/ci-cd-custom-deploy.md

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
---
2+
title: Customize repository deployments
3+
titleSuffix: Microsoft Sentinel
4+
description: This article describes how to customize repository deployments for the repositories feature in Microsoft Sentinel.
5+
author: austinmccollum
6+
ms.topic: how-to
7+
ms.date: 9/15/2022
8+
ms.author: austinmc
9+
#Customer intent: As a SOC collaborator or MSSP analyst, I want to know how to optimize my source control repositories for continuous integration and continuous delivery (CI/CD). Specifically as an MSSP content manager, I want to know how to deploy one solution to many customer workspaces and still be able to tailor custom content for their environments.
10+
---
11+
12+
# Customize repository deployments (Public Preview)
13+
14+
There are two primary ways to customize the deployment of your repository content to Microsoft Sentinel workspaces. Each method uses different files and syntax, so consider these examples to get you started.
15+
16+
- Modify the GitHub workflow or DevOps pipeline to customize deployment options such as your connection's deployment trigger, deployment path or usage of smart deployments.
17+
18+
- Utilize the newly introduced configuration file to control the prioritized order of your content deployments, choose to *exclude* specific content files from those deployments, or map parameter files to specific content files.
19+
20+
> [!IMPORTANT]
21+
>
22+
> The Microsoft Sentinel **Repositories** feature is currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
23+
24+
25+
## Prerequisites and scope
26+
27+
Microsoft Sentinel currently supports connections to GitHub and Azure DevOps repositories. Before connecting your Microsoft Sentinel workspace to your source control repository, make sure that you have:
28+
29+
- An **Owner** role in the resource group that contains your Microsoft Sentinel workspace *or* a combination of **User Access Administrator** and **Sentinel Contributor** roles to create the connection
30+
- Contributor access to your GitHub or Azure DevOps repository
31+
- Actions enabled for GitHub and Pipelines enabled for Azure DevOps
32+
- Ensure custom content files you want to deploy to your workspaces are in relevant [Azure Resource Manager (ARM) templates](../azure-resource-manager/templates/index.yml).
33+
34+
For more information, see [Validate your content](ci-cd-custom-content.md#validate-your-content)
35+
36+
37+
## Customize the workflow or pipeline
38+
39+
The default workflow only deploys content that has been modified since the last deployment based on commits to the repository. But you may want to turn off smart deployments or perform other customizations. For example, you can configure different deployment triggers, or deploy content exclusively from a specific root folder.
40+
41+
Select one of the following tabs depending on your connection type:
42+
43+
# [GitHub](#tab/github)
44+
45+
**To customize your GitHub deployment workflow**:
46+
47+
1. In GitHub, go to your repository and find your workflow in the *.github/workflows* directory.
48+
49+
The workflow file is the YML file starting with *sentinel-deploy-xxxxx.yml*. Open that file and the workflow name is shown in the first line and has the following default naming convention: `Deploy Content to <workspace-name> [<deployment-id>]`.
50+
51+
For example: `name: Deploy Content to repositories-demo [xxxxx-dk5d-3s94-4829-9xvnc7391v83a]`
52+
53+
1. Select the pencil button at the top-right of the page to open the file for editing, and then modify the deployment as follows:
54+
55+
- **To modify the deployment trigger**, update the `on` section in the code, which describes the event that triggers the workflow to run.
56+
57+
By default, this configuration is set to `on: push`, which means that the workflow is triggered at any push to the connected branch, including both modifications to existing content and additions of new content to the repository. For example:
58+
59+
```yml
60+
on:
61+
push:
62+
branches: [ main ]
63+
paths:
64+
- `**`
65+
- `!.github/workflows/**` # this filter prevents other workflow changes from triggering this workflow
66+
- `.github/workflows/sentinel-deploy-<deployment-id>.yml`
67+
```
68+
69+
You may want to change these settings, for example, to schedule the workflow to run periodically, or to combine different workflow events together.
70+
71+
For more information, see the [GitHub documentation](https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows#configuring-workflow-events) on configuring workflow events.
72+
73+
- **To disable smart deployments**:
74+
The smart deployments behavior is separate from the deployment trigger discussed above. Navigate to the `jobs` section of your workflow. Switch the `smartDeployment` default value from `true` to `false`. This will turn off the smart deployments functionality and all future deployments for this connection will redeploy all the repository's relevant content files to the connected workspace(s) once this change is committed.
75+
76+
- **To modify the deployment path**:
77+
78+
In the default configuration shown above for the `on` section, the wildcards (`**`) in the first line in the `paths` section indicate that the entire branch is in the path for the deployment triggers.
79+
80+
This default configuration means that a deployment workflow is triggered anytime that content is pushed to any part of the branch.
81+
82+
Later on in the file, the `jobs` section includes the following default configuration: `directory: '${{ github.workspace }}'`. This line indicates that the entire GitHub branch is in the path for the content deployment, without filtering for any folder paths.
83+
84+
To deploy content from a specific folder path only, add it to both the `paths` and the `directory` configuration. For example, to deploy content only from a root folder named `SentinelContent`, update your code as follows:
85+
86+
```yml
87+
paths:
88+
- `SentinelContent/**`
89+
- `!.github/workflows/**` # this filter prevents other workflow changes from triggering this workflow
90+
- `.github/workflows/sentinel-deploy-<deployment-id>.yml`
91+
92+
...
93+
directory: '${{ github.workspace }}/SentinelContent'
94+
```
95+
96+
For more information, see the [GitHub documentation](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#onpushpull_requestpaths) on GitHub Actions and editing GitHub workflows.
97+
98+
# [Azure DevOps](#tab/azure-devops)
99+
100+
**To customize your Azure DevOps deployment pipeline**:
101+
102+
1. In Azure DevOps, go to your repository and find your pipeline definition file in the *.sentinel* directory.
103+
104+
The pipeline name is shown in the first line of the pipeline file, and has the following default naming convention: `Deploy Content to <workspace-name> [<deployment-id>]`.
105+
106+
For example: `name: Deploy Content to repositories-demo [xxxxx-dk5d-3s94-4829-9xvnc7391v83a]`
107+
108+
1. Select the pencil button at the top-right of the page to open the file for editing, and then modify the deployment as follows:
109+
110+
- **To modify the deployment trigger**, update the `trigger` section in the code, which describes the event that triggers the workflow to run.
111+
112+
By default, this configuration is set to detect any push to the connected branch, including both modifications to existing content and additions of new content to the repository.
113+
114+
Modify this trigger to any available Azure DevOps Triggers, such as a scheduling trigger or a pull request triggers. For more information, see the [Azure DevOps trigger documentation](/azure/devops/pipelines/yaml-schema).
115+
116+
- **To disable smart deployments**:
117+
The smart deployments behavior is separate from the deployment trigger discussed above. Navigate to the `ScriptArguments` section of your pipeline. Switch the `smartDeployment` default value from `true` to `false`. This will turn off the smart deployments functionality and all future deployments for this connection will redeploy all the repository's relevant content files to the connected workspace(s) once this change is committed.
118+
119+
- **To modify the deployment path**:
120+
121+
The default configuration for the `trigger` section has the following code, which indicates that the `main` branch is in the path for the deployment triggers:
122+
123+
```yml
124+
trigger:
125+
branches:
126+
include:
127+
- main
128+
```
129+
130+
This default configuration means that a deployment pipeline is triggered anytime that content is pushed to any part of the `main` branch.
131+
132+
To deploy content from a specific folder path only, add the folder name to the `include` section, for the trigger, and the `steps` section, for the deployment path, below as needed.
133+
134+
For example, to deploy content only from a root folder named `SentinelContent` in your `main` branch, add `include` and `workingDirectory` settings to your code as follows:
135+
136+
```yml
137+
paths:
138+
exclude:
139+
- .sentinel/*
140+
include:
141+
- .sentinel/sentinel-deploy-39d8ekc8-397-5963-49g8-5k63k5953829.yml
142+
- SentinelContent
143+
....
144+
steps:
145+
- task: AzurePowerShell@5
146+
inputs:
147+
azureSubscription: `Sentinel_Deploy_ServiceConnection_0000000000000000`
148+
workingDirectory: `SentinelContent`
149+
```
150+
151+
For more information, see the [Azure DevOps documentation](/azure/devops/pipelines/yaml-schema) on the Azure DevOps YAML schema.
152+
153+
---
154+
155+
> [!IMPORTANT]
156+
> In both GitHub and Azure DevOps, make sure that you keep the trigger path and deployment path directories consistent.
157+
>
158+
159+
160+
## Customize your connection configuration
161+
162+
The deployment script for repositories supports the usage of a deployment configuration file for each repository branch as of July 2022. The configuration JSON file helps you map parameter files to relevant content files, prioritize specific content in deployments, and exclude specific content from deployments.
163+
164+
165+
1. Create the file *sentinel-deployment.config* at the root of your repository. Adding, deleting, or modifying this configuration file will cause a full deployment of all the content in the repository according to the updated configuration.
166+
167+
:::image type="content" source="media/ci-cd-custom-deploy/deployment-config.png" alt-text="Screenshot of a repository root directory. The RepositoriesSampleContent is shown with the location of the sentinel-deployment.config file." lightbox="media/ci-cd-custom-deploy/deployment-config.png":::
168+
169+
1. Include JSON structured content in three optional sections, `"prioritizedcontentfiles":`, `"excludecontentfiles":`, and `"parameterfilemappings":`. If no sections are included or the .config file is omitted, the deployment process will still run. Invalid or unrecognized sections will be ignored.
170+
171+
Here's an example of the entire contents of a valid *sentinel-deployment.config* file. This sample can also be found at the [Sentinel CICD repositories sample](https://github.com/SentinelCICD/RepositoriesSampleContent).
172+
173+
```json
174+
{
175+
"prioritizedcontentfiles": [
176+
"parsers/Sample/ASimAuthenticationAWSCloudTrail.json",
177+
"workbooks/sample/TrendMicroDeepSecurityAttackActivity_ARM.json",
178+
"Playbooks/PaloAlto-PAN-OS/PaloAltoCustomConnector/azuredeploy.json"
179+
],
180+
"excludecontentfiles": [
181+
"Detections/Sample/PaloAlto-PortScanning.json",
182+
"parameters"
183+
],
184+
"parameterfilemappings": {
185+
"879001c8-2181-4374-be7d-72e5dc69bd2b": {
186+
"Playbooks/PaloAlto-PAN-OS/Playbooks/PaloAlto-PAN-OS-BlockIP/azuredeploy.json": "parameters/samples/parameter-file-1.json"
187+
},
188+
"9af71571-7181-4cef-992e-ef3f61506b4e": {
189+
"Playbooks/Enrich-SentinelIncident-GreyNoiseCommunity-IP/azuredeploy.json": "path/to/any-parameter-file.json"
190+
}
191+
},
192+
"DummySection": "This shouldn't impact deployment"
193+
}
194+
```
195+
196+
> [!NOTE]
197+
> Don't use the backslash "\\" character in any of the content paths. Use the forward slash "/" instead.
198+
>
199+
200+
- **To prioritize content files**:
201+
202+
As the amount of content in your repository grows, deployment times may increase. Add time sensitive content to this section to prioritize its deployment when a trigger occurs.
203+
204+
Add full path names to the `"prioritizedcontentfiles":` section. Wildcard matching is not supported at this time.
205+
206+
- **To exclude content files**, modify the `"excludecontentfiles":` section with full path names of individual .json deployment files.
207+
208+
- **To map parameters**:
209+
210+
The deployment script will accept three methods to map parameters. The precedence is determined for each included .json deployment file in your repository as follows:
211+
212+
:::image type="content" source="media/ci-cd-custom-deploy/deploy-parameter-file-precedence.svg" alt-text="A diagram showing the precedence of parameter file mappings.":::
213+
214+
1. Is there a mapping in the sentinel-deployment.config?
215+
1. Is there a workspace parameter file?
216+
1. Is there a default parameter file?
217+
218+
Modifying the mapped parameter file listed in the sentinel-deployment.config will trigger the deployment of its paired content file. Adding or modifying a *.parameters-\<workspaceID\>.json* file or *.parameters.json* file triggers a deployment of that corresponding content file along with the newly modified parameters, unless a higher precedence parameter mappings is in place. Other content files won't be deployed if the smart deployments feature is still enabled.
219+
220+
221+
## Next steps
222+
223+
A sample repository is available with demonstrating the deployment config file and all three parameter mapping methods.
224+
225+
For more information, see:
226+
227+
- [Sentinel CICD repositories sample](https://github.com/SentinelCICD/RepositoriesSampleContent)
228+
- [Create Resource Manager parameter file](/../../azure/azure-resource-manager/templates/parameter-files.md)
229+
- [Parameters in ARM templates](/../../azure/azure-resource-manager/templates/parameters.md)
230+
231+

0 commit comments

Comments
 (0)