Merge pull request #76 from SQLPlayer/TriggersPack

Triggers pack

Author: NowinskiK

README.md

@@ -11,14 +11,16 @@ The main advantage of the module is the ability to publish all the Azure Data Fa
 * Finding the **right order** for deploying objects (no more worrying about object names)
 * Built-in mechanism to replace, remove or add the properties with the indicated values (CSV and JSON file formats supported)
 * Stopping/starting triggers
-* Dropping objects when not exist in the source (code)
-* Filtering (include or exclude) objects to be deployed by name and/or type and/or folder
+* Dropping objects when not exist in the source (code)  
+  * (new!) Optionally can skip deletion of excluded objects
+* Filtering (include or exclude) objects to be deployed by name and/or type **and/or folder** (new!)
 * Filtering supports wildcards
 * Publish options allow you to control:
   * Whether stop and restarting triggers
   * Whether delete or not objects not in the source
   * Whether create or not a new instance of ADF if it not exist
 * Tokenisation in config file allows replace any value by Environment Variable or Variable from DevOps Pipeline
+  * (new!) Allows to define multiple file (objects) by wildcarding
 * Global Parameters
 
 The following features coming in the future:
@@ -131,6 +133,9 @@ $opt = New-AdfPublishOption
 * [Boolean] **DeployGlobalParams** - indicates whether deploy Global Parameters of ADF. Nothing happens when parameters are not defined. (default: *true*)
 * [Boolean] **FailsWhenConfigItemNotFound** - indicates whether configuration items not found fails the script. (default: *true*)
 * [Boolean] **FailsWhenPathNotFound** - indicates whether missing paths fails the script. (default: *true*)
+* [Boolean] **DoNotStopStartExcludedTriggers** - specifies whether excluded triggers will be stopped before deployment (default: *false*)
+* [Boolean] **DoNotDeleteExcludedObjects** - specifies whether excluded objects can be removed. Applies when `DeleteNotInSource` is set to *True* only. (default: *true*) 
+
 
 Subsequently, you can define the needed options:
 
@@ -191,11 +196,13 @@ Therefore, an extra character should be provided before the name/pattern:
 trigger.*
 -*.SharedIR*
 -*.LS_SqlServer_DEV19_AW2017
+-*.*@testFolder
 ```
 
 The above file (if used) adds:
 - 2 items to *Includes* list (line 1-2)
 - 2 items to *Excludes* list (line 3-4)
+- all items located in `testFolder` to *Excludes* list (line 5)
 
 > The file should use UTF-8 encoding.
 
@@ -435,8 +442,10 @@ If you prefer using JSON rather than CSV for setting up configuration - JSON fil
 
 
 ## Step: Stoping triggers
-This block stops all triggers which must be stopped due to deployment.
-> Operation might be skipped when `StopStartTriggers = false` in *Publish Options*
+This block stops all triggers which must be stopped due to deployment.  
+Since version 0.30 you can better control which triggers you want to omit from stopping. Only need to add such triggers to `Excludes` list and set flag `DoNotStopStartExcludedTriggers` to *true*.
+
+> The step might be skipped when `StopStartTriggers = false` in *Publish Options*
 
 ## Step: Deployment of ADF objects
 This step is actually responsible for doing all the stuff.
@@ -445,13 +454,61 @@ The mechanism is smart enough to publish all objects in the right order, thence
 
 ## Step: Deleting objects not in source
 This process removes all objects from ADF service whom couldn't be found in the source (ADF code).  
-The mechanism is smart enough to dropping the objects in right order.
+The mechanism is smart enough to dropping the objects in right order.  
+Since version 0.30 you can better control which objects you want to omit from removing. Only need to add such objects to `Excludes` list and set flag `DoNotDeleteExcludedObjects` to *true*. 
 
-> Operation might be skipped when `DeleteNotInSource = false` in *Publish Options*
+> The step might be skipped when `DeleteNotInSource = false` in *Publish Options*
 
 ## Step: Restarting all triggers
 Restarting all triggers that should be enabled.
-> Operation might be skipped when `StopStartTriggers = false` in *Publish Options*
+> The step might be skipped when `StopStartTriggers = false` in *Publish Options*
+
+# Selective deployment, triggers and logic
+
+Publishing only selected objects of ADF is not an easy thing. If you add dependencies between objects and a need of stopping triggers before deploying on top of that - the situation becomes even more difficult. Therefore, not always it might be obvious what would happen during the deployment while you have flags set up, an object exist (or not) in the source and/or a trigger is Enabled (or Disabled) on target ADF service where you deploy to.  
+All these factors:
+* Does object exist in the source?
+* Is trigger is Enabled in the target?
+* What is the value for `DeleteNotInSource` flag?
+* What is the value for `StopStartTriggers` flag?
+* What is the value for `DoNotStopStartExcludedTriggers` flag?
+* What is the value for `DoNotDeleteExcludedObjects` flag?
+* Is an object on `Excludes` list?  
+
+had to be considered thoroughly, hence the following table arose:
+![Matrix of behaviour logic](./matrix-of-behaviour-logic.png)
+
+## Assumptions
+
+It's worth to explain a bit why the behaviour looks as above.
+
+### StopStartTriggers 
+It allows you to decide. It stops ALL existing and `Started` (Enabled) triggers.  
+Generally you should use TRUE (default) if you want to avoid troubles.  
+However, if you choose FALSE - you must accept that process WILL NOT touch triggers. You can still do it for yourself in pre or post-deployment script.
+
+### DoNotStopStartExcludedTriggers
+In some scenarios, people still want to let process to manage of triggers, but with some exceptions.
+Usage of this flag make sense along with `StopStartTriggers` only and some objects are excluded from deployment.
+It guarantees that no objects (excluded) will be even touched in target service during the deployment.
+
+### Excluded (collection) 
+This option should allow to completely separate group of objects from other group.
+In that way you can keep all files in the source but control behaviour with flags.
+Exclusion can be defined in two ways:
+1) Explicitly in 'Excludes' collection
+2) Implicitly by not adding an item in 'Includes' collection
+
+**Important:** Both collections can use wildcards as the name of objects.  
+It simplify defining multiple objects by one line, or defining objects which does not exist yet.
+These collections are very useful while you want to do a selective deployment.
+
+### DoNotDeleteExcludedObjects
+Excluded object can be deleted only if flag `DoNotDeleteExcludedObjects` = false.
+This situation could cause error when trigger has not been deleted prior.
+Otherwise, nothing will happen.  
+This option gives you a flexibility of deleting objects in the target, but still not touching objects from 'other' group.
+
 
 # Publish from Azure DevOps
 

azure.datafactory.tools.psd1

@@ -4,14 +4,23 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [0.30.0] - 2021-01-08
+### Added
+* Better control of which: (see new publish flags below)
+  * triggers could be stop/start
+  * objects could be removed (if apply)
+* Publish flag: `DoNotDeleteExcludedObjects` ([#47](https://github.com/SQLPlayer/azure.datafactory.tools/issues/47))
+* Publish flag: `DoNotStopStartExcludedTriggers` allows stopping selected triggers ([#51](https://github.com/SQLPlayer/azure.datafactory.tools/issues/51))
+* Numbers to all error messages ([full list](./en-us/messages_index.md))
+
 ## [0.20.0] - 2020-12-28
 ### Fixed
 * Do not start trigger which has not been deployed and thus does not exist (#51)
 
 ## [0.19.0] - 2020-12-23
 ### Added
 * Support wildcard when specifying object(s) name in config file (#58)
-* Added option $IgnoreLackOfReferencedObject (#64)
+* Added option `$IgnoreLackOfReferencedObject` (#64)
 * Add object name to the msg before action (#49)
 * Exit publish cmd when ADF name is already in use (#43)
 * Allow selecting objects in given folder (#68)

changelog.md

@@ -1,19 +1,39 @@
-# Messages:
-ADFTmxxx
-m = module/feature
-0 - Generic
+# Messages 
 
+> ADFTmxxx  
+m = module/feature  
+0 - Generic  
 xxx - number of error
 
-## Index 
-ADFT0005: Referenced object [$name] was not found.
-ADFT0006: Referenced object [$name] was not found. No error raised as user wanted to carry on.
 
-ADFT0007: Could not find object: $type.$name
-ADFT0008: Could not find object: $type.$name
-ADFT0009: Body of the object is empty!
-ADFT0010: Wrong path defined in config for object(path): $type.$name(properties.$path)"
-ADFT0011: adf.Location property has not been provided.
+## Index of errors
 
+Error number | Message
+-- | --
+ADFT0005 | Referenced object [*name*] was not found.
+ADFT0006 | Referenced object [*name*] was not found. No error raised as user wanted to carry on.
+ADFT0007 | Could not find object: *type.name*
+ADFT0008 | Could not find object: *type.name*
+ADFT0009 | Body of the object is empty!
+ADFT0010 | Wrong path defined in config for object(path): *type.name*(*properties.path*)"
+ADFT0011 | adf.Location property has not been provided.
+ADFT0012 | Deployment for this kind of Integration Runtime is not supported yet.
+ADFT0013 | Type *type* is not supported.
+ADFT0014 | Type *type* is not supported.
+ADFT0015 | Type *type* is not supported.
+ADFT0016 | Type '*type*' is not supported.
+ADFT0017 | Object [*name*] could not be found.
+ADFT0018 | Type *type* is not supported.
+ADFT0019 | Folder '*RootFolder*' doesn't exist.
+ADFT0020 | The header of config file is wrong. The header must have only 4 columns named: type, name, path, value.
+ADFT0021 | Config file, row x : Value in column 'Type' is empty.
+ADFT0022 | Config file, row x : Type *type* is not supported.
+ADFT0023 | Config file, row x : Value in column 'Name' is empty.
+ADFT0024 | Config file, row x : Value in column 'Path' is empty.
+ADFT0025 | Config file, row x has too many columns.
+ADFT0026 | File does not exist: *FilterFilePath*
+ADFT0027 | Azure Data Factory instance does not exist.
+ADFT0028 | Expected format of name for 'FullName' input parameter is: objectType.objectName[@folderName]
+ADFT0029 | Unknown object type: *type*.
 
 

en-us/messages_index.md

@@ -8,4 +8,6 @@ class AdfPublishOption {
     [Boolean] $FailsWhenConfigItemNotFound = $true
     [Boolean] $FailsWhenPathNotFound = $true
     [Boolean] $IgnoreLackOfReferencedObject = $false
+    [Boolean] $DoNotStopStartExcludedTriggers = $false
+    [Boolean] $DoNotDeleteExcludedObjects = $true
 }

matrix-of-behaviour-logic.png

@@ -1,25 +1,93 @@
 class AdfObjectName {
     [string] $Name
     [string] $Type
+    [string] $Folder
 
     AdfObjectName ([string] $Name, [string] $Type) 
     {
         $this.Name = $Name
         $this.Type = $Type
     }
 
+    AdfObjectName ([string] $Name, [string] $Type, [string] $Folder) 
+    {
+        $this.Name = $Name
+        $this.Type = $Type
+        $this.Folder = $Folder
+    }
+
     AdfObjectName ([string] $FullName) 
     {
-        if ($FullName.IndexOf('.') -lt 1) {
-            Write-Error "Expected format of name for 'FullName' input parameter is: objectType.objectName"
+        $m = [regex]::matches($FullName, '([a-zA-Z]+)\.([a-zA-Z 0-9\-_]+)@?(.*)')
+        if ($m.Success -eq $false) {
+            throw "ADFT0028: Expected format of name for 'FullName' input parameter is: objectType.objectName[@folderName]"
         }
-        $parts = $FullName.Split('.')
-        if ($parts[0] -notin [AdfObject]::allowedTypes ) { 
-            Write-Error -Message "Unknown object type: $parts[0]."
+        if ($m.Groups[1].Value -notin [AdfObject]::allowedTypes ) { 
+            throw "ADFT0029: Unknown object type: $($m.Groups[1].Value)."
         }
-        $this.Type = $parts[0]
-        $this.Name = $parts[1]
+        $this.Type = $m.Groups[1].Value
+        $this.Name = $m.Groups[2].Value
+        $this.Folder = $m.Groups[3].Value
     }
 
-}
+    [String] FullName ([boolean] $quoted)
+    {
+        if ($quoted) {
+            return "[$($this.Type)].[$($this.Name)]"
+        } else {
+            return "$($this.Type).$($this.Name)"
+        }
+    }
+
+    [String] FullNameWithFolder ()
+    {
+        if ($this.Folder.Length -gt 0) {
+            return "$($this.Type).$($this.Name)@$($this.Folder)"
+        } else {
+            return "$($this.Type).$($this.Name)"
+        }
+    }
+
+    [String] FullName ()
+    {
+        return $this.FullName($false)
+    }
+
+    [String] FullNameQuoted ()
+    {
+        return $this.FullName($true)
+    }
 
+    [Boolean] IsNameMatch ([array]$wildcardPatterns)
+    {
+        $fullname = $this.FullName()
+        $r = $wildcardPatterns | Where-Object { $fullname -like $_ }
+        return $null -ne $r
+    }
+
+    [Boolean] IsNameExcluded ([AdfPublishOption] $opt)
+    {
+        $fullname = $this.FullNameWithFolder()
+
+        # One can exclude objects by listing them explicitly in Excludes collection, ...
+        $excPatterns = $opt.Excludes.Keys
+        $r = $excPatterns | Where-Object { $fullname -like $_ }
+        if ($null -ne $r) 
+        { 
+            # Means: object is excluded if matches any item in (Excludes) collection
+            return $true    
+        }
+
+        # ... or by listing them implicitly in Includes collection:
+        $incPatterns = $opt.Includes.Keys
+        if ($incPatterns.Count -eq 0) 
+        { 
+            # If no items = all objects match => object is not excluded
+            return $false 
+        }
+        $r = $incPatterns | Where-Object { $fullname -like $_ }
+        # Means: object is excluded if not match any item in (Includes) collection
+        return ($null -eq $r)   
+    }
+
+}

private/!AdfPublishOption.class.ps1

@@ -73,7 +73,7 @@ function Deploy-AdfObjectOnly {
                 -Force | Out-Null
             }
             else {
-                Write-Error "Deployment for this kind of Integration Runtime is not supported yet."
+                Write-Error "ADFT0012: Deployment for this kind of Integration Runtime is not supported yet."
             }
         }
         'linkedService'
@@ -140,7 +140,7 @@ function Deploy-AdfObjectOnly {
         }
         default
         {
-            Write-Error "Type $($obj.Type) is not supported."
+            Write-Error "ADFT0013: Type $($obj.Type) is not supported."
         }
     }
 

private/AdfObjectName.class.ps1

@@ -41,7 +41,7 @@ function Get-AdfObjectByName {
         }
         default
         {
-            Write-Error "Type [$type] is not supported."
+            Write-Error "ADFT0014: Type [$type] is not supported."
         }
     }
 

private/Deploy-AdfObjectOnly.ps1

@@ -41,7 +41,7 @@ function Get-AdfObjectByPattern {
         }
         default
         {
-            Write-Error "Type [$type] is not supported."
+            Write-Error "ADFT0015: Type [$type] is not supported."
         }
     }