tokenize-ps3.ps1 does not work with environment variables

[danko-d]

In tokenize-ps3.ps1 it is no longer possible to use environment variables as replacement strings.

It was possible in tokenize.ps1 thanks to the following lines:

 if (Test-Path env:$matchedItem) {
          $variableValue = (get-item env:$matchedItem).Value
          Write-Verbose "Found custom variable '$matchedItem' in build or release definition with value '$variableValue'" 
 }

[TheBeardedLlama]

update: never mind, it was a red herring in my case...

I think I've hit the same problem.
Did you get anywhere with it @danko-d ?

[aolszowka]

This is an old thread; but the repository has not been abandoned yet and is one of the top results in Google.

They appear to have subtly changed how the environment variables are grabbed and this has changed the behavior of this script. In tokenize-ps3.ps1 they actually use Get-VstsTaskVariableInfo to grab the variables as seen here:

Extension-UtilitiesPack/Utilites/Tokenizer/tokenize-ps3.ps1

Line 22 in 4747cae

$allVars = ArrayToHash (Get-VstsTaskVariableInfo)

This has the unfortunate side effect of only grabbing the Job Variables as per the documentation for Get-VstsTaskVariableInfo.

You can see this behavior by setting up this toy pipeline:

sourcefile.json

{
    "FirstName": "__FIRST_NAME__",
    "Unknown": "__UNKNOWN_VARIABLE__"
}

azure-pipeline.yml

trigger: none

pool:
  vmImage: windows-latest

variables: # These are Job Variables
- name: FIRST_NAME
  value: Luke

steps:
- task: CmdLine@2
  displayName: 'Echo Out the Source File'
  inputs:
    script: |
      type sourcefile.json

- task: ms-devlabs.utilitytasks.task-tokenizer.Tokenizer@2
  displayName: 'Tokenize File'
  inputs:
    SourcePath: 'sourcefile.json'
  env:
    FIRST_NAME: 'Paul' # This is not respected

- task: CmdLine@2
  displayName: 'Echo Out the Source File (After Modifications)'
  inputs:
    script: |
      type sourcefile.json

And here are the screenshots showing this "working":
Showing an echo of the unmodified file:

Showing that the task found the variable for replacement:

Showing that Luke is the replacement:

Unexpectedly (at least if you don't understand the documentation) Luke is printed instead of Paul.

This causes issues with attempting to use a KeyVault Secret as you MUST explicitly set these as Environment Variables as per the documentation:

Unlike a normal variable, they are not automatically decrypted into environment variables for scripts. You need to explicitly map secret variables.

The following example shows how to use a secret variable called mySecret in PowerShell and Bash scripts. Unlike a normal pipeline variable, there's no environment variable called MYSECRET.

variables:
 GLOBAL_MYSECRET: $(mySecret) # this will not work because the secret variable needs to be mapped as env
 GLOBAL_MY_MAPPED_ENV_VAR: $(nonSecretVariable) # this works because it's not a secret.
 
 steps:

- powershell: |
    Write-Host "Using an input-macro works: $(mySecret)"
    Write-Host "Using the env var directly does not work: $env:MYSECRET"
    Write-Host "Using a global secret var mapped in the pipeline does not work either: $env:GLOBAL_MYSECRET"
    Write-Host "Using a global non-secret var mapped in the pipeline works: $env:GLOBAL_MY_MAPPED_ENV_VAR" 
    Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
  env:
    MY_MAPPED_ENV_VAR: $(mySecret) # the recommended way to map to an env variable

I have a few ideas that I'm willing to try and will respond back with my results.

[aolszowka]

Sorry for the double post; but I wanted to come back with that I also got this working with KeyVault Secrets. The key to getting this to work is to tell the KeyVault task to RunAsPreJob which according to the documentation will: Make secrets available to whole job. I suspect this is equivalent to setting a Job level variable.

I utilized the secrets filter to ensure we only pull the secret we care about.

The following snippet works:

sourcefile.json

{
    "MySecret": "__My-Secret__",
    "Unknown": "__UNKNOWN_VARIABLE__"
}

azure-pipeline.yml

trigger: none

pool:
  vmImage: windows-latest

steps:
- task: AzureKeyVault@2
  displayName: 'Azure Key Vault: MyKeyVault'
  inputs:
    azureSubscription: 'SPN'
    KeyVaultName: 'MyKeyVault'
    secretsFilter: 'My-Secret'
    RunAsPreJob: true

- task: CmdLine@2
  displayName: 'Echo Out the Source File'
  inputs:
    script: |
      type sourcefile.json

- task: ms-devlabs.utilitytasks.task-tokenizer.Tokenizer@2
  displayName: 'Tokenize File'
  inputs:
    SourcePath: 'sourcefile.json'

- task: CmdLine@2
  displayName: 'Echo Out the Source File (After Modifications)'
  inputs:
    script: |
      type sourcefile.json

Note that when looking at the output in the console the secret will be redacted in the logs due to the behavior called out in the documentation:

Specifically:

We make an effort to mask secrets from appearing in Azure Pipelines output, but you still need to take precautions. Never echo secrets as output. Some operating systems log command line arguments. Never pass secrets on the command line. Instead, we suggest that you map your secrets into environment variables.

We never mask substrings of secrets. If, for example, "abc123" is set as a secret, "abc" isn't masked from the logs. This is to avoid masking secrets at too granular of a level, making the logs unreadable. For this reason, secrets should not contain structured data. If, for example, "{ "foo": "bar" }" is set as a secret, "bar" isn't masked from the logs.