diff --git a/fabric/fabric_cicd_gitlab/README.md b/fabric/fabric_cicd_gitlab/README.md index 47d079068..e08384640 100644 --- a/fabric/fabric_cicd_gitlab/README.md +++ b/fabric/fabric_cicd_gitlab/README.md @@ -2,20 +2,75 @@ ## Introduction -This code sample provides an advanced mechanism for implementing a CI/CD process with Microsoft Fabric, accomodating multi-tenancy and generic-git deployments. While the sample currently has some [limitations](#known-limitations), the goal is to iteratively enhance its capabilities in alignment with Fabric’s advancements. +The code in this sample showcases ways to mimic the Microsoft Fabric [Git Integration](https://learn.microsoft.com/fabric/cicd/git-integration/intro-to-git-integration?tabs=azure-devops) functionality, accommodating multi-tenancy and generic-git deployments for currently unsupported scenarios. While the sample currently has some [limitations](#known-limitations), the goal is to iteratively enhance its capabilities in alignment with Fabric’s advancements. This sample is recommended if: -- your organization adopts multi-tenancy in their CI/CD processes, such as the different environments (such as Development, Staging and Production) are on different Microsoft Entra IDs. -- your organization's preferred git tool is today not yet supported by Fabric (i.e. such as GitLab, and Bitbucket). +- Your organization adopts multi-tenancy in their CI/CD processes where different environments (such as Development, Staging, and Production) are on different Microsoft Entra IDs. +- Your organization's preferred git provider is not yet supported by Microsoft Fabric (e.g. GitLab, Bitbucket). For more details, review the official and up-to-date list of [supported Git providers in Microsoft Fabric](https://learn.microsoft.com/fabric/cicd/git-integration/intro-to-git-integration?tabs=azure-devops). If none of the scenarios above match your current situation, consider using the [Fabric CI/CD sample for Azure DevOps](../fabric_ci_cd/README.md). +## Contents of This Sample + +This sample showcases two approaches: + +- A simplified way for tracking Microsoft Fabric items with source control. +- A more advanced example attempting to provide a comprehensive solution for CI/CD in unsupported scenarios. + +### Simplified Approach + +The simplified approach is suited for scenarios where multiple developers might be working on the same Fabric workspace and need to selectively download and store their work in source control, one Fabric item at a time. + +If you are interested in this approach, follow the [instructions for the simplified approach](./docs/simplified_approach.md). + +### Comprehensive Approach + +The comprehensive approach is suited for scenarios where each developer is working on different workspaces, and it is acceptable to have a one-to-one mapping between feature branches and Fabric workspaces. + +If you are interested in this approach, follow the [instructions for the comprehensive approach](./docs/full_cicd_approach.md). + +## Known limitations + +- Microsoft Information Protection labels are enforced by Fabric but there is still not a way to set MIP labels via APIs. When MIP labels are defaulting to Restricted/Confidential, then some of the API calls in the below scripts might fail. +- Service Principal authentication is currently not supported by Fabric APIs (See [Microsoft Documentation](https://learn.microsoft.com/rest/api/fabric/articles/using-fabric-apis#considerations-and-limitation)), therefore this sample is currently relying on user tokens. See the below instructions for generating a valid user token. + +### Generating a Fabric Bearer Token + +Service Principal and Managed Identity Authentication is currently supported by some Fabric REST APIs. When such authentication is not available, REST APIs need to be executed with user context, using a user token. Such user token is valid for one hour and needs to be refreshed after that. There are several ways to generate the token: + +- **(Recommended) Using a bash script**: Using the [`refresh_api_token.sh`](./src/refresh_api_token.sh) bash script. This script also supports generating the token with SPN/MI Authentication, using the `use_spn` parameter. + + ```bash + ./src/refresh_api_token.sh + ``` + +- **Using PowerShell**: The token can be generated by using the following PowerShell command: + + ```powershell + [PS]> Connect-PowerBIServiceAccount + [PS]> Get-PowerBIAccessToken + ``` + +- **Using Edge browser devtools (F12)**: If you are already logged into Fabric portal, you can invoke the following command from the Edge Browser DevTools (F12) console: + + ```sh + > copy(PowerBIAccessToken) + ``` + + This will copy the token to your clipboard. You can then paste its value in the `.env` file. + +## Roadmap + +- Triggering hydration of Lakehouse (via Data Pipelines and Notebook). + +## Comprehensive CI/CD Approach + ## How Does It Work? ### Using REST APIs for creating/updating Fabric items -Currently, Microsoft Fabric supports Git integration for Azure DevOps only. For this reason, this sample presents a way to use [Fabric REST APIs](https://learn.microsoft.com/rest/api/fabric/articles/using-fabric-apis) to integrate with other GIT source control mechanisms beyond Azure devOps. A brief summary of the steps involved are: +Your chosen Git provider might not be in the [list of supported Git providers](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration?tabs=azure-devops#supported-git-providers) for Microsoft Fabric. If this is the case and you want a comprehensive solution to track your work with source control, this sample presents a way to use [Fabric REST APIs](https://learn.microsoft.com/rest/api/fabric/articles/using-fabric-apis) to integrate with other Git source control mechanisms beyond the supported list. A brief summary of the steps involved are: 1. Creating a Fabric workspace (if not existing already) 2. Creating/updating Fabric items in the workspace to reflect what is on the developer branch. @@ -32,7 +87,7 @@ This approach assumes that the developer will operate by following the recommend This sample maintains a record of changes to Fabric items in source control to prevent the need for constant deletion and recreation of modified items. It does this by tracking the *Object Id*s (the GUIDs of the items in the Fabric workspace, as per the Fabric REST APIs) in an `item-config.json` configuration file. -All Fabric items come with a minimal definition (at the time of writing comprising of Name, Type and Description). Such minimal defintion is stored in the `item-metadata.json` file. +All Fabric items come with a minimal definition (at the time of writing comprising of Name, Type and Description). Such minimal definition is stored in the `item-metadata.json` file. Certain types of items in Fabric can have an [item definition](https://learn.microsoft.com/rest/api/fabric/articles/item-management/definitions/item-definition-overview). You can get this definition by using the [`getDefinition`](https://learn.microsoft.com/rest/api/fabric/core/items/get-item-definition) API. The result of this API call is saved in a file called `item-definition.json`. @@ -70,15 +125,15 @@ This sample uses PowerShell scripts to automate the CI/CD process. Below, you’ | Script/File | Description | |--------|-------------| -|[params.psd1](./src/params.psd1)|Parameters file - used to store the values of input arguments to the scripts. Update the values as needed.| +|[.env](./config/.envtemplate)|Environment variables file - used to store the parameter values required by the scripts. Update the values as needed.| |[update_from_git_to_ws.ps1](./src/update_from_git_to_ws.ps1)|Script to create a Fabric workspace (if non existing) and sync assets from source control (local git branch) to the workspace.| -|[update_from_ws_to_git.ps1](./src/update_from_ws_to_git.ps1)|Script to update the local repository from the item defintions in the Fabric workspace.| +|[update_from_ws_to_git.ps1](./src/update_from_ws_to_git.ps1)|Script to update the local repository from the item definitions in the Fabric workspace.| -> Note: to avoid committing secrets to your remote branch, make sure to ignore changes to the local version of your `params.psd1` file. +> Note: to avoid committing secrets to your remote branch, make sure to ignore changes to the local version of your `.env` file. ### Understanding The DevOps Pipelines -The [DevOps Pipelies README](./devops/README.md) provides a comprehensive explanation of the functionality of the DevOps Pipelines showcased in this example. +The [DevOps Pipelines README](./devops/README.md) provides a comprehensive explanation of the functionality of the DevOps Pipelines showcased in this example. ## Set-up Instructions @@ -86,6 +141,7 @@ The [DevOps Pipelies README](./devops/README.md) provides a comprehensive explan - Powershell version 7+ - Local IDE with `git` command installed. +- A `bash` shell - A DevOps source control system, like Azure DevOps or GitLab. - A Fabric tenant with at least one capacity running. - If you don't have a Fabric tenant you can create a [Fabric trial](https://learn.microsoft.com/fabric/get-started/fabric-trial) and use a trial capacity instead. @@ -109,37 +165,73 @@ The below picture illustrates these followed by a description of each of the num ![Fabric CI/CD Architecture](./images/architecture_and_workflow.png) -**Step 0. Prepare for local development**: +**Step 0. Update environment variables and prepare for local development**: -- In a `bash` shell run the following command. Replace `feat/feat_1` with your feature branch name and `dev` with you main development branch name. This commands creates a new feature branch locally from `dev` (or the specified branch). It then instructs git to disregard local modifications to any new or existing `item-config.json` file. The purpose of this is to prevent the `objectId`s in the `dev` branch from being replaced by the `objectId`s from the developer workspace during a commit. For more information see the [Source Control Mechanisms for Fabric Items](#source-control-mechanism-for-fabric-items) section. +- Verify that all the prerequisite software is installed and available in your system’s PATH. +- Create a copy of the [`.envtemplate`](./config/.envtemplate) file and rename it to `.env` - ```sh - .\new_branch.sh feat/feat_1 dev - ``` + ```bash + cp ./config/.envtemplate ./config/.env + ``` -> Note: this approach will also work if no Fabric assets are present on your branch, but you will still need a folder for storing Fabric items definitions later. +- Edit the newly created `.env` file, filling the required parameters (`TENANT_ID`, `ITEMS_FOLDER` and `FABRIC_CAPACITY_ID`) +- Load environment variables -**Step 1. Create/Update Fabric workspace and create Fabric items from local branch**: + ```bash + source config/.env + ``` -- Update the `params.psd1` file as needed. If you need to generate a new token refer to the [Generating a Fabric Bearer Token](#generating-a-fabric-bearer-token) section. Load the values of the parameters file as follows: +- Login to Azure CLI with your user or SPN/MI, use one of the below approaches: - ```pwsh - $config = Import-PowerShellDataFile .\src\params.psd1 + ```bash + # User Login + az login --use-device-code -t $TENANT_ID + ``` + + or + + ```bash + # SPN Login + az login --service-principal -u $APP_CLIENT_ID -p $APP_CLIENT_SECRET --tenant $TENANT_ID --allow-no-subscription + ``` + + or + + ```bash + # MI Login + az login --identity + ``` + +- Refresh the Fabric token: + + ```bash + ./src/refresh_api_token.sh + ``` + +- In a `bash` shell run the following command. Replace `feat/feat_1` with your feature branch name and `dev` with your main development branch name. This commands creates a new feature branch locally from `dev` (or the specified branch). It then instructs git to disregard local modifications to any new or existing `item-config.json` file. The purpose of this is to prevent the `objectId`s in the `dev` branch from being replaced by the `objectId`s from the developer workspace during a commit. For more information see the [Source Control Mechanisms for Fabric Items](#source-control-mechanism-for-fabric-items) section. + + ```bash + ./new_branch.sh feat/feat_1 dev ``` +> Note: this approach will also work if no Fabric assets are present on your branch, but you will still need a folder for storing Fabric items definitions later. + +**Step 1. Create/Update Fabric workspace and create Fabric items from local branch**: + - Run the [`update_from_git_to_ws.ps1`](./src/update_from_git_to_ws.ps1) script from the local repository folder. This step will create a new workspace and mirror what is on the repo to the workspace. > **CAUTION: Workspace items that are not in the local branch will be deleted from Fabric workspace.** - When running this for the first time on a new branch, utilize the `-resetConfig` setting it to `$true`. This ignores any existing `item-config.json` files and creates corresponding new objects in the workspace. This step is crucial as it prevents the script from failing due to a search for `objectId`s that are coming from the `dev` branch/workspace, which would not exist in the new Fabric workspace. - ```pwsh - .\src\update_from_git_to_ws.ps1 -baseUrl $config.baseUrl -fabricToken $config.fabricToken -workspaceName $config.workspaceName -capacityId $config.capacityId -folder $config.folder -resetConfig $true + ```bash + pwsh + ./src/update_from_git_to_ws.ps1 -workspaceName "" -resetConfig $true ``` - All other times you can omit the flag `-resetConfig` (it will default to `$false`). ```pwsh - .\src\update_from_git_to_ws.ps1 -baseUrl $config.baseUrl -fabricToken $config.fabricToken -workspaceName $config.workspaceName -capacityId $config.capacityId -folder $config.folder + ./src/update_from_git_to_ws.ps1 -workspaceName "" ``` **Step 2. Develop in the Fabric workspace**: @@ -148,16 +240,17 @@ The below picture illustrates these followed by a description of each of the num **Step 3. Sync the local branch with Fabric workspace**: -- Once you are ready to commit your changes to your branch, run the [`update_from_ws_to_git.ps1`](./src/update_from_ws_to_git.ps1). The script will update your local branch mirroring what is in your Fabric workspace. This creates/updates folders in the `$config.folder` on your local branch. For more information on folder structure see the [Fabric Items and Source Control](#fabric-items-and-source-control) section. +- Once you are ready to commit your changes to your branch, run the [`update_from_ws_to_git.ps1`](./src/update_from_ws_to_git.ps1). The script will update your local branch mirroring what is in your Fabric workspace. This creates/updates folders in the `$config.folder` on your local branch. For more information on folder structure see the [Source control mechanism for Fabric items](#source-control-mechanism-for-fabric-items) section. > **CAUTION: local branch items that are not in the workspace will be deleted from local branch.** ```pwsh - .\\update_from_ws_to_git.ps1 -baseUrl $config.baseUrl -fabricToken $config.fabricToken -workspaceName $config.workspaceName -capacityId $config.capacityId -folder $config.folder + ./src/update_from_ws_to_git.ps1 -workspaceName "" ``` **Step 4. Repeat until done**: - Iterate between step 1 to step 3 as needed. +- If your token has expired, use the `refresh_api_token.sh` script to store a new token in the environment file. **Step 5. Push changes and create a PR**: @@ -180,43 +273,15 @@ The below picture illustrates these followed by a description of each of the num ## Common errors -- `Error reponse: Response status code does not indicate success: 400 (Bad Request)` - - Likely one of the Fabric items you are trying to update has a defintion containing errors. We advise maintaining identifiers such as workspaceIds and LakehouseIds as **parameters** for Notebooks and Data Pipelines. Failure to do this may result in the script being unable to push your item definitions if they reference item ids that have been removed. -- `Error reponse: Response status code does not indicate success: 401 (Unauthorized)` +- `Error response: Response status code does not indicate success: 400 (Bad Request)` + - Likely one of the Fabric items you are trying to update has a definition containing errors. We advise maintaining identifiers such as workspaceIds and LakehouseIds as **parameters** for Notebooks and Data Pipelines. Failure to do this may result in the script being unable to push your item definitions if they reference item ids that have been removed. +- `Error response: Response status code does not indicate success: 401 (Unauthorized)` - Likely your user token has expired. Update it and source your params file and then try again. - If using Azure DevOps: if you are getting this error when running devops pipelines after refreshing the token variable, make sure you have toggled the secret variable type. -- `Error reponse: Response status code does not indicate success: 403 (Forbidden)` +- `Error response: Response status code does not indicate success: 403 (Forbidden)` - Likely one of the Fabric items you are trying to update has a MIP label that prevents you from updating its definition. -- `Error reponse: Response status code does not indicate success: 404 (Not Found)` +- `Error response: Response status code does not indicate success: 404 (Not Found)` - Likely the capacity you specified cannot be used. Make sure that your Fabric capacity is available and running, it might have been paused, in this case you can resume it and re-run the script. -- `Error reponse: A parameter cannot be found that matches parameter name 'ResponseHeadersVariable'` +- `Error response: A parameter cannot be found that matches parameter name 'ResponseHeadersVariable'` - Likely you need to update your Powershell version to 7+. - -## Known limitations - -- Microsoft Information Protection labels are enforced by Fabric but there is still not a way to set MIP labels via APIs. When MIP labels are defaulting to Restricted/Confidential, then some of the API calls in the below scripts might fail. -- Service Principal authentication is currently not supported by Fabric APIs (See [Microsoft Documentation](https://learn.microsoft.com/rest/api/fabric/articles/using-fabric-apis#considerations-and-limitation)), therefore this sample is currently relying on user tokens. See the below instructions for generating a valid user token. - -### Generating a Fabric Bearer Token - -Until Service Principal Authentication will be supported by Fabric APIs, a user token has to be manually generated and passed to the script as a variable. Such user token is valid for one hour and needs to be refreshed after that. There are several ways to generate the token: - -- **Using PowerShell**: The token can be generated by using the following PowerShell command: - - ```powershell - [PS]> Connect-PowerBIServiceAccount - [PS]> Get-PowerBIAccessToken - ``` - -- **Using Edge browser devtools (F12)**: If you are already logged into Fabric portal, you can invoke the following command from the Edge Browser DevTools (F12) console: - - ```sh - > copy(PowerBIAccessToken) - ``` - - This will copy the token to your clipboard. You can then paste its value in the `params.psd1` file. - -## Roadmap - -- Triggering hydration of Lakehouse (via Data Pipelines and Notebook). diff --git a/fabric/fabric_cicd_gitlab/config/.envtemplate b/fabric/fabric_cicd_gitlab/config/.envtemplate new file mode 100644 index 000000000..bd4c13746 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/config/.envtemplate @@ -0,0 +1,10 @@ +# Environment variables of this sample + +# Variables Required by simplified and comprehensive approach +TENANT_ID="" # Your Entra Tenant ID, used to authenticate and grab a user token +FABRIC_API_BASEURL="https://api.fabric.microsoft.com/v1" # the base URL of the Fabric APIs +FABRIC_USER_TOKEN="" # the Fabric user token used to authenticate against the Fabric APIs + +# Variables required only for the comprehensive approach +ITEMS_FOLDER=fabric # the folder name (existing) where Fabric item files will be stored +FABRIC_CAPACITY_ID="" # the capacity id (existing) of the capacity where the Fabric item will be created. Can be retrieved from Fabric Admin portal. \ No newline at end of file diff --git a/fabric/fabric_cicd_gitlab/docs/full_cicd_approach.md b/fabric/fabric_cicd_gitlab/docs/full_cicd_approach.md new file mode 100644 index 000000000..8c091f09a --- /dev/null +++ b/fabric/fabric_cicd_gitlab/docs/full_cicd_approach.md @@ -0,0 +1,221 @@ +# Comprehensive CI/CD Approach + +## How Does It Work? + +### Using REST APIs for creating/updating Fabric items + +Your chosen Git provider might not be in the [list of supported Git providers](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration?tabs=azure-devops#supported-git-providers) for Microsoft Fabric. If this is the case and you want a comprehensive solution to track your work with source control, this sample presents a way to use [Fabric REST APIs](https://learn.microsoft.com/rest/api/fabric/articles/using-fabric-apis) to integrate with other Git source control mechanisms beyond the supported list. A brief summary of the steps involved are: + +1. Creating a Fabric workspace (if not existing already) +2. Creating/updating Fabric items in the workspace to reflect what is on the developer branch. +3. Working as needed in the Fabric workspace. +4. Updating changes from workspace to reflect them back in source control. + +> **Note 1**: This sample follows a strategy where each feature branch is paired with a corresponding Fabric workspace, implementing a one-workspace-per-branch approach. +> +> **Note 2**: This example includes a set of Fabric items to demonstrate the functionality of the solution. You are welcome to substitute these with your own items. In doing so, we advise maintaining identifiers such as workspaceIds and LakehouseIds as parameters for Notebooks and Data Pipelines. Failure to do this may result in the script being unable to push your item definitions if they reference item ids that have been removed. + +This approach assumes that the developer will operate by following the recommended workflow, as described below in the [Recommended Workflow](#recommended-workflow) section. + +### Source control mechanism for Fabric items + +This sample maintains a record of changes to Fabric items in source control to prevent the need for constant deletion and recreation of modified items. It does this by tracking the *Object Id*s (the GUIDs of the items in the Fabric workspace, as per the Fabric REST APIs) in an `item-config.json` configuration file. + +All Fabric items come with a minimal definition (at the time of writing comprising of Name, Type and Description). Such minimal definition is stored in the `item-metadata.json` file. + +Certain types of items in Fabric can have an [item definition](https://learn.microsoft.com/rest/api/fabric/articles/item-management/definitions/item-definition-overview). You can get this definition by using the [`getDefinition`](https://learn.microsoft.com/rest/api/fabric/core/items/get-item-definition) API. The result of this API call is saved in a file called `item-definition.json`. + +To make things easier for developers, the definition, which is encoded in base64 in the `item-definition.json` file, is also stored in the repository. For instance, if the item is a Notebook, a notebook in ipynb format is saved in the repository. This is done by decoding the base64 data that the `getDefinition` API returns. This allows the developer to edit the notebook using their favorite IDE. Any changes made are then saved back in Fabric when the corresponding Fabric item is updated using the [`updateDefinition`](https://learn.microsoft.com/rest/api/fabric/core/items/update-item-definition) API. + +All the files that compose a specific Fabric item are stored in a corresponding folder (`Item1Folder.ItemType` in the example below). + +All these individual item folders are then stored in one main folder, named `fabric` in this sample. + +### Understanding repository structure + +The below folder structure offers a visual representation of how this code sample will organize source control files in the repository: + +```fsys +/ (root of this project) +│ +└───... +│ +└───fabric +│ └───Item1Folder.ItemType +│ | │ item-config.json +│ | │ item-definition.json (optional) +│ | │ item-metadata.json +│ | │ (other optional files that may vary by item type) +│ | +│ └───Item2Folder.ItemType +│ │ ... +│ +└───... +``` + +### Understanding The PowerShell Scripts + +This sample uses PowerShell scripts to automate the CI/CD process. Below, you’ll find a brief overview of their purpose. + +| Script/File | Description | +|--------|-------------| +|[.env](../config/.envtemplate)|Environment variables file - used to store the parameter values required by the scripts. Update the values as needed.| +|[update_from_git_to_ws.ps1](../src/update_from_git_to_ws.ps1)|Script to create a Fabric workspace (if non existing) and sync assets from source control (local git branch) to the workspace.| +|[update_from_ws_to_git.ps1](../src/update_from_ws_to_git.ps1)|Script to update the local repository from the item definitions in the Fabric workspace.| + +> Note: to avoid committing secrets to your remote branch, make sure to ignore changes to the local version of your `.env` file. + +### Understanding The DevOps Pipelines + +The [DevOps Pipelines README](../devops/README.md) provides a comprehensive explanation of the functionality of the DevOps Pipelines showcased in this example. + +## Set-up Instructions + +### Pre-Requisites + +- Powershell version 7+ +- Local IDE with `git` command installed. +- A `bash` shell +- A DevOps source control system, like Azure DevOps or GitLab. +- A Fabric tenant with at least one capacity running. + - If you don't have a Fabric tenant you can create a [Fabric trial](https://learn.microsoft.com/fabric/get-started/fabric-trial) and use a trial capacity instead. + +### Setting Up Your GIT Repository + +To use this sample it is advisable that you: + +1. Create a brand new repository with your source control tool of choice. +2. Clone the entire repository locally to a directory of your choice. +3. Copy everything that is under [this sample's folder](./) to the directory from step 2. +4. Read remaining instructions. + +### Deployment Steps + +Create Build (CI) and Release (CD) pipelines from the YML definitions provided in this sample. To do so, refer to the information in the [DevOps pipeline readme](../devops/README.md). + +### Recommended Workflow + +The below picture illustrates these followed by a description of each of the numbered step: + +![Fabric CI/CD Architecture](../images/architecture_and_workflow.png) + +**Step 0. Update environment variables and prepare for local development**: + +- Verify that all the prerequisite software is installed and available in your system’s PATH. +- Create a copy of the [`.envtemplate`](../config/.envtemplate) file and rename it to `.env` + + ```bash + cp ./config/.envtemplate ./config/.env + ``` + +- Edit the newly created `.env` file, filling the required parameters (`TENANT_ID`, `ITEMS_FOLDER` and `FABRIC_CAPACITY_ID`) +- Load environment variables + + ```bash + source config/.env + ``` + +- Login to Azure CLI with your user or SPN/MI, use one of the below approaches: + + ```bash + # User Login + az login --use-device-code -t $TENANT_ID + ``` + + or + + ```bash + # SPN Login + az login --service-principal -u $APP_CLIENT_ID -p $APP_CLIENT_SECRET --tenant $TENANT_ID --allow-no-subscription + ``` + + or + + ```bash + # MI Login + az login --identity + ``` + +- Refresh the Fabric token: + + ```bash + ./src/refresh_api_token.sh + ``` + +- In a `bash` shell run the following command. Replace `feat/feat_1` with your feature branch name and `dev` with your main development branch name. This commands creates a new feature branch locally from `dev` (or the specified branch). It then instructs git to disregard local modifications to any new or existing `item-config.json` file. The purpose of this is to prevent the `objectId`s in the `dev` branch from being replaced by the `objectId`s from the developer workspace during a commit. For more information see the [Source Control Mechanisms for Fabric Items](#source-control-mechanism-for-fabric-items) section. + + ```bash + ./new_branch.sh feat/feat_1 dev + ``` + +> Note: this approach will also work if no Fabric assets are present on your branch, but you will still need a folder for storing Fabric items definitions later. + +**Step 1. Create/Update Fabric workspace and create Fabric items from local branch**: + +- Run the [`update_from_git_to_ws.ps1`](../src/update_from_git_to_ws.ps1) script from the local repository folder. This step will create a new workspace and mirror what is on the repo to the workspace. + > **CAUTION: Workspace items that are not in the local branch will be deleted from Fabric workspace.** + + - When running this for the first time on a new branch, utilize the `-resetConfig` setting it to `$true`. This ignores any existing `item-config.json` files and creates corresponding new objects in the workspace. This step is crucial as it prevents the script from failing due to a search for `objectId`s that are coming from the `dev` branch/workspace, which would not exist in the new Fabric workspace. + + ```bash + pwsh + ./src/update_from_git_to_ws.ps1 -workspaceName "" -resetConfig $true + ``` + + - All other times you can omit the flag `-resetConfig` (it will default to `$false`). + + ```pwsh + ./src/update_from_git_to_ws.ps1 -workspaceName "" + ``` + +**Step 2. Develop in the Fabric workspace**: + +- Work as needed in your Fabric workspace (you can find it in Fabric looking for the Workspace `$config.workspaceName`). + +**Step 3. Sync the local branch with Fabric workspace**: + +- Once you are ready to commit your changes to your branch, run the [`update_from_ws_to_git.ps1`](../src/update_from_ws_to_git.ps1). The script will update your local branch mirroring what is in your Fabric workspace. This creates/updates folders in the `$config.folder` on your local branch. For more information on folder structure see the [Source control mechanism for Fabric items](#source-control-mechanism-for-fabric-items) section. + > **CAUTION: local branch items that are not in the workspace will be deleted from local branch.** + + ```pwsh + ../src/update_from_ws_to_git.ps1 -workspaceName "" + ``` + +**Step 4. Repeat until done**: + +- Iterate between step 1 to step 3 as needed. +- If your token has expired, use the `refresh_api_token.sh` script to store a new token in the environment file. + +**Step 5. Push changes and create a PR**: + +- When happy with the changes, create a PR to merge the changes. + + > **CAUTION**: Make sure that when creating the PR no `item-config.json` files are pushed to `dev`. These files are created in each of the workspace item folder as part of Step 3. This file contains the *logical ids and object ids* to identify each of the assets. However, these vary from workspace to another hence these should not be checked in. + > + > **Note**: After the PR is merged it is recommended that the developer delete the feature branch. When, in the local development environment, the developer will switch back to the `dev` branch they will be warned that local changes to `.gitignore` and `item-config.json` files will be lost. There is no harm in doing this. + +**Step 6. Follow PR approval process to DEV workspace**: + +- When the PR is approved, devops Build and Release pipelines are triggered: + + 1. the Build pipeline checks that no `item-config.json` files are being pushed to `dev`. For more information on usage of DevOps Pipelines in this sample, review the [DevOps Pipelines README](../devops/README.md). + 2. the release pipeline will mirror what is on `dev` to the development workspace by running `update_from_git_to_ws.ps1`. + +**Step 7 and 8. Use Release pipeline to deploy to all environments/stages**: + +- The release pipeline for STG and PRD can be identical or a variation to the release pipeline for DEV. For more information on usage of DevOps Pipelines in this sample, review the [DevOps Pipelines README](../devops/README.md). + +## Common errors + +- `Error response: Response status code does not indicate success: 400 (Bad Request)` + - Likely one of the Fabric items you are trying to update has a definition containing errors. We advise maintaining identifiers such as workspaceIds and LakehouseIds as **parameters** for Notebooks and Data Pipelines. Failure to do this may result in the script being unable to push your item definitions if they reference item ids that have been removed. +- `Error response: Response status code does not indicate success: 401 (Unauthorized)` + - Likely your user token has expired. Update it and source your params file and then try again. + - If using Azure DevOps: if you are getting this error when running devops pipelines after refreshing the token variable, make sure you have toggled the secret variable type. +- `Error response: Response status code does not indicate success: 403 (Forbidden)` + - Likely one of the Fabric items you are trying to update has a MIP label that prevents you from updating its definition. + +- `Error response: Response status code does not indicate success: 404 (Not Found)` + - Likely the capacity you specified cannot be used. Make sure that your Fabric capacity is available and running, it might have been paused, in this case you can resume it and re-run the script. +- `Error response: A parameter cannot be found that matches parameter name 'ResponseHeadersVariable'` + - Likely you need to update your Powershell version to 7+. diff --git a/fabric/fabric_cicd_gitlab/docs/simplified_approach.md b/fabric/fabric_cicd_gitlab/docs/simplified_approach.md new file mode 100644 index 000000000..44b78fb67 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/docs/simplified_approach.md @@ -0,0 +1,124 @@ +# Simplified Update Item Script – Usage Instructions + +## Overview + +The [`simplified_update_item_from_ws.sh`](../src/simplified_update_item_from_ws.sh) script takes care of downloading items from Fabric so that developers can then commit those into source control. The script is designed to: + +- Authenticate with the Fabric API (refreshing the API token if expired). +- Retrieve the specified workspace by name (provided as an input parameter). +- Locate an item within that workspace that matches the provided item name and type. +- Download the item's definition (or metadata if the item type does not support definition retrieval). +- Save the definition files locally for further processing (e.g., version control, CI/CD). + +The [`simplified_update_item_from_git.sh`](../src/simplified_update_item_from_git.sh) script takes care of pushing to Fabric the local changes to item metadata or definitions. The script is designed to: + +- Authenticate with the Fabric API (refreshing the API token if expired). +- Retrieve the workspace ID based on the provided workspace name. +- Locate an item within that workspace by looking at the item name and type specified in the item definition files stored in the provided local folder. +- If there is no matching item in the workspace, the script creates a new item. Otherwise the matching item is updated. Creation or update is achieved by sending via APIs the definition of the Fabric item (e.g., DataPipeline, Environment, Notebook, etc.) from the local filesystem to the specified Microsoft Fabric workspace. + +## Prerequisite Software + +- **Bash**: The script is written in Bash and requires a Unix-like shell. +- **Azure CLI (az)**: Used to generate a Fabric API token. Also used by the helper functions to make REST API calls to Fabric. +- **jq**: For processing JSON responses. +- **curl**: For HTTP requests made in the long-running operations. +- **Git**: To manage your repository. +- A proper configuration file located at `./config/.env` containing the following variables: + - `TENANT_ID`: Required. Your Entra Tenant ID, used to authenticate and retrieve an access token + - `FABRIC_API_BASEURL`: Required. The base URL for the Fabric API. No need to update this as it's fixed. + - `FABRIC_USER_TOKEN`: Optional. The authentication token for the Fabric API. This will be filled by the script, can be left as is. + +## Assumptions + +The sample assumes the following: + +- the script should be executed using a user that has access to the Fabric workspace for which items need to be downloaded, with minimal permissions of `Contributor`. +- The Fabric workspace from which items should be downloaded is assigned to a **running** Fabric capacity. If the capacity is paused the script will fail. + +## How to Use the Scripts + +### Before running the scripts + +1. **Ensure Environment Setup:** + - Verify that all the prerequisite software is installed and available in your system’s PATH. + - Create a copy of the [`.envtemplate`](../config/.envtemplate) file and rename it to `.env` + + ```bash + cp ./config/.envtemplate ./config/.env + ``` + + - Edit the newly created `.env` file, filling the required parameters (`TENANT_ID`) + - Load environment variables + + ```bash + source config/.env + ``` + + - Login to Azure CLI with your user or SPN/MI, below instructions for user login with device code: + + ```bash + az login --use-device-code -t $TENANT_ID + ``` + +### Running the script: `simplified_update_item_from_ws.sh` + +1. **Script Parameters:** + The script requires four parameters: + - **workspace_name**: The display name of the Fabric workspace. + - **item_name**: The name of the Fabric item to retrieve. + - **item_type**: The type of the Fabric item (e.g., Notebook, DataPipeline) used to disambiguate items with the same name. + - **folder**: The local destination folder in your filesystem (local repository) where the retrieved item definition (or metadata) will be stored. + +1. **Example Usage:** + + ```bash + ./src/simplified_update_item_from_ws.sh "MyWorkspaceName" "MyNotebookName" "Notebook" "./fabric" + ``` + + This command will: + - Check if the Fabric API token is expired; if so, it will refresh the token and reload environment variables. + - Retrieve the workspace ID corresponding to `MyWorkspaceName`. + - Look for the item `MyNotebookName` of type `Notebook` within that workspace. Review the list of supported [Fabric item types](https://learn.microsoft.com/rest/api/fabric/core/items/list-items?tabs=HTTP#itemtype). + - Download the item definition and store the resulting files in the `./fabric` folder. + +1. **Commit to source control** + - **Manual Step**: After the item definition is downloaded locally, the files can be committed to the feature branch with the preferred mechanism: for example using VS Code or by executing a `git commit` command. + +### Running the script: `simplified_update_item_from_git.sh` + +1. **Script Parameters:** + The script requires two parameters: + - **workspace_name**: The display name of the Fabric workspace. + - **item_folder**: The source folder where the item definition files are located. + +1. **Example Usage:** + + ```bash + ./src/simplified_update_item_from_git.sh "MyWorkspaceName" "./fabric/MyNotebookName.Notebook" + ``` + + This command will: + - Check if the Fabric API token is expired; if so, it will refresh the token and reload environment variables. + - Retrieve the workspace ID corresponding to `MyWorkspaceName`. + - Verify that the folder `./fabric/MyNotebookName.Notebook` contains required item metadata and/or definition files. + - Create or Update the corresponding Fabric item in the `MyWorkspaceName` workspace, by using metadata and definition files found in the `./fabric/MyNotebookName.Notebook` folder. + +## Troubleshooting + +- If the script cannot find the workspace or item, double-check the names and types. +- Ensure that the right Tenant ID is provided. The script will attempt to refresh expired tokens automatically. +- Verify that the necessary tools (az, jq, curl, git) are installed and accessible. + +## Remarks + +The simplified approach has only been tested with items of type: DataPipeline, Lakehouse, Notebook, Environment. Other item types have not been tested and the sample might not be mimicking the current Fabric git integration approach. If a specific item type is important for you/your company, contributions to this repository would be more than welcome! + +### Known Gaps + +When a Lakehouse or Environment item is synced only a `.platform` file is produced. This deviates from the current Git integration behavior: + +- for Lakehouse items, Fabric git integration generates also a `shortcuts.metadata.json` file, containing the list of shortcuts that point to other Lakehouse items (either in the same or other workspaces of the same tenant). +- for Environment items, Fabric git integration generates also a `Settings` folder that contains the Spark pool `yml` settings definition file. + +These omissions are an intentional simplification in this approach to streamline the process and focus on core functionality. Users requiring full parity with Fabric Git integration behavior may need to extend the scripts accordingly. diff --git a/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.png b/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.png new file mode 100644 index 000000000..49e95e062 Binary files /dev/null and b/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.png differ diff --git a/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.svg b/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.svg new file mode 100644 index 000000000..1278efdea --- /dev/null +++ b/fabric/fabric_cicd_gitlab/images/Fabric CICD-Fabric CI_CD Flow.drawio.svg @@ -0,0 +1,4 @@ + + + +
Push
Pull
update
from
wokspace
feature branch(es)
feature/*
develop workspace
PR & 
Merge
Branch
update
from
git
develop
feature workspace(s)
update
via GitHub action
main workspace
PR & 
Merge
main
update
via GitHub action
 \ No newline at end of file diff --git a/fabric/fabric_cicd_gitlab/src/fabric_api_helpers.sh b/fabric/fabric_cicd_gitlab/src/fabric_api_helpers.sh new file mode 100644 index 000000000..06818380a --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/fabric_api_helpers.sh @@ -0,0 +1,414 @@ +#!/bin/bash + +. ./src/utilities.sh + +############################# +## Fabric functions +############################# + +#---------------------------- +# Workspaces +#---------------------------- + +# Function to get all workspaces names +get_workspace_names(){ + rest_call get "workspaces" "value[].displayName" tsv + #az rest --method get --uri "$FABRIC_API_BASEURL/workspaces" --headers "Authorization=Bearer $FABRIC_USER_TOKEN" --query "value[].id" -o tsv +} + +# Get workspace displayName by specifying a workspace id +get_workspace_name(){ + local workspace_id=$1 + rest_call get "workspaces/$workspace_id" "displayName" tsv + #az rest --method get --uri "$FABRIC_API_BASEURL/workspaces/$workspace_id" --headers "Authorization=Bearer $token" +} + +# Get workspace id by specifying a workspace displayName +get_workspace_id(){ + local workspace_name=$1 + rest_call get "workspaces" "value[?displayName=='$workspace_name'].id" tsv | tr -d '\r' + #az rest --method get --uri "$FABRIC_API_BASEURL/workspaces/$workspace_id" --headers "Authorization=Bearer $token" +} + +# Create a new workspace with the specified name using capacity id +create_workspace(){ + local workspace_name=$1 + local capacity_id=$2 + body=$(echo '{"displayName" : "'$workspace_name'", "capacityId" : "'$capacity_id'"}') + + rest_call "POST" "workspaces" "id" "tsv" "$body" +} + +get_or_create_workspace() { + # given a workspace name returns its id + # if the workspace does not exist, creates it and returns its id + # requires a capacity id and a workspace name + + local workspace_name=$1 + local capacity_id=$2 + + workspace_id=$(get_workspace_id "$workspace_name") + + if [ -z "$workspace_id" ]; then + log "A workspace with the requested name $workspace_name was not found, creating new workspace." + workspace_id=$(create_workspace "$workspace_name" "$capacity_id") + log "Workspace $workspace_name with id $workspace_id was created." "success" + else + log "Workspace $workspace_name with id $workspace_id was found." "success" + fi + echo $workspace_id +} + +#---------------------------- +# Items +#---------------------------- + + +get_workspace_items(){ + local workspace_id=$1 + # get all workspace items + rest_call get "workspaces/$workspace_id/items" "value" "json" +} + +get_item_id(){ + local workspace_id=$1 + local item_name="$2" + local item_type=$3 + # rest_call get "workspaces/$workspace_id/items?type=$item_type" "value[?displayName=='$item_name'].id" tsv | tr -d '\r' + echo $(get_item_by_name "$workspace_id" "$item_name" "$item_type" | jq -r '.id') +} + +get_item_by_name(){ + local workspace_id=$1 + local item_name="$2" + local item_type=$3 + rest_call get "workspaces/$workspace_id/items?type=$item_type" "value[?displayName=='$item_name'] | [0].{description: description, displayName: displayName, type: type, id: id}" "json" | tr -d '\r' +} + +get_and_store_item(){ + local workspace_id="$1" + local item_name="$2" + local item_type="$3" + local folder="$4" + # This function retrieves the definition of an item + # requires as inputs the workspace id, the item name and the item type + # If the item type supports retrieving the definition then it will return that + # else it will return the item metadata + log "Retrieving item '$item_name' of type '$item_type' from workspace '$workspace_id'" + if [ $item_type == "Notebook" ] || [ $item_type == "DataPipeline" ]; then + # When the item supports definition then use the getDefinition API + item_definition=$(get_item_definition "$workspace_id" "$item_name" "$item_type") + if [ -z "$item_definition" ]; then + log "Failed to retrieve definition for item $item_name of type $item_type." + return 1 + fi + log "Saving definition to file..." + store_item_definition "$folder" "$item_name" "$item_type" "$item_definition" + else + item_metadata=$(rest_call get "workspaces/$workspace_id/items?type=$item_type" "value[?displayName=='$item_name'] | [0].{description: description, displayName: displayName, type: type}" "json" | tr -d '\r') + if [ -z "$item_metadata" ]; then + log "Item $item_name of type $item_type was not found in the workspace." + return 1 + fi + log "Saving item metadata..." + store_item_metadata "$folder" "$item_name" "$item_type" "$item_metadata" + fi +} + +store_item_metadata(){ + local folder=$1 + local item_name=$2 + local item_type=$3 + local item_metadata=$4 + local output_folder="$folder/$item_name.$item_type" + mkdir -p "$output_folder" + platform_file=$(cat < "$output_folder/.platform" + log "Item metadata saved to $output_folder" "success" +} + +get_item_definition(){ + local workspace_id=$1 + local item_name=$2 + local item_type=$3 + + # Get item id + item_id=$(get_item_id "$workspace_id" "$item_name" "$item_type") + if [ -z "$item_id" ]; then + log "Item $item_name of type $item_type was not found in the workspace." + return 1 + fi + log "Found item '$item_name' with ID: '$item_id'" + log "retrieving item definition for '$item_name'" + # then get item definition + # if itemType=Notebook then filter for format=ipynb + if [ "$item_type" == "Notebook" ]; then + uri="workspaces/$workspace_id/items/$item_id/getDefinition?format=ipynb" + else + uri="workspaces/$workspace_id/items/$item_id/getDefinition" + fi + response=$(curl -sSi -X POST -H "Authorization: Bearer $FABRIC_USER_TOKEN" "$FABRIC_API_BASEURL/$uri" --data "") + response=$(return_operation_response "$response" "$item_name" "$item_type") + + echo "$response" +} + +store_item_definition(){ + local folder=$1 + local item_name=$2 + local item_type=$3 + local definitionJson=$4 + local output_folder="$folder/$item_name.$item_type" + mkdir -p "$output_folder" + # for each path element in the item definition json + # convert the base64 encoded content to a file + # using the payloadType, payload and path elements + # for part in $(echo "$definitionJson" | jq -r '.definition.parts[] | @base64'); do + # part=$(echo "$part" | base64 --decode) + for part in $(echo "$definitionJson" | jq -r -c '.definition.parts[]'); do + path=$(echo "$part" | jq -r -c '.path') + payloadType=$(echo "$part" | jq -r -c '.payloadType') + payload=$(echo "$part" | jq -r -c '.payload') + log "Saving item definition part '$path' in '$output_folder'" + if [ "$payloadType" == "InlineBase64" ]; then + echo -n "$payload" | base64 -d > "$output_folder/$path" + else + echo -n "$payload" > "$output_folder/$path" + fi + done + log "Item definition saved to $output_folder" "success" +} + +#----------------------------- +# Long running operations +#----------------------------- + +return_operation_response() { + local response="$1" + local item_name="$2" + local item_type="$3" + + status_code=$(echo "$response" | head -n 1 | cut -d' ' -f2) + + if [ "$status_code" != 202 ] && [ "$status_code" != 200 ]; then + log "Failed to retrieve definition for item $item_name of type $item_type." + log "Response: $response" + return 1 + fi + if [ "$status_code" == 200 ]; then + response=$(echo "$response" | tail -n 1) + echo "$response" + return 0 + else + location=$(echo "$response" | grep -i ^location: | cut -d: -f2- | sed 's/^ *\(.*\).*/\1/' | tr -d '\r') + retry_after=$(echo "$response" | grep -i ^retry-after: | cut -d: -f2- | sed 's/^ *\(.*\).*/\1/' | tr -d '\r') + response=$(long_running_operation_polling "$location" "$retry_after") + if [ -z "$response" ]; then + log "Failed to retrieve definition for item $item_name of type $item_type." + return 1 + fi + fi + + echo "$response" + +} + +# Function to poll a long running operation +long_running_operation_polling() { + local uri=$1 + local retryAfter=$2 + local requestHeader="Authorization: Bearer $FABRIC_USER_TOKEN" + + log "Polling long running operation ID has been started with a retry-after time of $retryAfter seconds." + + while true; do + operationState=$(curl -s -H "$requestHeader" "$uri") + status=$(echo "$operationState" | jq -r '.status') + + if [[ "$status" == "NotStarted" || "$status" == "Running" ]]; then + sleep 20 + else + break + fi + done + + if [ "$status" == "Failed" ]; then + log "The long running operation has been completed with failure. Error response: $(echo "$operationState" | jq '.')" + else + log "Operation successfully completed." "success" + item=$(curl -s -H "$requestHeader" "$uri/result") + echo "$item" + fi +} + +#------------------------------ +# Item CRUD Operations +#------------------------------ + +# Function to create or update a workspace item +# This function takes a workspace ID, item name, item type, and folder as arguments +# It checks if the item already exists in the workspace and either creates or updates it accordingly +create_or_update_item() { + local workspace_id=$1 + local item_folder="$2" + + platform_file="$item_folder/.platform" + if [ ! -f "$platform_file" ]; then + log "Item folder '$item_folder' does not contain a .platform file." "danger" + exit 1 + fi + + item_name=$(jq -r '.metadata.displayName' "$platform_file") + item_type=$(jq -r '.metadata.type' "$platform_file") + if [ -z "$item_name" ] || [ -z "$item_type" ]; then + log "Error: Item name or type not found in the .platform file." + exit 1 + fi + log "Platform file contains item '$item_name' of type $item_type" + + + # check if the item already exists in the workspace + item_id=$(get_item_id "$workspace_id" "$item_name" "$item_type") + if [ -n "$item_id" ]; then + log "Item $item_name of type $item_type already exists in the workspace and has item ID: $item_id.\nUpdating item..." "warning" + returned_item=$(update_item "$workspace_id" "$item_id" "$item_name" "$item_type" "$item_folder") + else + log "Item $item_name of type $item_type does not exist in the workspace.\nCreating new item..." "info" + returned_item=$(create_item "$workspace_id" "$item_name" "$item_type" "$item_folder") + fi +} + +create_item() { + local workspace_id=$1 + local item_name=$2 + local item_type=$3 + local item_folder=$4 + # This function creates a new item in the workspace + # It takes the workspace ID, item name, item type, and folder as arguments + + body=$(echo '{"displayName": "'"$item_name"'", "description": "", "type": "'"$item_type"'"}') + + returned_item=$(rest_call post "workspaces/$workspace_id/items" "" "json" "$body") + item_id=$(echo "$returned_item" | jq -r '.id') + if [ -z "$item_id" ]; then + log "Failed to create item $item_name of type $item_type." + return 1 + fi + + # if the item type has a definition then use the definition + # count the number of files in item_folder that are not the .platform file + file_count=$(find "$item_folder" -type f ! -name ".platform" | wc -l) + if [ "$file_count" -gt 0 ]; then + log "Updating item definition..." "info" + returned_item=$(update_item_definition "$workspace_id" "$item_id" "$item_folder") + fi + echo "$returned_item" +} + +update_item() { + local workspace_id=$1 + local item_id=$2 + local item_name=$3 + local item_type=$4 + local item_folder=$5 + # This function updates an existing item in the workspace + # It takes the workspace ID, item id, item name, item type, and folder as arguments + + # check if the item folder contains a definition file + # if the item type is DataPipeline then the defintion file name is 'pipeline-content.json' + # if the item type is Notebook then the defintion file name is 'notebook-content.ipynb' or 'notebook-content.py' + platform_file="$item_folder/.platform" + + # if the item type requires definition files then make sure they exist + if [ "$item_type" == "Notebook" ] && [ ! -f "$item_folder/notebook-content.ipynb" ]; then + definition_file="$item_folder/notebook-content.py" + elif [ "$item_type" == "Notebook" ] && [ ! -f "$item_folder/notebook-content.py" ]; then + definition_file="$item_folder/notebook-content.ipynb" + elif [ "$item_type" == "DataPipeline" ]; then + definition_file="$item_folder/pipeline-content.json" + fi + + definition_file_exists=false + if ([ "$item_type" == "Notebook" ] || [ "$item_type" == "DataPipeline" ]) && [ ! -f "$definition_file" ]; then + log "No definition file found in the item folder '$item_folder', only metadata will be updated." "warning" + elif [ "$item_type" == "Notebook" ] || [ "$item_type" == "DataPipeline" ]; then + definition_file_exists=true + log "Definition file found in the item folder '$item_folder'." + fi + + if [ $definition_file_exists == "true" ]; then + returned_item=$(update_item_definition "$workspace_id" "$item_id" "$item_folder") + else + # update only metadata + item_metadata=$(jq -r ".metadata | {displayName, description}" "$platform_file") + returned_item=$(rest_call patch "workspaces/$workspace_id/items/$item_id" "" "json" "$item_metadata") + fi + + if [ -z "$returned_item" ]; then + log "Failed to update item $item_name of type $item_type." + return 1 + fi + if [ $definition_file_exists == "true" ]; then + log "Item $item_name of type $item_type was updated with definition." "success" + else + log "Item $item_name of type $item_type was updated." "success" + fi + + echo "$returned_item" +} + +update_item_definition() { + local workspace_id=$1 + local item_id=$2 + local item_folder="$3" + + # construct body or curl call by listing all files in the item folder + # for every file, get the base64 encoded content + # and create a part object with the path, payloadType and payload + shopt -s lastpipe + find "$item_folder" -type f | while IFS= read -r file; do + path=$(basename "$file") + payloadType="InlineBase64" + # Deal with Open Darwin's version of base64 + # which does not support the -w option + if [[ "$(uname)" == "Darwin" ]]; then # Mac OSX + payload=$(base64 < "$file" | tr -d '\n' | tr -d '\r') + else + payload=$(base64 -w 0 "$file") + fi + + file_extension="${path##*.}" + if [ $file_extension == "ipynb" ]; then + format="ipynb" + fi + echo -n '{"path" : "'"$path"'", "payloadType" : "'"$payloadType"'", "payload" : "'"$payload"'"}' >> parts.json + done + + parts=$(cat parts.json | jq -s '.' | tr -d '\r' | tr -d '\n') + if [ -z $format ]; then + echo -n '{"definition" : {"parts" : ' $parts "}}" > definition.json + else + echo -n '{"definition" : {"parts" : '$parts', "format" : "'$format'"}}' > definition.json + fi + shopt -u lastpipe + + uri="workspaces/$workspace_id/items/$item_id/updateDefinition?updateMetadata=true" + item_name=$(jq -r '.metadata.displayName' "$item_folder/.platform") + item_type=$(jq -r '.metadata.type' "$item_folder/.platform") + response=$(curl -sSi -X POST -H "Authorization: Bearer $FABRIC_USER_TOKEN" "$FABRIC_API_BASEURL/$uri" -H "Content-Type: application/json" --data "@definition.json") + response=$(return_operation_response "$response" "$item_name" "$item_type") + rm -f parts.json + rm -f definition.json + + echo "$response" +} \ No newline at end of file diff --git a/fabric/fabric_cicd_gitlab/src/new_branch.sh b/fabric/fabric_cicd_gitlab/src/new_branch.sh index 6e5b4cd61..6c70a66ab 100644 --- a/fabric/fabric_cicd_gitlab/src/new_branch.sh +++ b/fabric/fabric_cicd_gitlab/src/new_branch.sh @@ -6,4 +6,4 @@ git update-index --assume-unchanged .gitignore echo >> .gitignore echo "item-config.json" >> .gitignore git update-index --assume-unchanged $(git ls-files "**/item-config.json") -git update-index --assume-unchanged ./src/params.psd1 \ No newline at end of file +git update-index --assume-unchanged ./config/.env diff --git a/fabric/fabric_cicd_gitlab/src/refresh_api_token.sh b/fabric/fabric_cicd_gitlab/src/refresh_api_token.sh new file mode 100644 index 000000000..1db2f1430 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/refresh_api_token.sh @@ -0,0 +1,30 @@ +#!/bin/bash +set -o errexit + +# Set authenticate flag to false by default if not provided +authenticate=${1:-false} +use_spn=${2:-false} + +source ./config/.env + +if [ "$authenticate" != "false" ] && [ "$use_spn" == "false" ]; then + echo "Authenticating..." + az config set core.login_experience_v2=off + az login --tenant $TENANT_ID --use-device-code + az config set core.login_experience_v2=on +else + if [ "$authenticate" != "false" ] && [ "$use_spn" != "false" ]; then + # else if authenticate is not false and use_spn is not false authenticate with a service principal + echo "Authenticating with service principal..." + az login --service-principal -u $APP_CLIENT_ID -p $APP_CLIENT_SECRET --tenant $TENANT_ID --allow-no-subscription + fi +fi + +# grab an Entra token for the Fabric API +token=$(az account get-access-token \ + --resource "https://analysis.windows.net/powerbi/api" \ + --query "accessToken" \ + -o tsv | tr -d '\r') + +# Refresh the token in the .env file +sed -i "s/FABRIC_USER_TOKEN=.*/FABRIC_USER_TOKEN=$token/" ./config/.env \ No newline at end of file diff --git a/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_git.sh b/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_git.sh new file mode 100644 index 000000000..1e06bfe00 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_git.sh @@ -0,0 +1,70 @@ +#!/bin/bash + +# ----------------------------------------------------------------------------- +# Script to upload the definition of a Fabric item (e.g., Dataset, Pipeline, etc.) +# from local filesystem to the specified Microsoft Fabric workspace. +# The script checks if the API token is expired and refreshes it if needed. +# It also retrieves the workspace ID based on the provided workspace name. +# ----------------------------------------------------------------------------- +set -e + +. ./src/fabric_api_helpers.sh + +# Load environment variables from the .env file +source ./config/.env + +# Validate input arguments: workspaceName, itemName, itemType, folder +if [ "$#" -ne 2 ]; then + log "Usage: $0 " + exit 1 +fi + +workspaceName="$1" # Fabric workspace name +item_folder="$2" # The source folder where the item definition files are located + +# Check if the item folder exists +if [ ! -d "$item_folder" ]; then + log "Error: Item folder '$item_folder' does not exist." + exit 1 +fi +# Check if the item folder contains a .platform file +if [ ! -f "$item_folder/.platform" ]; then + log "Error: No .platform file found in the item folder '$item_folder'." + exit 1 +fi + +# Check required environment variables +if [ -z "$FABRIC_API_BASEURL" ] || [ -z "$FABRIC_USER_TOKEN" ]; then + log "FABRIC_API_BASEURL or FABRIC_USER_TOKEN is not set in the env file." + exit 1 +fi + +# ----------------------------------------------------------------------------- +# Check if the API token is expired and refresh if needed using refresh_api_token.sh +# ----------------------------------------------------------------------------- +if [[ $(is_token_expired) = "1" ]]; then + log "API token has expired. Refreshing token..." + # Call refresh_api_token.sh. + ./src/refresh_api_token.sh + # Reload environment variables after token refresh. + source ./config/.env + log "Token refreshed." +fi + +# ----------------------------------------------------------------------------- +# Get the workspace ID from Fabric +# ----------------------------------------------------------------------------- +workspaceId=$(get_workspace_id "$workspaceName") +if [ -z "$workspaceId" ]; then + log "Error: Could not find workspace $workspaceName." + exit 1 +fi +log "Found workspace '$workspaceName' with ID: '$workspaceId'" + +# ----------------------------------------------------------------------------- +# call the create_or_update_item function to create or update the item +# ----------------------------------------------------------------------------- + +create_or_update_item "$workspaceId" "$item_folder" + +log "Script successfully completed." "success" diff --git a/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_ws.sh b/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_ws.sh new file mode 100644 index 000000000..46ddc4372 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/simplified_update_item_from_ws.sh @@ -0,0 +1,64 @@ +#!/bin/bash + +# ----------------------------------------------------------------------------- +# Script to download the definition of a Fabric item (e.g., Dataset, Pipeline, etc.) +# and store it in a specified folder. +# The script checks if the API token is expired and refreshes it if needed. +# It also retrieves the workspace ID based on the provided workspace name. +# ----------------------------------------------------------------------------- +set -e + +. ./src/fabric_api_helpers.sh + +source ./config/.env + +# Validate input arguments: workspaceName, itemName, itemType, folder +if [ "$#" -ne 4 ]; then + log "Usage: $0 " + exit 1 +fi + +workspaceName="$1" # Fabric workspace name +itemName="$2" # Fabric item name whose definition should be downloaded +itemType="$3" # Fabric item type used to filter the results +folder="$4" # Destination folder for the definition file + +# Check required environment variables +if [ -z "$FABRIC_API_BASEURL" ] || [ -z "$FABRIC_USER_TOKEN" ]; then + log "FABRIC_API_BASEURL or FABRIC_USER_TOKEN is not set in the env file." + exit 1 +fi + +# create destination folder if needed +mkdir -p "$folder" + +# ----------------------------------------------------------------------------- +# Check if the API token is expired and refresh if needed using refresh_api_token.sh +# ----------------------------------------------------------------------------- +if [[ $(is_token_expired) = "1" ]]; then + log "API token has expired. Refreshing token..." + # Call refresh_api_token.sh. + ./src/refresh_api_token.sh + # Reload environment variables after token refresh. + source ./config/.env + log "Token refreshed." +fi + +# ----------------------------------------------------------------------------- +# Get the workspace ID from Fabric using a helper function +# ----------------------------------------------------------------------------- +workspace_id=$(get_workspace_id "$workspaceName") +if [ -z "$workspace_id" ]; then + log "Error: Could not find workspace $workspaceName." + exit 1 +fi +log "Found workspace '$workspaceName' with ID: '$workspace_id'" + +# ----------------------------------------------------------------------------- +# Retrieve the item definition for the specified item if it exists +# Handle items that don't have a definition such as Lakehouse, Environment +# For these items, the API returns only the .platform file +# ----------------------------------------------------------------------------- +get_and_store_item "$workspace_id" "$itemName" "$itemType" "$folder" + +log "Script successfully completed." "success" diff --git a/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.ps1 b/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.ps1 index 8437c2d9a..ee1eca43e 100644 --- a/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.ps1 +++ b/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.ps1 @@ -1,11 +1,11 @@ param ( - [parameter(Mandatory = $true)] [String] $baseUrl, - [parameter(Mandatory = $true)] [String] $fabricToken, + [parameter(Mandatory = $false)] [String] $baseUrl, # Optional, the fabric api base url + [parameter(Mandatory = $false)] [String] $fabricToken, # Optional, the fabric api token [parameter(Mandatory = $true)] [String] $workspaceName, # The name of the workspace, - [parameter(Mandatory = $true)] [String] $capacityId, # The capacity id of the workspace, - [parameter(Mandatory = $true)] [String] $folder, # The folder where the workspace items are located on the branch, should be: Join-Path $(Build.SourcesDirectory) $(directory_name) - [parameter(Mandatory = $false)] [bool] $resetConfig=$false # Used when the developer wants to reset the config files in the workspace (typically when a new feature branch is created) + [parameter(Mandatory = $false)] [String] $capacityId, # Optional, the capacity id of the workspace, + [parameter(Mandatory = $false)] [String] $folder, # Optional, the folder where the workspace items are located on the branch, should be: Join-Path $(Build.SourcesDirectory) $(directory_name) + [parameter(Mandatory = $false)] [bool] $resetConfig=$false # Optional, used when the developer wants to reset the config files in the workspace (typically when a new feature branch is created) ) ## FROM GIT TO WORKSPACE # Used when the developer creates a new branch from the development/main branch @@ -64,9 +64,19 @@ function getorCreateWorkspaceId($requestHeader, $contentType, $baseUrl, $workspa } } +function getWorkspaceItems($requestHeader, $contentType, $baseUrl, $workspaceId){ + $params = @{ + Uri = "$($baseUrl)/workspaces/$($workspaceId)/items" + Method = "GET" + Headers = $requestHeader + ContentType = $contentType + } + return (Invoke-RestMethod @params).value +} function createWorkspaceItem($baseUrl, $workspaceId, $requestHeader, $contentType, $itemMetadata, $itemDefinition){ if ($itemDefinition) { + Write-Host "Creating item $($itemMetadata.displayName) with definition." -ForegroundColor Yellow # if the item has a definition create the item with definition $body = @{ displayName = $itemMetadata.displayName @@ -76,6 +86,7 @@ function createWorkspaceItem($baseUrl, $workspaceId, $requestHeader, $contentTyp } } else { #item does not have definition, only create the item with metadata + Write-Host "Creating item $($itemMetadata.displayName) without definition." -ForegroundColor Yellow $body = @{ displayName = $itemMetadata.displayName description = $itemMetadata.description @@ -163,7 +174,7 @@ function createOrUpdateWorkspaceItem($requestHeader, $contentType, $baseUrl, $wo if ([System.IO.File]::Exists($definitionFilePath)){ $itemDefinition = Get-Content -Path $definitionFilePath -Raw | ConvertFrom-Json Write-Host "Found item definition for $($itemMetadata.displayName)" -ForegroundColor Green - $contentFiles = Get-ChildItem -Path $folder | Where-Object {$_.Name -notlike $itemMetadataFileName -and $_.Name -notlike $itemDefinitionFileName -and $_.Name -notlike $itemConfigFileName} + $contentFiles = Get-ChildItem -Path $folder -Force | Where-Object {$_.Name -notlike $itemMetadataFileName -and $_.Name -notlike $itemDefinitionFileName -and $_.Name -notlike $itemConfigFileName} #$contentFiles = Get-ChildItem -Path $folder | Where-Object {$_.Name -like "*content*"} if ($contentFiles -and $contentFiles.Count -ge 1){ # if there is at least a content file then update the definition payload Write-Host "Found $($contentFiles.Count) content file(s) for $($itemMetadata.displayName)" -ForegroundColor Green @@ -201,11 +212,11 @@ function createOrUpdateWorkspaceItem($requestHeader, $contentType, $baseUrl, $wo if (!$itemConfig.objectId) { # 3. if an objectId is not present and only a logicalId is present then # Create a new object and save the objectId in the config file - Write-Host "Item $folder does not have an associated objectId, creating new Fabric item of type $($itemMetadata.type) with name $($itemMetadata.displayName)." -ForegroundColor Yellow + Write-Host "Item $($itemMetadata.displayName) does not have an associated objectId, creating new Fabric item of type $($itemMetadata.type) with name $($itemMetadata.displayName)." -ForegroundColor Yellow $item = createWorkspaceItem $baseUrl $workspaceId $requestHeader $contentType $itemMetadata $itemDefinition - Write-Host "item is $($item.displayName) with id $($item.id)" + Write-Host "Created item $($item.displayName) with id $($item.id)" # update the config file with the returned objectId $itemConfig | add-member -Name "objectId" -value $item.id -MemberType NoteProperty -Force Write-Host "itemConfig objectId is $($itemConfig.objectId)" @@ -286,7 +297,33 @@ function longRunningOperationPolling($uri, $retryAfter){ } } +function loadEnvironmentVariables() { + Write-Host "Loading environment file..." + get-content config/.env | ForEach-Object { + if ($_ -match '^#' -or [string]::IsNullOrWhiteSpace($_)) { return } # skip comments and empty lines + $name, $value = $_.split('=') + $value = $value.split('#')[0].trim() # to support commented env files + $value = $value -replace '^"|"$' # remove leading and trailing double quotes + set-content env:\$name $value + } + Write-Host "Finished loading environment file. \nFabric REST API endpoint is $env:FABRIC_API_BASEURL" +} + try { + # if the following parameters are not set, the script will load env variables from the .env file + # else it will use the provided parameters + if (!$fabricToken -or !$baseUrl -or !$capacityId -or !$folder) { + Write-Host "Parameters fabricToken, baseUrl, capacityId or folder are not set, loading from .env file." + loadEnvironmentVariables + $baseUrl=$env:FABRIC_API_BASEURL + $fabricToken=$env:FABRIC_USER_TOKEN + $capacityId=$env:FABRIC_CAPACITY_ID + $folder=$env:ITEMS_FOLDER + } + else { + Write-Host "Required parameters are set, using them directly." + } + # TODO: consider removing the logicalId from the file as it's not used today. Write-Host "this task is running Powershell version " $PSVersionTable.PSVersion Write-Host "the folder we are working on is $folder" @@ -307,13 +344,7 @@ try { # 2. For every Fabric item on the branch, check if they exist in the workspace # first get a list of all items in the workspace - $params = @{ - Uri = "$($baseUrl)/workspaces/$($workspaceId)/items" - Method = "GET" - Headers = $requestHeader - ContentType = $contentType - } - $workspaceItems = (Invoke-RestMethod @params).value + $workspaceItems = getWorkspaceItems $requestHeader $contentType $baseUrl $workspaceId $repoItems = @() # keep track of found object ids (either from creation or config files) and remove all other object ids from the workspace # if they exist update them else create new ones $dir = Get-ChildItem -Path $folder -Recurse -Directory @@ -342,4 +373,4 @@ catch { $errorResponse = GetErrorResponse($_) Write-Host "Failed to run script to update workspace items for workspace $workspaceName. Error reponse: $errorResponse" -ForegroundColor Red exit 1 -} \ No newline at end of file +} diff --git a/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.sh b/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.sh new file mode 100644 index 000000000..4e4872047 --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/update_from_git_to_ws.sh @@ -0,0 +1,124 @@ +#!/bin/bash + +set -e + +. ./refresh_api_token.sh +. ./fabric_api_helpers.sh + +source ./config/.env + + +# Parameters +baseUrl=$FABRIC_API_BASEURL +fabricToken=$FABRIC_USER_TOKEN +capacityId=$FABRIC_CAPACITY_ID +folder=$ITEMS_FOLDER +resetConfig=${6:-false} +workspaceName=$1 + +# Functions +get_error_response() { + echo "$1" +} + +update_workspace_item_definition() { + local baseUrl=$1 + local workspace_id=$2 + local requestHeader=$3 + local contentType=$4 + local itemMetadata=$5 + local itemDefinition=$6 + local itemConfig=$7 + + uri="$baseUrl/workspaces/$workspace_id/items/$(echo "$itemConfig" | jq -r '.objectId')/updateDefinition" + if [ "$(echo "$itemMetadata" | jq -r '.type')" == "Notebook" ] && [ -z "$(echo "$itemDefinition" | jq -r '.definition.format')" ]; then + itemDefinition=$(echo "$itemDefinition" | jq '.definition.format = "ipynb"') + fi + body=$(jq -n --argjson definition "$(echo "$itemDefinition" | jq '.definition')" '{definition: $definition}') + + echo "Executing POST to update definition of item $(echo "$itemConfig" | jq -r '.objectId') $(echo "$itemMetadata" | jq -r '.displayName')" + curl -s -X POST -H "$requestHeader" -H "Content-Type: $contentType" -d "$body" "$uri" +} + +update_workspace_item() { + local baseUrl=$1 + local workspace_id=$2 + local requestHeader=$3 + local contentType=$4 + local itemMetadata=$5 + local itemDefinition=$6 + local itemConfig=$7 + + uri="$baseUrl/workspaces/$workspace_id/items/$(echo "$itemConfig" | jq -r '.objectId')" + body=$(jq -n --arg displayName "$(echo "$itemMetadata" | jq -r '.displayName')" --arg description "$(echo "$itemMetadata" | jq -r '.description')" '{displayName: $displayName, description: $description}') + + echo "Executing PATCH to update item $(echo "$itemConfig" | jq -r '.objectId') $(echo "$itemMetadata" | jq -r '.displayName')" + curl -s -X PATCH -H "$requestHeader" -H "Content-Type: $contentType" -d "$body" "$uri" + + if [ -n "$itemDefinition" ]; then + update_workspace_item_definition "$baseUrl" "$workspace_id" "$requestHeader" "$contentType" "$itemMetadata" "$itemDefinition" "$itemConfig" + fi +} + +long_running_operation_polling() { + local uri=$1 + local retryAfter=$2 + + echo "Polling long running operation ID $uri has been started with a retry-after time of $retryAfter seconds." + + while true; do + operationState=$(curl -s -H "$requestHeader" -H "Content-Type: $contentType" "$uri") + status=$(echo "$operationState" | jq -r '.Status') + + echo "Long running operation status: $status" + + if [[ "$status" == "NotStarted" || "$status" == "Running" ]]; then + sleep 20 + else + break + fi + done + + if [ "$status" == "Failed" ]; then + echo "The long running operation has been completed with failure. Error response: $(echo "$operationState" | jq '.')" + else + echo "The long running operation has been successfully completed." + if [ -n "$responseHeadersLocation" ]; then + uri="$responseHeadersLocation" + else + return + fi + item=$(curl -s -H "$requestHeader" -H "Content-Type: $contentType" "$uri") + echo "$item" + fi +} + +# Main script +echo "The folder we are working on is $folder" +echo "Updating workspace items for workspace $workspaceName" + +itemConfigFileName="item-config.json" +itemMetadataFileName="item-metadata.json" +itemDefinitionFileName="item-definition.json" + +workspace_id=$(get_or_create_workspace "$workspaceName" "$capacityId") +workspaceItems=$(get_workspace_items "$workspace_id") +repoItems=() + +dirs=$(find $folder -maxdepth 1 -mindepth 1 -type d) +for d in $dirs; do + echo "$d" + repoItems=($(create_or_update_workspace_item "$requestHeader" "$contentType" "$baseUrl" "$workspace_id" "$workspaceItems" "$d" "${repoItems[@]}")) +done + +for item in $(echo "$workspaceItems" | jq -r '.[] | @base64'); do + item=$(echo "$item" | base64 --decode) + itemId=$(echo "$item" | jq -r '.id') + itemType=$(echo "$item" | jq -r '.type') + if [[ ! " ${repoItems[@]} " =~ " ${itemId} " ]] && [[ "$itemType" != "SQLEndpoint" && "$itemType" != "SemanticModel" ]]; then + echo "Item $itemId $(echo "$item" | jq -r '.displayName') is in the workspace but not in the repository, deleting." + curl -s -X DELETE -H "$requestHeader" -H "Content-Type: $contentType" "$baseUrl/workspaces/$workspace_id/items/$itemId" + fi +done + +echo "Script execution completed successfully. Workspace items have been updated for workspace $workspaceName." diff --git a/fabric/fabric_cicd_gitlab/src/update_from_ws_to_git.ps1 b/fabric/fabric_cicd_gitlab/src/update_from_ws_to_git.ps1 index 78f19183c..3c49e3603 100644 --- a/fabric/fabric_cicd_gitlab/src/update_from_ws_to_git.ps1 +++ b/fabric/fabric_cicd_gitlab/src/update_from_ws_to_git.ps1 @@ -1,10 +1,6 @@ param ( - [parameter(Mandatory = $true)] [String] $baseUrl, - [parameter(Mandatory = $true)] [String] $fabricToken, - [parameter(Mandatory = $true)] [String] $workspaceName, # The name of the workspace, - [parameter(Mandatory = $false)] [String] $capacityId, # The capacity id of the workspace, - [parameter(Mandatory = $true)] [String] $folder # The folder where the workspace items are located on the branch, should be: Join-Path $(Build.SourcesDirectory) $(directory_name) + [parameter(Mandatory = $true)] [String] $workspaceName # The name of the workspace, ) ## FROM WORKSPACE TO GIT # Used when the developers have finished working on their workspace and want to sync back to their branch. @@ -219,7 +215,24 @@ function longRunningOperationPolling($uri, $retryAfter){ +function loadEnvironmentVariables() { + Write-Host "Loading environment file..." + get-content config/.env | ForEach-Object { + if ($_ -match '^#' -or [string]::IsNullOrWhiteSpace($_)) { return } # skip comments and empty lines + $name, $value = $_.split('=') + $value = $value.split('#')[0].trim() # to support commented env files + $value = $value -replace '^"|"$' # remove leading and trailing double quotes + set-content env:\$name $value + } + Write-Host "Finished loading environment file. \nFabric REST API endpoint is $env:FABRIC_API_BASEURL" +} + try { + loadEnvironmentVariables + $baseUrl=$env:FABRIC_API_BASEURL + $fabricToken=$env:FABRIC_USER_TOKEN + $folder=$env:ITEMS_FOLDER + Write-Host "this task is running Powershell version " $PSVersionTable.PSVersion Write-Host "the folder we are working on is $folder" Write-Host "Updating workspace items for workspace $workspaceName" diff --git a/fabric/fabric_cicd_gitlab/src/utilities.sh b/fabric/fabric_cicd_gitlab/src/utilities.sh new file mode 100644 index 000000000..33c18911f --- /dev/null +++ b/fabric/fabric_cicd_gitlab/src/utilities.sh @@ -0,0 +1,111 @@ +#!/bin/bash + +############################# +## Utility functions +############################# +print_style () { + case "$2" in + "info") + COLOR="96m" + ;; + "success") + COLOR="92m" + ;; + "warning") + COLOR="93m" + ;; + "danger") + COLOR="91m" + ;; + "action") + COLOR="32m" + ;; + *) + COLOR="0m" + ;; + esac + + STARTCOLOR="\e[$COLOR" + ENDCOLOR="\e[0m" + printf "$STARTCOLOR%b$ENDCOLOR" "$1" +} + +log() { + # This function takes a string as an argument and prints it to the console to stderr + # if a second argument is provided, it will be used as the style of the message + # Usage: log "message" "style" + # Example: log "Hello, World!" "info" + local message=$1 + local style=${2:-} + + if [[ -z "$style" ]]; then + echo -e "$(print_style "$message" "default")" >&2 + else + echo -e "$(print_style "$message" "$style")" >&2 + fi +} + +# Function to make REST API calls to Fabric API +rest_call(){ + local method=$1 + local uri=$2 + local query=${3:-} + local output=${4:-"json"} + local body=${5:-} + + if [ -z "$query" ] && [ -z "$body" ]; then + az rest --method $method --uri "$FABRIC_API_BASEURL/$uri" --headers "Authorization=Bearer $FABRIC_USER_TOKEN" --output $output + return + fi + + if [ -n "$query" ] && [ -z "$body" ]; then + az rest --method $method --uri "$FABRIC_API_BASEURL/$uri" --headers "Authorization=Bearer $FABRIC_USER_TOKEN" --query "$query" --output $output + return + fi + + if [ -z "$query" ] && [ -n "$body" ]; then + az rest --method $method --uri "$FABRIC_API_BASEURL/$uri" --headers "Authorization=Bearer $FABRIC_USER_TOKEN" --output $output --body "$body" + fi + + if [ -n "$query" ] && [ -n "$body" ]; then + az rest --method $method --uri "$FABRIC_API_BASEURL/$uri" --headers "Authorization=Bearer $FABRIC_USER_TOKEN" --query "$query" --output $output --body "$body" + return + fi + +} + +function is_token_expired { + # Ensure the token is set + if [ -z "$FABRIC_USER_TOKEN" ]; then + log "No FABRIC_USER_TOKEN set." + echo 1 + return + fi + + # Extract JWT payload (assumes token format: header.payload.signature) + payload=$(echo "$FABRIC_USER_TOKEN" | cut -d '.' -f2 | sed 's/-/+/g; s/_/\//g;') + # Add missing padding if needed + mod4=$(( ${#payload} % 4 )) + if [ $mod4 -ne 0 ]; then + payload="${payload}$(printf '%0.s=' $(seq 1 $((4 - mod4))))" + fi + + # Decode payload and extract the expiration field using jq + exp=$(echo "$payload" | base64 -d 2>/dev/null | jq -r '.exp') + if [ -z "$exp" ]; then + log "Unable to parse token expiration." + echo 1 + return + fi + + # Compare expiration with current time + current=$(date +%s) + if [ "$current" -ge "$exp" ]; then + # Token is expired + echo 1 + else + # Token is not expired + echo 0 + fi +} +