Architecture Breakdown

fabric_cicd Module

FabricWorkspace Class:

• The orchestration layer of fabric-cicd. It manages the deployment lifecycle – Retrieval, Processing, and Deploying
• Retrieval:
  • In this stage, the class retrieves the metadata of items and folders (repository and workspace), the parameters necessary for managing item deployment.
  • The retrieved information is stored into separate dictionaries: repository_items, deployed_items, repository_folders, deployed_folders, and environment_parameter
• Processing:
  • In this stage, the class processes item source control files in preparation for deployment via the Fabric REST APIs. This step is key to ensuring the item payload is valid for making a call to the item definition APIs
  • Various file processing can take place including custom file processing (defined per item type), replacement of logical Ids, replacement of parameterized values, and replacement of default workspace ids
  • Note: the only file that cannot be modified during processing is the .platform file to maintain item integrity
• Deploying:
  • In this stage, the Fabric Workspace Class performs publish/unpublish operation on each Fabric item through an API call which defines the actual deployment

Deployment Operations:

Deployment is coordinated through 2 kinds of operations: publishing and unpublishing. These operations are orchestrated by functions defined in publish.py.

publish.py contains 3 user-facing functions:

1. publish_all_items()
2. unpublish_all_orphan_items()
3. deploy_with_config() (a wrapper of the first 2 functions)

The specific order of Fabric item types during publish/unpublish is defined logically in the publish_all_items() and unpublish_all_orphan_items(), based on dependencies that exist between item types.

Item Dependency Management:

Types of Dependencies:

1. Dependencies between item types, ex: Notebook dependent on Environment
2. Dependencies within item types, ex: Pipeline dependent on Pipeline

Dependency Management during Deployment:

• Publish case:
  • Item types published according to logical order of publish_{itemtype-plural}() functions called in publish_all_items()
  • Within item type, logic sits at item level (publish_{itemtype-plural}()) to sort specific items within the given type.
• Unpublish case:
  • Item types unpublished according to ordered list defined in unpublish_all_orphan_items() (reverse order of the order set in publish_all_items())
  • Within item type, sorting based on dependencies is also managed within the unpublish_all_orphan_items() function.

Constants:

• The constants file is where constants are centrally defined and used across various files including the FabricWorkspace and Parameter classes
• The type of constants includes lists, regex patterns, dictionaries, strings
• Users have the option to override a constant value for a deployment instance
• The library version number is also defined in constants.py

Changelog:

• The changelog.md file is a markdown file that contains the change updates per release
• Specific emojis are used to categorize the specific change
• Under the title which includes the release version and release date, the listed changes include a link to the associated GitHub issue

Common Module (_common)

API Management:

• FabricEndpoint class is the communication layer between fabric-cicd and Microsoft Fabric’s REST APIs
• It handles HTTP requests to Fabric and manages authentication, retries, errors, and long-running operations automatically. Includes comprehensive logging.
• Authentication Management:
  • Uses the azure-identity library to manage the actual authentication while FabricEndpoint handles Azure AD token acquisition, token expiration, and token refresh.
• Resilience:
  • Fabric Endpoint includes retry logic, throttling handling, and error recovery.
• Long-Running Operations:
  • Polls asynchronous operations until completion, specifically polling the location header URL with GET operation requests, checks the status (succeeded, failed, running) until completion.
• Comprehensive Logging:
  • Provides detailed debug information for troubleshooting by printing API requests and responses to the terminal and also adds API traces to an error log file that gets created during each deployment run.
• Flow:
  • The FabricEndpoint object gets created within the FabricWorkspace class. The endpoint object then uses the passed in Azure TokenCredential object (upn, spn, mi) to authenticate when running the APIs.
  • Initialization:
    • Stores the Azure TokenCredential object (upn, spn, mi, etc.)
    • Immediately acquires an access token via _refresh_token() method
    • Sets up the requests module
    • Initializes the aad_token – the actual Bearer token sent with API requests, aad_token_expiration – when the token expires (automatically refreshed), token_credential – Azure credential object (from the azure-identity library).
  • invoke() Method:
    • The core method of the class – the method that is called in FabricWorkspace and in other files whenever calling an API
    • The method invokes an API call/sends an HTTP request to the specified URL using the provided API method and body, if applicable
    • Accepts other optional parameters and kwargs.
    • invoke() executes the following steps:
      1. Request Loop:
         • Supports operations with multiple HTTP requests (LRO)
      2. Build Headers:
         • Header gets passed into API call which specifies the type of file content in the request body (usually JSON content, but option to set other forms)
      3. Send Request:
         • Sends the HTTP request using the requests library
      4. Handle Response:
         • Different kinds of responses are handled based on the status code including LRO (regular and unique case), throttling, token expiration. Performs retries which use exponential backoff
      5. Return Standardized Response:
         • Returns a dictionary as the response with the header, body, and status code information which can be extracted
• Token Management:
  • Automatic token refresh (called within the invoke method) and uses helper function to decode the actual token to get token information including expiration, executing identity (upn, appid, oid)
• Response Handling:
  • Various responses can be returned by the HTTP requests which require specific handling. For example:
    • Long-Running Operations (202):
      • Without polling (special case) – parameter of invoke() set to specifically not poll until completion but instead exit the request loop
      • With polling (default behavior) – polls until completion, uses the GET operation to check status
    • Throttling (429):
      • Does a retry up to 5 attempts, with exponential backoff to avoid overwhelming the API
    • Item Name Conflicts:
      • Does a retry up to 5 attempts, with exponential backoff
    • Authorization Errors:
      • Raises an exception, resulting in deployment failure

Configuration Deployment Management:

Logic that is called within the deploy_with_config() function used in the Configuration Deployment approach.

Configuration utilities:

• Contains various functions responsible for loading a validated config.yml file to a Python dictionary, extracting the settings provided, and handling config value overrides

ConfigValidator class:

• A class responsible for the validation of user inputs provided in the config.yml file
• Validates the YAML contents and returns a Python dictionary

Formatting, Logging, and Error Handling:

• Color:
  • Color formatting defined and used in the console
• Logging:
  • Includes logging utilities
  • Defines different logging levels through specific formatting based on log type, e.g., debug (grey), error (red), info (white), warning (yellow)
• Custom Exceptions:
  • Custom exceptions specially defined for fabric-cicd
  • Used in error-handling

Item Metadata Management:

Item class:

• Used to store item metadata and process items during deployment

File class:

• Used to store file metadata and process source control files of a given item during deployment

Validation:

• Group of validation functions used to validate user input, such as workspace id, repository directory, environment, etc.
• Group of functions to check file types and library version

Items Module (_items)

• A file defined per Fabric item type supported in fabric-cicd
• Used specifically in the publishing of items to a Fabric workspace
• Contains a wrapper publish function which accepts a FabricWorkspace object and calls the _publish_item() method from FabricWorkspace class
• The file may include custom logic for special handling of the given item type during deployment (pre/post-processing)
• The wrapper functions of the item types are called within the publish_all_items() function in a defined order

Parameter Module (_parameter)

Supported Parameters:

• find_replace
• key_value_replace
• spark_pool
• semantic_model_binding

Parameter Utilities:

• Group of varying functions used in the FabricWorkspace and Parameter classes
• Functionalities include: parameter value extraction, parameter validation, and parameter processing

Parameter Validation:

• Validation is managed by the Parameter class
• Responsible for loading the parameter.yml file to a Python dictionary and validating its contents
• Validation areas include: file existence, YAML syntax, load, structure, data types, specific parameters (names, key/values, specific properties), environment keys, optional filters

Advanced Features:

• Include using regex pattern for find_value, variables for replace_value, file filters, etc.
• See documentation here

Parameter Templates:

• Use multiple parameter files and choose to apply or not apply during a deployment
• Splitting large parameter file into smaller files
• Reference parameter template paths within the main parameter.yml file
• See documentation here