Azure DevOps Templates and Variables

Hi All,

I’ve been working on a project that made use of Azure Data Factory and Azure DevOps recently.  In this post I would like to explain how I went about creating my Azure Devops Pipeline, this pipeline is a multi stage pipeline created in YAML, I am not using the classic release pipelines.

I’m going to assume you have some experience in Azure DevOps before I continue.

Repo Folder Setup

I would like to show you how my DevOps repo has been setup, my repo is called “KDH-People-Workforce” in this repo I have created the following folders:

• data-factory
  • md-organisation
    • credentials
    • data-flows
    • datasets
    • linked-services
    • pipelines
• pipelines
  • build
    • ci-templates
    • data-factory
      • ci-release
        • dev

I’ve included an image of my repo layout as well.

The Template

Within the ci-templates folder,  I created a YAML template file called DeployADFResources-template.yaml

the full code is shown below.

Discussing The Code

Now lets discuss the above YAML code.

1. Template Parameters

I am making use of template parameters, this template accepts the below parameters to make it reusable:

• subscription: Azure subscription ID where the resources are managed.
• resourceGroup: Azure resource group containing the ADF resources.
• datafactoryName: The name of the Azure Data Factory instance.
• workingDirectory: Directory path containing JSON files for the ADF resources.
• adfResourceType: The type of ADF resource to manage (e.g., pipelines, datasets).

2. First Task: Reading and Combining JSON Files

I’m using PowerShell to read the JSON files for the different ADF resources.

• Task: PowerShell@2
• Purpose: Reads JSON files in the specified directory and combines them into a single JSON array, then encodes this array in Base64 format.

Key Steps in the Script

1. Initialize Folder Path:
   $folderPath = "${{parameters.workingDirectory}}"

2. Check if Folder Exists:
   • If the folder exists, read all .json files.
3. Read JSON Files:
   • Use Get-ChildItem to find .json files in the directory.
   • Use ConvertFrom-Json to parse the content of each file into a PowerShell object.
   • Collect these objects in $jsonArray.
4. Combine and Encode JSON:
   • Convert the JSON array to a string using ConvertTo-Json.
   • Encode the string into Base64 using [Convert]::ToBase64String.
5. Store Output as Variable:
   • Use Write-Host with the ##vso[task.setvariable] syntax to set a variable (JsonResult) that can be accessed in later tasks.

3. Second Task: Creating/Updating ADF Resources

• Task: AzurePowerShell@5
• Purpose: Decode the combined JSON array, validate it, and make API calls to create or update ADF resources.

Key Steps in the Script

1. Decode Base64 JSON:
   $decodedJson = [System.Text.Encoding]::UTF8.GetString([Convert]::FromBase64String("$(SETVAR.JsonResult)"))

2. Validate JSON:
   • Use ConvertFrom-Json to parse the decoded string.
   • Log the parsed JSON contents for debugging.
3. Get Azure Token:
   • Retrieve an access token using Get-AzAccessToken for authorization in API calls.
4. Iterate Over JSON Array:
   • For each JSON object in the array:
     • Construct the API endpoint URL for the ADF resource type and name.
     • Define headers for the API request, including the token.
     • Convert the resource body back to JSON format.
     • Make a PUT API call to create or update the ADF resource.
5. Log Response:
   • Log the status code of the API call for each resource.

The YAML Pipeline

With the above template I now need to create the YAML release pipeline I named this pipeline “ci-build-release-md-organisation-dev.yaml“.  the code for this is showing below.

Discussing the Code

1. Triggers

• Branches: The pipeline triggers on changes to the develop branch.
• Paths: Limits the trigger to changes in the data-factory/md-organisation folder.

2. Variables

• Azure Subscription: Specifies the Service Connection for deploying resources.
• Agent Pool: Uses the windows-latest image for the pipeline agent.
• Working Directory: Points to the folder containing the ADF resource JSON definitions.
• Environment Details:
  • subscription: Azure subscription ID.
  • resourceGroupName: Resource group where ADF resides.
  • ADFName: Name of the target Azure Data Factory.

3. Parameters

• Template Folder: Points to the folder containing the reusable deployment template (DeployADFResources-template.yaml).

4. Stages

Each stage corresponds to a specific ADF resource type, with a similar structure for deploying:

• Credentials
• Linked Services
• Datasets
• Data Flows
• Pipelines

Example: Deploy ADF Credentials

• Stage and Job:
  • Each stage contains one or more jobs for deploying a specific resource type.
  • The job’s pool uses the $(vmImage) variable.
• Reusable Template:
  • The deployment logic is extracted into the DeployADFResources-template.yaml.
  • Parameters are passed to the template, including:
    • subscription, resourceGroup, datafactoryName: Environment-specific details.
    • workingDirectory: Path to the JSON files for the resource type (credentials/dev for this stage).
    • adfResourceType: Specifies the type of resource being deployed.

Other Stages

The structure is repeated for:

• Linked Services: Uses the linked-services/dev directory.
• Datasets: Uses the datasets/dev directory.
• Data Flows: Uses the data-flows/dev directory.
• Pipelines: Uses the pipelines/dev directory.

The ADF JSON Files

I got the JSON files from the code view for the resource in ADF Studio.  I then copied this code into a new file, I needed to modify the JSON slightly to include an additional property called “body“, below is an example of json for an ADF credential.

I proceeded to do the same for all the other needed ADF resources, I’ve include an image of how my repo looked.

 

That’s all, I hope this gave you some idea on how to deploy ADF resources using Azure DevOps.