Triggering Azure Pipelines with an offline source repository

This blog post is part of a series of posts covering the Challenges adopting the Azure DevOps SaaS offering in highly regulated industries. The first post covered some of the connectivity challenges experienced by some the enterprises which I have worked with, and some high-level options for mitigating these challenges.

I will not be addressing the Azure DevOps Server Self-host option which is the easiest way to deploy Azure DevOps to an air gapped environment, my goal is to introduce native cloud technologies which reduces the operations burden placed on these enterprises.

I will also not directly address how enterprises can whitelist incoming Azure DevOps traffic in their firewalls, but at a high level - keep in mind that an Azure DevOps customer's Organization is region bound and traffic will originate from that Azure Regions IP address space.  The Azure DevOps team are working to improve this story, this revolves around defining a new set of Azure Service Tags consisting of a smaller static IP Range.

There are currently two possibilities when leveraging Azure Pipelines.

1. YAML Pipelines, which is the modern approach to building pipelines and now supports both Continuous Integration & Continuous Deployment.
2. Release Pipelines, which is used solely for Continuous Deployment.

So, what does the connectivity look like when we talk about Azure Pipelines.

The key areas where we will see connectivity issues based on the above diagram are highlighted in the colour red these can be broken into two broad categories.

Azure DevOps Service Integration connectivity.

• YAML definition files are stored in Source Control, if you bring your own Source Control(GitHub Enterprise Server, Subversion or Some other Git based product) the Azure DevOps service requires connectivity to query the repository contents to discover & manage these YAML files.
• Automated setup of pipeline triggers, both YAML and Release Pipelines support the concept of triggers - one example of this is configuring a webhook on a GitHub repo as a CI/CD trigger.
• Pipeline Triggers & Release Artefacts typically allow a version to be provided, these values can be provided manually when executing a pipeline. The Azure DevOps Release Pipelines UI will also help you by allowing you to  select a version from a pre-populated list, this is done by querying for example in the case of GitHub the current commit at the head of the ‘master’ branch.

From a Developers perspective integration is where they will be impacted the most both from a usability perspective as well as fundamentally how their CI/CD process will be configured. Historically, in this scenario I have seen enterprises achieve success if they combine their existing CI infrastructure (Jenkins/TeamCity etc) and Release Pipelines. The reason being that up until today YAML pipelines supported only manual or scheduled triggers for a private resource deployed to a company's network i.e. GitHub Enterprise Server.

Azure DevOps Agent Connectivity.

• Polling for new Pipeline jobs to be executed on the Agent
• Publishing pipeline artefacts like build/release logs, build artefacts etc back to the Azure DevOps Service

By Self-hosting, their own Azure DevOps agents' enterprises can address most of the Azure DevOps Agent connectivity scenarios described above. They can run these agents both on-premises and/or within their cloud environment, the key being that the agent will need network connectivity to whatever resources are required during pipeline execution.

The Azure Pipelines team have been working on a spec and a new feature which will also ease the operations burden and cost for running and scaling these Self-Hosted agents leveraging the concept of a  "Elastic Agent Pools", initially they are supporting Azure Virtual Machine ScaleSets in a Private Preview.

YAML Pipelines offline Triggers

For the remainder of this post let's explore some of the changes which are coming down the pike which could be used to successfully configure CI/CD for an offline resource such as GitHub Enterprise Server.

The Azure Pipeline team are planning to introduce a Web Hook Trigger Resource to enable us to trigger our YAML Pipelines based on a simple call to the Webhook's HTTP Endpoint. This will allow us to setup our triggers without making any changes to our enterprise firewalls to allow additional inbound traffic.

In this scenario we will not be able to store the Pipelines YAML definition in our private GitHub Enterprise Repository therefore - I recommend leveraging Azure Repos for this task.

High-level the connectivity for this scenario would look something like the following.

Pre-requisites

• Windows or Linux based Azure DevOps Self-hosted Agent Pool.
• Azure DevOps Service Connection for our GitHub Enterprise Server do not click "verify" the Azure DevOps Service is unable to perform verification for offline resources.

• GitHub Enterprise Server Personal Access Token to be used with the Web hook creation script, with admin:repo_hook scope selected.

High-level Steps

1. Create a new "Incoming Webhook" service connection, this is a new Service Connection Type which will be introduced which allows us to define two important pieces of information.
   • HTTP Header - the name of the Http Header in the request which contains the payload hash value for request verification. In the case of the GitHub webhook request the header will be "X-Hub-Signature".
   • Secret - the shared secret which is used to compute the payload hash used for verification of the incoming request.
     
2. Setup our Azure Pipeline for CI or CD, this Pipeline will be triggered each time Azure DevOps receives a request for the incoming the web hook endpoint. I'm using the Pipeline tasks provided by the "GitHub Enterprise Release Artifact" extension to clone the repository. I'm also filtering the incoming Web Hook requests based on the source GitHub Repo & leveraging Pipeline parameters to access data related to the incoming request payload. We can identify the Branch and Commit Id and other important information related to the request. This Azure Pipeline could be used to build or deploy application artefacts.

   trigger: none
   pool: SelfHostedAgentPool
   
   resources:
     webhooks:
       - webhook: githubrepotrigger
         connection: github-repo-trigger
         filters:
           - path: repository.full_name
             value: gareth/HighlyRegulatedLab1
   
   steps:
   - checkout: none
   - task: DownloadArtifactsGitHubEnterprise@1
     displayName: 'Clone GitHub Enterprise Repository'
     inputs:
       GithubEnterpriseConnection: 'private-github-ent'
       Repository: ${{ parameters.githubrepotrigger.repository.full_name}}
       Branch: ${{ parameters.githubrepotrigger.repository.default_branch}}
       CommitId: ${{ parameters.githubrepotrigger.head_commit.id}}
   

3. Setup a web hook on the GitHub Enterprise Server repository which will trigger the CI/CD pipeline, below is a simple script for automating the web hook creation on a GitHub Enterprise Server Repository.

   param(
       [String] $githubBaseUrl,
       [String] $githubRepo,
       [String] $githubUsername,
       [String] $githubPAT,
       [String] $devOpsOrg,
       [String] $devOpsWebHookName,    
       [String] $devOpsSharedSecret
   )
   
   #optimzation for running non-interactively
   $progressPreference = 'silentlyContinue' 
   # Convert Json utils
   Import-Module Microsoft.PowerShell.Utility -Force
   
   # Trust Self signed Certificates
   [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true};
   
   # Setup our basic auth header value
   $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($githubUsername):$githubPAT")) 
   
   # compute the webhook url which should be used
   $devOpsWebHookUrl = "https://dev.azure.com/$devOpsOrg/_apis/public/distributedtask/webhooks/$($devOpsWebHookName)?api-version=6.0-preview"
   
   # Setup the payload for the GitHub WebHook creation request
   $body = @"
   {
       "name": "web",
       "active": true,
       "events": [
           "push"
       ],
       "config": {
           "secret":"$devOpsSharedSecret", 
           "url": "$devOpsWebHookUrl", 
           "content_type": "json"
       }
   }
   "@
   
   try{
   
       $response = Invoke-WebRequest -Method post -Uri "$githubBaseUrl/api/v3/repos/$githubRepo/hooks" -Body $body -ContentType "application/json" -Headers @{ "Authorization" ="Basic $encodedPAT" }
   
       Write-Host "Successfully configured WebHook"
   
   }
   catch
   {
       Write-Error "Failed to configure WebHook"
       Write-Host "StatusCode:" $_.Exception.Response.StatusCode.value__
       Write-Host "StatusDescription:" $_.Exception.Response.StatusDescription
   }
   

4. Developers would create & manage their YAML Pipeline definitions as normal in the Azure Pipelines UI and these would be stored in an Azure Repo Git repository.
5. When developers Push a new change to the repository the GitHub webhook will be triggered.
   
6. A new Azure Pipeline Run will be created each time the Incoming Webhook Resource is triggered, based on our Pipeline definition the first thing that will happen is that the GitHub Enterprise repository will be cloned onto the Agent.
   
   and we can see that the GitHub Enterprise Repository was successfully downloaded
   

Once this feature becomes available in Public Preview you will be able to try this out in your Azure DevOps Organization.