Stylized image of DevOps infinity symbol with a car driving on it and increasing speed over golden arrows.

Using Platform Hub to increase Supply Chain Security

Bob Walker
Bob Walker

Imagine this scenario: An application is built on Monday afternoon. That version is deployed to a test environment on Tuesday morning for verification. Testing is successful, and the production deployment occurs on Wednesday morning. But no one knew that early Tuesday morning a third-party package issued an update that closes a critical CVE. Ideally, a process exists to inform application teams about the vulnerability before testing starts on Tuesday and include it as part of the production deployment on Wednesday.

With Platform Hub, solving that problem at scale is much easier than ever before. In this post, I will walk you through the steps to increase supply chain security in Octopus Deploy using the Process Templates and Policies included in Platform Hub.

Nomenclature

If you read my previous post on supply chain security, you are familiar with SBOMs, Provenance, and Attestations. For those who haven’t read that post, I’ve included the definitions below to make it easier to follow along.

  • SBOM - Software Bill of Materials - a list of all the third-party libraries (and their third-party libraries) used to create the build artifact (container, .zip files, jar files, etc.).
  • Provenance - the record of who created the software change, how it was modified and built, and what inputs went into it. It shows how the build artifact was built.
  • Attestation - A cryptographically verifiable statement that asserts something about an artifact, specifically its Provenance. It is similar to the notary seal on a document. Doesn’t show the whole process, but certifies its validity.

SBOMs, Provenance, and Attestations are intertwined. Think of it like a cake.

  • SBOMs are the ingredient list.
  • Provenance is the recipe and kitchen log (who cooked it, when, and with which tools).
  • Attestation is a signed certificate that proves the ingredient list, recipe, and cooking process are trustworthy.

Responsibilities differences between build servers and Octopus Deploy

This post focuses on using Process Templates and Policies within Octopus Deploy. But they will rely on artifacts created by the build server. For example, the build server will create the SBOM, while Octopus Deploy will scan the SBOM for package references with known vulnerabilities. Below is a table showing the differences in responsibility between build servers and Octopus Deploy.

Build Server                                                                      Octopus Deploy                                                                    
Generating SBOMs                                                                 Attaching the SBOM to the release or forwarding it onto a third-party tool like Ortelius
Scanning source code referenced third-party packages for known vulnerabilities    Scanning for known vulnerabilities within package versions listed in the SBOM     
Generating Attestations                                                           Verifying attestations                                                           
Scanning recently built containers for known vulnerabilities                      Scanning containers about to be deployed for known vulnerabilities                
Create build information file for Octopus Deploy to consume                       Use the build information file to update third-party issue trackers like JIRA     

Tooling Used

For my build server, I’m using GitHub Actions because it includes built-in Attestation generation. I’ll use Trivy for vulnerability scanning and SBOM generation.

  • SBOM generation - GitHub Actions will use Trivy to create an SBOM in the spdx-json format via the provided step by AquaSecurity/trivy-action step.
  • SBOM scanning - Octopus Deploy will use Trivy to scan the SBOM for known fixed vulnerabilities.
  • Attestation generation - GitHub Actions will use the built in step actions/attest-build-provenance to create the attestation.
  • Attestation verification - Octopus Deploy will use the command gh attestation verify provided by GitHub CLI to verify the attestation.
  • Container scanning Post-build - GitHub Actions will use Trivy to scan the container for known fixed vulnerabilities after the container is built.
  • Container scanning Pre-deployment - Octopus Deploy will use Trivy to scan the container for known fixed vulnerabilities before a deployment.

In Octopus Deploy, I opted for execution containers instead of installing Trivy directly on my workers. There are two execution containers with Trivy installed you can use today:

Process Templates

My Process Template will contain all the necessary logic to attach the SBOM to the release, scan the SBOM for known vulnerabilities, verify the attestations from GitHub, and scan containers for known third-party vulnerabilities.

The Process Template documentation provides a step-by-step guide to creating Process Templates. Instead of a step-by-step guide, I’ll walk you through the specific configuration for my Process Template.

Process Template Configuration

Following our best practices, I created a Process Template focused on supply chain security, Deploy Process - Attach SBOM and Verify Build Artifacts.

List of Process Templates on an instance with an arrow pointing to the Process Template to attach SBOM and build artifacts.

The Process Template currently has two steps.

  1. The first step will extract the SBOM from a package, attach it as a deployment artifact, and then run Trivy to scan for known fixed vulnerabilities.
  2. The second step will loop through a list of packages and containers, run gh attestation verify on each item, and run Trivy on any containers for any known fixed vulnerabilities.

List of steps in the example Process Template

Process Template parameters

For my Process Template, there are four Process Template parameters.

  1. Template.SBOM.Artifact - a zip file containing the SBOM for a specific application.
  2. Template.Git.AuthToken - the GitHub PAT used for gh attestation verify to function properly.
  3. Template.Verify.Workerpool - the worker pool to run all the steps.
  4. Template.Verify.ExecutionContainerFeed - the container feed for the execution containers.

List of parameters for the Process Template

Processing the SBOM

The first step will use the package from the Template.SBOM.Artifact, Template.Verify.Workerpool, and Template.Verify.ExecutionContainer parameters. As a producer of this step, I opted to hardcode the execution container instead of passing it in as a parameter. The consumer shouldn’t need to worry about that configuration.

Step to attach the SBOM and run Trivy

I opted for inline scripts because Octopus Deploy stores Process Templates in git. Referencing a third-party repo felt redundant. Below is the script I used.

$OctopusEnvironmentName = $OctopusParameters["Octopus.Environment.Name"]
$extractedPath = $OctopusParameters["Octopus.Action.Package[Template.SBOM.Artifact].ExtractedPath"]

Write-Host "The SBOM extracted file path is this value $extractedPath"
      
$sbomFiles = Get-ChildItem -Path $extractedPath -Filter "*.json" -Recurse

foreach ($sbom in $sbomFiles)
{
  Write-Host "Attaching $($sbom.FullName) as an artifacts"
  New-OctopusArtifact -Path $sbom.FullName -Name "$OctopusEnvironmentName.SBOM.JSON"  

  Write-Host "Running trivy to scan the SBOM for any new vulnerabilities since the build was run"
  trivy sbom $sbom.FullName --severity "MEDIUM,HIGH,CRITICAL" --ignore-unfixed --quiet

  if ($LASTEXITCODE -eq 0)
  {
    Write-Highlight "Trivy successfully scanned the SBOM and no new vulnerabilities were found in the referenced third-party libraries."
  }
  else
  {
     Write-Error "Trivy found vulnerabilities that must be fixed before this application version can proceed. Please update the package references and rebuild the application."
  }
}

Verifying attestations

The second step in the Process Template is much more complex. The initial configuration is similar to the first step. It uses the Template.Git.AuthToken, Template.Verify.Workerpool, and Template.Verify.ExecutionContainer parameters. Just like with the first step, the execution container is hardcoded.

Step to verify attestations and run Trivy

The complexity stems from decisions I made as the producer of the step to make it easier for consumers to use the template.

  • Maximum re-use: the Process Template must support 1 to N packages/containers. The consumer shouldn’t need to provide a list of those packages/containers either. That information is stored in the variable manifest.
  • The consumer shouldn’t have to hardcode information that is already included in the variable manifest. For example the GitHub repo is stored in the build information variables.

The gh attestation verify command presented two challenges. That command needs a hash to lookup the attestation.

  1. For zip / JAR / WAR / NuGet files the hash is created from the file itself. gh attestation verify requires the path to the folder.
  2. For containers the digest hash is used. For private container repositories, verification must occur prior to invoking gh attestation verify.

For my applications, I have the following benefits:

  • All services and websites are hosted on Kubernetes clusters.
  • All containers publicly accessible on hub.docker.com.
  • All deployments to database backends or other services occur via a Kubernetes worker running on the same cluster.

That allowed me to perform a couple of shortcuts unique to my configuration.

  • Octopus will download all containers/packages to the same Kubernetes cluster at the start of the deployment.
  • Any package (including the SBOM package) needed for the deployment is stored in the octopus/files directory.
  • Being public containers on DockerHub, I didn’t have to worry about authentication when pulling the digest for the container.

Below is the PowerShell that works for my configuration. It will require some modifications if you wish to include it in your instance. I’m providing it for example use only.

$gitHubToken = $OctopusParameters["Template.Git.AuthToken"]

$buildInformation = $OctopusParameters["Octopus.Deployment.PackageBuildInformation"]
$OctopusEnvironmentName = $OctopusParameters["Octopus.Environment.Name"]

Write-Host "Getting a list of packages and containers to attest to from the variable manifest"
$objectArray = @()
foreach ($key in $OctopusParameters.Keys)
{
  if ($key -like "*.PackageId")
  {
    Write-Host "Found a package Id parameter: $key - checking to see if it already is in the packages to verify"
    
    $packageId = $OctopusParameters[$key]
    Write-Host "The package ID to check for is $packageId"

    $packageVersionKey = $key -replace ".PackageId", ".PackageVersion"
    Write-Host "The package version key is $packageVersionKey"
    $packageVersion = $OctopusParameters[$packageVersionKey]
    Write-Host "The package version is $packageVersion"

    $packageVersionToVerify = "$($packageId):$($packageVersion)"

    if ($objectArray -contains "$packageVersionToVerify")
    {
      Write-Host "$packageVersionToVerify already exists in the array"
    }
    else
    {
      Write-Host "$packageVersionToVerify does not exist - adding it"
      $objectArray += $packageVersionToVerify
    }    
  }  
}

Write-Host "Getting the GitHub repository from the build information"
$buildInfoObject = ConvertFrom-Json $buildInformation
$vcsRoot = $null

Write-Host "Getting the repo name from build information"
foreach ($packageItem in $objectArray)
{
  $artifactToCompare = $packageItem.Trim().Split(':')
  $packageVersion = $artifactToCompare[1]
  
  Write-Host "The version to look for is: $packageVersion"
  
  foreach ($package in $buildInfoObject)
  {
    Write-Host "Comparing $($package.Version) with $($packageVersion)"
    if ($packageVersion -eq $package.Version)
    {
      Write-Host "Versions match, getting the build URL"    
      $vcsRoot = $package.VcsRoot
      Write-Host "The vcsRoot is $vcsRoot"    
    }
  }
}

if ($null -eq $vcsRoot)
{
  Write-Error "Unable to pull the build information URL from the Octopus Build information using supplied versions in $packageName. Check that the build information has been supplied and try again."
}

$githubLessUrl = $vcsRoot -Replace "https://github.com/", ""

$env:GITHUB_TOKEN = $gitHubToken

Write-Host "Verifying the attestation of all the found packages and containers."
foreach($packageItem in $objectArray)
{    
  Write-Host "Verifying $packageItem"
  $artifactToCompare = $packageItem.Trim().Split(':')
  $packageName = $artifactToCompare[0].Replace("/", "")
  
  if ($packageItem.Contains("/"))
  {
      $imageToAttest = "oci://$packageItem"

      Write-Host "Attesting to $imageToAttest in the repo $githubLessUrl"
      $attestation=gh attestation verify "$imageToAttest" --repo $githubLessUrl --format json 

      if ($LASTEXITCODE -ne 0)
      {
         Write-Error "The attestation for $packageItem could not be verified"
      }

      Write-Highlight "$packageItem successfully passed attestation verification"
      Write-Verbose $attestation

      Write-Host "Writing the attest output to $packageName.$OctopusEnvironmentName.attestation.json"
      New-Item -Name "$packageName.$OctopusEnvironmentName.attestation.json" -ItemType "File" -Value $attestation
      New-OctopusArtifact -Path "$packageName.$OctopusEnvironmentName.attestation.json" -Name  "$packageName.$OctopusEnvironmentName.attestation.json"

      Write-Host "Running trivy to check the container for any known vulnerabilities that might have been discovered since the build."      
      trivy image --severity "MEDIUM,HIGH,CRITICAL" --ignore-unfixed --quiet $packageItem
      if ($LASTEXITCODE -eq 0)
      {
          Write-Highlight "Trivy successfully scanned $packageItem and no new vulnerabilities have been found in the container or base containers since they were built."
      }
      else
      {
         Write-Error "Trivy found vulnerabilities in the build artifacts that must be fixed. You can no longer deploy this release. Please update the base container version, rebuild the application, and create a new release."
      }
  }
  else
  {    
    if (Test-Path "/octopus/Files/")
    {
      Write-Host "$artifactToCompare is a package from our local repo, getting the information from /octopus/Files/"
      $zipFiles = Get-ChildItem -Path "/octopus/Files/" -Filter "*$($artifactToCompare[0])*$($artifactToCompare[1])@*.zip" -Recurse
    }
    else
    {
      Write-Host "$artifactToCompare is a package from our local repo, getting the information from /home/Octopus/Files"
      $zipFiles = Get-ChildItem -Path "/home/Octopus/Files" -Filter "*$($artifactToCompare[0])*$($artifactToCompare[1])@*.zip" -Recurse
    }

    $artifactVerified = $false
    foreach ($file in $zipFiles) 
    {
      if (test-path "$packageName.$OctopusEnvironmentName.attestation.json")
      {
        Continue
      }
      
      Write-Host "Attesting to $($file.FullName) in the repo $githubLessUrl"
      $attestation=gh attestation verify "$($file.FullName)" --repo $githubLessUrl --format json

      if ($LASTEXITCODE -ne 0)
      {
         Write-Error "The attestation for $packageItem could not be verified - this means no attestation was generated or the package has been tampered with since it was created - stopping the deployment to avoid a security incident."
      }

      Write-Highlight "$packageItem successfully passed attestation verification"
      Write-Verbose $attestation
      $artifactVerified = $true

      Write-Host "Writing the attest output to $packageName.$OctopusEnvironmentName.attestation.json"
      New-Item -Name "$packageName.$OctopusEnvironmentName.attestation.json" -ItemType "File" -Value $attestation
      New-OctopusArtifact -Path "$packageName.$OctopusEnvironmentName.attestation.json" -Name  "$packageName.$OctopusEnvironmentName.attestation.json"
    }

    if ($artifactVerified -eq $false)
    {
      Write-Error "The attestation for $packageItem could not be verified - this means no attestation was generated or the package has been tampered with since it was created - stopping the deployment to avoid a security incident."
    }
  }  
}

Policies

Platform Hub’s Process templates are half of the solution. The other half are Policies. The Policies in Octopus Deploy can fail deployments if specific steps (or Process Templates) are not present. That same logic can be applied to runbook runs, but obviously, they’d require different steps.

How Policies work

I want to explain Policies because they are a new concept in Octopus Deploy. Our policy engine uses Rego to query Octopus Deploy. The end goal of the policy engine is to provide “hooks” into various actions within Octopus Deploy. The first “hook” we provide is executing Deployments or Runbook Runs.

When a Deployment or Runbook Run occurs, Octopus Deploy will send input information to the policy engine that looks similar to the example below.

Input:
{
  "Environment": {
    "Id": "Environments-42",
    "Name": "Test",
    "Slug": "test"
  },
  "Project": {
    "Id": "Projects-541",
    "Name": "Trident",
    "Slug": "trident-aks"
  },
  "ProjectGroup": {
    "Id": "ProjectGroups-483",
    "Name": "Kubernetes",
    "Slug": "kubernetes"
  },
  "Space": {
    "Id": "Spaces-1",
    "Name": "Default",
    "Slug": "default"
  },
  "SkippedSteps": [],
  "Steps": [
    {
      "Id": "azure-key-vault-retrieve-secrets",
      "Slug": "azure-key-vault-retrieve-secrets",
      "ActionType": "Octopus.AzurePowerShell",
      "Enabled": true,
      "IsRequired": false,
      "Source": {
        "Type": "Step Template",
        "SlugOrId": "ActionTemplates-561",
        "Version": "2"
      }
    },
    {
      "Id": "verify-build-artifacts-attach-sbom-to-release",
      "Slug": "verify-build-artifacts-attach-sbom-to-release",
      "ActionType": "Octopus.Script",
      "Enabled": true,
      "IsRequired": true,
      "Source": {
        "Type": "Process Template",
        "SlugOrId": "deploy-process-verify-build-artifacts",
        "Version": "2.5.0"
      }
    },
    {
      "Id": "verify-build-artifacts-verify-docker-containers",
      "Slug": "verify-build-artifacts-verify-docker-containers",
      "ActionType": "Octopus.Script",
      "Enabled": true,
      "IsRequired": true,
      "Source": {
        "Type": "Process Template",
        "SlugOrId": "deploy-process-verify-build-artifacts",
        "Version": "2.5.0"
      }
    },
    {
      "Id": "deploy-k8s-manifest-deploy-container",
      "Slug": "deploy-k8s-manifest-deploy-container",
      "ActionType": "Octopus.KubernetesDeployRawYaml",
      "Enabled": true,
      "IsRequired": false,
      "Source": {
        "Type": "Process Template",
        "SlugOrId": "deploy-process-deploy-to-kubernetes-via-manifest",
        "Version": "1.1.0"
      }
    },
    {
      "Id": "deploy-k8s-manifest-verify-deployment",
      "Slug": "deploy-k8s-manifest-verify-deployment",
      "ActionType": "Octopus.Script",
      "Enabled": true,
      "IsRequired": false,
      "Source": {
        "Type": "Process Template",
        "SlugOrId": "deploy-process-deploy-to-kubernetes-via-manifest",
        "Version": "1.1.0"
      }
    },
    {
      "Id": "notify-team-of-deployment-status",
      "Slug": "notify-team-of-deployment-status",
      "ActionType": "Octopus.Script",
      "Enabled": true,
      "IsRequired": false,
      "Source": {
        "Type": "Step Template",
        "SlugOrId": "ActionTemplates-101",
        "Version": "15"
      }
    }
  ]
}

The policy engine will attempt to match that input to a policy. If it matches and the policy passes, then the Deployment (or Runbook Run) can proceed. The policy’s success (or failure) will be logged to the audit log.

Policy evaluation that was logged to the audit log

Requiring a Process Template

Using the input from before, the policy to require that the Process Template is as follows:

name = "Verify Build Artifacts Required"
description = "Requires the Process Template Deploy Process - Verify Build Artifacts for all deployments"
ViolationReason = "Deploy Process - Verify Build Artifacts is required on all deployments to K8s"

scope {
    rego = <<-EOT
        # The package name MUST match the file name that is stored in git. The file name should be verify_build_artifacts_required.ocl
        package verify_build_artifacts_required 

            default evaluate := false

        # Only run this policy for deployments in the projects in the Project Group Kubernetes
            evaluate := true if {
            input.ProjectGroup.Slug == "kubernetes" 
            not input.Runbook           
        }
    EOT
}

conditions {
    rego = <<-EOT
        # The package name MUST match the file name that is stored in git. The file name should be verify_build_artifacts_required.ocl 
        package verify_build_artifacts_required

        # Assume all evaluations will fail
        default result := {"allowed": false}

        result := {"allowed": true} if {
            some step in input.Steps
            # Match using the source.SlugOrId
            step.Source.SlugOrId == "deploy-process-verify-build-artifacts" 
            # Ensure the step is enabled - if it is not then fail it.
            step.Enabled == true           
            not verify_build_artifacts_skipped
        }

        result := {"allowed": false, "Reason": "The Process Template Deploy Process - Verify Build Artifacts is required and cannot be skipped for a deployment to K8s to any environment."} if {
            verify_build_artifacts_skipped
        }

        verify_build_artifacts_skipped if {
            # Fail the evaluation if the user elects to skip the Process Template when creating the deployment
            some step in input.Steps
            step.Id in input.SkippedSteps
            step.Source.SlugOrId == "deploy-process-verify-build-artifacts"
        }
    EOT

}

Bringing everything together

Adding the Process Template to the deployment process is the same as adding any other step to a deployment process.

Adding a Process Template to the deploy process with parameters being populated

A deployment to the test environment will then show both Policies and Process Templates in action. Because the Process Template is part of the deployment the Policy check passes and the deployment was successful.

Deployment with Policies and Process Templates

The eagle-eyed among you will likely notice that my scripts fail the deployment when a Trivy scan fails. What if you need to deploy a change to fix a show-stopping bug?  I solve that by using guided failures. When Trivy or an attestation verification fails, the deployment pauses and waits for a human to intervene. They decide to ignore the failure or cancel the deployment. Regardless of the decision, that information is logged in the audit log.

Conclusion

I’m not naive enough to believe what is described in this post will 100% secure the software supply chain. Leveraging Process Templates and Policies in Platform Hub makes it much easier to secure the software supply chain in Octopus Deploy. Using Pull Requests in GitHub, Process Templates, Policies, ITSM, and RBAC in Octopus Deploy, it’s much easier to get to SLSA Level 4 than ever before. The Process Template includes the necessary steps to guarantee the build artifact about to be deployed hasn’t been tampered with and no new fixed vulnerabilities have been reported. Policies guarantee that each deployment to production includes the appropriate Process Template.

Happy deployments!

Tags:

Bob Walker

Related posts