Octopus Deploy and GitHub logos connected by arrows.

Supply chain security with GitHub Actions and Octopus Deploy

Bob Walker

In May 2021, in response to a series of high-profile cyber attacks, President Biden issued Executive Order 14028 to improve the nation’s cybersecurity. A big reason for this was the SolarWinds supply chain attack in which Russian state-sponsored hackers compromised the Orion software platform. Section 4 specifically addressed that attack:

The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors. There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended.

Since 2021, we’ve seen a lot of new functionality to enable supply chain security. In this blog post, I will walk you through improving your supply chain security by leveraging GitHub, GitHub Actions, and Octopus Deploy.

Disclaimer

I’m under no illusions that this article will be the be-all and end-all solution for supply chain security. These solutions will not cover other avenues of attack. The solutions use new functionality added to both platforms since 2021, pre-existing functionality, and common-sense configurations.

A great third-party resource is Supply-chain Levels for Software Artifacts or SLSA(pronounced “salsa”). For additional information, consult your CISO (Chief Information Security Officer), security team, or other company experts. As with anything security-related, check your company’s policies to ensure compliance.

Nomenclature

This article will use relatively new terms in software deployment pipelines. I’ve included the definitions below to make it easier to follow along.

  • SBOM - Software Bill of Materials - a list of all the third-party libraries (and their third-party libraries) used to create the build artifact (container, .zip files, jar files, etc.).
  • Provenance - the record of who created the software change, how it was modified and built, and what inputs went into it. It shows how the build artifact was built.
  • Attestation - A cryptographically verifiable statement that asserts something about an artifact, specifically its Provenance. It is similar to the notary seal on a document. Doesn’t show the whole process, but certifies its validity.

SBOMs, Provenance, and Attestations are intertwined. Think of it like a cake.

  • SBOMs are the ingredient list.
  • Provenance is the recipe and kitchen log (who cooked it, when, and with which tools).
  • Attestation is a signed certificate that proves the ingredient list, recipe, and cooking process are trustworthy.

Supply chain security is more than SBOMs, Provenance, and Attestation

SBOMs, Provenance, and Attestations are new concepts to the typical software deployment pipeline. It is tempting to focus solely on them, but that is like only worrying about the tires on a car. There is much more to it. RBAC controls, branch protection policies, audit log streaming to SIEM, key vaults, approvals from ITSM, and authentication/authorization for cloud accounts, to name a few.

Responsibilities differences between GitHub and Octopus Deploy

Clear boundaries between tooling in the deployment pipeline are essential. It sets expectations within your organization, avoids overlapping effort, and enables the tooling to be used as designed.

I’ve seen many instances where tooling is misused because it supports a simple use case. For example, approvals should be handled via ITSM tooling. But Octopus Deploy has the manual intervention step. It was initially designed to let a deployment perform an act (generate a delta report), pause, and let someone review before proceeding. But for years, users tried to use it for deployment approvals. They wanted rules such as “the person who made the change can’t approve it.”

The responsibilities of GitHub and Octopus Deploy in a secure pipeline are as follows:

GitHub                                                              Octopus Deploy                                                            
Branch Protection Policies                                          Environmental progression and release orchestration                        
Pull Request workflow                                                Centralized dashboard for deployment status and latest version            
Linting, static code analysis, and vulnerability scanning            Creating and tracking approvals in ITSM tooling                            
Automated Testing (unit tests, integration tests, etc.)              RBAC and separation of duties for production deployments                  
Creating and publishing build artifacts (packages, containers, etc.)Ingesting SBOMs and verifying Attestations                                
Generating SBOMs and Attestations                                    Environmental modeling of infrastructure                                  
Calculating version numbers                                          Authentication and authorization to deployment targets and cloud providers
Creating releases in Octopus Deploy                                  Creating and destroying ephemeral environments                            

Example Pipeline

All the screenshots and scripts come from a real-world example I put together for demos and webinars. All the source code, builds, and deployment process are stored in a public repo on GitHub as well as my publicly available Octopus Cloud instance. You can log in as a guest.

As you can see, my GitHub workflow builds, tests, scans the code, generates the attestations, and then hands over to Octopus Deploy.

The latest GitHub action run from the example repo

The deployment process will pull secrets from an Azure key vault, attach the SBOM as a deployment artifact, verify the attestations of all the build artifacts, build a database delta report, check the report, and finally deploy the database and website changes.

The latest Octopus Deploy deployment from the example repo

Supply chain security doesn’t mean slowing down the pipeline. You’ll notice many steps in the build workflow can run in parallel to speed up the overall process. A fast pipeline is just as important as a secure pipeline. Developers will hesitate to check in if it takes two hours to build, test, and scan the code. From the logs, you can see it took 7 minutes to build and deploy the application; 3 minutes for GitHub action and 4 minutes for Octopus Deploy.

Deployment Pipeline Rules

I created deployment pipeline with the following rules:

  1. At least one person must review every change that can impact Production, be it source code, database schema, or production deployment.
  2. All changes must be made in a branch and merged via a pull request. The main branch must always be deployable.
  3. Builds must run anytime source code changes.
  4. All unit tests and third-party vulnerability scans must run for every build.
  5. All builds must generate attestations for their build artifacts.
  6. All deployments must verify the attestations before making changes.
  7. All interactions between systems (GitHub -> Octopus, Octopus -> Azure, etc.) must use OIDC whenever possible.
  8. When OIDC is impossible, store any secrets/passwords in a key vault that enables secret rotation.

GitHub Configuration

The focus for GitHub will be on securing code from tampering and minimizing third-party vulnerabilities. As such, the scope of the GitHub configuration extends beyond the GitHub build action.

Branch rulesets

To ensure a review occurs for any change that impacts the product, the main or primary branch should have an appropriate ruleset that will:

  • All commits are required to be made on a separate branch.
  • A pull request with at least one approval is required.
  • A code scanning result is required for critical alerts.

Consult with your security team or CISO for other appropriate settings for your company.

Built-in code scanning

Use the scanning GitHub provides to find problems proactively. These include:

  • Enabling Dependabot to alert you of vulnerabilities that impact your dependencies.
  • Enabling Secret Scanning to alert you if a secret is accidentally checked into version control.

These settings will proactively find problems in your codebase instead of waiting for a build to run.  

General GitHub Actions settings

Before getting into SBOMs and attestations, use the following with your GitHub Actions.

  • Use Secrets and Variables instead of storing them directly in the GitHub action.
  • Use OIDC to login to Octopus Deploy. Do not use log-lived API keys. When configuring the service account user in Octopus Deploy, you must provide a subject for the OIDC authentication to work. On my instance, I’m a little more relaxed about the subjects.
    • Pull Request Subject: repo:BobJWalker/*:pull_request - accept any connection from any pull request action in the BobJWalker repo.
    • Builds: repo:BobJWalker/*:ref:refs/heads/* - accept any connection from any build in the BobJWalker repo.
  • Trigger builds for main or primary branch, hotfix and feature branches, and manually.   Set up filters for specific folders or files to avoid a no-op build. For example, my build workflow triggers are:
on:
  push:
    branches: 
      - main
      - 'feature/**'
      - 'features/**'
      - 'hotfix/**'
    paths:
      - 'src/**'
      - 'db/**'
      - 'k8s/**'
      - '.github/workflows/build.yml' ## This is in here so I get a new build with each build change I make     
  workflow_dispatch:

Calculating Versions

Believe it or not, calculating the version number was the most challenging part of this entire process. I’m using SemVer and want to remain close to the rules.

  • Builds from the main branch would be use: {Major}.{Minor}.{Patch}, e.g. 6.16.26. The main branch is the only thing that can go to Production.
  • Builds from any other branch would use: {Major}.{Minor}.{Patch}-{EscapedBranchName}.{CommitsSinceVersionSource}, e.g. 6.16.27-feature-singleton.1. The escaped branch name would include feature or hotfix. The code on these branches is pre-release and should only go to Development for testing.
  • The first check-in to a non-main branch would auto increment the {Patch} using the version from the main branch. After that, only the {CommitsSinceVersionSource} would be incremented for the non-main branch check-ins. By default, all changes must be backward compatible.
  • Increasing the {Major}.{Minor} would be a manual process accomplished using commit messages, e.g., +semver: major for significant increments and +semver: minor for minor increments. The developer is making a deliberate decision to make a non-backward-compatible change, which they document via a git commit.

Of the tools I tried, I kept returning to gitversion. It met 90% of my needs with no additional configuration. But, it doesn’t work well with dynamic version formats based on branch names. The result is a step with a small amount of business logic and output variables.

runs-on: ubuntu-latest
name: Determine version
outputs:       
  sem_ver: ${{ steps.determine_version.outputs.AssemblySemFileVer }}
steps:      
  - name: Set environment variable based on branch
    id: set_env_var
    run: |
      echo "GITHUB_REF: $GITHUB_REF"
      echo "GITHUB_HEAD_REF: $GITHUB_HEAD_REF"
      BRANCH_NAME="${GITHUB_REF#refs/heads/}"
      echo "Branch detected: $BRANCH_NAME"
      
      if [ "$BRANCH_NAME" = "main" ]; then         
        echo "GIT_VERSION_INCREMENT=Patch" >> $GITHUB_ENV            
        echo "GIT_VERSION_MODE=ContinuousDeployment" >> $GITHUB_ENV 
        echo "GIT_VERSION_FORMAT={Major}.{Minor}.{Patch}" >> $GITHUB_ENV        
      else            
        echo "GIT_VERSION_INCREMENT=Patch" >> $GITHUB_ENV            
        echo "GIT_VERSION_MODE=ContinuousDelivery" >> $GITHUB_ENV
        echo "GIT_VERSION_FORMAT={Major}.{Minor}.{Patch}-{EscapedBranchName}.{CommitsSinceVersionSource}" >> $GITHUB_ENV
      fi
  - uses: actions/checkout@v1
    with:
      fetch-depth: '0'            
  - name: Install GitVersion
    uses: gittools/actions/gitversion/setup@v1
    with:
        versionSpec: 6.0.5
  - id: determine_version
    name: Determine Version
    uses: gittools/actions/gitversion/execute@v1
    with:
        additionalArguments: /overrideconfig assembly-file-versioning-format=${{ env.GIT_VERSION_FORMAT }} /overrideconfig increment=${{ env.GIT_VERSION_INCREMENT }} /overrideconfig mode=${{ env.GIT_VERSION_MODE }} /overrideconfig update-build-number=true

Trivy

My tool of choice for vulnerability scanning is Trivy, an open-source security scanner that can scan third-party package references, create containers, and generate SBOMs.

Trivy source code scanning

Trivy provides an action to scan third-party package references in the source code and upload them to GitHub. For my .NET application, Trivy is using the package.lock.json files. That file can be automatically generated by .NET’s build process by adding <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> to the project file.

For example:

<PropertyGroup>
 <TargetFramework>net9.0</TargetFramework>
 <RootNamespace>Trident.Web</RootNamespace>
 <VersionPrefix>6.16</VersionPrefix>
 <StartupObject>Trident.Web.Program</StartupObject>
 <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>       
</PropertyGroup>

Once the package.lock.json exists, only three steps are needed to scan and upload the results to GitHub. Trivy will report unfixed changes, which will create a lot of noise. I configured the action to ignore unfixed vulnerabilities, but stop the build if a fixable vulnerability is found.

- name: Checkout code
  uses: actions/checkout@v4

- name: Run Trivy vulnerability scanner on repo
  uses: aquasecurity/trivy-action@0.32.0
  with:
    scan-type: 'fs'
    ignore-unfixed: true # Prevent unfixed results from being flagged
    exit-code: '1' # Stop the build if a fixable vulnerability is discovered
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'LOW,MEDIUM,HIGH,CRITICAL' # Change the severity levels to match company policy

- name: Upload Trivy scan results to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: 'trivy-results.sarif'

It is best to consult your company policies to determine the best approach:

  • Report all vulnerabilities but allow the build to proceed.
  • Only report vulnerabilities that have been fixed and stop the build if one is found.
  • Only report vulnerabilities that have been fixed but allow the build to proceed.

Trivy container scanning

You should also scan any created containers for vulnerabilities. Your code might be referencing another container version that has known CVEs. Unfortunately, you have to wait until the container is built before scanning. So you’ll need to be strategic when you build the container vs. publishing it to your container registry.

- name: build website container
  id: build_container
  working-directory: src
  run: | 
      docker build -f "./Trident.Web/Dockerfile"  --build-arg APP_VERSION=${{ needs.prep.outputs.sem_Ver }} --tag ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} --tag ${{ vars.DOCKER_HUB_REPO }}:latest .                                               

- name: Run Trivy vulnerability scanner on docker container
  uses: aquasecurity/trivy-action@0.32.0
  with:
    image-ref: '${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}'
    vuln-type: 'os,library'
    ignore-unfixed: true # Prevent unfixed results from being flagged
    exit-code: '1' # Stop the build if a fixable vulnerability is discovered
    format: 'sarif'
    output: 'trivy-image-results.sarif'          
    severity: 'LOW,MEDIUM,HIGH,CRITICAL' # Change the severity levels to match company policy

- name: Upload Trivy scan results to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: 'trivy-image-results.sarif'

- name: Login to Docker Hub
  uses: docker/login-action@v2
  with:             
      username: ${{ secrets.DOCKERHUB_USERNAME }}
      password: ${{ secrets.DOCKERHUB_PAT }}
- name: push docker image
  working-directory: src
  id: push_docker_image
  run: |
      docker push ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}
      docker push ${{ vars.DOCKER_HUB_REPO }}:latest

      dockerSha=$(docker manifest inspect ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} -v | jq -r '.Descriptor.digest')
      echo "Docker sha is $dockerSha"            
      echo "TRIDENT_DOCKER_SHA=$dockerSha" >> $GITHUB_OUTPUT

Trivy SBOM generation, packaging, and publishing

Trivy can generate SBOMs from the samepackage.lock.json files (along with other package reference files) used to scan for vulnerabilities. For my deployment pipeline, I’m putting the SBOM into a .zip file and uploading it to Octopus Deploy so I can add it as a deployment artifact. If anyone asks for the SBOM, I can go directly to the production deployment and download it for them. I’m also publishing building information for this SBOM, so I have a record of all the commits that are part of this deployment.

runs-on: ubuntu-latest 
permissions:
  # Add any additional permissions your job requires here
  id-token: write # This is required to obtain the OIDC Token for Octopus Deploy
steps:      
  - name: Checkout the code for SBOM
    uses: actions/checkout@v1
    with:
      fetch-depth: '0'  
        
  - name: Run Trivy in GitHub SBOM mode and submit results to Dependency Graph
    uses: aquasecurity/trivy-action@0.32.0
    with:
      scan-type: 'fs'
      format: 'github'
      output: 'dependency-results.sbom.json'
      scan-ref: '.'
      github-pat: ${{ secrets.GITHUB_TOKEN }}

  - name: Package SBOM
    id: "sbom_package"
    uses: OctopusDeploy/create-zip-package-action@v3
    with:
      package_id: Trident.SBOM
      version: "${{ needs.prep.outputs.sem_Ver }}" # the version comes from an earlier step 
      base_path: "./"          
      files: "dependency-results.sbom.json"
      output_folder: packaged                               
  
  - name: Create the Subject Checksum file for Attestation Build Provenance 
    id: determine_sbom_hash  
    shell: pwsh        
    run: |
      $packageHash = Get-FileHash -path "packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip" -Algorithm SHA256
      $hashToSave = $packageHash.Hash 
      Write-Host "The SBOM package hash is $hashToSave"
      
      "SBOM_HASH=$hashToSave" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
  
  - name: Login to Octopus Deploy 🐙
    uses: OctopusDeploy/login@v1
    with: 
      server: ${{ vars.OCTOPUS_SERVER_URL }}
      service_account_id: ${{ secrets.OCTOPUS_OIDC_SERVICE_ACCOUNT_ID }}
  - name: Push packages to Octopus 🐙
    uses: OctopusDeploy/push-package-action@v3
    with:
      server: ${{ vars.OCTOPUS_SERVER_URL }}
      space: ${{ vars.OCTOPUS_SPACE }}
      packages: |
        packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip  # the version comes from an earlier step
  - name: Push build information to Octopus 🐙
    uses: OctopusDeploy/push-build-information-action@v3
    with:
      packages: |          
        Trident.SBOM                               
      version: "${{ needs.prep.outputs.sem_Ver }}" # the version comes from an earlier step
      server: ${{ vars.OCTOPUS_SERVER_URL }}
      space: ${{ vars.OCTOPUS_SPACE }}

Create Attestations in GitHub Actions

This sample workflow builds three artifacts.

  1. SBOM Package Trident.SBOM
  2. Database Schema Package Trident.Database.DBUp
  3. Website Container bobjwalker99/trident

I wanted to have a single attestation for all three build artifacts. To do that, I needed to get the SHA256 for each artifact and combine it into a file to send to the action.

In the example above, you likely saw the following for the container:

dockerSha=$(docker manifest inspect ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} -v | jq -r '.Descriptor.digest')
echo "Docker sha is $dockerSha"            
echo "TRIDENT_DOCKER_SHA=$dockerSha" >> $GITHUB_OUTPUT

For the packages I used:

$packageHash = Get-FileHash -path "packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip" -Algorithm SHA256
$hashToSave = $packageHash.Hash 
Write-Host "The SBOM package hash is $hashToSave"
      
"SBOM_HASH=$hashToSave" | Out-File -FilePath $env:GITHUB_OUTPUT -Append

I next needed to dump all of those hashes to a .txt file for the attestation action to consume. The result is:

runs-on: ubuntu-latest
permissions:
  id-token: write
  attestations: write # Required to publish attestations      
steps:         
  - name: Create the Subject Checksum file for Provenance        
    shell: pwsh        
    run: |          
      $cleanedPackageSha = $("${{ needs.build_and_publish_database.outputs.database_hash }}" -replace "sha256:", "").Trim()
      $cleanedSbomSha = $("${{ needs.sbom.outputs.sbom_hash }}" -replace "sha256:", "").Trim()
      $cleanedImageSha = $("${{ needs.build_and_publish_website.outputs.website_hash }}" -replace "sha256:", "").Trim()

      $imageSubject = "${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}".Trim()
      $packageSubject = "Trident.Database.DbUp.${{ needs.prep.outputs.sem_Ver }}.zip".Trim()
      $sbomSubject = "Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip".Trim()

      Write-Host "The website information is $cleanedImageSha  $imageSubject"
      Write-Host "The database information is $cleanedPackageSha  $packageSubject"
      Write-Host "The SBOM information is $cleanedSbomSha  $sbomSubject"

      $subjectText = @"
      $cleanedImageSha  $imageSubject
      $cleanedPackageSha  $packageSubject
      $cleanedSbomSha  $sbomSubject
      "@

      Write-Host "Creating the checksums file"
      New-Item -Path . -Name "subject.checksums.txt" -ItemType "File" -Value $subjectText      
  - name: Generate Attestation from Provenance
    uses: actions/attest-build-provenance@v2
    id: websiteattest
    with:
        subject-checksums: subject.checksums.txt

The resulting attestation is:

attestation in GitHub generated from an action

Handing over to Octopus Deploy

In my Octopus Deploy instance, all feature and hotfix branches are deployed to the Development environment. While all main or primary branches are deployed to Test -> Staging -> Production. That allows me to get feedback on the short-lived branches while keeping main always in a deployable state.

The challenge is that the GitHub action needs to know the specific channel based on the branch that triggered the workflow. It is easy to do, but the syntax is a little goofy. It ends up being: channel: ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_FEATURE_BRANCH_CHANNEL }} - which is saying when on main, use the release channel. Otherwise, use the channel that deploys to Development.

permissions:
  # Add any additional permissions your job requires here
  id-token: write # This is required to obtain the OIDC Token for Octopus Deploy 
steps:
  - name: Login to Octopus Deploy 🐙
    uses: OctopusDeploy/login@v1
    with: 
      server: ${{ vars.OCTOPUS_SERVER_URL }}
      service_account_id: ${{ secrets.OCTOPUS_OIDC_SERVICE_ACCOUNT_ID }}         
  - name: Create and deploy release in Octopus 🐙
    uses: OctopusDeploy/create-release-action@v3
    with:
      server: ${{ vars.OCTOPUS_SERVER_URL }}
      space: ${{ vars.OCTOPUS_SPACE }}
      project: ${{ vars.OCTOPUS_PROJECT_NAME }}
      channel: ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_FEATURE_BRANCH_CHANNEL }}
      package_version: "${{ needs.prep.outputs.sem_Ver }}" 
      release_number: "${{ needs.prep.outputs.sem_Ver }}"          
      git_ref: ${{ (github.ref_type == 'tag' && github.event.repository.default_branch ) || (github.head_ref || github.ref) }}
      git_commit: ${{ github.event.after || github.event.pull_request.head.sha }} 

Octopus Deploy Configuration

Octopus Deploy is the tooling that changes Production, impacting your users and customers. Because of that, this section will spend a lot of time ensuring only authorized changes and the deployment configuration make it to Production.

Permissions

We want to divide permissions between Platform/DevOps engineers and Developers. The Platform/DevOps engineer is the producer; they are experts in the tooling, deployment targets, and company policies. The developer is the consumer; they are experts in their application. The producers create all the pieces necessary for the consumer to use. Consumers can modify settings specific to their application. Producers make sure all pipelines are compliant.

Below is a list of permissions to illustrate the difference between Producers and Consumers.

PermissionDevOps / Platform Engineer (Producer)Developer (Consumer)
Create ProjectsYesNo
Create and modify Variable SetsYesNo
Create and modify Environments and LifecyclesYesNo
Create and modify cloud accounts, feeds, and GitHub accountsYesNo
Projects - Configure Version Control and branch protection policiesYesNo
Projects - Create and modify channelsYesNo
Projects - Modify ITSM settingsYesNo
Projects - Modify guided failure settingsYesNo
Projects - Modify runbooksYes - must be done in a branch and submitted via a PRYes - must be done in a branch and submitted via a PR
Projects - Modify deployment processYes - must be done in a branch and submitted via a PRYes - must be done in a branch and submitted via a PR
Projects - Modify variablesYes - must be done in a branch and submitted via a PRYes - must be done in a branch and submitted via a PR

To accomplish that separation, you will need to:

  1. Configure project version control for each project.
  2. Configure branch protection policies for the main or primary branch in project version control.
  3. Configure RBAC for your users (see below).

RBAC In Octopus

For producers (Platform/DevOps Engineers), you’ll need to create a custom role called Platform Engineer. For Consumers (developers), you’ll need to create a custom role called Developer.

The specific permissions for each role will be:

PermissionPlatform EngineersDeveloper
AccountCreateYesNo
AccountDeleteYesNo
AccountEditYesNo
AccountViewYesYes
ActionTemplateCreateYesNo
ActionTemplateDeleteYesNo
ActionTemplateEditYesNo
ActionTemplateViewYesYes
ArtifactCreateYesNo
ArtifactDeleteYesNo
ArtifactEditYesNo
ArtifactViewYesYes
BuiltInFeedPushYesYes
CertificateViewYesYes
DefectReportYesYes
DefectResolveYesYes
DeploymentViewYesYes
EnvironmentCreateYesNo
EnvironmentDeleteYesNo
EnvironmentEditYesNo
EnvironmentViewYesYes
EventViewYesYes
FeedEditYesNo
FeedViewYesYes
GitCredentialEditYesNo
GitCredentialViewYesYes
InsightsReportCreateYesNo
InsightsReportDeleteYesNo
InsightsReportEditYesNo
InsightsReportViewYesYes
InterruptionViewYesYes
InterruptionViewSubmitResponsibleYesYes
LibraryVariableSetCreateYesNo
LibraryVariableSetDeleteYesNo
LibraryVariableSetEditYesNo
LibraryVariableSetViewYesYes
LifecycleCreateYesNo
LifecycleDeleteYesNo
LifecycleEditYesNo
LifecycleViewYesYes
MachineCreateYesNo
MachineDeleteYesNo
MachineEditYesNo
MachineViewYesYes
MachinePolicyCreateYesNo
MachinePolicyDeleteYesNo
MachinePolicyEditYesNo
MachinePolicyViewYesYes
ProcessEditYesYes
ProcessViewYesYes
ProjectCreateYesNo
ProjectDeleteYesNo
ProjectEditYesNo
ProjectViewYesYes
ProjectGroupCreateYesNo
ProjectGroupDeleteYesNo
ProjectGroupEditYesNo
ProjectGroupViewYesYes
ProxyCreateYesNo
ProxyDeleteYesNo
ProxyEditYesNo
ProxyViewYesYes
ReleaseCreateYesYes
ReleaseDeleteYesNo
ReleaseViewYesYes
RetentionAdministerYesNo
RunbookEditYesYes
RunbookRunViewYesYes
RunbookViewYesYes
SubscriptionCreateYesNo
SubscriptionDeleteYesNo
SubscriptionEditYesNo
SubscriptionViewYesYes
TagSetCreateYesNo
TagSetDeleteYesNo
TagSetEditYesNo
TargetTagAdministerYesNo
TargetTagViewYesYes
TaskCancelYesNo
TaskCreateYesNo
TaskViewYesYes
TenantViewYesYes
TriggerCreateYesYes
TriggerDeleteYesYes
TriggerEditYesYes
TriggerViewYesYes
TeamCreateYesNo
TeamDeleteYesNo
TeamEditYesNo
TeamViewYesYes
VariableEditYesYes
VariableEditUnscopedYesYes
VariableViewYesYes
VariableViewUnscopedYesYes
WorkerEditYesNo
WorkerViewYesYes
System Perm - DeploymentFreezeAdministerYesNo
System Permission - SpaceCreateYesNo
System Permission - SpaceDeleteYesNo
System Permission - SpaceEditYesNo
System Permission - SpaceViewYesNo
System Perm - PlatformHubEditYesNo
System Perm - PlatformHubViewYesNo
System Permission - TaskCancelYesNo
System Permission - TaskCreateYesNo
System Permission - TeamCreateYesNo
System Permission - TeamDeleteYesNo
System Permission - TeamEditYesNo
System Permission - TeamViewYesYes
System Permission - UserInviteYesYes
System Permission - UserRoleViewYesYes
System Permission - UserViewYesYes

Once those roles are created, create the appropriate teams and assign them those roles. Do not scope them to any environment or tenant.

The new roles detailed above provide the appropriate editing capabilities within Octopus Deploy. They purposely exclude creating deployments and runbook runs. You’ll likely want to scope them to the appropriate environments or tenants.

Common scenarios we see:

  • Developers can deploy any project to Development and Test, but not Production.
  • Developers can deploy specific projects to Development, Test, and Production. But no other projects.
  • Release managers or web admins can deploy to Production.
RolePlatform EngineersDeveloper
Deployment CreatorYes - no scopingYes - scoped to specific environments, tenants, or projects
Runbook ConsumerYes - no scopingYes - scoped to specific environments, tenants, or projects
Tenant Manager (if leveraging multi-tenancy)Yes - no scopingYes - scoped to specific environments, tenants, or projects

The Developer team on my instance is configured to allow them to deploy to Development, Test, and Staging while allowing them to run runbooks on any environment.

Developer team user role assignment

Cloud Accounts and Third-Party Key Vaults

In my example deployment pipeline, I’m storing secrets inside of Azure Key Vault instead of using Octopus Deploy’s sensitive variables. My primary reason is that Azure Key Vault is focused on storing secrets and doesn’t try to be anything else. It offers features and functionality, such as secret rotation and versioning, for storing secrets that Octopus Deploy doesn’t provide.

This blog post walks you through configuring Azure Key Vault with Octopus Deploy. The primary difference between my configuration and that configuration is that my Azure Account in Octopus Deploy uses OIDC.

Azure account in Octopus Deploy using OIDC

With Third-Party Key Vaults and OIDC, I aim to eliminate Octopus Deploy from the secret-storing business altogether.

Lifecycles

Lifecycles are among Octopus Deploy’s most misused features.

The most common misconfiguration I see is:

  • Default lifecycle: Development -> Test -> Staging -> Production
  • Hotfix lifecycle: Staging -> Production

The common reason behind that configuration is “we need a clear path to production in the event of an emergency.”  While a valid point, the primary problem with that configuration is that Development is included in any path to production. Development should be used for fast feedback on a work in progress. At no point should code from Development ever be promoted to Production. It must go through a pull request/approval workflow, which is at the core of supply chain security.

The recommended configuration is:

  • Default Lifecycle: Development
  • Release Lifecycle: Test -> Staging -> Production

lifecycles in Octopus Deploy

The default lifecycle deployment destination for all branches except the main or primary branch is Development. The release lifecycle is only for main or the primary branch. With this configuration, only approved code (remember we configured the branch ruleset in GitHub) is deployed to Production.

ITSM Integration

Octopus Deploy supports ServiceNow and Jira Service Management for ITSM approvals. If you are using those services and have an Enterprise tier license, you should configure that integration as soon as possible.

Our ITSM integration will create a change request and wait until it reaches the appropriate state. Octopus will not even start the deployment until that state is reached.

ITSM enabled in Octopus

Not all environments require ITSM approval, so we require you to enable ITSM per environment. Because you can turn off/on ITSM integration, I recommend restricting EnvironmentEdit permissions to Platform Engineers.

Production environment with ITSM enabled

GitHub Octopus Deploy Integration

If you are using Octopus Deploy cloud, configure the Octopus Deploy GitHub Application. This allows you to connect to GitHub repositories without providing a PAT. You can still control the connection to specific repositories and organizations.

Octopus Deploy GitHub app integration

Build Information and Issue Tracking Integration

Octopus Deploy natively integrates with JIRA, GitHub Issue Tracker, and Azure DevOps Issue Tracker. If you are using JIRA Cloud, we also provide the capability to update the deployment status of JIRA Tickets.

JIRA Cloud deployment integration

That requires the configuration of JIRA Integration with Octopus. Our documentation provides a step-by-step guide for configuration. The result looks like this:

JIRA Integration configuration

If you recall, earlier, the GitHub action was publishing the build information for the SBOM. Octopus parses the build commit message to look for the issue. My commit messages were prefixed with TRID-43, which Octopus then linked to that ticket in JIRA.

parsing the commit message from build information

Project Settings

We want to configure the Octopus Deploy project to follow these rules:

  1. Any changes to the deployment process, variables, and runbooks require approval.
  2. Only the main or primary branch can be deployed to Production.
  3. Only pre-release versions can be deployed to Development; Production only accepts packages without pre-release versions.
  4. If applicable, ITSM integration is enabled.

Important: ProjectEdit permission was restricted to Platform Engineers because it can modify a project’s Project Version Control and ITSM Provider configuration. Specifically, anyone with those permissions can turn off branch protection policies and ITSM requirements. What used to be a permission to change the name and icon of the project has grown into a powerful setting. Be mindful of who grants access to it.

Configure Project Version Control

We want to use the same pull-request approval process for the deployment process, variables, and runbook updates as used for source code. To do that, configure project version control for your project. While doing that, configure the branch protection policy for the main or primary branch. In the Octopus Deploy UI, you’ll see notifications that the main or primary branch is protected and requires any changes to be made to the branch.

project version control settings

I also recommend using the same git repository as your source code for the following reasons:

  • The code repository should store the source code, how it is built, its dependencies, how it is hosted, and how it is deployed. That allows you to reproduce production in local and testing environments quickly.
  • Often, underlying code changes, such as migrating to Kubernetes, require an update to the build and deployment process. All the necessary changes to migrate to Kubernetes can be made in the same branch and merged using the same pull request process.

Important: Anyone with the ProjectEdit permission can change the branch protection policy setting. Using the recommended roles from earlier will limit that permission to Platform Engineers.

Configure Channels

Now that version control is enabled, we can configure two channels for our two lifecycles. We will include version and branch rules in those channels.

  • Default Channel
    • Lifecycle: Default Lifecycle
    • Package Version Rules: select the steps that deploy packages and enter this in the pre-release tag field ^[^\+].*. That will ensure only packages with a pre-release tag can be deployed to Development
    • Branch Protection Rules: Enter a pattern to ensure anything but the main or primary branch can be used. For my instance I use [feature|hotfix]*/*
  • Release Channel
    • Lifecycle: Release Lifecycle
    • Package Version Rules: select the steps that deploy packages and enter this into the pre-release tag field ^(|\+.*)$. That will block any packages with a pre-release tag from being selected.
    • Branch Protection rules: enter main into the branches field. That will ensure only the main branch can be used.   With these rules in place, we can ensure only artifacts and processes from the main branch will be used for Production deployments.

channel configuration with version and branch rules

ITSM

In the ITSM Providers screen, check the Change Controlled check box and select the appropriate ITSM provider connection. You can also select specific runbooks to be changed controlled in this screen.

project itsm provider screen

Important: Anyone with the ProjectEdit permission can change the ITSM setting. Using the recommended roles from earlier will limit that permission to Platform Engineers.

Deployment Process

The final piece of our supply chain security workflow is to add the appropriate steps to pull secrets from Azure Key Vault, publish the SBOM as a deployment artifact, and verify the attestations from GitHub.

Pulling secrets from Azure Key Vault

The deployment process uses community library step template to interact with Azure Key Vault.

I recommend using the execution container octopuslabs/azure-workertools, which has all the required CLIs to pull secrets from the key vault.

The step will pull one-to-many secrets. It uses the format [Secret Name] | [Output Variable Name] to determine the secrets to pull. For example:

azure-sql-server | SQLServerName
azure-sql-password | SQLUserPassword
azure-sql-username | SQLUserName
octopus-api-key | OctopusApiKey
github-token | GitHubToken
sumo-logic-url | SumoLogicUrl

The output variable name is how you can reference the secrets in subsequent steps. For example, the SQL Server Name can be accessed in subsequent steps by looking for the Octopus.Action[Azure Key Vault - Retrieve Secrets].Output.SQLServerName variable.

Azure key vault step in a deployment process

Publishing the SBOM as a deployment artifact

We can use pre-existing functionality to publish the SBOM as a deployment artifact. Add a script step to the process, then add a package reference. Ensure the package reference extracts the package during the deployment.

package reference that extracts a package

The script only needs to find the .json file and publish it as a deployment artifact. The following is the PowerShell script I used in the deployment pipeline.

$extractedPath = $OctopusParameters["Octopus.Action.Package[Template.SBOM.Artifact].ExtractedPath"]
$OctopusEnvironmentName = $OctopusParameters["Octopus.Environment.Name"]

$sbomFiles = Get-ChildItem -Path $extractedPath -Filter "*.json" -Recurse

foreach ($sbom in $sbomFiles)
{
  Write-Host "Attaching $($sbom.FullName) as an artifact"
  New-OctopusArtifact -Path $sbom.FullName -Name "$OctopusEnvironmentName.SBOM.JSON"

  break
} 

Verifying attestations

We will use the gh attestation verify CLI command as the means to verify the attestations. We’ve provided octopuslabs/github-workertools as an execution container so you don’t have to worry about downloading and installing the CLI.

That command converts the deployment artifact to a SHA256 hash and then looks for attestations matching that hash for your repository or organization. If the artifact has been tampered with, the attestation verification will fail because it won’t find a matching hash. For example, here is the API endpoint my process attempted to hit on a failed attestation verification.

https://api.github.com/repos/BobJWalker/Trident/attestations/sha256:d96a5d80bc2d7ce427dc1f543d7caedf4ea014a397cae0140909dfa306a48b1b?per_page=30&predicate_type=https://slsa.dev/provenance/v1

After the CLI finds the attestation for the artifact, it will perform an additional series of verifications. More information about how it verifies the attestation can be found in the GitHub’s docs.

For package attestation verification, you can use our community step template.

For Docker containers, you’ll need to authenticate to the container registry. Once you do that, you can run a script similar to this:

$packageVersion = $OctopusParameters["Octopus.Action.Package[YOUR CONTAINER].PackageVersion"]
$packageName = $OctopusParameters["Octopus.Action.Package[YOUR CONTAINER].PackageId"]

$GHToken = "YOUR GITHUB TOKEN"
$image = "oci://$($packageName)$:$($packageVersion)$"
$repo = "bobjwalker/trident"

$env:GITHUB_TOKEN=$GHToken
$attestation=gh attestation verify "$image" --repo $repo --format json
if ($LASTEXITCODE -ne 0)
{
    Write-Error "The attestation for $image could not be verified"
}

Write-Highlight "$image successfully passed attestation verification"
Write-Verbose $attestation
$artifactVerified = $true

Write-Host "Writing the attest output to $packageName.$OctopusEnvironmentName.attestation.json"
New-Item -Name "$packageName.$OctopusEnvironmentName.attestation.json" -ItemType "File" -Value $attestation
New-OctopusArtifact -Path "$packageName.$OctopusEnvironmentName.attestation.json" -Name  "$packageName.$OctopusEnvironmentName.attestation.json"

Conclusion

There will always be a healthy tension between security and usability. You can enact security policies that replicate the computer vault from the first Mission: Impossible film, but it isn’t very usable at scale. On the other side of the spectrum, everyone can be made an admin, and there is no red tape, but that isn’t very secure. Finding the “right” amount of security is always a challenge. My goal with this article was to provide low-effort recommendations that are as non-intrusive as possible but substantially increase supply chain security. My guiding “north star” was SLSA level 3 with this pipeline.

Despite the changes, when the rules are followed (pull request required, build on every check-in, always verify attestations, etc.), there should be no noticeable impact on the developer experience. It’s only when someone attempts to bypass the rules that they encounter friction.

Bob Walker

Related posts