In May 2021, in response to a series of high-profile cyber attacks, President Biden issued Executive Order 14028 to improve the nationâs cybersecurity. A big reason for this was the SolarWinds supply chain attack in which Russian state-sponsored hackers compromised the Orion software platform. Section 4 specifically addressed that attack:
The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors. There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended.
Since 2021, weâve seen a lot of new functionality to enable supply chain security. In this blog post, I will walk you through improving your supply chain security by leveraging GitHub, GitHub Actions, and Octopus Deploy.
Disclaimer
Iâm under no illusions that this article will be the be-all and end-all solution for supply chain security. These solutions will not cover other avenues of attack. The solutions use new functionality added to both platforms since 2021, pre-existing functionality, and common-sense configurations.
A great third-party resource is Supply-chain Levels for Software Artifacts or SLSA(pronounced âsalsaâ). For additional information, consult your CISO (Chief Information Security Officer), security team, or other company experts. As with anything security-related, check your companyâs policies to ensure compliance.
Nomenclature
This article will use relatively new terms in software deployment pipelines. Iâve included the definitions below to make it easier to follow along.
- SBOM - Software Bill of Materials - a list of all the third-party libraries (and their third-party libraries) used to create the build artifact (container, .zip files, jar files, etc.).
- Provenance - the record of who created the software change, how it was modified and built, and what inputs went into it. It shows how the build artifact was built.
- Attestation - A cryptographically verifiable statement that asserts something about an artifact, specifically its Provenance. It is similar to the notary seal on a document. Doesnât show the whole process, but certifies its validity.
SBOMs, Provenance, and Attestations are intertwined. Think of it like a cake.
- SBOMs are the ingredient list.
- Provenance is the recipe and kitchen log (who cooked it, when, and with which tools).
- Attestation is a signed certificate that proves the ingredient list, recipe, and cooking process are trustworthy.
Supply chain security is more than SBOMs, Provenance, and Attestation
SBOMs, Provenance, and Attestations are new concepts to the typical software deployment pipeline. It is tempting to focus solely on them, but that is like only worrying about the tires on a car. There is much more to it. RBAC controls, branch protection policies, audit log streaming to SIEM, key vaults, approvals from ITSM, and authentication/authorization for cloud accounts, to name a few.
Responsibilities differences between GitHub and Octopus Deploy
Clear boundaries between tooling in the deployment pipeline are essential. It sets expectations within your organization, avoids overlapping effort, and enables the tooling to be used as designed.
Iâve seen many instances where tooling is misused because it supports a simple use case. For example, approvals should be handled via ITSM tooling. But Octopus Deploy has the manual intervention step. It was initially designed to let a deployment perform an act (generate a delta report), pause, and let someone review before proceeding. But for years, users tried to use it for deployment approvals. They wanted rules such as âthe person who made the change canât approve it.â
The responsibilities of GitHub and Octopus Deploy in a secure pipeline are as follows:
GitHub                                | Octopus Deploy                               |
---|---|
Branch Protection Policies                      | Environmental progression and release orchestration             |
Pull Request workflow                         | Centralized dashboard for deployment status and latest version       |
Linting, static code analysis, and vulnerability scanning       | Creating and tracking approvals in ITSM tooling               |
Automated Testing (unit tests, integration tests, etc.)        | RBAC and separation of duties for production deployments          |
Creating and publishing build artifacts (packages, containers, etc.) | Ingesting SBOMs and verifying Attestations                 |
Generating SBOMs and Attestations                   | Environmental modeling of infrastructure                  |
Calculating version numbers                      | Authentication and authorization to deployment targets and cloud providers |
Creating releases in Octopus Deploy                  | Creating and destroying ephemeral environments               |
Example Pipeline
All the screenshots and scripts come from a real-world example I put together for demos and webinars. All the source code, builds, and deployment process are stored in a public repo on GitHub as well as my publicly available Octopus Cloud instance. You can log in as a guest.
As you can see, my GitHub workflow builds, tests, scans the code, generates the attestations, and then hands over to Octopus Deploy.
The deployment process will pull secrets from an Azure key vault, attach the SBOM as a deployment artifact, verify the attestations of all the build artifacts, build a database delta report, check the report, and finally deploy the database and website changes.
Supply chain security doesnât mean slowing down the pipeline. Youâll notice many steps in the build workflow can run in parallel to speed up the overall process. A fast pipeline is just as important as a secure pipeline. Developers will hesitate to check in if it takes two hours to build, test, and scan the code. From the logs, you can see it took 7 minutes to build and deploy the application; 3 minutes for GitHub action and 4 minutes for Octopus Deploy.
Deployment Pipeline Rules
I created deployment pipeline with the following rules:
- At least one person must review every change that can impact
Production
, be it source code, database schema, or production deployment. - All changes must be made in a branch and merged via a pull request. The
main
branch must always be deployable. - Builds must run anytime source code changes.
- All unit tests and third-party vulnerability scans must run for every build.
- All builds must generate attestations for their build artifacts.
- All deployments must verify the attestations before making changes.
- All interactions between systems (GitHub -> Octopus, Octopus -> Azure, etc.) must use OIDC whenever possible.
- When OIDC is impossible, store any secrets/passwords in a key vault that enables secret rotation.
GitHub Configuration
The focus for GitHub will be on securing code from tampering and minimizing third-party vulnerabilities. As such, the scope of the GitHub configuration extends beyond the GitHub build action.
Branch rulesets
To ensure a review occurs for any change that impacts the product, the main
or primary branch should have an appropriate ruleset that will:
- All commits are required to be made on a separate branch.
- A pull request with at least one approval is required.
- A code scanning result is required for critical alerts.
Consult with your security team or CISO for other appropriate settings for your company.
Built-in code scanning
Use the scanning GitHub provides to find problems proactively. These include:
- Enabling Dependabot to alert you of vulnerabilities that impact your dependencies.
- Enabling Secret Scanning to alert you if a secret is accidentally checked into version control.
These settings will proactively find problems in your codebase instead of waiting for a build to run. Â
General GitHub Actions settings
Before getting into SBOMs and attestations, use the following with your GitHub Actions.
- Use Secrets and Variables instead of storing them directly in the GitHub action.
- Use OIDC to login to Octopus Deploy. Do not use log-lived API keys. When configuring the service account user in Octopus Deploy, you must provide a subject for the OIDC authentication to work. On my instance, Iâm a little more relaxed about the subjects.
- Pull Request Subject:
repo:BobJWalker/*:pull_request
- accept any connection from any pull request action in the BobJWalker repo. - Builds:
repo:BobJWalker/*:ref:refs/heads/*
- accept any connection from any build in the BobJWalker repo.
- Pull Request Subject:
- Trigger builds for
main
or primary branch,hotfix
andfeature
branches, and manually. Â Set up filters for specific folders or files to avoid a no-op build. For example, my build workflow triggers are:
on:
push:
branches:
- main
- 'feature/**'
- 'features/**'
- 'hotfix/**'
paths:
- 'src/**'
- 'db/**'
- 'k8s/**'
- '.github/workflows/build.yml' ## This is in here so I get a new build with each build change I make
workflow_dispatch:
Calculating Versions
Believe it or not, calculating the version number was the most challenging part of this entire process. Iâm using SemVer and want to remain close to the rules.
- Builds from the
main
branch would be use:{Major}.{Minor}.{Patch}
, e.g.6.16.26
. Themain
branch is the only thing that can go toProduction
. - Builds from any other branch would use:
{Major}.{Minor}.{Patch}-{EscapedBranchName}.{CommitsSinceVersionSource}
, e.g.6.16.27-feature-singleton.1
. The escaped branch name would includefeature
orhotfix
. The code on these branches is pre-release and should only go toDevelopment
for testing. - The first check-in to a non-main branch would auto increment the
{Patch}
using the version from themain
branch. After that, only the{CommitsSinceVersionSource}
would be incremented for the non-main branch check-ins. By default, all changes must be backward compatible. - Increasing the
{Major}.{Minor}
would be a manual process accomplished using commit messages, e.g.,+semver: major
for significant increments and+semver: minor
for minor increments. The developer is making a deliberate decision to make a non-backward-compatible change, which they document via a git commit.
Of the tools I tried, I kept returning to gitversion. It met 90% of my needs with no additional configuration. But, it doesnât work well with dynamic version formats based on branch names. The result is a step with a small amount of business logic and output variables.
runs-on: ubuntu-latest
name: Determine version
outputs:
sem_ver: ${{ steps.determine_version.outputs.AssemblySemFileVer }}
steps:
- name: Set environment variable based on branch
id: set_env_var
run: |
echo "GITHUB_REF: $GITHUB_REF"
echo "GITHUB_HEAD_REF: $GITHUB_HEAD_REF"
BRANCH_NAME="${GITHUB_REF#refs/heads/}"
echo "Branch detected: $BRANCH_NAME"
if [ "$BRANCH_NAME" = "main" ]; then
echo "GIT_VERSION_INCREMENT=Patch" >> $GITHUB_ENV
echo "GIT_VERSION_MODE=ContinuousDeployment" >> $GITHUB_ENV
echo "GIT_VERSION_FORMAT={Major}.{Minor}.{Patch}" >> $GITHUB_ENV
else
echo "GIT_VERSION_INCREMENT=Patch" >> $GITHUB_ENV
echo "GIT_VERSION_MODE=ContinuousDelivery" >> $GITHUB_ENV
echo "GIT_VERSION_FORMAT={Major}.{Minor}.{Patch}-{EscapedBranchName}.{CommitsSinceVersionSource}" >> $GITHUB_ENV
fi
- uses: actions/checkout@v1
with:
fetch-depth: '0'
- name: Install GitVersion
uses: gittools/actions/gitversion/setup@v1
with:
versionSpec: 6.0.5
- id: determine_version
name: Determine Version
uses: gittools/actions/gitversion/execute@v1
with:
additionalArguments: /overrideconfig assembly-file-versioning-format=${{ env.GIT_VERSION_FORMAT }} /overrideconfig increment=${{ env.GIT_VERSION_INCREMENT }} /overrideconfig mode=${{ env.GIT_VERSION_MODE }} /overrideconfig update-build-number=true
Trivy
My tool of choice for vulnerability scanning is Trivy, an open-source security scanner that can scan third-party package references, create containers, and generate SBOMs.
Trivy source code scanning
Trivy provides an action to scan third-party package references in the source code and upload them to GitHub. For my .NET application, Trivy is using the package.lock.json
files. That file can be automatically generated by .NETâs build process by adding <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
to the project file.
For example:
<PropertyGroup>
<TargetFramework>net9.0</TargetFramework>
<RootNamespace>Trident.Web</RootNamespace>
<VersionPrefix>6.16</VersionPrefix>
<StartupObject>Trident.Web.Program</StartupObject>
<RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> Â Â Â
</PropertyGroup>
Once the package.lock.json
exists, only three steps are needed to scan and upload the results to GitHub. Trivy will report unfixed changes, which will create a lot of noise. I configured the action to ignore unfixed vulnerabilities, but stop the build if a fixable vulnerability is found.
- name: Checkout code
uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner on repo
uses: aquasecurity/trivy-action@0.32.0
with:
scan-type: 'fs'
ignore-unfixed: true # Prevent unfixed results from being flagged
exit-code: '1' # Stop the build if a fixable vulnerability is discovered
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'LOW,MEDIUM,HIGH,CRITICAL' # Change the severity levels to match company policy
- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'trivy-results.sarif'
It is best to consult your company policies to determine the best approach:
- Report all vulnerabilities but allow the build to proceed.
- Only report vulnerabilities that have been fixed and stop the build if one is found.
- Only report vulnerabilities that have been fixed but allow the build to proceed.
Trivy container scanning
You should also scan any created containers for vulnerabilities. Your code might be referencing another container version that has known CVEs. Unfortunately, you have to wait until the container is built before scanning. So youâll need to be strategic when you build the container vs. publishing it to your container registry.
- name: build website container
id: build_container
working-directory: src
run: |
docker build -f "./Trident.Web/Dockerfile" --build-arg APP_VERSION=${{ needs.prep.outputs.sem_Ver }} --tag ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} --tag ${{ vars.DOCKER_HUB_REPO }}:latest .
- name: Run Trivy vulnerability scanner on docker container
uses: aquasecurity/trivy-action@0.32.0
with:
image-ref: '${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}'
vuln-type: 'os,library'
ignore-unfixed: true # Prevent unfixed results from being flagged
exit-code: '1' # Stop the build if a fixable vulnerability is discovered
format: 'sarif'
output: 'trivy-image-results.sarif'
severity: 'LOW,MEDIUM,HIGH,CRITICAL' # Change the severity levels to match company policy
- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'trivy-image-results.sarif'
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PAT }}
- name: push docker image
working-directory: src
id: push_docker_image
run: |
docker push ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}
docker push ${{ vars.DOCKER_HUB_REPO }}:latest
dockerSha=$(docker manifest inspect ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} -v | jq -r '.Descriptor.digest')
echo "Docker sha is $dockerSha"
echo "TRIDENT_DOCKER_SHA=$dockerSha" >> $GITHUB_OUTPUT
Trivy SBOM generation, packaging, and publishing
Trivy can generate SBOMs from the samepackage.lock.json
files (along with other package reference files) used to scan for vulnerabilities. For my deployment pipeline, Iâm putting the SBOM into a .zip file and uploading it to Octopus Deploy so I can add it as a deployment artifact. If anyone asks for the SBOM, I can go directly to the production deployment and download it for them. Iâm also publishing building information for this SBOM, so I have a record of all the commits that are part of this deployment.
runs-on: ubuntu-latest
permissions:
# Add any additional permissions your job requires here
id-token: write # This is required to obtain the OIDC Token for Octopus Deploy
steps:
- name: Checkout the code for SBOM
uses: actions/checkout@v1
with:
fetch-depth: '0'
- name: Run Trivy in GitHub SBOM mode and submit results to Dependency Graph
uses: aquasecurity/trivy-action@0.32.0
with:
scan-type: 'fs'
format: 'github'
output: 'dependency-results.sbom.json'
scan-ref: '.'
github-pat: ${{ secrets.GITHUB_TOKEN }}
- name: Package SBOM
id: "sbom_package"
uses: OctopusDeploy/create-zip-package-action@v3
with:
package_id: Trident.SBOM
version: "${{ needs.prep.outputs.sem_Ver }}" # the version comes from an earlier step
base_path: "./"
files: "dependency-results.sbom.json"
output_folder: packaged
- name: Create the Subject Checksum file for Attestation Build Provenance
id: determine_sbom_hash
shell: pwsh
run: |
$packageHash = Get-FileHash -path "packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip" -Algorithm SHA256
$hashToSave = $packageHash.Hash
Write-Host "The SBOM package hash is $hashToSave"
"SBOM_HASH=$hashToSave" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
- name: Login to Octopus Deploy đ
uses: OctopusDeploy/login@v1
with:
server: ${{ vars.OCTOPUS_SERVER_URL }}
service_account_id: ${{ secrets.OCTOPUS_OIDC_SERVICE_ACCOUNT_ID }}
- name: Push packages to Octopus đ
uses: OctopusDeploy/push-package-action@v3
with:
server: ${{ vars.OCTOPUS_SERVER_URL }}
space: ${{ vars.OCTOPUS_SPACE }}
packages: |
packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip # the version comes from an earlier step
- name: Push build information to Octopus đ
uses: OctopusDeploy/push-build-information-action@v3
with:
packages: |
Trident.SBOM
version: "${{ needs.prep.outputs.sem_Ver }}" # the version comes from an earlier step
server: ${{ vars.OCTOPUS_SERVER_URL }}
space: ${{ vars.OCTOPUS_SPACE }}
Create Attestations in GitHub Actions
This sample workflow builds three artifacts.
- SBOM Package
Trident.SBOM
- Database Schema Package
Trident.Database.DBUp
- Website Container
bobjwalker99/trident
I wanted to have a single attestation for all three build artifacts. To do that, I needed to get the SHA256
for each artifact and combine it into a file to send to the action.
In the example above, you likely saw the following for the container:
dockerSha=$(docker manifest inspect ${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }} -v | jq -r '.Descriptor.digest')
echo "Docker sha is $dockerSha"
echo "TRIDENT_DOCKER_SHA=$dockerSha" >> $GITHUB_OUTPUT
For the packages I used:
$packageHash = Get-FileHash -path "packaged/Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip" -Algorithm SHA256
$hashToSave = $packageHash.Hash
Write-Host "The SBOM package hash is $hashToSave"
  Â
"SBOM_HASH=$hashToSave" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
I next needed to dump all of those hashes to a .txt
file for the attestation action to consume. The result is:
runs-on: ubuntu-latest
permissions:
id-token: write
attestations: write # Required to publish attestations
steps:
- name: Create the Subject Checksum file for Provenance
shell: pwsh
run: |
$cleanedPackageSha = $("${{ needs.build_and_publish_database.outputs.database_hash }}" -replace "sha256:", "").Trim()
$cleanedSbomSha = $("${{ needs.sbom.outputs.sbom_hash }}" -replace "sha256:", "").Trim()
$cleanedImageSha = $("${{ needs.build_and_publish_website.outputs.website_hash }}" -replace "sha256:", "").Trim()
$imageSubject = "${{ vars.DOCKER_HUB_REPO }}:${{ needs.prep.outputs.sem_Ver }}".Trim()
$packageSubject = "Trident.Database.DbUp.${{ needs.prep.outputs.sem_Ver }}.zip".Trim()
$sbomSubject = "Trident.SBOM.${{ needs.prep.outputs.sem_Ver }}.zip".Trim()
Write-Host "The website information is $cleanedImageSha $imageSubject"
Write-Host "The database information is $cleanedPackageSha $packageSubject"
Write-Host "The SBOM information is $cleanedSbomSha $sbomSubject"
$subjectText = @"
$cleanedImageSha $imageSubject
$cleanedPackageSha $packageSubject
$cleanedSbomSha $sbomSubject
"@
Write-Host "Creating the checksums file"
New-Item -Path . -Name "subject.checksums.txt" -ItemType "File" -Value $subjectText
- name: Generate Attestation from Provenance
uses: actions/attest-build-provenance@v2
id: websiteattest
with:
subject-checksums: subject.checksums.txt
The resulting attestation is:
Handing over to Octopus Deploy
In my Octopus Deploy instance, all feature and hotfix branches are deployed to the Development
environment. While all main
or primary branches are deployed to Test
-> Staging
-> Production
. That allows me to get feedback on the short-lived branches while keeping main
always in a deployable state.
The challenge is that the GitHub action needs to know the specific channel based on the branch that triggered the workflow. It is easy to do, but the syntax is a little goofy. It ends up being: channel: ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_FEATURE_BRANCH_CHANNEL }}
- which is saying when on main
, use the release channel. Otherwise, use the channel that deploys to Development
.
permissions:
# Add any additional permissions your job requires here
id-token: write # This is required to obtain the OIDC Token for Octopus Deploy
steps:
- name: Login to Octopus Deploy đ
uses: OctopusDeploy/login@v1
with:
server: ${{ vars.OCTOPUS_SERVER_URL }}
service_account_id: ${{ secrets.OCTOPUS_OIDC_SERVICE_ACCOUNT_ID }}
- name: Create and deploy release in Octopus đ
uses: OctopusDeploy/create-release-action@v3
with:
server: ${{ vars.OCTOPUS_SERVER_URL }}
space: ${{ vars.OCTOPUS_SPACE }}
project: ${{ vars.OCTOPUS_PROJECT_NAME }}
channel: ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_FEATURE_BRANCH_CHANNEL }}
package_version: "${{ needs.prep.outputs.sem_Ver }}"
release_number: "${{ needs.prep.outputs.sem_Ver }}"
git_ref: ${{ (github.ref_type == 'tag' && github.event.repository.default_branch ) || (github.head_ref || github.ref) }}
git_commit: ${{ github.event.after || github.event.pull_request.head.sha }}
Octopus Deploy Configuration
Octopus Deploy is the tooling that changes Production
, impacting your users and customers. Because of that, this section will spend a lot of time ensuring only authorized changes and the deployment configuration make it to Production
.
Permissions
We want to divide permissions between Platform/DevOps engineers and Developers. The Platform/DevOps engineer is the producer; they are experts in the tooling, deployment targets, and company policies. The developer is the consumer; they are experts in their application. The producers create all the pieces necessary for the consumer to use. Consumers can modify settings specific to their application. Producers make sure all pipelines are compliant.
Below is a list of permissions to illustrate the difference between Producers and Consumers.
Permission | DevOps / Platform Engineer (Producer) | Developer (Consumer) |
---|---|---|
Create Projects | Yes | No |
Create and modify Variable Sets | Yes | No |
Create and modify Environments and Lifecycles | Yes | No |
Create and modify cloud accounts, feeds, and GitHub accounts | Yes | No |
Projects - Configure Version Control and branch protection policies | Yes | No |
Projects - Create and modify channels | Yes | No |
Projects - Modify ITSM settings | Yes | No |
Projects - Modify guided failure settings | Yes | No |
Projects - Modify runbooks | Yes - must be done in a branch and submitted via a PR | Yes - must be done in a branch and submitted via a PR |
Projects - Modify deployment process | Yes - must be done in a branch and submitted via a PR | Yes - must be done in a branch and submitted via a PR |
Projects - Modify variables | Yes - must be done in a branch and submitted via a PR | Yes - must be done in a branch and submitted via a PR |
To accomplish that separation, you will need to:
- Configure project version control for each project.
- Configure branch protection policies for the
main
or primary branch in project version control. - Configure RBAC for your users (see below).
RBAC In Octopus
For producers (Platform/DevOps Engineers), youâll need to create a custom role called Platform Engineer
. For Consumers (developers), youâll need to create a custom role called Developer
.
The specific permissions for each role will be:
Permission | Platform Engineers | Developer |
---|---|---|
AccountCreate | Yes | No |
AccountDelete | Yes | No |
AccountEdit | Yes | No |
AccountView | Yes | Yes |
ActionTemplateCreate | Yes | No |
ActionTemplateDelete | Yes | No |
ActionTemplateEdit | Yes | No |
ActionTemplateView | Yes | Yes |
ArtifactCreate | Yes | No |
ArtifactDelete | Yes | No |
ArtifactEdit | Yes | No |
ArtifactView | Yes | Yes |
BuiltInFeedPush | Yes | Yes |
CertificateView | Yes | Yes |
DefectReport | Yes | Yes |
DefectResolve | Yes | Yes |
DeploymentView | Yes | Yes |
EnvironmentCreate | Yes | No |
EnvironmentDelete | Yes | No |
EnvironmentEdit | Yes | No |
EnvironmentView | Yes | Yes |
EventView | Yes | Yes |
FeedEdit | Yes | No |
FeedView | Yes | Yes |
GitCredentialEdit | Yes | No |
GitCredentialView | Yes | Yes |
InsightsReportCreate | Yes | No |
InsightsReportDelete | Yes | No |
InsightsReportEdit | Yes | No |
InsightsReportView | Yes | Yes |
InterruptionView | Yes | Yes |
InterruptionViewSubmitResponsible | Yes | Yes |
LibraryVariableSetCreate | Yes | No |
LibraryVariableSetDelete | Yes | No |
LibraryVariableSetEdit | Yes | No |
LibraryVariableSetView | Yes | Yes |
LifecycleCreate | Yes | No |
LifecycleDelete | Yes | No |
LifecycleEdit | Yes | No |
LifecycleView | Yes | Yes |
MachineCreate | Yes | No |
MachineDelete | Yes | No |
MachineEdit | Yes | No |
MachineView | Yes | Yes |
MachinePolicyCreate | Yes | No |
MachinePolicyDelete | Yes | No |
MachinePolicyEdit | Yes | No |
MachinePolicyView | Yes | Yes |
ProcessEdit | Yes | Yes |
ProcessView | Yes | Yes |
ProjectCreate | Yes | No |
ProjectDelete | Yes | No |
ProjectEdit | Yes | No |
ProjectView | Yes | Yes |
ProjectGroupCreate | Yes | No |
ProjectGroupDelete | Yes | No |
ProjectGroupEdit | Yes | No |
ProjectGroupView | Yes | Yes |
ProxyCreate | Yes | No |
ProxyDelete | Yes | No |
ProxyEdit | Yes | No |
ProxyView | Yes | Yes |
ReleaseCreate | Yes | Yes |
ReleaseDelete | Yes | No |
ReleaseView | Yes | Yes |
RetentionAdminister | Yes | No |
RunbookEdit | Yes | Yes |
RunbookRunView | Yes | Yes |
RunbookView | Yes | Yes |
SubscriptionCreate | Yes | No |
SubscriptionDelete | Yes | No |
SubscriptionEdit | Yes | No |
SubscriptionView | Yes | Yes |
TagSetCreate | Yes | No |
TagSetDelete | Yes | No |
TagSetEdit | Yes | No |
TargetTagAdminister | Yes | No |
TargetTagView | Yes | Yes |
TaskCancel | Yes | No |
TaskCreate | Yes | No |
TaskView | Yes | Yes |
TenantView | Yes | Yes |
TriggerCreate | Yes | Yes |
TriggerDelete | Yes | Yes |
TriggerEdit | Yes | Yes |
TriggerView | Yes | Yes |
TeamCreate | Yes | No |
TeamDelete | Yes | No |
TeamEdit | Yes | No |
TeamView | Yes | Yes |
VariableEdit | Yes | Yes |
VariableEditUnscoped | Yes | Yes |
VariableView | Yes | Yes |
VariableViewUnscoped | Yes | Yes |
WorkerEdit | Yes | No |
WorkerView | Yes | Yes |
System Perm - DeploymentFreezeAdminister | Yes | No |
System Permission - SpaceCreate | Yes | No |
System Permission - SpaceDelete | Yes | No |
System Permission - SpaceEdit | Yes | No |
System Permission - SpaceView | Yes | No |
System Perm - PlatformHubEdit | Yes | No |
System Perm - PlatformHubView | Yes | No |
System Permission - TaskCancel | Yes | No |
System Permission - TaskCreate | Yes | No |
System Permission - TeamCreate | Yes | No |
System Permission - TeamDelete | Yes | No |
System Permission - TeamEdit | Yes | No |
System Permission - TeamView | Yes | Yes |
System Permission - UserInvite | Yes | Yes |
System Permission - UserRoleView | Yes | Yes |
System Permission - UserView | Yes | Yes |
Once those roles are created, create the appropriate teams and assign them those roles. Do not scope them to any environment or tenant.
The new roles detailed above provide the appropriate editing capabilities within Octopus Deploy. They purposely exclude creating deployments and runbook runs. Youâll likely want to scope them to the appropriate environments or tenants.
Common scenarios we see:
- Developers can deploy any project to
Development
andTest
, but notProduction
. - Developers can deploy specific projects to
Development
,Test
, andProduction
. But no other projects. - Release managers or web admins can deploy to
Production
.
Role | Platform Engineers | Developer |
---|---|---|
Deployment Creator | Yes - no scoping | Yes - scoped to specific environments, tenants, or projects |
Runbook Consumer | Yes - no scoping | Yes - scoped to specific environments, tenants, or projects |
Tenant Manager (if leveraging multi-tenancy) | Yes - no scoping | Yes - scoped to specific environments, tenants, or projects |
The Developer team on my instance is configured to allow them to deploy to Development
, Test
, and Staging
while allowing them to run runbooks on any environment.
Cloud Accounts and Third-Party Key Vaults
In my example deployment pipeline, Iâm storing secrets inside of Azure Key Vault instead of using Octopus Deployâs sensitive variables. My primary reason is that Azure Key Vault is focused on storing secrets and doesnât try to be anything else. It offers features and functionality, such as secret rotation and versioning, for storing secrets that Octopus Deploy doesnât provide.
This blog post walks you through configuring Azure Key Vault with Octopus Deploy. The primary difference between my configuration and that configuration is that my Azure Account in Octopus Deploy uses OIDC.
With Third-Party Key Vaults and OIDC, I aim to eliminate Octopus Deploy from the secret-storing business altogether.
Lifecycles
Lifecycles are among Octopus Deployâs most misused features.
The most common misconfiguration I see is:
- Default lifecycle:
Development
->Test
->Staging
->Production
- Hotfix lifecycle:
Staging
->Production
The common reason behind that configuration is âwe need a clear path to production in the event of an emergency.â Â While a valid point, the primary problem with that configuration is that Development
is included in any path to production. Development
should be used for fast feedback on a work in progress. At no point should code from Development
ever be promoted to Production
. It must go through a pull request/approval workflow, which is at the core of supply chain security.
The recommended configuration is:
- Default Lifecycle:
Development
- Release Lifecycle:
Test
->Staging
->Production
The default lifecycle deployment destination for all branches except the main
or primary branch is Development
. The release lifecycle is only for main
or the primary branch. With this configuration, only approved code (remember we configured the branch ruleset in GitHub) is deployed to Production
.
ITSM Integration
Octopus Deploy supports ServiceNow and Jira Service Management for ITSM approvals. If you are using those services and have an Enterprise tier license, you should configure that integration as soon as possible.
Our ITSM integration will create a change request and wait until it reaches the appropriate state. Octopus will not even start the deployment until that state is reached.
Not all environments require ITSM approval, so we require you to enable ITSM per environment. Because you can turn off/on ITSM integration, I recommend restricting EnvironmentEdit
permissions to Platform Engineers.
GitHub Octopus Deploy Integration
If you are using Octopus Deploy cloud, configure the Octopus Deploy GitHub Application. This allows you to connect to GitHub repositories without providing a PAT. You can still control the connection to specific repositories and organizations.
Build Information and Issue Tracking Integration
Octopus Deploy natively integrates with JIRA, GitHub Issue Tracker, and Azure DevOps Issue Tracker. If you are using JIRA Cloud, we also provide the capability to update the deployment status of JIRA Tickets.
That requires the configuration of JIRA Integration with Octopus. Our documentation provides a step-by-step guide for configuration. The result looks like this:
If you recall, earlier, the GitHub action was publishing the build information for the SBOM. Octopus parses the build commit message to look for the issue. My commit messages were prefixed with TRID-43
, which Octopus then linked to that ticket in JIRA.
Project Settings
We want to configure the Octopus Deploy project to follow these rules:
- Any changes to the deployment process, variables, and runbooks require approval.
- Only the
main
or primary branch can be deployed toProduction
. - Only
pre-release
versions can be deployed toDevelopment
;Production
only accepts packages withoutpre-release
versions. - If applicable, ITSM integration is enabled.
Important: ProjectEdit
permission was restricted to Platform Engineers because it can modify a projectâs Project Version Control and ITSM Provider configuration. Specifically, anyone with those permissions can turn off branch protection policies and ITSM requirements. What used to be a permission to change the name and icon of the project has grown into a powerful setting. Be mindful of who grants access to it.
Configure Project Version Control
We want to use the same pull-request approval process for the deployment process, variables, and runbook updates as used for source code. To do that, configure project version control for your project. While doing that, configure the branch protection policy for the main
or primary branch. In the Octopus Deploy UI, youâll see notifications that the main
or primary branch is protected and requires any changes to be made to the branch.
I also recommend using the same git repository as your source code for the following reasons:
- The code repository should store the source code, how it is built, its dependencies, how it is hosted, and how it is deployed. That allows you to reproduce production in local and testing environments quickly.
- Often, underlying code changes, such as migrating to Kubernetes, require an update to the build and deployment process. All the necessary changes to migrate to Kubernetes can be made in the same branch and merged using the same pull request process.
Important: Anyone with the ProjectEdit
permission can change the branch protection policy setting. Using the recommended roles from earlier will limit that permission to Platform Engineers.
Configure Channels
Now that version control is enabled, we can configure two channels for our two lifecycles. We will include version and branch rules in those channels.
- Default Channel
- Lifecycle: Default Lifecycle
- Package Version Rules: select the steps that deploy packages and enter this in the pre-release tag field
^[^\+].*
. That will ensure only packages with a pre-release tag can be deployed toDevelopment
- Branch Protection Rules: Enter a pattern to ensure anything but the
main
or primary branch can be used. For my instance I use[feature|hotfix]*/*
- Release Channel
- Lifecycle: Release Lifecycle
- Package Version Rules: select the steps that deploy packages and enter this into the pre-release tag field
^(|\+.*)$
. That will block any packages with a pre-release tag from being selected. - Branch Protection rules: enter
main
into the branches field. That will ensure only the main branch can be used. Â With these rules in place, we can ensure only artifacts and processes from themain
branch will be used forProduction
deployments.
ITSM
In the ITSM Providers screen, check the Change Controlled
check box and select the appropriate ITSM provider connection. You can also select specific runbooks to be changed controlled in this screen.
Important: Anyone with the ProjectEdit
permission can change the ITSM setting. Using the recommended roles from earlier will limit that permission to Platform Engineers.
Deployment Process
The final piece of our supply chain security workflow is to add the appropriate steps to pull secrets from Azure Key Vault, publish the SBOM as a deployment artifact, and verify the attestations from GitHub.
Pulling secrets from Azure Key Vault
The deployment process uses community library step template to interact with Azure Key Vault.
I recommend using the execution container octopuslabs/azure-workertools, which has all the required CLIs to pull secrets from the key vault.
The step will pull one-to-many secrets. It uses the format [Secret Name] | [Output Variable Name]
to determine the secrets to pull. For example:
azure-sql-server | SQLServerName
azure-sql-password | SQLUserPassword
azure-sql-username | SQLUserName
octopus-api-key | OctopusApiKey
github-token | GitHubToken
sumo-logic-url | SumoLogicUrl
The output variable name is how you can reference the secrets in subsequent steps. For example, the SQL Server Name can be accessed in subsequent steps by looking for the Octopus.Action[Azure Key Vault - Retrieve Secrets].Output.SQLServerName
variable.
Publishing the SBOM as a deployment artifact
We can use pre-existing functionality to publish the SBOM as a deployment artifact. Add a script step to the process, then add a package reference. Ensure the package reference extracts the package during the deployment.
The script only needs to find the .json
file and publish it as a deployment artifact. The following is the PowerShell script I used in the deployment pipeline.
$extractedPath = $OctopusParameters["Octopus.Action.Package[Template.SBOM.Artifact].ExtractedPath"]
$OctopusEnvironmentName = $OctopusParameters["Octopus.Environment.Name"]
$sbomFiles = Get-ChildItem -Path $extractedPath -Filter "*.json" -Recurse
foreach ($sbom in $sbomFiles)
{
Write-Host "Attaching $($sbom.FullName) as an artifact"
New-OctopusArtifact -Path $sbom.FullName -Name "$OctopusEnvironmentName.SBOM.JSON"
break
}
Verifying attestations
We will use the gh attestation verify
CLI command as the means to verify the attestations. Weâve provided octopuslabs/github-workertools
as an execution container so you donât have to worry about downloading and installing the CLI.
That command converts the deployment artifact to a SHA256 hash and then looks for attestations matching that hash for your repository or organization. If the artifact has been tampered with, the attestation verification will fail because it wonât find a matching hash. For example, here is the API endpoint my process attempted to hit on a failed attestation verification.
https://api.github.com/repos/BobJWalker/Trident/attestations/sha256:d96a5d80bc2d7ce427dc1f543d7caedf4ea014a397cae0140909dfa306a48b1b?per_page=30&predicate_type=https://slsa.dev/provenance/v1
After the CLI finds the attestation for the artifact, it will perform an additional series of verifications. More information about how it verifies the attestation can be found in the GitHubâs docs.
For package attestation verification, you can use our community step template.
For Docker containers, youâll need to authenticate to the container registry. Once you do that, you can run a script similar to this:
$packageVersion = $OctopusParameters["Octopus.Action.Package[YOUR CONTAINER].PackageVersion"]
$packageName = $OctopusParameters["Octopus.Action.Package[YOUR CONTAINER].PackageId"]
$GHToken = "YOUR GITHUB TOKEN"
$image = "oci://$($packageName)$:$($packageVersion)$"
$repo = "bobjwalker/trident"
$env:GITHUB_TOKEN=$GHToken
$attestation=gh attestation verify "$image" --repo $repo --format json
if ($LASTEXITCODE -ne 0)
{
Write-Error "The attestation for $image could not be verified"
}
Write-Highlight "$image successfully passed attestation verification"
Write-Verbose $attestation
$artifactVerified = $true
Write-Host "Writing the attest output to $packageName.$OctopusEnvironmentName.attestation.json"
New-Item -Name "$packageName.$OctopusEnvironmentName.attestation.json" -ItemType "File" -Value $attestation
New-OctopusArtifact -Path "$packageName.$OctopusEnvironmentName.attestation.json" -Name "$packageName.$OctopusEnvironmentName.attestation.json"
Conclusion
There will always be a healthy tension between security and usability. You can enact security policies that replicate the computer vault from the first Mission: Impossible film, but it isnât very usable at scale. On the other side of the spectrum, everyone can be made an admin, and there is no red tape, but that isnât very secure. Finding the ârightâ amount of security is always a challenge. My goal with this article was to provide low-effort recommendations that are as non-intrusive as possible but substantially increase supply chain security. My guiding ânorth starâ was SLSA level 3 with this pipeline.
Despite the changes, when the rules are followed (pull request required, build on every check-in, always verify attestations, etc.), there should be no noticeable impact on the developer experience. Itâs only when someone attempts to bypass the rules that they encounter friction.
Tags:
Related posts

AI deployments best practices
AI deployments present some unique challenges for the DevOps team. And yet, existing DevOps best practices still apply.

The productivity delusion
Find out why measuring productivity is a fast-track to failure and what to do instead.

Deploying LLMs with Octopus
LLMs are now a common component of modern applications, but deploying them with Kubernetes requires careful consideration. This article explores the challenges and best practices for deploying LLMs in a Kubernetes environment.