Migration

Ask Octopus Episode #14 - Configure your HA Cluster

You may already have an existing Octopus Server that you wish to make highly available. The process to migrate to Octopus High Availability is the same as the process detailed in Configuring High Availability for Octopus, except your existing server will be the first node in the cluster. Migrating to HA will involve:

1. Moving the SQL Server Database to a dedicated SQL Server.
2. Moving all the task logs, packages, artifacts, imports, event exports etc., to a shared storage folder (BLOB data).
3. Configuring a load balancer.

This guide is generic and purposely avoids mentioning specific technologies such as Azure File Storage, AWS RDS SQL Server, etc. Please refer to the guide matching your hosting solution for specifics.

• Designing Octopus for High Availability On-Premises
• Designing Octopus for High Availability in Azure
• Designing Octopus for High Availability in AWS

Prep Work

These actions will require downtime. You can do prep work to keep the downtime to a minimum.

Moving the database

Moving the SQL Server database involves performing a backup and restore of the database. That backup and restore have to occur during the outage window.

You can prepare for that by doing the following:

• Provision the SQL Server Instance (if it doesn’t already exist).
• Create the SQL Server user Octopus will use to log into SQL Server (if it doesn’t already exist).

After the SQL Server has been provisioned and the user has been created, you’ll want to ensure Octopus Deploy can connect to the SQL Server. It is important to do this on the server hosting Octopus Deploy with the same user the Octopus Deploy Windows Service is running as. If the Octopus Deploy Windows Service is running as a Local System any administrator account should work for this test.

$userName = ""
$password = ""
$newSQLServer = ""

if ([string]::IsNullOrWhiteSpace($userName) -eq $true){
    Write-Host "No username found, using integrated security"
    $connectionString = "Server=$newSQLServer;Database=master;integrated security=true;"
}
else {
    Write-Host "Username found, using SQL Authentication"
    $connectionString = "Server=$newSQLServer;Database=master;User ID=$userName;Password=$password;"
}

$sqlConnection = New-Object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = $connectionString

Write-Host "Attempting to connect to $newSQLServer"
$sqlConnection.Open()

Write-Host "Connection successful.  Closing connection."
$sqlConnection.Close()

Moving BLOB data

Most of the BLOB data (task logs, artifacts, packages, imports, event exports etc) stored on the file system can be copied to the new location prior to the outage window. Doing so will reduce the amount of copying you have to do during the outage windows. In addition, you can make sure your Octopus Deploy instance can use that shared location by running a test script to create and delete a file.

• Provision the shared storage folder.
• If you are going to create a symbolic link to that shared folder, do that now.
• Use the script console to ensure Octopus can connect to the shared folder and create files.

$filePath = "YOUR DIRECTORY"
New-Item "$filePath\file.txt" -ItemType file
Remove-Item "$filepath\file.txt"

• Run a tool such RoboCopy to copy the folder contents.

An example PowerShell script using RoboCopy will be:

robocopy C:\Octopus\TaskLogs \\YOURFILESHARE\OctopusHA\TaskLogs /mir /r:5
robocopy C:\Octopus\Artifacts \\YOURFILESHARE\OctopusHA\Artifacts /mir /r:5
robocopy C:\Octopus\Packages \\YOURFILESHARE\OctopusHA\Packages /mir /r:5
robocopy C:\Octopus\Imports \\YOURFILESHARE\OctopusHA\Imports /mir /r:5
robocopy C:\Octopus\EventExports \\YOURFILESHARE\OctopusHA\EventExports /mir /r:5

Configure load balancer

Octopus Deploy must sit behind a load balancer when configured in HA mode. We recommend creating a new URL for your Octopus HA cluster. For example, if the current URL for your Octopus Instance is octopus.mydomain.local, the load-balanced URL could be octopusha.mydomain.local.

The advantage of a new URL is:

1. You can still access each server directly (if need be).
2. The process of redirecting users and applications to the new URL should only need to be done once.
3. You can configure and test it before the outage window, along with working through any connection issues.

Octopus Server provides a health check endpoint for your load balancer to ping: /api/octopusservernodes/ping.

Making a standard HTTP GET request to this URL on your Octopus Server nodes will return:

• HTTP Status Code 200 OK as long as the Octopus Server node is online and not in drain mode.
• HTTP Status Code 418 I'm a teapot when the Octopus Server node is online, but it is currently in drain mode preparing for maintenance.
• Anything else indicates the Octopus Server node is offline, or something has gone wrong with this node.

We typically recommend using a round-robin (or similar) approach for sharing traffic between the nodes in your cluster, as the Octopus Web Portal is stateless.

All package uploads are sent as a POST to the REST API endpoint /api/[SPACE-ID]/packages/raw. Because the REST API will be behind a load balancer, you’ll need to configure the following on the load balancer:

• Timeout: Octopus is designed to handle 1 GB+ packages, which takes longer than the typical http/https timeout to upload.
• Request Size: Octopus does not have a size limit on the request body for packages. Some load balancers only allow 2 or 3 MB files by default.

Outage windows

The below steps will cause an outage in Octopus Deploy. With all the prep work, the outage window should be small. If possible, we recommend making these change off-hours. In addition, you don’t have to do them all in one outage window. You can move the database in one outage window, and the file system in the other outage window.

During your outage window, perform the following steps (skip the sections that don’t apply).

1. Ensure you have a backup of your master key.
2. Enable Maintenance Mode to prevent anyone from deploying or making changes during the upgrade.
3. Stop the Octopus Deploy windows service.

Move the database

1. Back up the database.
2. Restore the database on the new SQL Server.
3. On your Octopus Server, run the following command to update the connection string (where “VALUE” is your connection string).

Set-Location "C:\Program Files\Octopus Deploy\Octopus"

& .\Octopus.Server.exe database --connectionString="VALUE"

Move the file storage

1. Run RoboCopy one final time to pick up any new files.
2. Run the following command to update the paths.

Set-Location "C:\Program Files\Octopus Deploy\Octopus"
$filePath = "YOUR ROOT DIRECTORY"

& .\Octopus.Server.exe path --clusterShared "$filePath"
& .\Octopus.Server.exe path --artifacts "$filePath\Artifacts"
& .\Octopus.Server.exe path --taskLogs "$filePath\TaskLogs"
& .\Octopus.Server.exe path --nugetRepository "$filePath\Packages"
& .\Octopus.Server.exe path --imports "$filePath\Imports"
& .\Octopus.Server.exe path --eventExports "$filePath\EventExports"
& .\Octopus.Server.exe path --telemetry "$filePath\Telemetry"

After moving database and file storage

After you finish moving the database and file storage, it is time to turn back on your Octopus Deploy instance.

1. Turn back on the Octopus Deploy instance. If the instance fails to start, that indicates a database connection issue.
2. Log in to your instance.
3. Navigate to previous deployments. If you cannot see the task logs, that indicates a file storage issue.
4. Perform a couple of test deployments.
5. Assuming all goes well, disable maintenance mode.
6. Notify everyone of the new URL (if there is one).

Adding additional nodes

After configuring a load balancer and moving the database and files, adding a new node is trivial.

1. Create a new server to host Octopus Deploy.
2. Install the same version by downloading it from our download archive.
3. When the Octopus Manager loads, click the Add this instance to a High Availability cluster and follow the wizard.
4. Add that server to your load balancer.

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback

Page updated on Wednesday, October 4, 2023