This section walks through the different options and considerations for the components required to set up Octopus High Availability in AWS.
Setting up Octopus: High Availability
This guide assumes that all of the servers used for your Octopus High Availability instance are hosted in AWS and are running Windows Server.
Some assembly required
Octopus High Availability is designed for mission-critical enterprise scenarios and depends heavily on infrastructure and Microsoft components. At a minimum:
- You should be familiar with SQL Server failover clustering, AWS RDS, or have DBAs available to create and manage the database.
- You should be familiar with SANs, AWS FSx, or other approaches to sharing storage between servers.
- You should be familiar with load balancing for applications.
IaaS vs PaaS: If you are planning on using IaaS exclusively in AWS and don’t intend to use their PaaS offerings (such as AWS RDS), then the On-Premises guide might be a better approach for you as management of your virtual machines, Domain Controllers, SQL Database Servers, and load balancers will be your responsibility.
To install Octopus nodes you need at least two machines running Windows Server 2016+. There’s only one choice when building virtual machines in AWS, and that’s EC2 Instances. There are a number of different instance types to choose from. When selecting the size of the instance, we generally find sticking with the General purpose size is the best option.
We recommend starting with either 2 cores / 4 GB of RAM or 4 cores / 8 GB of RAM and limiting the task cap to 20 for each node. In our experience, it is much better to have 4 smaller VMs, each with 4 cores / 8 GB of RAM than 2 large VMs, each with 8 cores / 16 GB of RAM. With 2 servers, if one of them were to go down, you’d lose 50% of your capacity. With 4 servers, if one of them were to go down, you’d lose 25% of your capacity. The difference in cost between the 4 smaller VMs and 2 large VMs is typically minimal.
Due to how Octopus stores the paths to various BLOB data (task logs, artifacts, packages, imports, event exports etc.), you cannot run a mix of both Windows Servers, and Octopus Linux containers connected to the same Octopus Deploy instance. A single instance should only be hosted using one method.
Each Octopus Server node stores project, environment, and deployment-related data in a shared Microsoft SQL Server Database. Since this database is shared, it’s important that the database server is also highly available.
If you don’t have a SQL cluster in AWS then AWS provides a fully managed and highly available SQL database as a service called Amazon RDS for SQL Server.
Choosing a SQL edition is an important decision, and will depend on your organization requirements and Octopus usage.
It’s not possible to change the edition of a SQL Server RDS instance by modifying it without taking a snapshot and restoring that to a new instance.
For a highly available SQL Server in AWS RDS, we recommend SQL Server Standard or higher.
Once you’ve settled on an edition, the great thing about using AWS RDS is that you can start small and scale the size of your instance on demand as your Octopus usage grows. AWS provide a list of the instance sizes for each SQL Server edition and versions.
How the database is made highly available is really up to you; to Octopus, it’s just a connection string. We are not experts on SQL Server high availability, so if you have an on-site DBA team, we recommend using them. There are many options for high availability with SQL Server, and Brent Ozar also has a fantastic set of resources on SQL Server Failover Clustering if you are looking for an introduction and practical guide to setting it up.
Octopus High Availability works with:
Octopus High Availability has not been tested with Log Shipping or Database Mirroring, and does not support SQL Server replication. More information
See also the SQL Server Database page, which explains the editions and versions of SQL Server that Octopus supports and explains the requirements for how the database must be configured.
Octopus stores several files that are not suitable to store in the database. These include:
- Packages used by the built-in repository. These packages can often be very large in size.
- Artifacts collected during a deployment. Teams using Octopus sometimes use this feature to collect large log files and other files from machines during a deployment.
- Task logs are text files that store all of the log output from deployments and other tasks.
- Imported zip files used by the Export/Import Projects feature.
- Archived audit logs by the Archived audit logs feature.
As with the database, you’ll tell the Octopus Servers where to store them as a file path within your operating system. The shared storage needs to be accessible by all Octopus nodes. Each of these three types of data can be stored in a different location.
Whichever way you provide the shared storage, there are a few considerations to keep in mind:
- To Octopus, it needs to appear as either:
A mapped network drive e.g.
A UNC path to a file share e.g.
A symbolic link pointing at a local folder, e.g.
C:\OctopusShared\Artifacts <<===>> \\server\share\Artifacts
- The service account that Octopus runs needs full control over the directory.
- Drives are mapped per-user, so you should map the drive using the same service account that Octopus is running under.
AWS offers file storage called
Amazon FSx, It’s a native Windows file system built on Windows Server. It includes full support for the SMB protocol, Windows NTFS, and requires Microsoft Active Directory (AD) integration. This makes it an ideal choice for connecting to your EC2 instances hosting Octopus to store all your Octopus packages and log files.
If you choose to go with Amazon FSx there are some resources that will help you get started:
- AWS have a starter guide which explains how to configure Amazon FSx and connect it up to an EC2 machine.
- AWS have a hands-on lab on using DataSync to support multi-region FSx data across AWS regions. This could be useful when considering disaster recovery options for Octopus High Availability.
- We have an AWS FSx High Availability blog post which is a step-by-step guide to connecting Amazon FSx to your Octopus High Availability Server nodes on Windows.
Load Balancing in AWS
To distribute traffic to the Octopus web portal on multiple nodes, you need to use a HTTP load balancer. AWS provides a solution to distribute HTTP/HTTPS traffic to EC2 instances, Elastic Load Balancing is a highly available, secure, and elastic load balancer. There are three implementations of ELB;
If you are only using Listening Tentacles, we recommend using the Application Load Balancer.
However, Polling Tentacles don’t work well with the Application Load Balancer, so instead, we recommend using the Network Load Balancer. To setup a Network Load Balancer for Octopus High Availability with Polling Tentacles take a look at this knowledge base article.
Octopus Server provides a health check endpoint for your load balancer to ping:
Making a standard
HTTP GET request to this URL on your Octopus Server nodes will return:
- HTTP Status Code
200 OKas long as the Octopus Server node is online and not in drain mode.
- HTTP Status Code
418 I'm a teapotwhen the Octopus Server node is online, but it is currently in drain mode preparing for maintenance.
- Anything else indicates the Octopus Server node is offline, or something has gone wrong with this node.
The Octopus Server node configuration is also returned as JSON in the HTTP response body.
We typically recommend using a round-robin (or similar) approach for sharing traffic between the nodes in your cluster, as the Octopus Web Portal is stateless.
All package uploads are sent as a POST to the REST API endpoint
/api/[SPACE-ID]/packages/raw. Because the REST API will be behind a load balancer, you’ll need to configure the following on the load balancer:
- Timeout: Octopus is designed to handle 1 GB+ packages, which takes longer than the typical http/https timeout to upload.
- Request Size: Octopus does not have a size limit on the request body for packages. Some load balancers only allow 2 or 3 MB files by default.
Polling Tentacles with HA
Listening Tentacles require no special configuration for Octopus High Availability. Polling Tentacles, however, poll a server at regular intervals to check if there are any tasks waiting for the Tentacle to perform. In a High Availability scenario Polling Tentacles must poll all of the Octopus Server nodes in your configuration.
Connecting Polling Tentacles
Whilst a Tentacle could poll a load balancer in an Octopus High Availability cluster, there is a risk, depending on your load balancer configuration, that the Tentacle will not poll all servers in a timely manner.
We recommend two options when configuring Polling Tentacles to connect to your Octopus High Availability cluster:
- Using a unique address, and the same listening port (
10943by default) for each node.
- Using the same address and a unique port for each node.
These are discussed further in the next sections.
Using a unique address
In this scenario, no load balancer is required. Instead, each Octopus node would be configured to listen on the same port (
10943 by default) for inbound traffic. In addition, each node would be able to be reached directly by your Polling Tentacle on a unique address for the node.
For each node in your HA cluster:
- Ensure the communication port Octopus listens on (
10943by default) is open, including any firewall.
- Register the node with the Poll Server command line option. Specify the unique address for the node, including the listening port. For example, in a three-node cluster:
- Node1 would use address: Octo1.domain.com:10943
- Node2 would use address: Octo2.domain.com:10943
- Node3 would use address: Octo3.domain.com:10943
The important thing to remember is that each node should be using a unique address and the same port.
A Polling Tentacle will connect to the Octopus Rest API over ports 80 or 443 when it is registering itself with the Octopus Server. After that, it will connect over port
10943 (by default) with the Octopus Server node.
It’s important to ensure that any firewalls also allow port 80 or 443 for the initial Tentacle registration.
Using a unique port
In this scenario, a type of Network Address Translation (NAT) is leveraged by using the same address and unique ports, usually routed through a load balancer or other network device. Each Octopus node would be configured to listen on a different port (starting at
10943 by default) for inbound traffic.
The advantage of using unique ports is that the Polling Tentacle doesn’t need to know each node’s address, only the port. The address translation is handled by the load balancer. This allows each node to have a private IP address, with no public access from outside your network required.
Imagine a three-node HA cluster. For each one, we expose a different port to listen on using the Octopus.Server configure command:
- Node1 - Port
- Node2 - Port
- Node3 - Port
Next on the load balancer, create Network Address Translation (NAT) rules and point them to each node in your HA Cluster:
- Open port
10943and route traffic to Node1 in your HA Cluster
- Open port
10944and route traffic to Node2 in your HA Cluster
- Open port
10945and route traffic to Node3 in your HA Cluster
- Continue for any additional nodes in your HA cluster.
If you configured your nodes to use a different listening port, replace
10945 with your port range.
The important thing to remember is that each node should be using the same address and a different port.
Registering Polling Tentacles
There are two options to add Octopus Servers to a Polling Tentacle, via the command-line or via editing the Tentacle.config file directly.
Configuring a Polling Tentacle via the command-line is the preferred option with the command executed once per server; an example command using the default instance can be seen below:
C:\Program Files\Octopus Deploy\Tentacle>Tentacle poll-server --server=http://my.Octopus.server --apikey=API-77751F90F9EEDCEE0C0CD84F7A3CC726AD123FA6
For more information on this command please refer to the Tentacle Poll Server command line options.
Alternatively you can edit Tentacle.config directly to add each Octopus Server (this is interpreted as a JSON array of servers). This method is not recommended as the Tentacle service for each server will need to be restarted to accept incoming connections via this method.
Notice there is an address entry for each Octopus Server in the High Availability configuration.
Help us continuously improve
Please let us know if you have any feedback about this page.
Page updated on Sunday, January 1, 2023