How do workers... work?
Until now, there were two places that Octopus Deploy ran your deployment code: on the servers you’re deploying to, using Tentacle, and directly on the Octopus server. Workers give you a third place to run parts of your deployment; a place to run code on servers that you deploy from, without running it on the Octopus server.
Workers are the best place to run deployment steps like:
- Publishing to Azure websites
- Deploying AWS CloudFormation templates
- Deploying to AWS Elastic Beanstalk
- Uploading files to Amazon S3
- Backing up databases
- Performing database schema migrations
- Configuring load balancers.
With all of these steps you usually only want them to run "once" in a deployment, and from a specific machine or set of machines. It doesn’t make sense to execute them on the web and application servers you’re deploying to. For this reason, these tasks tend to run on the Octopus server. With workers, you can now run them on special servers dedicated to the task, and away from the Octopus server.
Workers are grouped into worker pools. When a deployment step is configured to run using a worker pool, at deployment time, Octopus selects the first available worker from the pool. It’s the same paradigm as build agents on a build server: we just need one worker to run the task, and we don’t care which one so long as it’s not busy.
Octopus takes care of assigning workers to run deployment steps and transferring packages to workers so you don't have to. Octopus will spread the workload when it can, while also trying to reuse workers when it gives you the advantages of package caching.
Workers with different platforms and capabilities
When a team configures Octopus to run a script, there’s often an assumption about the platform and dependencies on the server that will run the script. Maybe the script assumes it's running on a Linux server, or maybe it assumes a certain version of the Azure SDK is installed. With workers, you can create a pool for each of the platforms and dependencies your team needs:
- Pools for scripts that expect different operating systems (Linux vs. Windows)
- Pools for scripts that rely on different versions of cloud CLI tools: Azure CLI, AWS CLI, Terraform CLI, Kubernetes CLI (kubectl) etc
- Pools in special subnets or access to protected servers, such as for database deployments or in DMZs.
Performance and scalability
Workers enable you to shift deployment work onto other machines running in pools. This can greatly reduce the work performed on your Octopus server, freeing it to more readily execute and orchestrate more deployments and make the UI & API much more responsive.
Worker pools can improve scalability by enabling dedicated pools of machines that can be utilized by multiple projects and teams. When a deployment uses a worker, Octopus will select a worker from the pool. Since you can have many machines in a pool, you can scale your workers to match your deployment load. This can significantly improve deployment speed and resource utilization for both large and small teams.
Great examples of this include utilizing workers with database deployments, deployments to cloud infrastructure like Azure, AWS and Kubernetes (K8s) and specialized tasks like network configuration and firewall changes.
Workers make Octopus Server more secure
Without workers, when teams tell Octopus to run a PowerShell script on the Octopus server, they have the ability to execute arbitrary code on the Octopus server. This is by design; after all, the whole point of Octopus is to give you an easy place to run scripts.
Of course, there’s the chance that this power could be abused. Between permissions management and auditing, this isn’t an issue for most teams.
With workers, that risk is eliminated, as you can prevent teams from running scripts on the Octopus server and make them use workers instead. This can make it much easier to pass your next PCI-DSS compliance and other security audits.
Unlike Octopus server nodes, workers and worker pools can be geographically distributed, greatly improving the speed of deployments to different geographical regions. Placing your workers close to your production networks can help to reduce latency and round-trip communication times and avoid potentially significant package transfer times once a worker has a copy of the application or service being deployed.
Where you place your workers, your package feeds, and how your networks are configured can all affect your networking costs (though, of course, worker VMs cost too) and deployment efficiency. For example, colocation of workers and targets is likely to reduce the round trip time (compared to the Octopus server to target time), and options for workers to acquire their own packages reduces the amount of traffic the Octopus server has to push out. Regardless of how you get the packages there, you'll see some improvements in network use and CPU cost on your Octopus server. If you use the same package more than once, you'll also save because the worker will already have it.
And of course, reduced network traffic means one thing: reduced deployment windows and application downtime.
If you deploy to multiple regions, you can, of course, provision worker pools for each region, match them up with your targets, and Octopus will use the right workers when deploying to the different regions.