Some customers move fast. They read the release notes, spin up a test instance, and upgrade the same week. Others move much more carefully, and for good reason. When your deployment pipelines touch thousands of servers across a global business, caution is necessary.
This is the story of one of those customers: a large financial services organization running Octopus Server as the backbone of its deployment process. They’d been on the same version for a long time, well behind current, and running a single Octopus Server instance which was beginning to show signs of stress under the load, so they wanted to move to a High Availability (HA) cluster.
A cautious, multinational customer
I’ve worked with the Infrastructure team at this multinational FSI long enough to know they are deliberate in their decision making, only making changes to existing structures after they have done extensive planning and testing. Stability is key when so much depends on their infrastructure being available at all times.
The upgrade itself wasn’t just a version bump. It meant also moving from a single server to an HA cluster in a different Active Directory domain; which touched the database, the Octopus home directory and shared storage, authentication, and eventually DNS too. Any one of those pieces going wrong during business hours could affect deployments across their entire estate. Understandably, they wanted to slow down and understand each piece before committing to any of it.
Answering the questions, one at a time
My job as their Customer Engineer was to make the unfamiliar feel manageable. That meant drawing on past experience plus my knowledge of both the customer and Octopus, to work through every question they raised, however small: from how the database migration would work, to what moving the Octopus home directory actually involved, to whether existing Tentacles would need any changes, through to how authentication would behave once the HA instance was added to the other Active Directory domain.
Small, safe steps
Answering the customer’s questions built confidence, but confidence alone doesn’t get you through a migration. What actually got this customer moving was a draft plan that broke the upgrade down into small, ordered, and reversible stages, rather than one high-stakes cutover:
- Notify users of the upcoming upgrade well in advance.
- Select the version of Octopus Server they were going to upgrade to.
- Set up their load balancer to include two of the five HA nodes as UI-only nodes.
- Besides the Migration Plan, have a clearly defined Roll-back Plan, just in case.
- Build the five new HA node Virtual Machines, and add them to the other AD domain well ahead of time.
- On the day, put the existing Octopus server into maintenance mode and then let running tasks drain, so nothing is mid-flight when the migration starts.
- Back up the database and restore it into the new environment, rather than pointing the new environment straight at the live database - which also means rolling back was safe and easy if it came to it.
- Move shared storage across separately, and confirm it before moving on.
- Install Octopus Server on a single High Availability node first, then:
- Test authentication with a couple of local Octopus users before touching anything related to sign-in for their teams.
- Perform a pre-selected list of test deployments.
- Now install Octopus Server on the other HA nodes, pausing to check local Octopus login access and to do test deployments after each one.
- Once the basics are proven to work, only then enable Active Directory.
- Last of all, point DNS at the load balancer’s IP address for the HA cluster once everything else is verified. That way, nobody can accidentally start using the new Octopus Deploy instance before it is actually ready.
Each stage had a clear checkpoint before moving to the next. If something looked wrong at any stage then they could pause or roll back without having already committed to the whole change. For an Infrastructure team whose instinct was to slow down and check everything, that mattered more than any amount of reassurance could on its own.
The outcome
The customer completed their move to an HA cluster, and it gave them a foundation to build on. Instead of staying several versions behind, they settled into a regular 6-monthly upgrade cadence, staying close enough to current that they didn’t feel like they were on the bleeding edge. Years later, they’re still on that cadence, still running a large and complex Octopus environment, and rarely need to raise a support ticket, which is its own kind of compliment. As their team lead said:
“…we could not have done our Octopus instance migration without you; and we don’t know of any company that would provide the level of services you have - your help saved us a lot of money!”
What this means for other cautious teams
If you’re a customer who’s stuck on an old version because the upgrade feels too big to risk, the fix is to get solid answers to your questions that are grounded in your needs and clear in what is required of you. So answer the difficult questions and break the upgrade down into stages small enough that each one is genuinely low-risk on its own. After all, hesitation is often just a request for a plan that hasn’t been written yet.
So if you’re planning a similar move, start with our Designing Octopus for High Availability and our Upgrade guides; and if you need more help, then by all means get in touch with your account team here at Octopus.
Happy deployments!