Automation's importance to Continuous Delivery

From the start, Continuous Delivery called for the automation of the whole deployment pipeline. Automation is an obvious choice for some stages, such as the build process, but do you really have to automate everything?

Barriers to automation

When you first investigate Continuous Delivery or DevOps, you might find some teams looking for ways to avoid automation. A team might take shortcuts, like quick, hot-copy deployments to pre-live environments. Another might work outside the deployment pipeline, like running quarterly performance tests instead of running them for each change.

When you find a fault in your quarterly performance test, you may need to review hundreds of changes that could have caused the drop in performance. When you run it as part of your deployment pipeline, you know exactly where to look as there’s only one change. Whether it’s performance, security, or some other quality attribute you’re testing, it’s easier when you know there’s a problem straight away.

Another technique some teams use is reducing how often they deploy. If it takes the team 8 hours to deploy to production each week, you can reduce the ratio of time spent on deployments by making them monthly. Mathematically, you move from 1/5th of the team’s time spent on deployments to 1/20th. However, this logic breaks when it makes contact with reality because when you deploy less often:

1. The batch size increases
2. The manual stages in your deployment pipeline get more expensive
3. Testers will find more faults during testing

You soon wonder if you should move from quarterly to annual production deployments!

The critical problem with this negative spiral is the longer you wait to deploy, the larger the batch of changes becomes. These large batches cause problems during integration, testing, and deployments that are feeding the team’s decision to deploy less often.

The alternative to this bad feedback loop is to reduce batch sizes and focus on lowering the transaction cost of your deployment pipeline. The best way to make your deployment pipeline cheaper and more reliable is automation.

Instead of trending towards an infinitely large batch that is never deployed, trend toward deploying every change immediately.

What you can automate

Most teams have an automated build process they can run to create a software version. There are plenty of other stages to fully or partially automate.

Here’s a list of some typical stages that you might be doing manually that are prime candidates for automation:

• Triggering a build when code changes
• Building and packaging the software
• Creating a release note that describes changes
• Creating the infrastructure for an environment
• Deploying the software to an environment
• Running regression tests
• Speed and load tests
• Security scanning
• Database backups
• Database upgrades
• New software version notifications

The stages of your deployment pipeline must also become continuous if you want to deliver software continuously. There are practices named after this need, like Continuous Integration and continuous testing. There are tools that make most deployment pipeline automation easy to do.

Where automation is more challenging, it often due to a complex manual process. Once you automate the easier stages, you increase your skills and confidence to tackle the more difficult ones. If you have a list of tasks you can’t automate when you start, it will likely shrink over time. The number of stages that people must do is much smaller than you first think.

What you believe is possible will grow as you learn new skills.

Reasons to automate your deployment pipeline

The positive spiral at Continuous Delivery is the relationship between batch size and deployment frequency. However, there are other reasons to automate as much as possible.

Automation brings people together

Your organization might have several teams involved in the deployment pipeline. After developers make changes, different teams test, deploy, and run the software in production. The deployment pipeline captures the interactions between these roles.

You can use the process of mapping and automating the deployment pipeline to find and resolve conflicts without a heavyweight process. For example, you can use automation to define how you deploy a software version rather than writing a detailed deployment process document to pass to another team.

Automation can prevent miscommunication and reduce friction between roles.

Repetitive work is punishment

People don’t enjoy repeating the same task over and over. That’s why repetitive tasks were traditionally used as a form of punishment (19th-century prisons used treadmills and crank machines).

You might have seen this first-hand if your teacher ever forced you to write lines at school (where you have to write the same sentence many times, usually related to your perceived offense). Your first line might be neat and error-free, but it doesn’t take long for the quality to drop and for mistakes to happen.

How much easier would it have been to automate the lines? The script below reliably writes the lines, and each one is the same high-quality as the first:

for ($i = 0; $i -lt 100; $i++) {
    Write-Host 'I must not write on the table.'
}

Computers are experts at repetitive tasks. After you automate a stage in your deployment pipeline, it’s done the same way, in the same order, with nothing missed - every time.

Automation is faster

It won’t surprise you to hear that automated deployments are faster than manual ones. No matter what steps you need in your deployment pipeline, the process will be faster the more you automate.

There are arguments against rushing software delivery’s design and code phases, as it can reduce software quality. However, the reverse is true for the deployment pipeline. Once you integrate code into the main branch in version control, you should aim to get fast feedback about the new software version. The sooner you find a problem, the less damage it can do.

You can get feedback sooner with an automated deployment pipeline. This isn’t limited to the detection of functional bugs. By including tests for the quality attributes of your software, you can find out as soon as a change impacts system performance, resource usage, or fails a security check.

You can also get feedback from your users earlier, which you can use to improve features while you still have a budget to refine the idea. Rather than bet your whole budget on a single attempt to deliver a feature, you make incremental investments that let you learn from actual feature use.

You can re-use automation

Automation is reusable. After you create an automated deployment pipeline, you can use many of the tools, scripts, and skills to build the next one. This is true even when you build a deployment pipeline for software written in different languages or targeting different hosting environments.

The novelty of finding ways to automate tasks reduces as the organization builds a toolbox of automation solutions.

Developers often consider reuse as part of their role. Applying reuse to a manual process document is challenging, but for automation, it’s a valuable skill.

Automation can go bigger

When you automate your whole deployment pipeline, you can create a new environment at the push of a button. You can now solve problems that previously seemed inherent in software delivery:

• Test environments: Teams often compete for a limited number of test environments. They either wait for their turn or ruin shared test data. You can now generate temporary test environments on demand.
• Incident response: When an incident occurs or a high-priority issue is reported, you can quickly create an environment with the appropriate software version to reproduce the problem.
• Disaster recovery: When you have a critical infrastructure issue, you can create a new production environment using the deployment pipeline and restore the database backups.

When you manually perform testing, deployment, or operations tasks, there are low limits on how far you can scale. For example, if you had to manually update the database connection string for all instances of a production system, this would be easy on one server, more difficult on 10, and problematic on 100.

Automation can perform tasks on a vast scale.

Self-funding

You need to make an up-front investment in automation that provides quick returns. If you automate a task that takes an hour a day to perform manually, you now have 4 hours each week to invest in further automation. You also save time previously spent on resolving issues caused by mistakes in the manual process.

Very quickly, you save more hours each week than you need to reinvest in automation. You can then spend more time on feature development. This will improve your relationship with non-technical people in your organization.

Focus on simpler automation for more frequent tasks to get the best return on your initial investment. Then use what you learn to tackle more challenging tasks.

Delaying automation

You will come across some areas that could be automated but either:

1. Have a high cost relative to the benefit
2. Are technically impossible at the time

You need to delay automation in these areas until you can change the cost and benefit profile, or get a toolset that changes what’s possible.

Imagine you have to test a website to ensure all the pages match the brand style guide. To do this manually, you select 10 pages at random and review the styles for any colors, fonts, sizes, or geometry that fails to meet the appropriate styles.

You could automate some of this by creating a web crawler that checks each element on the page is on-brand. It could raise errors for many common brand issues:

• Incorrect foreground or background color
• The wrong font
• Headings that are the wrong size
• Body copy in the wrong style
• Insufficient padding around the company logo

You would spend a great deal of time and money creating and maintaining this test suite. You won’t remove significant manual effort as many of the tests are sub-second decisions by an experienced tester, who would still need to look at pages to check for items you can’t automate.

However, this is not an uncommon problem for organizations to solve, so there’s every chance a specialist tool supplier could create a product in this space. This could increase the benefits of automating the checks and might be cheaper than a home-grown tool. There are already tools that assist manual visual testing by highlighting areas on pages that visually change.

Signs your process should be automated

Your existing manual process may signal it’s ready to be automated. The instructions you created for manual work might include a checklist that describes what the automation would do. The only part you are missing is the automation.

For example, a checklist for a deployment might include a series of steps and instructions:

1. Build the application
2. Package the application
3. Copy the package to the server
4. Stop the existing application
5. Copy the current files from d:\application\ to d:\backup\[date]
6. Copy the files from the new package to d:\application\
7. Restart the application
8. Check the application is working

With several open file explorer windows, you’ll likely copy a file to the wrong place. If you accidentally deploy the new software version to the backup folder, you could waste time investigating why your new feature isn’t visible.

Instead, use the checklist to begin the automation process by writing a script to handle each step or using a more advanced toolset to get additional benefits like pre-built steps, approvals, and audit trails.

Automation ideas

Widely automated deployment pipeline stages include:

• Builds
• Tests
• Packaging
• Database deployments
• Software deployments
• Infrastructure
• Operations runbooks
• Monitoring and alerting
• Log management

You may have more stages in your deployment pipeline. If you have compliance or regulatory requirements that need extra steps, you can also automate those. Some stages may become redundant, like approvals added to the process following past manual deployment errors.

Summary

You need to apply automation aggressively to your deployment pipeline to get the benefits of Continuous Delivery. You should automate as many stages and approvals as possible and appropriate to your organization.

Use the automation process to discover and improve your deployment pipeline and build trust. When you achieve high levels of trust, some stages in your deployment pipeline may become redundant. Remove steps you don’t need as part of your continuous improvement efforts.

Further reading

These resources will help you find out more about Continuous Delivery:

• Continuous Delivery by Jez Humble and Dave Farley (2011)
• Continuous Delivery Pipelines by Dave Farley (2021)
• An overview of Continuous Delivery on Octopus DevOps
• Continuous Delivery principles on Octopus DevOps
• Start using Continuous Delivery on Octopus DevOps
• Our white paper on the importance of Continuous Delivery

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback