With some basic support for Docker Engine steps released in the Octopus Deploy 3.5.1 build, we have turned our attention to the next level of abstraction... Docker Compose
Having now started down the path to providing first-class support for Docker we still want your feedback for this next step (pun intended).
Naming is hard
Docker Compose allows you to manage multiple containers, networks and volumes together through a single docker-compose.yml configuration file. It uses a project name to isolate groups of resources to allow managing multiple environments within a single host and in this way you can have several Docker Compose apps running on a given host, isolated within their own network.
One idea we are playing with is namespacing these apps at deployment-time based on a combination of the Octopus Deploy project + step + environment. This would allow the deployment to start multiple Docker Compose apps across Octopus environments, but ensures that when a deployment takes place and we only want to update an existing service, then it would be only those containers that get updated and be restarted (as defined by changes to the configuration file such as an image version update). For example the next release may involve updating the version of the image used for our web container as well as scaling up the number of web containers in that app. Then ideally only the web containers in the relevant Docker Compose app should be affected by that deployment. Docker Compose gives us this ability to target the group of containers and update them in one hit, rather than multiple stop/run steps individually however some naming convention is needed to achieve that.
The flip side, is that this would slightly differ with the standard package extraction that you may be used to with Octopus Deploy, whereby multiple versions of the deployed project may exist at any one time in the same environment since they are just stored in different locations on the file system. Potentially providing a user-provided element to the identifier (such as the channel name) would allow similar behaviors if desired.
Configuration Source
As noted above, the required element of a Docker Compose step is the docker-compose.yml file that describes the app. In order to provide integration to Octopus Deploy package lifecycles, we expect that users will want to bind one or more images to a Docker image through the Octopus Deploy package selection process. When a release is created, all the package versions bound to services in that step will be then available to be selected by the user. The configuration would also be able to work with standard variable substitution to allow for environment specific configurations that resolve at deploy time. If variable substitution is supported do you think you would have the need to use multiple compose files?
Perhaps it would be preferable to extract the configuration file out of a versioned package itself (similar to how scripts sourced from packages currently work)? In this case you may need the option to provide the list of images that are intended to be selected at release creation time without having the list of services extracted for you automatically from the configuration.
Build at Deployment
By providing a mechanism to source the configuration from a separate package, then perhaps it may additionally provide the ability to provide Dockerfiles that can be used to build an image at deployment time as part of the docker-compose process. We have deliberately avoided providing a docker build step as part of our first 3.5.1 release since this feels like a process that should take place during the build phase of your CI pipeline. The single immutable build artifact, the image, should be created once and progress through the relevant environments. Do you feel like creating images would be useful in the deployment process, taking into consideration the philosophy of both Octopus Deploy and Docker?
Microsoft is also getting behind Docker and have announced first-class support in Visual Studio 2017 for deploying directly to containers (so only a little behind us and our first Docker integration ;) ) ...Docker may soon play a more important part of your CI/CD pipeline.
Request For Comment
Hopefully you have the opportunity to try out the new Docker steps available in the 3.5.1 to get a feel for where we are taking these features in Octopus Deploy. Ultimately this work is being done to make your lives easier when it comes to deploying your applications so it's your opinion that matter most.
- What do you think about namespacing the Docker Compose groups to provide easy modification at upgrade time?
- What would your preferred mechanism be to providing the configuration file, inline or package sourced?
- Do you feel that providing image build capability makes sense?
- What else have we completely missed?
If you have any other thoughts or opinions on how Docker Compose should look, add your comments below or comment on and follow the open GitHub ticket #2875.
