Upload to AWS S3 template

Octopus supports the uploading an entire package or the files contained within a package through the Upload a package to an AWS S3 bucket step. This step uploads the package or the file(s) contained within the package using the AWS managed by Octopus.

The following instructions can be used to configure the Upload a package to an AWS S3 bucket step.

Add the upload a package to an AWS S3 bucket step

Add the Upload a package to an AWS S3 bucket step to the project, and give it a name.

AWS section

Select the variable that references the Amazon Web Services Account under the AWS Account section or select whether you wish to execute using the service role of an EC2 instance. If you don’t have an AWS Account Variable yet, check our documentation on how to create one.

The supplied account can optionally be used to assume a different AWS service role. This can be used to run the AWS commands with a role that limits the services that can be affected.

Package section

Under the Package section, define how the target package and the associated file uploads should behave for the step. The entire package can be uploaded or individual file(s) from the package can be specified for upload.

Entire package

By default, the entire package will be uploaded to the S3 bucket untouched with the given bucket key, metadata, and tags.

Variable Substitution File Patterns

A newline-separated list of file names to transform, relative to the package contents. Extended wildcard syntax is supported. E.g., Notes.txt, Config\*.json, **\specific-folder\*.config. This field supports extended template syntax. Conditional if and unless:

#{if MyVar}...#{/if}

Iteration over variable sets or comma-separated values with each:

#{each mv in MyVar}...#{mv}...#{/each}

Structured Variable File Patterns

A list of files to perform structured variable substitution on.

Target files need to be newline-separated, relative to the package contents. Extended wildcard syntax is supported. E.g., appsettings.json, Config\*.xml, **\specific-folder\*.yaml. Learn more about the Structured Configuration Variables feature and view Structured Variables examples.

Use filename with embedded content hash

Select this option to allow the hash of the contents of the package to be included in the resulting bucket key.

The hash should appear before the extension in the format of filename@hash.extension. The hash value is based on the contents of the zip package and is calculated after any variable substitutions/replacements.

Additionally, the hash value is available as a variable named Octopus.Action.Package.PackageContentHash to be used as a custom key. Note that this variable can only be used in this step.

Individual files from the package

If you have chosen to upload individual files from the package, you will be presented with an additional Files Section where you can add one or more file selections where each selection can be for a single file or for multiple files depending on your use case.

Adding and removing file selections

A new file selection can be added by clicking on the Add Another File Selection button located under the Files Section.

A file selection can be removed by expanding the appropriate selection and clicking on the Remove File Selection button.

Bucket key prefix

A prefix delimited with forward slash / characters is seen in S3 as folders. Here are a few examples to help you get those files in the right folder structure:

Single file selection

The single file selection lets you upload a single file to an S3 bucket which must exist within the package. If the file is not found an associated error will be raised. This selection also allows for the bucket key to be explicit.

Bucket keys

A custom key delimited with forward slash / characters is seen in S3 as folders. Here are a few examples to help you get those files in the right structure:

Multiple file selections

Multiple file selections allow one or more files from the package to be uploaded to the S3 bucket using globbing patterns. The behavior is slightly different from single file selections as you have less control over the bucket key which will be used while no error will be raised if no files matched the globbing pattern. Files uploaded will have the bucket key prefix added to the file name. That is a file matched by the globbing pattern path/to/file/File.template with a prefix of templates/ will be uploaded with the bucket key templates/File.template.

Metadata and tags

Metadata and tags can be provided for the package, for file selections for the package, or for individual files. See the AWS documentation for more information regarding the usage of metadata.

Canned ACL

The canned ACL must be specified when uploading files as it dictates the permissions for a file within the S3 bucket. Please see the AWS documentation for information regarding Canned ACLs.

Storage class

The storage class for files specify the performance access requirements for a file. Please see the AWS documentation for more information regarding Storage classes.

Upload behavior

Uploads are skipped if the content hash is the same as an existing object in the target bucket. This is done to avoid unnecessary uploads and may require special care to be taken when using custom bucket keys.

Output variables

Octopus will create several output variables storing the properties of any uploaded files. The values can then be used in subsequent steps to access the uploaded files.

Here is the list of variables created after a single file has been uploaded:

Here is the list of variables created after multiple files have been uploaded:

Error messages

The AWS deployment steps include a number of unique error codes that may be displayed in the output if there was an error. Below is a list of the errors and any additional troubleshooting steps that can be taken to rectify them.

AWS-S3-ERROR-0001

The AWS account used to perform the operation does not have the required permissions to describe the CloudFormation stack. This means that the step is not able to generate any output variables.

An exception was thrown while contacting the AWS API.

This can happen when accessing AWS via a proxy, and the response from AWS indicated an error. The response body is printed to the logs in these cases.

An incorrect AWS region can result in this error. Ensure that the region matches one from the AWS documentation.

AWS-S3-ERROR-0002

The AWS account used to perform the operation does not have the required permissions to upload to the bucket.

An exception was thrown while contacting the AWS API.

This can happen when accessing AWS via a proxy, and the response from AWS indicated an error. The response body is printed to the logs in these cases.

An incorrect AWS region can result in this error. Ensure that the region matches one from the AWS documentation.

AWS-S3-ERROR-0003

An error occurred uploading a file to a bucket possibly due to metadata. Specified value has invalid HTTP header characters.

This can happen if the metadata key and or value has invalid characters. Ensure characters as per the AWS documentation is not used as part of metadata.

An unrecognized Forbidden error was thrown while uploading to bucket [bucket-name]

The AWS account or role used to perform the operation does not have the required permissions to create and/or upload to the bucket. This error can be caused by a user and/or bucket policy that is too restrictive. The following user and bucket policy examples contain the minimum required permissions for this step:

Example User Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAccountOperations",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        },
        {
            "Sid": "AllowBucketOperations",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:CreateBucket"
            ],
            "Resource": "arn:aws:s3:::my-bucket-name"
        },
        {
            "Sid": "AllowObjectOperations",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket-name/*"
            ]
        }
    ]
}

Example Bucket Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::my_bucket_name/*",
            "Condition": {
                "StringNotLike": {
                    "aws:userId": [
                        "aws-user-id:*",
                        "aws-user-id"
                    ]
                }
            }
        }
    ]
}

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback

Page updated on Sunday, January 1, 2023