Structured configuration variables

With the Structured Configuration Variables feature you can define variables in Octopus for use in JSON, YAML, XML, and Properties configuration files of your applications. This lets you define different values based on the scope of the deployment. Settings are located using a structure-matching syntax, so you can update values nested inside structures such as JSON objects and arrays, YAML mappings and sequences, and XML elements and attributes. XPath is used for XML files, and similar expressions are used for the other formats.

Configuring the structured configuration variables feature

1. To enable Structured Configuration Variables on a step that supports the feature, click the CONFIGURE FEATURES link, select Structured Configuration Variables, then click OK.
2. In the Structured Configuration Variables section of the step, specify the relative paths to your structured configuration files, relative to the working directory. For instance:

approot\packages\ASPNET.Core.Sample\1.0.0\root\appSettings.json

or

**/application.yaml

Octopus will find the target files, match structures described by the names of Octopus variables, and replace their contents with the values of the variables.

Selecting target files

Specify files that should have variable replacement applied to them. Multiple files can be supplied by separating them with a new line.

You can supply full paths to files, use wildcards to find multiple files in a directory, or use wildcards for a directory to find all files at that level or deeper:

Specific file path

ExampleProject\appSettings.json

Match any .yaml files in the root directory

*.yaml

Match any .json files in the specified directory

Config\*.json

Match any .xml files in the specified directory or deeper

Application/**/*.xml

The Target File field also supports Variable Substitution Syntax, to allow things like referencing environment-specific files, or conditionally including them based on scoped variables. Extended template syntax allows conditionals and loops to be used.

How the file type for target files is determined

Structured Configuration Variables allows for replacement in JSON, YAML, XML, and Properties files. To determine what file type is being used, Octopus will first try and parse the file as JSON, and if it succeeds, it will treat the file as JSON. This is to ensure backwards compatibility, because this feature previously only supported JSON files.

If the file doesn’t parse as JSON, Octopus refers to its file extension. If it is yaml or yml, the file will be parsed as YAML, if the extension is xml, the file will be parsed as XML, and finally if the extension is properties the file will be parsed as a Java Properties format.

If the file extension is not recognized (for example, a file with a config file extension), Octopus will try to parse the files using each of the supported formats until a matching format is found.

Variable replacement

Octopus uses variable names to identify the structures that should be replaced within the target files. If a structure within a target file has a hierarchical location that matches a variable name, its content will be replaced with the variable’s value. The hierarchical location is identified differently depending on the type of target file:

• In JSON and YAML files, each location is identified by the sequence of keys leading to it from the root level, separated by :.
• In XML files, structures can be identified by setting Octopus variable names to XPath expressions.
• In Java Properties files, they have their keys matched against Octopus variable names.

An example for each supported file type can be found in the following table:

Variable names starting with the word Octopus

When targeting JSON and YAML files, care should be taken when naming variables to be used with the Structured Configuration Variables feature; Specifically, to avoid the use of the word Octopus in the name where possible. This is because Octopus provides a number of system variables that start with the word Octopus that aren’t intended for use with this feature.

Consider the following JSON input file:

{
  "OctopusServer": {
    "WebPort": "80"
  }
}

If you had a variable named OctopusServer:WebPort with value 8080, the value would not be replaced as the variable name starts with the word Octopus.

The easiest way to workaround this is to change the name of your variable to start with something other than the word Octopus.

Variable casing

Octopus matches variable names to the structure in target files in a case insensitive way.

For example, given the following JSON input file:

{
  "app": {
    "port": "80"
  }
}

If you had a variable named APP:PORT with value 8080, the value would be replaced despite the name of the variable being in upper case. The output file would become:

{
  "app": {
    "port": "8080"
  }
}

For more information, refer to our variable casing documentation.

JSON and YAML

Simple variables

Given this example of a target config file:

{
   "weatherApiUrl": "dev.weather.com",
   "weatherApiKey": "DEV1234567",
   "tempImageFolder": "C:\temp\img",
   "port": 8080,
   "debug": true
}

If you define variables in your Octopus project called weatherApiUrl, weatherApiKey, port, and debug with the values test.weather.com, TEST7654321, 80, and false, the target config file is updated to become:

{
   "weatherApiUrl": "test.weather.com",
   "weatherApiKey": "TEST7654321",
   "tempImageFolder": "C:\temp\img",
   "port": 80,
   "debug": false
}

Note that the tempImageFolder setting remains untouched, and the types of port and debug have not been changed. Octopus will attempt to keep the original type if the new value matches the type of the old value.

Hierarchical variables

It is common (and encouraged) to use hierarchical variables in Structured configuration files. This is supported in Octopus variables by using a nested path syntax delimited by colon characters.

For example, to update the value of weatherApi.url and weatherApi.key in the target config file you would configure the Octopus variables weatherApi:url and weatherApi:key.

Hierarchical JSON

{
   "weatherApi": {
      "url": "dev.weather.com",
      "key": "DEV1234567"
   }
}

Hierarchical YAML

weatherApi:
  url: dev.weather.com
  key: DEV1234567

You can also replace an entire object. For the example above, you could set Octopus variable weatherApi to a value of {"url":"test.weather.com","key":"TEST7654321"}, which will result in this:

Replaced Hierarchical JSON

{
   "weatherApi": {
      "url": "test.weather.com",
      "key": "TEST7654321"
   }
}

Replaced Hierarchical YAML

weatherApi:
  url: test.weather.com
  key: TEST7654321

JSON Array or YAML Sequence variables

Octopus can replace a value in a JSON array or a YAML sequence by using the zero-based index of the array or sequence in the variable name. If we take the following examples:

Example Hierarchical JSON

{
   "foo": {
      "bar": [
         "item1",
         "item2"
     ]
   }
}

Example Hierarchical YAML

foo:
  bar:
    - item1
    - item2

Variables can be set for foo:bar:1 with a value qux which will update the value of the second element in the array or sequence to be qux, like so:

Replaced Array Index Hierarchical JSON

{
   "foo": {
      "bar": [
         "item1",
         "qux"
     ]
   }
}

Replaced Sequence Index Hierarchical YAML

foo:
  bar:
    - item1
    - qux

It’s possible to replace an entire array or sequence too. In the previous example, if the Octopus variable foo:bar was set to ["baz","qux"], it would create outputs like:

Replaced Array Hierarchical JSON

{
   "foo": {
      "bar": [
         "baz",
         "qux"
     ]
   }
}

Replaced Sequence Hierarchical YAML

foo:
  bar:
    - baz
    - qux

The properties of objects in arrays can be replaced. In the example below, defining an Octopus variable foo:bar:0:url with the value of test.weather.com replaces the url property of the first object in the array:

Replaced Object Property in Array Hierarchical JSON

{
   "foo": {
      "bar": [
         {
            "url": "test.weather.com",
            "key": "DEV1234567"
         }
     ]
   }
}

Replaced Map Property in Sequence Hierarchical YAML

foo:
  bar:
    -
      url: test.weather.com
      key: DEV1234567

XML

For XML files, the values to replace are located using the standard XPath syntax. Octopus supports both XPath 1 and XPath 2.

Octopus variables with names that are valid XPath expressions are matched against the target XML files. For example, if you have a variable called //environment with the value production, it will replace the contents of all <environment> elements with production.

Replacing content

When replacing content, the replacement can only be as rich as what was originally there. If you select an element that contains only text, the replacement will be treated as text and structure-defining characters will be encoded as entity references. However, if you select an element that contains further element structures, the replacement is treated as an XML fragment, and structure-defining characters will be added as is.

This means that if you replace a password or connection string, any characters like &, < and > will be safely encoded within the string. For example, assume the target file contains the following:

<connectionString>Server=.;Database=db;User Id=admin;Password=password;</connectionString>

If you define a variable called //connectionString with the value Server=.;Database=db;User Id=admin&boss;Password=Pass<word>1; the structure will be updated as follows:

<connectionString>Server=.;Database=db;User Id=admin&amp;boss;Password=Pass&lt;word&gt;1;</connectionString>

It’s worth noting that an empty element, such as <rules />, contains no element structures and will only be filled with text. For example, assume the target file contains the following:

Empty XML Element

<configuration>
   <logging>
      <rules />
   </logging>
</configuration>

If the Octopus variable /configuration/logging/rules is specified with the value <rule level="trace" />, the value will be encoded as text, becoming:

Empty XML Element Filled

<configuration>
  <logging>
    <rules>&lt;rule level='trace' /&gt;</rules>
  </logging>
</configuration>

However, if the variable is named /configuration/logging to match the parent element, with the value <rules><rule level="trace" /></rules>, the value will be treated as an XML fragment because it is replacing an element structure (the <rules /> element). This becomes:

Empty XML Element Parent Replaced

<configuration>
  <logging>
    <rules>
      <rule level="trace" />
    </rules>
  </logging>
</configuration>

Replacing mixed content elements

Sometimes an element will contain a mixture of text and element structures. An example of this is:

<document>This is <b>mixed</b> content</document>

Because it contains an element structure, a replacement will be treated as an XML fragment. A variable named /document with the value of <logger /> would result in:

<document>
  <logger />
</document>

Another option is to match and replace individual text nodes. A variable named /document/child::text()[1] with the value just <text> would result in:

<document>just &lt;text&gt;<b>mixed</b> content</document>

Replacing attributes

Matching and replacing attribute values is supported with XPath. For example, assume the target file contains the following:

<configuration>
    <email role="admin">admin@example.com</email>
    <email role="user">user@example.com</email>
</configuration>

With the Octopus variable /configuration/email/@role with the value developer, the output will look like:

<configuration>
  <email role="developer">admin@example.com</email>
  <email role="developer">user@example.com</email>
</configuration>

Alternatively, to replace an element based on its attribute, you can apply the condition as a predicate. With a variable named /configuration/email[@role='admin'] with the value chief@example.org, the output will look like:

<configuration>
  <email role="admin">chief@example.org</email>
  <email role="user">user@example.com</email>
</configuration>

Similar to the examples above, you can also replace other attribute values. With a variable named /configuration/email[@role='admin']/@address with the value chief@example.org, the output will look like:

<configuration>
  <email role="admin" address="chief@example.org"></email>
  <email role="user" address="user@example.com"></email>
</configuration>

XML CDATA sections

CDATA sections can be replaced just like any other node by selecting them with the XPath. When the content of the CDATA section is replaced, the CDATA presentation is maintained in the output. In the following example, development in the CDATA tag can be replaced with prod<1> by having a variable /document/environment/text() with the value prod<1>:

XML Structure with CDATA

<document>
    <environment><![CDATA[development]]></environment>
</document>

XML Structure with CDATA Replaced

<document>
  <environment><![CDATA[prod<1>]]></environment>
</document>

Processing instructions

Processing instructions can be replaced using the XPath processing instruction selector like so: /document/processing-instruction('xml-stylesheet'). When replacing a processing instruction, it’s not possible to replace the individual attributes. The whole processing instruction gets replaced with the supplied value. Take the following example:

XML Structure Processing Instruction

<document>
   <?xml-stylesheet type="text/xsl" href="/Content/Glossary/main.xsl"?>
</document>

When the Octopus variable /document/processing-instruction('xml-stylesheet') is set to new value the output will be the following:

XML Structure Processing Instruction Replaced

<document>
   <?xml-stylesheet new-value ?>
</document>

Namespaces

When parsing the XML document, Octopus collects all namespace declarations for use in XPath expressions, so you can use any of the declared prefixes.

One limitation is that if the same prefix is declared more than once in a document, only the first will be available in XPath expressions. Because this is a potentially surprising situation, a warning will be logged, similar to the following:

The namespace 'http://octopus.com' could not be mapped to the 'octopus' prefix, as another namespace 'http://octopus.com/xml' is already mapped to that prefix. XPath selectors using this prefix may not return the expected nodes. You can avoid this by ensuring all namespaces in your document have unique prefixes.

Root elements with namespaces If you have xml files that have a namespace on the root element, you might find your XPath expression doesn’t match the root node. XPath provides different ways to select an element. One option to try is using a wildcard namespace in your XPath expression like /*:rootelement/*:childelement

Given the following xml:

<server xmlns="urn:my:domain:1.0">
  <properties>
    <property name="host.name" value="localhost" />
  </properties>
</server>

If you wanted to replace the value localhost, you could use the XPath expression of: /*:server/*:properties/*:property[@name='host.name']/@value

Java properties

Given this example of a target properties file:

weatherApiUrl = dev.weather.com
weatherApiKey = DEV1234567
tempImageFolder = C:\\temp\\img
logsFolder = C:\\logs
port = 8080
debug = true

If you define variables in your Octopus project called weatherApiUrl, weatherApiKey, tempImageFolder, port, and debug with the values test.weather.com, TEST7654321, D:\temp\img, 80, and false, the target properties file is updated to become:

weatherApiUrl = test.weather.com
weatherApiKey = TEST7654321
tempImageFolder = D:\\temp\\img
logsFolder = C:\\logs
port = 80
debug = false

Note that the logsFolder setting remains untouched as there was no variable defined to override the value and that tempImageFolder has been encoded with the double \. Octopus will encode the variable in the correct encoding for the properties file format.

Unlike JSON, YAML, and XML, it’s not possible to do hierarchical replacement in a properties file as properties files are simple key value files.

Help us continuously improve

Please let us know if you have any feedback about this page.

Send feedback

Page updated on Sunday, January 1, 2023