Octopus and Elastic with OpenTelemetry

Octopus supports exporting audit logs to an OpenTelemetry provider, which can then be combined with the Elastic Distributions for OpenTelemetry Collector (EDOT Collector) to send audit logs to an Elastic stack.

In this post we’ll step through the process of setting up a test on-premises Elastic stack and configuring Octopus to send audit logs to it using OpenTelemetry.

Installing Elastic stack

The open source Elastic stack consists of Elasticsearch and Kibana. Instructions for running a test version of this stack locally are found in the GitHub repo. They involve running the following script to create a Docker Compose and supporting configuration files:

curl -fsSL https://elastic.co/start-local | sh

A number of randomly generated passwords are saved in the .env file. In particular, we will need the ES_LOCAL_PASSWORD and ES_LOCAL_API_KEY values, which will be used in subsequent configuration files:

START_LOCAL_VERSION=0.12.0
ES_LOCAL_VERSION=9.2.4
ES_LOCAL_CONTAINER_NAME=es-local-dev
ES_LOCAL_PASSWORD=password
ES_LOCAL_PORT=9200
ES_LOCAL_URL=http://localhost:${ES_LOCAL_PORT}
ES_LOCAL_DISK_SPACE_REQUIRED=1gb
ES_LOCAL_JAVA_OPTS="-Xms128m -Xmx2g"
KIBANA_LOCAL_CONTAINER_NAME=kibana-local-dev
KIBANA_LOCAL_SETTINGS_CONTAINER_NAME=kibana-local-settings
KIBANA_LOCAL_PORT=5601
KIBANA_LOCAL_PASSWORD=password
KIBANA_ENCRYPTION_KEY=encryption_key_goes_here
ES_LOCAL_API_KEY=api_key_goes_here

Start the Docker Compose stack with:

./start.sh

You can now open Kibana in your browser at http://localhost:5601 and log in with the username elastic and the password stored in the ES_LOCAL_PASSWORD environment variable.

Configuring the EDOT Collector

We need to create three files to support the EDOT Collector.

Create these files in a new directory to avoid any clashes with the files created by the Elastic stack from the previous step.

The first is the otel-collector-config.yml file, which contains the configuration for the EDOT Collector. You must replace the exporters.elasticsearch.password value with the password stored in the ES_LOCAL_PASSWORD environment variable:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

processors:
  batch:
    send_batch_size: 100
    timeout: 10s

exporters:
  debug:
    verbosity: detailed
  elasticsearch:
    endpoint: "http://localhost:9200"
    user: elastic
    # Replace this value with the ES_LOCAL_PASSWORD environment variable
    password: password

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticsearch]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticsearch]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticsearch]

The second file is .env, which contains environment variables for the EDOT Collector. You must replace the ELASTIC_API_KEY value with the API key stored in the Elastic stack ES_LOCAL_API_KEY environment variable:

HOST_FILESYSTEM=/
DOCKER_SOCK=/var/run/docker.sock
ELASTIC_AGENT_OTEL=true
COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:9.2.4
ELASTIC_API_KEY=api_key_goes_here
ELASTIC_ENDPOINT=localhost:9200
OTEL_COLLECTOR_CONFIG=otel-collector-config.yml

The third file is docker-compose.yml, which contains the Docker Compose configuration for the EDOT Collector:

services:
  otel-collector:
    image: ${COLLECTOR_CONTRIB_IMAGE}
    container_name: otel-collector
    deploy:
      resources:
        limits:
          memory: 1.5G
    restart: unless-stopped
    command: ["--config", "/etc/otelcol-config.yml" ]
    network_mode: host
    user: 0:0
    volumes:
      - ${HOST_FILESYSTEM}:/hostfs:ro
      - ${DOCKER_SOCK}:/var/run/docker.sock:ro
      - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml
    environment:
      - HOST_FILESYSTEM
      - ELASTIC_AGENT_OTEL
      - ELASTIC_API_KEY
      - ELASTIC_ENDPOINT
      - STORAGE_DIR=/usr/share/elastic-agent

Start the EDOT Collector with:

docker-compose up -d

Exposing OpenTelemetry endpoint publicly

The EDOT Collector is now running locally, but it only exposes the OpenTelemetry endpoint on localhost. This means Octopus cloud instances can not access it.

To expose the endpoint publicly, we can use ngrok. After installing ngrok, run the following command to expose the HTTP port used by OpenTelemetry:

ngrok http 4318

Configuring Octopus

Now that we have a publicly accessible OpenTelemetry endpoint, we can configure Octopus to send audit logs to it.

Configure the Octopus Audit Stream:

• To use the OpenTelemetry provider
• With the ngrok URL (which is typically a randomly generated value like https://5abbd123f26c.ngrok.app)
• Using the HTTP/protobuf protocol
• With no secrets or headers

Once these values are configured, Octopus will send audit events to the EDOT Collector, which will then forward them to the Elastic stack.

Conclusion

In this post we have seen how to configure Octopus to send audit logs to an Elastic stack using OpenTelemetry and the EDOT Collector. This setup provides a powerful way to analyze and visualize audit logs from Octopus, enabling better monitoring and security auditing capabilities.

Happy Deployments!