Automating Octopus with Azure Functions

With Microsoft and Amazon releasing Azure Functions and Lambda respectively, the effort and cost barriers to hosting small services and scripts have been lowered. These services can be combined with the Octopus REST API and the Subscriptions Feature to provide automation above what Octopus provides out of the box.

This post explores two such integrations. The first example escalates unclaimed manual interventions and the second responds to machines going offline. Both examples use the Octopus.Client library with C# Azure Functions. The Twilio SMS integration is used as well, but this can be replaced with your notification method of choice, or just a log message.

Example 1 - Unclaimed Interventions

This example queries the Octopus server on a regular basis and finds any deployments that have pending manual interventions. If at the time of the check, those manual interventions have not been assigned to anyone and it has been over one minute, an SMS is sent. Guided failures are also included as they are a special case of manual intervention.

Function Setup

After creating a new Function app using your Azure account, add a new Function and select the TimerTrigger-CSharp template. Give it a name, and keep the default schedule.

To reference the Octopus.Client nuget package, select View Files (see above sceenshot) and add a file named project.json with the following contents. For more details on this see Azure Functions package management.

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "Octopus.Client": "4.5.0"
      }
    }
   }
}

Next setup the Octopus public URL and API key. Do this by selecting the Function app settings item (see screenshot) and then select Configure app settings. Add the two app settings, OctopusUrl and OctopusApiKey.

Finally, setup the Twilo integration. This is optional, so if it is skipped, remove the references to Twilo and SMSMessage from the code. Select the the function again, and then select Integrate. Create a new Output, selecting Twilio SMS. Enter your Twilo details. Note that the SID and Token are references to App Settings, so you will need to go there to input the actual SID and App Setting.

The Function

Replace the body of the function (run.csx) with the following:

#r "Twilio.Api"

using System.Net;
using Octopus.Client;
using Octopus.Client.Model;
using Twilio;

public static async Task Run(TimerInfo myTimer, TraceWriter log, IAsyncCollector<SMSMessage> message)
{
    var endpoint = new OctopusServerEndpoint(GetSetting("OctopusUrl"), GetSetting("OctopusApiKey"));
    var client = await OctopusAsyncClient.Create(endpoint);

    var tasks = await client.Repository.Tasks.GetAllActive();
    foreach (var task in tasks.Where(t => t.HasPendingInterruptions))
    {
        var interruptions = await client.Repository.Interruptions.List(pendingOnly: true, regardingDocumentId: task.Id);
        var unhandled = from i in interruptions.Items
                        where i.IsPending && i.ResponsibleUserId == null
                        select (TimeSpan?) DateTimeOffset.Now.Subtract(i.Created);
        var oldest = unhandled.Max();
        if (oldest.HasValue && oldest > TimeSpan.FromMinutes(1))
        {
            var sms = new SMSMessage();
            sms.Body = $"The task {task.Description} has not been actioned after {oldest.Value.TotalMinutes:n0} minutes";
            log.Info(sms.Body);
            await message.AddAsync(sms);
        }
    }  
}

public static string GetSetting(string name) =>  System.Environment.GetEnvironmentVariable(name, EnvironmentVariableTarget.Process);

Click the Save and run button and check the log output, it should run successfully.

Testing

To test the integration, create a new project (or use an existing one) with a manual intervention step and deploy that. Alternatively cause a deployment to pause with a guided failure. Wait for a minute and then run the function again manually (or wait for the function to run on the timer).

Example 2 - Offline Machines

This example is triggered by a subscription when a machine transitions from available to unavailable as a result of a health check or deployment. It then determines whether to delete the machine or take some remedial action. If the machine cannot be recovered an SMS is sent. For brevity the implementation below always determines the machine should not be deleted and the remedial action is to do nothing.

Setup

Add a new function based on the HttpTrigger-CSharp template (keep the Authorisation Level as Function) and follow the instructions above to setup Octopus.Client and Twilio.

Next add a subscription in your Octopus instance, with an Event filter of Machine found to be unavailable and a payload URL of your Azure Function.

Testing

Since this function responds to an external event and payload, to speed up development and testing of the function, the payload can be captured and then replayed using the built in testing feature. To get started, replace the body of the function (run.csx) with the following and save it:

#r "Twilio.Api"

using System.Net;
using Twilio;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log, IAsyncCollector<SMSMessage> message)
{
    dynamic data = await req.Content.ReadAsAsync<object>();
    log.Info(data.ToString());
    return req.CreateResponse(HttpStatusCode.OK);
}

Next stop one of the server's tentacles and run a health check for that tentacle. This check should fail.

Look at the log output of your function, you should see the Octopus request payload. If you do not, check the server log for warnings.

Copy the request payload from the log output and select the Test menu item (next to View Files). Paste payload into the Request body section. Now click the Run button again, and the payload will print to the output again.

The Function

For brevity this function has two placeholder methods HasBeenDecommissioned and AttemptToBringOnline. In a full implementation, these methods would interact with external systems such as the Azure management API or some infrastructure monitoring software.

Replace the body of the function (run.csx) with the following:

#r "Twilio.Api"

using System.Net;
using Octopus.Client;
using Octopus.Client.Model;
using Twilio;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log, IAsyncCollector<SMSMessage> message)
{
    var endpoint = new OctopusServerEndpoint(GetSetting("OctopusUrl"), GetSetting("OctopusApiKey"));
    var client = await OctopusAsyncClient.Create(endpoint);

    var machineIds = await GetAffectedMachineIds(req);
    log.Info($"Machines {string.Join(", ", machineIds)} have gone offline");

    var failedMachines = new List<MachineResource>();
    foreach(var id in machineIds)
    {
        var machine = await client.Repository.Machines.Get(id);
        if(HasBeenDecommissioned(machine))
        {
            log.Info($"Machine {machine.Id} is no longer required, removing");
            await client.Repository.Machines.Delete(machine);
        }
        else
        {
            log.Info($"Machine {machine.Id} should not have gone offline");
            failedMachines.Add(machine);
        }
    }

    await SendOutageSms(failedMachines.Count, message);
    await AttemptToBringOnline(failedMachines);
    var task = await client.Repository.Tasks.ExecuteHealthCheck();

    return req.CreateResponse(HttpStatusCode.OK);
}

public static string GetSetting(string name) =>  System.Environment.GetEnvironmentVariable(name, EnvironmentVariableTarget.Process);

static async Task<string[]> GetAffectedMachineIds(HttpRequestMessage req)
{
    dynamic data = await req.Content.ReadAsAsync<object>();
    var ids = (string[]) data.Payload.Event.RelatedDocumentIds.ToObject<string[]>();
    return ids.Where(i => i.StartsWith("Machines-")).ToArray();
}

// Call out to cloud provider or monitoring software
static bool HasBeenDecommissioned(MachineResource machine) => false;

static Task SendOutageSms(int count, IAsyncCollector<SMSMessage> message)
{
    var sms = new SMSMessage();
    sms.Body = $"{count} machines has become unavailable";
    return message.AddAsync(sms);
}

// Take some remedial action
static Task AttemptToBringOnline(List<MachineResource> machines) => Task.CompletedTask;

Click the Save and run button and check the log output, it should report that a machine has gone offline.

More Information

See the Coordinating Multiple Projects documentation page outlines some further scenarios along with some sample code.

For more code samples, see the OctopusDeploy-Api GitHub repository.

Finally, LinqPad is a great tool for editing and testing parts of functions.