Reach Slack, Teams and PagerDuty without something in between

CHANGELOG.md

@@ -37,6 +37,14 @@ its code behaves exactly as it did.
   that only cared about a component going down reached into `OldState`/`NewState.LastResult?.Health`
   and worked it out again. The alerting and uptime packages were doing that identically; both now
   read `PreviousHealth` and `CurrentHealth` from the event instead of digging for them.
+- **Slack, Microsoft Teams and PagerDuty alert sinks**, in `Healthie.NET.Alerting` beside the
+  webhook. Each of those three rejects arbitrary JSON and wants its own shape, so the generic
+  webhook never actually reached them without something in between to reshape it -- which its own
+  remarks admitted, in the phrase "Teams through a Power Automate flow". They need no dependency the
+  package did not already have, so they are sinks rather than a package each. Teams targets the
+  Workflows URL and an Adaptive Card, because the Office 365 connectors and the `MessageCard`
+  payload they took are retired. PagerDuty resolves the incident it opened rather than raising a
+  second one: `Alert.DeduplicationKey` and `Alert.IsRecovery` already existed for exactly that.
 - **Schedules.** `PulseSchedule` says either "every this long" or "on this cron expression", and
   sits alongside `PulseInterval` rather than replacing it. The enum stopped at five minutes, which
   is short of what a certificate-expiry or disk-space check wants. Cron is standard Unix syntax and

README.md

@@ -824,12 +824,11 @@ Upgrading from v1.x? See the [v1 to v2 migration guide](https://github.com/ivanv
 Shipped since 3.1.4: alerting on transitions, OpenTelemetry metrics and traces, arbitrary intervals
 and cron, PostgreSQL / SQL Server / SQLite state providers, Hangfire / Coravel / Temporal
 scheduling, ready-made checkers, uptime reporting, leader election, optimistic concurrency
-on `IStateProvider`, and `HealthChanged` on the state-changed event. What is left:
+on `IStateProvider`, `HealthChanged` on the state-changed event, and Slack / Teams /
+PagerDuty alert sinks. What is left:
 
 - **A Redis state provider** -- the fastest option for state written on every tick, and a natural
   lease store for leader election.
-- **Alert sinks beyond the webhook** -- Slack, Teams and PagerDuty as packages rather than as a
-  documented payload.
 
 ---
 

src/Healthie.Alerting/Healthie.Alerting.csproj

@@ -2,8 +2,8 @@
 
   <PropertyGroup>
     <PackageId>Healthie.NET.Alerting</PackageId>
-    <Description>Alerting for Healthie.NET: turns health changes into alerts and delivers them to a webhook or your own sink, with deduplication and delivery failures isolated from the checks.</Description>
-    <PackageTags>$(HealthieCommonTags);alerting;notifications;webhook;incidents</PackageTags>
+    <Description>Alerting for Healthie.NET: turns health changes into alerts and delivers them to Slack, Microsoft Teams, PagerDuty, a webhook or your own sink, with deduplication and delivery failures isolated from the checks.</Description>
+    <PackageTags>$(HealthieCommonTags);alerting;notifications;webhook;slack;teams;pagerduty;incidents</PackageTags>
   </PropertyGroup>
 
   <ItemGroup>

src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs

@@ -0,0 +1,202 @@
+using Healthie.Abstractions.Enums;
+using System.Net.Http.Json;
+using System.Text.Json.Serialization;
+
+namespace Healthie.Alerting;
+
+/// <summary>
+/// Posts each alert to a Microsoft Teams channel as an Adaptive Card.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Targets the Workflows URL, not the old Office 365 connector. Microsoft has retired those
+/// connectors and the <c>MessageCard</c> payload they took, so a sink written to that shape would
+/// have arrived already dead. A Workflows webhook takes the same envelope the Bot Framework uses:
+/// a message whose attachment is an Adaptive Card.
+/// </para>
+/// <para>
+/// This is what the webhook sink meant by "Teams through a Power Automate flow" -- the flow existed
+/// to reshape the payload, and shaping it correctly here removes the flow.
+/// </para>
+/// </remarks>
+public sealed class MicrosoftTeamsAlertSink : IAlertSink
+{
+    /// <summary>The name this sink resolves its <see cref="HttpClient"/> under.</summary>
+    public const string HttpClientName = "Healthie.Alerting.MicrosoftTeams";
+
+    private readonly IHttpClientFactory _clients;
+    private readonly Uri _webhookUrl;
+
+    /// <summary>Initializes a new instance of the <see cref="MicrosoftTeamsAlertSink"/> class.</summary>
+    /// <param name="clients">The factory the request's client comes from.</param>
+    /// <param name="webhookUrl">The Workflows URL of the channel to post to.</param>
+    public MicrosoftTeamsAlertSink(IHttpClientFactory clients, Uri webhookUrl)
+    {
+        _clients = clients ?? throw new ArgumentNullException(nameof(clients));
+        _webhookUrl = webhookUrl ?? throw new ArgumentNullException(nameof(webhookUrl));
+    }
+
+    /// <inheritdoc />
+    public async Task SendAsync(Alert alert, CancellationToken cancellationToken = default)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        var client = _clients.CreateClient(HttpClientName);
+
+        using var response = await client
+            .PostAsJsonAsync(_webhookUrl, TeamsMessage.From(alert), cancellationToken)
+            .ConfigureAwait(false);
+
+        // Throwing is how a sink reports a failed delivery; the dispatcher logs it and carries on.
+        response.EnsureSuccessStatusCode();
+    }
+}
+
+/// <summary>
+/// The envelope a Teams Workflows webhook takes.
+/// </summary>
+/// <param name="Type">Always <c>message</c>.</param>
+/// <param name="Attachments">Exactly one, holding the card.</param>
+public sealed record TeamsMessage(
+    [property: JsonPropertyName("type")] string Type,
+    [property: JsonPropertyName("attachments")] IReadOnlyList<TeamsAttachment> Attachments)
+{
+    /// <summary>Builds the message for an alert.</summary>
+    /// <param name="alert">The alert to describe.</param>
+    public static TeamsMessage From(Alert alert)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        return new TeamsMessage(
+            "message",
+            [new TeamsAttachment(
+                "application/vnd.microsoft.card.adaptive",
+                ContentUrl: null,
+                AdaptiveCard.From(alert))]);
+    }
+}
+
+/// <summary>
+/// One attachment on a Teams message.
+/// </summary>
+/// <param name="ContentType">Always the Adaptive Card content type.</param>
+/// <param name="ContentUrl">Always <c>null</c>; the card is inline.</param>
+/// <param name="Content">The card.</param>
+public sealed record TeamsAttachment(
+    [property: JsonPropertyName("contentType")] string ContentType,
+    [property: JsonPropertyName("contentUrl")] string? ContentUrl,
+    [property: JsonPropertyName("content")] AdaptiveCard Content);
+
+/// <summary>
+/// The Adaptive Card a Teams channel renders.
+/// </summary>
+/// <param name="Type">Always <c>AdaptiveCard</c>.</param>
+/// <param name="Schema">The Adaptive Card schema URL, which Teams requires.</param>
+/// <param name="Version">The card schema version.</param>
+/// <param name="Body">The card's elements.</param>
+public sealed record AdaptiveCard(
+    [property: JsonPropertyName("type")] string Type,
+    [property: JsonPropertyName("$schema")] string Schema,
+    [property: JsonPropertyName("version")] string Version,
+    [property: JsonPropertyName("body")] IReadOnlyList<object> Body)
+{
+    /// <summary>Builds the card for an alert.</summary>
+    /// <param name="alert">The alert to describe.</param>
+    public static AdaptiveCard From(Alert alert)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        var headline = alert.IsRecovery
+            ? $"{alert.DisplayName} recovered"
+            : $"{alert.DisplayName} is {alert.CurrentHealth}";
+
+        var facts = new List<AdaptiveFact>
+        {
+            new("Checker", alert.CheckerName),
+            new("Status", alert.CurrentHealth.ToString()),
+            new("Was", alert.PreviousHealth?.ToString() ?? "never run"),
+            new("Observed", $"{alert.OccurredAt:yyyy-MM-dd HH:mm:ss} UTC"),
+        };
+
+        if (alert.Group is { } group)
+        {
+            facts.Add(new AdaptiveFact("Group", group));
+        }
+
+        if (alert.Tags.Count > 0)
+        {
+            facts.Add(new AdaptiveFact("Tags", string.Join(", ", alert.Tags)));
+        }
+
+        var body = new List<object>
+        {
+            new AdaptiveTextBlock(headline, Weight: "Bolder", Size: "Medium", Color: ColourOf(alert.CurrentHealth), Wrap: true),
+            new AdaptiveFactSet(facts),
+        };
+
+        if (!string.IsNullOrWhiteSpace(alert.Message))
+        {
+            body.Add(new AdaptiveTextBlock(alert.Message, Weight: null, Size: null, Color: null, Wrap: true));
+        }
+
+        return new AdaptiveCard("AdaptiveCard", "http://adaptivecards.io/schemas/adaptive-card.json", "1.4", body);
+    }
+
+    /// <summary>
+    /// Adaptive Cards name their colours rather than taking hex.
+    /// </summary>
+    /// <remarks>
+    /// Lower case, because the schema's <c>Colors</c> enum is
+    /// <c>default|dark|light|accent|good|warning|attention</c> and matching is case-sensitive. A
+    /// value outside it is not an error: the card renders in the default colour and the POST still
+    /// returns 2xx, so getting this wrong loses the colour on every alert and reports success.
+    /// </remarks>
+    private static string ColourOf(PulseCheckerHealth health) => health switch
+    {
+        PulseCheckerHealth.Unhealthy => "attention",
+        PulseCheckerHealth.Suspicious => "warning",
+        PulseCheckerHealth.Healthy => "good",
+        _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."),
+    };
+}
+
+/// <summary>
+/// A line of text on an Adaptive Card.
+/// </summary>
+/// <param name="Text">The text.</param>
+/// <param name="Weight">How bold, or <c>null</c> for the default.</param>
+/// <param name="Size">How large, or <c>null</c> for the default.</param>
+/// <param name="Color">One of the Adaptive Card colour names, or <c>null</c> for the default.</param>
+/// <param name="Wrap">Whether long text wraps rather than being cut off.</param>
+public sealed record AdaptiveTextBlock(
+    [property: JsonPropertyName("text")] string Text,
+    [property: JsonPropertyName("weight"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Weight,
+    [property: JsonPropertyName("size"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Size,
+    [property: JsonPropertyName("color"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Color,
+    [property: JsonPropertyName("wrap")] bool Wrap)
+{
+    /// <summary>Always <c>TextBlock</c>.</summary>
+    [JsonPropertyName("type")]
+    public string Type => "TextBlock";
+}
+
+/// <summary>
+/// A table of labelled values on an Adaptive Card.
+/// </summary>
+/// <param name="Facts">The rows.</param>
+public sealed record AdaptiveFactSet(
+    [property: JsonPropertyName("facts")] IReadOnlyList<AdaptiveFact> Facts)
+{
+    /// <summary>Always <c>FactSet</c>.</summary>
+    [JsonPropertyName("type")]
+    public string Type => "FactSet";
+}
+
+/// <summary>
+/// One row of an <see cref="AdaptiveFactSet"/>.
+/// </summary>
+/// <param name="Title">The label.</param>
+/// <param name="Value">The value.</param>
+public sealed record AdaptiveFact(
+    [property: JsonPropertyName("title")] string Title,
+    [property: JsonPropertyName("value")] string Value);

src/Healthie.Alerting/PagerDutyAlertSink.cs

@@ -0,0 +1,146 @@
+using Healthie.Abstractions.Enums;
+using System.Net.Http.Json;
+using System.Text.Json.Serialization;
+
+namespace Healthie.Alerting;
+
+/// <summary>
+/// Opens and closes PagerDuty incidents through the Events API v2.
+/// </summary>
+/// <remarks>
+/// <para>
+/// The one destination a generic webhook cannot reach usefully. PagerDuty does not want a health
+/// change posted at it; it wants to be told an incident has started or ended, and which incident,
+/// so that a checker flapping between suspicious and unhealthy pages somebody once rather than
+/// every time it moves.
+/// </para>
+/// <para>
+/// <see cref="Alert.DeduplicationKey"/> and <see cref="Alert.IsRecovery"/> already existed for
+/// exactly this: the key becomes <c>dedup_key</c>, and a recovery becomes <c>resolve</c> rather
+/// than another <c>trigger</c>.
+/// </para>
+/// </remarks>
+public sealed class PagerDutyAlertSink : IAlertSink
+{
+    /// <summary>The name this sink resolves its <see cref="HttpClient"/> under.</summary>
+    public const string HttpClientName = "Healthie.Alerting.PagerDuty";
+
+    /// <summary>Where the Events API takes events.</summary>
+    public static readonly Uri EventsEndpoint = new("https://events.pagerduty.com/v2/enqueue");
+
+    private readonly IHttpClientFactory _clients;
+    private readonly string _routingKey;
+    private readonly Uri _endpoint;
+
+    /// <summary>Initializes a new instance of the <see cref="PagerDutyAlertSink"/> class.</summary>
+    /// <param name="clients">The factory the request's client comes from.</param>
+    /// <param name="routingKey">The integration key of the PagerDuty service to alert.</param>
+    /// <param name="endpoint">Where to send events. Defaults to <see cref="EventsEndpoint"/>.</param>
+    public PagerDutyAlertSink(IHttpClientFactory clients, string routingKey, Uri? endpoint = null)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(routingKey);
+
+        _clients = clients ?? throw new ArgumentNullException(nameof(clients));
+        _routingKey = routingKey;
+        _endpoint = endpoint ?? EventsEndpoint;
+    }
+
+    /// <inheritdoc />
+    public async Task SendAsync(Alert alert, CancellationToken cancellationToken = default)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        var client = _clients.CreateClient(HttpClientName);
+
+        using var response = await client
+            .PostAsJsonAsync(_endpoint, PagerDutyEvent.From(alert, _routingKey), cancellationToken)
+            .ConfigureAwait(false);
+
+        // Throwing is how a sink reports a failed delivery; the dispatcher logs it and carries on.
+        response.EnsureSuccessStatusCode();
+    }
+}
+
+/// <summary>
+/// One PagerDuty Events API v2 event.
+/// </summary>
+/// <param name="RoutingKey">The integration key of the service to alert.</param>
+/// <param name="EventAction">Either <c>trigger</c> or <c>resolve</c>.</param>
+/// <param name="DedupKey">Identifies the incident, so repeats update it instead of opening another.</param>
+/// <param name="Payload">What the incident says. Omitted on a resolve, which PagerDuty allows.</param>
+public sealed record PagerDutyEvent(
+    [property: JsonPropertyName("routing_key")] string RoutingKey,
+    [property: JsonPropertyName("event_action")] string EventAction,
+    [property: JsonPropertyName("dedup_key")] string DedupKey,
+    [property: JsonPropertyName("payload"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] PagerDutyEventPayload? Payload)
+{
+    /// <summary>Builds the event for an alert.</summary>
+    /// <param name="alert">The alert to describe.</param>
+    /// <param name="routingKey">The integration key of the service to alert.</param>
+    public static PagerDutyEvent From(Alert alert, string routingKey)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        // A recovery closes the incident the failure opened, which is the whole reason the
+        // deduplication key excludes the health and the time.
+        return alert.IsRecovery
+            ? new PagerDutyEvent(routingKey, "resolve", alert.DeduplicationKey, Payload: null)
+            : new PagerDutyEvent(
+                routingKey,
+                "trigger",
+                alert.DeduplicationKey,
+                PagerDutyEventPayload.From(alert));
+    }
+}
+
+/// <summary>
+/// What a triggered PagerDuty incident says.
+/// </summary>
+/// <param name="Summary">The one line shown in the incident list and the page.</param>
+/// <param name="Severity">PagerDuty's own scale: <c>critical</c>, <c>warning</c> or <c>info</c>.</param>
+/// <param name="Source">Where the problem is, which PagerDuty groups and searches by.</param>
+/// <param name="Component">The component within that source.</param>
+/// <param name="Group">The checker's group, or <c>null</c>.</param>
+/// <param name="CustomDetails">Everything else, shown on the incident.</param>
+public sealed record PagerDutyEventPayload(
+    [property: JsonPropertyName("summary")] string Summary,
+    [property: JsonPropertyName("severity")] string Severity,
+    [property: JsonPropertyName("source")] string Source,
+    [property: JsonPropertyName("component")] string Component,
+    [property: JsonPropertyName("group"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Group,
+    [property: JsonPropertyName("custom_details")] IReadOnlyDictionary<string, string> CustomDetails)
+{
+    /// <summary>Builds the payload for an alert.</summary>
+    /// <param name="alert">The alert to describe.</param>
+    public static PagerDutyEventPayload From(Alert alert)
+    {
+        ArgumentNullException.ThrowIfNull(alert);
+
+        return new PagerDutyEventPayload(
+            $"{alert.DisplayName} is {alert.CurrentHealth}",
+            SeverityOf(alert.CurrentHealth),
+            alert.CheckerName,
+            alert.DisplayName,
+            alert.Group,
+            new Dictionary<string, string>(StringComparer.Ordinal)
+            {
+                ["message"] = alert.Message,
+                ["previous_health"] = alert.PreviousHealth?.ToString() ?? "none",
+                ["tags"] = string.Join(", ", alert.Tags),
+                ["observed_at_utc"] = alert.OccurredAt.ToString("O"),
+            });
+    }
+
+    /// <summary>Maps a health onto PagerDuty's severity scale.</summary>
+    /// <remarks>
+    /// Suspicious is a warning rather than critical on purpose: it is the state that says "failing,
+    /// but not past the threshold yet", and paging on it would defeat having a threshold.
+    /// </remarks>
+    private static string SeverityOf(PulseCheckerHealth health) => health switch
+    {
+        PulseCheckerHealth.Unhealthy => "critical",
+        PulseCheckerHealth.Suspicious => "warning",
+        PulseCheckerHealth.Healthy => "info",
+        _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."),
+    };
+}