From b57fb96cb598755a22fb8feb5e2f7ad761761199 Mon Sep 17 00:00:00 2001 From: Ivan Vydrin Date: Wed, 29 Jul 2026 21:22:43 +0300 Subject: [PATCH 1/3] Reach Slack, Teams and PagerDuty without something in between The webhook sink's own remarks claimed it was "the one sink worth shipping, because it is the one that reaches everything else", and listed Slack, Teams and PagerDuty among the things it reached. It does not reach any of them. All three reject arbitrary JSON: a Slack incoming webhook wants text and attachments or blocks, Teams wants an Adaptive Card in a Bot Framework envelope, and PagerDuty wants an Events API event. Posting the generic payload at a Slack webhook returns invalid_payload. The remarks half admitted it, in the phrase "Teams through a Power Automate flow" -- the flow existed to reshape the payload, which is work this library was leaving to its users. Three sinks, in the alerting package rather than a package each. That is the one place this deviates from what the roadmap said: it called for packages, and a package buys isolation of dependencies. These have none to isolate -- every one is HttpClient and System.Text.Json, both already here -- so three more package ids, readmes, and release entries would cost real maintenance and give nothing back. PagerDuty is the one that is not just a payload shape. It does not want to be told a health changed; it wants to be told an incident opened or closed, and which one. Alert.DeduplicationKey and Alert.IsRecovery already existed for exactly that, with a comment saying a flapping checker is one incident rather than several -- so a failure triggers keyed on the checker and a recovery resolves that same incident instead of raising a second one saying everything is fine. Suspicious maps to warning rather than critical, because it is the state that means "failing, but not past the threshold yet" and paging on it would defeat having a threshold. Teams targets the Workflows URL and an Adaptive Card. The Office 365 connectors and the MessageCard payload they took are retired, so a sink written to that shape would have arrived already dead. The payload shapes are asserted against the request body rather than against the records that build them, because the failure mode is a field named right in C# and wrong in JSON, which shows up only at the far end where no test would see it. The webhook sink had no coverage at all and now shares the same harness. --- CHANGELOG.md | 8 + README.md | 5 +- .../Healthie.Alerting.csproj | 4 +- .../MicrosoftTeamsAlertSink.cs | 194 +++++++++++ src/Healthie.Alerting/PagerDutyAlertSink.cs | 146 ++++++++ src/Healthie.Alerting/README.md | 22 +- src/Healthie.Alerting/SlackAlertSink.cs | 135 ++++++++ src/Healthie.Alerting/StartupExtensions.cs | 76 ++++- src/Healthie.Alerting/WebhookAlertSink.cs | 13 +- tests/Healthie.Tests.Unit/AlertSinkTests.cs | 320 ++++++++++++++++++ 10 files changed, 911 insertions(+), 12 deletions(-) create mode 100644 src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs create mode 100644 src/Healthie.Alerting/PagerDutyAlertSink.cs create mode 100644 src/Healthie.Alerting/SlackAlertSink.cs create mode 100644 tests/Healthie.Tests.Unit/AlertSinkTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index ae3cff4..65d0987 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,14 @@ its code behaves exactly as it did. that only cared about a component going down reached into `OldState`/`NewState.LastResult?.Health` and worked it out again. The alerting and uptime packages were doing that identically; both now read `PreviousHealth` and `CurrentHealth` from the event instead of digging for them. +- **Slack, Microsoft Teams and PagerDuty alert sinks**, in `Healthie.NET.Alerting` beside the + webhook. Each of those three rejects arbitrary JSON and wants its own shape, so the generic + webhook never actually reached them without something in between to reshape it -- which its own + remarks admitted, in the phrase "Teams through a Power Automate flow". They need no dependency the + package did not already have, so they are sinks rather than a package each. Teams targets the + Workflows URL and an Adaptive Card, because the Office 365 connectors and the `MessageCard` + payload they took are retired. PagerDuty resolves the incident it opened rather than raising a + second one: `Alert.DeduplicationKey` and `Alert.IsRecovery` already existed for exactly that. - **Schedules.** `PulseSchedule` says either "every this long" or "on this cron expression", and sits alongside `PulseInterval` rather than replacing it. The enum stopped at five minutes, which is short of what a certificate-expiry or disk-space check wants. Cron is standard Unix syntax and diff --git a/README.md b/README.md index c483be8..c657e96 100644 --- a/README.md +++ b/README.md @@ -824,12 +824,11 @@ Upgrading from v1.x? See the [v1 to v2 migration guide](https://github.com/ivanv Shipped since 3.1.4: alerting on transitions, OpenTelemetry metrics and traces, arbitrary intervals and cron, PostgreSQL / SQL Server / SQLite state providers, Hangfire / Coravel / Temporal scheduling, ready-made checkers, uptime reporting, leader election, optimistic concurrency -on `IStateProvider`, and `HealthChanged` on the state-changed event. What is left: +on `IStateProvider`, `HealthChanged` on the state-changed event, and Slack / Teams / +PagerDuty alert sinks. What is left: - **A Redis state provider** -- the fastest option for state written on every tick, and a natural lease store for leader election. -- **Alert sinks beyond the webhook** -- Slack, Teams and PagerDuty as packages rather than as a - documented payload. --- diff --git a/src/Healthie.Alerting/Healthie.Alerting.csproj b/src/Healthie.Alerting/Healthie.Alerting.csproj index 21ad91c..ab88a24 100644 --- a/src/Healthie.Alerting/Healthie.Alerting.csproj +++ b/src/Healthie.Alerting/Healthie.Alerting.csproj @@ -2,8 +2,8 @@ Healthie.NET.Alerting - Alerting for Healthie.NET: turns health changes into alerts and delivers them to a webhook or your own sink, with deduplication and delivery failures isolated from the checks. - $(HealthieCommonTags);alerting;notifications;webhook;incidents + Alerting for Healthie.NET: turns health changes into alerts and delivers them to Slack, Microsoft Teams, PagerDuty, a webhook or your own sink, with deduplication and delivery failures isolated from the checks. + $(HealthieCommonTags);alerting;notifications;webhook;slack;teams;pagerduty;incidents diff --git a/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs b/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs new file mode 100644 index 0000000..bc45b6c --- /dev/null +++ b/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs @@ -0,0 +1,194 @@ +using Healthie.Abstractions.Enums; +using System.Net.Http.Json; +using System.Text.Json.Serialization; + +namespace Healthie.Alerting; + +/// +/// Posts each alert to a Microsoft Teams channel as an Adaptive Card. +/// +/// +/// +/// Targets the Workflows URL, not the old Office 365 connector. Microsoft has retired those +/// connectors and the MessageCard payload they took, so a sink written to that shape would +/// have arrived already dead. A Workflows webhook takes the same envelope the Bot Framework uses: +/// a message whose attachment is an Adaptive Card. +/// +/// +/// This is what the webhook sink meant by "Teams through a Power Automate flow" -- the flow existed +/// to reshape the payload, and shaping it correctly here removes the flow. +/// +/// +public sealed class MicrosoftTeamsAlertSink : IAlertSink +{ + /// The name this sink resolves its under. + public const string HttpClientName = "Healthie.Alerting.MicrosoftTeams"; + + private readonly IHttpClientFactory _clients; + private readonly Uri _webhookUrl; + + /// Initializes a new instance of the class. + /// The factory the request's client comes from. + /// The Workflows URL of the channel to post to. + public MicrosoftTeamsAlertSink(IHttpClientFactory clients, Uri webhookUrl) + { + _clients = clients ?? throw new ArgumentNullException(nameof(clients)); + _webhookUrl = webhookUrl ?? throw new ArgumentNullException(nameof(webhookUrl)); + } + + /// + public async Task SendAsync(Alert alert, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(alert); + + var client = _clients.CreateClient(HttpClientName); + + using var response = await client + .PostAsJsonAsync(_webhookUrl, TeamsMessage.From(alert), cancellationToken) + .ConfigureAwait(false); + + // Throwing is how a sink reports a failed delivery; the dispatcher logs it and carries on. + response.EnsureSuccessStatusCode(); + } +} + +/// +/// The envelope a Teams Workflows webhook takes. +/// +/// Always message. +/// Exactly one, holding the card. +public sealed record TeamsMessage( + [property: JsonPropertyName("type")] string Type, + [property: JsonPropertyName("attachments")] IReadOnlyList Attachments) +{ + /// Builds the message for an alert. + /// The alert to describe. + public static TeamsMessage From(Alert alert) + { + ArgumentNullException.ThrowIfNull(alert); + + return new TeamsMessage( + "message", + [new TeamsAttachment( + "application/vnd.microsoft.card.adaptive", + ContentUrl: null, + AdaptiveCard.From(alert))]); + } +} + +/// +/// One attachment on a Teams message. +/// +/// Always the Adaptive Card content type. +/// Always null; the card is inline. +/// The card. +public sealed record TeamsAttachment( + [property: JsonPropertyName("contentType")] string ContentType, + [property: JsonPropertyName("contentUrl")] string? ContentUrl, + [property: JsonPropertyName("content")] AdaptiveCard Content); + +/// +/// The Adaptive Card a Teams channel renders. +/// +/// Always AdaptiveCard. +/// The Adaptive Card schema URL, which Teams requires. +/// The card schema version. +/// The card's elements. +public sealed record AdaptiveCard( + [property: JsonPropertyName("type")] string Type, + [property: JsonPropertyName("$schema")] string Schema, + [property: JsonPropertyName("version")] string Version, + [property: JsonPropertyName("body")] IReadOnlyList Body) +{ + /// Builds the card for an alert. + /// The alert to describe. + public static AdaptiveCard From(Alert alert) + { + ArgumentNullException.ThrowIfNull(alert); + + var headline = alert.IsRecovery + ? $"{alert.DisplayName} recovered" + : $"{alert.DisplayName} is {alert.CurrentHealth}"; + + var facts = new List + { + new("Checker", alert.CheckerName), + new("Status", alert.CurrentHealth.ToString()), + new("Was", alert.PreviousHealth?.ToString() ?? "never run"), + new("Observed", $"{alert.OccurredAt:yyyy-MM-dd HH:mm:ss} UTC"), + }; + + if (alert.Group is { } group) + { + facts.Add(new AdaptiveFact("Group", group)); + } + + if (alert.Tags.Count > 0) + { + facts.Add(new AdaptiveFact("Tags", string.Join(", ", alert.Tags))); + } + + var body = new List + { + new AdaptiveTextBlock(headline, Weight: "Bolder", Size: "Medium", Color: ColourOf(alert.CurrentHealth), Wrap: true), + new AdaptiveFactSet(facts), + }; + + if (!string.IsNullOrWhiteSpace(alert.Message)) + { + body.Add(new AdaptiveTextBlock(alert.Message, Weight: null, Size: null, Color: null, Wrap: true)); + } + + return new AdaptiveCard("AdaptiveCard", "http://adaptivecards.io/schemas/adaptive-card.json", "1.4", body); + } + + /// Adaptive Cards name their colours rather than taking hex. + private static string ColourOf(PulseCheckerHealth health) => health switch + { + PulseCheckerHealth.Unhealthy => "Attention", + PulseCheckerHealth.Suspicious => "Warning", + PulseCheckerHealth.Healthy => "Good", + _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."), + }; +} + +/// +/// A line of text on an Adaptive Card. +/// +/// The text. +/// How bold, or null for the default. +/// How large, or null for the default. +/// One of the Adaptive Card colour names, or null for the default. +/// Whether long text wraps rather than being cut off. +public sealed record AdaptiveTextBlock( + [property: JsonPropertyName("text")] string Text, + [property: JsonPropertyName("weight"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Weight, + [property: JsonPropertyName("size"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Size, + [property: JsonPropertyName("color"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Color, + [property: JsonPropertyName("wrap")] bool Wrap) +{ + /// Always TextBlock. + [JsonPropertyName("type")] + public string Type => "TextBlock"; +} + +/// +/// A table of labelled values on an Adaptive Card. +/// +/// The rows. +public sealed record AdaptiveFactSet( + [property: JsonPropertyName("facts")] IReadOnlyList Facts) +{ + /// Always FactSet. + [JsonPropertyName("type")] + public string Type => "FactSet"; +} + +/// +/// One row of an . +/// +/// The label. +/// The value. +public sealed record AdaptiveFact( + [property: JsonPropertyName("title")] string Title, + [property: JsonPropertyName("value")] string Value); diff --git a/src/Healthie.Alerting/PagerDutyAlertSink.cs b/src/Healthie.Alerting/PagerDutyAlertSink.cs new file mode 100644 index 0000000..31b2ce8 --- /dev/null +++ b/src/Healthie.Alerting/PagerDutyAlertSink.cs @@ -0,0 +1,146 @@ +using Healthie.Abstractions.Enums; +using System.Net.Http.Json; +using System.Text.Json.Serialization; + +namespace Healthie.Alerting; + +/// +/// Opens and closes PagerDuty incidents through the Events API v2. +/// +/// +/// +/// The one destination a generic webhook cannot reach usefully. PagerDuty does not want a health +/// change posted at it; it wants to be told an incident has started or ended, and which incident, +/// so that a checker flapping between suspicious and unhealthy pages somebody once rather than +/// every time it moves. +/// +/// +/// and already existed for +/// exactly this: the key becomes dedup_key, and a recovery becomes resolve rather +/// than another trigger. +/// +/// +public sealed class PagerDutyAlertSink : IAlertSink +{ + /// The name this sink resolves its under. + public const string HttpClientName = "Healthie.Alerting.PagerDuty"; + + /// Where the Events API takes events. + public static readonly Uri EventsEndpoint = new("https://events.pagerduty.com/v2/enqueue"); + + private readonly IHttpClientFactory _clients; + private readonly string _routingKey; + private readonly Uri _endpoint; + + /// Initializes a new instance of the class. + /// The factory the request's client comes from. + /// The integration key of the PagerDuty service to alert. + /// Where to send events. Defaults to . + public PagerDutyAlertSink(IHttpClientFactory clients, string routingKey, Uri? endpoint = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(routingKey); + + _clients = clients ?? throw new ArgumentNullException(nameof(clients)); + _routingKey = routingKey; + _endpoint = endpoint ?? EventsEndpoint; + } + + /// + public async Task SendAsync(Alert alert, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(alert); + + var client = _clients.CreateClient(HttpClientName); + + using var response = await client + .PostAsJsonAsync(_endpoint, PagerDutyEvent.From(alert, _routingKey), cancellationToken) + .ConfigureAwait(false); + + // Throwing is how a sink reports a failed delivery; the dispatcher logs it and carries on. + response.EnsureSuccessStatusCode(); + } +} + +/// +/// One PagerDuty Events API v2 event. +/// +/// The integration key of the service to alert. +/// Either trigger or resolve. +/// Identifies the incident, so repeats update it instead of opening another. +/// What the incident says. Omitted on a resolve, which PagerDuty allows. +public sealed record PagerDutyEvent( + [property: JsonPropertyName("routing_key")] string RoutingKey, + [property: JsonPropertyName("event_action")] string EventAction, + [property: JsonPropertyName("dedup_key")] string DedupKey, + [property: JsonPropertyName("payload"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] PagerDutyEventPayload? Payload) +{ + /// Builds the event for an alert. + /// The alert to describe. + /// The integration key of the service to alert. + public static PagerDutyEvent From(Alert alert, string routingKey) + { + ArgumentNullException.ThrowIfNull(alert); + + // A recovery closes the incident the failure opened, which is the whole reason the + // deduplication key excludes the health and the time. + return alert.IsRecovery + ? new PagerDutyEvent(routingKey, "resolve", alert.DeduplicationKey, Payload: null) + : new PagerDutyEvent( + routingKey, + "trigger", + alert.DeduplicationKey, + PagerDutyEventPayload.From(alert)); + } +} + +/// +/// What a triggered PagerDuty incident says. +/// +/// The one line shown in the incident list and the page. +/// PagerDuty's own scale: critical, warning or info. +/// Where the problem is, which PagerDuty groups and searches by. +/// The component within that source. +/// The checker's group, or null. +/// Everything else, shown on the incident. +public sealed record PagerDutyEventPayload( + [property: JsonPropertyName("summary")] string Summary, + [property: JsonPropertyName("severity")] string Severity, + [property: JsonPropertyName("source")] string Source, + [property: JsonPropertyName("component")] string Component, + [property: JsonPropertyName("group")] string? Group, + [property: JsonPropertyName("custom_details")] IReadOnlyDictionary CustomDetails) +{ + /// Builds the payload for an alert. + /// The alert to describe. + public static PagerDutyEventPayload From(Alert alert) + { + ArgumentNullException.ThrowIfNull(alert); + + return new PagerDutyEventPayload( + $"{alert.DisplayName} is {alert.CurrentHealth}", + SeverityOf(alert.CurrentHealth), + alert.CheckerName, + alert.DisplayName, + alert.Group, + new Dictionary(StringComparer.Ordinal) + { + ["message"] = alert.Message, + ["previous_health"] = alert.PreviousHealth?.ToString() ?? "none", + ["tags"] = string.Join(", ", alert.Tags), + ["observed_at_utc"] = alert.OccurredAt.ToString("O"), + }); + } + + /// Maps a health onto PagerDuty's severity scale. + /// + /// Suspicious is a warning rather than critical on purpose: it is the state that says "failing, + /// but not past the threshold yet", and paging on it would defeat having a threshold. + /// + private static string SeverityOf(PulseCheckerHealth health) => health switch + { + PulseCheckerHealth.Unhealthy => "critical", + PulseCheckerHealth.Suspicious => "warning", + PulseCheckerHealth.Healthy => "info", + _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."), + }; +} diff --git a/src/Healthie.Alerting/README.md b/src/Healthie.Alerting/README.md index 992953b..97a90ed 100644 --- a/src/Healthie.Alerting/README.md +++ b/src/Healthie.Alerting/README.md @@ -22,10 +22,28 @@ using Healthie.Alerting; builder.Services .AddHealthie(typeof(Program).Assembly) .AddHealthieAlerts() - .AddHealthieWebhookAlerts(new Uri("https://hooks.example.com/healthie")); + .AddHealthieSlackAlerts(new Uri("https://hooks.slack.com/services/...")) + .AddHealthiePagerDutyAlerts("your-integration-key"); ``` -Add as many webhooks as you like; each gets every alert. For anywhere a webhook cannot reach, implement `IAlertSink` and register it — the dispatcher finds every one that is registered. +Add as many sinks as you like; each gets every alert. For anywhere none of them reaches, implement `IAlertSink` and register it — the dispatcher finds every one that is registered. + +## Where alerts go + +| Sink | Registered with | Sends | +|---|---|---| +| Slack | `AddHealthieSlackAlerts(url)` | A message with a colour-coded attachment, to an incoming webhook | +| Microsoft Teams | `AddHealthieMicrosoftTeamsAlerts(url)` | An Adaptive Card, to a **Workflows** URL | +| PagerDuty | `AddHealthiePagerDutyAlerts(key)` | An Events API v2 `trigger`, resolved when the checker recovers | +| Anything else | `AddHealthieWebhookAlerts(url)` | The generic JSON payload below | + +Three of these are not the webhook wearing a hat. Slack, Teams and PagerDuty each reject arbitrary JSON, so posting the generic payload at them fails — which is why reaching them used to need something in between to reshape it. None needs a dependency this package did not already have, so they are here rather than in a package each. + +**Teams takes the Workflows URL, not an Office 365 connector.** Microsoft has retired those connectors along with the `MessageCard` payload they accepted; in Teams, add a *Workflow* to the channel and use the URL it gives you. + +**PagerDuty closes what it opens.** A failure raises an incident keyed on the checker, and the recovery resolves that same incident rather than posting a second message saying everything is fine. A checker flapping between suspicious and unhealthy stays one incident, because the deduplication key deliberately leaves out the health and the time. `Suspicious` maps to `warning` rather than `critical` — it is the state that means "failing, but not past the threshold yet", and paging on it would defeat having a threshold. + +Configure any sink's `HttpClient` — its timeout, its handler, an auth header — by naming its `HttpClientName` constant in your own `AddHttpClient` call. ## What alerts diff --git a/src/Healthie.Alerting/SlackAlertSink.cs b/src/Healthie.Alerting/SlackAlertSink.cs new file mode 100644 index 0000000..4871eda --- /dev/null +++ b/src/Healthie.Alerting/SlackAlertSink.cs @@ -0,0 +1,135 @@ +using Healthie.Abstractions.Enums; +using System.Net.Http.Json; +using System.Text.Json.Serialization; + +namespace Healthie.Alerting; + +/// +/// Posts each alert to a Slack incoming webhook. +/// +/// +/// A Slack webhook does not accept arbitrary JSON: it wants text, and optionally attachments +/// or blocks. Posting the generic payload at one produces invalid_payload, which is why +/// reaching Slack used to need something in between to reshape it. +/// +public sealed class SlackAlertSink : IAlertSink +{ + /// The name this sink resolves its under. + public const string HttpClientName = "Healthie.Alerting.Slack"; + + private readonly IHttpClientFactory _clients; + private readonly Uri _webhookUrl; + + /// Initializes a new instance of the class. + /// The factory the request's client comes from. + /// The incoming-webhook URL of the channel to post to. + public SlackAlertSink(IHttpClientFactory clients, Uri webhookUrl) + { + _clients = clients ?? throw new ArgumentNullException(nameof(clients)); + _webhookUrl = webhookUrl ?? throw new ArgumentNullException(nameof(webhookUrl)); + } + + /// + public async Task SendAsync(Alert alert, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(alert); + + var client = _clients.CreateClient(HttpClientName); + + using var response = await client + .PostAsJsonAsync(_webhookUrl, SlackMessage.From(alert), cancellationToken) + .ConfigureAwait(false); + + // Throwing is how a sink reports a failed delivery; the dispatcher logs it and carries on. + response.EnsureSuccessStatusCode(); + } +} + +/// +/// The JSON a posts. +/// +/// The notification line, which is what a push notification shows. +/// The coloured detail block. +public sealed record SlackMessage( + [property: JsonPropertyName("text")] string Text, + [property: JsonPropertyName("attachments")] IReadOnlyList Attachments) +{ + /// Builds the message for an alert. + /// The alert to describe. + public static SlackMessage From(Alert alert) + { + ArgumentNullException.ThrowIfNull(alert); + + var headline = alert.IsRecovery + ? $"{alert.DisplayName} recovered" + : $"{alert.DisplayName} is {alert.CurrentHealth}"; + + var fields = new List + { + new("Status", alert.CurrentHealth.ToString(), Short: true), + new("Was", alert.PreviousHealth?.ToString() ?? "never run", Short: true), + }; + + if (alert.Group is { } group) + { + fields.Add(new SlackField("Group", group, Short: true)); + } + + if (alert.Tags.Count > 0) + { + fields.Add(new SlackField("Tags", string.Join(", ", alert.Tags), Short: true)); + } + + if (!string.IsNullOrWhiteSpace(alert.Message)) + { + fields.Add(new SlackField("Message", alert.Message, Short: false)); + } + + return new SlackMessage( + headline, + [new SlackAttachment(ColourOf(alert.CurrentHealth), alert.CheckerName, fields, ToUnixSeconds(alert.OccurredAt))]); + } + + /// Slack's own status colours, which it renders as the bar down the attachment. + private static string ColourOf(PulseCheckerHealth health) => health switch + { + PulseCheckerHealth.Unhealthy => "danger", + PulseCheckerHealth.Suspicious => "warning", + PulseCheckerHealth.Healthy => "good", + _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."), + }; + + /// + /// Slack timestamps events in Unix seconds. + /// + /// + /// Read as UTC rather than local: an alert's time is recorded in UTC, and a DateTime carrying + /// Unspecified would otherwise be shifted by the server's offset on the way out. + /// + private static long ToUnixSeconds(DateTime occurredAt) => + new DateTimeOffset(DateTime.SpecifyKind(occurredAt, DateTimeKind.Utc)).ToUnixTimeSeconds(); +} + +/// +/// One coloured block under a Slack message. +/// +/// Slack's good, warning or danger. +/// Shown small under the fields; the checker's full name goes here. +/// The labelled values. +/// When it happened, in Unix seconds. +public sealed record SlackAttachment( + [property: JsonPropertyName("color")] string Color, + [property: JsonPropertyName("footer")] string Footer, + [property: JsonPropertyName("fields")] IReadOnlyList Fields, + [property: JsonPropertyName("ts")] long Timestamp); + +/// +/// One labelled value in a Slack attachment. +/// +/// The label. +/// The value. +/// Whether Slack may put it beside another rather than on its own line. +public sealed record SlackField( + [property: JsonPropertyName("title")] string Title, + [property: JsonPropertyName("value")] string Value, + [property: JsonPropertyName("short")] bool Short); diff --git a/src/Healthie.Alerting/StartupExtensions.cs b/src/Healthie.Alerting/StartupExtensions.cs index 4268796..f0e7aec 100644 --- a/src/Healthie.Alerting/StartupExtensions.cs +++ b/src/Healthie.Alerting/StartupExtensions.cs @@ -16,7 +16,9 @@ public static class StartupExtensions /// Optionally adjusts which changes alert, and how hard to try. /// The service collection for fluent chaining. /// - /// Add at least one sink -- , or your own + /// Add at least one sink -- , + /// , , + /// , or your own /// . With none, the dispatcher says so once at startup and does nothing, /// rather than queueing alerts nobody will read. /// @@ -56,4 +58,76 @@ public static IServiceCollection AddHealthieWebhookAlerts(this IServiceCollectio return services.AddSingleton(provider => new WebhookAlertSink(provider.GetRequiredService(), url)); } + + /// + /// Posts every alert to a Slack channel. + /// + /// The service collection to register with. + /// The incoming-webhook URL of the channel to post to. + /// The service collection for fluent chaining. + /// + /// One more sink rather than a replacement, so this can sit alongside a webhook and a pager and + /// each gets every alert. Configure the client by naming + /// in your own AddHttpClient call. + /// + public static IServiceCollection AddHealthieSlackAlerts(this IServiceCollection services, Uri webhookUrl) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(webhookUrl); + + services.AddHttpClient(); + + return services.AddSingleton(provider => + new SlackAlertSink(provider.GetRequiredService(), webhookUrl)); + } + + /// + /// Posts every alert to a Microsoft Teams channel as an Adaptive Card. + /// + /// The service collection to register with. + /// The Workflows URL of the channel to post to. + /// The service collection for fluent chaining. + /// + /// The URL is the one a Teams Workflow gives you, not an Office 365 connector -- those + /// are retired, along with the payload they took. Configure the client by naming + /// in your own AddHttpClient call. + /// + public static IServiceCollection AddHealthieMicrosoftTeamsAlerts(this IServiceCollection services, Uri webhookUrl) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(webhookUrl); + + services.AddHttpClient(); + + return services.AddSingleton(provider => + new MicrosoftTeamsAlertSink(provider.GetRequiredService(), webhookUrl)); + } + + /// + /// Opens and closes PagerDuty incidents as checkers fail and recover. + /// + /// The service collection to register with. + /// The integration key of the PagerDuty service to alert. + /// + /// Where to send events. Defaults to ; override it + /// for the EU service region. + /// + /// The service collection for fluent chaining. + /// + /// Unlike the chat sinks, this one closes what it opened: a recovery resolves the incident the + /// failure raised rather than posting a second message saying everything is fine. + /// + public static IServiceCollection AddHealthiePagerDutyAlerts( + this IServiceCollection services, + string routingKey, + Uri? endpoint = null) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentException.ThrowIfNullOrWhiteSpace(routingKey); + + services.AddHttpClient(); + + return services.AddSingleton(provider => + new PagerDutyAlertSink(provider.GetRequiredService(), routingKey, endpoint)); + } } diff --git a/src/Healthie.Alerting/WebhookAlertSink.cs b/src/Healthie.Alerting/WebhookAlertSink.cs index b60eceb..f53774d 100644 --- a/src/Healthie.Alerting/WebhookAlertSink.cs +++ b/src/Healthie.Alerting/WebhookAlertSink.cs @@ -6,10 +6,15 @@ namespace Healthie.Alerting; /// Posts each alert as JSON to a URL. /// /// -/// The one sink worth shipping, because it is the one that reaches everything else. Slack, Teams -/// through a Power Automate flow, Discord, PagerDuty's Events API, an internal service and a -/// no-code automation tool all accept an HTTP POST, so a webhook covers them with a payload shape -/// documented once rather than a package each. +/// The sink for anything that accepts a POST and does not care what shape it is: an internal +/// service, a no-code automation tool, a queue. It is deliberately the generic one -- the payload is +/// documented once and never reshaped per destination. +/// +/// It does not reach Slack, Teams or PagerDuty. Each of those rejects arbitrary JSON and wants its +/// own shape, which is what the sinks beside this one are: , +/// and . They need no +/// dependency this package did not already have, so they are here rather than in a package each. +/// /// public sealed class WebhookAlertSink : IAlertSink { diff --git a/tests/Healthie.Tests.Unit/AlertSinkTests.cs b/tests/Healthie.Tests.Unit/AlertSinkTests.cs new file mode 100644 index 0000000..2606803 --- /dev/null +++ b/tests/Healthie.Tests.Unit/AlertSinkTests.cs @@ -0,0 +1,320 @@ +using Healthie.Abstractions.Enums; +using Healthie.Alerting; +using System.Net; +using System.Text.Json; + +namespace Healthie.Tests.Unit; + +/// +/// What each sink actually puts on the wire. +/// +/// +/// Asserted against the request body rather than against the payload records, because the thing that +/// breaks is the shape a service receives: Slack, Teams and PagerDuty each reject arbitrary JSON, so +/// a field named right in C# and wrong in JSON fails only at the far end, where nothing here would +/// see it. +/// +public class AlertSinkTests +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static Alert Failing( + PulseCheckerHealth current = PulseCheckerHealth.Unhealthy, + PulseCheckerHealth? previous = PulseCheckerHealth.Healthy) => + new( + "Acme.Checkers.PaymentsPulseChecker", + "Payments API", + "Tier 1", + ["payments", "external"], + previous, + current, + "502 Bad Gateway", + new DateTime(2026, 7, 29, 14, 30, 0, DateTimeKind.Utc)); + + private static Alert Recovered() => Failing(PulseCheckerHealth.Healthy, PulseCheckerHealth.Unhealthy); + + /// Captures the one request a sink makes, and answers it. + private sealed class CapturingHandler(HttpStatusCode status = HttpStatusCode.OK) : HttpMessageHandler + { + public HttpRequestMessage? Request { get; private set; } + + public string Body { get; private set; } = string.Empty; + + protected override async Task SendAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) + { + Request = request; + Body = request.Content is null + ? string.Empty + : await request.Content.ReadAsStringAsync(cancellationToken); + + return new HttpResponseMessage(status); + } + } + + private sealed class SingleClientFactory(HttpMessageHandler handler) : IHttpClientFactory + { + public HttpClient CreateClient(string name) => new(handler, disposeHandler: false); + } + + private static (SingleClientFactory Clients, CapturingHandler Handler) Capture( + HttpStatusCode status = HttpStatusCode.OK) + { + var handler = new CapturingHandler(status); + return (new SingleClientFactory(handler), handler); + } + + private static JsonElement Json(string body) => JsonDocument.Parse(body).RootElement; + + // ---- Slack --------------------------------------------------------------------------------- + + [Fact] + public async Task Slack_SendsTheShapeAnIncomingWebhookAccepts() + { + var (clients, handler) = Capture(); + var sink = new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")); + + await sink.SendAsync(Failing(), Ct); + + var root = Json(handler.Body); + + // `text` is what a push notification shows, so it must carry the headline on its own. + Assert.Contains("Payments API", root.GetProperty("text").GetString()!, StringComparison.Ordinal); + + var attachment = root.GetProperty("attachments")[0]; + Assert.Equal("danger", attachment.GetProperty("color").GetString()); + Assert.Equal("Acme.Checkers.PaymentsPulseChecker", attachment.GetProperty("footer").GetString()); + + var fields = attachment.GetProperty("fields").EnumerateArray() + .ToDictionary(f => f.GetProperty("title").GetString()!, f => f.GetProperty("value").GetString()!); + + Assert.Equal("Unhealthy", fields["Status"]); + Assert.Equal("Healthy", fields["Was"]); + Assert.Equal("Tier 1", fields["Group"]); + Assert.Equal("payments, external", fields["Tags"]); + Assert.Equal("502 Bad Gateway", fields["Message"]); + } + + [Theory] + [InlineData(PulseCheckerHealth.Unhealthy, "danger")] + [InlineData(PulseCheckerHealth.Suspicious, "warning")] + [InlineData(PulseCheckerHealth.Healthy, "good")] + public async Task Slack_ColoursTheAttachmentByHealth(PulseCheckerHealth health, string expected) + { + var (clients, handler) = Capture(); + var sink = new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")); + + await sink.SendAsync(Failing(health, PulseCheckerHealth.Suspicious), Ct); + + Assert.Equal(expected, Json(handler.Body).GetProperty("attachments")[0].GetProperty("color").GetString()); + } + + /// + /// The alert's time is UTC. Read as anything else it is shifted by the server's offset, and the + /// message says a component failed at a time it did not. + /// + [Fact] + public async Task Slack_TimestampsInUtcRegardlessOfTheServersZone() + { + var (clients, handler) = Capture(); + var sink = new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")); + + await sink.SendAsync(Failing(), Ct); + + var expected = new DateTimeOffset(2026, 7, 29, 14, 30, 0, TimeSpan.Zero).ToUnixTimeSeconds(); + + Assert.Equal(expected, Json(handler.Body).GetProperty("attachments")[0].GetProperty("ts").GetInt64()); + } + + // ---- Microsoft Teams ----------------------------------------------------------------------- + + [Fact] + public async Task Teams_SendsAnAdaptiveCardInTheEnvelopeWorkflowsExpects() + { + var (clients, handler) = Capture(); + var sink = new MicrosoftTeamsAlertSink(clients, new Uri("https://prod.westeurope.logic.azure.test/workflows/abc")); + + await sink.SendAsync(Failing(), Ct); + + var root = Json(handler.Body); + Assert.Equal("message", root.GetProperty("type").GetString()); + + var attachment = root.GetProperty("attachments")[0]; + Assert.Equal("application/vnd.microsoft.card.adaptive", attachment.GetProperty("contentType").GetString()); + + var card = attachment.GetProperty("content"); + Assert.Equal("AdaptiveCard", card.GetProperty("type").GetString()); + + // Teams refuses a card with no schema. + Assert.Equal("http://adaptivecards.io/schemas/adaptive-card.json", card.GetProperty("$schema").GetString()); + + var body = card.GetProperty("body").EnumerateArray().ToList(); + Assert.Equal("TextBlock", body[0].GetProperty("type").GetString()); + Assert.Equal("Attention", body[0].GetProperty("color").GetString()); + Assert.Contains("Payments API", body[0].GetProperty("text").GetString()!, StringComparison.Ordinal); + + Assert.Equal("FactSet", body[1].GetProperty("type").GetString()); + + var facts = body[1].GetProperty("facts").EnumerateArray() + .ToDictionary(f => f.GetProperty("title").GetString()!, f => f.GetProperty("value").GetString()!); + + Assert.Equal("Unhealthy", facts["Status"]); + Assert.Equal("Healthy", facts["Was"]); + Assert.Contains("UTC", facts["Observed"], StringComparison.Ordinal); + } + + /// + /// A card element carrying a null weight or colour is invalid to Teams, so the optional ones are + /// left out rather than written as null. + /// + [Fact] + public async Task Teams_OmitsOptionalCardPropertiesRatherThanWritingThemNull() + { + var (clients, handler) = Capture(); + var sink = new MicrosoftTeamsAlertSink(clients, new Uri("https://prod.westeurope.logic.azure.test/workflows/abc")); + + await sink.SendAsync(Failing(), Ct); + + var body = Json(handler.Body) + .GetProperty("attachments")[0].GetProperty("content").GetProperty("body") + .EnumerateArray().ToList(); + + // The message block is the plain one: no weight, size or colour of its own. + var message = body.Last(); + Assert.False(message.TryGetProperty("weight", out _)); + Assert.False(message.TryGetProperty("size", out _)); + Assert.False(message.TryGetProperty("color", out _)); + Assert.True(message.GetProperty("wrap").GetBoolean()); + } + + // ---- PagerDuty ----------------------------------------------------------------------------- + + [Fact] + public async Task PagerDuty_TriggersAnIncidentKeyedOnTheChecker() + { + var (clients, handler) = Capture(HttpStatusCode.Accepted); + var sink = new PagerDutyAlertSink(clients, "routing-key-1"); + + var alert = Failing(); + await sink.SendAsync(alert, Ct); + + var root = Json(handler.Body); + Assert.Equal("routing-key-1", root.GetProperty("routing_key").GetString()); + Assert.Equal("trigger", root.GetProperty("event_action").GetString()); + Assert.Equal(alert.DeduplicationKey, root.GetProperty("dedup_key").GetString()); + + var payload = root.GetProperty("payload"); + Assert.Equal("critical", payload.GetProperty("severity").GetString()); + Assert.Equal("Acme.Checkers.PaymentsPulseChecker", payload.GetProperty("source").GetString()); + Assert.Equal("Payments API", payload.GetProperty("component").GetString()); + Assert.Equal("Tier 1", payload.GetProperty("group").GetString()); + Assert.Equal("502 Bad Gateway", payload.GetProperty("custom_details").GetProperty("message").GetString()); + } + + /// + /// The reason the deduplication key leaves out the health and the time: a recovery has to close + /// the incident the failure opened, not open a second one saying everything is fine. + /// + [Fact] + public async Task PagerDuty_ResolvesTheSameIncidentOnRecovery() + { + var (clients, handler) = Capture(HttpStatusCode.Accepted); + var sink = new PagerDutyAlertSink(clients, "routing-key-1"); + + await sink.SendAsync(Failing(), Ct); + var opened = Json(handler.Body).GetProperty("dedup_key").GetString(); + + await sink.SendAsync(Recovered(), Ct); + var root = Json(handler.Body); + + Assert.Equal("resolve", root.GetProperty("event_action").GetString()); + Assert.Equal(opened, root.GetProperty("dedup_key").GetString()); + + // A resolve carries no payload at all -- not one written as null. + Assert.False(root.TryGetProperty("payload", out _)); + } + + /// + /// Suspicious means "failing, but not past the threshold yet". Paging on it would defeat having + /// a threshold at all. + /// + [Theory] + [InlineData(PulseCheckerHealth.Unhealthy, "critical")] + [InlineData(PulseCheckerHealth.Suspicious, "warning")] + public async Task PagerDuty_MapsHealthOntoItsOwnSeverityScale(PulseCheckerHealth health, string expected) + { + var (clients, handler) = Capture(HttpStatusCode.Accepted); + var sink = new PagerDutyAlertSink(clients, "routing-key-1"); + + await sink.SendAsync(Failing(health, PulseCheckerHealth.Healthy), Ct); + + Assert.Equal(expected, Json(handler.Body).GetProperty("payload").GetProperty("severity").GetString()); + } + + [Fact] + public async Task PagerDuty_PostsToTheEventsApiUnlessToldOtherwise() + { + var (clients, handler) = Capture(HttpStatusCode.Accepted); + + await new PagerDutyAlertSink(clients, "routing-key-1").SendAsync(Failing(), Ct); + Assert.Equal(PagerDutyAlertSink.EventsEndpoint, handler.Request!.RequestUri); + + var eu = new Uri("https://events.eu.pagerduty.com/v2/enqueue"); + await new PagerDutyAlertSink(clients, "routing-key-1", eu).SendAsync(Failing(), Ct); + Assert.Equal(eu, handler.Request!.RequestUri); + } + + [Fact] + public void PagerDuty_WithoutARoutingKey_SaysSoAtConstruction() + { + var (clients, _) = Capture(); + + Assert.Throws(() => new PagerDutyAlertSink(clients, " ")); + } + + // ---- Every sink ---------------------------------------------------------------------------- + + /// + /// Throwing is how a sink reports a failed delivery: the dispatcher catches it, logs it, and no + /// check is affected. A sink that swallowed the failure would report a delivery that never + /// happened. + /// + [Fact] + public async Task ARejectedDelivery_Throws() + { + var (clients, _) = Capture(HttpStatusCode.BadRequest); + + var sinks = new IAlertSink[] + { + new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")), + new MicrosoftTeamsAlertSink(clients, new Uri("https://prod.westeurope.logic.azure.test/workflows/abc")), + new PagerDutyAlertSink(clients, "routing-key-1"), + new WebhookAlertSink(clients, new Uri("https://example.test/hook")), + }; + + foreach (var sink in sinks) + { + await Assert.ThrowsAsync(() => sink.SendAsync(Failing(), Ct)); + } + } + + [Fact] + public async Task EverySink_PostsRatherThanAnythingElse() + { + var (clients, handler) = Capture(); + + foreach (var sink in new IAlertSink[] + { + new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")), + new MicrosoftTeamsAlertSink(clients, new Uri("https://prod.westeurope.logic.azure.test/workflows/abc")), + new WebhookAlertSink(clients, new Uri("https://example.test/hook")), + }) + { + await sink.SendAsync(Failing(), Ct); + + Assert.Equal(HttpMethod.Post, handler.Request!.Method); + Assert.Equal("application/json", handler.Request.Content!.Headers.ContentType!.MediaType); + } + } +} From 8704fe176fb7e46b26fdba8080f9a3a5342f2903 Mon Sep 17 00:00:00 2001 From: Ivan Vydrin Date: Wed, 29 Jul 2026 21:31:04 +0300 Subject: [PATCH 2/3] Stop a Coravel test failing whenever the machine is busy ASingleDueOccurrence_TriggersOnceEvenIfTicksKeepComing scheduled a checker on a one-millisecond period and then asserted that a second tick would not fire it. With that period the second tick is legitimately due after a single millisecond, so the test failed whenever anything put a millisecond between the two calls -- which says nothing at all about whether one occurrence can fire twice, the thing the test is named for. Its own comment described the arrangement it wanted, "due immediately, then not again for an hour", and the code underneath did not build that. The period is half a second now, and the wait is for that one occurrence to come due. The two ticks then have 500ms of slack between them rather than one. This is pre-existing and unrelated to the sinks; it is here because adding a test class changed how xUnit schedules the collections and made it fail about half the time on net10.0, which would otherwise have looked like the sinks breaking something. On main it fails roughly one run in two in isolation, and it is almost certainly the one unexplained red run earlier in this branch's history. Still catches what it is for: with the scheduler mutated to leave a consumed occurrence due, it goes red. --- .../Healthie.Tests.Unit/CoravelSchedulerTests.cs | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs b/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs index b8501fc..7cbaeba 100644 --- a/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs +++ b/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs @@ -81,14 +81,21 @@ public async Task ASingleDueOccurrence_TriggersOnceEvenIfTicksKeepComing() var scheduler = new CoravelPulseScheduler(); var checker = new FakePulseChecker("once-per-occurrence"); - await scheduler.ScheduleAsync(checker, PulseSchedule.Every(TimeSpan.FromHours(1)), Ct); + // Long enough that the occurrence this consumes cannot come round again between the two + // ticks below. A millisecond here made the second tick legitimately due, so the test failed + // whenever the machine was busy enough to put a millisecond between them -- which said + // nothing about whether one occurrence can fire twice. + var period = TimeSpan.FromMilliseconds(500); - // Due immediately, then not again for an hour. - await scheduler.ScheduleAsync(checker, PulseSchedule.Every(TimeSpan.FromMilliseconds(1)), Ct); - await Task.Delay(20, Ct); + await scheduler.ScheduleAsync(checker, PulseSchedule.Every(period), Ct); + + // Wait for that one occurrence to come due. + await Task.Delay(period + TimeSpan.FromMilliseconds(50), Ct); await scheduler.TickAsync(Ct); var afterFirst = checker.TriggerCount; + + // Straight away again: the occurrence is spent and the next is half a second off. await scheduler.TickAsync(Ct); Assert.Equal(1, afterFirst); From 14d1eaecc209c32a595d528597f2396fb4274d72 Mon Sep 17 00:00:00 2001 From: Ivan Vydrin Date: Wed, 29 Jul 2026 22:05:34 +0300 Subject: [PATCH 3/3] Send colours Teams understands, and stop trusting the wall clock Review, checking the payloads against the services' own documentation rather than against these tests, found three things. The Adaptive Card colours were PascalCase. The schema's Colors enum is lowercase and matching is case-sensitive, and an unrecognised value is not an error: the card renders in the default colour and the POST still returns 2xx. Every Unhealthy and Suspicious alert would have lost its colour in Teams and nothing would ever have said so. The test asserted "Attention", so it pinned the bug rather than catching it. Confirmed against the published schema, which lists default|dark|light|accent|good|warning|attention. Slack parses the top-level text as mrkdwn, so &, < and > have to be escaped there. DisplayName is virtual and "Auth & Session" is an ordinary thing to call a checker. The attachment's fields are plain text unless mrkdwn_in says otherwise, and it does not, so only the headline needed it. PagerDuty's group was the one nullable field here without JsonIgnore(WhenWritingNull), so a checker with no group -- the common case -- sent "group": null while every other absent field was omitted. The UTC test was vacuous. It built its alert with DateTimeKind.Utc, and DateTimeOffset takes its offset from the Kind, so deleting the SpecifyKind call it was named after produced byte-identical output. It now uses an unmarked time, which is what a state round-tripped through a store that does not preserve Kind gives you, and which is the only case that tells the two apart. The Coravel test is deterministic rather than merely less flaky. Widening the window from one millisecond to five hundred made the failure rarer without changing its nature -- the second tick still raced the wall clock, and a long enough stall would still lose. The scheduler takes a TimeProvider now, optional so nothing about constructing it changes, and the test moves the clock by hand instead of sleeping. It asserts the next occurrence still arrives, which sleeping never did, and the suite spends 550ms less doing it. --- .../MicrosoftTeamsAlertSink.cs | 16 +++-- src/Healthie.Alerting/PagerDutyAlertSink.cs | 2 +- src/Healthie.Alerting/SlackAlertSink.cs | 21 ++++++- .../CoravelPulseScheduler.cs | 19 +++++- tests/Healthie.Tests.Unit/AlertSinkTests.cs | 58 ++++++++++++++++--- .../CoravelSchedulerTests.cs | 28 +++++---- tests/Healthie.Tests.Unit/TestDoubles.cs | 17 ++++++ 7 files changed, 133 insertions(+), 28 deletions(-) diff --git a/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs b/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs index bc45b6c..d221ce7 100644 --- a/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs +++ b/src/Healthie.Alerting/MicrosoftTeamsAlertSink.cs @@ -142,12 +142,20 @@ public static AdaptiveCard From(Alert alert) return new AdaptiveCard("AdaptiveCard", "http://adaptivecards.io/schemas/adaptive-card.json", "1.4", body); } - /// Adaptive Cards name their colours rather than taking hex. + /// + /// Adaptive Cards name their colours rather than taking hex. + /// + /// + /// Lower case, because the schema's Colors enum is + /// default|dark|light|accent|good|warning|attention and matching is case-sensitive. A + /// value outside it is not an error: the card renders in the default colour and the POST still + /// returns 2xx, so getting this wrong loses the colour on every alert and reports success. + /// private static string ColourOf(PulseCheckerHealth health) => health switch { - PulseCheckerHealth.Unhealthy => "Attention", - PulseCheckerHealth.Suspicious => "Warning", - PulseCheckerHealth.Healthy => "Good", + PulseCheckerHealth.Unhealthy => "attention", + PulseCheckerHealth.Suspicious => "warning", + PulseCheckerHealth.Healthy => "good", _ => throw new ArgumentOutOfRangeException(nameof(health), health, "Unknown health."), }; } diff --git a/src/Healthie.Alerting/PagerDutyAlertSink.cs b/src/Healthie.Alerting/PagerDutyAlertSink.cs index 31b2ce8..888deea 100644 --- a/src/Healthie.Alerting/PagerDutyAlertSink.cs +++ b/src/Healthie.Alerting/PagerDutyAlertSink.cs @@ -107,7 +107,7 @@ public sealed record PagerDutyEventPayload( [property: JsonPropertyName("severity")] string Severity, [property: JsonPropertyName("source")] string Source, [property: JsonPropertyName("component")] string Component, - [property: JsonPropertyName("group")] string? Group, + [property: JsonPropertyName("group"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Group, [property: JsonPropertyName("custom_details")] IReadOnlyDictionary CustomDetails) { /// Builds the payload for an alert. diff --git a/src/Healthie.Alerting/SlackAlertSink.cs b/src/Healthie.Alerting/SlackAlertSink.cs index 4871eda..f4a6b53 100644 --- a/src/Healthie.Alerting/SlackAlertSink.cs +++ b/src/Healthie.Alerting/SlackAlertSink.cs @@ -60,9 +60,13 @@ public static SlackMessage From(Alert alert) { ArgumentNullException.ThrowIfNull(alert); + // The display name is a consumer's own string, and DisplayName is virtual -- "Auth & Session" + // is an ordinary thing to call a checker. + var name = Escape(alert.DisplayName); + var headline = alert.IsRecovery - ? $"{alert.DisplayName} recovered" - : $"{alert.DisplayName} is {alert.CurrentHealth}"; + ? $"{name} recovered" + : $"{name} is {alert.CurrentHealth}"; var fields = new List { @@ -90,6 +94,19 @@ public static SlackMessage From(Alert alert) [new SlackAttachment(ColourOf(alert.CurrentHealth), alert.CheckerName, fields, ToUnixSeconds(alert.OccurredAt))]); } + /// + /// Escapes the three characters Slack reads as markup. + /// + /// + /// Only the top-level text needs this: Slack parses it as mrkdwn by default, so an + /// unescaped < starts what it takes for a link or a mention. The attachment's fields + /// are plain text unless mrkdwn_in asks otherwise, and it does not. + /// + private static string Escape(string text) => text + .Replace("&", "&", StringComparison.Ordinal) + .Replace("<", "<", StringComparison.Ordinal) + .Replace(">", ">", StringComparison.Ordinal); + /// Slack's own status colours, which it renders as the bar down the attachment. private static string ColourOf(PulseCheckerHealth health) => health switch { diff --git a/src/Healthie.Scheduling.Coravel/CoravelPulseScheduler.cs b/src/Healthie.Scheduling.Coravel/CoravelPulseScheduler.cs index 020e8b3..7c234df 100644 --- a/src/Healthie.Scheduling.Coravel/CoravelPulseScheduler.cs +++ b/src/Healthie.Scheduling.Coravel/CoravelPulseScheduler.cs @@ -25,8 +25,21 @@ namespace Healthie.Scheduling.Coravel; /// same thing without the dependency. /// /// -public sealed class CoravelPulseScheduler(ILogger? logger = null) : IPulseScheduler +public sealed class CoravelPulseScheduler( + ILogger? logger = null, + TimeProvider? timeProvider = null) : IPulseScheduler { + /// + /// Where "now" comes from. + /// + /// + /// Optional, so adding it does not change how this is constructed, and + /// is the wall clock this always used. A test can supply its + /// own instead of sleeping: due times are the whole of this class's behaviour, and a test that + /// waits for real milliseconds to pass is testing the machine's load as much as the scheduler. + /// + private readonly TimeProvider _time = timeProvider ?? TimeProvider.System; + private sealed record Entry(IPulseChecker Checker, PulseSchedule Schedule, CronExpression? Cron, DateTime DueAt); private readonly ConcurrentDictionary _scheduled = new(StringComparer.Ordinal); @@ -52,7 +65,7 @@ public Task ScheduleAsync( // is already running on a good one. var cron = schedule.IsCron ? ParseCron(schedule.CronExpression!, checker.Name) : null; - var due = NextDueAt(schedule, cron, DateTime.UtcNow); + var due = NextDueAt(schedule, cron, _time.GetUtcNow().UtcDateTime); if (due is null) { @@ -89,7 +102,7 @@ public Task UnscheduleAsync(IPulseChecker checker, CancellationToken cancellatio /// internal async Task TickAsync(CancellationToken cancellationToken = default) { - var now = DateTime.UtcNow; + var now = _time.GetUtcNow().UtcDateTime; foreach (var entry in _scheduled.Values) { diff --git a/tests/Healthie.Tests.Unit/AlertSinkTests.cs b/tests/Healthie.Tests.Unit/AlertSinkTests.cs index 2606803..17b22ab 100644 --- a/tests/Healthie.Tests.Unit/AlertSinkTests.cs +++ b/tests/Healthie.Tests.Unit/AlertSinkTests.cs @@ -67,7 +67,6 @@ private static (SingleClientFactory Clients, CapturingHandler Handler) Capture( private static JsonElement Json(string body) => JsonDocument.Parse(body).RootElement; - // ---- Slack --------------------------------------------------------------------------------- [Fact] public async Task Slack_SendsTheShapeAnIncomingWebhookAccepts() @@ -111,23 +110,68 @@ public async Task Slack_ColoursTheAttachmentByHealth(PulseCheckerHealth health, } /// - /// The alert's time is UTC. Read as anything else it is shifted by the server's offset, and the + /// An alert's time is UTC. Read as anything else it is shifted by the server's offset, and the /// message says a component failed at a time it did not. /// + /// + /// The time here carries , which is what makes this test + /// mean something: new DateTimeOffset(DateTime) takes its offset from the Kind, so a time + /// already marked Utc converts correctly whether or not the sink says so. Only an unmarked one + /// -- a state round-tripped through a store that does not preserve Kind -- tells the two apart. + /// [Fact] - public async Task Slack_TimestampsInUtcRegardlessOfTheServersZone() + public async Task Slack_TimestampsInUtcEvenWhenTheTimeDoesNotSaySo() { var (clients, handler) = Capture(); var sink = new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")); - await sink.SendAsync(Failing(), Ct); + var unmarked = Failing() with + { + OccurredAt = new DateTime(2026, 7, 29, 14, 30, 0, DateTimeKind.Unspecified), + }; + + await sink.SendAsync(unmarked, Ct); var expected = new DateTimeOffset(2026, 7, 29, 14, 30, 0, TimeSpan.Zero).ToUnixTimeSeconds(); Assert.Equal(expected, Json(handler.Body).GetProperty("attachments")[0].GetProperty("ts").GetInt64()); } - // ---- Microsoft Teams ----------------------------------------------------------------------- + /// + /// Slack parses the top-level text as mrkdwn, so an unescaped angle bracket starts what it takes + /// for a link. DisplayName is virtual and "Auth & Session" is an ordinary name to give a checker. + /// + [Fact] + public async Task Slack_EscapesTheCharactersItWouldOtherwiseReadAsMarkup() + { + var (clients, handler) = Capture(); + var sink = new SlackAlertSink(clients, new Uri("https://hooks.slack.test/services/abc")); + + var awkward = Failing() with { DisplayName = "Auth & " }; + + await sink.SendAsync(awkward, Ct); + + var text = Json(handler.Body).GetProperty("text").GetString()!; + + Assert.Contains("Auth & <Session>", text, StringComparison.Ordinal); + Assert.DoesNotContain("", text, StringComparison.Ordinal); + } + + /// + /// A checker with no group must not send "group": null -- every other optional field here + /// is omitted when it is absent, and this one was not. + /// + [Fact] + public async Task PagerDuty_OmitsTheGroupWhenTheCheckerHasNone() + { + var (clients, handler) = Capture(HttpStatusCode.Accepted); + var sink = new PagerDutyAlertSink(clients, "routing-key-1"); + + await sink.SendAsync(Failing() with { Group = null }, Ct); + + Assert.False(Json(handler.Body).GetProperty("payload").TryGetProperty("group", out _)); + } + [Fact] public async Task Teams_SendsAnAdaptiveCardInTheEnvelopeWorkflowsExpects() @@ -151,7 +195,7 @@ public async Task Teams_SendsAnAdaptiveCardInTheEnvelopeWorkflowsExpects() var body = card.GetProperty("body").EnumerateArray().ToList(); Assert.Equal("TextBlock", body[0].GetProperty("type").GetString()); - Assert.Equal("Attention", body[0].GetProperty("color").GetString()); + Assert.Equal("attention", body[0].GetProperty("color").GetString()); Assert.Contains("Payments API", body[0].GetProperty("text").GetString()!, StringComparison.Ordinal); Assert.Equal("FactSet", body[1].GetProperty("type").GetString()); @@ -188,7 +232,6 @@ public async Task Teams_OmitsOptionalCardPropertiesRatherThanWritingThemNull() Assert.True(message.GetProperty("wrap").GetBoolean()); } - // ---- PagerDuty ----------------------------------------------------------------------------- [Fact] public async Task PagerDuty_TriggersAnIncidentKeyedOnTheChecker() @@ -273,7 +316,6 @@ public void PagerDuty_WithoutARoutingKey_SaysSoAtConstruction() Assert.Throws(() => new PagerDutyAlertSink(clients, " ")); } - // ---- Every sink ---------------------------------------------------------------------------- /// /// Throwing is how a sink reports a failed delivery: the dispatcher catches it, logs it, and no diff --git a/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs b/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs index 7cbaeba..1fec1be 100644 --- a/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs +++ b/tests/Healthie.Tests.Unit/CoravelSchedulerTests.cs @@ -78,28 +78,36 @@ public async Task ReschedulingAChecker_ReplacesItsScheduleRatherThanAddingOne() [Fact] public async Task ASingleDueOccurrence_TriggersOnceEvenIfTicksKeepComing() { - var scheduler = new CoravelPulseScheduler(); + // The clock is controlled, so this asks the question directly instead of sleeping and hoping. + // It used to schedule a one-millisecond period and assert a second tick would not fire -- + // but with that period the second tick is legitimately due after a millisecond, so it failed + // whenever the machine was busy enough to put one between the two calls, which says nothing + // about whether a single occurrence can fire twice. + var clock = new SettableTimeProvider(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)); + var scheduler = new CoravelPulseScheduler(timeProvider: clock); var checker = new FakePulseChecker("once-per-occurrence"); - // Long enough that the occurrence this consumes cannot come round again between the two - // ticks below. A millisecond here made the second tick legitimately due, so the test failed - // whenever the machine was busy enough to put a millisecond between them -- which said - // nothing about whether one occurrence can fire twice. - var period = TimeSpan.FromMilliseconds(500); - + var period = TimeSpan.FromMinutes(1); await scheduler.ScheduleAsync(checker, PulseSchedule.Every(period), Ct); - // Wait for that one occurrence to come due. - await Task.Delay(period + TimeSpan.FromMilliseconds(50), Ct); + // Exactly one occurrence has come due. + clock.Advance(period); await scheduler.TickAsync(Ct); var afterFirst = checker.TriggerCount; - // Straight away again: the occurrence is spent and the next is half a second off. + // Ticks keep coming, and the clock has not moved: the occurrence is spent. + await scheduler.TickAsync(Ct); await scheduler.TickAsync(Ct); Assert.Equal(1, afterFirst); Assert.Equal(1, checker.TriggerCount); + + // And the next occurrence still arrives when it should. + clock.Advance(period); + await scheduler.TickAsync(Ct); + + Assert.Equal(2, checker.TriggerCount); } /// One checker throwing must not stop the rest of the tick. diff --git a/tests/Healthie.Tests.Unit/TestDoubles.cs b/tests/Healthie.Tests.Unit/TestDoubles.cs index ca58e9e..e95debc 100644 --- a/tests/Healthie.Tests.Unit/TestDoubles.cs +++ b/tests/Healthie.Tests.Unit/TestDoubles.cs @@ -182,3 +182,20 @@ public Task ScheduleAsync(IPulseChecker checker, PulseInterval interval, Cancell public Task UnscheduleAsync(IPulseChecker checker, CancellationToken cancellationToken = default) => Task.CompletedTask; } + +/// +/// A clock a test moves by hand. +/// +/// +/// Only is overridden, because that is all the schedulers read. A test that +/// waits for real time to pass is testing the machine's load as much as the code. +/// +internal sealed class SettableTimeProvider(DateTimeOffset now) : TimeProvider +{ + private DateTimeOffset _now = now; + + public override DateTimeOffset GetUtcNow() => _now; + + /// Moves the clock forward. + public void Advance(TimeSpan by) => _now = _now.Add(by); +}