diff --git a/CHANGELOG.md b/CHANGELOG.md index 10f1de6..45b95eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,37 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] -Eleven new packages, and the schedule model that several of them needed. Everything here is -additive: no public API was removed or changed, and an application that upgrades without touching -its code behaves exactly as it did. +## [4.0.0] - 2026-07-30 -### Fixed +Twelve new packages, the schedule model several of them needed, and optimistic concurrency on the +state contract. -- **The dashboard's own page did not encode its title.** `MapHealthieUI` builds that one page as a - string rather than through Razor, which encodes every interpolation for you, so a - `HealthieUIOptions.DashboardTitle` built from anything the host did not write itself could close - the title element and open a script one. -- **A dashboard component left its handler behind when it was disposed.** The service is scoped to a - circuit, so it was assumed the circuit ending released everything; that holds only while the - dashboard is mounted once per circuit, and a host that routes to it inside its own layout builds a - new one every time the user navigates back. `IHealthieDashboardService.UnsubscribeFromStateChangesAsync` - is the missing half, and the component calls it. -- **Two callers scheduling one checker at once could leave a timer nobody could stop.** Installing a - schedule was "cancel the old one, then start the new one", and only the last one stored was - reachable; the other kept triggering, could not be unscheduled, and held a linked - `CancellationTokenSource` that was never disposed. -- **`HealthieMcpOptions.MaxHistoryPageSize` was documented and never read.** `get_check_history` - clamped to a hard-coded 200 instead, so a host that lowered the option got no such thing. -- **The relational table-name guard admitted a name with a trailing newline.** In .NET `$` matches - immediately before one, so a table name ending in one passed a check whose own error message says - it does not. Nothing could be smuggled through it -- a lone trailing newline is whitespace to - every engine here -- but the guard now ends at `\z` and means what it says. -- **The release workflow granted `contents: write` to every job in it.** Only the one job that - creates the GitHub release needs it, and the top level now grants nothing but read, so a job added - later starts with no write access rather than inheriting it. -- **A checker name from a REST route reached the log with its control characters intact.** The - not-found branch logs a name precisely when it matches nothing, percent-encoded CR and LF arrive - decoded, and a log sink writing plain text writes them as line breaks. +**Source-compatible: an application that upgrades without touching its code still compiles and still +behaves as it did.** Nothing public was removed or renamed, and every new interface member is a +defaulted one, so a provider or scheduler written against 3.x keeps working. Two things to know +before upgrading: + +- **One binary break.** `HealthieTools`, the MCP read-only tool class, gained a defaulted + `HealthieMcpOptions?` parameter on its constructor so it could read the page-size option it had + been ignoring. Calling code still compiles unchanged; an assembly compiled against 3.1.4 and not + rebuilt does not. Recompiling is enough, and that break is why this is a major rather than a minor. +- **The relational providers alter an existing table on startup** to add the version column + optimistic concurrency needs. See *Optimistic concurrency* below. ### Added @@ -128,8 +113,111 @@ its code behaves exactly as it did. - **State removal.** `IStateProvider.DeleteStateAsync` cleans up after a checker that was renamed or removed. Defaulted to refuse rather than to silently do nothing. +- **The dashboard shows what the feature packages know.** Installing one used to change what the + application did and nothing about the one screen an operator looks at, so uptime, alerts and + leadership were only visible to whatever the host wired up itself. Each now has a small read-only + contract in `Healthie.NET.Abstractions` that its package implements, and the board renders the + panel when the container can resolve it: uptime over the last day and the longest outage in it + beside the run-based percentage, a drawer of recent alerts saying which reached their sinks and + which did not, a badge saying whether this replica is the one running the checks, and a button + that asks the model why a checker has been failing. An application that installs none of them gets + the board exactly as it was. + + Read-only throughout, so all of it shows under `HealthieUIOptions.AllowMutations = false` -- the + one exception is asking the model, which is still a read but spends money on the host's account, + and is gated with the controls that change things. +- **The alert history is persisted and paged.** It is written through the application's own + `IStateProvider`, so a deployment on CosmosDB, PostgreSQL, SQL Server, SQLite or Redis keeps its + alerts across a redeploy and one left on the in-memory provider does not. There is no second + storage contract to configure and no provider had to learn about alerts. `IAlertInsights` reads a + page at a time and reports the total, because the question asked most often is asked just after a + restart, about what happened before it -- which is exactly what a last-twenty list throws away. + Bounded at `HistoryLength`, and the view says how many are kept, of what cap, in which provider. +- **The dashboard shows where alerts are delivered**, with each sink's delivered and failed counts + and its last error. An application that installed alerting and never configured a sink raises + alerts that reach nobody, and a healthy-looking list of them was indistinguishable from one that + was notifying people; sinks are now listed from startup rather than from their first delivery, and + one that recovers stops being reported as failing. +- **Alerting is configurable from the dashboard**: minimum severity, deduplication window, delivery + timeout, and whether recoveries alert. Those four and no others, because they are the ones the + dispatcher reads on every alert -- its queue capacity and history length are fixed when it is + built, so they are shown as facts rather than offered as controls that would quietly do nothing. + There is also a test alert, which goes through the real sinks: the only way to find out that a + webhook URL is wrong is to use it. +- **`AddHealthieMetrics()`** collects the library's own instruments in-process behind + `IMetricsInsights`, and the dashboard grows a metrics view: checks run, the share that reported + healthy, transitions, mean and slowest check duration, and results by health. A `MeterListener` on + the `Healthie.NET` meter, so it reads what is already being emitted and runs alongside an + OpenTelemetry exporter rather than instead of one. Opt-in, because a listener costs a callback on + every measurement and an application with an APM has somewhere better to look. + + Overlapped triggers get their own figure. A checker whose check outlasts its own interval looks + healthy and is quietly running at a fraction of the rate it was asked to, and this is the only + place that shows. +- **A side menu on the dashboard**, listing an overview, a section per group with its tally and worst + state, and the alerts, metrics, event-log and about views. Picking a group narrows the list and picking it + again is the way back; it combines with the search box and the tag filter rather than clearing + them, and the groups it lists come from the store rather than from the rows currently surviving + those filters -- a menu that dropped a group because a search had narrowed it away would take away + the means of getting back to it. + + A rail rather than an overlay drawer, and collapsing to its icons rather than disappearing. This + package ships no JavaScript of its own, and an off-canvas drawer needs a focus trap to be honest + about keyboard use. Below 820px it becomes a scrolling strip above the list. +- **The dashboard opens sectioned by group** rather than as one flat list. A group is a partition, so + the sectioned view answers what is wrong and where at a glance; a flat list of forty checkers asks + the reader to do that grouping themselves. Checkers with no group collect under one heading, so the + default hides nothing. The `GROUP` button still switches to the flat list. +- **A cron expression can be set from the dashboard**, beside the interval picker, and through + `PUT /healthie/{checkerName}/schedule` on the REST API. The scheduler judges the expression before + anything is stored -- `IPulseScheduler.TryValidateSchedule`, defaulted to accept so an existing + scheduler is unaffected -- because Cronos, Quartz and Temporal do not agree on cron dialects and + the only answer worth having is from the implementation that will run it. A refusal carries that + implementation's own reason and leaves the stored schedule alone; storing first and failing on the + reschedule would leave a checker that no longer runs and a store that says it should. + + `IPulseChecker.SetScheduleAsync` and `IPulsesScheduler.SetScheduleAsync` are the API behind it. A + schedule an interval can express exactly is stored as that interval, so only a genuinely custom + period or a cron expression occupies `PulseCheckerState.Schedule`. + ### Fixed +- **The dashboard ignored `PulseSchedule` everywhere it showed a cadence.** It read + `PulseCheckerState.Interval` for the rate column, for the aggregate checks-per-minute, and for the + interval picker -- and that field is documented as ignored once `Schedule` is set. So a checker on + a cron expression advertised a rate it was not running at, was summed into the aggregate at that + rate, and offered a picker whose every change stored a field nothing reads. The board now reads + `EffectiveSchedule`, shows a cron expression as one, counts cron checkers separately rather than + inventing a rate for them, and disables the picker while an expression is in force. +- **`SetIntervalAsync` did nothing to a checker that had a schedule.** The schedule overrides the + interval, so choosing one left the checker at its old cadence with nothing to say so. It now clears + the schedule, which is what choosing an interval means. +- **The dashboard's own page did not encode its title.** `MapHealthieUI` builds that one page as a + string rather than through Razor, which encodes every interpolation for you, so a + `HealthieUIOptions.DashboardTitle` built from anything the host did not write itself could close + the title element and open a script one. +- **A dashboard component left its handler behind when it was disposed.** The service is scoped to a + circuit, so it was assumed the circuit ending released everything; that holds only while the + dashboard is mounted once per circuit, and a host that routes to it inside its own layout builds a + new one every time the user navigates back. `IHealthieDashboardService.UnsubscribeFromStateChangesAsync` + is the missing half, and the component calls it. +- **Two callers scheduling one checker at once could leave a timer nobody could stop.** Installing a + schedule was "cancel the old one, then start the new one", and only the last one stored was + reachable; the other kept triggering, could not be unscheduled, and held a linked + `CancellationTokenSource` that was never disposed. +- **`HealthieMcpOptions.MaxHistoryPageSize` was documented and never read.** `get_check_history` + clamped to a hard-coded 200 instead, so a host that lowered the option got no such thing. +- **The relational table-name guard admitted a name with a trailing newline.** In .NET `$` matches + immediately before one, so a table name ending in one passed a check whose own error message says + it does not. Nothing could be smuggled through it -- a lone trailing newline is whitespace to + every engine here -- but the guard now ends at `\z` and means what it says. +- **The release workflow granted `contents: write` to every job in it.** Only the one job that + creates the GitHub release needs it, and the top level now grants nothing but read, so a job added + later starts with no write access rather than inheriting it. +- **A checker name from a REST route reached the log with its control characters intact.** The + not-found branch logs a name precisely when it matches nothing, percent-encoded CR and LF arrive + decoded, and a log sink writing plain text writes them as line breaks. + - A cron schedule whose next occurrence was more than about fifty days out **stopped the checker permanently and silently**. `Task.Delay` refuses a longer wait and throws, and that throw is not a cancellation, so the scheduler's loop ended with nothing logged. An annual certificate check @@ -432,6 +520,7 @@ Blazor dashboard, and the CosmosDB and Quartz.NET providers. - Dashboard UI improvements and additional sample pulse checkers. -[Unreleased]: https://github.com/ivanvyd/Healthie.NET/compare/v3.1.4...HEAD +[Unreleased]: https://github.com/ivanvyd/Healthie.NET/compare/v4.0.0...HEAD +[4.0.0]: https://github.com/ivanvyd/Healthie.NET/compare/v3.1.4...v4.0.0 [3.0.0]: https://github.com/ivanvyd/Healthie.NET/compare/v2.3.0...v3.0.0 [2.3.0]: https://github.com/ivanvyd/Healthie.NET/releases/tag/v2.3.0 diff --git a/README.md b/README.md index 2af2d89..d495598 100644 --- a/README.md +++ b/README.md @@ -446,6 +446,7 @@ The `Healthie.NET.Dashboard` package is a pulse monitor for your services, shipp - **Aggregate pulse trace** -- an EKG across the header, with the combined checks-per-minute of everything you monitor. - **A row per checker** -- status light, name, checks per minute, and a pulse strip of the last N runs, one blip per run, coloured by the health it reported. - **Detail panel** -- select a checker for its uptime, failure streak, last message, and its interval, failure threshold, group and tags, all editable in place. +- **A panel per feature package you install** -- 24-hour uptime and longest outage from `Healthie.NET.Uptime`, a recent-alerts drawer from `Healthie.NET.Alerting`, a replica badge from `Healthie.NET.LeaderElection`, and an `EXPLAIN` button from `Healthie.NET.AI`. Nothing to configure: the panel appears because the package is registered, and an application that installs none of them gets the board unchanged. All read-only, so all of it survives `AllowMutations = false` -- bar `EXPLAIN`, which spends money on your account. - **Groups and tags** -- a checker sits in at most one group and carries any number of tags. Sections the list by group, filters it by tag. Both are declared in code and can be changed here; see [Groups and tags](#groups-and-tags). - **Pin** -- keep the checkers worth watching at the top. A pin is shared, not personal. - **Rows or cards** -- the same list laid out either way, flat or sectioned by group with per-group healthy/suspicious/failing tallies. @@ -814,6 +815,8 @@ Both are development-time only. Nothing under `src/` depends on Aspire or Docker ## Migration +Upgrading from v3.x? See the [v3 to v4 migration guide](https://github.com/ivanvyd/Healthie.NET/blob/main/docs/migration-v3-to-v4.md). 4.0.0 is source-compatible: rebuild and you are done. + Upgrading from v2.x? See the [v2 to v3 migration guide](https://github.com/ivanvyd/Healthie.NET/blob/main/docs/migration-v2-to-v3.md). Most applications need no code changes. Upgrading from v1.x? See the [v1 to v2 migration guide](https://github.com/ivanvyd/Healthie.NET/blob/main/docs/migration-v1-to-v2.md) first. @@ -822,11 +825,12 @@ Upgrading from v1.x? See the [v1 to v2 migration guide](https://github.com/ivanv ## Roadmap -Shipped since 3.1.4: alerting on transitions, OpenTelemetry metrics and traces, arbitrary intervals +Shipped in 4.0.0: alerting on transitions, OpenTelemetry metrics and traces, arbitrary intervals and cron, PostgreSQL / SQL Server / SQLite state providers, Hangfire / Coravel / Temporal scheduling, ready-made checkers, uptime reporting, leader election, optimistic concurrency on `IStateProvider`, `HealthChanged` on the state-changed event, Slack / Teams / PagerDuty alert -sinks, and a Redis state provider. +sinks, a Redis state provider, and a dashboard that surfaces each of those packages as it is +installed. Every item that was on this list has shipped. Two open questions are decisions rather than features, and both are deliberate as they stand: diff --git a/docs/migration-v3-to-v4.md b/docs/migration-v3-to-v4.md new file mode 100644 index 0000000..bbb8189 --- /dev/null +++ b/docs/migration-v3-to-v4.md @@ -0,0 +1,59 @@ +# Migrating from v3.x to v4.0 + +Short guide, because there is little to do. 4.0.0 is a large release — twelve new packages — but it is +source-compatible: **an application that upgrades without touching its code still compiles and still +behaves as it did.** The major number is here for one binary break and one startup behaviour, both +below. + +## The one binary break + +`HealthieTools`, the read-only MCP tool class in `Healthie.NET.Mcp`, gained a defaulted parameter on +its constructor so it could read the page-size option it had been ignoring: + +```csharp +// 3.x +public HealthieTools(IPulsesScheduler pulsesScheduler) + +// 4.0 +public HealthieTools(IPulsesScheduler pulsesScheduler, HealthieMcpOptions? options = null) +``` + +Calling code compiles unchanged. An assembly compiled against 3.1.4 and **not rebuilt** will fail to +find the old constructor at runtime. Rebuilding is the whole fix, and almost nobody constructs this +type by hand — it is resolved by the MCP server. + +## The relational providers alter your table on startup + +`Healthie.NET.Postgres`, `Healthie.NET.SqlServer` and `Healthie.NET.Sqlite` add a `version` column to +their state table the first time they start against a table created before it existed. This is what +optimistic concurrency writes against. The initializer checks whether the column is there before +touching anything, so restarting is safe and repeating it is a no-op. + +If your database user cannot `ALTER TABLE`, add the column yourself before deploying — the providers +log what they wanted and carry on without concurrency rather than failing startup. + +## Things that changed shape but not behaviour + +| Area | What happened | Do you need to act? | +|---|---|---| +| `IStateProvider` | Gained optimistic-concurrency members, all defaulted | No. A provider written against 3.x compiles and reports `SupportsOptimisticConcurrency == false` | +| `IPulseChecker` | Gained `SetScheduleAsync`, defaulted | No, unless you implement the interface directly rather than deriving from `PulseChecker` | +| `IPulseScheduler` | Gained `TryValidateSchedule`, defaulted to accept | No | +| `PulseChecker.SetIntervalAsync` | Now clears any `Schedule` | Only if you relied on setting an interval having no effect on a cron-scheduled checker, which was a bug | +| Dashboard | Opens sectioned by group rather than flat | No. The `GROUP` button still switches to the flat list | + +## Worth turning on while you are here + +None of these is required, and none changes anything until you add it. + +```csharp +builder.Services + .AddHealthieAlerts() // health changes become alerts; add a sink to deliver them + .AddHealthieUptime() // uptime over real time, not just the rolling history + .AddHealthieMetrics(); // the library's own meter, read in-process for the dashboard +``` + +Each one adds a panel or a view to the dashboard on its own — there is no dashboard configuration to +match. See the [dashboard README](https://github.com/ivanvyd/Healthie.NET/blob/main/src/Healthie.Dashboard/README.md). + +The full list of what landed is in the [changelog](https://github.com/ivanvyd/Healthie.NET/blob/main/CHANGELOG.md). diff --git a/samples/Healthie.Sample.BlazorUI/Healthie.Sample.BlazorUI.csproj b/samples/Healthie.Sample.BlazorUI/Healthie.Sample.BlazorUI.csproj index 05d1a53..23caa16 100644 --- a/samples/Healthie.Sample.BlazorUI/Healthie.Sample.BlazorUI.csproj +++ b/samples/Healthie.Sample.BlazorUI/Healthie.Sample.BlazorUI.csproj @@ -13,6 +13,9 @@ + + + diff --git a/samples/Healthie.Sample.BlazorUI/Program.cs b/samples/Healthie.Sample.BlazorUI/Program.cs index 7a2b6b3..884432c 100644 --- a/samples/Healthie.Sample.BlazorUI/Program.cs +++ b/samples/Healthie.Sample.BlazorUI/Program.cs @@ -1,8 +1,12 @@ -using Healthie.DependencyInjection; +using Healthie.DependencyInjection; using Healthie.Sample.BlazorUI.Components; using Healthie.Scheduling.Quartz; using Healthie.StateProviding.CosmosDb; +using Healthie.Abstractions.Enums; +using Healthie.Alerting; using Healthie.Dashboard; +using Healthie.LeaderElection; +using Healthie.Uptime; using Microsoft.Azure.Cosmos; var builder = WebApplication.CreateBuilder(args); @@ -23,6 +27,28 @@ options.AllowMutations = builder.Configuration.GetValue("Healthie:AllowMutations", true); }); +// The dashboard shows a panel per feature package the application installs, and shows none of them +// otherwise. Installing them here is what makes this sample demonstrate the whole board rather than +// the core of it. +builder.Services + .AddHealthieAlerts(options => + { + // Suspicious as well as unhealthy, and a short window, so the alerts panel actually has + // something in it while somebody is looking at the sample. + options.MinimumSeverity = PulseCheckerHealth.Suspicious; + options.DeduplicationWindow = TimeSpan.FromSeconds(20); + }) + .AddHealthieUptime() + // Reads the library's own meter in-process, so the board can show what it has counted. + .AddHealthieMetrics(); + +// Leader election off by default: with one replica it is always the leader, and the badge would be +// a permanent reassurance about a problem nobody has. Healthie:LeaderElection=true shows it. +if (builder.Configuration.GetValue("Healthie:LeaderElection", false)) +{ + builder.Services.AddHealthieLeaderElection(); +} + // Swap the scheduler with Healthie:Scheduler=Quartz. AddHealthie has already registered the // built-in timer, and the last registration wins, so this overrides it. if (string.Equals(builder.Configuration["Healthie:Scheduler"], "Quartz", StringComparison.OrdinalIgnoreCase)) diff --git a/samples/Healthie.Sample.BlazorUI/Pulses/CertificateExpiryPulseChecker.cs b/samples/Healthie.Sample.BlazorUI/Pulses/CertificateExpiryPulseChecker.cs index e9fbdf4..381fbe2 100644 --- a/samples/Healthie.Sample.BlazorUI/Pulses/CertificateExpiryPulseChecker.cs +++ b/samples/Healthie.Sample.BlazorUI/Pulses/CertificateExpiryPulseChecker.cs @@ -1,6 +1,7 @@ using Healthie.Abstractions; using Healthie.Abstractions.Enums; using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; using Healthie.Abstractions.StateProviding; namespace Healthie.Sample.BlazorUI.Pulses; @@ -9,8 +10,12 @@ public class CertificateExpiryPulseChecker : PulseChecker { private readonly Random _random = new(); + // On a cron expression rather than an interval, because that is what a certificate check + // actually wants and because it is the case the board has to render: no rate a minute, and an + // interval picker that must not pretend to apply. Every two minutes here so the sample shows it + // running rather than waiting until 03:20. public CertificateExpiryPulseChecker(IStateProvider stateProvider) - : base(stateProvider, PulseInterval.Every2Minutes, 1) + : base(stateProvider, PulseSchedule.Cron("*/2 * * * *"), 1) { } diff --git a/samples/Healthie.Sample.BlazorUI/packages.lock.json b/samples/Healthie.Sample.BlazorUI/packages.lock.json index 4a5073c..c55b3da 100644 --- a/samples/Healthie.Sample.BlazorUI/packages.lock.json +++ b/samples/Healthie.Sample.BlazorUI/packages.lock.json @@ -33,21 +33,50 @@ "resolved": "1.1.0", "contentHash": "J2G1k+u5unBV+aYcwxo94ip16Rkp65pgWFb0R6zwJipzWNMgvqlWeuI7/+R+e8bob66LnSG+llLJ+z8wI94cHg==" }, + "Microsoft.Extensions.Configuration": { + "type": "Transitive", + "resolved": "10.0.10", + "contentHash": "plJWK2zpWuuyxI8F8s2scx6Je7N1Ajjs6HvYUGKwRnDMWIVIz9FHwAkiT7ASgrvAOd10T0FPVlh9BzAJJME+jg==", + "dependencies": { + "Microsoft.Extensions.Configuration.Abstractions": "10.0.10", + "Microsoft.Extensions.Primitives": "10.0.10" + } + }, "Microsoft.Extensions.Configuration.Abstractions": { "type": "Transitive", - "resolved": "8.0.0", - "contentHash": "3lE/iLSutpgX1CC0NOW70FJoGARRHbyKmG7dc0klnUZ9Dd9hS6N/POPWhKhMLCEuNN5nXEY5agmlFtH562vqhQ==", + "resolved": "10.0.10", + "contentHash": "5Vnd2I75DmZCVEjSynIdJ/0EGafgnLQwgR3t2C2/fkjx/nRG+cLwxLLdInoHeCEpkD5K4Ov/g9ZCRYrl4TRsaA==", "dependencies": { - "Microsoft.Extensions.Primitives": "8.0.0" + "Microsoft.Extensions.Primitives": "10.0.10" + } + }, + "Microsoft.Extensions.Configuration.Binder": { + "type": "Transitive", + "resolved": "10.0.10", + "contentHash": "GqmN2o1CkJvk7uWp+p4CwBYW0w/zfoEbvsiFDbO2G8l1Uz+mrDAbAcZiXhU2lufKPby1cjAUdd5GTWpebYOkOA==", + "dependencies": { + "Microsoft.Extensions.Configuration": "10.0.10", + "Microsoft.Extensions.Configuration.Abstractions": "10.0.10" + } + }, + "Microsoft.Extensions.Diagnostics": { + "type": "Transitive", + "resolved": "10.0.10", + "contentHash": "Kr/e7lUf4+N8tacbqJ2Ctwe/HarKdAc9ZkgKVVqvtJDBKbez+T/KnUwu82KSlnBp/SrpBcxc7u7xkE2oUZT/5Q==", + "dependencies": { + "Microsoft.Extensions.Configuration": "10.0.10", + "Microsoft.Extensions.Diagnostics.Abstractions": "10.0.10", + "Microsoft.Extensions.Options.ConfigurationExtensions": "10.0.10" } }, "Microsoft.Extensions.Diagnostics.Abstractions": { "type": "Transitive", - "resolved": "8.0.1", - "contentHash": "elH2vmwNmsXuKmUeMQ4YW9ldXiF+gSGDgg1vORksob5POnpaI6caj1Hu8zaYbEuibhqCoWg0YRWDazBY3zjBfg==", + "resolved": "10.0.10", + "contentHash": "9uWiKpeOVac355STyChWR/pliFX/5CeLqChW9kKsaxyDH4EUTZxMkT4Jwp/J/peLm0GBFmSX5c0WCse3yCnq1Q==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "8.0.2", - "Microsoft.Extensions.Options": "8.0.2" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10", + "Microsoft.Extensions.Options": "10.0.10", + "System.Diagnostics.DiagnosticSource": "10.0.10" } }, "Microsoft.Extensions.FileProviders.Abstractions": { @@ -58,27 +87,50 @@ "Microsoft.Extensions.Primitives": "8.0.0" } }, + "Microsoft.Extensions.Logging": { + "type": "Transitive", + "resolved": "10.0.10", + "contentHash": "Tf6z5HsL0VDYRTfvsoNrTGHGheCwkTsZBA2FFh5ATJUbkAwug+FFNISJK2gjpUNemlAOoWllAK52HOWCjto3EQ==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection": "10.0.10", + "Microsoft.Extensions.Logging.Abstractions": "10.0.10", + "Microsoft.Extensions.Options": "10.0.10" + } + }, "Microsoft.Extensions.Logging.Abstractions": { "type": "Transitive", - "resolved": "8.0.3", - "contentHash": "dL0QGToTxggRLMYY4ZYX5AMwBb+byQBd/5dMiZE07Nv73o6I5Are3C7eQTh7K2+A4ct0PVISSr7TZANbiNb2yQ==", + "resolved": "10.0.10", + "contentHash": "zkFxGYUvdxAvIKTyXHrmW+Sux53D4SezD9dMyZ6hrwwzPQJNuwCRy1f5W7AvYTqacEGhWF2XderRQG1OvbV8og==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "8.0.2" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10", + "System.Diagnostics.DiagnosticSource": "10.0.10" } }, "Microsoft.Extensions.Options": { "type": "Transitive", - "resolved": "8.0.2", - "contentHash": "dWGKvhFybsaZpGmzkGCbNNwBD1rVlWzrZKANLW/CcbFJpCEceMCGzT7zZwHOGBCbwM0SzBuceMj5HN1LKV1QqA==", + "resolved": "10.0.10", + "contentHash": "srnhnk7nE8krBiIXp71LvBmKBtraBONWSRzdjJgRv1Ko9Mp8IVNqv4vIS9hGeVteBig8aQkva9ZG+sC+o5sVcA==", "dependencies": { - "Microsoft.Extensions.DependencyInjection.Abstractions": "8.0.0", - "Microsoft.Extensions.Primitives": "8.0.0" + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10", + "Microsoft.Extensions.Primitives": "10.0.10" + } + }, + "Microsoft.Extensions.Options.ConfigurationExtensions": { + "type": "Transitive", + "resolved": "10.0.10", + "contentHash": "tnBmu/LwF25ZQK+HBNCu2xrwnkKoB/XEbJyooGGoYxHrhvxbSKi7eOFiJ4AXBy/QU4vtCvCJfoi8k9Ej72qzOQ==", + "dependencies": { + "Microsoft.Extensions.Configuration.Abstractions": "10.0.10", + "Microsoft.Extensions.Configuration.Binder": "10.0.10", + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10", + "Microsoft.Extensions.Options": "10.0.10", + "Microsoft.Extensions.Primitives": "10.0.10" } }, "Microsoft.Extensions.Primitives": { "type": "Transitive", - "resolved": "8.0.0", - "contentHash": "bXJEZrW9ny8vjMF1JV253WeLhpEVzFo1lyaZu1vQ4ZxWUlVvknZ/+ftFgVheLubb4eZPSwwxBeqS1JkCOjxd8g==" + "resolved": "10.0.10", + "contentHash": "5wu/GrYVd8mG2DVUw3vFJzF+O336TyTGg/Kmcgw9bfwYhCoFiV5lR5QeEmKecJyrW4W54nMfD3p3589E8a7czQ==" }, "Microsoft.Win32.SystemEvents": { "type": "Transitive", @@ -123,8 +175,8 @@ }, "System.Diagnostics.DiagnosticSource": { "type": "Transitive", - "resolved": "8.0.1", - "contentHash": "vaoWjvkG1aenR2XdjaVivlCV9fADfgyhW5bZtXT23qaEea0lWiUljdQuze4E31vKM7ZWJaSUsbYIKE3rnzfZUg==" + "resolved": "10.0.10", + "contentHash": "cjtKi6ERMYWp6b9UTVPcwDT29PjKDtlM3W9OwnWL5abRsI8ku42Q2wqZoLIIXJnT/XF2s2CjuK8Nl4a3mmTxQQ==" }, "System.Drawing.Common": { "type": "Transitive", @@ -217,6 +269,13 @@ "Microsoft.Extensions.Hosting.Abstractions": "[8.0.1, )" } }, + "Healthie.NET.Alerting": { + "type": "Project", + "dependencies": { + "Healthie.NET.Abstractions": "[1.0.0, )", + "Microsoft.Extensions.Http": "[10.0.10, )" + } + }, "Healthie.NET.CosmosDb": { "type": "Project", "dependencies": { @@ -238,6 +297,12 @@ "Microsoft.Extensions.Diagnostics.HealthChecks": "[8.0.29, )" } }, + "Healthie.NET.LeaderElection": { + "type": "Project", + "dependencies": { + "Healthie.NET.Abstractions": "[1.0.0, )" + } + }, "Healthie.NET.Quartz": { "type": "Project", "dependencies": { @@ -247,6 +312,12 @@ "Quartz.Extensions.Hosting": "[3.19.1, )" } }, + "Healthie.NET.Uptime": { + "type": "Project", + "dependencies": { + "Healthie.NET.Abstractions": "[1.0.0, )" + } + }, "Cronos": { "type": "CentralTransitive", "requested": "[0.13.0, )", @@ -273,11 +344,20 @@ "System.ValueTuple": "4.5.0" } }, + "Microsoft.Extensions.DependencyInjection": { + "type": "CentralTransitive", + "requested": "[8.0.1, )", + "resolved": "10.0.10", + "contentHash": "ANyvsgkNBRvcJh2XLgn8veGmajf+8m0AbKK+HPWdRL1yraSNVVSmQhFntLtdz/C795jxqqup+k05cs/3jZQPOA==", + "dependencies": { + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10" + } + }, "Microsoft.Extensions.DependencyInjection.Abstractions": { "type": "CentralTransitive", "requested": "[8.0.2, )", - "resolved": "8.0.2", - "contentHash": "3iE7UF7MQkCv1cxzCahz+Y/guQbTqieyxyaWKhrRO91itI9cOKO76OHeQDahqG4MmW5umr3CcCvGmK92lWNlbg==" + "resolved": "10.0.10", + "contentHash": "z/2xXlFw2aLGjHyEm6E0tQ+In6VfzQzTrtArbQ2c0TQE16ZbyDCMGPvaUT9I0s8rgy9sRWlU2P9waW37qV04qA==" }, "Microsoft.Extensions.Diagnostics.HealthChecks": { "type": "CentralTransitive", @@ -310,6 +390,20 @@ "Microsoft.Extensions.Logging.Abstractions": "8.0.2" } }, + "Microsoft.Extensions.Http": { + "type": "CentralTransitive", + "requested": "[10.0.10, )", + "resolved": "10.0.10", + "contentHash": "DuEBLw2y7ZBfilnaJtlge8f2M49932v43t7j1InceXVjcZzxNrSEkobYXdgZkXuPFnFbKcbWpAS+fZgYEW1BhA==", + "dependencies": { + "Microsoft.Extensions.Configuration.Abstractions": "10.0.10", + "Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.10", + "Microsoft.Extensions.Diagnostics": "10.0.10", + "Microsoft.Extensions.Logging": "10.0.10", + "Microsoft.Extensions.Logging.Abstractions": "10.0.10", + "Microsoft.Extensions.Options": "10.0.10" + } + }, "Quartz.Extensions.DependencyInjection": { "type": "CentralTransitive", "requested": "[3.19.1, )", diff --git a/src/Healthie.AI/DiagnosisInsights.cs b/src/Healthie.AI/DiagnosisInsights.cs new file mode 100644 index 0000000..c1ac77e --- /dev/null +++ b/src/Healthie.AI/DiagnosisInsights.cs @@ -0,0 +1,23 @@ +using Healthie.Abstractions.Insights; + +namespace Healthie.AI; + +/// +/// Lets the dashboard ask for an explanation of why a checker has been failing. +/// +/// +/// A string rather than the full , because the board shows prose and +/// this keeps the shared contract free of this package's own types. Asked for on demand: the call +/// goes to a language model, so nothing should make it on a board that redraws every second. +/// +/// The diagnostician that talks to the model. +internal sealed class DiagnosisInsights(IPulseDiagnostician diagnostician) : IDiagnosisInsights +{ + /// + public async Task ExplainAsync(string checkerName, CancellationToken cancellationToken = default) + { + var diagnosis = await diagnostician.DiagnoseAsync(checkerName, cancellationToken).ConfigureAwait(false); + + return diagnosis.Summary; + } +} diff --git a/src/Healthie.AI/StartupExtensions.cs b/src/Healthie.AI/StartupExtensions.cs index 7323fe9..37147ea 100644 --- a/src/Healthie.AI/StartupExtensions.cs +++ b/src/Healthie.AI/StartupExtensions.cs @@ -1,3 +1,4 @@ +using Healthie.Abstractions.Insights; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; @@ -37,6 +38,11 @@ public static IServiceCollection AddHealthieAI(this IServiceCollection services) services.TryAddSingleton(); + // What the dashboard offers when this package is installed: a button that asks why. Not + // referenced from there, so a dashboard does not pull a model client in with it. + services.TryAddSingleton(provider => + new DiagnosisInsights(provider.GetRequiredService())); + return services; } } diff --git a/src/Healthie.Abstractions/IPulseChecker.cs b/src/Healthie.Abstractions/IPulseChecker.cs index 53ceeee..5bc4a81 100644 --- a/src/Healthie.Abstractions/IPulseChecker.cs +++ b/src/Healthie.Abstractions/IPulseChecker.cs @@ -1,5 +1,6 @@ using Healthie.Abstractions.Enums; using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; namespace Healthie.Abstractions; @@ -45,6 +46,27 @@ public interface IPulseChecker : IPulse, IState, IAsyncDisposable /// A task that represents the asynchronous operation. Task SetIntervalAsync(PulseInterval interval, CancellationToken cancellationToken = default); + /// + /// Sets the schedule the pulse check runs on, which a may not be + /// able to express. + /// + /// The schedule to run on, or null to go back to the interval. + /// A token to monitor for cancellation requests. + /// A task that represents the asynchronous operation. + /// + /// This implementation does not support being rescheduled. The default throws; anything deriving + /// from PulseChecker overrides it. + /// + /// + /// Defaulted rather than abstract so a checker written against the older interface keeps + /// compiling. It throws rather than doing nothing, because a scheduling call that silently has + /// no effect leaves a checker running on the old schedule and a caller believing otherwise. + /// + Task SetScheduleAsync(PulseSchedule? schedule, CancellationToken cancellationToken = default) => + throw new NotSupportedException( + $"{GetType().Name} does not support being given a {nameof(PulseSchedule)}. Derive from " + + "PulseChecker, or implement this method."); + /// /// Sets the unhealthy threshold for the pulse checker asynchronously. /// diff --git a/src/Healthie.Abstractions/Insights/AlertInsight.cs b/src/Healthie.Abstractions/Insights/AlertInsight.cs new file mode 100644 index 0000000..806b884 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/AlertInsight.cs @@ -0,0 +1,26 @@ +using Healthie.Abstractions.Enums; + +namespace Healthie.Abstractions.Insights; + +/// +/// One alert, as the dashboard shows it. +/// +/// The checker the alert is about. +/// That checker's display name. +/// The health before, or null if it had never run. +/// The health that raised the alert. +/// The check's own message. +/// When it was raised, in UTC. +/// Whether every sink accepted it. +/// +/// Raised and delivered are separate facts, and both are carried: an alert that fired and reached +/// nobody is the failure worth seeing on the board. +/// +public sealed record AlertInsight( + string CheckerName, + string DisplayName, + PulseCheckerHealth? PreviousHealth, + PulseCheckerHealth CurrentHealth, + string Message, + DateTime OccurredAt, + bool Delivered); diff --git a/src/Healthie.Abstractions/Insights/AlertPage.cs b/src/Healthie.Abstractions/Insights/AlertPage.cs new file mode 100644 index 0000000..022087f --- /dev/null +++ b/src/Healthie.Abstractions/Insights/AlertPage.cs @@ -0,0 +1,20 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// One page of the alert history, newest first. +/// +/// The alerts on this page. +/// How many are held in total, which is what the pager counts against. +/// +/// The name of the state provider the history is kept in, so the board can say where it went and +/// whether it will still be there after a restart. +/// +/// +/// How many alerts are kept before the oldest is discarded. The history is bounded on purpose: it is +/// a window onto what happened, and the record of record is wherever the sinks deliver to. +/// +public sealed record AlertPage( + IReadOnlyList Alerts, + int Total, + string StoredIn, + int Capacity); diff --git a/src/Healthie.Abstractions/Insights/AlertSinkStatus.cs b/src/Healthie.Abstractions/Insights/AlertSinkStatus.cs new file mode 100644 index 0000000..2f24802 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/AlertSinkStatus.cs @@ -0,0 +1,23 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// One place alerts are delivered to, and how it has been going. +/// +/// The sink's type name, which is what identifies it on the board. +/// How many alerts it has accepted. +/// How many it refused, timed out on, or threw over. +/// The most recent failure's message, or null if it has never failed. +/// +/// Counted per sink rather than in total because that is the question being asked: with three sinks +/// configured, "one alert did not get through" is a very different morning from "Slack is down". +/// +public sealed record AlertSinkStatus(string Name, int Delivered, int Failed, string? LastError) +{ + /// Whether this sink is currently getting alerts through. + /// + /// Judged on the last attempt rather than the ratio: a sink that failed a hundred times and is + /// working now is working, and one that has delivered thousands and just started failing is the + /// thing worth showing in red. + /// + public bool IsHealthy => LastError is null; +} diff --git a/src/Healthie.Abstractions/Insights/IAlertConfiguration.cs b/src/Healthie.Abstractions/Insights/IAlertConfiguration.cs new file mode 100644 index 0000000..de5584d --- /dev/null +++ b/src/Healthie.Abstractions/Insights/IAlertConfiguration.cs @@ -0,0 +1,53 @@ +using Healthie.Abstractions.Enums; + +namespace Healthie.Abstractions.Insights; + +/// +/// The alerting settings that can be changed while the application is running. +/// +/// +/// +/// Only the ones that take effect. The dispatcher reads these four on every alert, so a change here +/// applies to the next one; its queue capacity and history length are fixed when it is built and +/// cannot be moved without a restart, so they are not offered. Showing a control that quietly does +/// nothing is the failure this whole release has been fixing. +/// +/// +/// In memory and not persisted: this is the same object the host configured at startup, so a +/// restart returns to whatever AddHealthieAlerts was given. The board says so. +/// +/// +public interface IAlertConfiguration +{ + /// What the dispatcher is using now. + AlertSettings Current { get; } + + /// Applies new settings, from the next alert onwards. + /// The settings to apply. + void Apply(AlertSettings settings); + + /// + /// Sends a test alert to every registered sink and reports what happened. + /// + /// A token to monitor for cancellation requests. + /// Each sink, and whether it took the alert. + /// + /// The only way to find out that a webhook URL is wrong is to use it. Deduplication and the + /// severity threshold are bypassed, because the point is to exercise delivery rather than to + /// decide whether this alert is worth sending. + /// + Task> SendTestAlertAsync(CancellationToken cancellationToken = default); +} + +/// +/// The alerting settings a running application will honour a change to. +/// +/// The least severe health that raises an alert. +/// Whether returning to healthy raises one too. +/// How long the same checker is quiet for after alerting. +/// How long a single sink gets before it is abandoned. +public sealed record AlertSettings( + PulseCheckerHealth MinimumSeverity, + bool SendRecoveries, + TimeSpan DeduplicationWindow, + TimeSpan DeliveryTimeout); diff --git a/src/Healthie.Abstractions/Insights/IAlertInsights.cs b/src/Healthie.Abstractions/Insights/IAlertInsights.cs new file mode 100644 index 0000000..6b07af5 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/IAlertInsights.cs @@ -0,0 +1,48 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// The alerts that have been raised, and whether they are being delivered. +/// +/// +/// Declared here rather than in the alerting package, so the dashboard can render the panel without +/// referencing it -- see for why that matters. Reads only. +/// +public interface IAlertInsights +{ + /// + /// One page of alerts, newest first. + /// + /// How many of the newest to pass over. + /// How many to return. + /// A token to monitor for cancellation requests. + /// The page, and how many there are in total. + /// + /// Paged rather than capped at a handful because the history outlives the process: an operator + /// arriving after a restart is looking for what happened before it, which is exactly the part a + /// "last twenty" list throws away. + /// + Task GetAlertsAsync( + int skip, + int take, + CancellationToken cancellationToken = default); + + /// + /// How many alerts were dropped because the queue was full. + /// + /// + /// Worth showing rather than only logging: a dropped alert is one nobody was told about, and the + /// board is where somebody would look to find out that alerting itself is behind. + /// + int DroppedCount { get; } + + /// + /// Where alerts are being delivered, and how that is going. + /// + /// + /// Empty when nothing is registered to deliver to, which is the case worth surfacing: an + /// application that installed alerting and never configured a sink raises alerts that reach + /// nobody, and a board showing a healthy list of alerts looks exactly like one that is notifying + /// people. Startup logs it once; this is what puts it where somebody is looking. + /// + IReadOnlyList Sinks { get; } +} diff --git a/src/Healthie.Abstractions/Insights/IDiagnosisInsights.cs b/src/Healthie.Abstractions/Insights/IDiagnosisInsights.cs new file mode 100644 index 0000000..dbfbdf7 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/IDiagnosisInsights.cs @@ -0,0 +1,23 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// An explanation of why a checker has been failing. +/// +/// +/// Declared here rather than in the AI package, so the dashboard can offer the button without +/// referencing it -- see for why that matters. +/// +public interface IDiagnosisInsights +{ + /// + /// Explains a checker's recent failures. + /// + /// The checker to explain. + /// A token to monitor for cancellation requests. + /// The explanation. + /// + /// Asked for rather than shown: this goes to a language model, which costs money and takes + /// seconds, so nothing should call it on a board that redraws every second. + /// + Task ExplainAsync(string checkerName, CancellationToken cancellationToken = default); +} diff --git a/src/Healthie.Abstractions/Insights/ILeadershipInsights.cs b/src/Healthie.Abstractions/Insights/ILeadershipInsights.cs new file mode 100644 index 0000000..528e3c3 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/ILeadershipInsights.cs @@ -0,0 +1,18 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// Whether this replica is the one running the checks. +/// +/// +/// With leader election on, only one replica runs a checker on each interval. A board that did not +/// say so would show a follower with everything idle and look broken. Declared here rather than in +/// the leader-election package -- see for why. Reads only. +/// +public interface ILeadershipInsights +{ + /// Whether this replica currently holds the lease. + bool IsLeader { get; } + + /// Something identifying this replica, for a board an operator is comparing across tabs. + string ReplicaId { get; } +} diff --git a/src/Healthie.Abstractions/Insights/IMetricsInsights.cs b/src/Healthie.Abstractions/Insights/IMetricsInsights.cs new file mode 100644 index 0000000..899af84 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/IMetricsInsights.cs @@ -0,0 +1,23 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// What the library's own instruments have counted, for a board to show. +/// +/// +/// +/// The Healthie.NET meter is the real home of this data and OpenTelemetry is the real way to +/// read it -- an APM keeps history, does percentiles properly, and alerts on them. This exists for +/// the case where there is no APM in front of the operator, which is most first runs and most small +/// deployments: the numbers are already being emitted, and nobody was looking at them. +/// +/// +/// So: a live window, not a time series. Nothing here is persisted or survives a restart, and the +/// board says as much beside it. +/// +/// +public interface IMetricsInsights +{ + /// What has been counted since the process started. + /// A token to monitor for cancellation requests. + MetricsSnapshot Snapshot(CancellationToken cancellationToken = default); +} diff --git a/src/Healthie.Abstractions/Insights/IUptimeInsights.cs b/src/Healthie.Abstractions/Insights/IUptimeInsights.cs new file mode 100644 index 0000000..fb0bdb4 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/IUptimeInsights.cs @@ -0,0 +1,38 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// How much of a window a checker spent healthy, for the dashboard to show. +/// +/// +/// +/// Declared here rather than in the package that implements it, so the dashboard can render the +/// panel without referencing that package. The dashboard is a Razor class library that references +/// this one and nothing else, and referencing alerting, uptime, leader election and the AI package +/// to display them would put all four on everyone who installs a dashboard. The board asks the +/// container what it has: a panel appears because a service is registered, not because a flag was +/// set. +/// +/// +/// Reads only, as every contract in this namespace does. Mutation stays on the interfaces that +/// already own it, which is what lets the dashboard show all of this in read-only mode. +/// +/// +public interface IUptimeInsights +{ + /// + /// The share of a window a checker spent healthy. + /// + /// The checker to report on. + /// How far back to look. + /// A token to monitor for cancellation requests. + /// The report, or null if nothing is recorded for that checker. + /// + /// Distinct from the percentage the board already shows, which is the share of the runs still in + /// the rolling history -- a few minutes at a fast interval. This is measured over real time and + /// survives history being trimmed. + /// + Task GetUptimeAsync( + string checkerName, + TimeSpan window, + CancellationToken cancellationToken = default); +} diff --git a/src/Healthie.Abstractions/Insights/MetricsSnapshot.cs b/src/Healthie.Abstractions/Insights/MetricsSnapshot.cs new file mode 100644 index 0000000..edc13db --- /dev/null +++ b/src/Healthie.Abstractions/Insights/MetricsSnapshot.cs @@ -0,0 +1,40 @@ +using Healthie.Abstractions.Enums; + +namespace Healthie.Abstractions.Insights; + +/// +/// What the library's instruments have counted since the process started. +/// +/// Total checks completed. +/// Checks completed, by the health each reported. +/// Checks that produced a state different from the stored one. +/// Triggers that returned immediately because the previous check had not finished. +/// Mean check duration, or null before anything has run. +/// The slowest single check, or null before anything has run. +/// When collection started, which is when the process did. +public sealed record MetricsSnapshot( + long Checks, + IReadOnlyDictionary ResultsByHealth, + long Transitions, + long OverlappedTriggers, + TimeSpan? MeanDuration, + TimeSpan? SlowestDuration, + DateTime Since) +{ + /// + /// The share of completed checks that reported healthy, or null before anything has run. + /// + public double? HealthyShare => Checks == 0 + ? null + : 100d * ResultsByHealth.GetValueOrDefault(PulseCheckerHealth.Healthy) / Checks; + + /// + /// Whether any trigger has been skipped for overlapping the previous one. + /// + /// + /// Called out separately because it is the one number here that means something is wrong rather + /// than something happened: a checker whose check outlasts its own interval looks healthy and is + /// quietly running at a fraction of the rate it was asked to. This is the only place it shows. + /// + public bool HasOverlaps => OverlappedTriggers > 0; +} diff --git a/src/Healthie.Abstractions/Insights/UptimeInsight.cs b/src/Healthie.Abstractions/Insights/UptimeInsight.cs new file mode 100644 index 0000000..67a0955 --- /dev/null +++ b/src/Healthie.Abstractions/Insights/UptimeInsight.cs @@ -0,0 +1,13 @@ +namespace Healthie.Abstractions.Insights; + +/// +/// What a checker's uptime looked like over a window. +/// +/// The share of the window spent healthy, from 0 to 100. +/// The window measured. +/// The longest unbroken unhealthy stretch, or null if there was none. +/// +/// The longest outage is carried beside the percentage because a percentage alone cannot tell a +/// hundred one-second blips from one long outage, and those are very different mornings. +/// +public sealed record UptimeInsight(double Percentage, TimeSpan Window, TimeSpan? LongestOutage); diff --git a/src/Healthie.Abstractions/PulseChecker.cs b/src/Healthie.Abstractions/PulseChecker.cs index 1eb8826..85a8959 100644 --- a/src/Healthie.Abstractions/PulseChecker.cs +++ b/src/Healthie.Abstractions/PulseChecker.cs @@ -352,9 +352,46 @@ private async Task UpdateStateAsync(Action apply, Cance } /// + /// + /// Clears any , because that overrides the interval: + /// without this, choosing an interval on a cron-scheduled checker stored a field nothing reads + /// and the checker carried on at its old cadence. + /// public async Task SetIntervalAsync(PulseInterval interval, CancellationToken cancellationToken = default) { - await UpdateStateAsync(state => state.Interval = interval, cancellationToken).ConfigureAwait(false); + await UpdateStateAsync( + state => + { + state.Interval = interval; + state.Schedule = null; + }, + cancellationToken).ConfigureAwait(false); + } + + /// + /// + /// A schedule an interval can express exactly is stored as that interval, so the common case + /// keeps the shape every stored state and every older reader already understands, and only a + /// genuinely custom period or a cron expression occupies . + /// + public async Task SetScheduleAsync(PulseSchedule? schedule, CancellationToken cancellationToken = default) + { + if (schedule is null) + { + // Back to whatever interval is stored, which is what the field is there to hold. + await UpdateStateAsync(state => state.Schedule = null, cancellationToken).ConfigureAwait(false); + + return; + } + + if (schedule.TryToInterval(out var interval)) + { + await SetIntervalAsync(interval, cancellationToken).ConfigureAwait(false); + + return; + } + + await UpdateStateAsync(state => state.Schedule = schedule, cancellationToken).ConfigureAwait(false); } /// diff --git a/src/Healthie.Abstractions/Scheduling/IPulseScheduler.cs b/src/Healthie.Abstractions/Scheduling/IPulseScheduler.cs index 8d36f20..6385f14 100644 --- a/src/Healthie.Abstractions/Scheduling/IPulseScheduler.cs +++ b/src/Healthie.Abstractions/Scheduling/IPulseScheduler.cs @@ -50,6 +50,30 @@ Task ScheduleAsync(IPulseChecker checker, PulseSchedule schedule, CancellationTo $"{nameof(ScheduleAsync)}({nameof(PulseSchedule)}) to use it."); } + /// + /// Judges a schedule before anything is asked to store it. + /// + /// The schedule to judge. + /// Why it was refused, or null when it was accepted. + /// true when this scheduler could run it. + /// + /// Asked of the scheduler because the scheduler is the authority: Cronos, Quartz and Temporal do + /// not agree on cron dialects, so the only implementation whose answer means anything is the one + /// that will run it. This exists so a schedule typed into the dashboard is refused before it is + /// persisted -- storing it first and discovering the problem while rescheduling leaves a checker + /// with a schedule nothing can run. + /// + /// Defaulted to accept, so a scheduler written against the older interface is unaffected and + /// reports the problem when scheduling, as it does today. + /// + /// + bool TryValidateSchedule(PulseSchedule schedule, out string? error) + { + error = null; + + return true; + } + /// /// Unschedules a previously scheduled pulse checker. /// diff --git a/src/Healthie.Abstractions/Scheduling/IPulsesScheduler.cs b/src/Healthie.Abstractions/Scheduling/IPulsesScheduler.cs index 09672e2..d7d5531 100644 --- a/src/Healthie.Abstractions/Scheduling/IPulsesScheduler.cs +++ b/src/Healthie.Abstractions/Scheduling/IPulsesScheduler.cs @@ -33,6 +33,23 @@ public interface IPulsesScheduler : IHostedService /// Thrown when no pulse checker with the specified exists. Task SetIntervalAsync(string name, PulseInterval interval, CancellationToken cancellationToken = default); + /// + /// Sets the schedule a pulse checker runs on, and reschedules it. + /// + /// The name of the pulse checker. + /// The schedule to run it on, or null to go back to its interval. + /// A token to monitor for cancellation requests. + /// A task that represents the asynchronous operation. + /// + /// No pulse checker with the specified exists, or the registered + /// scheduler will not run the schedule. + /// + /// + /// Refused before it is stored rather than after: a schedule the scheduler cannot run, persisted + /// and then failed on, leaves a checker that no longer runs and a store that says it should. + /// + Task SetScheduleAsync(string name, PulseSchedule? schedule, CancellationToken cancellationToken = default); + /// /// Sets the unhealthy threshold for a specific pulse checker. /// diff --git a/src/Healthie.Abstractions/Scheduling/PulsesScheduler.cs b/src/Healthie.Abstractions/Scheduling/PulsesScheduler.cs index c619b80..c2b9e9b 100644 --- a/src/Healthie.Abstractions/Scheduling/PulsesScheduler.cs +++ b/src/Healthie.Abstractions/Scheduling/PulsesScheduler.cs @@ -98,6 +98,23 @@ public async Task SetIntervalAsync(string name, PulseInterval interval, Cancella await ScheduleAsync(pulseChecker, cancellationToken).ConfigureAwait(false); } + /// + public async Task SetScheduleAsync(string name, PulseSchedule? schedule, CancellationToken cancellationToken = default) + { + var pulseChecker = GetCheckerOrThrow(name); + + if (schedule is not null && !_pulseScheduler.TryValidateSchedule(schedule, out var error)) + { + // No parameter name: this message is written to be shown to whoever typed the + // expression, and "(Parameter 'schedule')" is plumbing to everyone but a debugger. The + // caller named the checker, so the message does not repeat it back. + throw new ArgumentException($"'{schedule}' cannot be scheduled. {error}"); + } + + await pulseChecker.SetScheduleAsync(schedule, cancellationToken).ConfigureAwait(false); + await ScheduleAsync(pulseChecker, cancellationToken).ConfigureAwait(false); + } + /// public async Task SetUnhealthyThresholdAsync(string name, uint threshold, CancellationToken cancellationToken = default) { diff --git a/src/Healthie.Alerting/AlertConfiguration.cs b/src/Healthie.Alerting/AlertConfiguration.cs new file mode 100644 index 0000000..8f32604 --- /dev/null +++ b/src/Healthie.Alerting/AlertConfiguration.cs @@ -0,0 +1,53 @@ +using Healthie.Abstractions.Insights; + +namespace Healthie.Alerting; + +/// +/// Lets the dashboard change the alerting settings that a running dispatcher honours. +/// +/// +/// Writes straight to the options object the dispatcher holds, which is the same singleton the host +/// configured: the dispatcher reads these four on every alert rather than snapshotting them, so a +/// change applies to the next one with nothing to restart or re-register. +/// +/// The options the dispatcher is reading. +/// The dispatcher, for sending a test alert through the real sinks. +internal sealed class AlertConfiguration(HealthieAlertOptions options, AlertDispatcher dispatcher) + : IAlertConfiguration +{ + /// + public AlertSettings Current => new( + options.MinimumSeverity, + options.SendRecoveries, + options.DeduplicationWindow, + options.DeliveryTimeout); + + /// + public void Apply(AlertSettings settings) + { + ArgumentNullException.ThrowIfNull(settings); + + if (settings.DeduplicationWindow < TimeSpan.Zero) + { + throw new ArgumentException( + "A deduplication window cannot be negative. Zero means alert on every change.", + nameof(settings)); + } + + if (settings.DeliveryTimeout <= TimeSpan.Zero) + { + throw new ArgumentException( + "A delivery timeout must be positive, or no sink would ever get long enough to answer.", + nameof(settings)); + } + + options.MinimumSeverity = settings.MinimumSeverity; + options.SendRecoveries = settings.SendRecoveries; + options.DeduplicationWindow = settings.DeduplicationWindow; + options.DeliveryTimeout = settings.DeliveryTimeout; + } + + /// + public Task> SendTestAlertAsync(CancellationToken cancellationToken = default) => + dispatcher.SendTestAlertAsync(cancellationToken); +} diff --git a/src/Healthie.Alerting/AlertDispatcher.cs b/src/Healthie.Alerting/AlertDispatcher.cs index 9aee9d6..e7f07f3 100644 --- a/src/Healthie.Alerting/AlertDispatcher.cs +++ b/src/Healthie.Alerting/AlertDispatcher.cs @@ -1,3 +1,4 @@ +using Healthie.Abstractions.Insights; using Healthie.Abstractions; using Healthie.Abstractions.Enums; using Healthie.Abstractions.Models; @@ -31,6 +32,7 @@ public sealed class AlertDispatcher : BackgroundService private readonly IReadOnlyList _checkers; private readonly IReadOnlyList _sinks; private readonly HealthieAlertOptions _options; + private readonly AlertHistory? _history; private readonly ILogger? _logger; private readonly Channel _queue; @@ -43,11 +45,16 @@ public sealed class AlertDispatcher : BackgroundService /// Every registered pulse checker. /// Every registered alert sink. /// Which changes alert, and how hard to try. + /// + /// Keeps the last few alerts for the dashboard. Optional, so an application without one carries + /// no history at all. + /// /// An optional logger for diagnostic output. public AlertDispatcher( IEnumerable checkers, IEnumerable sinks, HealthieAlertOptions options, + AlertHistory? history = null, ILogger? logger = null) { ArgumentNullException.ThrowIfNull(checkers); @@ -56,6 +63,7 @@ public AlertDispatcher( _checkers = [.. checkers]; _sinks = [.. sinks]; _options = options ?? throw new ArgumentNullException(nameof(options)); + _history = history; _logger = logger; _queue = Channel.CreateBounded( @@ -84,11 +92,17 @@ public AlertDispatcher( /// public override Task StartAsync(CancellationToken cancellationToken) { - // Nothing to deliver to, so there is no reason to subscribe or to hold a queue. + // Still subscribes with no sink registered: the dashboard's alerts panel reads the same + // history this fills, so "nowhere to deliver" is no longer "nothing to do". if (_sinks.Count == 0) { - _logger?.LogInformation("Alerting is registered but no sink is; no alerts will be sent."); - return Task.CompletedTask; + _logger?.LogInformation( + "Alerting is registered but no sink is; alerts will show on the dashboard and be sent nowhere."); + } + + foreach (var sink in _sinks) + { + _history?.Register(sink.GetType().Name); } Subscribe(); @@ -172,6 +186,8 @@ private void OnStateChanged(IPulseChecker checker, PulseCheckerStateChangedEvent /// private void OnAlertDropped(Alert alert) { + _history?.RecordDropped(); + var dropped = Interlocked.Increment(ref _dropped); _logger?.LogWarning( @@ -226,20 +242,30 @@ private bool IsWithinDeduplicationWindow(string checkerName) /// private async Task DeliverAsync(Alert alert, CancellationToken stoppingToken) { + // "Raised" and "delivered" are different facts, and the board shows both: an alert that + // fired and reached nobody is the failure worth seeing. + var delivered = true; + foreach (var sink in _sinks) { using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken); timeout.CancelAfter(_options.DeliveryTimeout); + var name = sink.GetType().Name; + try { await sink.SendAsync(alert, timeout.Token).ConfigureAwait(false); + _history?.RecordDelivery(name, error: null); } catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested) { + delivered = false; + _history?.RecordDelivery(name, $"Did not respond within {_options.DeliveryTimeout}."); + _logger?.LogWarning( "Alert sink {Sink} did not deliver the alert for '{CheckerName}' within {Timeout}.", - sink.GetType().Name, + name, alert.CheckerName, _options.DeliveryTimeout); } @@ -249,13 +275,44 @@ private async Task DeliverAsync(Alert alert, CancellationToken stoppingToken) } catch (Exception ex) { + delivered = false; + _history?.RecordDelivery(name, ex.Message); + _logger?.LogError( ex, "Alert sink {Sink} failed to deliver the alert for '{CheckerName}'.", - sink.GetType().Name, + name, alert.CheckerName); } } + + _history?.Record(alert, delivered); + } + + /// + /// Puts one alert through every sink and reports what each did, bypassing the queue. + /// + /// A token to monitor for cancellation requests. + /// Each sink's tally, including this attempt. + /// + /// Delivered directly rather than enqueued so the caller can be told the outcome; the queue + /// exists to keep a slow sink away from the checks, and nothing is checking here. + /// + public async Task> SendTestAlertAsync(CancellationToken cancellationToken = default) + { + var alert = new Alert( + "healthie.test", + "Healthie test alert", + Group: null, + Tags: [], + PulseCheckerHealth.Healthy, + PulseCheckerHealth.Unhealthy, + "Test alert raised from the dashboard. Nothing is wrong.", + DateTime.UtcNow); + + await DeliverAsync(alert, cancellationToken).ConfigureAwait(false); + + return _history?.Sinks ?? []; } /// diff --git a/src/Healthie.Alerting/AlertHistory.cs b/src/Healthie.Alerting/AlertHistory.cs new file mode 100644 index 0000000..61ace8b --- /dev/null +++ b/src/Healthie.Alerting/AlertHistory.cs @@ -0,0 +1,252 @@ +using Healthie.Abstractions.Insights; +using Healthie.Abstractions.StateProviding; +using Microsoft.Extensions.Logging; + +namespace Healthie.Alerting; + +/// +/// The alerts that have been raised, kept so the dashboard can show them. +/// +/// +/// +/// Alerting is fire-and-forget by design: an alert goes to its sinks and is gone. That is right for +/// delivery and wrong for the one screen an operator looks at, where "what fired, and did it get +/// through" is the first question -- and it is asked most often just after a restart, about what +/// happened before it. +/// +/// +/// So the log is written through the application's own : a deployment on +/// CosmosDB, Postgres or Redis keeps its alert history across a redeploy, and one left on the +/// in-memory provider does not. There is no second storage contract to configure, and no provider +/// had to learn about alerts. +/// +/// +/// Bounded, and written whole on each alert. Both are affordable because alerts are transitions +/// rather than checks -- a checker running every second raises nothing until its health changes -- and +/// both are deliberate: an unbounded log in a state document would grow without limit, and the record +/// of record is wherever the sinks deliver to. +/// +/// +/// How many alerts to keep before the oldest is discarded. +/// Where to persist the log, or null to keep it in memory only. +/// An optional logger for diagnostic output. +public sealed class AlertHistory( + int capacity, + IStateProvider? stateProvider = null, + ILogger? logger = null) : IAlertInsights +{ + /// The key the whole log is stored under. + /// + /// Deliberately not a checker's name, and prefixed so it cannot collide with one: a state + /// provider is keyed by checker name, and this is the one entry that is not a checker. + /// + private const string StorageKey = "healthie.alerts.log"; + + private readonly Queue _recent = new(capacity); + // A plain object, not System.Threading.Lock: this package targets net8.0 as well. + private readonly object _gate = new(); + + private bool _loaded; + + private readonly Dictionary _sinks = []; + + private int _dropped; + + /// + public int DroppedCount => Volatile.Read(ref _dropped); + + /// + public IReadOnlyList Sinks + { + get + { + lock (_gate) + { + return [.. _sinks.Select(entry => new AlertSinkStatus( + entry.Key, entry.Value.Delivered, entry.Value.Failed, entry.Value.LastError))]; + } + } + } + + /// Registers a sink so it appears on the board before it has done anything. + /// The sink's type name. + /// + /// Named at startup rather than discovered on first delivery, because "no sinks configured" and + /// "sinks configured, nothing has alerted yet" look identical otherwise and mean opposite things. + /// + public void Register(string name) + { + lock (_gate) + { + _sinks.TryAdd(name, new SinkTally()); + } + } + + /// Records the outcome of one sink's attempt at one alert. + /// The sink's type name. + /// The failure, or null when it was accepted. + public void RecordDelivery(string name, string? error) + { + lock (_gate) + { + if (!_sinks.TryGetValue(name, out var tally)) + { + tally = new SinkTally(); + _sinks[name] = tally; + } + + if (error is null) + { + tally.Delivered++; + + // Cleared on success, so a sink that failed once and recovered stops being shown as + // broken -- what matters is whether it is working now. + tally.LastError = null; + } + else + { + tally.Failed++; + tally.LastError = error; + } + } + } + + private sealed class SinkTally + { + public int Delivered { get; set; } + + public int Failed { get; set; } + + public string? LastError { get; set; } + } + + /// Records an alert and whether every sink took it. + /// The alert that was raised. + /// Whether every sink accepted it. + public void Record(Alert alert, bool delivered) + { + var insight = new AlertInsight( + alert.CheckerName, + alert.DisplayName, + alert.PreviousHealth, + alert.CurrentHealth, + alert.Message, + alert.OccurredAt, + delivered); + + List toPersist; + + lock (_gate) + { + // Trims after enqueuing rather than before. Dropping the oldest first has to special-case + // an empty queue, and a capacity of zero makes every call the empty case. + _recent.Enqueue(insight); + + while (_recent.Count > capacity) + { + _recent.Dequeue(); + } + + toPersist = [.. _recent]; + } + + // Outside the lock: this is a round trip to the state store, and holding a lock across it + // would stall every reader of the board for the duration of a database write. + _ = PersistAsync(toPersist); + } + + /// Records that an alert never reached the queue. + public void RecordDropped() => Interlocked.Increment(ref _dropped); + + /// + public async Task GetAlertsAsync( + int skip, + int take, + CancellationToken cancellationToken = default) + { + await EnsureLoadedAsync(cancellationToken).ConfigureAwait(false); + + lock (_gate) + { + var newestFirst = _recent.Reverse().ToList(); + + IReadOnlyList page = + [.. newestFirst.Skip(Math.Max(skip, 0)).Take(Math.Max(take, 0))]; + + return new AlertPage(page, newestFirst.Count, StoreName, capacity); + } + } + + /// What the board calls the place this history is kept. + private string StoreName => stateProvider?.GetType().Name ?? "memory"; + + /// + /// Reads the stored log once, the first time anything asks for a page. + /// + /// + /// Lazily rather than at startup: the dispatcher subscribes while the host is still starting, and + /// a state provider may not have finished initializing its container or table by then. Nothing + /// needs the history until somebody opens the board. + /// + private async Task EnsureLoadedAsync(CancellationToken cancellationToken) + { + if (_loaded || stateProvider is null) + { + return; + } + + // Set before the read, not after: a failed read must not leave every later page request + // retrying a store that is not answering. + _loaded = true; + + try + { + var stored = await stateProvider + .GetStateAsync>(StorageKey, cancellationToken) + .ConfigureAwait(false); + + if (stored is null or { Count: 0 }) + { + return; + } + + lock (_gate) + { + // In front of anything raised while this was loading, then trimmed: the stored log is + // older by definition, and a restart that raises an alert immediately must not lose it. + var live = _recent.ToList(); + _recent.Clear(); + + foreach (var insight in stored.Concat(live).TakeLast(capacity)) + { + _recent.Enqueue(insight); + } + } + } + catch (Exception ex) + { + // A history that cannot be read is not a reason to fail the board: it shows what this + // process has seen, and says where the rest was meant to be. + logger?.LogWarning(ex, "Could not read the stored alert history from the state provider."); + } + } + + private async Task PersistAsync(List log) + { + if (stateProvider is null) + { + return; + } + + try + { + await stateProvider.SetStateAsync(StorageKey, log).ConfigureAwait(false); + } + catch (Exception ex) + { + // Never propagated. This runs on the alert-delivery path, and a state store that is down + // must not take alerting down with it -- the alert has already reached its sinks. + logger?.LogWarning(ex, "Could not persist the alert history to the state provider."); + } + } +} diff --git a/src/Healthie.Alerting/HealthieAlertOptions.cs b/src/Healthie.Alerting/HealthieAlertOptions.cs index 5c7af0b..4049a12 100644 --- a/src/Healthie.Alerting/HealthieAlertOptions.cs +++ b/src/Healthie.Alerting/HealthieAlertOptions.cs @@ -55,4 +55,20 @@ public sealed class HealthieAlertOptions /// leak in the process being monitored. A dropped alert is counted and logged. /// public int QueueCapacity { get; set; } = 1024; + + private int _historyLength = 50; + + /// + /// How many recent alerts the dashboard can show. Defaults to 50, minimum 1. + /// + /// + /// A window onto what just happened rather than a record; the record is wherever the sinks + /// deliver to. Kept in memory and bounded, so it costs nothing to leave on. Clamped rather than + /// rejected, as MaxHistoryLength is. + /// + public int HistoryLength + { + get => _historyLength; + set => _historyLength = Math.Max(value, 1); + } } diff --git a/src/Healthie.Alerting/StartupExtensions.cs b/src/Healthie.Alerting/StartupExtensions.cs index f0e7aec..3d83561 100644 --- a/src/Healthie.Alerting/StartupExtensions.cs +++ b/src/Healthie.Alerting/StartupExtensions.cs @@ -1,3 +1,6 @@ +using Healthie.Abstractions.Insights; +using Healthie.Abstractions.StateProviding; +using Microsoft.Extensions.Logging; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; @@ -32,7 +35,29 @@ public static IServiceCollection AddHealthieAlerts( configure?.Invoke(options); services.TryAddSingleton(options); - services.AddHostedService(); + + // What the dashboard renders when this package is installed: the last few alerts and whether + // they were delivered. Registered here rather than referenced there, so installing a + // dashboard does not drag alerting in with it. + // Given the application's own state provider, so the alert log lands wherever checker state + // already does and survives a redeploy on any durable one. GetService, not GetRequiredService: + // alerting can be registered before AddHealthie has put a provider in. + services.TryAddSingleton(provider => new AlertHistory( + options.HistoryLength, + provider.GetService(), + provider.GetService>())); + services.TryAddSingleton(p => p.GetRequiredService()); + services.TryAddSingleton(); + + // The dispatcher is resolved as itself and handed to the host, rather than registered + // straight as a hosted service: a test alert has to go through the sinks of the dispatcher + // that is actually running, not a second copy of it. Guarded by hand because + // TryAddEnumerable refuses a factory-built descriptor, and this must stay callable twice. + if (services.All(descriptor => descriptor.ServiceType != typeof(AlertDispatcher))) + { + services.AddSingleton(); + services.AddHostedService(provider => provider.GetRequiredService()); + } return services; } diff --git a/src/Healthie.Api/Controllers/HealthCheckersController.cs b/src/Healthie.Api/Controllers/HealthCheckersController.cs index 0584a65..c5e071f 100644 --- a/src/Healthie.Api/Controllers/HealthCheckersController.cs +++ b/src/Healthie.Api/Controllers/HealthCheckersController.cs @@ -113,6 +113,65 @@ public async Task SetCheckerInterval(string checkerName, [FromQue } } + /// + /// Sets the schedule a pulse checker runs on, for a cadence no interval expresses. + /// + /// The name of the pulse checker. + /// + /// A standard Unix cron expression, in five fields or six with a leading seconds field. Omit it + /// to clear the schedule and go back to the checker's interval. + /// + /// A token to monitor for cancellation requests. + /// + /// 204 No Content on success, 404 if the checker is not found, or 400 if the name is empty or + /// the registered scheduler will not run the expression. + /// + /// + /// The 400 carries the scheduler's own reason, because the schedulers do not agree on cron + /// dialects and "invalid expression" would leave the caller guessing which rule they broke. + /// + [HttpPut("{checkerName}/schedule")] + [ProducesResponseType(StatusCodes.Status204NoContent)] + [ProducesResponseType(StatusCodes.Status404NotFound)] + [ProducesResponseType(StatusCodes.Status400BadRequest)] + public async Task SetCheckerSchedule( + string checkerName, + [FromQuery] string? cron, + CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(checkerName)) + { + return BadRequest("Checker name cannot be empty."); + } + + try + { + var checkers = await pulsesScheduler.GetPulseCheckersAsync(cancellationToken).ConfigureAwait(false); + if (!checkers.ContainsKey(checkerName)) + { + logger?.LogWarning("Checker '{CheckerName}' not found for setting a schedule.", ForLog(checkerName)); + return NotFound($"Checker '{checkerName}' not found."); + } + + var schedule = string.IsNullOrWhiteSpace(cron) ? null : PulseSchedule.Cron(cron); + + await pulsesScheduler.SetScheduleAsync(checkerName, schedule, cancellationToken).ConfigureAwait(false); + + return NoContent(); + } + catch (ArgumentException ex) + { + // The schedule itself was refused -- by PulseSchedule for its shape, or by the scheduler + // for its dialect. Either way it is the caller's input, not a server fault. + return BadRequest(ex.Message); + } + catch (Exception ex) + { + logger?.LogError(ex, "Error setting the schedule for checker '{CheckerName}'.", ForLog(checkerName)); + return StatusCode(StatusCodes.Status500InternalServerError, "An unexpected error occurred."); + } + } + /// /// Sets the unhealthy threshold for a specific pulse checker. /// diff --git a/src/Healthie.Dashboard/Components/HealthieDashboard.Insights.cs b/src/Healthie.Dashboard/Components/HealthieDashboard.Insights.cs new file mode 100644 index 0000000..7c541c2 --- /dev/null +++ b/src/Healthie.Dashboard/Components/HealthieDashboard.Insights.cs @@ -0,0 +1,250 @@ +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Insights; +using Microsoft.AspNetCore.Components; +using Microsoft.Extensions.DependencyInjection; + +namespace Healthie.Dashboard.Components; + +/// +/// The parts of the board that appear only when a feature package is installed. +/// +/// +/// +/// Each of these is resolved from the container rather than injected, because [Inject] throws +/// when a service is absent and absent is the normal case: an application with no alerting installed +/// should get a board without an alerts panel, not an exception. +/// +/// +/// All of them read. Nothing here changes a checker, which is what lets the whole set show in +/// read-only mode -- the one exception is asking a language model to explain a failure, which is +/// still a read but costs money on someone else's account, so it is treated as an action. +/// +/// +public sealed partial class HealthieDashboard +{ + /// How far back the uptime panel looks. + /// + /// A day, because that is the window an operator asks about first and the one an overnight + /// incident falls inside. The board's own percentage covers the last few minutes. + /// + private static readonly TimeSpan UptimeWindow = TimeSpan.FromHours(24); + + [Inject] + private IServiceProvider Services { get; set; } = default!; + + /// How many alerts a page of the history holds. + private const int AlertsPerPage = 25; + + private IUptimeInsights? _uptimeInsights; + private IAlertInsights? _alertInsights; + private ILeadershipInsights? _leadershipInsights; + private IDiagnosisInsights? _diagnosisInsights; + private IMetricsInsights? _metricsInsights; + private IAlertConfiguration? _alertConfiguration; + + private UptimeInsight? _uptime; + + private AlertPage? _alerts; + private int _alertPage; + private bool _undeliveredOnly; + + private MetricsSnapshot? _metrics; + + private AlertSettings? _settingsDraft; + private string? _settingsError; + private IReadOnlyList? _testResult; + private bool _testing; + + private string? _diagnosis; + private bool _diagnosing; + + /// Whether this replica is the one running the checks. + /// + /// True when nothing is elected: a single replica runs everything, and saying "follower" there + /// would be a warning about a situation that does not exist. + /// + private bool IsLeader => _leadershipInsights?.IsLeader ?? true; + + /// Picks up whatever feature packages the application installed. + private void ResolveInsights() + { + _uptimeInsights = Services.GetService(); + _alertInsights = Services.GetService(); + _leadershipInsights = Services.GetService(); + _diagnosisInsights = Services.GetService(); + _metricsInsights = Services.GetService(); + _alertConfiguration = Services.GetService(); + _settingsDraft = _alertConfiguration?.Current; + } + + /// Reads the uptime for the selected checker. + /// + /// On selection, not on the clock tick: it is a query over recorded segments and the board + /// redraws every second. Re-selecting a checker re-reads rather than short-circuiting, so + /// clicking the row you are on is how you refresh a window that has been open a while. + /// + private async Task LoadUptimeAsync(string? checkerName) + { + if (_uptimeInsights is null || checkerName is null) + { + _uptime = null; + + return; + } + + _uptime = await _uptimeInsights.GetUptimeAsync(checkerName, UptimeWindow); + } + + /// Reads one page of the alert history, newest first. + private async Task LoadAlertsAsync() + { + if (_alertInsights is null) + { + return; + } + + _alerts = await _alertInsights.GetAlertsAsync(_alertPage * AlertsPerPage, AlertsPerPage); + } + + /// The alerts this page shows, after the undelivered filter. + /// + /// Filtered here rather than in the query: the store pages over everything it holds, and a filter + /// pushed into it would make the page counts disagree with the pager. This narrows what is shown + /// on the page you are on, which is what the toggle says it does. + /// + private IReadOnlyList VisibleAlerts => + _alerts is null + ? [] + : _undeliveredOnly + ? [.. _alerts.Alerts.Where(alert => !alert.Delivered)] + : _alerts.Alerts; + + private int AlertPageCount => + _alerts is null || _alerts.Total == 0 ? 1 : (_alerts.Total + AlertsPerPage - 1) / AlertsPerPage; + + private async Task GoToAlertPageAsync(int page) + { + _alertPage = Math.Clamp(page, 0, AlertPageCount - 1); + + await LoadAlertsAsync(); + } + + private async Task ToggleUndeliveredOnlyAsync() + { + _undeliveredOnly = !_undeliveredOnly; + + await LoadAlertsAsync(); + } + + /// Reads what the library's own instruments have counted. + private void LoadMetrics() => _metrics = _metricsInsights?.Snapshot(); + + /// + /// Applies the alerting settings in the form, from the next alert onwards. + /// + private void ApplyAlertSettings() + { + if (_alertConfiguration is null || _settingsDraft is null) + { + return; + } + + _settingsError = null; + + try + { + _alertConfiguration.Apply(_settingsDraft); + AddEvent("CONF", Status.Paused, "Alerting settings changed"); + } + catch (ArgumentException ex) + { + _settingsError = ex.Message; + } + } + + /// + /// Sends one alert through the real sinks, because the only way to find out that a webhook URL is + /// wrong is to use it. + /// + private async Task SendTestAlertAsync() + { + if (_alertConfiguration is null || _testing) + { + return; + } + + _testing = true; + _testResult = null; + + try + { + _testResult = await _alertConfiguration.SendTestAlertAsync(); + AddEvent("TEST", Status.Ok, "Test alert sent to every sink"); + await LoadAlertsAsync(); + } + catch (Exception ex) + { + _settingsError = $"Could not send the test alert: {ex.Message}"; + } + finally + { + _testing = false; + } + } + + /// Asks the model why a checker has been failing. + /// + /// Guarded against a second click while the first is in flight: the call takes seconds and costs + /// money, and a button that looks unresponsive invites exactly that. + /// + private async Task ExplainAsync(string checkerName) + { + if (_diagnosisInsights is null || _diagnosing) + { + return; + } + + _diagnosing = true; + _diagnosis = null; + + try + { + _diagnosis = await _diagnosisInsights.ExplainAsync(checkerName); + } + catch (Exception ex) + { + // Shown rather than swallowed: a model that is misconfigured or out of quota should say + // so on the board, not leave a button that appears to do nothing. + _diagnosis = $"Could not explain this: {ex.Message}"; + } + finally + { + _diagnosing = false; + } + } + + /// Formats an uptime percentage the way the rest of the board formats one. + private static string Percent(double value) => $"{value:0.##}%"; + + /// A check duration, in the unit a person reads durations of checks in. + private static string Milliseconds(TimeSpan span) => + span.TotalMilliseconds < 1000 ? $"{span.TotalMilliseconds:0} ms" : $"{span.TotalSeconds:0.##} s"; + + /// The status class for a health, so an alert row is coloured like everything else. + private static string HealthClass(PulseCheckerHealth? health) => health switch + { + PulseCheckerHealth.Healthy => "hpm-ok", + PulseCheckerHealth.Suspicious => "hpm-warn", + PulseCheckerHealth.Unhealthy => "hpm-crit", + _ => "hpm-paused", + }; + + /// A short, human length: "4m", "2h 10m". + private static string Duration(TimeSpan span) => span switch + { + { TotalSeconds: < 60 } => $"{span.TotalSeconds:0}s", + { TotalMinutes: < 60 } => $"{span.TotalMinutes:0}m", + { TotalHours: < 24 } => $"{(int)span.TotalHours}h {span.Minutes}m", + _ => $"{(int)span.TotalDays}d {span.Hours}h", + }; +} diff --git a/src/Healthie.Dashboard/Components/HealthieDashboard.razor b/src/Healthie.Dashboard/Components/HealthieDashboard.razor index 92b5b0e..1eb3b7e 100644 --- a/src/Healthie.Dashboard/Components/HealthieDashboard.razor +++ b/src/Healthie.Dashboard/Components/HealthieDashboard.razor @@ -1,4 +1,4 @@ -@* Healthie.NET dashboard -- Pulse Monitor. *@ +@* Healthie.NET dashboard -- Pulse Monitor. *@ @implements IAsyncDisposable @using Healthie.Abstractions.Extensions @using Healthie.Dashboard.Components @@ -51,6 +51,35 @@ value="@_searchFilter" @oninput="OnSearchInput" /> + @* + Only when something is elected: an application with no leader election installed has + one replica running everything, and a permanent LEADER badge there would reassure + about a problem nobody has. + *@ + @if (_leadershipInsights is not null) + { + + @(IsLeader ? "LEADER" : "FOLLOWER") + + } + + @if (_alertInsights is not null) + { + + } +
@if (Options.AllowMutations) { @@ -101,11 +130,13 @@ -
AGGREGATE PULSE · @ChecksPerMinute CHECKS/MIN
+
+ AGGREGATE PULSE · @ChecksPerMinute CHECKS/MIN@(CronCheckerCount > 0 ? $" · {CronCheckerCount} CRON" : null) +
- @* The event log at full size. The sidebar copy is a glance; this is the one to read when something has gone wrong and the interesting line has already scrolled away. @@ -236,7 +267,7 @@
- @RatePerMinute(state) - per min + @RateLabel(state) + @RateUnit(state)
@if (history.Count == 0) @@ -312,13 +344,364 @@ ; } -
-
+
+ + @* + Reads and filters only, so it shows in read-only mode: every entry either scrolls to a + section or opens a panel. Rendered before the list in the DOM as well as to its left, so + tab order matches reading order. + *@ + + + @if (_view == BoardView.Alerts && _alertInsights is not null) + { +
+
+
+

ALERTS

+

+ @if (_alerts is { } stored) + { + @stored.Total kept of the last @stored.Capacity · stored in @stored.StoredIn + } +

+
+ +
+ + @if (_alertInsights.DroppedCount > 0) + { + + @_alertInsights.DroppedCount DROPPED + + } + + +
+ + @* + Where alerts go, before the alerts themselves: an empty list means nothing has + fired, and that reads completely differently depending on whether anything is + configured to deliver. + *@ +
+ @if (_alertInsights.Sinks.Count == 0) + { +
+ +
+
NO SINK CONFIGURED
+
+ Alerts are recorded here and sent nowhere. Add one with + AddHealthieSlackAlerts, AddHealthieWebhookAlerts, + AddHealthieMicrosoftTeamsAlerts or + AddHealthiePagerDutyAlerts. +
+
+
+ } + else + { + @foreach (var sink in _alertInsights.Sinks) + { +
+ +
+
@sink.Name
+
+ @sink.Delivered delivered · @sink.Failed failed + @if (sink.LastError is { } error) + { + — @error + } +
+
+
+ } + } +
+ + @if (VisibleAlerts.Count == 0) + { +
+ @(_undeliveredOnly + ? "Every alert on this page reached its sinks." + : "Nothing has alerted yet. Alerts appear here as checkers change health.") +
+ } + else + { +
    + @foreach (var alert in VisibleAlerts) + { +
  • +
    + @alert.DisplayName + + @(alert.PreviousHealth?.ToString().ToUpperInvariant() ?? "NEW") + + @alert.CurrentHealth.ToString().ToUpperInvariant() + +
    + @if (!alert.Delivered) + { + NOT DELIVERED + } + @Relative(alert.OccurredAt) +
    +
    @alert.Message
    +
    • + } +
+ + @if (AlertPageCount > 1) + { +
+ + PAGE @(_alertPage + 1) OF @AlertPageCount + +
+ } + } + + @* + Only the settings a running dispatcher actually honours. Its queue capacity and + history length are fixed when it is built, so they are shown as read-only facts + above rather than offered as controls that would quietly do nothing. + *@ + @if (_alertConfiguration is not null && Options.AllowMutations && _settingsDraft is { } draft) + { +
+

SETTINGS

+

+ Applies from the next alert. Held in memory, so a restart returns to whatever + AddHealthieAlerts was given. +

+ +
+
+ + +
+
+ + +
+
+ + +
+
+ + + + @if (_settingsError is not null) + { + + } + +
+ + +
+ + @if (_testResult is { } results) + { +
+ @if (results.Count == 0) + { + Nothing to deliver to, so the test alert was recorded and sent nowhere. + } + else + { + @results.Count(r => r.IsHealthy) of @results.Count sink(s) took it. + } +
+ } +
+ } +
+ } + else if (_view == BoardView.Metrics && _metricsInsights is not null) + { +
+
+
+

METRICS

+

+ @if (_metrics is { } snapshot) + { + Since @snapshot.Since.ToString("u") · this process only + } +

+
+
+ +
+ +

+ Read from the Healthie.NET meter in this process. It is a live count, + not a time series: nothing here survives a restart, and an OpenTelemetry exporter + on the same meter is the way to keep history. +

+ + @if (_metrics is { } m) + { +
+
+
CHECKS RUN
+
@m.Checks
+
+
+
HEALTHY
+
@(m.HealthyShare is { } share ? Percent(share) : "--")
+
+
+
TRANSITIONS
+
@m.Transitions
+
+
+
OVERLAPS
+
@m.OverlappedTriggers
+
+
+
MEAN CHECK
+
@(m.MeanDuration is { } mean ? Milliseconds(mean) : "--")
+
+
+
SLOWEST CHECK
+
@(m.SlowestDuration is { } slowest ? Milliseconds(slowest) : "--")
+
+
+ +

RESULTS BY HEALTH

+
+ @foreach (var health in new[] { PulseCheckerHealth.Healthy, PulseCheckerHealth.Suspicious, PulseCheckerHealth.Unhealthy }) + { +
+
@health.ToString().ToUpperInvariant()
+
@m.ResultsByHealth.GetValueOrDefault(health)
+
+ } +
+ } +
+ } + +
-
@@ -367,21 +750,21 @@ }
} - else if (_groupByTags) + else if (_sectionByGroup) { foreach (var group in GroupedRows()) { - var collapsed = _collapsedGroups.Contains(group.Tag); + var collapsed = _collapsedGroups.Contains(group.Name);
+ @* Beside the run-based figure above, so the two are read against each other. *@ + @if (_uptime is { } uptime) + { +
+
24H
+
@Percent(uptime.Percentage)
+
+ @if (uptime.LongestOutage is { } outage) + { +
+
WORST
+
@Duration(outage)
+
+ } + }
FAILS
@@ -457,6 +855,26 @@ legible elsewhere: the interval is the rate on the row, the threshold is the denominator in FAILS, and the group and tags are chips on the row. *@ + @* + Explaining is a read, but it spends money on the host's account and takes + seconds, so it sits with the actions rather than with the readings. + *@ + @if (_diagnosisInsights is not null && Options.AllowMutations) + { +
+ + @if (_diagnosis is { } diagnosis) + { +

@diagnosis

+ } +
+ } + @if (Options.AllowMutations) {
@@ -466,9 +884,16 @@
+ @* + Disabled while a cron expression is in force rather than left + live: the schedule overrides the interval, so a picker that + still moved would store a value nothing reads. + *@
- - @foreach (var interval in Intervals) { @@ -481,6 +906,25 @@
+
+ + + @if (_scheduleError is not null) + { + + } + else + { +
Enter to apply, Escape to undo. e.g. 0 6 * * MON-FRI
+ } +
+
GROUP
@if (_isNamingGroup) diff --git a/src/Healthie.Dashboard/Components/HealthieDashboard.razor.cs b/src/Healthie.Dashboard/Components/HealthieDashboard.razor.cs index 8f9cec4..aa8e1f9 100644 --- a/src/Healthie.Dashboard/Components/HealthieDashboard.razor.cs +++ b/src/Healthie.Dashboard/Components/HealthieDashboard.razor.cs @@ -1,6 +1,7 @@ using Healthie.Abstractions.Enums; using Healthie.Abstractions.Extensions; using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; using Healthie.Dashboard.Services; using Microsoft.AspNetCore.Components; using Microsoft.AspNetCore.Components.Web; @@ -66,11 +67,64 @@ public sealed partial class HealthieDashboard : IAsyncDisposable private bool _showAbout; private bool _showLog; private bool _asCards; - private bool _groupByTags; + + /// + /// Sectioned by group on open, flat on request. + /// + /// + /// A group is a partition -- every checker under exactly one heading, tallies that add up -- so + /// the sectioned view is the one that answers "what is wrong, and where" at a glance. A flat + /// list of forty checkers makes the reader do that grouping in their head. Checkers given no + /// group collect under one heading rather than disappearing, so nothing is hidden by the + /// default. + /// + private bool _sectionByGroup = true; + private bool _isNamingGroup; private string? _tagDraft; private string? _groupDraft; + /// + /// Whether the side menu is expanded. Open on a desktop-width first load. + /// + /// + /// The rail collapses to its icons rather than disappearing, so the toggle is always reachable + /// and the layout does not reflow to a different set of controls. + /// + private bool _navOpen = true; + + /// + /// The group the menu has narrowed the list to, or null for everything. + /// + /// + /// A separate filter from _tagFilter and the search box, and combined with them rather + /// than replacing them: picking a group in the menu should not silently clear a search someone + /// has typed. + /// + private string? _navSection; + + /// What the middle column is showing. + /// + /// A view rather than a panel that pushes the board down. The alerts list used to open as a + /// full-width band above everything, which displaced every row on the page and had nowhere to sit + /// once the side menu existed. Views switch what the middle column renders and leave the menu and + /// the detail panel where they are. + /// + private enum BoardView + { + Checkers, + Alerts, + Metrics, + } + + private BoardView _view = BoardView.Checkers; + + /// What is in the cron box, which is not the stored schedule until it is applied. + private string? _cronDraft; + + /// Why the last schedule was refused, shown beside the box that was typed into. + private string? _scheduleError; + /// Marks the "new group" choice in the group picker, which no real group can collide with. private const string NewGroupOption = "\u0000new"; @@ -149,12 +203,22 @@ private string OverallLabel _ => "var(--hpm-ok)", }; - /// How many checks a minute the active checkers add up to. + /// + /// How many checks a minute the active checkers add up to. + /// + /// + /// Cron checkers are left out rather than guessed at: "every weekday at 06:00" has no rate a + /// minute, and folding in the interval they are not running on is what this used to do. + /// reports them separately so they are not silently missing. + /// private string ChecksPerMinute => _states.Values - .Where(state => state.IsActive) - .Sum(state => 60d / state.Interval.ToTimeSpan().TotalSeconds) + .Where(state => state.IsActive && !state.EffectiveSchedule.IsCron) + .Sum(state => 60d / state.EffectiveSchedule.Period!.Value.TotalSeconds) .ToString("0"); + /// How many active checkers run on a cron expression instead of a rate. + private int CronCheckerCount => _states.Values.Count(state => state.IsActive && state.EffectiveSchedule.IsCron); + /// How many runs the sparkline can show, which is however many are kept. private int HistoryWindow => _states.Count == 0 ? 0 : _states.Values.Max(s => s.History.Count); @@ -169,6 +233,9 @@ protected override async Task OnInitializedAsync() _initialized = true; _isDarkMode = ThemeState.IsDarkMode; + // Whatever feature packages the application installed. Absent is the normal case. + ResolveInsights(); + // Hand the states the prerender read across to the interactive render. Without this the // circuit starts from an empty board and reads every checker again; the board it renders in // the meantime -- momentarily empty, then repopulating -- replaces the one the prerender @@ -194,6 +261,11 @@ protected override async Task OnInitializedAsync() await LoadAsync(); } + // The first checker is selected by MarkLoaded rather than by a click, so it has not been + // through the path that reads its uptime and fills the editors. Without this the panel opens + // missing the columns and drafts that every later selection has. + await SelectAsync(_selected); + _clockLoop = RunClockAsync(); } @@ -354,6 +426,15 @@ private void Refresh() entry.Value.Tags.Contains(_tagFilter, StringComparer.OrdinalIgnoreCase)); } + // Narrowed from the side menu, and combined with the two above rather than replacing them. + if (_navSection is { } section) + { + filtered = filtered.Where(entry => string.Equals( + string.IsNullOrWhiteSpace(entry.Value.Group) ? UngroupedName : entry.Value.Group, + section, + StringComparison.OrdinalIgnoreCase)); + } + // Pinned first, then by name. Pinning is only useful if it survives the sort. _filtered = [ @@ -386,17 +467,17 @@ private void OnSearchInput(ChangeEventArgs args) /// several of them can be on one checker, which would put it under several headings and make /// the tallies count it twice. /// - private IEnumerable GroupedRows() => + private IEnumerable GroupedRows() => _filtered .GroupBy(entry => string.IsNullOrWhiteSpace(entry.Value.Group) ? UngroupedName : entry.Value.Group!, StringComparer.OrdinalIgnoreCase) - .Select(group => new TagGroup(group.Key, [.. group])) + .Select(group => new CheckerGroup(group.Key, [.. group])) // Ungrouped last: it is the leftovers, not a heading anyone chose. - .OrderBy(group => group.Tag == UngroupedName) - .ThenBy(group => group.Tag, StringComparer.OrdinalIgnoreCase); + .OrderBy(group => group.Name == UngroupedName) + .ThenBy(group => group.Name, StringComparer.OrdinalIgnoreCase); /// One group's checkers, and the tallies its header shows. - private sealed record TagGroup(string Tag, List> Rows) + private sealed record CheckerGroup(string Name, List> Rows) { public int Healthy => Rows.Count(r => HealthOf(r.Value) == PulseCheckerHealth.Healthy); @@ -423,7 +504,65 @@ private void ToggleGroup(string tag) private void ToggleGrouping() { - _groupByTags = !_groupByTags; + _sectionByGroup = !_sectionByGroup; + Refresh(); + } + + private void ToggleNav() => _navOpen = !_navOpen; + + /// + /// The groups the menu lists, which are the ones in the store rather than the ones surviving the + /// current filters. + /// + /// + /// Off , not off the filtered rows: a menu that dropped a group because the + /// search box had narrowed it away would take away the means of getting back to it. + /// + private IEnumerable NavGroups() => + _states + .GroupBy(entry => string.IsNullOrWhiteSpace(entry.Value.Group) ? UngroupedName : entry.Value.Group!, + StringComparer.OrdinalIgnoreCase) + .Select(group => new CheckerGroup(group.Key, [.. group])) + .OrderBy(group => group.Name == UngroupedName) + .ThenBy(group => group.Name, StringComparer.OrdinalIgnoreCase); + + private Task ShowEverythingAsync() + { + _view = BoardView.Checkers; + _navSection = null; + Refresh(); + + return Task.CompletedTask; + } + + /// Narrows the list to one group, or back to everything when it is picked again. + private Task ShowGroupAsync(string group) + { + _view = BoardView.Checkers; + _navSection = string.Equals(_navSection, group, StringComparison.OrdinalIgnoreCase) ? null : group; + Refresh(); + + return Task.CompletedTask; + } + + /// Switches the middle column, loading whatever the view needs on the way in. + private async Task ShowViewAsync(BoardView view) + { + // Picking the view you are already on goes back to the checkers, so every menu entry toggles + // and none of them is a trap. + _view = _view == view ? BoardView.Checkers : view; + + switch (_view) + { + case BoardView.Alerts: + _alertPage = 0; + await LoadAlertsAsync(); + break; + case BoardView.Metrics: + LoadMetrics(); + break; + } + Refresh(); } @@ -434,14 +573,34 @@ private void OnTagFilterChanged(ChangeEventArgs args) Refresh(); } - private void OnRowKeyDown(KeyboardEventArgs args, string name) + private async Task OnRowKeyDown(KeyboardEventArgs args, string name) { if (args.Key is "Enter" or " ") { - _selected = name; + await SelectAsync(name); } } + /// + /// Selects a checker and reads anything the feature packages can add about it. + /// + /// + /// On selection rather than on the clock tick: uptime is a query over recorded segments and the + /// board redraws every second. + /// + private async Task SelectAsync(string? name) + { + _selected = name; + _diagnosis = null; + + // The cron box follows the selection: left alone it would show one checker's expression + // while Enter applied it to another. + _cronDraft = _selectedState?.Schedule?.CronExpression; + _scheduleError = null; + + await LoadUptimeAsync(name); + } + private void ToggleAbout() => _showAbout = !_showAbout; private void ToggleLog() => _showLog = !_showLog; @@ -597,6 +756,59 @@ private async Task OnIntervalChanged(ChangeEventArgs args) AddEvent("CONF", Status.Paused, $"{DisplayNameOf(_selected)} interval set to {interval}"); } + /// + /// Applies what is in the cron box: a schedule, or none when it has been emptied. + /// + /// + /// On Enter rather than on every keystroke, because a half-typed expression is not a schedule + /// and every attempt reschedules the checker. A refusal is shown where it was typed -- the + /// schedulers disagree about cron dialects, so which rule was broken is the useful part, and the + /// scheduler is asked before anything is stored. + /// + private async Task ApplyCronAsync() + { + if (_selected is null) + { + return; + } + + _scheduleError = null; + + var typed = _cronDraft?.Trim(); + + try + { + if (string.IsNullOrEmpty(typed)) + { + await DashboardService.SetScheduleAsync(_selected, null); + AddEvent("CONF", Status.Paused, $"{DisplayNameOf(_selected)} back on its interval"); + + return; + } + + await DashboardService.SetScheduleAsync(_selected, PulseSchedule.Cron(typed)); + AddEvent("CONF", Status.Paused, $"{DisplayNameOf(_selected)} scheduled on '{typed}'"); + } + catch (ArgumentException ex) + { + _scheduleError = ex.Message; + } + } + + private async Task OnCronKeyDown(KeyboardEventArgs args) + { + switch (args.Key) + { + case "Enter": + await ApplyCronAsync(); + break; + case "Escape": + _cronDraft = _selectedState?.Schedule?.CronExpression; + _scheduleError = null; + break; + } + } + private async Task OnThresholdChanged(ChangeEventArgs args) { if (_selected is null || @@ -660,13 +872,33 @@ private static string StatusWord(PulseCheckerState state) => : state.LastResult is null ? "PENDING" : state.LastResult.Health.ToString().ToUpperInvariant(); - private static string RatePerMinute(PulseCheckerState state) + /// + /// What the rate column reads: a number of runs a minute, or the cron expression when the + /// schedule is one. + /// + /// + /// Off , not off Interval. Interval is + /// documented as ignored once a schedule is set, so a cron checker used to advertise a rate it + /// was not running at -- and the aggregate summed that rate into its total. + /// + private static string RateLabel(PulseCheckerState state) { - var perMinute = 60d / state.Interval.ToTimeSpan().TotalSeconds; + var schedule = state.EffectiveSchedule; + + if (schedule.CronExpression is { } cron) + { + return cron; + } + + var perMinute = 60d / schedule.Period!.Value.TotalSeconds; return perMinute >= 1 ? Math.Round(perMinute).ToString("0") : perMinute.ToString("0.0"); } + /// The unit under the rate, which a cron expression does not have. + private static string RateUnit(PulseCheckerState state) => + state.EffectiveSchedule.IsCron ? "cron" : "per min"; + private static string FailuresLabel(PulseCheckerState state) => state.UnhealthyThreshold > 0 ? $"{state.ConsecutiveFailureCount}/{state.UnhealthyThreshold}" diff --git a/src/Healthie.Dashboard/Components/HealthieIcons.cs b/src/Healthie.Dashboard/Components/HealthieIcons.cs index ba99bb9..4daf8ea 100644 --- a/src/Healthie.Dashboard/Components/HealthieIcons.cs +++ b/src/Healthie.Dashboard/Components/HealthieIcons.cs @@ -58,6 +58,9 @@ internal static class HealthieIcons public static MarkupString X => Svg( ""); + public static MarkupString Menu => Svg( + ""); + public static MarkupString Rows => Svg( ""); diff --git a/src/Healthie.Dashboard/README.md b/src/Healthie.Dashboard/README.md index 6259032..86180dc 100644 --- a/src/Healthie.Dashboard/README.md +++ b/src/Healthie.Dashboard/README.md @@ -1,4 +1,4 @@ -![Healthie.NET - Trust your uptime](https://raw.githubusercontent.com/ivanvyd/Healthie.NET/main/healthie.net.banner.png) +![Healthie.NET - Trust your uptime](https://raw.githubusercontent.com/ivanvyd/Healthie.NET/main/healthie.net.banner.png) # Healthie.NET.Dashboard @@ -57,12 +57,14 @@ app.MapHealthieUI().RequireAuthorization("AdminPolicy"); // With auth ## Features - Event-driven real-time updates via `IPulseChecker.StateChanged` (no polling) -- Per-checker management: start, stop, trigger, reset, change interval, change threshold +- Per-checker management: start, stop, trigger, reset, retime by interval or cron expression, change threshold - Bulk actions: Start All, Stop All, Trigger All - A read-only mode that reports everything and changes nothing — see below +- Panels for the feature packages you install — uptime, alerts, leadership, AI — see below - Groups and tags, both editable here and seeded from code — see below +- A collapsible left menu: overview, a section per group with its tally, and the alerts, log and about views - Pin a checker to the top of the list -- Rows or cards, flat or sectioned by group with per-group tallies +- Rows or cards, sectioned by group when it opens or flat on request, with per-group tallies - Live event log, with a full-size view behind the expand icon - Legend and about behind the `?` in the header - Dark/light theme toggle @@ -83,10 +85,11 @@ builder.Services.AddHealthieUI(options => options.AllowMutations = false); That leaves a board that only reports. Every state, sparkline, group, tag, and event stays exactly where it was; the controls that would change any of it are not rendered. Nothing is lost to the -reader, because the values behind the editors are on the board already — the interval is the row's -rate, the threshold is the denominator in `FAILS`, and the group and tags are the chips under each -name. Searching, filtering, grouping, switching to cards, opening the event log, and the theme -toggle all still work: they change your view, not the checker. +reader, because the values behind the editors are on the board already — the schedule is the row's +rate, or its cron expression where it runs on one; the threshold is the denominator in `FAILS`; and +the group and tags are the chips under each name. The side menu, searching, filtering, grouping, +switching to cards, opening the event log, and the theme toggle all still work: they change your +view, not the checker. **This is not authorization.** It is one setting for the whole application, applied to every viewer alike, so it cannot hand the controls to an admin and withhold them from everyone else. It answers @@ -99,6 +102,66 @@ builder.Services.AddHealthieUI(options => options.AllowMutations = false); app.MapHealthieUI().RequireAuthorization(); ``` +## Panels for the packages you install + +The board grows a panel for each feature package the application registers, and shows none of them +otherwise. There is nothing to switch on: it renders the panel when the container can resolve the +contract, so installing the package is the whole configuration. + +| Install | What appears | +|---|---| +| `Healthie.NET.Uptime` | `24H` on the selected checker — uptime measured over real time — and `WORST`, the longest unbroken outage inside that window | +| `Healthie.NET.Alerting` | An **ALERTS** view: the paged alert history, where each alert is delivered, and the settings a running dispatcher honours — see below | +| `AddHealthieMetrics()` | A **METRICS** view: checks run, the share that reported healthy, transitions, overlapped triggers, and mean and slowest check duration | +| `Healthie.NET.LeaderElection` | A `LEADER` or `FOLLOWER` badge, hover-naming the replica — on a follower every checker sits still, which is otherwise indistinguishable from a broken board | +| `Healthie.NET.AI` | An `EXPLAIN` button on a failing checker, which asks your `IChatClient` why it has been failing | + +### The alerts view + +Three things, in the order you need them. + +**Where alerts go.** Every registered `IAlertSink`, listed from startup rather than from its first +delivery, with its delivered and failed counts and its last error. With none configured it says so +outright — alerts raised and sent nowhere reads identically to alerts being delivered, and those are +opposite situations. A sink that recovers stops being shown as failing. + +**What has fired.** The history, newest first, a page at a time, with a filter for the ones that did +not reach every sink. It is written through your application's own `IStateProvider`, so on CosmosDB, +PostgreSQL, SQL Server, SQLite or Redis it **survives a redeploy**; on the in-memory provider it does +not. The header names the provider it went to and the cap it is kept at, so neither is a guess. + +**What alerting is doing.** Minimum severity, deduplication window, delivery timeout and whether +recoveries alert, all editable and applied from the next alert. Only those four: the dispatcher reads +them on every alert, whereas its queue capacity and history length are fixed when it is built, and a +control that quietly did nothing would be worse than no control. `SEND TEST ALERT` puts one through +the real sinks — the only way to find out a webhook URL is wrong is to use it. + +Settings and the test button are gated by `AllowMutations`; everything else in the view is a read and +stays under read-only mode. + +### The metrics view + +`AddHealthieMetrics()` attaches a `MeterListener` to the `Healthie.NET` meter, which is where this +library already emits `healthie.check.duration`, `.results`, `.transitions` and `.overlaps`. It reads +what is being published rather than instrumenting anything again, so it runs **alongside** an +OpenTelemetry exporter on the same meter, not instead of one. + +It is a live count and not a time series: nothing survives a restart, and an exporter is still the +way to keep history. Opt-in for that reason — a listener costs a callback on every measurement, and +an application with an APM in front of it has somewhere better to look. + +`OVERLAPS` is the figure worth watching. A checker whose check takes longer than its own interval +returns immediately, looks healthy, and is quietly running at a fraction of the rate it was asked to; +this is the only place it shows. + +`24H` sits beside the board's own `UPTIME`, which is the share of the runs still in the rolling +history — a hundred results, so at a one-second interval, the last hundred seconds. They answer +different questions and disagree for good reasons, which is why both are shown. + +All of it reads and none of it writes, so it all stays under `AllowMutations = false`. The exception +is `EXPLAIN`: still a read, but it spends money on your account, so it is gated with the controls +that change things. + ## Groups and tags The two look similar and answer different questions. diff --git a/src/Healthie.Dashboard/Services/HealthieDashboardService.cs b/src/Healthie.Dashboard/Services/HealthieDashboardService.cs index d4273a2..88cd57b 100644 --- a/src/Healthie.Dashboard/Services/HealthieDashboardService.cs +++ b/src/Healthie.Dashboard/Services/HealthieDashboardService.cs @@ -50,6 +50,22 @@ await pulsesScheduler.SetIntervalAsync(name, interval, cancellationToken) } } + /// + public async Task SetScheduleAsync(string name, PulseSchedule? schedule, + CancellationToken cancellationToken = default) + { + try + { + await pulsesScheduler.SetScheduleAsync(name, schedule, cancellationToken) + .ConfigureAwait(false); + } + catch (Exception ex) + { + logger?.LogError(ex, "Failed to set the schedule for checker '{CheckerName}'.", name); + throw; + } + } + /// public async Task SetThresholdAsync(string name, uint threshold, CancellationToken cancellationToken = default) diff --git a/src/Healthie.Dashboard/Services/IHealthieDashboardService.cs b/src/Healthie.Dashboard/Services/IHealthieDashboardService.cs index 5b97ebf..48754fc 100644 --- a/src/Healthie.Dashboard/Services/IHealthieDashboardService.cs +++ b/src/Healthie.Dashboard/Services/IHealthieDashboardService.cs @@ -1,5 +1,6 @@ using Healthie.Abstractions.Enums; using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; namespace Healthie.Dashboard.Services; @@ -69,6 +70,20 @@ Task> GetAllStatesAsync( Task SetIntervalAsync(string name, PulseInterval interval, CancellationToken cancellationToken = default); + /// + /// Sets the schedule a pulse checker runs on, for a cadence no + /// expresses. + /// + /// The name of the pulse checker. + /// The schedule, or null to go back to the interval. + /// A token to monitor for cancellation requests. + /// + /// The registered scheduler will not run this schedule. The message is written to be shown to + /// whoever typed it. + /// + Task SetScheduleAsync(string name, PulseSchedule? schedule, + CancellationToken cancellationToken = default); + /// /// Sets the unhealthy threshold for a specific pulse checker. /// diff --git a/src/Healthie.Dashboard/wwwroot/healthie.css b/src/Healthie.Dashboard/wwwroot/healthie.css index ed9565e..6feea7a 100644 --- a/src/Healthie.Dashboard/wwwroot/healthie.css +++ b/src/Healthie.Dashboard/wwwroot/healthie.css @@ -1,4 +1,4 @@ -/* +/* * Healthie.NET dashboard -- Pulse Monitor. * * No third-party CSS and no web fonts: this ships inside someone else's application, which may run @@ -279,15 +279,21 @@ /* ------------------------------------------------------------------ body -- */ .hpm-body { + --hpm-nav-w: 208px; + max-width: 1500px; margin: 0 auto; padding: 22px 26px 70px; display: grid; - grid-template-columns: minmax(0, 1fr) 400px; + grid-template-columns: var(--hpm-nav-w) minmax(0, 1fr) 400px; gap: 18px; align-items: start; } +.hpm-body--nav-collapsed { + --hpm-nav-w: 54px; +} + .hpm-list { display: flex; flex-direction: column; @@ -722,12 +728,14 @@ /* ------------------------------------------------------------ responsive -- */ @media (max-width: 1100px) { + /* The detail panel drops below the list; the rail keeps its column. */ .hpm-body { - grid-template-columns: minmax(0, 1fr); + grid-template-columns: var(--hpm-nav-w) minmax(0, 1fr); } .hpm-side { position: static; + grid-column: 1 / -1; } } @@ -1537,3 +1545,555 @@ from { opacity: 0; transform: translate(-50%, calc(-50% - 6px)); } to { opacity: 1; transform: translate(-50%, -50%); } } + +/* --------------------------------------------------------------------------------------------- + Panels that appear only when a feature package is installed: leadership, alerts, and the + explanation. Built from the same variables as everything above, so they inherit both themes and + need no colours of their own. + --------------------------------------------------------------------------------------------- */ + +.hpm-replica { + padding: 4px 9px; + border: 1px solid var(--hpm-line2); + border-radius: 999px; + font-family: var(--hpm-mono); + font-size: 10.5px; + font-weight: 700; + letter-spacing: 0.08em; + white-space: nowrap; +} + +.hpm-replica--leader { + color: var(--hpm-ok); + border-color: var(--hpm-ok); + background: var(--hpm-ok-bg); +} + +/* A follower is not a fault, so it reads as information rather than as a warning. */ +.hpm-replica--follower { + color: var(--hpm-faint); +} + +.hpm-btn--alerts-dropped { + color: var(--hpm-crit); + border-color: var(--hpm-crit); +} + +/* --------------------------------------------------------------------------- + Side menu + + A rail rather than a drawer, and no JavaScript, because this package ships + without any: an overlay drawer needs a backdrop and a focus trap to be + honest about keyboard use, and neither is doable in CSS alone. Collapsed it + keeps its icons and its toggle, so nothing becomes unreachable and the page + does not reflow onto a different set of controls. + --------------------------------------------------------------------------- */ + +.hpm-nav { + position: sticky; + top: 14px; + display: flex; + flex-direction: column; + gap: 2px; + min-width: 0; + padding: 8px; + border: 1px solid var(--hpm-line); + border-radius: 8px; + background: var(--hpm-panel); +} + +.hpm-nav-toggle { + display: flex; + align-items: center; + gap: 10px; + width: 100%; + padding: 8px 9px; + border: 0; + border-radius: 6px; + background: none; + color: var(--hpm-faint); + font: inherit; + font-size: 11.5px; + letter-spacing: 0.09em; + cursor: pointer; +} + +.hpm-nav-toggle:hover { + color: var(--hpm-text); + background: var(--hpm-line); +} + +.hpm-nav-burger { + flex: 0 0 auto; + width: 16px; + height: 16px; +} + +.hpm-nav-heading { + margin: 12px 0 4px; + padding: 0 9px; + color: var(--hpm-faint); + font-size: 9.5px; + letter-spacing: 0.16em; +} + +.hpm-nav-item { + display: flex; + align-items: center; + gap: 9px; + width: 100%; + padding: 7px 9px; + border: 0; + border-left: 2px solid transparent; + border-radius: 0 6px 6px 0; + background: none; + color: var(--hpm-text); + font: inherit; + font-size: 11.5px; + letter-spacing: 0.05em; + text-align: left; + cursor: pointer; +} + +.hpm-nav-item:hover { + background: var(--hpm-line); +} + +.hpm-nav-item--current { + border-left-color: var(--hpm-accent, var(--hpm-ok)); + background: var(--hpm-line); +} + +.hpm-nav-dot { + flex: 0 0 auto; + width: 7px; + height: 7px; + border-radius: 50%; + background: var(--hpm-faint); +} + +.hpm-nav-dot.hpm-ok { background: var(--hpm-ok); } +.hpm-nav-dot.hpm-warn { background: var(--hpm-warn); } +.hpm-nav-dot.hpm-crit { background: var(--hpm-crit); } + +.hpm-nav-text { + flex: 1 1 auto; + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.hpm-nav-count { + flex: 0 0 auto; + color: var(--hpm-faint); + font-size: 10px; +} + +.hpm-nav-count--warn { + color: var(--hpm-crit); +} + +/* Collapsed: the labels go, the icons and the toggle stay. */ +.hpm-body--nav-collapsed .hpm-nav-text, +.hpm-body--nav-collapsed .hpm-nav-count, +.hpm-body--nav-collapsed .hpm-nav-label, +.hpm-body--nav-collapsed .hpm-nav-heading { + display: none; +} + +.hpm-body--nav-collapsed .hpm-nav-item, +.hpm-body--nav-collapsed .hpm-nav-toggle { + justify-content: center; + padding-inline: 0; +} + +.hpm-body--nav-collapsed .hpm-nav-dot { + width: 9px; + height: 9px; +} + +/* The cron box sits under the interval and threshold pair, full width. */ +.hpm-field--cron { + margin-top: 10px; +} + +.hpm-field--cron input { + font-family: var(--hpm-mono); +} + +.hpm-field--cron input[aria-invalid="true"] { + border-color: var(--hpm-crit); +} + +.hpm-field-hint, +.hpm-field-error { + margin-top: 5px; + font-size: 10.5px; + line-height: 1.45; +} + +.hpm-field-hint { + color: var(--hpm-faint); +} + +.hpm-field-hint code { + font-family: var(--hpm-mono); + color: var(--hpm-muted); +} + +.hpm-field-error { + color: var(--hpm-crit); +} + +/* A cron expression is text, not a number, so it does not get the big numeric treatment. */ +.hpm-rate--cron { + font-family: var(--hpm-mono); + font-size: 11px; + letter-spacing: 0; +} + +/* + * A disabled control that still looks live invites the click it will ignore -- which is the same + * complaint as the interval picker silently doing nothing, moved one step later. The browser's own + * 0.7 opacity is not enough against this palette. + */ +.hpm-field select:disabled, +.hpm-field input:disabled { + color: var(--hpm-faint); + background: var(--hpm-panel2); + border-style: dashed; + cursor: not-allowed; +} + +.hpm-field:has(select:disabled) > label, +.hpm-field:has(input:disabled) > label { + color: var(--hpm-faint); +} +/* + * Below this a rail costs more width than it earns, so it becomes a strip above the list: one + * horizontally scrolling row of the same items. Still every entry, still no JavaScript, and nothing + * overlays the content -- which is what an off-canvas drawer would need a focus trap to do honestly. + */ +@media (max-width: 820px) { + .hpm-body, + .hpm-body--nav-collapsed { + grid-template-columns: minmax(0, 1fr); + } + + .hpm-nav { + position: static; + padding: 6px; + } + + /* The items live in this wrapper, so this is what has to become the row. */ + .hpm-nav-body { + display: flex; + flex-direction: row; + align-items: center; + gap: 6px; + overflow-x: auto; + scrollbar-width: thin; + scrollbar-color: var(--hpm-line2) transparent; + } + + .hpm-nav-toggle, + .hpm-nav-heading { + display: none; + } + + .hpm-nav-item { + width: auto; + flex: 0 0 auto; + border-left: 0; + border-bottom: 2px solid transparent; + border-radius: 6px; + } + + .hpm-nav-item--current { + border-left-color: transparent; + border-bottom-color: var(--hpm-accent, var(--hpm-ok)); + } + + /* The labels come back here even when the rail was left collapsed on a wider screen. */ + .hpm-body--nav-collapsed .hpm-nav-text, + .hpm-body--nav-collapsed .hpm-nav-count { + display: inline; + } +} + +/* --------------------------------------------------------------------------- + Views + + Alerts and metrics render in the middle column, in place of the checker list, + rather than as a band above it. The band displaced every row on the page when + it opened and had nowhere to sit once the side menu existed. + --------------------------------------------------------------------------- */ + +.hpm-list--hidden { + display: none; +} + +.hpm-view { + display: flex; + flex-direction: column; + gap: 14px; + min-width: 0; + padding: 16px 18px 20px; + border: 1px solid var(--hpm-line); + border-radius: 8px; + background: var(--hpm-panel); +} + +.hpm-view-head { + display: flex; + align-items: flex-start; + gap: 10px; + flex-wrap: wrap; +} + +.hpm-view-title { + margin: 0; + font-family: var(--hpm-mono); + font-size: 13px; + font-weight: 600; + letter-spacing: 0.14em; +} + +.hpm-view-subtitle { + margin: 6px 0 0; + color: var(--hpm-faint); + font-family: var(--hpm-mono); + font-size: 10px; + font-weight: 600; + letter-spacing: 0.16em; +} + +.hpm-view-sub, +.hpm-view-note { + margin: 4px 0 0; + color: var(--hpm-muted); + font-size: 11.5px; + line-height: 1.5; +} + +.hpm-view-sub code, +.hpm-view-note code, +.hpm-sink-note code { + font-family: var(--hpm-mono); + font-size: 10.5px; + color: var(--hpm-text); +} + +.hpm-view-empty { + padding: 26px 4px; + color: var(--hpm-faint); + font-size: 12px; + text-align: center; +} + +.hpm-view-section { + margin-top: 6px; + padding-top: 14px; + border-top: 1px solid var(--hpm-line); +} + +.hpm-view-actions { + display: flex; + gap: 8px; + margin-top: 12px; + flex-wrap: wrap; +} + +/* ------------------------------------------------------------------ sinks -- */ + +.hpm-sinks { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); + gap: 8px; +} + +.hpm-sink { + display: flex; + align-items: flex-start; + gap: 9px; + padding: 9px 11px; + border: 1px solid var(--hpm-line); + border-left: 2px solid var(--hpm-status, var(--hpm-faint)); + border-radius: 6px; + background: var(--hpm-bg); +} + +.hpm-sink--ok { --hpm-status: var(--hpm-ok); } +.hpm-sink--failing { --hpm-status: var(--hpm-crit); } +.hpm-sink--none { --hpm-status: var(--hpm-warn); } + +.hpm-sink-dot { + flex: 0 0 auto; + width: 7px; + height: 7px; + margin-top: 4px; + border-radius: 50%; + background: var(--hpm-status); +} + +.hpm-sink-body { + min-width: 0; +} + +.hpm-sink-name { + font-family: var(--hpm-mono); + font-size: 11.5px; + letter-spacing: 0.05em; +} + +.hpm-sink-note { + margin-top: 3px; + color: var(--hpm-muted); + font-size: 10.5px; + line-height: 1.5; +} + +.hpm-sink-error { + color: var(--hpm-crit); +} + +/* ------------------------------------------------------------------ alerts -- */ + +.hpm-alert-list { + list-style: none; + margin: 0; + padding: 0; + display: flex; + flex-direction: column; + gap: 6px; +} + +.hpm-alert { + padding: 9px 12px; + border: 1px solid var(--hpm-line); + border-left: 3px solid var(--hpm-status, var(--hpm-faint)); + border-radius: 6px; + background: var(--hpm-bg); +} + +.hpm-alert--undelivered { + background: var(--hpm-crit-bg); +} + +.hpm-alert-main { + display: flex; + align-items: center; + gap: 9px; + flex-wrap: wrap; +} + +.hpm-alert-name { + font-size: 12.5px; + font-weight: 600; +} + +.hpm-alert-move { + display: inline-flex; + align-items: center; + gap: 5px; +} + +.hpm-alert-arrow { + color: var(--hpm-faint); +} + +/* A health, as a chip, coloured by the health it names. */ +.hpm-hchip { + padding: 1px 6px; + border-radius: 3px; + color: var(--hpm-status); + background: var(--hpm-status-bg); + font-family: var(--hpm-mono); + font-size: 9.5px; + letter-spacing: 0.08em; +} + +.hpm-alert-when { + color: var(--hpm-faint); + font-family: var(--hpm-mono); + font-size: 10.5px; + white-space: nowrap; +} + +.hpm-alert-msg { + margin-top: 5px; + color: var(--hpm-muted); + font-size: 11.5px; + line-height: 1.5; +} + +.hpm-pill { + padding: 2px 7px; + border-radius: 3px; + color: var(--hpm-status, var(--hpm-faint)); + background: var(--hpm-status-bg, transparent); + font-family: var(--hpm-mono); + font-size: 9.5px; + letter-spacing: 0.08em; + white-space: nowrap; +} + +.hpm-pill--crit { --hpm-status: var(--hpm-crit); --hpm-status-bg: var(--hpm-crit-bg); } + +.hpm-pager { + display: flex; + align-items: center; + justify-content: center; + gap: 12px; + padding-top: 4px; +} + +.hpm-pager-at { + color: var(--hpm-faint); + font-family: var(--hpm-mono); + font-size: 10.5px; + letter-spacing: 0.08em; +} + +.hpm-check { + display: flex; + align-items: center; + gap: 8px; + margin-top: 10px; + color: var(--hpm-muted); + font-size: 11.5px; +} + +/* ----------------------------------------------------------------- metrics -- */ + +.hpm-metrics { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); + gap: 8px; +} + +.hpm-metric { + padding: 11px 13px; + border: 1px solid var(--hpm-line); + border-radius: 6px; + background: var(--hpm-bg); +} + +.hpm-metric--warn { + border-color: var(--hpm-warn); +} + +.hpm-metric-label { + color: var(--hpm-faint); + font-family: var(--hpm-mono); + font-size: 9.5px; + letter-spacing: 0.14em; +} + +.hpm-metric-value { + margin-top: 5px; + font-family: var(--hpm-mono); + font-size: 19px; + font-weight: 600; +} diff --git a/src/Healthie.DependencyInjection/MeterMetricsInsights.cs b/src/Healthie.DependencyInjection/MeterMetricsInsights.cs new file mode 100644 index 0000000..48a65da --- /dev/null +++ b/src/Healthie.DependencyInjection/MeterMetricsInsights.cs @@ -0,0 +1,145 @@ +using Healthie.Abstractions.Diagnostics; +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Insights; +using System.Diagnostics.Metrics; + +namespace Healthie.DependencyInjection; + +/// +/// Collects the library's own instruments in-process, so a board can show them. +/// +/// +/// +/// A subscribed by meter name, which is how it reads instruments that are +/// internal to another assembly without that assembly exposing them. It is the same mechanism +/// OpenTelemetry uses, so this does not compete with an exporter: both can listen at once, and +/// neither sees the other. +/// +/// +/// Opt-in through AddHealthieMetrics(). A listener costs a callback on every recorded +/// measurement, which is small but not free, and an application exporting to an APM already has +/// somewhere better to look. +/// +/// +public sealed class MeterMetricsInsights : IMetricsInsights, IDisposable +{ + private readonly MeterListener _listener; + private readonly object _gate = new(); + private readonly Dictionary _resultsByHealth = []; + + private long _checks; + private long _transitions; + private long _overlaps; + private double _durationTotal; + private long _durationCount; + private double _durationMax; + + private readonly DateTime _since; + + /// Starts listening. + /// Where the start time comes from. + public MeterMetricsInsights(TimeProvider? timeProvider = null) + { + _since = (timeProvider ?? TimeProvider.System).GetUtcNow().UtcDateTime; + + _listener = new MeterListener + { + InstrumentPublished = (instrument, listener) => + { + if (instrument.Meter.Name == HealthieDiagnostics.MeterName) + { + listener.EnableMeasurementEvents(instrument); + } + }, + }; + + _listener.SetMeasurementEventCallback(OnLong); + _listener.SetMeasurementEventCallback(OnDouble); + _listener.Start(); + } + + /// + public MetricsSnapshot Snapshot(CancellationToken cancellationToken = default) + { + lock (_gate) + { + return new MetricsSnapshot( + _checks, + new Dictionary(_resultsByHealth), + _transitions, + _overlaps, + _durationCount == 0 ? null : TimeSpan.FromSeconds(_durationTotal / _durationCount), + _durationCount == 0 ? null : TimeSpan.FromSeconds(_durationMax), + _since); + } + } + + private void OnLong( + Instrument instrument, + long measurement, + ReadOnlySpan> tags, + object? state) + { + // The health tag has to be read here: the span cannot outlive the callback, so it is resolved + // before taking the lock rather than stashed. + var health = HealthOf(tags); + + lock (_gate) + { + switch (instrument.Name) + { + case "healthie.check.results": + _checks += measurement; + + if (health is { } reported) + { + _resultsByHealth[reported] = _resultsByHealth.GetValueOrDefault(reported) + measurement; + } + + break; + case "healthie.check.transitions": + _transitions += measurement; + break; + case "healthie.check.overlaps": + _overlaps += measurement; + break; + } + } + } + + private void OnDouble( + Instrument instrument, + double measurement, + ReadOnlySpan> tags, + object? state) + { + if (instrument.Name != "healthie.check.duration") + { + return; + } + + lock (_gate) + { + _durationTotal += measurement; + _durationCount++; + _durationMax = Math.Max(_durationMax, measurement); + } + } + + private static PulseCheckerHealth? HealthOf(ReadOnlySpan> tags) + { + foreach (var tag in tags) + { + if (tag.Key == HealthieDiagnostics.ResultTag + && Enum.TryParse(tag.Value?.ToString(), ignoreCase: true, out var health)) + { + return health; + } + } + + return null; + } + + /// + public void Dispose() => _listener.Dispose(); +} diff --git a/src/Healthie.DependencyInjection/StartupExtensions.cs b/src/Healthie.DependencyInjection/StartupExtensions.cs index 8b7e819..3283949 100644 --- a/src/Healthie.DependencyInjection/StartupExtensions.cs +++ b/src/Healthie.DependencyInjection/StartupExtensions.cs @@ -1,5 +1,6 @@ using Healthie.Abstractions; using Healthie.Abstractions.Initialization; +using Healthie.Abstractions.Insights; using Healthie.Abstractions.Scheduling; using Healthie.Abstractions.StateProviding; using Microsoft.Extensions.DependencyInjection; @@ -96,6 +97,27 @@ private static void RegisterCheckers( } } + /// + /// Collects the library's own metrics in-process, so the dashboard can show them. + /// + /// The service collection. + /// The same service collection, for chaining. + /// + /// Opt-in, because a MeterListener costs a callback on every recorded measurement and an + /// application already exporting to an APM has somewhere better to read them. It listens by + /// meter name alongside any exporter rather than instead of one, so adding this does not take + /// anything away from OpenTelemetry. + /// + public static IServiceCollection AddHealthieMetrics(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.TryAddSingleton(); + services.TryAddSingleton(provider => provider.GetRequiredService()); + + return services; + } + private static IEnumerable GetLoadableTypes(Assembly assembly) { try diff --git a/src/Healthie.DependencyInjection/TimerPulseScheduler.cs b/src/Healthie.DependencyInjection/TimerPulseScheduler.cs index f1e29a7..6e47593 100644 --- a/src/Healthie.DependencyInjection/TimerPulseScheduler.cs +++ b/src/Healthie.DependencyInjection/TimerPulseScheduler.cs @@ -252,18 +252,51 @@ private async Task TriggerAsync(IPulseChecker checker, CancellationToken token) } } + /// + /// + /// Cronos is what actually drives the timer, so asking Cronos is the only answer worth giving. + /// + public bool TryValidateSchedule(PulseSchedule schedule, out string? error) + { + ArgumentNullException.ThrowIfNull(schedule); + + error = null; + + if (schedule.CronExpression is not { } expression) + { + return true; + } + + try + { + CronExpression.Parse(expression, CronFormatFor(expression)); + + return true; + } + catch (CronFormatException ex) + { + // Cronos names the field and the range it wanted, which is more use than restating the + // format -- the field this is shown beside already gives an example of one. + error = ex.Message; + + return false; + } + } + + /// Six fields or more means the leading one is seconds. + private static CronFormat CronFormatFor(string expression) => + expression.Split(' ', StringSplitOptions.RemoveEmptyEntries).Length >= 6 + ? CronFormat.IncludeSeconds + : CronFormat.Standard; + /// /// Parses a standard Unix cron expression, in five fields or six with a leading seconds field. /// private static CronExpression ParseCron(string expression, string checkerName) { - var format = expression.Split(' ', StringSplitOptions.RemoveEmptyEntries).Length >= 6 - ? CronFormat.IncludeSeconds - : CronFormat.Standard; - try { - return CronExpression.Parse(expression, format); + return CronExpression.Parse(expression, CronFormatFor(expression)); } catch (CronFormatException ex) { diff --git a/src/Healthie.LeaderElection/LeadershipInsights.cs b/src/Healthie.LeaderElection/LeadershipInsights.cs new file mode 100644 index 0000000..61302b5 --- /dev/null +++ b/src/Healthie.LeaderElection/LeadershipInsights.cs @@ -0,0 +1,24 @@ +using Healthie.Abstractions.Insights; + +namespace Healthie.LeaderElection; + +/// +/// Tells the dashboard whether this replica is the one running the checks. +/// +/// +/// Without it, a board served by a follower shows every checker sitting still and reads as broken: +/// nothing is running here, and nothing is meant to be. The replica id is there because an operator +/// comparing two tabs needs to know which is which. +/// +/// The scheduler that holds, or does not hold, the lease. +/// Where the replica's own identity comes from. +internal sealed class LeadershipInsights( + LeaderElectedPulseScheduler scheduler, + LeaderElectionOptions options) : ILeadershipInsights +{ + /// + public bool IsLeader => scheduler.IsLeader; + + /// + public string ReplicaId => options.HolderId; +} diff --git a/src/Healthie.LeaderElection/StartupExtensions.cs b/src/Healthie.LeaderElection/StartupExtensions.cs index 8f0f9c8..ad0acd4 100644 --- a/src/Healthie.LeaderElection/StartupExtensions.cs +++ b/src/Healthie.LeaderElection/StartupExtensions.cs @@ -1,4 +1,5 @@ using Healthie.Abstractions.Scheduling; +using Healthie.Abstractions.Insights; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; @@ -52,6 +53,12 @@ public static IServiceCollection AddHealthieLeaderElection( services.AddSingleton(provider => provider.GetRequiredService()); services.AddHostedService(); + // So a board served by a follower says so, instead of showing everything idle and looking + // broken. + services.TryAddSingleton(provider => new LeadershipInsights( + provider.GetRequiredService(), + provider.GetRequiredService())); + return services; } diff --git a/src/Healthie.Scheduling.Quartz/QuartzPulseScheduler.cs b/src/Healthie.Scheduling.Quartz/QuartzPulseScheduler.cs index 01bb08e..0587982 100644 --- a/src/Healthie.Scheduling.Quartz/QuartzPulseScheduler.cs +++ b/src/Healthie.Scheduling.Quartz/QuartzPulseScheduler.cs @@ -70,6 +70,37 @@ public Task ScheduleAsync( : ScheduleCoreAsync(checker, cronExpression: null, schedule.Period, cancellationToken); } + /// + /// + /// Answered by the same translation that runs it. Quartz takes a six-field expression with its + /// own day-of-week numbering and refuses to constrain both day fields at once, so an expression + /// Cronos is happy with is not automatically one this scheduler can run. + /// + public bool TryValidateSchedule(PulseSchedule schedule, out string? error) + { + ArgumentNullException.ThrowIfNull(schedule); + + error = null; + + if (schedule.CronExpression is not { } expression) + { + return true; + } + + try + { + CronScheduleBuilder.CronSchedule(UnixCron.ToQuartz(expression)); + + return true; + } + catch (Exception ex) when (ex is NotSupportedException or FormatException or ArgumentException) + { + error = ex.Message; + + return false; + } + } + /// /// Replaces this checker's Quartz job with one on the given trigger. /// diff --git a/src/Healthie.StateProviding.Relational/RelationalDialect.cs b/src/Healthie.StateProviding.Relational/RelationalDialect.cs index 0f5da4e..8e17c04 100644 --- a/src/Healthie.StateProviding.Relational/RelationalDialect.cs +++ b/src/Healthie.StateProviding.Relational/RelationalDialect.cs @@ -29,7 +29,9 @@ namespace Healthie.StateProviding.Relational; /// /// /// Statement adding the version column to a table that predates it, with {0} for the table -/// name. Run only when the column is missing. +/// name. Run only when the column is missing. Optional: a dialect that does not supply one is left +/// alone by the migration, which is right for a table this library created and wrong only for one +/// that predates versioning, where the statement is the whole point. /// /// /// Statement inserting one row only if the name is not taken, reporting the outcome through rows @@ -40,7 +42,7 @@ public sealed record RelationalDialect( string Name, string CreateTableFormat, string UpsertFormat, - string AddVersionColumnFormat, + string? AddVersionColumnFormat = null, string? InsertIfAbsentFormat = null) { /// @@ -193,14 +195,15 @@ internal static void ValidateTableName(string tableName) internal const string DeleteFormat = "DELETE FROM {0} WHERE name = @name"; /// - /// Adds the version column to a table created before it existed. + /// Adds the version column to a table created before it existed, or null when the + /// dialect does not supply the statement. /// /// /// A plain ALTER, run only when the column is genuinely missing -- the initializer checks first /// rather than relying on an IF NOT EXISTS that SQLite does not have for ADD COLUMN. /// - internal string AddVersionColumn(string tableName) => - Format(AddVersionColumnFormat, tableName); + internal string? AddVersionColumn(string tableName) => + AddVersionColumnFormat is null ? null : Format(AddVersionColumnFormat, tableName); internal static string Select(string tableName) => Format(SelectFormat, tableName); diff --git a/src/Healthie.StateProviding.Relational/RelationalStateProviderInitializer.cs b/src/Healthie.StateProviding.Relational/RelationalStateProviderInitializer.cs index 3396501..00ce165 100644 --- a/src/Healthie.StateProviding.Relational/RelationalStateProviderInitializer.cs +++ b/src/Healthie.StateProviding.Relational/RelationalStateProviderInitializer.cs @@ -26,7 +26,7 @@ public sealed class RelationalStateProviderInitializer( private readonly string _createTableSql = (dialect ?? throw new ArgumentNullException(nameof(dialect))) .CreateTable(Validated(tableName)); - private readonly string _addVersionColumnSql = dialect.AddVersionColumn(Validated(tableName)); + private readonly string? _addVersionColumnSql = dialect.AddVersionColumn(Validated(tableName)); private readonly string _tableName = Validated(tableName); private static string Validated(string tableName) @@ -73,7 +73,10 @@ public async Task InitializeAsync(CancellationToken cancellationToken = default) /// private async Task AddVersionColumnIfMissingAsync(DbConnection connection, CancellationToken cancellationToken) { - if (await HasVersionColumnAsync(connection, cancellationToken).ConfigureAwait(false)) + // Nothing to add, and nothing to add it with: a hand-built dialect that supplied no + // statement is telling us its table already has the column, or that it is not our business. + if (_addVersionColumnSql is null + || await HasVersionColumnAsync(connection, cancellationToken).ConfigureAwait(false)) { return; } diff --git a/src/Healthie.Uptime/StartupExtensions.cs b/src/Healthie.Uptime/StartupExtensions.cs index e6fdf84..2d46192 100644 --- a/src/Healthie.Uptime/StartupExtensions.cs +++ b/src/Healthie.Uptime/StartupExtensions.cs @@ -1,3 +1,4 @@ +using Healthie.Abstractions.Insights; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; @@ -25,6 +26,11 @@ public static IServiceCollection AddHealthieUptime(this IServiceCollection servi services.TryAddSingleton(); services.AddHostedService(); + // What the dashboard renders when this package is installed. Registered here rather than + // referenced there, so installing a dashboard does not drag uptime in with it. + services.TryAddSingleton(provider => + new UptimeInsights(provider.GetRequiredService())); + return services; } } diff --git a/src/Healthie.Uptime/UptimeInsights.cs b/src/Healthie.Uptime/UptimeInsights.cs new file mode 100644 index 0000000..5f31772 --- /dev/null +++ b/src/Healthie.Uptime/UptimeInsights.cs @@ -0,0 +1,82 @@ +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Insights; + +namespace Healthie.Uptime; + +/// +/// Reports this package's uptime to the dashboard, without the dashboard having to reference it. +/// +/// +/// The board already shows a percentage of its own: the share of the runs still in the rolling +/// history, which at a one-second interval is the last couple of minutes. This one is measured over +/// real time and survives history being trimmed, which is what makes it worth showing separately +/// rather than replacing the other. +/// +internal sealed class UptimeInsights(IUptimeStore store, TimeProvider? timeProvider = null) : IUptimeInsights +{ + private readonly TimeProvider _time = timeProvider ?? TimeProvider.System; + + /// + public async Task GetUptimeAsync( + string checkerName, + TimeSpan window, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(checkerName); + + var to = _time.GetUtcNow().UtcDateTime; + var from = to - window; + + var segments = await store.GetSegmentsAsync(checkerName, from, to, cancellationToken) + .ConfigureAwait(false); + + if (segments.Count == 0) + { + return null; + } + + var report = UptimeCalculator.Calculate(checkerName, segments, from, to); + + // Null when nothing was observed, which is not the same as nothing being recorded: the + // dashboard shows the difference rather than printing a confident zero. + if (report.UptimePercentage is not { } percentage) + { + return null; + } + + return new UptimeInsight(percentage, window, LongestOutage(segments, from, to)); + } + + /// + /// The longest unbroken unhealthy stretch inside the window. + /// + /// + /// A percentage alone cannot tell a hundred one-second blips from one long outage, and those are + /// very different mornings. Segments are clipped to the window so a stretch that started before + /// it is counted only for the part that falls inside. + /// + private static TimeSpan? LongestOutage(IReadOnlyList segments, DateTime from, DateTime to) + { + TimeSpan? longest = null; + + foreach (var segment in segments.Where(s => s.Health == PulseCheckerHealth.Unhealthy)) + { + var start = segment.StartedAt < from ? from : segment.StartedAt; + var end = segment.EndedAt is { } ended && ended < to ? ended : to; + + if (end <= start) + { + continue; + } + + var length = end - start; + + if (longest is null || length > longest) + { + longest = length; + } + } + + return longest; + } +} diff --git a/tests/Healthie.Tests.E2E/DashboardGroupingTests.cs b/tests/Healthie.Tests.E2E/DashboardGroupingTests.cs index 510ef33..b985c55 100644 --- a/tests/Healthie.Tests.E2E/DashboardGroupingTests.cs +++ b/tests/Healthie.Tests.E2E/DashboardGroupingTests.cs @@ -1,4 +1,4 @@ -using Microsoft.Playwright; +using Microsoft.Playwright; namespace Healthie.Tests.E2E; @@ -22,9 +22,27 @@ public class DashboardGroupingTests(BrowserFixture browser) private const string TargetGroup = "Data Stores"; + /// The one checker in the sample that runs on a cron expression rather than an interval. + private const string CronChecker = "TLS Certificate"; + private static ILocator RowFor(IPage page, string displayName) => page.Locator(".hpm-row").Filter(new() { HasTextString = displayName }); + + /// + /// Selects a checker and waits for the panel to actually be showing it. + /// + /// + /// The click is a round trip to the server. Waiting on a panel control instead proves nothing -- + /// they are already on screen for whichever checker was selected on load, so a test that typed + /// straight after the click edited the wrong checker and failed somewhere else entirely. + /// + private static async Task SelectCheckerAsync(IPage page, string displayName) + { + await RowFor(page, displayName).ClickAsync(); + await Assertions.Expect(page.Locator(".hpm-sel-name")).ToHaveTextAsync(displayName); + } + private static ILocator GroupFor(IPage page, string group) => page.Locator(".hpm-group").Filter(new() { Has = page.Locator(".hpm-group-name", new() { HasTextString = group }) }); @@ -74,16 +92,128 @@ public async Task Dashboard_WhenGrouped_ShowsEveryCheckerExactlyOnce(ProviderSet await using var app = await SampleApp.StartAsync(setup, Ct); var page = await OpenDashboardAsync(app); - var flatCount = await page.Locator(".hpm-row").CountAsync(); + // The board opens grouped, so this is the state under test without touching anything. + await page.Locator(".hpm-group").First.WaitForAsync(); + var groupedCount = await page.Locator(".hpm-row").CountAsync(); + Assert.Equal(1, await RowFor(page, TargetChecker).CountAsync()); + + // The same checkers laid out flat: the button goes the other way now, which makes this the + // comparison the test always wanted rather than a count taken before grouping was applied. await page.GetByRole(AriaRole.Button, new() { Name = "GROUP", Exact = true }).ClickAsync(); - await page.Locator(".hpm-group").First.WaitForAsync(); + await Assertions.Expect(page.Locator(".hpm-group")).ToHaveCountAsync(0); - Assert.Equal(flatCount, await page.Locator(".hpm-row").CountAsync()); + Assert.Equal(groupedCount, await page.Locator(".hpm-row").CountAsync()); Assert.Equal(1, await RowFor(page, TargetChecker).CountAsync()); browser.AssertNoErrors(page); } + /// + /// The board opens sectioned by group rather than as one flat list, and the GROUP button + /// reflects that rather than inviting a click that turns it off unannounced. + /// + [Theory] + [MemberData(nameof(DashboardTests.Setups), MemberType = typeof(DashboardTests))] + public async Task Dashboard_OnOpen_IsSectionedByGroup(ProviderSetup setup) + { + await using var app = await SampleApp.StartAsync(setup, Ct); + var page = await OpenDashboardAsync(app); + + await Assertions.Expect(page.Locator(".hpm-group").First).ToBeVisibleAsync(); + await Assertions.Expect(page.GetByRole(AriaRole.Button, new() { Name = "GROUP", Exact = true })) + .ToHaveAttributeAsync("aria-pressed", "true"); + + browser.AssertNoErrors(page); + } + + /// + /// The side menu narrows the list to one group, and says which one it is narrowed to. + /// + [Theory] + [MemberData(nameof(DashboardTests.Setups), MemberType = typeof(DashboardTests))] + public async Task Dashboard_SideMenu_NarrowsTheListToOneGroup(ProviderSetup setup) + { + await using var app = await SampleApp.StartAsync(setup, Ct); + var page = await OpenDashboardAsync(app); + + var everything = await page.Locator(".hpm-row").CountAsync(); + var inTarget = await GroupFor(page, TargetGroup).Locator(".hpm-row").CountAsync(); + Assert.True(inTarget > 0 && inTarget < everything); + + var entry = page.Locator(".hpm-nav-item").Filter(new() { HasTextString = TargetGroup }); + await entry.ClickAsync(); + + await Assertions.Expect(page.Locator(".hpm-row")).ToHaveCountAsync(inTarget); + await Assertions.Expect(entry).ToHaveAttributeAsync("aria-current", "true"); + + // Picking it again is the way back, so the menu cannot strand anyone in one group. + await entry.ClickAsync(); + await Assertions.Expect(page.Locator(".hpm-row")).ToHaveCountAsync(everything); + + browser.AssertNoErrors(page); + } + + /// + /// A cron expression typed on the board reaches the scheduler, and a bad one is refused without + /// touching what is stored. + /// + /// + /// Ordered deliberately: the refusal is asserted first, so a passing "it applied" cannot be the + /// result of the field simply ignoring everything typed into it. + /// + [Theory] + [MemberData(nameof(DashboardTests.Setups), MemberType = typeof(DashboardTests))] + public async Task Dashboard_CronEditor_AppliesAValidExpressionAndRefusesABadOne(ProviderSetup setup) + { + await using var app = await SampleApp.StartAsync(setup, Ct); + var page = await OpenDashboardAsync(app); + + // Waiting on the panel's title, not on the box: the box is already visible for whichever + // checker was selected on load, so it tells you nothing about whether the click has landed. + await RowFor(page, CronChecker).ClickAsync(); + await Assertions.Expect(page.Locator(".hpm-sel-name")).ToHaveTextAsync(CronChecker); + + var box = page.Locator("#hpm-cron"); + var before = await box.InputValueAsync(); + Assert.False(string.IsNullOrWhiteSpace(before)); + + await box.FillAsync("99 99 * * *"); + await box.PressAsync("Enter"); + + await Assertions.Expect(page.Locator(".hpm-field-error")).ToBeVisibleAsync(); + await Assertions.Expect(RowFor(page, CronChecker).Locator(".hpm-rate")).ToHaveTextAsync(before); + + await box.FillAsync("0 6 * * MON-FRI"); + await box.PressAsync("Enter"); + + await Assertions.Expect(page.Locator(".hpm-field-error")).ToHaveCountAsync(0); + await Assertions.Expect(RowFor(page, CronChecker).Locator(".hpm-rate")).ToHaveTextAsync("0 6 * * MON-FRI"); + + browser.AssertNoErrors(page); + } + + /// + /// The interval picker is inert while a cron expression is in force, because the schedule + /// overrides it -- a live picker would store a value nothing reads. + /// + [Theory] + [MemberData(nameof(DashboardTests.Setups), MemberType = typeof(DashboardTests))] + public async Task Dashboard_ForACronChecker_DisablesTheIntervalPicker(ProviderSetup setup) + { + await using var app = await SampleApp.StartAsync(setup, Ct); + var page = await OpenDashboardAsync(app); + + await RowFor(page, CronChecker).ClickAsync(); + await Assertions.Expect(page.Locator(".hpm-sel-name")).ToHaveTextAsync(CronChecker); + await Assertions.Expect(page.Locator("#hpm-interval")).ToBeDisabledAsync(); + + await RowFor(page, TargetChecker).ClickAsync(); + await Assertions.Expect(page.Locator(".hpm-sel-name")).ToHaveTextAsync(TargetChecker); + await Assertions.Expect(page.Locator("#hpm-interval")).ToBeEnabledAsync(); + + browser.AssertNoErrors(page); + } + /// A group's header reports what is inside it, so its tallies must add up to that. [Theory] [MemberData(nameof(DashboardTests.Setups), MemberType = typeof(DashboardTests))] @@ -92,7 +222,6 @@ public async Task Dashboard_GroupHeader_TalliesMatchTheRowsInsideIt(ProviderSetu await using var app = await SampleApp.StartAsync(setup, Ct); var page = await OpenDashboardAsync(app); - await page.GetByRole(AriaRole.Button, new() { Name = "GROUP", Exact = true }).ClickAsync(); var group = GroupFor(page, TargetGroup); await group.WaitForAsync(); @@ -112,7 +241,6 @@ public async Task Dashboard_CollapsingAGroup_HidesItsRowsAndLeavesTheOthers(Prov await using var app = await SampleApp.StartAsync(setup, Ct); var page = await OpenDashboardAsync(app); - await page.GetByRole(AriaRole.Button, new() { Name = "GROUP", Exact = true }).ClickAsync(); var group = GroupFor(page, TargetGroup); await group.WaitForAsync(); diff --git a/tests/Healthie.Tests.E2E/DashboardTests.cs b/tests/Healthie.Tests.E2E/DashboardTests.cs index e379f0c..d9ba99a 100644 --- a/tests/Healthie.Tests.E2E/DashboardTests.cs +++ b/tests/Healthie.Tests.E2E/DashboardTests.cs @@ -1,4 +1,4 @@ -using Microsoft.Playwright; +using Microsoft.Playwright; namespace Healthie.Tests.E2E; @@ -101,7 +101,19 @@ public async Task SelectingAChecker_ShowsItsDetail(ProviderSetup setup) await page.Locator(".hpm-sel-name", new() { HasTextString = TargetChecker }) .WaitForAsync(new() { Timeout = 10_000 }); Assert.Equal(TargetChecker, (await page.Locator(".hpm-sel-name").TextContentAsync())?.Trim()); - Assert.Equal(3, await page.Locator(".hpm-stat").CountAsync()); + + // A count would be a timing assertion, not a detail one: the sample installs the uptime + // package, whose 24H cell appears once a segment has been recorded and whose WORST cell + // appears only after an outage. So: the three that are always there, and nothing unexpected. + // Scoped to the stats grid: the same label class dresses the GROUP and TAGS editors below it. + var labels = await page.Locator(".hpm-stats .hpm-stat-label").AllTextContentsAsync(); + var trimmed = labels.Select(label => label.Trim()).ToList(); + + Assert.Contains("UPTIME", trimmed); + Assert.Contains("FAILS", trimmed); + Assert.Contains("STATE", trimmed); + Assert.Empty(trimmed.Except(["UPTIME", "FAILS", "STATE", "24H", "WORST"])); + browser.AssertNoErrors(page); } diff --git a/tests/Healthie.Tests.Unit/AlertingTests.cs b/tests/Healthie.Tests.Unit/AlertingTests.cs index 6a223d9..151ad89 100644 --- a/tests/Healthie.Tests.Unit/AlertingTests.cs +++ b/tests/Healthie.Tests.Unit/AlertingTests.cs @@ -1,4 +1,5 @@ -using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Insights; using Healthie.Alerting; using Microsoft.Extensions.Hosting; @@ -98,6 +99,50 @@ public async Task AFailure_ReachesTheSink() } } + /// + /// An application can install alerting for the dashboard's panel alone and wire up delivery + /// later. The dispatcher used to skip subscribing when no sink was registered -- correct while + /// sinks were the only consumer, and the reason that panel could never fill once it was one. + /// + [Fact] + public async Task WithNoSinkRegistered_AlertsStillReachTheHistoryTheDashboardReads() + { + var checker = new FakePulseChecker("alerting-target"); + var history = new AlertHistory(capacity: 4); + var dispatcher = new AlertDispatcher( + [checker], + [], + new HealthieAlertOptions { DeduplicationWindow = TimeSpan.Zero }, + history); + + await ((IHostedService)dispatcher).StartAsync(Ct); + + try + { + Assert.Equal(1, checker.SubscriberCount); + checker.RaiseStateChanged(PulseCheckerHealth.Unhealthy); + + IReadOnlyList recent = []; + var deadline = DateTime.UtcNow.AddSeconds(5); + + while (recent.Count == 0 && DateTime.UtcNow < deadline) + { + recent = (await history.GetAlertsAsync(0, 10, Ct)).Alerts; + + if (recent.Count == 0) + { + await Task.Delay(20, Ct); + } + } + + Assert.Equal(PulseCheckerHealth.Unhealthy, Assert.Single(recent).CurrentHealth); + } + finally + { + await ((IHostedService)dispatcher).StopAsync(CancellationToken.None); + } + } + /// /// A change carrying no result says nothing about health, so it must not alert. This is the one /// branch of the health-change test that the other tests here cannot reach: every other way of diff --git a/tests/Healthie.Tests.Unit/InsightsTests.cs b/tests/Healthie.Tests.Unit/InsightsTests.cs new file mode 100644 index 0000000..b7f3676 --- /dev/null +++ b/tests/Healthie.Tests.Unit/InsightsTests.cs @@ -0,0 +1,330 @@ +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Insights; +using Healthie.Abstractions.StateProviding; +using Healthie.Alerting; +using Healthie.DependencyInjection; +using Healthie.LeaderElection; +using Healthie.Uptime; +using Microsoft.Extensions.DependencyInjection; + +namespace Healthie.Tests.Unit; + +/// +/// The read-only views the dashboard renders when a feature package is installed. +/// +/// +/// The point of the contract is that the dashboard shows a panel because a service is registered, +/// not because a flag was set -- so what is asserted here is mostly presence and absence, and that +/// nothing appears in an application that installed nothing. +/// +public class InsightsTests +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static ServiceProvider Build(Action configure) + { + var services = new ServiceCollection(); + services.AddHealthie(typeof(InsightsTests).Assembly); + configure(services); + + return services.BuildServiceProvider(); + } + + /// + /// The case that matters most: an application with none of the feature packages must resolve + /// none of the contracts, so the board renders exactly what it rendered before. + /// + [Fact] + public void WithNoFeaturePackages_NoInsightsAreRegistered() + { + using var provider = Build(_ => { }); + + Assert.Null(provider.GetService()); + Assert.Null(provider.GetService()); + Assert.Null(provider.GetService()); + Assert.Null(provider.GetService()); + } + + [Fact] + public void AddHealthieUptime_RegistersUptimeInsightsAndNothingElse() + { + using var provider = Build(services => services.AddHealthieUptime()); + + Assert.NotNull(provider.GetService()); + Assert.Null(provider.GetService()); + Assert.Null(provider.GetService()); + } + + [Fact] + public void AddHealthieAlerts_RegistersAlertInsights() + { + using var provider = Build(services => services.AddHealthieAlerts()); + + Assert.NotNull(provider.GetService()); + Assert.Null(provider.GetService()); + } + + [Fact] + public void AddHealthieLeaderElection_RegistersLeadershipInsights() + { + using var provider = Build(services => services.AddHealthieLeaderElection()); + + var leadership = provider.GetService(); + + Assert.NotNull(leadership); + Assert.False(string.IsNullOrWhiteSpace(leadership.ReplicaId)); + } + + /// + /// Nothing recorded is not the same as nothing observed, and neither is a zero. A checker that + /// has never run reports no uptime rather than a confident 0%. + /// + [Fact] + public async Task UptimeInsights_ForACheckerThatNeverRan_ReportsNothing() + { + using var provider = Build(services => services.AddHealthieUptime()); + + var uptime = provider.GetRequiredService(); + + Assert.Null(await uptime.GetUptimeAsync("never-ran", TimeSpan.FromHours(24), Ct)); + } + + [Fact] + public async Task UptimeInsights_ReportsTheShareOfTheWindowSpentHealthy() + { + using var provider = Build(services => services.AddHealthieUptime()); + + var store = provider.GetRequiredService(); + var now = DateTime.UtcNow; + + // Transitions, which is how the recorder writes them: each one closes the segment before it. + // Healthy for an hour, then unhealthy for the hour up to now, inside a four-hour window. + await store.RecordAsync("split", PulseCheckerHealth.Healthy, now.AddHours(-2), Ct); + await store.RecordAsync("split", PulseCheckerHealth.Unhealthy, now.AddHours(-1), Ct); + + var uptime = await provider.GetRequiredService() + .GetUptimeAsync("split", TimeSpan.FromHours(4), Ct); + + Assert.NotNull(uptime); + + // Half of the observed time, not of the window: the two hours nothing was recorded are time + // nobody was watching, and counting them either way would be a claim about nothing. + Assert.Equal(50, uptime.Percentage, 0); + + Assert.NotNull(uptime.LongestOutage); + Assert.Equal(1, uptime.LongestOutage.Value.TotalHours, 1); + } + + /// + /// A percentage cannot tell a hundred blips from one long outage, which is why the longest + /// stretch is reported beside it. + /// + [Fact] + public async Task UptimeInsights_ReportsTheLongestOutageNotTheTotal() + { + using var provider = Build(services => services.AddHealthieUptime()); + + var store = provider.GetRequiredService(); + var now = DateTime.UtcNow; + + await store.RecordAsync("blips", PulseCheckerHealth.Unhealthy, now.AddMinutes(-50), Ct); + await store.RecordAsync("blips", PulseCheckerHealth.Healthy, now.AddMinutes(-45), Ct); + await store.RecordAsync("blips", PulseCheckerHealth.Unhealthy, now.AddMinutes(-30), Ct); + await store.RecordAsync("blips", PulseCheckerHealth.Healthy, now.AddMinutes(-10), Ct); + + var uptime = await provider.GetRequiredService() + .GetUptimeAsync("blips", TimeSpan.FromHours(1), Ct); + + // 25 minutes unhealthy in total, but the longest single stretch is 20. + Assert.Equal(20, uptime!.LongestOutage!.Value.TotalMinutes, 1); + } + + [Fact] + public async Task AlertHistory_KeepsTheNewestAndDropsTheOldest() + { + var history = new AlertHistory(capacity: 2); + + foreach (var i in Enumerable.Range(1, 3)) + { + history.Record(Alert($"checker-{i}"), delivered: true); + } + + var recent = (await history.GetAlertsAsync(0, 10, Ct)).Alerts; + + Assert.Equal(2, recent.Count); + Assert.Equal("checker-3", recent[0].CheckerName); + Assert.Equal("checker-2", recent[1].CheckerName); + } + + /// + /// An alert that fired and reached nobody is the failure worth seeing, so it is recorded as + /// raised-but-undelivered rather than not recorded at all. + /// + [Fact] + public async Task AlertHistory_RemembersWhetherAnAlertWasDelivered() + { + var history = new AlertHistory(capacity: 4); + + history.Record(Alert("delivered"), delivered: true); + history.Record(Alert("failed"), delivered: false); + + var recent = (await history.GetAlertsAsync(0, 10, Ct)).Alerts; + + Assert.False(recent[0].Delivered); + Assert.True(recent[1].Delivered); + } + + /// + /// Trimming used to happen before the enqueue, which made a capacity of zero the empty-queue + /// case on every single call -- so the first alert raised took down the dispatcher's delivery + /// loop rather than being discarded. + /// + [Fact] + public async Task AlertHistory_WithNoRoomAtAll_DiscardsRatherThanThrows() + { + var history = new AlertHistory(capacity: 0); + + history.Record(Alert("nowhere"), delivered: true); + + Assert.Empty((await history.GetAlertsAsync(0, 10, Ct)).Alerts); + } + + /// + /// The point of writing the log through the state provider: an operator arriving after a + /// redeploy is looking for what happened before it. + /// + [Fact] + public async Task AlertHistory_OnADurableProvider_SurvivesTheProcessThatWroteIt() + { + // One provider, two histories: the second is the same application after a restart. + var store = new InMemoryStateProvider(); + + var before = new AlertHistory(capacity: 10, store); + before.Record(Alert("checker-1"), delivered: true); + before.Record(Alert("checker-2"), delivered: false); + + // The write is off the delivery path, so it is finished when the read can see it. + await WaitForAsync(async () => (await new AlertHistory(10, store).GetAlertsAsync(0, 10, Ct)).Total == 2); + + var after = await new AlertHistory(capacity: 10, store).GetAlertsAsync(0, 10, Ct); + + Assert.Equal(2, after.Total); + Assert.Equal("checker-2", after.Alerts[0].CheckerName); + Assert.False(after.Alerts[0].Delivered); + } + + /// + /// A page is a window on the whole history, and the total it reports is what a pager counts + /// against -- not the size of the page it happens to be looking at. + /// + [Fact] + public async Task AlertHistory_PagesOverEverythingItHolds() + { + var history = new AlertHistory(capacity: 10); + + foreach (var i in Enumerable.Range(1, 7)) + { + history.Record(Alert($"checker-{i}"), delivered: true); + } + + var first = await history.GetAlertsAsync(0, 3, Ct); + var second = await history.GetAlertsAsync(3, 3, Ct); + var last = await history.GetAlertsAsync(6, 3, Ct); + + Assert.Equal(7, first.Total); + Assert.Equal(7, second.Total); + + Assert.Equal(3, first.Alerts.Count); + Assert.Equal(3, second.Alerts.Count); + Assert.Single(last.Alerts); + + // Newest first, and no alert appears on two pages. + Assert.Equal("checker-7", first.Alerts[0].CheckerName); + Assert.Equal("checker-4", second.Alerts[0].CheckerName); + Assert.Equal("checker-1", last.Alerts[0].CheckerName); + } + + /// + /// "Nothing has alerted" and "nothing is configured to deliver" look identical on a board and + /// mean opposite things, so a sink is listed from startup rather than from its first delivery. + /// + [Fact] + public void AlertHistory_ListsASinkBeforeItHasDoneAnything() + { + var history = new AlertHistory(capacity: 4); + + Assert.Empty(history.Sinks); + + history.Register("SlackAlertSink"); + + var sink = Assert.Single(history.Sinks); + Assert.Equal("SlackAlertSink", sink.Name); + Assert.Equal(0, sink.Delivered); + Assert.True(sink.IsHealthy); + } + + /// + /// A sink that failed once and recovered is working, so the board must stop showing it in red. + /// + [Fact] + public void AlertHistory_ASinkThatRecovers_StopsBeingReportedAsFailing() + { + var history = new AlertHistory(capacity: 4); + + history.RecordDelivery("WebhookAlertSink", error: "500 Internal Server Error"); + + Assert.False(Assert.Single(history.Sinks).IsHealthy); + + history.RecordDelivery("WebhookAlertSink", error: null); + + var sink = Assert.Single(history.Sinks); + + Assert.True(sink.IsHealthy); + Assert.Equal(1, sink.Delivered); + Assert.Equal(1, sink.Failed); + } + + private static async Task WaitForAsync(Func> condition) + { + var deadline = DateTime.UtcNow.AddSeconds(5); + + while (DateTime.UtcNow < deadline && !await condition()) + { + await Task.Delay(20, Ct); + } + } + + /// + /// And the supported route there cannot reach zero: the option clamps, as MaxHistoryLength does, + /// so a board configured with no history shows the last alert rather than nothing at all. + /// + [Fact] + public void HistoryLength_BelowOne_IsClamped() + { + Assert.Equal(1, new HealthieAlertOptions { HistoryLength = 0 }.HistoryLength); + Assert.Equal(1, new HealthieAlertOptions { HistoryLength = -5 }.HistoryLength); + } + + [Fact] + public void AlertHistory_CountsWhatNeverReachedTheQueue() + { + var history = new AlertHistory(capacity: 4); + + Assert.Equal(0, history.DroppedCount); + + history.RecordDropped(); + history.RecordDropped(); + + Assert.Equal(2, history.DroppedCount); + } + + private static Alert Alert(string name) => new( + name, + name, + Group: null, + Tags: [], + PulseCheckerHealth.Healthy, + PulseCheckerHealth.Unhealthy, + "down", + DateTime.UtcNow); +} diff --git a/tests/Healthie.Tests.Unit/ScheduleMutationTests.cs b/tests/Healthie.Tests.Unit/ScheduleMutationTests.cs new file mode 100644 index 0000000..e0aae4a --- /dev/null +++ b/tests/Healthie.Tests.Unit/ScheduleMutationTests.cs @@ -0,0 +1,187 @@ +using Healthie.Abstractions; +using Healthie.Abstractions.Enums; +using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; +using Healthie.Abstractions.StateProviding; +using Healthie.DependencyInjection; + +namespace Healthie.Tests.Unit; + +/// +/// Setting a schedule from outside the code that declared it. +/// +/// +/// The trap these cover is that is ignored once +/// is set. Anything that writes one without considering the +/// other stores a value nothing reads, and the checker carries on at its old cadence while the caller +/// believes otherwise. +/// +public class ScheduleMutationTests +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + /// + /// A real on a cron schedule, because the behaviour under test lives + /// in the base class rather than in the interface. + /// + /// + /// Takes only a state provider, like the other checkers in this assembly: assembly scanning + /// registers every concrete here, so one whose constructor the + /// container cannot satisfy fails every registration test in the suite rather than only its own. + /// + internal sealed class CronScheduledChecker(IStateProvider stateProvider) + : PulseChecker(stateProvider, PulseSchedule.Cron(InitialCron)) + { + public const string InitialCron = "0 3 * * *"; + + public override string Name => "schedule-target"; + + public override Task CheckAsync(CancellationToken cancellationToken = default) => + Task.FromResult(new PulseCheckerResult(PulseCheckerHealth.Healthy, "ok")); + } + + private static async Task<(CronScheduledChecker Checker, IStateProvider Provider)> DailyCheckerAsync() + { + var provider = new InMemoryStateProvider(); + var checker = new CronScheduledChecker(provider); + + // Seeds the initial state, which is what makes the cron schedule the stored one. + await checker.TriggerAsync(Ct); + + return (checker, provider); + } + + private static async Task StateOf(IStateProvider provider) => + (await provider.GetStateAsync("schedule-target", Ct))!; + + /// + /// The headline bug: choosing an interval on a cron checker used to write a field nothing reads. + /// + [Fact] + public async Task SettingAnInterval_ClearsACronSchedule_SoTheChoiceTakesEffect() + { + var (checker, provider) = await DailyCheckerAsync(); + using var _ = checker; + + Assert.True((await StateOf(provider)).EffectiveSchedule.IsCron); + + await checker.SetIntervalAsync(PulseInterval.Every30Seconds, Ct); + + var state = await StateOf(provider); + + Assert.Null(state.Schedule); + Assert.Equal(PulseInterval.Every30Seconds, state.Interval); + Assert.Equal(TimeSpan.FromSeconds(30), state.EffectiveSchedule.Period); + } + + [Fact] + public async Task SettingACronSchedule_StoresIt() + { + var (checker, provider) = await DailyCheckerAsync(); + using var _ = checker; + + await checker.SetScheduleAsync(PulseSchedule.Cron("*/5 * * * *"), Ct); + + Assert.Equal("*/5 * * * *", (await StateOf(provider)).Schedule?.CronExpression); + } + + /// + /// A schedule the enum can name exactly is stored as that interval, so the common case keeps the + /// shape every stored state and every older reader already understands. + /// + [Fact] + public async Task SettingAScheduleAnIntervalCanExpress_StoresTheInterval_NotTheSchedule() + { + var (checker, provider) = await DailyCheckerAsync(); + using var _ = checker; + + await checker.SetScheduleAsync(PulseSchedule.Every(TimeSpan.FromMinutes(5)), Ct); + + var state = await StateOf(provider); + + Assert.Null(state.Schedule); + Assert.Equal(PulseInterval.Every5Minutes, state.Interval); + } + + [Fact] + public async Task ClearingTheSchedule_GoesBackToTheStoredInterval() + { + var (checker, provider) = await DailyCheckerAsync(); + using var _ = checker; + + await checker.SetScheduleAsync(null, Ct); + + var state = await StateOf(provider); + + Assert.Null(state.Schedule); + Assert.False(state.EffectiveSchedule.IsCron); + } + + /// + /// A period no names has to survive as a schedule rather than being + /// rounded to the nearest one the enum happens to have. + /// + [Fact] + public async Task SettingAnAwkwardPeriod_KeepsItExactly() + { + var (checker, provider) = await DailyCheckerAsync(); + using var _ = checker; + + await checker.SetScheduleAsync(PulseSchedule.Every(TimeSpan.FromSeconds(90)), Ct); + + Assert.Equal(TimeSpan.FromSeconds(90), (await StateOf(provider)).Schedule?.Period); + } + + [Fact] + public void TimerScheduler_AcceptsAValidCronExpression() + { + using var scheduler = new TimerPulseScheduler(); + + Assert.True(scheduler.TryValidateSchedule(PulseSchedule.Cron("0 6 * * MON-FRI"), out var error)); + Assert.Null(error); + } + + /// + /// Refused with a reason, not just refused: the schedulers disagree about cron dialects, so which + /// rule was broken is the part worth showing whoever typed it. + /// + [Theory] + [InlineData("99 99 * * *")] + [InlineData("not a cron expression")] + [InlineData("* * *")] + public void TimerScheduler_RefusesABadCronExpression_WithAReason(string expression) + { + using var scheduler = new TimerPulseScheduler(); + + Assert.False(scheduler.TryValidateSchedule(PulseSchedule.Cron(expression), out var error)); + Assert.False(string.IsNullOrWhiteSpace(error)); + } + + [Fact] + public void TimerScheduler_AcceptsAPeriodWithoutOpinion() + { + using var scheduler = new TimerPulseScheduler(); + + Assert.True(scheduler.TryValidateSchedule(PulseSchedule.Every(TimeSpan.FromSeconds(90)), out _)); + } + + /// + /// Refused before it is stored. Persisting first and failing on the reschedule leaves a checker + /// that no longer runs and a store that says it should. + /// + [Fact] + public async Task PulsesScheduler_RefusingASchedule_LeavesTheStoredOneAlone() + { + var provider = new InMemoryStateProvider(); + using var checker = new CronScheduledChecker(provider); + await checker.TriggerAsync(Ct); + + using var scheduler = new TimerPulseScheduler(); + var pulses = new PulsesScheduler([checker], scheduler, new HealthieOptions()); + + await Assert.ThrowsAsync(() => + pulses.SetScheduleAsync("schedule-target", PulseSchedule.Cron("99 99 * * *"), Ct)); + + Assert.Equal(CronScheduledChecker.InitialCron, (await StateOf(provider)).Schedule?.CronExpression); + } +} diff --git a/tests/Healthie.Tests.Unit/StubDashboardService.cs b/tests/Healthie.Tests.Unit/StubDashboardService.cs index cce2521..9aac4ef 100644 --- a/tests/Healthie.Tests.Unit/StubDashboardService.cs +++ b/tests/Healthie.Tests.Unit/StubDashboardService.cs @@ -1,5 +1,6 @@ using Healthie.Abstractions.Enums; using Healthie.Abstractions.Models; +using Healthie.Abstractions.Scheduling; using Healthie.Dashboard.Services; namespace Healthie.Tests.Unit; @@ -31,6 +32,8 @@ private static NotSupportedException NotStubbed(string member) => public virtual Task SetIntervalAsync(string name, PulseInterval interval, CancellationToken cancellationToken = default) => throw NotStubbed(nameof(SetIntervalAsync)); + public virtual Task SetScheduleAsync(string name, PulseSchedule? schedule, CancellationToken cancellationToken = default) => throw NotStubbed(nameof(SetScheduleAsync)); + public virtual Task SetThresholdAsync(string name, uint threshold, CancellationToken cancellationToken = default) => throw NotStubbed(nameof(SetThresholdAsync)); public virtual Task SetTagsAsync(string name, IReadOnlyList tags, CancellationToken cancellationToken = default) => throw NotStubbed(nameof(SetTagsAsync));