Semantic activity logs

[scotwells]

High-Level Summary

Users today have no intuitive way to see what happened inside their projects — only low-level audit log data. This enhancement introduces Activity Logs, a curated and semantic layer that translates raw platform events into clear, human-readable summaries of meaningful actions.

At the core of this system are Activity Descriptors — small, declarative definitions that describe how a given type of system event should be presented to a human user.

Each descriptor specifies:

• Matching rules for which events it applies to (e.g., a Deployment update, a DNS record creation).
• A template for how the event should be phrased (e.g., {{ actor }} created a DNS zone for {{ params.domainName }}).
• Optional conditions and labels that provide context such as product area, category, or audience.

Descriptors act as a translation layer between structured machine data and human understanding. By evaluating them dynamically at read time, the platform can:

• Render human-friendly summaries for any product area (DNS, Proxy, Domains, Access, etc.).
• Evolve or reword messages without re-ingesting historical data.
• Tailor phrasing and visibility to different audiences (Customer vs. Staff).
• Support localization, A/B testing, or thematic variations in the future.

The result is a flexible, extensible framework that enables consistent, semantic Activity Logs across the entire platform — giving users immediate insight into what changed, who did it, and where it happened.

Motivation

Raw audit logs record the technical details of every API call — who made it, when, and what resource was touched. While invaluable for debugging or compliance, this information is too technical and low-level for most users trying to understand how their project has changed over time.

Activity Logs make that same information human-readable and contextual. They don’t analyze runtime behavior or service health — instead, they rephrase platform actions into clear, semantic descriptions of what configuration change occurred and who performed it.

This helps:

• Customers understand how their project state is evolving without parsing audit payloads.
• Support staff quickly verify user or system actions when troubleshooting.
• Operators maintain consistent visibility into user-driven changes across product areas (DNS, Proxy, Domains, Access, etc.).

Goals

• Provide a clear, semantic, and readable activity timeline for each project.
• Define and manage reusable ActivityDescriptors for translating raw audit events into curated summaries.
• Enable consistent phrasing and structure across all product areas.
• Allow easy iteration on copy, labels, and presentation without data migrations.
• Maintain high performance and low latency in the portal and API.

Non-Goals

• Exporting activity logs to external sources
• Capturing runtime or health events
• Alerting / notifications

Notes

Refer to the product alignment document for more information on the high-level goals of this enhancement.

[scotwells]

Brainstorming a little on what this could look like based on the goals outlined above. The key piece here being all activity log descriptions are driven by API resources so they can be dynamically changed and retroactively apply to pre-existing audit log data.

Note

Below is an illustration of an audit log for the creation of a DNS zone being used to generate an activity log with a human-friendly description of:

alice@datum created DNS Zone for example.com

The audit log generated when a DNS Zone is created:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "8a2f4cbe-8d8f-4c1e-bb68-2d3beef20110",
  "requestURI": "/apis/dns.networking.datumapis.com/v1alpha1/namespaces/default/dnszones",
  "verb": "create",
  "user": { 
    "username": "alice@datum",
    "groups": ["devs"] 
  },
  "userAgent": "kubectl/v1.30 (linux/amd64)",
  "objectRef": {
    "resource": "dnszones",
    "apiGroup": "dns.datum.io",
    "apiVersion": "v1alpha1",
    "namespace": "proj-123",
    "name": "example-com"
  },
  "requestObject": {
    "apiVersion": "dns.networking.datumapis.com/v1alpha1",
    "kind": "DNSZone",
    "metadata": { "name": "example-com", "namespace": "proj-123" },
    "spec": {
      "domain": "example.com"
    }
  },
  "responseObject": {
    "apiVersion": "dns.networking.datumapis.com/v1alpha1",
    "kind": "DNSZone",
    "metadata": { "name": "example-com", "namespace": "proj-123", "uid": "a1b2c3" },
    "spec": { "domain": "example.com" },
    "status": { "state": "Ready", "nameServers": ["ns1.example.net.", "ns2.example.net."] }
  },
  "responseStatus": { "code": 201 },
  "requestReceivedTimestamp": "2025-11-10T15:42:10.123Z"
}

Example activity log descriptor that could be used to match the audit log entry and produce a human readable summary:

apiVersion: activity.miloapis.com/v1alpha1
kind: ActivityDescriptor
metadata:
  name: dns-zone-create
spec:
  match:
    apiGroup: "dns.datum.io"
    resource: "dnszones"
    verb: "create"
  when: "true"
  summary: "{{ actor }} created DNS Zone for ({{ domain }})"
  params:
    actor: "audit.user.username"
    zone:  "audit.objectRef.name"
    domain: "coalesce(req.spec.domain, resp.spec.domain)"
  labels:
    product: "dns"
    category: "zone"
    action: "create"

The resulting Activity Log for the above Activity Descriptor and Audit Log:

{
  "ts": "2025-11-10T15:42:10.123Z",
  "type": "com.datum.activity.dns.zone.create",
  "descriptorID": "dns-zone-create.v1",
  "actor": "alice@datum",
  "subject": { "kind": "DNSZone", "name": "example-com", "namespace": "proj-123" },
  "action": "create",
  "outcome": "success",
  "http": 201,
  "requestID": "8a2f4cbe-8d8f-4c1e-bb68-2d3beef20110",
  "params": { "zone": "example-com", "domain": "example.com" },
  "summary": "alice@datum created DNS Zone for `example.com`",
  "links": {
    "resource": "/projects/proj-123/dns/zones/example-com"
  },
  "labels": { "product": "dns", "category": "zone", "env": "prod", "project": "web" }
}

[scotwells]

One limitation that came up in discussion in conversation with @brian-toresdahl-datum around Kubernetes audit logs was the lack of the state of the existing resource in the audit log payload. The audit log only contains the request and response bodies.

I did some research around how we could augment the existing apiserver to capture the existing object in the audit log entry so we can use it to render a rich-diff to users in the UI or use it in the summary that's generated.

We can expose the full pre-update object in audit logs without changing the audit schema by emitting it as an audit annotation on UPDATE requests. Audit annotations are key–value pairs attached to the audit event during request handling.

This would require modifying the storage layer in our apiserver to register a BeforeUpdateFunc to hook into the storage layer before an object is persisted and add the annotation to the audit log at the well-known key activity.miloapis.com/previous-object.

Optionally, we can also include a compact, machine-readable diff (e.g., RFC 6902 JSON Patch) under the annotation delta.miloapis.com/rfc6902. This approach uses existing hooks and APIs: add the annotation with audit.AddAuditAnnotation(...), and if desired, compute the diff alongside.

Here's an example of how an audit log would look for an update to a DNS Record Set:

{
  "apiVersion": "audit.k8s.io/v1",
  "kind": "Event",
  "level": "Metadata",
  "stage": "ResponseComplete",
  "verb": "update",
  "requestURI": "/apis/dns.networking.miloapis.com/v1alpha1/namespaces/edge/dnsrecordsets/www",
  "user": { "username": "alice@example.com" },
  "objectRef": {
    "resource": "dnsrecordsets",
    "namespace": "edge",
    "name": "www",
    "apiGroup": "dns.networking.miloapis.com",
    "apiVersion": "v1alpha1"
  },
  "responseStatus": { "code": 200 },
  "annotations": {
    "activity.miloapis.com/previous-object": "{\"apiVersion\":\"dns.networking.miloapis.com/v1alpha1\",\"kind\":\"DNSRecordSet\",\"metadata\":{\"name\":\"www\",\"namespace\":\"edge\"},\"spec\":{\"zoneRef\":{\"name\":\"example-com\"},\"type\":\"A\",\"ttl\":60,\"records\":[\"203.0.113.10\"]}}",
    "delta.miloapis.com/rfc6902": "[{\"op\":\"replace\",\"path\":\"/spec/ttl\",\"value\":120},{\"op\":\"add\",\"path\":\"/spec/records/-\",\"value\":\"203.0.113.11\"}]"
  }
}

[scotwells]

Adding some thoughts around how this can be implemented:

• Keep the API definition outside of Milo (e.g. activity.miloapis.com) in a new repo (e.g. datum-cloud/activity)
• Repo would contain the following components:
  • Aggregated apiserver implementation for querying audit log and activity log data from a backend storage system (e.g. Clickhouse)
  • Instrumentation library that wraps the storage layer to add the previous object to the audit log annotations
  • Ingestion pipeline configuration for receiving audit logs from an APIserver through the standard Kubernetes Audit Log format
• Activity Descriptors would be treated as CRDs, ideally registered automatically by the aggregated apiserver. Audit logs should be stored in Clickhouse.
• End-to-end tests should be setup with chainsaw to confirm we can deploy an instance of Milo configured to send audit log data to the activity system's log ingestion pipeline and the aggregated apiserver configured to handle the activity.miloapis.com api group.

[drewr]

Assuming https://github.com/datum-cloud/activity/ closes this one out. What's the status?

[scotwells]

@drewr development on this hasn't started yet. The activity repo is only scoped to #535 as of now. This task is the follow up improvements we'd tackle to make audit logs more human friendly.