Add integrated logger-status system with .status() API, health checks and hierarchical view

[kriszyp]

Introduces logger.status({ problem: 'key' }).error('msg') API that both logs and updates component status, enabling real-time health monitoring without relying on external log shipping to DataDog.

New modules:

• logBridge.ts: bridges logger.status() calls to ComponentStatusRegistry
• hierarchy.ts: builds tree-structured status view from flat component map
• healthChecks.ts: periodic disk/memory/CPU monitoring via status system

Key changes:

• HarperLogger gains .status() method returning a logger-like wrapper
• ComponentStatus extended with source, expiresAt, occurrenceCount fields
• get_status operation now returns hierarchical status tree
• Status API adopted across replication, threads, server, and database code

The longer design document that Claude is working from:

Integrated Status System: Explicit Logger-Status API + System Health + Hierarchical View

Context

Harper relies on shipping log files to DataDog for problem detection -- expensive and high overhead. Harper already has a component status system (core/components/status/) used only by the component loader. The goal: let developers explicitly connect log calls to the status system via a .status() API on the logger, add system health monitoring, and expose everything through get_status as a hierarchical status tree a front-end can display.

Part 1: Logger .status() API

Design

Add a .status() method to the logger that returns a logger-like object. When you log through it, the message is both logged normally AND updates component status:

// Report a problem -- sets status to error/warning for the given key
logger.status({ problem: 'replication-connection' }).error('Connection failed to node-1');
logger.status({ problem: 'replication-connection', expires: 300 }).warn('Connection unstable');

// Resolve a problem -- clears the status for that key
logger.status({ resolves: 'replication-connection' }).info('Connected to node-1');

• problem: string -- A status key identifying this problem. Sets/updates a status entry.
• resolves: string -- A status key to clear. Marks the problem as resolved (sets healthy).
• expires: number -- Optional seconds until the status auto-clears. Only valid with problem.
• The log level determines the status severity: fatal/error -> error status, warn -> warning status.
• info/debug/trace with problem would set warning (since they're lower severity). With resolves, any level works.

Implementation

File: core/utility/logging/harper_logger.js

Add a status() method to HarperLogger:

status(options) {
    return statusLogger(options, this);
}

Add a statusLogger function (similar pattern to loggerWithTag):

let statusHandler; // injected from TS side to avoid circular deps

function statusLogger(options, logger) {
    // Returns an object with all log methods that both log AND update status
    const wrapper = {};
    for (const level of ['notify', 'fatal', 'error', 'warn', 'info', 'debug', 'trace']) {
        wrapper[level] = function (...args) {
            // First, log normally
            logger[level](...args);
            // Then, update status if handler is set
            if (statusHandler) {
                statusHandler(options, level, logger.tag || currentTag, args);
            }
        };
    }
    return wrapper;
}

function setStatusHandler(handler) {
    statusHandler = handler;
}

Export setStatusHandler from the module. The statusHandler is a function injected from the TS status bridge module at startup, avoiding circular dependency (JS logger cannot import TS status modules directly).

Also add .status() to:

• The loggerWithTag return object
• The externalLogger facade (so user code can use it)
• The module-level exports (module.exports.status = ...)

Status Handler Module

New file: core/components/status/logBridge.ts

import { statusForComponent } from './api.ts';
import { componentStatusRegistry } from './registry.ts';

const LOG_TO_STATUS_LEVEL = {
    fatal: 'error',
    error: 'error',
    warn: 'warning',
    notify: 'warning',
    info: 'warning',   // unusual to use info with problem, but treat as warning
    debug: 'warning',
    trace: 'warning',
};

// Track expiry timers
const expiryTimers = new Map<string, ReturnType<typeof setTimeout>>();

export function handleStatusLog(
    options: { problem?: string; resolves?: string; expires?: number },
    level: string,
    componentTag: string | undefined,
    args: any[]
) {
    const message = args.map(a => typeof a === 'string' ? a : String(a)).join(' ');

    if (options.resolves) {
        // Clear the status
        const key = options.resolves;
        clearExpiry(key);
        statusForComponent(key).healthy('Resolved: ' + message.substring(0, 200));
        return;
    }

    if (options.problem) {
        const key = options.problem;
        const statusLevel = LOG_TO_STATUS_LEVEL[level] || 'warning';

        if (statusLevel === 'error') {
            const errorArg = args.find(a => a instanceof Error);
            statusForComponent(key).error(message.substring(0, 500), errorArg);
        } else {
            statusForComponent(key).warning(message.substring(0, 500));
        }

        // Handle expiry
        if (options.expires) {
            scheduleExpiry(key, options.expires * 1000);
        }
    }
}

function scheduleExpiry(key: string, ms: number) {
    clearExpiry(key);
    const timer = setTimeout(() => {
        statusForComponent(key).healthy('Auto-expired');
        expiryTimers.delete(key);
    }, ms);
    timer.unref();
    expiryTimers.set(key, timer);
}

function clearExpiry(key: string) {
    const existing = expiryTimers.get(key);
    if (existing) {
        clearTimeout(existing);
        expiryTimers.delete(key);
    }
}

Initialization

In the component startup path (likely componentLoader.ts or a dedicated status start hook), after the status system is ready:

import { setStatusHandler } from '../../utility/logging/harper_logger.js';
import { handleStatusLog } from './logBridge.ts';

setStatusHandler(handleStatusLog);

External/User Code

The external logger facade already delegates to the internal logger. Add .status() to the facade:

// In module.exports.externalLogger
module.exports.externalLogger.status = function(options) {
    return externalLogger.status(options);
};

This is also exposed via _assignPackageExport('logger', ...), so user application code gets it automatically:

// User code in a Harper application
const { logger } = require('harperdb');
logger.status({ problem: 'my-api.upstream' }).error('Upstream API returned 503');
logger.status({ resolves: 'my-api.upstream' }).info('Upstream API recovered');

Part 2: Status Lifecycle Extensions

Extend ComponentStatus (core/components/status/ComponentStatus.ts)

Add optional fields for richer status tracking:

• source?: 'log' | 'explicit' | 'health-check' -- how the status was set
• expiresAt?: number -- epoch ms for auto-clear
• occurrenceCount?: number -- incremented on repeated reports of same problem

Extend ComponentStatusRegistry (core/components/status/ComponentStatusRegistry.ts)

Add to setStatus:

• Accept optional source and expiresAt parameters
• If setting status for a key that already exists at the same level, increment occurrenceCount

No acknowledge/dismiss/adjustSeverity for now (deferred to later phase).

Extend ComponentStatusBuilder (core/components/status/api.ts)

Add source tracking -- when called from the log bridge, pass source: 'log'; when called from health checks, pass source: 'health-check'; direct calls default to source: 'explicit'.

Part 3: System Health Monitoring

New Module: core/components/status/healthChecks.ts

Periodic checks using existing systemInformation.js functions:

import { statusForComponent } from './api.ts';
// Import existing system info functions
import { getDiskInfo, getMemoryInfo, getCPUInfo } from '../../utility/environment/systemInformation.js';

Checks:

• Disk: filesystem usage percent -> statusForComponent('system.disk')
• Memory: used/total percent -> statusForComponent('system.memory')
• CPU: load average vs core count -> statusForComponent('system.cpu')

Thresholds (configurable via harperdb-config.yaml):

• Warning: 80% utilization
• Error: 95% utilization
• Check interval: 60 seconds

Health checks self-heal: when the metric drops below the threshold, status reverts to healthy. No expiry needed.

Runs on main thread only. Uses .unref() on interval timer.

export function startHealthChecks(config?) {
    const thresholds = { ...DEFAULT_THRESHOLDS, ...config?.thresholds };
    const interval = (config?.intervalSeconds || 60) * 1000;

    const timer = setInterval(async () => {
        await checkDisk(thresholds.disk);
        await checkMemory(thresholds.memory);
        await checkCPU(thresholds.cpu);
    }, interval);
    timer.unref();

    // Run immediately
    checkDisk(thresholds.disk);
    checkMemory(thresholds.memory);
    checkCPU(thresholds.cpu);
}

Part 4: Hierarchical Status View

New Module: core/components/status/hierarchy.ts

Builds a tree from the flat component status map by splitting names on .:

interface StatusNode {
    status: ComponentStatusLevel;
    message?: string;
    source?: string;
    lastChecked?: { main?: number; workers: Record<number, number> };
    children?: Record<string, StatusNode>;
}

export function buildHierarchy(
    statuses: Map<string, AggregatedComponentStatus>
): Record<string, StatusNode> {
    const root: Record<string, StatusNode> = {};

    for (const [name, status] of statuses) {
        const parts = name.split('.');
        let current = root;

        for (let i = 0; i < parts.length; i++) {
            const part = parts[i];
            if (!current[part]) {
                current[part] = {
                    status: 'healthy',
                    children: {},
                };
            }
            if (i === parts.length - 1) {
                // Leaf node -- set actual status
                current[part].status = status.status;
                current[part].message = status.latestMessage;
                current[part].lastChecked = status.lastChecked;
            }
            current = current[part].children!;
        }
    }

    // Roll up: each parent's status = worst of its children
    rollUpStatus(root);
    return root;
}

Parent status = worst child status (error > warning > loading > unknown > healthy).

Extend GET_STATUS Response

File: core/server/status/index.ts

Modify getAllStatus() to include the hierarchy:

import { buildHierarchy } from '../../components/status/hierarchy.ts';

async function getAllStatus(): Promise<AllStatusSummary> {
    const statusRecords = getStatusTable().search([]);
    const aggregatedStatuses = await statusInternal.query.allThreads();

    const componentStatusArray = Array.from(aggregatedStatuses.entries()).map(
        ([name, status]) => ({ name, ...status })
    );

    return {
        systemStatus: statusRecords,
        componentStatus: componentStatusArray,
        hierarchy: buildHierarchy(aggregatedStatuses),  // NEW
        restartRequired: restartNeeded(),
    };
}

Part 5: Configuration

Add to harperdb-config.yaml schema:

status:
  healthChecks:
    enabled: true
    intervalSeconds: 60
    thresholds:
      disk: { warning: 80, error: 95 }
      memory: { warning: 80, error: 95 }
      cpu: { warning: 85, error: 95 }

The log bridge itself needs no configuration -- it's explicitly opted into per log call via .status().

Implementation Phases

Phase 1: Logger .status() API + Bridge

1. Add status() method to HarperLogger class
2. Add statusLogger() function (returns logger-like wrapper)
3. Add setStatusHandler export and module-level variable
4. Add .status() to loggerWithTag return objects
5. Add .status() to externalLogger facade
6. Create core/components/status/logBridge.ts with handleStatusLog
7. Wire up: call setStatusHandler(handleStatusLog) during component startup

Phase 2: Status Lifecycle Extensions

1. Add source, expiresAt, occurrenceCount to ComponentStatus
2. Update ComponentStatusRegistry.setStatus to handle these fields
3. Implement expiry timer management in the registry

Phase 3: Hierarchical Status View

1. Create core/components/status/hierarchy.ts
2. Extend getAllStatus() in core/server/status/index.ts to include hierarchy
3. Update AllStatusSummary type

Phase 4: System Health Checks

1. Create core/components/status/healthChecks.ts
2. Add configuration support (config schema)
3. Start health checks during server startup (main thread only)

Phase 5: Testing

1. Unit tests for .status() logger wrapper
2. Unit tests for handleStatusLog (problem, resolves, expires)
3. Unit tests for buildHierarchy (tree construction, status rollup)
4. Unit tests for health check threshold evaluation

Files to Modify

File	Changes
core/utility/logging/harper_logger.js	Add status() to HarperLogger, statusLogger(), setStatusHandler, update exports and facades
core/components/status/ComponentStatus.ts	Add source, expiresAt, occurrenceCount fields
core/components/status/ComponentStatusRegistry.ts	Handle new fields in setStatus, expiry timer mgmt
core/components/status/api.ts	Pass source through builder methods
core/server/status/index.ts	Add hierarchy to getAllStatus() response

New Files

File	Purpose
core/components/status/logBridge.ts	handleStatusLog -- bridges .status() calls to component status
core/components/status/hierarchy.ts	buildHierarchy -- builds tree from flat status map
core/components/status/healthChecks.ts	Periodic system health monitoring (disk, memory, CPU)

Verification

1. Use .status({ problem: 'test' }).error('test error') -> verify it appears in get_status response under componentStatus and hierarchy
2. Use .status({ resolves: 'test' }).info('resolved') -> verify status clears to healthy
3. Use .status({ problem: 'temp', expires: 5 }).warn('temporary') -> verify it auto-clears after 5 seconds
4. Verify hierarchy in get_status response groups system.disk, system.memory, system.cpu under a system parent
5. Verify parent nodes in hierarchy roll up worst-case status from children
6. Verify external logger (user code) can use .status() API

[kriszyp]

This PR's design is now the foundation of the holistic monitoring overhaul: design doc (Holistic Fleet Monitoring & Status Overhaul, incl. a "Design notes: PR #372 re-land" section — keep/extend/reconsider + phasing), tracking issue HarperFast/harper-pro#532, and re-land issue #1639. Plan is to re-land in phases (extended with audience and remediation fields) rather than rebase this branch, so this PR will likely be superseded by the Phase 1 PR.

— Claude (KrAIs), on behalf of Kris

[claude]

Both previously flagged blockers (double-dispatch, missing test coverage) are fixed in commit 13c539a with regression tests. Found 1 new blocker: startHealthChecks runs unconditionally at import time with no config wiring. See inline comment.

[claude]

Health checks start unconditionally at import time, with no config wiring

File: components/status/internal.ts:35-41 (see also components/status/healthChecks.ts:116-132)

What: initLogBridge() and startHealthChecks() run as side effects of importing internal.ts — which happens transitively whenever anything imports components/status/index.ts (it re-exports * as internal). startHealthChecks() is called with no arguments, so it always uses the hardcoded defaults (60s interval, 80%/95% thresholds) and unconditionally schedules a real setInterval polling disk/memory/CPU via systeminformation on the main thread. There is no wiring to the status.healthChecks.enabled/intervalSeconds/thresholds config this PR's own design describes (Part 5) — grepping the repo for healthChecks/intervalSeconds in .yaml/config schema turns up nothing outside this PR's TS files.

Why it matters: This is a monitoring feature that, per its own design, is supposed to be configurable (including disabled) via harperdb-config.yaml, but as shipped it can never be turned off and always uses the hardcoded thresholds — a real operational regression for any deployment that wants to opt out. It's also reachable from unitTests/server/status/status.test.js (require('#src/components/status/index') at module scope, line 7), which means every run of that suite kicks off real, unmocked systeminformation disk/mem/cpu polling against the shared componentStatusRegistry singleton — writing system.disk/system.memory/system.cpu entries that persist unless a test explicitly resets the registry (only status.test.js itself does, via beforeEach/after).

Suggested fix: Have startHealthChecks() read enabled/intervalSeconds/thresholds from the server config (or accept an explicit config object from the startup path) instead of running unconditionally, and move the module-level initLogBridge()/startHealthChecks() calls out of a plain import side effect and into an explicit startup hook so importing the status module for its types/registry doesn't also start background OS polling.