Fix server unit tests

[heskew]

Fixes #84

Moved some tests to proper integration tests were real server or threads were being tested against.

Files Deleted (4 unit test files)

Deleted Unit Test	Reason
unitTests/server/fastifyRoutes/customFunctionsServer.test.js	Started real Fastify server - integration test behavior
unitTests/server/harperdb/hdbServer.test.js	Started real Fastify server - integration test behavior
unitTests/server/threads/manageThreads.test.js	Required real worker thread infrastructure
unitTests/server/threads/socketRouter.test.js	Required real worker thread infrastructure

Files Added (3 integration tests)

New Integration Test	Coverage
integrationTests/server/operations-server.test.ts	Content negotiation (JSON, MessagePack, CBOR, CSV), error handling, auth
integrationTests/server/storage-reclamation.test.ts	Caching table expiration, record reclamation, audit logging
integrationTests/server/thread-management.test.ts	Concurrent request handling, server recovery from errors

Files Modified (9)

File	Changes
unitTests/server/storageReclamation.test.js	Rewrote with 23 comprehensive unit tests using rewire/sinon mocks
unitTests/server/fastifyRoutes/operations.test.js	Removed enterprise-only tests (SSH keys) and tests requiring real servers
unitTests/server/itc/serverHandlers.test.js	Removed tests requiring ITC/schema infrastructure
unitTests/server/ipc/utility/ipcUtils.test.js	Removed tests requiring real IPC infrastructure
unitTests/server/serverHelpers/serverUtilities.test.js	Removed enterprise-only tests (CLUSTER_STATUS)
unitTests/server/jobs/jobs.test.js	Removed skipped test
unitTests/server/threads/thread-for-tests.js	Supporting changes
unitTests/test_utils.js	Supporting changes
utility/environment/environmentManager.js	Supporting changes

Stats

• Net change: -1,176 lines (2,649 removed, 1,473 added)
• Unit tests: 257 passing (0 skipped)
• Integration tests: 28 passing (0 skipped)

Mapping: Deleted Unit Tests → New Integration Tests

The integration test names don't match 1:1 because functionality was consolidated:

Deleted Unit Test	Covered By
customFunctionsServer.test.js	operations-server.test.ts (server behavior, content negotiation)
hdbServer.test.js	operations-server.test.ts (server behavior, content negotiation)
manageThreads.test.js	thread-management.test.ts (concurrent requests, thread handling)
socketRouter.test.js	thread-management.test.ts (concurrent requests, thread handling)
(storageReclamation integration coverage)	storage-reclamation.test.ts (end-to-end reclamation)

Unit Tests

% npm run test:unit:server

> harper@5.0.0-unreleased test:unit:server
> mocha 'unitTests/server/**/*.js' --config unitTests/.mocharc.json --enable-source-maps

Current directory /Users/heskew/src/harper/harper/bin


  Test custom functions operations
    ✔ Test initial cf status values
    Test custom function project operations
      ✔ Test addComponent creates the project folder with the correct name
      ✔ Test getCustomFunctions returns object with proper length and content
      ✔ Test packageCustomFunctionProject properly tars up a project directory
      ✔ Test setCustomFunction creates a function file as expected
      ✔ Test setCustomFunction updates a function file as expected
      ✔ Test dropCustomFunctionProject drops project as expected
    Test component operations
      ✔ Test getComponents happy path (5003ms)
      ✔ Test getComponents includes status information when component status exists
      ✔ Test getComponents shows unknown status when component not in status map (5001ms)
      ✔ Test getComponents handles getAggregatedFromAllThreads error gracefully
      ✔ Test getComponents passes consolidatedStatuses to getAggregatedStatusFor
      ✔ Test getComponents handles getAggregatedStatusFor errors gracefully
      ✔ Test getComponents shows different statuses for different components
      ✔ Test getComponentFile happy path
      ✔ Test setComponentFile happy path
    Test deployComponent force flag
      ✔ Test deployComponent allows overwriting existing user component without force flag (283ms)
      ✔ Test deployComponent allows deploying new component without force flag
      ✔ Test deployComponent prevents overwriting core component without force flag
      ✔ Test deployComponent allows overwriting core component with force flag
      ✔ Test deployComponent prevents overwriting multiple core component names

  Test operationsValidation module
    Test checkProjectExists function
      ✔ Test message returned if project does not exist
      ✔ Test project is returned if project exists
      ✔ Test message is returned if fs exists throws error
    Test checkFileExists function
      ✔ Test message is returned if file does not exist
      ✔ Test file is returned if it does exist
      ✔ Test message is returned if fs exists throws error
    Test getDropCustomFunctionValidator function
      ✔ Test validation messages are returned
      ✔ Test alphanumeric validation messages are returned
    Test setCustomFunctionValidator function
      ✔ Test validation messages are returned
      ✔ Test alphanumeric validation messages are returned
    Test addCustomFunctionProjectValidator function
      ✔ Test validation messages are returned
      ✔ Test alphanumeric validation messages are returned
    Test dropCustomFunctionProjectValidator function
      ✔ Test validation messages are returned
      ✔ Test alphanumeric validation messages are returned

  Test ipcUtils module
    Test validateEvent function
      ✔ Test non object error returned
      ✔ Test missing type error returned
      ✔ Test missing message error returned
      ✔ Test invalid event type error returned
      ✔ Test missing originator error returned
    Test constructor functions
      ✔ Test SchemaEventMsg
      ✔ Test UserEventMsg

  Test hdbChildIpcHandler module
    Test user event handler function
      ✔ Test User Handler log error upon setUsersWithRolesCache failure
      ✔ Test User Handler calls setUsersWithRolesCache on valid event
      ✔ Test User Handler logs error on invalid event (missing type)
      ✔ Test User Handler logs error on invalid event (missing message)
      ✔ Test User Handler addListener functionality
    Test schema event handler function
      ✔ Test Schema Handler logs error on invalid event (missing type)
      ✔ Test Schema Handler logs error on invalid event (missing message)
    Test componentStatusRequestHandler function
      ✔ Test componentStatusRequestHandler logs error on invalid event (missing type)
      ✔ Test componentStatusRequestHandler logs error on invalid event (missing message)
      ✔ Test componentStatusRequestHandler logs error on invalid event (missing originator)
      ✔ Test componentStatusRequestHandler processes valid event without error

  Test ITCEventObject class
    ✔ Test new ITCEventObject is correct shape

  Test itcUtils module
    Test validateEvent function
      ✔ Test non object error returned
      ✔ Test missing type error returned
      ✔ Test missing message error returned
      ✔ Test invalid event type error returned
      ✔ Test missing originator error returned
    Test constructor functions
      ✔ Test SchemaEventMsg
      ✔ Test UserEventMsg

  Test parseMessage
    ✔ Nominal case, parse with no errors
    ✔ Invalid message json
    ✔ Invalid operation specified
    ✔ Invalid job id
    ✔ Invalid job
    ✔ Invalid json
    ✔ Invalid runner message
    ✔ Test the operation switch statement

  Test runJob
    ✔ Nominal case, run stubbed dataload
    ✔ Throw exception during update to test error handling
    ✔ Invalid message json
    ✔ Invalid job id
    ✔ Invalid job
    ✔ Invalid json
    ✔ Invalid runner message
    ✔ Throw exception during csv data load

  Test jobs.js
    Test getJob & getJobsByStartDate handlers
      ✔ nominal case, call handleGetJobsByStartDate.
      ✔ nominal case, call handleGetJobsByStartDate if end_datetime
      ✔ call getJobsInDateRange, throw an error to test catch.
      ✔ nominal case, call handleGetJob
      ✔ nominal case, call handleGetJob if end_datetime
      ✔ call handleGetJob, throw an error to test catch.
    Test addJob
      ✔ nominal case, add a job to the schema.
      ✔ test calling addJob, invalid job type, expect false.
      ✔ test calling addJob with first search id collision, expect true 
      ✔ test calling addJob with null job.
      ✔ test validation msg from CSV URL load is handled as expected
      ✔ test validation msg from CSV data load is handled as expected
      ✔ test validation msg from import S3 file is handled as expected
      ✔ test error result is returned
      ✔ test error result is returned when error with duplicate id search
      ✔ test error result is returned when error with insert
      ✔ test error result is returned when inserted hashes length is zero
    Test getJobsInDateRange
      ✔ nominal case, search in date ranges.
      ✔ Search with invalid from date, expect error.
      ✔ Search with invalid to date, expect error.
      ✔ Search valid input, no results expected.
      ✔ test custom error message is thrown
    Test getJobById
      ✔ nominal case, find 1 job by ID.
      ✔ Search with null id, expect error
      ✔ test custom error message is thrown
    Test updateJob
      ✔ Nominal case of updateJob
      ✔ Nominal case of updateJob, check end time updated
      ✔ Test bad object check
      ✔ Test missing id check

  Test operationsServer module
    Test calculateRestHttpURL
      ✔ Chooses https when httpSecurePort provided
      ✔ Chooses can use non-standard ports
      ✔ Chooses http when httpSecurePort not provided
      ✔ Uses configured port instead of the one from the request
      ✔ Uses configured port instead of the one from the request
      ✔ Uses port from the request if not otherwise configured
      ✔ Uses the hostname from the request
      ✔ Assume port forwarding and SSL termination with non-localhost, non-ips

  REST - parseHeaderValue
    ✔ should not be vulnerable to denial of service

  Test Headers
    Create and modify headers
      ✔ should handle headers
      ✔ should handle append with commas
      ✔ should handle append with commas on a Map
      ✔ construct headers from object
      ✔ construct headers from Map and merge
      ✔ should handle multiple Set-Cookie headers correctly
      ✔ should keep non-Set-Cookie headers with arrays merged (not split)
      ✔ should handle set() with array values and ensure array elements are strings
      ✔ should handle appendHeader with headers.set path (no append method)
      ✔ should handle appendHeader with plain object (no set or append)
      ✔ should handle mergeHeaders with Set-Cookie on target without append method
      ✔ should handle mergeHeaders with Set-Cookie on target with append method
      ✔ should handle mergeHeaders with plain object target
      ✔ should handle mergeHeaders with single Set-Cookie value (not array)
      ✔ should handle mergeHeaders with Map target and single existing Set-Cookie
      ✔ should handle mergeHeaders with Map target and array existing Set-Cookie
      ✔ should verify mergeHeaders returns the same target (modifies in place)

  Test JSONStream module 
    Streaming
      ✔ Streams object
      ✔ Streams array
      ✔ Streams generator
      ✔ Streams async generator

  Request class
    peerCertificate getter
      ✔ should call getPeerCertificate with true to get full certificate chain
      ✔ should return null when socket has no certificate
      ✔ should handle empty certificate object
      ✔ should ensure certificate chain is available for OCSP verification
    other getters
      ✔ should return correct protocol based on socket encryption
      ✔ should return correct IP address
      ✔ should return authorized status
      ✔ should return mtlsConfig from server
      ✔ should return correct pathname
      ✔ should return correct host
      ✔ should return correct absoluteURL
      ✔ should handle pathname setter
      ✔ should return httpVersion
      ✔ should return isAborted status
    body getter
      ✔ should create RequestBody instance lazily
      ✔ should proxy event handling to node request
      ✔ should proxy pipe to node request
    sendEarlyHints method
      ✔ should send early hints with link header
      ✔ should merge link with additional headers
    Headers class
      ✔ should get headers case-insensitively
      ✔ should set headers case-insensitively
      ✔ should check header existence case-insensitively
      ✔ should delete headers case-insensitively
      ✔ should iterate over headers
      ✔ should return header keys
      ✔ should return header values
      ✔ should iterate with forEach

  test requestTimePlugin
    ✔ test Server-Timing header

  Test serverUtilities.js module 
    Test chooseOperation
      ✔ Nominal path with insert operation.
      ✔ Invalid operation specified in json.
    test getOperationFunction
      ✔ test insert
      ✔ test update
      ✔ test upsert
      ✔ test SEARCH_BY_HASH
      ✔ test SEARCH_BY_VALUE
      ✔ test SEARCH
      ✔ test SQL
      ✔ test CSV_DATA_LOAD
      ✔ test CSV_FILE_LOAD
      ✔ test CSV_URL_LOAD
      ✔ test CREATE_SCHEMA
      ✔ test CREATE_TABLE
      ✔ test CREATE_ATTRIBUTE
      ✔ test DROP_SCHEMA
      ✔ test DROP_TABLE
      ✔ test DROP_ATTRIBUTE
      ✔ test DESCRIBE_SCHEMA
      ✔ test DESCRIBE_TABLE
      ✔ test DESCRIBE_ALL
      ✔ test DELETE
      ✔ test ADD_USER
      ✔ test ALTER_USER
      ✔ test DROP_USER
      ✔ test LIST_USERS
      ✔ test LIST_ROLES
      ✔ test ADD_ROLE
      ✔ test ALTER_ROLE
      ✔ test DROP_ROLE
      ✔ test USER_INFO
      ✔ test READ_LOG
      ✔ test SET_CONFIGURATION
      ✔ test EXPORT_TO_S3
      ✔ test DELETE_FILES_BEFORE
      ✔ test EXPORT_LOCAL
      ✔ test SEARCH_JOBS_BY_START_DATE
      ✔ test GET_JOB
      ✔ test RESTART
      ✔ test CATCHUP
      ✔ test SYSTEM_INFORMATION
      ✔ test DELETE_AUDIT_LOGS_BEFORE
      ✔ test READ_AUDIT_LOG
    Test processLocalTransaction
      ✔ Should return results from callOperationFunctionAsAwait() method
      ✔ Should handle error thrown from callOperationFunctionAsAwait() method
      ✔ Should handle error returned from operation function caller
      ✔ Should wrap non-object results in message object
      ✔ Should not log request body for read_log operation
      ✔ Should strip sensitive fields from logged request body

  server.status.definitions
    constants
      ✔ should export STATUS_DEFINITIONS with correct structure
      ✔ should export STATUS_IDS with all status types
      ✔ should export DEFAULT_STATUS_ID as primary
    exported constants structure
      ✔ should ensure STATUS_IDS contains all keys from STATUS_DEFINITIONS
      ✔ should validate the structure of exported constants

  server.status
    ✔ should set status
    ✔ should get specific status
    ✔ should get complete status with just primary set
    ✔ should get complete status
    ✔ should fail validation on test status
    ✔ should validate availability status values
    getAllStatus functionality
      ✔ should return system status, component status, and restart flag when calling get without id
      ✔ should handle empty component status gracefully
      ✔ should continue working if component status functions are unavailable

  storageReclamation module
    onStorageReclamation
      ✔ should register handler when skipThreadCheck is true
      ✔ should register handler on last worker thread
      ✔ should not register handler on non-last worker thread
      ✔ should register multiple handlers for the same path
      ✔ should register handlers for different paths
      ✔ should set reclamation timer after first handler registration
      ✔ should not create duplicate timers on subsequent registrations
      ✔ should initialize handler entry with priority 0
    setAvailableSpaceRatioGetter
      ✔ should allow setting custom space ratio getter
      ✔ should reset to default getter when passed null
    runReclamationHandlers
      ✔ should not call handler when space is above threshold
      ✔ should call handler when space is below threshold
      ✔ should call handler with priority 0 after space is reclaimed
      ✔ should handle multiple paths independently
      ✔ should handle errors in space ratio getter gracefully
      ✔ should handle errors in handler gracefully
      ✔ should call multiple handlers for the same path
      ✔ should not log when handler returns undefined
      ✔ should not call handler when space is exactly at threshold
      ✔ should reschedule timer after running handlers
      ✔ should update entry priority after each run
      ✔ should not call handler on third run when space stays normal
      ✔ should continue processing other paths after one path errors

  throttle test
    ✔ will throttle calls to a function
    ✔ will limit the queue length of throttled functions (101ms)
    ✔ throttled calls propagate errors


  253 passing (11s)

Integration Tests

% node --test integrationTests/server/*.test.ts
▶ Operations Server
  ✔ health endpoint returns 200 (17.325167ms)
  ✔ POST request without body returns 400 (3.400958ms)
  ✔ POST request with invalid JSON returns 400 (2.535791ms)
  ✔ describe_all operation returns JSON by default (3.798125ms)
  ✔ returns MessagePack when Accept: application/x-msgpack (2.425417ms)
  ✔ parses MessagePack request body (3.733417ms)
  ✔ returns 400 with invalid MessagePack (12.4425ms)
  ✔ returns CBOR when Accept: application/cbor (3.187083ms)
  ✔ parses CBOR request body (2.503333ms)
  ✔ returns 400 with invalid CBOR (6.470125ms)
  ✔ returns CSV when Accept: text/csv (476.625459ms)
  ✔ request without auth works in dev mode (1.69275ms)
  ✔ request with invalid credentials returns 401 (0.614709ms)
✔ Operations Server (2557.484ms)
▶ Storage reclamation
  ✔ verify Harper is running (19.889083ms)
  ✔ create test database and caching table with audit logging (25.307209ms)
  ✔ insert records into caching table (270.83925ms)
  ✔ audit logs are created for insert operations (3.846875ms)
  ✔ records are reclaimed after expiration and reclamation cycle (6010.482125ms)
✔ Storage reclamation (8585.589667ms)
▶ Thread Management
  ✔ server handles concurrent requests across threads (27.375208ms)
  ✔ server recovers from malformed requests without affecting subsequent requests (3.067459ms)
  ✔ server handles mixed concurrent valid and invalid requests (4.493042ms)
✔ Thread Management (2610.012458ms)
ℹ tests 21
ℹ suites 3
ℹ pass 21
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 8691.938833

[cap10morgan]

Looks great! Thanks for the attention to unit vs. integration detail (I should probably go back over the tests I already migrated with this in mind) and for the very helpful tables explaining all the changes!