v2 refactor: @devrev/airsync-sdk (rename, state/adapter split, emit-from-return)#208
Draft
radovanjorgic wants to merge 30 commits into
Draft
v2 refactor: @devrev/airsync-sdk (rename, state/adapter split, emit-from-return)#208radovanjorgic wants to merge 30 commits into
radovanjorgic wants to merge 30 commits into
Conversation
Self-contained plan for the v2 rebuild: branch strategy, 12-commit sequence, hard rules, and baked-in reference data (enum old->new tables, deprecated-file list, connector import surface). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Package identity change for the v2 AirSync rebrand. Scoped to package identity only: - package.json name, version (2.0.0-beta.0), description - package-lock.json (regenerated) - README install command - release workflow references Deferred to later commits: README Airdrop->AirSync prose (C2), the api-extractor report filename + config under backwards-compatibility (C8, test/report territory). Ref: V2_PROGRESS.md C0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the entire src/deprecated/ tree (Adapter, DemoExtractor, HTTPClient, deprecated Uploader, demo metadata) and its four barrel re-exports from src/index.ts. These were unused by production code and any of the inspectable connectors (zero live references). Add tsconfig.build.json that extends tsconfig.json and excludes test files (**/*.test.ts, src/tests). Point the 'build' script at it so 'npm run build' compiles only shippable source. This keeps the build green across subsequent v2 commits even while the test suite (run separately via ts-jest) lags behind the rename/contract changes until it is fixed in Phase 2. BREAKING CHANGE: removed deprecated exports Adapter, DemoExtractor, HTTPClient, and the legacy Uploader from the public API. Ref: V2_PROGRESS.md C1 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d prose Rename the public types AirdropEvent -> AirSyncEvent and AirdropMessage -> AirSyncMessage across all production code (no back-compat alias), and update stale 'ADaaS'/'Airdrop' branding in comments and JSDoc to 'AirSync'. Left untouched (platform contracts): the /internal/airdrop.* API route strings, the AIRDROP_* mapping enum members and their 'airdrop_*' values, and the external_system_type: 'ADaaS' string literal. BREAKING CHANGE: AirdropEvent and AirdropMessage are renamed to AirSyncEvent and AirSyncMessage. Connectors must update their imports and type references. Ref: V2_PROGRESS.md C2 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backend now sends and accepts only the new event-type values, so the old->new translation indirection and the deprecated enum members are no longer needed. - Delete deprecated members from EventType and ExtractorEventType (the EXTRACTION_* values) and the typo/plural dupes from LoaderEventType (DataLoadingDelay, AttachmentsLoading*), keeping only the new members. - Delete src/common/event-type-translation.ts and its test. - Rewire the four callers to use event types directly with no translation: process-task.ts, spawn.ts (incoming), control-protocol.ts emit() and worker-adapter.ts emit() (outgoing). - Drop the now-dead old-member case arms in spawn.helpers.ts (return values unchanged). Behavior is unchanged: translation was identity for new values, which are now the only values on the wire. BREAKING CHANGE: removed deprecated enum members. Connectors must use the new event-type members (e.g. EventType.StartExtractingMetadata instead of EventType.ExtractionMetadataStart; LoaderEventType.DataLoadingDelayed instead of DataLoadingDelay). Ref: V2_PROGRESS.md C3 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tate Structural split of the monolithic State class into a shared abstract BaseState plus mode-specific ExtractionState and LoadingState subclasses. The on-disk state shape is unchanged: still the flat AdapterState = ConnectorState & SdkState (the connector/SDK envelope split comes in a later commit). - base-state.ts: abstract BaseState owns the shared lifecycle (init, fetch, postState) and installInitialDomainMappingIfNeeded, extracted from the old createAdapterState factory. - extraction-state.ts: ExtractionState seeds extractionSdkState and adds resolveExtractionWindow (time-value resolution, pending-boundary reuse, lastSyncStarted, window validation); createExtractionState factory. - loading-state.ts: LoadingState seeds loadingSdkState; createLoadingState factory. - state.ts: thin module re-exporting the classes/factories; createAdapterState is now a dispatcher selecting the mode-specific state by event_context.mode. - Consumers (types/workers.ts, worker-adapter.ts) reference BaseState. Behavior is preserved. Loading mode previously ran extraction-window code that was inert for loading events (no matching event types, no pending boundaries in loadingSdkState); routing it to LoadingState only drops those inert log lines. Ref: V2_PROGRESS.md C4a Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gin/v2) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Separate connector-owned state from SDK bookkeeping on disk. State is now
persisted as a v2 envelope { connectorState, sdkState } instead of a single
flat blob that merged both, so SDK internals stay encapsulated and can never
collide with connector keys.
- state.interfaces.ts: add AdapterStateEnvelope and V1_SDK_STATE_KEYS (the
union of the SDK-owned initial-state keys); mark the flat AdapterState
deprecated. SdkState stays a single combined interface (narrowing into
per-mode variants is deferred to the adapter split).
- base-state.ts: hold _connectorState and _sdkState separately; the state
getter/setter now exposes connector state only; add an sdkState getter/setter
for SDK internals; init() routes fetched state through a normalizeFetchedState()
migration shim; postState() persists the envelope.
- Migration shim: a v2 envelope is used as-is; a legacy flat v1 blob is split
by V1_SDK_STATE_KEYS (SDK keys to sdkState, the rest to connectorState);
a half-envelope or non-object fails loud. Existing syncs migrate on first read.
- extraction-state.ts: resolve the extraction window against sdkState.
- worker-adapter.ts: adapter.state now returns ConnectorState; add an
adapter.sdkState getter; all internal SDK-field access goes through sdkState.
- attachments-streaming-pool.ts: read toDevRev via adapter.sdkState.
BREAKING CHANGE: adapter.state no longer exposes SDK bookkeeping fields
(lastSyncStarted, workersOldest, toDevRev, fromDevRev, snapInVersionId, ...).
Connector state is now disjoint from SDK state. Persisted v1 state is migrated
automatically on read.
Ref: V2_PROGRESS.md C4b
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the monolithic WorkerAdapter with a BaseAdapter holding the shared state access and the emit() control-protocol flow as a template method, plus ExtractionAdapter and LoadingAdapter subclasses that own their mode-specific surface. - base-adapter.ts: abstract BaseAdapter owns event/options/isTimeout/ hasWorkerEmitted, adapterState, uploader, state/sdkState/extractionScope accessors, postState(), and emit(). emit() is a template method with three hooks: beforeEmit (pre-emit work), buildEmitPayload (mode payload extras), afterEmit (post-emit cleanup). - extraction-adapter.ts: ExtractionAdapter owns repos, artifacts, attachment streaming/processing. beforeEmit uploads repos and updates extraction boundaries (incl. lastSuccessfulSyncStarted promotion); buildEmitPayload adds artifacts; afterEmit clears them. An override emit() handles the inline external-sync-unit upload before delegating to BaseAdapter.emit(). - loading-adapter.ts: LoadingAdapter owns mappers, loader reports, item and attachment loading. buildEmitPayload adds reports + processed_files. - worker-adapter.helpers.ts moved to adapters/loading-adapter.helpers.ts (loader-only). Old worker-adapter.ts removed. - WorkerAdapter is now a type alias ExtractionAdapter | LoadingAdapter. processTask builds the concrete adapter from the event's sync mode and passes it to task/onTimeout. Attachment processor types and the streaming pool are typed to ExtractionAdapter. Behavior is preserved: emit() runs the same steps in the same order, the mode-specific payloads are unchanged, and the extraction-boundary bookkeeping is identical. The emit(eventType, data) signature is kept; the emit-from-return contract change is a separate commit. BREAKING CHANGE: WorkerAdapter is no longer a constructable class. Extraction tasks receive an ExtractionAdapter and loading tasks a LoadingAdapter; the methods available are scoped to the worker's mode. Ref: V2_PROGRESS.md C5 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Connectors now signal how a phase ended by RETURNING a TaskResult from their
task/onTimeout callbacks; the SDK maps that status to the phase-appropriate
platform event and emits it exactly once. Connectors never call emit directly.
- types/workers.ts: add TaskResult ({status: 'success'|'progress'|'delay'
(+delaySeconds)|'error' (+error)}) and TaskStatus. ProcessTaskInterface.task
now returns Promise<TaskResult>; onTimeout is optional and also returns
Promise<TaskResult>. TaskAdapterInterface is generic over the adapter. The
WorkerAdapter union alias is removed.
- spawn.helpers.ts: add getEventTypeForResult + EVENT_PHASE_MAP, mapping each
incoming EventType + status to the outgoing event. Resumable phases (data/
attachment extraction, data/attachment loading) honour every status; non-
resumable phases (ESU, metadata, state deletions) only have done/error, so
a progress/delay there is illegal and emits an error with a descriptive msg.
- base-adapter.ts: emit() is now protected (SDK-internal). New public
emitFromResult(result) — the driver-invoked bridge that picks the event from
the mapping and emits with the right payload. State is still saved before
emit for non-stateless events.
- extraction-adapter.ts: streamAttachments returns a TaskResult (timeout ->
progress, pool error/delay -> error/delay, otherwise success) instead of
emitting and exiting. The inline external-sync-unit emit override is removed
(ESUs flow through the repo; connectors push then return success).
- loading-adapter.ts: loadItemTypes and loadAttachments return a TaskResult;
their former mid-flight emits (rate-limit -> delay, timeout -> progress,
error -> error) become returns. They no longer call process.exit.
- process-task.ts: processTask is split into processExtractionTask and
processLoadingTask over a shared runWorkerTask driver that runs task (then
onTimeout, or a default progress result, on timeout) and emits once from the
returned TaskResult.
- index.ts: export processExtractionTask + processLoadingTask; remove
processTask. emit is no longer part of the public surface.
BREAKING CHANGE: connectors must return a TaskResult from task/onTimeout
instead of calling adapter.emit(...), and import processExtractionTask /
processLoadingTask instead of processTask. adapter.emit is no longer public;
the loader and attachment-streaming methods return a TaskResult to be returned
from the task. onTimeout is now optional (defaults to a progress handoff).
Ref: V2_PROGRESS.md C6
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documentation-only pass (C7). Brings the v2-new structural code and the under-documented older modules up to the `src/mappers/mappers.ts` style bar (one-line what + "Used to/for ..." usage line + typed @param/@returns). Covered: - adapters: BaseAdapter template-method emit + emitFromResult, ExtractionAdapter (repos/artifacts/attachment streaming), LoadingAdapter (mappers/reports/loading). - state: BaseState lifecycle, the v1->v2 normalizeFetchedState migration shim, ExtractionState window resolution, the mode-based createAdapterState dispatcher, and state.interfaces (SdkState, envelope, V1_SDK_STATE_KEYS). - multithreading: processExtractionTask/processLoadingTask + runWorkerTask driver, spawn/Spawn supervision, and the getEventTypeForResult / EVENT_PHASE_MAP status->event mapping. - repo, uploader (+helpers/interfaces), attachments-streaming-pool. - common: control-protocol emit, install-initial-domain-mapping, errors; types/loading (previously zero JSDoc) and the types barrel. No executable code, type signatures, imports, or string literals changed (verified: every changed line is a comment). Build green, lint clean.
This reverts commit d05434b.
Three independent cleanups from the v2 self-review: 1. Drop the pre-1.15.2 processed-attachments migration. `migrateProcessedAttachments` only upgraded the legacy `string[]` form of `lastProcessedAttachmentsIdsList` to the structured `ProcessedAttachment[]` form. That `string[]` format predates v1.15.2 (commit c3aa151, Feb 2026); from v1.15.2 on, state is already written as `ProcessedAttachment[]`, and the list is per-cycle scratch state cleared at the end of each completed attachment phase. v2 assumes on-disk state written by >= v1.15.2 (connectors are on >= 1.16, typically 1.19, before moving to v2), so the shim is dead weight. The call site now just initializes the list to [] when absent. 2. Move the `emit` primitive `common/control-protocol.ts` -> `multithreading/emit.ts`. The file's sole export is `emit`, used only by the multithreading layer (base-adapter, spawn) — co-locating it with its consumers and naming it after its content. Import paths updated. 3. Remove the dead `createAdapterState` dispatcher from `state.ts`. After C6 split the single entry point into processExtractionTask/processLoadingTask, process-task.ts calls createExtractionState/createLoadingState directly; the mode dispatcher had zero callers. `state.ts` is now a thin re-export barrel. Build green, lint clean.
- attachments: yield via setImmediate instead of sleep(100) in the streaming progress path; drop the noisy "Sleeping for Nms" log from the sleep() helper. Same soft-timeout observability, ~200s less artificial latency on large syncs. - public API: collapse to a single explicit barrel in src/index.ts (no export *); delete the dead src/http/index.ts and redundant src/types/index.ts; unexport BaseAdapter; promote Mappers/Item/ ItemTypeToLoad onto the root barrel. - timeout: once isTimeout fired, the timeout outcome always wins -- the SDK runs onTimeout (or default progress) and ignores the task return. - types: widen AirSyncEvent.context with user_id/dev_oid/source_id/ service_account_id so connectors stop casting. - docs: add v1 -> v2 MIGRATION.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stop exposing axios as public API while keeping it as an internal
dependency. The internal retry-enabled client (http/axios-client-internal)
is unchanged and still backs uploader, emit, state, mappers and IDM.
Public-surface changes (all v2 breaking):
- Delete the public `axios`/`axiosClient` exports and the
src/http/axios-client.ts file. Connectors needing axios import it directly.
- Replace `formatAxiosError`/`serializeAxiosError` barrel exports with a
single `serializeError(unknown): string`. `serializeAxiosError` stays
exported off-barrel (used internally + by its unit tests).
- Introduce `HttpStreamResponse` (minimal structural supertype of
AxiosResponse: `{ data; headers }`) and use it on the public
`ExternalSystemAttachmentStreamingResponse.httpStream` and the
Uploader/ExtractionAdapter stream params, so AxiosResponse no longer
leaks through public type signatures.
- Mappers methods now unwrap `.data` and return the payload types directly
instead of AxiosResponse envelopes; loading-adapter call sites updated.
Drive-by: remove the dead duplicate `truncateFilename` from common/helpers
(the live copy is in uploader/uploader.helpers), and delete the unused
http/constants.ts and http/types.ts (no remaining importers).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pure file moves/renames + import-path rewrites. No behavior changes;
the public barrel's exported symbol set is unchanged (paths only).
Renames:
- http/axios-client-internal.ts -> http/client.ts (+ .test)
(drop redundant '-internal'; barrel-absence already means internal)
- logger/logger.context.ts -> logger/logger-context.ts
- logger/logger.constants.ts -> logger/logger-constants.ts
- mappers/mappers.interface.ts -> mappers/mappers.interfaces.ts
common/ decomposition (by single-owner rule; common/ now = constants + helpers):
- install-initial-domain-mapping.{ts,test} -> state/
- time-value-resolver.{ts,test} -> state/
- errors.ts -> types/errors.ts
New testing/ module (consolidates all public test-support surface):
- mock-server/{mock-server,mock-server.interfaces,README} -> testing/
- common/test-utils.ts split: createMockEvent + DeepPartial ->
testing/mock-event.ts; MOCK_SERVER_DEFAULT_URL -> testing/mock-server.ts
Deleted: src/multithreading/worker-adapter/ (6 dead test files referencing
the pre-refactor WorkerAdapter/State/AirdropEvent symbols removed in the
adapter split; replacements get written in the deferred test pass).
Allowed companion-file suffixes are now exactly {.interfaces,.helpers,.test};
the one-off .context/.constants suffixes are gone.
Verified: production source typechecks and builds clean; no stale references
to any moved path. Tests remain intentionally broken from the prior adapter
split (they reference deleted modules/symbols) and are fixed in a later pass;
this commit introduces no new module-resolution breakage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merge the two one-off logger peer files back into logger.ts and condense their JSDoc. Both were rejected for folding in the earlier structure pass on cycle/importer-count grounds; re-checking the live tree showed the fold is clean once one misfiled helper moves: - logger-context.ts is a pure leaf (only node:async_hooks) and 5 of its 6 importers already import logger.ts; the 6th (repo.ts) pulls it transitively via Uploader. No cycle. - logger-constants.ts alone would cycle: common/helpers.ts:truncateMessage reads MAX_LOG_STRING_LENGTH and logger.ts imports truncateMessage. Resolved by moving truncateMessage itself into logger.ts — it is a log-formatting helper misfiled in the common/ grab-bag (only callers: logger.ts + base-adapter.ts). This severs the cycle and improves cohesion. Importers repointed to ./logger for runWith*/getSdkLogContextValue/ serializeError/truncateMessage. logger/ is now logger.ts + logger.interfaces.ts + logger.test.ts. Public barrel unchanged (these symbols were always internal). Production typecheck, lint, and build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete three root markdown files that had drifted out of sync with the v2 public surface, and drop README's link to the removed reference: - V2_PROGRESS.md — internal refactor-tracking scaffold, spent. - REFERENCE.md — 1337-line hand-written API doc, ~52 dead-symbol references. - MIGRATION.md — contained false claims about axios still being exported. A correct v1->v2 guide will be written in the deferred docs pass once the public surface is settled and the api report regenerates. Root markdown is now README.md + CONTRIBUTING.md (both accurate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…kerPath Drop the remaining deprecated public surface ahead of v2.0 GA, and align MIGRATION.md. - EventData: remove external_sync_units (ESUs ride out as repo artifacts) and the dead progress field; fix artifacts' incorrect @deprecated tag — it is load-bearing (buildEmitPayload emits it). - SdkState: remove lastSyncStarted / lastSuccessfulSyncStarted. The time-value resolver now degrades WORKERS_NEWEST(_PLUS_WINDOW) to UNBOUNDED when workersNewest is unset (was a lastSuccessfulSyncStarted fallback). Both legacy keys are retained in V1_SDK_STATE_KEYS so old flat blobs are still stripped on migration. - Remove orphaned deprecated types/enums: ExtractionMode, EventContextIn, EventContextOut, DomainObjectState, ErrorLevel, LogRecord, AdapterUpdateParams, AdapterState. - Remove duplicate EventType.UnknownEventType / ExtractorEventType .UnknownEventType (LoaderEventType.UnknownEventType retained). - Remove deprecated spawn() workerPath option (use baseWorkerPath). - Prune the public barrel accordingly. Cross-checked against 9 connector repos: none of the removed symbols are imported by any v2 connector. Build typechecks clean. The pre-existing broken test suite is repaired in a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v2 refactor of
@devrev/airsync-sdk(formerly@devrev/ts-adaas)Clean, commit-by-commit rebuild of the SDK for v2. Breaking changes are intended. Each commit is single-purpose and keeps
npm run buildgreen; the test suite and api-extractor report are intentionally left for the closing steps.Rename & branding
@devrev/ts-adaas→@devrev/airsync-sdk(2.0.0-beta.0)AirdropEvent/AirdropMessage→AirSyncEvent/AirSyncMessage(hard rename, no alias); proseAirdrop/ADaaS→AirSync/internal/airdrop.*API routes,AIRDROP_*mapping enum, the'ADaaS'literalDeletions / cleanup
src/deprecated/**and its barrel exportsEXTRACTION_*enum members +event-type-translation.ts); backend now sends/accepts only the new event typesmigrateProcessedAttachmentsshim (v2 assumes on-disk state ≥ v1.15.2)createAdapterStatedispatcher (obviated by the C6 entry-point split)State redesign
StateintoBaseState+ExtractionState+LoadingState{ connectorState, sdkState }, with a v1-flat → v2-envelope migration shim on readadapter.statenow returns connector state only; SDK fields move toadapter.sdkStateAdapter redesign
WorkerAdapterintoBaseAdapter+ExtractionAdapter+LoadingAdapteremit()is now a template method (beforeEmit / buildEmitPayload / afterEmit) and SDK-internalExecution model (emit-from-return)
task/onTimeoutnow return aTaskResult(success/progress/delay/error); the SDK maps status → phase event and emits exactly onceprocessTasksplit intoprocessExtractionTask+processLoadingTaskTaskResultinstead of emitting mid-flightMisc
emitprimitivecommon/control-protocol.ts→multithreading/emit.ts(co-located with its only consumers)Remaining before ready-for-review
airsync-sdk.api.md)migrate-v2skill (v1→v2 catalog)🤖 Generated with Claude Code