feat: add `progress_data` to `worker_metadata` by gasperzgonec · Pull Request #202 · devrev/adaas-sdk

gasperzgonec · 2026-05-19T11:03:47Z

This PR flattens extraction progress metadata into worker_metadata and adds timestamp bounds the platform can use to detect incremental syncs that are not advancing.

NormalizedAttachment.created_date and NormalizedAttachment.modified_date (optional): attachments can now contribute source timestamps the same way normalized items do.
Repo.dateRanges: each repo now tracks the oldest/newest valid created_date and modified_date seen across uploaded batches.
Flat worker_metadata progress fields: extraction progress/done events now send the latest extracted repo’s item_type, oldest_created_date, newest_created_date, oldest_modified_date, and newest_modified_date.
State window fields in worker_metadata: oldest_state_date and newest_state_date are copied from event_context.extract_from / extract_to on emitted callback payloads.

Connected Issues

#ISS-252036

Checklist

Tests added/updated and ran with npm run test OR no tests needed.
Ran backwards compatibility tests with npm run test:backwards-compatibility.
Code formatted and checked with npm run lint.
Tested airdrop-template linked to this PR.
Documentation updated and provided a link to PR / new docs OR no docs needed.

Migration note

If your connector normalizes attachments, populate created_date and/or modified_date on each NormalizedAttachment using the source system timestamp in RFC3339 format.

const normalizedAttachment: NormalizedAttachment = {
  id: attachment.id,
  url: attachment.downloadUrl,
  file_name: attachment.name,
  parent_id: attachment.parentId,
  created_date: attachment.createdAt,
  modified_date: attachment.updatedAt,
};

Notes:

Attachment timestamp fields are optional, so existing connectors keep working.
Attachment repos only contribute created/modified bounds when those fields are populated.
Invalid timestamp strings are ignored and do not corrupt the tracked ranges.
Normalized items already support created_date / modified_date; no connector-side change is needed there.

What `worker_metadata` now contains

worker_metadata is part of the callback payload. It is not inside event_data.

Repo progress fields

These fields are attached only for:

DATA_EXTRACTION_PROGRESS
DATA_EXTRACTION_DONE
ATTACHMENT_EXTRACTION_PROGRESS
ATTACHMENT_EXTRACTION_DONE

Shape:

worker_metadata: {
  item_type?: string;
  oldest_created_date?: string;
  newest_created_date?: string;
  oldest_modified_date?: string;
  newest_modified_date?: string;
  oldest_state_date?: string;
  newest_state_date?: string;
  adaas_library_version?: string;
}

Behavior:

The repo progress fields describe only the latest extracted repo.
Values are emitted as RFC3339 strings.
Unset bounds are omitted.
Loader events and non-progress extraction events do not include repo progress fields.

Example:

{
  "worker_metadata": {
    "item_type": "issues",
    "oldest_created_date": "2024-01-01T00:00:00.000Z",
    "newest_created_date": "2024-05-01T00:00:00.000Z",
    "oldest_modified_date": "2024-02-01T00:00:00.000Z",
    "newest_modified_date": "2024-06-01T00:00:00.000Z",
    "oldest_state_date": "2024-01-01T00:00:00.000Z",
    "newest_state_date": "2024-06-01T00:00:00.000Z",
    "adaas_library_version": "..."
  }
}

How repo progress fields are computed

On each Repo.upload(), the SDK scans uploaded items for valid created_date and modified_date values.
It updates that repo’s running dateRanges.creationDate and dateRanges.modifiedDate.
On the extraction progress/done events above, WorkerAdapter emits the bounds for the repo identified by the latest uploaded itemType.

How state window fields are computed

oldest_state_date is copied from event_context.extract_from.
newest_state_date is copied from event_context.extract_to.
These values are injected by control-protocol.emit() and override caller-provided state-date values.

Why this exists

The platform can compare the emitted repo timestamp bounds together with the absolute extraction window. If repeated runs keep reporting the same latest-item bounds and state window, that is a signal that the incremental sync is not making forward progress.

Started returning `progress_data` in `worker_metadata`.

radovanjorgic

No tests at all? :/

radovanjorgic · 2026-05-20T05:29:32Z

    const itemsToUpload = batch || this.items;

    if (itemsToUpload.length > 0) {
+      for (const item of itemsToUpload) {


Huh, I have few questions:

Do we really need to do this for each item? How is that from performance perspective? Let's say you have 100+ repos at once each with 5000 items in it?

What if timeout comes at this point?

Can we offload this work to backend? After storing the files, extractor-adapter/snap-in manager scans them and picks progress from there.

For the #3, we'll need @GasperSenk 's input, but I don't think so.

For the #1 and #2, I think this is fine.
This logic takes O(n) in time and O(1) in memory use.
There's just integer comparisons, and it should be fast enough to not block any isTimeout checks.

No, adapter doesn't know the normalization function to find the correct field name for the dates.

radovanjorgic · 2026-05-20T05:29:57Z

    if (itemsToUpload.length > 0) {
+      for (const item of itemsToUpload) {
+        if (
+          item != null &&


Simply just if (item?.created_date)?

This didn't work before I added created_date to the NormalizedAttachment.
Good catch!

radovanjorgic · 2026-05-20T05:30:15Z

+          'created_date' in item &&
+          item.created_date != null
+        ) {
+          const created_date = new Date(item['created_date']).getTime();


Don't use snake case for variable names please.

radovanjorgic · 2026-05-20T05:31:21Z

+    min: 0,
+    max: 0,


What do min and max mean here? Maybe we should use oldest and newest instead?

These are min and max timestamps. Due to them being numbers, we decided to go with the min and max.
Discussed in the ISS comments.

Might make sense to use newest and oldest, we do that in the backend, but I see why you would use min and max when you are dealing with longs.

GasperSenk · 2026-06-03T10:10:15Z

  eventType: ExtractorEventType | LoaderEventType;
  data?: EventData;
+  worker_metadata?: {
+    progress_data: Record<string, { min: number; max: number }>;


This is only a single entry?

GasperSenk · 2026-06-03T10:11:01Z

+            newEventType == ExtractorEventType.AttachmentExtractionProgress)
+        ) {
+          for (const repo of this.repos) {
+            itemTimestamps[repo.itemType] = repo.itemTimestamps;


We said we only want one entry for progress.
The latest from the list.

GasperSenk · 2026-06-03T10:12:22Z

  public uploadedArtifacts: Artifact[];
+  public itemTimestamps: { min: number; max: number } = {
+    min: 0,
+    max: 0,


We said to have a single record type and contain created and modified dates.

radovanjorgic · 2026-06-12T07:48:23Z

Closing because we opened new PR which merges #202 and #205

feat: added created_date to the NormalizedAttachment

d86af02

Started returning `progress_data` in `worker_metadata`.

gasperzgonec requested review from a team and radovanjorgic as code owners May 19, 2026 11:03

radovanjorgic requested changes May 20, 2026

View reviewed changes

gasperzgonec added 7 commits May 20, 2026 10:34

chore: renamed variable and improved nullability check

0d841c1

chore: added tests for timestamp reporting

785b327

feat: Updated to only report on the last item being processed.

cab6a5e

fix(bug): Invalid dates would get stuck on NaN.

7c8b476

feat: Changed timestamps to RFC3339 strings

643d1a4

Merge branch 'main' into gasperz/ISS-252036

99027f8

feat: Flattened the worker_metadata, added state_date fields.

a0ce4ee

GasperSenk approved these changes Jun 9, 2026

View reviewed changes

radovanjorgic marked this pull request as draft June 9, 2026 09:57

gasperzgonec mentioned this pull request Jun 12, 2026

feat: Add timestamps to artifacts and worker_metadata for easier tracking #210

Open

5 tasks

radovanjorgic mentioned this pull request Jun 12, 2026

feat: add modified date timestamps to artifacts #205

Closed

5 tasks

radovanjorgic closed this Jun 12, 2026

Uh oh!

Conversation

gasperzgonec commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Connected Issues

Checklist

Migration note

What worker_metadata now contains

Repo progress fields

How repo progress fields are computed

How state window fields are computed

Why this exists

Uh oh!

radovanjorgic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radovanjorgic commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gasperzgonec commented May 19, 2026 •

edited

Loading

What `worker_metadata` now contains