Add WarehouseLambda core with S3 WarehouseDumper - #974
Merged
Conversation
jwils
requested review from
BrianSigafoos-SQ,
ayousufi,
bsorbo,
jwondrusch,
lancethomps,
myronmarston,
nicolewoch and
thomasmahoney
as code owners
January 15, 2026 15:53
jwils
force-pushed
the
joshuaw/warehouse-lambda-1-scaffold
branch
from
January 15, 2026 16:01
26fc5bf to
7d17137
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 15, 2026 16:02
0a4c78d to
bd44498
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-1-scaffold
branch
2 times, most recently
from
January 16, 2026 17:54
2cfa97e to
4cf4c6e
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 16, 2026 17:54
bd44498 to
85b800b
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-1-scaffold
branch
3 times, most recently
from
January 17, 2026 15:05
6174878 to
735851d
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
2 times, most recently
from
January 17, 2026 15:15
182a666 to
3ecf91f
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-1-scaffold
branch
from
January 17, 2026 15:32
735851d to
3d20dca
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
3 times, most recently
from
January 17, 2026 20:03
03a355f to
acbf663
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-1-scaffold
branch
from
January 17, 2026 20:21
3d20dca to
b2e296b
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 17, 2026 20:32
acbf663 to
af3f70c
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
5 times, most recently
from
January 17, 2026 22:13
8c6089c to
1c62fe0
Compare
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
7 times, most recently
from
January 18, 2026 02:11
315be59 to
7cea41d
Compare
myronmarston
requested changes
Jan 18, 2026
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 19, 2026 03:41
1a6f1d7 to
1229b07
Compare
myronmarston
requested changes
Jan 19, 2026
myronmarston
left a comment
Collaborator
There was a problem hiding this comment.
Left some more feedback. Most of it is cosmetic...with the exception of the version vs json_schema_version mixup--that one matters!
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 19, 2026 14:31
1229b07 to
e201d75
Compare
myronmarston
approved these changes
Jan 19, 2026
myronmarston
left a comment
Collaborator
There was a problem hiding this comment.
LGTM apart from the build failure. Left a suggestion which should fix it.
jwils
enabled auto-merge (squash)
January 19, 2026 23:25
jwils
disabled auto-merge
January 19, 2026 23:26
This commit implements the core S3 data export functionality for the elasticgraph-warehouse_lambda gem, including the main WarehouseLambda class and WarehouseDumper implementation. Components added: - WarehouseLambda: Main class that orchestrates indexer, S3 client, and warehouse dumper with from_parsed_yaml factory method - WarehouseDumper: Implements DatastoreIndexingRouter interface to dump indexing operations to S3 as gzipped JSONL files - Configures indexer with WarehouseDumper as the datastore_router to redirect all indexing operations to S3 instead of the datastore Key features: - Groups operations by GraphQL type and writes separate JSONL files per type - Generates deterministic S3 keys with versioning, date partitioning, and UUIDs: s3://bucket/dumped-data/<prefix>/<Type>/v<version>/<date>/<uuid>.jsonl.gz - Structured logging for observability (batch counts, file sizes, S3 keys) - Extracts data from operation payloads including id, __eg_version, and record fields - Filters operations to only include primary index updates - Skips operations with no datastore payload (e.g., delete operations) - Returns BulkResult with success status for all operations - Leverages aws-sdk-s3 with SHA256 checksum validation and conditional writes - S3 client initialization with optional region override from config Testing: - 37 comprehensive test examples covering all components - Tests verify S3 operations, JSONL formatting, compression, and logging - Uses stubbed AWS S3 client for fast, deterministic tests - Full RBS type signatures for type safety - 100% test coverage (from_env marked :nocov: until LambdaFunction added in next commit) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Myron Marston <myron.marston@gmail.com>
jwils
force-pushed
the
joshuaw/warehouse-lambda-2-core
branch
from
January 20, 2026 01:09
33a6d6d to
b716416
Compare
…da.rbs Co-authored-by: Myron Marston <myron.marston@gmail.com>
2 tasks
jwils
added a commit
that referenced
this pull request
Apr 23, 2026
Distills the patterns Myron Marston applies when reviewing merged PRs (naming at the caller's level of abstraction, respond_to? as a code smell, wrapper-class DI pattern, load-bearing tests, RBS/YARD hygiene, etc.) into an edit-first skill. Invoked as /myron-polish, the skill walks the current branch's diff against main and applies fixes directly rather than producing review comments. It loops — re-scan, apply edits, run script/lint --fix / script/spellcheck -w / script/type_check / script/run_gem_specs — until a full iteration makes zero edits and every verification command passes. Capped at 8 iterations. Source material: Myron's review bodies and inline comments on merged PRs #974, #973, #1067, #1066, #1108, #1120, #1131, #1134, #1144, #1107. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit implements the core S3 data export functionality for the
elasticgraph-warehouse_lambda gem, including the main WarehouseLambda
class and WarehouseDumper implementation.
Components added:
warehouse dumper with from_parsed_yaml factory method
indexing operations to S3 as gzipped JSONL files
redirect all indexing operations to S3 instead of the datastore
Key features:
s3://bucket/dumped-data///v//.jsonl.gz
Testing: