Group JSON ingestion schema state - #1281
Closed
jwils wants to merge 1 commit into
Closed
Conversation
jwils
force-pushed
the
joshuaw/json-ingestion-state-container
branch
2 times, most recently
from
June 30, 2026 23:14
c443df8 to
d660189
Compare
jwils
force-pushed
the
joshuaw/json-ingestion-state-container
branch
from
July 1, 2026 18:16
d660189 to
ee70fc0
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the Protocol Buffers schema artifact generation logic. Running `schema_artifacts:dump` emits: - `schema.proto` — the generated Protobuf schema for the indexed types - `proto_field_numbers.yaml` — a sidecar that reserves field numbers and enum value numbers so they stay wire-stable as the schema evolves (including across field renames, and reserving removed values so numbers are not reused) Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars. - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281. The generator emits `proto3` by default and can emit `proto2` via `syntax: :proto2` (which labels every field `optional`/`repeated`). Arbitrary file-level headers (e.g. `option` declarations) can be injected verbatim via `headers:`, so language-specific options can be set without baking any particular convention into the gem.
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the core Protocol Buffers schema generation logic. Running `schema_artifacts:dump` emits a `proto3` `schema.proto` artifact covering the schema's indexed types. Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars (resolved via `type_ref.with_reverted_override` so built-in scalars renamed with `type_name_overrides` keep working). - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. - `schema.proto` uses the public GraphQL field names, and validates proto package names (each dot-separated segment must be a valid identifier). Field and enum value numbers are assigned sequentially in definition order; keeping them wire-stable across schema evolution comes in a stacked follow-up that adds a `proto_field_numbers.yaml` sidecar artifact. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281.
jwils
added a commit
that referenced
this pull request
Jul 11, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the core Protocol Buffers schema generation logic. Running `schema_artifacts:dump` emits a `proto3` `schema.proto` artifact covering the schema's indexed types. Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars (resolved via `type_ref.with_reverted_override` so built-in scalars renamed with `type_name_overrides` keep working). - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. - `schema.proto` uses the public GraphQL field names, and validates proto package names (each dot-separated segment must be a valid identifier). Field and enum value numbers are assigned sequentially in definition order; keeping them wire-stable across schema evolution comes in a stacked follow-up that adds a `proto_field_numbers.yaml` sidecar artifact. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281.
jwils
added a commit
that referenced
this pull request
Jul 11, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the core Protocol Buffers schema generation logic. Running `schema_artifacts:dump` emits a `proto3` `schema.proto` artifact covering the schema's indexed types. Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars (resolved via `type_ref.with_reverted_override` so built-in scalars renamed with `type_name_overrides` keep working). - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. - `schema.proto` uses the public GraphQL field names, and validates proto package names (each dot-separated segment must be a valid identifier). Field and enum value numbers are assigned sequentially in definition order; keeping them wire-stable across schema evolution comes in a stacked follow-up that adds a `proto_field_numbers.yaml` sidecar artifact. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281.
jwils
added a commit
that referenced
this pull request
Jul 15, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the core Protocol Buffers schema generation logic. Running `schema_artifacts:dump` emits a `proto3` `schema.proto` artifact covering the schema's indexed types. Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars (resolved via `type_ref.with_reverted_override` so built-in scalars renamed with `type_name_overrides` keep working). - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. - `schema.proto` uses the public GraphQL field names, and validates proto package names (each dot-separated segment must be a valid identifier). Field and enum value numbers are assigned sequentially in definition order; keeping them wire-stable across schema evolution comes in a stacked follow-up that adds a `proto_field_numbers.yaml` sidecar artifact. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281.
jwils
added a commit
that referenced
this pull request
Jul 20, 2026
## Why - continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem - revive the earlier protobuf prototype from #1056 on top of the new serializer extension points ## What - fill in the `elasticgraph-proto_ingestion` extension gem with core generation: `schema_artifacts:dump` emits a `proto3` `schema.proto` covering the schema's indexed types - map built-in ElasticGraph scalars to proto types, with `t.protobuf type:` for custom scalars (resolved via `type_ref.with_reverted_override` so built-ins renamed with `type_name_overrides` keep working) - generate messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid - keep `schema.proto` on public GraphQL field names; validate proto package names - hold extension state on a `ProtoIngestionState` container behind a single `proto_ingestion_state` reader (matching #1281) Field and enum value numbers are assigned sequentially in definition order in this PR; the stacked follow-up adds the `proto_field_numbers.yaml` sidecar that keeps them wire-stable across schema evolution. ## Stacked follow-ups 1. this PR — core `schema.proto` generation 2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml` sidecar 3. `syntax: :proto2` support and custom file-level `headers:` 4. #1286 — enum value sourcing from existing proto enums + external proto type references ## Verification - `script/run_gem_specs elasticgraph-proto_ingestion` (100% line + branch coverage at this commit) - `script/type_check`, `script/lint`, `script/spellcheck` - `script/quick_build` green at the stack head (whose tree is identical to the previously reviewed single-PR revision) ## References - #1059 - #1056 - #1079 ## Update — 2026-07-10 - The current stack is #1080 → #1304 → #1306 → #1305 → #1286. - Proto extension state now uses a mutable Struct, and keyword package-name segments are validated without being rewritten. - Lists of lists now raise an actionable schema error instead of generating wrapper messages; this supersedes the earlier wrapping note above. --- ## Update — 2026-07-15 - Interface and union messages now wrap concrete subtype messages in a `oneof`, matching the JSON Schema `oneOf` representation. - Concrete subtype messages omit the redundant `__typename` discriminator. - Proto type rendering is now stateless: the generator selects the reachable type graph up front, then each extended type renders itself without mutating shared traversal state. - No-block extension coverage now completes the definitions so the fixture remains valid under CI GraphQL-schema validation. --- ## Update — 2026-07-19 - Replaced keyword suffixing with fully qualified local message and enum references, preserving source type and field names while disambiguating contextual protobuf words and built-in scalar names. - Removed the now-unnecessary keyword collision tracking and verified a generated schema containing contextual names with `protoc` 35.1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Keep JSON ingestion-specific schema definition state off the shared
SchemaDefinition::Statesurface. This follows up on #1259 by grouping the extension state behind one accessor.What
JSONIngestionStateand attach it viaState#json_ingestion_stateRisk Assessment
Low — internal schema-definition state refactor; public DSL behavior remains unchanged.
References
Validation:
script/lint --fixscript/spellcheck -wscript/type_checkRUBYOPT=--disable-frozen-string-literal bundle exec rake site:doctestbundle exec rspec elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/deprecated_element_spec.rb elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/json_schema_spec.rb elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/indexing/json_schema_with_metadata_spec.rbscript/run_gem_specs elasticgraph-json_ingestionscript/run_gem_specs elasticgraph-schema_definitionandscript/quick_build; local datastore boot is blocked because Docker is not running (~/.docker/run/docker.sockis unavailable).RUBYOPT=--disable-frozen-string-literal bundle exec rake site:validate; the CI-relevant doctest failure is fixed, but local execution later stops insite:npm_installwith an npm dependency resolution issue.