Skip to content

Frankkim/update tap to store nested data as jsonb field - #1

Merged
francisace merged 4 commits into
masterfrom
frankkim/update-tap-to-store-nested-data-as-jsonb-field
Nov 1, 2023
Merged

Frankkim/update tap to store nested data as jsonb field#1
francisace merged 4 commits into
masterfrom
frankkim/update-tap-to-store-nested-data-as-jsonb-field

Conversation

@francisace

@francisace francisace commented Nov 1, 2023

Copy link
Copy Markdown

PR Description for target-postgres

Title

Disable Table De-Nesting and Store All Nested JSON Objects in JSONB Columns

Description

Summary

This Pull Request addresses the handling of nested JSON objects in target-postgres. Instead of de-nesting the tables, all nested structures—including arrays—will now be stored directly in JSONB columns.

Motivation

Handling nested objects by creating separate tables (de-nesting) can result in a complex schema that's hard to manage and query. By storing the nested objects directly as JSONB, we simplify the schema and make it easier to query nested data.

Changes

  • Disabled the table de-nesting logic.
  • All nested JSON objects and arrays are now saved in JSONB columns.
  • Updated the test cases to reflect these changes.

How it works

Here is an example to illustrate the new behavior:

Previous Behavior

For a JSON payload like the following:

{
  "name": "John",
  "info": {
    "email": "john@example.com",
    "address": {
      "city": "NY",
      "state": "NY"
    }
  }
}

The data would be saved in multiple tables, such as main_table, info_table, and address_table.

New Behavior
Now, the data will be saved in a single table with JSONB columns:

name info
John {"email": "john@example.com", "address": {"city": "NY", "state": "NY"}}

@clarkeRS
clarkeRS self-requested a review November 1, 2023 19:58
@francisace
francisace merged commit ee56077 into master Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants