Skip to content

Latest commit

 

History

History
220 lines (172 loc) · 4.6 KB

restructure.md

File metadata and controls

220 lines (172 loc) · 4.6 KB

restructure operator

The restructure operator facilitates changing the structure of a record by adding, removing, moving, and flattening fields.

The operator is configured with a list of ops, which are small operations that are applied to a record in the order they are defined.

Configuration Fields

Field Default Description
id restructure A unique identifier for the operator
output Next in pipeline The connected operator(s) that will receive all outbound entries
ops required A list of ops. The available op types are defined below
on_error send The behavior of the operator if it encounters an error. See on_error
if An expression that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers.

Op types

Add

The add op adds a field to a record. It must have a field key and exactly one of value or value_expr.

field is a field that will be set to value or the result of value_expr

value is a static string that will be added to each entry at the field defined by field

value_expr is an expression with access to the record object

Example usage:

- type: restructure
  ops:
    - add:
        field: "key1"
        value: "val1"
    - add:
        field: "key2"
        value_expr: 'record["key1"] + "-suffix"'
Input record Output record 
{}
 
{
  "key1": "val1",
  "key2": "val1-suffix"
}

Remove

The remove op removes a field from a record.

Example usage:

- type: restructure
  ops:
    - remove: "key1"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2"
}
 
{
  "key2": "val2"
}

Retain

The retain op keeps the specified list of fields, and removes the rest.

Example usage:

- type: restructure
  ops:
    - retain:
      - "key1"
      - "key2"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2",
  "key3": "val3",
  "key4": "val4"
}
 
{
  "key1": "val1",
  "key2": "val2"
}

Move

The move op moves (or renames) a field from one location to another. Both the from and to fields are required.

Example usage:

- type: restructure
  ops:
    - move:
        from: "key1"
        to: "key3"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2"
}
 
{
  "key3": "val1",
  "key2": "val2"
}

Flatten

The flatten op flattens a field by moving its children up to the same level as the field.

Example usage:

- type: restructure
  ops:
    - flatten: "key1"
Input record Output record 
{
  "key1": {
    "nested1": "nestedval1",
    "nested2": "nestedval2"
  },
  "key2": "val2"
}
 
{
  "nested1": "nestedval1",
  "nested2": "nestedval2",
  "key2": "val2"
}