Skip to content

Latest commit

 

History

History
177 lines (139 loc) · 5.47 KB

timestamp.md

File metadata and controls

177 lines (139 loc) · 5.47 KB

timestamp parsing parameters

Parser operators can parse a timestamp and attach the resulting time value to a log entry.

Field Default Description
parse_from required A field that indicates the field to be parsed as JSON
layout_type strptime The type of timestamp. Valid values are strptime, gotime, and epoch
layout required The exact layout of the timestamp to be parsed
preserve_to Preserves the unparsed value at the specified field
location Local The geographic location (timezone) to use when parsing a timestamp that does not include a timezone. The available locations depend on the local IANA Time Zone database. This page contains many examples, such as America/New_York.

How to specify timestamp parsing parameters

Most parser operators, such as regex_parser support these fields inside of a timestamp block.

If a timestamp block is specified, the parser operator will perform the timestamp parsing after performing its other parsing actions, but before passing the entry to the specified output operator.

- type: regex_parser
  regexp: '^Time=(?P<timestamp_field>\d{4}-\d{2}-\d{2}), Host=(?P<host>[^,]+)'
  timestamp:
    parse_from: timestamp_field
    layout_type: strptime
    layout: '%Y-%m-%d'

As a special case, the time_parser operator supports these fields inline. This is because time parsing is the primary purpose of the operator.

- type: time_parser
  parse_from: timestamp_field
  layout_type: strptime
  layout: '%Y-%m-%d'

Example Configurations

Parse a timestamp using a strptime layout

The default layout_type is strptime, which uses "directives" such as %Y (4-digit year) and %H (2-digit hour). A full list of supported directives is found here.

Configuration:

- type: time_parser
  parse_from: timestamp_field
  layout_type: strptime
  layout: '%a %b %e %H:%M:%S %Z %Y'
Input entry Output entry 
{
  "timestamp": "",
  "record": {
    "timestamp_field": "Jun 5 13:50:27 EST 2020"
  }
}
 
{
  "timestamp": "2020-06-05T13:50:27-05:00",
  "record": {}
}

Parse a timestamp using a gotime layout

The gotime layout type uses Golang's native time parsing capabilities. Golang takes an unconventional approach to time parsing. Finer details are well-documented here.

Configuration:

- type: time_parser
  parse_from: timestamp_field
  layout_type: gotime
  layout: Jan 2 15:04:05 MST 2006
Input entry Output entry 
{
  "timestamp": "",
  "record": {
    "timestamp_field": "Jun 5 13:50:27 EST 2020"
  }
}
 
{
  "timestamp": "2020-06-05T13:50:27-05:00",
  "record": {}
}

Parse a timestamp using an epoch layout (and preserve the original value)

The epoch layout type uses can consume epoch-based timestamps. The following layouts are supported:

Layout Meaning Example parse_from data type support
s Seconds since the epoch 1136214245 string, int64, float64
ms Milliseconds since the epoch 1136214245123 string, int64, float64
us Microseconds since the epoch 1136214245123456 string, int64, float64
ns Nanoseconds since the epoch 1136214245123456789 string, int64, float64[2]
s.ms Seconds plus milliseconds since the epoch 1136214245.123 string, int64[1], float64
s.us Seconds plus microseconds since the epoch 1136214245.123456 string, int64[1], float64
s.ns Seconds plus nanoseconds since the epoch 1136214245.123456789 string, int64[1], float64[2]

[1] Interpretted as seconds. Equivalent to using s layout.
[2] Due to floating point precision limitations, loss of up to 100ns may be expected.

Configuration:

- type: time_parser
  parse_from: timestamp_field
  layout_type: epoch
  layout: s
  preserve: true
Input entry Output entry 
{
  "timestamp": "",
  "record": {
    "timestamp_field": 1136214245
  }
}
 
{
  "timestamp": "2006-01-02T15:04:05-07:00",
  "record": {
    "timestamp_field": 1136214245
  }
}