Skip to content

Staging data ingestion and query optimizations #452

@nitisht

Description

@nitisht
  • Defer the memtable.push flow outside of the write path. For example make the memtable push async / use a separate thread.
  • The disk.push should be made aysnc, so server doesn't wait for the event to be written always. Also discussed in Mitigating blocking IO in highly batched payloads #441
  • Concat of events (in the memtable) happens at 16384 events in memory. We need to either get rid of concat completely or move it out of ingest / write flow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions