Entity Store: Make log extraction scalable across spaces

[uri-weisman]

Summary

In 9.4, Entity Store is enabled by default in the default space only. We want it enabled in all spaces, but the current design does not scale: resource usage grows linearly with the number of spaces without a corresponding benefit, because client logs are not space-aware.

Current behavior

• Entity Store runs log extraction on the indices defined in the default security data view.
• It aggregates those logs and writes results into one index per space.
• The source logs (client/ingestion logs) are not space-aware; the same underlying data is effectively processed once per space.
• So we repeat the same extraction work for every space, multiplying cost without adding value for non-default spaces.

Problem

• Enabling Entity Store in all spaces would cause extraction to run N times (once per space) over the same log data.
• This makes the feature scale linearly with the number of spaces and blocks enabling Entity Store by default in all spaces.

Desired outcome

• Examine and propose a solution where:
  • Log extraction is scalable (e.g. not duplicated per space when logs are not space-specific).
  • Resource usage does not grow linearly with the number of spaces.
• Options to consider might include: running extraction once and sharing/partitioning results by space where meaningful, or making extraction space-aware only where the data source is space-aware.

Acceptance criteria (suggested)

• Document current behavior and why it scales linearly with spaces.
• Propose an approach that avoids redundant extraction for non–space-aware log sources.
• Identify any trade-offs (e.g. space-specific vs shared indices, migration path for existing per-space indices).