Skip to content

#409 Parallel file processing with rayon #413

Description

@ajianaz

Problem

File discovery, parsing, and DB write is fully sequential. No use of multi-core.

Location

src/index/mod.rs:222-258 -- for entry in walker { ... index_file() }

Solution

  1. Add rayon dependency
  2. Collect files into Vec, then par_iter() for parsing
  3. Parse files in parallel (CPU-bound)
  4. Write results via channel to single writer thread

Expected Impact

2-4x faster indexing on multi-core.

Dependency

Blocked by #404, #405, #404b.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions