Indexing performance on cold starts

[evgenyop]

I have performance issues on every cold start, and Codex times out on Semble MCP calls, despite increased timeouts in the config.
My environment: Windows 11, i7 3GHz CPU, 16 GB RAM.
To reproduce the issue, close all Codex instances and ensure in Task Manager no Codex or Python processes are running. Start Codex (Semble MCP is available) and ask to search the repo. The search times out, and Codex resorts to the default rg.
The test repo is 400k lines of c# and sql code and md docs in 2k files. .gitignore is properly configured. See the test screenshot from Codex:

Config in Codex:
[mcp_servers.semble] command = "uvx" args = ["--from", "C:\\GitHub-public\\AI\\semble[mcp]", "semble", "--include-text-files"] startup_timeout_sec = 120 tool_timeout_sec = 120
I started using the code directly to get --include-text-files as it was not in the release, and I have used it like that since, without installing.
Even if I try using it again in the same Codex session, it times out again.

We only have large-to-very-large repos in our environment, so this issue is very relevant to us. I hope it's a bug which can be easily fixed.
If not, maybe the multi-thread processing should be utilised. That'd be a quick and effective win if you could use the multiprocessing module or port the scanning part into a Rust extension.

[stephantul]

Hey, thanks for raising the issue from the other thread.

Can you manually index using the Python client? e.g.,

# Or whatever path you have
s = SembleIndex.from_path(".")
files = sorted(s._file_mapping)
print(len(files))
print(len(s.chunks))
languages = Counter([x.language for x in s.chunks])
print(languages)

In interactive mode, you can go through the files this indexes and see whether anything is anomalous. We'll be adding a kind of diagnose tool soon.

[evgenyop]

I did some further testing:

1. Direct Python indexing (SembleIndex.from_path(path, include_text_files=True)) of the large repo works in ~31-45s.
2. CLI search (uvx --from "C:\GitHub-public\AI\semble[mcp]" semble search "..." "C:\repo" --include-text-files) works in ~30s.
3. MCP on-demand indexing of the full large repo consistently hangs past 120s.
4. MCP on-demand indexing of smaller repos/subfolders works.
5. MCP startup pre-indexing (via passing the startup repo path parameter in the MCP config) of the same large repo works, and then searches return in ~0.1s.

In essence, startup pre-indexing works as a workaround; the real bug is still full-repo on-demand MCP indexing.

[rtolsma]

• the cli cold start is taking over a minute
• semble should just default create a .semble/{commit hash}.index or smth equivalent to read from warm

[Pringled]

@rtolsma we're working on this and will release this very soon (most likely today or tomorrow).

[rtolsma]

sweet. i was running some profiling as well.

the current cache invalidation -> search -> re-index loop each time is still brutal

on my personal repo:

indexed files: 6,988 -> 6,251
cache size: 101M -> 97M

cold-no-index-cache 17.347s
warm-cache-1 2.449s
warm-cache-2 2.228s
after-touch-reindex 17.439s
warm-after-reindex 2.407s
touch-search-loop-1 17.731s
touch-search-loop-2 17.317s
touch-search-loop-3 17.526s

[rtolsma]

my goal is to make indexing just ambient and fast enough to add to pre commit for fast iteration & automatically staying up to date in a repo

[Pringled]

@rtolsma and @evgenyop we've added persistent caching in v0.3.0. It's automatically invalidated and rebuilt on file changes. Let us know if this works for you! @rtolsma we will look at incremental re-indexing next but that will take some more time to develop.