Constrained query expansion#998
Merged
Merged
Conversation
Add constrained query expansion step to /graphify query skill
## Problem
`graphify query` matches via case-folded substring + IDF — no stemming, no synonyms, no cross-language match. When the user's question uses different vocabulary than the graph labels (Slavic → English, "handlers" → "handler", "обработчик" → "handler"), the literal matcher returns 0
hits and the LLM consumer either gets empty subgraph or improvises an ungrounded keyword list from training memory (e.g. expanding "auth" to `{passport, sso, saml, oauth, jwt, scim, …}` regardless of whether those tokens exist in the corpus).
## Fix
Adds a `Step 0 — Constrained query expansion` block to the skill's `/graphify query` section. The LLM consumer extracts vocabulary from graph labels (CamelCase/snake_case split, length-filtered) and is instructed to pick **only** tokens present in that vocabulary, explicitly forbidden from inventing terms.
Effects:
- Bounded improvisation — fantom tokens (terms not in corpus) cannot be expanded, even when LLM "knows" they're related to the intent.
- Honest negative signal — if vocab is poor on a query's topic, expansion returns [] and the LLM tells the user, instead of fabricating a search.
- Auditability — selected tokens are printed to the user, and saved into `save-result` for the next --update to graph as Q&A nodes.
## Scope
Patches the canonical `graphify/skill.md`. The 11 host-variant skills (skill-codex.md, skill-aider.md, …) follow the same query-section contract but inline Python rather than calling `graphify query` CLI; those need a parallel patch with the inline form. Happy to follow up in a separate PR after review on the canonical patch.
## Test
On a graph built from the graphify repo itself (1284 nodes, 1454 vocab tokens), an unconstrained expansion of "укрупненная архитектура аутентификации" yields {auth, oauth, jwt, saml, sso, ldap, scim, mfa, 2fa, pin, passport, session, login, token} — of which 11/15 are absent
from the corpus. Constrained expansion against the actual vocab yields {credential, security, token, signature, user, architecture, component, module, overview} — 9 tokens, 0 fantom. Same retrieval, dramatically higher precision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add constrained query expansion step to /graphify query skill
Problem
graphify querymatches via case-folded substring + IDF — no stemming, no synonyms, no cross-language match. When the user's question uses different vocabulary than the graph labels (Slavic → English, "handlers" → "handler", "обработчик" → "handler"), the literal matcher returns 0 hits and the LLM consumer either gets empty subgraph or improvises an ungrounded keyword list from training memory (e.g. expanding "auth" to{passport, sso, saml, oauth, jwt, scim, …}regardless of whether those tokens exist in the corpus).Fix
Adds a
Step 0 — Constrained query expansionblock to the skill's/graphify querysection. The LLM consumer extracts vocabulary from graph labels (CamelCase/snake_case split, length-filtered) and is instructed to pick only tokens present in that vocabulary, explicitly forbidden from inventing terms.Effects:
save-resultfor the next --update to graph as Q&A nodes.Scope
Patches the canonical
graphify/skill.md. The 11 host-variant skills (skill-codex.md, skill-aider.md, …) follow the same query-section contract but inline Python rather than callinggraphify queryCLI; those need a parallel patch with the inline form. Happy to follow up in a separate PR after review on the canonical patch.Test
On a graph built from the graphify repo itself (1284 nodes, 1454 vocab tokens), an unconstrained expansion of "укрупненная архитектура аутентификации" yields {auth, oauth, jwt, saml, sso, ldap, scim, mfa, 2fa, pin, passport, session, login, token} — of which 11/15 are absent from the corpus. Constrained expansion against the actual vocab yields {credential, security, token, signature, user, architecture, component, module, overview} — 9 tokens, 0 fantom. Same retrieval, dramatically higher precision.