Adding dynamic retrieval pipeline: 3rd attempt

[backurs]

This PR implements a pipeline where we dynamically decide when to call search tool or prune tool, or get document tool. The search tool executes vector searches (over different sources) and fulltext searches (over different sources). It retrieves more documents than we can put in the context, so we use a reranker to filter them down to a reasonable count. The prune tool prunes the list of documents retrieved so far that are most relevant to the query. The get document tool obtains the full document content given a document id (search tool retrieves only snippets of documents).

The pipeline does prune calls, search calls and get documents tool calls in an arbitrary order (as decided by the llm) until the llm decides that there is enough information to answer the question.

[Copilot]

Pull request overview

Adds a standalone “dynamic retriever” script that lets an LLM orchestrate retrieval by dynamically calling search, get_document, and prune tools over Cosmos DB sources, with optional semantic reranking. Includes an example YAML config to run the pipeline.

Changes:

• Introduces dynamic_retriever.py implementing tool-driven search/prune/get-doc loops, vector + fulltext queries, and optional ranker reranking.
• Adds config_dynamic.yaml.example documenting required configuration for LLM, embeddings, ranker, and Cosmos sources.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 20 comments.

File	Description
dynamic_retriever.py	Implements the dynamic tool-calling retrieval pipeline over Cosmos DB + optional ranker.
config_dynamic.yaml.example	Provides an example configuration for running the dynamic retriever script.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.