feat: introduce PageIndex SDK with Collection-based local/cloud API

.github/workflows/publish.yml

@@ -0,0 +1,63 @@
+name: Publish to PyPI
+
+# Release flow (the git tag IS the version — nothing to bump in the repo):
+#   1. git tag -a v0.3.0.dev2 -m "Release 0.3.0.dev2"
+#   2. git push origin v0.3.0.dev2
+#   3. This workflow derives the version from the tag, injects it into
+#      pyproject.toml, builds, publishes to PyPI via OIDC trusted publishing
+#      (no stored secret), and creates a GitHub Release with generated notes.
+#
+# The tag must be a PEP 440 version with a leading `v`:
+#   v0.3.0   v0.3.0rc1   v0.3.0.dev2
+# PyPI rejects duplicate version uploads, so each tag must be a new version.
+# `pip install pageindex` skips dev/pre releases — install one with
+# `pip install pageindex==0.3.0.dev2` or `pip install --pre pageindex`.
+#
+# One-time setup this workflow depends on:
+#   - PyPI: add a Trusted Publisher on the `pageindex` project pointing at
+#     repo VectifyAI/PageIndex, workflow `publish.yml`, environment `pypi`.
+#   - GitHub: create an Environment named `pypi` (Settings -> Environments).
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # OIDC trusted publishing to PyPI
+      contents: write   # create the GitHub Release
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7.0.0
+
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1  # v6.3.0
+        with:
+          python-version: "3.12"
+
+      - name: Set version from tag and build
+        run: |
+          set -euo pipefail
+          python -m pip install --upgrade build packaging
+          VERSION="${GITHUB_REF_NAME#v}"
+          echo "Publishing version: $VERSION"
+          # Fail early on a malformed tag instead of publishing a junk version.
+          python -c "from packaging.version import Version; Version('$VERSION')"
+          # The git tag is the single source of truth; overwrite the static
+          # placeholder in [tool.poetry] so the built artifacts carry $VERSION.
+          sed -i "s/^version = .*/version = \"$VERSION\"/" pyproject.toml
+          grep '^version = ' pyproject.toml
+          python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b  # release/v1.14.0
+
+      - name: Create GitHub Release
+        uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b  # v3.0.1
+        with:
+          tag_name: ${{ github.ref_name }}
+          name: ${{ github.ref_name }}
+          generate_release_notes: true
+          files: dist/*

.gitignore

@@ -4,3 +4,12 @@ __pycache__
 .env*
 .venv/
 logs/
+pageindex.egg-info/
+dist/
+*.db
+venv/
+uv.lock
+
+# local SDK test-run artifacts (generated by demos)
+examples/workspace/files/
+examples/workspace/*.db

README.md

@@ -143,6 +143,66 @@ You can generate PageIndex tree structures with this open-source repo. Or use ou
 
 ---
 
+# 🚀 SDK Usage
+
+A unified `PageIndexClient` powers both local self-hosted and cloud-managed modes. Mode is auto-detected by whether you pass an `api_key`.
+
+### Install
+
+```bash
+pip install pageindex
+```
+
+### Quick start
+
+```python
+from pageindex import PageIndexClient
+
+# Local mode — uses your LLM key (e.g. OPENAI_API_KEY in env).
+# `model` drives indexing; agent QA uses `retrieve_model` (default: gpt-5.4).
+client = PageIndexClient(model="gpt-4o-2024-11-20")
+
+collection = client.collection()
+doc_id = collection.add("path/to/your.pdf")
+
+print(collection.query("What is the main contribution?", doc_ids=doc_id))
+
+# Cloud mode — fully managed, no LLM key needed:
+# client = PageIndexClient(api_key="your-pageindex-api-key")
+```
+
+`collection.query(...)` returns the answer string by default. Always pass `doc_ids` for reliable single-document QA — omitting it queries the entire collection, which is experimental (see below).
+
+### Streaming queries
+
+```python
+import asyncio
+
+async def main():
+    async for ev in collection.query("Explain multi-head attention", doc_ids=doc_id, stream=True):
+        if ev.type == "text_delta":
+            print(ev.data, end="", flush=True)
+        elif ev.type == "tool_call":
+            print(f"\n[tool] {ev.data['name']}")
+
+asyncio.run(main())
+```
+
+`ev.type` is one of: `tool_call`, `tool_result`, `text_delta`, `text_done`. A `text_done` fires each time a text message completes — a local agentic query may emit several as the agent narrates between tool calls; the last `text_done` before the stream ends carries the final answer.
+
+### Multi-document collections (experimental)
+
+Passing `doc_ids` scopes the query to a specific subset of documents — this is the recommended path. `doc_ids` accepts a single id (`str`) or a list:
+
+```python
+collection.query("What does this paper say?", doc_ids=doc1)            # single
+collection.query("Compare these two papers", doc_ids=[doc1, doc2])     # multi
+```
+
+Omitting `doc_ids` queries the **entire collection** and lets the agent pick which docs to read. This is an **experimental** feature with a naive first implementation — we're actively working on better cross-document retrieval. A `UserWarning` is emitted; set `PAGEINDEX_EXPERIMENTAL_MULTIDOC=1` to silence it.
+
+---
+
 # ⚙️ Package Usage
 
 > **Note:** This package uses standard PDF parsing. For use cases with complex PDFs, our [cloud service](https://pageindex.ai/developer) (via MCP and API) offers enhanced OCR, tree building, and retrieval.
@@ -181,8 +241,10 @@ You can customize the processing with additional optional arguments:
 --max-tokens-per-node   Max tokens per node (default: 20000)
 --if-add-node-id        Add node ID (yes/no, default: yes)
 --if-add-node-summary   Add node summary (yes/no, default: yes)
---if-add-doc-description Add doc description (yes/no, default: yes)
+--if-add-doc-description Add doc description (yes/no, default: no)
+--if-add-node-text      Add raw text to nodes (yes/no, default: no)
 ```
+A bare flag is shorthand for `yes` (e.g. `--if-add-node-id` turns the option on).
 </details>
 
 <details>

examples/agentic_vectorless_rag_demo.py

@@ -32,8 +32,7 @@
 from agents.stream_events import RawResponsesStreamEvent, RunItemStreamEvent
 from openai.types.responses import ResponseTextDeltaEvent, ResponseReasoningSummaryTextDeltaEvent
 
-from pageindex import PageIndexClient
-import pageindex.utils as utils
+from pageindex import LocalClient
 
 PDF_URL = "https://arxiv.org/pdf/2603.15031"
 
@@ -52,7 +51,17 @@
 """
 
 
-def query_agent(client: PageIndexClient, doc_id: str, prompt: str, verbose: bool = False) -> str:
+def _normalize_model_for_agents_sdk(model: str) -> str:
+    """The OpenAI Agents SDK only recognizes 'openai/' and 'litellm/' model
+    prefixes; route any other LiteLLM-style provider path (e.g. 'anthropic/...')
+    through litellm explicitly, mirroring what PageIndex itself does internally
+    for its built-in agent."""
+    if model and "/" in model and not model.startswith(("litellm/", "openai/")):
+        return f"litellm/{model}"
+    return model
+
+
+def query_agent(collection, doc_id: str, prompt: str, model: str, verbose: bool = False) -> str:
     """Run a document QA agent using the OpenAI Agents SDK.
 
     Streams text output token-by-token and returns the full answer string.
@@ -62,12 +71,14 @@ def query_agent(client: PageIndexClient, doc_id: str, prompt: str, verbose: bool
     @function_tool
     def get_document() -> str:
         """Get document metadata: status, page count, name, and description."""
-        return client.get_document(doc_id)
+        doc = collection.get_document(doc_id)
+        doc.pop("structure", None)  # keep tool output small for the LLM context
+        return json.dumps(doc, ensure_ascii=False)
 
     @function_tool
     def get_document_structure() -> str:
         """Get the document's full tree structure (without text) to find relevant sections."""
-        return client.get_document_structure(doc_id)
+        return json.dumps(collection.get_document_structure(doc_id), ensure_ascii=False)
 
     @function_tool
     def get_page_content(pages: str) -> str:
@@ -76,13 +87,13 @@ def get_page_content(pages: str) -> str:
         Use tight ranges: e.g. '5-7' for pages 5 to 7, '3,8' for pages 3 and 8, '12' for page 12.
         For Markdown documents, use line numbers from the structure's line_num field.
         """
-        return client.get_page_content(doc_id, pages)
+        return json.dumps(collection.get_page_content(doc_id, pages), ensure_ascii=False)
 
     agent = Agent(
         name="PageIndex",
         instructions=AGENT_SYSTEM_PROMPT,
         tools=[get_document, get_document_structure, get_page_content],
-        model=client.retrieve_model,
+        model=_normalize_model_for_agents_sdk(model),
         # model_settings=ModelSettings(reasoning={"effort": "low", "summary": "auto"}),  # Uncomment to enable reasoning
     )
 
@@ -130,10 +141,10 @@ async def _run():
 
     try:
         asyncio.get_running_loop()
-        with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
-            return pool.submit(asyncio.run, _run()).result()
     except RuntimeError:
         return asyncio.run(_run())
+    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
+        return pool.submit(asyncio.run, _run()).result()
 
 
 if __name__ == "__main__":
@@ -152,37 +163,33 @@ async def _run():
                         f.write(chunk)
         print("Download complete.\n")
 
-    # Setup
-    client = PageIndexClient(workspace=WORKSPACE)
+    # Setup: self-hosted local client + a collection
+    client = LocalClient(storage_path=str(WORKSPACE))
+    collection = client.collection("agentic-demo")
 
     # Step 1: Index PDF and view tree structure
     print("=" * 60)
     print("Step 1: Index PDF and view tree structure")
     print("=" * 60)
-    doc_id = next(
-        (did for did, doc in client.documents.items() if doc.get('doc_name') == PDF_PATH.name),
-        None,
-    )
-    if doc_id:
-        print(f"\nLoaded cached doc_id: {doc_id}")
-    else:
-        doc_id = client.index(PDF_PATH)
-        print(f"\nIndexed. doc_id: {doc_id}")
+    # Content-hash dedup: re-running reuses the existing doc_id, no re-index.
+    doc_id = collection.add(str(PDF_PATH))
+    print(f"\ndoc_id: {doc_id}")
     print("\nTree Structure (top-level sections):")
-    structure = json.loads(client.get_document_structure(doc_id))
-    utils.print_tree(structure)
+    for node in collection.get_document_structure(doc_id):
+        print(f"  - {node.get('title', '(untitled)')}")
 
     # Step 2: View document metadata
     print("\n" + "=" * 60)
     print("Step 2: View document metadata")
     print("=" * 60)
-    doc_metadata = client.get_document(doc_id)
-    print(f"\n{doc_metadata}")
+    meta = collection.get_document(doc_id)
+    meta.pop("structure", None)
+    print("\n" + json.dumps(meta, ensure_ascii=False, indent=2))
 
     # Step 3: Agent Query
     print("\n" + "=" * 60)
     print("Step 3: Agent Query (auto tool-use)")
     print("=" * 60)
     question = "Explain Attention Residuals in simple language."
     print(f"\nQuestion: '{question}'")
-    query_agent(client, doc_id, question, verbose=True)
+    query_agent(collection, doc_id, question, client.retrieve_model, verbose=True)

examples/cloud_demo.py

@@ -0,0 +1,67 @@
+"""
+Agentic Vectorless RAG with PageIndex SDK - Cloud Demo
+
+Uses CloudClient for fully-managed document indexing and QA.
+No LLM API key needed — the cloud service handles everything.
+
+Steps:
+  1 — Upload and index a PDF via PageIndex cloud
+  2 — Stream a question with tool call visibility
+
+Requirements:
+    pip install pageindex
+    export PAGEINDEX_API_KEY=your-api-key
+"""
+import asyncio
+import os
+import sys
+from pathlib import Path
+import requests
+from pageindex import CloudClient
+
+_EXAMPLES_DIR = Path(__file__).parent
+PDF_URL = "https://arxiv.org/pdf/2603.15031"
+PDF_PATH = _EXAMPLES_DIR / "documents" / "attention-residuals.pdf"
+
+api_key = os.environ.get("PAGEINDEX_API_KEY")
+if not api_key:
+    sys.exit("PAGEINDEX_API_KEY not set — get a key at https://dash.pageindex.ai")
+
+# Download PDF if needed
+if not PDF_PATH.exists():
+    print(f"Downloading {PDF_URL} ...")
+    PDF_PATH.parent.mkdir(parents=True, exist_ok=True)
+    with requests.get(PDF_URL, stream=True, timeout=30) as r:
+        r.raise_for_status()
+        with open(PDF_PATH, "wb") as f:
+            for chunk in r.iter_content(chunk_size=8192):
+                if chunk:
+                    f.write(chunk)
+    print("Download complete.\n")
+
+client = CloudClient(api_key=api_key)
+collection = client.collection()
+
+doc_id = collection.add(str(PDF_PATH))
+print(f"Indexed: {doc_id}\n")
+
+# Streaming query
+stream = collection.query("What is the main contribution of this paper?", stream=True)
+
+async def main():
+    streamed_text = False
+    async for event in stream:
+        if event.type == "text_delta":
+            print(event.data, end="", flush=True)
+            streamed_text = True
+        elif event.type == "tool_call":
+            if streamed_text:
+                print()
+                streamed_text = False
+            args = event.data.get("args", "")
+            print(f"[tool call] {event.data['name']}({args})")
+        elif event.type == "text_done":
+            print()
+            streamed_text = False
+
+asyncio.run(main())