Skip to content

feat(daemon): POST /sync endpoint for dashboard Sync Now button#196

Merged
Gradata merged 1 commit into
mainfrom
feat/daemon-sync-endpoint
May 15, 2026
Merged

feat(daemon): POST /sync endpoint for dashboard Sync Now button#196
Gradata merged 1 commit into
mainfrom
feat/daemon-sync-endpoint

Conversation

@Gradata

@Gradata Gradata commented May 15, 2026

Copy link
Copy Markdown
Owner

Adds POST /sync to the local daemon so the cloud dashboard's Sync Now button works.

Bug

Dashboard SyncStatus.tsx POSTs to http://127.0.0.1:8765/sync but the daemon only had /health and per-feature endpoints. Button always failed with 'Local Gradata daemon not running' banner.

Change

  • New POST /sync handler runs the cron-sync logic inline: reads system.db events past watermark, POSTs to api.gradata.ai/api/v1/sync, advances watermark.
  • CORS headers on /sync and OPTIONS preflight so https://app.gradata.ai can call into 127.0.0.1.
  • Returns {status, pushed, last_sync_at} on success, matching what the dashboard expects.

Test plan

tests/test_daemon_sync.py: green.
Manual: daemon running, curl -X POST http://127.0.0.1:8765/sync returns pushed count.

Layering

No Layer 0 -> 2 imports. daemon.py already uses _brain_lock, sqlite3 stdlib, urllib.request stdlib.

Adds POST /sync to the local daemon so the cloud dashboard's Sync Now
button (SyncStatus.tsx) works. The dashboard POSTs to
http://127.0.0.1:8765/sync but the daemon previously only had /health
and per-feature endpoints, so the button always failed with
'Local Gradata daemon not running' banner.

The new handler mirrors sync_cron.py exactly:
- Reads watermark from $BRAIN_DIR/.sync_cron_watermark
- SELECT * FROM events WHERE id > watermark ORDER BY id LIMIT 500
- Builds (events, corrections) payload with same severity normalization
  and dedup-by-(session, description) as the cron pusher
- POSTs to GRADATA_CLOUD_API_URL (default api.gradata.ai/api/v1/sync)
  with Bearer auth
- API key resolution: --api-key flag > GRADATA_API_KEY env > ~/.gradata/key
- Atomic watermark write on success
- Returns {status, pushed, last_sync_at} on success
- 502 on cloud HTTP/network failure (watermark NOT advanced)
- 500 on unexpected exceptions (logged with exc_info)

CORS: dashboard at https://app.gradata.ai is cross-origin to 127.0.0.1.
Added Access-Control-Allow-Origin to all responses and an OPTIONS
preflight handler returning 204 with the allow-methods/-headers.

Tests: tests/test_daemon_sync.py covers happy path, no-new-events,
HTTP error, network error, missing API key, OPTIONS preflight, CORS
header on POST, plus direct unit tests for payload dedup and key
resolution priority.

Layering check: daemon.py stays in Layer 2. New helpers use stdlib
(sqlite3, urllib, json, os, pathlib) — no Layer 0 -> 2 imports
introduced.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

@coderabbitai

coderabbitai Bot commented May 15, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough
  • New POST /sync endpoint on local daemon (http://127.0.0.1:8765/sync) enables dashboard "Sync Now" button to trigger syncs
  • Event sync logic reads events from system.db past a watermark, posts them to api.gradata.ai/api/v1/sync with event deduplication by (session, description), and advances watermark atomically on success
  • CORS support added with OPTIONS preflight handler and CORS headers for cross-origin requests from https://app.gradata.ai
  • API key resolution implements priority order: --api-key CLI flag > GRADATA_API_KEY env var > ~/.gradata/key file
  • Error handling returns HTTP 502 for cloud failures (without advancing watermark) and HTTP 500 for unexpected exceptions; includes detailed logging
  • New GradataDaemon parameter api_key: str | None = None added to __init__ (optional, backward compatible)
  • Response format returns {status, pushed, last_sync_at} JSON matching dashboard expectations
  • Comprehensive test coverage with 406 lines of tests covering happy path, edge cases (no new events), cloud errors, missing credentials, CORS contract, payload deduplication, and API key priority resolution
  • No layering violations — all changes use stdlib modules and existing locks, no new Layer 0→2 imports

Walkthrough

Adds a cloud synchronization endpoint (POST /sync) to the Gradata daemon that pushes newly created events to Gradata Cloud past a stored watermark. Implements cloud request helpers (API key resolution, event/correction payload construction with deduplication), HTTP endpoint wiring with CORS support, CLI integration for the API key, and comprehensive integration and unit tests.

Changes

Cloud Sync Endpoint

Layer / File(s) Summary
Cloud sync payload construction and helpers
Gradata/src/gradata/daemon.py
Imports urllib.error and urllib.request for HTTP operations. Defines three reusable helpers: _resolve_api_key (checks explicit arg → env var → ~/.gradata/key); _cloud_post (POSTs to sync endpoint with robust error handling); _build_sync_payload (transforms SQLite event rows into events and deduplicated corrections with severity normalization and (session, description) deduplication rules).
HTTP endpoint, CORS, and route wiring
Gradata/src/gradata/daemon.py
Documents the new POST /sync endpoint. Implements do_OPTIONS for CORS preflight (204 + allow-methods/max-age headers). Adds _write_cors_headers() and updates _send_json() to emit CORS headers on JSON responses. Routes /sync to _handle_sync(), which orchestrates the sync workflow: reads/validates watermark file, queries SQLite events past watermark, builds cloud payload, resolves API key, posts to cloud with error handling (returns 502 without advancing watermark on failure), atomically updates watermark on success, returns structured JSON status or error responses.
CLI parameter and daemon initialization
Gradata/src/gradata/daemon.py
Extends GradataDaemon.__init__ signature with optional api_key parameter; stores on instance as self._api_key. Adds --api-key CLI flag with help text. Passes parsed CLI flag value to daemon constructor.
Integration and unit tests
Gradata/tests/test_daemon_sync.py
Test fixture initializes temporary SQLite brain with pre-seeded events and starts daemon on ephemeral port. HTTP helper _post() sends JSON POST and returns (status_code, parsed_json). Happy-path test mocks cloud call, asserts HTTP 200 with status=ok, verifies payload contents (events/corrections/deduplication), confirms watermark advancement. No-events test confirms pushed=0 and no cloud call. Error-path tests (cloud HTTP failure, network/DNS failure) assert HTTP 502 without watermark advancement. Missing-API-key test confirms HTTP 502 with error message. CORS contract tests validate OPTIONS /sync returns 204 with headers and POST /sync includes Access-Control-Allow-Origin. Unit tests for _build_sync_payload verify severity normalization and deduplication by (session, description). Unit tests for _resolve_api_key verify resolution priority and None fallback.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • Gradata/gradata#162: Main PR adds the daemon-side /sync HTTP endpoint, watermark handling, and API-key-based cloud request payload construction, while retrieved PR implements the cloud client/client sync flow that posts batched watermarked events to that same /sync contract (plus backfill); together they directly align on the sync/watermark mechanism.

Suggested labels

feature, breaking-change

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 65.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: adding a POST /sync endpoint to the daemon for the dashboard's Sync Now button functionality.
Description check ✅ Passed The description is directly related to the changeset, explaining the bug, the solution, test plan, and layering considerations for the new /sync endpoint.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/daemon-sync-endpoint

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 OpenGrep (1.20.0)

OpenGrep fatal error (exit code 2):
┌──────────────┐
│ Opengrep CLI │
└──────────────┘

�[32m✔�[39m �[1mOpengrep OSS�[0m
�[32m✔�[39m Basic security coverage for first-party code vulnerabilities.

�[1m Loading rules from local config...�[0m
[00.19][ERROR]: Error: exception Glob.Lexer.Syntax_error("malformed glob pattern: missing ']'")
Raised at Glob__Lexer.syntax_error in file "libs/glob/Lexer.mll", line 8, characters 2-26
Called from Glob__Lexer.__ocaml_lex_token_rec in file "libs/glob/Lexer.mll", line 29, characters 26-53
Cal


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Gradata/src/gradata/daemon.py`:
- Around line 299-302: Remove the unnecessary credential header from the CORS
helper: in the _write_cors_headers method remove the call that sets
"Access-Control-Allow-Credentials" so only "Access-Control-Allow-Origin" (and
any other non-credential CORS headers) are sent; this prevents sending
Access-Control-Allow-Credentials: true alongside Access-Control-Allow-Origin:
"*" which violates the CORS spec for the /sync endpoint that uses server-side
API key auth.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 058d3c77-645d-4ccd-947c-224486793824

📥 Commits

Reviewing files that changed from the base of the PR and between 7000062 and 1229140.

📒 Files selected for processing (2)
  • Gradata/src/gradata/daemon.py
  • Gradata/tests/test_daemon_sync.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: pytest windows-latest / py3.11
  • GitHub Check: pytest macos-latest / py3.11
  • GitHub Check: pytest ubuntu-latest / py3.12
  • GitHub Check: pytest ubuntu-latest / py3.11
  • GitHub Check: pytest macos-latest / py3.12
  • GitHub Check: pytest windows-latest / py3.12
  • GitHub Check: pytest (py3.12)
🧰 Additional context used
📓 Path-based instructions (2)
Gradata/src/**/*.py

📄 CodeRabbit inference engine (Gradata/AGENTS.md)

Gradata/src/**/*.py: Prefer sentence-transformers for local embeddings, google-genai for Gemini embeddings, cryptography for AES-GCM encrypted system.db, bm25s for BM25 rule ranking, and mem0ai for external memory adapters — guard all optional dependency imports with try / except ImportError at the call site, never at module level
Maintain strict layering: Layer 0 (Primitives: _types.py, _db.py, _events.py, _paths.py, _file_lock.py; Patterns: contrib/patterns/) must never import from Layer 1 (Enhancements: enhancements/, rules/) or Layer 2 (Public API: brain.py, cli.py, daemon.py, mcp_server.py)
Never use bare except: pass — use typed exceptions or at minimum logger.warning(...) with exc_info=True to avoid silent failure in a memory product
Never import from out-of-scope sibling directories ../Sprites/ or ../Hausgem/ within gradata/* code — that is a layering bug
Never leak private-sibling paths into public docs/code — no references to ../Sprites/, ../Hausgem/, email addresses, OneDrive paths, or Sprites-specific examples from inside gradata/*
Use atomic-write helper when writing JSON files to prevent corruption from mid-write crashes

Files:

  • Gradata/src/gradata/daemon.py
Gradata/tests/**/*.py

📄 CodeRabbit inference engine (Gradata/AGENTS.md)

Gradata/tests/**/*.py: Set BRAIN_DIR environment variable via tmp_path in conftest.py for test isolation — ensure _paths.py module cache refreshes when calling Brain.init() directly inside tests
Add unit tests in tests/test_*.py for every CI push without LLM calls (deterministic); mark integration tests with @pytest.mark.integration and skip them by default (they hit real LLM APIs)

Files:

  • Gradata/tests/test_daemon_sync.py
🔇 Additional comments (21)
Gradata/src/gradata/daemon.py (10)

38-39: LGTM!


109-116: LGTM!


118-133: LGTM!


136-153: LGTM!


156-222: LGTM!


253-264: LGTM!


278-278: LGTM!

Also applies to: 309-309


777-905: LGTM!


925-941: LGTM!


1236-1253: LGTM!

Gradata/tests/test_daemon_sync.py (11)

1-30: LGTM!


35-109: LGTM!


112-125: LGTM!


131-168: LGTM!


171-204: LGTM!


210-231: LGTM!


234-247: LGTM!


250-291: LGTM!


297-321: LGTM!


327-382: LGTM!


385-406: LGTM!

Comment on lines +299 to +302
def _write_cors_headers(self) -> None:
"""Write CORS response headers (called between send_response and end_headers)."""
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Credentials", "true")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | 💤 Low value

Remove unnecessary Access-Control-Allow-Credentials header.

Since the /sync endpoint authenticates via a server-side API key (not browser credentials), Access-Control-Allow-Credentials: true is unnecessary. More importantly, per CORS spec, when this header is set to true, the Access-Control-Allow-Origin header must specify an explicit origin—not *. Browsers will reject credentialed requests if both are present as shown.

Since no browser credentials are used, simply remove the credentials header:

Proposed fix
     def _write_cors_headers(self) -> None:
         """Write CORS response headers (called between send_response and end_headers)."""
         self.send_header("Access-Control-Allow-Origin", "*")
-        self.send_header("Access-Control-Allow-Credentials", "true")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def _write_cors_headers(self) -> None:
"""Write CORS response headers (called between send_response and end_headers)."""
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Credentials", "true")
def _write_cors_headers(self) -> None:
"""Write CORS response headers (called between send_response and end_headers)."""
self.send_header("Access-Control-Allow-Origin", "*")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Gradata/src/gradata/daemon.py` around lines 299 - 302, Remove the unnecessary
credential header from the CORS helper: in the _write_cors_headers method remove
the call that sets "Access-Control-Allow-Credentials" so only
"Access-Control-Allow-Origin" (and any other non-credential CORS headers) are
sent; this prevents sending Access-Control-Allow-Credentials: true alongside
Access-Control-Allow-Origin: "*" which violates the CORS spec for the /sync
endpoint that uses server-side API key auth.

@Gradata Gradata merged commit 2a00706 into main May 15, 2026
9 checks passed
@Gradata Gradata deleted the feat/daemon-sync-endpoint branch May 15, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant