Skip to content

sync-process.md

Latest commit

 

History

History
327 lines (255 loc) · 9.98 KB

sync-process.md

File metadata and controls

327 lines (255 loc) · 9.98 KB

Granola Sync Process Documentation

This document explains how the Granola Sync plugin synchronizes notes and transcripts from Granola to Obsidian.

Overview

The sync process retrieves documents and transcripts from the Granola API and saves them to the Obsidian vault. The plugin supports multiple sync destinations and handles deduplication through Granola ID tracking.

High-Level Sync Flow

flowchart TD
    A[Sync Triggered] --> B[Load Credentials]
    B --> C{Credentials Loaded?}
    C -->|No| D[Show Error Notice]
    C -->|Yes| E[Build Granola ID Cache]
    E --> F[Fetch Documents from API]
    F --> G{Documents Found?}
    G -->|No| H[End Sync]
    G -->|Yes| I{Transcripts Enabled?}
    I -->|Yes| J[Sync Transcripts]
    I -->|No| K[Skip Transcripts]
    J --> L{Notes Enabled?}
    K --> L
    L -->|Yes| M[Sync Notes]
    L -->|No| H
    M --> O[Update Last Sync Time]
    O --> P[Update Status Bar]
    P --> H
Loading

Credentials Loading

The plugin loads credentials by reading directly from the filesystem. This approach provides secure access to credentials stored in the Granola application's data directory without requiring a temporary HTTP server.

Credentials Loading Flow

sequenceDiagram
    participant Plugin
    participant CredentialsService
    participant FileSystem
    participant GranolaApp

    Plugin->>CredentialsService: loadCredentials()
    CredentialsService->>FileSystem: Read {OS-specific}/supabase.json
    FileSystem-->>CredentialsService: JSON Content or Error
    
    alt File Read Success
        CredentialsService->>CredentialsService: Parse JSON
        CredentialsService->>CredentialsService: Extract workos_tokens.access_token
        CredentialsService->>CredentialsService: Check Token Expiration
        alt Token Expired
            CredentialsService->>GranolaApp: Refresh Token via API
            GranolaApp-->>CredentialsService: New Access Token
        end
        CredentialsService-->>Plugin: { accessToken, error: null }
    else File Read Error
        CredentialsService->>CredentialsService: Generate Error Message
        CredentialsService-->>Plugin: { accessToken: null, error }
    end
Loading

Credentials Service Details

  • File Location: {OS-specific}/supabase.json
    • Windows: %APPDATA%/Granola/supabase.json
    • Linux: ~/.config/Granola/supabase.json
    • macOS: ~/Library/Application Support/Granola/supabase.json
  • Token Path: workos_tokens.access_token within the JSON structure
  • Token Refresh: Automatically refreshes expired tokens using the refresh token
  • Error Handling: Provides clear error messages for:
    • File not found (ENOENT)
    • Permission errors (EACCES)
    • Invalid JSON format
    • Missing required fields

Document Fetching

The plugin fetches documents from the Granola API using the loaded access token.

API Request Flow

sequenceDiagram
    participant Plugin
    participant GranolaAPI
    participant GranolaServer

    Plugin->>GranolaAPI: fetchGranolaDocuments(accessToken)
    GranolaAPI->>GranolaServer: POST /v2/get-documents
    Note over GranolaAPI: Headers: Authorization Bearer token<br/>Body: {limit: 100, offset: 0,<br/>include_last_viewed_panel: true}
    GranolaServer-->>GranolaAPI: {docs: GranolaDoc[]}
    GranolaAPI->>GranolaAPI: Validate Response Structure
    GranolaAPI-->>Plugin: GranolaDoc[]
Loading

Document Structure

Each GranolaDoc contains:

  • id: Unique Granola document identifier
  • title: Document title
  • created_at: Creation timestamp (optional)
  • updated_at: Last update timestamp (optional)
  • last_viewed_panel.content: ProseMirror document structure (optional)
  • notes_markdown: Raw private notes in markdown format (optional)

Error Handling

The plugin handles various HTTP error codes:

  • 401: Authentication failed (token expired)
  • 403: Access forbidden
  • 404: API endpoint not found
  • 500+: Server errors
  • Other: Network or connection errors

Granola ID Cache

Before syncing, the plugin builds a cache mapping Granola IDs to existing Obsidian files. This enables efficient deduplication.

Cache Building Process

flowchart LR
    A[Scan All Markdown Files] --> B[Read Frontmatter]
    B --> C{Has granola_id?}
    C -->|Yes| D[Add to Cache Map]
    C -->|No| E[Skip File]
    D --> F[Next File]
    E --> F
    F --> G{More Files?}
    G -->|Yes| B
    G -->|No| H[Cache Ready]
Loading

Cache Structure: Map<cacheKey, TFile>

  • Key: Composite key in format {granolaId}-{type} where type is 'note' or 'transcript'
  • Value: Obsidian TFile reference
  • This allows notes and transcripts with the same Granola ID to coexist without conflicts

Frontmatter Structure

All synced files include frontmatter with metadata for tracking and identification.

Note Frontmatter

---
granola_id: doc-123
title: "Meeting Title"
type: note
created: 2024-01-15T10:00:00Z
updated: 2024-01-15T12:00:00Z
---

Transcript Frontmatter

---
granola_id: doc-123
title: "Meeting Title - Transcript"
type: transcript
created: 2024-01-15T10:00:00Z
updated: 2024-01-15T12:00:00Z
---

Key Features:

  • granola_id: Consistent across both note and transcript for the same source document
  • type: Distinguishes between 'note' and 'transcript' files
  • created and updated: Timestamps from Granola API (when available)
  • Both file types can share the same granola_id while being uniquely identified by type

Legacy Frontmatter Migration

The plugin automatically migrates legacy frontmatter formats on load:

  • Removes -transcript suffix from granola_id in transcript files
  • Adds type field to all files (note/transcript)
  • Adds missing timestamps to transcript files (when available)
  • Migration runs silently in the background
  • Will be removed in version 2.0.0

Note Syncing

Notes can be synced to three different destinations, each with different behavior:

Note Sync Destinations

flowchart TD
    A[Sync Notes] --> B{Destination Type?}
    B -->|Daily Notes| C[Group by Date]
    B -->|Daily Note Folder Structure| D[Individual Files]
    B -->|Granola Folder| D

    C --> E[Create/Update Daily Note]
    E --> F[Update Section with Heading]

    D --> G[For Each Document]
    G --> H[Convert ProseMirror to Markdown]
    H --> H1{Include Private Notes?}
    H1 -->|Yes & Has Content| H2[Add Private Notes + Enhanced Notes Sections]
    H1 -->|No or Empty| H3[Add Note Content Only]
    H2 --> I[Add Frontmatter]
    H3 --> I
    I --> J[Save to Disk]
Loading

Daily Notes Destination

When syncing to daily notes, documents are grouped by date and inserted into sections within daily note files.

flowchart TD
    A[Documents List] --> B[For Each Document]
    B --> C[Extract Date from document timestamps]
    C --> D[Group by Date YYYY-MM-DD]
    D --> E[Convert ProseMirror to Markdown]
    E --> E1{Include Private Notes?}
    E1 -->|Yes & Has Content| E2[Add Private Notes Section]
    E1 -->|No or Empty| E3[Skip Private Notes]
    E2 --> F[Add to Daily Notes Map]
    E3 --> F
    F --> G{More Documents?}
    G -->|Yes| B
    G -->|No| H[For Each Date Group]
    H --> I[Get/Create Daily Note File]
    I --> J[Build Section Content]
    J --> K[Update Section with Heading]
    K --> L{More Dates?}
    L -->|Yes| H
    L -->|No| M[Complete]
Loading

Section Format:

  • Heading: User-configured section heading (trimmed)
  • Each note includes:
    • H3 title
    • Granola ID
    • Created/Updated timestamps
    • Optional transcript link
    • Note content (with optional Private Notes and Enhanced Notes sections if enabled)

Private Notes Support:

When the "Include Private Notes" setting is enabled and notes_markdown has content:

  • Notes include a "## Private Notes" section at the top with the raw private notes
  • Followed by a "## Enhanced Notes" section with the processed note content
  • When disabled or notes_markdown is empty, notes display content directly without section headings

File Saving and Deduplication

The saveToDisk method handles file creation, updates, and deduplication.

Save to Disk Flow

flowchart TD
    A[saveToDisk Called] --> B[Determine Folder Path]
    B --> C{File Type?}
    C -->|Transcript| D[Use Transcript Destination]
    C -->|Note| E[Use Note Destination]
    D --> F[Ensure Folder Exists]
    E --> F
    F --> G{Has granolaId?}
    G -->|Yes| H[Search Cache for Existing File]
    G -->|No| I[Search by File Path]
    H --> J{File Found?}
    I --> J
    J -->|Yes| K[Update Existing File]
    J -->|No| L[Create New File]
    K --> M{Path Changed?}
    M -->|Yes| N[Attempt Rename]
    M -->|No| O[Update Cache]
    N --> O
    L --> O
    O --> P[Return Success]
Loading

Deduplication Strategy

  1. Primary: Search by Granola ID in cache
  2. Fallback: Search by computed file path
  3. Update: If file exists, update content
  4. Rename: If path changed (e.g., title changed), rename file
  5. Cache Update: Always update cache after save

File Path Computation

Path computation depends on the destination type:

Daily Note Folder Structure:

  • Uses daily note format settings
  • Extracts folder structure from date format
  • Combines with base daily notes folder

Granola Folder / Granola Transcripts Folder:

  • Uses configured folder path directly
  • Normalizes path separators

Filename Sanitization:

  • Removes invalid characters: <>:"/\|?*
  • Replaces spaces with underscores
  • Truncates to 200 characters maximum

Summary

The sync process orchestrates multiple components:

  1. Credentials Management: Direct filesystem reading with automatic token refresh
  2. API Communication: Fetching documents and transcripts from Granola API
  3. Deduplication: Using Granola ID cache to prevent duplicates
  4. Content Conversion: ProseMirror to Markdown transformation
  5. File Management: Creating/updating files in appropriate locations
  6. Error Handling: Comprehensive error reporting and recovery

The plugin supports flexible sync destinations and maintains data integrity through ID-based deduplication and careful file management.