Skip to content

Latest commit

 

History

History
119 lines (91 loc) · 3.25 KB

README.md

File metadata and controls

119 lines (91 loc) · 3.25 KB

synq-steep

ETL pipeline that syncs Steep analytics data (metrics, entities, modules) to SYNQ data catalog as custom entities.

Installation

# Using uv
uv sync

# Or pip
pip install -e .

Usage

# Dry run (preview entities without uploading)
uv run synq-steep --dry-run --mock-dir ./v1

# Sync all entity types
uv run synq-steep \
  --steep-token $STEEP_TOKEN \
  --synq-client-id $SYNQ_CLIENT_ID \
  --synq-client-secret $SYNQ_CLIENT_SECRET

# Sync specific types only
uv run synq-steep \
  --steep-token $STEEP_TOKEN \
  --synq-client-id $SYNQ_CLIENT_ID \
  --synq-client-secret $SYNQ_CLIENT_SECRET \
  --types metrics,modules

# Sync modules with Snowflake upstream table relationships
uv run synq-steep \
  --steep-token $STEEP_TOKEN \
  --synq-client-id $SYNQ_CLIENT_ID \
  --synq-client-secret $SYNQ_CLIENT_SECRET \
  --snowflake-account $SNOWFLAKE_ACCOUNT \
  --snowflake-database $SNOWFLAKE_DATABASE

Options

Option Env Variable Description
--steep-url STEEP_URL Steep API base URL (default: https://api.steep.app)
--steep-token STEEP_TOKEN Steep API bearer token
--synq-client-id SYNQ_CLIENT_ID SYNQ OAuth2 client ID
--synq-client-secret SYNQ_CLIENT_SECRET SYNQ OAuth2 client secret
--synq-host SYNQ_HOST SYNQ API host (default: developer.synq.io)
--snowflake-account SNOWFLAKE_ACCOUNT Snowflake account identifier for upstream table relationships
--snowflake-database SNOWFLAKE_DATABASE Snowflake database name for upstream table relationships
--dry-run - Print entities instead of uploading
--mock-dir - Use local JSON files instead of Steep API
--types - Comma-separated types: metrics, entities, modules
--skip-types - Skip custom type definition phase
--skip-relationships - Skip relationship creation phase

GitHub Actions Workflow

Create .github/workflows/sync-steep.yml:

name: Sync Steep to SYNQ

on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:        # Manual trigger

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Install dependencies
        run: uv sync

      - name: Sync Steep data to SYNQ
        env:
          STEEP_TOKEN: ${{ secrets.STEEP_TOKEN }}
          SYNQ_CLIENT_ID: ${{ secrets.SYNQ_CLIENT_ID }}
          SYNQ_CLIENT_SECRET: ${{ secrets.SYNQ_CLIENT_SECRET }}
        run: uv run synq-steep

Required Secrets

Add these secrets in GitHub repository settings:

  • STEEP_TOKEN - Steep API bearer token
  • SYNQ_CLIENT_ID - SYNQ OAuth2 client ID
  • SYNQ_CLIENT_SECRET - SYNQ OAuth2 client secret

Development

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=synq_steep

# Type checking
uv run pyright

Entity Mapping

Steep Type SYNQ Identifier Annotations
Metric steep::metric::{id} url, identifier, category, owners, schema, table
Entity steep::entity::{id} moduleId, createdAt, updatedAt
Module steep::module::{id} identifier, schema, table, externalSource