Skip to content

Latest commit

 

History

History
601 lines (448 loc) · 13.5 KB

File metadata and controls

601 lines (448 loc) · 13.5 KB

API Guide - K2 Reference Data Platform

Production-ready REST API for querying crypto instrument specifications with bitemporal support.

Quick Start

# Start API server
make api-dev

# API available at
http://localhost:8001

# Interactive documentation
http://localhost:8001/docs

# Alternative documentation (ReDoc)
http://localhost:8001/redoc

Core Concepts

Bitemporal Queries

The API supports point-in-time queries using dual temporality:

  • Business Time (valid_from/valid_to): When spec was effective in reality
  • System Time (record_created_at): When we learned about it

Example:

# What was BTCUSDT tick_size on 2024-01-15 at 10:00 UTC?
GET /v1/instruments?exchange=binance&symbol=BTCUSDT&as_of=2024-01-15T10:00:00Z

Canonical IDs

Instruments have exchange-native symbols and canonical IDs:

Exchange Native Symbol Canonical ID
Binance BTCUSDT BTC-USD-SPOT
Kraken XBT/USD BTC-USD-SPOT
Coinbase BTC-USD BTC-USD-SPOT

Use symbology endpoints to map between representations.


Authentication

Phase 1: No authentication (development only)

Production: API key authentication via X-API-Key header

curl -H "X-API-Key: your-api-key" https://api.k2.com/v1/instruments

Endpoints

Health Check

GET /health

Verify API and dependencies are operational.

Response:

{
  "status": "healthy",
  "version": "1.0.0",
  "checks": {
    "duckdb": "ok",
    "iceberg_catalog": "ok",
    "s3": "ok"
  }
}

Status Values:

  • healthy: All checks passed
  • degraded: Some non-critical checks failed
  • unhealthy: Critical checks failed

List Instruments

GET /v1/instruments

Query current (active) instruments with optional filters.

Query Parameters:

  • exchange (optional): Filter by exchange (binance, kraken)
  • symbol (optional): Filter by symbol (BTCUSDT)
  • instrument_type (optional): Filter by type (spot, perpetual, future)
  • as_of (optional): Point-in-time timestamp (ISO 8601)
  • limit (default: 100, max: 1000): Results limit

Examples:

# All current instruments
curl http://localhost:8001/v1/instruments

# Binance instruments only
curl http://localhost:8001/v1/instruments?exchange=binance

# Spot instruments only
curl http://localhost:8001/v1/instruments?instrument_type=spot

# Specific instrument (current state)
curl http://localhost:8001/v1/instruments?exchange=binance&symbol=BTCUSDT

# Historical query (as of specific date)
curl "http://localhost:8001/v1/instruments?exchange=binance&symbol=BTCUSDT&as_of=2024-01-15T10:00:00Z"

Response:

{
  "data": [
    {
      "exchange": "binance",
      "symbol": "BTCUSDT",
      "instrument_type": "spot",
      "base_asset": "BTC",
      "quote_asset": "USDT",
      "status": "active",
      "tick_size": 0.01,
      "lot_size": 0.00001,
      "min_notional": 10.0,
      "max_leverage": null,
      "funding_interval_hours": null,
      "settlement_asset": null,
      "valid_from": "2024-01-10T00:00:00Z",
      "valid_to": null,
      "record_created_at": "2024-01-10T10:00:00Z"
    }
  ],
  "count": 1,
  "query_timestamp": "2024-01-23T15:30:00Z",
  "request_id": "req_abc123"
}

Field Descriptions:

Field Description
tick_size Minimum price increment (e.g., 0.01 = $0.01 steps)
lot_size Minimum quantity increment (e.g., 0.001 BTC)
min_notional Minimum order value in quote currency
max_leverage Maximum allowed leverage (null for spot)
valid_from When this spec became effective
valid_to When this spec was superseded (null = current)
record_created_at When we first recorded this version

Instrument History

GET /v1/instruments/{exchange}/{symbol}/history

Get full audit trail showing all specification changes over time.

Path Parameters:

  • exchange: Exchange identifier
  • symbol: Exchange-native symbol

Query Parameters:

  • limit (default: 100, max: 1000): Maximum versions to return

Example:

curl http://localhost:8001/v1/instruments/binance/BTCUSDT/history

Response:

{
  "exchange": "binance",
  "symbol": "BTCUSDT",
  "history": [
    {
      "tick_size": 0.01,
      "lot_size": 0.00001,
      "status": "active",
      "valid_from": "2024-01-10T00:00:00Z",
      "valid_to": "2024-01-15T00:00:00Z",
      "record_created_at": "2024-01-10T10:00:00Z",
      "change_reason": "initial_load"
    },
    {
      "tick_size": 0.05,
      "lot_size": 0.00001,
      "status": "active",
      "valid_from": "2024-01-15T00:00:00Z",
      "valid_to": null,
      "record_created_at": "2024-01-14T16:00:00Z",
      "change_reason": "spec_update"
    }
  ],
  "count": 2,
  "request_id": "req_xyz789"
}

Use Case: Regulatory compliance, debugging pricing issues, understanding spec evolution.


Symbology Lookup (Canonical → Exchange)

GET /v1/symbology/{canonical_id}

Lookup exchange-specific symbols from canonical ID.

Path Parameters:

  • canonical_id: Canonical instrument identifier (e.g., BTC-USD-SPOT)

Example:

curl http://localhost:8001/v1/symbology/BTC-USD-SPOT

Response:

{
  "canonical_id": "BTC-USD-SPOT",
  "base_asset": "BTC",
  "quote_asset": "USD",
  "instrument_class": "spot",
  "binance_symbol": "BTCUSDT",
  "kraken_symbol": "XBT/USD",
  "bybit_symbol": null,
  "coinbase_symbol": "BTC-USD",
  "parent_canonical_id": null,
  "exchange_count": 3,
  "created_at": "2024-01-10T00:00:00Z",
  "updated_at": "2024-01-15T10:00:00Z"
}

Use Case: Multi-exchange trading (submit orders to all venues listing an instrument).


Symbology Reverse Lookup (Exchange → Canonical)

GET /v1/symbology/resolve

Resolve exchange-specific symbol to canonical ID.

Query Parameters:

  • exchange: Exchange identifier (binance, kraken, etc.)
  • symbol: Exchange-native symbol

Example:

curl "http://localhost:8001/v1/symbology/resolve?exchange=binance&symbol=BTCUSDT"

Response:

{
  "canonical_id": "BTC-USD-SPOT",
  "base_asset": "BTC",
  "quote_asset": "USD",
  "instrument_class": "spot",
  "binance_symbol": "BTCUSDT",
  "kraken_symbol": "XBT/USD",
  ...
}

Use Case: Normalize symbols from different exchanges for aggregation.


Error Handling

All errors return consistent JSON structure:

{
  "detail": {
    "code": "INSTRUMENT_NOT_FOUND",
    "message": "Instrument binance/BTCUSDT not found as of 2024-01-15T10:00:00Z",
    "details": {
      "exchange": "binance",
      "symbol": "BTCUSDT",
      "as_of": "2024-01-15T10:00:00Z"
    },
    "request_id": "req_abc123"
  }
}

HTTP Status Codes:

Code Meaning Example
200 Success Instrument found
404 Not Found Instrument doesn't exist
400 Bad Request Invalid parameter
413 Payload Too Large Request body > 10MB
500 Internal Error Database failure
503 Service Unavailable Iceberg catalog down

Error Codes:

Code Description
INSTRUMENT_NOT_FOUND Instrument doesn't exist at given timestamp
SYMBOLOGY_NOT_FOUND Canonical ID or symbol not found
INVALID_PARAMETER Query parameter validation failed
INTERNAL_SERVER_ERROR Unexpected server error

Response Headers

All responses include standard headers:

Header Description Example
X-Correlation-ID Request tracing ID req_abc123
Cache-Control HTTP caching policy public, max-age=300
Content-Type Response format application/json

Caching: Responses are cacheable for 5 minutes (reference data changes infrequently).


Rate Limiting

Phase 1: No rate limiting

Production: 1000 requests/minute per API key

Rate limit headers (when implemented):

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 987
X-RateLimit-Reset: 1640000000

OpenAPI Specification

Interactive documentation: http://localhost:8001/docs

Download OpenAPI spec: http://localhost:8001/openapi.json

Generate client SDKs:

# Python client
openapi-generator generate -i http://localhost:8001/openapi.json -g python -o ./clients/python

# TypeScript client
openapi-generator generate -i http://localhost:8001/openapi.json -g typescript-fetch -o ./clients/typescript

Code Examples

Python (requests)

import requests
from datetime import datetime

# Current instruments
response = requests.get(
    "http://localhost:8001/v1/instruments",
    params={"exchange": "binance", "limit": 100}
)
instruments = response.json()["data"]

# Point-in-time query
as_of = datetime(2024, 1, 15, 10, 0, 0).isoformat() + "Z"
response = requests.get(
    "http://localhost:8001/v1/instruments",
    params={
        "exchange": "binance",
        "symbol": "BTCUSDT",
        "as_of": as_of
    }
)
historical_spec = response.json()["data"][0]
print(f"Tick size on {as_of}: {historical_spec['tick_size']}")

# Symbology lookup
response = requests.get(
    "http://localhost:8001/v1/symbology/BTC-USD-SPOT"
)
symbology = response.json()
print(f"Binance: {symbology['binance_symbol']}")
print(f"Kraken: {symbology['kraken_symbol']}")

JavaScript (fetch)

// Current instruments
const response = await fetch(
  'http://localhost:8001/v1/instruments?exchange=binance&limit=100'
);
const { data: instruments } = await response.json();

// Point-in-time query
const asOf = '2024-01-15T10:00:00Z';
const historicalResponse = await fetch(
  `http://localhost:8001/v1/instruments?exchange=binance&symbol=BTCUSDT&as_of=${asOf}`
);
const { data: [spec] } = await historicalResponse.json();
console.log(`Tick size on ${asOf}: ${spec.tick_size}`);

// Symbology reverse lookup
const symbologyResponse = await fetch(
  'http://localhost:8001/v1/symbology/resolve?exchange=binance&symbol=BTCUSDT'
);
const symbology = await symbologyResponse.json();
console.log(`Canonical ID: ${symbology.canonical_id}`);

curl

# Current instruments
curl -X GET "http://localhost:8001/v1/instruments?exchange=binance&limit=10" \
  -H "Accept: application/json"

# Point-in-time query
curl -X GET "http://localhost:8001/v1/instruments?exchange=binance&symbol=BTCUSDT&as_of=2024-01-15T10:00:00Z" \
  -H "Accept: application/json"

# Instrument history
curl -X GET "http://localhost:8001/v1/instruments/binance/BTCUSDT/history" \
  -H "Accept: application/json"

# Symbology lookup
curl -X GET "http://localhost:8001/v1/symbology/BTC-USD-SPOT" \
  -H "Accept: application/json"

# Reverse symbology
curl -X GET "http://localhost:8001/v1/symbology/resolve?exchange=binance&symbol=BTCUSDT" \
  -H "Accept: application/json"

Performance

Latency Targets:

  • p50: < 50ms
  • p95: < 100ms
  • p99: < 200ms

Optimization:

  • Connection pooling (5-50 connections)
  • HTTP caching (5 minutes)
  • DuckDB query optimization
  • Iceberg metadata pruning

Monitor:

# Check response times
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8001/v1/instruments

# curl-format.txt content:
time_namelookup:  %{time_namelookup}\n
time_connect:  %{time_connect}\n
time_starttransfer:  %{time_starttransfer}\n
time_total:  %{time_total}\n

Troubleshooting

API Returns 503 (Service Unavailable)

Cause: Iceberg catalog or S3/MinIO unreachable

Fix:

# Check infrastructure
make docker-up

# Verify services
docker compose -f infrastructure/compose/docker-compose.refdata.yml ps

# Check health endpoint
curl http://localhost:8001/health

Empty Results ({data: [], count: 0})

Cause: No data in Silver/Gold tables

Fix:

# Run ingestion
make ingest-now

# Run DBT transformations
make dbt-run

# Verify data
duckdb -c "SELECT COUNT(*) FROM iceberg_scan('s3://refdata-warehouse/silver/instruments')"

404 for Point-in-Time Query

Cause: Instrument didn't exist at that timestamp

Solution: Check valid_from/valid_to in history:

curl http://localhost:8001/v1/instruments/binance/BTCUSDT/history

Development

Running Locally

# Start infrastructure
make docker-up

# Start API (dev mode with auto-reload)
make api-dev

# Run integration tests
make test-integration

Adding New Endpoints

  1. Create router (src/refdata/api/routers/new_endpoint.py)
  2. Define Pydantic models (src/refdata/api/models.py)
  3. Add query function (src/refdata/api/queries.py)
  4. Register router (src/refdata/api/main.py)
  5. Add tests (tests/integration/api/test_new_endpoint.py)
  6. Update this guide

Production Deployment

Docker

# Build image
docker build -t k2-refdata-api:latest .

# Run container
docker run -d \
  -p 8001:8001 \
  -e DBT_S3_ENDPOINT=s3.amazonaws.com \
  -e DBT_S3_ACCESS_KEY_ID=AKIA... \
  -e DBT_S3_SECRET_ACCESS_KEY=... \
  k2-refdata-api:latest

Environment Variables

Variable Description Default
DBT_S3_ENDPOINT S3/MinIO endpoint localhost:9000
DBT_S3_ACCESS_KEY_ID S3 access key admin
DBT_S3_SECRET_ACCESS_KEY S3 secret key password
DBT_ICEBERG_CATALOG_URI Iceberg REST catalog http://localhost:8181
LOG_LEVEL Logging level INFO

Support


Last Updated: 2026-01-23 API Version: 1.0.0