Ghostfolio ETF Data Sync

Proof-of-concept community service for discovering ETF assets from Ghostfolio, fetching profile data such as countries, holdings and sectors from source providers like iShares, SPDR or Vanguard, then syncing that data back through the Ghostfolio asset profiles API.

Status

This project is a local proof of concept, not a polished or production-ready integration. It was vibe-coded with AI assistance to explore how Ghostfolio's read/write asset profile APIs can be used with external ETF provider data.

Expect rough edges: provider endpoints can change, mappings should be reviewed before injecting data, and generated payloads should be checked against the source provider before trusting them for anything important.

Direction

The service is intentionally separate from Ghostfolio:

• Ghostfolio remains the source for existing assets
• this service tracks assets, provider fetches and Ghostfolio injections
• reads and writes to Ghostfolio go through the asset profiles API
• storage can use file, SQLite or Postgres depending on .env

Installation

npm install
npm run build

Configuration

The service reads these environment variables:

PORT=5000
DATA_DIR=./data
GHOSTFOLIO_BASE_URL=http://localhost:3333
GHOSTFOLIO_TOKEN=...
ALLOW_SELF_SIGNED=false
GHOSTFOLIO_DATA_SOURCES=YAHOO,COINGECKO,MANUAL,GHOSTFOLIO
ISHARES_BASE_URL=https://www.ishares.com/uk/individual/en
ISHARES_AUTOCOMPLETE_URL=https://www.ishares.com/uk/individual/en/autoComplete.search
ISHARES_HOLDINGS_DOWNLOAD_ID=1506575576011
ISHARES_SCREENER_URL=https://www.ishares.com/uk/individual/en/product-screener/product-screener-v3.1.jsn
ISHARES_SCREENER_DCR_PATH=/templatedata/config/product-screener-v3/data/en/uk/product-screener/ishares-product-screener-backend-config
SPDR_HOLDINGS_BASE_URL=https://www.ssga.com/library-content/products/fund-data/etfs
SPDR_DEFAULT_REGION=us
SPDR_LANGUAGE=en
SPDR_FUNDFINDER_URL=https://www.ssga.com/bin/v1/ssmp/fund/fundfinder
SPDR_SEARCH_CONTEXTS=us|en|individual,lu|en_gb|institutional
VANGUARD_US_BASE_URL=https://investor.vanguard.com
VANGUARD_EU_BASE_URL=https://www.ie.vanguard
VANGUARD_EU_PRODUCT_LIST_URL=https://www.ie.vanguard/products
VANGUARD_EU_GRAPHQL_URL=https://www.ie.vanguard/gpx/graphql
VANGUARD_EU_CONSUMER_ID=ie0
DATABASE_PROVIDER=sqlite
DATABASE_URL=./data/sync.sqlite

Create a local .env from .env.example. GHOSTFOLIO_BASE_URL defaults to http://localhost:3333, because self-hosted Ghostfolio usually listens on port 3333. For a development setup, you can change it to something like https://localhost:4200.

ALLOW_SELF_SIGNED=true is useful for local Ghostfolio HTTPS certificates. Only enable it when you run Ghostfolio locally with self-signed HTTPS.

GHOSTFOLIO_DATA_SOURCES controls the dropdown for the Ghostfolio asset identifier source. This is different from the ETF provider source: YAHOO/IWDA.AS identifies the asset in Ghostfolio, while ishares/251882 identifies where profile data is fetched from.

The iShares connector downloads the public holdings CSV from iShares. ISHARES_BASE_URL defaults to the UK individual site, and ISHARES_HOLDINGS_DOWNLOAD_ID is the current public holdings download id used by the product pages. Provider search uses the public iShares product screener JSON endpoint configured by ISHARES_SCREENER_URL and ISHARES_SCREENER_DCR_PATH, with ISHARES_AUTOCOMPLETE_URL as an extra lookup for alternate tickers. Matches are enriched from the product page listing table so Ghostfolio symbol suggestions prefer exchange RICs such as IWDA.AS.

The SPDR connector downloads official SSGA holdings XLSX files from SPDR_HOLDINGS_BASE_URL. sourceSymbol is stored as a compact provider id:

• a ticker using SPDR_DEFAULT_REGION, for example SPY
• a region-qualified ticker, for example us:SPY or emea:WTCH NA

Full holdings XLSX URLs are still accepted by the fetch command for backwards compatibility, but provider search stores the compact provider id.

The US holdings files often include holdings only. EMEA/UCITS files can also include country and sector columns; the connector injects those only when they are present in the source file.

SPDR provider search uses the public SSGA fund-finder endpoint. SPDR_SEARCH_CONTEXTS is a comma-separated list of country|language|role contexts. The defaults search US ETFs and Luxembourg EMEA/UCITS ETFs.

The Vanguard connector supports two official Vanguard surfaces:

• US ETFs use investor.vanguard.com holdings APIs and store sourceSymbol as us:<ticker>, for example us:VOO
• European UCITS ETFs use the Irish Vanguard product list and GPX API and store sourceSymbol as eu:<portId>, for example eu:9679 for Vanguard FTSE All-World UCITS ETF (USD) Accumulating

Vanguard EU search reads the product ids from VANGUARD_EU_PRODUCT_LIST_URL and enriches matches through VANGUARD_EU_GRAPHQL_URL. The defaults point at https://www.ie.vanguard/products, which is the broad Irish-domiciled funds list. Set VANGUARD_EU_BASE_URL, VANGUARD_EU_PRODUCT_LIST_URL, VANGUARD_EU_GRAPHQL_URL and VANGUARD_EU_CONSUMER_ID together if you want to target another Vanguard locale such as the Netherlands site.

DATABASE_PROVIDER supports:

• file: JSON in DATA_DIR/store.json
• sqlite: SQLite database in DATABASE_URL, or DATA_DIR/sync.sqlite by default
• postgres: PostgreSQL through DATABASE_URL, or through POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER and POSTGRES_PASSWORD

For Postgres, the target database configuration is:

DATABASE_PROVIDER=postgres
DATABASE_URL=
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=ghostfolio_etf_sync
POSTGRES_USER=ghostfolio_etf_sync
POSTGRES_PASSWORD=change-me
POSTGRES_SSL=false

By default, the app can create the target role and database during startup:

POSTGRES_AUTO_CREATE=true
POSTGRES_ADMIN_HOST=postgres
POSTGRES_ADMIN_PORT=5432
POSTGRES_ADMIN_DB=postgres
POSTGRES_ADMIN_USER=postgres
POSTGRES_ADMIN_PASSWORD=change-me
POSTGRES_CONNECT_RETRIES=20
POSTGRES_CONNECT_RETRY_DELAY_MS=1000

This mirrors the convenience of the Ghostfolio Docker setup: the Postgres container exposes an admin database/user, and the application initializes what it needs. For an external Postgres server, POSTGRES_ADMIN_USER must have permission to create roles and databases. If you do not want the app to create them, set POSTGRES_AUTO_CREATE=false and create the role/database manually.

If you prefer a full connection string:

DATABASE_PROVIDER=postgres
DATABASE_URL=postgres://ghostfolio_etf_sync:change-me@postgres:5432/ghostfolio_etf_sync

When switching from file to an empty SQLite/Postgres store, the app automatically uses the existing DATA_DIR/store.json as a seed.

Workflow

1. Discover ETFs From Ghostfolio

npm run dev -- discover

This reads ETF asset profiles from Ghostfolio:

GET /api/v1/asset-profiles?assetSubClasses=ETF

The dashboard has the same flow through the Discover button. Discovery upserts assets in the sync store and keeps existing provider mappings such as source and sourceSymbol when Ghostfolio does not provide those fields.

The Ghostfolio token must belong to a user with admin access, because the endpoint is guarded by accessAdminControl.

2. Link Provider Data

Provider search fills source metadata such as provider id, ISIN and name. The Ghostfolio symbol is separate and should match the symbol used in your Ghostfolio portfolio. Choose one of the exchange ticker suggestions when available, or type it manually when the provider does not expose a matching symbol.

ETFs can also be added manually through the dashboard or CLI:

npm run dev -- add-asset \
  --dataSource YAHOO \
  --symbol IWDA.AS \
  --isin IE00B4L5Y983 \
  --name "iShares Core MSCI World UCITS ETF" \
  --source ishares \
  --sourceSymbol 251882

For iShares, sourceSymbol is the iShares product id. For example, 251882 is the iShares product id for iShares Core MSCI World UCITS ETF. Full iShares product URLs are still accepted by fetch commands for backwards compatibility, but provider search stores the product id.

3. Fetch Provider Data

npm run dev -- fetch --source ishares --symbol 251882 --dataSource YAHOO --assetSymbol IWDA.AS --output ./profile.json

The iShares connector parses the provider holdings CSV and creates:

• holdings from every row with a positive Weight (%)
• sectors by aggregating row weights by Sector
• countries by aggregating row weights by Location and mapping country names to ISO-2 country codes

SPDR fetches use the official SSGA holdings XLSX:

npm run dev -- fetch --source spdr --symbol us:SPY --dataSource YAHOO --assetSymbol SPY --output ./profile.json
npm run dev -- fetch --source spdr --symbol "emea:WTCH NA" --dataSource YAHOO --assetSymbol WTCH.AS --output ./profile.json

Vanguard fetches use compact source symbols for the US and EU routes:

npm run dev -- fetch --source vanguard --symbol us:VOO --dataSource YAHOO --assetSymbol VOO --output ./profile.json
npm run dev -- fetch --source vanguard --symbol eu:9679 --dataSource YAHOO --assetSymbol VWCE.AS --output ./profile.json

Fetch results are logged in the sync store.

4. Inject Data Into Ghostfolio

npm run dev -- inject --exchange YAHOO --symbol WTCH.AS --file ./profile.json

Equivalent curl example:

curl -k -i -X PATCH "https://localhost:4200/api/v1/asset-profiles/YAHOO/WTCH.AS" \
  -H "Authorization: Bearer $GF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "countries": [
      { "code": "US", "weight": 0.6 },
      { "code": "NL", "weight": 0.4 }
    ],
    "holdings": [
      { "name": "Test Holding A", "weight": 0.55 },
      { "name": "Test Holding B", "weight": 0.45 }
    ],
    "sectors": [
      { "name": "Technology", "weight": 0.9 },
      { "name": "Financial Services", "weight": 0.1 }
    ]
  }'

Ghostfolio currently exposes this update route behind experimental features. Enable experimental features for the token user before running injection jobs.

Dashboard

Start the dashboard server:

npm run dev:server

Open http://localhost:5000.

The dashboard shows:

• ETFs known by the sync service
• latest fetch per ETF
• latest injection per ETF
• recent fetch and injection logs

The top-level Refresh button reloads the dashboard from the sync database. It does not fetch provider data. Use Discover to import ETF asset profiles from Ghostfolio, Fetch in an asset detail modal to queue a provider fetch for that asset, and Fetch all on the assets view to queue fetches for every configured asset.

Fetch and inject actions can be queued from the asset detail modal:

• Fetch: downloads fresh provider data and writes a fetch history record
• after a successful fetch, an inject job is queued automatically and sends the latest payload to Ghostfolio
• failed fetches or failed injections can be retried from the asset detail history

The Queue tab shows only open jobs (queued and running). Completed and failed work is visible in each asset's detail history. The first queue implementation is in-process and persistent in the selected store. It processes one job at a time while the dashboard server is running. For multi-container workers, scheduled bulk syncs or stronger retry semantics, this should later move to a dedicated worker queue.

The frontend is prepared for translations. English is the default locale; additional locales can be added in ui/app.js.

Provider Architecture

Provider modules implement the SourceProvider contract from src/sources/provider.ts:

• id: stable provider id used by assets, for example ishares, spdr or vanguard
• search(query, limit): returns provider metadata and exchange ticker suggestions
• fetchProfile(sourceSymbol): fetches provider data and returns Ghostfolio-ready profile data

Provider-specific response shapes are handled in functions, not provider-local interfaces. Raw provider rows use generic RawHoldingRow and RawSearchRow records from src/sources/shared.ts. Holdings are normalized as early as possible to NormalizedHoldingRow, then converted to AssetProfileData through the shared buildAssetProfile() helper. This keeps country mapping, holdings sorting and sector/country aggregation consistent across providers.

To add another provider:

• add provider-specific config defaults in src/config.ts
• create a provider module that exports const provider: SourceProvider
• parse the provider response into NormalizedHoldingRow[]
• register the provider in src/sources/index.ts

Docker

docker build -t ghostfolio-etf-data-sync .
docker run --rm -p 5000:5000 \
  -e GHOSTFOLIO_BASE_URL=http://host.docker.internal:3333 \
  -e GHOSTFOLIO_TOKEN="$GF_TOKEN" \
  -e DATABASE_PROVIDER=sqlite \
  -e DATABASE_URL=/usr/src/app/data/sync.sqlite \
  -v "$PWD/data:/usr/src/app/data" \
  ghostfolio-etf-data-sync

Or with Docker Compose:

docker compose up --build

For Postgres through Docker Compose:

DATABASE_PROVIDER=postgres \
DATABASE_URL= \
POSTGRES_HOST=postgres \
POSTGRES_DB=ghostfolio_etf_sync \
POSTGRES_USER=ghostfolio_etf_sync \
POSTGRES_PASSWORD=change-me \
POSTGRES_AUTO_CREATE=true \
POSTGRES_ADMIN_USER=postgres \
POSTGRES_ADMIN_PASSWORD=change-me \
docker compose --profile postgres up --build

Storage

The store contains:

• assets: ETFs discovered from Ghostfolio or added manually
• fetches: provider fetch history, including status and payload
• injections: Ghostfolio injection history, including status and response metadata

In this first database version, the store is saved as a JSON document in the selected backend. This keeps adapters simple and makes migration from the original file store straightforward. Later, this can be normalized into separate SQL tables.

Initial setup behavior:

• file: creates DATA_DIR/store.json
• sqlite: creates the SQLite database file and sync_store table
• postgres: can create the role and database when POSTGRES_AUTO_CREATE=true, then creates the sync_store table

Next Steps

• Add automatic provider matching for discovered ETFs where ISINs are available
• Add more SPDR search contexts for additional local sites if needed
• Add UI actions for fetch, preview, inject and retry
• Add a queue/jobs view for bulk fetches, scheduled syncs and retries