Note	The previous wis2downloader Python package has been archived to https://github.com/world-Meteorological-Organization/wis2downloader-v1.

WIS2 Downloader

A scalable system for downloading data from WIS2 (WMO Information System 2.0) Global Brokers.

Overview

WIS2 Downloader subscribes to MQTT topics on WIS2 Global Brokers, receives notification messages, and downloads the referenced data files. It features:

• Reliability - Redis-backed state with automatic reconnection

• Scalability - Celery workers for parallel downloads

• Deduplication - Prevents duplicate downloads via message ID, data ID, and file hash tracking

• Filtering - Filter by media type at subscription level

• Monitoring - Prometheus metrics and Grafana dashboards

• Centralized Logging - Loki for log aggregation

Architecture

                                    ┌─────────────────┐
                                    │  WIS2 Global    │
                                    │  Broker (MQTT)  │
                                    └────────┬────────┘
                                             │
                                             ▼
┌─────────────┐    Redis PubSub    ┌─────────────────┐
│ Subscription│◄──────────────────►│   Subscriber    │
│   Manager   │    (commands)      │   (MQTT Client) │
│  (REST API) │                    └────────┬────────┘
└──────┬──────┘                             │
       │                                    │ Celery Task
       │ Metrics                            ▼
       │                           ┌─────────────────┐
       ▼                           │  Celery Worker  │
┌─────────────┐                    │  (Downloads)    │
│ Prometheus  │◄───────────────────┤                 │
└──────┬──────┘    Metrics         └────────┬────────┘
       │                                    │
       ▼                                    ▼
┌─────────────┐                    ┌─────────────────┐
│   Grafana   │                    │   File System   │
└─────────────┘                    │   (/data)       │
                                   └─────────────────┘

              ┌─────────────────────────────────┐
              │             Redis               │
              │    (State + Task Queue)         │
              └─────────────────────────────────┘

Quick Start

Download the latest release tarball and extract it, then enter the extracted directory.

Then configure and start:

# Generate .env with randomised secrets
bash setup.sh

# Set up downloads directory permissions (containers run as UID 10001)
# Note - this is only required when run on a Linux workstation/server.
sudo groupadd -g 988 wis2
sudo useradd -u 10001 -g wis2 -M -s /usr/sbin/nologin wis2
mkdir -p downloads && sudo chown wis2:wis2 downloads

# Start services
docker compose up -d

# Create a subscription
curl -X POST http://localhost:5002/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
    "target": "surface-obs"
  }'

# View downloaded files
ls -la downloads/surface-obs/

Note	The containers run as a non-root user (UID 10001, GID 988). If you encounter permission errors writing to the downloads directory, see the Admin Guide for alternative approaches including ACLs.

Tip	To follow the latest development version instead, clone the repository with git clone https://github.com/World-Meteorological-Organization/wis2downloader.git.

Documentation

Full documentation is available at world-meteorological-organization.github.io/wis2downloader in English, French, Spanish, Arabic, Chinese, and Russian.

Document	Description
Admin Guide	Deployment, configuration, environment variables, monitoring
User Guide	Creating subscriptions, filtering, viewing downloads
API Reference	REST API documentation (also available via Swagger UI at /swagger)
Developer Guide	Architecture, module documentation, extending the system

Services

Service	Port	Description
subscription-manager	5002	REST API for managing subscriptions, Prometheus metrics endpoint
subscriber	-	MQTT client connecting to WIS2 Global Broker
celery	-	Background workers processing download tasks
grafana	3000	Metrics visualization (admin/admin)
redis	6379 (internal)	Redis for state storage and task queue

Configuration

Key environment variables (see Admin Guide for complete list):

Variable	Default	Description
GLOBAL_BROKER_HOST	See docker-compose.yaml	WIS2 Global Broker hostname (two brokers pre-configured)
DATA_PATH	/data	Base directory for downloaded files
LOG_LEVEL	DEBUG	Logging verbosity
GC_EXCLUDE	-	Comma-separated list of global caches to exclude

License

See LICENSE file.