diff --git a/README.md b/README.md index e489396..b3c6f99 100644 --- a/README.md +++ b/README.md @@ -154,6 +154,88 @@ See [examples/datatool_dashboard.py](examples/datatool_dashboard.py) for a full To regenerate the screenshots after UI changes, see [docs/generate_screenshots.py](docs/generate_screenshots.py). +## Server-side downsampling + +Large time series are downsampled on the server so the plot only transports the +points the current zoom level can show. This is driven by `PlotConfig.downsample`: + +```python +from opensemantic.base.view._config import DashboardConfig, PlotConfig, DownsampleConfig + +config = DashboardConfig( + plot=PlotConfig( + downsample=DownsampleConfig( + enabled=True, # downsample when the backend supports it + max_points=2000, # target points per channel + method="auto", # auto | sample | average | minmax + edge_anchors=True, # keep the window's first/last real datapoints + ) + ) +) +``` + +It only engages on a PostgREST/TimescaleDB backend (the `downsample_tool_channel` +RPC). On a SQLite/local backend, the RPC being absent, or any error, the read +silently falls back to the full-resolution path - downsampling never breaks a +read. The `DataToolView` plot also reloads at a finer resolution when you zoom in. + +Strategies (N = number of buckets): + +- `sample` (default): one real datapoint nearest each bucket center. Schema + agnostic; works for scalar and composite channels. N rows. +- `average`: structure-preserving deep average per bucket (every numeric leaf + averaged, non-numeric keys carried), bucket-center timestamp. N rows. +- `minmax`: the real min and max datapoint of every numeric sub-characteristic + per bucket. Scalar: 2N real rows; composite: the real per-leaf extremes. Best + at preserving spikes. +- `auto`: `minmax` for numeric channels, `sample` for text channels. + +`average`/`minmax` skip non-numeric leaves (comments, labels) and fall back to +`sample` when a channel has no numeric leaf at all. + +**Unit normalization caveat:** `average` and `minmax` compare and combine the +bare stored numbers per leaf, so they are only correct when all stored values of +a leaf share the same unit. The archive stores base-unit-normalized values, but +data ingested without normalization (mixed units in one channel) will produce +wrong `average`/`minmax` results. `sample` returns whole real rows and is +unaffected. + +The RPC lives in pgstack's `postgres/config/optional/100_init_tsdb_schema.sql` +(uses only core, Apache-2 TimescaleDB; no toolkit dependency). It is created at +database init; apply it manually (`psql -f` / pgAdmin) on an already-running +cluster. To measure the speedup, see +[benchmarks/bench_downsample.py](benchmarks/bench_downsample.py). + +[examples/downsample_demo.py](examples/downsample_demo.py) is an interactive +demo: four channels carry the same 100k-point signal (with narrow spikes), one +per strategy - the channels are named `raw`, `sample`, `average`, `minmax`. +Selecting `raw` loads slowly with full detail; `sample`/`average` load fast but +drop the spikes; `minmax` keeps them. Box-zoom into a flat stretch and click +"Load current range" to re-fetch that window at finer resolution - the hidden +spikes reappear on `sample`/`average`. Needs a running pgstack with the RPC +applied; seed once with `python examples/downsample_demo.py`, then +`panel serve examples/downsample_demo.py`. + +![Downsampling demo](docs/downsample_demo.gif) + +![Strategy comparison](docs/screenshot_downsample_raw.png) + +*Full window, all four channels: `raw` and `minmax` keep the spikes; `sample` +and `average` smooth them away at the coarse full-window resolution.* + +![Zoomed, before reload](docs/screenshot_downsample_zoom_select.png) + +*Box-zoomed around a spike on the `sample` channel: only the coarse full-window +points are shown, so the spike is still hidden.* + +![Zoom reveals the peak](docs/screenshot_downsample_zoom.png) + +*After "Load current range": the window is re-fetched at finer buckets and the +hidden spike reappears. The toolbar reset returns to the full window.* + +To regenerate these, see +[docs/generate_downsample_screenshots.py](docs/generate_downsample_screenshots.py). + ## ProcessObjectView (Process/Object Dashboard UI) Where `DataToolView` is centered on data tools, `ProcessObjectView` is centered diff --git a/benchmarks/bench_downsample.py b/benchmarks/bench_downsample.py new file mode 100644 index 0000000..7dd577b --- /dev/null +++ b/benchmarks/bench_downsample.py @@ -0,0 +1,219 @@ +"""Benchmark the server-side downsampling RPC against a live pgstack. + +Seeds escalating series sizes for a scalar and a composite channel, then +times a full-resolution read against each downsampling strategy and reports +wall time, rows returned and approximate payload size, so the speedup vs +full-resolution and the relative cost of the deep aggregates are visible. + +Gated on the same env vars as the integration tests; prints a skip notice +otherwise: + + TEST_PGRST_URL, TEST_PGRST_JWT_SECRET (required) + TEST_PGRST_JWT_ROLE, TEST_PGRST_SCHEMA (optional) + BENCH_SIZES comma-separated point counts (default "10000,100000") + BENCH_MAXPTS target points per downsampled read (default 2000) + BENCH_REPEAT timed repeats per case (default 3) + TEST_PG_DSN optional libpq DSN; if psycopg is installed an + EXPLAIN ANALYZE of one RPC call is printed. + +Run: python benchmarks/bench_downsample.py +""" + +import asyncio +import datetime as dt +import json +import os +import statistics +import sys +import time +from pathlib import Path + +# Load tests/.env if present, mirroring tests/conftest.py, so the benchmark +# can be run with the same configuration as the integration tests. +_env_path = Path(__file__).resolve().parent.parent / "tests" / ".env" +if _env_path.exists(): + try: + from dotenv import load_dotenv + + load_dotenv(_env_path) + except ImportError: + pass + +_URL = os.environ.get("TEST_PGRST_URL") +_SECRET = os.environ.get("TEST_PGRST_JWT_SECRET") +_ROLE = os.environ.get("TEST_PGRST_JWT_ROLE", "api_user") +_SCHEMA = os.environ.get("TEST_PGRST_SCHEMA", "api") +_SIZES = [int(s) for s in os.environ.get("BENCH_SIZES", "10000,100000").split(",")] +_MAXPTS = int(os.environ.get("BENCH_MAXPTS", "2000")) +_REPEAT = int(os.environ.get("BENCH_REPEAT", "3")) +_PG_DSN = os.environ.get("TEST_PG_DSN") + +WRITE_CHUNK = 5000 + + +def _make_db(): + import jwt + from postgrest import AsyncPostgrestClient + + from opensemantic.base import PostgrestTimeSeriesDatabaseController + from opensemantic.core import Label + + token = jwt.encode({"role": _ROLE}, _SECRET, algorithm="HS256") + client = AsyncPostgrestClient( + base_url=_URL, + schema=_SCHEMA, + headers={"Authorization": f"Bearer {token}"}, + ) + db = PostgrestTimeSeriesDatabaseController( + name="bench", label=[Label(text="Bench")], buffered=False + ) + db.set_client(client) + return db + + +def _make_controller(db): + """A DataToolController with a scalar and a composite channel, bound to db.""" + from uuid import uuid4 + + from opensemantic import compute_scoped_uuid + from opensemantic.base.v1 import DataChannel, DataToolController + from opensemantic.core.v1 import Label as LabelV1 + + parent = uuid4() + ctrl = DataToolController( + uuid=str(parent), + name="Bench", + label=[LabelV1(text="Bench")], + data_channels=[ + DataChannel( + uuid=str(compute_scoped_uuid(parent, "scalar")), + osw_id="placeholder", + name="scalar", + label=[LabelV1(text="scalar")], + ), + DataChannel( + uuid=str(compute_scoped_uuid(parent, "composite")), + osw_id="placeholder", + name="composite", + label=[LabelV1(text="composite")], + ), + ], + ) + ctrl.archive_database = db + return ctrl + + +async def _timed_read(db, tool_id, ch, start, end, method): + from opensemantic.base import DownsampleParams, ReadToolChannelRawParams + + ds = None + if method != "raw": + ds = DownsampleParams(max_points=_MAXPTS, method=method) + times = [] + rows = [] + for _ in range(_REPEAT): + t0 = time.perf_counter() + rows = await db.read_tool_channel_raw( + ReadToolChannelRawParams( + tool_osw_id=tool_id, + channel_osw_id=ch, + start=start, + end=end, + downsample=ds, + ) + ) + times.append((time.perf_counter() - t0) * 1000.0) + payload_kb = len(json.dumps(rows)) / 1024.0 + return statistics.median(times), len(rows), payload_kb + + +def _explain(tool_id, ch, start, end): + if not _PG_DSN: + return + try: + import psycopg + except Exception: + print("\n(psycopg not installed; skipping EXPLAIN ANALYZE)") + return + sql = ( + "EXPLAIN (ANALYZE, BUFFERS) " + "SELECT * FROM api.downsample_tool_channel(%s, %s, %s, %s, %s, NULL, 'minmax')" + ) + print("\nEXPLAIN ANALYZE (minmax):") + try: + with psycopg.connect(_PG_DSN) as conn, conn.cursor() as cur: + cur.execute(sql, (tool_id, ch, start, end, _MAXPTS)) + for (line,) in cur.fetchall(): + print(" " + line) + except Exception as e: + print(f" EXPLAIN failed: {e}") + + +async def _run(): + from opensemantic.base import DeleteToolParams + from opensemantic.base._demo_data import seed_channel_series + + db = _make_db() + header = ( + f"{'size':>9} {'channel':>9} {'method':>8} {'rows':>8} {'ms':>9} {'KB':>10}" + ) + print(header) + print("-" * len(header)) + base = dt.datetime(2023, 1, 1, tzinfo=dt.timezone.utc) + for idx, n in enumerate(_SIZES): + is_last = idx == len(_SIZES) - 1 + ctrl = _make_controller(db) + tool_id = ctrl.get_osw_id() + ch_scalar = ctrl.get_channel_by_name("scalar").get_osw_id().split("#")[-1] + ch_comp = ctrl.get_channel_by_name("composite").get_osw_id().split("#")[-1] + + def _value(channel, i, _n=n): + if channel.name == "composite": + return { + "temperature": {"value": float(i)}, + "humidity": {"value": float(_n - i)}, + } + return {"value": float(i)} + + try: + await seed_channel_series( + ctrl, n_points=n, base_ts=base, value_fn=_value, chunk_size=WRITE_CHUNK + ) + await asyncio.sleep(1.0) # let PostgREST settle after the writes + start, end = base, base + dt.timedelta(seconds=n - 1) + for label, ch in (("scalar", ch_scalar), ("composite", ch_comp)): + for method in ("raw", "sample", "average", "minmax"): + ms, rows, kb = await _timed_read( + db, tool_id, ch, start, end, method + ) + print( + f"{n:>9} {label:>9} {method:>8} {rows:>8} " + f"{ms:>9.1f} {kb:>10.1f}" + ) + if is_last: + # EXPLAIN while the tool still exists (largest size). + _explain(tool_id, ch_scalar, start, end) + finally: + try: + await db.delete_tool(DeleteToolParams(tool_osw_id=tool_id)) + except Exception: + pass + + +def main(): + if not (_URL and _SECRET): + print( + "Skipping benchmark: set TEST_PGRST_URL and TEST_PGRST_JWT_SECRET " + "to run against a live pgstack." + ) + return 0 + try: + asyncio.run(_run()) + except Exception as e: + print(f"Benchmark failed (is pgstack up and the RPC applied?): {e}") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/_screenshot_utils.py b/docs/_screenshot_utils.py new file mode 100644 index 0000000..584b282 --- /dev/null +++ b/docs/_screenshot_utils.py @@ -0,0 +1,91 @@ +"""Shared Playwright/Panel helpers for the docs screenshot generators. + +Used by generate_downsample_screenshots.py and generate_process_screenshots.py +so the Wunderbaum-checkbox clicking, frame capture and panel-serve lifecycle +live in one place. +""" + +import glob +import io +import os +import subprocess +import sys +import time + +import imageio.v3 as iio + +DOCS_DIR = os.path.dirname(os.path.abspath(__file__)) +PACKAGE_DIR = os.path.dirname(DOCS_DIR) + + +def all_wb_shadows_js(): + """JS defining allWbShadows(root): every Wunderbaum shadow root, in order.""" + return """ + function allWbShadows(root) { + const out = []; + function rec(r) { + for (const el of r.querySelectorAll('*')) { + if (el.shadowRoot) { + if (el.shadowRoot.querySelectorAll('.wb-row').length > 0) + out.push(el.shadowRoot); + rec(el.shadowRoot); + } + } + } + rec(root); + return out; + } + """ + + +def click_tree_checkbox(page, tree_idx, cb_idx): + """Click the cb_idx-th checkbox of the tree_idx-th Wunderbaum (0 = first).""" + page.evaluate( + f"""() => {{ + {all_wb_shadows_js()} + const sh = allWbShadows(document)[{tree_idx}]; + if (sh) {{ + const cbs = sh.querySelectorAll('i.wb-checkbox'); + if (cbs[{cb_idx}]) cbs[{cb_idx}].click(); + }} + }}""" + ) + + +def capture(page, frames, delay=800): + """Wait `delay` ms, then append a screenshot to `frames`.""" + page.wait_for_timeout(delay) + frames.append(iio.imread(io.BytesIO(page.screenshot()))) + + +def start_server(example, port, env=None, clean_sqlite=False, settle=12): + """Start ``panel serve --port `` and return the process. + + ``clean_sqlite`` first removes leftover ``*_db.sqlite`` in the package dir + (so a SQLite demo reseeds fresh). ``settle`` is the seconds to wait for the + first module execution before returning. + """ + if clean_sqlite: + for db in glob.glob(os.path.join(PACKAGE_DIR, "*_db.sqlite")): + os.remove(db) + logf = open(os.path.join(PACKAGE_DIR, f"_gen_server_{port}.log"), "w") + proc_env = dict(os.environ) + if env: + proc_env.update(env) + proc = subprocess.Popen( + [sys.executable, "-m", "panel", "serve", example, "--port", str(port)], + stdout=logf, + stderr=subprocess.STDOUT, + cwd=PACKAGE_DIR, + env=proc_env, + ) + time.sleep(settle) + return proc + + +def stop_server(proc): + proc.terminate() + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + proc.kill() diff --git a/docs/downsample_demo.gif b/docs/downsample_demo.gif new file mode 100644 index 0000000..345b190 Binary files /dev/null and b/docs/downsample_demo.gif differ diff --git a/docs/generate_downsample_screenshots.py b/docs/generate_downsample_screenshots.py new file mode 100644 index 0000000..ac22e57 --- /dev/null +++ b/docs/generate_downsample_screenshots.py @@ -0,0 +1,315 @@ +"""Generate screenshots and a GIF for the downsampling demo README. + +Serves examples/downsample_demo.py (which reads from a running pgstack with the +downsampling RPC applied and pre-seeded data) and drives the UI with Playwright. + +The script tells the whole story on a single full-window server: + 1. Full window: select each channel (one strategy each). minmax keeps the + spikes, sample/average lose them at the coarse full-window resolution, raw + shows full detail (slow 100k load). + 2. Zoom range: narrow the shared x-range to a window around the first spike + (this is what a horizontal box-zoom produces) - the spikes are still + missing because only the already-loaded coarse points are shown. + 3. Load current range: click the button - the zoomed window is re-fetched at + finer buckets and the hidden spikes reappear on sample/average. + +Steps 2-3 exercise the real interactive feature (set zoom range -> "Load +current range" re-fetch), so this script doubles as an end-to-end validation of +the zoom path: if the zoomed window were loaded at the wrong position (e.g. a +timezone offset bug), the spike would not land inside the narrowed axis. + +Prerequisites: + pip install playwright imageio + playwright install chromium + # a running pgstack on localhost:3000 with the demo data seeded + # (run `python examples/downsample_demo.py` once to seed) + +Usage: + python docs/generate_downsample_screenshots.py +""" + +import datetime as dt +import os + +import imageio.v3 as iio +from _screenshot_utils import ( + capture, + click_tree_checkbox, + start_server, + stop_server, +) +from playwright.sync_api import sync_playwright + +DOCS_DIR = os.path.dirname(os.path.abspath(__file__)) +PACKAGE_DIR = os.path.dirname(DOCS_DIR) +EXAMPLE = os.path.join(PACKAGE_DIR, "examples", "downsample_demo.py") +VIEWPORT = {"width": 1400, "height": 1300} + +# Tree checkbox order: 0 = DataTool root, then channels in declaration order +# (raw, sample, average, minmax). +CB_RAW = 1 +CB_SAMPLE = 2 +CB_AVERAGE = 3 +CB_MINMAX = 4 + +# Demo series constants (mirror examples/downsample_demo.py) used to compute the +# zoom window directly, instead of reading possibly-stale Bokeh range models. +BASE_TS = dt.datetime(2024, 1, 1, tzinfo=dt.timezone.utc) +# Zoom window in data seconds, around the first spike (at 12 000 s) so the finer +# buckets reveal it on sample/average after "Load current range". +ZOOM_SEC = (4_000, 20_000) + +LOAD_RANGE_LABEL = "Load current range" + + +def axis_ms(sec): + """Epoch-ms x-axis position of data second ``sec``. + + The plot draws timestamps in local wall time (naive), so a box-zoom range is + expressed as the local-wall-clock epoch. Mirror that: take the UTC instant + ``BASE_TS + sec``, read it as local wall time, and return that as an epoch. + """ + utc = BASE_TS + dt.timedelta(seconds=sec) + naive_local = utc.astimezone().replace(tzinfo=None) + return naive_local.replace(tzinfo=dt.timezone.utc).timestamp() * 1000.0 + + +def click_button(page, label): + """Click the first