Add six real-world large-data notebooks

README.md

@@ -218,6 +218,19 @@ For the full design, see the [design dossier](spec/design-dossier.md).
 - Large-data views that adapt from direct rendering to decimated and density
   representations as the visible range changes.
 
+## Examples
+
+Each notebook fetches working rows from its linked public source; raw datasets
+are not stored in this repository. See the
+[example guide](examples/real_world/README.md) for source links, workload
+controls, and setup. Counts describe the data behind each featured chart; the
+notebooks can scale further.
+
+|  |  |  |
+| :---: | :---: | :---: |
+| **Gaia DR3 · cosmic observatory**<br><sub>250,000 plotted stars</sub><br><br>![Gaia DR3 stellar color versus absolute magnitude.](examples/real_world/assets/01-gaia-hr-diagram.png)<br><br>[Open notebook](examples/real_world/01_gaia_hr_diagram.ipynb) | **gnomAD v4.1 · genomic atlas**<br><sub>164,000 plotted variants</sub><br><br>![gnomAD allele frequency across all autosomes.](examples/real_world/assets/02-gnomad-allele-frequency.png)<br><br>[Open notebook](examples/real_world/02_gnomad_allele_frequency.ipynb) | **Pan-UKBB · biobank editorial**<br><sub>814,294 plotted variants</sub><br><br>![Pan-UKBB standing-height associations across all autosomes.](examples/real_world/assets/03-pan-ukbb-manhattan.png)<br><br>[Open notebook](examples/real_world/03_pan_ukbb_manhattan.ipynb) |
+| **Dukascopy · trading terminal**<br><sub>101,427 plotted ticks</sub><br><br>![Dukascopy EUR/USD midpoint quotes.](examples/real_world/assets/04-dukascopy-fx-ticks.png)<br><br>[Open notebook](examples/real_world/04_dukascopy_fx_ticks.ipynb) | **LIGO · signal-lab oscilloscope**<br><sub>16,777,216 raw · 3,441 shown</sub><br><br>![GWOSC reconstructed Hanford waveform for GW150914.](examples/real_world/assets/05-ligo-gw150914-strain.png)<br><br>[Open notebook](examples/real_world/05_ligo_gw150914_strain.ipynb) | **NYC TLC · night cartography**<br><sub>300,000 pickup records</sub><br><br>![Locally projected NYC yellow-taxi pickup hexbin density.](examples/real_world/assets/06-nyc-taxi-density.png)<br><br>[Open notebook](examples/real_world/06_nyc_taxi_density.ipynb) |
+
 ## Documentation
 
 Start with the [XY documentation](https://reflex.dev/docs/xy/) for installation,

examples/real_world/.gitignore

@@ -0,0 +1 @@
+data/

examples/real_world/01_gaia_hr_diagram.ipynb

@@ -0,0 +1,284 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7fb27b941602401d91542211134fc71a",
+   "metadata": {},
+   "source": [
+    "# Gaia DR3: a million-star Hertzsprung–Russell diagram\n",
+    "\n",
+    "This notebook queries the official ESA Gaia Archive and gives the raw\n",
+    "color–magnitude points to XY. The main sequence, red-giant branch, and\n",
+    "white-dwarf sequence emerge as density structure; no pre-binning is\n",
+    "required.\n",
+    "\n",
+    "Gaia DR3 contains roughly 1.8 billion sources. The default query keeps\n",
+    "this run practical at one million high-quality stars. Increase\n",
+    "`GAIA_ROWS` to exercise a larger slice.\n",
+    "\n",
+    "**Source:** [Gaia Archive programmatic access](https://www.cosmos.esa.int/web/gaia-users/archive/programmatic-access)\n",
+    "and [`gaiadr3.gaia_source`](https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html).\n",
+    "\n",
+    "Install beside XY with `python -m pip install numpy requests xy`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "acae54e37e7d407bbb7b55eff062a284",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import numpy as np\n",
+    "import requests\n",
+    "\n",
+    "import xy\n",
+    "\n",
+    "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\"))\n",
+    "DATA_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "ROW_LIMIT = int(os.getenv(\"GAIA_ROWS\", \"1000000\"))\n",
+    "if ROW_LIMIT <= 0:\n",
+    "    raise ValueError(\"GAIA_ROWS must be positive\")\n",
+    "\n",
+    "TAP_SYNC = \"https://gea.esac.esa.int/tap-server/tap/sync\"\n",
+    "query = f\"\"\"\n",
+    "SELECT TOP {ROW_LIMIT}\n",
+    "    bp_rp,\n",
+    "    phot_g_mean_mag,\n",
+    "    parallax\n",
+    "FROM gaiadr3.gaia_source\n",
+    "WHERE bp_rp IS NOT NULL\n",
+    "    AND phot_g_mean_mag IS NOT NULL\n",
+    "    AND parallax > 0\n",
+    "    AND parallax_over_error > 10\n",
+    "    AND phot_g_mean_flux_over_error > 50\n",
+    "\"\"\"\n",
+    "\n",
+    "csv_path = DATA_DIR / f\"gaia-dr3-hr-{ROW_LIMIT}.csv\"\n",
+    "if not csv_path.exists():\n",
+    "    with requests.post(\n",
+    "        TAP_SYNC,\n",
+    "        data={\n",
+    "            \"REQUEST\": \"doQuery\",\n",
+    "            \"LANG\": \"ADQL\",\n",
+    "            \"FORMAT\": \"csv\",\n",
+    "            \"QUERY\": query,\n",
+    "        },\n",
+    "        stream=True,\n",
+    "        timeout=(30, 3600),\n",
+    "    ) as response:\n",
+    "        response.raise_for_status()\n",
+    "        partial = csv_path.with_suffix(\".csv.part\")\n",
+    "        with partial.open(\"wb\") as output:\n",
+    "            for chunk in response.iter_content(chunk_size=1024 * 1024):\n",
+    "                output.write(chunk)\n",
+    "        partial.replace(csv_path)\n",
+    "\n",
+    "print(f\"cached query result: {csv_path}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "stars = np.genfromtxt(\n",
+    "    csv_path,\n",
+    "    delimiter=\",\",\n",
+    "    names=True,\n",
+    "    dtype=np.float64,\n",
+    "    encoding=\"utf-8\",\n",
+    ")\n",
+    "stars = np.atleast_1d(stars)\n",
+    "\n",
+    "color_index = stars[\"bp_rp\"]\n",
+    "absolute_g = stars[\"phot_g_mean_mag\"] + 5 * np.log10(stars[\"parallax\"]) - 10\n",
+    "finite = np.isfinite(color_index) & np.isfinite(absolute_g)\n",
+    "color_index = color_index[finite]\n",
+    "absolute_g = absolute_g[finite]\n",
+    "\n",
+    "print(f\"{color_index.size:,} stars\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8dd0d8092fe74a7c96281538738b07e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "axis_style = {\n",
+    "    \"axis_color\": \"#65708d\",\n",
+    "    \"axis_width\": 1.0,\n",
+    "    \"grid_color\": \"#343755\",\n",
+    "    \"grid_dash\": \"dotted\",\n",
+    "    \"grid_opacity\": 0.58,\n",
+    "    \"grid_width\": 0.8,\n",
+    "    \"label_color\": \"#d7d9e8\",\n",
+    "    \"label_size\": 14,\n",
+    "    \"tick_color\": \"#65708d\",\n",
+    "    \"tick_label_color\": \"#aeb4cb\",\n",
+    "    \"tick_label_size\": 12.5,\n",
+    "    \"tick_length\": 5,\n",
+    "    \"tick_width\": 0.8,\n",
+    "}\n",
+    "sequence_label_style = {\n",
+    "    \"background\": \"#090b18f2\",\n",
+    "    \"border\": \"1px solid #8b83b880\",\n",
+    "    \"border_radius\": 7,\n",
+    "    \"font_size\": 14.5,\n",
+    "    \"font_weight\": 700,\n",
+    "    \"letter_spacing\": \"0.055em\",\n",
+    "    \"padding\": \"4px 8px\",\n",
+    "}\n",
+    "\n",
+    "chart = xy.scatter_chart(\n",
+    "    xy.scatter(\n",
+    "        color_index,\n",
+    "        absolute_g,\n",
+    "        color=absolute_g,\n",
+    "        color_domain=(-6.0, 16.0),\n",
+    "        colormap=\"magma_r\",\n",
+    "        size=1.0,\n",
+    "        opacity=0.82,\n",
+    "        density=True,\n",
+    "    ),\n",
+    "    xy.callout(\n",
+    "        2.35,\n",
+    "        0.8,\n",
+    "        \"RED GIANT BRANCH\",\n",
+    "        dx=88,\n",
+    "        dy=-32,\n",
+    "        color=\"#f6a15f\",\n",
+    "        width=1.2,\n",
+    "        opacity=0.8,\n",
+    "        style={**sequence_label_style, \"label_color\": \"#ffc48d\"},\n",
+    "    ),\n",
+    "    xy.callout(\n",
+    "        1.72,\n",
+    "        6.1,\n",
+    "        \"MAIN SEQUENCE\",\n",
+    "        dx=116,\n",
+    "        dy=-22,\n",
+    "        color=\"#bca8ff\",\n",
+    "        width=1.2,\n",
+    "        opacity=0.76,\n",
+    "        style={**sequence_label_style, \"label_color\": \"#ddd4ff\"},\n",
+    "    ),\n",
+    "    xy.callout(\n",
+    "        0.65,\n",
+    "        9.0,\n",
+    "        \"WHITE DWARFS\",\n",
+    "        dx=-86,\n",
+    "        dy=34,\n",
+    "        color=\"#8ed8ff\",\n",
+    "        width=1.2,\n",
+    "        opacity=0.78,\n",
+    "        anchor=\"end\",\n",
+    "        style={**sequence_label_style, \"label_color\": \"#bde9ff\"},\n",
+    "    ),\n",
+    "    xy.text(\n",
+    "        0.08,\n",
+    "        0.965,\n",
+    "        \"GAIA DR3\",\n",
+    "        dx=0,\n",
+    "        dy=0,\n",
+    "        color=\"#bca8ff\",\n",
+    "        style={\n",
+    "            \"coordinate_space\": \"figure_fraction\",\n",
+    "            \"font_size\": 11.5,\n",
+    "            \"font_weight\": 750,\n",
+    "            \"letter_spacing\": \"0.18em\",\n",
+    "            \"vertical_align\": \"top\",\n",
+    "        },\n",
+    "    ),\n",
+    "    xy.text(\n",
+    "        0.08,\n",
+    "        0.928,\n",
+    "        \"Hertzsprung\\u2013Russell diagram\",\n",
+    "        dx=0,\n",
+    "        dy=0,\n",
+    "        color=\"#fff5e8\",\n",
+    "        style={\n",
+    "            \"coordinate_space\": \"figure_fraction\",\n",
+    "            \"font_size\": 25,\n",
+    "            \"font_weight\": 760,\n",
+    "            \"letter_spacing\": \"0.005em\",\n",
+    "            \"vertical_align\": \"top\",\n",
+    "        },\n",
+    "    ),\n",
+    "    xy.text(\n",
+    "        0.08,\n",
+    "        0.872,\n",
+    "        f\"{color_index.size:,} HIGH-CONFIDENCE STARS  ·  BRIGHTER ↑  ·  BLUE → RED\",\n",
+    "        dx=0,\n",
+    "        dy=0,\n",
+    "        color=\"#9299b8\",\n",
+    "        style={\n",
+    "            \"coordinate_space\": \"figure_fraction\",\n",
+    "            \"font_size\": 11.5,\n",
+    "            \"font_weight\": 600,\n",
+    "            \"letter_spacing\": \"0.09em\",\n",
+    "            \"vertical_align\": \"top\",\n",
+    "        },\n",
+    "    ),\n",
+    "    xy.x_axis(\n",
+    "        label=\"STELLAR COLOR  ·  BP \\u2212 RP (mag)\",\n",
+    "        domain=(-1.0, 5.0),\n",
+    "        tick_values=[-1, 0, 1, 2, 3, 4, 5],\n",
+    "        tick_labels=[\"\\u22121\", \"0\", \"1\", \"2\", \"3\", \"4\", \"5\"],\n",
+    "        style=axis_style,\n",
+    "    ),\n",
+    "    xy.y_axis(\n",
+    "        label=\"ABSOLUTE G MAGNITUDE\",\n",
+    "        label_offset=-28,\n",
+    "        domain=(-6.0, 16.0),\n",
+    "        reverse=True,\n",
+    "        tick_values=[-6, -2, 2, 6, 10, 14],\n",
+    "        tick_labels=[\"\\u22126\", \"\\u22122\", \"2\", \"6\", \"10\", \"14\"],\n",
+    "        style=axis_style,\n",
+    "    ),\n",
+    "    xy.theme(\n",
+    "        background=\"#03040c\",\n",
+    "        plot_background=\"#080916\",\n",
+    "        text_color=\"#e9eaf2\",\n",
+    "        grid_color=\"#343755\",\n",
+    "        axis_color=\"#65708d\",\n",
+    "        crosshair_color=\"#ffc48d\",\n",
+    "        selection_color=\"#bca8ff\",\n",
+    "        selection_fill=\"#bca8ff26\",\n",
+    "    ),\n",
+    "    styles={\n",
+    "        \"annotation_label\": {\"line_height\": 1.2},\n",
+    "        \"axis_title\": {\"font_weight\": 680, \"letter_spacing\": \"0.055em\"},\n",
+    "        \"tick_label\": {\"font_variant_numeric\": \"tabular-nums\"},\n",
+    "    },\n",
+    "    padding=(112, 48, 70, 116),\n",
+    "    width=1200,\n",
+    "    height=700,\n",
+    ")\n",
+    "print(chart.memory_report()[\"canonical_bytes\"], \"canonical bytes\")\n",
+    "chart"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}