diff --git a/README.md b/README.md index ca544f87..0c038227 100644 --- a/README.md +++ b/README.md @@ -218,6 +218,19 @@ For the full design, see the [design dossier](spec/design-dossier.md). - Large-data views that adapt from direct rendering to decimated and density representations as the visible range changes. +## Examples + +Each notebook fetches working rows from its linked public source; raw datasets +are not stored in this repository. See the +[example guide](examples/real_world/README.md) for source links, workload +controls, and setup. Counts describe the data behind each featured chart; the +notebooks can scale further. + +| | | | +| :---: | :---: | :---: | +| **Gaia DR3 · cosmic observatory**
250,000 plotted stars

![Gaia DR3 stellar color versus absolute magnitude.](examples/real_world/assets/01-gaia-hr-diagram.png)

[Open notebook](examples/real_world/01_gaia_hr_diagram.ipynb) | **gnomAD v4.1 · genomic atlas**
164,000 plotted variants

![gnomAD allele frequency across all autosomes.](examples/real_world/assets/02-gnomad-allele-frequency.png)

[Open notebook](examples/real_world/02_gnomad_allele_frequency.ipynb) | **Pan-UKBB · biobank editorial**
814,294 plotted variants

![Pan-UKBB standing-height associations across all autosomes.](examples/real_world/assets/03-pan-ukbb-manhattan.png)

[Open notebook](examples/real_world/03_pan_ukbb_manhattan.ipynb) | +| **Dukascopy · trading terminal**
101,427 plotted ticks

![Dukascopy EUR/USD midpoint quotes.](examples/real_world/assets/04-dukascopy-fx-ticks.png)

[Open notebook](examples/real_world/04_dukascopy_fx_ticks.ipynb) | **LIGO · signal-lab oscilloscope**
16,777,216 raw · 3,441 shown

![GWOSC reconstructed Hanford waveform for GW150914.](examples/real_world/assets/05-ligo-gw150914-strain.png)

[Open notebook](examples/real_world/05_ligo_gw150914_strain.ipynb) | **NYC TLC · night cartography**
300,000 pickup records

![Locally projected NYC yellow-taxi pickup hexbin density.](examples/real_world/assets/06-nyc-taxi-density.png)

[Open notebook](examples/real_world/06_nyc_taxi_density.ipynb) | + ## Documentation Start with the [XY documentation](https://reflex.dev/docs/xy/) for installation, diff --git a/examples/real_world/.gitignore b/examples/real_world/.gitignore new file mode 100644 index 00000000..8fce6030 --- /dev/null +++ b/examples/real_world/.gitignore @@ -0,0 +1 @@ +data/ diff --git a/examples/real_world/01_gaia_hr_diagram.ipynb b/examples/real_world/01_gaia_hr_diagram.ipynb new file mode 100644 index 00000000..4d158be2 --- /dev/null +++ b/examples/real_world/01_gaia_hr_diagram.ipynb @@ -0,0 +1,284 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# Gaia DR3: a million-star Hertzsprung–Russell diagram\n", + "\n", + "This notebook queries the official ESA Gaia Archive and gives the raw\n", + "color–magnitude points to XY. The main sequence, red-giant branch, and\n", + "white-dwarf sequence emerge as density structure; no pre-binning is\n", + "required.\n", + "\n", + "Gaia DR3 contains roughly 1.8 billion sources. The default query keeps\n", + "this run practical at one million high-quality stars. Increase\n", + "`GAIA_ROWS` to exercise a larger slice.\n", + "\n", + "**Source:** [Gaia Archive programmatic access](https://www.cosmos.esa.int/web/gaia-users/archive/programmatic-access)\n", + "and [`gaiadr3.gaia_source`](https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html).\n", + "\n", + "Install beside XY with `python -m pip install numpy requests xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import requests\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\"))\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "ROW_LIMIT = int(os.getenv(\"GAIA_ROWS\", \"1000000\"))\n", + "if ROW_LIMIT <= 0:\n", + " raise ValueError(\"GAIA_ROWS must be positive\")\n", + "\n", + "TAP_SYNC = \"https://gea.esac.esa.int/tap-server/tap/sync\"\n", + "query = f\"\"\"\n", + "SELECT TOP {ROW_LIMIT}\n", + " bp_rp,\n", + " phot_g_mean_mag,\n", + " parallax\n", + "FROM gaiadr3.gaia_source\n", + "WHERE bp_rp IS NOT NULL\n", + " AND phot_g_mean_mag IS NOT NULL\n", + " AND parallax > 0\n", + " AND parallax_over_error > 10\n", + " AND phot_g_mean_flux_over_error > 50\n", + "\"\"\"\n", + "\n", + "csv_path = DATA_DIR / f\"gaia-dr3-hr-{ROW_LIMIT}.csv\"\n", + "if not csv_path.exists():\n", + " with requests.post(\n", + " TAP_SYNC,\n", + " data={\n", + " \"REQUEST\": \"doQuery\",\n", + " \"LANG\": \"ADQL\",\n", + " \"FORMAT\": \"csv\",\n", + " \"QUERY\": query,\n", + " },\n", + " stream=True,\n", + " timeout=(30, 3600),\n", + " ) as response:\n", + " response.raise_for_status()\n", + " partial = csv_path.with_suffix(\".csv.part\")\n", + " with partial.open(\"wb\") as output:\n", + " for chunk in response.iter_content(chunk_size=1024 * 1024):\n", + " output.write(chunk)\n", + " partial.replace(csv_path)\n", + "\n", + "print(f\"cached query result: {csv_path}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "stars = np.genfromtxt(\n", + " csv_path,\n", + " delimiter=\",\",\n", + " names=True,\n", + " dtype=np.float64,\n", + " encoding=\"utf-8\",\n", + ")\n", + "stars = np.atleast_1d(stars)\n", + "\n", + "color_index = stars[\"bp_rp\"]\n", + "absolute_g = stars[\"phot_g_mean_mag\"] + 5 * np.log10(stars[\"parallax\"]) - 10\n", + "finite = np.isfinite(color_index) & np.isfinite(absolute_g)\n", + "color_index = color_index[finite]\n", + "absolute_g = absolute_g[finite]\n", + "\n", + "print(f\"{color_index.size:,} stars\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "axis_style = {\n", + " \"axis_color\": \"#65708d\",\n", + " \"axis_width\": 1.0,\n", + " \"grid_color\": \"#343755\",\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.58,\n", + " \"grid_width\": 0.8,\n", + " \"label_color\": \"#d7d9e8\",\n", + " \"label_size\": 14,\n", + " \"tick_color\": \"#65708d\",\n", + " \"tick_label_color\": \"#aeb4cb\",\n", + " \"tick_label_size\": 12.5,\n", + " \"tick_length\": 5,\n", + " \"tick_width\": 0.8,\n", + "}\n", + "sequence_label_style = {\n", + " \"background\": \"#090b18f2\",\n", + " \"border\": \"1px solid #8b83b880\",\n", + " \"border_radius\": 7,\n", + " \"font_size\": 14.5,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.055em\",\n", + " \"padding\": \"4px 8px\",\n", + "}\n", + "\n", + "chart = xy.scatter_chart(\n", + " xy.scatter(\n", + " color_index,\n", + " absolute_g,\n", + " color=absolute_g,\n", + " color_domain=(-6.0, 16.0),\n", + " colormap=\"magma_r\",\n", + " size=1.0,\n", + " opacity=0.82,\n", + " density=True,\n", + " ),\n", + " xy.callout(\n", + " 2.35,\n", + " 0.8,\n", + " \"RED GIANT BRANCH\",\n", + " dx=88,\n", + " dy=-32,\n", + " color=\"#f6a15f\",\n", + " width=1.2,\n", + " opacity=0.8,\n", + " style={**sequence_label_style, \"label_color\": \"#ffc48d\"},\n", + " ),\n", + " xy.callout(\n", + " 1.72,\n", + " 6.1,\n", + " \"MAIN SEQUENCE\",\n", + " dx=116,\n", + " dy=-22,\n", + " color=\"#bca8ff\",\n", + " width=1.2,\n", + " opacity=0.76,\n", + " style={**sequence_label_style, \"label_color\": \"#ddd4ff\"},\n", + " ),\n", + " xy.callout(\n", + " 0.65,\n", + " 9.0,\n", + " \"WHITE DWARFS\",\n", + " dx=-86,\n", + " dy=34,\n", + " color=\"#8ed8ff\",\n", + " width=1.2,\n", + " opacity=0.78,\n", + " anchor=\"end\",\n", + " style={**sequence_label_style, \"label_color\": \"#bde9ff\"},\n", + " ),\n", + " xy.text(\n", + " 0.08,\n", + " 0.965,\n", + " \"GAIA DR3\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#bca8ff\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.18em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.08,\n", + " 0.928,\n", + " \"Hertzsprung\\u2013Russell diagram\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#fff5e8\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 25,\n", + " \"font_weight\": 760,\n", + " \"letter_spacing\": \"0.005em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.08,\n", + " 0.872,\n", + " f\"{color_index.size:,} HIGH-CONFIDENCE STARS · BRIGHTER ↑ · BLUE → RED\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#9299b8\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 600,\n", + " \"letter_spacing\": \"0.09em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"STELLAR COLOR · BP \\u2212 RP (mag)\",\n", + " domain=(-1.0, 5.0),\n", + " tick_values=[-1, 0, 1, 2, 3, 4, 5],\n", + " tick_labels=[\"\\u22121\", \"0\", \"1\", \"2\", \"3\", \"4\", \"5\"],\n", + " style=axis_style,\n", + " ),\n", + " xy.y_axis(\n", + " label=\"ABSOLUTE G MAGNITUDE\",\n", + " label_offset=-28,\n", + " domain=(-6.0, 16.0),\n", + " reverse=True,\n", + " tick_values=[-6, -2, 2, 6, 10, 14],\n", + " tick_labels=[\"\\u22126\", \"\\u22122\", \"2\", \"6\", \"10\", \"14\"],\n", + " style=axis_style,\n", + " ),\n", + " xy.theme(\n", + " background=\"#03040c\",\n", + " plot_background=\"#080916\",\n", + " text_color=\"#e9eaf2\",\n", + " grid_color=\"#343755\",\n", + " axis_color=\"#65708d\",\n", + " crosshair_color=\"#ffc48d\",\n", + " selection_color=\"#bca8ff\",\n", + " selection_fill=\"#bca8ff26\",\n", + " ),\n", + " styles={\n", + " \"annotation_label\": {\"line_height\": 1.2},\n", + " \"axis_title\": {\"font_weight\": 680, \"letter_spacing\": \"0.055em\"},\n", + " \"tick_label\": {\"font_variant_numeric\": \"tabular-nums\"},\n", + " },\n", + " padding=(112, 48, 70, 116),\n", + " width=1200,\n", + " height=700,\n", + ")\n", + "print(chart.memory_report()[\"canonical_bytes\"], \"canonical bytes\")\n", + "chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/02_gnomad_allele_frequency.ipynb b/examples/real_world/02_gnomad_allele_frequency.ipynb new file mode 100644 index 00000000..b472d2b9 --- /dev/null +++ b/examples/real_world/02_gnomad_allele_frequency.ipynb @@ -0,0 +1,365 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# gnomAD v4.1: allele frequency across the human genome\n", + "\n", + "gnomAD v4.1 genomes contain hundreds of millions of short variants.\n", + "This notebook uses the official tabix indexes to read evenly spaced\n", + "windows from all 22 autosomes without downloading the multi-gigabyte\n", + "chromosome VCFs in full. XY receives the sampled variants as one dense\n", + "scatter and keeps the source rows available as the view refines.\n", + "\n", + "`GNOMAD_WINDOWS` and `GNOMAD_VARIANTS_PER_WINDOW` control the sample.\n", + "Set `GNOMAD_CHROMOSOMES=22` for a quick first run.\n", + "\n", + "**Source:** [gnomAD v4.1 release](https://gnomad.broadinstitute.org/news/2024-04-gnomad-v4-1/)\n", + "and the public Google Cloud mirror under\n", + "`gs://gcp-public-data--gnomad/release/4.1/`.\n", + "\n", + "Install beside XY with `python -m pip install numpy pysam requests xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import pysam\n", + "import requests\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\")) / \"gnomad\"\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "CHROMOSOME_LENGTHS = {\n", + " 1: 248_956_422,\n", + " 2: 242_193_529,\n", + " 3: 198_295_559,\n", + " 4: 190_214_555,\n", + " 5: 181_538_259,\n", + " 6: 170_805_979,\n", + " 7: 159_345_973,\n", + " 8: 145_138_636,\n", + " 9: 138_394_717,\n", + " 10: 133_797_422,\n", + " 11: 135_086_622,\n", + " 12: 133_275_309,\n", + " 13: 114_364_328,\n", + " 14: 107_043_718,\n", + " 15: 101_991_189,\n", + " 16: 90_338_345,\n", + " 17: 83_257_441,\n", + " 18: 80_373_285,\n", + " 19: 58_617_616,\n", + " 20: 64_444_167,\n", + " 21: 46_709_983,\n", + " 22: 50_818_468,\n", + "}\n", + "chromosomes = [\n", + " int(value)\n", + " for value in os.getenv(\n", + " \"GNOMAD_CHROMOSOMES\",\n", + " \",\".join(str(value) for value in CHROMOSOME_LENGTHS),\n", + " ).split(\",\")\n", + "]\n", + "unknown = set(chromosomes).difference(CHROMOSOME_LENGTHS)\n", + "if unknown:\n", + " raise ValueError(f\"unknown autosomes: {sorted(unknown)}\")\n", + "\n", + "windows_per_chromosome = int(os.getenv(\"GNOMAD_WINDOWS\", \"8\"))\n", + "variants_per_window = int(os.getenv(\"GNOMAD_VARIANTS_PER_WINDOW\", \"20000\"))\n", + "window_width = int(os.getenv(\"GNOMAD_WINDOW_BP\", \"2000000\"))\n", + "if min(windows_per_chromosome, variants_per_window, window_width) <= 0:\n", + " raise ValueError(\"gnomAD window controls must be positive\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "genomic_position_parts = []\n", + "allele_frequency_parts = []\n", + "chromosome_parts = []\n", + "\n", + "offsets = {}\n", + "running_offset = 0\n", + "for chromosome, length in CHROMOSOME_LENGTHS.items():\n", + " offsets[chromosome] = running_offset\n", + " running_offset += length\n", + "\n", + "base_url = \"https://storage.googleapis.com/gcp-public-data--gnomad/release/4.1/vcf/genomes\"\n", + "for chromosome in chromosomes:\n", + " length = CHROMOSOME_LENGTHS[chromosome]\n", + " centers = (\n", + " (np.arange(windows_per_chromosome, dtype=np.float64) + 0.5)\n", + " * length\n", + " / windows_per_chromosome\n", + " )\n", + " starts = np.clip(\n", + " centers - window_width / 2,\n", + " 0,\n", + " max(0, length - window_width),\n", + " ).astype(np.int64)\n", + " url = f\"{base_url}/gnomad.genomes.v4.1.sites.chr{chromosome}.vcf.bgz\"\n", + " index_path = DATA_DIR / f\"gnomad.genomes.v4.1.sites.chr{chromosome}.vcf.bgz.tbi\"\n", + " if not index_path.exists():\n", + " response = requests.get(f\"{url}.tbi\", timeout=120)\n", + " response.raise_for_status()\n", + " partial = index_path.with_suffix(\".tbi.part\")\n", + " partial.write_bytes(response.content)\n", + " partial.replace(index_path)\n", + " positions = []\n", + " frequencies = []\n", + " with pysam.VariantFile(url, index_filename=str(index_path)) as variants:\n", + " for start in starts:\n", + " kept = 0\n", + " records = variants.fetch(\n", + " f\"chr{chromosome}\",\n", + " int(start),\n", + " int(start + window_width),\n", + " )\n", + " for record in records:\n", + " allele_frequencies = record.info.get(\"AF\")\n", + " if allele_frequencies is None:\n", + " continue\n", + " for frequency in allele_frequencies:\n", + " if frequency is None or not 0 < frequency <= 1:\n", + " continue\n", + " positions.append(offsets[chromosome] + record.pos)\n", + " frequencies.append(float(frequency))\n", + " kept += 1\n", + " if kept >= variants_per_window:\n", + " break\n", + " if kept >= variants_per_window:\n", + " break\n", + "\n", + " genomic_position_parts.append(np.asarray(positions, dtype=np.float64))\n", + " allele_frequency_parts.append(np.asarray(frequencies, dtype=np.float64))\n", + " chromosome_parts.append(np.full(len(positions), chromosome, dtype=np.float64))\n", + " print(f\"chr{chromosome}: {len(positions):,} variants\")\n", + "\n", + "genomic_position = np.concatenate(genomic_position_parts)\n", + "allele_frequency = np.concatenate(allele_frequency_parts)\n", + "chromosome_number = np.concatenate(chromosome_parts)\n", + "print(f\"{genomic_position.size:,} variants total\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "tick_chromosomes = [*range(1, 23, 2), 22]\n", + "tick_values = [\n", + " offsets[chromosome] + CHROMOSOME_LENGTHS[chromosome] / 2 for chromosome in tick_chromosomes\n", + "]\n", + "tick_labels = [str(chromosome) for chromosome in tick_chromosomes]\n", + "chromosome_boundaries = [offsets[chromosome] for chromosome in range(2, 23)]\n", + "chromosome_palette_position = np.where(\n", + " chromosome_number.astype(np.int64) % 2 == 0,\n", + " 0.56,\n", + " 0.28,\n", + ")\n", + "rare_variant_share = 100.0 * np.count_nonzero(allele_frequency < 1e-2) / allele_frequency.size\n", + "\n", + "x_axis_style = {\n", + " \"axis_color\": \"#8094a5\",\n", + " \"axis_width\": 1.0,\n", + " \"grid_opacity\": 0,\n", + " \"label_color\": \"#19364b\",\n", + " \"label_size\": 14,\n", + " \"tick_color\": \"#8094a5\",\n", + " \"tick_label_color\": \"#40596a\",\n", + " \"tick_label_size\": 11.5,\n", + " \"tick_length\": 5,\n", + " \"tick_width\": 0.8,\n", + "}\n", + "y_axis_style = {\n", + " **x_axis_style,\n", + " \"grid_color\": \"#b9cbd6\",\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.78,\n", + " \"grid_width\": 0.8,\n", + " \"tick_label_size\": 12.5,\n", + "}\n", + "\n", + "chart = xy.scatter_chart(\n", + " xy.y_band(\n", + " 1e-7,\n", + " 1e-2,\n", + " color=\"#0b8f93\",\n", + " opacity=0.045,\n", + " ),\n", + " xy.scatter(\n", + " genomic_position,\n", + " allele_frequency,\n", + " color=chromosome_palette_position,\n", + " color_domain=(0.0, 1.0),\n", + " colormap=\"viridis\",\n", + " size=1.0,\n", + " opacity=0.72,\n", + " density=True,\n", + " ),\n", + " *[\n", + " xy.vline(\n", + " boundary,\n", + " color=\"#9db2c0\",\n", + " width=0.7,\n", + " opacity=0.5,\n", + " style={\"dash\": \"2,5\"},\n", + " )\n", + " for boundary in chromosome_boundaries\n", + " ],\n", + " xy.hline(\n", + " 1e-2,\n", + " color=\"#087e8b\",\n", + " width=1.2,\n", + " opacity=0.82,\n", + " ),\n", + " xy.text(\n", + " float(genomic_position.max()),\n", + " 2.5e-6,\n", + " f\"{rare_variant_share:.1f}%\",\n", + " dx=-8,\n", + " dy=0,\n", + " color=\"#086b75\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 26,\n", + " \"font_weight\": 760,\n", + " \"letter_spacing\": \"-0.015em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " float(genomic_position.max()),\n", + " 1.45e-6,\n", + " \"OF THIS SAMPLE BELOW 1% AF\",\n", + " dx=-8,\n", + " dy=0,\n", + " color=\"#486a78\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.075em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.08,\n", + " 0.965,\n", + " \"gnomAD v4.1 / Allele-frequency atlas\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#102f44\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 24,\n", + " \"font_weight\": 760,\n", + " \"letter_spacing\": \"0.01em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.08,\n", + " 0.912,\n", + " (\n", + " f\"{genomic_position.size:,} VARIANTS · {len(chromosomes)} AUTOSOMES · \"\n", + " f\"{windows_per_chromosome} INDEXED WINDOWS EACH\"\n", + " ),\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#597487\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 600,\n", + " \"letter_spacing\": \"0.085em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.95,\n", + " 0.955,\n", + " \"AF · LOG₁₀ SCALE\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#087e8b\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.09em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"CHROMOSOME\",\n", + " tick_values=tick_values,\n", + " tick_labels=tick_labels,\n", + " tick_label_min_gap=0,\n", + " style=x_axis_style,\n", + " ),\n", + " xy.y_axis(\n", + " label=\"ALTERNATE ALLELE FREQUENCY\",\n", + " label_offset=-28,\n", + " type_=\"log\",\n", + " domain=(1e-7, 1.0),\n", + " tick_values=[1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1.0],\n", + " tick_labels=[\"10⁻⁷\", \"10⁻⁶\", \"10⁻⁵\", \"10⁻⁴\", \"10⁻³\", \"10⁻²\", \"10⁻¹\", \"1\"],\n", + " style=y_axis_style,\n", + " ),\n", + " xy.theme(\n", + " background=\"#f2f8fb\",\n", + " plot_background=\"#fbfdfe\",\n", + " text_color=\"#19364b\",\n", + " grid_color=\"#b9cbd6\",\n", + " axis_color=\"#8094a5\",\n", + " crosshair_color=\"#087e8b\",\n", + " selection_color=\"#075985\",\n", + " selection_fill=\"#0891b226\",\n", + " ),\n", + " styles={\n", + " \"annotation_label\": {\"line_height\": 1.2},\n", + " \"axis_title\": {\"font_weight\": 680, \"letter_spacing\": \"0.055em\"},\n", + " \"tick_label\": {\"font_variant_numeric\": \"tabular-nums\"},\n", + " },\n", + " padding=(96, 48, 78, 116),\n", + " width=1200,\n", + " height=700,\n", + ")\n", + "print(chart.memory_report()[\"canonical_bytes\"], \"canonical bytes\")\n", + "chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/03_pan_ukbb_manhattan.ipynb b/examples/real_world/03_pan_ukbb_manhattan.ipynb new file mode 100644 index 00000000..f53975d4 --- /dev/null +++ b/examples/real_world/03_pan_ukbb_manhattan.ipynb @@ -0,0 +1,421 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# Pan-UK Biobank: a multi-million-variant Manhattan plot\n", + "\n", + "Each Pan-UKBB per-phenotype file contains 28,987,534 variants. This\n", + "notebook reads indexed windows from the standing-height GWAS and plots\n", + "the released `−log10(p)` statistic across all autosomes. The full\n", + "2.0 GB flat file stays remote; only its 2 MB tabix index and requested\n", + "BGZF blocks are transferred.\n", + "\n", + "Increase `PANUKBB_WINDOWS` or `PANUKBB_VARIANTS_PER_WINDOW` for a\n", + "denser run.\n", + "\n", + "**Source:** [Pan-UKBB downloads](https://pan.ukbb.broadinstitute.org/downloads/index.html)\n", + "and [per-phenotype file documentation](https://pan.ukbb.broadinstitute.org/docs/per-phenotype-files/index.html).\n", + "The data are CC BY 4.0; publications should acknowledge Pan-UKBB and\n", + "UK Biobank as requested on the download page.\n", + "\n", + "Install beside XY with `python -m pip install numpy pysam requests xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import pysam\n", + "import requests\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\"))\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "DATA_URL = (\n", + " \"https://pan-ukb-us-east-1.s3.amazonaws.com/sumstats_flat_files/\"\n", + " \"continuous-50-both_sexes-irnt.tsv.bgz\"\n", + ")\n", + "INDEX_URL = (\n", + " \"https://pan-ukb-us-east-1.s3.amazonaws.com/\"\n", + " \"sumstats_flat_files_tabix/\"\n", + " \"continuous-50-both_sexes-irnt.tsv.bgz.tbi\"\n", + ")\n", + "index_path = DATA_DIR / \"continuous-50-both_sexes-irnt.tsv.bgz.tbi\"\n", + "if not index_path.exists():\n", + " response = requests.get(INDEX_URL, timeout=120)\n", + " response.raise_for_status()\n", + " index_path.write_bytes(response.content)\n", + "\n", + "CHROMOSOME_LENGTHS = {\n", + " 1: 249_250_621,\n", + " 2: 243_199_373,\n", + " 3: 198_022_430,\n", + " 4: 191_154_276,\n", + " 5: 180_915_260,\n", + " 6: 171_115_067,\n", + " 7: 159_138_663,\n", + " 8: 146_364_022,\n", + " 9: 141_213_431,\n", + " 10: 135_534_747,\n", + " 11: 135_006_516,\n", + " 12: 133_851_895,\n", + " 13: 115_169_878,\n", + " 14: 107_349_540,\n", + " 15: 102_531_392,\n", + " 16: 90_354_753,\n", + " 17: 81_195_210,\n", + " 18: 78_077_248,\n", + " 19: 59_128_983,\n", + " 20: 63_025_520,\n", + " 21: 48_129_895,\n", + " 22: 51_304_566,\n", + "}\n", + "windows_per_chromosome = int(os.getenv(\"PANUKBB_WINDOWS\", \"8\"))\n", + "variants_per_window = int(os.getenv(\"PANUKBB_VARIANTS_PER_WINDOW\", \"15000\"))\n", + "window_width = int(os.getenv(\"PANUKBB_WINDOW_BP\", \"3000000\"))\n", + "if min(windows_per_chromosome, variants_per_window, window_width) <= 0:\n", + " raise ValueError(\"Pan-UKBB window controls must be positive\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "offsets = {}\n", + "running_offset = 0\n", + "for chromosome, length in CHROMOSOME_LENGTHS.items():\n", + " offsets[chromosome] = running_offset\n", + " running_offset += length\n", + "\n", + "position_parts = []\n", + "significance_parts = []\n", + "chromosome_parts = []\n", + "\n", + "# The immutable quantitative-trait schema starts with:\n", + "# chr, pos, ref, alt, af_meta_hq, beta_meta_hq, se_meta_hq,\n", + "# neglog10_pval_meta_hq. Its plain TSV header is skipped by tabix.\n", + "position_column = 1\n", + "pvalue_column = 7\n", + "\n", + "with pysam.TabixFile(DATA_URL, index=str(index_path)) as summary:\n", + " for chromosome, length in CHROMOSOME_LENGTHS.items():\n", + " centers = (\n", + " (np.arange(windows_per_chromosome, dtype=np.float64) + 0.5)\n", + " * length\n", + " / windows_per_chromosome\n", + " )\n", + " starts = np.clip(\n", + " centers - window_width / 2,\n", + " 0,\n", + " max(0, length - window_width),\n", + " ).astype(np.int64)\n", + " positions = []\n", + " significance = []\n", + " for start in starts:\n", + " kept = 0\n", + " records = summary.fetch(\n", + " str(chromosome),\n", + " int(start),\n", + " int(start + window_width),\n", + " )\n", + " for line in records:\n", + " fields = line.split(\"\\t\")\n", + " value = fields[pvalue_column]\n", + " if value == \"NA\":\n", + " continue\n", + " positions.append(offsets[chromosome] + int(fields[position_column]))\n", + " significance.append(float(value))\n", + " kept += 1\n", + " if kept >= variants_per_window:\n", + " break\n", + "\n", + " position_parts.append(np.asarray(positions, dtype=np.float64))\n", + " significance_parts.append(np.asarray(significance, dtype=np.float64))\n", + " chromosome_parts.append(np.full(len(positions), chromosome, dtype=np.float64))\n", + " print(f\"chr{chromosome}: {len(positions):,} variants\")\n", + "\n", + "genomic_position = np.concatenate(position_parts)\n", + "neglog10_pvalue = np.concatenate(significance_parts)\n", + "chromosome_number = np.concatenate(chromosome_parts)\n", + "print(f\"{genomic_position.size:,} variants total\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "tick_values = [\n", + " offsets[chromosome] + CHROMOSOME_LENGTHS[chromosome] / 2 for chromosome in CHROMOSOME_LENGTHS\n", + "]\n", + "tick_labels = [str(chromosome) for chromosome in CHROMOSOME_LENGTHS]\n", + "chromosome_boundaries = [offsets[chromosome] for chromosome in range(2, 23)]\n", + "\n", + "# Equal-luminance aubergine and ochre distinguish adjacent chromosomes\n", + "# without giving either half of the genome more visual weight.\n", + "aubergine = np.array([116, 77, 104, 255], dtype=np.float32) / 255\n", + "ochre = np.array([129, 84, 47, 255], dtype=np.float32) / 255\n", + "point_rgba = np.empty((chromosome_number.size, 4), dtype=np.float32)\n", + "odd_chromosome = chromosome_number % 2 == 1\n", + "point_rgba[odd_chromosome] = aubergine\n", + "point_rgba[~odd_chromosome] = ochre\n", + "\n", + "chr20_indices = np.flatnonzero(chromosome_number == 20)\n", + "chr20_peak_index = int(chr20_indices[np.argmax(neglog10_pvalue[chr20_indices])])\n", + "other_peak_index = int(np.argmax(np.where(chromosome_number == 20, -np.inf, neglog10_pvalue)))\n", + "peak_specs = (\n", + " (chr20_peak_index, -14, 18, \"end\"),\n", + " (other_peak_index, 14, -24, \"start\"),\n", + ")\n", + "peak_labels = []\n", + "for peak_index, _, _, _ in peak_specs:\n", + " chromosome = int(chromosome_number[peak_index])\n", + " position = int(genomic_position[peak_index] - offsets[chromosome])\n", + " peak_labels.append(\n", + " f\"CHR {chromosome} · {position / 1e6:.1f} MB · −LOG₁₀(P) {neglog10_pvalue[peak_index]:.1f}\" # noqa: RUF001\n", + " )\n", + "\n", + "genome_wide_threshold = -np.log10(5e-8)\n", + "\n", + "chart = xy.scatter_chart(\n", + " xy.scatter(\n", + " genomic_position,\n", + " neglog10_pvalue,\n", + " color=point_rgba,\n", + " size=1.35,\n", + " opacity=0.78,\n", + " density=True,\n", + " ),\n", + " *[\n", + " xy.vline(\n", + " boundary,\n", + " color=\"#b7ab9b\",\n", + " width=0.7,\n", + " opacity=0.46,\n", + " style={\"dash\": \"2,5\"},\n", + " )\n", + " for boundary in chromosome_boundaries\n", + " ],\n", + " xy.hline(\n", + " genome_wide_threshold,\n", + " color=\"#a33a32\",\n", + " width=2.3,\n", + " style={\"dash\": \"7,4\"},\n", + " ),\n", + " xy.text(\n", + " float(genomic_position.max()),\n", + " genome_wide_threshold,\n", + " \"GENOME-WIDE · P = 5 \\u00d7 10⁻⁸\",\n", + " dx=-10,\n", + " dy=-20,\n", + " color=\"#923a34\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"background\": \"#fffdf8f2\",\n", + " \"border\": \"1px solid #d8c7b8\",\n", + " \"border_radius\": 2,\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.045em\",\n", + " \"padding\": \"3px 6px\",\n", + " },\n", + " ),\n", + " xy.scatter(\n", + " genomic_position[[chr20_peak_index, other_peak_index]],\n", + " neglog10_pvalue[[chr20_peak_index, other_peak_index]],\n", + " color=\"#fffdf8\",\n", + " stroke=\"#81542f\",\n", + " stroke_width=2,\n", + " size=6,\n", + " opacity=1,\n", + " ),\n", + " *[\n", + " xy.text(\n", + " float(genomic_position[peak_index]),\n", + " float(neglog10_pvalue[peak_index]),\n", + " label,\n", + " dx=dx,\n", + " dy=dy,\n", + " color=\"#50354a\",\n", + " anchor=anchor,\n", + " style={\n", + " \"background\": \"#fffdf8f2\",\n", + " \"border\": \"1px solid #cdbfaf\",\n", + " \"border_radius\": 2,\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.025em\",\n", + " \"padding\": \"3px 6px\",\n", + " },\n", + " )\n", + " for (peak_index, dx, dy, anchor), label in zip(peak_specs, peak_labels, strict=True)\n", + " ],\n", + " xy.text(\n", + " 0.102,\n", + " 0.965,\n", + " \"PAN-UK BIOBANK / STANDING-HEIGHT GWAS\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#50354a\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 22,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.025em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.102,\n", + " 0.918,\n", + " f\"{genomic_position.size:,} VARIANTS · 22 AUTOSOMES\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#766b60\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 600,\n", + " \"letter_spacing\": \"0.045em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.965,\n", + " 0.918,\n", + " \"PAN-ANCESTRY META-ANALYSIS\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#766b60\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 600,\n", + " \"letter_spacing\": \"0.045em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.535,\n", + " 0.04,\n", + " \"CHROMOSOME\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#342f2a\",\n", + " anchor=\"middle\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 11.5,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.07em\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=None,\n", + " tick_values=tick_values,\n", + " tick_labels=tick_labels,\n", + " style={\n", + " \"grid_opacity\": 0,\n", + " \"axis_color\": \"#6f665c\",\n", + " \"axis_width\": 1.2,\n", + " \"tick_color\": \"#8f8375\",\n", + " \"tick_width\": 1,\n", + " \"tick_length\": 5,\n", + " \"tick_label_color\": \"#4a433c\",\n", + " \"label_color\": \"#342f2a\",\n", + " },\n", + " ),\n", + " xy.y_axis(\n", + " label=\"Association significance, \\u2212log₁₀(P)\",\n", + " label_offset=-16,\n", + " tick_count=7,\n", + " style={\n", + " \"grid_color\": \"#d8d0c3\",\n", + " \"grid_width\": 1,\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.9,\n", + " \"axis_color\": \"#6f665c\",\n", + " \"axis_width\": 1.2,\n", + " \"tick_color\": \"#8f8375\",\n", + " \"tick_label_color\": \"#4a433c\",\n", + " \"label_color\": \"#342f2a\",\n", + " },\n", + " ),\n", + " xy.legend(show=False),\n", + " xy.tooltip(\n", + " title=\"Standing-height association\",\n", + " format={\"x\": \",.0f\", \"y\": \".2f\"},\n", + " ),\n", + " xy.interaction_config(\n", + " crosshair=True,\n", + " wheel_zoom=True,\n", + " box_zoom=True,\n", + " double_click_reset=True,\n", + " ),\n", + " xy.theme(\n", + " background=\"#f3eee4\",\n", + " plot_background=\"#fffdf8\",\n", + " text_color=\"#2c2824\",\n", + " grid_color=\"#d8d0c3\",\n", + " axis_color=\"#6f665c\",\n", + " crosshair_color=\"#a33a32\",\n", + " selection_color=\"#71465f\",\n", + " selection_fill=\"#71465f24\",\n", + " ),\n", + " styles={\n", + " \"axis_title\": {\n", + " \"font_size\": 12,\n", + " \"font_weight\": 650,\n", + " \"letter_spacing\": \"0.03em\",\n", + " },\n", + " \"tick_label\": {\n", + " \"font_size\": 11,\n", + " \"font_variant_numeric\": \"tabular-nums\",\n", + " },\n", + " \"annotation_label\": {\"font_weight\": 650},\n", + " },\n", + " style={\n", + " \"border\": \"1px solid #d4cabc\",\n", + " \"font_family\": \"Georgia, 'Times New Roman', serif\",\n", + " },\n", + " width=1150,\n", + " height=620,\n", + " padding=(100, 42, 94, 122),\n", + ")\n", + "print(chart.memory_report()[\"canonical_bytes\"], \"canonical bytes\")\n", + "chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/04_dukascopy_fx_ticks.ipynb b/examples/real_world/04_dukascopy_fx_ticks.ipynb new file mode 100644 index 00000000..9724206b --- /dev/null +++ b/examples/real_world/04_dukascopy_fx_ticks.ipynb @@ -0,0 +1,411 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# Dukascopy: millions of EUR/USD ticks\n", + "\n", + "This notebook downloads Dukascopy's hourly BI5 tick files, decodes\n", + "bid/ask quotes, and sends every midpoint observation to one XY line.\n", + "The overview is M4-decimated to the viewport; wheel-zooming restores\n", + "detail from the canonical tick series.\n", + "\n", + "The default is five calendar days beginning 2024-01-02. Change\n", + "`DUKASCOPY_START`, `DUKASCOPY_DAYS`, `DUKASCOPY_HOURS`, or\n", + "`DUKASCOPY_SYMBOL` to explore\n", + "a longer period. The price divisor below is correct for EUR/USD and\n", + "most five-decimal FX pairs.\n", + "\n", + "**Source:** [Dukascopy Historical Data Export](https://www.dukascopy.com/swiss/english/marketwatch/historical/).\n", + "\n", + "Install beside XY with `python -m pip install numpy requests xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import lzma\n", + "import os\n", + "import time as time_module\n", + "from datetime import UTC, date, datetime, time, timedelta\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import requests\n", + "from requests.adapters import HTTPAdapter\n", + "from urllib3.util.retry import Retry\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\")) / \"dukascopy\"\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "SYMBOL = os.getenv(\"DUKASCOPY_SYMBOL\", \"EURUSD\").upper()\n", + "START = date.fromisoformat(os.getenv(\"DUKASCOPY_START\", \"2024-01-02\"))\n", + "DAYS = int(os.getenv(\"DUKASCOPY_DAYS\", \"5\"))\n", + "HOURS = int(os.getenv(\"DUKASCOPY_HOURS\", \"24\"))\n", + "PRICE_DIVISOR = float(os.getenv(\"DUKASCOPY_PRICE_DIVISOR\", \"100000\"))\n", + "REQUEST_DELAY = float(os.getenv(\"DUKASCOPY_REQUEST_DELAY\", \"0.2\"))\n", + "if DAYS <= 0 or not 1 <= HOURS <= 24 or PRICE_DIVISOR <= 0 or REQUEST_DELAY < 0:\n", + " raise ValueError(\"use positive days/divisor, 1-24 hours, and a non-negative delay\")\n", + "\n", + "SESSION = requests.Session()\n", + "SESSION.headers[\"User-Agent\"] = \"xy-real-world-notebook/1.0\"\n", + "SESSION.mount(\n", + " \"https://\",\n", + " HTTPAdapter(\n", + " max_retries=Retry(\n", + " total=6,\n", + " backoff_factor=1.0,\n", + " status_forcelist=(429, 500, 502, 503, 504),\n", + " allowed_methods={\"GET\"},\n", + " respect_retry_after_header=True,\n", + " )\n", + " ),\n", + ")\n", + "\n", + "TICK_DTYPE = np.dtype(\n", + " [\n", + " (\"millisecond\", \">u4\"),\n", + " (\"ask\", \">u4\"),\n", + " (\"bid\", \">u4\"),\n", + " (\"ask_volume\", \">f4\"),\n", + " (\"bid_volume\", \">f4\"),\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "def load_hour(day: date, hour: int) -> tuple[np.ndarray, np.ndarray]:\n", + " relative = f\"{SYMBOL}/{day.year}/{day.month - 1:02d}/{day.day:02d}/{hour:02d}h_ticks.bi5\"\n", + " url = f\"https://datafeed.dukascopy.com/datafeed/{relative}\"\n", + " path = DATA_DIR / relative\n", + " if not path.exists():\n", + " response = SESSION.get(url, timeout=120)\n", + " time_module.sleep(REQUEST_DELAY)\n", + " if response.status_code == 404:\n", + " return np.empty(0, dtype=\"datetime64[ms]\"), np.empty(0)\n", + " response.raise_for_status()\n", + " path.parent.mkdir(parents=True, exist_ok=True)\n", + " path.write_bytes(response.content)\n", + "\n", + " compressed = path.read_bytes()\n", + " if not compressed:\n", + " return np.empty(0, dtype=\"datetime64[ms]\"), np.empty(0)\n", + " ticks = np.frombuffer(lzma.decompress(compressed), dtype=TICK_DTYPE)\n", + " hour_start = datetime.combine(\n", + " day,\n", + " time(hour=hour),\n", + " tzinfo=UTC,\n", + " )\n", + " epoch_ms = int(hour_start.timestamp() * 1000)\n", + " timestamps = (epoch_ms + ticks[\"millisecond\"].astype(np.int64)).astype(\"datetime64[ms]\")\n", + " midpoint = (ticks[\"ask\"].astype(np.float64) + ticks[\"bid\"].astype(np.float64)) / (\n", + " 2 * PRICE_DIVISOR\n", + " )\n", + " return timestamps, midpoint\n", + "\n", + "\n", + "timestamp_parts = []\n", + "midpoint_parts = []\n", + "for day_offset in range(DAYS):\n", + " day = START + timedelta(days=day_offset)\n", + " for hour in range(HOURS):\n", + " timestamps, midpoint = load_hour(day, hour)\n", + " if midpoint.size:\n", + " timestamp_parts.append(timestamps)\n", + " midpoint_parts.append(midpoint)\n", + "\n", + "if not midpoint_parts:\n", + " raise RuntimeError(\"the selected interval returned no Dukascopy ticks\")\n", + "\n", + "timestamps = np.concatenate(timestamp_parts)\n", + "midpoint = np.concatenate(midpoint_parts)\n", + "print(f\"{midpoint.size:,} {SYMBOL} ticks from {timestamps[0]} through {timestamps[-1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "pair = f\"{SYMBOL[:3]}/{SYMBOL[3:]}\"\n", + "price_decimals = max(0, int(round(np.log10(PRICE_DIVISOR))))\n", + "price_span = float(np.ptp(midpoint))\n", + "raw_step = max(price_span / 6, 1 / PRICE_DIVISOR)\n", + "step_magnitude = 10 ** np.floor(np.log10(raw_step))\n", + "step_multiplier = next(\n", + " candidate for candidate in (1, 2, 5, 10) if candidate >= raw_step / step_magnitude\n", + ")\n", + "price_step = float(step_multiplier * step_magnitude)\n", + "price_floor = np.floor(midpoint.min() / price_step) * price_step\n", + "price_ceiling = np.ceil(midpoint.max() / price_step) * price_step\n", + "price_ticks = np.arange(price_floor, price_ceiling + price_step / 2, price_step)\n", + "price_tick_labels = [f\"{value:.4f}\" for value in price_ticks]\n", + "session_rail_floor = price_ceiling + price_step * 0.02\n", + "session_rail_label_y = price_ceiling + price_step * 0.20\n", + "\n", + "last_day_start = timestamps[-1].astype(\"datetime64[D]\").astype(\"datetime64[ms]\")\n", + "last_day_index = int(np.searchsorted(timestamps, last_day_start, side=\"left\"))\n", + "day_open = float(midpoint[last_day_index])\n", + "day_change = float(midpoint[-1] - day_open)\n", + "day_change_percent = day_change / day_open * 100\n", + "pip_size = 0.01 if SYMBOL.endswith(\"JPY\") else 0.0001\n", + "day_change_pips = day_change / pip_size\n", + "day_direction = \"UP\" if day_change > 0 else \"DOWN\" if day_change < 0 else \"FLAT\"\n", + "day_change_color = \"#7ce8bc\" if day_change >= 0 else \"#ff9f8c\"\n", + "last_timestamp_ms = int(timestamps[-1].astype(\"datetime64[ms]\").astype(np.int64))\n", + "\n", + "# Subtle regional-session bands make the intraday rhythm legible while\n", + "# preserving every tick in the interactive canonical series.\n", + "session_bands = []\n", + "session_boundaries = []\n", + "session_labels = []\n", + "session_specs = (\n", + " (0, 7, \"ASIA\", \"#17364a\"),\n", + " (7, 13, \"LONDON\", \"#173b32\"),\n", + " (13, 21, \"NEW YORK\", \"#3b3020\"),\n", + ")\n", + "for day_offset in range(DAYS):\n", + " session_start = datetime.combine(START + timedelta(days=day_offset), time())\n", + " for start_hour, end_hour, label, color in session_specs:\n", + " session_bands.append(\n", + " xy.x_band(\n", + " session_start + timedelta(hours=start_hour),\n", + " session_start + timedelta(hours=end_hour),\n", + " color=color,\n", + " opacity=0.12,\n", + " )\n", + " )\n", + " if day_offset == 0:\n", + " label_color = {\"ASIA\": \"#79b9d8\", \"LONDON\": \"#6fd4ad\", \"NEW YORK\": \"#d8b36f\"}[label]\n", + " session_labels.append(\n", + " xy.text(\n", + " session_start + timedelta(hours=(start_hour + end_hour) / 2),\n", + " session_rail_label_y,\n", + " label,\n", + " dx=0,\n", + " dy=0,\n", + " color=label_color,\n", + " anchor=\"middle\",\n", + " style={\n", + " \"font_size\": 10,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.08em\",\n", + " },\n", + " )\n", + " )\n", + " for boundary_hour in (7, 13, 21):\n", + " session_boundaries.append(\n", + " xy.vline(\n", + " session_start + timedelta(hours=boundary_hour),\n", + " color=\"#405268\",\n", + " width=1,\n", + " opacity=0.65,\n", + " )\n", + " )\n", + "\n", + "period_end = START + timedelta(days=DAYS - 1)\n", + "period_label = START.isoformat() if DAYS == 1 else f\"{START.isoformat()} → {period_end.isoformat()}\"\n", + "\n", + "chart = xy.line_chart(\n", + " *session_bands,\n", + " *session_boundaries,\n", + " xy.hline(\n", + " session_rail_floor,\n", + " color=\"#405268\",\n", + " width=1,\n", + " opacity=0.72,\n", + " ),\n", + " *session_labels,\n", + " xy.line(\n", + " timestamps,\n", + " midpoint,\n", + " name=f\"{pair} midpoint\",\n", + " color=\"#42e6a4\",\n", + " width=1.7,\n", + " ),\n", + " xy.line(\n", + " [timestamps[0], timestamps[-1]],\n", + " [midpoint[-1], midpoint[-1]],\n", + " color=\"#8af7cb\",\n", + " width=1,\n", + " opacity=0.62,\n", + " dash=[5, 5],\n", + " ),\n", + " xy.text(\n", + " timestamps[-1],\n", + " float(midpoint[-1]),\n", + " f\"LAST · {midpoint[-1]:.{price_decimals}f}\",\n", + " dx=10,\n", + " dy=0,\n", + " color=\"#8af7cb\",\n", + " anchor=\"start\",\n", + " style={\n", + " \"background\": \"#071521\",\n", + " \"border\": \"1px solid #42e6a480\",\n", + " \"border_radius\": 4,\n", + " \"font_size\": 10.5,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.035em\",\n", + " \"padding\": \"3px 7px\",\n", + " \"vertical_align\": \"middle\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.097,\n", + " 0.965,\n", + " f\"{pair} / INTRADAY TICK TAPE\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#8af7cb\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 20,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.045em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.097,\n", + " 0.918,\n", + " f\"{period_label} · {midpoint.size:,} TICKS · DUKASCOPY HISTORICAL FEED\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#8292a5\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10.5,\n", + " \"font_weight\": 600,\n", + " \"letter_spacing\": \"0.035em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.882,\n", + " 0.959,\n", + " f\"DAY {day_direction} {day_change_percent:+.2f}% · {day_change_pips:+.1f} PIPS\",\n", + " dx=0,\n", + " dy=0,\n", + " color=day_change_color,\n", + " anchor=\"end\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 12,\n", + " \"font_weight\": 750,\n", + " \"letter_spacing\": \"0.025em\",\n", + " \"vertical_align\": \"top\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"TIME · UTC\",\n", + " type_=\"time\",\n", + " domain=(int(timestamps[0].astype(np.int64)), last_timestamp_ms),\n", + " style={\n", + " \"grid_color\": \"#223044\",\n", + " \"grid_width\": 1,\n", + " \"grid_opacity\": 0.45,\n", + " \"axis_color\": \"#55657a\",\n", + " \"axis_width\": 1.2,\n", + " \"tick_color\": \"#55657a\",\n", + " \"tick_label_color\": \"#a8b5c5\",\n", + " \"label_color\": \"#8af7cb\",\n", + " },\n", + " ),\n", + " xy.y_axis(\n", + " label=f\"{pair} · MID\",\n", + " label_offset=-28,\n", + " domain=(price_floor - price_step * 0.2, price_ceiling + price_step * 0.38),\n", + " tick_values=price_ticks,\n", + " tick_labels=price_tick_labels,\n", + " style={\n", + " \"grid_color\": \"#26354a\",\n", + " \"grid_width\": 1,\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.8,\n", + " \"axis_color\": \"#55657a\",\n", + " \"axis_width\": 1.2,\n", + " \"tick_color\": \"#55657a\",\n", + " \"tick_label_color\": \"#c5d0dc\",\n", + " \"label_color\": \"#8af7cb\",\n", + " },\n", + " ),\n", + " xy.legend(show=False),\n", + " xy.tooltip(\n", + " title=f\"{pair} midpoint\",\n", + " format={\"y\": f\".{price_decimals}f\"},\n", + " ),\n", + " xy.interaction_config(\n", + " crosshair=True,\n", + " wheel_zoom=True,\n", + " box_zoom=True,\n", + " double_click_reset=True,\n", + " ),\n", + " xy.theme(\n", + " background=\"#05080d\",\n", + " plot_background=\"#08111b\",\n", + " text_color=\"#c8d3df\",\n", + " grid_color=\"#26354a\",\n", + " axis_color=\"#55657a\",\n", + " crosshair_color=\"#f0c96a\",\n", + " selection_color=\"#42e6a4\",\n", + " selection_fill=\"#42e6a424\",\n", + " ),\n", + " styles={\n", + " \"axis_title\": {\"font_weight\": 700, \"letter_spacing\": \"0.06em\"},\n", + " \"tick_label\": {\n", + " \"font_family\": \"ui-monospace, monospace\",\n", + " \"font_variant_numeric\": \"tabular-nums\",\n", + " },\n", + " \"annotation_label\": {\"font_family\": \"ui-monospace, monospace\"},\n", + " \"tooltip\": {\n", + " \"background\": \"#071019\",\n", + " \"color\": \"#d9e4ee\",\n", + " \"border\": \"1px solid #2a4b4b\",\n", + " \"border_radius\": 3,\n", + " \"font_family\": \"ui-monospace, monospace\",\n", + " },\n", + " },\n", + " style={\n", + " \"border\": \"1px solid #172838\",\n", + " \"font_family\": \"ui-monospace, SFMono-Regular, Menlo, monospace\",\n", + " },\n", + " width=1150,\n", + " height=560,\n", + " padding=(98, 142, 68, 116),\n", + ")\n", + "payload = chart.figure().build_payload()[0]\n", + "print(\"render tier:\", payload[\"traces\"][0][\"tier\"])\n", + "chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/05_ligo_gw150914_strain.ipynb b/examples/real_world/05_ligo_gw150914_strain.ipynb new file mode 100644 index 00000000..a282275b --- /dev/null +++ b/examples/real_world/05_ligo_gw150914_strain.ipynb @@ -0,0 +1,592 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# LIGO GW150914: an hour of gravitational-wave strain\n", + "\n", + "GWOSC publishes 4096-second strain files at 4096 Hz and 16384 Hz.\n", + "That is 16.8 million or 67.1 million samples in a single line. The first\n", + "chart keeps the complete raw record so XY can exercise M4 decimation. The\n", + "event detail then extracts 320 ms around `t = 0` and applies a tapered\n", + "35–350 Hz FFT bandpass so the chirp is legible without presenting the\n", + "result as a whitened or template-matched detection product.\n", + "\n", + "The default 4 kHz HDF5 file is about 134 MB. Set\n", + "`LIGO_SAMPLE_RATE=16384` for the full-rate, roughly 536 MB file.\n", + "\n", + "**Source:** [GWOSC GW150914 event page](https://gwosc.org/events/GW150914/),\n", + "the [official Hanford template reconstruction](https://gwosc.org/GW150914data/P150914/fig2-unfiltered-template-reconstruction-H.txt),\n", + "and [GWOSC URL lookup documentation](https://gwosc.readthedocs.io/en/stable/locate.html).\n", + "GWOSC event data are released under CC BY 4.0; follow the\n", + "acknowledgement guidance linked from the event page.\n", + "\n", + "Install beside XY with\n", + "`python -m pip install numpy requests h5py gwosc xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import h5py\n", + "import numpy as np\n", + "import requests\n", + "from gwosc.locate import get_event_urls\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\")) / \"gwosc\"\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "EVENT = \"GW150914\"\n", + "EVENT_GPS = 1_126_259_462.4\n", + "DETECTOR = os.getenv(\"LIGO_DETECTOR\", \"H1\")\n", + "SAMPLE_RATE = int(os.getenv(\"LIGO_SAMPLE_RATE\", \"4096\"))\n", + "DURATION = 4096\n", + "DETECTOR_SITES = {\"H1\": \"HANFORD\", \"L1\": \"LIVINGSTON\"}\n", + "if SAMPLE_RATE not in {4096, 16384}:\n", + " raise ValueError(\"LIGO_SAMPLE_RATE must be 4096 or 16384\")\n", + "if DETECTOR not in DETECTOR_SITES:\n", + " raise ValueError(\"LIGO_DETECTOR must be H1 or L1\")\n", + "\n", + "urls = get_event_urls(\n", + " EVENT,\n", + " catalog=\"GWTC-1-confident\",\n", + " version=3,\n", + " detector=DETECTOR,\n", + " duration=DURATION,\n", + " sample_rate=SAMPLE_RATE,\n", + " format=\"hdf5\",\n", + ")\n", + "if not urls:\n", + " raise RuntimeError(\"GWOSC returned no matching strain file\")\n", + "url = urls[0]\n", + "hdf5_path = DATA_DIR / Path(url).name\n", + "if not hdf5_path.exists():\n", + " with requests.get(\n", + " url,\n", + " stream=True,\n", + " timeout=(30, 3600),\n", + " ) as response:\n", + " response.raise_for_status()\n", + " partial = hdf5_path.with_suffix(\".hdf5.part\")\n", + " with partial.open(\"wb\") as output:\n", + " for chunk in response.iter_content(chunk_size=4 * 1024 * 1024):\n", + " output.write(chunk)\n", + " partial.replace(hdf5_path)\n", + "\n", + "reconstruction_url = (\n", + " \"https://gwosc.org/GW150914data/P150914/fig2-unfiltered-template-reconstruction-H.txt\"\n", + ")\n", + "reconstruction_path = DATA_DIR / Path(reconstruction_url).name\n", + "if not reconstruction_path.exists():\n", + " with requests.get(reconstruction_url, stream=True, timeout=(30, 300)) as response:\n", + " response.raise_for_status()\n", + " partial = reconstruction_path.with_suffix(\".txt.part\")\n", + " with partial.open(\"wb\") as output:\n", + " for chunk in response.iter_content(chunk_size=1024 * 1024):\n", + " output.write(chunk)\n", + " partial.replace(reconstruction_path)\n", + "\n", + "print(f\"cached strain file: {hdf5_path}\")\n", + "print(f\"cached reconstruction: {reconstruction_path}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "with h5py.File(hdf5_path, \"r\") as data:\n", + " strain = np.asarray(data[\"strain\"][\"Strain\"], dtype=np.float64)\n", + " gps_start = float(np.asarray(data[\"meta\"][\"GPSstart\"]))\n", + " x_spacing = float(\n", + " data[\"strain\"][\"Strain\"].attrs.get(\n", + " \"Xspacing\",\n", + " 1 / SAMPLE_RATE,\n", + " )\n", + " )\n", + "\n", + "seconds_from_event = gps_start + np.arange(strain.size, dtype=np.float64) * x_spacing - EVENT_GPS\n", + "print(\n", + " f\"{strain.size:,} samples · {1 / x_spacing:,.0f} Hz · \"\n", + " f\"{strain.nbytes / 2**20:,.1f} MiB canonical strain\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "full_x_bounds = (float(seconds_from_event[0]), float(seconds_from_event[-1]))\n", + "full_y_bounds = (float(np.nanmin(strain)), float(np.nanmax(strain)))\n", + "\n", + "SIGNAL_FONT = \"IBM Plex Mono, ui-monospace, monospace\"\n", + "SIGNAL_AXIS_STYLE = {\n", + " \"grid_color\": \"#173640\",\n", + " \"grid_width\": 1,\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.62,\n", + " \"axis_color\": \"#3d7380\",\n", + " \"axis_width\": 1,\n", + " \"tick_color\": \"#43f4ff\",\n", + " \"tick_width\": 1,\n", + " \"tick_length\": 5,\n", + " \"tick_label_color\": \"#a9ccd2\",\n", + " \"label_color\": \"#d7f9fc\",\n", + " \"tick_label_size\": 12,\n", + " \"label_size\": 12,\n", + "}\n", + "SIGNAL_THEME = xy.theme(\n", + " background=\"#03070b\",\n", + " plot_background=\"#061116\",\n", + " text_color=\"#d7f9fc\",\n", + " grid_color=\"#173640\",\n", + " axis_color=\"#3d7380\",\n", + " crosshair_color=\"#ff4f91\",\n", + " selection_color=\"#43f4ff\",\n", + " selection_fill=\"#43f4ff22\",\n", + ")\n", + "SIGNAL_STYLES = {\n", + " \"tick_label\": {\"font_family\": SIGNAL_FONT},\n", + " \"axis_title\": {\"font_family\": SIGNAL_FONT, \"letter_spacing\": \"0.05em\"},\n", + " \"annotation_label\": {\"font_family\": SIGNAL_FONT},\n", + " \"tooltip\": {\n", + " \"background\": \"#07151a\",\n", + " \"color\": \"#d7f9fc\",\n", + " \"border\": \"1px solid #2b7680\",\n", + " \"border_radius\": 5,\n", + " \"font_family\": SIGNAL_FONT,\n", + " },\n", + "}\n", + "SIGNAL_CARD = {\"border\": \"1px solid #173640\", \"font_family\": SIGNAL_FONT}\n", + "\n", + "overview_chart = xy.line_chart(\n", + " xy.line(\n", + " seconds_from_event,\n", + " strain,\n", + " name=f\"{DETECTOR} · raw strain\",\n", + " color=\"#43f4ff\",\n", + " width=1.0,\n", + " opacity=0.72,\n", + " ),\n", + " xy.vline(0, color=\"#ff4f91\", width=2, opacity=0.95, style={\"dash\": \"6,5\"}),\n", + " xy.text(\n", + " 0.111,\n", + " 0.925,\n", + " \"4096-SECOND RAW RECORD\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#f1fdff\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 20,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.04em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.111,\n", + " 0.85,\n", + " f\"{EVENT} / {DETECTOR} / {1 / x_spacing:,.0f} HZ / {strain.size / 1e6:.1f}M SAMPLES\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#79a7af\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.09em\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"TIME FROM EVENT / seconds\",\n", + " domain=full_x_bounds,\n", + " bounds=full_x_bounds,\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.y_axis(\n", + " label=\"RAW DETECTOR STRAIN h(t)\",\n", + " label_offset=-16,\n", + " bounds=full_y_bounds,\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.tooltip(title=f\"{DETECTOR} RAW STRAIN\", format={\"x\": \"+.1f\", \"y\": \".3e\"}),\n", + " xy.legend(show=False),\n", + " xy.interaction_config(crosshair=True, wheel_zoom=True, box_zoom=True),\n", + " SIGNAL_THEME,\n", + " styles=SIGNAL_STYLES,\n", + " style=SIGNAL_CARD,\n", + " width=1150,\n", + " height=360,\n", + " padding=(82, 32, 66, 128),\n", + ")\n", + "overview_payload = overview_chart.figure().build_payload()[0]\n", + "print(\"overview render tier:\", overview_payload[\"traces\"][0][\"tier\"])\n", + "overview_chart" + ] + }, + { + "cell_type": "markdown", + "id": "dc0ea1b7d35d4e1ebff9e5ef733cff00", + "metadata": {}, + "source": [ + "## Event detail: a truthful signal-first view\n", + "\n", + "A Hann-tapered segment with a linearly rolled frequency response suppresses\n", + "frequencies below 25 Hz, passes 35–350 Hz, and rolls off by 400 Hz. The\n", + "chart labels that processing directly and keeps the GWOSC event reference\n", + "at `t = 0` as the visual anchor.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d56d14a6bbf74725a3806ab939010e00", + "metadata": {}, + "outputs": [], + "source": [ + "detail_mask = (seconds_from_event >= -0.24) & (seconds_from_event <= 0.08)\n", + "detail_time = seconds_from_event[detail_mask]\n", + "detail_raw = strain[detail_mask]\n", + "detail_centered = np.nan_to_num(detail_raw - np.nanmean(detail_raw))\n", + "detail_frequency = np.fft.rfftfreq(detail_centered.size, d=x_spacing)\n", + "low_rolloff = np.clip((detail_frequency - 25) / 10, 0, 1)\n", + "high_rolloff = np.clip((400 - detail_frequency) / 50, 0, 1)\n", + "bandpass_response = np.minimum(low_rolloff, high_rolloff)\n", + "bandpassed_strain = (\n", + " np.fft.irfft(\n", + " np.fft.rfft(detail_centered * np.hanning(detail_centered.size)) * bandpass_response,\n", + " n=detail_centered.size,\n", + " )\n", + " * 1e21\n", + ")\n", + "\n", + "visible_domain = (-0.18, 0.05)\n", + "visible_mask = (detail_time >= visible_domain[0]) & (detail_time <= visible_domain[1])\n", + "visible_strain = bandpassed_strain[visible_mask]\n", + "detail_y_bound = max(4.0, np.ceil(np.nanpercentile(np.abs(visible_strain), 99.8) * 2) / 2)\n", + "late_mask = (detail_time >= -0.09) & (detail_time <= -0.05)\n", + "late_indices = np.flatnonzero(late_mask)\n", + "late_index = late_indices[np.argmax(bandpassed_strain[late_indices])]\n", + "\n", + "bandpassed_chart = xy.line_chart(\n", + " xy.line(\n", + " detail_time,\n", + " bandpassed_strain,\n", + " name=f\"{DETECTOR} · 35-350 Hz bandpass glow\",\n", + " color=\"#43f4ff\",\n", + " width=5.0,\n", + " opacity=0.10,\n", + " ),\n", + " xy.line(\n", + " detail_time,\n", + " bandpassed_strain,\n", + " name=f\"{DETECTOR} · 35-350 Hz bandpass\",\n", + " color=\"#55e8ff\",\n", + " width=1.35,\n", + " opacity=0.96,\n", + " ),\n", + " xy.hline(0, color=\"#2b7680\", width=1, opacity=0.68),\n", + " xy.vline(0, color=\"#ff4f91\", width=2.25, opacity=0.98, style={\"dash\": \"7,5\"}),\n", + " xy.text(\n", + " 0,\n", + " detail_y_bound * 0.88,\n", + " \"GWOSC EVENT REFERENCE / t = 0\",\n", + " dx=-12,\n", + " dy=-6,\n", + " color=\"#ff4f91\",\n", + " anchor=\"end\",\n", + " style={\"font_size\": 12, \"font_weight\": 700, \"letter_spacing\": \"0.06em\"},\n", + " ),\n", + " xy.callout(\n", + " float(detail_time[late_index]),\n", + " float(bandpassed_strain[late_index]),\n", + " \"LATE INSPIRAL\",\n", + " dx=-86,\n", + " dy=-46,\n", + " color=\"#8ddfeb\",\n", + " width=1.25,\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 12,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#d7f9fc\",\n", + " \"letter_spacing\": \"0.06em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.136,\n", + " 0.94,\n", + " EVENT,\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#f1fdff\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 26,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.025em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.136,\n", + " 0.895,\n", + " f\"{DETECTOR_SITES[DETECTOR]} {DETECTOR} / {1 / x_spacing:,.0f} HZ / 35-350 HZ BANDPASS\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#79a7af\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.09em\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"TIME FROM GWOSC EVENT REFERENCE / seconds\",\n", + " domain=visible_domain,\n", + " bounds=(float(detail_time[0]), float(detail_time[-1])),\n", + " tick_values=[-0.15, -0.10, -0.05, 0, 0.05],\n", + " tick_labels=[\"-0.15\", \"-0.10\", \"-0.05\", \"0\", \"+0.05\"],\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.y_axis(\n", + " label=\"BANDPASSED STRAIN / \\u00d7 10\\u207b\\u00b2\\u00b9\",\n", + " label_offset=-18,\n", + " domain=(-detail_y_bound, detail_y_bound),\n", + " bounds=(float(np.nanmin(bandpassed_strain)), float(np.nanmax(bandpassed_strain))),\n", + " tick_values=[-4, -2, 0, 2, 4],\n", + " tick_labels=[\"-4\", \"-2\", \"0\", \"+2\", \"+4\"],\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.tooltip(\n", + " title=f\"{DETECTOR} 35-350 Hz BANDPASS\",\n", + " format={\"x\": \"+.5f\", \"y\": \"+.3f\"},\n", + " ),\n", + " xy.legend(show=False),\n", + " xy.interaction_config(\n", + " hover=True,\n", + " crosshair=True,\n", + " wheel_zoom=True,\n", + " box_zoom=True,\n", + " double_click_reset=True,\n", + " ),\n", + " SIGNAL_THEME,\n", + " styles=SIGNAL_STYLES,\n", + " style=SIGNAL_CARD,\n", + " width=1150,\n", + " height=620,\n", + " padding=(96, 36, 80, 156),\n", + ")\n", + "detail_payload = bandpassed_chart.figure().build_payload()[0]\n", + "print(\"event-detail render tier:\", detail_payload[\"traces\"][1][\"tier\"])\n", + "bandpassed_chart" + ] + }, + { + "cell_type": "markdown", + "id": "e4e2eebea324443f93d03edee3dbe786", + "metadata": {}, + "source": [ + "## Reconstructed waveform: the signal as the visual hero\n", + "\n", + "GWOSC also publishes the numerical-relativity reference and reconstructed\n", + "Hanford strain used for its event figure. This final view peak-aligns the\n", + "reconstructed column, preserves its documented `strain × 10²¹` scale, and\n", + "labels the inspiral-to-ringdown story directly.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5ae90e724c04bff8c5eb22b9b25489f", + "metadata": {}, + "outputs": [], + "source": [ + "reconstruction_seconds, _nr_strain, reconstructed_strain = np.loadtxt(\n", + " reconstruction_path,\n", + " comments=\"#\",\n", + " unpack=True,\n", + ")\n", + "peak_index = int(np.argmax(np.abs(reconstructed_strain)))\n", + "reconstruction_seconds = reconstruction_seconds - reconstruction_seconds[peak_index]\n", + "inspiral_index = int(np.argmin(np.abs(reconstruction_seconds + 0.028)))\n", + "ringdown_index = int(np.argmin(np.abs(reconstruction_seconds - 0.031)))\n", + "reconstruction_bounds = (\n", + " float(np.nanmin(reconstructed_strain)),\n", + " float(np.nanmax(reconstructed_strain)),\n", + ")\n", + "\n", + "chart = xy.line_chart(\n", + " xy.line(\n", + " reconstruction_seconds,\n", + " reconstructed_strain,\n", + " name=\"H1 reconstructed strain · glow\",\n", + " color=\"#43f4ff\",\n", + " width=5.5,\n", + " opacity=0.10,\n", + " ),\n", + " xy.line(\n", + " reconstruction_seconds,\n", + " reconstructed_strain,\n", + " name=\"H1 reconstructed strain\",\n", + " color=\"#55e8ff\",\n", + " width=1.45,\n", + " opacity=0.98,\n", + " ),\n", + " xy.hline(0, color=\"#2b7680\", width=1, opacity=0.68),\n", + " xy.vline(0, color=\"#ff4f91\", width=2.25, opacity=0.98, style={\"dash\": \"7,5\"}),\n", + " xy.text(\n", + " 0,\n", + " 1.40,\n", + " \"PEAK STRAIN / t = 0\",\n", + " dx=-12,\n", + " dy=-6,\n", + " color=\"#ff4f91\",\n", + " anchor=\"end\",\n", + " style={\"font_size\": 13, \"font_weight\": 700, \"letter_spacing\": \"0.06em\"},\n", + " ),\n", + " xy.callout(\n", + " float(reconstruction_seconds[inspiral_index]),\n", + " float(reconstructed_strain[inspiral_index]),\n", + " \"AMPLITUDE + FREQUENCY RISE\",\n", + " dx=-132,\n", + " dy=-58,\n", + " color=\"#8ddfeb\",\n", + " width=1.25,\n", + " anchor=\"start\",\n", + " style={\n", + " \"font_size\": 13,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#d7f9fc\",\n", + " \"letter_spacing\": \"0.05em\",\n", + " },\n", + " ),\n", + " xy.callout(\n", + " float(reconstruction_seconds[ringdown_index]),\n", + " float(reconstructed_strain[ringdown_index]),\n", + " \"RINGDOWN\",\n", + " dx=-66,\n", + " dy=-42,\n", + " color=\"#ff6fa7\",\n", + " width=1.25,\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 13,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#ffd3e4\",\n", + " \"letter_spacing\": \"0.07em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.137,\n", + " 0.94,\n", + " EVENT,\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#f1fdff\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 26,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.025em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.137,\n", + " 0.895,\n", + " \"HANFORD H1 / OFFICIAL GWOSC TEMPLATE RECONSTRUCTION / PEAK-ALIGNED\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#79a7af\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.085em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " 0.963,\n", + " 0.925,\n", + " \"14 SEP 2015 / 09:50:45 UTC\",\n", + " dx=0,\n", + " dy=0,\n", + " color=\"#ff6fa7\",\n", + " anchor=\"end\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.075em\",\n", + " },\n", + " ),\n", + " xy.x_axis(\n", + " label=\"TIME FROM PEAK STRAIN / seconds\",\n", + " domain=(-0.15, 0.055),\n", + " bounds=(float(reconstruction_seconds[0]), float(reconstruction_seconds[-1])),\n", + " tick_values=[-0.15, -0.10, -0.05, 0, 0.05],\n", + " tick_labels=[\"-0.15\", \"-0.10\", \"-0.05\", \"0\", \"+0.05\"],\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.y_axis(\n", + " label=\"RECONSTRUCTED STRAIN / \\u00d7 10\\u207b\\u00b2\\u00b9\",\n", + " label_offset=-18,\n", + " domain=(-1.55, 1.55),\n", + " bounds=reconstruction_bounds,\n", + " tick_values=[-1.0, -0.5, 0, 0.5, 1.0],\n", + " tick_labels=[\"-1.0\", \"-0.5\", \"0\", \"+0.5\", \"+1.0\"],\n", + " style=SIGNAL_AXIS_STYLE,\n", + " ),\n", + " xy.tooltip(\n", + " title=\"H1 GWOSC TEMPLATE RECONSTRUCTION\",\n", + " format={\"x\": \"+.5f\", \"y\": \"+.3f\"},\n", + " ),\n", + " xy.legend(show=False),\n", + " xy.interaction_config(\n", + " hover=True,\n", + " crosshair=True,\n", + " wheel_zoom=True,\n", + " box_zoom=True,\n", + " double_click_reset=True,\n", + " ),\n", + " SIGNAL_THEME,\n", + " styles=SIGNAL_STYLES,\n", + " style=SIGNAL_CARD,\n", + " width=1200,\n", + " height=700,\n", + " padding=(102, 44, 90, 164),\n", + ")\n", + "chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/06_nyc_taxi_density.ipynb b/examples/real_world/06_nyc_taxi_density.ipynb new file mode 100644 index 00000000..40a2a60e --- /dev/null +++ b/examples/real_world/06_nyc_taxi_density.ipynb @@ -0,0 +1,517 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": {}, + "source": [ + "# NYC TLC taxi pickups: street-grid density without a basemap\n", + "\n", + "NYC OpenData's 2015 TLC trip table retains pickup longitude and latitude.\n", + "The SODA API returns only those three requested columns, then\n", + "XY renders the same rows three ways: automatic scatter density,\n", + "explicit hexbin aggregation, and 24 hourly facets. A local projection at\n", + "40.75° N corrects longitude scale, so Manhattan's street grid emerges from\n", + "pickup points alone without a basemap.\n", + "\n", + "The default reads one million valid pickups in 50,000-row pages.\n", + "Increase `TLC_MAX_ROWS` to scale the example; each page is cached so an\n", + "interrupted run resumes without repeating completed downloads.\n", + "\n", + "**Source:** [NYC TLC Trip Record Data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page)\n", + "and the [2015 Yellow Taxi Trip Data table](https://data.cityofnewyork.us/d/2yzn-sicd).\n", + "TLC notes that trip records are published as submitted and may contain\n", + "inaccuracies.\n", + "\n", + "Install beside XY with `python -m pip install numpy requests xy`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import time\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import requests\n", + "from requests.adapters import HTTPAdapter\n", + "from urllib3.util.retry import Retry\n", + "\n", + "import xy\n", + "\n", + "DATA_DIR = Path(os.getenv(\"XY_REAL_WORLD_DATA\", \"data\")) / \"nyc-tlc\"\n", + "DATA_DIR.mkdir(parents=True, exist_ok=True)\n", + "\n", + "MAX_ROWS = int(os.getenv(\"TLC_MAX_ROWS\", \"1000000\"))\n", + "PAGE_SIZE = int(os.getenv(\"TLC_PAGE_SIZE\", \"50000\"))\n", + "REQUEST_DELAY = float(os.getenv(\"TLC_REQUEST_DELAY\", \"0.1\"))\n", + "if MAX_ROWS <= 0 or not 1 <= PAGE_SIZE <= 50_000 or REQUEST_DELAY < 0:\n", + " raise ValueError(\"use positive rows, a page size <= 50,000, and a non-negative delay\")\n", + "\n", + "SESSION = requests.Session()\n", + "SESSION.headers[\"User-Agent\"] = \"xy-real-world-notebook/1.0\"\n", + "app_token = os.getenv(\"SOCRATA_APP_TOKEN\")\n", + "if app_token:\n", + " SESSION.headers[\"X-App-Token\"] = app_token\n", + "SESSION.mount(\n", + " \"https://\",\n", + " HTTPAdapter(\n", + " max_retries=Retry(\n", + " total=6,\n", + " backoff_factor=1.0,\n", + " status_forcelist=(429, 500, 502, 503, 504),\n", + " allowed_methods={\"GET\"},\n", + " respect_retry_after_header=True,\n", + " )\n", + " ),\n", + ")\n", + "\n", + "API_URL = \"https://data.cityofnewyork.us/resource/2yzn-sicd.csv\"\n", + "parts = []\n", + "for offset in range(0, MAX_ROWS, PAGE_SIZE):\n", + " limit = min(PAGE_SIZE, MAX_ROWS - offset)\n", + " page_path = DATA_DIR / f\"pickups-{offset:09d}-{limit:05d}.csv\"\n", + " if not page_path.exists():\n", + " response = SESSION.get(\n", + " API_URL,\n", + " params={\n", + " \"$select\": \"pickup_longitude,pickup_latitude,pickup_datetime\",\n", + " \"$where\": (\n", + " \"pickup_longitude between -74.10 and -73.70 \"\n", + " \"and pickup_latitude between 40.55 and 40.95\"\n", + " ),\n", + " \"$order\": \":id\",\n", + " \"$limit\": limit,\n", + " \"$offset\": offset,\n", + " },\n", + " stream=True,\n", + " timeout=(30, 600),\n", + " )\n", + " response.raise_for_status()\n", + " partial = page_path.with_suffix(\".csv.part\")\n", + " with partial.open(\"wb\") as output:\n", + " for chunk in response.iter_content(chunk_size=1024 * 1024):\n", + " output.write(chunk)\n", + " partial.replace(page_path)\n", + " time.sleep(REQUEST_DELAY)\n", + "\n", + " page = np.loadtxt(\n", + " page_path,\n", + " delimiter=\",\",\n", + " quotechar='\"',\n", + " skiprows=1,\n", + " dtype=[(\"longitude\", \"f8\"), (\"latitude\", \"f8\"), (\"pickup_time\", \"U23\")],\n", + " encoding=\"utf-8\",\n", + " ndmin=1,\n", + " )\n", + " if page.size == 0:\n", + " break\n", + " parts.append(page)\n", + " print(f\"loaded {sum(part.size for part in parts):,} pickups\")\n", + " if page.size < limit:\n", + " break\n", + "\n", + "if not parts:\n", + " raise RuntimeError(\"NYC OpenData returned no pickup rows\")\n", + "pickups = np.concatenate(parts)\n", + "longitude = pickups[\"longitude\"]\n", + "latitude = pickups[\"latitude\"]\n", + "pickup_time = pickups[\"pickup_time\"].astype(\"datetime64[ms]\")\n", + "hour = (pickup_time.astype(\"datetime64[h]\").astype(np.int64) % 24).astype(np.int16)\n", + "print(f\"{longitude.size:,} valid pickup locations\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": {}, + "outputs": [], + "source": [ + "MAP_LONGITUDE_DOMAIN = (-74.035, -73.765)\n", + "MAP_Y_DOMAIN = (40.635, 40.865)\n", + "MAP_REFERENCE_LONGITUDE = -73.90\n", + "MAP_REFERENCE_LATITUDE = 40.75\n", + "MAP_LONGITUDE_SCALE = float(np.cos(np.deg2rad(MAP_REFERENCE_LATITUDE)))\n", + "HERO_WIDTH = 1200\n", + "HERO_HEIGHT = 700\n", + "HERO_PADDING = (116, 168, 76, 148)\n", + "\n", + "\n", + "def project_longitude(value):\n", + " return (np.asarray(value, dtype=np.float64) - MAP_REFERENCE_LONGITUDE) * MAP_LONGITUDE_SCALE\n", + "\n", + "\n", + "projected_longitude = project_longitude(longitude)\n", + "MAP_X_TICKS = [\n", + " float(project_longitude(value)) for value in [-74.10, -74.00, -73.90, -73.80, -73.70]\n", + "]\n", + "MAP_X_LABELS = [\"74.10° W\", \"74.00° W\", \"73.90° W\", \"73.80° W\", \"73.70° W\"]\n", + "MAP_Y_TICKS = [40.65, 40.70, 40.75, 40.80, 40.85]\n", + "MAP_Y_LABELS = [\"40.65° N\", \"40.70° N\", \"40.75° N\", \"40.80° N\", \"40.85° N\"]\n", + "\n", + "# Derive the hero domain from its exact inner plot dimensions. One projected\n", + "# longitude degree therefore occupies the same number of screen pixels as one\n", + "# latitude degree, even on the wide gallery canvas.\n", + "MAP_PLOT_WIDTH = HERO_WIDTH - HERO_PADDING[1] - HERO_PADDING[3]\n", + "MAP_PLOT_HEIGHT = HERO_HEIGHT - HERO_PADDING[0] - HERO_PADDING[2]\n", + "MAP_PLOT_ASPECT = MAP_PLOT_WIDTH / MAP_PLOT_HEIGHT\n", + "MAP_X_SPAN = (MAP_Y_DOMAIN[1] - MAP_Y_DOMAIN[0]) * MAP_PLOT_ASPECT\n", + "MAP_X_DOMAIN = (-MAP_X_SPAN / 2, MAP_X_SPAN / 2)\n", + "\n", + "\n", + "def nyc_x_axis(*, compact=False):\n", + " return xy.x_axis(\n", + " label=None if compact else \"LONGITUDE / WEST\",\n", + " domain=MAP_X_DOMAIN,\n", + " bounds=MAP_X_DOMAIN,\n", + " tick_values=None if compact else MAP_X_TICKS,\n", + " tick_labels=None if compact else MAP_X_LABELS,\n", + " tick_label_strategy=\"none\" if compact else None,\n", + " style={\n", + " \"grid_color\": \"#193743\",\n", + " \"grid_width\": 1,\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.72,\n", + " \"axis_color\": \"#496979\",\n", + " \"axis_width\": 1,\n", + " \"tick_color\": \"#e6b969\",\n", + " \"tick_width\": 1,\n", + " \"tick_length\": 4,\n", + " \"tick_label_color\": \"#b9cbd0\",\n", + " \"label_color\": \"#f3d59a\",\n", + " \"tick_label_size\": 11,\n", + " \"label_size\": 12,\n", + " },\n", + " )\n", + "\n", + "\n", + "def nyc_y_axis(*, compact=False):\n", + " return xy.y_axis(\n", + " label=None if compact else \"LATITUDE / NORTH\",\n", + " label_offset=-18,\n", + " domain=MAP_Y_DOMAIN,\n", + " bounds=MAP_Y_DOMAIN,\n", + " tick_values=None if compact else MAP_Y_TICKS,\n", + " tick_labels=None if compact else MAP_Y_LABELS,\n", + " tick_label_strategy=\"none\" if compact else None,\n", + " style={\n", + " \"grid_color\": \"#193743\",\n", + " \"grid_width\": 1,\n", + " \"grid_dash\": \"dotted\",\n", + " \"grid_opacity\": 0.72,\n", + " \"axis_color\": \"#496979\",\n", + " \"axis_width\": 1,\n", + " \"tick_color\": \"#e6b969\",\n", + " \"tick_width\": 1,\n", + " \"tick_length\": 4,\n", + " \"tick_label_color\": \"#b9cbd0\",\n", + " \"label_color\": \"#f3d59a\",\n", + " \"tick_label_size\": 11,\n", + " \"label_size\": 12,\n", + " },\n", + " )\n", + "\n", + "\n", + "def nyc_theme():\n", + " return xy.theme(\n", + " background=\"#08131b\",\n", + " plot_background=\"#071821\",\n", + " text_color=\"#edf1e8\",\n", + " grid_color=\"#193743\",\n", + " axis_color=\"#496979\",\n", + " crosshair_color=\"#ffca68\",\n", + " selection_color=\"#ffb000\",\n", + " selection_fill=\"#ffb00024\",\n", + " )\n", + "\n", + "\n", + "def nyc_landmarks():\n", + " return (\n", + " xy.callout(\n", + " float(project_longitude(-73.8740)),\n", + " 40.7769,\n", + " \"LGA / QUEENS\",\n", + " color=\"#f3c86d\",\n", + " width=1.15,\n", + " dx=22,\n", + " dy=-24,\n", + " style={\n", + " \"font_size\": 12,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#ffe3a0\",\n", + " \"letter_spacing\": \"0.07em\",\n", + " },\n", + " ),\n", + " xy.callout(\n", + " float(project_longitude(-73.7781)),\n", + " 40.6413,\n", + " \"JFK / QUEENS\",\n", + " color=\"#f3c86d\",\n", + " width=1.15,\n", + " dx=-18,\n", + " dy=-25,\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 12,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#ffe3a0\",\n", + " \"letter_spacing\": \"0.07em\",\n", + " },\n", + " ),\n", + " xy.callout(\n", + " float(project_longitude(-73.9855)),\n", + " 40.7580,\n", + " \"MIDTOWN CORE\\nPICKUP SPINE\",\n", + " color=\"#67d7e5\",\n", + " width=1.25,\n", + " dx=-18,\n", + " dy=-64,\n", + " anchor=\"end\",\n", + " style={\n", + " \"font_size\": 13,\n", + " \"font_weight\": 700,\n", + " \"label_color\": \"#c8f5f7\",\n", + " \"letter_spacing\": \"0.055em\",\n", + " },\n", + " ),\n", + " )\n", + "\n", + "\n", + "def nyc_header(mode):\n", + " header_anchor_y = 1 - (HERO_PADDING[0] - 16) / HERO_HEIGHT\n", + " return (\n", + " xy.text(\n", + " HERO_PADDING[3] / HERO_WIDTH,\n", + " header_anchor_y,\n", + " \"NYC NIGHT MOVES\",\n", + " dx=0,\n", + " dy=-56,\n", + " color=\"#fff7e3\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 26,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.025em\",\n", + " },\n", + " ),\n", + " xy.text(\n", + " HERO_PADDING[3] / HERO_WIDTH,\n", + " header_anchor_y,\n", + " f\"{longitude.size:,} PICKUPS · YELLOW CAB 2015 · {mode}\",\n", + " dx=0,\n", + " dy=-12,\n", + " color=\"#7fa8b2\",\n", + " style={\n", + " \"coordinate_space\": \"figure_fraction\",\n", + " \"font_size\": 10,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.09em\",\n", + " },\n", + " ),\n", + " )\n", + "\n", + "\n", + "NYC_CHROME_STYLES = {\n", + " \"title\": {\n", + " \"text_align\": \"left\",\n", + " \"font_size\": 22,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.055em\",\n", + " },\n", + " \"tick_label\": {\"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\"},\n", + " \"axis_title\": {\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " \"letter_spacing\": \"0.08em\",\n", + " },\n", + " \"annotation_label\": {\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " \"letter_spacing\": \"0.08em\",\n", + " },\n", + " \"colorbar\": {\n", + " \"background\": \"#0d202a\",\n", + " \"color\": \"#f3d59a\",\n", + " \"border\": \"1px solid #294652\",\n", + " \"border_radius\": 5,\n", + " \"padding\": \"8px 7px\",\n", + " },\n", + " \"colorbar_title\": {\"font_size\": 11, \"font_weight\": 700, \"letter_spacing\": \"0.055em\"},\n", + " \"colorbar_tick\": {\"font_size\": 11},\n", + " \"tooltip\": {\n", + " \"background\": \"#0d202a\",\n", + " \"color\": \"#edf1e8\",\n", + " \"border\": \"1px solid #496979\",\n", + " \"border_radius\": 5,\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " },\n", + "}\n", + "\n", + "density_chart = xy.scatter_chart(\n", + " xy.scatter(\n", + " projected_longitude,\n", + " latitude,\n", + " name=\"pickup intensity\",\n", + " color=\"#ffd166\",\n", + " size=1.1,\n", + " opacity=1.0,\n", + " density=True,\n", + " ),\n", + " xy.scatter(\n", + " projected_longitude[:: max(1, longitude.size // 75_000)],\n", + " latitude[:: max(1, latitude.size // 75_000)],\n", + " name=\"pickup detail\",\n", + " color=\"#ffe7a3\",\n", + " size=0.7,\n", + " opacity=0.16,\n", + " density=False,\n", + " ),\n", + " *nyc_landmarks(),\n", + " *nyc_header(\"ALL-HOUR · ADAPTIVE SCATTER DENSITY\"),\n", + " nyc_x_axis(),\n", + " nyc_y_axis(),\n", + " xy.tooltip(title=\"YELLOW-TAXI PICKUP\"),\n", + " xy.legend(show=False),\n", + " xy.interaction_config(crosshair=True, wheel_zoom=True, box_zoom=True),\n", + " nyc_theme(),\n", + " styles=NYC_CHROME_STYLES,\n", + " style={\n", + " \"border\": \"1px solid #294652\",\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " },\n", + " width=HERO_WIDTH,\n", + " height=HERO_HEIGHT,\n", + " padding=HERO_PADDING,\n", + ")\n", + "print(\n", + " \"scatter tier:\",\n", + " density_chart.figure().build_payload()[0][\"traces\"][0][\"tier\"],\n", + ")\n", + "density_chart" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": {}, + "outputs": [], + "source": [ + "hexbin_chart = xy.hexbin_chart(\n", + " xy.hexbin(\n", + " projected_longitude,\n", + " latitude,\n", + " name=\"pickups per hex\",\n", + " gridsize=(210, 260),\n", + " mincnt=2,\n", + " bins=\"log\",\n", + " colormap=\"magma\",\n", + " opacity=0.96,\n", + " ),\n", + " *nyc_landmarks(),\n", + " *nyc_header(\"ALL-HOUR · LN-SCALED HEX DENSITY\"),\n", + " nyc_x_axis(),\n", + " nyc_y_axis(),\n", + " xy.colorbar(\n", + " title=\"LN(1 + PICKUPS / HEX)\",\n", + " ticks=[2, 3, 4, 5, 6, 7],\n", + " orientation=\"vertical\",\n", + " ),\n", + " xy.tooltip(title=\"LN(1 + PICKUPS / HEX)\", format={\"value\": \".2f\"}),\n", + " xy.legend(show=False),\n", + " xy.interaction_config(crosshair=True, wheel_zoom=True, box_zoom=True),\n", + " nyc_theme(),\n", + " styles=NYC_CHROME_STYLES,\n", + " style={\n", + " \"border\": \"1px solid #294652\",\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " },\n", + " width=HERO_WIDTH,\n", + " height=HERO_HEIGHT,\n", + " padding=HERO_PADDING,\n", + ")\n", + "hexbin_chart" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "72eea5119410473aa328ad9291626812", + "metadata": {}, + "outputs": [], + "source": [ + "hour_order = np.argsort(hour, kind=\"stable\")\n", + "hour_labels = np.array([f\"{value:02d}:00\" for value in range(24)])[hour[hour_order]]\n", + "hourly_data = {\n", + " \"map_x\": projected_longitude[hour_order],\n", + " \"latitude\": latitude[hour_order],\n", + " \"local_hour\": hour_labels,\n", + "}\n", + "hourly_chart = xy.facet_chart(\n", + " xy.scatter(\n", + " x=\"map_x\",\n", + " y=\"latitude\",\n", + " name=\"pickup intensity\",\n", + " color=\"#ffd166\",\n", + " size=1.0,\n", + " opacity=1.0,\n", + " density=True,\n", + " ),\n", + " xy.scatter(\n", + " x=\"map_x\",\n", + " y=\"latitude\",\n", + " name=\"pickup detail\",\n", + " color=\"#ffe7a3\",\n", + " size=0.65,\n", + " opacity=0.18,\n", + " density=False,\n", + " ),\n", + " nyc_x_axis(compact=True),\n", + " nyc_y_axis(compact=True),\n", + " xy.legend(show=False),\n", + " nyc_theme(),\n", + " by=\"local_hour\",\n", + " data=hourly_data,\n", + " cols=6,\n", + " share_x=True,\n", + " share_y=True,\n", + " title=\"NYC NIGHT MOVES · PICKUP DENSITY BY LOCAL HOUR\",\n", + " styles={\n", + " **NYC_CHROME_STYLES,\n", + " \"title\": {\n", + " \"font_size\": 11,\n", + " \"font_weight\": 700,\n", + " \"letter_spacing\": \"0.08em\",\n", + " },\n", + " },\n", + " style={\n", + " \"border\": \"1px solid #294652\",\n", + " \"font_family\": \"Avenir Next, ui-sans-serif, sans-serif\",\n", + " },\n", + " width=1320,\n", + " height=210,\n", + " padding=(30, 16, 16, 16),\n", + " gap=0,\n", + ")\n", + "hourly_chart" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/real_world/README.md b/examples/real_world/README.md new file mode 100644 index 00000000..42814862 --- /dev/null +++ b/examples/real_world/README.md @@ -0,0 +1,92 @@ +# Real-world large-data notebooks + +These six notebooks use public datasets large enough to exercise XY's density, +decimation, and faceting paths with recognizable scientific and operational +data: + +| Notebook | Dataset | Default workload | XY path | +| --- | --- | ---: | --- | +| `01_gaia_hr_diagram.ipynb` | [ESA Gaia DR3](https://www.cosmos.esa.int/web/gaia-users/archive/programmatic-access) | 1M stars | scatter density | +| `02_gnomad_allele_frequency.ipynb` | [gnomAD v4.1 genomes](https://gnomad.broadinstitute.org/news/2024-04-gnomad-v4-1/) | up to 3.52M variants | scatter density + log axis | +| `03_pan_ukbb_manhattan.ipynb` | [Pan-UKBB standing-height GWAS](https://pan.ukbb.broadinstitute.org/downloads/index.html) | up to 2.64M variants | scatter density | +| `04_dukascopy_fx_ticks.ipynb` | [Dukascopy EUR/USD ticks](https://www.dukascopy.com/swiss/english/marketwatch/historical/) | five calendar days | M4 line decimation | +| `05_ligo_gw150914_strain.ipynb` | [GWOSC GW150914 strain](https://gwosc.org/events/GW150914/) | 16.8M samples | M4 line decimation | +| `06_nyc_taxi_density.ipynb` | [NYC TLC yellow-taxi pickups](https://data.cityofnewyork.us/d/2yzn-sicd) | 1M trips | density, hexbin, 24 facets | + +## Screenshots + +These are rendered chart outputs, not copies of the source datasets. The PNGs +below are the only data-derived artifacts checked in; each notebook downloads +its working rows from the linked public source into the git-ignored cache. + +### 1. Gaia DR3 — Hertzsprung–Russell diagram + +![Gaia DR3 stellar color versus absolute magnitude rendered as an XY scatter-density chart.](assets/01-gaia-hr-diagram.png) + +**Theme:** Cosmic observatory · scatter density of stellar color versus +absolute magnitude + +### 2. gnomAD v4.1 — allele frequency across the genome + +![gnomAD allele frequency by chromosome rendered as an XY scatter-density chart with a logarithmic axis.](assets/02-gnomad-allele-frequency.png) + +**Theme:** Clinical genomic atlas · scatter density of genomic position versus +allele frequency + +### 3. Pan-UKBB — standing-height Manhattan plot + +![Pan-UKBB standing-height associations across all autosomes rendered as an XY Manhattan plot.](assets/03-pan-ukbb-manhattan.png) + +**Theme:** Warm biobank editorial · scatter density of genomic position versus +−log10 p-value + +### 4. Dukascopy — EUR/USD tick history + +![Dukascopy EUR/USD midpoint quotes rendered as an XY decimated line chart.](assets/04-dukascopy-fx-ticks.png) + +**Theme:** Trading terminal · M4 line decimation of UTC time versus EUR/USD +midpoint + +### 5. LIGO — GW150914 detector strain + +![GWOSC's reconstructed Hanford waveform for GW150914 rendered as a signal-lab line chart showing inspiral, peak strain, and ringdown.](assets/05-ligo-gw150914-strain.png) + +**Theme:** Signal-lab oscilloscope · a 16.8M-sample M4 overview, a bandpassed +detector detail, and the official GWOSC waveform reconstruction + +### 6. NYC TLC — yellow-taxi pickup density + +![Locally projected NYC yellow-taxi pickup coordinates rendered as a night-map hexbin-density chart with Midtown and airport callouts.](assets/06-nyc-taxi-density.png) + +**Theme:** Night cartography · locally projected hexbin density using +`ln(1 + pickups / hex)`. The notebook also renders the same rows as automatic +scatter density and 24 hourly facets. + +## Setup + +From a Python 3.11+ environment: + +```bash +python -m pip install xy jupyter numpy requests pysam h5py gwosc +jupyter lab +``` + +Open this directory in Jupyter and run a notebook from top to bottom. Each +notebook documents its official source, attribution notes, expected download, +and environment variables for scaling the workload. + +Downloads and indexes are cached under `data/` by default. Set +`XY_REAL_WORLD_DATA=/absolute/path` to use a shared cache outside the checkout. +The directory is git-ignored. + +Start with a smaller remote sample when checking connectivity: + +```bash +GNOMAD_CHROMOSOMES=22 GNOMAD_WINDOWS=2 jupyter lab +PANUKBB_WINDOWS=2 PANUKBB_VARIANTS_PER_WINDOW=2000 jupyter lab +TLC_MAX_ROWS=1000000 jupyter lab +``` + +The notebooks intentionally keep acquisition separate from visualization. +Once a dataset is cached, chart cells can be rerun and restyled without another +download. diff --git a/examples/real_world/assets/01-gaia-hr-diagram.png b/examples/real_world/assets/01-gaia-hr-diagram.png new file mode 100644 index 00000000..f26adcbb Binary files /dev/null and b/examples/real_world/assets/01-gaia-hr-diagram.png differ diff --git a/examples/real_world/assets/02-gnomad-allele-frequency.png b/examples/real_world/assets/02-gnomad-allele-frequency.png new file mode 100644 index 00000000..3d4a20a7 Binary files /dev/null and b/examples/real_world/assets/02-gnomad-allele-frequency.png differ diff --git a/examples/real_world/assets/03-pan-ukbb-manhattan.png b/examples/real_world/assets/03-pan-ukbb-manhattan.png new file mode 100644 index 00000000..964a8f49 Binary files /dev/null and b/examples/real_world/assets/03-pan-ukbb-manhattan.png differ diff --git a/examples/real_world/assets/04-dukascopy-fx-ticks.png b/examples/real_world/assets/04-dukascopy-fx-ticks.png new file mode 100644 index 00000000..fd939990 Binary files /dev/null and b/examples/real_world/assets/04-dukascopy-fx-ticks.png differ diff --git a/examples/real_world/assets/05-ligo-gw150914-strain.png b/examples/real_world/assets/05-ligo-gw150914-strain.png new file mode 100644 index 00000000..f2dba1ba Binary files /dev/null and b/examples/real_world/assets/05-ligo-gw150914-strain.png differ diff --git a/examples/real_world/assets/06-nyc-taxi-density.png b/examples/real_world/assets/06-nyc-taxi-density.png new file mode 100644 index 00000000..51c8aed9 Binary files /dev/null and b/examples/real_world/assets/06-nyc-taxi-density.png differ