Skip to content

Feature/faster ci notebooks #561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FBumann merged 3 commits into from
Jan 10, 2026
Merged

Feature/faster ci notebooks #561

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b6e1445
Build job (PRs & pushes):
FBumann Jan 10, 2026
3cb2b3d
1. Identified slow notebooks - 08b-rolling-horizon (5 min) and 08c2…
FBumann Jan 10, 2026
3d07f55
Small fixes and add missing file
FBumann Jan 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 19 additions & 9 deletions .github/workflows/docs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ concurrency:
env:
PYTHON_VERSION: "3.11"
MPLBACKEND: Agg
PLOTLY_RENDERER: json
PLOTLY_RENDERER: notebook_connected

jobs:
build:
Expand Down Expand Up @@ -72,14 +72,14 @@ jobs:
path: docs/notebooks/**/*.ipynb
key: notebooks-${{ steps.notebook-cache-key.outputs.hash }}

- name: Execute notebooks in parallel
- name: Execute fast notebooks
if: steps.notebook-cache.outputs.cache-hit != 'true'
run: |
set -eo pipefail
# Execute all notebooks in parallel (4 at a time)
# Run from notebooks directory so relative imports work
cd docs/notebooks && find . -name '*.ipynb' -print0 | \
xargs -0 -P 4 -I {} sh -c 'jupyter execute --inplace "$1" || exit 255' _ {}
# Execute fast notebooks in parallel (4 at a time), excluding slow ones
cd docs/notebooks && find . -name '*.ipynb' | \
grep -vFf slow_notebooks.txt | \
xargs -P 4 -I {} sh -c 'jupyter execute --inplace "$1" || exit 255' _ {}
Comment on lines +75 to +82
Copy link
Contributor

@coderabbitai coderabbitai bot Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify slow_notebooks.txt exists and validate path format

echo "=== Checking slow_notebooks.txt existence ==="
if [ -f docs/notebooks/slow_notebooks.txt ]; then
  echo "✓ File exists at docs/notebooks/slow_notebooks.txt"
else
  echo "✗ File NOT found at docs/notebooks/slow_notebooks.txt"
  exit 1
fi

echo -e "\n=== Contents of slow_notebooks.txt ==="
cat docs/notebooks/slow_notebooks.txt

echo -e "\n=== Sample find output (first 5 notebooks) ==="
cd docs/notebooks && find . -name '*.ipynb' | head -5

echo -e "\n=== Validating path format and notebook existence ==="
cd docs/notebooks
invalid_count=0
missing_count=0
while IFS= read -r line; do
  # Skip empty lines
  [ -z "$line" ] && continue
  
  # Check format: should start with ./ and end with .ipynb
  if [[ ! "$line" =~ ^\./.*\.ipynb$ ]]; then
    echo "WARNING: Path '$line' doesn't match expected ./...ipynb format"
    ((invalid_count++))
  fi
  
  # Check if notebook file exists
  if [ ! -f "$line" ]; then
    echo "ERROR: Notebook '$line' doesn't exist"
    ((missing_count++))
  fi
done < slow_notebooks.txt

echo -e "\n=== Summary ==="
echo "Invalid format count: $invalid_count"
echo "Missing notebooks count: $missing_count"

if [ $invalid_count -eq 0 ] && [ $missing_count -eq 0 ]; then
  echo "✓ All paths are valid and notebooks exist"
  exit 0
else
  echo "✗ Issues found with slow_notebooks.txt"
  exit 1
fi

Repository: flixOpt/flixopt

Length of output: 158

slow_notebooks.txt is missing and will cause the workflow to fail.

The file docs/notebooks/slow_notebooks.txt does not exist, but the step attempts to use it with grep -vFf slow_notebooks.txt. This will cause grep to exit with an error, which will fail the entire step due to set -eo pipefail. The file must be created in the repository with paths matching the find output format (e.g., ./notebook-name.ipynb), or the step must be modified to handle its absence.

🤖 Prompt for AI Agents 
In @.github/workflows/docs.yaml around lines 75 - 82, The workflow step "Execute
fast notebooks" uses grep -vFf slow_notebooks.txt which will fail if
docs/notebooks/slow_notebooks.txt is missing; either add the file at
docs/notebooks/slow_notebooks.txt with entries matching the find output (e.g.,
./notebook-name.ipynb) or modify the step to guard the grep call (e.g., only use
-f when the file exists) so the sh command (with set -eo pipefail) won't error
out when the file is absent.


- name: Build docs
env:
Expand Down Expand Up @@ -136,12 +136,22 @@ jobs:
path: docs/notebooks/**/*.ipynb
key: notebooks-${{ steps.notebook-cache-key.outputs.hash }}

- name: Execute notebooks in parallel
- name: Execute fast notebooks
if: steps.notebook-cache.outputs.cache-hit != 'true'
run: |
set -eo pipefail
cd docs/notebooks && find . -name '*.ipynb' -print0 | \
xargs -0 -P 4 -I {} sh -c 'jupyter execute --inplace "$1" || exit 255' _ {}
# Execute fast notebooks in parallel (4 at a time), excluding slow ones
cd docs/notebooks && find . -name '*.ipynb' | \
grep -vFf slow_notebooks.txt | \
xargs -P 4 -I {} sh -c 'jupyter execute --inplace "$1" || exit 255' _ {}

- name: Execute slow notebooks
if: steps.notebook-cache.outputs.cache-hit != 'true'
run: |
set -eo pipefail
# Execute slow notebooks (only on release)
cd docs/notebooks && cat slow_notebooks.txt | \
xargs -I {} sh -c 'jupyter execute --inplace "$1" || exit 255' _ {}

- name: Configure Git
run: |
Expand Down
2 changes: 2 additions & 0 deletions docs/notebooks/slow_notebooks.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
08b-rolling-horizon.ipynb
08c2-clustering-storage-modes.ipynb
11 changes: 8 additions & 3 deletions flixopt/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -818,11 +818,15 @@ def notebook(cls) -> type[CONFIG]:
"""Configure for Jupyter notebook environments.

Optimizes settings for notebook usage:
- Sets plotly renderer to 'notebook' for inline display
- Sets plotly renderer to 'notebook' for inline display (unless PLOTLY_RENDERER env var is set)
- Disables automatic plot.show() calls (notebooks display via _repr_html_)
- Enables SUCCESS-level console logging
- Disables solver console output for cleaner notebook cells

Note:
The plotly renderer can be overridden by setting the PLOTLY_RENDERER
environment variable (e.g., 'notebook_connected' for CDN-based rendering).

Examples:
```python
# At the start of your notebook
Expand All @@ -836,8 +840,9 @@ def notebook(cls) -> type[CONFIG]:
"""
import plotly.io as pio

# Set plotly to render inline in notebooks
pio.renderers.default = 'notebook'
# Set plotly to render inline in notebooks (respect PLOTLY_RENDERER env var)
if 'PLOTLY_RENDERER' not in os.environ:
pio.renderers.default = 'notebook'
pio.templates.default = 'plotly_white'

# Disable default show since notebooks render via _repr_html_
Expand Down