diff --git a/contributing-docs/images/iterm2-enable-mouse-reporting.png b/contributing-docs/images/iterm2-enable-mouse-reporting.png new file mode 100644 index 0000000000000..a8e23f83d02d1 Binary files /dev/null and b/contributing-docs/images/iterm2-enable-mouse-reporting.png differ diff --git a/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md b/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md index 25041f185d26d..f50549e1b2a7c 100644 --- a/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md +++ b/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md @@ -25,6 +25,11 @@ breeze start-airflow --use-mprocs ``` +Breeze will start Airflow components using mprocs to manage multiple processes in a single terminal window. +It will generate dynamically the required configuration based on the selected executor and options and use it, +the generated configuration file is stored in a `files` folder inside the container, so that you can also +use it outside of Breeze if needed. + ## Common Usage Patterns | Command | Description | @@ -38,16 +43,15 @@ breeze start-airflow --use-mprocs ## mprocs Keyboard Shortcuts -| Key | Action | -|---------|------------------------------| -| `↑↓` | Navigate between processes | -| `Space` | Show/hide process output | -| `r` | Restart selected process | -| `k` | Kill selected process | -| `s` | Start selected process | -| `a` | Toggle showing all processes | -| `q` | Quit mprocs | -| `?` | Show help | +| Key | Action | +|------|----------------------------| +| `↑↓` | Navigate between processes | +| `r` | Restart selected process | +| `x` | Stop selected process | +| `s` | Start selected process | +| `a` | Add new process | +| `q` | Quit mprocs | +| `?` | Show help | ## Components Managed @@ -97,3 +101,32 @@ curl -L "https://github.com/pvolok/mprocs/releases/download/v${MPROCS_VERSION}/m | tar -xz -C /usr/local/bin/ mprocs chmod +x /usr/local/bin/mprocs ``` + +## Mac OS X and iTerm2 + +There are some known issues with mprocs and iTerm2 (which is often used as terminal by +developers using MacOS) with default settings, but they have solutions and workarounds: + +* Mouse clicks are not captured correctly by default + +**Solution:** you need to configure "Enable Mouse reporting": + +![Enable mouse reporting](../images/iterm2-enable-mouse-reporting.png) + +* Ctrl-a does not work correctly when in breeze Docker container (issue https://github.com/pvolok/mprocs/issues/179) + +**Workaround**: You need to use `Cmd - <-` to change focus instead of Ctrl-a. Also you can switch +between process list and output with mouse click providing that you applied the solution above. + +## Standalone execution + +You can run mprocs outside of Breeze for custom setups. Create a `mprocs.yaml` configuration +file defining your processes, then start mprocs: + +```bash +mprocs -f mprocs.yaml +``` + +An example [mprocs.yaml](mprocs.yaml) file for Airflow components can be found in this directory - it has +default configurations for all major Airflow components, but you can customize it as needed by copying it +elsewhere and modifying the process definitions, uncommenting the commented or adding new processes. diff --git a/contributing-docs/mprocs/README.md b/contributing-docs/mprocs/README.md deleted file mode 100644 index 8c0f48216997d..0000000000000 --- a/contributing-docs/mprocs/README.md +++ /dev/null @@ -1,123 +0,0 @@ - - -# mprocs Configuration for Airflow - -This directory contains mprocs configuration files for running Airflow components. - -## Overview - -mprocs is a modern alternative to tmux for managing multiple processes. It provides: - -- Intuitive TUI with better visual feedback -- Easy process management (start, stop, restart) -- Mouse and keyboard navigation -- Process status indicators -- Better cross-platform support - -## Files - -- `basic.yaml` - Static mprocs configuration example -- `generate_mprocs_config.py` - Script to generate dynamic mprocs configuration (located in `scripts/in_container/bin/`) -- `run_mprocs` - Entry point script for starting Airflow with mprocs (located in `scripts/in_container/bin/`) - -## Usage - -### Using Breeze - -The easiest way to use mprocs with Airflow is through the Breeze command: - -```bash -breeze start-airflow --use-mprocs -``` - -This will: - -1. Start the Breeze container -2. Dynamically generate an mprocs configuration based on your environment -3. Launch mprocs with all configured Airflow components - -### With Debug Support - -You can combine mprocs with debugging: - -```bash -breeze start-airflow --use-mprocs --debug scheduler --debug triggerer -``` - -### Manual Usage - -Inside the Breeze container, you can manually generate and use mprocs configuration: - -```bash -# Generate configuration -python3 /opt/airflow/scripts/in_container/bin/generate_mprocs_config.py /tmp/airflow-mprocs.yaml - -# Start mprocs -mprocs --config /tmp/airflow-mprocs.yaml -``` - -## Dynamic Configuration - -The `generate_mprocs_config.py` script reads environment variables to determine which components to run: - -- `INTEGRATION_CELERY` - Enables Celery worker -- `CELERY_FLOWER` - Enables Flower UI -- `STANDALONE_DAG_PROCESSOR` - Enables standalone DAG processor -- `AIRFLOW__CORE__EXECUTOR` - Determines if Edge worker should run -- `USE_AIRFLOW_VERSION` - Determines if API server (3.x+) or webserver (2.x) should run -- `BREEZE_DEBUG_*` - Enables debug mode for specific components -- `DEV_MODE` - Enables development mode features - -## Components - -The following Airflow components can be managed by mprocs: - -1. **scheduler** - Main Airflow scheduler -2. **api_server** (Airflow 3.x+) or **webserver** (Airflow 2.x) -3. **triggerer** - Handles deferred tasks -4. **dag_processor** - Standalone Dag processor (optional) -5. **celery_worker** - Celery worker (when using CeleryExecutor) -6. **flower** - Celery Flower monitoring UI (optional) -7. **edge_worker** - Edge worker (when using EdgeExecutor) - -## mprocs Controls - -When mprocs is running: - -- **Arrow keys** - Navigate between processes -- **Space** - Show/hide process output -- **r** - Restart selected process -- **k** - Kill selected process -- **s** - Start selected process -- **a** - Toggle showing all processes -- **q** - Quit mprocs -- **?** - Show help - -## Requirements - -mprocs is pre-installed in the Breeze CI image starting from Airflow 3.x development. No additional setup is required. - -For older images or custom setups, you can manually install mprocs by adding it to your Dockerfile or by customizing your Breeze image. See the installation section in the Dockerfile.ci for reference. - -## See Also - -- [Debugging Airflow Components](../../contributing-docs/20_debugging_airflow_components.rst) -- [Contributors Quick Start](../../contributing-docs/03_contributors_quick_start.rst) -- [mprocs Quick Reference](../../contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md) diff --git a/contributing-docs/mprocs/basic.yaml b/contributing-docs/mprocs/mprocs.yaml similarity index 100% rename from contributing-docs/mprocs/basic.yaml rename to contributing-docs/mprocs/mprocs.yaml diff --git a/scripts/in_container/bin/generate_mprocs_config.py b/scripts/in_container/bin/generate_mprocs_config.py index 7be9cf26b43c7..f793d5461ae34 100755 --- a/scripts/in_container/bin/generate_mprocs_config.py +++ b/scripts/in_container/bin/generate_mprocs_config.py @@ -15,7 +15,13 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. - +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "rich>=13.6.0", +# "PyYAML>=6.0", +# ] +# /// """Generate mprocs configuration dynamically based on environment variables.""" from __future__ import annotations diff --git a/scripts/in_container/bin/run_mprocs b/scripts/in_container/bin/run_mprocs index a60377a0ba67f..778b393eb7ad3 100755 --- a/scripts/in_container/bin/run_mprocs +++ b/scripts/in_container/bin/run_mprocs @@ -15,19 +15,11 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. - -# Use LocalExecutor if not set and if backend is not sqlite as it gives -# better performance -if [[ ${BACKEND} != "sqlite" ]]; then - export AIRFLOW__CORE__EXECUTOR=${AIRFLOW__CORE__EXECUTOR:-LocalExecutor} -fi - # Generate mprocs configuration dynamically -MPROCS_CONFIG_PATH="/tmp/mprocs-airflow.yaml" -python /opt/airflow/scripts/in_container/bin/generate_mprocs_config.py "${MPROCS_CONFIG_PATH}" +MPROCS_CONFIG_PATH="/files/mprocs.yaml" -if [[ $? -ne 0 ]]; then +if ! uv run "/opt/airflow/scripts/in_container/bin/generate_mprocs_config.py" "${MPROCS_CONFIG_PATH}"; then echo "Failed to generate mprocs configuration" exit 1 fi