Use `mprocs` for `airflow standalone` when --interactive flag is passed

[potiuk]

Description

Currently airlfow standalone multiplexes logs at the stdout, and while it is useful for non-interactive usage, when you run airflow standalone as an interactive user (for example when you want to do some Dag development related tasks, or even when you develop airflow itself), it's not really helpful to see those logs multiplexed.

Recently in breeze we switched to mprocs as a default terminal multiplexer and after discussion in https://lists.apache.org/thread/lj1lj2typ8mwj75pmjt3gs1o6k9p7dmf the conclusion is that mprocs would be a great terminal/log multiplexer for airflow standalone --interactive mode (new mode that we would like to add).

It has some nice featuers:

• intuitive TUI interface
• ability to control, restart and generally manage each of the sub-processes
• ability to see logs separately for each process
• intuitive keyboard and mouse navigation
• nice copy&paste support for issue reporting and diagnostics

Use case/motivation

Interactive airflow standalone would be nice for anyone using airflow standalone to test and debug Dags.

It is also good to test and debug airflow itself - for local virtualenve development it might be a convenien,t local equivalent of start-airflow without the need of having docker and docker compose (with limitations of course - only working for sqlite, and only supporting local executor).

Related issues

• Add mprocs support for breeze start-airflow command #58702
• Improve documentation and behaviour of mprocs #58718
• Use mprocs by default with start-airflow #60844

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Prab-27]

@potiuk Is it fine to run it in the Docker environment (Breeze), or is it necessary to test them in a virtual environment (venv)?

[jason810496]

If I understand it correctly based on the context from dev mailing list, I think it supports this out of the box, so when user pip install apache-airflow should be able to use --interactive directly.

[potiuk]

Yes. It shoud be "truly standalone".

The idea to make it totally independent from breeze - so that it does not require docker at all. Also this is both for those who want to just start airflow, try it and - for example - do some testing with their Dags, as well as for those who might want to have a nicer and quicker (but very limited) replacement for start-airflow - without the overhead of containers.

This means that it should not rely on mprocs being installed - it should download and install (well make it runnable internally in some cache in AIRFLOW_HOME most likely) mprocs automatically (ideally with some consensus of the user that should be asked for it unless they specify something like --answer yes. This is easy as mprocs publishes multi-platform binaries (it's a rust project) that are somewhat big binaries - but not too big and "standalone" as well - means they do not need any 3rd-party libraries. So it's just the matter of pulling and unpacking the binary that is right for the architecture we are in (on failing in case our architecture is not supported) .

[potiuk]

Also - Aritra volunteered on the devlist so I assigned him, But It could also bee cooperation between several people.
Also @ferruzzi mentors some people with MLH (Major League Hacking) who could do it or help with it.

It's not super complex to do, I think most of the work will be to work-out the abstraction in airflow standalone to be able to easily switch between "regular" and "mprocs" proces management - in case of mprocs - it handles a lot of things already - things like signal propagation, quitting, restarting etc.

But we already have a number of features that should be thoroughly tested and reviewed and maybe combined with the dev mode as well - to make airflow standalone more usable for "locally developing Airflow" case:

• hot reload - it should nicely work with airflow standalone in this case, but maybe we want to do some better integration (mprocs already supports server that accept commands and it could easily be used instead of built-in hot-reload - because the built-in one has some limitations (module reloading might be problematic)

• (re)building assets automatically when we start stand-alone from sources

And probably some more interesting things that can be found while implementing and playing with such --interactive mode.