[Feature] Setup dashboard for Airflow monitoring

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

This is an open issue for new contributors. Apache Airflow is a widely used workflow scheduler. We are encouraging someone new to the community to add a new level catalog(workflow) for Airflow.

Metrics

Airflow exposes metrics through StatsD, https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html.
We could use StatsD + OpenTelemetry StatesD(https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md) + OpenTelemetry OTEL exporter to ship the metrics to SkyWalking OTEL receiver.
Then use MAL to build metrics as well as a dashboard for those metrics. Notice, a new layer and new UI menu should be added.

Logging

Airflow supports Fluents to ship metrics, https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-architecture.html. SkyWalking already has FluentD setup support, so we should be able to receive and catalog the logs.

Additionally, Task Logs seems an interesting think. We could use LAL(Log Analysis) to group the logs by task name(or ID) by treating tasks as endpoints(SkyWalking concept).

Use case

Add more observability for Airflow server.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[mufiye]

I have done one issue about the agent and I'm interested in oap. I think I can do this task, could you please assign it to me?

[wu-sheng]

Assigned. Good luck.

[mufiye]

Assigned. Good luck.

Ok, I will do my best.

[wu-sheng]

@mufiye You could take one step at a time. Make metrics available for airflow first. Then move forward on logs.

[mufiye]

@mufiye You could take one step at a time. Make metrics available for airflow first. Then move forward on logs.

Ok, I will do the metrics part first. And I think I can refer to other similar issues about how to add the metrics dashboard, such as issue #9677.

[mufiye]

Hello, @wu-sheng . I find that all data opentelemetry collector received use tag to compose "metrics name" but have no "tag attributes", such as "airflow.ti.finish.tutorial.templated.up_for_reschedule", "airflow.ti.finish.tutorial.print_date.shutdown". I have no idea how to write the mal rules to process these data.

[wu-sheng]

What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.

[mufiye]

What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.

I mean that all info is contained in the "metrics name" such as <job_name>, <job_id>, <dag_id>, <task_id>, <operator_name>, and so on. But I have no way to filter and process. Or I just don't consider these metrics? Because of the statsD data format, the otel data collected will not have "key value pair" tag attributes.

[wu-sheng]

Does the original statsd have these metadata?

[mufiye]

These metadata compose the name, such as "local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>".