Explore integration with OpenLineage

[astrojuanlu]

OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used.

https://openlineage.io/

Reference implementation: Marquez https://github.com/MarquezProject/marquez

Validation:

With the release of Airflow 2.7.0, OpenLineage became a built-in feature of Airflow.

What does this mean for us? Should Kedro-Viz consume OpenLineage exports coming from Kedro? (See previous discussion at kedro-org/kedro-viz#1833 (comment))

[astrojuanlu]

A user asked

Is there any way to show log in Databricks for running pipelines/nodes execution such as time spending to complete specific node, memory usage, or cost in graphical manner like the image below?

I was just wondering there are any ways to visualize pipeline execution results/logs in graphical manner like for example, this node takes this much time to complete, etc.

Exporting OL data addresses this.

[astrojuanlu]

Another user:

What we are trying to capture and track is column-level spark transformations across the whole kedro pipeline

(hinted that only Kedro-related lineage would likely be insufficient)

[astrojuanlu]

OpenLineage allows solutions like DataHub to support more integrations. For example

DataHub's REST endpoint allows users to send OpenLineage events directly to DataHub.

https://datahubproject.io/docs/lineage/openlineage/

To automatically extract lineage information, the v2 plugin builds on top of Airflow's built-in OpenLineage extractors.

https://datahubproject.io/docs/lineage/airflow/#automatic-lineage-extraction

[astrojuanlu]

Well, here's a proof of concept!

https://github.com/astrojuanlu/kedro-openlineage