Factory datasets are not getting validated

[kacper-ki]

Description

A factory dataset with schema defined isn't validated by kedro-pandera

Context

Unable to validate datasets defined as a factory

Steps to Reproduce

For this catalog entry:

"{foo}_feature":
  type: pandas.ParquetDataset
  filepath: data/04_feature/{foo}_feature.parquet
  metadata:
    pandera:
      schema: ${pa.python:my_kedro_project.pipelines.feature_preprocessing.schemas.GenericFeatureSchema}

Expected Result

The factory dataset is validated

Actual Result

The dataset isn't validated (in my case its the output dataset). Removing factory specification fixes the problem

Investigation / workaround

Looking into the source code the line dataset = catalog._datasets.get(name) returns None for a factory dataset, which makes metadata become None too. That stops the validation.

It is a bigger issue with the catalog and dataset factories.

I managed to fix the issue by wrapping the code inside the for loop:

for name, data in datasets.items():
  if catalog.exists(name):
    dataset = catalog._datasets.get(name)
    metadata = getattr(dataset, "metadata", None)
    ...

That makes the dataset pop up in catalog._datasets and it's getting validated properly

Another workaround that I can think of is move from before/after_node_run hook to before/after_dataset_loaded, but not 100% sure that it will work

Your Environment

• kedro-pandera=0.2.2
• Python 3.10.11
• Apple M2 Pro, Sonoma 14.5

Does the bug also happen with the last version on main?

Yes

[Galileo-Galilei]

Thank you very much for the detailed report. I'll accept PR if you want to solve it fast ;)