ColumnNotFoundError when using unnest

[micouy]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.LazyFrame({
    "a": [
        [1],
        [2, 2],
        [3, 3, 3],
    ]
})


def field_name(index: int) -> str:
    return f"a_split_{index}"


df = df.with_columns(
    pl.col("a")
        .list
        .to_struct(
            n_field_strategy="max_width",
            fields=field_name,
        )
).unnest("a")

# Comment this step out and the "a_split_0" will be present in the final output.
df = df.with_columns(
    pl.col("a_split_0") * 2
)

print(df.collect())

Log output

No response

Issue description

The code above raises the following error:

polars.exceptions.ColumnNotFoundError: a_split_0

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'with_columns' <---
UNNEST by:[a]
   WITH_COLUMNS:
   [col("a").list.to_struct()] 
    DF ["a"]; PROJECT */1 COLUMNS; SELECTION: None

I suspect this might be an expected behaviour. I know in case of LazyFrame the schema is validated before executing the query. Since the number of columns cannot be known before execution and since the naming function might be indeterministic, the presence of the resulting columns cannot be verified early.

In case the error is expected, I think it would be useful for users to add a comment to LazyFrame.unnest docs.

Expected behavior

The query above would complete successfully or an informative warning would be raised.

Installed versions

--------Version info---------
Polars:              1.9.0
Index type:          UInt32
Platform:            macOS-14.5-arm64-arm-64bit
Python:              3.11.9 (main, Aug  2 2024, 11:33:02) [Clang 15.0.0 (clang-1500.3.9.4)]

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         1.6.0
numpy                2.1.2
openpyxl             <not installed>
pandas               <not installed>
pyarrow              <not installed>
pydantic             2.9.2
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

[ritchie46]

The dtype of list.to_struct cannot be known statically and therefore requires a user to provide a dtype. We must add that option.

[ritchie46]

In the mean time you can add a cast to inform Polars of the type.

[cmdlineluser]

There are probably a few issues on the topic that could all be rolled into one:

• ColumnNotFoundError after unnest #7764
• .struct.field('*') PanicException when used after .list.to_struct() #17092
• Some struct functions do not work on .list.to_struct #16370
• .list.to_struct() should update the schema in lazy context when fields are provided #15742
• .list.to_struct() has non-deterministic behavior #16450

[DarkAmoeba]

I think I also encountered a variant of this issue which seemed specifically related to when the I aliased the structed column with a new name. This is a re-producible example of the issue I experienced:

import datetime

data = [{'timestamp': datetime.datetime(2024, 9, 30, 16, 5, 19),
         'flight': 1,
         'conformances': [True,
                          True,
                          False]},
        {'timestamp': datetime.datetime(2024, 9, 30, 16, 5, 20),
         'flight': 2,
         'conformances': [False,
                          False,
                          False]},
        {'timestamp': datetime.datetime(2024, 9, 30, 16, 5, 21),
         'flight': 3,
         'conformances': [False,
                          False,
                          True]}]

ldf = pl.LazyFrame(data)
ldf2 = ldf.with_columns(pl.col('conformances').list.to_struct(fields=['heading', 'level', 'speed']).alias('new_conformances'))

# this works fine
ldf2.unnest('new_conformances').collect().select('heading', 'level', 'speed')

# this raises polars.exceptions.ColumnNotFoundError: heading
ldf2.unnest('new_conformances').drop('flight').collect().select('heading', 'level', 'speed')

[cmdlineluser]

@DarkAmoeba I can reproduce that on 1.10.0

I think it was fixed by #19439 in 1.11.0