Converting Pandas / Arrow to Polars Error

[Sajith]

Hi there,

I’m using Knime 5.2.0 and I’ve got the polars package installed as well.

However when I try to convert pandas to polars or arrow to polars I’m unable to.
I get below error when trying to change pandas to polars:

Similarly when I try to change arrow to polars, I get the below error:

Any ideas to resolve the issue would be greatly appreciated

[Sajith]

Digging into the error a little more I get the below message
’ cannot create series from Extension(“knime.logical_type”, Int64, Some(“{"value_factory_class":"org.knime.core.data.v2.time.LocalDateValueFactory"}”))’

[Sajith]

Soo… I debugged with chatgpt and I implemented some suggestions it mentioned, i.e.

1. enforce the pandas datetime to the pandas dataframe and
2. also changed encoding to UTF-8

• If anyone else has this issue may this can help you as well, but if someone can advise if I had to fix something to avoid this issue in general - let me know

[sanket_2012]

Hi @Sajith ,
Thanks for posting the issue and sharing the solution.
Let me ask internally about this.

Thanks,
Sanket

[bwilhelm]

Hi @Sajith,

The issue you’re encountering stems from how KNIME handles data types when transferring data to pandas. KNIME uses extension arrays (pyarrow, pandas) for certain columns, such as date and time, because KNIME’s date and time types can have different ranges and precision compared to Python’s native datetime. This ensures that no data is lost during the transfer.

However, not all libraries (like polars) can handle these extension types directly. Therefore, the solution of explicitly converting these columns to standard datetime types in pandas before passing them to the library is reasonable to ensure compatibility.

It’s interesting that you had to set sys.stdout encoding to UTF-8. I’m curious why this step was necessary. When reading stdout back in, we generally expect it to use the system’s encoding. Could there have been a specific issue you were running into that made this adjustment helpful?

[Sajith]

Well after getting the date sorted, I still had an error with the dataframe as follows:

An error occurred while converting to polars dataframe: ‘charmap’ codec can’t encode characters in position 15-98: character maps to

And once I placed the import sys statements it worked fine.