We have a column with type “Date time” and we try to store the table to local using “parquet writer” node
then we use the “parquet reader” node to read the stored parquet file, we are able to notice that the data type was automatically converted from “date time” to “zoned date time”.
Note: this automatic data type conversion happens if we try to use .parquet fille type for storing and retrieving the data and does not happen with .csv or .xlsx files
Please let us know if there is any resolution to this.
Could you check in the “Transformation” tab of the reader node if the type is set to “Zoned Date Time”. If yes, changing this setting to “Local Date Time” will fix the issue.
Thank you for taking you time to respond, From my understanding the solution you provided works when we manually change the type of the column in the node settings, but the reader node we are using is within a component and from the point of view of automation this seems like a manual intervention which we are trying to avoided in some way.
And also we could see that this issue is not happening when we go with .csv or .xlsx formats, hence we were curious to find out if there would be an easy way to resolve it.
I asked to check if you have set the type to “Zoned Date Time” manually in the reader node when you were configuring the node (maybe unintentionally?).
Could you try a new Parquet Reader node to read the same file and check if the type is still Zoned Date Time?
And if yes, would you double check the table before the Parquet Writer node and see if the type is zoned date time?
Also, it would be very helpful if you provide us with an example workflow which reproduces the same issue.