Error converting DECIMAL value while reading a Parquet file

#1

Hi All,

I am trying to read a folder of Parquet files, however the extension fails to find a mapping for DECIMAL input type. For some reason cannot add a mapping type in the extension.

Anyone else having the same problem ?

thanks

0 Likes

#2

Moving this to the Big Data subforum. Let me ask one of our Big Data developers and see if they have an idea.

0 Likes

#3

Hi @milonb,
welcome to the KNIME community!
Unfortunately we do not have the possibility to read Parquet DECIMAL values with the Parquet Reader right now.
Parquet offers different possibilities to write DECIMAL values, including unlimited precision (Depending on the underlying Parquet Primitive Type).
What would be your expectation for the destination Type in KNIME?
Do you know the Primitive Type the DECIMAL value was stored with?

best regards
Mareike

0 Likes

#4

The data stored in the DECIMAL field would be of temperature. I would expect that a Float would suffice as a destination type. However, I am unable to add a conversion rule to the Reader menu (equivalent to typecasting in typical programming languages).

0 Likes

#5

Hey @milonb,
yes, we currently have no converter for DECIMAL. Depending on the Primitive Type you might be able to read the file by adding a name based rule but the values would not be correct.
I will add a feature request for small DECIMAL values to a double value. However right now it is not possible to read those values in KNIME. Sorry.
best regards Mareike

1 Like

#6

Made it work… somewhat.

Converted the parquet files into CSV using Scala on Spark-shell (for some reason the Pyspark Wrapper is broken) and read the column using the CSV reader as a NUMBER(double).

Not a good engineering practice though.

Eagerly waiting for the patch @mareike.hoeger !

1 Like