I have CSV files that I am reading in with the ‘CSV Reader’ node, and I noticed that the table it produces does not exactly match the data in the file itself. Specifically, in the file itself all numbers are entered to 7 digits past the decimal yet in the table that is produced by CSV Reader I see only 3. If I look at the column spec for the output from CSV Reader it also shows these to have only 3 digits past the decimal. I don’t see any obvious parameters in CSV Reader that I can change to change this.
Can someone shed some light on what is going on? Should I use a different method to read in these CSV files to retain the full numeric precision that was in the file?
Use File reader instead. It allows to change data type, so you can change your numbers to double to have 7 digits after dot.
Thank you for the suggestion. I just started looking at File Reader, but I don’t see a way to specify how many digits it should take in after the decimal. I set it to “Number (double)” - which is what it detected it to be - and it still produced only 3 digits after the decimal. Where should I look for an option to specify the number of digits?
For more complex imports I use the R package Readr which seems to be quite robust and has a lot of configurations going for it.
Looks like with standard nodes you need to load fields as text and then convert them to double.
That works, right until I convert to double (where I end up with only 3 digits after the decimal again). I wonder if there is a setting in KNIME that I’m not aware of that is doing this? After your suggestion I did ‘File Reader’ -> ‘Column Rename’ -> ‘String to Number’. After the first two steps I see the correct number of digits, after the third they are all silently truncated to 3 digits after the decimal. I don’t see an option in there to tell it how many digits to keep.
Thank you for the suggestion! It appears based on what I’ve found with File Reader (as described in my reply to the suggestion to use that) that I may have something misconfigured in how KNIME handles double values in general. Similarly I just tried using Constant Value Column to add on a column to an existing table - and added a value with many digits past the decimal - and KNIME silently truncated it to three digits past the decimal.
However I then found that I can do math on the truncated values and they are treated correctly (as in the digits that appear to have been truncated off are still included), so maybe this is actually a display issue? For example if I put in “123.456789” in the constant value column, it shows up in the table as “123.457”; however if I then use a Math Formula node and multiply it by 100 the resulting column will be “12345.679”.
So at least it appears my math should be OK.
Actually, it occurs that I may have spoken too soon on that though based on what I need to do next with the data. What I need to do with the values - which were input with 7 digits past the decimal - is as follows:
Truncate the values to four digits past the decimal (which apparently KNIME won’t display for me currently)
Find the unique values from this list and write them out as a list of values with four digits after the decimal (sorted numerically).
1 Like
Numeric columns are only cut to three digits in the table view. Internally still all digits are stored. You can change the display by right-clicking on the column header and selecting a different renderer. There should be one for full precision.
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.