Stuck with csv reading nodes

KNIME Analytics Platform
1

I’m stuck with reading csv files.

AFAIK there are 2 Nodes, CSV Reader and File Reader with this capability. My CSVs often change their structure that means new columns are added somewhere in the file. There are also fields with newlines inside but properly escaped as strings. Also there are fields with mixed values (numeric and strings) but all properly escaped as strings.

CSV Reader won’t work because of the newlines inside the strings but seems to work with changing structures.

File Reader works with the newlines inside escaped strings but does not work with changing csv file structures. Also has issues with mixed fields, eg. if there is are properly escaped column values like “1234”,“2345”,“im a string”,“3456” the column is recognized by the scan as integer, so I have to manually define it as a string column. If the structure of the underlying csv file changes and there are manually defined columns everything get mixed up completely.

Next thing I tried was using a java snipped with some libraries (SuperCSV) but the java snipped seems to be row oriented and cannot output a complete table. Google outputs about using R snippets tried that too but there seems to be issues with R compatibility and Knime which I couldn’t solve.

csvs are such a pain but unfortunately widely in use. Anybody an Tipp or an idea how to fix this ?

2

The best approach in my view is to use Readr with R. And yes you will have to install R and compile RServe 1.8.6 which can be a problem. But one you overcome this problem you will have an amazing set of capabilities.

1 Like
3

Tried that already a few weeks ago but had a lot of issues with “eval failed” messages due to incompatibilities between rserve knime. After updating everything today it seems to work.

It is indeed a solution maybe The solution, but having two standard nodes in KNIME itself and due to the fact that dealing with csv’s is common, I don’t feel very happy to add this complexity to my production only to read perfectly compliant csv’s. But this one goes out to the maintainers of the csv/file reader Nodes an not to you, so thank you very much for the quick reply and for the working solution.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.