I’m stuck with reading csv files. AFAIK there are 2 Nodes, CSV Reader and File Reader with this capability. My CSVs often change their structure that means new columns are added somewhere in the file. There are also fields with newlines inside but properly escaped as strings. Also there are fields with mixed values (numeric and strings) but all properly escaped as strings. CSV Reader won’t work because of the newlines inside the strings but seems to work with changing structures. File Reader works with the newlines inside escaped strings but does not work with changing csv file structures. Also has issues with mixed fields, eg. if there is are properly escaped column values like “1234”,“2345”,“im a string”,“3456” the column is recognized by the scan as integer, so I have to manually define it as a string column. If the structure of the underlying csv file changes and there are manually defined columns everything get mixed up completely. Next thing I tried was using a java snipped with some libraries (SuperCSV) but the java snipped seems to be row oriented and cannot output a complete table. Google outputs about using R snippets tried that too but there seems to be issues with R compatibility and Knime which I couldn’t solve. csvs are such a pain but unfortunately widely in use. Anybody an Tipp or an idea how to fix this ?

The best approach in my view is to use Readr with R. And yes you will have to install R and compile RServe 1.8.6 which can be a problem. But one you overcome this problem you will have an amazing set of capabilities. [image] Force R’s readr to import all columns of a CSV file as strings – mlauber71 Force R’s readr to import all columns as strings ——— library(readr) file_location <- knime.flow.in[["Location"]] knime.out <- as.data.frame(read_delim(file=fil… [image] import 'dirty' CSV files with the help of R's readr – mlauber71 import 'dirty' CSV files with the help of R's readr [image] use R library(readr) to read (messy) CSV file into KNIME – mlauber71 use R library(readr) to read (messy) CSV file into KNIME Export the data to SQLite, ARFF and Parquet and demonstrate to read the files back into KNIME

Stuck with csv reading nodes

KNIME Analytics Platform

mlauber71 February 14, 2020, 10:27am 2

The best approach in my view is to use Readr with R. And yes you will have to install R and compile RServe 1.8.6 which can be a problem. But one you overcome this problem you will have an amazing set of capabilities.

1 Like

Not reading all Commas in "Column delimiter"