I’m stuck with reading csv files.
AFAIK there are 2 Nodes, CSV Reader and File Reader with this capability. My CSVs often change their structure that means new columns are added somewhere in the file. There are also fields with newlines inside but properly escaped as strings. Also there are fields with mixed values (numeric and strings) but all properly escaped as strings.
CSV Reader won’t work because of the newlines inside the strings but seems to work with changing structures.
File Reader works with the newlines inside escaped strings but does not work with changing csv file structures. Also has issues with mixed fields, eg. if there is are properly escaped column values like “1234”,“2345”,“im a string”,“3456” the column is recognized by the scan as integer, so I have to manually define it as a string column. If the structure of the underlying csv file changes and there are manually defined columns everything get mixed up completely.
Next thing I tried was using a java snipped with some libraries (SuperCSV) but the java snipped seems to be row oriented and cannot output a complete table. Google outputs about using R snippets tried that too but there seems to be issues with R compatibility and Knime which I couldn’t solve.
csvs are such a pain but unfortunately widely in use. Anybody an Tipp or an idea how to fix this ?