File Reader and irregular CSV files

[Batjesen]

Hi,

I will have thousands of files reading historic weather data in Sweden, there may be one one or more actual data columns besides timestamps and such, depending on what type of weather data it is, and that is handled via variables.

The files themselves, for historic data are highly irregular in design and can only be obtained as CSV:s. Newer data from active stations can be obtained as JSON as well.

My problem is that some 5 years ago I used File Reader(?) versions that allowed checking on number of a certain characters and delete, f ex, the 5th “;” (or “,”) and on, on a row. That fixed my problems at that time.

I use Knime 5.3 and opens a file “skip first lines 8”, and has header, set the character set to utf8, but gets “X Row 12: The data row has to few elements”. Yes the lines no longer contains any comments and a spacer column, which makes the file readable in an editor - but that is not the purpose of a CSV-file, is it?
smhi-opendata_1_107330_20240903_092400.csv (2.2 KB)

I would be greatfull for any suggestions excluding fixing the files outside Knime

Regards,

Batjesen/Ingemar

[mlauber71]

@Batjesen the main thing to do is to allow short lines when importing the data

CSV - Import Weather Files from Sweden - KNIME Forum (82460) – mlauber71

[Batjesen]

Thanks mlauber71,

Yeah that did the trick! Skip first lines I knew of as well as UTF-8!

Never understood the meaning of short data rows, but now I do.

Again Thanks!

Batjesen/Ingemar