Import of strange csv data

[carllum26]

Dear KNIME Forum,
Today I received a strange data log of four sensors on
6: Volt (DC)
7: Volt (DC)
8: Volt (DC)
9: Ohm (Resistance)

The data ara strange because the csv-reader read them in a non straight way.
The reasons ara several line breaks in the raw data.
e. g. row 0 contains time and date
row 1: volt channels 6, 7, 8
row2: resistance channel 9

With which setting in the csv reader do I receive a straight data set for further analysis of that time-series?

Best
Carl

[carllum26]

DataImport_HyperTerm.knwf (6.6 KB)

[ipazin]

Helo @carllum26,

and welcome to KNIME Community!

Is there possibility for you to share data? You shared workflow in a reset state with CSV Reader pointing to location that is obviously not accessible to us

Br,
Ivan

[carllum26]

Dear Ivan plus forum readers,
Thanks for the hint, here is the log-file of the voltage and resistance data.
2021-10-28_08-03-52.log (37.6 KB)
In the meantime I will try some alternative KNIME nodes and keep you informed.

[mlauber71]

@carllum26 i don’t think you will be able to just import this data with a CSV reader. You very likely will have to import the data as a whole line and then do some splits and joins.

The good news is that the data always seem to have the same structure so this should not be too complicated a task. I will see if I can come up with something.

It might be something like this

Reference Column Filter Node with LIKE operator

@Brotfahrer I had another look at the data and I would like to point you to another systematic approach. Using the underlying structure of the files to mark and split the relevant data. the big blocks seem to be defined by parts marked ECU. With the help of row numbers, you can identify and mark these blocks as “Level1” value pair. You would have to deal with the header values separately and maybe add the file name as constant column. The new LevelB is formed by Blocks that can be identified b…

Or this

HL7 output

@AAM i used your example and set up a workflow that hopefully takes you further in the right direction. Feel free to test it and ask additional questions. I saw you used an advanced Column expression. I left that as it was. In your case I would suggest to use a Loop and handle each message (consisting of several lines as one case and export that into a txt file). Therefor you would need a unique ID that would also server as a file name. I like to use the KNIME protocol that makes it easy to p…

[carllum26]

Thanks so far for the hints .
I will test your input during the day.

[ipazin]

Hello @carllum26,

although this is kind of a @mlauber71’s territory as he is import expert I have taken a look at file you provided and came up with following solution.

Idea is simple. Read all data as one column and by splitting data address each input pattern separately (some didn’t need any manipulation steps) to have every info in their own column. Here is workflow example.
2021_10_29_Line_Reader_Approach.knwf (28.1 KB)

Although idea is simple implementation features couple of tricks and steps that might raise some questions for a beginner so check it out and if any feel free to ask

Br,
Ivan

[carllum26]

Dear Ivan,
dear Forum readers,
Thank you very much for your support in solving the data import features with a couple of line breaks
I performed further string manipulations, calculations, and finally I received what I expected (but not without your support).

@Ivan you will stay as author in the WF

The converted voltages to humidity over time.

[ipazin]

Glad got what you wanted
Ivan