Which method for data input is most efficient?

I would like to adopt a best practice when it comes to minimizing the resources and processing time during the data input and subsequent workflow process. Is there a difference between selecting/deselecting the desired fields for input at the File Reader or CSV Reader, etc. node level at the beginning of the flow, or by using a Column Filter node after the File Reader?

Also, is there a way to select or deselect all fields within the File Reader?

Thank you for your time!

Manuel

Hi @mmunozjr and welcome to the forum.

For your first question, I suspect it’s going to be quicker to deselect fields in the reader node, as opposed to filtering them out after the fact. How quick will depend on the file size and number of variables, of course. That said, you can prove this to yourself using the Timer Info node - do it both ways and measure how long it takes! There’s no substitute for a proper experiment, right? :slight_smile:

As to your second question, it’s not currently supported. But we have a ticket in our system to add that feature (AP-16733), and I’m adding a +1 on that ticket from you.

3 Likes

Thank you, Scott! I greatly appreciate your time and the response provided. I am loving my experience with Knime, such a game changer. :slight_smile:

Warm regards,
Manuel

4 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.