Variable filter after merge variable node

Hello,

how could I eliminate the unnecessary flow variables after the Merge Variable Node?

Thanks!

Roland

Hi

You can transfer them into “table row” by using “variable to table row” and then use column filter to omit the extra variables.

Mona

The approach suggested here doesn’t work - you can transfer flow variables to a table using the Variable to Table Row node, but the flow variables themselves still remain.

There isn’t a straightforward way to delete or remove flow variables (see previous forum discussion on this topic here: delete flow variable). As Iris mentions in that thread, a workaround might be to use the inherent local flow variable scope of components (previously called wrapped metanodes in the link) to effectively filter out flow variables created inside them.

Maybe there is a need for something like a “Flow Variable Filter” node. I would be interested to hear thoughts from others in the forum community about it.

13 Likes

Thanks for the clarification!

+1 @ScottF - would be great to have a filter to cleanup variables rather than having to use components and metanodes to drop them off.

3 Likes

Yes, definitely +1 from me for a Variable Filter node.

In discovering the issue raised by the following question, I realised that the ability to filter out Flow Variables would have been the perfect solution!

So it isn’t just a need to remove superflous variables, but a single flow split into different processing streams and then merged back via Merge Variables will always result in the flow variables from the first input stream trumping the flow variables from inputs 2 and 3, as there is no way to filter them out.

From the Merge Variables documentation:

“Note: If there are the same variables defined in different inputs, the standard conflict handling is applied: Top most inputs take priority and define the value of a variable.”

3 Likes

Noted! Thanks for the feedback in this thread. There is more interest in this feature than I would have guessed - we will definitely mention it to the developers. :+1:

8 Likes

Yes I can see the value of being able to remove any superfluous variables… although would suggest keeping any deletions (however this is done) to a minimum to ensure that the flow variablea are available towards the end of the workflow. This helps another future user identify what variables exist and can be used for further development of the workflow (as required). If variables are not carried through then in the future, a second variable for the same thing might be created. Obviously this results in duplication which can be problematic for transparency and overall workflow robustness.

Thanks, Ben

1 Like

I use compontents for this. Junk variables go in and only the important ones go out.

1 Like

Hi, just to add that I would appreciate a more explicit way to manage variable filtering as suggested above. Actually I am having difficulties to get to work via component configuration anyway. I have tried applying a manual filtering and wildcard/regex method to the exit of the component (choose variables to be visible outside the component) but still they seem to persist after that point. I have a load of junk variables arising from a couple of ‘case end switches’ and reading in a previously save variables file (# version exclusion using regex ^((?!#).)*$ ). Does anyone have an example work flow to share ?

Hi @ScottF

I discover this thread right now and definitely, I would also find really very helpful to have a kind of -Flow Variable Filter- node to tide up the workflow at a flow point where they are not any more needed.

So +1 from me too although @dr_snglr is a nice trick to achieve this too :wink:

Thanks & best regards
Ael

1 Like

Thanks for that feedback and +1 but I’m still finding the component exit trick is not filtering out the variables as it apparently should. As I said, I have tried various regex and wildcard methods but also manual exactly as @dr_snglr indicates, i.e.


But the excluded variable (same with force inclusion or exclusion) stubbornly remains downstream, i.e.
image
In this case I actually want to replace the variable ‘bnumber’ value with ‘bnumber_to_use’ and thought that by filtering it out I could just do a rename downstream using ‘string manipulation’ node… Incidentally I had originally tried using a variable rule engine, but this does not offer a ‘replace’ option like the data version - I suppose because of the way variables are handled (?)

Hello @ScottF,

hoping to keep this thread alive- That’s also a +1 from me!

Best regards
Michael

Hi @mwakileh

Make sure to cast your vote next to the title header.

1 Like

Hi @ScottF

This topic has the highest numbers of votes so far. Is there something in the pipeline for this? :slight_smile:

Hi @ArjenEX -

There is a ticket for this (AP-16515) and I’ll ping the devs about it again since as you mention, it’s at the top of the vote list. As you can imagine, the recent UI changes have been the focus of a lot of development time lately, but hopefully flow variables can get some additional attention soon.

As always, I can’t promise anything, even if a particular request has a lot of votes. But fingers crossed! :crossed_fingers:

5 Likes

Since it’s the weekend, I thought I’d build a theoretical Flow Variable filter demonstration workflow…

It’s not theoretical in the sense that it doesn’t actually work… it does work… it’s just the implementation is impractical for real-world use (unless you are desperate!):

In this “pseudo-imaginary world”, the data generated by the Data Generator at the top branch, is available to the lower branch. However, whilst the Variable Creator has created 4 variables, only three of those variables are present on the lower branch until it gets to the final node when all 4 variables are reunited :slight_smile:

The filtering is performed in the “Continue At (with Variable Filter)” component:

This feels a little like I’m writing an April Fool, but it does actually work, but as I said… it’s impractical in this form.

Oh, and one other thing… it uses magic :wink:

I definitely support the flow variable filter idea, but I can imagine its implementation would be a fundamental change to the way KNIME currently behaves, even though it sounds like it should be straightforward.

3 Likes