Row Filter: Non-intuitive handling of missing values

[HiZ]

Hi,

I ran into a strange, non-intuitive issue and I would like to understand the background if possible.

Assumig we have a (string) column with “A” / “B” / missing value and the Row Filter logic is set to:

1. Equals “A” + Non-matching → Result: “B” + missing value
2. Does not equal “A” + Matching → Results: “B” only

Is it the intended behaviour?

I did the same with Row Filter (Labs) and it had “B” + missing value in both ways.

Thanks,

(KNIME 5.3.2)

[lgknime]

Hey @HiZ

Good catch. The behaviour is intended. Without going too much into depth there are multiple approaches out there on how to handle missing values and the unequal operator in the context of missing values.
In this case, the logic is that the value of a missing field is unknown so it could be A but we don’t know. Therefore the A != ? can’t be negated with certainty.

Hope this makes sense to you.
All the best
Linus

[HiZ]

Hello @lgknime

Thanks, for the explanation. Now I understand the concept behind.
I struggled a while till I found the problem for the inconsistent results.

I did not find it in the node documentation. Or is it a general rule which is valid for all nodes? (The Row Filter (Labs) behaves differently…)

I hope the others will be more trained then I was!

[Geo]

Hello,

Even though I understand the logic behind this behaviour, I do find it rather unintuitive and unnecessarily complex. The better strategy would be imo to offer the user a tick box to let them decide whether missing values should be handled with certainty or not, the default being as you wish.

Kind regards
Geo

[lgknime]

@HiZ

To my knowledge the Expression node works different because Python handles these cases differently.
The Row Filter is aligned with DB Row Filter.

@Geo
Thanks for your feedback. You are more than welcome to open feature request, so we can get some feedback from the community on that.