Concatenate: Add Option to fail for Column Differernces

Feedback & Ideas
,
1

Hi,

when executing complex workflows, columns can change either in type, get lost or new appear. For better control and to ensure data consistency, I’d like to suggest to add an option to allow the the Concatenate node to fail if column disjoints are identified.

Here is a test workflow:

PS: Also applying bug flag since this might not be intentional.

Best
Mike

2

Hi Mike,

Thanks for this suggestion as well. For now, I highly recommend that you create a structure after the concatenation in which you check the column types (use node Extract Table Spec – KNIME Community Hub) and then decide what you want to keep based on that.

Cheers,
Aline

3

Hi Aline,

thanks for your suggestion. I know how to handle the situation but generally this scenario should be handled more strictly triggering an error or, as suggested, to make the behavior voluntary by choice of the user, wouldn’t you agree?

Best
Mike

1 Like
4

Hey Mike, I personally have never faced this issue without using, say, a try/catch structure, but a “builtin check” could be nice indeed.

Cheers,
Aline

5

Hey Aline,

I faced in in different scenarios i.e. when ingesting data from various sources like data from a EU vs. NA company. The first uses a comma and the later a full stop as a decimal sign.

Most often however it’s just time itself pranking you because of software getting updated causing a change in data structure or types (int vs. long) or an initially forgotten setting got changed resulting in the same. I could almost go on forever :wink:

Best
Mike

1 Like