I have seen some topics related to loops on this, but haven’t seen anything on concatenate node. The problem that I am having is: I have many data sources that each get transformed to a unified format and are concatenated with concatenate node. If one of those sources has error (either in the CSV reader or in the transformation nodes between the reader and concatenate nodes) then, the whole workflow gets stopped and the final data doesn’t get uploaded to database. Is there a way to use concatenate node in such a way, that if one of the inputs has error, it still concatenates the rest and the workflow continues?
I tried Catch Error (Data ports) node, but that doesn’t seem to work as expected - if error is in the reader, then the workflow doesn’t reach the catch error node.
Have you tried a construct with Try (Variable Ports) before the reader node? And if that doesn’t work, could you share an example of your workflow with generic data?
Please find the workflow attach below. I tried to use “Try (Variable Ports)”, but couldn’t work it out.
As you can see in the example, there are 3 data sources - first data source has no errors, second has error in the transformation, third has error in reading the data. Is it possible to continue executing the workflow even if some of the inputs of concatenate node fail?
you should “wrap” every branch from reader node to last node before Concatenate node in a Try/Catch sequence. Just keep in mind that errors during configure phase can not be detect with Try/Catch. Check this topic for a bit more info:
Could you maybe clarify what you mean by “wrap every branch from reader node to last node”? Maybe you could create an example with the attached workflow?
Thank you! So if I understand correctly, it is not possible to achieve fully what I am trying to achieve, because if the CSV reader fails, the workflow stops?
yes and no. As said before errors during configure phase can not be detected with Try/Catch sequence. For example if you point Reader node to a file which doesn’t exist your Concatenate node will not work and data will not be written inside Database. But there should be a workaround (or at least idea) for it in linked topic. In general building fully (or at least highly) bullet proof workflow for a complex process with proper logging and fallback scenarios require quite some time and effort. But it’s possible
