Hi, I have a case where we need to use lots of try/catch and if/endif also case switch/end statements on our spark jobs. However, AFAIK, Knime does not have special try/catch and if/endif nodes for spark.
I am using variables try/catch but also very limited because we cannot use the catch on the spark data ports.
The same goes with case switch variable where I can control a flow by putting the case selection as input variable, but in the end I cannot concatenate them together as spark concatenate even if it says has “optional in”, still needs the previous node to get done.
- Can I find any good example for these case switch, if and try/catch in spark?
2. And additionally, I need a workflow that even if the job fails it will destroy the spark context. I have tried some to no avail (with sample workflow attached)
Okay, I managed to get number 2 with try/catch…
Still, need the number one, especially case switch start and case switch end in spark with spark data.
(See attached… I need the same logic in spark). However taking the spark to table is not feasible as our dataset is very large
to spark.zip (14.9 KB)
Thank you, and sorry for asking way too many questions on the forum!