Parallel chunk loop

KNIME Extensions
#1

Hi all,

I managed to get a workflow with parallel chuck loop working, but I noticed some weird behaviour:

1. Disable Main Branch can not be checked - it is greyed out

2. The chunks are numbered in a non-continuous manner

3. The parallel chunk loop cannot be placed in a metanode, which is unfortunate because by nature is quite complex

Perhaps the parallel chunk nodes are going to be improved in a future version? Any ideas?

Many thanks,

Panos

#2

Hi Panos,

well - there is a reason those nodes are still on Labs :-)

Your #1 is still work in progress, which is why it is not yet accessible. This is only a matter of making things nicer, though, as then all of the work will be done in the temporary metanode. From a functionality point of view, nothing will change.

I am not sure I understand your #2 and #3 though - the rows arrive at the end node in excatly the same order as they were fed into the start node and over here parallel chunk loops run very nicely inside a metanode?

I guess I am misunderstanding your problems - can you clarify? Thanks!

Cheers, Michael

#3

Hi Michael,

I have to say that I do not have access to my office computer at the moment, so I tried to reproduce the problem on my home Mac, using the latest version of KNIME (not sure which is installed in the office, probably 2.5.x).

I attach the workflow example I came up with. The workflow does not do something meaningful. The java snippet simply returnsĀ 

return "chunk_"+$${Ichunk_index}$$+"_"+$$ROWINDEX$$;

to be used as a rowId. At the same time it allows to see the chunk index, that is not continuous (my point 2, see attached). Despite the non-continuous numbering of the chunks the workflow works fine.Ā 

If I dump everything from the loop start to the loop end in a metanode it does indeed work, so it will be difficult to reproduce my point 3. The workflow at the office does a lot more things though, including loading an external .jar file. I am not sure if this is the reason...

I will post another message once I have the chance to retry the implementation in the office with regard to my point 3.

Many thanks,

Panos

#4

Hi Panos,

thanks for the clarification - I see your point. You are wondering about the chunks' indices, not the order of chunks (which is what you did wrote originally, of course, but I managed to misunderstand ;-)Ā  Interesting and easily fixable (it actually has to do with the last chunk which is processed by the original piece of the workflow). I would hope that this is not a critical problem, though?

Let me know if you can reproduce this metanode issue. Maybe some of your nested loops/chunks aren't contained in the same workflow/metanode? That's the only limitation I am aware of that could cause problems when collapsing (parts of) a workflow.

Cheers, Michael

#5

Hi,

got to resurrect this. The option to ā€œDisable Main Branchā€ was once accessible in one of my workflows. Unfortunately it isnā€™t anymore now for an unspecified reason ā€¦

35
29

Any plans to get the Parallel Chunk nodes stabilized?

Thanks
Mike

#6

Hi @mw -

Interesting observation - thanks for posting it. Let me ask internally and see if someone can shed some light on this.

#7

I checked older versions of KNIME and donā€™t find any in which that option was ever enabled. (The oldest installation I could find is 2.9 ā€¦ years old!).

I think we wanted to enable that option to make all ā€˜parallel calculationsā€™ have equal rights but then never found a reason to do it. I could ponder more about semi-technical details but maybe I can start asking how you came across this and if you think it bears any problem?

Cheers,
Bernd

PS: Your question started a quick messaging flood internally as to what that option would do and if we broke anything along the way. :slight_smile:

1 Like