parallel chunker node

I was wondering if there are plans for a parallel chunker node end with "column append" option, similar to the one for regular loops? I have a workflow that could greatly benefit for that kind of node.

Thanks,

Natasja

 

Good idea. It's not yet on the feature list, but I am happy to put it on it. However, we need to discuss if and how we can support parallel processing of columns... As a work around you could transpose the table before and after the loop start node, I guess?

Hi Gabriel,

Thanks for putting it on the list. In this case, I don't think I can use a transpose. I use the chunker loop start followed by the 'table row to variable loop start' to turn the rows into the original table into column name variables. Then the loop goes over a 2nd table, doing calculations on the columns matching the variables created earlier in the loop using the first table. I then use the 'loop end (column append)'. The final node would have to be the chunker loop end, but it doesn't accept new columns, so I'm in a bit of a bind here in this case. Hopefully this makes sense, otherwise please let me know.

Thanks,

Natasja

 

Hi Natasja, cool idea of using the parallel chunking concept to loop over columns. I wasn't aware of the fact that a second table is involved. So I assume you are almost there, what if you just transpose the table before going into the loop end and afterwards transpose it back, that should fix it? Cheers, Thomas

Hi Thomas,

That does work, but is kind of counterintuitive. After the transpose (which is inserted between the 'loop end (column append)' and 'chunker end' nodes) there are ~100K columns that didn't exist in either input table, yet the chunker is ok with this? 

Thanks,

Natasja

 

Hi Natasja, yes the chunker is okay with that, it's only a matter of collecting the data in the Loop End which is indeed not optimal. I currently don't see any other solution, but I will open a feature request to have this tracked. Thanks for your feedback, Thomas

There's definitely still a need for a chunker loop with column append end note, as the transposing is only 9% through after 4-5 hours of churning. 

 
 
 

Got it. I vote for it, too :) I don't know your workflow in all details, but would it be an option using the Create and Split Collection node to have the rows/columns combined before going into the chunk loop?