Loops inside a loop?

Hello *,
is it save to execute loops inside a loop?
What do you think (or know)??

For a very big table I want to process this table in chunks. And the chunks shall be processed using two consecutive loops, that iterate on a second table (TableRow to Variable Loop Start).
As I am currently facing different issues with Java Heap or I/O errors (not enough space left on device (even with 43 GB free....) while writing the results in one run.

BR, Susanne

 

Hi Susanne

 

I regularly run loops within loops, it can make things easier if you put your loops within meta nodes to make things cleaner. Using a chunking loop with tablerow to variable loops shouldn't be a problem.

 

Are you currently running at your max available heap space?

 

 

Regards,

 

Sam

Hi Sam,

current heap settings are: "-Xmx2048m" and "-XX:MaxPermSize=256m" used on a machine with 15.6 GiB (4 x Intel i5-2500 CPU @ 3.30Ghz):
I guess that might not yet be the maximum.
[Once after setting the MaxPermSize as well, I run into real problems..., so I didn't do further experiments....]
Do you have a recommendation regarding heap space for me?

BR & thanxs, Susanne
 

Well, I imagine there is an arguement for getting the workflows to run under a lower memory limit but for a short term fix you could increase your heap size and see if your dataset can be processed. 

 

I have an 8GB memory availability on my machine and set the heap space to 5-6Gb so that I can do a lot of work at the same time without worrying. 

 

One option is to change the memory management for the node and get it to write to disk. However you seem to be having I/O errors also? I don't know how this would behave in this regard. 

Hi Susanne, 

Nested loops are supported, but they do hinder performance a due to the overhead for each iteration.  I generally try to minimize the number of loops I use in a workflow for this reason.  

Regarding the memory issue, I would try changing "-Xmx2048m" to "-Xmx10g" to start with since you have it available.  As for disk space, 43g does seem like a lot, but it also sounds like you are doing something quite heavy.

Maybe if you post a simplified screenshot of your workflow we can see what is going on and if there are any design changes we can make to get it to run more efficiently?

 

 

Hi Aaron,

thanks for the advice (10g) and the offer to look at the flow.
After putting chunk looping in place and writing the results from inside the loop, I already succeeded.
I still would like to know how I can set the "temp_path" differently?

Best regards, Susanne