A question that immediately comes to mind is where you currently have the garbage collection located within your workflow, and more importantly in relationship to the loops.
Regarding how memory usage works on the server, this post explains that pretty well in case you haven’t seen it yet!