Hi Arjen,
I just tried your workflow locally.
Generally, you can modify your query in Find Elements to the following XPath:
//img[@src='images/collapsedMidNode.gif']/..
This will search for all collapsed folder icons and get you the parent link. Then simply click this link and continue the iteration until there are no more collapsed folders found. I did this using an Extract Table Dimension node which serves as condition to a Variable Condition Loop End (end iterating when no more results are found):
I have modified the workflow like this and have it running since ten minutes and it seems to work okay.
There’s the following gotchas:
- With each expanded folder, new unexpanded folders will pop up. This seems like the (Germish) “barrel without floor”, thus the workflow will be running for quite a long time
- As the page size will increase with each folder expanded, the loading time per request will grow. It will start rather quickly, but with each request it will get a bit slower (as more data needs to be transferred)
- Interestingly the server doesn’t seem to put any limits to this
Still, I think that you’ll need to think of some approach to cut this into several smaller pieces so that it will scale (and eventually finish 
). Probably it makes sense to process all main categories one after the other?
Anyways, I’ll attach my current version here – hope this helps to bring you further!
–Philipp