Recursive Loops and Pagination For Scraping

[ArjenEX]

Hi @dinkyc

Welcome to the KNIME community!

There is some reading material available about this:

https://knime.tips/paginating-get-results-via-recursion-loop-in-knime/

An example that deals with paginating through API calls, which is comparable to your situation.

Loop Variables for API

@badger101 Went back and fourth a few times between different setups but ended up with these two that stuck. They are producing the same output. [image] General notes: Recursive Loop is the way to go here. It’s tricky to master but it has to be used since the loop needs to “remember” what was just queried and what the URL of the next GET request should subsequently be. You need to properly control the exit of the loop. In this case, whenever the nextPageToken is not retrievable it means no…

In general, it is impossible to troubleshoot these kind of web interaction workflows without seeing it in action. It’s all down to the configuration of your nodes. Moreover, it is not clear which forum you are working with, how it is coded, etc, etc. As such, please upload your workflow and provide your current vs expected output.

You can use the Export KNIME Workflow option for this.