Recursive Loops and Pagination For Scraping

[dinkyc]

Hi,

I am new to KNIME and have been experiencing some problems while using a recursive loop in my workflow. I’m using KNIME 4.7.0.

Essentially, the purpose of my workflow is to pull text off of a forum(s) and flip through to the next page to fulfill the same task. Loop until the last page is equal to itself, i.e. there are no more pages left. I’m unsure of two things:

A) Apriori to recursive loops and pagination, what might be causing the error on my second loop? A lot of missing data and the columns sizes are all out of wack.

B) How can I make flow variables to handle pagination with a workflow like this.

I realize that my workflow is very elementary. Thanks in advance

[ArjenEX]

Hi @dinkyc

Welcome to the KNIME community!

There is some reading material available about this:

https://knime.tips/paginating-get-results-via-recursion-loop-in-knime/

An example that deals with paginating through API calls, which is comparable to your situation.

Loop Variables for API

@badger101 Went back and fourth a few times between different setups but ended up with these two that stuck. They are producing the same output. [image] General notes: Recursive Loop is the way to go here. It’s tricky to master but it has to be used since the loop needs to “remember” what was just queried and what the URL of the next GET request should subsequently be. You need to properly control the exit of the loop. In this case, whenever the nextPageToken is not retrievable it means no…

In general, it is impossible to troubleshoot these kind of web interaction workflows without seeing it in action. It’s all down to the configuration of your nodes. Moreover, it is not clear which forum you are working with, how it is coded, etc, etc. As such, please upload your workflow and provide your current vs expected output.

You can use the Export KNIME Workflow option for this.