Recursive Loops and Pagination For Scraping


I am new to KNIME and have been experiencing some problems while using a recursive loop in my workflow. I’m using KNIME 4.7.0.

Essentially, the purpose of my workflow is to pull text off of a forum(s) and flip through to the next page to fulfill the same task. Loop until the last page is equal to itself, i.e. there are no more pages left. I’m unsure of two things:

A) Apriori to recursive loops and pagination, what might be causing the error on my second loop? A lot of missing data and the columns sizes are all out of wack.

B) How can I make flow variables to handle pagination with a workflow like this.

I realize that my workflow is very elementary. Thanks in advance

Hi @dinkyc

Welcome to the KNIME community!

There is some reading material available about this:

An example that deals with paginating through API calls, which is comparable to your situation.

In general, it is impossible to troubleshoot these kind of web interaction workflows without seeing it in action. It’s all down to the configuration of your nodes. Moreover, it is not clear which forum you are working with, how it is coded, etc, etc. As such, please upload your workflow and provide your current vs expected output.

You can use the Export KNIME Workflow option for this.


1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.