Loop an API call with JSON field

I have the above workflow - I have a GET Request that has a URL (and credentials) to pull data from an API. Unfortunately, the API limits data pulls to 300 records. I have several thousand records to pull. IN the API / JSON output, there is a field called “next”.

I need to loop that “next” field back into the GET Request, overwriting the original URL and keep looping until there are no more “next” fields. I did some research and saw that I need to use the Loop start, then create a table row to variable row to extract that ‘next’ field, then feed back into the GET Request as a flow variable? But the input is already taken by the loop start. How do I start the first call with the original API, then extract the 'next; field and insert back into the URL field of the GET node, and continually looping until complete?

Hi @ebarr,

You can use the Variable Condition Loop End node to close the Generic Loop Start. The condition here should be based on the “next” value.

If you get the URL for the next page in the “next” field, then I would suggest Recursive Loop Start and Recursive Loop End nodes instead. Where you end the loop based on the existence of the next value.

Here you can read more about loops:

1 Like

Hi @ebarr

This could also help to get started:

2 Likes

I was initially having some troubles as YouTube has a unique key to ride along with URL, whereas my use case simply has a unique URL (called “next”) after every run. Took me a bit to understand what I was doing, but I have been able to get the required data and recursive loop.

Thank you for the detailed tutorial, was very helpful. I’m impressed with the KNIME community. I used to use Alteryx and am finding KNIME more powerful with a community that has really advanced my understanding.

3 Likes

Great to hear that it was useful! The KNIME Community is definitely a good place to be :+1:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.