I am working on building a KNIME workflow to handle paginated API requests, where the response includes a “hasNextPage” value to indicate if more data is available. Here’s what I want to achieve:
Initial API Call: Perform the first GET request to fetch data.
Process the Response: Convert the JSON response to a table and filter the column containing “hasNextPage” to check if it’s true.
Dynamic URL Creation: If “hasNextPage” is true, dynamically generate the next URL using the “endcolumn” value from the API response.
Repeat Requests: Continue making GET requests in a loop, updating the URL dynamically each time, until “hasNextPage” is false.
Combine Data: Append all the fetched data into a single table.
Here’s what I’ve done so far:
Used the GET Request node to fetch the first page.
Parsed the JSON response using JSON Path and JSON to Table nodes.
Filtered rows to check for “hasNextPage” = true and extracted the “endColumn” value for the next request.
Tried to pass the dynamic URL into a loop using the Table Row to Variable Loop Start and String Manipulation nodes.
However, I’m facing challenges with:
Ensuring the dynamic URL is updated correctly in the loop for each subsequent request.
Breaking the loop when “hasNextPage” becomes false.