Sending multiple GET requests with different page parameter

[aalpersa]

Hello guys,
I want to retrieve data from an API and it only supports 500 entries in one call. That is why I need to make more than one Api call to access all data. After that I want to put them in a table. If you can guide me in that manner, I would be happy.

Here is my example url:https://s/api/order&limit=500&currentPage=2. Current page parameter will change in every call.

I think I need to do a loop, but I am fairly new to KNIME

Thank you.

[Daniel_Weikert]

Welcome
Have a look at the table row to variable loop start or counting loop start. You should find examples on the KNIME Hub
examples can be found here

Loops | KNIME

br

[Kyle_Watkins]

Hi @aalpersa!

I made a sample workflow you could try out to make multiple GET requests that iterates through and passes a new page parameter with each call. My example is pretty basic and only will iterate from pages 0 to a max number set in the config of the Counting Loop Start, but it could be a good starting place.

API Loop.knar.knwf (17.7 KB)

Best regards,
Kyle

[aalpersa]

Hi @Kyle_Watkins, thanks for your fast and nice example. However, I need to solve one problem here. I have a number of pages parameter, and I need to start and iterate the loop according to that number. I also need to pass that one in to my GET Request Node, and combine all data into a single table. Can you help me in this manner. I already got the number of page parameter from my request .

[aalpersa]

Hey @Kyle_Watkins,
Here I made this workflow,

The number of loops needs to be set dynamically since number of pages are dynamic (number of pages should be equal to the number of iterations). And I also need to save all the data into a one table in each loop. However, it is now not saving and I only be able to see the last iteration data. Before the Loop End Node, I need to save the data.

[Kyle_Watkins]

Hi @aalpersa,

To make the loop dynamic for your GET requests, I think the best way would be to start with a Generic Loop Start node, and end with a Variable Condition Loop End node.

Then you take your output from the GET request, turn the “status” into a flow variable, and check the status in the config of the Variable Condition Loop End node and stop the loop whenever the status is missing or provides a number thats different than when the GET request is successful. The loop end nodes should concatenate all of the rows from all of the get requests as long as you don’t have “Collect rows from last iteration only” checked.

Best Regards,
Kyle

[gonhaddock]

Hello @aalpersa
Just trying to bring one more example to discussion… you may want to check this working workflow to get insights.

Just KNIME It _ Challenge 007 – gonhaddock

BR