Loop through api with pagination

Hello.

I need your help.
I have an API with pagination. And send request with parameter (date). Server returns first page with some IDs and url for next page.
So I don’t know how make such loop which can do:

  • get page by url
  • if page is empty - break the loop
  • if page have data write it and start new iteration with new page url
    What kind of loop I need to choose?

Hi @dbolshev , this sounds like a “do … while” loop, and for this kind of loop, you can use the Generic Loop and end the loop with a Variable Condition Loop End. So, basically start your loop, call your api, and if result is empty, set exit loop to true.

They’re these nodes:

And here’s an example of how to use them:

2 Likes

Hello, @bruno29a Thanks a lot!

Hello,
for looping through the pages I also found the Recursive Loop very useful. Especially in cases where you have not only one query, but several different queries, which might have a different number of pages each. Let me know if you want more information on how to set this up.
If you only have one query, the solution by @bruno29a will be easier.

1 Like

@daniela_digles Hello. Thanks for your post. At the end I also chose recursive loop. I send url of the next page to the next iteration. If body of page is empty I finish my loop with special value of variable.

Are you referring to a recursive loop because you wanted to always take the top row (url) and fed the other rows in the next iteration? Or is there any specific reason for this? Thx

@Daniel_Weikert Hello. In this API you receive a page with N elements and url of the next page. So at the end of current iteration you need to send url of the next page to the next iteration.

ok thanks for clarification,
if the next page would have been defined by the url itself at the end like "https://…page=1
then my first idea would not have been a recursive loop.
br

On every page I have specified number of items (by default 100). I check the number of items on each iteration. If the number of items less then 100 than it is the last page. In this case we end the loop by variable value.

Hello @Daniel_Weikert,
yes, initially I also used the current loop iteration variable to build up the url with the page. But I often want to run several queries at once, and not all of them return the same amount of data (and therefore pages). With the recursive loop, I only feed the available “next page” urls back to the beginning of the loop, and don’t collect empty pages for the finished queries.

Hello @dbolshev,
by chance it could happen that the last page has exactly 100 items. Be careful that you don’t create an endless loop in this case. Does the API still generate a “next page” uri even though there is no data left? If this is not the case, and you only feed the next page uri to the lower recursive loop end port, it will automatically end if it is empty.

In this API in such situation next page will exist but will be empty. So my workflow will work correctly. But thanks for your warning anyway :slight_smile:

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.