I’m reading information about an API. The API brings 10000 records per call.
If there are more than 10000 records, a code appears in the sgtepagina column,
I don’t know how I could do to call the API iteratively in case sgtepagina is with values.
Thanks in advance
Hi @DanielaQ,
If I have understood the issue correctly, you receive responses as 10000 per page and the responses with “sgtepagina” include more than one page, so you have to send another request for the next pages, right?
If yes, is there an endpoint to get the total number of results before requesting them?
If not, I need more information about how you handle those with more than 10000 at the moment then we may be able to optimize your current approach.
Hello ! thank you for your answer. Indeed, if I call the API and it has more than 10,000 records, then from 10,001 to 9,999 if it has, it will always bring information in the sgtepagina field, which is used to call the API again, placing that parameter in the request.
There is no endpoint to get the total results before requesting them.
Currently what I do is manually. I call the API, and once I process the json, I see if sgtepagina comes empty or with data. If it comes with data, then I call the API again, as the following image shows
If in this second call comes the field in sgtepagina, then I copy/paste again all the nodes, change the parameter of sgtepagina by the value that the call gives me, get the next 10.000 data and at the end I concatenate
In this case, I’d suggest using Recursive Loop Start and Recursive Loop End nodes.
The flow would be something like below:
In this approach, first, you would prepare the initial requests and then send them to the recursive loop. In the loop, you send the requests and parse them then send them to the top port of the loop end node. Also, you would create a new branch with values containing the “sgtepagina” value by using the Row Filter node and prepare the requests before sending them to the bottom port of the loop end node.
In the first iteration, your initial requests would be sent and in the next requests, data from the second input port of the loop end node would be used to feed the loop.
When there is no response with “sgtepagina” value, the loop will end.
You can also read more about the recursive loop here.
Thank you very much for your reply ! I wanted to know what exactly should go in the first string manipulation before it enters the loop and if the url of the request should go with the parameters as a variable
It’s just an example. It just means that you prepare all the requests in this step and then send them to the Get Request node in the loop. So the input for Get Request would be the same in all iterations since you prepare the next requests before the loop end (bottom port).
I assumed that the response from the first set of requests are the same as the ones you get for the next page requests and you follow the same approach to parse them.
I understand, I tried to take your idea to understand better and I get the following message “ERROR Recursive Loop End 9:88 Execute failed: Cell count in row “Row0#1” is not equal to length of column names array: 10 vs. 6”
Checking it is for the fact that I get repeated columns and add the name (#)
How can I get out of this problem?
Yes, you need to pass the same table structure to the loop end in each iteration. If you clean the table which you send to the bottom port of the end loop, and make it look like the table you sent to the loop start node, then I think it would solve the problem here.
Hi guys,
Just a point here… “Loop End” node allow you to make changes at the table structure if the data changes too… Just mark the option as the print below.
Loop End node
As you can see here, you can set it, but check the result if is necessary to convert it to the right format/type too.
Thanks,
Denis
@denisfi Indeed, this option is for the Loop End node not the Recursive Loop End node which was suggested.
The Recursive Loop End node won’t accept changing table specifications unless the number of columns and the column types (based on column index) is the same in all iterations. Apparently, this node takes the column names and types from the first iteration output and then fills the next rows based on column index if the column type matches. So, if there is a mismatched column type or column index, it will fail. This behavior doesn’t apply to “Collect data from last iteration only” mode in which it collects data from the last iteration regardless of table spec in previous iterations.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.
