If I have understood the issue correctly, you receive responses as 10000 per page and the responses with “sgtepagina” include more than one page, so you have to send another request for the next pages, right?
If yes, is there an endpoint to get the total number of results before requesting them?
If not, I need more information about how you handle those with more than 10000 at the moment then we may be able to optimize your current approach.
Hello ! thank you for your answer. Indeed, if I call the API and it has more than 10,000 records, then from 10,001 to 9,999 if it has, it will always bring information in the sgtepagina field, which is used to call the API again, placing that parameter in the request.
There is no endpoint to get the total results before requesting them.
Currently what I do is manually. I call the API, and once I process the json, I see if sgtepagina comes empty or with data. If it comes with data, then I call the API again, as the following image shows
If in this second call comes the field in sgtepagina, then I copy/paste again all the nodes, change the parameter of sgtepagina by the value that the call gives me, get the next 10.000 data and at the end I concatenate
In this approach, first, you would prepare the initial requests and then send them to the recursive loop. In the loop, you send the requests and parse them then send them to the top port of the loop end node. Also, you would create a new branch with values containing the “sgtepagina” value by using the Row Filter node and prepare the requests before sending them to the bottom port of the loop end node.
In the first iteration, your initial requests would be sent and in the next requests, data from the second input port of the loop end node would be used to feed the loop.
When there is no response with “sgtepagina” value, the loop will end.
You can also read more about the recursive loop here.
Thank you very much for your reply ! I wanted to know what exactly should go in the first string manipulation before it enters the loop and if the url of the request should go with the parameters as a variable
It’s just an example. It just means that you prepare all the requests in this step and then send them to the Get Request node in the loop. So the input for Get Request would be the same in all iterations since you prepare the next requests before the loop end (bottom port).
I assumed that the response from the first set of requests are the same as the ones you get for the next page requests and you follow the same approach to parse them.
I understand, I tried to take your idea to understand better and I get the following message “ERROR Recursive Loop End 9:88 Execute failed: Cell count in row “Row0#1” is not equal to length of column names array: 10 vs. 6”
Checking it is for the fact that I get repeated columns and add the name (#)
Yes, you need to pass the same table structure to the loop end in each iteration. If you clean the table which you send to the bottom port of the end loop, and make it look like the table you sent to the loop start node, then I think it would solve the problem here.
The Recursive Loop End node won’t accept changing table specifications unless the number of columns and the column types (based on column index) is the same in all iterations. Apparently, this node takes the column names and types from the first iteration output and then fills the next rows based on column index if the column type matches. So, if there is a mismatched column type or column index, it will fail. This behavior doesn’t apply to “Collect data from last iteration only” mode in which it collects data from the last iteration regardless of table spec in previous iterations.