REST GET/POST Request rate limit

Is there a way to loop RESTful Requests with 429 status?

I’ve found the error handling in the GET Request node to be somewhat hard to optimize; even after I had a rough estimate of runtime to reduce the requests under the rate-limit, there are still small numbers of 429 in the results. I tried playing with the “retry after x seconds” option but it added too much delay in runtime.

The solution I’m thinking is a “brute force” conditional loop between the GET Request node that would check if the status of the request is 429 to repeat the request. What loop nodes would I need for something like this?

Hi @Muzume,

I didn’t quite wrap my head around why the “pause execution on 429” setting doesn’t work for you, but assuming that is the case, I would also opt for a loop around the request node.

Likely, the Recursive Loop is what you’d want here. I believe that will re-run with the results left in the top input table:

Some services may give additional information to the 429 response, e.g. how long you have to wait. If present, you can extract that duration and set the waiting time of the Wait node via flow variable.

Kind regards
Marvin

5 Likes

Thanks Marvin, your solution looks solid. As you’ve mentioned, I’ve managed something working with “pause execution on 429” but it still lets out a loose 429 once in a while (1-2 requests in 1000) so it was bugging me that I needed at least 2 GET nodes to run the workflow unsupervised. Definitely a learning experience, I was just curious on the “how” to implement the looping logic.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.