Loop for only specific number of times within specific number of hours (because API/site has low request limit per day)

Hi all, hope you are doing well.
I am interacting with a site or API (unofficial backdoor API) that has an un-publicized quota of, let’s say, 10,000 requests per day. If I have 30,000 row job, I want to chunk 10,000 rows per every 24 hours for 3 consecutive days unattended. If I let it go beyond the quota limit, it’ll eventually block the IP for an extended period of time.

I looked at Chunk Loop Run-to-time Loop Start and Chunk Loop Run-for-time Loop Start and both don’t seem to quite do exactly what I need it to do. Is there any other nodes or a combination of that can work here? Or do I need to build a custom logic? And also can a workflow run for days?

Thank you very much in advance for your time and help. Stay safe.

you can try wait node inside the loop

1 Like

Hi @chandrika_99, thank you for your reply.

I have something like this (still testing out the loop portion). With the Wait node, how do I limit the # of rows to process to a specific count per day? Are you suggesting 24 hours divided by # of rows to process ==> use that #of mins as Wait time as variable? Thanks for your help.

image

My suggestion is…
Don’t use Chunk loop Run To Time loop start. Just use Chunk loop start (by specifying the number of rows as 10000) that will do your job. Use the Wait loop in between Table row to variable and Try Nodes. Try for few rows and two minutes let me know if you face any issues.
Hope this works.
image
This what i have done

3 Likes

@chandrika_99, that makes sense. Let me try that, and THANK YOU SO MUCH!!

In case you use the GET Request node to access the API you could also try this without a loop: it has an option called delay, where you can specify how long the node should wait between two requests.

3 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.