Selenium nodes: too many requests on internet page - how to bypass?

[brandy71]

Hi,

I have a table with a lot of URLs (roughly 2800) referring to notices on TED portal (Place of performance (Map) - TED Tenders Electronic Daily).

I want to save the source code of each URL. However, after 500 I receive an error since too many requests were sent and I can try in 24 hours again.

I noticed that having a different IP-address for the next 500 URLs can bypass this problem, but I am not sure how and if this can be implemented and if there are other possible ways to bypass this error.

Thx and kind regards

[qqilihq]

Please see here:

Help with 429 error on website

Hi, I think I already sent you an email about this. But as it’s probably relevant to all, I’ll paste my response here: Hi Fernando, I have just executed one iteration of the workflow and I do not receive any errors. Generally a 429 “too many requests” status indicates a rate limitation error. I’d guess that the page you’re accessing detects automated, bot-like behavior and thus throws the mentioned error. There’s a couple of options: (1) Try adding further and longer timeouts, (2) Use a …

HTH - Philipp