Hi @natanzi @aworker , the most important thing here is to create delays between the requests, or Knime will try to send 40k requests at once for 40k URLs, or you can risk getting banned from
I’m not sure about the HTTP Retriever, but the GET Request node has this option embeded:
Delay is the delay in ms between each batch, and Concurrency means how many requests per batch.
Alternatively if you still want to use the HTTP Retriever, then use a Chunk Loop where you can configure the batch size, and you can use the Wait node to create the delay between each iteration of the loop.
You should check on whois.nic.ir what are the limits it allows you to send. If you respect their limits, you should not encounter any errors.