Whois domain with get request

[natanzi]

Dears,

I want to find the whois info by the getting method; Although there is a python library that address this case, I want to learn how to use the get method.

note:

Domain example: natanzi.ir

URL of NIC for.IR domain: http://whois.nic.ir/WHOIS?name=natanzi.ir

Does anybody know how it should be implemented?

Whois.nic.ir.knwf (13.0 KB)

[aworker]

Dear @natanzi

Please find below a possible solution to your need:

20220110 Pikairos Whois domain with get request.knwf (187.0 KB)

Hope this helps.

Best wishes

Ael

[natanzi]

Dear @aworker
Many thanks, its work.,
I have add 10 URL and get the below error, I don’t know the root cause, also its possible to send 40K URL? I mean we need to set 10ms interval maybe

ERROR HTTP Retriever       6:760      Execute failed: Row5: Error retrieving http://whois.nic.ir/WHOIS?name=izbank.ir: Exception org.apache.http.conn.ConnectTimeoutException: Connect to whois.nic.ir:80 timed out for URL "http://whois.nic.ir/WHOIS?name=izbank.ir": Connect to whois.nic.ir:80 timed out

[aworker]

Dear @natanzi

I have tried the faulty URL ("") but it worked for me and I cannot hence reproduce the error.

Having said this, one can play with the parameters:

It is possible for instance to set up a ‘number of retries after error’ and a ‘fail on network error’.
Thus, you could for instance try to increase the # of retries and to uncheck the ‘fail on network error’.

Hope this helps.

Best wishes

Ps: Thanks for having validated the solution

[bruno29a]

Hi @natanzi @aworker , the most important thing here is to create delays between the requests, or Knime will try to send 40k requests at once for 40k URLs, or you can risk getting banned from whois.nic.ir.

I’m not sure about the HTTP Retriever, but the GET Request node has this option embeded:

Delay is the delay in ms between each batch, and Concurrency means how many requests per batch.

Alternatively if you still want to use the HTTP Retriever, then use a Chunk Loop where you can configure the batch size, and you can use the Wait node to create the delay between each iteration of the loop.

You should check on whois.nic.ir what are the limits it allows you to send. If you respect their limits, you should not encounter any errors.

[natanzi]

Hi @bruno29a ,
Thanks, Yes I used chunk loop and its work for 40K URL.