Knime Error GET/POST/DELETE Requests

Hello,

recently I got serious problems when sending requests via Knime. In the last years everything worked fine, but recently if I send multiple requests sometimes they get me my desired result, but sometimes I only get a ? as response. See:

The “funny” thing is, that this depends not on my request settings, because if I restart the node it will fail at different rows and work for those, which we here declared with a questionmark. This behaviour is present on the Knime Server and locally and for completely different services from different systems. Has anyone of you ever seen this weird behaviour? Increasing timeout or delay didn’t work for me.

I can clearly see when the node will produce those errors, because then there is not percent mark below the node when processing.

Best regards
Armin

Hi @ArminFan , I’m guessing you are NOT sending these in batches? If that is the case, then you are abusing the server and flooding it with the requests, and that is why sometimes it works and other times it does not, and at different rows.

You need to limit the amount of requests you are sending at a time and put some delays in between the requests, or you will get this behaviour, and eventually can be banned from the api server.

The GET Request already has a feature/option included for that in the Delay and Concurrency:

Delay represents the amount in milliseconds (ms) of delay between each request, and Concurrency represents the amount of requests to send. In my example, I set it up to send 5 requests with a wait of 2 seconds (1000 ms = 1 sec) in between.

You may want to check the api documentation for the allowed limit

1 Like

Hi @bruno29a,

thanks for your answer. I tried everything out and was sending 45 requests (45 rows) to the API with concurrency = 5 and Delay = 2000ms. I even increased the Timeout to 20 seconds, but still got one row with a questionmark…

Regards,
Armin

@bruno29a

The “funny” thing is, that the same workflow works totally fine at a colleagues machine, but not on mine at home.

Hi @ArminFan ,

There might be some network issue too indeed. You can try to decrease the concurrency to 3, which should take 30 seconds to run for the 45 requests. You can even decrease to 1, which will take 1.5 minutes (90 seconds) to run for the 45 requests.

Alternatively, you can split the results that are empty “?” and re-run them, and then concatenate the whole thing. Something like this:
image

Your first GET Request can be more aggressive (Concurrency 5, Delay 2000), which will run faster but with bigger risk of failing, and your second GET Request can be more conservative (Concurrency 1, Delay 2000-3000ms)

1 Like

@ArminFan
Sorry have to ask. Are you working for a german pharmaceutical company?

regarding the issue. Can you check the logs? Is there any hint regarding the issue. Normally an error mesage tells the story on what’s going on.
br

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.