POST Request API speed

KNIME Analytics Platform
1

Hi,
We use the Post Request node to send data and then get a response back from API. Data size can range anywhere between 50K to 1 Million. We have a i7, 64GB RAM, KNIME 5.2. However, the speed for the API returns is very low. Sometimes it can take hours to 2 days. We moved to a faster upload speed- say going up to almost 500 MBPS (upload), changed xmx to 50G and added -Dorg.knime.container.cellsinmemory=10000000 to the config file but nothing has improved the speed of API returns. Can someone please help?

Yes, we can break files and run things in parallel but we want to improve speed, either way. What should I be doing?

Can someone please help.

Ace

2

If you are calling out to external HTTP APIs the bottleneck is the network latency. The bandwidth of your internet connection doesn’t even matter much neither do any memory settings in KNIME. You can increase the Concurrency setting in the dialog, which masks the latency a bit.

1 Like
3

Thanks, Thor. Right now, it’s 1. Can it go to any number (subject to API restrictions), or do you have any suggestions on a particular number?

4

This solely depends on the API that you are calling. Therefore I cannot suggest a number.

5

You can try making the request with a Python script (use the AI to write the script, check if it works and then ask it again to optimize the script and make it concurrent/multithread).

Compare the result vs using the POST request node combined with increasing concurrency and using the parallelism nodes.

6

Thanks. Will try that option too.

7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.