REST nodes (GET/POST) not streaming

[pfpmeijers]

Hello, in an older thread POST Node and Streaming a problem was identified with POST request not streaming. A ticket was added for this.
In latest release the problem is still there. Is there a ticket number for tracking the status?
Is there also some more info (example) of the proposed workaround?
Thanks in advance.
PS, the original topic was closed so I could not reply directly to that and instead had to open a new topic.

[ipazin]

Hi there @pfpmeijers,

I have added +1 on it. (Internal reference: AP-13322).

@pfpmeijers

Is there a ticket number for tracking the status?

Ticketing system is not public.

@pfpmeijers

Is there also some more info (example) of the proposed workaround?

You mean for the workaround proposed by @hornm?

@pfpmeijers

the original topic was closed so I could not reply directly to that and instead had to open a new topic.

You did exactly as suggested in this topic

Bug Reporting Best Practices

Hello Bug Reporters! I would like to share some of best practices on reporting bugs you encounter while working with KNIME. First to say tnx to those who are already following these or similar practices and encouraging others to do so. Secondly, as much as bugs are annoying and not pleasant to see I think that this kind of feedback is also very valuable and hopefully there will be once a series of badges to acknowledge it. Finally to say that place for reporting bugs is by opening a topic on K…

Br,
Ivan

[pfpmeijers]

Indeed I meant the workaround by @hornm by splitting and merging.
Thanks in advance for any more explanation on this.

[hornm]

Hi @pfpmeijers,

sorry for getting back to you so late.

The respective ticket to improve the ‘streamability’ of the REST nodes is still in our backlog, unfortunately (i.e. not yet tackled but definitely something we won’t loose sight of).

If it’s about the parallelization of REST node(s) (e.g. POST Request) it can also be done by using the parallel chunk loop start/end. This, however, requires the called web service being able to deal with requests in rapid succession.

If the REST node is the bottleneck in a longer and otherwise completely streamable workflow, I don’t have a good idea except this ugly workaround (as suggested earlier):

• split your input data into multiple ports (e.g. using multiple Partitioning nodes), either right at the beginning in the streamed component or outside of it; the more ports the higher the parallelization obviously
• process the data at the multiple ports the exact same way by duplicating the respective (streamable) workflow branch
• concatenate the final output back into one table again (either inside or outside of the component).

Let me know if you need more details.

Best,
Martin

[pfpmeijers]

Thanks for the reply.

“This, however, requires the called web service being able to deal with requests in rapid succession”

Isn’t that also the case when the REST node is properly streaming?
I thought that for any row in the input table a separate REST call would be invoked?

[hornm]

@pfpmeijers

“This, however, requires the called web service being able to deal with requests in rapid succession”

Isn’t that also the case when the REST node is properly streaming?
I thought that for any row in the input table a separate REST call would be invoked?

Yes, that’s right. However, for the REST nodes you can specify a delay between consecutive requests - which obviously would also slow down the streaming (if supported properly).

One more thing: There is also a ‘concurrency’ option to specify the number of concurrent requests. Another possible approach to parallelize http requests.