REST nodes (GET/POST) not streaming

Hello, in an older thread POST Node and Streaming a problem was identified with POST request not streaming. A ticket was added for this.
In latest release the problem is still there. Is there a ticket number for tracking the status?
Is there also some more info (example) of the proposed workaround?
Thanks in advance.
PS, the original topic was closed so I could not reply directly to that and instead had to open a new topic.

Hi there @pfpmeijers,

I have added +1 on it. (Internal reference: AP-13322).

Ticketing system is not public.

You mean for the workaround proposed by @hornm?

You did exactly as suggested in this topic :smiley:

Br,
Ivan

Indeed I meant the workaround by @hornm by splitting and merging.
Thanks in advance for any more explanation on this.

Hi @pfpmeijers,

sorry for getting back to you so late.

The respective ticket to improve the ‘streamability’ of the REST nodes is still in our backlog, unfortunately (i.e. not yet tackled but definitely something we won’t loose sight of).

If it’s about the parallelization of REST node(s) (e.g. POST Request) it can also be done by using the parallel chunk loop start/end. This, however, requires the called web service being able to deal with requests in rapid succession.

If the REST node is the bottleneck in a longer and otherwise completely streamable workflow, I don’t have a good idea except this ugly workaround (as suggested earlier):

  • split your input data into multiple ports (e.g. using multiple Partitioning nodes), either right at the beginning in the streamed component or outside of it; the more ports the higher the parallelization obviously
  • process the data at the multiple ports the exact same way by duplicating the respective (streamable) workflow branch
  • concatenate the final output back into one table again (either inside or outside of the component).

Let me know if you need more details.

Best,
Martin

2 Likes

Thanks for the reply.

“This, however, requires the called web service being able to deal with requests in rapid succession”

Isn’t that also the case when the REST node is properly streaming?
I thought that for any row in the input table a separate REST call would be invoked?

Yes, that’s right. However, for the REST nodes you can specify a delay between consecutive requests - which obviously would also slow down the streaming (if supported properly).

One more thing: There is also a ‘concurrency’ option to specify the number of concurrent requests. Another possible approach to parallelize http requests.