POST Node and Streaming

The POST node seems to only partially support streaming in a way that makes the streaming less useful. Simply said the node waits till it has all records before it actual starts making requests. So if one has a somewhat long running process before the POST node, streaming doesn’t help to hide the POST node being slow as it will always wait for all data before starting to make requests.

The expected behavior would be for the node to start make request as soon as the first batch of rows is available.

Hi @beginner

my dev colleagues are currently checkin this.
We might need till next week, because we are currently in pre-Release. time :slight_smile:

Hi @beginner,

yes, that’s true, the rows are collected first which not optimal indeed. I opened a ticket to improve that.

The only way to get at least a bit more parallelization during the streaming I can think of right now, is to split your list of requests, feed them into multiple POST nodes in parallel and finally concatenate them (with 4.1, ports can be added dynamically to the Concatenate node :)).

(A note on the split: table rows are collected at source nodes of diamond shaped (<>) workflow structures, too - i.e. it might be beneficial to do the splitting outside of the streamed component)

Best,
Martin

3 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.