Get Request Node

KNIME Analytics Platform
1

I have a Problem with a Get Request Node… The Nod Output stop by 100.000 Datarows…
Have someone a solution for this Problem?

1 Like
2

Hi,
Welcome to the KNIME Forum! Can you tell us a bit more about the workflow? Are you passing a table with more than 100k URLs to the Get Request and then it stops after 100k and is still in green state, or does it throw an error? I am not aware of any implemented limitations in the node, but you can try to work around this issue by using a Chunk Loop Start with a chunk size of 100k, followed by the GET Request node and then a Loop End node. This chunks your input table and calls the GET Request node multiple times. However, that would be a workaround. It should theoretically also work with more than 100k rows, but to analyze this we need more info about the workflow.
Kind regards,
Alexander

1 Like
4

We load data from an ODATA interface via the GET Request node.
The data source contains more than 100,000 records.
However, only 100,000 records are displayed in the Get Request node.
The Get Request Node is green after the execution.
In another post I read that the problem can be solved if the KNIME .ini is adjusted, with this additional line:

-Dorg.knime.container.cellsinmemory=10000000

I have set the value to 10,000,000. Unfortunately, that didn’t work.

5

Hi,
So the OData interface outputs a JSON containing more than 100k records? Are you sure you are not running in some API limitation or pagination? Because to KNIME all JSON is the same and if KNIME would just cut the JSON document somewhere, it would not be valid JSON anymore and could not be parsed. Can you run the same GET Request with your browser and check if you get back more results than with KNIME?
Kind regards,
Alexander

6

Hi,

the the OData interface outputs does not contain jason but an xlm format. Containing more than 100k records.
I fetched the data using the excel ODBC interface and received 197.539 data records in a excel file.
The WF contains the following nods: Get Request → XPAth → Column Filter

When I look into the body of the Get Request Node I see 4385891 Lines and in the last Data record I see the following line:
<d:ID m:type=“Edm.Int32”>100000</d:ID>

7

Hi,
In that case I assume that the ODBC interface either
a) sends some additional parameters to the endpoint to instruct it to return all/more values, or
b) does pagination, i.e. executes multiple requests in the background using the $skip and $top query options (see here for more info).
I think you might need to do multiple queries in a loop, e.g. using a Recursive Loop Start and Recursive Loop End and adapting the $top and $skip query options in each iteration. Another option would be the Interval Loop Start together with a Variable Condition Loop End.
Kind regards,
Alexander

3 Likes
8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.