I have been looking to work with timeseries from an internal database using the GET Request. Since these timeseries are quite long, I seem to only get a result for a day or two (see fig 1) but not for more than 2 days (see fig 2). So I would guess that there is a limit to the amount of data that the GET request provides.
To check, I pasted the URL into an explorer and got the entire result without any delay. Memory space is also unlikely to be the issue, given the significant resources allocated to KNIME.
So is there a chance this is a problem with Knime? (currently using version 4.7.3)
Or is there any additional way to configure the GET Request Node?
@Daniel_Weikert
I agree that a loop might be a quick fix. However, the plan for this workflow is to get data for fairly long time series, and looping over days over several years would most likely slow down the workflow significantly.
The option in the Error Handling tab provides us with the following reply. Unfortunately, this is also no different from the info we had so far.
My guess here is that the json is cut at some point which leads to a syntax error in the json format.
Finally, I also tried the suggestion using the POST Method. I don’t believe that the API accepts that leading me into an error 405.
Interestingly, I have another workflow that I’ve been using for a while where the GET request, which returns a very large JSON code, works fine. It is a different API but the node configuration is identical…
Do you have another idea where some configuration might be missing?
Short the period of your request, it’ll bring less data from the request and you can test if the problem can be only about the content or a limitation from knime.
I saw at the knime’s preferences a setting that can help/test for this moment… called “Preferred Renderers” > “Others” > “HTTP Result”
By default, it’s set to result firsts 1000 characters, but you can change it to full. MAYBE it can help you.
The response from the server using post said that you need to inform a media type at the header request… try to set as “Content-type” “application/json” and “Accept” “application/json”.
POST method works with large pack of data, Get can have some limitations.
Check if the endpoint works with sync or assync information too.
Can you try to extract all HTTP headers from the response into columns? It could be that the server is sending a chunked response which - ironically - I believe the REST nodes cannot deal with properly. You should see this in the Transfer-Encoding response header.
Hi @thor , but the node was set to large data in chunk, so I understand and agree with you to add some information throw the response headers to see something else…
Yes loops are generally slow, maybe parallel chunk loop could help a little bit. If not to much overhead I would give it a try just to see the performance. Also have you looked at the api doc? Maybe some additional useful info there
br
I did look at the API doc and there are no anomalies. As for the chunk loop, I agree that it might work better than a regular loop, but my reservations performance wise however still remain.
Great, thank you very much. In the meantime I will try to work my way around this issue with some loops or even better some temporary database storing. Let’s see what works best.
