GET Request Node

[MoonUnit]

I want to automate data retrieval from a site called Stock Data. https://www.stockdata.org/documentation.

I’ve struggled with the GET Request node because while I can get a JSON table, I cant figure out how to transform the output from the request into rows and columns that are useable. The KNIME example workflow is too simplistic to help. Any ideas on how I can find out how to configure this node so that I could end up with data structured in rows and column, i.e. a csv file?

[umutcankurt]

Hello; can you share the link to the full page you are trying to get data from.

[ArjenEX]

Hi @MoonUnit

Welcome to the KNIME Community!

You can achieve this with the JSON Path node. Some examples are here:

JSON Path

But as mentioned by @umutcankurt, please share the JSON output that you got and your expected output in terms of rows and columns so that people can help you more effectively.

[MoonUnit]

I’m looking to retrieve data from stockdata.org where I have a subscription to access stock data via their rest api. You can see more information from their documentation page, which I previously referenced. I’ve included a screenshot of the specific endpoint (adjusted intraday) which is the subject of this issue.

[MoonUnit]

Thanks for your reply and willingness to help me on this. I’ve included screenshots of my custom workflow, the KNIME sample workflow and some of the associated customer configurations as well as a .csv output file which represents my end goal for what I’d like the data to look like.

A second related question has come up. The KNIME sample method calls for a parameter table with which to construct GET urls. Do you have a suggestion on how to configure the workflow to construct subsequent GET calls based on an initial table of multiple rows?
KNIME Sample -

Current custom workflow-

JSON Path Node Configuration -

I’m unable to convert this JSON Path node output to anything but it does seem like the columns I’ve attempted to create yield some sort of string date. JSON to Table does not interact with the output however.

Desired output-

Suggestions?

[umutcankurt]

Can you add the workflow you’ve already created here? without reset

Since resources are created with different infrastructures in data mining, you may need to proceed with many different solution methods.

Sometimes just json, html, xml, selenium nodes with last resort browser behavior, sometimes json to xml from there to result… etc. etc.

If you can add the workflow, I’ll try to look at the workflow you created when I have time. Or the experts in the form will definitely help.

[MoonUnit]

Thanks. I was actually able to achieve the general results I was looking for by changing the JSON path column configurations in the JSON Path node to something like this: $[‘data’][*][‘data’][‘open’]. This unlocked the data for me.

Now the question is do you have a suggestion on repeating the construction of GET request urls from multiple rows of the REST request table and then appending the successive results to a table?

My data provider throttles the data so I have to make multiple requests.

[umutcankurt]

This answer helps with the solution.

String to JSON help to fix the error - code 83

Hi @umutcankurt , I’ve finally downloaded and imported your workflow. If you are going to hit the website multiple times, I suggest you add some delays in between each request. You can do this directly in the GET Request: [image] It takes values in milliseconds (ms), so in my example, I’m setting a 3 seconds delay between each request (3000 ms). From your Table Creator, I see that you have 6 pages only (0 to 5). Are you only interested in the first 6 pages? I checked the website, and it has …

[Daniel_Weikert]

The get request can use a table as input and applies the request for each row. So if you put the specific url in the rows. Each request would be excecuted
br