GET Request Node

I want to automate data retrieval from a site called Stock Data. https://www.stockdata.org/documentation.

I’ve struggled with the GET Request node because while I can get a JSON table, I cant figure out how to transform the output from the request into rows and columns that are useable. The KNIME example workflow is too simplistic to help. Any ideas on how I can find out how to configure this node so that I could end up with data structured in rows and column, i.e. a csv file?

Hello; can you share the link to the full page you are trying to get data from.

Hi @MoonUnit

Welcome to the KNIME Community!

You can achieve this with the JSON Path node. Some examples are here:

But as mentioned by @umutcankurt, please share the JSON output that you got and your expected output in terms of rows and columns so that people can help you more effectively.

3 Likes

I’m looking to retrieve data from stockdata.org where I have a subscription to access stock data via their rest api. You can see more information from their documentation page, which I previously referenced. I’ve included a screenshot of the specific endpoint (adjusted intraday) which is the subject of this issue.

Thanks for your reply and willingness to help me on this. I’ve included screenshots of my custom workflow, the KNIME sample workflow and some of the associated customer configurations as well as a .csv output file which represents my end goal for what I’d like the data to look like.

A second related question has come up. The KNIME sample method calls for a parameter table with which to construct GET urls. Do you have a suggestion on how to configure the workflow to construct subsequent GET calls based on an initial table of multiple rows?
KNIME Sample -
KNIME Custom Flow

Current custom workflow-

JSON Path Node Configuration -

I’m unable to convert this JSON Path node output to anything but it does seem like the columns I’ve attempted to create yield some sort of string date. JSON to Table does not interact with the output however.

Desired output-
desired_table

Suggestions?

1 Like

Can you add the workflow you’ve already created here? without reset

Since resources are created with different infrastructures in data mining, you may need to proceed with many different solution methods.

Sometimes just json, html, xml, selenium nodes with last resort browser behavior, sometimes json to xml from there to result… etc. etc.

If you can add the workflow, I’ll try to look at the workflow you created when I have time. Or the experts in the form will definitely help.

Thanks. I was actually able to achieve the general results I was looking for by changing the JSON path column configurations in the JSON Path node to something like this: $[‘data’][*][‘data’][‘open’]. This unlocked the data for me.

Now the question is do you have a suggestion on repeating the construction of GET request urls from multiple rows of the REST request table and then appending the successive results to a table?

My data provider throttles the data so I have to make multiple requests.

This answer helps with the solution.

image

1 Like

The get request can use a table as input and applies the request for each row. So if you put the specific url in the rows. Each request would be excecuted
br