Save real-time data stream from an API

KNIME Analytics Platform
1

Hi there,

I’m trying to figure out how to save the data stream from an API with the weather forecast in a table. Do you know how to do that?

The purpose is to create historical data with the past hourly forecast.

I’m attaching the workflow that accesses the API.

Thank you.

Carlos

2
4

Hi @CarlosEnrique84

Directly tagging people after your post has been up for just one day is usually not appreciated by people :wink:

Anyway, what’s your desired output here? The data is in table format already right or am I missing the point? I see you hardcoded one city in the row filter so I assume to want to process this data for all cities?

2 Likes
5

Hello @ArjenEX,

I appreciate your kindness in responding to me.

I’m sorry for rushing.

I’m going to untag Markus.

That’s right. In my workflow, I am filtering a city, and the output of the Missing Value node is a table with the weather forecast for the next 48 hours in the city.

This forecast is updated every hour.

Can the hourly forecast for all cities be recorded/saved? I want to create a historical weather forecast database and compare it with the actual weather data.

Beforehand, thank you very much.

Carlos

6

@CarlosEnrique84 if the question is about storing the data. You could have a dedicated database (like Oracle or Postgres) or you could store your results locally with timestamps.

Either as for example parquet files that could then be read back as a large file or being archived by time if you include the timestamp in the name.

Or you could use a local SQL database like SQLite or H2 to insert the latest round of your data including a timestamp.

In any case you might want to invest in some planning of how to set such a system up and also how to run and backup the system to prevent the loss of data.

1 Like
7

Hi @mlauber71 !!!

Thank you so much for your response! I appreciate your time and dedication! The information you gave me is a completely new topic for me. I need to spend a lot of time trying to understand!

Have a nice week!

Carlos

1 Like
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.