KNIME + CLOUD (GCP/AWS) and Streaming Data

KNIME Extensions
,
1

Anyone has some advices? Can KNIME be used to cooperate with KAFKA or equivalents like AWS (Kinesis). Are there any cloud connectors to service AWS/GCP?

2

All I’ve done was connect KNIME with GCP, but it doesn’t have any native connectors. I used a python script node and just made the connection there and fetched the data.

3

And what you think about KNIME and Streaming? Can KNIME be used to successfully analyse streaming data - some kind of Workflow Daemon equivalent? Or its better to just create daemon script that would be do it? Just thinking about it - how to tackle something like that…

4

Hi @zebov
we have Kafka nodes that you can use to connect, produce and consume to/from Kafka.

We do not have dedicated nodes for Kineses, but you may be able to use the Javs Snippet nodes with the Java KCL and KPL. However I personally have not tried that yet.

best regards Mareike

5

Great! Thanks for answer.

6

Hey I’m sorry, since you didn’t tag me, I didn’t receive the notification. I tested this in production environment and it works yeah. If you have a KNIME Server you can set the workflow to run on a a schedule, so set it to run every minute and it will retrieve all data added to the cloud between executions.