Kafka in Knime

KNIME Analytics Platform
1

Hii,

             How can i use Kafka through Knime. Please get me some solutions.

 

Regards,

Prasanna

2

Hi Prasanna,

I do know Kafka, but I suppose it is not about books in this case.

Which Kafka methodology are you refering? Pure guessing: maybe our external tools node can help you.

Best, Iris

3

I guess Prasanna was referring to Apache Kafka.

4

Hi,

              Yeah ! am looking for Apache Kafka Only. Upto my knowledge there is no direct nodes available to connect to Kafka. Am looking in an alternate way to connect to Kafka. 

              REST nodes are available in KNIME. If i i get any video on REST nodes usage in detail it will be much helpful.

        

Regards,

Prasanna

5

Hi,

there is a really nice documentation for the rest nodes you can find it here https://tech.knime.org/book/krest-rest-nodes-for-knime-trusted-extension

 

Best, Iris

6

Hi,

         Yeah. I will look at this document and whether spark streaming is possible in KNIME?

 

Regards,

Prasanna

7

Hello

I am interested to know if Knime can be integrated with Apache Kafka in order to implement a streaming analyzis?

if not, does Knime plan to create a such connector in its roadmap? if yes, when will the connector be available?

if Knime does not plan to build a such connector, what is the alternative suggested by Knime to implement a streaming analyzis.

Thanks beforehand for your answer

regards, Paul

 

 

 

8

Hello Paul,

KNIME supports streaming execution and analysis as described in the Streaming data in KNIME blog post. However I don't think that a Kafka consumer/producer exists as of yet. Since KNIME is Open Source anyone with some Java experience can implement new nodes. For a starting point have a look at the KNIME Developer Guide.

We are watching Kafka but do not plan an integration in the near future. However this might change depending on user demand.

Bye

Tobias

9

This is more an FYI. We pretty much had the same questions as Paul. We want to trigger an analysis each time a new event/msg is found on a kafka topic. If there are details on this we would be very happy to have them.

Ron

10

Hi all,

since we had the same needs of a Kafka Consumer - and Kafka Producer node, we just developed these (currently quite rudimentary) nodes for ourselves by using the KNIME-SDK. If interested, I can provide you with these Beta-Nodes.
Basically the real challenge is the current streaming behavior of the KNIME sink nodes running in KNIME-Streaming mode. In my opinion, the current streaming sinks are not able - or only under specific constraints - to process data packets (batches), which are produced by a permanent (ever-ongoing) streaming source correctly.

The reason seems to be that the current streaming sinks (for example, "CSV Writer" or "Database Writer") first buffer the received data packets internally and partially flush these packets only after (!) completion of the workflow. In continuous workflows this situation leads to situations in which the sinks write either nothing or incomplete partial deliveries (e.g. on disk or database).

For this reason, we have also written our own CSV and database sink nodes, which always flush data packets after receiving,   regardless of whether the Worlfklow has been terminated or not. 

Greetings,

Stefan

 

11

Hi spost,
Would you be willing to share your custom nodes that enable for Kafka streaming?
I don’t want to recreate the wheel if a possible solution already exists…

Thanks!
DK

12

Hi Prasanna,

it took us some time, but KNIME finally supports Kafka, see Apache Kafka integration.

Best
Mark

1 Like