I followed the (light) documentation on Kafka connector and Kafka consumer in order to consume from Kafka and yet, cannot receive any message.
My Kafka cluster is working, as I can post and consume messages using Python or Java client: in either case, I only need the cluster location (e.g. localhost:9092) and the topic to consume from (e.g. test).
So I configured the cluster location on the Kafka connector, that seem to connect fine. But on the Kafka consumer I don’t understand the concept of Client ID and Group ID, that I didn’t have to use to successfully produce and consume (with Python or Java).
If I set Client ID and Group ID to, say “KNIME” then I keep having “Node created an empty data table”, even though the topic has messages in it. And if I remove Client ID and Group ID I have an exception:
Can someone explain how to successfully consume Kafka using the KNIME Kafka connector and consumer?
Thanks in advance!
Hey @jbuusao,
for more details on consumer groups please have a look at https://kafka.apache.org/documentation/#intro_consumers.
Now to your problem. When you create a new consumer group this group will start reading all entries that have been created after this consumer group has been created, i.e., if you have already pushed msgs to your test topic and execute the consumer afterwards with a newly created group this node will return an empty table.
How to solve this problem
- Push new data to the same topic (after you’ve created that consumer group) and either use the stop #msg criterion or change the timestamp to, e.g., use execution time & data
- In the consumer go to the Advanced settings an add the entry auto.offset.reset value earliest this way also new consumer groups will read from the beginning of that topic, i.e., every single msg that is still available in that topic. Note that using earliest however will not change anything for already existent groups. This post might be useful Knime Kafka nodes - kafka version.
I really hope this solves your problem. If not don’t hesitate to reach out to me.
Best and happy KNIMing
Mark
1 Like
Hello all,
After configuring the Knime we got the following message after trying to get a message stream from KSQL .
After checking with Knime forum we found a same issue:
https://forum.knime.com/t/how-to-consume-messages-from-kafka-using-knime-kafka-extension/22601/2
we are checking this , please current message that we got.
