How to consume messages from Kafka using KNIME Kafka extension

KNIME Extensions
2

Hey @jbuusao,

for more details on consumer groups please have a look at https://kafka.apache.org/documentation/#intro_consumers.

Now to your problem. When you create a new consumer group this group will start reading all entries that have been created after this consumer group has been created, i.e., if you have already pushed msgs to your test topic and execute the consumer afterwards with a newly created group this node will return an empty table.

How to solve this problem

  1. Push new data to the same topic (after you’ve created that consumer group) and either use the stop #msg criterion or change the timestamp to, e.g., use execution time & data
  2. In the consumer go to the Advanced settings an add the entry auto.offset.reset value earliest this way also new consumer groups will read from the beginning of that topic, i.e., every single msg that is still available in that topic. Note that using earliest however will not change anything for already existent groups. This post might be useful Knime Kafka nodes - kafka version.

I really hope this solves your problem. If not don’t hesitate to reach out to me.

Best and happy KNIMing
Mark

1 Like