Kafka Node and Configuration of Cloud Kafka Provider (Cloud Karafka)

KNIME Analytics Platform
1

Hey guys,

I’ve been trying to setup the Kafka connector node and I just can’t figure out how to get it done. I am using Cloud Karafka as my cloud provider and they gave me some connections info but I can’t seem to figure out how to configure it. Cloud Karafka is free. Do you guys think you could create a free account and then connect KNIME to it and then share with us what to do there please?

2

Hey cageybee,

you are absolutely right; connecting to Cloud Karafka is tricky.

To my understanding you already have the Certs file? Otherwise please download it. These are the required steps to connect to Cloud Karafka.

  • Kafka Connector Dialog - Settings:

    1. Copy and paste the CLOUDKARAFKA_BROKERS list from the Certs into the Kafka cluster field (please remove the leading and trailing quotation marks).

    2. Just to make sure that the connector has enough time increase the validation timeout to, e.g., 5000.

Now comes the tricky part.

  • Kafka Connector Dialog - Advanced Settings:

    • Add the following three keys (you can change the key by clicking on it): security.protocol, sasl.mechanism, sasl.jaas.config and set the following values:

    • security.protocol: SASL_SSL

    • sasl.mechanism: SCRAM-SHA-256

    • sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="<user>" password="<pw>";

Please replace <user> by your username and_ <pw>_ by your password, see Certs. Click OK and you should be able to connect to the cluster.

Notes:

  • You can consume messages from and send messages to Topic(s) <user>-default .

  • For the Kafka Consumer node, I had to increase the Poll timeout (ms).

  • Setting auto.offset.reset to earliest in the Advanced Settings tab of the Kafka Consumer node might be of interest for you.

6 Likes
3

THANK YOU!!!

4

Hey, Mark. Today, I tried connecting to Kafka with the same Kafka Connector we got to work yesterday and I am getting an error now connecting. Also, I can’t seem to get the Kafka Producer node to work either. Could you take a look at this too, please? Thank you!

5

It all works now.
image

1 Like
6

That’s great news!

7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.