Can we deploy Kafka connect which uses External Kafka clusters ?

[prakashpsgcse]

We have Strimzi operator deployed in k8s for full scale clusters and we have Kafka clusters running in VM's as well . We use standalone user operator to manage Users . Can we deploy KafkaConnect in k8s which points to External VM based Kafka cluster ?

Can we specify External ips (X:9094,Y:9094) in KafkaConnect CR ?

spec:
authentication:
bootstrapServers: 9094,Y:9094

[prakashpsgcse]

In latest version 0.46, https://strimzi.io/docs/operators/latest/configuring.html

81. KafkaConnectSpec schema reference

Exceptions

You can specify and configure the options listed in the Apache Kafka documentation.

However, Strimzi takes care of configuring and managing options related to the following, which cannot be changed:

Kafka cluster bootstrap address

Security (encryption, authentication, and authorization)

Listener and REST interface configuration

Plugin path configuration

Properties with the following prefixes cannot be set:

bootstrap.servers

consumer.interceptor.classes

listeners.

[scholzj]

You can deploy Kafka Connect with Strimzi and connect it to any Kafka cluster (assuming it uses the supported authentication etc.). The configuration works as with a Strimzi-based Kafka cluster. So you for example specify the bootstrap servers in .spec.bootstrapServers in KafkaConnect CR etc.

[prakashpsgcse]

Thanks for your answer. So whatever mentioned in Exception section of KafkaConnectSpec is not applicable or its it like once we deploy kafkaConnect CR then it cannot be changed ?

[scholzj]

That is about the .spec.config section. You cannot specify the bootstrap.servers there. As I said, you have to configure this in spec.bootstrapServers.

Changing the spec.bootstrapServers field is possible even after the cluster is deployed. But ... normally it should be the the same Kafka cluster. Otherwise you would be basically deploying a new cluster.