Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confluent Cloud Topic Creation #36

Merged
merged 2 commits into from
Feb 6, 2025
Merged

Confluent Cloud Topic Creation #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cf60a45
adding confluent cloud support to jikkou for creating topics
Michael7371 Feb 6, 2025
94e4f13
adding confluent credentials to the sample.env and docker compose
Michael7371 Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The JPO ITS utilities repository serves as a central location for deploying open
- [Quick Run](#quick-run)
- [3. Kafka](#3-kafka)
- [Configure Topic Creation](#configure-topic-creation)
- [Confluent Cloud Support](#confluent-cloud-support)
- [Quick Run](#quick-run-1)
- [4. MongoDB Kafka Connect](#4-mongodb-kafka-connect)
- [Configuration](#configuration)
Expand Down Expand Up @@ -106,6 +107,17 @@ The following enviroment variables can be used to configure Kafka Topic creation
| `KAFKA_TOPIC_RETENTION_MS` | Retention time for stream topics, milliseconds |
| `KAFKA_TOPIC_DELETE_RETENTION_MS` | Tombstone retention time for compacted topics, milliseconds |

#### Confluent Cloud Support

The following environment variables are used to configure the Kafka client for Confluent Cloud.

| Environment Variable | Description |
|---|---|
| `KAFKA_SECURITY_PROTOCOL` | Security protocol for Kafka |
| `KAFKA_SASL_MECHANISM` | SASL mechanism for Kafka |
| `KAFKA_SASL_JAAS_CONFIG` | SASL JAAS configuration for Kafka |
| `KAFKA_SSL_ENDPOINT_ALGORITHM` | SSL endpoint algorithm for Kafka |

### Quick Run

1. Create a copy of `sample.env` and rename it to `.env`.
Expand Down
5 changes: 5 additions & 0 deletions docker-compose-kafka.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@ services:
KAFKA_TOPIC_CREATE_CONFLICTMONITOR: ${KAFKA_TOPIC_CREATE_CONFLICTMONITOR}
KAFKA_TOPIC_CREATE_DEDUPLICATOR: ${KAFKA_TOPIC_CREATE_DEDUPLICATOR}
KAFKA_TOPIC_CREATE_MECDEPOSIT: ${KAFKA_TOPIC_CREATE_MECDEPOSIT}

KAFKA_SECURITY_PROTOCOL: ${KAFKA_SECURITY_PROTOCOL:-PLAINTEXT}
KAFKA_SASL_MECHANISM: ${KAFKA_SASL_MECHANISM}
KAFKA_SASL_JAAS_CONFIG: ${KAFKA_SASL_JAAS_CONFIG}
KAFKA_SSL_ENDPOINT_ALGORITHM: ${KAFKA_SSL_ENDPOINT_ALGORITHM}
logging:
options:
max-size: "10m"
Expand Down
19 changes: 11 additions & 8 deletions jikkou/application.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

jikkou {

extension.providers {
Expand All @@ -23,13 +22,17 @@ jikkou {
client {
bootstrap.servers = "kafka:9092"
bootstrap.servers = ${?KAFKA_BOOTSTRAP_SERVERS}
# security.protocol = "SSL"
# ssl.keystore.location = "/tmp/client.keystore.p12"
# ssl.keystore.password = "password"
# ssl.keystore.type = "PKCS12"
# ssl.truststore.location = "/tmp/client.truststore.jks"
# ssl.truststore.password = "password"
# ssl.key.password = "password"

# Security protocol defaults to PLAINTEXT for local development
security.protocol = "PLAINTEXT"
security.protocol = ${?KAFKA_SECURITY_PROTOCOL}

# SASL Configuration - only used when security.protocol is SASL_SSL
sasl.mechanism = ${?KAFKA_SASL_MECHANISM}
sasl.jaas.config = ${?KAFKA_SASL_JAAS_CONFIG}

# SSL Configuration for Confluent Cloud - only used with SASL_SSL
ssl.endpoint.identification.algorithm = ${?KAFKA_SSL_ENDPOINT_ALGORITHM}
}
brokers {
# If 'True'
Expand Down
4 changes: 2 additions & 2 deletions jikkou/kafka-topics-template.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ metadata:
labels:
app: "{{ app.name }}"
spec:
partitions: 1
replicas: 1
partitions: {{ topic.partitions | default(values.partitions) }}
replicas: {{ system.env.KAFKA_TOPIC_REPLICAS | default(values.replicas) }}
configs:
cleanup.policy: compact
min.insync.replicas: {{ system.env.KAFKA_TOPIC_MIN_INSYNC_REPLICAS | default(values.minInsyncReplicas) }}
Expand Down
7 changes: 7 additions & 0 deletions sample.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ KAFKA_TOPIC_CREATE_GEOJSONCONVERTER=true # Create topics for GeoJSON Converte
KAFKA_TOPIC_CREATE_CONFLICTMONITOR=true # Create topics for Conflict Monitor
KAFKA_TOPIC_CREATE_DEDUPLICATOR=false # Create topics for Deduplicator
KAFKA_TOPIC_CREATE_MECDEPOSIT=false # Create topics for MecDeposit

# Confluent Cloud Support
KAFKA_SECURITY_PROTOCOL=PLAINTEXT
KAFKA_SASL_MECHANISM=PLAIN
KAFKA_SASL_JAAS_CONFIG=
KAFKA_SSL_ENDPOINT_ALGORITHM=

### KAFKA variables - END ###

### MONGODB variables - START ###
Expand Down