Installed Strimzi Kafka Operator version 0.45.0

Three Kafka nodes + Three ZooKeeper nodes
Can tolerate 1 node failure
3 replicas distributed across different nodes
Use Azure Disk SSD SCRAM-SHA-512 + ACLs

cluster/manifests/freeleaps-data-platform/kafka/README.md

@@ -0,0 +1,75 @@
+# Kafka Configuration for Freeleaps Data Platform
+
+This directory contains the Kafka configuration for the Freeleaps Data Platform.
+
+## Files
+
+- `values.yaml` - Original Bitnami Kafka Helm values (legacy)
+- `vpa.yaml` - Vertical Pod Autoscaler configuration
+- `strimzi-kafka-cluster.yaml` - Strimzi Kafka Operator cluster configuration
+- `README.md` - This file
+
+## Migration from Bitnami to Strimzi
+
+### Current State
+- **Existing**: Bitnami Kafka deployment (single node)
+- **Target**: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes)
+
+### Migration Steps
+
+1. **Install Strimzi Operator** (already done)
+   ```bash
+   helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
+     --namespace freeleaps-data-platform
+   ```
+
+2. **Delete existing Kafka deployment**
+   ```bash
+   kubectl delete statefulset kafka-controller -n freeleaps-data-platform
+   kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform
+   kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform
+   kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform
+   ```
+
+3. **Deploy new Strimzi Kafka cluster**
+   ```bash
+   kubectl apply -f strimzi-kafka-cluster.yaml
+   ```
+
+### New Cluster Features
+
+- **High Availability**: 3 Kafka brokers + 3 Zookeeper nodes
+- **Authentication**: SCRAM-SHA-512
+- **Storage**: 20Gi per broker, 10Gi per ZK node
+- **Replication**: 3x replication factor
+- **Security**: TLS support for internal communication
+
+### Monitoring
+
+Check cluster status:
+```bash
+kubectl get kafka -n freeleaps-data-platform
+kubectl get kafkatopic -n freeleaps-data-platform
+kubectl get kafkauser -n freeleaps-data-platform
+```
+
+### Client Configuration
+
+For applications connecting to the new Kafka cluster, use:
+- **Bootstrap servers**: `kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092`
+- **Authentication**: SCRAM-SHA-512
+- **Username**: `freeleaps-user`
+- **Password**: Generated by Strimzi (check the secret)
+
+## Storage Requirements
+
+- **Kafka brokers**: 3 × 20Gi = 60Gi
+- **Zookeeper**: 3 × 10Gi = 30Gi
+- **Total**: 90Gi
+
+## Network Policy
+
+The cluster includes network policies that allow:
+- Internal cluster communication
+- External client access (with proper authentication)
+- Metrics collection (if enabled)

cluster/manifests/freeleaps-data-platform/kafka/storage-class.yaml

@@ -0,0 +1,14 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: kafka-storage-class
+  annotations:
+    storageclass.kubernetes.io/is-default-class: "false"
+provisioner: kubernetes.io/azure-disk
+parameters:
+  skuName: StandardSSD_LRS
+  cachingMode: ReadWrite
+  kind: Managed
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: true

cluster/manifests/freeleaps-data-platform/kafka/strimzi-kafka-cluster.yaml

@@ -0,0 +1,90 @@
+apiVersion: kafka.strimzi.io/v1beta2
+kind: Kafka
+metadata:
+  name: kafka-cluster
+  namespace: freeleaps-data-platform
+  labels:
+    app: kafka
+    component: messaging
+spec:
+  kafka:
+    version: 3.8.0
+    replicas: 3
+    listeners:
+      - name: plain
+        port: 9092
+        type: internal
+        tls: false
+        authentication:
+          type: scram-sha-512
+      - name: tls
+        port: 9093
+        type: internal
+        tls: true
+        authentication:
+          type: tls
+    config:
+      offsets.topic.replication.factor: 3
+      transaction.state.log.replication.factor: 3
+      transaction.state.log.min.isr: 2
+      default.replication.factor: 3
+      min.insync.replicas: 2
+      inter.broker.protocol.version: "3.8"
+      log.retention.hours: 168  # 7 days
+      log.segment.bytes: 1073741824  # 1GB
+      num.partitions: 8
+      log.retention.check.interval.ms: 300000
+    storage:
+      type: persistent-claim
+      size: 20Gi
+      deleteClaim: false
+  zookeeper:
+    replicas: 3
+    storage:
+      type: persistent-claim
+      size: 10Gi
+      deleteClaim: false
+  entityOperator:
+    topicOperator: {}
+    userOperator: {}
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaTopic
+metadata:
+  name: system-events
+  namespace: freeleaps-data-platform
+  labels:
+    strimzi.io/cluster: kafka-cluster
+spec:
+  partitions: 3
+  replicas: 3
+  config:
+    retention.ms: 604800000  # 7 days
+    segment.bytes: 1073741824  # 1GB
+    cleanup.policy: delete
+---
+apiVersion: kafka.strimzi.io/v1beta2
+kind: KafkaUser
+metadata:
+  name: freeleaps-user
+  namespace: freeleaps-data-platform
+  labels:
+    strimzi.io/cluster: kafka-cluster
+spec:
+  authentication:
+    type: scram-sha-512
+  authorization:
+    type: simple
+    acls:
+      - resource:
+          type: topic
+          name: "*"
+          patternType: literal
+        operation: All
+        host: "*"
+      - resource:
+          type: group
+          name: "*"
+          patternType: literal
+        operation: All
+        host: "*"