prometheus 监控 kafka ¶
部署 zookeeper 与 kafka ¶
1. helm 添加 repo ¶
1. 添加 repo
helm repo add bitnami https://charts.bitnami.com/bitnami
2. 更新 repo
helm repo update
3. 搜索 zookeeper
helm search repo zookeeper
4. 搜索 kafka
helm search repo kafka
2. 部署 zookeeper ¶
1. 创建工作目录
mkdir kafkadir && cd kafkadir
2. helm 拉取 zookeeper 文件
helm pull bitnami/zookeeper && ls
3. 解压 zookeeper 资源
tar -xf zookeeper-13.1.1.tgz && cd zookeeper && ls
1. 配置时区、持久化存储、副本数等 ¶
首次部署时可直接修改 values 中的配置，方便后期更新（upgrade）, 也可以使用 --set 设置
1. 修改 values.yaml 文件
# 关闭认证（默认关闭）
108 auth:
109 client:
110 ## @param auth.client.enabled Enable ZooKeeper client-server authentication. It uses SASL/Digest-MD5
111 ##
112 enabled: false
...
# 修改时区
226 extraEnvVars:
227 - name: TZ
228 value: "Asia/Shanghai"
...
# 修改副本数
248 replicaCount: 3
...
# 配置持久化存储
691 persistence:
692 ## @param persistence.enabled Enable ZooKeeper data persistence using PVC. If false, use emptyDir
693 ##
694 enabled: true
695 ## @param persistence.existingClaim Name of an existing PVC to use (only when deploying a single replica)
696 ##
697 existingClaim: ""
698 ## @param persistence.storageClass PVC Storage Class for ZooKeeper data volume
699 ## If defined, storageClassName: <storageClass>
700 ## If set to "-", storageClassName: "", which disables dynamic provisioning
701 ## If undefined (the default) or set to null, no storageClassName spec is
702 ## set, choosing the default provisioner. (gp2 on AWS, standard on
703 ## GKE, AWS & OpenStack)
704 ##
705 storageClass: "nfs-client"
...
834 serviceMonitor:
835 ## @param metrics.serviceMonitor.enabled Create ServiceMonitor Resource for scraping metrics using Prometheus Operator
836 ##
837 enabled: true # 默认为 false，修改为 true
838 ## @param metrics.serviceMonitor.namespace Namespace for the ServiceMonitor Resource (defaults to the Release Namespace)
839 ##
840 namespace: "kafka" # 默认为空，修改为 kafka
841 ## @param metrics.serviceMonitor.interval Interval at which metrics should be scraped.
842 ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
843 ##
844 interval: "10" # 默认为空，修改为 10
845 ## @param metrics.serviceMonitor.scrapeTimeout Timeout after which the scrape is ended
846 ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
847 ##
848 scrapeTimeout: "20" # 默认为空，修改为 20
创建 zookeeper 资源 ¶
1. 创建 zookeeper 资源
helm install zookeeper ./ -f values.yaml -n kafka --create-namespace
2. 查看 zookeeper 资源
kubectl get all -n kafka
3. 查看 zookeeper 状态
kubectl exec -it zookeeper-0 -n kafka -- bash zkServer.sh status
3. 部署 kafka ¶
1. helm 拉取 kafka 文件
helm pull bitnami/kafka && ls
2. 解压 zookeeper 资源
tar -xf kafka-28.0.4.tgz && cd kafka && ls
1. 配置时区、副本数、持久化存储、zookeeper 连接等。 ¶
首次部署时可直接修改 values 中的配置，方便后期更新（upgrade）, 也可以使用 --set 设置
1. 修改 values.yaml 文件
402 extraEnvVars:
403 - name: TZ
404 value: "Asia/Shanghai"
...
809 persistence:
810 ## @param controller.persistence.enabled Enable Kafka data persistence using PVC, note that ZooKeeper persistence is unaffected
811 ##
812 enabled: true
813 ## @param controller.persistence.existingClaim A manually managed Persistent Volume and Claim
814 ## If defined, PVC must be created manually before volume will be bound
815 ## The value is evaluated as a template
816 ##
817 existingClaim: "nfs-client"
...
2595 zookeeper:
2596 ## @param zookeeper.enabled Switch to enable or disable the ZooKeeper helm chart. Must be false if you use KRaft mode.
2597 ##
2598 enabled: false # 默认为 true ，修改为 false ,不使用内部 zookeeper
...
2636 externalZookeeper:
2637 ## @param externalZookeeper.servers List of external zookeeper servers to use. Typically used in combination with 'zookeeperChrootPath'. Must be empty if you use KRaft mode.
2638 ##
2639 servers: zookeeper # server 名称
2. 高可用配置。 ¶
设置默认分区、默认副本数、日志过期时间，需要根据 kafka 节点数设定。
142 heapOpts: -Xmx1024m -Xms1024m
143 ## @param interBrokerProtocolVersion Override the setting 'inter.broker.protocol.version' during the ZK migration.
144 ## Ref. https://docs.confluent.io/platform/current/installation/migrate-zk-kraft.htm
145
146 deleteTopicEnable: true # 允许删除topic
147 logRetentionHours: 168 # 默认日志保留时间，为一周
148 defaultReplicationFactor: 2 # 自动创建 topic 默认的副本数
149 offsetsTopicReplicationFactor: 2 # 用于配置 offse t记录的 topic 的 partition 的副本个数
150 transactionStateLogReplicationFactor: 2 # 事务主题的复制因子
151 transactionStateLogMinIsr: 2 # 默认为 1 修改为 2
152 numPartitions: 3 # 新建Topic时默认的分区数
...
2307 ## @param provisioning.numPartitions Default number of partitions for topics when unspecified
2308 ##
2309 numPartitions: 3
创建 kafka ¶
1. 创建 kafka 资源
helm install kafka ./ -f values.yaml -n kafka
2. 查看 zookeeper 资源
kubectl get all -n kafka
3. 查看 zookeeper 状态
kubectl exec -it zookeeper-0 -n kafka -- bash zkServer.sh status