helm-Chart包开发 ¶
Chart 目录结构 ¶
| 文件 | 说明 |
|---|---|
| Chart.yaml | 用于描述Chart的基本信息; helm show chart stable/mysql命令查看的内容就是此文件内容 |
| values.yaml | Chart的默认配置文件; helm show values stable/mysql命令查看的内容就是此文件内容 |
| README.md | [可选] 当前Chart的介绍 |
| LICENS | [可选] 协议 |
| requirements.yaml | [可选] 用于存放当前Chart依赖的其它Chart的说明文件 |
| charts/ | [可选]: 该目录中放置当前Chart依赖的其它Chart |
| templates/ | [可选]: 部署文件模版目录 |
创建不可配置的 chart ¶
1. 创建目录与 chart.yaml ¶
1. 创建项目目录
[root@k8smaster002 /]# mkdir -p /helm/nginx/templates
[root@k8smaster002 /]# cd /helm/nginx
2. 编写 Chart.yaml
echo 'name: helm-nginx
version: 1.0.0
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Kubernetes' > 1.txt
2. 创建 deployment.yaml ¶
1. 编写 deployment.yaml
cat > templates/deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: helm-nginx
spec:
replicas: 1
selector:
matchLabels:
app: helm-nginx
template:
metadata:
labels:
app: helm-nginx
spec:
containers:
- name: c1
image: nginx:1.15-alpine
imagePullPolicy: IfNotPresent
EOF
3. 创建 service.yaml ¶
1. 编写 service.yaml
cat > templates/service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: helm-nginx
labels:
app: helm-nginx
spec:
selector:
app: helm-nginx
ports:
- port: 80
targetPort: 80
protocol: TCP
EOF
4. 使用 chart 安装应用 ¶
1. 使用 --generate-name 选项时，Helm 会自动生成一个唯一的名称，通常是在 chart 名称的基础上添加一个随机的后缀。
[root@k8smaster002 nginx]# helm install /helm/nginx/ --generate-name
NAME: nginx-1712747429
LAST DEPLOYED: Wed Apr 10 19:10:29 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
5. 查看与验证 ¶
1. 查看相关信息
...
[root@k8smaster002 nginx]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nginx-1712747429 default 1 2024-04-10 19:10:29.510237252 +0800 CST deployed helm-nginx-1.0.0 1.0
...
...
[root@k8smaster002 nginx]# kubectl get pods,service -l app=helm-nginx
NAME READY STATUS RESTARTS AGE
pod/helm-nginx-7f584fb795-kvcb7 1/1 Running 0 14s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helm-nginx ClusterIP 10.96.207.37 <none> 80/TCP 14s
...
...
[root@k8smaster002 nginx]# curl -I 10.96.207.37
HTTP/1.1 200 OK
Server: nginx/1.15.12
Date: Wed, 10 Apr 2024 11:17:30 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Sat, 11 May 2019 00:35:53 GMT
Connection: keep-alive
ETag: "5cd618e9-264"
Accept-Ranges: bytes
...
6. 删除 release ¶
1. 使用 helm 并删除 release
...
[root@k8smaster002 nginx]# helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nginx-1712747793 default 1 2024-04-10 19:16:33.203275026 +0800 CST deployed helm-nginx-1.0.0 1.0
...
...
[root@k8smaster002 nginx]# helm uninstall nginx-1712747793
release "nginx-1712747793" uninstalled
...
创建可配置的 Chart ¶
官方的预定义变量
- Release.Name：发布的名称（不是chart）
- Release.Time：chart发布上次更新的时间。这将匹配Last ReleasedRelease对象上的时间。
- Release.Namespace：chart发布到的名称空间。
- Release.Service：进行发布的服务。
- Release.IsUpgrade：如果当前操作是升级或回滚，则设置为true。
- Release.IsInstall：如果当前操作是安装，则设置为true。
- Release.Revision：修订号。它从1开始，每个都递增helm upgrade。
- Chart：内容Chart.yaml。因此，chart版本可以Chart.Version和维护者一样获得 Chart.Maintainers。
- Files：类似于chart的对象，包含chart中的所有非特殊文件。这不会授予您访问模板的权限，但可以访问存在的其他文件（除非使用它们除外.helmignore）。可以使用{{index .Files "file.name"}}或使用{{.Files.Get name}}或 {{.Files.GetStringname}}函数访问文件。您也可以访问该文件的内容，[]byte使用{{.Files.GetBytes}}
- Capabilities：类似于地图的对象，包含有关Kubernetes（{{.Capabilities.KubeVersion}}，Tiller（{{.Capabilities.TillerVersion}}和支持的Kubernetes API）版本（{{.Capabilities.APIVersions.Has "batch/v1"）的版本的信息
1. 新增 values.yaml 文件 ¶
1. 新增 values.yaml 文件
cat > /helm/nginx/values.yaml <<EOF
image:
repository: nginx
tag: '1.15-alpine'
replicas: 2
EOF
2. 配置 deploy 引用 values 的值 ¶
2. 修改 deployment.yaml 文件
cat > /helm/nginx/templates/deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: helm-nginx
spec:
replicas: {{ .Values.replicas }}
selector:
matchLabels:
app: helm-nginx
template:
metadata:
labels:
app: helm-nginx
spec:
containers:
- name: helm-nginx
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: IfNotPresent
EOF
3. 查看文件的不同
[root@k8smaster002 nginx]# diff templates/deployment-bak.yaml templates/deployment.yaml
6c6
< replicas: 1
---
> replicas: {{ .Values.replicas }}
16,17c16,17
< - name: c1
< image: nginx:1.15-alpine
---
> - name: helm-nginx
> image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
18a19
3. 测试 ¶
1. 直接使用应用测试 ¶
1. deployment.yaml 将直接使用 values.yaml 中的配置
[root@k8smaster002 nginx]# helm install helm-nginx-new /helm/nginx
NAME: helm-nginx-new
LAST DEPLOYED: Wed Apr 10 19:42:45 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
2. 查看相关配置
[root@k8smaster002 nginx]# kubectl get pods,service -l app=helm-nginx
NAME READY STATUS RESTARTS AGE
pod/helm-nginx-5cc778b4f6-9vdkn 1/1 Running 0 104s
pod/helm-nginx-5cc778b4f6-w6ggk 1/1 Running 0 104s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helm-nginx ClusterIP 10.111.125.161 <none> 80/TCP 104s
2. 通过命令行设置变量后干运行测试 ¶
1. 通过在命令行设置变量为 deployment.yaml 赋值，使用 --set 选项，使用 --dry-run 选项来打印出生成的清单文件内容，而不执行部署。
[root@k8smaster002 nginx]# helm install helm-nginx-new --set replicas=3 /helm/nginx/ --dry-run
NAME: helm-nginx-new
LAST DEPLOYED: Wed Apr 10 19:46:01 2024
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
MANIFEST:
---
# Source: helm-nginx/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: helm-nginx
labels:
app: helm-nginx
spec:
selector:
app: helm-nginx
ports:
- port: 80
targetPort: 80
protocol: TCP
---
# Source: helm-nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: helm-nginx
spec:
replicas: 3
selector:
matchLabels:
app: helm-nginx
template:
metadata:
labels:
app: helm-nginx
spec:
containers:
- name: helm-nginx
image: nginx:1.15-alpine
imagePullPolicy: IfNotPresent
2. 使用 upgrade 更新
[root@k8smaster002 nginx]# helm upgrade helm-nginx-new --set replicas=3 /helm/nginx/
Release "helm-nginx-new" has been upgraded. Happy Helming!
NAME: helm-nginx-new
LAST DEPLOYED: Wed Apr 10 19:49:34 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
3. 查看 -l app=helm-nginx 相关资源
...
[root@k8smaster002 nginx]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helm-nginx-new default 2 2024-04-10 19:49:34.27588687 +0800 CST deployed helm-nginx-1.0.0 1.0
my-mysql default 3 2024-04-10 18:32:38.844401614 +0800 CST deployed mysql-1.6.9 5.7.30
...
...
[root@k8smaster002 nginx]# kubectl get pods,service -l app=helm-nginx
NAME READY STATUS RESTARTS AGE
pod/helm-nginx-5cc778b4f6-9vdkn 1/1 Running 0 7m36s
pod/helm-nginx-5cc778b4f6-j56kc 1/1 Running 0 47s
pod/helm-nginx-5cc778b4f6-w6ggk 1/1 Running 0 7m36s
...
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/helm-nginx ClusterIP 10.111.125.161 <none> 80/TCP 7m36s
...
4. 将 Chart 包进行打包 ¶
将 chart 打包成一个压缩文件，便于存储与分享。
1. 使用 package 打包 chart 项目
[root@k8smaster002 helm]# helm package /helm/nginx/
Successfully packaged chart and saved it to: /helm/helm-nginx-1.0.0.tgz
[root@k8smaster002 helm]# ls
helm-nginx-1.0.0.tgz nginx
5. 使用 Chart 安装 ¶
1. 使用 /helm/helm-nginx-1.0.0.tgz 来进行安装
...
[root@k8smaster002 helm]# helm install helm-nginx /helm/helm-nginx-1.0.0.tgz
NAME: helm-nginx1
LAST DEPLOYED: Wed Apr 10 21:13:56 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
...
2. 查看 helm 相关信息
...
[root@k8smaster002 helm]# helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helm-nginx default 1 2024-04-10 21:13:56.724476081 +0800 CST deployed helm-nginx-1.0.0 1.0
...
...
[root@k8smaster002 helm]# kubectl get pod -l app
NAME READY STATUS RESTARTS AGE
helm-nginx-5cc778b4f6-jzbjj 1/1 Running 0 17s
helm-nginx-5cc778b4f6-wf8g8 1/1 Running 0 17s
...
...
[root@k8smaster002 helm]# kubectl get pod -l app=helm-nginx
NAME READY STATUS RESTARTS AGE
helm-nginx-5cc778b4f6-jzbjj 1/1 Running 0 27s
helm-nginx-5cc778b4f6-wf8g8 1/1 Running 0 27s
...
Chart 包托管至 Harbor 方案 ¶
1. 集群外 harbor 服务器准备 ¶
harbor 服务器安装 ¶
❯ mkdir /harbor && cd /harbor
> wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
❯ tar -xf harbor-offline-installer-v2.5.3.tgz && cd harbor
❯ mv harbor.yml.tmpl harbor.yml
❯ grep -v "#" harbor.yml |grep -v "^$"
hostname: harbor.huichengcheng.com
http:
port: 180
https:
port: 60000
certificate: /harbor/harbor/harbor.huichengcheng.com_bundle.pem
private_key: /harbor/harbor/harbor.huichengcheng.com.key
harbor_admin_password: Harbor12345
database:
password: root123
max_idle_conns: 100
max_open_conns: 900
data_volume: /data
trivy:
ignore_unfixed: false
skip_update: false
offline_scan: false
insecure: false
jobservice:
max_job_workers: 10
notification:
webhook_job_max_retry: 10
chart:
absolute_url: disabled
log:
level: info
local:
rotate_count: 50
rotate_size: 200M
location: /var/log/harbor
_version: 2.5.0
proxy:
http_proxy:
https_proxy:
no_proxy:
components:
- core
- jobservice
- trivy
upload_purging:
enabled: true
age: 168h
interval: 24h
dryrun: false
❯ docker load -i harbor.v2.5.3.tar.gz
❯ ./install.sh --with-chartmuseum
[Step 0]: checking if docker is installed ...
Note: docker version: 25.0.3
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.29.2
[Step 2]: loading Harbor images ...
Loaded image: goharbor/harbor-portal:v2.5.3
Loaded image: goharbor/harbor-core:v2.5.3
Loaded image: goharbor/redis-photon:v2.5.3
Loaded image: goharbor/prepare:v2.5.3
Loaded image: goharbor/harbor-db:v2.5.3
Loaded image: goharbor/chartmuseum-photon:v2.5.3
Loaded image: goharbor/harbor-jobservice:v2.5.3
Loaded image: goharbor/harbor-registryctl:v2.5.3
Loaded image: goharbor/nginx-photon:v2.5.3
Loaded image: goharbor/notary-signer-photon:v2.5.3
Loaded image: goharbor/harbor-log:v2.5.3
Loaded image: goharbor/harbor-exporter:v2.5.3
Loaded image: goharbor/registry-photon:v2.5.3
Loaded image: goharbor/notary-server-photon:v2.5.3
Loaded image: goharbor/trivy-adapter-photon:v2.5.3
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /harbor/harbor
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/chartserver/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/portal/nginx.conf
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /config/chartserver/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Note: stopping existing Harbor instance ...
Stopping harbor-jobservice ... done
Stopping harbor-core ... done
Stopping registryctl ... done
Stopping harbor-db ... done
Stopping redis ... done
Stopping registry ... done
Stopping harbor-portal ... done
Stopping chartmuseum ... done
Stopping harbor-log ... done
Removing harbor-jobservice ... done
Removing nginx ... done
Removing harbor-core ... done
Removing registryctl ... done
Removing harbor-db ... done
Removing redis ... done
Removing registry ... done
Removing harbor-portal ... done
Removing chartmuseum ... done
Removing harbor-log ... done
Removing network harbor_harbor
Removing network harbor_harbor-chartmuseum
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating network "harbor_harbor-chartmuseum" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating chartmuseum ... done
Creating harbor-db ... done
Creating registry ... done
Creating redis ... done
Creating registryctl ... done
Creating harbor-core ... done
Creating nginx ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----
[root@k8smaster002 helm]# helm repo add harborhelm https://harbor.huichengcheng.com:16000/chartrepo/nginx --username admin --password Harbor12345
"harborhelm" has been added to your repositories
[root@k8smaster002 helm]# helm repo list
NAME URL
prometheus-community https://prometheus-community.github.io/helm-charts
bitnami https://charts.bitnami.com/bitnami
stable http://mirror.azure.cn/kubernetes/charts/
harborhelm https://harbor.huichengcheng.com:16000/chartrepo/nginx
[root@k8smaster002 helm]# helm search repo helm-nginx
NAME CHART VERSION APP VERSION DESCRIPTION
harborhelm/helm-nginx 1.0.0 1.0 A Helm chart for Kubernetes
[root@k8smaster002 helm]# helm install helm-nginx-test harborhelm/helm-nginx
NAME: helm-nginx-test
LAST DEPLOYED: Wed Apr 10 22:04:25 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
[root@k8smaster002 helm]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helm-nginx-test default 1 2024-04-10 22:04:25.563924361 +0800 CST deployed helm-nginx-1.0.0 1.0
my-mysql default 3 2024-04-10 18:32:38.844401614 +0800 CST deployed mysql-1.6.9 5.7.30
[root@k8smaster002 helm]# kubectl get pods -l app=helm-nginx
NAME READY STATUS RESTARTS AGE
helm-nginx-5cc778b4f6-l5dx7 1/1 Running 0 56s
helm-nginx-5cc778b4f6-rftn4 1/1 Running 0 56s
安装helmpush插件 ¶
需要安装helmpush插件才能上传
- 在线直接安装
[root@k8smaster002 helm]# helm plugin install https://github.com/chartmuseum/helm-push
Downloading and installing helm-push v0.10.4 ...
https://github.com/chartmuseum/helm-push/releases/download/v0.10.4/helm-push_0.10.4_linux_amd64.tar.gz
Installed plugin: cm-push
[root@k8smaster002 helm]# ls /root/.local/share/helm/plugins/helm-push/bin/
helm-cm-push
将打包应用 push 到 harbor ¶
[root@k8smaster002 helm]# ls nginx/
Chart.yaml templates values.yaml
[root@k8smaster002 helm]# sed -i "s/1/2/g" nginx/Chart.yaml
[root@k8smaster002 helm]# helm package nginx/
Successfully packaged chart and saved it to: /helm/helm-nginx-2.0.0.tgz
[root@k8smaster002 helm]# ls
helm-nginx-2.0.0.tgz nginx
[root@k8smaster002 helm]# helm -h
The Kubernetes package manager
...
Available Commands:
cm-push Please see https://github.com/chartmuseum/helm-push for usage
[root@k8smaster002 helm]# helm cm-push --username admin --password Harbor12345 helm-nginx-2.0.0.tgz harborhelm
Pushing helm-nginx-2.0.0.tgz to harborhelm...
Done.
Helm Chart包可视化管理 Kubeapps应用商店 ¶
[root@k8smaster002 helm]# helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" already exists with the same configuration, skipping
[root@k8smaster002 helm]# helm search repo kubeapps
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/kubeapps 15.0.2 2.10.0 Kubeapps is a web-based UI for launching and ma...
[root@k8smaster002 helm]# helm install kubeapps bitnami/kubeapps --namespace kubeapps
NAME: kubeapps
LAST DEPLOYED: Wed Apr 10 22:37:23 2024
NAMESPACE: kubeapps
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kubeapps
CHART VERSION: 15.0.2
APP VERSION: 2.10.0** Please be patient while the chart is being deployed **
Tip:
Watch the deployment status using the command: kubectl get pods -w --namespace kubeapps
Kubeapps can be accessed via port 80 on the following DNS name from within your cluster:
kubeapps.kubeapps.svc.cluster.local
To access Kubeapps from outside your K8s cluster, follow the steps below:
1. Get the Kubeapps URL by running these commands:
echo "Kubeapps URL: http://127.0.0.1:8080"
kubectl port-forward --namespace kubeapps service/kubeapps 8080:80
2. Open a browser and access Kubeapps using the obtained URL.
WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
- apprepository.resources
- dashboard.resources
- frontend.resources
- kubeappsapis.resources
- postgresql.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
访问 kubeapps ¶