Kubernetes对接GlusterFS（把GlusterFS部署在Kubernetes集群内）

容器云技术

一、环境版本信息

1. 版本信息

CentOS 7.6.1810
Kubernetes v1.11.0
GlusterFS gluster3u13_centos7
Heketi 4

2. 服务器信息

server03 <-> 172.16.170.130 <-> kubernetes master
server07 <-> 172.16.170.134 <-> kubernetes node
server08 <-> 172.16.170.135 <-> kubernetes node
server09 <-> 172.16.170.136 <-> kubernetes node

二、相关注意事项

1. 不要使用链接 https://github.com/gluster/gluster-kubernetes/tree/v1.1 中的部署脚本；

笔者最开始用了，这个过程中还参考了网上很多的帖子，可谓是吃尽了苦头。。。。。。

2. Heketi只有ServiceAccount是不够用的，需要给这个ServiceAccount创建ClusterRoleBinding对象，为其分配比较高的API访问权限；

比如说笔者这里就简单处理，为其分配cluster-admin权限。

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: heketi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: heketi
  namespace: storage

3. 所有宿主机都要执行modprobe dm_thin_pool，否则创建volume会报错；

1
modprobe dm_thin_pool

4. GlusterFS和Heketi的Docker镜像不要像社区代码中那样使用latest和dev这样镜像内容会随时更新的标签，建议选择内容固定的标签对应的镜像。

笔者选择的Docker镜像如下：

1
2
gluster/gluster-centos:gluster3u13_centos7
heketi/heketi:4

三、完整的实现过程记录

1. Kubernetes集群的所有节点上（既包括Master也包括Node）上都执行：

1
2
3
4
5
6
# 安装GlusterFS的yum源
yum install -y centos-release-gluster
# 安装挂载GlusterFS Volume所需要的驱动
yum install -y glusterfs glusterfs-fuse
# 开启Linux内核的 device-mapper target(s) 支持
modprobe dm_thin_pool

2. 参考社区的YAML，编写部署GlusterFS的YAML：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
mkdir -p gluster-kubernetes/deploy/gluster-kubernetes/
cd gluster-kubernetes/deploy/gluster-kubernetes/

cat <<EOF > 01-common-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: storage
EOF

cat <<EOF > 02-glusterfs-daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: glusterfs
  namespace: storage
  labels:
    glusterfs: daemonset
  annotations:
    description: GlusterFS DaemonSet
    tags: glusterfs
spec:
  template:
    metadata:
      name: glusterfs
      labels:
        glusterfs-node: pod
    spec:
      nodeSelector:
        storagenode: glusterfs
      hostNetwork: true
      containers:
      - image: gluster/gluster-centos:gluster3u13_centos7
        imagePullPolicy: IfNotPresent
        name: glusterfs
        volumeMounts:
        - name: glusterfs-heketi
          mountPath: "/var/lib/heketi"
        - name: glusterfs-run
          mountPath: "/run"
        - name: glusterfs-lvm
          mountPath: "/run/lvm"
        - name: glusterfs-etc
          mountPath: "/etc/glusterfs"
        - name: glusterfs-logs
          mountPath: "/var/log/glusterfs"
        - name: glusterfs-config
          mountPath: "/var/lib/glusterd"
        - name: glusterfs-dev
          mountPath: "/dev"
        - name: glusterfs-misc
          mountPath: "/var/lib/misc/glusterfsd"
        - name: glusterfs-cgroup
          mountPath: "/sys/fs/cgroup"
          readOnly: true
        - name: glusterfs-ssl
          mountPath: "/etc/ssl"
          readOnly: true
        securityContext:
          capabilities: {}
          privileged: true
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 60
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - systemctl status glusterd.service
        livenessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 60
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - systemctl status glusterd.service
      volumes:
      - name: glusterfs-heketi
        hostPath:
          path: "/var/lib/heketi"
      - name: glusterfs-run
      - name: glusterfs-lvm
        hostPath:
          path: "/run/lvm"
      - name: glusterfs-etc
        hostPath:
          path: "/etc/glusterfs"
      - name: glusterfs-logs
        hostPath:
          path: "/var/log/glusterfs"
      - name: glusterfs-config
        hostPath:
          path: "/var/lib/glusterd"
      - name: glusterfs-dev
        hostPath:
          path: "/dev"
      - name: glusterfs-misc
        hostPath:
          path: "/var/lib/misc/glusterfsd"
      - name: glusterfs-cgroup
        hostPath:
          path: "/sys/fs/cgroup"
      - name: glusterfs-ssl
        hostPath:
          path: "/etc/ssl"
EOF

cat <<EOF > 03-heketi-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heketi
  namespace: storage
EOF

cat <<EOF > 04-heketi-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: heketi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: heketi
  namespace: storage
EOF

cat <<EOF > 05-heketi-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heketi
  namespace: storage
  labels:
    glusterfs: heketi-deployment
    deploy-heketi: heketi-deployment
  annotations:
    description: Defines how to deploy Heketi
spec:
  replicas: 1
  template:
    metadata:
      name: heketi
      labels:
        name: deploy-heketi
        glusterfs: heketi-pod
    spec:
      serviceAccountName: heketi
      containers:
      - image: heketi/heketi:4
        imagePullPolicy: IfNotPresent
        name: heketi
        env:
        - name: HEKETI_EXECUTOR
          value: kubernetes
        - name: HEKETI_FSTAB
          value: "/var/lib/heketi/fstab"
        - name: HEKETI_SNAPSHOT_LIMIT
          value: '14'
        - name: HEKETI_KUBE_GLUSTER_DAEMONSET
          value: "y"
        - name: HEKETI_CLI_SERVER
          value: "http://127.0.0.1:8080"
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: db
          mountPath: "/var/lib/heketi"
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 3
          httpGet:
            path: "/hello"
            port: 8080
        livenessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 30
          httpGet:
            path: "/hello"
            port: 8080
      volumes:
      - name: db
EOF

cat <<EOF > 06-heketi-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: heketi
  namespace: storage
  labels:
    glusterfs: heketi-service
    deploy-heketi: support
  annotations:
    description: Exposes Heketi Service
spec:
  selector:
    name: deploy-heketi
  ports:
  - name: deploy-heketi
    port: 8080
    targetPort: 8080
EOF

3. 在Kubernetes集群上创建Heketi和GlusterFS集群：

1
2
3
4
5
6
7
8
9
cd ../../
kubectl create -f deploy/gluster-kubernetes/

kubectl get pod -n storage -o wide
NAME                      READY     STATUS    RESTARTS   AGE       IP               NODE
glusterfs-6shxs           1/1       Running   0          1m        172.16.170.135   server08
glusterfs-9vpjg           1/1       Running   0          1m        172.16.170.136   server09
glusterfs-pmq2n           1/1       Running   0          1m        172.16.170.134   server07
heketi-5bdc8f9db5-nmjrw   1/1       Running   0          1m        10.211.1.12      server07

4. 使用Heketi初始化GlusterFS集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
cat <<EOF > topology.json
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "server07"
              ],
              "storage": [
                "172.16.170.134"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "server08"
              ],
              "storage": [
                "172.16.170.135"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "server09"
              ],
              "storage": [
                "172.16.170.136"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        }
      ]
    }
  ]
}
EOF

kubectl cp topology.json heketi-5bdc8f9db5-nmjrw:/ -n storage

kubectl exec -it heketi-5bdc8f9db5-nmjrw /bin/bash -n storage
[root@heketi-5bdc8f9db5-nmjrw /]# heketi-cli topology load --json=topology.json
Creating cluster ... ID: dde6f33a6cb852e04cb0685e210d3959
	Creating node server07 ... ID: 2818eb53b31234c2cc852b339538d856
		Adding device /dev/sdb ... OK
	Creating node server08 ... ID: d5773e74abd6f3041d07b095a2f4ae99
		Adding device /dev/sdb ... OK
	Creating node server09 ... ID: 4caafd29101b77da4a6cd265dad65140
		Adding device /dev/sdb ... OK

heketi-cli topology info

Cluster Id: dde6f33a6cb852e04cb0685e210d3959

    Volumes:

    Nodes:

	Node Id: 2818eb53b31234c2cc852b339538d856
	State: online
	Cluster Id: dde6f33a6cb852e04cb0685e210d3959
	Zone: 1
	Management Hostname: server07
	Storage Hostname: 172.16.170.134
	Devices:
		Id:25bbec25514b711d48d0abef98809310   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

	Node Id: 4caafd29101b77da4a6cd265dad65140
	State: online
	Cluster Id: dde6f33a6cb852e04cb0685e210d3959
	Zone: 1
	Management Hostname: server09
	Storage Hostname: 172.16.170.136
	Devices:
		Id:78385691f170d5ae6aae78448ab7d710   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

	Node Id: d5773e74abd6f3041d07b095a2f4ae99
	State: online
	Cluster Id: dde6f33a6cb852e04cb0685e210d3959
	Zone: 1
	Management Hostname: server08
	Storage Hostname: 172.16.170.135
	Devices:
		Id:2678d65a18f0c21b38288d218d209a3f   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

5. 编写验证Heketi和GlusterFS集群对接Kubernetes集群的相关YAML：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
mkdir -p example/
cd example/

kubectl get service -n storage
NAME      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
heketi    ClusterIP   10.96.162.3   <none>        8080/TCP   1h

cat <<EOF > 01-glusterfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.96.162.3:8080"
  restauthenabled: "false"
  volumetype: "replicate:3"
EOF

cat <<EOF > 02-persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: glusterfs
EOF

cat <<EOF > 03-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - image: alpine:3.8
    command:
    - sleep
    - "3600"
    imagePullPolicy: IfNotPresent
    name: alpine
    volumeMounts:
    - name: myvolume
      mountPath: "/data"
  restartPolicy: Always
  volumes:
  - name: myvolume
    persistentVolumeClaim:
      claimName: myclaim
EOF

6. 验证Heketi和GlusterFS集群对接Kubernetes集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
# 创建StorageClass对接到GlusterFS
kubectl create -f 01-glusterfs-storageclass.yaml
storageclass.storage.k8s.io/glusterfs created

# 创建PVC使用上面的StorageClass
kubectl create -f 02-persistentvolumeclaim.yaml
persistentvolumeclaim/myclaim created

# 验证PVC动态供应PV
kubectl get pvc -o wide
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myclaim   Bound     pvc-f910f4b2-239d-11e9-b3b6-000c295a4c4c   1Gi        RWX            glusterfs      1m

kubectl get pv -o wide
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
pvc-f910f4b2-239d-11e9-b3b6-000c295a4c4c   1Gi        RWX            Delete           Bound     default/myclaim   glusterfs                1m

# 创建测试Pod使用上面的PVC
kubectl create -f 03-pod.yaml
pod/alpine created

kubectl get pod -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP            NODE
alpine    1/1       Running   0          5s        10.211.2.10   server08

# 验证PVC是否挂载到Pod中
kubectl exec -it alpine /bin/sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      4.3G     12.7G  25% /
tmpfs                   909.8M         0    909.8M   0% /dev
tmpfs                   909.8M         0    909.8M   0% /sys/fs/cgroup
172.16.170.134:vol_7f5ea2a6035e2f7885687db4ef46d2e3
                       1015.3M     42.8M    982.6M   4% /data
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /dev/termination-log
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hostname
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   909.8M     12.0K    909.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   909.8M         0    909.8M   0% /proc/kcore
tmpfs                   909.8M         0    909.8M   0% /proc/timer_list
tmpfs                   909.8M         0    909.8M   0% /proc/timer_stats
tmpfs                   909.8M         0    909.8M   0% /proc/sched_debug
tmpfs                   909.8M         0    909.8M   0% /sys/firmware

# 创建测试文件，验证存储容量的限制
/data # fallocate -l 982M onebox.disk
/data # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      4.3G     12.7G  25% /
tmpfs                   909.8M         0    909.8M   0% /dev
tmpfs                   909.8M         0    909.8M   0% /sys/fs/cgroup
172.16.170.134:vol_7f5ea2a6035e2f7885687db4ef46d2e3
                       1015.3M      1.0G    572.0K 100% /data
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /dev/termination-log
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hostname
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   909.8M     12.0K    909.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   909.8M         0    909.8M   0% /proc/kcore
tmpfs                   909.8M         0    909.8M   0% /proc/timer_list
tmpfs                   909.8M         0    909.8M   0% /proc/timer_stats
tmpfs                   909.8M         0    909.8M   0% /proc/sched_debug
tmpfs                   909.8M         0    909.8M   0% /sys/firmware
/data # fallocate -l 982M onebox.disk
fallocate: fallocate 'onebox.disk': No space left on device

# 测试文件删除后，验证容量是否恢复
/data # rm -rf onebox.disk
/data # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      4.3G     12.7G  25% /
tmpfs                   909.8M         0    909.8M   0% /dev
tmpfs                   909.8M         0    909.8M   0% /sys/fs/cgroup
172.16.170.134:vol_7f5ea2a6035e2f7885687db4ef46d2e3
                       1015.3M     42.8M    982.6M   4% /data
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /dev/termination-log
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hostname
/dev/mapper/cl-root      17.0G      4.3G     12.7G  25% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   909.8M     12.0K    909.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   909.8M         0    909.8M   0% /proc/kcore
tmpfs                   909.8M         0    909.8M   0% /proc/timer_list
tmpfs                   909.8M         0    909.8M   0% /proc/timer_stats
tmpfs                   909.8M         0    909.8M   0% /proc/sched_debug
tmpfs                   909.8M         0    909.8M   0% /sys/firmware

四、参考资料

https://github.com/gluster/gluster-kubernetes/tree/v1.1
https://hub.docker.com/r/gluster/gluster-centos/tags
https://blog.csdn.net/q1403539144/article/details/86566796
http://www.cnitblog.com/xijia0524/archive/2014/04/19/89466.html