Kubernetes对接GlusterFS（把GlusterFS部署在Kubernetes集群外）

容器云技术

一、环境版本信息

1. 版本信息

CentOS 7.6.1810
Kubernetes v1.11.0
GlusterFS 5.3-1.el7 @centos-gluster5
Heketi 8.0.0-1.el7 @centos-gluster5

2. 服务器信息

server01 <-> 172.16.170.128 <-> kubernetes master
server02 <-> 172.16.170.129 <-> kubernetes node
server03 <-> 172.16.170.130 <-> kubernetes node

server07 <-> 172.16.170.134 <-> glusterfs node
server08 <-> 172.16.170.135 <-> glusterfs node
server09 <-> 172.16.170.136 <-> glusterfs node

二、相关注意事项

1. Heketi所在Node到GlusterFS Cluster的所有Node需要配置root用户的SSH免密登录；

2. /etc/heketi/heketi.json 中需要修改配置，不能不做任何修改就拿来直接使用；

（1）安全的做法是需要开启jwt认证，两个key字段分别用来设置admin用户和user用户的访问密码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
...
  "use_auth": true,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "12345678"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "12345678"
    }
  },
...

（2）需要把executor修改为ssh，不能使用默认的mock：

1
2
3
4
5
6
7
8
...
    "executor": "ssh",

    "sshexec": {
      "keyfile": "/root/.ssh/id_rsa",
      "user": "root"
    },
...

（3）json文件中有些配置想使用默认值，需要注释掉或者删除掉，不然无法启动Heketi，因为这些字段的value部分写得是字段的使用说明而非默认值。

3. /lib/systemd/system/heketi.service 中的User值应该为root，不修改的话默认为heketi，无法成功启动。

1
2
3
...
User=root
...

三、完整的实现过程记录

1. Kubernetes集群的所有节点上（既包括Master也包括Node）上都执行：

1
2
3
4
# 安装GlusterFS的yum源
yum install -y centos-release-gluster
# 安装挂载GlusterFS Volume所需要的驱动
yum install -y glusterfs glusterfs-fuse

2. 在Kubernetes集群外部安装GlusterFS集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
# 在server07、server08和server09三台服务器上执行
yum update -y
setenforce 0
systemctl disable firewalld.service
systemctl stop firewalld.service

systemctl disable NetworkManager.service
systemctl stop NetworkManager.service

cat <<EOF >> /etc/hosts

# For GlusterFS Cluster
172.16.170.134 server07
172.16.170.135 server08
172.16.170.136 server09

EOF

yum install -y centos-release-gluster
yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma glusterfs-geo-replication glusterfs-devel
mkdir /opt/glusterd
sed -i 's/var\/lib/opt/g' /etc/glusterfs/glusterd.vol
systemctl enable glusterd.service
systemctl start glusterd.service
systemctl status glusterd.service

# 在server07服务器上执行
yum install -y heketi heketi-client
ssh-keygen
ssh-copy-id root@server07
ssh-copy-id root@server08
ssh-copy-id root@server09

cat <<EOF > /etc/heketi/heketi.json
{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": true,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "12345678"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "12345678"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "sshexec": {
      "keyfile": "/root/.ssh/id_rsa",
      "user": "root"
    },

    "_kubeexec_comment": "Kubernetes configuration",
    "kubeexec": {
      "host" :"https://kubernetes.host:8443",
      "cert" : "/path/to/crt.file",
      "insecure": false,
      "user": "kubernetes username",
      "password": "password for kubernetes user",
      "namespace": "OpenShift project or Kubernetes namespace",
      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
    },

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "loglevel" : "debug"
  }
}
EOF

cat <<EOF > /lib/systemd/system/heketi.service
[Unit]
Description=Heketi Server

[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
User=root
ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable heketi.service
systemctl start heketi.service
systemctl status heketi.service

curl -XGET http://localhost:8080/hello
Hello from Heketi

3. 使用Heketi初始化GlusterFS集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
cat <<EOF > /etc/heketi/topology.json
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "server07"
              ],
              "storage": [
                "172.16.170.134"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "server08"
              ],
              "storage": [
                "172.16.170.135"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "server09"
              ],
              "storage": [
                "172.16.170.136"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        }
      ]
    }
  ]
}
EOF

heketi-cli --user admin --secret 12345678 topology load --json=/etc/heketi/topology.json
Creating cluster ... ID: 4e01ca44d2e6f0077ee3abe9c6783183
	Allowing file volumes on cluster.
	Allowing block volumes on cluster.
	Creating node server07 ... ID: 2e8b83add06cde8713d56ecb7d424033
		Adding device /dev/sdb ... OK
	Creating node server08 ... ID: 56a23b0579e7f79511843e046e69008f
		Adding device /dev/sdb ... OK
	Creating node server09 ... ID: 9fb13189ba78ef302046fe4414f633bf
		Adding device /dev/sdb ... OK

heketi-cli --user admin --secret 12345678 cluster list
Clusters:
Id:4e01ca44d2e6f0077ee3abe9c6783183 [file][block]

heketi-cli --user admin --secret 12345678 --json=true node list
Id:2e8b83add06cde8713d56ecb7d424033	Cluster:4e01ca44d2e6f0077ee3abe9c6783183
Id:56a23b0579e7f79511843e046e69008f	Cluster:4e01ca44d2e6f0077ee3abe9c6783183
Id:9fb13189ba78ef302046fe4414f633bf	Cluster:4e01ca44d2e6f0077ee3abe9c6783183


heketi-cli --user admin --secret 12345678 topology info

Cluster Id: 4e01ca44d2e6f0077ee3abe9c6783183

    File:  true
    Block: true

    Volumes:

    Nodes:

	Node Id: 2e8b83add06cde8713d56ecb7d424033
	State: online
	Cluster Id: 4e01ca44d2e6f0077ee3abe9c6783183
	Zone: 1
	Management Hostnames: server07
	Storage Hostnames: 172.16.170.134
	Devices:
		Id:4995671aaef70cb1f640d0f411e94d2f   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

	Node Id: 56a23b0579e7f79511843e046e69008f
	State: online
	Cluster Id: 4e01ca44d2e6f0077ee3abe9c6783183
	Zone: 1
	Management Hostnames: server08
	Storage Hostnames: 172.16.170.135
	Devices:
		Id:c4a56208419516d4fbc437d9dc3b265e   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

	Node Id: 9fb13189ba78ef302046fe4414f633bf
	State: online
	Cluster Id: 4e01ca44d2e6f0077ee3abe9c6783183
	Zone: 1
	Management Hostnames: server09
	Storage Hostnames: 172.16.170.136
	Devices:
		Id:be2a9558f0634233be72f0c55d051898   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19
			Bricks:

4. 编写验证Heketi和GlusterFS集群对接Kubernetes集群的相关YAML：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
mkdir -p glusterfs/
cd glusterfs/

cat <<EOF > 01-heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
  key: MTIzNDU2Nzg=
type: kubernetes.io/glusterfs
EOF

cat <<EOF > 02-glusterfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://172.16.170.134:8080"
  clusterid: "4e01ca44d2e6f0077ee3abe9c6783183"
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"
  volumetype: "replicate:3"
EOF

cat <<EOF > 03-persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: glusterfs
EOF

cat <<EOF > 04-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - image: alpine:3.8
    command:
    - sleep
    - "3600"
    imagePullPolicy: IfNotPresent
    name: alpine
    volumeMounts:
    - name: myvolume
      mountPath: "/data"
  restartPolicy: Always
  volumes:
  - name: myvolume
    persistentVolumeClaim:
      claimName: myclaim
EOF

5. 验证Heketi和GlusterFS集群对接Kubernetes集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# 创建StorageClass对接到GlusterFS
kubectl create -f 01-heketi-secret.yaml
secret/heketi-secret created

kubectl create -f 02-glusterfs-storageclass.yaml
storageclass.storage.k8s.io/glusterfs created

# 创建PVC使用上面的StorageClass
kubectl create -f 03-persistentvolumeclaim.yaml
persistentvolumeclaim/myclaim created

# 验证PVC动态供应PV
kubectl get pvc -o wide
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myclaim   Bound     pvc-9d5a1b14-2456-11e9-a588-000c2921eba0   1Gi        RWX            glusterfs      40s

kubectl get pv -o wide
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
pvc-9d5a1b14-2456-11e9-a588-000c2921eba0   1Gi        RWX            Delete           Bound     default/myclaim   glusterfs                40s

# 创建测试Pod使用上面的PVC
kubectl create -f 04-pod.yaml
pod/alpine created

kubectl get pod -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP           NODE
alpine    1/1       Running   0          17s       10.211.1.2   server02

# 验证PVC是否挂载到Pod中
kubectl exec -it alpine /bin/sh
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      2.2G     14.8G  13% /
tmpfs                   911.8M         0    911.8M   0% /dev
tmpfs                   911.8M         0    911.8M   0% /sys/fs/cgroup
172.16.170.134:vol_b46e22b816b18672ecb5a8c3289b0193
                       1014.0M     42.7M    971.3M   4% /data
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /dev/termination-log
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hostname
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   911.8M     12.0K    911.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   911.8M         0    911.8M   0% /proc/kcore
tmpfs                   911.8M         0    911.8M   0% /proc/timer_list
tmpfs                   911.8M         0    911.8M   0% /proc/timer_stats
tmpfs                   911.8M         0    911.8M   0% /proc/sched_debug
tmpfs                   911.8M         0    911.8M   0% /sys/firmware

# 创建测试文件，验证存储容量的限制
/data # fallocate -l 971M onebox.disk
/data # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      2.2G     14.8G  13% /
tmpfs                   911.8M         0    911.8M   0% /dev
tmpfs                   911.8M         0    911.8M   0% /sys/fs/cgroup
172.16.170.134:vol_b46e22b816b18672ecb5a8c3289b0193
                       1014.0M   1013.7M    348.0K 100% /data
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /dev/termination-log
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hostname
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   911.8M     12.0K    911.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   911.8M         0    911.8M   0% /proc/kcore
tmpfs                   911.8M         0    911.8M   0% /proc/timer_list
tmpfs                   911.8M         0    911.8M   0% /proc/timer_stats
tmpfs                   911.8M         0    911.8M   0% /proc/sched_debug
tmpfs                   911.8M         0    911.8M   0% /sys/firmware
/data # fallocate -l 971M onebox.disk
fallocate: fallocate 'onebox.disk': No space left on device

# 测试文件删除后，验证容量是否恢复
/data # rm -rf onebox.disk
/data # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  17.0G      2.2G     14.8G  13% /
tmpfs                   911.8M         0    911.8M   0% /dev
tmpfs                   911.8M         0    911.8M   0% /sys/fs/cgroup
172.16.170.134:vol_b46e22b816b18672ecb5a8c3289b0193
                       1014.0M     42.7M    971.3M   4% /data
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /dev/termination-log
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/resolv.conf
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hostname
/dev/mapper/cl-root      17.0G      2.2G     14.8G  13% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   911.8M     12.0K    911.8M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   911.8M         0    911.8M   0% /proc/kcore
tmpfs                   911.8M         0    911.8M   0% /proc/timer_list
tmpfs                   911.8M         0    911.8M   0% /proc/timer_stats
tmpfs                   911.8M         0    911.8M   0% /proc/sched_debug
tmpfs                   911.8M         0    911.8M   0% /sys/firmware