Kubernetes对接Ceph RBD关键点记录

容器云技术

Ceph RBD External Provisioner版本

https://github.com/kubernetes-incubator/external-storage/blob/v5.1.0/ceph/rbd/

对Linux操作系统的内核版本有要求，经初步测试需要版本为4.x

CentOS 7.5.1804如何升级操作系统内核？
https://www.howtoforge.com/tutorial/how-to-upgrade-kernel-in-centos-7-server/
https://zhuanlan.zhihu.com/p/29617407

Kubernetes各个Node上需要安装匹配版本的ceph-common

这里以宿主机操作系统为CentOS 7.5.1804，ceph版本为mimic为例

1
2
rpm -Uvh https://download.ceph.com/rpm-mimic/el7/noarch/ceph-release-1-1.el7.noarch.rpm
yum install -y --nogpgcheck ceph-common

参考资料：
https://github.com/kubernetes-incubator/external-storage/blob/v5.1.0/ceph/rbd/Dockerfile

StorageClass和Ceph RBD External Provisioner的正常工作依赖于Kubernetes内部的DNS解析

  1. 配置DNS记录，让Kubernetes集群内部可以把ceph monitor解析到对应的外部地址上；

    1
    2
    3
    4
    5
    6
    7
    8
    kind: Service
    apiVersion: v1
    metadata:
      name: ceph-mon
      namespace: kube-system
    spec:
      type: ExternalName
      externalName: 172.16.170.134.xip.io # ceph monitor的地址

  2. 配置StorageClass，让monitors参数使用1中配置的DNS记录。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: managed-cephrbd-storage
    provisioner: yonghui.cn/cephrbd
    parameters:
      monitors: ceph-mon.kube-system.svc.cluster.local:6789 # 这里需要使用Kubernetes内部的DNS配置ceph monitor的地址
      pool: kube
      adminId: admin
      adminSecretNamespace: kube-system
      adminSecretName: cephrbd-admin-secret
      userId: kube
      userSecretNamespace: kube-system
      userSecretName: cephrbd-secret
      imageFormat: "2"
      imageFeatures: layering

推荐参考资料：
http://blog.51cto.com/ygqygq2/2163656
https://segmentfault.com/q/1010000011440882

Ceph RBD的简单操作记录

1
2
3
4
5
# rbd ls -p kube
kubernetes-dynamic-pvc-4392eab0-fc4c-11e8-a9f7-ee2ca7a031f1
# rbd rm kube/kubernetes-dynamic-pvc-4392eab0-fc4c-11e8-a9f7-ee2ca7a031f1
Removing image: 100% complete...done.
# rbd ls -p kube