2020-03-04

Kubernetes Harbor 的安装、配置和使用

一、实验环境版本信息

1. 操作系统的版本信息

CentOS Linux release 7.6.1810 (Core)

2. 各组件的版本信息

kubernetes cluster v1.17.0，推荐使用kubeadm v1.17.0 进行试验

etcd v3.4.3
kube-apiserver v1.17.0
kube-controller-manager v1.17.0
kube-scheduler v1.17.0
kubectl v1.17.0
helm v3.1.0

docker 18.09.9
kubelet v1.17.0
calico v3.11.1

kubernetes harbor，使用容器化的方式部署

kubernetes harbor v1.10.1

二、在 Node 节点上准备Docker镜像

docker pull goharbor/clair-adapter-photon:v1.0.1-v1.10.1
docker pull goharbor/clair-photon:v2.1.1-v1.10.1
docker pull goharbor/notary-server-photon:v0.6.1-v1.10.1
docker pull goharbor/harbor-core:v1.10.1
docker pull goharbor/harbor-portal:v1.10.1
docker pull goharbor/chartmuseum-photon:v0.9.0-v1.10.1
docker pull goharbor/redis-photon:v1.10.1
docker pull goharbor/notary-signer-photon:v0.6.1-v1.10.1
docker pull goharbor/harbor-registryctl:v1.10.1
docker pull goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.1
docker pull goharbor/harbor-jobservice:v1.10.1
docker pull goharbor/harbor-db:v1.10.1

三、安装和配置 Kubernetes Harbor

1. 安装 Helm 3 包管理工具

# curl -o helm-v3.1.0-linux-amd64.tar.gz https://get.helm.sh/helm-v3.1.0-linux-amd64.tar.gz
# tar -zxvf helm-v3.1.0-linux-amd64.tar.gz
# cd linux-amd64/
# cp helm /usr/local/bin/

# helm version
version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}

2. 在 Kubernetes 集群中加入动态存储供应的支持，并配置相关的 StorageClass 对象

## 本文以安装了NFS Client Provisioner为例，下面是我环境中的相关资源对象展示，请依据实际环境进行配置。我环境中的 StorageClass 对象叫 managed-nfs-storage 
# kubectl get pod -n storage -o wide
NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
nfs-client-provisioner-6c45d64447-hrjhq   1/1     Running   2          21h   10.211.196.155   node01   <none>           <none>

# kubectl get storageclass managed-nfs-storage -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2020-03-03T12:10:27Z"
  name: managed-nfs-storage
  resourceVersion: "27152"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/managed-nfs-storage
  uid: 05c23ff9-5f9a-4a6b-89dd-ed99013cc344
mountOptions:
- vers=4
parameters:
  archiveOnDelete: "false"
provisioner: fuseim.pri/ifs
reclaimPolicy: Delete
volumeBindingMode: Immediate

# kubectl get storageclass -o wide
NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs   Delete          Immediate           false                  21h

3. 使用 Helm 3 安装 Kubernetes Harbor

# git clone https://github.com/goharbor/harbor-helm.git
# cd harbor-helm/
# git checkout -b v1.3.1.tag v1.3.1
# git diff values.yaml
diff --git a/values.yaml b/values.yaml
index b201d98..fdc5593 100644
--- a/values.yaml
+++ b/values.yaml
@@ -26,8 +26,8 @@ expose:
     commonName: ""
   ingress:
     hosts:
-      core: core.harbor.domain
-      notary: notary.harbor.domain
+      core: core.harbor.singhwang.com
+      notary: notary.harbor.singhwang.com
     # set to the type of ingress controller if it has specific requirements.
     # leave as `default` for most ingress controllers.
     # set to `gce` if using the GCE ingress controller
@@ -98,7 +98,7 @@ expose:
 # the IP address of k8s node
 #
 # If Harbor is deployed behind the proxy, set it as the URL of proxy
-externalURL: https://core.harbor.domain
+externalURL: https://core.harbor.singhwang.com

 # The persistence is enabled by default and a default StorageClass
 # is needed in the k8s cluster to provision volumes dynamicly.
@@ -120,19 +120,19 @@ persistence:
       # Specify the "storageClass" used to provision the volume. Or the default
       # StorageClass will be used(the default).
       # Set it to "-" to disable dynamic provisioning
-      storageClass: ""
+      storageClass: "managed-nfs-storage"
       subPath: ""
       accessMode: ReadWriteOnce
       size: 5Gi
     chartmuseum:
       existingClaim: ""
-      storageClass: ""
+      storageClass: "managed-nfs-storage"
       subPath: ""
       accessMode: ReadWriteOnce
       size: 5Gi
     jobservice:
       existingClaim: ""
-      storageClass: ""
+      storageClass: "managed-nfs-storage"
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
@@ -140,7 +140,7 @@ persistence:
     # be ignored
     database:
       existingClaim: ""
-      storageClass: ""
+      storageClass: "managed-nfs-storage"
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
@@ -148,7 +148,7 @@ persistence:
     # be ignored
     redis:
       existingClaim: ""
-      storageClass: ""
+      storageClass: "managed-nfs-storage"
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
@@ -250,7 +250,7 @@ updateStrategy:
 logLevel: info

 # The initial password of Harbor admin. Change it from portal after launching Harbor
-harborAdminPassword: "Harbor12345"
+harborAdminPassword: "190708"
 # The secret key used for encryption. Must be a string of 16 chars.
 secretKey: "not-a-secure-key"
## 按照上述git对比出来的变化进行修改，storageClass 字段的请依据自己的环境进行配置，比如我环境里的叫 managed-nfs-storage

# helm install harbor harbor-helm --namespace registry
# helm list -n registry
NAME  	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART       	APP VERSION
harbor	registry 	1       	2020-03-04 11:05:18.297145404 +0800 CST	deployed	harbor-1.3.1	1.10.1

# kubectl get pod -n registry -o wide
NAME                                           READY   STATUS    RESTARTS   AGE    IP               NODE     NOMINATED NODE   READINESS GATES
harbor-harbor-chartmuseum-6cc998b954-b65jc     1/1     Running   2          6h7m   10.211.140.95    node02   <none>           <none>
harbor-harbor-clair-654dcfd8bf-tk98w           2/2     Running   10         6h7m   10.211.196.153   node01   <none>           <none>
harbor-harbor-core-85b965488d-tp7jv            1/1     Running   4          6h7m   10.211.196.156   node01   <none>           <none>
harbor-harbor-database-0                       1/1     Running   2          6h7m   10.211.140.94    node02   <none>           <none>
harbor-harbor-jobservice-5f489d87b9-sl57q      1/1     Running   2          6h7m   10.211.140.89    node02   <none>           <none>
harbor-harbor-notary-server-795ccb7bb6-ngfml   1/1     Running   7          6h7m   10.211.196.152   node01   <none>           <none>
harbor-harbor-notary-signer-7595696bf9-kvkk7   1/1     Running   8          6h7m   10.211.140.87    node02   <none>           <none>
harbor-harbor-portal-5cbc6d5897-gm2rc          1/1     Running   2          6h7m   10.211.196.154   node01   <none>           <none>
harbor-harbor-redis-0                          1/1     Running   2          6h7m   10.211.140.93    node02   <none>           <none>
harbor-harbor-registry-75c4f4cc9b-8h72h        2/2     Running   4          6h7m   10.211.140.88    node02   <none>           <none>

四、使用说明

获取 ingress 资源对象中的 HOSTS 和 ADDRESS 在访问端做好 hosts 映射，条件允许的话，也可以配置为网络中的 DNS 记录

1
2
3

# kubectl get ingress -n registry -o wide
NAME                    HOSTS                                                   ADDRESS                           PORTS     AGE
harbor-harbor-ingress   core.harbor.singhwang.com,notary.harbor.singhwang.com   192.168.112.129,192.168.112.130   80, 443   7h7m

访问 Kubernetes Harbor 服务并登陆 https://core.harbor.singhwang.com
在 Kubernetes Harbor 中创建项目

需要使用镜像仓库的Node节点上完成证书的配置

## 创建证书目录
# mkdir -p /etc/docker/certs.d/core.harbor.singhwang.com/

## 获取证书内容
# kubectl get secrets harbor-harbor-ingress -n registry -o jsonpath="{.data.ca\.crt}" | base64 --decode

## 根据证书内容生成证书文件
cat <<EOF > /etc/docker/certs.d/core.harbor.singhwang.com/ca.crt
。。。这里替换为证书内容。。。
EOF

在命令行窗口中推送镜像到项目下

# docker login core.harbor.singhwang.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

# docker tag nginx:1.7.9 core.harbor.singhwang.com/demo/nginx:1.7.9

# docker push core.harbor.singhwang.com/demo/nginx:1.7.9
The push refers to repository [core.harbor.singhwang.com/demo/nginx]
5f70bf18a086: Pushed
4b26ab29a475: Pushed
ccb1d68e3fb7: Pushed
e387107e2065: Pushed
63bf84221cce: Pushed
e02dce553481: Pushed
dea2e4984e29: Pushed
1.7.9: digest: sha256:b1f5935eb2e9e2ae89c0b3e2e148c19068d91ca502e857052f14db230443e4c2 size: 3012

# docker logout core.harbor.singhwang.com
Removing login credentials for core.harbor.singhwang.com

在 Kubernetes Harbor 中查看推送上去的镜像和推送日志

五、参考资料

https://github.com/goharbor/harbor-helm/tree/v1.3.1
https://www.cnblogs.com/longgor/p/11203820.html