kubernetes pod挂载nfs持久化数据权限问题

Posted on | In

问题描述

在k8s中部署gocd-server,因需要持久化数据,gocd-server相关数据目录挂载到了nfs volume,
但这导致gocd-server启动失败.错误提示如下:

1
2
3
4
5
6
7
8
9
root@master01:~/k8s-test/gocd-server# kubectl logs -f test-gocd-server-deployment-5c849557d9-5prc2 -n test
/docker-entrypoint.sh: Creating directories and symlinks to hold GoCD configuration, data, and logs
$ mkdir /go-working-dir
$ chown go:go /go-working-dir
$ ln -sv /godata/artifacts /go-working-dir/artifacts
'/go-working-dir/artifacts' -> '/godata/artifacts'
$ chown go:go /go-working-dir/artifacts
chown: /go-working-dir/artifacts: Operation not permitted
/docker-entrypoint.sh: cannot chown go:go /go-working-dir/artifacts

原因

经分析，是如下原因导致的:

(1) nfs export出来的目录加了”root_squash”,如在nfs client side 用root(uid=0)身份操作该nfs目录,在其内创建的任何文件目录它的owner:grouper会是nobody:nogroup

(2) nfs export出来的目录uid=0,gid=0,即在nfs server side该目录也是uid=0,gid=0,则在client挂载该目录后，只有client side的uid=0的用户拥有该目录读写权限. 但uid=0是root用户，按照(1)的原因，root会变为nobody. 哈哈，这是矛盾的。

(3) 现实情况是，客户给的nfs目录，权限是777, 这样虽然root被压缩成nobody了，但client side能在该目录创建文件和目录，
只不过他们owner:grouper会是nobody:nogroup,虽然能创建文件,但你想chown他们的owner:grouper是不能的

docker-gocd-server github

解决办法

分析gocd-server的dockerfile和entrypoint.sh，你会发现它容器是以root用户启动，但最后gocd-server却是以go用户启动的。

修改其docker-entrypoint.sh，让gocd-server以nobody用户启动即可.

修改后重新打包的镜像：docker pull itwye/gocd-server:fornfs

验证过程

严格模拟线上nfs

nfs配置(该配置模拟的真实环境)

1
2
3
4
5
6
7
#注意:将要export的/opt/nfs目录权限是777
root@node02:/opt# ls -al
drwxrwxrwx  3 root root 4096 Jul  5 12:46 nfs

#注意: root_squash
root@node02:/opt# cat /etc/exports
/opt/nfs           *(rw,sync,root_squash,no_subtree_check)

k8s编排文件

(1) deployment.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-gocd-server-deployment
  labels:
    app: test-gocd-server
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-gocd-server
  template:
    metadata:
      labels:
        app: test-gocd-server
    spec:
      nodeName: node01
      containers:
        - name: test-serve
          ports:
            - containerPort: 8153
            - containerPort: 8154
          livenessProbe:
            httpGet:
              path: /go/api/v1/health
              port: 8153
            initialDelaySeconds: 90
            periodSeconds: 15
            failureThreshold: 10
          readinessProbe:
            httpGet:
              path: /go/api/v1/health
              port: 8153
            initialDelaySeconds: 90
            periodSeconds: 15
            failureThreshold: 10
          #image: gocd/gocd-server:v19.5.0
          image: itwye/gocd-server:fornfs
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: goserver-vol
              mountPath: /docker-entrypoint.d
              subPath: scripts
            - name: goserver-vol
              mountPath: /home/go
              subPath: homego
            - name: goserver-vol
              mountPath: /godata
              subPath: godata
      restartPolicy: Always
      volumes:
        - name: goserver-vol
          persistentVolumeClaim:
            claimName: test-pvc-nfs

(2) service.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: Service
metadata:
  name: test-gocd-server-service
  namespace: test
spec:
  type: NodePort
  selector:
    app: test-gocd-server
  ports:
  - protocol: TCP
    name: http
    port: 8153
    targetPort: 8153
  - protocol: TCP
    name: https
    port: 8154
    targetPort: 8154

(3) pv.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: PersistentVolume
metadata:
  name:  test-pv-nfs
  labels:
    app: test-nfs
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /opt/nfs
    server: 172.16.20.12

(4) pvc.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc-nfs
  labels:
    app: test-nfs
  namespace: test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      app: test-nfs

参考

(1) nfs root_squash

1
2
no_root_squash：登入 NFS 主机使用分享目录的使用者，如果是 root 的话，那么对于这个分享的目录来说，他就具有 root 的权限！这个项目『极不安全』，不建议使用！ 
root_squash：在登入 NFS 主机使用分享之目录的使用者如果是 root 时，那么这个使用者的权限将被压缩成为匿名使用者，通常他的 UID 与 GID 都会变成 nobody 那个系统账号的身份；