Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EFS Volume creation automation #2

Open
jazzyarchitects opened this issue Jan 21, 2019 · 6 comments
Open

EFS Volume creation automation #2

jazzyarchitects opened this issue Jan 21, 2019 · 6 comments

Comments

@jazzyarchitects
Copy link

I am trying to create a Hyperledger network and I get the following error for the pods when RCA is being deployed.

Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv --scope -- mount -t nfs -o rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv
Output: Running as unit run-25323.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 failed, reason given by server: No such file or directory
  Warning  FailedMount  26s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-data-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv --scope -- mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv
Output: Running as unit run-25324.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data failed, reason given by server: No such file or directory
  Warning  FailedMount  26s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running as unit run-25326.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directory
  Warning  FailedMount  26s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv --scope -- mount -t nfs -o timeo=600,retrans=2,nfsvers=4.1,rsize=1048576,wsize=1048576,hard fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv
Output: Running as unit run-25339.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 failed, reason given by server: No such file or directory
  Warning  FailedMount  26s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-data-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv --scope -- mount -t nfs -o retrans=2,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv
Output: Running as unit run-25337.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data failed, reason given by server: No such file or directory
  Warning  FailedMount  26s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1,rsize=1048576 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running as unit run-25338.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directory
  Warning  FailedMount  25s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-data-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv --scope -- mount -t nfs -o rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-data-org0-pv
Output: Running as unit run-25353.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-data failed, reason given by server: No such file or directory
  Warning  FailedMount  25s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv --scope -- mount -t nfs -o wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1,rsize=1048576 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-org0-pv
Output: Running as unit run-25352.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-org0 failed, reason given by server: No such file or directory
  Warning  FailedMount  25s  kubelet, ip-172-20-44-136.us-west-2.compute.internal  MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running as unit run-25360.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directory
  Warning  FailedMount  10s (x9 over 23s)  kubelet, ip-172-20-44-136.us-west-2.compute.internal  (combined from similar events): MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o hard,timeo=600,retrans=2,nfsvers=4.1,rsize=1048576,wsize=1048576 fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/a52c6f76-1d3e-11e9-a94b-024d18c16a92/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running as unit run-25669.scope.
mount.nfs: mounting fs-80deba28.efs.us-west-2.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directory

I can see that it is because the mount directories don't exist in the NFS server but isn't the script supposed to take care of that instead of manually mounting and creating directories?

@hieu29791
Copy link

Hi, I have the same problem here, any update?

@jazzyarchitects
Copy link
Author

@hieu29791 I did some workaround for this. I had to create a kubernetes job which is run after the provisioner is created. This kubernetes job basically runs the below shell script:

          mkdir -p /nfs/${id}/rca-scripts
          mkdir -p /nfs/${id}/ica
          mkdir -p /nfs/${id}/rca
          mkdir -p /nfs/${id}/rca-data
          mkdir -p /nfs/${id}/orderer

          chmod a+rwx /nfs/${id}/rca-scripts
          chmod a+rwx /nfs/${id}/ica
          chmod a+rwx /nfs/${id}/rca
          chmod a+rwx /nfs/${id}/rca-data
          chmod a+rwx /nfs/${id}/orderer

          rm -rf /nfs/${id}/rca-scripts/*
          rm -rf /nfs/${id}/ica/*
          rm -rf /nfs/${id}/rca/*
          rm -rf /nfs/${id}/rca-data/*
          rm -rf /nfs/${id}/orderer/*

where ${id} is the org name.

Complete config:

Job:

apiVersion: 'batch/v1',
    kind: 'Job',
    metadata: {
      name: 'nfs-partitions',
      namespace: `${domain}`,
    },
    spec: {
      template: {
        spec: {
          containers: [
            {
              name: 'partitioner',
              image: 'alpine',
              command: ['sh', '/conf/init.sh'],
              volumeMounts: [
                {
                  name: 'init',
                  mountPath: '/conf',
                },
                {
                  name: 'nfs',
                  mountPath: '/nfs',
                },
              ],
            },
          ],
          volumes: [
            {
              name: 'init',
              configMap: {
                name: 'partition-create-script',
              },
            },
            {
              name: 'nfs',
              persistentVolumeClaim: {
                claimName: `raw-nfs-mount-${id}`,
              },
            },
          ],
          restartPolicy: 'Never',
        },
      },
      backoffLimit: 1,
    }

PVCs:

kind: 'PersistentVolumeClaim',
    apiVersion: 'v1',
    metadata: {
      name: `raw-nfs-mount-${id}`,
      namespace: `${domain}`,
    },
    spec: {
      accessModes: ['ReadWriteMany'],
      resources: {
        requests: {
          storage: '1Gi',
        },
      },
      storageClassName,
      volumeName: `raw-nfs-mount-${id}`,
    },

PV:

 kind: 'PersistentVolume',
    apiVersion: 'v1',
    metadata: {
      name: `raw-nfs-mount-${id}`,
      namespace: `${domain}`,
    },
    spec: {
      capacity: {
        storage: '1Gi',
      },
      storageClassName,
      accessModes: ['ReadWriteMany'],
      persistentVolumeReclaimPolicy: 'Retain',
      mountOptions: ['nfsvers=4.1', 'rsize=1048576', 'wsize=1048576', 'hard', 'timeo=600', 'retrans=2'],
      nfs: {
        server: efsServer,
        path: '/',
      },
    },

Configmap:

apiVersion: 'v1',
    kind: 'ConfigMap',
    metadata: {
      name: 'partition-create-script',
      labels: {
        name: 'partition-create-script',
      },
      namespace: `${domain}`,
    },
    data: {
      'init.sh': `
          mkdir -p /nfs/${id}/rca-scripts
          mkdir -p /nfs/${id}/ica
          mkdir -p /nfs/${id}/rca
          mkdir -p /nfs/${id}/rca-data
          mkdir -p /nfs/${id}/orderer

          chmod a+rwx /nfs/${id}/rca-scripts
          chmod a+rwx /nfs/${id}/ica
          chmod a+rwx /nfs/${id}/rca
          chmod a+rwx /nfs/${id}/rca-data
          chmod a+rwx /nfs/${id}/orderer

          rm -rf /nfs/${id}/rca-scripts/*
          rm -rf /nfs/${id}/ica/*
          rm -rf /nfs/${id}/rca/*
          rm -rf /nfs/${id}/rca-data/*
          rm -rf /nfs/${id}/orderer/*
        `,
    },

Hope this helps

@hieu29791
Copy link

@jazzyarchitects Thanks for your help. Could you explain more to me about this step? When I run start-fabric.sh I want automatically create folders on EFS. How to I migration your script and this example, I just run start-fabric.sh and this automatically do the rest.

@MCLDG-zz
Copy link

MCLDG-zz commented Feb 23, 2019 via email

@hieu29791
Copy link

When you run start-fabric.sh, it calls a function makeDirs. This creates the directories on efs. Can you check the output of start-fabric.sh and confirm any errors that are seen here? You can check the dirs on efs by doing ls /opt/share on the bastion, since efs is mounted on the bastion also .

On Fri, 22 Feb 2019, 17:04 hieu29791, @.***> wrote: @jazzyarchitects https://github.com/jazzyarchitects Thanks for your help. Could you explain more to me about this step? When I run start-fabric.sh I want automatically create folders on EFS. How to I migration your script and this example, I just run start-fabric.sh and this automatically do the rest. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AL0HGpGQ94-NlWX8yjomu5Aq0hkX-zhZks5vP7KIgaJpZM4aKJLl .

Hi,
When I run start-fabric.sh, on the step Starting RCA in K8s, error display like this:

kubectl describe pod -n org0
Name:               rca-org0-7c4977ccc5-4lswg
Namespace:          org0
Priority:           0
PriorityClassName:  <none>
Node:               ip-192-168-229-54.ap-southeast-1.compute.internal/192.168.229.54
Start Time:         Mon, 25 Feb 2019 04:34:42 +0000
Labels:             app=hyperledger
                    name=rca-org0
                    org=org0
                    pod-template-hash=3705337771
                    role=ca
Annotations:        <none>
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/rca-org0-7c4977ccc5
Containers:
  rca-org0:
    Container ID:  
    Image:         hyperledger/fabric-ca:x86_64-1.1.0
    Image ID:      
    Port:          7054/TCP
    Host Port:     0/TCP
    Command:
      sh
    Args:
      -c
      /scripts/start-root-ca.sh 2>&1
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      FABRIC_CA_SERVER_HOME:         /etc/hyperledger/fabric-ca
      FABRIC_CA_SERVER_TLS_ENABLED:  true
      FABRIC_CA_SERVER_CSR_CN:       rca-org0.org0
      FABRIC_CA_SERVER_CSR_HOSTS:    rca-org0.org0
      FABRIC_CA_SERVER_DEBUG:        true
      BOOTSTRAP_USER_PASS:           rca-org0-admin:rca-org0-adminpw
      TARGET_CERTFILE:               /data/org0-ca-cert.pem
      FABRIC_ORGS:                   org0.org0 org1.org1 org2.org2
    Mounts:
      /data from rca-data (rw)
      /etc/hyperledger/fabric-ca from rca (rw)
      /scripts from rca-scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-v69sv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  rca:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rca-org0-pvc
    ReadOnly:   false
  rca-scripts:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rca-scripts-org0-pvc
    ReadOnly:   false
  rca-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rca-data-org0-pvc
    ReadOnly:   false
  default-token-v69sv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-v69sv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                                                        Message
  ----     ------       ----               ----                                                        -------
  Warning  FailedMount  19m (x29 over 1h)  kubelet, ip-192-168-229-54.ap-southeast-1.compute.internal  Unable to mount volumes for pod "rca-org0-7c4977ccc5-4lswg_org0(abc00cc9-38b6-11e9-aef9-0a59596ba446)": timeout expired waiting for volumes to attach or mount for pod "org0"/"rca-org0-7c4977ccc5-4lswg". list of unmounted volumes=[rca rca-scripts rca-data]. list of unattached volumes=[rca rca-scripts rca-data default-token-v69sv]
  Warning  FailedMount  1m (x211 over 2h)  kubelet, ip-192-168-229-54.ap-southeast-1.compute.internal  (combined from similar events): MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/abc00cc9-38b6-11e9-aef9-0a59596ba446/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o hard,timeo=600,retrans=2,nfsvers=4.1,rsize=1048576,wsize=1048576 fs-87913ec6.efs.ap-southeast-1.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/abc00cc9-38b6-11e9-aef9-0a59596ba446/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running scope as unit run-26504.scope.
mount.nfs: mounting fs-87913ec6.efs.ap-southeast-1.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directory

On Bastion, I see folder already create on /op/share/ I don't know what wrong is it. Could you explain to me.

tree /opt/share         
/opt/share
├── ica-org0
├── ica-org1
├── ica-org2
├── orderer
├── rca-data
├── rca-org0
├── rca-org1
├── rca-org2
└── rca-scripts
    ├── addorg-fabric-join.sh
    ├── addorg-fabric-setup.sh
    ├── create-channel.sh
    ├── env3orgs.sh
    ├── env.sh
    ├── fabric-delete-org.sh
    ├── fabric-installcc.sh
    ├── fabric-signconf.sh
    ├── fabric-updateconf.sh
    ├── fabric-upgradecc.sh
    ├── gen-channel-artifacts.sh
    ├── gen-channel-artifacts-template.sh
    ├── install-marbles-cc.sh
    ├── load-fabric-marbles.sh
    ├── load-fabric.sh
    ├── peer-join-channel.sh
    ├── register-orderer.sh
    ├── register-org.sh
    ├── register-peer.sh
    ├── start-intermediate-ca.sh
    ├── start-orderer.sh
    ├── start-peer.sh
    ├── start-root-ca.sh
    ├── test-fabric-abac.sh
    ├── test-fabric-marbles.sh
    ├── test-fabric-marbles-workshop.sh
    └── test-marbles-cc.sh

@msolefonte
Copy link

Excuse me by starting this topic again. Can't be this problem related to execute the code in a local machine with kubectl and not in the ec2 instance? Executed in local, makeDir is going to create the folders in the local path, default /opt/share, which is going to allow the script continue at that point, but will show the error when data is required and not set in the EFS.

I would recommend to create the files not in local but in a job as the default way, as shown by @jazzyarchitects. This way fabric could be started from everywhere. One problem of this would be that start-fabric.shstill requires to access data from local path (assuming that it is the EFS volume), so one patch would be to create that on a folder of /tmp and deleting in after the installation. It can be completely redone, of course, but it is harder and requires more time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants