We are trying to configure vsphere cloud provider for kubernetes cluster according to below support link for dynamic provisioning from vmware. There was no problem with kubernetres version v1.8, the issue appeared after deploying new kubernetes cluster with version 1.12. https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html
Kubernetes version
v1.12.7
Environment
Cluster: master1,node1,node2
Virtualization: VMware ESXi, 6.5.0, 7388607 , each virtual machine node placed on different esxi host, diskuuid is true
Network: Cisco Apic, Version 3.2(4e)
Problem definition:
Storage class, pv, pvc successfull created and bounded, vmdk disk added to virtual machine successfully but node cannot mount this physical disk to virtual path of the pod, there are two different log not sure which one is right, first log mention _"cloud provider not initialized"_, if not how vmdk created and attached to virtual machine, second log mention _"mount failed because not found the ...vmdk"_ but i can browse from the vcenter mentioned vmdk is in the datastore"
[root@kubemaster ~]# kubectl get pods pvpod
NAME READY STATUS RESTARTS AGE
pvpod 0/1 ContainerCreating 0 17h
kubectl descrice pod pvpod
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe pod pvpod
Name: pvpod
Namespace: default
Priority: 0
PriorityClassName:
Node: kubenode2/10.1.1.12
Start Time: Tue, 14 May 2019 12:52:18 +0300
Labels:
Annotations: opflex.cisco.com/computed-endpoint-group: {"policy-space":"Kubernetes","name":"kubernetes|kube-default"}
opflex.cisco.com/computed-security-group: []
Status: Pending
IP:
Containers:
test-container:
Container ID:
Image: gcr.io/google_containers/test-webserver
Image ID:
Port:
Host Port:
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/test-vmdk from test-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pvf97 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
test-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvcsc001
ReadOnly: false
default-token-pvf97:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pvf97
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m22s default-scheduler Successfully assigned default/pvpod to kubenode2
Warning FailedMount 8m21s kubelet, kubenode2 MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87284.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
Warning FailedMount 8m21s kubelet, kubenode2 MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87288.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
Warning FailedMount 8m20s kubelet, kubenode2 MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87290.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
Warning FailedMount 8m18s kubelet, kubenode2 MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87329.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
Normal SuccessfulAttachVolume 8m17s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3"
Warning FailedMount 8m14s kubelet, kubenode2 MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
/var/log/messages on the node
May 8 16:05:32 kubenode1 kubelet: E0508 16:05:32.730158 53933 vsphere_volume_util.go:187] Cloud provider not initialized properly
May 8 16:05:32 kubenode1 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3.
May 8 16:05:32 kubenode1 kubelet: E0508 16:05:32.744639 53933 mount_linux.go:152] Mount failed: exit status 32
May 8 16:05:32 kubenode1 kubelet: Mounting command: systemd-run
May 8 16:05:32 kubenode1 kubelet: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3
May 8 16:05:32 kubenode1 kubelet: Output: Running scope as unit run-57611.scope.
May 8 16:05:32 kubenode1 kubelet: mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist
May 8 16:05:32 kubenode1 kubelet: E0508 16:05:32.744865 53933 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\" (\"b52b1433-718a-11e9-a3ef-005056b068e3\")" failed. No retries permitted until 2019-05-08 16:07:34.74479901 +0300 +03 m=+619.028655111 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-e3705fec-7189-11e9-a3ef-005056b068e3\" (UniqueName: \"kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\") pod \"pvpod\" (UID: \"b52b1433-718a-11e9-a3ef-005056b068e3\") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\nOutput: Running scope as unit run-57611.scope.\nmount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\n\n"
May 8 16:05:32 kubenode1 kubelet: E0508 16:05:32.830478 53933 vsphere_volume_util.go:187] Cloud provider not initialized properly
Pv,PVC,SC yaml files
Sc-fast.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: datastore: KUBEDATASTORE diskformat: thin fstype: ext3
pvc.yaml
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcsc001 annotations: volume.beta.kubernetes.io/storage-class: fast spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi
pod.yaml
apiVersion: v1 kind: Pod metadata: name: pvpod spec: containers: - name: test-container image: gcr.io/google_containers/test-webserver volumeMounts: - name: test-volume mountPath: /test-vmdk volumes: - name: test-volume persistentVolumeClaim: claimName: pvcsc001
Output of applied yaml files
[root@kubemaster vcp]# kubectl create -f vsphere-volume-sc-fast.yaml
storageclass.storage.k8s.io/fast created
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe storageclass fast
Name: fast
IsDefaultClass: No
Annotations:
Provisioner: kubernetes.io/vsphere-volume
Parameters: datastore=KUBEDATASTORE,diskformat=thin,fstype=ext3
AllowVolumeExpansion:
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:
[root@kubemaster vcp]# kubectl create -f vsphere-volume-pvcsc.yaml persistentvolumeclaim/pvcsc001 created [root@kubemaster vcp]# [root@kubemaster vcp]# kubectl describe pvc pvcsc001 Name: pvcsc001 Namespace: default StorageClass: fast Status: Bound Volume: pvc-4aa82312-72ec-11e9-8b74-005056b068e3 Labels: Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-class: fast volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pvc-protection] Capacity: 2Gi Access Modes: RWO Events: Type Reason Age From Message Normal ProvisioningSucceeded 13s persistentvolume-controller Successfully provisioned volume pvc-4aa82312-72ec-11e9-8b74-005056b068e3 using kubernetes.io/vsphere-volume Mounted By: <.none>
[root@kubemaster vcp]# [root@kubemaster vcp]# kubectl describe pv pvc-4aa82312-72ec-11e9-8b74-005056b068e3 Name: pvc-4aa82312-72ec-11e9-8b74-005056b068e3 Labels: Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner pv.kubernetes.io/bound-by-controller: yes pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pv-protection] StorageClass: fast Status: Bound Claim: default/pvcsc001 Reclaim Policy: Delete Access Modes: RWO Capacity: 2Gi Node Affinity: Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-4aa82312-72ec-11e9-8b74-005056b068e3.vmdk FSType: ext3 StoragePolicyName: Events:
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl create -f vsphere-volume-pvcscpod.yaml
pod/pvpod created
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl get pod pvpod
NAME READY STATUS RESTARTS AGE
pvpod 0/1 ContainerCreating 0 49s
first progress
in the first place when i run “ vsphere-volume-pvcsc.yaml” it was not successfully creating the volume, output of the “kubectl describe pvc” message was No VM found, than i applied https://github.com/kubernetes/kubernetes/issues/65933#issuecomment-408358655 method to match uuids( even providerid and product serial was same), after than volume created successfully and attached into virtual machine. But Pod still ContainerCreating state.
cat /sys/class/dmi/id/product_serial | sed -e 's/^VMware-//' -e 's/-/ /' | awk '{ print toupper($1$2$3$4 "-" $5$6 "-" $7$8 "-" $9$10 "-" $11$12$13$14$15$16) }' kubectl patch node kubemaster -p '{"spec":{"providerID":"vsphere://4230456E-D8B2-AED1-E270-740256CBD273"}}'
Second progress
We have disgussed with @shahbour at #77663 and we made sure it is not a cloud provider issue even the message "cloud provider not initialized properly" message take to the stage while container creating.