Skip to main content
Solved

Backup-Policy with Blueprint for MariaDB Deployment fails


Hi!

I have problems with using a blueprint to backup a mariadb deployment in kubernetes. After running trough a series of error messages I made ot to the last that did not let me successful back up the workload with a blueprint: 

 

Show content
cause:
  cause:
    fields:
      - name: message
        value: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod failed
          or did not transition into complete state: Pod kanister-job-t8pw4
          failed. Pod details (&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
          APPNAME-CUSTOMER-test  754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
          2022-04-29 09:55:17 +0000 UTC <nil> <nil> map[createdBy:kanister]
          map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
          cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
          []  [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC FieldsV1
          {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
          } {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
          {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
          } {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
          {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
          }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
          00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
          UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
          00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
          UTC,Reason:ContainersNotReady,Message:containers with unready status:
          [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
          00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
          UTC,Reason:ContainersNotReady,Message:containers with unready status:
          [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
          00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
          UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
          09:55:17 +0000
          UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
          09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
          UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
      - name: actionSet
        value:
          metadata:
            creationTimestamp: 2022-04-29T09:55:17Z
            generateName: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-deployment-
            generation: 4
            labels:
              kanister.io/JobID: 20f47d60-c7a2-11ec-9afe-ce64f5560a62
            managedFields:
              - apiVersion: cr.kanister.io/v1alpha1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                    f:labels:
                      .: {}
                      f:kanister.io/JobID: {}
                  f:spec:
                    .: {}
                    f:actions: {}
                  f:status:
                    .: {}
                    f:actions: {}
                    f:error:
                      .: {}
                      f:message: {}
                    f:state: {}
                manager: Go-http-client
                operation: Update
                time: 2022-04-29T09:55:17Z
            name: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d655zh
            namespace: kasten-io
            resourceVersion: "42480091"
            uid: fa5d502f-bd86-48bb-be43-7ad3e345caef
          spec:
            actions:
              - blueprint: APPNAME-mariadb-blueprint
                name: backup
                object:
                  apiVersion: v1
                  group: apps
                  kind: deployment
                  name: mariadb
                  namespace: APPNAME-CUSTOMER-test
                  resource: deployments
                options:
                  objectStorePath: repo/dbab9b79-db59-4905-b6bf-635c8699b479/
                preferredVersion: v1.0.0-alpha
                profile:
                  apiVersion: v1alpha1
                  group: ""
                  kind: profile
                  name: kanister-phase-kopia-profile-j8fjf
                  namespace: kasten-io
                  resource: ""
          status:
            actions:
              - artifacts:
                  mariaBackup:
                    kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
                blueprint: APPNAME-mariadb-blueprint
                deferPhase:
                  name: ""
                  state: ""
                name: backup
                object:
                  apiVersion: v1
                  group: apps
                  kind: deployment
                  name: mariadb
                  namespace: APPNAME-CUSTOMER-test
                  resource: deployments
                phases:
                  - name: dumpToStore
                    state: failed
            error:
              message: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod
                failed or did not transition into complete state: Pod
                kanister-job-t8pw4 failed. Pod details
                (&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
                APPNAME-CUSTOMER-test  754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
                2022-04-29 09:55:17 +0000 UTC <nil> <nil>
                map[createdBy:kanister]
                map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
                cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
                []  [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC
                FieldsV1
                {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
                } {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
                {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
                } {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
                {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
                }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
                00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
                UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
                00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
                UTC,Reason:ContainersNotReady,Message:containers with unready
                status:
                [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
                00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
                UTC,Reason:ContainersNotReady,Message:containers with unready
                status:
                [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
                00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
                UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
                09:55:17 +0000
                UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
                09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
                UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
            state: failed
    file: kasten.io/k10/kio/kanister/operation.go:114
    function: kasten.io/k10/kio/kanister.(*Operation).Execute
    linenumber: 114
    message: ActionSet Failed
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:676
  function: kasten.io/k10/kio/exec/phases/backup.RunKanisterSnapshotActions
  linenumber: 676
  message: Failed to create backup kanister phase
message: Job failed to be executed
fields: []

 

 

 

 

 

Details of my Environment: 

Kubernetes: v1.21.5

K10: Upgraded today to 4.5.14

MariaDB in Deployment: 10.6 (This is not the StatefulSet from the Docs! Its just a Deployment-Type)

 

The Snapshot creation with the normal mode has no problems. Only my try to setup a blueprint for that mariadb database fails. 

 

 

If it helps: I add my blueprint here: 

Show content
apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: mariadb-blueprint
actions:
  backup:
    outputArtifacts:
      mariaBackup:
        # Capture the kopia snapshot information for subsequent actions
        # The information includes the kopia snapshot ID which is essential for restore and delete to succeed
        # `kopiaOutput` is the name provided to kando using `--output-name` flag
        kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
    phases:
    - func: KubeTask
      name: dumpToStore
      objects:
        mariaSecret:
          kind: Secret
          name: '{{ .Deployment.Name }}'
          namespace: '{{ .Deployment.Namespace }}'
      args:
        image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
        namespace: "{{ .Deployment.Namespace }}"
        command:
        - bash
        - -o
        - errexit
        - -o
        - pipefail
        - -c
        - |
          backup_file_path="dump.sql"
          root_password="{{ index .Phases.dumpToStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
          exclude_db="('information_schema', 'mysql', 'performance_schema')"
          dump_databases=""
          for db in `mysql -u root  -h {{ .Deployment.Name }} --password=${root_password} -ANe "use information_schema; select schema_name from schemata where schema_name not in ${exclude_db}"`
          do
                  dump_databases="${dump_databases} ${db}"
          done
          dump_cmd="mysqldump -u root --password=${root_password} -h {{ .Deployment.Name }} --column-statistics=0 --single-transaction --databases ${dump_databases}"
          ${dump_cmd} | kando location push --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --output-name "kopiaOutput" -
  restore:
    inputArtifactNames:
    # The kopia snapshot info created in backup phase can be used here
    # Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
    - mariaBackup
    phases:
    - func: KubeTask
      name: restoreFromStore
      objects:
        mariaSecret:
          kind: Secret
          name: '{{ .Deployment.Name }}'
          namespace: '{{ .Deployment.Namespace }}'
      args:
        image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
        namespace: "{{ .Deployment.Namespace }}"
        command:
        - bash
        - -o
        - errexit
        - -o
        - pipefail
        - -c
        - |
          backup_file_path="dump.sql"
          kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
          root_password="{{ index .Phases.restoreFromStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
          kando location pull --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}" - | mysql -u root --password=${root_password} -h {{ .Deployment.Name }}
  delete:
    inputArtifactNames:
    # The kopia snapshot info created in backup phase can be used here
    # Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
    - mariaBackup
    phases:
    - func: KubeTask
      name: deleteFromStore
      args:
        image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
        namespace: "{{ .Namespace.Name }}"
        command:
        - bash
        - -o
        - errexit
        - -o
        - pipefail
        - -c
        - |
          backup_file_path="dump.sql"
          kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
          kando location delete --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}"

 

Can you guys tell me where I am doing something wrong? 

 

Thanks a lot and have a nice day!

Best answer by jaiganeshjk

@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?

This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?

View original
Did this topic help you find an answer to your question?

5 comments

Forum|alt.badge.img+1
  • Comes here often
  • 89 comments
  • May 2, 2022

Hello SMT-Almato,

 

Could you please rerun the policy and within the first ten minutes provide the events for kasten-io namespace.

 

kubectl get events -n kasten-io

 

Thanks

Emmanuel


  • Author
  • Comes here often
  • 19 comments
  • May 5, 2022
EBrockman wrote:

kubectl get events -n kasten-io

Hi Emmanuel, 

thank you for your reply. Of course: 

Show content
0s          Warning   ActionSetFailed Action: backup   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dvrkd8   Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test  f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Warning   ActionSetFailed Action: backup   blueprint/APPNAME-mariadb-blueprint                                           Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test  f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Normal    Killing                          pod/data-mover-svc-ctc2g                                                    Stopping container container
0s          Normal    Scheduled                        pod/data-mover-svc-l2mps                                                    Successfully assigned kasten-io/data-mover-svc-l2mps to K8SNODE-cyo102
0s          Normal    Pulled                           pod/data-mover-svc-l2mps                                                    Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s          Normal    Created                          pod/data-mover-svc-l2mps                                                    Created container container
0s          Normal    Started                          pod/data-mover-svc-l2mps                                                    Started container container
0s          Normal    Started Action                   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl   Executing action backup
0s          Normal    Started Phase                    actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl   Executing phase dumpToStore
0s          Warning   ActionSetFailed Action: backup   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl   Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test  d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Warning   ActionSetFailed Action: backup   blueprint/APPNAME-mariadb-blueprint                                           Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test  d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Normal    Scheduled                        pod/backup-data-stats-wr98h                                                 Successfully assigned kasten-io/backup-data-stats-wr98h to K8SNODE-cyo102
0s          Normal    Pulled                           pod/backup-data-stats-wr98h                                                 Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s          Normal    Created                          pod/backup-data-stats-wr98h                                                 Created container container
0s          Normal    Started                          pod/backup-data-stats-wr98h                                                 Started container container
0s          Normal    Killing                          pod/backup-data-stats-wr98h                                                 Stopping container container
0s          Normal    Killing                          pod/data-mover-svc-l2mps                                                    Stopping container container
0s          Warning   FailedToUpdateEndpoint           endpoints/data-mover-svc-f4kqg                                              Failed to update endpoint kasten-io/data-mover-svc-f4kqg: Operation cannot be fulfilled on endpoints "data-mover-svc-f4kqg": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/kasten-io/data-mover-svc-f4kqg, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 39e1cda0-7b17-4ac9-98b1-da3efddb0c7c, UID in object meta:
0s          Normal    Scheduled                        pod/data-mover-svc-njqxz                                                    Successfully assigned kasten-io/data-mover-svc-njqxz to K8SNODE-cyo102
0s          Normal    Pulled                           pod/data-mover-svc-njqxz                                                    Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s          Normal    Created                          pod/data-mover-svc-njqxz                                                    Created container container
0s          Normal    Started                          pod/data-mover-svc-njqxz                                                    Started container container
0s          Normal    Started Action                   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf   Executing action backup
0s          Normal    Started Phase                    actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf   Executing phase dumpToStore
0s          Warning   ActionSetFailed Action: backup   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf   Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test  84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Warning   ActionSetFailed Action: backup   blueprint/APPNAME-mariadb-blueprint                                           Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test  84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s          Normal    Scheduled                        pod/backup-data-stats-swhj5                                                 Successfully assigned kasten-io/backup-data-stats-swhj5 to K8SNODE-cyo102
0s          Normal    Pulled                           pod/backup-data-stats-swhj5                                                 Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s          Normal    Created                          pod/backup-data-stats-swhj5                                                 Created container container
0s          Normal    Started                          pod/backup-data-stats-swhj5                                                 Started container container
0s          Normal    Killing                          pod/backup-data-stats-swhj5                                                 Stopping container container
0s          Normal    Killing                          pod/data-mover-svc-njqxz                                                    Stopping container container

 

 


  • Author
  • Comes here often
  • 19 comments
  • May 11, 2022

Is there anything you could also need? Maybe someone could give me a hint, in which direction I have to search. 

 

Thanks a lot!


jaiganeshjk
Forum|alt.badge.img+2
  • Experienced User
  • 275 comments
  • Answer
  • May 31, 2022

@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?

This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?


  • Author
  • Comes here often
  • 19 comments
  • June 3, 2022

I will do this, but this will need a few days. I will respond here, hopefully with a solution. 


Comment