Skip to main content
Solved

Backup-Policy with Blueprint for MariaDB Deployment fails


Hi!

I have problems with using a blueprint to backup a mariadb deployment in kubernetes. After running trough a series of error messages I made ot to the last that did not let me successful back up the workload with a blueprint: 

 

Show content
1cause:
2 cause:
3 fields:
4 - name: message
5 value: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod failed
6 or did not transition into complete state: Pod kanister-job-t8pw4
7 failed. Pod details (&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
8 APPNAME-CUSTOMER-test 754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
9 2022-04-29 09:55:17 +0000 UTC <nil> <nil> map[createdBy:kanister]
10 map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
11 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
12 [] [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC FieldsV1
13 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
14 } {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
15 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
16 } {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
17 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
18 }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
19 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
20 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
21 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
22 UTC,Reason:ContainersNotReady,Message:containers with unready status:
23 [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
24 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
25 UTC,Reason:ContainersNotReady,Message:containers with unready status:
26 [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
27 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
28 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
29 09:55:17 +0000
30 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
31 09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
32 UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
33 - name: actionSet
34 value:
35 metadata:
36 creationTimestamp: 2022-04-29T09:55:17Z
37 generateName: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-deployment-
38 generation: 4
39 labels:
40 kanister.io/JobID: 20f47d60-c7a2-11ec-9afe-ce64f5560a62
41 managedFields:
42 - apiVersion: cr.kanister.io/v1alpha1
43 fieldsType: FieldsV1
44 fieldsV1:
45 f:metadata:
46 f:generateName: {}
47 f:labels:
48 .: {}
49 f:kanister.io/JobID: {}
50 f:spec:
51 .: {}
52 f:actions: {}
53 f:status:
54 .: {}
55 f:actions: {}
56 f:error:
57 .: {}
58 f:message: {}
59 f:state: {}
60 manager: Go-http-client
61 operation: Update
62 time: 2022-04-29T09:55:17Z
63 name: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d655zh
64 namespace: kasten-io
65 resourceVersion: "42480091"
66 uid: fa5d502f-bd86-48bb-be43-7ad3e345caef
67 spec:
68 actions:
69 - blueprint: APPNAME-mariadb-blueprint
70 name: backup
71 object:
72 apiVersion: v1
73 group: apps
74 kind: deployment
75 name: mariadb
76 namespace: APPNAME-CUSTOMER-test
77 resource: deployments
78 options:
79 objectStorePath: repo/dbab9b79-db59-4905-b6bf-635c8699b479/
80 preferredVersion: v1.0.0-alpha
81 profile:
82 apiVersion: v1alpha1
83 group: ""
84 kind: profile
85 name: kanister-phase-kopia-profile-j8fjf
86 namespace: kasten-io
87 resource: ""
88 status:
89 actions:
90 - artifacts:
91 mariaBackup:
92 kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
93 blueprint: APPNAME-mariadb-blueprint
94 deferPhase:
95 name: ""
96 state: ""
97 name: backup
98 object:
99 apiVersion: v1
100 group: apps
101 kind: deployment
102 name: mariadb
103 namespace: APPNAME-CUSTOMER-test
104 resource: deployments
105 phases:
106 - name: dumpToStore
107 state: failed
108 error:
109 message: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod
110 failed or did not transition into complete state: Pod
111 kanister-job-t8pw4 failed. Pod details
112 (&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
113 APPNAME-CUSTOMER-test 754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
114 2022-04-29 09:55:17 +0000 UTC <nil> <nil>
115 map[createdBy:kanister]
116 map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
117 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
118 [] [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC
119 FieldsV1
120 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
121 } {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
122 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
123 } {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
124 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
125 }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
126 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
127 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
128 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
129 UTC,Reason:ContainersNotReady,Message:containers with unready
130 status:
131 [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
132 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
133 UTC,Reason:ContainersNotReady,Message:containers with unready
134 status:
135 [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
136 00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
137 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
138 09:55:17 +0000
139 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
140 09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
141 UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
142 state: failed
143 file: kasten.io/k10/kio/kanister/operation.go:114
144 function: kasten.io/k10/kio/kanister.(*Operation).Execute
145 linenumber: 114
146 message: ActionSet Failed
147 file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:676
148 function: kasten.io/k10/kio/exec/phases/backup.RunKanisterSnapshotActions
149 linenumber: 676
150 message: Failed to create backup kanister phase
151message: Job failed to be executed
152fields: []
153

 

 

 

 

 

Details of my Environment: 

Kubernetes: v1.21.5

K10: Upgraded today to 4.5.14

MariaDB in Deployment: 10.6 (This is not the StatefulSet from the Docs! Its just a Deployment-Type)

 

The Snapshot creation with the normal mode has no problems. Only my try to setup a blueprint for that mariadb database fails. 

 

 

If it helps: I add my blueprint here: 

Show content
1apiVersion: cr.kanister.io/v1alpha1
2kind: Blueprint
3metadata:
4 name: mariadb-blueprint
5actions:
6 backup:
7 outputArtifacts:
8 mariaBackup:
9 # Capture the kopia snapshot information for subsequent actions
10 # The information includes the kopia snapshot ID which is essential for restore and delete to succeed
11 # `kopiaOutput` is the name provided to kando using `--output-name` flag
12 kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
13 phases:
14 - func: KubeTask
15 name: dumpToStore
16 objects:
17 mariaSecret:
18 kind: Secret
19 name: '{{ .Deployment.Name }}'
20 namespace: '{{ .Deployment.Namespace }}'
21 args:
22 image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
23 namespace: "{{ .Deployment.Namespace }}"
24 command:
25 - bash
26 - -o
27 - errexit
28 - -o
29 - pipefail
30 - -c
31 - |
32 backup_file_path="dump.sql"
33 root_password="{{ index .Phases.dumpToStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
34 exclude_db="('information_schema', 'mysql', 'performance_schema')"
35 dump_databases=""
36 for db in `mysql -u root -h {{ .Deployment.Name }} --password=${root_password} -ANe "use information_schema; select schema_name from schemata where schema_name not in ${exclude_db}"`
37 do
38 dump_databases="${dump_databases} ${db}"
39 done
40 dump_cmd="mysqldump -u root --password=${root_password} -h {{ .Deployment.Name }} --column-statistics=0 --single-transaction --databases ${dump_databases}"
41 ${dump_cmd} | kando location push --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --output-name "kopiaOutput" -
42 restore:
43 inputArtifactNames:
44 # The kopia snapshot info created in backup phase can be used here
45 # Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
46 - mariaBackup
47 phases:
48 - func: KubeTask
49 name: restoreFromStore
50 objects:
51 mariaSecret:
52 kind: Secret
53 name: '{{ .Deployment.Name }}'
54 namespace: '{{ .Deployment.Namespace }}'
55 args:
56 image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
57 namespace: "{{ .Deployment.Namespace }}"
58 command:
59 - bash
60 - -o
61 - errexit
62 - -o
63 - pipefail
64 - -c
65 - |
66 backup_file_path="dump.sql"
67 kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
68 root_password="{{ index .Phases.restoreFromStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
69 kando location pull --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}" - | mysql -u root --password=${root_password} -h {{ .Deployment.Name }}
70 delete:
71 inputArtifactNames:
72 # The kopia snapshot info created in backup phase can be used here
73 # Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
74 - mariaBackup
75 phases:
76 - func: KubeTask
77 name: deleteFromStore
78 args:
79 image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
80 namespace: "{{ .Namespace.Name }}"
81 command:
82 - bash
83 - -o
84 - errexit
85 - -o
86 - pipefail
87 - -c
88 - |
89 backup_file_path="dump.sql"
90 kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
91 kando location delete --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}"

 

Can you guys tell me where I am doing something wrong? 

 

Thanks a lot and have a nice day!

Best answer by jaiganeshjk

@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?

This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?

View original
Did this topic help you find an answer to your question?

5 comments

Forum|alt.badge.img+1
  • Comes here often
  • 89 comments
  • May 2, 2022

Hello SMT-Almato,

 

Could you please rerun the policy and within the first ten minutes provide the events for kasten-io namespace.

 

kubectl get events -n kasten-io

 

Thanks

Emmanuel


  • Author
  • Comes here often
  • 19 comments
  • May 5, 2022
EBrockman wrote:

kubectl get events -n kasten-io

Hi Emmanuel, 

thank you for your reply. Of course: 

Show content
10s Warning ActionSetFailed Action: backup actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dvrkd8 Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
20s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
30s Normal Killing pod/data-mover-svc-ctc2g Stopping container container
40s Normal Scheduled pod/data-mover-svc-l2mps Successfully assigned kasten-io/data-mover-svc-l2mps to K8SNODE-cyo102
50s Normal Pulled pod/data-mover-svc-l2mps Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
60s Normal Created pod/data-mover-svc-l2mps Created container container
70s Normal Started pod/data-mover-svc-l2mps Started container container
80s Normal Started Action actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Executing action backup
90s Normal Started Phase actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Executing phase dumpToStore
100s Warning ActionSetFailed Action: backup actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
110s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
120s Normal Scheduled pod/backup-data-stats-wr98h Successfully assigned kasten-io/backup-data-stats-wr98h to K8SNODE-cyo102
130s Normal Pulled pod/backup-data-stats-wr98h Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
140s Normal Created pod/backup-data-stats-wr98h Created container container
150s Normal Started pod/backup-data-stats-wr98h Started container container
160s Normal Killing pod/backup-data-stats-wr98h Stopping container container
170s Normal Killing pod/data-mover-svc-l2mps Stopping container container
180s Warning FailedToUpdateEndpoint endpoints/data-mover-svc-f4kqg Failed to update endpoint kasten-io/data-mover-svc-f4kqg: Operation cannot be fulfilled on endpoints "data-mover-svc-f4kqg": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/kasten-io/data-mover-svc-f4kqg, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 39e1cda0-7b17-4ac9-98b1-da3efddb0c7c, UID in object meta:
190s Normal Scheduled pod/data-mover-svc-njqxz Successfully assigned kasten-io/data-mover-svc-njqxz to K8SNODE-cyo102
200s Normal Pulled pod/data-mover-svc-njqxz Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
210s Normal Created pod/data-mover-svc-njqxz Created container container
220s Normal Started pod/data-mover-svc-njqxz Started container container
230s Normal Started Action actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Executing action backup
240s Normal Started Phase actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Executing phase dumpToStore
250s Warning ActionSetFailed Action: backup actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test 84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
260s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test 84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
270s Normal Scheduled pod/backup-data-stats-swhj5 Successfully assigned kasten-io/backup-data-stats-swhj5 to K8SNODE-cyo102
280s Normal Pulled pod/backup-data-stats-swhj5 Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
290s Normal Created pod/backup-data-stats-swhj5 Created container container
300s Normal Started pod/backup-data-stats-swhj5 Started container container
310s Normal Killing pod/backup-data-stats-swhj5 Stopping container container
320s Normal Killing pod/data-mover-svc-njqxz Stopping container container

 

 


  • Author
  • Comes here often
  • 19 comments
  • May 11, 2022

Is there anything you could also need? Maybe someone could give me a hint, in which direction I have to search. 

 

Thanks a lot!


jaiganeshjk
Forum|alt.badge.img+2
  • Experienced User
  • 276 comments
  • Answer
  • May 31, 2022

@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?

This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?


  • Author
  • Comes here often
  • 19 comments
  • June 3, 2022

I will do this, but this will need a few days. I will respond here, hopefully with a solution. 


Comment