Solved

Backup-Policy with Blueprint for MariaDB Deployment fails


Userlevel 4

Hi!

I have problems with using a blueprint to backup a mariadb deployment in kubernetes. After running trough a series of error messages I made ot to the last that did not let me successful back up the workload with a blueprint: 

 

cause:
cause:
fields:
- name: message
value: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod failed
or did not transition into complete state: Pod kanister-job-t8pw4
failed. Pod details (&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
APPNAME-CUSTOMER-test 754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
2022-04-29 09:55:17 +0000 UTC <nil> <nil> map[createdBy:kanister]
map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
[] [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC FieldsV1
{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
} {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
} {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:ContainersNotReady,Message:containers with unready status:
[container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:ContainersNotReady,Message:containers with unready status:
[container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
09:55:17 +0000
UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
- name: actionSet
value:
metadata:
creationTimestamp: 2022-04-29T09:55:17Z
generateName: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-deployment-
generation: 4
labels:
kanister.io/JobID: 20f47d60-c7a2-11ec-9afe-ce64f5560a62
managedFields:
- apiVersion: cr.kanister.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:generateName: {}
f:labels:
.: {}
f:kanister.io/JobID: {}
f:spec:
.: {}
f:actions: {}
f:status:
.: {}
f:actions: {}
f:error:
.: {}
f:message: {}
f:state: {}
manager: Go-http-client
operation: Update
time: 2022-04-29T09:55:17Z
name: k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d655zh
namespace: kasten-io
resourceVersion: "42480091"
uid: fa5d502f-bd86-48bb-be43-7ad3e345caef
spec:
actions:
- blueprint: APPNAME-mariadb-blueprint
name: backup
object:
apiVersion: v1
group: apps
kind: deployment
name: mariadb
namespace: APPNAME-CUSTOMER-test
resource: deployments
options:
objectStorePath: repo/dbab9b79-db59-4905-b6bf-635c8699b479/
preferredVersion: v1.0.0-alpha
profile:
apiVersion: v1alpha1
group: ""
kind: profile
name: kanister-phase-kopia-profile-j8fjf
namespace: kasten-io
resource: ""
status:
actions:
- artifacts:
mariaBackup:
kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
blueprint: APPNAME-mariadb-blueprint
deferPhase:
name: ""
state: ""
name: backup
object:
apiVersion: v1
group: apps
kind: deployment
name: mariadb
namespace: APPNAME-CUSTOMER-test
resource: deployments
phases:
- name: dumpToStore
state: failed
error:
message: 'Failed while waiting for Pod kanister-job-t8pw4 to complete: Pod
failed or did not transition into complete state: Pod
kanister-job-t8pw4 failed. Pod details
(&Pod{ObjectMeta:{kanister-job-t8pw4 kanister-job-
APPNAME-CUSTOMER-test 754772e9-403e-447e-8c4a-d6a6f3dbe006 42480080 0
2022-04-29 09:55:17 +0000 UTC <nil> <nil>
map[createdBy:kanister]
map[cni.projectcalico.org/containerID:ab374f1fc96af0c13fd75443989d41e27533a3fc5cf85e8c4fee40dc8996a748
cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] []
[] [{Go-http-client Update v1 2022-04-29 09:55:17 +0000 UTC
FieldsV1
{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
} {calico Update v1 2022-04-29 09:55:18 +0000 UTC FieldsV1
{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}
} {kubelet Update v1 2022-04-29 09:55:19 +0000 UTC FieldsV1
{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.129\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
}]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-ghfdj,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-ghfdj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:alm-k8w-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:ContainersNotReady,Message:containers with unready
status:
[container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:ContainersNotReady,Message:containers with unready
status:
[container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01
00:00:00 +0000 UTC,LastTransitionTime:2022-04-29 09:55:17 +0000
UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.129,StartTime:2022-04-29
09:55:17 +0000
UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-04-29
09:55:18 +0000 UTC,FinishedAt:2022-04-29 09:55:19 +0000
UTC,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://43abea729be4588ce1021786ffb3c15163c33bbcde30bb5b2944c666eae3ff2c,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.129,},},EphemeralContainerStatuses:[]ContainerStatus{},},})'
state: failed
file: kasten.io/k10/kio/kanister/operation.go:114
function: kasten.io/k10/kio/kanister.(*Operation).Execute
linenumber: 114
message: ActionSet Failed
file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:676
function: kasten.io/k10/kio/exec/phases/backup.RunKanisterSnapshotActions
linenumber: 676
message: Failed to create backup kanister phase
message: Job failed to be executed
fields: []

 

 

 

 

 

Details of my Environment: 

Kubernetes: v1.21.5

K10: Upgraded today to 4.5.14

MariaDB in Deployment: 10.6 (This is not the StatefulSet from the Docs! Its just a Deployment-Type)

 

The Snapshot creation with the normal mode has no problems. Only my try to setup a blueprint for that mariadb database fails. 

 

 

If it helps: I add my blueprint here: 

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
name: mariadb-blueprint
actions:
backup:
outputArtifacts:
mariaBackup:
# Capture the kopia snapshot information for subsequent actions
# The information includes the kopia snapshot ID which is essential for restore and delete to succeed
# `kopiaOutput` is the name provided to kando using `--output-name` flag
kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
phases:
- func: KubeTask
name: dumpToStore
objects:
mariaSecret:
kind: Secret
name: '{{ .Deployment.Name }}'
namespace: '{{ .Deployment.Namespace }}'
args:
image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
namespace: "{{ .Deployment.Namespace }}"
command:
- bash
- -o
- errexit
- -o
- pipefail
- -c
- |
backup_file_path="dump.sql"
root_password="{{ index .Phases.dumpToStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
exclude_db="('information_schema', 'mysql', 'performance_schema')"
dump_databases=""
for db in `mysql -u root -h {{ .Deployment.Name }} --password=${root_password} -ANe "use information_schema; select schema_name from schemata where schema_name not in ${exclude_db}"`
do
dump_databases="${dump_databases} ${db}"
done
dump_cmd="mysqldump -u root --password=${root_password} -h {{ .Deployment.Name }} --column-statistics=0 --single-transaction --databases ${dump_databases}"
${dump_cmd} | kando location push --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --output-name "kopiaOutput" -
restore:
inputArtifactNames:
# The kopia snapshot info created in backup phase can be used here
# Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
- mariaBackup
phases:
- func: KubeTask
name: restoreFromStore
objects:
mariaSecret:
kind: Secret
name: '{{ .Deployment.Name }}'
namespace: '{{ .Deployment.Namespace }}'
args:
image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
namespace: "{{ .Deployment.Namespace }}"
command:
- bash
- -o
- errexit
- -o
- pipefail
- -c
- |
backup_file_path="dump.sql"
kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
root_password="{{ index .Phases.restoreFromStore.Secrets.mariaSecret.Data "MYSQL_ROOT_PASSWORD" | toString }}"
kando location pull --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}" - | mysql -u root --password=${root_password} -h {{ .Deployment.Name }}
delete:
inputArtifactNames:
# The kopia snapshot info created in backup phase can be used here
# Use the `--kopia-snapshot` flag in kando to pass in `mariaBackup.KopiaSnapshot`
- mariaBackup
phases:
- func: KubeTask
name: deleteFromStore
args:
image: ghcr.io/kanisterio/mysql-sidecar:0.78.0
namespace: "{{ .Namespace.Name }}"
command:
- bash
- -o
- errexit
- -o
- pipefail
- -c
- |
backup_file_path="dump.sql"
kopia_snap='{{ .ArtifactsIn.mariaBackup.KopiaSnapshot }}'
kando location delete --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}"

 

Can you guys tell me where I am doing something wrong? 

 

Thanks a lot and have a nice day!

icon

Best answer by jaiganeshjk 31 May 2022, 09:07

View original

5 comments

Userlevel 3
Badge +1

Hello SMT-Almato,

 

Could you please rerun the policy and within the first ten minutes provide the events for kasten-io namespace.

 

kubectl get events -n kasten-io

 

Thanks

Emmanuel

Userlevel 4

kubectl get events -n kasten-io

Hi Emmanuel, 

thank you for your reply. Of course: 

0s          Warning   ActionSetFailed Action: backup   actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dvrkd8   Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test  f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] []  [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-ff4gb to complete: Pod failed or did not transition into complete state: Pod kanister-job-ff4gb failed. Pod details (&Pod{ObjectMeta:{kanister-job-ff4gb kanister-job- APPNAME-CUSTOMER-test f7502bc3-9cd8-4b1f-bc49-59ec626f2cb9 45744997 0 2022-05-05 14:28:32 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:822705ebad818bd23bb3c9aa49a3f5f38154c56fae61b4cd3b5e428cc2463d87 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:28:32 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:28:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:28:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.160\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-698qz,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-698qz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:35 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:28:32 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.160,StartTime:2022-05-05 14:28:32 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:28:33 +0000 UTC,FinishedAt:2022-05-05 14:28:34 +0000 UTC,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://407510dd106fa29965050a0e9db6b4f681120e6d3e71fe8800ec3a7289570f23,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.160,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Normal Killing pod/data-mover-svc-ctc2g Stopping container container
0s Normal Scheduled pod/data-mover-svc-l2mps Successfully assigned kasten-io/data-mover-svc-l2mps to K8SNODE-cyo102
0s Normal Pulled pod/data-mover-svc-l2mps Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s Normal Created pod/data-mover-svc-l2mps Created container container
0s Normal Started pod/data-mover-svc-l2mps Started container container
0s Normal Started Action actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Executing action backup
0s Normal Started Phase actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Executing phase dumpToStore
0s Warning ActionSetFailed Action: backup actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-dm5nfl Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-2j8wm to complete: Pod failed or did not transition into complete state: Pod kanister-job-2j8wm failed. Pod details (&Pod{ObjectMeta:{kanister-job-2j8wm kanister-job- APPNAME-CUSTOMER-test d56ca63a-bc14-4d05-8a73-90f71541eb47 45745375 0 2022-05-05 14:29:24 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:25b39245e1534a0839bb195a1c6af13c9d57447cb4d5b80856f766c04d0d39ad cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:29:24 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:29:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:29:26 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7hkk8,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7hkk8,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:29:24 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.172,StartTime:2022-05-05 14:29:24 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:29:25 +0000 UTC,FinishedAt:2022-05-05 14:29:26 +0000 UTC,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://95bb92103c01ff9aab4265d816a08194e670dbd8925ca749de2a4b7fe4c720b3,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.172,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Normal Scheduled pod/backup-data-stats-wr98h Successfully assigned kasten-io/backup-data-stats-wr98h to K8SNODE-cyo102
0s Normal Pulled pod/backup-data-stats-wr98h Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s Normal Created pod/backup-data-stats-wr98h Created container container
0s Normal Started pod/backup-data-stats-wr98h Started container container
0s Normal Killing pod/backup-data-stats-wr98h Stopping container container
0s Normal Killing pod/data-mover-svc-l2mps Stopping container container
0s Warning FailedToUpdateEndpoint endpoints/data-mover-svc-f4kqg Failed to update endpoint kasten-io/data-mover-svc-f4kqg: Operation cannot be fulfilled on endpoints "data-mover-svc-f4kqg": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/kasten-io/data-mover-svc-f4kqg, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 39e1cda0-7b17-4ac9-98b1-da3efddb0c7c, UID in object meta:
0s Normal Scheduled pod/data-mover-svc-njqxz Successfully assigned kasten-io/data-mover-svc-njqxz to K8SNODE-cyo102
0s Normal Pulled pod/data-mover-svc-njqxz Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s Normal Created pod/data-mover-svc-njqxz Created container container
0s Normal Started pod/data-mover-svc-njqxz Started container container
0s Normal Started Action actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Executing action backup
0s Normal Started Phase actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Executing phase dumpToStore
0s Warning ActionSetFailed Action: backup actionset/k10-backup-APPNAME-mariadb-blueprint-mariadb-APPNAME-CUSTOMER-test-d47pcf Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test 84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Warning ActionSetFailed Action: backup blueprint/APPNAME-mariadb-blueprint Failed to execute phase: v1alpha1.Phase{Name:"dumpToStore", State:"pending", Output:map[string]interface {}(nil)}: Failed while waiting for Pod kanister-job-zq7g4 to complete: Pod failed or did not transition into complete state: Pod kanister-job-zq7g4 failed. Pod details (&Pod{ObjectMeta:{kanister-job-zq7g4 kanister-job- APPNAME-CUSTOMER-test 84e2db00-a97d-42b9-8a98-428402630437 45745806 0 2022-05-05 14:30:20 +0000 UTC <nil> <nil> map[createdBy:kanister] map[cni.projectcalico.org/containerID:5ab201690b94e7ed6c7d6dfb08a1198218570f677a202aeb99c0dc6b95cb1e39 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs:] [] [] [{Go-http-client Update v1 2022-05-05 14:30:20 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:createdBy":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"container\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}} } {calico Update v1 2022-05-05 14:30:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}} } {kubelet Update v1 2022-05-05 14:30:22 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.192.154\"}":{".":{},"f:ip":{}}},"f:startTime":{}}} }]},Spec:PodSpec{Volumes:[]Volume{Volume{Name:kube-api-access-7jgmr,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:nil,NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,},StorageOS:nil,CSI:nil,Ephemeral:nil,},},},Containers:[]Container{Container{Name:container,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,Command:[<*****>],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7jgmr,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:K8SNODE-cyo102,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:*PreemptLowerPriority,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},SetHostnameAsFQDN:nil,OS:nil,},Status:PodStatus{Phase:Failed,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:ContainersReady,Status:False,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:ContainersNotReady,Message:containers with unready status: [container],},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2022-05-05 14:30:20 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.23.19.14,PodIP:10.244.192.154,StartTime:2022-05-05 14:30:20 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:container,State:ContainerState{Waiting:nil,Running:nil,Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-05-05 14:30:21 +0000 UTC,FinishedAt:2022-05-05 14:30:22 +0000 UTC,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,},},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:false,RestartCount:0,Image:ghcr.io/kanisterio/mysql-sidecar:0.78.0,ImageID:ghcr.io/kanisterio/mysql-sidecar@sha256:73d5cbb5d53d35cb4213521d5b32398ca20fd0b71a8635dcd99e59b38deb2858,ContainerID:containerd://d0d9790fb5709ac639f68399f48d9fd006161648510b8b9d6718120e220633e7,Started:*false,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.244.192.154,},},EphemeralContainerStatuses:[]ContainerStatus{},},})
0s Normal Scheduled pod/backup-data-stats-swhj5 Successfully assigned kasten-io/backup-data-stats-swhj5 to K8SNODE-cyo102
0s Normal Pulled pod/backup-data-stats-swhj5 Container image "ghcr.io/kanisterio/kanister-tools:0.78.0" already present on machine
0s Normal Created pod/backup-data-stats-swhj5 Created container container
0s Normal Started pod/backup-data-stats-swhj5 Started container container
0s Normal Killing pod/backup-data-stats-swhj5 Stopping container container
0s Normal Killing pod/data-mover-svc-njqxz Stopping container container

 

 

Userlevel 4

Is there anything you could also need? Maybe someone could give me a hint, in which direction I have to search. 

 

Thanks a lot!

Userlevel 6
Badge +2

@SMT-Almato Would you be able to re-run the backup(since this is old and probably we might not have relevant logs anymore) and generate a debug bundle ?

This needs further troubleshooting. Would you be able to create a case with us by selecting Kasten By Veeam K10 Trial under the select products drop-down including the debug logs ?

Userlevel 4

I will do this, but this will need a few days. I will respond here, hopefully with a solution. 

Comment