Logical MySQL restore fail

@jaiganeshjk 

@jaiganeshjk  thanks for your reply.

I already upgrade the kasten k10 from 5.0.11 to 5.5.6 use the commands “helm upgrade k10 /root/k10/k10-5.5.6.tgz --namespace=kasten-io -f /root/mjw/k10/test/k10_val.yaml”. 

but there are two pods fail like below screenshot.

detail error info for pod catalog-svc-5c7c6764f7-ntswr as below:

detail error info for pod metering-svc-667fb78dc4-k72cd as below:

yes, I know this reason is the volume pvc-14f8ae51-7f71-4966-871a-f38599165cd7 and pvc-c8849df6-25ea-45c1-8964-668ae7b5893b are deleted, because I changed the backend nas-130 to nas-131 and removed them manually。I dont know that k10 won’t create them automatically after upgrade or reboot. now how can i recover them? could you guide me? thank you very much.

Strangely，I do the same operates for pods jobs-svc-848d69b956-fmg7g and logging-svc-68d94d9b45-xm7dk, the two pods can recreate the volumes automatically after upgrade the k10, I don’t know why catalog-svc-5c7c6764f7-ntswr and metering-svc-667fb78dc4-k72cd can not?

PVC recreate alutomatically after upgrade k10.

This is usually a problem with the storage backends which exposes client access to snapshot directory.

K10 tries to run chown recursively and fails since `.snapshot` directory is read-only

May I know what NAS backend you are using in this case ?

Is there a way you can disable client access snapshot directory for this volume ?

Recent versions probably has improvements in how we handle the phonehome reports 

@jaiganeshjk Update again and ignore that problem above。I installed k10 version 5.5.8 successfully。

But the mysql data is still not restored. Is there something wrong with any of my steps？below are detail steps。

1. deploy statefulset and all pods’s state are running. 

2. create a new database named test.

3. backup the namespace k10-mysql in kasten k10

4. restore to a new namespace names restore-2.

5. the application restore-2 restored successfully.

6. But the database name test create before is not restored?

The logical blueprint that you are using is an example created for mysql Standalone and not HA.

I don’t know how the data is synced between the replicas. You will have to keep it in mind how the HA works before backing it up.

Snapshot field computes the capacity of the local snapshots created by the CSI driver or the provider.

However, your backup was blueprint based and it is pushed to your location target. So it comes under object storage and not Snapshots.

 @jaiganeshjk I verifed that the backup and restore did not use injectKanisterSidecar just use snapshot. but the database data is still not restored.

Steps as below:

1. Deplpoy mysql application and login in mysql.

2. create database test and insert three records.

3. Create policy on K10 dashboard and backup the namespace deploy-mysql and the result is fine.

During the buckup I can see the snapshot is created.

4.  Then restore to the new namespace named restore-11 using the restore point of the backup.

Similarly, I can also see the restored snapshot when restoring.

5. Check the pod status in namespace restore-11 and login in the mysql to check the database test. The result is the database test have not restored?

Could you help me with the problem？thank you very much.

my k8s is openshift  4.10.0-0.okd-2022-07-09-073606

Kubernetes Version is v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty

@Hagag thanks for your response.

Both statefulset and deployment I have tested，but got the same result.

I disabled the injection of the Kanister sidecar by paramter “--set injectKanisterSidecar.enabled=true”. 

the yaml of statefulset and deployment as below:

apiVersion: v1
kind: Service
metadata:
  name: mysql-bk
  namespace: mysql-test
  labels:
    app: mysql
spec:
  type: NodePort
  selector:
    app: mysql
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-bk-sfs
  namespace: mysql-test
spec:
  selector:
    matchLabels:
      app: mysql # 必须匹配 .spec.template.metadata.labels
  serviceName: mysql-bk
  replicas: 3 # 默认值是 1
  minReadySeconds: 10 # 默认值是 0
  template:
    metadata:
      labels:
        app: mysql # 必须匹配 .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mysql
        image: registry.example.com:8443/mysql/mysql:8.0.26
        imagePullPolicy: IfNotPresent
        env:
          - name: MYSQL_ROOT_PASSWORD
            value: "123456"
        ports:
          - containerPort: 3306
            name: mysql-svc
        volumeMounts:
        - name: mysql-bk-pvc
          mountPath: /root/mjw
  volumeClaimTemplates:
  - metadata:
      name: mysql-bk-pvc
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: mysc
      resources:
        requests:
          storage: 100Gi 

============below is the yaml of deployment=================

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: deploy-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: registry.example.com:8443/mysql/mysql:5.7
          imagePullPolicy: IfNotPresent
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "123456"
          ports:
            - containerPort: 3306
          volumeMounts:
            - mountPath: /root/mjw
              name: mysql-pvc-test
      volumes:
        - name: mysql-pvc-test
          persistentVolumeClaim:
            claimName: mysql-pvc-test

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc-test
  namespace: deploy-mysql
spec:
  accessModes:
    - ReadWriteOnce
  # 替换成你的StorageClass
  storageClassName: mysc
  resources:
    requests:
      storage: 50Gi
---

apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
  namespace: deploy-mysql
  labels:
    app: mysql
spec:
  type: NodePort
  ports:
    - port: 3306
      protocol: TCP
      name: http
      nodePort: 32356
  selector:
    app: mysql

============below is the k10 value file content==============

global:
  airgapped:
    repository: registry.example.com:8443/kasten-images
  persistence:
    storageClass: mysc
injectKanisterSidecar:
  enabled: true
metering:
  mode: airgap

@Hagag Thanks for you guidance。But the database data is still not restored. Please help me to fix the problem.

First the Arifacts and CRDs are cleaned in my env.

Second, below steps are following the logical MySQL backup guide。

1. Edit the blueprint file which get from the link .

https://raw.githubusercontent.com/kanisterio/kanister/0.90.0/examples/mysql-deploymentconfig/blueprint-v2/mysql-dep-config-blueprint.yaml

2. create the blueprint into namespace kasten-io

  3. annotate the statefullset with the below annotations to instruct K10 to use this Blueprint.

kubectl --namespace mysql-test annotate statefulset/mysql-bk-sfs kanister.kasten.io/blueprint=mysql-dep-config-blueprint

and we can see the blueprint annotated in statfulset yaml.

4. Insert three records in database table like below.

5. Begain to backup the application on k10 dashbord.

6. remove the databases test as a disaster。

7. restore data using the backup point before.

8. The mysql application is restored.

8. check mysql database, but the database test is not restored.

below is my blueprint yaml file.

apiVersion: cr.kanister.io/v1alpha1
kind: Blueprint
metadata:
  name: mysql-dep-config-blueprint
  namespace: kasten-io
actions:
  backup:
    outputArtifacts:
      mysqlBackup:
        # Capture the kopia snapshot information for subsequent actions
        # The information includes the kopia snapshot ID which is essential for restore and delete to succeed
        # `kopiaOutput` is the name provided to kando using `--output-name` flag
        kopiaSnapshot: "{{ .Phases.dumpToStore.Output.kopiaOutput }}"
    phases:
      - func: KubeTask
        name: dumpToStore
        objects:
          mysqlsecret:
            kind: Secret
            #name: "{{ .DeploymentConfig.Name }}"
            name: mysql-password
            #namespace: "{{ .DeploymentConfig.Namespace }}"
            namespace: "{{ .StatefulSet.Namespace }}"
        args:
          #image: ghcr.io/kanisterio/mysql-sidecar:0.83.0
          image: registry.example.com:8443/kasten-images/mysql-sidecar:0.90.0
          namespace: "{{ .StatefulSet.Namespace }}"
          command:
            - bash
            - -o
            - errexit
            - -o
            - pipefail
            - -c
            - |
              backup_file_path="dump.sql"
              root_password="{{ index .Phases.dumpToStore.Secrets.mysqlsecret.Data "password" | toString }}"
              dump_cmd="mysqldump --column-statistics=0 -u root --password=${root_password} -h mysql-bk --single-transaction --all-databases"
              ${dump_cmd} | kando location push --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --output-name "kopiaOutput" -
  restore:
    inputArtifactNames:
      # The kopia snapshot info created in backup phase can be used here
      # Use the `--kopia-snapshot` flag in kando to pass in `mysqlBackup.KopiaSnapshot`
      - mysqlBackup
    phases:
      - func: KubeTask
        name: restoreFromStore
        objects:
          mysqlsecret:
            kind: Secret
            name: mysql-password
            namespace: "{{ .StatefulSet.Namespace }}"
        args:
          image: registry.example.com:8443/kasten-images/mysql-sidecar:0.90.0
          namespace: "{{ .StatefulSet.Namespace }}"
          command:
            - bash
            - -o
            - errexit
            - -o
            - pipefail
            - -c
            - |
              backup_file_path="dump.sql"
              kopia_snap='{{ .ArtifactsIn.mysqlBackup.KopiaSnapshot }}'
              root_password="{{ index .Phases.restoreFromStore.Secrets.mysqlsecret.Data "password" | toString }}"
              restore_cmd="mysql -u root --password=${root_password} -h mysql-bk"
              kando location pull --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}" - | ${restore_cmd}
  delete:
    inputArtifactNames:
      # The kopia snapshot info created in backup phase can be used here
      # Use the `--kopia-snapshot` flag in kando to pass in `mysqlBackup.KopiaSnapshot`
      - mysqlBackup
    phases:
      - func: KubeTask
        name: deleteFromStore
        args:
          image: registry.example.com:8443/kasten-images/mysql-sidecar:0.90.0
          namespace: "{{ .StatefulSet.Namespace }}"
          command:
            - bash
            - -o
            - errexit
            - -o
            - pipefail
            - -c
            - |
              backup_file_path="dump.sql"
              kopia_snap='{{ .ArtifactsIn.mysqlBackup.KopiaSnapshot }}'
              kando location delete --profile '{{ toJson .Profile }}' --path "${backup_file_path}" --kopia-snapshot "${kopia_snap}"
 

@meijianwei I have successfully reproduced the issue. It appears to occur when using replicas in your workload. I will need some time to investigate this behavior and provide you with further information. In the meantime, a possible workaround is to remove the annotation from your workload, such as StatefulSets ( to disable blueprint usage ) , and perform a normal backup

kubectl --namespace mysql-test annotate statefulset/mysql-bk-sfs kanister.kasten.io/blueprint=mysql-dep-config-blueprint-

P.S The blueprint will work with one replica
 

@Hagag thanks very much. By the way, how can I manage the database synchronization between the endpoints? Could you give me a guide? 

My current situation is that the test database cannot be restored no matter whether use blueprint or not.