@jaiganeshjk 

Hello @widura it seems, the catalog deployment is already scaled down and the restore pod is trying to scale it down too, and eventually, you got the error “unable to scale down...)

plz share the output of the below command:

​​​​​​​kubectl get deployments catalog-svc -n kasten-io

and please clarify if you trying to restore on the same cluster.
If you are reinstalling K10 on the same cluster, cleaning up the namespace in which K10 was previously installed before the above passphrase creation is important.

and K10 must be reinstalled before recovery.
 

Ahmed Hagag

Hello,

thanks for all the input so far and sorry for the late reply! I was fully booked yesterday.

I tried to reproduce this now with a fresh k10 installation, with nothing but the DR enabled. I deleted k10 completely, including the namespace, sticked with the identical cluster to redeploy k10, deploy the DR passphrase as secret, create a location profile manually using the “Authenticate With AWS IAM Role” option and deploy the k10restore-job as helm chart with sourceClusterID, profile.name and secrets.awsIamRole set.

I receive this error message:

{"File":"kasten.io/k10/kio/tools/restorectl/root.go","Function":"kasten.io/k10/kio/tools/restorectl.Execute","Line":24,"cluster_name":"7fe66dd6-231b-48e4-834a-87a128b2d777","error":{"message":"Failed to restore DR snapshot","function":"kasten.io/k10/kio/tools/restorectl.restoreK10","linenumber":160,"file":"kasten.io/k10/kio/tools/restorectl/restore.go:160","cause":{"message":"Failed to restore K10 DR backup","function":"kasten.io/k10/kio/kanister/function.RestoreDataDR","linenumber":135,"file":"kasten.io/k10/kio/kanister/function/kio_restore_data_dr.go:135","cause":{"message":"Failed to generate repository connect command","function":"kasten.io/k10/kio/kopia.ConnectToKopiaRepository","linenumber":536,"file":"kasten.io/k10/kio/kopia/repository.go:536","cause":{"message":"Failed to generate blob store args","function":"kasten.io/k10/kio/kopia.repositoryConnectCommand","linenumber":358,"file":"kasten.io/k10/kio/kopia/repository.go:358","cause":{"message":"Failed to get AWS credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 337285c9-28e5-4a90-bfc7-daaece35c61f"}}}}},"hostname":"k10-restore-k10restore-bgcft","level":"error","msg":"Failed","time":"20230301-08:40:45.172Z"}
Stream closed EOF for kasten-io/k10-restore-k10restore-bgcft (k10restore)

I noticed that the secret used by the location is empty. I tried to copy the content from the one secret containing the IRSA role ARN to the empty secret but without a result.  

I realized that re-running the k10restore job with this issue, results in the catalog-svc deployment being set to “0” but not to “1” again, resulting in the issue described in my initial post.

Though, I’m now stuck with this authentication issue. I double-checked the Service Account of the k10restore-job but this is correctly annotated with the IRSA role ARN. I’m using the same role for the restore as for creating the DR backup, so that is also not the issue.

Any input from your end?

All the best,

Widura

I have managed to sort it out. I was missing to reference the correct serviceaccount in the k10restore helm chart. By default, a new service account is created, which did not match the IRSA policy. Referencing the SA name from the k10 deployment solved the authentication issue. I’ve added theses two lines to the list of values in the k10restore helm chart:

• serviceAccount.create = false

• serviceAccount.name = <irsa_service_account_name>

Thanks again to all for your efforts!

Widura