• EN
Solved

ambassador:2.2.2 not avilable

L
Userlevel 2

Hello,

I’ve upgraded from 4.5.10 to 4.5.13 this week, since then my backups are failing.

Looks like there is a new ambassador version since 4.5.12, which can’t be pulled (Back-off pulling image "quay.io/datawire/ambassador:2.2.2")

Can’t find that image version on the quay.io registry.

 

Is this a known issue?

 

icon

Best answer by jaiganeshjk 21 April 2022, 07:57

View original

6 comments

jaiganeshjk
Userlevel 6
Badge +2

@lemassacre Thank you for posting this question.

K10 has started using Emissary . There is a change in the image registry for emissary and it is now docker.io/emissaryingress/emissary:2.2.2

You might probably be using older values.yaml file.

Please have a look at this published KB article and let us know if it helps

Jaiganesh
L
Userlevel 2

Thanks for the quick reply, this solved the issue with that pod not starting. Now it’s running again!

Unfortunatelly it doesn’t solve the problems that i have with volume snapshots since the upgrade:
cause:
  cause:
    cause:
      cause:
        cause:
          message: context deadline exceeded
        file: kasten.io/k10/kio/poll/poll.go:95
        function: kasten.io/k10/kio/poll.waitWithBackoffWithRetriesHelper
        linenumber: 95
        message: Context done while polling
      fields:
        - name: duration
          value: 44m59.963318481s
      file: kasten.io/k10/kio/poll/poll.go:65
      function: kasten.io/k10/kio/poll.waitWithBackoffWithRetries
      linenumber: 65
      message: Timeout while polling
    fields:
      - name: actionSet
        value: k10-backuptoserver-k10-deployment-generic-volume-2.0.17-piznnbl
    file: kasten.io/k10/kio/kanister/operation.go:284
    function: kasten.io/k10/kio/kanister.(*Operation).waitForActionSetCompletion
    linenumber: 284
    message: Error waiting for ActionSet
  fields:
    - name: appName
      value: app
    - name: appType
      value: deployment
    - name: namespace
      value: namespace
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:612
  function: kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot
  linenumber: 612
  message: Failed to snapshot volumes
message: Job failed to be executed
fields: []

 

 

jaiganeshjk
Userlevel 6
Badge +2

Thanks for confirming that your issue with the gateway service is resolved.

kanister.backupTimeoutI would suggest you to mark the comment with the KB article as the answer for this question.

 

Unfortunatelly it doesn’t solve the problems that i have with volume snapshots since the upgrade:
cause:
  cause:
    cause:
      cause:
        cause:
          message: context deadline exceeded
        file: kasten.io/k10/kio/poll/poll.go:95
        function: kasten.io/k10/kio/poll.waitWithBackoffWithRetriesHelper
        linenumber: 95
        message: Context done while polling
      fields:
        - name: duration
          value: 44m5kanister.backupTimeout9.963318481s
      file: kasten.io/k10/kio/poll/poll.go:65
      function: kasten.io/k10/kio/poll.waitWithBackoffWithRetries
      linenumber: 65
      message: Timeout while polling
    fields:
      - name: actionSet
        value: k10-backuptoserver-k10-deployment-generic-volume-2.0.17-piznnbl
    file: kasten.io/k10/kio/kanister/operation.go:284
    function: kasten.io/k10/kio/kanister.(*Operation).waitForActionSetCompletion
    linenumber: 284
    message: Error waiting for ActionSet
  fields:
    - name: appName
      value: app
    - name: appType
      value: deployment
    - name: namespace
      value: namespace
  file: kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:612
  function: kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot
  linenumber: 612
  message: Failed to snapshot volumes
message: Job failed to be executed
fields: []

 

As for the above issue, I assume that you are using GVS to backup you application.

The error message means that it is timing out after the default timeout of 45 mins. Probably the copy operations were not completed within the 45 min timeframe to your NFS.

You can increase this time out by upgrading k10 with the value kanister.backupTimeout set to the desired duration.

Full list of the supported helm values can be found here.

Jaiganesh
L
Userlevel 2

It’s just a few MB that are copied on that snapshot, only took a few seconds before the upgrade.

Are there any logs that i can investiagte?

jaiganeshjk
Userlevel 6
Badge +2

It’s just a few MB that are copied on that snapshot, only took a few seconds before the upgrade.

Are there any logs that i can investiagte?

This shouldn’t take that much time for few MB. You can look at the kanister-svc logs to see if there are any clues.

If not, You can raise a case with us by signing up to veeam portal https://my.veeam.com/ with the debug logs.

Jaiganesh
L
Userlevel 2

This shouldn’t take that much time for few MB. You can look at the kanister-svc logs to see if there are any clues.

 

Ok, I found something in that logs that could explain the problem, but I have no clue how to solve that:
 

{

    "Container": "kanister-sidecar",

    "File": "pkg/format/format.go",

    "Function": "github.com/kanisterio/kanister/pkg/format.LogWithCtx",

    "Line": 61,

    "Out": "kopia: error: unable to list sources: 401 Unauthorized, try --help",

    "Pod": "Pod",

    "cluster_name": "cluster_name",

    "hostname": "kanister-svc-5fc6fbf99d-9cdwh",

    "level": "info",

    "msg": "Pod Update",

    "time": "2022-04-21T10:58:58.534152852Z",

    "version": "4.5.13"

}

 

Comment


Terms & Conditions