kasten 7.0.8 failed to pull gcr.io/kasten-images/admin:7.0.8

Hi,

I have upgrade k10 this morning and i have this error on one container:

Failed to pull image "gcr.io/kasten-images/admin:7.0.8": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kasten-images/admin:7.0.8": failed to resolve reference "gcr.io/kasten-images/admin:7.0.8": gcr.io/kasten-images/admin:7.0.8: not found

If i check the repo, this image is missing :

Page 1 / 1

Hi @Vecteur IT

This service/image is no longer in use, so you shouldn't encounter this error during the upgrade. Which version of K10 are you upgrading from?

Thanks,
Ahmed Hagag

Hi,

I also encountered the same issue. Just upgraded from 7.0.6 to 7.0.7 and then 7.0.8… and then back to 7.0.7

I tried 7.0.8 and 7.0.9 also, but still no success.

Cheers,

Andrei

Hi @gavu
This image is a part of state-svc deployment, please scale down this deployment, delete it and upgrade again.
this should be enough.

Thanks
Ahmed Hagag

Hi Ahmed,

thanks for the fast reply! unfortunately it did not work. Here’s what I did:

disconnected the app from argocd
just for testing: ran “helm upgrade --install kasten kasten/k10 --namespace=kasten-io --values values.yaml --version 7.0.7” to make sure everything still works
scaled down state-svc
deleted state-svc
ran “helm upgrade --install kasten kasten/k10 --namespace=kasten-io --values values.yaml --version 7.0.8”

state-svc is back, and failing.

Cheers,

Andrei

Hi @gavu so you mean the upgrade is done without this image error any more but now the state-svc pod is fialing?if so please describe the pod and share the output

kubectl describe pod -l component=state -n kasten-io

Thanks
Ahmed Hagag

Hi @Hagag,

here’s the requested output after running “helm upgrade --install kasten kasten/k10 --namespace=kasten-io --values values.yaml --version 7.0.8”

Name:             state-svc-7586bb89c9-kw62c
Namespace:        kasten-io
Priority:         0
Service Account:  kasten-k10
Node:             hive02/10.0.0.32
Start Time:       Mon, 30 Sep 2024 16:19:48 +0200
Labels:           app=k10
                  app.kubernetes.io/instance=kasten
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=k10
                  component=state
                  helm.sh/chart=k10-7.0.8
                  heritage=Helm
                  pod-template-hash=7586bb89c9
                  release=kasten
                  run=state-svc
Annotations:      checksum/config: a8572616465aad6f1e4baa4d866bc129e6e07c182ca71639dfc89769660cdb56
                  checksum/frontend-nginx-config: b2f6e11d3f3eeaf8abf7eb87ccfbe356ca671ece18d258cf7686be8e02f50199
                  checksum/secret: 4e2c403c91eabfefa5ea85a97a3b709ef42b6c6d35ac5cf96bd3054e7e18b563
Status:           Pending
SeccompProfile:   RuntimeDefault
IP:               10.42.2.133
IPs:
  IP:           10.42.2.133
Controlled By:  ReplicaSet/state-svc-7586bb89c9
Containers:
  state-svc:
    Container ID:   containerd://a824d3962999a514da79a943c0867218a4ad18ccb7be92915edfb3be3f06f9a5
    Image:          gcr.io/kasten-images/state:7.0.8
    Image ID:       gcr.io/kasten-images/state@sha256:575141d899a75117e2cab1b08a2f579cb44ba3de7ae7438a3364bf1140cf3d8d
    Port:           8000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 30 Sep 2024 16:19:49 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      2m
      memory:   30Mi
    Liveness:   http-get http://:8000/v0/healthz delay=300s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8000/v0/healthz delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      VERSION:                                        <set to the key 'version' of config map 'k10-config'>  Optional: false
      K10_CAPABILITIES:                               mc
      K10_HOST_SVC:                                   state
      MODEL_STORE_DIR:                                /tmp/k10store
      LOG_LEVEL:                                      <set to the key 'loglevel' of config map 'k10-config'>  Optional: false
      POD_NAMESPACE:                                  kasten-io (v1:metadata.namespace)
      CONCURRENT_SNAP_CONVERSIONS:                    <set to the key 'concurrentSnapConversions' of config map 'k10-config'>               Optional: false
      CONCURRENT_WORKLOAD_SNAPSHOTS:                  <set to the key 'concurrentWorkloadSnapshots' of config map 'k10-config'>             Optional: false
      K10_DATA_STORE_PARALLEL_UPLOAD:                 <set to the key 'k10DataStoreParallelUpload' of config map 'k10-config'>              Optional: false
      K10_DATA_STORE_PARALLEL_DOWNLOAD:               <set to the key 'k10DataStoreParallelDownload' of config map 'k10-config'>            Optional: false
      K10_DATA_STORE_GENERAL_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreGeneralContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_GENERAL_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreGeneralMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_DATA_STORE_RESTORE_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreRestoreContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_RESTORE_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreRestoreMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_LIMITER_GENERIC_VOLUME_SNAPSHOTS:           <set to the key 'K10LimiterGenericVolumeSnapshots' of config map 'k10-config'>        Optional: false
      K10_LIMITER_GENERIC_VOLUME_COPIES:              <set to the key 'K10LimiterGenericVolumeCopies' of config map 'k10-config'>           Optional: false
      K10_LIMITER_GENERIC_VOLUME_RESTORES:            <set to the key 'K10LimiterGenericVolumeRestores' of config map 'k10-config'>         Optional: false
      K10_LIMITER_CSI_SNAPSHOTS:                      <set to the key 'K10LimiterCsiSnapshots' of config map 'k10-config'>                  Optional: false
      K10_LIMITER_PROVIDER_SNAPSHOTS:                 <set to the key 'K10LimiterProviderSnapshots' of config map 'k10-config'>             Optional: false
      K10_LIMITER_IMAGE_COPIES:                       <set to the key 'K10LimiterImageCopies' of config map 'k10-config'>                   Optional: false
      K10_EPHEMERAL_PVC_OVERHEAD:                     <set to the key 'K10EphemeralPVCOverhead' of config map 'k10-config'>                 Optional: false
      K10_PERSISTENCE_STORAGE_CLASS:                  <set to the key 'K10PersistenceStorageClass' of config map 'k10-config'>              Optional: false
      AWS_ASSUME_ROLE_DURATION:                       <set to the key 'AWSAssumeRoleDuration' of config map 'k10-config'>                   Optional: false
      KANISTER_TOOLS:                                 <set to the key 'KanisterToolsImage' of config map 'k10-config'>                      Optional: false
      K10_RELEASE_NAME:                               kasten
      KANISTER_FUNCTION_VERSION:                      <set to the key 'kanisterFunctionVersion' of config map 'k10-config'>  Optional: false
    Mounts:
      /mnt/k10-features from k10-features (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jvsw8 (ro)
  admin-svc:
    Container ID:
    Image:         gcr.io/kasten-images/admin:7.0.8
    Image ID:
    Port:          8001/TCP
    Host Port:     0/TCP
    Args:
      --port=8001
      --host=0.0.0.0
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      2m
      memory:   160Mi
    Liveness:   http-get http://:8001/v0/healthz delay=300s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8001/v0/healthz delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      VERSION:                                        <set to the key 'version' of config map 'k10-config'>  Optional: false
      K10_CAPABILITIES:                               mc
      K10_HOST_SVC:                                   state
      MODEL_STORE_DIR:                                <set to the key 'modelstoredirname' of config map 'k10-config'>  Optional: false
      LOG_LEVEL:                                      <set to the key 'loglevel' of config map 'k10-config'>           Optional: false
      POD_NAMESPACE:                                  kasten-io (v1:metadata.namespace)
      CONCURRENT_SNAP_CONVERSIONS:                    <set to the key 'concurrentSnapConversions' of config map 'k10-config'>               Optional: false
      CONCURRENT_WORKLOAD_SNAPSHOTS:                  <set to the key 'concurrentWorkloadSnapshots' of config map 'k10-config'>             Optional: false
      K10_DATA_STORE_PARALLEL_UPLOAD:                 <set to the key 'k10DataStoreParallelUpload' of config map 'k10-config'>              Optional: false
      K10_DATA_STORE_PARALLEL_DOWNLOAD:               <set to the key 'k10DataStoreParallelDownload' of config map 'k10-config'>            Optional: false
      K10_DATA_STORE_GENERAL_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreGeneralContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_GENERAL_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreGeneralMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_DATA_STORE_RESTORE_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreRestoreContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_RESTORE_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreRestoreMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_LIMITER_GENERIC_VOLUME_SNAPSHOTS:           <set to the key 'K10LimiterGenericVolumeSnapshots' of config map 'k10-config'>        Optional: false
      K10_LIMITER_GENERIC_VOLUME_COPIES:              <set to the key 'K10LimiterGenericVolumeCopies' of config map 'k10-config'>           Optional: false
      K10_LIMITER_GENERIC_VOLUME_RESTORES:            <set to the key 'K10LimiterGenericVolumeRestores' of config map 'k10-config'>         Optional: false
      K10_LIMITER_CSI_SNAPSHOTS:                      <set to the key 'K10LimiterCsiSnapshots' of config map 'k10-config'>                  Optional: false
      K10_LIMITER_PROVIDER_SNAPSHOTS:                 <set to the key 'K10LimiterProviderSnapshots' of config map 'k10-config'>             Optional: false
      K10_LIMITER_IMAGE_COPIES:                       <set to the key 'K10LimiterImageCopies' of config map 'k10-config'>                   Optional: false
      K10_EPHEMERAL_PVC_OVERHEAD:                     <set to the key 'K10EphemeralPVCOverhead' of config map 'k10-config'>                 Optional: false
      K10_PERSISTENCE_STORAGE_CLASS:                  <set to the key 'K10PersistenceStorageClass' of config map 'k10-config'>              Optional: false
      AWS_ASSUME_ROLE_DURATION:                       <set to the key 'AWSAssumeRoleDuration' of config map 'k10-config'>                   Optional: false
      KANISTER_TOOLS:                                 <set to the key 'KanisterToolsImage' of config map 'k10-config'>                      Optional: false
      K10_RELEASE_NAME:                               kasten
      KANISTER_FUNCTION_VERSION:                      <set to the key 'kanisterFunctionVersion' of config map 'k10-config'>  Optional: false
    Mounts:
      /mnt/k10-features from k10-features (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jvsw8 (ro)
  events-svc:
    Container ID:  containerd://b6e5c595a6a6a01308946e9499fb7c0109ef793b2b826747efed04048561991e
    Image:         gcr.io/kasten-images/events:7.0.8
    Image ID:      gcr.io/kasten-images/events@sha256:b12322e6b9c0b2660c6dfa825fd8f1af236696cedf01b1c48bc43cac4a5d9da2
    Port:          8002/TCP
    Host Port:     0/TCP
    Args:
      --port=8002
      --host=0.0.0.0
    State:          Running
      Started:      Mon, 30 Sep 2024 16:19:50 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8002/v0/healthz delay=300s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8002/v0/healthz delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      VERSION:                                        <set to the key 'version' of config map 'k10-config'>  Optional: false
      K10_CAPABILITIES:                               mc
      K10_HOST_SVC:                                   state
      MODEL_STORE_DIR:                                <set to the key 'modelstoredirname' of config map 'k10-config'>  Optional: false
      LOG_LEVEL:                                      <set to the key 'loglevel' of config map 'k10-config'>           Optional: false
      POD_NAMESPACE:                                  kasten-io (v1:metadata.namespace)
      CONCURRENT_SNAP_CONVERSIONS:                    <set to the key 'concurrentSnapConversions' of config map 'k10-config'>               Optional: false
      CONCURRENT_WORKLOAD_SNAPSHOTS:                  <set to the key 'concurrentWorkloadSnapshots' of config map 'k10-config'>             Optional: false
      K10_DATA_STORE_PARALLEL_UPLOAD:                 <set to the key 'k10DataStoreParallelUpload' of config map 'k10-config'>              Optional: false
      K10_DATA_STORE_PARALLEL_DOWNLOAD:               <set to the key 'k10DataStoreParallelDownload' of config map 'k10-config'>            Optional: false
      K10_DATA_STORE_GENERAL_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreGeneralContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_GENERAL_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreGeneralMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_DATA_STORE_RESTORE_CONTENT_CACHE_SIZE_MB:   <set to the key 'k10DataStoreRestoreContentCacheSizeMB' of config map 'k10-config'>   Optional: false
      K10_DATA_STORE_RESTORE_METADATA_CACHE_SIZE_MB:  <set to the key 'k10DataStoreRestoreMetadataCacheSizeMB' of config map 'k10-config'>  Optional: false
      K10_LIMITER_GENERIC_VOLUME_SNAPSHOTS:           <set to the key 'K10LimiterGenericVolumeSnapshots' of config map 'k10-config'>        Optional: false
      K10_LIMITER_GENERIC_VOLUME_COPIES:              <set to the key 'K10LimiterGenericVolumeCopies' of config map 'k10-config'>           Optional: false
      K10_LIMITER_GENERIC_VOLUME_RESTORES:            <set to the key 'K10LimiterGenericVolumeRestores' of config map 'k10-config'>         Optional: false
      K10_LIMITER_CSI_SNAPSHOTS:                      <set to the key 'K10LimiterCsiSnapshots' of config map 'k10-config'>                  Optional: false
      K10_LIMITER_PROVIDER_SNAPSHOTS:                 <set to the key 'K10LimiterProviderSnapshots' of config map 'k10-config'>             Optional: false
      K10_LIMITER_IMAGE_COPIES:                       <set to the key 'K10LimiterImageCopies' of config map 'k10-config'>                   Optional: false
      K10_EPHEMERAL_PVC_OVERHEAD:                     <set to the key 'K10EphemeralPVCOverhead' of config map 'k10-config'>                 Optional: false
      K10_PERSISTENCE_STORAGE_CLASS:                  <set to the key 'K10PersistenceStorageClass' of config map 'k10-config'>              Optional: false
      AWS_ASSUME_ROLE_DURATION:                       <set to the key 'AWSAssumeRoleDuration' of config map 'k10-config'>                   Optional: false
      KANISTER_TOOLS:                                 <set to the key 'KanisterToolsImage' of config map 'k10-config'>                      Optional: false
      K10_RELEASE_NAME:                               kasten
      KANISTER_FUNCTION_VERSION:                      <set to the key 'kanisterFunctionVersion' of config map 'k10-config'>  Optional: false
    Mounts:
      /mnt/k10-features from k10-features (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jvsw8 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  k10-features:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      k10-features
    Optional:  false
  kube-api-access-jvsw8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  49s                default-scheduler  Successfully assigned kasten-io/state-svc-7586bb89c9-kw62c to hive02
  Normal   Pulling    49s                kubelet            Pulling image "gcr.io/kasten-images/state:7.0.8"
  Normal   Pulled     49s                kubelet            Successfully pulled image "gcr.io/kasten-images/state:7.0.8" in 419ms (419ms including waiting). Image size: 114896174 bytes.
  Normal   Created    49s                kubelet            Created container state-svc
  Normal   Started    49s                kubelet            Started container state-svc
  Normal   Pulled     48s                kubelet            Successfully pulled image "gcr.io/kasten-images/events:7.0.8" in 368ms (368ms including waiting). Image size: 113453441 bytes.
  Normal   Pulling    48s                kubelet            Pulling image "gcr.io/kasten-images/events:7.0.8"
  Normal   Created    48s                kubelet            Created container events-svc
  Normal   Started    48s                kubelet            Started container events-svc
  Warning  Failed     31s (x2 over 48s)  kubelet            Failed to pull image "gcr.io/kasten-images/admin:7.0.8": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kasten-images/admin:7.0.8": failed to resolve reference "gcr.io/kasten-images/admin:7.0.8": gcr.io/kasten-images/admin:7.0.8: not found
  Warning  Failed     31s (x2 over 48s)  kubelet            Error: ErrImagePull
  Normal   BackOff    18s (x5 over 47s)  kubelet            Back-off pulling image "gcr.io/kasten-images/admin:7.0.8"
  Warning  Failed     18s (x5 over 47s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    7s (x3 over 49s)   kubelet            Pulling image "gcr.io/kasten-images/admin:7.0.8"

Cheers,
Andrei

Hi @gavu please share the output of the below command

kubectl get deployments.apps -n kasten-io state-svc -o jsonpath="{range .spec.template.spec.containers}{.image}{'\n'}{end}"

Thanks

Hi @Hagag,

sure, here it is:

$ kubectl get deployments.apps -n kasten-io state-svc -o jsonpath="{range .spec.template.spec.containers}{.image}{'\n'}{end}"

gcr.io/kasten-images/state:7.0.8
gcr.io/kasten-images/admin:7.0.8
gcr.io/kasten-images/events:7.0.8

Hi @Hagag ,

I fixed it. Apparently I used the entire values.yaml file and just customized some values. I removed everything which was “default” and now k10 updated as expected. I guess some older “default” values were deprecated.

Hope this helps someone else too

Cheers,

Andrei

Comment

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded