Hey everyone, after the session yesterday with Geoff I wanted to share some of the notes I put together with everyone so you can see the importance of storage within Kubernetes, also you can see for those in the vSphere space how many similarities there are here when it comes to being enterprise ready to store your data services in Kubernetes. 

A topic we spoke about was, who remembers when we were told we should not run SQL or Exchange on Virtual Machines? 

Kubernetes release 1.24 (May 2022) (Codename: Stargazer)

Storage Capacity and Volume Expansion Are Generally Available

• Storage capacity tracking supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding.
• Volume expansion adds support for resizing existing persistent volumes

Storage Plugin Migration

Work is underway to migrate the internals of in-tree storage plugins to call out to CSI Plugins while maintaining the original API. The Azure Disk and OpenStack Cinder plugins have both been migrated.

CSI Snapshot

VolumeSnapshot v1beta1 CRD has been removed.

Volume snapshot and restore functionality for Kubernetes and the Container Storage Interface (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, moved to GA in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.20 and is now unsupported. Refer to KEP-177: CSI Snapshot and Volume Snapshot GA blog for more information.

Kubernetes release 1.25 (August 2022) (Codename: Combiner)

Promoted Local Ephemeral Storage Capacity Isolation to Stable

The Local Ephemeral Storage Capacity Isolation feature moved to GA. This was introduced as alpha in 1.8, moved to beta in 1.10, and it is now a stable feature. It provides support for capacity isolation of local ephemeral storage between pods, such as EmptyDir, so that a pod can be hard limited in its consumption of shared resources by evicting Pods if its consumption of local ephemeral storage exceeds that limit.

Promoted core CSI Migration to Stable

CSI Migration is an ongoing effort that SIG Storage has been working on for a few releases. The goal is to move in-tree volume plugins to out-of-tree CSI drivers and eventually remove the in-tree volume plugins. The core CSI Migration feature moved to GA. CSI Migration for GCE PD and AWS EBS also moved to GA. CSI Migration for vSphere remains in beta (but is on by default). CSI Migration for Portworx moved to Beta (but is off-by-default).

Promoted CSI Ephemeral Volume to Stable

The CSI Ephemeral Volume feature allows CSI volumes to be specified directly in the pod specification for ephemeral use cases. They can be used to inject arbitrary states, such as configuration, secrets, identity, variables or similar information, directly inside pods using a mounted volume. This was initially introduced in 1.15 as an alpha feature, and it moved to GA. This feature is used by some CSI drivers such as the secret-store CSI driver.

Introduced KMS v2 API

Introduce KMS v2alpha1 API to add performance, rotation, and observability improvements. Encrypt data at rest (ie Kubernetes Secrets) with DEK using AES-GCM instead of AES-CBC for kms data encryption. No user action is required. Reads with AES-GCM and AES-CBC will continue to be allowed. See the guide Using a KMS provider for data encryption for more information.

Kubernetes release 1.26 (December 2022) (Codename: Electrifying)

Storage improvements

Following the GA of the core Container Storage Interface (CSI) Migration feature in the previous release, CSI migration is an on-going effort that we've been working on for a few releases now, and this release continues to add (and remove) features aligned with the migration's goals, as well as other improvements to Kubernetes storage.

CSI migration for Azure File and vSphere graduated to stable

Both the vSphere and Azure in-tree driver migration to CSI have graduated to Stable. You can find more information about them in the vSphere CSI driver and Azure File CSI driver repositories.

Delegate FSGroup to CSI Driver graduated to stable

This feature allows Kubernetes to supply the pod's fsGroup to the CSI driver when a volume is mounted so that the driver can utilize mount options to control volume permissions. Previously, the kubelet would always apply the fsGroupownership and permission change to files in the volume according to the policy specified in the Pod's .spec.securityContext.fsGroupChangePolicy field. Starting with this release, CSI drivers have the option to apply the fsGroup settings during attach or mount time of the volumes.

In-tree GlusterFS driver removal

Already deprecated in the v1.25 release, the in-tree GlusterFS driver was removed in this release.

In-tree OpenStack Cinder driver removal

This release removed the deprecated in-tree storage integration for OpenStack (the cinder volume type). You should migrate to external cloud provider and CSI driver from https://github.com/kubernetes/cloud-provider-openstack instead. For more information, visit Cinder in-tree to CSI driver migration.

What can we expect in 2023?

Kubernetes 1.27 release information

Alpha.1 released Jan 24th 2023

Alpha.2 planned for Feb 14th 2023

Alpha.3 planned for Feb 28th 2023

Code freeze on Mar 14th 2023

RC0 release planned on Mar 21st 2023

RC1 release planned on Apr 4th 2023

V1.27.0 GA scheduled for Apr 11th 2023

Based on enhancement tracking we may or may not see the following items in the next release, more detail can be found here - https://github.com/orgs/kubernetes/projects/117

• PV Health Monitor - Provide a PV health monitor to watch the health status of volumes and mark it if a PV becomes unhealthy.
• Support Recovery from Volume Expansion failures - A user may expand a PersistentVolumeClaim(PVC) to a size which may not be supported by underlying storage provider. In which case - typically expansion controller forever tries to expand the volume and keeps failing.
• ReadWriteOncePod PersistentVolume Access Mode - restricts access to a single pod on a single node. This access mode differs from the existing ReadWriteOnce (RWO) access mode, which restricts access to a single node, but allows simultaneous access from many pods on that node.
• Runtime assisted mounting of Persistent Volumes
• CSI Differential Snapshot for Block Volumes
• VolumeGroup and VolumeGroupSnapshot - Introduce a VolumeGroup API to manage multiple volumes together and a VolumeGroupSnapshot API to take a snapshot of a VolumeGroup.

Some further out interesting storage related enhancements

Changed Block Tracking with CSI VolumeSnapshotDelta - https://github.com/kubernetes/enhancements/pull/3367