In-tree volume plugins¶

An object can be at an API version the target serves and still break on upgrade, because a field inside it stops working. The commonest case is a pod volume of an in-tree plugin that upstream removed when CSI migration finished: a Deployment with a glusterfs volume is valid apps/v1, and on Kubernetes 1.26 or later its pods do not start. upgradescope reports these as volume-plugin findings (#351). A StorageClass whose provisioner is the plugin's in-tree one (kubernetes.io/rbd) has the same problem for the claims it provisions (#362).

What is checked¶

The knowledge base carries a hand-authored dataset, internal/kb/data/volumeplugins.json. Each entry names a plugin by its field in a pod's volumes[] (corev1.VolumeSource) and a PersistentVolume's spec (corev1.PersistentVolumeSource), the minor its end came in, a classification, the replacement, and the upstream release notes and pull requests that minor comes from. A unit test refuses an entry without a citation, with a minor that does not parse, or whose name is not a field of either type.

An entry may also name the plugin's in-tree StorageClass provisioner (provisioner, as kubernetes.io/<name>, each once), with a citation of its own (provisionerCitation): the upstream source that names it, the plugin's pkg/volume package in the last release that had it, or k8s.io/csi-translation-lib for a CSI-migrated plugin. cephfs never had an in-tree provisioner, nor gitRepo or flexVolume, so a StorageClass cannot name one of them. The provisioners are part of the knowledge base version.

The classification follows the k8s.io/api field comments: "the in-tree type is no longer supported" is removed, and "all operations are redirected to the CSI driver" is CSI migration.

Plugin	Class	Minor	CSI driver	StorageClass provisioner
`scaleIO`	removed	1.22		`kubernetes.io/scaleio`
`flocker`, `quobyte`, `storageos`	removed	1.25		`kubernetes.io/flocker`, `kubernetes.io/quobyte`, `kubernetes.io/storageos`
`glusterfs`	removed	1.26		`kubernetes.io/glusterfs`
`cephfs`	removed	1.31		none
`rbd`	removed	1.31		`kubernetes.io/rbd`
`gitRepo`	removed (disabled)	1.33		none
`cinder`	CSI migration	1.26	`cinder.csi.openstack.org`	`kubernetes.io/cinder`
`awsElasticBlockStore`	CSI migration	1.27	`ebs.csi.aws.com`	`kubernetes.io/aws-ebs`
`azureDisk`	CSI migration	1.27	`disk.csi.azure.com`	`kubernetes.io/azure-disk`
`gcePersistentDisk`	CSI migration	1.28	`pd.csi.storage.gke.io`	`kubernetes.io/gce-pd`
`vsphereVolume`	CSI migration	1.29	`csi.vsphere.vmware.com`	`kubernetes.io/vsphere-volume`
`azureFile`	CSI migration	1.30	`file.csi.azure.com`	`kubernetes.io/azure-file`
`portworxVolume`	CSI migration	1.36	`pxd.portworx.com`	`kubernetes.io/portworx-volume`
`flexVolume`	deprecated (1.23)			none

Notes:

gitRepo has been deprecated since 1.11. Kubernetes 1.33 disabled it by default (the GitRepoVolumeDriver feature gate can turn it back on), and 1.36 locked it off. It counts as removed from 1.33.
1.31 removed rbd's CSI migration support with the plugin. So, like cephfs, which never had CSI migration, the field stops working.
vsphereVolume's minor is the release that removed the GA CSIMigrationvSphere feature gate (1.29), which made CSI migration unconditional.
azureFile's in-tree plugin was removed for 1.28 and the removal reverted before release (kubernetes/kubernetes#118388). It was removed again in 1.30.
photonPersistentDisk is not in the dataset: its removal minor was not confirmed from release notes. A field the dataset does not list is not judged.

Severity¶

Class	Blocker	Warning	Info
removed	at or after the minor	the minor before it	earlier
CSI migration	never	at or after the minor: name the CSI driver that must be installed	earlier
deprecated	never	never	always

A removed plugin's pods do not start whether they name it or mount a claim bound to a PersistentVolume that does, and a StorageClass with its in-tree provisioner no longer provisions new claims. A CSI-migrated plugin's StorageClass keeps working, its new claims provisioned by the named CSI driver. A CSI-migrated plugin is never a blocker on its own. The API field keeps working, and every operation goes to the named CSI driver, so a cluster with that driver installed is fine. upgradescope does not check whether the driver is installed. Each finding is keyed volume-plugin/<plugin>, so ignore rules, upgradescope.basit.engineer/ignore annotations on the workload and baselines work on it as on any other finding.

Where the volumes are read from¶

scan --files and the gate. The pod templates of Deployments, DaemonSets, StatefulSets, ReplicaSets, Jobs and CronJobs, a Pod's own volumes, PersistentVolume manifests, and StorageClass manifests with an in-tree provisioner. Each one is located by file and line. A document that cannot be decoded makes the volumes capability partial. With ?cluster=, the gate adds the posted manifests' plugins to the cluster's.
Live (scan, the agent): pods. The pods the collector lists anyway: the kube-system pods from the versions step and the rest from the addons step. Reading them makes no request of its own. Pods are counted per plugin and namespace, and no pod is named. Between full pod passes (--pod-pass-every), the pods outside kube-system are reported as the last full pass read them, as add-ons are (addOnEvidenceAgeSeconds). If no pod can be listed, volumes is not assessed, and the two lists below are not made. If a later page fails, it is partial.
Live: PersistentVolumes and StorageClasses (#362). The capability makes two requests of its own each collection, more on a cluster with more than 500 of either: one paged list of PersistentVolumes and one of StorageClasses, 500 objects a page (TestCollectLiveVolumesTwoExtraRequests counts them). They are listed every collection, never held between pod passes. A PersistentVolume counts for the in-tree plugin of its source, under the namespace of its claim when it is bound, and under none (cluster-scoped) when it is not (a Released one, whose claim is gone, included). A StorageClass whose provisioner is an in-tree one counts for that plugin, under none; a CSI provisioner is not counted. Both are named in the finding, and the ignore annotations on them work as on a workload. The pods the finding counts beside them are not named, so they are counted as omitted (objectsOmitted, "…and N more"): accepting every named PersistentVolume and StorageClass, by annotation or by a name: rule, leaves the finding standing for the pods, with "N object(s) suppressed". Only a rule without object selectors takes the whole finding. The gate does the same for the cluster's pods beside a PR's objects. The agent's ClusterRole grants get and list on persistentvolumes and storageclasses for this. A refused or failed list makes volumes partial, with a reason naming what was not read (for example "list persistentvolumes: ... forbidden ...; PersistentVolumes were not checked"). It is not a hard error, but it is a required gap when the unread list could hide a blocker (#374): the knowledge base has a removed plugin, removed at or before the target, that a PersistentVolume can name (for the PersistentVolume list) or that has an in-tree provisioner (for the StorageClass list). From 1.22 (scaleIO) that is every target, so the verdict is unknown and the gate exits 2 unless --allow-incomplete, where the same cluster with the lists allowed reads blocked on an rbd PersistentVolume. csi-migration plugins only warn and never make the gap required. A hand-written agent role (rbac.create=false) without the two rules therefore reads unknown, not ready, at those targets. See Verdict and score.
Still not traced. A pod that uses an in-tree plugin through a claim is found through the PersistentVolume the claim is bound to, not traced from the pod: the finding names the PersistentVolume and counts it under the claim's namespace, but does not name the pods that mount it. A claim still Pending is not read, so a claim of an in-tree StorageClass is found through the class only. A PersistentVolume of a CSI StorageClass names the csi source and is not counted.

Older agents¶

volumes is an optional capability. A collector that predates it does not report it: a v0.1.x or v0.2.0 release-candidate agent, or a files inventory saved by an older CLI. Its report then has a volumes gap, "not reported by the collector", so the cluster reads as not assessed for volume plugins, never as clean. That gap, unlike a partial one that skipped a PersistentVolume or StorageClass list, is optional and changes no verdict.

A newer agent can push to an older server. The server ignores fields and capabilities it does not know, so it accepts the push and judges what it knows. No collectorSchema bump was needed, and agents and servers can be upgraded in either order.

Validation at ingest¶

The server checks each volumePlugins entry with the checks it makes of an API usage entry (apiUsage, apiAuthorshipUnknown, a CRD's usage, a Helm release's manifestApis): no count, objectsOmitted or namespace count may be negative, namespace keys and object refs must be namespace and object names, and an entry lists at most 100 objects. A plugin must also be a field name (at most 64 ASCII letters and digits, starting with a letter), and each plugin is listed once. A push that breaks one of these is refused (422), and the error names the field. The agent repairs its inventory before pushing: it drops an entry with a bad plugin or count, and the bad objects and namespace keys of the rest, and marks volumes partial with the reason. One hostile pod cannot make every push of the cluster fail.

Not checked¶

Field-level removals other than in-tree volume plugins are not checked. These include the seccomp alpha annotations (ignored since 1.27), the kubernetes.io/ingress.class annotation, Service.spec.externalIPs and beta.kubernetes.io/os.