Claims ledger¶
Every public claim upgradescope makes (README, chart README, Action, CLI help, docs) is listed here, with the automated tests that keep it true or the issue that tracks why there are none. A claim that stops being true should turn CI red, not wait for the next audit.
The ledger comes out of the October 2026 claims verification (#99): red-team missions attacked each claim on real clusters and integrations. Round 1 is in #131 to #137; round 2 filed #165 to #169 and #189 to #200; round 3 ran against v0.2.0-rc.3 (the red team on a source build of the same commit) and filed #366 among others. Each claim is in one of three places:
- the tables by area: claims that held when audited, the part of a claim that held, or claims that hold since their fix landed with a test, with what proves it. A few are still checked by hand only; those say "not automated" and name #99;
- Under repair: claims that did not hold, with the issue that fixes them; they move up when the fix lands with a test;
- Not yet audited: claims that no mission has confirmed in full, or that passed in part (the row says which part did not), with the tests that cover them so far.
IDs are the audit's, so a row can be traced back to its mission; the
DS- rows, for the documentation's own contracts, came after it.
How to read "Proven by". Each reference is checked by
hack/claims-check.sh (make claims-check, run by CI's repo-checks job), so a
renamed or deleted test fails CI:
TestName: a Go test (unit, golden, or an env-gated integration test);e2e:fn: a gate of the kind end-to-end run,hack/e2e.sh, on Kubernetes 1.31 and 1.37 for every PR that changes code (IR-24) and 1.29 to 1.37 weekly (CI's kube job);ci:job: a CI job of.github/workflows/ci.yml;make target, or a path such as ahack/test script.
A row with no reference says "not automated" (or, in the last table, "not
audited") and names the issue that tracks it. That issue must be open:
hack/claims-trackers.sh (CI's repo-checks job on pull requests and pushes,
not on a release, which depends on the tagged code and never on live issue
state; it asks GitHub, so it is not part of make claims-check) fails when
the issue after not true yet:, not automated:, not audited: or
under repair: is closed, so a gap whose tracker closed is either fixed (move
the row up with its proof) or re-pointed.
A claim that is only checked by hand names #99.
API usage: removed and deprecated APIs¶
| ID | Claim | Proven by |
|---|---|---|
| API-01 | Removed and deprecated APIs are found by who writes them, from managedFields and the last-applied annotation, not by whether the API is still served. An object written through a deprecated version is reported with its field manager; once re-applied through the GA version it is not, and apiserver-maintained objects never are. Four control-plane field managers are trusted and never reported: kube-apiserver, kube-controller-manager, kube-scheduler (which a custom scheduler built on the kube-scheduler framework may report too, so its deprecated writes are not seen) and api-priority-and-fairness-config-producer-v1; the architecture page and API usage detection list all four (#239). |
TestCollectAPIUsageFlagsOnlyObjectsAuthoredViaDeprecatedVersion TestCollectAPIUsageObjectWrittenViaGAVersionIsNotAFinding TestCollectAPIUsageClearsAfterManagerMigrates TestCollectAPIUsageAPFBootstrapObjectsAreNotBlockers TestAuthoringManagerIgnoresInternalManagersAndStatusEntries e2e:deprecated_object_reported |
| API-01b | A freshly created cluster, scanned at its next minor, has no removed-API blocker (#3). | e2e:no_removed_api_blockers |
| API-01c | When the kind itself goes away (no surviving version), every stored object counts. | TestCollectAPIUsageTypeRemovedKindCountsEveryObject TestCollectAPIUsageRealKBPodSecurityPolicyCountsEveryObject |
| API-01d | An object of a flagged kind with no managedFields entry left to judge by (none outside the status subresource and the control plane's managers) and no usable last-applied annotation (absent, unparseable, or without an apiVersion; a DeviceClass created through resource.k8s.io/v1beta1 with spec: {}, which the apiserver stores the same as one created through v1) cannot be attributed, and is reported as a deprecated-api info finding titled "authorship unknown", with the key suffix /authorship-unknown. It is never counted as use of the API, so it changes neither verdict nor score (#199). An object a controller later wrote the status of is as unknown as before. A kind with several flagged versions served at once gets one such finding, not one per version. Objects the control plane alone wrote, or that carry a usable last-applied annotation or a non-status entry, are attributed or the control plane's own, and are not reported; the architecture page says so too, instead of calling such objects undetected (#239). |
TestCollectAPIUsageFieldlessObjectsAreAuthorshipUnknown TestCollectAPIUsageAuthorshipUnknownOncePerKind TestEvalAuthorshipUnknown TestValidateIdentifiersRefusesInvalid TestValidateLimitsRefusesBeyond |
| API-02 | removed-api is a blocker when the API is removed at or before the target and a warning when it is removed in the next minor. |
TestEvalAPIUsageRemovedAtTarget TestEvalAPIUsageRemovedAtTargetPlusOne TestEvaluateGolden |
| API-02b | A deprecation that takes effect after the target is titled as a future deprecation, and "(projected)" past the knowledge base's horizon, not as current. A removal past the horizon (a k8s.io/api lifecycle default, not a shipped release) is titled "(projected)" too, as a blocker and as a warning, and says so in its detail, in a live or manifest finding and in the Helm finding for a release's stored manifest ("removed in 1.38 (projected)", "is projected not to serve"); one at or inside the horizon (a shipped release, the horizon itself included) is not marked. A target past the horizon says in its required kb-coverage gap that removals after the horizon are projected (#266). |
TestEvalAPIUsageFutureDeprecationTitle TestRemovalPastHorizonIsProjected TestRemovalAtHorizonIsNotProjected TestEvalHelmManifestRemovalPastHorizonIsProjected TestEvalHelmManifestRemovalAtHorizonIsNotProjected TestEvaluateGolden |
| API-02c | In --files mode and in the gate, a manifest at an API version the target does not serve yet (introduced after it) is a removed-api blocker titled "not served until X", because applying it fails like a removed API; the Deprecated branch does not answer for it first, and the remediation names the newest version of the kind the target can serve (it says alpha and beta APIs may need enabling when that is one; the GA version wins a tie at one release), or the release to upgrade to. A live cluster's stored objects are never judged so, and the same manifest at a target that serves it is clean (#266). Its key is removed-api/<group>/<version>/<kind>/unserved (API-02d). |
TestFilesModeUnservedAPIIsABlocker TestLiveInventoryIsSilentAboutIntroduced TestScanFilesAPINotServedYetIsABlocker TestGateManifestAtAPINotServedYetIsABlocker TestServedAlternative TestServedAlternativeIsServed TestServedAlternativePrefersStabilityAtTheSameRelease TestServedAlternativeIsNeverTheEntryItself TestEvaluateGolden |
| API-02d | The "not served until X" blocker has its own key, removed-api/<group>/<version>/<kind>/unserved; the removal blocker and the warning for a removal in the next minor keep removed-api/<group>/<version>/<kind> and share it (an ignore rule accepts a key whatever the severity, a baseline marks a risen severity new). So a rule or a baseline taken from a scan at a target that does not serve the API yet never accepts the removal blocker at a later target (--target 1.30 then --target 1.37 on a v1beta1 ServiceCIDR: the blocker is visible, new, and the gate fails). A rule with the bare key of an API whose finding is the unserved one matches nothing and scan says which key to write; a baseline written before reads without its not-served-yet entries, so they come back once. A deprecated-call row still folds into the unserved finding, and a partial api-usage gap that skipped the API still hides it, also in the server's carried-finding index for notifications; a Helm release named unserved has the key of a release, not of this phase (#300). |
TestUnservedBlockerHasItsOwnKey TestUnservedKeySuffix TestUnservedKeyStillMatchesItsAPI TestDeprecatedCallFoldsIntoUnservedFinding TestRuleAndBaselineForUnservedDoNotHoldTheRemoval TestRemovalWarningAndBlockerShareTheirKey TestBareKeyRuleForUnservedAPIWarns TestOldBaselineDoesNotHoldARemovalByAnUnservedEntry TestBaseOfUnservedKeyIgnoresHelmReleases TestCarriedFoldMatchesFoldsInto TestFilesModeUnservedAPIIsABlocker TestEvaluateGolden |
| API-08 | engine.Evaluate grows linearly with the number of namespaces: the namespace-to-team map is built once per evaluation pass and threaded to every caller, with no package-level cache, so a result is the same deduplicated, sorted list and every golden is byte-identical (#333). A finding for an object with no metadata.namespace is worded "no namespace set", never "namespace unset", and its key, suppressions and baselines are unchanged (#334). |
TestEvaluateScalesLinearlyWithNamespaces TestEvaluateGolden TestEvalAPIUsageManifestObjects TestEvalAPIUsageManifestsInAClusterRow TestCRDUsageManifestsInAClusterRow |
| API-03 | CRD versions are judged, live and in --files: a custom resource at a version its CRD does not serve is a blocker, a deprecated served version in use is a warning that quotes its deprecationWarning, and a status.storedVersions entry the CRD no longer serves is a warning with storage-migration remediation. |
TestEvalCRDVersionsUnservedInUse TestEvalCRDVersionsDeprecatedInUse TestEvalCRDVersionsStoredUnserved TestCollectCRDs TestCollectFilesCRDVersions |
| API-04 | The scanner lists every resource at a version that is not deprecated whenever the cluster serves one. On the e2e kind cluster, which serves one deprecated group/version on purpose, neither scan nor the agent calls a deprecated API (the e2e's own request through that version shows the audit log records them; hack/e2e/deprecated-request-allowlist.txt is empty since #139); a cluster that serves a kind only at deprecated versions still gets a LIST there (API-04c). |
TestCollectAPIUsageNeverListsDeprecatedVersionWhenAnotherIsServed TestCollectAPIUsageListsAtReplacementGroupWhenOwnGroupIsAllDeprecated e2e:audit_no_deprecated_requests |
| API-04b | Repeat scans of an unchanged cluster give identical reports: the scanner's own deprecated-endpoint LISTs are never reported as callers, also on a scan whose API discovery fails or skips a group, when the metric still holds an earlier scan's rows: the rows at the group/versions where the knowledge base schedules a removal (the only ones it lists at a deprecated version) are then named as skipped by deprecated-calls for that scan, not attributed (#239). When discovery fails outright that is every such group/version, also one the scanner would not list at on that cluster (flowcontrol.apiserver.k8s.io/v1beta3 where v1 is served), so other clients' real calls there are withheld for that scan too, and the reason says the withholding is that broad; and kinds the knowledge base deprecates but never removes (core v1 Endpoints, ComponentStatus) are not listed at all (#123). |
TestCollect_SelfFeedingDeprecatedCalls TestSelfListedRowsAreNotAttributedWhenDiscoveryFails TestCollectDeprecatedCallsWithholdsRowsDiscoveryCouldNotAttribute TestDiscoveryFailureWithholdingIsWordedAsBroadAsItIs TestUnknownSelfCallsAreTheRemovalGroupVersions TestCollectAPIUsageReportsItsOwnDeprecatedLists TestCollectAPIUsageMakesNoDeprecatedRequestOn137 e2e:audit_no_deprecated_requests |
| API-04c | A kind served only at a deprecated version that is being removed (coordination.k8s.io/v1beta1 leasecandidates, policy/v1beta1 podsecuritypolicies on 1.24) is still listed there, since its objects can block the upgrade. That LIST is not scored as a caller: the deprecated-calls capability comes back partial, naming those endpoints as skipped, because the metric cannot tell other clients from the scanner there (apiserver audit logs can). |
TestCollect_SelfFeedingDeprecatedCalls TestCollectAPIUsageReportsItsOwnDeprecatedLists TestCollectDeprecatedCallsMarksSelfListedResources |
| API-05 | A caller with no stored objects is a standalone deprecated-api-in-use finding; caller evidence otherwise merges onto the object finding, as a sentence of its detail and as structured callers, each the standalone finding it would have been. Caller evidence fails closed under suppression (#236): when an upgradescope.basit.engineer/ignore annotation or a rule with namespace, name or file takes every object of the finding, each folded caller stands as its own deprecated-api-in-use finding at its own severity, so a blocker caller keeps the verdict blocked. Only a rule without object selectors that takes the object finding, or a rule without object selectors for the caller's own key or category, suppresses the caller (a caller lists no objects or namespaces, so selectors match nothing). A re-emitted caller keeps findings[] in severity order, a blocker caller ahead of the report's warnings and infos. This holds for scan, /gate and the agent's spec.ignore, which share the code. At /gate, caller rows come only from a cluster's apiserver metrics, so only ?cluster= has them (a team-scoped gate's share has none): a re-emitted caller is tagged source: cluster, like every deprecated-api-in-use finding there, and a blocker caller keeps clusterVerdict blocked, not the gate's own verdict, ready, X-Upgradescope-Verdict or ?fail-on status, which judge only what the manifests introduce. |
TestEvaluateCallerWithoutObjectsStaysStandalone TestEvaluateMergesCallersIntoAPIUsageFinding TestEvaluateMergesSubresourceCallers TestEvaluateMoreSevereCallerIsNotFolded TestFoldKeepsCallersAsData TestApplyObjectSuppressionKeepsFoldedCallers TestApplyWholeFindingRulesSuppressCallers TestApplyPartialObjectSuppressionKeepsCallersOnFinding TestApplyCallerRuleWithSelectorsTakesNothing TestApplyReemittedCallerKeepsSeverityOrder TestGateClusterReemittedCallerBlocksClusterVerdictOnly TestJSONReportMatchesSchema TestEvaluateGolden |
| API-06 | CRD versions are judged per CRD, whatever the target (#48): custom resources at a version the CRD does not serve are a crd-version blocker; custom resources written through a deprecated: true version are a warning quoting its deprecationWarning (info when none is found); a status.storedVersions entry the CRD no longer serves is a warning whose remediation is the storage version migration. |
TestEvalCRDVersionsDeprecatedInUse TestEvalCRDVersionsUnservedInUse TestEvalCRDVersionsStoredUnserved TestEvalCRDVersionsCleanCRD TestEvaluateGolden |
| API-06b | Live, CRDs are read through apiextensions.k8s.io/v1. The custom resources of a CRD with a deprecated or unserved version are listed once, metadata only, at a served version that is not deprecated, and counted by managedFields authorship. A forbidden list, or a CRD with only deprecated versions to list at, makes crds partial, naming the versions left unchecked. |
TestCollectCRDs TestCollectCRDsForbiddenCustomResources TestCollectCRDsEveryServedVersionDeprecated TestCollectCRDsForbiddenCRDs TestCollectCRDsEndToEnd TestEvalCRDVersionsUncheckedUsage |
| API-06c | With --files, a cert-manager CRD that no longer serves v1alpha2 next to a cert-manager.io/v1alpha2 Certificate is a blocker at the Certificate's file and line. A custom resource whose CRD is not in the files is not assessed (crds partial), never a finding. |
TestCollectFilesCRDVersions TestCollectManifestsCRDVersions |
| API-06d | An inventory from an agent (or a saved files inventory) that predates CRD checks reports crds as not assessed (not required), never as clean. |
TestEvaluateCRDsGap |
| API-09 | In-tree volume plugin removals in pod specs are reported (#351): a pod template, Pod or PersistentVolume manifest (--files, the gate) or a live pod naming a plugin of the cited dataset is a volume-plugin finding, located in manifests. A plugin removed with no migration path (glusterfs 1.26, cephfs and rbd 1.31, gitRepo disabled 1.33) is a blocker at or after its removal and a warning the minor before; one served only through CSI migration (awsElasticBlockStore 1.27, gcePersistentDisk 1.28) is a warning naming the CSI driver, never a blocker; a deprecated one (flexVolume) is info. Every finding is cited, and a manifest without these fields is unchanged (score 100). |
TestVolumePluginsData TestVolumePluginNamesAreAPIFields TestCollectFilesVolumePlugins TestEvaluateVolumePluginWindows TestEvaluateNoVolumePluginsIsClean TestScanFilesVolumePluginRepro TestScanFilesVolumePluginIgnoreAnnotation TestMergeManifestsVolumePlugins TestGateClusterVolumePluginBlamedOnManifests TestGateClusterVolumePluginNotBlamedOnUnrelatedManifest TestEvaluateGolden |
| API-10 | Live, the in-tree volume plugins of pods come from the pods the collector lists anyway, with no request of their own, and from the held pod pass between full passes. The capability's own requests are exactly two paged lists a collection, of PersistentVolumes and of StorageClasses (#362). | TestCollectLiveVolumesTwoExtraRequests TestCollectLiveVolumesFromHeldPass TestCollectLiveVolumesPodListFails |
| API-13 | Live, the volumes capability lists PersistentVolumes and StorageClasses (#362), paged at 500 a page and never held between pod passes: a PersistentVolume's in-tree source (rbd, awsElasticBlockStore, any plugin of the dataset a PersistentVolumeSource can name) counts for that plugin under its bound claim's namespace, or under none when unbound; a StorageClass whose provisioner is the in-tree one of a plugin (kubernetes.io/rbd, kubernetes.io/aws-ebs, ..., mapped in the dataset, each mapping cited to the upstream source that names it; a CSI provisioner is not counted) counts for that plugin. Each is named in the finding. A refused or failed list (on any page) makes volumes partial with a reason naming what was not read, never a hard error, and that gap is required, so the verdict is unknown and the gate exits 2 unless --allow-incomplete, when the knowledge base has a removed plugin (not a csi-migration one) removed at or before the target that the unread list could show: a PersistentVolume source field for the PersistentVolume list, an in-tree provisioner for the StorageClass list (#374); no pod read leaves it not assessed, with neither list made. An rbd PersistentVolume is a blocker at 1.31 and a warning at 1.30. The pods a finding counts beside its named PersistentVolumes and StorageClasses are counted omitted, so accepting every named object (an ignore annotation or a name: rule), in a scan, the agent or the gate beside a PR's objects, leaves the blocker standing for the pods; only a rule without object selectors takes it whole. |
TestEvaluateVolumePluginUnlistedPodsCountedOmitted TestApplyAcceptedPersistentVolumeLeavesThePodsBlocker TestGateClusterVolumeSuppressionKeepsUnlistedPods TestGateAcceptedManifestVolumeKeepsClusterPods TestCollectLiveVolumesPersistentVolumesAndStorageClasses TestCollectLiveVolumesPagesPersistentVolumes TestCollectLiveVolumesListRefused TestCollectLiveVolumesListFailedIsRequired TestAssessmentGapsVolumesPartialRequired TestPVSourcePluginsMatchesTheAPI TestScanRefusedVolumeListsHidingRBDIsNotReady TestCollectLiveVolumesNoPodReadListsNoVolumes TestCollectLiveVolumesObjectsListedEveryCollection TestCollectFilesStorageClassProvisioner TestVolumePluginProvisioners TestParseVolumePluginsRefuses TestEvaluateVolumePluginPersistentVolumesAndStorageClasses TestEvaluateVolumePluginGateMixedObjects TestMergeManifestsVolumePluginsWithClusterObjects TestEvaluateGolden |
| API-11 | Older agents read not assessed for volume plugins: an inventory without the volumes capability (a v0.1.x or v0.2.0 release-candidate agent's, or a files inventory an older CLI saved) has an optional volumes gap, never a clean pass, and its verdict is unchanged. A newer agent's unknown fields and capabilities are accepted, so no collectorSchema bump was needed. |
TestEvaluateVolumesNotReported TestOlderAgentVolumesNotAssessed TestIngestAcceptsUnknownFieldsAndCapabilities TestIngestRefusesInvalidVolumePlugins TestAdmitVolumePlugins |
| API-12 | Volume plugin entries are validated at ingest as API usage entries are, in the identifiers (namespace keys, object refs), the cap on objects and the non-negative counts (API-14), and in their plugin name: a plugin that is not a field name (control characters, over 64 bytes), a negative count or namespace count, a namespace key or object ref that is not a namespace or object name, or objects over the cap is refused (422) naming the field and quoting the value cut; the agent drops such an entry, or the bad objects and keys, before pushing, marking volumes partial with the reason, so one pod's volume cannot make every push of the cluster fail. |
TestAdmitRefusesHostileVolumePlugins TestConformRepairsVolumePlugins TestIngestRefusesHostileVolumePlugins |
| API-14 | A count is what a collector counted, so none is negative: a pushed count, objectsOmitted or namespace count below zero, in apiUsage, apiAuthorshipUnknown, a CRD's usage or a Helm release's manifestApis, is refused (422) naming the field (a count of 0 is accepted), where it used to be stored and read "(-5 objects)" or, for a Helm manifest row, dropped by the engine with no finding. The agent drops the bad entry (a bad Helm manifest row drops its release, a CRD's usage row its CRD) and marks the capability partial with the reason, so one bad row cannot make every push of the cluster fail; an API usage entry is named in api-usage's skipped list and a release in helm's, while a dropped CRD is only counted in crds' reason, as that list names the CRDs whose versions could not be checked and the engine reads it so. |
TestAdmitRefusesNegativeAPIUsageCount TestIngestRefusesNegativeUsageCounts TestConformRepairsNegativeUsageCounts |
| API-07 | A namespace-scoped ignore rule never takes an add-on finding that also covers an install in no named namespace (a manifest object without metadata.namespace, or a cluster-scoped IngressClass): the finding carries unnamespaced: true and stays, so the namespace: sandbox rule that accepts the EOL ingress-nginx in sandbox leaves the blocker for an install that can land in production. A finding whose installs are all in named namespaces the rule matches is still suppressed (#237). |
TestAddOnFindingsMarkInstallsInNoNamedNamespace TestChartIncompatMarksOnlyTheIncompatibleInstalls TestNodeRuntimeFindingsAreNotUnnamespaced TestApplyNamespaceSelectorLeavesFindingsCoveringUnnamespacedInstalls TestEndToEndNamespaceRuleDoesNotTakeAnInstallWithoutNamespace |
| PF-02 | Every cluster-wide list is paged (500 objects a page); the api-usage and Helm lists are metadata-only, and object references are capped. The pod and node lists size each page after the first by the largest object of the one before: as many objects as fit 8 MiB encoded at that size, at least 500 and at most 1,000 (#228), so no page holds more than 1,000 objects, twice what a page held before. No limit set before a page is read bounds the size of what it will hold, so a page's worst case is 1,000 times the largest object, reached when small objects are followed by large ones: 500 pods of 137 bytes followed by 1,000 of up to 41,685 bytes (39.4 MiB encoded) held between 122.3 and 125.3 MiB of live heap on GitHub's ubuntu-latest runner (linux/amd64), where users' agents run: the test's lower bounds read 122.3 to 123.4 MiB and its upper bounds 125.0 to 125.3, in four CI runs at 23f8739 and 04c39fa (runs 37950213827 and 37964795504, attempts 1 to 3; this change before its rebase onto 377fd77) on 9 October 2026 (250 MiB when a page could hold 2,000, on an Apple M1 Pro on 4 October); a run of those pods (500 a page) between 61.7 and 68.0 MiB, and production-sized pods (about 8 KiB each, managedFields included; about 990 a page) between 32.9 and 34.3 MiB. A single reading after a forced collection can also count the garbage of reading the response (up to 153.8 MiB in those runs, 153.0 in #251's CI, which failed the 128 MiB the test enforced then), so the test lists each case 10 times and passes on the lowest upper bound; five dispatched CI runs of that test at 443a542 (before this change's last rebase onto main; 37987299524, 37987318295, 37990925915, 37994320874, 37996739083; 9 October 2026) all passed, the worst case between 123.0 and 125.1 MiB, with 8 of their 50 worst-case attempts past 144 MiB (up to 153.7) and at least 6 of each run's 10 under it. It enforces 144 MiB, 18.7 MiB (14.9%) above the highest upper bound read and 86.4 MiB under the agent's GOMEMLIMIT (90% of 256Mi, 230.4 MiB), what the rest of the agent and a collection's garbage may take. At the worst case's 3.2 bytes of live heap per encoded byte (125.3 MiB for 39.4 MiB), 1,000 pods of about 60 KiB after small ones take a page past that GOMEMLIMIT beside the 39.2 MiB the rest of the agent's heap peaked at in the benchmark (PF-09; (230.4 − 39.2) MiB / 3.18 / 1,000 is 61.6 KiB, 74.2 KiB with nothing else held), where pages of 500 before #228 took about 120 KiB (123.1 KiB; computed, PF-05). The pod, node and namespace lists of add-on and version collection read whole objects, on purpose: add-on detection needs the pods' spec images and labels, the kubelet versions need status.nodeInfo, and the lists are paged as above, which bounds them. No public page claims metadata-only for them (only api-usage, crds and Helm are), and the namespace list, which uses only a name and the team label, is a full list too, where a metadata list would do (internal/collect/versions.go; not changed: #99). |
TestCollectAPIUsageFollowsListPagination TestCollectAddOnsFollowsListPagination TestCollectVersionsFollowsListPagination TestPageLimit TestListPagesAreSizedByTheirLargestObject TestPodPagePeakHeapIsBounded TestCollectHelmFollowsListPagination TestCollectHelmFetchesOnlyTheChosenRevision TestCollectAPIUsageCapsObjectRefs make test-heap |
Deprecated-API callers¶
| ID | Claim | Proven by |
|---|---|---|
| DC-01 | Clients still calling deprecated APIs are found from the apiserver's apiserver_requested_deprecated_apis metric; the endpoints the scanner had to list itself are not reported as callers, and the capability names them as skipped (API-04c). The metric shows whether any client still calls a deprecated API, not which, and the docs and the chart's README and RBAC comment say that. |
TestCollectDeprecatedCalls TestEvalDeprecatedCallsSubresource TestDocsDeprecatedCallsDoNotSayWho |
| DC-02 | When /metrics is forbidden (401 or 403) or lacks the metric, the capability is reported as unavailable with the reason; the rest of the scan still runs. |
TestCollectDeprecatedCallsForbidden TestCollectDeprecatedCallsFamilyAbsent TestCollectDeprecatedCallsOtherErrorNotRewritten |
| DC-02b | A /metrics that hangs no longer spends the scan budget: the request is given up after --request-timeout (default 30s), only deprecated-calls is not assessed, and every other capability is still read (#94). deprecated-calls is not required, so an otherwise clean cluster is still ready, with the gap reported. |
TestCollectStalledMetricsEndsAtRequestTimeout TestCollectStalledMetricsVerdictCanBeReady |
| DC-03 | deprecated-api-in-use severity follows the removal window, and is info when the removal release is unknown. |
TestEvalDeprecatedCallsSeverityVsTarget TestEvalDeprecatedCallsUnparseableReleaseIsInfo TestEvalDeprecatedCallsUnparseableIsInfo |
| DC-04 | A caller row's removal release is the knowledge base's (removed or inferred-removed) when it has a removal for that group, version and resource, else the metric's removed_release label; when they differ the finding names the knowledge base as the source (#236). storagemigration.k8s.io/v1alpha1 StorageVersionMigration, labelled 1.36 and gone in 1.35, blocks at 1.35; networking.k8s.io/v1alpha1 IPAddress, labelled 1.33 and gone in 1.31, blocks at 1.31. The knowledge base wins when its removal is later than the label too, so such a caller can be a warning where the label alone would block. When two of its kinds share a REST resource name, the first entry's removal counts, as in RemovalOfCall. A row for an API the knowledge base does not know, or knows without a removal, is judged by its label. |
TestEvalDeprecatedCallsJudgedByKBRemoval TestEvalDeprecatedCallsIPAddressKBRemoval TestEvalDeprecatedCallsRemovalSource TestKBRemovalsFirstEntryWins |
| DC-05 | Folding caller rows into object findings indexes API usage by group, version and resource once per evaluation, so it is linear in rows and usage entries (#236): Evaluate with 16,000 usage entries and 16,000 caller rows, none folding, took 27.8 to 31.7 ms (BenchmarkFoldDeprecatedCalls16k, 3 runs of 10; 60 MB and 256k allocations each) at commit d3a4380a, where origin/main 76f40efb took 12.9 and 18.4 s (2 runs), on an Apple M1 Pro under load averages of 6 to 16, 9 October 2026. Time and allocations grow linearly: 4,000, 8,000 and 16,000 of each took 7.6, 15.7 and 33.5 ms (12, 28 and 60 MB; one run of 10 each, load average 66, commit bd0db987, same day). The test enforces 1 s. A row folds into the first usage entry whose kind has its resource name, as before the index. A folded row is charged against EvaluateWithin's limit at least what it takes on its finding (its callers entry repeats the requested group, version, resource and subresource). The bound is the engine's: the server's notification delta still compares each new blocker with every carried finding (engine.FoldsInto, tracked in #241), and ingest bounds the number of caller rows only by the push's node budget, not by a cap of its own. |
TestFoldDeprecatedCallsIsLinear TestFoldIndexMatchesKindMatchesResource TestFoldIndexFirstUsageEntryWins TestEvaluateFoldsCoreAndIrregularPlurals TestEvaluateWithinChargesFoldedCallers |
Add-ons past end of life¶
| ID | Claim | Proven by |
|---|---|---|
| AO-01 | Add-ons past end of life are detected by container image and by Helm chart (and by pod labels and IngressClass, AO-09). When the cluster-wide pod list is forbidden, Helm releases and IngressClass controllers still identify them (the capability is then partial, and a required gap, VS-06b) (#199). | TestCollectAddOnsUsesPodImagesAndHelmReleases TestCollectAddOnsPodsForbiddenStillMatchesHelmReleasesAndIngressClasses TestGitOpsChartKeepsAddOnsPartialWhenPodsAreForbidden TestGitOpsAbsentLeavesAddOnsUnavailableWhenNothingIsRead TestMatchAddOns TestMatchAddOnsRealWorldImages TestEvalAddOnsHelmEvidence |
| AO-01b | Each install of an add-on (one per namespace; a second one where images or labels in a Helm release's namespace are on another release line than its appVersion, #165) is judged on its own, grouped by release line: a finding names only the namespaces and teams on its line, so a newer install neither masks an older one nor shares its blame (#129). On a real cluster, Istio on three lines in three teams' namespaces gets findings that each name only their own line's, and EOL 1.28 blocks. |
TestMatchAddOns TestMatchAddOnsFromLabelsAndIngressClass TestHelmReleaseDoesNotHideAnOlderRevision TestUpsertAddOns TestEvalAddOnsGroupsByReleaseLine TestEvalAddOnsGroupsByReleaseLineRealKB TestEvalAddOnsProductEOLOncePerAddOn TestEvaluateGolden e2e:istio_lines_judged_per_install |
| AO-02 | Ingress NGINX, past EOL since March 2026, is a blocker: installed from the upstream chart on a real cluster, scan reports it and exits 2. |
TestIngressNginxRetirement TestImageOnlyIngressNginxVerdicts TestScanIntegration_KindEOLIngressNginx e2e:eol_ingress_nginx_blocks e2e:cr_has_verdict |
| AO-03 | An image matcher of two or more segments is a suffix of the normalised repository path on whole segments (a one-segment matcher is that repository exactly, unless an entry opts a distinctive name into every registry prefix with "*/name", as etcd does), so mirrors and pull-through caches match and provider builds (GKE, AKS) match only entries written for them; a chart matcher is an exact chart name. The image tag, v stripped, is the detected version. A matcher with a tag pattern ("rancher/nginx-ingress-controller:*-hardened*") claims only the tags it matches, ahead of another entry's path-only matcher of the repository; an image without a tag (pinned by digest) is the tag-qualified entry's when its pod's labels or a Helm release in its namespace name that entry, and the path-only entry's otherwise. A component image (Flux's controllers, versioned on their own) is the product at the release line its tag's line ships in, from the entry's cited line table; a line the table lacks is the product without a version, never a guess (#265). |
TestMatchAddOns TestMatchAddOnsRealWorldImages TestSplitTagPattern TestTagMatches TestImageTagPatternTakesPrecedence TestMatchAddOnsUntaggedImageNamedByLabelsOrRelease TestComponentProductLine TestMatchAddOnsFluxFromControllers TestImageMatchersNeedTwoSegmentsToSuffixMatch TestPathMatches TestValidate TestAnyPrefixMatchersAreEtcdOnly TestMatchAddOnsRealWorldCharts TestParseImage TestVersionFromTag TestHelmInstallsJudgedByAppVersion |
| AO-04 | Image repositories no matcher claims go to unrecognizedImages (at most 200, the rest counted in unrecognizedImagesOmitted) and never become findings, nor change score or verdict. An image repository over 16 KiB cannot be an add-on's and is counted in unrecognizedImagesOmitted, not listed, with addons partial saying so (#268). An image another registry version claims can hide a blocker, so a report whose knowledge base differs from its server's reads addons as a required gap (SV-20). The report lists them in JSON, the table, Markdown and the dashboard's cluster view, worded as a detection gap, not a finding. |
TestCollectImageRepositoryOverTheLimitIsCountedNotListed TestMatchAddOnsUnrecognizedCap TestEvaluateReportsUnrecognizedImages TestWriteTableUnrecognizedImages TestWriteMarkdownUnrecognizedImages TestWriteJSONUnrecognizedImages web/src/UnrecognizedImages.test.tsx |
| AO-05 | eol status is a blocker, an EOL date within 90 days a warning, a version older than the oldest tracked release line a blocker when that line has ended (cert-manager 1.5, Istio 1.5, Calico 3.24, Cilium 1.12, Kyverno 1.7, Argo CD 0.12, Flux 1.24; #165), and a target above a compat row's k8s_max a compat finding. |
TestEvalAddOnsEOLBlocker TestEvalAddOnsApproachingEOLWarning TestEvalAddOnsChartIncompatBlocker TestEvalAddOnsCycleLifecycle TestEvalAddOnsJudgesTheInstalledCycle TestEvalAddOnsPredatesTrackedLinesDetail TestEvalAddOnsPredatesTrackedLinesRealKB TestEvalAddOnsPredatesSupportedOldestLine TestProviderBuildsGetNoUpstreamEOL TestEvaluateGolden |
| AO-07 | Node container runtimes past EOL are flagged per release line, naming the nodes; the containerd compat blocker names the nodes that cannot run the target (at most 10, the rest counted), even when they all run one version (#169). | TestEvalAddOnsNodeRuntimes TestEvalAddOnsNodeRuntimeDetail TestEvaluateNodeRuntimeVerdict TestEvalAddOnsNodeRuntimeCompatNamesNodes |
| AO-08 | An add-on the registry has no lifecycle data for (no version, a version between tracked release lines or newer than the newest, a product without lines) says so instead of passing silently; a version older than every tracked line is end of life instead (AO-05). A node container runtime no entry covers (cri-o, docker) is disclosed too: an info finding naming its nodes (#169). The finding's detail says what was missing for the evidence the add-on was found by (an image from which no version was read: no version in its tag and no pod version label naming the add-on, or a component image whose release line the registry does not map yet; a Helm release without an appVersion, ...), never a source the detector did not consult (#301): for an inventory from an agent that predates the pod-label rule (no collectorSchema), the detail names only the tag or component reason and says the agent did not read the pod's version label; for a Helm-found add-on it does not say the release records no appVersion either, because a v0.1.x agent reported the chart version in its place (the server blanks it, legacyView) and the release may record one. |
TestEvalAddOnsNoDataDetail TestEvalAddOnsNoDataDetailSaysWhenTheAgentDidNotReadPodLabels TestLegacyChartAddOnNoDataDetailClaimsNothingItCannotKnow TestEvalAddOnsNodeRuntimeNoDataDetail TestCollectFilesUnreadableImageVersionDetailIsTrue TestEvalAddOnsCycleLifecycle TestEvalAddOnsPredatesTrackedLinesRealKB TestEvalUncoveredNodeRuntimes |
| AO-09 | A pod whose images no matcher claims is an add-on when its app.kubernetes.io/name, helm.sh/chart chart name or app.kubernetes.io/part-of names one, at its app.kubernetes.io/version; provider builds are never claimed through labels. An image an entry claims but that names no version (pinned by digest, or :latest) takes the app.kubernetes.io/version of its pod under the same rule (the name label, or the chart label on a pod without one), the oldest across pods (on the RKE2 path that a pod's labels settle too, in any pod order), so a digest-pinned cert-manager 1.12 is the end-of-life blocker and not "version unknown"; a tag that names a version is never overridden, a label naming another add-on gives nothing, and a component image's product line is not guessed from a label; the cost is linear in the pods sharing the image (#301). An IngressClass with controller k8s.io/ingress-nginx is Ingress NGINX when nothing found it, its RKE2 or AKS builds, or Traefik. Either alone blocks an Ingress NGINX install that runs a mirrored or rebuilt image (#18). |
TestMatchAddOnsFromLabelsAndIngressClass TestLabelAndIngressClassDetectionVerdicts TestCollectAddOnsReadsLabelsAndIngressClasses TestCollectAddOnsIngressClassesUnreadable TestMatchAddOnsImageWithoutVersionTakesLabelVersion TestMatchAddOnsUntaggedClaimTakesTheOldestLabelVersion TestCollectFilesImageWithoutVersionTakesLabelVersion TestPodPassReusedEvidenceCarriesTheLabelVersion TestMatchAddOnsLabelledPodsOfOneImageAreLinear |
| AO-10 | The registry holds 34 add-ons. Five products retired as a whole are end-of-life blockers at any version, each with citations and matchers for the images and charts it publishes: Ingress NGINX, Kubernetes Dashboard, Promtail, Grafana Agent and Weave Net. Karpenter, Gatekeeper and Fluent Bit have release lines synced from endoflife.date. Ingress NGINX images are found under any registry host or mirror prefix that keeps their path (ingress-nginx/controller); the AKS build and RKE2's (rancher/nginx-ingress-controller with a -hardenedN tag) map to their own entries, while RKE1's build of that repository (-rancherN tags; RKE1 has been end of life since 2025-07-31), which RKE2's releases up to v1.20.11 and v1.21.2 also shipped, and Bitnami's are Ingress NGINX (#265). Flux installed from its own manifests is found by its v2 controllers, Flux v1 by fluxcd/flux and weaveworks/flux, and the Kubernetes Dashboard v1.x by its per-architecture images (kubernetes-dashboard-amd64 and others, #265); no image in the table test matches two entries and a bare controller repository matches none. No image or chart is claimed by two entries, and no entry id is another entry's chart name (a pod label names an add-on by either), embedded or operator-supplied: --registry-dir entries that would claim one stop the command at start, whether the matcher is a plain path or carries a tag pattern (the tag-over-path precedence is the embedded registry's alone, so an operator's entry of a new id cannot take an embedded entry's tags with one, and an unedited copy of an embedded file still loads), and when the entry replaces an embedded one whose claim the current registry gives to another entry (a copy taken from an earlier release), the error names the current matcher and says to copy the embedded file again (#265). Crossplane (its release lines and end dates read from its own release table, which endoflife.date does not track), Linkerd and the AWS Load Balancer Controller are in it, found by their images and charts, and so are Cluster Autoscaler and the AWS EBS and EFS CSI drivers (AO-14); the Amazon VPC CNI and the EKS Pod Identity Agent have no entry. The Dashboard metrics-scraper sidecars are not matched, as they carry their own version. |
TestRetiredProductsAreEOL TestEmbeddedRetiredProductsAreBlockers TestMatchAddOnsRealWorldImages TestMatchAddOnsRealWorldCharts TestMatchAddOnsFluxFromControllers TestMatchAddOnsMeshAndLoadBalancerImages TestMatchAddOnsMeshAndLoadBalancerCharts TestEvalAddOnsGitOpsPolicyAutoscalingRealKB TestClaimConflicts TestMergeConflictsOutdatedCopy TestMatchersOverlap TestLoadWithRegistryRejectsDoubleClaims TestLoadWithRegistryRejectsTagQualifiedTakeover TestMergeConflictsFaithfulCopies TestExtraSingleSegmentMatcherLeavesEmbeddedImagesAlone TestEmbeddedEntriesProperties TestDocsRegistryCounts ci:registry |
| AO-11 | An entry with split support (support.extended_eol_date and extended_support_condition: support ends for everyone on eol_date and lasts to the later date only under a condition the collector cannot see) is an eol-approaching warning stating the condition between the two dates, or as the first nears, and an eol-addon blocker after the later date, never a silent pass. RKE2's Ingress NGINX build carries its sources' dates: SUSE builds for community users ended 2026-03-31, Prime LTS support runs until 2027-11-30 (#265). |
TestEvalAddOnsSplitSupport TestEvalAddOnsRKE2IngressNginxSupportEnd TestValidate |
| AO-12 | The Kubernetes compatibility ranges in the registry are read from the upstream page each row cites, and encoded as far as that page states them (#341, #343): a maximum only where upstream says the line is unsupported above it (cert-manager's supported range per release line; Karpenter's matrix of the oldest release that runs each Kubernetes minor, from which a line's maximum is derived; Metrics Server 0.3.x, 1.8 to 1.21; Istio's supported Kubernetes versions per release from istio.io, 1.13 to 1.31, kept as compat rows that must agree with the ranges endoflife.date syncs into its cycles; Linkerd's minimum and maximum per release, 2.10 to 2.20; the AWS Load Balancer Controller 2.0 to 2.3, up to 1.21), a minimum where it says older Kubernetes is unsupported (Calico 3.25 to 3.32, Cilium 1.13 and 1.14, Metrics Server 0.6 and later, Velero 1.14 to 1.18, Prometheus Operator, the AWS Load Balancer Controller 2.4 and later). A tested-with list (Argo CD, Cilium 1.15 and later, Calico's tested versions, kube-state-metrics' client-go version, Ingress NGINX's supported-versions table, and KEDA's and Kyverno's tables, which endoflife.date copies into its cycles as ranges and tools/eol-sync drops for entries carrying the # eol-sync: ignore-k8s-range directive) and a missing matrix (Flux, Traefik, Gatekeeper, Crossplane, CoreDNS, AKS application routing) are not bounds and are not encoded, each entry saying so. At Kubernetes 1.37, Karpenter 1.6.0, cert-manager 1.19.0 and Metrics Server 0.3.6 are blockers citing their rows, Cilium 1.18 and Metrics Server 0.8.1 are not, and a forward-upgrade scan cannot trip a minimum. |
TestEmbeddedCompatRanges TestEmbeddedCompatRangesNetworking TestIstioCompatRowsAgreeWithCycleRanges TestIgnoresK8sRange TestRunDropsK8sRangeOnlyWhereTheEntryAsksFor TestEmbeddedCompatGoldenAt137 TestEvalAddOnsChartIncompatBlocker TestValidate TestEmbeddedEntriesProperties TestDocsRegistryCounts |
| AO-13 | CoreDNS, Metrics Server and kube-state-metrics are found at the image paths AKS (mcr.microsoft.com/oss/kubernetes/*, oss/v2/kubernetes/*), EKS Distro, GKE (gcr.io/gke-release/*, gke.gcr.io/*), RKE2 and k3s (rancher/mirrored-*) and Bitnami (bitnami/, bitnamilegacy/) publish them under, and Cilium, Calico and Prometheus Operator at Rancher's mirrored-* copies; the vendor's build suffix (-7, -hotfix.N, -gke.N, -eks-1-27-N, -debian-12-rN) never reaches the version. Where the provider's build follows its own support policy and no entry covers it (AKS's and GKE's Cilium and Calico, Rancher's hardened-*), the image stays unrecognized instead of raising an upstream end of life, and an image whose tag names no version is an install with none, an info finding (#342). |
TestMatchAddOnsManagedDistroImagePaths TestMatchAddOnsManagedDistroImageWithoutVersionIsUnjudged TestProviderBuildsGetNoUpstreamEOL TestMatchAddOnsRealWorldImages TestClaimConflicts |
| AO-14 | Cluster Autoscaler, the AWS EBS CSI driver, the AWS EFS CSI driver and Longhorn are in the registry, found at the image paths the projects, AWS (ECR Public, the EKS add-on's private eks/ repository) and Rancher's mirrored-longhornio-* copies publish them under, with the build suffix (-eksbuild.N, -debian-12-rN) never reaching the version; their sidecars and date-stamped images are not matched. Their Kubernetes ranges follow the same rule as AO-12, read on 2026-10-12: Longhorn's minimums (1.13 needs Kubernetes 1.34, 1.8 to 1.12 need 1.25, 1.5 to 1.7 need 1.21) and the AWS EFS CSI driver's matrix (v1.2 to v2.1 need 1.17, v0.2 to v1.1 need 1.14, v0.1.0 supports 1.11 to 1.13) are encoded; Cluster Autoscaler's per-minor pairing ("tested on GCP", "no cross version testing"), the EBS driver's "compatible with all Kubernetes versions supported by the Kubernetes project and/or Amazon EKS" and kube-state-metrics' client-go matrix are not bounds, and each entry says so with the URL checked. Longhorn's release lines are synced from endoflife.date; the others publish no dated lines. The support status of Promtail, Grafana Agent, Kubernetes Dashboard, Weave Net and etcd was re-read against the upstream statements on 2026-10-12 and held. Fluent Bit's was re-read too, and its synced release-line dates disagree with the project's SECURITY.md (5.0 ends 2026-11-06 in the registry, 2026-11-30 in SECURITY.md), which the entry's comment records. |
TestEmbeddedCompatRangesStorageAndOperations TestLonghornCyclesJudgeInstalledLine TestMatchAddOnsStorageAndOperationsImagePaths TestEmbeddedEntriesProperties TestClaimConflicts |
Helm releases¶
| ID | Claim | Proven by |
|---|---|---|
| HE-01 | Helm releases whose chart kubeVersion excludes the target, or whose stored manifest uses a removed API, are findings; the revision judged is the installed one (HE-03), and a release with none to judge is a gap, not a clean result. |
TestEvalHelmChartKubeVersion TestEvalHelmManifestRemovedAPIBlocks TestEvalHelmManifestDeprecatedAPIWarns TestHelmManifestWithRemovedAPIBlocksEndToEnd TestCollectHelmChartKubeVersionAndManifestAPIs |
| HE-02 | Every version range that passes registry validation is matchable. | TestMatchesRange |
| HE-03 | Only the installed revision is read: a failed upgrade is judged by what still runs (the newest deployed revision, else the newest superseded one), a release uninstalled with --keep-history yields nothing, and a release with failed revisions only (a failed install) is never dropped silently: its chart and stored manifest are not assessed, and the helm capability is partial and names it (#239). |
TestCollectHelmInstalledRevision TestHelmFailedOnlyReleaseIsANamedGapEndToEnd TestCollectHelmFetchesOnlyTheChosenRevision TestUninstalledHelmReleaseRaisesNoEOLBlocker TestEvalHelmSkipsUninstalledReleases e2e:keep_history_not_reported |
| HE-04 | One corrupt release never fails the capability, and memory stays bounded by one release however long the history: one release is decoded at a time, and beside it at most 7 more payloads are held while they are fetched or wait (8 GETs in flight at most, #226), each, while it is read, its response and the object decoded from it (at most etcd's 1.5 MiB each), and once fetched at most the 1 MiB a Secret holds, so at most 7 × 3 = 21 MiB above one release's decoding (computed; the 300×10×150 KiB and 50×10×450 KiB histories peaked at 19.7 and 17.5 MiB of heap fetching 8 at a time, 4 October 2026, Apple M1 Pro). One release is bounded too, in size and in parsing: 4 MiB stored and 16 MiB of decompressed JSON (a gzip bomb is skipped as too large), and its manifest is parsed in runs of at most 1 MiB and 64Ki YAML nodes (a document over 2 MiB or 64Ki nodes is not parsed, and the release is recorded with its other documents). Either is named on a partial helm capability and the other releases are still read. A payload is one gzip member decompressed no further than its size trailer says, so a lying trailer cannot grow the buffer. On the test harness the gzip bomb peaks at 17 MiB of heap, whatever its trailer says, and the worst manifests that fit the cap (newlines, tiny objects, documents at the size or node bound) at 32–46 MiB of live heap on a GitHub-hosted runner (CI run 37160469086, the test-heap job: the test reads the heap objects after collections forced back to back at GOGC=10, since under the chart's GOMEMLIMIT the runtime sheds garbage and live heap is what must fit, and enforces 64 MiB; read as a heap that holds garbage, at the default GOGC=100, the same cases peaked at 24–93 MiB), where a 127 KiB Secret of tiny ConfigMaps took 564 MiB parsed whole (#168). The run bounds themselves, 1 MiB and 64Ki nodes, are held by a test of the split, without a heap figure. Parsed in runs, a manifest yields what it yields parsed whole. A YAML document is parsed once (#285): kubectl's YAML-to-JSON step, which the walk is checked against, is made from the tree the walk read where that gives the JSON kubectl's own decoder makes, byte for byte, and from the text otherwise (anchors, tags, merge keys, keys that are not strings, duplicate keys, .nan and .inf, text its line reader changes, and text with a Unicode line break: U+0085, U+2028 or U+2029); a plain scalar that starts like a number is typed as go-yaml v2 types it (a copy of its rules, held to it by a test over generated scalars), never put to kubectl's decoder on its own. So a manifest yields what it yields parsed twice, and scan -f and the manifests of CollectManifests, which share the reader, are parsed once too. 32.5% less CPU to decode the bench fill's 1,000 releases (9.160 s to 6.186 s, mean of four runs each, the ThinkPad, 4b1c2893 and a14f737c, 10 October 2026); on a 1 MiB manifest dense in distinct number-like scalars 0.55 to 0.67 of the CPU of parsing twice, and 1.15 to 1.22 times it where a document is walked to its last value and then declined (b3eba1cc, whose production code is a14f737c's, the ThinkPad). These figures were measured at a14f737c and not re-measured after two later commits that change production code: 82927f11 makes the converter decline a document that holds U+0085, U+2028 or U+2029 (it goes to kubectl's decoder as before; the lab's payloads hold none, checked from the seed generator, not re-measured), and f1f2af12, a refactor that does not change what runs (Scale and cost). TestParseOnce_NumberDenseStreamIsNotSlower, a CPU ratio, runs in make test-heap with the heap-bound tests and not in a plain go test ./.... The secrets and configmaps drivers degrade independently. |
TestCollectHelmSkipsCorruptSecretKeepsValid TestCollectHelmPeakHeapIsBoundedByOneRelease TestStartFetchingHandsOverInOrderWithinItsBound TestCollectHelmGzipBombIsBounded TestCollectHelmManifestParsingIsBounded TestCollectHelmManifestDocumentOverTheBoundsIsAGap TestManifestAPIsInRunsMatchesWholeParse TestSplitManifestRunsHoldTheBounds TestParseOnce_CorpusStreamsMatchTwoParses TestParseOnce_EveryHelmDocumentMatchesTwoParses TestParseOnce_ReleasesHaveTheSameFindings TestParseOnce_RandomManifestsMatchTwoParses TestParseOnce_BoundsAreUnchanged TestParseOnce_UnicodeLineBreaksMatchTwoParses TestTreeJSON_MatchesKubectl TestTreeJSON_Documents TestTreeJSON_RandomDocuments TestV2Number_MatchesKubectl TestV2Number_TracksV2Version TestParseOnce_ShortcutIsWired TestParseOnce_NumberDenseStreamIsNotSlower TestDecodeHelmReleaseBoundsDecompressedSize TestDecodeHelmReleaseBoundsStoredSize TestDecodeHelmReleaseChecksTheWholeStream TestCollectHelmReadsConfigMapDriver TestCollectHelmDegradesPerDriver TestCollectHelmPrefersSecretsDriver |
| HE-05 | Helm releases are read from the secrets and configmaps storage drivers; the sql driver (an external database) is not supported, and the GitOps page, the chart README and the README say so. | TestCollectHelmReadsConfigMapDriver TestCollectHelmPrefersSecretsDriver |
| HE-06 | The charts that Argo CD and Flux deploy to the scanned cluster are read from their own resources and matched to the registry like a Helm release's chart: an Argo CD Application's spec.source and spec.sources[] entries with chart set, or a native OCI source (repoURL: oci://..., path: ., no chart: the chart is the last element of the repository path, taken from the redacted URL, and its version the targetRevision; an oci:// source with another path is read as the chart too, since a plain-manifest artifact cannot be told from one, which over-reports the helm gap), and a Flux HelmRelease's spec.chart.spec or its chartRef to an OCIRepository (the newest of helm.toolkit.fluxcd.io v2, v2beta2 and v2beta1 served). spec.sources replaces spec.source as Argo CD does; the in-cluster API server URL is recognised in its common spellings; a Git or Bucket chart path is reduced to its last element. A HelmRelease whose targetNamespace is not its own namespace is a second install beside the release Secret in its own namespace (not merged), as for any chart with a namespace override. Applications and HelmReleases that deploy to other clusters, or to a target that is not a namespace name (which would make the server refuse the whole inventory), are not counted. The chart gives the add-on and its namespace, not an app version, so an add-on retired as a whole (Ingress NGINX) is found and blocks; a GitOps chart version is shown only when exact and no Helm release records the real one (a Flux OCIRepository's ref is read as Flux applies it, digest over semver over tag: a digest is recorded as the digest and is no chart version, a semver as the range it is, a tag alone as the version); chart kubeVersion and stored-manifest checks are not assessed for these charts. |
TestGitOpsArgoApplicationSingleSource TestGitOpsArgoApplicationMultipleSources TestGitOpsArgoSkipsOtherClusters TestGitOpsFluxHelmRelease TestGitOpsFluxReadsTheNewestServedAPIVersion TestGitOpsFluxChartRefOCIRepository TestGitOpsFluxOCIRepositoriesAreListed TestGitOpsFluxMissingOCIRepositoryIsUnresolved TestGitOpsFluxOCIRepositoryErrorIsInTheReason TestGitOpsListsInPages TestGitOpsIgnoresImplausibleChartNames TestGitOpsInvalidTargetNamespaceIsDroppedNotPushed TestGitOpsChartVersionIsEvidenceOnlyWhenExact TestGitOpsArgoSourcesReplaceSource TestGitOpsFluxChartPathFromGitRepository TestArgoInClusterSpellings TestGitOpsDeployedIngressNginxIsFoundEndToEnd TestGitOpsFluxTargetNamespaceApartFromStorageIsTwoInstalls TestGitOpsArgoNativeOCIChartSingleSource TestGitOpsArgoNativeOCIChartMultiSource TestGitOpsArgoOCISourceThatNamesNoChartIsNotRead TestGitOpsArgoRefSourceIsNotAChart TestGitOpsArgoOCISourceWithAPathIsStillReadAsTheRepositoryChart TestGitOpsArgoNativeOCIChartFeedsAddOnDetection TestGitOpsFluxOCIRepositoryVersionFollowsFluxPrecedence TestGitOpsOCIChartNameDropsADigest TestEvaluateGolden |
| HE-07 | A cluster with no Helm release that shows Argo CD or Flux (its CRD is served, or a workload carries its tracking label or annotation) reports helm as partial, naming the tool in skipped and saying why in the reason, so chart-derived checks read as not assessed, never clean. Every chart read from an Argo CD Application, a native OCI source included, is the same gap whether or not the cluster also has Helm releases, since helm template leaves it no release (Argo CD is usually installed with Helm, so the mixed cluster is the common one); Argo CD Applications that name no chart add none while releases exist. A Flux whose HelmRelease list is served and empty, or whose HelmReleases all deploy elsewhere (a kubeConfig, a target that is not a namespace name), deploys no chart here and is no gap; Flux's helm-controller leaves release Secrets, so Flux is a gap only while the cluster shows no release. A resource the role cannot read, or that is not served, degrades the same way or not at all; it never fails the capability. |
TestGitOpsMarkersWithoutHelmReleasesAreAGap TestGitOpsArgoGapExplainsHelmTemplate TestGitOpsWorkloadMarkersWithoutHelmReleasesAreAGap TestGitOpsUnreadableWorkloadsAreNotAGap TestGitOpsWorkloadsAreNotListedWhenHelmReleasesExist TestGitOpsAbsentChangesNothing TestGitOpsForbiddenDegradesToPartial TestGitOpsForbiddenToolKeepsTheOthersCharts TestGitOpsNotServedIsNotAFailure TestGitOpsKeepsHelmUnavailable TestGitOpsWithoutDynamicClientStillReportsTheGap TestGitOpsFluxOCIRepositoryErrorIsInTheReason TestGitOpsDiscoveryFailureIsWorded TestGitOpsArgoApplicationWithoutChartsIsNoGapWhenReleasesExist TestGitOpsFluxWithoutHelmReleasesIsNoGap TestGitOpsFluxOnlyForeignHelmReleasesIsNoGap TestGitOpsFluxOnlyInvalidHelmReleasesIsNoGap TestGitOpsFluxForeignBesideLocalHelmReleaseIsAGap TestGitOpsHelmUnavailableReasonSaysTheChartsWereRead TestGitOpsArgoNativeOCIChartSingleSource TestGitOpsArgoNativeOCIChartWithoutReleasesIsOneGap |
| HE-08 | A chart repository URL read from an Argo CD Application or a Flux OCIRepository is recorded without userinfo, query string or fragment, whether or not it parses (an scp-like Git address loses its user too), so a credential in its userinfo, query or fragment never reaches the inventory, the server's store or the CLI's output; a token embedded in the path, as in Cloudsmith's dl.cloudsmith.io/<token>/..., cannot be told from a path and is recorded. An OCI reference pinned by digest keeps its digest. The cut does not rely on URL parsing: everything up to the last @ is dropped first, so a token or password containing /, ? or # does not leak either, and a URL with a ? or # before its last @ (a query such as ?user=ci@example.com&token=..., which Helm, curl and git send) is recorded as empty; control characters are removed and a URL left with no host is recorded as empty. The chart name read from an OCI URL, an Argo CD native OCI source's included, is not polluted by its query. |
TestGitOpsArgoNativeOCIChartRedactsCredentials TestRedactRepoURL TestRedactRepoURLCredentialWithDelimiter TestGitOpsRepoCredentialsNeverReachTheInventory |
| HE-09 | More evidence never lowers a Helm manifest finding's severity (#236): an object the live scan also flags is left to the live finding only when that finding is at least as severe (a removed-api blocker, or a removed-api warning when the API is removed in the minor after the target); a live deprecated-api info leaves the release's deprecated-api warning in place, so the score and --fail-on warning are those without the live object. |
TestHelmManifestWarningSurvivesLiveInfo TestHelmManifestDedupesEquallySevereLive TestEvaluateGolden |
| HE-10 | The collectors send nothing the server refuses (#268). A Helm release's name comes from its name label, a free label value; a label that is not an RFC 1123 subdomain is not the release's name, and the release is recorded under the name its object carries (the apiserver validated it). A release whose chart metadata (name, version, appVersion, kubeVersion) has a value over 16 KiB is not decodable (a cut kubeVersion would be a narrower constraint, not the chart's), and a stored manifest object whose name or namespace is no identifier, or a namespace key of a count that is not a namespace name, is dropped from the release's API usage, the object counted in objectsOmitted. An add-on's version is read from an image tag, which a pod's creator chooses: a "version" over 128 bytes is no version (the install is kept, versionless), and an add-on install with any string over 16 KiB (however it got there) is dropped in one pass, named in addons as v1 pods so the gap stays required. Whatever is left out is named: helm is partial and skipped names the release (namespace/name, an invalid name quoted and cut to 40 bytes, at most 20 named and the rest counted), never clean. The same repair covers every list of the inventory (Inventory.Conform): an element that still fails the server's checks is dropped (at most 256 per inventory: each drop validates it again) and named in its capability, which stays absent when the collector never reported it. Every shape a collector is known to produce (releases, manifest objects, add-on installs, images) is repaired in bulk, with no count limit, so one hostile Secret, ConfigMap or Pod, or any number of them across namespaces, cannot make the server refuse (422) every push of a cluster; the 256-drop net under them is for shapes not yet known, and an inventory with more than that to drop is not pushed (AG-18). |
TestCollectHelmReleaseLabelThatIsNoNameFallsBackToTheObjectName TestCollectHelmChartMetadataOverTheLimitIsAGapNotARefusedPush TestCollectHelmManifestObjectWithAnInvalidNameIsLeftOutAndNamed TestCollectAdversarialClusterIsAdmitted TestConformLeavesAGenuineInventoryAlone TestConformRepairsWhatAdmitRefuses TestConformKeepsAnUnavailableCapability TestConformLeavesAnUnreportedCapabilityAbsent TestConformFallbackIsBounded TestConformBoundsWhatItNames TestConformReturnsWhatItCannotRepair TestConformQuotesAndCutsWhatItNames TestConformDropsEveryAddOnWithAStringOverTheLimitInOnePass TestCollectAddOnsWithAHostileImageTagInManyNamespacesAreAdmitted |
| HE-11 | The agent remembers a refused OCIRepository list (403) across ticks and nothing else: a list that failed another way (a timeout, a 5xx, a 429) is asked for again on the next call and does not clear an earlier refusal either. | TestGitOpsCacheNeverRemembersANon403Error TestGitOpsListFailingOtherwiseIsAskedForEveryCall TestGitOpsRefusedOCIRepositoryListsAreRememberedAcrossTicks |
Version skew¶
| ID | Claim | Proven by |
|---|---|---|
| SK-01 | Kubelets that would fall outside the skew policy after the upgrade are blockers, ones already outside it warnings, and kubelets newer than the apiserver violations. | TestEvalSkewPostUpgradeOnlyBlocker TestEvalSkewCurrentViolationWarnsAndBlocksPostUpgrade TestEvalSkewKubeletNewerThanAPIServer TestEvalSkewKubeletBehindNewestHAApiserver TestEvalSkewWithinPolicyNoFindings |
| SK-02 | Control-plane skew covers HA apiserver spread, controller-manager, scheduler and kube-proxy, read from per-architecture images too (GKE's kube-proxy-amd64) and from RKE2's hardened-kubernetes, which runs every component; a component pod whose version cannot be read (digest-only, a tag that is not a version) makes the versions capability partial, naming it, never silent; that gap is required, so the verdict is unknown, when upstream would have told the version (an image named like the component, or any kube-apiserver, kube-controller-manager or kube-scheduler pod), and optional for a kube-proxy on a vendor image of another name (Oracle OKE's digest-pinned oke-public-kube-proxy), which leaves every listed platform's verdict ready (#169); scheduler-plugins' kube-scheduler is read by the Kubernetes minor its v0.<minor> tag is built on, not as Kubernetes v0; a managed control plane that hides its pods yields no false finding. |
TestEvalControlPlaneSkewHASpread TestEvalControlPlaneSkewCtrlMgrNewerIsBlocker TestEvalControlPlaneSkewSchedulerBehind TestEvalControlPlaneSkewKubeProxy TestEvalControlPlaneSkewEmptyControlPlaneNoFindings TestCollectVersionsControlPlane TestCollectVersionsUnreadableControlPlaneVersionIsPartial TestCollectVersionsRKE2HardenedKubernetes TestComponentImageTagUnreadableReasons TestCollectVersionsPlatformImagesVerdict TestEvaluatePartialAndAddOnGaps |
| SK-04 | Managed-cluster version strings (EKS, GKE, k3s, RKE2, OpenShift suffixes) parse. | TestParseVersion TestParseVersionObservedGitVersions TestCollectVersionsManagedClusterEmptyControlPlane |
| SK-06 | A cluster whose Node list is empty is not read as clean on kubelet skew and node runtimes: versions is partial (reason "no nodes listed: kubelet skew and node runtimes not assessed", nodes in skipped), and that gap is required on a live cluster, as a forbidden Node list is, so the verdict is unknown (#174). |
TestCollectVersionsNoNodesIsPartial TestCollectVersionsNoNodesAndUnreadablePodReportBoth TestCollectVersionsNodesForbiddenKeepsEarlierFields TestEvaluatePartialAndAddOnGaps TestEnvtestMatrix |
| SK-07 | Each kube-proxy is recorded with its node and not deduped across nodes; one more than 3 minors (2 before 1.25) older or newer than the kubelet on its own node is a warning keyed version-skew/kube-proxy-kubelet/<node> citing the skew policy; a kube-proxy with no node, or on a node not listed, is not paired, and an inventory from before the field evaluates as it did (#148). |
TestCollectVersionsKubeProxyRecordsNodeNotDeduped TestEvalKubeProxyKubeletSkew TestEvalKubeProxyKubeletSkewKeysAndOldInventories TestEvaluateGolden TestValidateIdentifiersRefusesInvalid |
| SK-05 | The policy follows upstream: n-3 kubelets from 1.28, n-2 before. | TestDefaultSkewPolicy TestSkewLegacyComponentsAllowTwoMinors |
Upgrade plans¶
scan --plan (#78) came after the audit; its rows are UP-.
| ID | Claim | Proven by |
|---|---|---|
| UP-01 | scan --plan judges the cluster at every minor from the one it runs (a live scan's oldest kube-apiserver, or --from with --files) to --target, and lists each finding in full at the first upgrade it affects, with the upgrade where its severity changes (a warning at one upgrade and a blocker at a later one shows both); a 1.31 to 1.36 plan is five upgrades. |
TestPlanGolden TestPlanSeverityTransitions TestScanPlanTable TestScanPlanMarkdown TestScanPlanJSON TestScanPlanFiles |
| UP-02 | Hops are one minor each unless the knowledge base has a cited upgrade step that skips minors; it ships none. | TestHopTargetsDefault TestHopTargetsCustomPath |
| UP-03 | --plan never changes the verdict, score or exit code: the plan's last upgrade has the blockers, score and verdict of the report at --target, over hundreds of random inventories; ignore rules apply to every upgrade. |
TestPlanFinalHopMatchesEvaluate TestScanPlanLastHopIsTheReport TestScanPlanTable TestScanPlanFiles |
| UP-04 | The upgrade-plan guide's table and JSON examples are real scan --plan output. |
TestDocsUpgradePlanExample |
Verdict, score and exit codes¶
| ID | Claim | Proven by |
|---|---|---|
| VS-01 | engine.Evaluate is pure: the same inventory, knowledge base, target and time give the same bytes, whatever the order of the inventory's rows, a CRD's spec.versions and a capability's skipped list included (#165, #367). The server's dedup hash is over the same order-free form, so a push that only reorders them is a duplicate. |
TestEvaluateGolden TestEvaluateOrderInvariant TestEvaluateOrderInvariantSkipped TestEvaluateDeepShuffleInvariant TestEvaluateKeepsCRDVersionOrder TestCanonicalOrdersCRDVersionsAndSkipped TestIngestDuplicateWhenCRDVersionsReordered TestScanBaselineGolden |
| VS-02 | score = max(0, 100 - min(75, 25 x blockers) - min(20, 5 x warnings)); info findings are never scored. |
TestScore TestRescore TestEvaluateGolden |
| VS-03 | ready in every output is verdict == "ready": blocked on any blocker, unknown when a required check was not assessed, ready otherwise. The README, the verdict page and the architecture never say ready = (blockers == 0), and the README names unknown and --allow-incomplete. |
TestEvaluateVerdict TestTargetStatusFromReportVerdict TestScanGateOnUnknownVerdict TestDocsReadinessContract |
| VS-16 | The table and markdown reports end with a static line saying that field-level removals other than in-tree volume plugins (which are checked, #351) are not checked by this version, and the README, Verdict and score and Knowledge base say so; the line changes no score, verdict, exit code or JSON, which stays as it was (#331). | TestReportsStateWhatIsNotChecked TestWriteTableNoFindings TestWriteMarkdownNoFindings |
| VS-04 | Severity is set per category, as the verdict page's table says, and only some categories depend on the target: an add-on past end of life blocks whatever the target, a node runtime past end of life and kb-stale only warn, and a controller-manager or scheduler newer than the apiserver blocks whatever the target. Every category the engine defines has a row there. |
TestEvalAddOnsEOLBlocker TestEvalAddOnsNodeRuntimes TestEvalKBStale TestEvalControlPlaneSkewCtrlMgrNewerIsBlocker TestDocsListEveryCategory |
| VS-05 | Exit codes: 0 below the --fail-on threshold, 1 on a scan error (an unreachable or unreadable cluster included), 2 when the gate fails. |
TestScanFailOnExitCodeMapping TestScanPipelineErrorIsExitOne TestScanUnreadableClusterIsError e2e:unreachable_scan_exits_1 e2e:eol_ingress_nginx_blocks |
| VS-05b | A report that cannot be written (stdout on a full disk) is an error, exit 1, in every output format, also when the gate would have failed. | TestWriteReportPropagatesWriteErrors |
| VS-06 | --fail-on blocker, warning or never; an unknown verdict fails the gate unless --allow-incomplete, which never excuses a --target that is not an upgrade (only --fail-on never, which always exits 0, passes one: scan --help and the generated CLI reference say so); an invalid value is an error. |
TestScanGateOnUnknownVerdict TestScanRejectsBadFailOn TestScanBaselineStillGatesIncomplete TestScanTargetNotAnUpgrade |
| VS-06b, VS-07, VS-08 | A 403, timeout or discovery gap on part of a capability is a partial gap, shown in every output (JSON, table, Markdown, SARIF, the ClusterReadiness status). It is required, so the verdict is unknown and never ready, when what it skipped includes an API removed at or before the target, or, for versions on a live cluster, a component whose version upstream would have told (SK-02), or when no Node was listed (SK-06); add-on detection is required on a live cluster, and stays so when only the pod list is denied: add-ons are then matched from the Helm releases, GitOps chart sources and IngressClasses that were read (a chart-installed EOL ingress-nginx still blocks) and the capability is partial (#199, #70); a partial volumes on a live cluster that skipped the PersistentVolume or StorageClass list is required when a removed plugin that list could show is removed at or before the target (from 1.22 in the shipped knowledge base), and one that skipped only pods is optional, addons carrying it (#374). A role narrower than the chart's (no pods, Secrets or ingresses; or no PersistentVolume and StorageClass reads, over an rbd PersistentVolume at 1.31) reads unknown, naming each gap, and its ClusterReadiness Ready condition is Unknown, where the full role reads blocked. |
TestCollectNarrowRoleIsNeverReady TestAssessmentGapsVolumesPartialRequired TestCollectLiveVolumesListFailedIsRequired TestScanRefusedVolumeListsHidingRBDIsNotReady TestCollectAddOnsPodsForbiddenStillMatchesHelmReleasesAndIngressClasses TestCollectAPIUsageForbiddenIngressesIsPartialNamingIngressAPIs TestForbiddenPodSecurityPolicyListIsARequiredGapAtRemoval TestEvaluatePartialAndAddOnGaps TestCollectVersionsNoNodesIsPartial TestWriteJSONPartialGap TestWriteTablePartialGaps TestWriteMarkdownPartialGaps TestWriteSARIFReportsGaps TestStatusFromReportsPartialGaps |
| VS-09 | A live scan in which every capability failed is an error, not "no findings"; --context is honoured. |
TestScanUnreadableClusterIsError e2e:unreachable_scan_exits_1 TestScanIntegration_KindEOLIngressNginx |
| VS-09b | A stalled or unresponsive API server cannot hang a live scan (#94): each request is bounded by --request-timeout (scan and agent, default 30s, 0 = no per-request limit), /version and discovery honour the scan's deadline, and each collector step runs under its own share of the 5-minute budget, so one stalled step degrades only its own capability, its reason naming the step deadline. |
TestCollectStalledStepDegradesOnlyItsCapability TestCollectStalledMetricsEndsAtRequestTimeout TestCollectVersionsHonoursContextOnStalledVersion TestCollectAPIUsageHonoursContextOnStalledDiscovery TestRunStepsSharesTheDeadline TestScanRequestTimeoutFlag TestScanRESTConfigAppliesRequestTimeout TestAgentRequestTimeoutFlag |
| VS-09c | A live scan's report names the kubeconfig context and the API server it read, in the table and markdown headers and as JSON kubeContext and apiServer; the server is reduced to scheme, host and port, so kubeconfig credentials, proxy paths and token queries never reach a report. |
TestScanNamesTheCluster TestAPIServerURL TestWriteTableNamesTheCluster TestWriteMarkdownNamesTheCluster |
| VS-10 | --target takes a minor such as 1.36; 2.0 and other typos are rejected, and so is a minor below 1.16, the oldest the knowledge base covers (pinned to the dataset's earliest removal), which is what YAML makes of an unquoted target: 1.30 (the number 1.3) and would read ready. The refusal says to quote it, in scan --target (exit 1, live or --files), the agent's --targets and spec.targets (a skipped entry's note in status.notAssessed carries the message), serve --targets, the fleet and per-cluster target parameters (422) and the gate's target (400, not the gate's own 422); a cluster's own version has no floor, a --from with --files --plan included (a 1.15 cluster plans). The YAML hint is for minors 1 to 9, not 1.0. The Action validates ^[0-9]+\.[0-9]+$ and the same floor before it downloads anything, and a test pins its floor to the dataset's, as another pins the floor in the Action's metadata and README, the OpenAPI document and these docs, and scan --help computes it (#237). |
TestParseTarget TestParseTargetFloorHintOnlyForTruncatedMinors TestAdmitServerVersionHasNoFloor TestParseClusterVersionHasNoFloor TestScanPlanFilesFromHasNoFloor TestOldestCoveredMinorIsTheFirstRemoval TestActionFloorIsTheKnowledgeBaseFloor TestStaticFilesNameTheKnowledgeBaseFloor TestScanHelpNamesTheKnowledgeBaseFloor TestScanRejectsBadTarget TestScanRejectsNonMajorOneTarget TestScanRejectsTargetBelowTheKnowledgeBase TestScanRequiresTarget TestGateErrors TestGateRefusesTargetBelowTheKnowledgeBase TestConfigRejectsTargetsBelowTheKnowledgeBase TestResolveTargetsSkipsTargetsBelowTheKnowledgeBase TestServeRejectsBadTargets TestNewRefusesExtraTargetsBelowTheKnowledgeBase TestFleetMatrixBadTargets hack/action_test.sh |
| VS-10b | On a live cluster, a --target at or below the minor the oldest kube-apiserver runs (a downgrade or the same minor; a typo such as 1.4 is below the knowledge base and refused up front, VS-10) is a required target gap: the verdict is unknown and scan exits 2, even with --allow-incomplete. Files mode has no cluster version, so no such gap. --fail-on never always exits 0, this included. |
TestEvaluateTargetNotAnUpgrade TestScanTargetNotAnUpgrade |
| VS-11 | A target newer than the knowledge base is allowed and reported with a kb-stale warning and a required kb-coverage gap, so the verdict is unknown: on kind's newest minor, a vanilla cluster scanned at the next one exits 2, and 0 with --allow-incomplete, and the agent's default ClusterReadiness names the gap and, with no blocker, is unknown with Ready Unknown/NotAssessed (#130). |
TestEvalKBStale TestEvaluateUnknownKeepsScoreAndKBStaleWarning TestTickDefaultTargetPastKBHorizonIsUnknown e2e:past_horizon_is_unknown e2e:cr_reports_kb_coverage_gap e2e:cr_unknown_once_blockers_accepted |
| VS-12 | Findings are sorted by severity, then category, then title. | TestSortFindings TestEvaluateGolden |
| VS-13 | Finding keys are unique within a report: control-plane skew is keyed version-skew/<component>-newer or -behind, so two skew findings never share a key, and a --baseline counts a finding whose severity rose as new. |
TestControlPlaneSkewKeysUnique TestEvaluateGolden TestBaselineSkewEscalationIsNew TestBaselineSeverityIncreaseIsNew |
| VS-14 | Each team's rollup (the JSON report's and the API's teams, the dashboard, the HTML export) has a verdict: blocked by a blocker of its own or an unattributed one, which cannot be ruled out as the team's; otherwise unknown when the report has a required not-assessed gap; otherwise ready. A team's ready is verdict == "ready", so no team reads ready while the cluster is unknown or blocked by a blocker no team owns; another team's blocker leaves it ready (#196). Its score, blockers and warnings count only its own findings; a team with no findings is not listed, and there is no per-namespace score. The fleet rollup (/api/v1/fleet/teams) gives each team the worst of its verdicts across clusters (blocked over unknown over ready), and the Teams page shows it and never colours a blocked or unknown team green (#243). The bucket of findings no team owns is (unattributed), in the API, the CLI and the dashboard: a name no label value can take and serve --team-map refuses (the map does not load), so a team named unattributed is its own row and no team's score or blockers are overwritten by the bucket's. |
TestParseTeamMapRejectsBadInput TestRenderTeamScoresNeverOverwritesATeamOfTheBucketsName TestTeamScores TestTeamVerdictFollowsTheCluster TestWriteJSON TestExportHTMLGolden TestFleetTeamsCarryTheTeamVerdict TestRenderTeamScores TestTeamScoresForOutputRealTeamNamedUnattributed make web-test |
| VS-15 | Every change to the JSON report's contract since v0.1.1 is in the changelog. Round 3 built v0.1.1 from its tag's source and diffed its JSON and SARIF output against main's on a PSP fixture: every difference is in CHANGELOG.md (ready semantics, schemaVersion 1, the SARIF ruleId, (unattributed), added keys). The check was by hand; the release preflight only refuses an empty section. |
hack/check-changelog.sh; the key diff: not automated, #99 |
Offline scans, SARIF, ignore rules and baselines¶
| ID | Claim | Proven by |
|---|---|---|
| FS-01 | scan --files reads rendered manifests offline and assesses API usage and add-ons; version skew, deprecated-API requests and Helm releases are reported as not assessed ("files mode"). |
TestCollectFiles TestManifestInventoriesAreFilesSource TestWriteTableFilesModeGolden TestScanFilesNoManifestsIsExitOne TestCollectFilesAssessesAddOns |
| FS-02 | --files reads *.yaml, *.yml and *.json, expands kind: List, and turns documents that do not parse into warnings, not errors. |
TestCollectFilesExpandsLists TestCollectFilesListVariants TestCollectFilesUnrenderedTemplateDoc TestScanFilesWarnsOnInvalidFiles TestCollectFilesSkipsVCSAndDependencyDirs |
| FS-02b | --files and /gate decode a stream as kubectl does: concatenated, pretty-printed and adjacent JSON objects, a YAML node after the first, a duplicated apiVersion or kind (the last value counts, with a warning), and typed Lists with untyped items; the two give the same findings. |
TestParseManifestStream_ConcatenatedJSON TestParseManifestStream_YAMLTrailingNode TestCollectFilesDuplicateKeys TestAdversarialCorpus TestGateParityWithScanFiles |
| FS-03 | --output sarif places each result on its file and line, and GitHub code scanning accepts it. An alert reaches a pull request's diff only when that file is committed to the repository: the README's helm template --output-dir rendered example files its alerts under an uncommitted path, in the Security tab but never on the diff, and the CI gate page and the Action's README say so. |
TestWrite TestWriteGitHubAcceptable TestScanFilesSARIFLocations ci:action TestDocsRound2Gaps |
| FS-04 | The server gate example on the CI gate page, run as written, fails the step on a removed API and still writes the SARIF report: /gate defaults to fail-on=blocker (never always answers 200), and ?path= gives the SARIF results a file location. |
TestGateDocsExample TestGateFailOn TestGateSARIFPath e2e:documented_gate_command e2e:gate_fails_removed_api (the same command, run on kind against the agent's cluster) |
| FS-05 | The CLI page says what helm template renders without a cluster (Helm's built-in .Capabilities, not the cluster's or the target's), how to pin them (--kube-version, --api-versions), and where a release's installed manifests are (helm get manifest). |
TestDocsHelmCapabilities |
| FS-07 | --files finds add-ons in the container and init-container images and the labels of Pod, Deployment, DaemonSet, StatefulSet, ReplicaSet, Job and CronJob pod templates (List items included) and in IngressClasses, matched as a live scan matches them, so an end-of-life add-on in a render fails the gate (#47). Images injected at admission time are not in the manifests and are not seen. |
TestManifestPodTemplates TestManifestAddOnEvidenceOnlyFromWorkloads TestCollectFilesAssessesAddOns TestCollectFilesAddOnNamespaceUnset TestScanFilesAssessesAddOns |
| FS-08 | /gate without ?cluster= judges add-ons in the posted manifests as scan --files does: the same verdict and findings on every regression-corpus input, a workload with an add-on image included. |
TestGateParityWithScanFiles |
| FS-06 | A known finding can be accepted with an expiring, reasoned ignore rule or an object annotation, and --baseline gates only on new findings. |
TestApplyCategoryRule TestApplyAnnotations TestApplyExpiredRuleWarnsAndDoesNotSuppress TestBaselineMark TestScanWriteBaselineRoundTrip TestScanFilesSARIFSuppressedOmitted ci:action |
| FS-06b | /gate suppresses as scan does, with the same code: upgradescope.basit.engineer/ignore annotations, and the ignore rules of a .upgradescope.yaml sent in config, parsed by the same code as scan --config (an expired rule warns and stops applying; an invalid config, one over 32 KiB or config given twice is a 422). Suppressed findings count toward neither the verdict nor fail-on; the JSON answer lists them with suppressedCount, SARIF has no result for them (see FS-15), JUnit as skipped, Code Quality leaves them out; the verdict, findings and suppressions match scan --files --config. With ?cluster=, what the manifests introduce is decided after suppression: a posted object accepted by its annotation or an object-scoped rule does not make the cluster's remaining objects at that key the pull request's. The suppressions guide's example works as written. |
TestGateAnnotationSuppresses TestGateConfigRules TestGateSuppressedCIFormats TestGateClusterSuppression TestGateClusterSuppressedManifestObject TestGateClusterSuppressedCustomResource TestGateDocsConfigExample TestGateConfigBounds TestGateParityWithScanFiles TestParseConfig |
| FS-15 | GitHub code scanning ignores a SARIF result's suppressions and opens an alert for every located result, so a finding accepted by an ignore rule or annotation is not a SARIF result: scan --output sarif, /gate?format=sarif and the Action's sarif-file leave it out, with one run-level note that counts the located ones ("N findings accepted by ignore rules or annotations are not uploaded; see the JSON report") and run.properties.suppressed. The JSON report, the step summary and JUnit still list them with their reasons. An unsuppressed finding is still a result, and so is a baseline-unchanged one (baselineState: unchanged), which code scanning opens an alert for: the guide says so and no page says accepted findings show as dismissed. GitHub itself is asked: a workflow_dispatch run of the action job on a branch other than the default, with upload-fixture-sarif, uploads the suppressed fixture's SARIF under its own category and checks through GET /code-scanning/alerts that the branch has no open alert in it, with the unsuppressed fixture's two open blockers as the control (not run on every push; hack/ci-sarif_test.sh pins the step's condition and the check). |
TestWriteSuppressedOmitted TestWriteNoSuppressedNoNote TestScanFilesSARIFSuppressedOmitted TestGateSARIFSuppressedOmitted TestGateSuppressedCIFormats TestDocsNoDismissedClaim ci:action hack/ci-sarif_test.sh |
| FS-09 | --output junit writes JUnit XML that validates against the Jenkins JUnit schema (the xUnit plugin's junit-10.xsd, vendored): one test per finding, failing exactly when the finding fails --fail-on, skipped when suppressed or unchanged since the baseline, and an error per required check not assessed; the output is stable across runs. A scan with no findings and every check assessed (a live scan) is one passing test, readiness/no findings; a clean --files scan is three skipped tests (not-assessed/deprecated-calls, helm, versions) and no passed test, which Jenkins still counts as tests (#308). |
TestJUnitGolden TestValidate TestWriteFollowsTheGate TestWriteTargetGap TestScanFilesJUnit TestWriteCleanReport TestScanFilesCleanJUnit TestDocsOtherCIJUnitCleanReport |
| FS-10 | --output gitlab-codequality writes a GitLab Code Quality report that GitLab's own entry schema (vendored) and documented requirements accept, every finding on its file and line (line 1 when only the file is known) or on the virtual path upgradescope/<key>, with fingerprints that stay the same across runs and when an object moves within its file. |
TestCodeQualityGolden TestWriteEntries TestWriteFileWithoutLine TestFingerprintsStable TestScanFilesGitLabCodeQuality |
| FS-11 | The exit code is the gate's whatever --output is, and /gate answers format=junit and format=gitlab-codequality with the same status and verdict as JSON, holding only what the manifests introduce. |
TestScanExitCodeIgnoresOutputFormat TestGateCIFormats TestGateCIFormatsClusterBaseline |
| FS-12 | Team attribution needs a cluster: scan --files reads no Namespace objects, so every finding of a files scan is unattributed (a rendered Namespace labelled team: attributes nothing), and --team-label with --files is refused with a message that says why, not the flag parser's generic one. The fleet, architecture and JSON report pages say so (#237). |
TestScanFilesAttributesNoTeamsAndRefusesTeamLabel |
| FS-13 | Scanning an unrendered Helm chart directory (scan --files <chart dir>) never reads ready: when files under a directory holding a Chart.yaml cannot be decoded and contain {{, one stderr hint names the chart and gives the helm template NAME <dir> --output-dir rendered command in place of a warning per file, and a required api-usage gap counts the templates not read, so the verdict is unknown and the gate exits 2; one template is enough. Templated files outside a chart are one count line and leave the verdict unchanged, unparseable non-template YAML is still warned about file by file, and a tree where nothing was read says so and names Helm (#335). |
TestScanUnrenderedChartIsUnknownWithOneHint TestScanUnrenderedChartOneTemplateJSON TestScanTemplatedOutsideChartIsOneCountLine TestScanBadYAMLStillWarnedPerFile TestScanFilesNoManifestsIsExitOne TestScanFilesUnassessedRemovedAPIIsUnknown |
| FS-14 | A scan with no --target says --target is required: the Kubernetes minor to upgrade to, e.g. --target <horizon> (the newest this build knows; see upgradescope version), the example being the knowledge base's horizon, which is also the example in the flag's help; exit 1 (#336). With no kubeconfig the live scan names the paths it looked at and offers --kubeconfig, --context and scan --files <dir> --target X, without client-go's KUBERNETES_MASTER hint (#337). A mistyped subcommand prints cobra's suggestions on lines of their own, with the typed name still escaped (#338). -o is the shorthand of --output (the format) on scan and version, and nothing else uses it (#339). |
TestScanMissingTargetNamesAnExample TestScanNoKubeconfigNamesPathsAndFilesAlternative TestUnknownCommandLayout TestOutputShorthandIsFormatEverywhere TestHorizonMatchesLoad |
GitHub Action¶
| ID | Claim | Proven by |
|---|---|---|
| AC-02 | Exit code 2 (findings at or above fail-on, or an unknown verdict without allow-incomplete) fails the step. |
ci:action hack/action_test.sh |
| AC-02b | sarif-file is set only when the scan produced a complete SARIF (exit 0 or 2), so the documented upload guard !cancelled() && sarif-file != '' skips the upload after a scan error (#197). |
hack/action_test.sh |
| AC-03 | version installs a release archive verified against its checksums.txt, or preinstalled uses the binary on PATH. |
hack/action_test.sh ci:action |
| AC-03b | With version unset, the Action used at a release tag ref (@vX.Y.Z or @vX.Y.Z-rc.N) runs that release, not GitHub's latest, which skips prereleases; at a full commit SHA it tries the release tags that git ls-remote --tags shows at that commit (an annotated tag peeled) newest first and runs the first whose release is published (probed with GET /repos/abd-ulbasit/upgradescope/releases/tags/<tag>, authenticated with the install step's token when it has one and the runner is on github.com, anonymous on GitHub Enterprise Server and GHE.com, where api.github.com rejects that host's token (401): a 404 is "not published"; on a 403 or 429, the rate limit of the anonymous probe, the tag's checksums.txt release download stands in for it: 200 is published, 404 is not, and anything else fails naming both statuses), fails, naming the tags and saying to pin a published version: vX.Y.Z or wait for the release workflow, when the commit has release tags and none is published (never latest, and never a suggestion of verify-provenance: false, #303), fails when that API answers anything but 200 or 404 (outside the rate-limit fallback), a 2xx other than 200 or a redirect included (the probe follows none, and curl -f exits 0 for them, so only 200 counts as published; the step cannot tell, and the message names the HTTP status), or runs latest with one ::warning naming why when no release tag points at the commit or the tag lookup fails; at another ref it runs latest; the ref counts only when github.action_repository is abd-ulbasit/upgradescope (case-insensitive), so a wrapping action's ref never picks the version; version: latest at a tag ref warns when the latest is older than the ref's release (#197). |
hack/action_test.sh |
| AC-05 | Inputs never reach a run: script unquoted, and are validated before use. |
hack/action_test.sh |
| AC-05b | Text that reaches the log from an input or the scanned tree cannot start a workflow command (no forged annotation, no ::add-mask::): an input value in a validation error has %, CR and LF escaped, and the gate pass's stderr, which repeats the path input and file names, and the Markdown report, which repeats the names of files with findings, are printed with a | in front of every line (the runner strips all leading Unicode whitespace before it looks for ::) and every ##[ (the runner's legacy command form, read anywhere in a line) written # #[ (#197). |
hack/action_test.sh |
| AC-06 | Before the Action installs a downloaded archive it checks, by default (verify-provenance: true), that its sha256 matches the same release's checksums.txt and, for every release except v0.1.0 and v0.1.1 (an exact list of the two releases that predate provenance, never a version comparison: a prerelease suffix such as -beta is verified, and v0.2.0-rc.2 and later release candidates are verified like releases), that this repository's release.yml built it for that release's tag on a GitHub-hosted runner: gh attestation verify <archive> --repo abd-ulbasit/upgradescope --signer-workflow abd-ulbasit/upgradescope/.github/workflows/release.yml --source-ref refs/tags/<tag> --deny-self-hosted-runners (the install step passes github.token as GH_TOKEN), or, without gh (or with a gh older than 2.68, which lacks a flag of that call: --source-ref is first in gh 2.68.0, cli/cli#10308; the Action asks the gh once, with gh attestation verify --help, and requires every flag the call passes to be listed, rather than matching an error) and with cosign, cosign verify-blob of checksums.txt against the release's checksums.txt.sigstore.json for the identity https://github.com/abd-ulbasit/upgradescope/.github/workflows/release.yml@refs/tags/<tag> and GitHub's OIDC issuer. An archive replaced together with a checksums.txt that matches it fails, as does an attestation for another tag, a missing bundle, any other gh failure (cosign is not tried after it), or no usable gh (2.68+) and no cosign on PATH (the error says "gh 2.68 or later (or cosign) is needed to verify provenance", whether the gh is too old or absent, never a replaced asset, and how to opt out); then nothing is put on PATH, and the verifier's output is logged behind | (#244). v0.2.0-rc.2 is the first release with attestations (gh attestation verify of its linux/amd64 archive passes at refs/tags/v0.2.0-rc.2; v0.2.0-rc.1 was never published as a release). v0.1.0 and v0.1.1 (the only two stable releases before v0.2.0) and verify-provenance: false are checked against checksums.txt only, with a ::warning; neither detects a tampered release. With version: latest (explicit, or the default at a branch, @v0 or a commit SHA with no release tag) resolving to v0.1.0 or v0.1.1, from any ref, the step fails before downloading anything and never falls back to go install, naming the tag (GitHub's latest is v0.2.0 or newer, so such a latest was moved back), unless verify-provenance: false; an explicit version: v0.1.1 is unchanged. With verify-provenance: true, an archive that cannot be downloaded, or a latest that does not resolve, fails the step with curl's final error (not its retry warnings) and installs nothing: there is no source-build fallback. Only verify-provenance: false falls back to go install (with Go set up), with a ::warning that the build is unverified (module checksum database only); version: preinstalled is not provenance-checked. CI's action job installs version: v0.2.0-rc.2 on both runner OSes with a gh spy on PATH and asserts that the action asked gh for attestation verify --help once, then ran one passing gh attestation verify for the archive (hack/action_test.sh runs that check against run.sh's gh path offline), and that the release archive was installed; its version: latest install resolves to GitHub's latest stable release, which from v0.2.0 on is attested and verified like any other; no CI step asserts which path it takes (#127). The install needs no attestations: permission in the consumer's workflow: this repository's attestations are public. |
hack/action_test.sh |
| AC-07 | Run as a consumer runs it, on Linux and macOS, the Action installs the latest release and fails on a fixture holding a removed API. | ci:action |
| AC-08 | The Action sets score, verdict, blockers and the other outputs, annotations, and a job summary listing suppressed findings and the baseline diff. |
hack/action_test.sh ci:action |
| AC-09 | allow-incomplete: true passes scan --allow-incomplete to the gate (#130): a target past the knowledge base's horizon, with no blocker, passes with the verdict output still unknown, and blockers still fail; false or unset passes nothing, so the unknown verdict fails the step; any other value is refused before anything is downloaded. |
hack/action_test.sh |
Agent and ClusterReadiness CRD¶
| ID | Claim | Proven by |
|---|---|---|
| AG-02 | One tick at start, then every --interval (minimum 1m) with jitter. |
TestRunFirstTickThenGracefulStop TestConfigIntervalMinimum TestAgentIntervalMinimum TestJitterBounds |
| AG-03 | A tick has a deadline, and a stop mid-tick is not a failure. Collection gets the tick deadline minus a reserve (30s, or half the deadline when that is under a minute) kept for what follows, each on its own slice: the ClusterReadiness calls end by half the reserve before the deadline (a CRD check the tick retries, the one at startup having failed, by three quarters of it, so a hung check or a re-created CRD's wait for Established leaves the spec read and the status write at least a quarter, and a CRD Established within the check's part still gets that tick's status written), the upgradescope.basit.engineer/status-error marker then gets a quarter of the reserve of its own, and the push runs until the deadline. A collection that uses all its time still leaves the status written (or, when the write runs out its slice, the object marked) and the push sent; a stop mid-collection still cancels every phase, writing and pushing nothing. A last collector step that runs out its time names the step deadline in its reason (#238). |
TestRunTickHasDeadline TestTickTimeout TestRunStopMidTickIsNotAFailure TestTickReserve TestTickWithASlowCollectionStillWritesTheStatusAndPushes TestTickWithAStalledMetricsScrapeWritesTheStatus TestTickWithAStalledStatusWriteStillMarksTheCR TestTickRecreatingTheCRDStillWritesTheStatus TestTickWithAHungCRDCheckStillWritesTheStatus TestTickStoppedMidCollectionWritesNothing TestRunStepsLastStepNamesTheStepDeadline |
| AG-04 | With no server, the agent still writes the CRD status every tick, except a tick that could not read (or decode) the object's spec, or set its spec.targets to --targets: that tick writes no status, so no observedGeneration is claimed for targets it did not read, marks the object with upgradescope.basit.engineer/status-error, fails, and the next tick reads the spec again (#238). |
TestTickCRDOnlyModeNoPusher TestAgentIntegration_CRDStatusOnKind TestTickFailedSpecReadWritesNoStatus TestTickUndecodableSpecWritesNoStatus TestTickFailedSpecTargetsPatchWritesNoStatus TestTickFailedSpecTargetsPatchKeepsTheCreateError |
| AG-05 | Pushes happen only when the canonical inventory hash changes, retry transient failures with capped backoff, keep only the latest payload, and drop on a permanent 4xx. A redirect is a permanent failure that is never followed, and Retry-After on a 429 or 503 is honoured up to the 1m backoff cap, before the jitter below. Retry waits are jittered: the backoff step fully (anywhere from 0 to the step), a Retry-After as the least wait plus up to a quarter of it more (so one wait is at most 75s), so agents started together, answered with the same 503 and Retry-After, do not retry together (#238). A URL or token no request could carry is a permanent failure, not retried. | TestTickDedupsUnchangedInventory TestSnapshotHashIgnoresCollectedAt TestFlushRetriesTransientWithBackoff TestBackoffCapped TestFlushLatestOnlyBuffer TestFlushOther4xxPermanentDropsPayload TestFlush401DropsPayloadNoRetry TestFlushRedirectIsPermanentFailureNeverFollowed TestTickRedirectedPushFailsEveryTick TestFlushHonoursRetryAfter TestFlushRetryAfterOnEveryAttempt TestRetryDelayJitterBand TestFlushRetriesOfAgentsStartedTogetherSpreadOut TestFlushUnusableURLOrTokenIsPermanent |
| AG-33 | A snapshot whose push failed at a tick and stayed buffered (a transient failure, or the tick ran out of time) is retried between ticks, so a server that was not listening yet at the first tick, as in a combined install (server.enabled=true), gets the first snapshot at the first retry after it starts listening, not at the next tick (#386); the kind e2e's combined install (agent.interval=1m) requires that delivery before the agent's second tick line, or a first push that succeeded. Each retry is one push (its usual attempts) after 10s, then 20s, 40s and 80s, then every 2m, each ±20%. Retrying stops at the first delivery, which advances the content hash gate as a tick's push does (the next tick's unchanged inventory is not pushed again), at a permanent refusal (the snapshot dropped, as at a tick), or when the next tick is due, whose own snapshot replaces the buffered one; ticks keep their interval. The ticks and the retries run in one goroutine, so a retry never overlaps a tick and never sends an older snapshot after a newer one, and a stop ends a retry promptly, while it waits or while its request hangs. Each retry logs one line, buffered snapshot pushed (INFO) or buffered snapshot push failed (WARN, with pushError and retrying), never the token; a failed retry counts in upgradescope_agent_push_errors_total, as a failed tick push does. The tick line's consecutiveFailures counts the failure the line reports: failed pushes in a row on a line whose push failed (1 after the first), failed ticks in a row on a tick failed line, 0 otherwise; a delivered retry ends the push streak. |
TestLoopRetriesTheBufferedSnapshotBeforeTheNextTick TestLoopNextTickSupersedesTheBufferedSnapshot TestLoopStopEndsTheRetry TestLoopDoesNotRetryAPermanentlyRefusedPush TestBufferedRetryDelay TestObserverConsecutiveFailuresCountsTheReportedFailure TestObserverTickLogLevels e2e:first_snapshot_before_second_tick hack/e2e_test.sh |
| AG-06 | The status write retries on conflict. | TestWriteStatusRetriesOnConflict |
| AG-07 | With --manage-crd the agent converges the CRD to its embedded manifest and leaves foreign metadata alone; without it, it never touches the CRD. A startup check that fails or runs out its 30 seconds (other than a CRD that is not installed, which stops the agent) is retried every tick until it succeeds, so a hung apiserver at pod start does not hold the first tick back; meanwhile each tick line carries crdError at WARN and status.notAssessed leads with a note, without failing the tick (#238). |
TestEnsureCRDUpdatesExisting TestEnsureCRDNoWriteWhenInSync TestEnsureCRDPreservesForeignMetadata TestRunSkipsCRDWhenNotManaged TestTickRetriesTheCRDUpgradeUntilItSucceeds TestRunRetriesEnsureCRDOnTheFirstTick TestRunStartupCRDCheckIsBounded TestObserverTickLogLevels |
| AG-08 | The ClusterReadiness is created if absent and never overwritten; user-set spec.targets survive. |
TestEnsureObjectCreatesAndIsIdempotent TestTickHonorsSpecTargets TestTickNoSpecWriteWhenFlagTargetsMatch |
| AG-09 | Empty spec.targets means the next minor above the server; invalid entries are skipped and reported. |
TestResolveTargetsDefaultNextMinor TestResolveTargetsSkipsInvalidWithNote TestResolveTargetsFromSpec TestTickWritesStatusWithDefaultTarget |
| AG-10 | Status holds, per target, the score, verdict, counts and at most the top 20 findings, and its size is bounded: spec.targets takes at most 8 (schema maxItems, items at most 8 characters); an agent given more by a CR older than that limit evaluates the first 8 and lists the rest in status.notAssessed; finding titles and remediation are clipped to 512 and 1024 bytes as stored (JSON escapes counted: a < is six bytes, so a limit in characters let a title of them reach 1.5 MiB), the upgrade-path title lists its hops only up to 3 and notAssessed is capped. The worst case is under 1 MiB (about 273 KiB, the same for ASCII, accented, emoji, escaped and invalid text, measured as the apiserver stores it), where 1000 targets had made a 4 MB status the apiserver refused on every tick. |
TestTargetStatusFromReportTop20Cap TestTargetStatusFromReportCounts TestStatusFromReports TestManifestBoundsSpecTargets TestTargetStatusFromReportClipsLongText TestWorstCaseStatusStaysUnder1MiB TestJSONCostNeverUndercounts TestBoundNotAssessedSummarizesTheRest TestResolveTargetsCapsAtMaxTargets TestResolveTargetsBoundsSkipNotes TestConfigRejectsTooManyTargets TestTickWithThousandSpecTargetsWritesBoundedStatus TestEvaluateUpgradePathTitleIsBounded TestSchemaAgentTargetsAtMostEight TestSchemaAgentTargetsCapIsTheCRDs |
| AG-11 | ClusterReadiness is cluster-scoped, upgradescope.basit.engineer/v1alpha1, short name ucr, and serves kubectl get ucr and Argo CD health checks. Policy engines as consumers: example Kyverno and Gatekeeper policies (audit/warn by default, failing open on stale, missing (an empty object in the Kyverno CLI test) or unassessed data, in-cluster operations only) and a static Renovate preset, in examples/. |
TestManifestShape TestConstantsMatchManifest TestArgoHealthScriptAgainstWrittenStatus (the guide's Lua under gopher-lua, not in a live Argo CD); the examples: TestFixturesAreWhatTheAgentWrites TestPoliciesNameTheObjectTheAgentWrites TestReaderRolesAreReadOnly TestRenovatePresetMatchesTheRegistry ci:examples make examples-test (the Kyverno CLI, gator and Renovate's config validator on fixtures, not a live cluster or admission webhook) |
| AG-12 | Installed from the chart on a real cluster, the agent keeps a ClusterReadiness with a score and verdict, and agent.targets reaches spec.targets. The server's /history is a change log: a row is added when a snapshot or a re-evaluation brings a new result (verdict, score or finding keys), and an unchanged re-evaluation refreshes the row in place, team attribution included; the retention page says exactly what produces a row. |
e2e:cr_has_verdict e2e:upgrade_with_targets TestAgentIntegration_CRDStatusOnKind TestDocsRound2Gaps |
| AG-15 | --cr-name (chart agent.crName) names the object, and the write rule follows it. |
TestAgentCmdFlagDefaults TestAgentRefusesInvalidNames TestRenderedRBACCustomCRName TestSchemaCRName TestSchemaCRNameAgreesWithTheAgent |
| AG-16 | Settings that could never work stop the agent before it touches the cluster, naming the flag: a --server-url that is not an http:// or https:// URL with a host, a server token with whitespace or a control character inside it (surrounding whitespace is trimmed from the flag, $UPGRADESCOPE_SERVER_TOKEN and --server-token-file alike), --cluster-name "", and a --force-sync-every of 0 or less. A --force-sync-every at or below --interval means every tick, with a warning: it takes effect as nine tenths of the interval, the shortest gap the tick jitter (integer, never under that floor) leaves between two pushes, so a tick the jitter brings early still force-syncs. A push with no cluster name at all (no flag, no readable cluster UID) is skipped, failing the push with an error that names --cluster-name (#238). The chart's schema refuses every non-empty agent.serverUrl the agent would refuse at start (it may refuse more: a host of letters, digits, ., -, _ and ~, or a bracketed IPv6 address). |
TestAgentRefusesUnusablePushSettings TestAgentTrimsTheEnvironmentToken TestAgentForceSyncEveryFlag TestValidateServerURL TestValidateServerToken TestConfigRefusesUnusablePushSettings TestValidateForceSyncEvery TestRunForceSyncBelowTheIntervalMeansEveryTick TestForceSyncBelowTheIntervalPushesEveryEarlyTick TestJitterFloorIsMinTickSpacing TestTickSkipsThePushWithoutAClusterName TestSchemaServerURLAgreesWithTheAgent TestSchemaServerURL |
| AG-30 | The Go profiler is off unless --pprof-addr is set, serves /debug/pprof/ on a listener of its own (never on the health listener), and only on a loopback host and port: the agent refuses a port with no host, a wildcard, or a routable address before touching the cluster, naming the flag. |
TestValidatePprofAddr TestRunServesPprofOnlyOnItsOwnLoopbackListener TestRunServesNoPprofByDefault TestAgentPprofAddrFlag |
| AG-17 | --server-token-file is re-read when the file changes, so a rotated push token needs no restart: the next push after the check uses it (checked at most every 5 seconds, only when a push is made, the new value swapped atomically). A new file that is empty, unreadable, or holds a token with whitespace inside keeps the old token and logs an error naming the file and never the token; a file that is missing, empty or unacceptable at start stops the agent. A token from --server-token or $UPGRADESCOPE_SERVER_TOKEN is read once, as before. Safe under -race. |
TestPushTokenRotatesWithoutARestart TestPushTokenFileEmptyOrBadKeepsTheOldToken TestConfigServerTokenFile TestPushTokenRotationIsRaceFree TestRunnerPushesWithTheRotatedTokenFile TestAgentServerTokenFileFollowsRotation TestEmptyOrUnreadableFileKeepsTheOldValue |
| AG-18 | The agent never offers the server an inventory it refuses for good (#268). It conforms what its collector returns (Inventory.Conform: left out and named, as HE-10) before it writes the status and pushes, so both describe one inventory, and checks it with the server's own admission (Inventory.Admit) before a push; one that is still refused (a server version that is no Kubernetes version) is not sent, and the tick says push skipped: the server would refuse this inventory (422) instead of offering the same content every tick (reached only by a shape the collectors and Conform do not know: the shapes they know are repaired without limit, HE-10). A cluster with a release named Bad_Name, a manifest object Bad/Name, a 17 KiB image and a 19 KiB kubeVersion pushes a snapshot the server accepts (202), the genuine releases kept and the rest named in helm and addons. What the collector (collect.Options.OnConform) or the tick left out is logged once per tick as counts and kinds, never an identifier. |
TestTickPushesAnInventoryTheServerAccepts TestTickDoesNotPushAnInventoryNoRepairMends TestTickLogsWhatTheCollectorLeftOut TestTickLogsWhatConformLeftOut TestCollectReportsWhatConformLeftOut TestCollectAdversarialClusterIsAdmitted |
| AG-19 | The CRD group moved to upgradescope.basit.engineer from one the project never owned (#68), and the pre-v0.2.0 ignore and ignore-reason annotation keys keep working for one minor release: the scanner reads them when the object has no new key (the new key wins when both are set), accepts the finding, and says once, in one bounded line that names the first five objects and counts the rest, that the old keys are deprecated and read only until v0.3.0. An old-key object without a reason is refused as a new-key one is, and the warning names the keys it carries. The scanner never writes the old keys. |
TestReadIgnore TestApplyLegacyAnnotationsWarnDeprecated TestApplyManyLegacyAnnotationsWarnOnce |
| AG-20 | An agent that manages the CRD and finds the pre-v0.2.0 CRD still installed logs one WARN with the command that removes it, and never deletes it. A chart upgraded across the group move without installing the new CRD, which Helm does not do and the chart's agent may not, makes the agent exit at startup with an error that names the group move and the exact kubectl apply of the release's CRD manifest at the binary's version (a placeholder tag, explained, for a dev build or any Go pseudo-version, tagged base or not); with --manage-crd=false the same command is in every failed tick's error, which /readyz carries. The chart's NOTES show the command on upgrade, and say what the agent does without it as the chart runs it (exits at startup with agent.manageCRD, stays up unready without it, no pod with agent.enabled=false). |
TestRunWarnsOnceAboutTheLegacyCRD TestRunNoLegacyCRDNoWarning TestRunNamesTheGroupMoveWhenOnlyTheLegacyCRDIsInstalled TestRunNamesTheInstallCommandWhenTheCRDIsMissing TestRunDevBuildSaysWhichTagToInstallTheCRDFrom TestTickNamesTheInstallCommandWhenTheCRDIsMissing TestInstallHintRelease TestNotesNameTheCRDInstallCommandOnUpgrade TestNotesUpgradeBlockMatchesWhatTheAgentDoes TestInstallHintDevBuildSaysWhatTagToUse |
| AG-31 | The AllTargetsReady condition is False when any spec target is blocked (its message names the target and its blocker count), Unknown when none is blocked but any is not assessed or no target was evaluated, and True only when every target is ready; Ready keeps the first target's meaning, so a ready first target with a blocked second leaves Ready=True and AllTargetsReady=False (#349). The message is bounded at 1 KiB with 8 targets, and the Argo CD health check in the GitOps guide reads it (a later blocked target is Degraded). |
TestAllTargetsReadyCondition TestAllTargetsReadyLeavesReadyAlone TestWriteStatusSetsAllTargetsReady TestAllTargetsReadyMessageStaysBoundedAtMaxTargets TestArgoHealthScriptAgainstWrittenStatus |
| AG-32 | kubectl get ucr shows a BLOCKERS column, the first target's blocker count (described as such, pointing to AllTargetsReady), and the chart's crds/ copy of the CRD is the embedded manifest byte for byte (#349). |
TestManifestHasBlockersColumn TestChartCRDCopyMatchesManifest |
| RB-07 | An agent that cannot write its ClusterReadiness status does not leave the object showing a stale verdict unmarked: it patches the annotation upgradescope.basit.engineer/status-error (the time and a short reason of the failure; the chart already grants patch on the object) and clears it with the next successful status write (a status written whose marker could not be cleared is not a failed tick, and does not mark the object again); and once at least 3 ticks in a row have failed and the last success is older than 2 intervals + 12m (after the UpgradescopeAgentNotTicking alert's 2 intervals + 10m, at any interval, so a firing blocked alert never resolves with nothing else firing) its verdict, score, findings and capability gauges are withdrawn, while the last-success timestamp stays. If patch on the object is revoked too, no marker can be written and /readyz, the metrics and the UpgradescopeAgentNotTicking alert are the only signals (#199). |
TestMarkStatusErrorAndWriteStatusClearsIt TestWriteStatusTellsAFailedClearFromAFailedWrite TestTickWithAFailedMarkerClearIsNotAFailedTick TestMarkStatusErrorForbidden TestWriteStatusPatchesNothingWhenNotMarked TestTickMarksTheCRWhenTheStatusWriteFails TestObserverDropsStaleVerdictGaugesAfterFailedTicks |
| AG-13 | A passing EOL date reaches the ClusterReadiness within one interval, with no Kubernetes event. | not automated: #99 (the clock-shifted run did not finish) |
Chart, RBAC and what the agent touches¶
| ID | Claim | Proven by |
|---|---|---|
| RB-01 | The README, SECURITY.md and the security page describe the agent's role as the chart renders it: get and list, never watch; writes (update and patch) only on its own ClusterReadiness, its status and the ClusterReadiness CRD, each by resourceNames, except that create on clusterreadinesses cannot be limited by name, so the agent can create a ClusterReadiness under any name (it never deletes); and the cluster-wide Secrets and ConfigMaps read that rbac.helmSecrets adds, or, with rbac.helmSecretsNamespaces, the Role per listed namespace that replaces it. SECURITY.md and the security page name the ingressclasses list that add-on detection reads, which the role grants. All three name the persistentvolumes and storageclasses lists of the volumes capability, which the role grants as two rules of get and list only, with no watch and no write (#362). |
TestRBACDocsMatchRole TestRBACDocProblems TestDocsAgentRBAC TestRenderedRBACDefault TestRenderedRBACHelmSecretsOff TestRenderedRBACHelmSecretsNamespaces TestRenderedRBACVolumes |
| RB-02 | The agent writes only its own ClusterReadiness, that object's status, and (with manageCRD) the ClusterReadiness CRD; nothing else, ever. Measured from the API server's audit log of a real install. |
e2e:audit_agent_writes_only_its_cr TestRenderedRBACDefault |
| RB-03 | Update and patch are limited to the agent.crName object and the one CRD, with no delete. create on clusterreadinesses cannot be limited by name, so the agent's ServiceAccount can create a ClusterReadiness under any name; the values comment of rbac.create, the chart README, the values reference and the security page say so. Round 3 saw kubectl create --as=system:serviceaccount:<ns>:<release>-upgradescope of a ClusterReadiness named otherwise succeed (rc.3); it did not record a delete, so "no delete" rests on the rendered role, not on a run; that run is not automated: #99. |
TestRenderedRBACDefault TestRenderedRBACCustomCRName TestRenderedRBACManageCRDOff TestRBACDocsNameTheUnrestrictedCreate hack/test-chart.sh |
| RB-04 | No webhooks, no finalizers, nothing in the cluster changes because of a finding: with an EOL blocker installed, the chart install adds no admission webhook, the ClusterReadiness carries no finalizer or owner reference, and the agent writes nothing but that object and its CRD. | e2e:no_webhooks_or_finalizers e2e:audit_agent_writes_only_its_cr TestRenderedRBACDefault |
| RB-05 | Helm detection reads Secrets only as owner=helm lists and GETs of Helm release Secrets, cluster-wide or, with rbac.helmSecretsNamespaces, in each listed namespace only; rbac.helmSecrets=false removes the grant. |
e2e:audit_secrets_helm_only TestCollectHelmFetchesOnlyTheChosenRevision TestRenderedRBACHelmSecretsOff TestCollectHelmListsOnlyTheListedNamespaces |
| RB-23 | With rbac.helmSecretsNamespaces set (and rbac.helmSecrets=true), the ClusterRole grants no Secrets or ConfigMaps; each listed namespace gets one Role (get/list on secrets and configmaps, no wildcard) and one RoleBinding to the agent's ServiceAccount, and the agent, passed --helm-namespaces, lists Helm storage only in those namespaces, never cluster-wide, reporting the helm capability partial with the namespaces read. Each Role still reads every Secret of its namespace. |
TestRenderedRBACHelmSecretsNamespaces TestRenderedRBACHelmSecretsNamespacesModes TestSchemaHelmSecretsNamespaces TestCollectHelmListsOnlyTheListedNamespaces TestCollectHelmRefusedNamespaceIsAGap TestTickReadsHelmReleasesOnlyInHelmNamespaces TestAgentHelmNamespacesFlag |
| RB-06 | The chart grants no non-resource URL but /version and /metrics (discovery, /api and /apis, comes from Kubernetes' default system:discovery role). |
TestRenderedRBACNonResourceURLs TestRenderedRBACDefault |
| RB-09 | Token wiring: existingSecret, the in-chart ingest token, readToken, and a token required with serverUrl. |
hack/test-chart.sh |
| RB-10 | agent.interval is at least 1m and targets are MAJOR.MINOR strings, enforced by the values schema. |
hack/test-chart.sh TestConfigIntervalMinimum |
| RB-11 | Pods run as non-root 65532 with a read-only root, no privilege escalation, RuntimeDefault seccomp and all capabilities dropped. | hack/test-chart.sh |
| RB-12 | The chart lints strictly, every documented values combination renders and validates at the oldest and newest tested minor, and crds/ matches the embedded CRD. |
ci:helm hack/helm-test.sh hack/test-chart.sh TestKBRBACRulesInSync |
| RB-13 | Every chart knob is documented in the commented values.yaml. |
not automated: #99 (verified by hand in #137: every value path the templates read is in values.yaml, and every leaf is read) |
| RB-14 | helm uninstall removes everything but the CRD (and the CR the agent made). |
e2e:uninstall_leaves_nothing |
| RB-15 | server.persistence.enabled=false falls back to emptyDir; the server Deployment uses Recreate (one SQLite writer). |
hack/test-chart.sh |
| RB-16 | Reading the Argo CD and Flux chart sources is opt-in: rbac.gitops.argocd and rbac.gitops.flux default to false and grant only get/list on Applications, HelmReleases and OCIRepositories (OCIRepositories listed since #248, fetched by name only where a list is refused), never watch, a write, or the rest of those groups. |
TestRenderedRBACGitOpsOptIn |
| RB-17 | The documented chart upgrade is helm upgrade --reset-then-reuse-values (Helm 3.14 or later), never --reuse-values, which makes the old release's defaults the new chart's: the new chart's defaults, its image digest included, apply under the documented flag, and the user values of an earlier release (the files in deploy/chart/testdata/upgrade, each checked to render on its own release's chart and to set only keys that chart's values.yaml has, the one guard for v0.1.1, whose chart has no schema) still render on the current chart. No doc recommends --reuse-values in a command. |
TestUpgradeWithResetThenReuseValues TestDocsDoNotRecommendReuseValues TestUpgradeValuesFilesAreRealUserValues |
| RB-18 | A SQLite server on the chart's read-only root filesystem has a writable temp directory (an emptyDir at /tmp, server.tmp.sizeLimit, and SQLITE_TMPDIR), so a cluster delete of a 60 MB backlog succeeds, as does the retention prune of one; without one SQLite fails the cluster delete with disk I/O error (6410) (reproduced under sandbox-exec on macOS only, the other test runs everywhere). The batched prune (SV-22) no longer failed without a temp directory in that emulation either (#263), because of the transaction and not the statement: the same 60 MB delete, in the pre-batching form and in the batched form, fails with 6410 inside an explicit transaction (as the cluster delete runs) and succeeds on its own, and succeeds inside a transaction with a writable SQLITE_TMPDIR; that SQLite's statement journal is the temp-file user is its documented behaviour and fits the four results, not observed directly. What one full batch of 5,000 rows needs was not measured (the run deletes 150). A Postgres server gets neither. |
TestServerSQLiteHasAWritableTempDir TestLargeDeleteWithSQLiteTmpDir TestLargeDeleteOnReadOnlyRootFilesystem TestTempSpaceIsNeededInsideATransactionNotByTheStatement |
| RB-19 | No object the chart renders other than a Secret carries a function of a token, a webhook URL or any other secret value: no checksum, salted or not, in a pod template, a Deployment's metadata or anywhere else (the only pod-template checksums are of the team map and the extra registry, which are not secret). So changing a token, a webhook value or the contents of an existingSecret does not restart the pods and needs no restart: serve and the agent re-read the mounted files (SE-23, AG-17, RB-22), and the chart NOTES, the upgrade page and the security page say so, with the latency and what still needs kubectl rollout restart (the Postgres URL, extraEnv and extraArgs, a notification URL not set at start). Every Secret key is written under data, so from this chart version on a value removed on upgrade leaves no key. The upgrade to this version is the exception: a Secret written by an earlier chart wrote readToken, adminToken, slackWebhook, webhook, webhookSecret and the agent's serverToken as stringData, which Helm's merge (computed from the previous manifest, which never listed data.<key>) cannot remove, so a value removed in that same upgrade keeps an inert key until kubectl patch secret ... --type=json -p '[{"op":"remove","path":"/data/<key>"}]' (reasoned from Helm's merge and the same mechanism documented for ingestToken, not reproduced on a cluster). |
TestNoObjectCarriesAFunctionOfASecret TestNoSecretDerivedValueInPodMetadata TestNotesDescribeRotationWithoutARestart TestSecretKeysAreRemovedWithTheirValues TestDocsDescribeRotationWithoutARestart TestDocsQualifyKeyRemovalOnTheFirstUpgrade |
| RB-20 | With server.staleAfter unset the server's stale threshold and the UpgradescopeClusterStale alert are the larger of 2h and three agent.intervals (2h up to a 40m interval); a staleAfter or clusterStaleAfterSeconds not above the in-chart agent's interval fails the render. |
TestStaleThresholdFollowsTheAgentInterval |
| RB-21 | server.service.port may be 80 or 443: serve listens on server.containerPort (8080). A Service name, the only name Kubernetes limits to 63 characters, is cut to fit at a 53-character release name, and what names a Service (the Ingress backend, the agent's push URL, the certificate's DNS names, the server's --allowed-host list, the ServiceMonitor, the alert's job) follows; no other resource is renamed, so a release upgraded from an earlier chart keeps its PVC, token Secret and ConfigMaps whatever the length of its name (a renamed PVC is deleted by Helm with the history in it, and a renamed token Secret loses the generated ingest token). networkPolicy.enabled with an Ingress or a ServiceMonitor fails the render unless networkPolicy.serverIngressFrom is set. More than one server replica gets a PodDisruptionBudget (a budget of 0 renders as written, and one with neither bound fails the render) and a soft topology spread; one replica gets neither. |
TestServicePortIsNotTheListenPort TestServiceNamesFitSixtyThreeCharacters TestServerAllowedHostsFollowTheCutServiceName TestLongReleaseKeepsTheNamesOfResourcesThatExist TestNetworkPolicyGuards TestServerHighAvailability TestPodDisruptionBudgetBounds |
| RB-22 | The chart mounts the server's tokens, webhook URLs and webhook key, and the agent's push token, as files from their Secrets (chart-managed, existingSecret, agent.existingSecret) and passes the *-file flags (--ingest-token-file, --read-token-file, --admin-token-file, --slack-webhook-file, --webhook-file, --webhook-secret-file, --server-token-file): never with subPath (a subPath mount never updates), read-only, listing only the keys in use, and no such secret reaches a container as an environment variable or an argument (the Postgres URL still does). The keys of an existingSecret that may be absent (the ingest token of a hub with no in-chart agent, and the notification URLs and key) come from a second volume marked optional, and serve is told with --optional-secret-file; the others are required, so a missing key keeps the pod from starting as before. A secret passed as a flag in server.extraArgs or agent.extraArgs beside the *-file flag the chart passes for it fails the render, naming the flag, instead of a pod that exits at start (serve and the agent refuse the pair). Deleting the ingestToken key from an existingSecret revokes the shared ingest token only on a hub with no in-chart agent, where the key is optional; with the in-chart agent it is required, so deleting it revokes nothing and the kubelet stops updating that Secret volume until the key is back (reasoned from the kubelet, not reproduced on a cluster), and the upgrade page, the security page and values.yaml say so. The install NOTES, the chart README, values.yaml, the upgrade and security pages give the rotation latency (the kubelet's Secret sync, up to about 60 to 90 seconds at its default settings by the Kubernetes documentation, not measured here, plus the 5-second check) and what still needs a restart; with server.replicas above 1 they also say each replica's kubelet syncs on its own schedule, so for up to that window a push can get a 401 from one replica and a 200 from another, and that a UPGRADESCOPE_*_TOKEN in extraEnv no longer overrides the chart's file. The NOTES' stale-threshold warning follows --force-sync-every in agent.extraArgs (either spelling, the last one winning, 1h when absent or unreadable) and says which force-sync it assumed (the default 1h, named as assumed, when the flag is not a duration the chart can read), and also warns when the default threshold is below the resulting gap. |
TestServerTokensAreMountedFilesNotEnvironment TestServerExistingSecretKeysThatMayBeAbsentAreOptional TestAgentPushTokenIsAMountedFile TestNoSecretDerivedValueInPodMetadata TestNotesDescribeRotationWithoutARestart TestNotesWarnOfAFlappingStaleThreshold TestNotesWarnOfADefaultThresholdBelowTheForceSync TestNotesSayWhenTheForceSyncIsAssumed TestExtraArgsSecretFlagBesideTheChartsFileFlagFailsTheRender TestIngestKeyDeletionIsDocumentedAsRevokingOnlyWhereItIsOptional TestDocsDescribeRotationWithoutARestart TestDocsSayReplicasSyncSeparatelyAndExtraEnvNoLongerOverrides |
Server, fleet and notifications¶
| ID | Claim | Proven by |
|---|---|---|
| SV-01 | SQLite (WAL, no cgo) or Postgres, and both pass one conformance suite; Postgres 17 on every PR that changes code (IR-24) and every push, 14 to 18 weekly. | TestSQLiteConformance TestPostgresConformance TestOpenSetsPragmas ci:pg-conformance |
| SV-02 | Duplicate pushes are detected by the canonical inventory hash; key order and whitespace never change it. | TestIngestDuplicateCanonicalHash TestInsertSnapshotDedup TestInsertSnapshotConcurrentIngest |
| SV-03 | Fleet, reports and exports serve stored evaluations, re-judged when the knowledge base or the team map changes, and after a date change on the next push or background re-evaluation (SV-03b). | TestReadPathsUseLatestSnapshotOnly TestDuplicatePushReevaluatesAcrossEOLDate TestRestartWithNewKBReevaluatesOnDuplicatePush TestTeamMapChangeReevaluates TestStartRunsBackgroundPassAndDelivery |
| SV-03b | The background re-evaluation also runs just after each UTC midnight, when end-of-life dates move a day, and until a stored verdict is re-judged every read of it (fleet cells, cluster summaries, reports) is marked outdated. A read of an outdated verdict starts the next pass early, at most one pass every 5 minutes after the last one started; a cluster the pass could not bring up to date (a report now over --max-snapshot-bytes, an inventory that does not decode, a failed evaluation) stays outdated, and is skipped by passes, and starts none from its reads, until its snapshot or the UTC day changes; a store error in a pass (a busy database, a lost connection) marks nothing, and the next pass tries again (#241). |
TestFleetReadAfterEOLCrossingIsFreshOrStale TestReevaluationWakesAtUTCMidnight TestKickedPassesAreRateLimited TestUnrefreshableRowDoesNotDrivePasses TestTransientStoreErrorDoesNotMarkTheClusterUnrefreshable TestCorruptInventoryMarksTheClusterUnrefreshable TestFailedEvaluationMarksTheClusterUnrefreshable |
| SV-04, SV-12 | Ingest refuses, with 422 and before anything is written, an inventory it cannot judge: null or empty, an unparseable server version, a schemaVersion other than 1, a source other than cluster (only the agent pushes, and a files inventory is judged without versions or add-ons, so claiming one turned a blocked cluster ready), or a collectorSchema other than 1 or none. A push that lacks the server version is judged at the cluster's last reported version, so a blocked cluster stays blocked. A schema-valid cluster inventory that does not report api-usage, versions or (when the knowledge base has add-ons) addons, including one with no or an empty capabilities map, is stored and judged with each as a required not-assessed gap: unknown at best, never ready (#194); so is a files inventory (scan --files) without api-usage. Collectors stamp collectorSchema; an inventory without it is judged as a v0.1.x agent's unless its agentVersion is a semantic version at or after 0.2.0-0, so dev, empty and unknown (what a v0.1.x agent built without a version reports) are, and a v0.2.0 release candidate's 0.2.0-rc.N is not. |
TestIngestRejectsMalformedInventory TestIngestStoreFailureLeavesNoClusterRow TestIngestDegradedPushKeepsFleetCell TestIngestCapabilityLessInventoryNeverReady TestEvaluateUnreportedRequiredCapabilities TestLegacyInventory TestLegacyViewJudgesSnapshotsAsCluster TestLegacyChartSourcedAddOnNotReady |
| SV-06 | With ?cluster=, /gate judges manifests inside the cluster's stored inventory and blames only what they add, per object: a pull request that adds a removed-API object fails even when the cluster already runs that kind, and one that adds an in-tree volume plugin (a PersistentVolume with spec.rbd, a pod template with an inline gitRepo) fails even when the cluster already has pods or volumes on that plugin (#365; the finding stays whole, listing the cluster's objects and the PR's even when the cluster's row already lists 100, and SARIF holds the PR's object only), and a harmless manifest is not blamed for the cluster's own deprecated calls, nor for a Helm release's stored-manifest blocker, and a cluster object's upgradescope.basit.engineer/ignore annotation (or a ?config= rule selecting it) accepts its finding, however many objects the PR posts at that API: the engine and suppression see every ref of the cluster's and the PR's, and only then is each finding's listing cut to 100 objects, the PR's kept and the cluster's cut first, as in the baseline (#72). |
TestGateClusterNewObjectOfResidentGVK TestGateClusterBenignManifestKeepsFold TestGateClusterNumericName TestUpsertUsageKeepsManifestRefs TestGateHelmVerdictIgnoresTheRefsThePRPushesOut TestGateSuppressionIgnoresTheRoomThePRTakes TestGateAnswerListsThePRsRefsFirst TestGateClusterVolumePluginBlamedOnManifests TestGateClusterVolumePluginNotBlamedOnUnrelatedManifest TestGateVolumePluginSARIFLocatedAtPath e2e:gate_passes_clean_manifest e2e:gate_fails_removed_api (on kind: a ConfigMap passes although the cluster has an EOL ingress-nginx; an extensions/v1beta1 Ingress fails) |
| SV-06b | With ?cluster=, /gate merges the posted manifests' add-ons and CRDs into the cluster's inventory too (#150): an EOL ingress-nginx the manifests deploy into a clean cluster is the manifests' blocker, as it is without ?cluster=; a posted custom resource at a version the cluster's CRD, or a posted CRD replacing it, does not serve is the manifests' crd-version blocker, as it is without ?cluster= when the CRD is posted with it; an add-on install the manifests carry replaces the cluster's in its namespace; custom resources the cluster already has at such a version stay the cluster's. |
TestGateClusterManifestAddOn TestGateClusterManifestCustomResource TestGateClusterManifestCRDReplacesClusters TestGateClusterResidentCustomResources TestMergeManifestsCRDs TestUpsertAddOns |
| SV-08 | An empty cluster UID is not a wildcard: a push that cannot say which cluster it is gets 409 once the name is bound to a UID, and never joins that cluster's history. | TestIngestRejectsUIDLessPushToBoundName TestSQLiteConformance |
| SV-24 | A cluster name is bound to the first non-empty cluster UID pushed under it: a name first registered by a push with no clusterId is bound by the first clusterId pushed afterwards, from whichever cluster sends it, and that push continues the name's history; then another UID, or none, gets 409. Two pushers that both send no clusterId share the name. The adopting push logs one line naming the cluster, the adopted clusterId and the snapshot (never the token), and sends no became-ready, since the history it continues may be another cluster's (#368). Both stores pass the same conformance case. |
TestIngestUIDPushAdoptsNameRegisteredWithoutUID TestIngestLogsUIDAdoption TestAdoptingPushSendsNoBecameReady TestSQLiteConformance TestPostgresConformance (the CommitAdoptsUIDOfNameRegisteredWithoutUID and UpsertClusterRejectsUIDChange subtests); on the built binary over HTTP, with no cluster (202, 202, history 75 then 100, then 409 for another UID and for none, one adoption log line and no token in the log): hack/claims-server.sh make claims-server ci:repo-checks |
| SV-07 | The fleet matrix and per-team rollups (worst score, total blockers, affected clusters). | TestFleetMatrixExplicitTargets TestFleetMatrixDefaultTargets TestFleetTeams TestTeamsEndpoint TestTeamMapApply |
| SV-09 | Auditor exports: one self-contained HTML report and a CSV per cluster and target. | TestExportHTMLGolden TestExportCSVGolden |
| SV-10 | Targets, from serve --targets or a read's target/targets parameter, are Kubernetes 1.x minors: 2.0 is refused (422 on a read), as scan refuses it. |
TestServerRejectsNonV1Targets |
| SV-11 | Read and write timeouts carry the default 20 MiB snapshot over a link of about 350 KiB/s or faster (slower gets a retryable 408: SV-11b), and a stalled body cannot hold a connection. | TestLargeSnapshotWithinDefaultTimeouts TestStalledBodyDisconnectedByReadTimeout TestShutdownWithStalledClient |
| SV-13 | Wrong methods get 405 with Allow; unknown paths under /api/, /metrics/ and the probes get a JSON 404, not the dashboard. |
TestMethodNotAllowed TestReservedPathsNeverServeDashboard TestStaticDoesNotShadowAPI TestOpenAPIUnknownPathIsJSONError |
| SV-13b | Bare /api is the API's too: a JSON 404, never the dashboard's index.html. |
TestComposedHandlerBareAPIIsJSON404 |
| SV-14 | Fleet views read stored snapshot heads, not whole inventories (a latest snapshot stored without a server version, which migration 0009 could not fill from its inventory, is read once per server start, and a duplicate push records the version: #241): 500 clusters under 10 concurrent readers keep the heap under 512 MiB, inside the chart's 1Gi memory limit, and /fleet p95 under 1 s given at least one full core of an Apple M1 Pro-class CPU (October 2026, across runs: 0.33 to 0.36 s unconstrained, 0.56 to 0.68 s with GOMAXPROCS=1, which caps Go's threads and is not a cgroup CPU quota; a --cpus 1 quota ran about 1.26 times slower than GOMAXPROCS=1 on the red-team's host, #196). The p95 does not hold at the chart's default 500m CPU limit (round 2 measured 3.9 s at --cpus 0.5 on a 2017 i3, #196); Running the server says to give such a fleet a core. |
make bench-server TestBenchServerFleet (env-gated: run by hand, not in CI; ingest's memory under concurrent large pushes is SE-05b and PF-06) TestFleetDecodesNoInventories TestLegacyRowWithoutVersionIsReadOnce TestMigration0009Backfills TestMigration0009BackfillsPostgres |
| SV-15 | Stored times are fixed-width UTC, so they sort in instant order. | TestTimeFormatFixedWidthUTC TestTimesStoredUTCFixedWidth TestParseStoredTimeRoundTrip |
| SV-16 | --db creates its parent directory and is mutually exclusive with --db-url. |
TestRunServeCreatesDBParentDir TestServeDBAndDBURLMutuallyExclusive |
| SV-17 | The background re-evaluation pass decodes no inventory to learn what is stale: it reads every snapshot head in one query and each target's evaluation summary (knowledge-base version, team-map hash, evaluatedAt, whether it holds a deprecated caller), and loads an inventory only for a cluster with a stale, missing or held target (a held target is one whose report carries a deprecated caller held after an apiserver restart: only the inventory says whether the scrape it judges ends or moves the hold), so a pass with nothing stale loads none. /history, a stored report, its findings and teams, and the exports read the version a snapshot is judged at from its head, without its inventory (#241). |
TestReevaluationPassLoadsNoInventoryWhenNothingIsStale TestPassLoadsTheInventoryOfAHeldTarget TestReadsLoadNoInventoryForTheServerVersion TestSQLiteConformance TestPostgresConformance (the CarriesHold and DuplicateFillsServerVersion subtests) |
| SV-18 | The newest evaluation of a cluster and target, its newest decided one (the notification baseline) and score history are read through indexes by evaluation id, without sorting the pair's history or its reports, so they do not slow down as history grows: on SQLite, behind 4,000 history rows with 30 KB reports, the baseline is read in under 2 ms and 100 points of history in under 5 ms (#241). | TestEvaluationReadsUseIndexes TestEvaluationReadsDoNotGrowWithHistory TestLatestSnapshotHeadsLooksUpEachCluster TestPostgresEvaluationReadsUseIndexes |
| SV-19 | The fleet teams rollup at a target with no stored evaluation keeps each cluster's what-if (by snapshot, target, knowledge base, team map and UTC day; a stored evaluation by its id and evaluatedAt), so the next rollup at that target computes only what changed and answers the same; it takes the per-cluster read slot one cluster at a time, so a per-cluster read waits for one cluster's part; and a rollup still computing after 20 s answers 503 with Retry-After, keeping what it computed for the retry (#241). |
TestFleetTeamsWhatIfIsCached TestFleetTeamsDoesNotHoldTheReadSlot TestFleetTeamsTimeBound TestFleetTeamsUnstoredTargetMatchesPerCluster |
| SV-21 | A retention prune that fails is visible: upgradescope_retention_prune_failures_total{store} counts it, upgradescope_retention_last_success_timestamp_seconds is set only when a prune ran to the end (absent before the first), and upgradescope_retention_rows_deleted_total{table} counts the rows deleted, those a failed run had committed included; a prune stopped by shutdown is neither a failure nor a success. The series exist only with a --retention window. /readyz does not change, since the server still ingests and serves. With the server and a non-zero server.retention, the chart's UpgradescopeRetentionStale alert fires when the gauge is more than 2 days old, or absent more than 2 days after the server started, or absent while the failures counter is above 0, a failure counted by a process that has completed no prune, whatever its age: a server that restarts every 12h with a startup prune that fails before Prometheus first scrapes it, so that every counter series is first seen at 1, fires, where increase() over such a series is 0; a restart after a successful prune, a counter reset and nothing scraped stay quiet (#263, #296; evaluated with the Prometheus PromQL engine in tools/promrule-test, which the pinned-helm job runs through hack/test-chart.sh with UPGRADESCOPE_CHART_TEST=1, so the cases cannot skip there). |
TestRetentionPruneFailureIsCountedAndLeavesTheGauge TestRetentionCountsRowsDeletedEvenWhenThePruneFails TestRetentionShutdownIsNotAFailure TestRetentionOffExportsNoRetentionSeries TestReadyzIgnoresPruneFailures TestMetricsReferenceListsEveryServerMetric TestRetentionStaleAlert TestRetentionStaleFiresWhenEverySeriesIsFirstSeenAtOne TestRetentionStaleFiresWhenTheFailureLandsAfterTheFirstScrape TestRetentionStaleStaysQuiet TestRetentionStaleOtherArms hack/test-chart.sh |
| SV-22 | Retention deletes in batches of at most 5,000 rows a transaction (500 ids a statement for the baselines it no longer spares), on SQLite and Postgres: a backlog larger than one batch drains over several transactions, each bounded, and a run cut short keeps the batches it committed and the next run resumes, so one failure does not repeat as one giant delete. Cost, measured on SQLite with 60,000 snapshots and evaluations: about 20% more time at best-of than one transaction (docs/operations/retention-and-backup.md); a Postgres prune was not timed (#263). The evaluation step finds old rows from the evaluations (created_at, id) index (migration 0010, ordering by created_at, id), so a prune that deletes nothing reads no report: on SQLite with 20,000 evaluations of 30 KB inside the window it took 0.27 to 0.43 ms where the scan it replaced took 177 to 270 ms under the write lock, and the backlog drain costs about 15% more at best-of for the index. On Postgres the migration is CREATE INDEX IF NOT EXISTS, so an operator can build the index CONCURRENTLY before the upgrade, which blocks writes to evaluations for the whole build otherwise (docs/operations/upgrade.md) (docs/operations/retention-and-backup.md; the snapshot step still scans the snapshots table, 29 to 31 ms for 20,000 snapshots of 30 KB and 9 to 10 ms for 20,000 of 1 KB) (#297). A first start whose 0010 build takes longer than 10 seconds is no longer cut short and restarted into the same failure: the startupProbe bounds it, and a start killed during it does not leave the build running server-side (SV-23, #364). |
TestPruneDefaultBatchBoundsEveryTransaction TestSQLiteConformance TestPostgresConformance (the PruneDrainsABacklogInBatches and PruneResumesAfterACutShortRun subtests) TestPruneEvaluationDrainUsesCreatedAtIndex TestPostgresPruneEvaluationDrainUsesCreatedAtIndex TestPruneWithNothingToDeleteIsCheap TestPostgresMigration0010CreatesItsIndexIfNotExists TestPostgresMigration0010AcceptsAnIndexBuiltConcurrently TestPostgresOpenSlowMigrationApplies TestPostgresKilledMigratorStopsServerSide |
| SV-23 | The server migrates its database before it opens its port, and on SQLite and Postgres alike only the startupProbe bounds how long that takes (docs/operations/upgrade.md, the startup window). On Postgres only the connection check at open has a bound of its own, 10 seconds, so an unreachable or wrong DSN still fails fast; the wait on the migration advisory lock and every migration run on the command's context with no deadline: a replica starting while another holds the lock for 15 s queues and then opens, and a 12 s migration applies (each failed at 10 s before #364, in serve, tokens and clusters). Cancelling that context mid-migration (SIGINT or SIGTERM cancels serve's) returns promptly with an error that wraps context.Canceled and says the migrations were interrupted, and the interrupted migration is not recorded, so the next start re-runs it (serve given a SIGTERM 13 s into a wait on the lock exited with status 1 and that error: checked by hand against PostgreSQL 17.11, 11 October 2026, not automated). The migration session sets client_connection_check_interval to 2s (PostgreSQL 14 and later; older servers ignore it), so a migrator killed mid-statement, with no cancel request sent, leaves no statement running server-side and frees the migration lock within 10 s. |
TestPostgresOpenWaitsForMigrationLock TestPostgresOpenSlowMigrationApplies TestPostgresOpenCancelledMigrationStopsServerSide TestPostgresKilledMigratorStopsServerSide TestPostgresOpenUnreachableHostFailsFast TestPostgresOpenBadDSN TestOpenStorePassesCommandContext |
| SV-16b | --db is the file SQLite opens, exactly as given: a path with %, ? or # (a%3Fb.db, x%20y.db, q?b.db) opens that file and no other, and a path with a NUL byte is refused (#195). |
TestOpenUsesExactPath TestOpenRejectsNUL TestTokensCreateDBPathIsExact |
| SV-20 | A snapshot collected with another knowledge base than the server's is never ready on evidence the agent's data did not collect (#268). The agent lists API usage, and reads a Helm release's manifest, only for the APIs its lifecycle data flags, and matches add-ons by its own registry (what no entry claims goes to unrecognizedImages, untagged), so a server whose data is newer would read ready on a cluster its own data blocks (an rc.2 agent's weave-net, a retired product, sent as docker.io/weaveworks/weave-kube). The server compares the lifecycle and registry digests in the push's kbVersion with its own: where the lifecycle digests differ api-usage and helm are partial, where the registry digests differ addons is, each naming knowledge base differs from the agent's in skipped, and the reason names both digests (never the agent's label) and says to upgrade the agent, and for the registry that --registry-dir entries the server lacks differ the same way; api-usage and addons are then required gaps, so the verdict is unknown at best, in the stored evaluation and in a what-if alike. Evidence is complete only when the digests are the same (a digest cannot say which side is newer, so a newer agent reads the same way); a kbVersion naming no digests is not known to be complete either; matching digests are judged exactly as before. The inventory carries nothing extra, so its size is unchanged. The push's agentVersion (at most 128 bytes) and kbVersion (at most 512) are agent-controlled text: ingest refuses, with 422 and without echoing it, one that is longer or not printable ASCII, no reason quotes either (a snapshot stored before that check is judged the same way), and every reason the skew extends stays within MaxReasonBytes with what the server added whole (K1: a 64 KiB kbVersion of control characters, quoted with %q into three reasons, would have made about 590 KB of reasons per evaluation; computed from the code at 8593a68b, 3 × 196,605 bytes, not measured). The verdict clears when the agent runs the server's data even if the cluster is unchanged: a duplicate push whose kbVersion or agentVersion differs from the stored one re-judges every target (the hash covers the inventory only). Under lifecycle skew helm is partial too, so a Helm release's findings are listed as hidden by it. The CI gate judges the same way: POST /api/v1/gate?cluster= applies the same view to the cluster's snapshot, and keeps an api-usage gap the judgement made (a skewed knowledge base, an agent before v0.2.0, count-only rows that name no object) when it merges the PR's manifests into the proposed state, so a PR against such a cluster reads unknown (ready false, clusterVerdict unknown), never ready. The move to unknown, and the agent catching up, send no notification (an unknown pass behind a required gap that hides findings, here KB skew, is not news and not a baseline). |
TestGateClusterSkewIsNeverReady TestGateClusterViewedUsageIsNeverReady TestKBSkewViewKeepsReasonsWithinTheBound TestUnattributedUsageViewKeepsTheReasonWithinTheBound TestKBSkewMovesNoNotifications TestKBSkewNeverEchoesTheAgentsLabels TestKBSkewClearsWhenTheAgentIsUpgraded TestKBSkewRegistryDirMismatchIsNamed TestAppendReason TestKBDigests TestKBSkewView TestKBSkewOlderAgentIsNeverReady TestEvaluatePartialAndAddOnGaps TestHiddenByGaps |
| KB-14 | A stored snapshot is kept as the agent pushed it, fields this server does not know included, so a newer server re-judges all of it; an inventory from a v0.1.1 agent or older (no collectorSchema, and an agentVersion that is not a release at or after 0.2.0-0: 0.1.1, dev or empty alike) is judged by what its collectors meant (its api-usage and deprecated calls not assessed, a chart-found add-on version kept as evidence only), so it is never ready on stale semantics. In an unmarked inventory otherwise judged as current, an api-usage count that names no object, the v0.1.x residency shape, is not judged: it is a partial api-usage gap naming the API, required when the target removes it, never a blocker. |
TestSnapshotRoundTripsUnknownFields TestLegacyResidencyFlagged TestLegacyChartSourcedAddOnNotReady TestLegacyInventory TestUnmarkedResidencyWithoutObjectsNotJudged |
| NT-01 | Notifications fire on changes only: a new blocker, a warning entering its EOL window, a cluster turning ready. Each pass is compared with the target's last baseline: a decided evaluation, or an unknown one whose only required gaps hide no finding (kb-coverage, a target past the KB's horizon; target), which saw every capability a blocker comes from (#380). So on a cluster first seen on the newest minor (default target past the horizon, unknown when clean) the first EOL add-on is a new-blocker, and a blocker an unknown pass saw gone is announced again when it returns; blockers such a pass resolved keep became-ready pending for the first ready pass. An unknown pass behind a required gap that may hide a blocker (a collector failure) sends nothing and is no baseline, so its recovery re-alerts nothing that never went away, and a cluster first seen that way gets a silent first baseline. Both stores select such a baseline, check it at commit (NT-05) and keep it from retention (NT-06) as a decided one. A blocker or EOL warning that a pass could not see is not a change (#189): one from a capability the pass did not assess (unavailable, or partial over it; required or optional alike) but the pass that last saw it did is carried forward in the baseline. Add-on findings count helm among their capabilities (an add-on found through its chart alone). It is not announced again when it is seen again, nor when a deprecated caller folded into a carried API usage finding stands on its own while api-usage does not see that API, and no became-ready is sent while one is carried; it is resolved by the first pass that assessed it and no longer finds it. A capability that was not assessed when a finding was seen does not hold it later, so an install that never assesses one (rbac.helmSecrets=false, a role that cannot list IngressClasses) still gets became-ready for a fix and a new alert for a finding that returns; an add-on seen while helm was partial is likewise resolved if helm then fails outright, and announced again when helm returns. Carried findings that would take the stored report over --max-snapshot-bytes are dropped, not the push. A deprecated-call blocker missing from a scrape of an apiserver up for less than 24 hours (apiserver_requested_deprecated_apis starts empty when the apiserver starts; the agent reports the apiserver's process_start_time_seconds) is held the same way (#204), and resolved by the first scrape at least 24 hours after that apiserver's start, measured on the scrape's collectedAt, not the server's clock: on a quiet cluster the next force-sync duplicate, so within 24 hours plus --force-sync-every of the restart, background passes and knowledge-base updates included. The start time is not part of the snapshot hash, so moves between HA apiservers store no new snapshot; one before 2014 or more than 10 minutes after collectedAt is ignored, and a recorded hold end more than 24 hours and 10 minutes after a push's collectedAt (from a clock that was ahead for both the agent and the apiserver) is dropped by that push, so once the clock is corrected the hold ends within that bound; background passes keep a recorded end rather than judge it by the older stored scrape. A control-plane upgrade that removes the API is not held: a caller of an API the scraped server version no longer serves (the KB's removal release at or below it, such as v1beta3 FlowSchema after v1.31 to v1.32, removed in 1.32) is resolved at once, so became-ready is not delayed by the window. A push without a start time honours an end an earlier scrape recorded, and the hold stays bounded by it. Limit: without a start time (an older agent) and with no recorded hold, a deprecated-call blocker that vanishes after a restart is resolved, and can send became-ready while the caller still exists; a client that calls less often than daily is resolved when the hold ends; either is announced again when the caller next calls. With several kube-apiservers, each counts deprecated calls on its own, so a caller that a scraped apiserver up for over a day never served is resolved the same way (operations.md, Notifications). |
TestComputeDelta TestIngestEmitsDeltaNotifications TestUnassessedCapabilityKeepsItsBlockers TestUnassessedCapabilityNeverBecomesReady TestRequiredCapabilityOutageKeepsItsBlockers TestUnassessedCapabilityStillAnnouncesNewBlockers TestAPIUsageOutageKeepsFoldedCaller TestHelmOutageKeepsChartFoundAddOn TestKeepCarriedOverLimitKeepsTheEvaluation TestAPIServerResetResolvesVanishedCaller TestRestartHoldEndsOnDuplicatePushes TestRestartHoldKeepsCallerWithOtherBlocker TestRestartHoldSurvivesKBRefresh TestRestartHoldUsesTheAgentsClock TestRestartHoldCarriesItsEnd TestAPIServerMoveIsNotANewSnapshot TestImplausibleAPIServerStartIsIgnored TestRestartHoldFromAFutureClockEnds TestRestartHoldSkipsAnAPIThatUpgradeRemoved TestRestartHoldKeepsAnAPIThePatchStillServes TestRestartHoldFromAnOldAgentKeepsTheCarriedEnd TestBackgroundPassKeepsALaterRecordedEnd TestRemovalOfCall TestRemovalOfCallShippedKB TestSnapshotHashIgnoresAPIServerStart TestCollectDeprecatedCallsRecordsAPIServerStart TestEveryCategoryHasSources TestSources TestHiddenByGaps TestHiddenBy TestFoldsInto TestPersistentGapStillResolvesAndReAlerts TestPersistentGapReAlertsEOLApproaching TestHelmOutageAfterHelmlessSighting TestHelmOutageAfterPartialHelmSighting TestCarriedFindingKeepsWhatItWasSeenWithout TestUnknownThenBlockedAnnouncesNewBlocker TestNewestMinorWebhookReceiverGetsOneNewBlocker TestBlockerResolvedByUnknownPassIsAnnouncedOnReturn TestUnknownWithRequiredGapIsNotABaseline TestUnknownVerdictNeverReadyOrAlerting TestUnknownBaselineCarriesWhatItCouldNotSee TestBlockerResolvedByUnknownPassBecomesReadyWhenDecided TestUnknownPassKeepsBecameReadyPending TestNotificationBaseline TestHidesNoFindingHidesNothing TestSQLiteConformance TestPostgresConformance (the LatestKnownEvaluationTakesUnknownBaseline, CommitRefusesReplacedUnknownBaseline and PruneKeepsUnknownBaseline subtests) |
| NT-01b | With several targets, a pass sends one notification per cluster, a change merged across targets only when it reads the same for each; a backwards clock step replays nothing. Round 3 saw one message per cluster per push for a webhook and for Slack on a live agent push (rc.3 source build, #99); the merge across targets and the clock step are covered by the unit tests only, not exercised live. | TestOneNotificationPerPassAcrossTargets TestGroupedChangeKeepsPerTargetWording TestClockStepDoesNotReplayNotifications |
| NT-01c | After a cluster upgrade, a blocker its new default target adds (v1beta1 ServiceCIDR, removed in 1.37, once the cluster is on 1.36) is notified; blockers it already had at its previous default target are not announced again, and a target newly added to --targets stays a silent baseline (#34). This includes a default target first decided after a knowledge-base update: it is compared with the cluster's last baseline of that target, its unknown passes past the horizon included (#380), so the blockers the update adds are announced once, with or without a decided evaluation of a lower target, and blockers the baseline had are not; a cluster whose passes there were all unknown behind a collector failure is compared with its lower target's, and without one stays a silent baseline. A target added to --targets is silent whether its first evaluation is blocked or unknown. |
TestUpgradeNotifiesBlockerNewAtNextTarget TestUpgradeDoesNotReannounceKnownBlockers TestUpgradeAnnouncesOnlyNewBlockersInOnePass TestUpgradeFromBlockedToReady TestNewlyConfiguredTargetIsStillABaseline TestClusterFirstSeenOnNewVersionIsStillBaseline TestUpgradeBaselineSpansASkippedMinor TestUpgradeBaselineIsTheNearestLowerTarget TestKBHorizonUnknownThenDecidedNotifies TestKBHorizonUnknownThenDecidedNotifiesWithoutLowerEvaluation TestKBHorizonUnknownThenDecidedSilentWithoutLowerEvaluation TestNewTargetDoesNotReannounceStandingBlockers TestKBUpdateDoesNotReannounceStandingBlockers |
| NT-02 | Notifications are best-effort: a hung sink never blocks ingest, nor delays another sink's messages (each sink's are delivered in queue order, one at a time, up to four sinks at once: #241), and a message is delivered once after retries. A delivery starts only when its timeout ends within the message's claim, so another replica never claims a message still being delivered. | TestIngestDoesNotWaitForNotifiers TestOutboxRetriesThenDeliversOnce TestOutboxGivesUpAfterMaxAttempts TestSlackDefaultTimeoutIsTwoSeconds TestHungSinkDoesNotDelayOtherSinks TestDeliveryNeverOutlastsItsLease TestOutboxLeaseCoversABatch |
| NT-05 | A pass's notifications are compared with the baselines it read (decided evaluations, or unknown ones past the KB horizon, #380) only if they are still the newest when it commits: another replica's push or the background pass landing in between makes the commit write nothing, and the push is evaluated again against what is there (up to three times, then 503 and the agent's retry), so a race never loses a became-ready or new-blocker nor sends one twice; a push that loses all three answers 503 with Retry-After: 10 and stores nothing (#241). |
TestPushRacingAnotherPushKeepsTheBaseline TestPushRacingTheBackgroundPassSendsOnce TestPushLosingTheRaceEveryTimeAnswers503AndStoresNothing TestFirstPushLosingTheRegistrationRaceNotifiesOnce TestFirstPushRacingRegistrationRetriesWithTheRegisteredID TestFirstPushRacingTheSameSnapshotIsJudgedForTheRegisteredCluster TestSQLiteConformance TestPostgresConformance (the CommitRefusesStaleBaseline, ReevaluateRefusesStaleBaseline and CommitRefusesReplacedUnknownBaseline subtests) |
| NT-06 | Retention never prunes a notification baseline in use: each cluster's last notification baseline (a decided evaluation, or an unknown one past the KB horizon, #380) of each target the server evaluates for it and of the three minors below its default target, and its snapshot, are kept however old, so a target unknown for longer than --retention, or the previous default target after an upgrade, is still compared with that baseline when next decided; an older decided evaluation under a newer unknown baseline ages out; the baselines of other targets (dropped from --targets, or older than the three minors below the default) age out like any evaluation, so a cluster keeps at most one older snapshot and evaluation for each target in use, and a cluster whose server version is unknown keeps every target's (#241). |
TestRetentionKeepsTheNotificationBaseline TestRetentionKeepsBaselinesOfTargetsInUseOnly TestRetentionKeepsEveryBaselineWhenTheVersionIsUnknown TestSQLiteConformance TestPostgresConformance (the PruneKeepsNotificationBaseline, PruneLimitsBaselinesToTargetsInUse and PruneKeepsUnknownBaseline subtests) |
| NT-07 | A pass's notification delta is linear in the findings it compares: 20,000 new deprecated callers against 20,000 carried API usage findings take under a second (each caller was compared with every carried finding, the code before PR #276: 46 s measured by an independent reviewer on an arm64 Mac on 10 October 2026, and 1m32.5s on an earlier run, probably on a loaded machine: the range is 46 s to 1m32.5s), now 38 to 54 ms for the 20,000 case in two runs (BenchmarkComputeDelta20k, same Mac, 10 October 2026), with the same answer (#241). |
TestComputeDeltaIsLinearInCarriedFindings TestCarriedFoldMatchesFoldsInto TestComputeDeltaFoldsCallerIntoCarriedUsage |
| NT-02b | A sink that answers 429 or 503 with Retry-After is not called again by that server before that delay (each replica keeps its own hold, in memory), capped at an hour, for any of its queued messages, and the held messages keep their attempts; other sinks are unaffected. A message is given up shortly after it has been queued for 8 hours (within a few minutes), however many are queued behind a rate-limited sink, so no queue outlives that. The hold is in memory: a restart forgets it. |
TestRetryAfterOn429 TestOutboxHonoursRetryAfter TestOutboxHoldsRateLimitedSink TestOutboxHoldDoesNotConsumeAttempts TestOutboxHoldBoundsMessageLifetime TestOutboxExpiredMessageNotSent TestSQLiteConformance TestPostgresConformance (the OutboxDefer subtest) |
| NT-03 | A Slack or webhook endpoint that answers with a redirect has not received the message: the delivery fails, and the redirect is never followed. | TestNotifierRedirectIsFailure |
| NT-04 | The generic webhook sends a versioned JSON body (schemaVersion 1, lowercase keys) and, with --webhook-secret, an X-Upgradescope-Signature HMAC-SHA256 of the raw body. |
TestNotificationJSONGolden TestGenericWebhookSignsBody TestWebhookDeliveryIsSigned TestServeWebhookSecret |
| SV-05 | What-if and the gate store nothing. | not automated: #99 (verified by hand in #132) |
| SV-25 | Object upgradescope.basit.engineer/ignore annotations, with their reason, apply in every view that judges objects: scan, the agent's ClusterReadiness status, /gate (and its ?cluster= objects), and the server's stored and read-time evaluations (the stored verdict and score, fleet cells, history, notifications, exports, GET /report), never the agent's spec.ignore on the server (#381). An accepted finding is in suppressed with source: annotation; annotating a blocked cluster's last blocker sends became-ready, a new already-annotated object sends no new-blocker, removing the annotation is a new-blocker; the CSV export lists each accepted finding as a suppressed row with its severity, source and reason, and the HTML export in a suppressed section. |
TestStoredEvaluationAppliesObjectAnnotations TestStoredAnnotationLifecycleNotifications TestAnnotatedBlockerPastTheHorizonNotifies TestExportListsSuppressedFindings TestFleetWideReadsMatchMainBeforeReadScopes TestApplyAnnotations TestTickLegacyAnnotationsKeepCapabilityGaps TestTickAppliesSpecIgnore TestGateClusterSuppressedManifestObject |
AI assistants (MCP)¶
| ID | Claim | Proven by |
|---|---|---|
| MC-01 | upgradescope mcp serves only read-only tools: scan, list_findings, get_report and registry_lookup, plus fleet_summary in fleet mode, and the MCP page documents exactly that set. Each tool carries readOnlyHint and none is marked destructive, so a tool added later, or one that drops the hint, fails the build. |
TestToolsAreReadOnly TestMCPStdioListsOnlyReadOnlyToolsAndListsFindings TestMCPFleetModeHonoursTheServersReadAuth TestDocsMCPPageListsExactlyTheTools |
| MC-02 | The server speaks MCP on stdio by default. With --http ADDR it serves streamable HTTP at /mcp, on 127.0.0.1 for a bare port or :PORT; any other address is refused without --allow-remote. Without --http-token the endpoint has no authentication, which the start message says; with it (or --http-token-file, $UPGRADESCOPE_MCP_HTTP_TOKEN), a request without that bearer token, or with another, is refused with 401. A request from another browser origin is refused with 403, and so is one with a foreign Host header (DNS rebinding) arriving on a loopback address; a request arriving on the non-loopback address --allow-remote opens gets no Host check. A 401 closes the connection without reading the body. Token or not, a peer cannot hold connections: a request's headers must arrive within 10 s and the whole request within 60 s, headers take at most 64 KiB (431) and a body 4 MiB (413), and a keep-alive connection idle for 120 s is closed, while a scan that takes longer than the read timeout still returns its result. An MCP session with no request for 10 minutes is closed (a call in it then gets 404), and at most 100 are open at once: a new one past that is refused with 503 and the reason. |
TestMCPStdioListsOnlyReadOnlyToolsAndListsFindings TestMCPHTTP TestMCPHTTPToken TestNormalizeMCPAddr TestMCPOptionsAreChecked TestMCPHTTPClosesIdleAndStalledConnections TestMCPHTTPRefusalClosesTheConnection TestMCPHTTPScanOutlastsTheReadTimeout TestMCPHTTPIdleSessionsExpire TestMCPHTTPSessionsAreCapped TestDocsMCPPageStatesTheHTTPLimits |
| MC-03 | scan runs the CLI's own scan and reads only the cluster --kubeconfig, --context and the environment name, with no other default (no ignore file is looked up, and there is no gate). Without --context, the kubeconfig's current context at start is pinned, so a later context switch does not move the server; when there is none to pin at start (no kubeconfig, no current context, an unreadable kubeconfig, a pod with no kubeconfig, where mcp does not fall back to the service account as scan does), every scan is refused, naming --context, even after a context becomes current, and the start message says so. A call names target minors and nothing else: a kubeconfig, a context or a directory of manifests in the call is refused. A call reads the cluster once and judges it at each of at most four targets; one scan runs at a time, and a call the client cancels stops its cluster read, or its wait for another scan, and reports nothing. Over HTTP, so does a call whose connection the client closes (on any protocol version), so another session's scan starts at once. |
TestMCPScanHonoursOnlyKubeconfigContextAndTheEnvironment TestMCPScanWithoutAContextAtStartIsRefused TestScanInputIsChecked TestMCPScanFeedsTheOtherTools TestMCPScanTargetsAndErrors TestScanFeedsTheOtherTools TestScanHonoursTheCallsContext TestRunMCPScanStopsWhenTheCallEnds TestMCPHTTPDisconnectCancelsTheCall |
| MC-04 | The tools' output is the published contract. get_report's output schema is api/report.schema.json; the output schemas of scan and list_findings refer to its definitions; every golden report of the engine, once it carries the CLI's envelope, validates against it, and the output of scan, get_report and list_findings validates against it (list_findings also over HTTP). |
TestReportSchemaValidatesGoldenReports TestOutputSchemasRejectWhatTheContractForbids TestOutputSchemasAreThePublishedContract TestMCPStdioListsOnlyReadOnlyToolsAndListsFindings TestMCPScanFeedsTheOtherTools TestMCPHTTP |
| MC-05 | list_findings filters by severity and category and says when limit cut the list. report_file and inventory_file read only a regular file: a directory, a FIFO, a device or /dev/stdin (on stdio, the client's own frames) is a tool error at once, and on Unix the open does not wait on a FIFO swapped in after the check (Windows has only the check). A regular file whose read blocks gives up when the client cancels the call. They read only an upgradescope report that follows api/report.schema.json, of at most 8 MiB, or an inventory of at most 20 MiB, through that bound whatever size the file reports (0 under /proc) or grows to after it is checked, and anything else is refused as a tool error with the reason rather than scored or failed as a protocol error ({"schemaVersion":1} included, a report given as an inventory does not score 100, and a report judged from an inventory is checked against the schema too). An inventory is checked as ingest checks a push (inventory.Admit, which the server uses): an invalid identifier, a serverVersion that is not 1.x, a collectorSchema this build does not know (later, or negative) and a value over the limits are refused, and a capability reason over its limit is cut and judged. A report (from a report_file, the fleet server or an inventory judged at a target) is also refused when an object repeats a member name, or has a member that differs only in case from a property of the schema beside it (Target beside target, FINDINGS, a finding's SEVERITY), because the schema leaves objects open and a case-insensitive decoder would read the member the schema check never judged: list_findings reads exactly target, kbVersion, findings and each finding's severity and category, so it and get_report cannot disagree about one document, and a report judged at 1.38 is never taken for target 1.40. The reason names the field and the rule and quotes nothing of the file, neither a value nor a map key. |
TestTheReportReadersDecodeExactlyTheNamesTheSchemaCheckJudged TestAReportWithAnAmbiguousMemberNameIsRefused TestARefusalAtTheRootReadsAtTheTopLevel TestACaseFoldedMemberCannotMakeGetReportAndListFindingsDisagree TestMCPInventoryFileIsJudgedAsIngestJudgesIt TestAdmitChecksAsIngestDoes TestAdmitErrorsKeepIngestsMessages TestValidateLimitsRefusesBeyond TestIdentifierErrorUnquoted TestReadFileIsBoundedWhateverTheFileSays TestReadFileReadsAFileThatReportsNoSize TestAnInventoryReportOutsideTheSchemaIsAToolError TestListFindingsFromReportFile TestSourceErrorsAreToolErrors TestReportFileIsBounded TestReportFileMustBeARegularFile TestReportFileRefusesWhatIsNotAFile TestMCPStdioDoesNotReadItsOwnStdin TestMCPInventoryFileIsBoundedAndARegularFile TestReadFileDoesNotBlockOnAFIFOSwappedIn TestAReadThatBlocksGivesUpWhenTheCallEnds TestInventoryIsJudgedUnderTheCallsContext TestErrorsDoNotQuoteTheFile TestMCPStdioListsOnlyReadOnlyToolsAndListsFindings |
| MC-06 | Fleet mode (--server-url with the read token from --read-token, --read-token-file or $UPGRADESCOPE_READ_TOKEN) reads the serve REST API, so the server's own read authentication applies: against a server that requires a token, a call without it, or with a wrong one, comes back as the tool's error carrying the server's 401, and with it the same call works. An open server needs no token. A redirect is not followed, a server report that does not follow the schema is a tool error with the reason, and a response over 8 MiB is refused; a read token over plain http:// to a host that is not loopback is warned about. |
TestMCPFleetModeHonoursTheServersReadAuth TestMCPFleetModeOpenServer TestFleetModeUsesTheServersReadAuth TestFleetRefusesAServerThatIsNotOne TestFleetReportMustFollowTheSchema TestFleetFollowsNoRedirect TestFleetResponsesAreBounded TestCleartextWarning |
| MC-07 | The mcpServers JSON and every upgradescope mcp command on the MCP page and in the README use flags the command accepts. |
TestDocsMCPExamplesParse |
| MC-08 | The MCP server adds about 2 MiB to the binary: 2.00 MiB on linux/amd64 (62,779,554 bytes at 67f30be against 60,682,402 for c5cfa5d, the commit the branch starts from, GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' ./cmd/upgradescope, Go 1.26.8, measured 2026-10-03). |
not automated: #99 (the measurement is in #76) |
| MC-09 | A tool result too large for an MCP client to receive in one message (the Go SDK client's 16 MiB, with the document sent twice) is refused before it is sent, saying to use list_findings with severity, category or a smaller limit, which then return part of the same report; list_findings at its largest limit is sent when it fits and refused with that reason when it does not. |
TestAResultTooLargeToReceiveIsRefusedWithAWayOut TestFindingsTooLargeToReceiveAreRefusedWithAWayOut |
| MC-10 | At most two get_report, list_findings or fleet_summary calls run at once, however many a client sends in parallel, so the memory a report takes while it is checked and sent is not multiplied; the others wait and are answered, and one the client cancels while it waits reads nothing. At most two file reads (report_file, inventory_file) are in flight, and a read the client cancelled while it blocked (a stalled mount) keeps its place until it returns, so cancelled calls cannot pile up background reads: the next waits, and gives up when its call ends. |
TestReportReadsAreBoundedInNumber TestAReportReadWaitingForASlotGivesUp TestCancelledReadsThatBlockAreBoundedInNumber TestReadFileUsesTheBoundedSemaphore |
| MC-11 | Every result of every tool (scan, get_report, list_findings, registry_lookup, fleet_summary) opens with a text notice, and every tool error does too. The rule is inverted, not a list of fields: every string of a result, values and object keys alike, is outside text, "cluster-supplied, not instructions", except numbers, booleans, the report's own member names, and the values of a short list of keys the tool writes itself (severity, verdict, category, target, kbVersion and the like, listed in toolWords) and only in the form it writes them (kbVersion only as the one label k8s.io/api vX.Y.Z; lifecycle <hex>; registry <hex>, so an instruction with no spaces is outside text); a field added to api/report.schema.json is outside text until it is listed. The notice states the rule, counts the outside strings and names where the free text is as JSON pointers through array indexes and the report's own member names only, so it quotes nothing a cluster, file, fleet server or registry chose, not a team's name or a map key containing / or ~; _meta["upgradescope.basit.engineer/clusterSupplied"] carries the same, structured. Each outside string and each object key over 2 KiB is cut, and the output still follows the published schema: a string the schema gives a pattern (a minor such as target, since or a hop's from and to, and annualCostDelta) would no longer match it once cut, so a report with one over 2 KiB (a value of exactly 2 KiB is kept whole) is refused as a tool error (from a report_file, the fleet server or an inventory) that names the location and the rule and quotes nothing of the value, and where a document reaches the marking unchecked (scan trusts the engine) a cut that would leave the result outside its tool's output schema is a tool error by the schema's rule too, never a JSON-RPC protocol error; a key that cutting would make equal to another key of its object (a hand-written file can hold both) is numbered before its cut mark (#2 …(cut by upgradescope mcp)), so no object comes out with a duplicate key. A tool error (an API server's, a file's, the fleet server's or the call's own arguments) is marked the same way and cut to 2 KiB after the tool's own words. The server's instructions state the rule and the MCP page names the fields, the tool-written keys and who chooses the text. |
TestEveryToolMarksOutsideText TestTheNoticeQuotesNoClusterKey TestToolWordsAreTheReportSchemas TestKBVersionIsOnlyTheShapeUpgradescopeWrites TestClusterSuppliedTextIsMarkedAndCut TestMarkClusterTextKeepsTheDocument TestCutKeysStayDistinct TestAReportFileWhoseNamesCutToOneFormKeepsDistinctNames TestAnOverlongPatternValueIsRefusedNotCut TestThePatternValueBoundIsOverTwoKiB TestACutThatBreaksTheOutputSchemaIsAToolError TestANormalReportIsUnchangedByTheChecks TestInstructionsSayClusterTextIsData TestMCPScanMarksTheClustersText TestDocsMCPPageSaysWhatTheClusterChooses |
Security¶
| ID | Claim | Proven by |
|---|---|---|
| SE-01 | Ingest and reads take bearer tokens (the Bearer scheme in any case); with no read credential (no --read-token, no read token minted with tokens create --read, no trusted team header), serve refuses to run unless the address it actually bound is loopback (localhost mapped elsewhere, a host name or :8080 is refused, and nothing is served in between) or --allow-anonymous-read is set (--require-read-credential, SE-24, starts on any address: the read API is then never open). --read-token equal to --ingest-token is refused, by serve and by server.New. |
TestIngestAuth TestReadAuth TestServeAnonymousReadGuard TestFleetReadAuth TestStartRejectsNonLoopbackBoundAddr TestStartWithScopedReadsOnNonLoopback TestBearerSchemeCaseInsensitive TestServeRefusesReadTokenEqualToIngestToken TestNewRefusesReadTokenEqualToIngestToken |
| SE-02 | A per-cluster token writes only its own cluster (403 before anything is written) and can be revoked. A push authenticated before its token is revoked stores nothing and answers 401, a duplicate push included. A push with a per-cluster token whose cluster is deleted or renamed at any point after it was authenticated (while its body uploads, while it waits for its ingest slot, or after its lookup) stores nothing and answers 409 "renamed or deleted": the token's cluster id is resolved with the token, and the commit re-checks the token and the cluster in its own transaction, so a revoked token never writes, a deleted cluster is not registered again by it and a rename leaves no second cluster with the same UID. A push with the shared ingest token names its cluster only in its body, so for it the guarantee starts at the server's cluster lookup, after the body is read: a delete or rename from then on is a 409 that stores nothing, and one that lands before it lets the push register the name like a fresh push (#369). Both stores pass the same conformance cases. | TestIngestPerClusterTokens TestIngestMismatchedTokenWritesNothing TestTokensRevoke TestTokensRevokeByID TestPushRacingAnAdminOperationWritesNothing TestPerClusterPushRacingRenameBeforeLookupAnswers409 TestPerClusterPushRevokedBeforeLookupAnswers401 TestSharedTokenPushRacingDeleteBeforeLookupRegistersTheName TestSQLiteConformance TestPostgresConformance (the CommitRechecksIngestToken, CommitRefusesChangedCluster and IngestTokenCluster subtests); on the built binary over HTTP, with no cluster, a throttled push (curl --limit-rate) held mid-upload while clusters rename or clusters delete runs, asserting the status and clusters list for both token kinds (per-cluster: 409 and nothing stored; shared: 202 and the name registered again): hack/claims-server.sh make claims-server ci:repo-checks |
| SE-03 | A per-cluster token is printed once; the server keeps its sha256 hash and its first 8 characters (which tokens list shows), never the token, and tokens create and the docs say so. Every tokens command the docs show parses. |
TestTokensCreatePrintsTokenOnceAndStoresHash TestTokensCreateMessage TestTokensList TestDocsTokenCommandsParse |
| SE-16 | The trusted-proxy mode is off by default and needs both --trust-team-header and --trusted-proxy-cidr (a set of ranges that together hold every address of IPv4 or of IPv6 is refused, 0.0.0.0/0 and ::/0 alone included, however the ranges are split across flags; a partial cover such as 0.0.0.0/1 is accepted, keeping the ranges to the proxy being the operator's job): the header sets the scope only on a request whose TCP peer is in those ranges, never the whole fleet (* in it is a team), and is ignored from anywhere else, where it can neither open nor widen a read. The oauth2-proxy example holds the first two of the three conditions the docs state for the mode to be safe: serve listens on 127.0.0.1 in the pod and the one Service exposes only the proxy's port, and the proxy, pinned to oauth2-proxy v7.15.5, pins --skip-auth-strip-headers=true, under which that version deletes a client's copies of the headers it injects (X-Forwarded-Groups, in any case, with - or _) on every route and sets the header from the session only, as one comma-joined line. Its skip-auth routes match the requests machines make, the gate's with its query (that version matches the path without the query), and no dashboard read; with --reverse-proxy the proxy matches them against a client's X-Forwarded-Uri, so the server's token check, not the proxy's sign-in, is what guards a read, as the docs say. It trusts only connections over the pod's loopback (its sidecar's, and, as the docs and the manifest warn, kubectl port-forward's to the pod), opts the pod out of Istio and Linkerd sidecar injection, leaves the client's Authorization header alone (--pass-basic-auth=false, no --pass-authorization-header or basic-auth password) and parses as serve. In this mode every read answer varies on the header and Authorization, so a shared cache in front of the proxy cannot serve one team's answer to another. Only space and tab around a listed team are trimmed, never another Unicode space, and the docs make it a third condition of the mode that nobody can name an identity-provider group that reads as a team they may not read (the list is comma separated and oauth2-proxy does not encode group names), with --allowed-group named as no filter of the claim. |
TestTrustedTeamHeader TestServeTrustedProxyFlags TestTrustedProxyCIDRsCoveringEverythingRefused TestPrefixesCoverFamily TestOAuth2ProxyExample TestDecodeTeamsTrimsOnlySpaceAndTab deploy/examples/oauth2-proxy/upgradescope-oauth2-proxy.yaml |
| SE-17 | Read tokens can be scoped to teams (tokens create --read --teams a --teams b, one free-text team per flag, or * for the fleet), stored as their sha256 hash and first 8 characters, listed and revoked (tokens list --read, tokens revoke --read --id <id>) without a restart; a revoked one is refused, and revoking the last does not open the read API again. A team-scoped token reads, from every read endpoint, only the clusters one of its teams owns a namespace in (by the engine's namespace attribution, after --team-map, as its current evaluations written with the current team map record it: a removed target's never-rewritten evaluation does not count) and of those only its teams' findings and team scores, a finding that spans teams (one API in several teams' namespaces) cut to its teams, their namespaces and the objects in them, with a title that counts only those, a detail that names only those and none of its folded apiserver caller rows (callers, which no team owns) (a finding whose namespace list the engine capped is always cut, so its namespace count and "and N more" are never shown), suppressed findings likewise; a --team-map team is free text, as before, and is named in the trusted header and X-Upgradescope-Teams by the team list encoding (percent-encoded, comma separated), except that a team called * is read as (*) with a startup warning (which also says so when the map names a team (*) too, the two then being one team); a scoped answer is sent with Cache-Control: private, no-store; a scoped report has no unrecognized images, and the helm capability's reason and skipped releases are withheld wherever gaps appear; a gate with ?cluster= evaluates the manifests within the scope's share of the cluster only, never the whole cluster (its teams' namespaces' usage, custom resources, releases, GitOps charts, add-ons and in-tree volume plugin counts, and the cluster's version, nodes, CRD definitions and capabilities; no cluster-scoped usage, unbound PersistentVolume or StorageClass counted on a volume plugin, live PersistentVolume or StorageClass name, unowned namespace or apiserver caller row), so a finding the PR's objects are in counts, titles and details only the PR's and the scope's evidence, even when the PR names another team's namespace, and the verdict, its header, the status under every ?fail-on, the cluster verdict and the score are the share's: clusters that differ only outside the scope give the same answer byte for byte, in every format, even where the whole cluster's findings, score, verdict and status differ (so a PR that breaks only another team's workloads passes a team-scoped gate, as the docs say, with no marker that would reveal it), except where the collector's 100-objects-per-API cap, which lists objects across the cluster in namespace order, left some of the scope's objects unlisted: then which of them are listed, objectsOmitted, the finding's detail and what is derived from the objects the collector listed (whether a Helm release's stored-manifest object gets a finding of its own, in the baseline as in the proposed state; whether the scope's objects' upgradescope.basit.engineer/ignore annotations, and ?config= rules that select objects by namespace, name or file, accept a finding: they accept listed objects only (a namespace rule takes a finding that lists none whole when it matches every namespace), so a finding with some of the scope's objects unlisted stays open, the cluster's; and with these the findings and suppressed findings, suppressedCount, the score, the cluster verdict and the answer's size) depend on how many objects at that API sort before them, as a read's cut finding (the objects and namespaces it lists, its title's count, its capped note) depends on the collector's and the engine's caps, and the docs say what a scoped caller can learn from that; the verdict, its header and the status never do; the room the PR's own objects take in a finding's 100-object listing changes which of the cluster's refs it lists (the cluster's are cut first, after evaluation and suppression, which see every ref), objectsOmitted, and with them the answer's size and whether it is over --max-gate-bytes (a 413), and nothing else, for scoped and fleet-wide gates alike: a ref the PR pushes out of the listing is still matched to a release's stored copy, still accepted by its annotation or a rule selecting it, and still counted in what is worded from the objects (the detail, the CRD-version remediation's note on managers); a cluster row that lists only the PR's objects while it counts the cluster's is worded as both, not as manifests; one without it answers as fleet-wide; any other cluster answers the same 404 as an unknown one, on every per-cluster endpoint and /gate?cluster= (which, for a scoped read, matches a name among the scope's clusters only and never looks up an id outside it), and is absent from /clusters, /fleet and /fleet/teams; /metrics answers it 403. --read-token, a * token and the admin token answer byte for byte what --read-token answered before. Both stores pass the same read-token conformance cases. |
TestScopedTokenReadsOnlyItsTeams TestScopedReadCutsFindingsThatSpanTeams TestScopedReadCutsCappedNamespaceList TestScopeCutsSuppressedFindings TestScopeCutDropsFoldedCallers TestScopedGateAnswersFromTheScopesShareOnly TestScopedGateIsDecidedByTheScopesShare TestScopedGateScoreIsNoProbe TestScopedGateDoesNotMergeAnotherTeamsUsage TestScopedGateObjectListingFollowsTheCollectorsCap TestScopedGateScoreFollowsTheCollectorsCap TestScopedGateSuppressionFollowsTheCollectorsCap TestGateHelmVerdictIgnoresTheRefsThePRPushesOut TestGateSuppressionIgnoresTheRoomThePRTakes TestGateAnswerListsThePRsRefsFirst TestEvalAPIUsageManifestsInAClusterRow TestCRDUsageManifestsInAClusterRow TestClusterShare TestClusterShareVolumePlugins TestScopedGateVolumePluginsAreTheScopesShare TestManifestsOwnVolumePlugins TestParseTeamMapKeepsFreeTextTeams TestParseTeamMapWarnsWhenStarTeamIsTaken TestScopedReadsOfFreeTextTeams TestTokensCreateReadFreeTextTeams TestScopedTokenOutOfScopeClusterIsNotFound TestTeamMapChangeLeavesTheOldTeamsScope TestEvaluationWithoutTeamsEntersScopeAfterThePass TestOutOfScopeClusterIsNeverLookedUp TestFleetWideTokensReadAsBefore TestFleetWideReadsMatchMainBeforeReadScopes TestRevokedReadTokenIsRejected TestEvaluationWithUnknownTeamsIsStale TestReadTokenStoresOnlyHashAndPrefix TestEvaluationWrittenWithoutTeamsIsUnknown TestTokensCreateRead TestTokensCreateReadRejectsBadScopes TestTokensListAndRevokeRead TestSQLiteConformance TestPostgresConformance (the ReadTokens and ClustersOfTeams subtests) |
| SE-04 | Tokens, webhook URLs and the database URL can come from the environment or a file instead of argv; a value from either is trimmed of surrounding whitespace, and one of whitespace only is refused. | TestSecretFlagPrecedence TestServeSecretsFromEnvAndFiles TestAgentServerTokenSources TestServeValidatesNotificationURLs |
| SE-18 | When serve listens on loopback, or trusts a team header, or the read API is open and the request presents no bearer (an anonymous read of an open read API: decided on each request, on any listen address, a wildcard :8080 included, so minting a read token ends it without a restart, within one second of the mint (SE-25), and a store that cannot say whether the API is open counts as open), a request whose Host is not localhost, a loopback address, the address the request arrived on (an IP literal: probes and scrapes of the pod's IP), the --listen host (not 0.0.0.0 or ::, which are addresses to bind) or an --allowed-host name (any port; repeatable, comma separated or $UPGRADESCOPE_ALLOWED_HOSTS; a port, scheme or wildcard is refused at startup) gets 421 before any route, token, team header or scope is looked at, on every path (the API, /metrics, the dashboard, /healthz, ingest and the gate), counted as upgradescope_http_requests_total{route="host-refused",code="421"} and logged at most once a minute (the Host quoted, escaped to ASCII and cut to 256 bytes, with the refusals since the last line), so a DNS-rebinding page reads nothing from an open loopback read API or through a kubectl port-forward in header mode; on a routable address with a read credential (--read-token, a minted read token or --require-read-credential) and no trusted header any Host is answered, as before, and on an open read API a request that presents a bearer is left to its credential (an unknown one is 401, SE-24). The chart passes the server Service's four DNS names (the in-chart agent's push host among them; the last ends in the clusterDomain value, default cluster.local, which the cert-manager Certificate uses too), server.ingress.host and server.allowedHosts (a port or scheme fails the render) as --allowed-host, its probes set no Host, and the oauth2-proxy example sends serve its upstream's Host (--pass-host-header=false, 127.0.0.1:8080). |
TestLoopbackServerRefusesForeignHosts TestTrustedHeaderModeRefusesForeignHosts TestHostCheckPrecedesCredentials TestHostRefusalsAreCountedAndLogged TestHostAllowed TestAllowedHostsSkipAnUnspecifiedListenHost TestNewRefusesInvalidAllowedHosts TestRoutableServerWithTokenAnswersAnyHost TestWildcardOpenReadAPIRefusesForeignHosts TestWildcardOpenReadAPIKeepsProbesAndScrapes TestWildcardServerWithAReadTokenAnswersAnyHost TestOpenReadAPIHostGuardFollowsTheStore TestOpenReadAPIQueriesTheStoreOncePerWindow TestOpenReadAPIHostGuardLeavesBearerRequestsToTheirCredentials TestLoopbackBindKeepsItsGuardWithAReadToken TestServeAllowedHosts TestServerAllowedHosts TestServerAllowedHostsClusterDomain TestServerAllowedHostsSchema TestOAuth2ProxyExampleSendsTheUpstreamHost |
| SE-19 | Slack and generic webhook URLs never reach the server's log or the outbox's last_error past their scheme and host (https://hooks.slack.com/…): not the path, query, fragment or userinfo, for a refused connection, a timeout, a URL that does not parse or an error that quotes the URL, and the host is withheld too when the URL holds an @, and then the transport error is cut to its final reason (no such host, connection refused, i/o timeout or the TLS reason), so neither the DNS name nor the dialed ip:port (a resolved address included) reaches the log or last_error; a URL without an @ keeps its host and Go's own wording of the cause, for diagnosis; the outbox scrubs the sink's URL from every error itself before logging or storing it, whether or not the notifier redacted it. serve refuses to start unless --slack-webhook and --webhook are absolute http(s) URLs with a host, naming the flag and never the value, nor a fragment of it: CheckURL gives a fixed text per failure class (bad port, bad escape, bad host, control character, no scheme) and never net/url's own error, which quotes part of the value. |
TestDeliveryErrorsNeverCarryTheURLsSecrets TestDeliveryErrorCauseIsScrubbed TestRedactURL TestOutboxNeverLogsOrStoresSinkURLs TestOutboxWithheldHostIsNotInTheDialError TestWithheldHostKeepsTheTLSReason TestOutboxScrubsTheSinkURLFromAnyError TestCheckURL TestServeValidatesNotificationURLs |
| SE-20 | clusters list, delete and rename with --server-url (or its deprecated alias --server) never follow a redirect: any 3xx is an error naming its status and Location, and the redirect target is never asked anything, so an http→https 301 or 302 cannot turn a delete or rename into a GET reported as done. |
TestClustersRefuseRedirects |
| SE-21 | The table and Markdown output, the warning: lines and the error printed on stderr show control characters from a manifest, a file name or a config as visible escapes, never as themselves: the C0 controls (a tab, CR and a newline too, so a field stays on its line), DEL, the C1 controls, the bidirectional controls U+202A to U+202E, U+2066 to U+2069, U+200E, U+200F and U+061C, U+2028 and U+2029, and bytes that are not UTF-8 (\x1b, \r, \n, \u202e). Markdown shows the whitespace controls as escapes too (it escapes before it folds spaces), and a newline in a file name quoted by an error is an escape, not a second line of stderr: the only newlines the error text keeps are those between the errors of a newline-joined multi-error (errors.Join's, or several %w joined by newlines alone). The clusters list and tokens list tables escape names the same way, and so do the suppression warnings of scan and of the mcp scan tool; the startup log and errors of serve --team-map, and the server's log lines that name a cluster (the notification outbox's among them), quote names with %q, which escapes the same characters. JSON, SARIF and JUnit are not escaped by this: their encoders write no raw control byte from a field (only their own formatting newlines), but they keep U+202E and the C1 controls as the valid characters they are (a JSON report printed to a terminal can still carry them). |
TestEscape TestEscapeLeavesNothingUnsafe TestTableAndMarkdownEscapeEveryField TestTableEscapesObjectNamesTitlesDetailsAndFiles TestMarkdownShowsTheEscapeNotTheCharacter TestMarkdownShowsWhitespaceControlsAsEscapes TestScanFilesNeutralisesHostileManifestText TestMCPScanSuppressWarningEscapesObjectNames TestTeamMapLogsAndErrorsQuoteHostileText TestAdminListsEscapeNames TestErrorTextEscapesNewlinesInQuotedNames TestErrorTextKeepsOnlyTheJoinSeparators TestMainPrintsErrorsEscaped TestOutboxLogsQuoteClusterNames TestServerLogLinesQuoteClusterNames |
| SE-22 | On a pull_request the gate trusts the pull request's own tree: the .upgradescope.yaml found in the scan root or at the repository root, the upgradescope.basit.engineer/ignore annotations in the manifests, the baseline file and the workflow, so a pull request can suppress its own findings. --config (the Action's config) names the one config read and stops that search; annotations always apply and have no off switch. docs/getting-started/ci-gate.md, docs/guides/suppressions-and-baselines.md and action/README.md say so and show one workflow that takes the config and baseline from the base commit, and an annotation guard. The workflow copies the two files as its last step before the gate, after removing the destinations, so a pull request cannot have committed them as symlinks to a file that an earlier step (such as helm template) writes, and it clears the path the base commit is checked out to. |
TestGateTrustsTheFilesOfThePullRequest TestGateNamedConfigStopsDiscoveryButNotAnnotations TestDocsSayWhoCanTurnTheGateOff TestDocsAnnotationGuardMatchesTheReport |
| SE-23 | serve re-reads --ingest-token-file, --read-token-file, --admin-token-file, --slack-webhook-file, --webhook-file and --webhook-secret-file when they change, checked at most every 5 seconds and only when a request (or a delivery) needs the value, swapped atomically: a rotated read, ingest or admin token takes effect and the old one is refused, with no restart. A new file that is empty, unreadable, larger than 64 KiB, or that would make two of the read, admin and ingest tokens equal (any of the three pairs, in either direction, and also when two files change at the same moment: their reloads run one at a time) keeps the old value, so an emptied read-token file never opens the read API, and logs an error naming the file and never a value. --optional-secret-file names the files that may be missing or empty at start (a key an existingSecret lacks): such a token or key that appears later is used, and a removed ingest-token file stops that token working (logged once, at INFO: it is the revocation the operator asked for, and an optional file that was never there logs nothing at all), which is what deleting the key from an existingSecret does only where the chart mounts it as optional (a hub with no in-chart agent; with the in-chart agent the key is required, deleting it revokes nothing and the kubelet stops updating that volume, reasoned from its behaviour and not reproduced on a cluster); a required ingest-token file, and every other file, that goes away keeps the old value and logs an error naming the file, so rotating by deleting and rewriting a file revokes nothing; a notification URL missing at start configures no sink, so one added later needs a restart. A delivery to a webhook reads its URL once, and scrubs its errors with the URL it posted to, not one a rotation changed meanwhile. A value from a flag or the environment is read once. server.New refuses an admin token equal to the read or ingest token, as the CLI always did. Safe under -race. |
TestReadTokenRotatesWithoutARestart TestIngestTokenRotatesWithoutARestart TestAdminTokenRotatesWithoutARestart TestEmptyOrUnreadableTokenFileKeepsTheOldToken TestRotationToACollidingTokenIsRefused TestRotationToACollidingTokenIsRefusedInEveryDirection TestSimultaneousCollidingRotationsNeverLeaveEqualTokens TestReloadGroupSerializesReloadsWithoutBlocking TestOutboxScrubsWithTheURLItPostedTo TestOptionalIngestTokenFileAppearsAndRevokes TestRequiredIngestTokenFileRemovalKeepsTheOldToken TestNewFailsOnABadTokenFile TestTokenRotationUnderConcurrentRequests TestNotifiersFollowTheirURLAndSecretSources TestServePassesSecretFilesForRotation TestServeOptionalSecretFile TestRotationIsSeenWithinTheCheckInterval TestChecksAtMostOncePerInterval TestSymlinkSwapLikeAKubeletSecretVolume TestConcurrentValueAndRotation TestOptionalFileMayAppearLaterAndRemovalCanRevoke TestOptionalFileMissingOnPurposeLogsNothing TestOptionalRemovalIsLoggedOnceAtInfo TestRequiredFileMissingStillErrors TestSwapToAnotherInodeWithSameSizeAndMtimeIsPickedUp TestEachDistinctFailureIsLoggedOnce |
| SE-24 | serve --require-read-credential keeps the read API closed whatever the database holds: it is never open, so a read with no credential, or with a bearer nothing knows, is 401 on every read endpoint and the gate even when the store has no read token, and the same after the store is swapped for an empty one (a lost PVC, an emptyDir restart, a restored older backup); serve starts on a non-loopback address with it and refuses it together with --allow-anonymous-read (by serve and by server.New); while no credential exists (no --read-token, no active read token, no trusted team header) it logs a WARN saying reads return 401 until one is minted and how, and does not when one exists. Without it, the read API is open only while no read credential exists (a lost or emptied database reopens it, as the docs and the backup section say), and on an open read API a request that presents a bearer matching no credential (not --read-token, the admin token, a stored read token or a trusted team mapping) is 401, never the whole fleet, while a request with no bearer still reads and a non-Bearer Authorization is no bearer; /healthz and /readyz need no credential. tokens create --read works against a SQLite database the running server holds open (a second store handle on the one WAL-mode file; the server answers 200 to the minted token on its next request, with no restart, and the API stays closed to anonymous reads). On Postgres only the store half is tested, by the pg-conformance job: a read token created, and then revoked, through a second store handle on the same schema is listed, validated and refused by the first handle's next query, with no restart of either (TestPostgresConformance, subtest ReadTokenMintedThroughAnotherHandle); the tokens create process against a Postgres the running serve holds, with the URL taken from the container's UPGRADESCOPE_DB_URL as the install notes' command does, is not run by any test. The chart passes --require-read-credential when server.readToken is empty and neither server.allowAnonymousRead nor server.ingress.allowAnonymousRead opts into an open read API (then --allow-anonymous-read), passes neither flag with a read token, refuses a server.extraArgs entry that contradicts the read-API flag it passes (--allow-anonymous-read, bare or true, while it closes the API; --require-read-credential, bare or true, while it opens it; either set false when it is the one the chart passes, which would undo it; both set true, which serve refuses) and accepts the other flag set false, which is its default (every strconv.ParseBool spelling), renders an Ingress in front of a closed read API, requires server.ingress.allowAnonymousRead for an open one, fails a metrics.serviceMonitor that would scrape a closed read API with no read token to send, and its install notes print the first-token command for SQLite and Postgres. |
TestRequireReadCredentialEmptyStoreStartsAndRefusesReads TestRequireReadCredentialSurvivesAStoreSwap TestNewRefusesRequireReadCredentialWithAllowAnonymousRead TestRequireReadCredentialAcceptsEveryCredential TestOpenReadAPIRefusesAnUnknownBearer TestOpenReadAPIBearerValidationErrorIsNotAnOpenRead TestRequireReadCredentialStartupWarning TestOpenReadAPIStartupWarningRemains TestServeRequireReadCredential TestServeRequireReadCredentialStartsOnAnExposedAddress TestReadTokenMintedWhileServeHoldsTheDatabase TestPostgresConformance TestServerReadAPIFlags TestServerExtraArgAllowAnonymousReadIsRefused TestServerExtraArgReadFlagsMustNotContradictTheChart TestIngressAndTheReadAPI TestServiceMonitorNeedsAReadCredentialItCanSend TestReadAPINotes make chart-test |
| SE-25 | On an open read API the question "is a read token minted yet?" is a store query that every anonymous request asks, whatever its path (the dashboard, assets, /healthz), including the ones whose Host the server does not answer for, before any concurrency limit. serve keeps the answer "no" for one second (readOpenWindow, on a monotonic clock, not the clock that tests pin): any number of anonymous requests, concurrent ones sharing a single query, make at most one ListReadTokens per second, and the next request after the window asks again. An error from the store is never kept (a failing store is asked again by the next request and is read as open for the Host check, as before). "Yes" is kept for good, so a closed API asks nothing more. The price is documented: a first read token minted by another process closes the API to anonymous reads, and ends the Host check for them, within that second, not on the next request; the token itself is looked up on every request and works at once. --read-token, --require-read-credential and a trusted team header never ask. |
TestOpenReadAPIQueriesTheStoreOncePerWindow TestOpenReadAPIWindowRenewsAndDoesNotCacheErrors TestOpenReadAPIConcurrentRequestsShareOneQuery TestOpenReadAPIHostGuardFollowsTheStore TestRevokedReadTokenIsRejected |
| SE-26 | clusters list, delete and rename warn on stderr, once, when a token would go over plain http:// to a host that is not loopback (the read token for list, the admin token for delete and rename); the warning names the host and never contains the token, and it is a warning only: the request is still made. Loopback hosts (localhost, 127.0.0.1, ::1), https:// URLs and calls with no token print none. tokens talks to the database directly and is out of scope. |
TestClustersWarnOfCleartextTokens TestMCPWarnsOfCleartextReadToken TestCleartextWarning |
| SE-27 | clusters --server-ca-file and mcp --server-ca-file trust a private CA's PEM bundle on top of the system roots, for the server API they call (agent.LoadServerCAs, as the agent's push does): they work against a TLS server whose certificate that CA issued and fail with an x509 error without the flag (Linux: signed by unknown authority; macOS: certificate is not trusted, which is not an x509.UnknownAuthorityError; Windows: the platform verifier's own error, read from Go's source and not run); there is no option to skip verification, and redirects stay refused. The flag needs a server URL and an https:// one, and a bundle with no certificate or a certificate that does not parse stops the command. $SSL_CERT_FILE and $SSL_CERT_DIR are the older workaround on Linux and other Unix systems only: Go ignores both on macOS and Windows, where the flag is the way. |
TestClustersTrustServerCAFile TestClustersServerCAFileIsChecked TestMCPFleetTrustsServerCAFile TestMCPServerCAFileIsChecked TestServerCAUsageQualifiesSSLCertFile TestDocsQualifySSLCertFile |
| SE-28 | --server-url is the server flag on clusters as on agent and mcp; --server stays as a hidden alias that prints cobra's deprecation note, and both are exclusive with --db, --db-url and --db-url-file and with each other. serve --targets is a repeatable list: --targets 1.37 --targets 1.38 evaluates both, --targets 1.37,1.38 (the form the chart renders) is unchanged, and the 4-minor limit counts distinct minors over all repeats. |
TestClustersServerURLIsTheFlagAndServerStaysAsAHiddenAlias TestClustersRedirectErrorNamesServerURL TestServeTargetsRepeatAndCSV TestServeTargetsAsTheChartRendersThem |
| SE-05 | Snapshot bodies are capped at 20 MiB on the wire and after gunzip, and nothing is read before auth. | TestIngestBodyLimits TestSnapshotBodyCapIsConfigurable TestIngestEncodingErrors |
| SE-05b, PF-06 | Ingest and /gate memory is bounded by structure, not only bytes: JSON values and YAML nodes (aliases at what they expand to) are counted against a node budget before anything is decoded (413), with a meter fuzzed never to undercount the decoders, and a stream with its aliases expanded must fit the body cap (413; 136 documents within their own 4 MiB each answered 532 MB). Buffered bodies share one budget per endpoint, and one request at a time measures aliases, decodes and evaluates (503 with Retry-After when full or after the wait). A /gate answer is bounded before it is encoded, in every format, and over --max-gate-bytes is 413; ?path= is at most 512 bytes (422): unbounded, a 60 KB path made an 817 MB answer and names of < a 72 MB SARIF one. /gate encodes its answer in that slot and holds it for its client under the budget the reads use (SE-05c): 10 clients whose 4.6 MB /gate?cluster= answers they never read grow the live heap by at most that budget (unbounded, 10 against a 17 MB push held 320 MiB). A ?cluster= gate whose cluster lists 100 refs at each of the KB's deprecated GVKs and whose PR adds 100 more at each (200 refs per finding before the listing is cut to 100, names up to 199 bytes, a ?path=) grows the heap by ~30 MiB, where the test fails above the 200 MiB every /gate heap test holds a request to. A push is 422, naming the field and the rule, before anything is stored, unless its identifiers are what the apiserver accepts for what they name (cluster name an RFC 1123 subdomain of at most 253 bytes, as tokens create and the agent's --cluster-name also require; namespaces RFC 1123 labels; object names at most 253 bytes without / or %) and its other values within the limits collectors keep to (strings of at most 16 KiB, 100 objects per API usage entry, each group/version/kind once per list, 32 capabilities; the free text a collector copies whole from the cluster, capability reasons and skipped entries and the ignore annotations, is cut to them by the agent and at ingest, not refused, so ~320 forbidden CRD lists in one reason no longer get every push of an agent refused): namespace keys of 190 apostrophes had made a 21 MB push three 42 MB reports and a 200 MB HTML export. Every report the server evaluates, stores or exports is at most --max-snapshot-bytes (the engine stops building it there; 413 above), since within those limits a report still repeats what its push names (5 MB of Helm releases had built three 41 MB reports). Strings are stored and served as long as pushed: no HTML or line-separator escapes in stored reports and JSON responses, a dedup hash that holds no escaped copy, and pushes that are not UTF-8 refused (422); unescaped, names of < grew one ingest 323 MiB. The live heap of the worst inputs within the budgets, ?cluster= included, measured on SQLite against the dearest snapshots the server stores (object names of x, < and U+2028; namespace maps; teams; Helm releases, field managers, callers and capability gaps of characters the exports lengthen), adds up with one read, two reads of a 500-cluster fleet (their evaluations listing the most of what they could not assess), the responses held for their clients and the re-evaluation pass to ~865 MiB (the worst of six runs of each part), inside the ~921 MiB GOMEMLIMIT of the chart's 1Gi, measured at four serve --targets, the most it takes (more is refused at startup and by the chart's schema), with notifications configured, for a cluster's later push against an earlier snapshot as large (~216 MiB: a push reads the previous snapshot's head only, and each previous report only for its findings' keys and severities, not the inventory or any whole report) and a re-evaluation pass that changes every finding (~198 MiB): each extra target adds about one report of up to --max-snapshot-bytes to a push and one to the re-evaluation pass (18-33 MiB each at the default 20 MiB: ~54 MiB of the sum per extra target, ~650 MiB with none). UTF-16 streams are refused (422). Outside these bounds, each listed with its measured cost in the operations guide: open connections (uncapped, ~18 KiB of heap and stack each), the kernel's socket buffers, a fleet larger than 500 clusters, and snapshots a v0.1 server stored before the budgets. |
TestUnreadGateResponsesAreBounded TestGateSlotIsNotHeldWhileSending TestNewRefusesMoreThanMaxExtraTargets TestServeCapsTargets TestIngestDecodeHeapIsBounded TestSnapshotNodeBudgetPrecedesTheCollectorView TestIngestLoadsNoPreviousInventory TestBuildNotification TestComputeDelta TestIngestBufferedBodiesAreBounded TestJSONMeterIsExact TestGateDecodeHeapIsBounded TestGateClusterContextHeapIsBounded TestGateClusterRefsAtEveryGVKHeapIsBounded TestStoredSnapshotHeapIsBounded TestGateChargesAliasesAtTheirExpandedSize TestGateAliasCheckWaitsForTheSlot TestGateWildcardsAreNotAliases TestGateNodeBudgetCoversTheStream TestGateRefusesUTF16 TestGateBudgetReturnsToZeroOnEveryError TestGateChargesAliasesAgainstTheStreamCap TestGatePathLengthIsCapped TestGateAnswerBoundHolds TestGateAnswerOverTheLimitIs413 TestGateAnswerHeapIsBounded TestMarshalJSONDoesNotAmplify TestSnapshotStringsAreNotAmplified TestIngestRefusesInvalidUTF8 TestIngestRefusesInvalidIdentifiers TestIngestRefusesInvalidClusterNames TestIngestRefusesValuesBeyondLimits TestIngestAcceptsTheGoldenInventories TestValidateLimitsRefusesBeyond TestTokensCreateRefusesInvalidClusterNames TestAgentRefusesInvalidNames TestClustersRenameTakesOnlyValidNewNames TestRenameMigratesAnInvalidV01ClusterName TestCutFreeTextCutsToTheLimits TestRunStepsCutsFreeTextToTheLimits TestIngestCutsTheFreeTextOfOlderAgents TestIngestRefusesAReportOverTheLimit TestEvaluatingReadsRefuseAReportOverTheLimit TestEvaluateWithinStopsAtItsLimit TestEvaluateWithinIsEvaluateWithinItsLimit TestFindingNamespacesAreCapped TestCanonicalHashIsJSONMarshals TestMeasureYAMLNeverUndercounts TestMeasureYAMLDepthLimit TestMeasureYAMLIsExactOnRealisticManifests make test-heap (the in-pod RSS under load: PF-06b, under repair) |
| SE-05c | Reads are bounded too: every per-cluster read (detail, report, findings, teams, history, export) and the fleet teams rollup run one at a time in a read slot of their own (503 with Retry-After after a 30s wait), and only a what-if decodes the whole stored inventory. One such read costs up to ~133 MiB on SQLite (the HTML export of a report at the report limit, --max-snapshot-bytes), and ten concurrent requests to each whose clients take their responses at once, against every snapshot the heap tests store (the dearest to decode, those whose reports are as large as their inventories or about the report limit, of strings the exports lengthen), grow the heap by at most 160 MiB (measured ~133, no more than one; unbounded, ~400 MiB); an export over the report limit is 413. A response is sent after the slot is released, with its Content-Length, when it fits a budget of twice --max-snapshot-bytes (40 MiB) for responses waiting on their clients, which /gate and the fleet reads share; one that does not fit what is left is 503 with Retry-After, and one larger than the whole budget (never a /gate answer at the default caps) is sent in the slot under a 20s write deadline: 8 or 20 clients that never read a ~20.9 MB report, the largest a push stores, over real sockets, grow the live heap by at most that budget and its peak by at most 200 MiB with the read in the slot included (measured 142; unbounded, 20 that never read a 17.5 MB one held 366 MiB). /clusters, /fleet and /metrics load no inventory and no report: they read each evaluation's summary columns, notAssessed included as a bounded summary (each gap's capability cut to 64 bytes, its reason to 256, at most 3 skipped entries listed, and at most 1 KiB of gaps per evaluation, the required first: 50 clusters of 32 16-KiB capabilities had made /fleet a 108 MB answer that grew the heap 516 MiB, and 500, each evaluated at five targets, the default and four --targets, now grow it 11.4 MiB, on a GitHub-hosted runner, CI run 37152753746, half of them pushed as a v0.1.x agent would, whose summaries the test checks are the larger), so 30 concurrent requests to each grow the heap by at most 16 MiB (285-584 MiB when they loaded reports). They run two at a time in fleet slots of their own (503 with Retry-After after a 30s wait, 5s for a /metrics scrape) and hold their responses under the same budget: against 2000 clusters, 100 clients that never read any of them, a 16-target /fleet included, grow the live heap by at most that budget and its peak by at most 168 MiB (measured 59 to 125 MiB, /metrics the most, on a GitHub-hosted runner, CI run 37160469086, against fleets whose clusters carry the gaps a v0.1.x agent is given; unbounded, 201 MiB for /clusters, 251 MiB for /fleet, and 433 MiB for 30 of /metrics). /fleet?targets= takes at most 16 distinct minors (422 above; 8,718 held a fleet slot 2m13s and grew the heap 418 MiB), and without ?targets= it opens at most 16 columns, the rest counted in targetsOmitted (500 clusters pushed at 500 minors grew the heap 36 MiB uncapped, 1.9 MiB now). The budget bounds memory, not who gets it: clients that keep it full of unread answers make the answers that do not fit 503 for up to the 120s write timeout (documented in SECURITY.md). |
TestReadHeapIsBounded TestUnreadResponsesAreBounded TestUnreadFleetResponsesAreBounded TestFleetTargetsAreCapped TestFleetDefaultColumnsAreCapped TestMetricsWaitsLessThanAScrapeTimeout TestReadConcurrencyLimit TestReadSlotIsNotHeldWhileSending TestHeldBudgetBusyAnswers503 TestReadSlotSendsWhatTheHeldBudgetCannotTake TestOpenAPIReadBusyMatchesSpec TestFleetDecodesNoInventories TestFleetReadsLoadNoReport TestSQLiteConformance TestMigration0007BackfillsNotAssessedSQLite TestNotAssessedOfIsBounded TestFleetSummariesAreBoundedPerEvaluation TestFleetReadsOfTheWidestGapsAreBounded TestExportOverTheLimitIs413 TestFleetDefaultColumnsAreMeasured make test-heap |
| SV-11b | A body that does not arrive within the read timeout gets a retryable 408, on ingest and /gate; a chunked body over the cap is 413, not a retryable 503, whether or not the body budget has room; with the budget full, an Expect: 100-continue client gets the 503 before it sends its body. |
TestIngestBodyReadTimeoutIs408 TestGateBodyReadTimeoutIs408 TestGateChunkedOverflowIs413EvenWithoutBudget TestGateBodyBudgetFullSendsNo100Continue |
| PF-08 | GOMEMLIMIT is set to 90% of the memory limit when unset: by the chart from each container's limit (a byte count or a quantity), and by serve and agent from the cgroup's (v2 or v1) outside it; an explicit value wins. |
TestGoMemLimitFollowsTheContainerLimit TestMemoryLimitFromCgroup |
| SE-06 | Security headers on every response the server's handler writes: the dashboard, its assets, the API, exports and their errors, /healthz. The few responses net/http writes before routing (a 400 for a malformed request, a 431, a 501) carry none, and the security page says so. |
TestSecurityHeadersOnDashboard TestSecurityHeadersOnAPIAndExport TestDocsSecurityHeadersScope |
| SE-07b | The CSV export puts a ' before a formula trigger wherever a spreadsheet could start a cell (a field's start or after , ; tab or a line break), past any leading white space. |
TestCSVSafeBlocksFormulaInjection TestWriteExportCSV_FormulaAfterSeparator TestExportCSVGolden |
| SE-08 | --tls-cert-file and --tls-key-file serve HTTPS (TLS 1.2 minimum, Go's default cipher suites), both or neither, the pair validated at startup and re-read when either file changes (checked at most once a second on a new handshake), so a renewal needs no restart; a pair that does not load keeps the previous certificate in service. The agent's --server-ca-file trusts a private CA's PEM bundle for pushes on top of the system roots; it needs an https --server-url, and a bundle with a certificate that does not parse fails the start. There is no option to skip verification, and a CA file leaves redirects refused as AG-05 says. |
TestServeTLS TestNewTLSValidation TestServeTLSFlags TestServeTLSReloadsRotatedKeyPair TestServeTLSMinimumVersion TestPushTrustsServerCAFile TestPushWithServerCAFileNeverFollowsRedirects TestLoadServerCAsRejectsUnusableFiles TestAgentServerCAFileFlag TestRunAgentRejectsUnusableServerCAFile |
| SE-09 | The chart serves the server over HTTPS (server.tls: an existing Secret or a cert-manager Certificate in its own -server-https Secret); the in-chart agent then pushes to https:// trusting the Secret's CA (server.tls.caKey), probes and the ServiceMonitor use HTTPS, and the chart's Ingress is told the backend speaks HTTPS. The TLS Secret is mounted without subPath, so a renewal reaches the running server. agent.serverCA (one key of a ConfigMap or a Secret) reaches the agent as --server-ca-file. server.ingress (off by default) terminates TLS for its host. server.sharedIngestToken=false leaves only per-cluster tokens: the chart writes the shared token under data, not stringData, so an upgrade that switches it off removes the key (checked by rendering; the upgrade on a live API server is not automated), and a Secret written by rc.2 or earlier needs a one-time kubectl patch, which the values comment gives. The agent warns when it would send its token over plain HTTP to a host that is not loopback. |
TestServerTLSFromSecret TestServerTLSFromCertManager TestServerTLSSecretIsNotTheIngressSecret TestServerTLSBehindIngressAndPublicCA TestServerTLSMountFollowsSecretUpdates TestAgentServerCA TestIngressTerminatesTLS TestSharedIngestTokenOptional TestCleartextPushWarning TestRunAgentWarnsOnCleartextPush (on a cluster: SE-09b, under repair) |
| DB-09 | serve creates the SQLite database and its -wal and -shm files 0600, a directory it creates 0700, and tightens an existing database on open; the fleet quickstart says so. |
TestOpenStoreFileMode TestOpenUsesExactPath TestOpenStoreCreatesPrivateDirectory |
| SE-10 | The collectors are read-only: scan makes no write request, measured from the API server's audit log. |
e2e:audit_scan_writes_nothing |
| SE-29 | scan and the agent name the release in the User-Agent of every request to the API server, upgradescope/<version> (<os>/<arch>) (the version upgradescope version reports; the agent's is its image's build arg), not client-go's unstamped upgradescope/v0.0.0 ... kubernetes/$Format, so the audit log tells releases apart (#379). |
TestKubernetesUserAgentNamesTheRelease TestScanRESTConfigUserAgent TestBuildAgentRESTConfigUserAgent TestKubernetesRequestsNameTheRelease e2e:audit_user_agent_names_release |
| SE-12 | Reachable vulnerabilities in the shipped binary fail CI, weekly too, with an expiring per-advisory allowlist. | ci:vuln hack/vulncheck_test.sh |
| SE-14 | Integration tests refuse any context that is not a kind cluster on loopback, unless UPGRADESCOPE_IT_CONTEXT names one, and the e2e fails when an integration test skips. |
TestITKubeContext e2e:integration_tests hack/e2e_test.sh |
Dashboard¶
| ID | Claim | Proven by |
|---|---|---|
| DB-01 | serve embeds the dashboard at /: the built binary serves index.html and every asset it references, with the right content types. |
make dashboard-smoke hack/dashboard-smoke_test.sh TestSPAHandlerServesIndexAndAssets ci:build |
| DB-02 | The committed bundle is the byte-for-byte Vite build of web/; CI fails on a stale one. |
ci:web hack/web-test.sh |
| DB-04 | The dashboard keeps the read token in sessionStorage (gone when the tab closes), or in localStorage only when the user ticks "Remember on this device", never in both; it is sent as a bearer header on every API call. |
web/src/api.test.ts (its sessionStorage-by-default and remember-opt-in cases) |
| DB-03 | No runtime JavaScript dependency beyond react and react-dom; the charts are hand-rolled SVG. | not automated: #99 (verified by hand in #133: npm ls lists react, react-dom and scheduler; every request is same-origin) |
| DB-05 | Without a built bundle (or with -tags nodashboard) the API still serves. The dashboard routes by URL hash (#/cluster/3), so it is served as the page at /, /index.html and one extensionless segment (/teams); at any other extensionless path (/cluster/3, /cluster/3/, /foo/, /index.html/) it answers 302 with a hand-written relative Location (../#/cluster/3, ../../#/cluster/3) that lands on the root with the route in the hash, so the page loads there, also behind a proxy that strips a path prefix (an absolute redirect would drop the prefix), and a path with a file extension that is not a file, or under /assets/, is a JSON 404, and /assets/ lists nothing. After the redirect every asset the page references answers 200 and its relative API base is on the same prefix, for the embedded bundle too. |
TestSPAHandlerWithoutBuiltDashboard TestSPAHandlerFallbackForClientRoutes TestSPAHandlerLoadsTheDashboardAtEveryRoute TestSPAHandlerRedirectsNestedRoutesRelatively TestSPAHandlerDoesNotListDirectories TestEmbeddedDashboardWorksUnderPathPrefix |
| DB-06 | The dashboard dev loop: cd web && npm run dev proxies /api to :8080. |
not automated: #99 (verified by hand in #133, macOS included) |
| DB-07 | Before a tag, the release binary is proven to serve the dashboard and every asset it references. | ci:release-check hack/dashboard-smoke_test.sh |
| DB-11 | Each point of a cluster's score history (GET /api/v1/clusters/{id}/history) carries the evaluation's verdict, and the dashboard's trend colours each dot by it, not by the score: an evaluation whose required check did not run can score 100 and is drawn unknown (hollow, titled "unknown"), never ready. A not-ready point from a server that sends no verdict is drawn neutral (a ready one stays green). |
TestScoreHistoryOrderingAndLimit TestHistory make web-test |
| DB-12 | The fleet team rollup says why each cluster was left out: excluded gives the reason (no-snapshot, too-large for a report over --max-snapshot-bytes, unreadable for a stored inventory that does not decode), missing keeps every name, and the Teams page labels each group by its reason, never "No snapshot yet" for a cluster that has snapshots. |
TestFleetTeamsSaysWhyAClusterWasLeftOut TestEvaluatingReadsRefuseAReportOverTheLimit make web-test |
| DB-10 | With a team-scoped read token the dashboard shows the server's filtered data and says which teams it shows; a 404 under a scoped token says the cluster may be another team's. | web/src/App.test.tsx TestScopedTokenReadsOnlyItsTeams |
| DB-13 | The dashboard's registry page shows an entry's split support as the scan report does: a product whose support ends for everyone on one date and continues to a later one only under a condition (RKE2 Ingress NGINX) reads, while both dates are ahead, "Supported until 2027-11-30 only if the cluster has a SUSE Rancher Prime LTS subscription; for everyone else support ends 2026-03-31", and in the past tense once a date has passed (see DB-18), never the first date alone, and an entry's component images (Flux's controllers) are listed beside its images (#265). | web/src/views/Registry.test.tsx |
| DB-18 | The dashboard's registry page derives an entry's status from its dates and today's UTC date, not from the stored status alone, which stays as data: a date on or before today has passed (the engine's rule, so the eol_date day itself counts). A passed eol_date with an extended_eol_date still ahead reads "extended support only · until eol_date with no extended date ahead (none, or also passed) reads "eol · |
web/src/views/Registry.test.tsx |
| DB-14 | The fleet page's search, quick filter, sort and summary target are kept in the hash query (#/?q=&filter=&sort=&target=), so a reload or a pasted link restores them. A name search matches a cluster name as a case-insensitive substring; blocked, unknown, stale and has-blockers look only at the targets the matrix shows (or at the one pinned target), never a target the server left out; the default sort is worst score first; the matrix draws at most 100 rows with a "show more" control. |
web/src/views/FleetTriage.test.tsx |
| DB-15 | The fleet summary counts every cluster once, in one of ready, blocked, unknown, stale, n/a or no stored evaluation for the selected target, so the counts sum to the number of clusters returned (for a team-scoped token, the ones in its scope). A cluster is ready only when its stored verdict is ready: an unknown verdict is never counted as ready (#243). Clicking a bucket applies the matching filter. |
web/src/views/FleetTriage.test.tsx |
| DB-16 | Every dashboard view keeps the data it shows while it refetches (with an inline spinner, never the blocking loading state), has a Refresh button and an "updated HH:MM:SS" stamp, and refetches when the tab becomes visible only if the data is older than 60 seconds. There is no interval polling. | web/src/hooks.test.tsx web/src/views/FleetTriage.test.tsx web/src/views/Cluster.test.tsx |
| DB-17 | The cluster page keeps the findings category, severity, team and text search in the hash query next to the target, and finding=<key> scrolls to and highlights one finding, so a reload or a pasted link restores the same view. Picking another target drops the filters. |
web/src/views/Cluster.test.tsx |
Knowledge base and add-on registry¶
| ID | Claim | Proven by |
|---|---|---|
| KB-01 | The API-lifecycle dataset is generated from k8s.io/api, and CI fails when the committed copy is stale. |
ci:kb-freshness TestDatasetSanity TestDeprecationGuideCoverage |
| KB-01b | Built-in APIs deleted from k8s.io/api (DRA v1alpha1 to v1alpha3, ClusterCIDR, ServiceCIDR and IPAddress v1alpha1, LeaseCandidate v1alpha1 and others since v0.17) are tombstones in the knowledge base, removed at the release that stopped serving them, so --files blocks them. |
TestKBCoversDeletedAlphaGVKs TestScanFilesDeletedAlphaAPI TestDeletedTypes |
| KB-01c | An object at a version or kind the knowledge base cannot place, in any group the generator's scheme registers (k8s.io/api plus the apiextensions and apiregistration schemes; core included, with or without lifecycle entries: internal.apiserver.k8s.io StorageVersion, imagepolicy.k8s.io ImageReview), is an unknown-api info finding; CRD and aggregated API groups stay silent. |
TestEvalAPIUsageUnknownGVK TestEvalAPIUsageUnknownGVKInGroupWithoutEntries TestEvalAPIUsageUnknownGVKEmbeddedKB TestBuiltinGroupsDataset |
| KB-01d | Kinds kube-apiserver never stored, and that no manifest can create as a resource (PodStatusResult, EphemeralContainers, ReplicationControllerDummy, batch JobTemplate, the removed Scale and DeploymentRollback bodies, the v1beta1 AdmissionReview, ConversionReview and policy Eviction, and the apidiscovery.k8s.io/v2beta1 APIGroupDiscovery) are not in the knowledge base, so they never block as removed; rbac.authorization.k8s.io/v1alpha1 is removed in 1.23 and node.k8s.io/v1alpha1 RuntimeClass in 1.24, each cited to the Kubernetes release notes. Registered types with no lifecycle markers and no such source (scheduling.k8s.io/v1alpha3) stay unknown-api infos. |
TestNonResourceKindsAreNotInTheKB TestKBCoversUntaggedRemovals TestScanFilesNonPersistedAndUntaggedAPIs TestNonPersistedKindsAreNotRecorded TestUntaggedLifecyclesAreCitedAndUntagged TestInferredRemovalsAreAudited |
| KB-02 | The API-lifecycle dataset is the generated file alone, with no hand-written overlay; the hand-written facts the generator adds for types k8s.io/api does not tag each carry a citation. |
TestLifecycleDatasetIsGeneratedOnly TestUntaggedLifecyclesAreCitedAndUntagged TestDeprecationGuideDeletedTypes |
| KB-04 | The README hard-codes no knowledge-base horizon (upgradescope version prints it), and every example target in the README and on the CI gate page is one the embedded knowledge base can judge, so a copied example gives a verdict, not unknown. |
TestDocsKBHorizon |
| KB-05 | Every add-on EOL claim carries an upstream citation, enforced by registry.Validate: a citation on a placeholder or local host (example.com, localhost, an IP address) is rejected, a date with status: unknown is rejected, and a file in registry/data that is not *.yaml (so would never be embedded) fails the tests. That a citation resolves and supports its fact is checked by hand: CI does not fetch. |
TestValidate TestEmbeddedEntriesProperties TestDataDirHoldsOnlyYAMLEntries |
| KB-06 | tools/eol-sync reconciles registry entries with endoflife.date, and a PR touching registry/ fails when they drift. |
TestRun TestComputeCycles ci:registry |
| KB-19 | Kubernetes ranges of an endoflife.date-synced entry live in compat rows, which tools/eol-sync does not rewrite: a write run replaces the cycles and leaves the compat block and the comment above it byte for byte, on a sample entry and on every embedded synced entry that carries one, so the weekly refresh cannot drop them (#341). |
TestRunKeepsCompatRows TestRunKeepsEmbeddedCompatRows |
| KB-09 | The knowledge base and the add-on registry are compiled into the binary; nothing is fetched at runtime, so a KB update reaches users only through a release. | TestLoad TestLoadFS; no outbound connection: not automated, #99 (strace in #133 saw none for scan --files and serve) |
| KB-10 | The knowledge-base version names the k8s.io/api release and digests of both datasets. |
TestDatasetVersion |
| KB-13 | registry is importable on its own. |
TestLoadFS |
| KB-15 | --registry-dir (scan, agent, serve; agent.extraRegistry in the chart) adds operator-supplied registry entries, a file or a directory of <id>.yaml, validated like the embedded ones: an invalid entry, or a path with no entry, stops the command at start naming the file. An entry with an embedded entry's id replaces it outright. The entries change the knowledge-base version. serve judges stored inventories with its own registry, so it needs the same entries as its agents; GET /api/v1/registry lists the embedded registry only. |
TestLoadExtra TestMerge TestLoadWithRegistry TestScanRegistryDirJudgesOperatorEntries TestScanRegistryDirRejectsInvalidEntries TestAgentAndServeRegistryDirFailTheStart TestRegistryDirFlagIsOnScanAgentAndServe TestAgentExtraRegistry TestAgentExtraRegistryKeysMustBeYAMLFiles |
| KB-16 | A removed or deprecated alpha or beta entry whose kind went GA carries a replacement: where k8s.io/api tags none, tools/gen-kb defaults it to the newest GA version of the kind, flagging each replacementDefaulted in the dataset and logging it with the migration guide and the successor's release changelog, and a test lists all 33. A finding that advises a defaulted replacement cites the successor's changelog beside the migration guide, which ends at v1.32. The remediation of a removed-API blocker then reads "migrate to admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy" at 1.34 or "networking.k8s.io/v1 ServiceCIDR" at 1.37, and where the GA version is not served yet at the removal, names the newest version served or the release it is served from (#266). |
TestEveryRemovedEntryWithAGASuccessorHasARemedy TestRemovedBetaBlockersCarryARemediation TestDefaultGAReplacements TestDefaultedReplacementsAreListed TestDefaultedReplacementCitesTheSuccessorsChangelog TestScanFilesRemovedBetaBlockersNameTheirReplacement |
| KB-17 | storage.k8s.io/v1alpha1 VolumeAttachment is dated by when kube-apiserver stopped serving it, 1.23 (its storage was dropped with the beta APIs removed in 1.22), not by the upstream tag (1.24): a cited taggedRemovalFixes override, applied only while earlier than the tag, marks the removal inferred and audited. scan --files at --target 1.23 is a blocker, at 1.22 the warning (#266). |
TestTaggedRemovalFixesAreEarlierThanTheTag TestExtractDatesVolumeAttachmentByTheServedRelease TestFixRemovalOfARegisteredTaggedType TestVolumeAttachmentV1alpha1RemovedInTheReleaseItStoppedBeingServed TestInferredRemovalsAreAudited TestScanFilesVolumeAttachmentV1alpha1 |
| KB-18 | A removed or deprecated entry with no replacement chain the target serves, whose kind has a later version that the target does serve (introduced after the entry, not removed by the target), remediates to the newest such version with its stability: "migrate to scheduling.k8s.io/v1beta1 Workload (beta; may need enabling)", or "(alpha; must be enabled), itself removed in 1.38 (projected)" for an alpha that goes later ("projected" when that removal is past the knowledge base's horizon, as the finding for it is titled; a removal at or inside the horizon is not marked). A Helm release's remediation words the same without nesting parentheses: "(scheduling.k8s.io/v1beta1 Workload - beta; may need enabling)". It is the version the dataset says the target serves, not always the beta (coordination.k8s.io/v1alpha1 LeaseCandidate removed in 1.32 names v1alpha2, since v1beta1 is served from 1.33). A replacement chain still wins. The live finding, the manifest finding and the Helm finding for a release's stored manifest say it alike (a test runs both a live and a manifest inventory) and none nests parentheses; a test lists the seven entries this adds a remedy to and derives the rest from the dataset (#302). |
TestRemovedAlphasWithAPreGASuccessorHaveARemedy TestServedSuccessor TestRemovedAlphaRemediatesToTheServedSuccessor TestAlphaSuccessorSaysWhenItIsRemovedLater TestSuccessorFallbackDoesNotReplaceTheChain TestHelmManifestRemediatesToTheServedSuccessor TestSuccessorRemedyWording TestSuccessorRemediationsDoNotNestParentheses |
| KB-20 | Every removed kind with no replacement and no served successor at the knowledge base's horizon carries a hand-written, cited migration note in internal/kb/data/migrations.json, which kb.Load merges and the generator and the weekly refresh never write; a note whose kind is not in the data, a duplicate, an empty note, or one without an https citation fails the load. A finding's remediation is its generated hint, then the note (the note alone when there is no hint), and the note's citations follow the deprecation guide's: PodSecurityPolicy at 1.25 names Pod Security Admission and cites the migration page, and Ingress, CustomResourceDefinition and both admission webhook configurations at 1.22 name the schema changes (#330). |
TestRemovedKindWithoutSuccessorHasAMigrationNote TestApplyMigrationsRefusesAnUntrustworthyFile TestShippedMigrationNotesAreCited TestSchemaChangingMigrationsAreNoted TestMigrationNotesInFindings TestOutputsCarryTheMigrationNote |
| KB-21 | A remediation never names a version less mature than the one in use (alpha, then beta, then GA, from the version name), for every entry at every target from 1.16 to the horizon; when the GA replacement is not served yet, the finding says the version in use is the right one for the target and names the GA version with the minor that first serves it. The introduced minors gen-kb clamped are corrected by fixups with changelog citations (EndpointSlice v1alpha1 1.16 and v1beta1 1.17, and batch/v2alpha1 CronJob, scheduling.k8s.io/v1alpha1 PriorityClass and settings.k8s.io/v1alpha1 PodPreset), so a 1.16 target does not report them "not served until 1.17". The same holds for a manifest at a version the target does not serve yet: its "write it as" alternative is at least as mature as the manifest, and with none the finding names the minor that first serves it ("upgrade the cluster to Kubernetes X") (#332). |
TestRemediationNeverNamesALessMatureVersion TestServedAlternative TestFilesModeUnservedAPIIsABlocker TestBetaSourceIsNotSentBackToAnAlpha TestClampedIntroducedMinorsAreCorrected TestRemediationNeverMovesBackToLessMatureAPI TestFixIntroduced TestIntroducedFixesAreCitedAndInTheDataset |
| RM-01 | A remediation never points at an API the target does not serve: not one removed at or before the target, nor one introduced after it (#237). When the replacement the knowledge base names is not served yet, the newest version of its kind that the target serves and that is newer than the API being replaced is named instead (storage.k8s.io/v1alpha1 VolumeAttributesClass at 1.33: v1beta1); with none, the finding says no replacement the target serves is known and when the next one is served from (certificates.k8s.io/v1beta1 ClusterTrustBundle at 1.36: v1 from 1.37). The served successor of a kind (KB-18) is held to the same rule at every target: of the entry's own kind, introduced after it, served and not removed at the target. A Helm release's stored-manifest finding names replacements the same way, and lists the APIs it knows no served replacement for, with the release that serves the next one when known. |
TestRemediationNeverPointsAtRemovedAPI TestResolveReplacement TestLaterReplacement TestResolveReplacementShippedServedOnly TestRemediationNamesOnlyServedReplacement TestHelmManifestRemediationNamesOnlyServedReplacement |
Managed-provider support and cost¶
| ID | Claim | Proven by |
|---|---|---|
| SUP-01 | The collector sets Inventory.Provider (eks, gke, aks, other) only from signals the provider produces: a server-version suffix or a managed node pool's label. A providerID scheme alone (kubeadm on EC2, Azure VMs or GCE) claims nothing and can only contradict a claim (a node with none contradicts nothing by scheme, so a -gke. version on providerID-less nodes is GKE, unless a node carries a Google Distributed Cloud baremetal.cluster.gke.io/ label, which contradicts it); kind and vanilla clusters are other; two claims are other; a cluster whose nodes could not be listed (or whose list failed partway) and whose version names no provider is left undetermined, not other. |
TestCollectVersionsProvider TestCollectVersionsProviderWithoutNodes TestCollectVersionsProviderNodeListFailsMidway |
| SUP-02 | The server accepts an inventory whose provider is a name it does not know (a newer agent's; the evaluation has no calendar for it) and refuses (422) one that is not a bounded name of at most 63 bytes. A pushed provider reaches the stored evaluation, so the server's report endpoint carries support and the finding. |
TestValidateIdentifiersRefusesInvalid TestSupportLifecycleThroughTheServer |
| SUP-03 | The provider calendars (registry/data/providers) carry both support dates per minor and citations; a price needs a currency, an as-of date and a cited page, and the extended price must exceed the standard one. EKS and AKS versions are generated from endoflife.date and CI fails on drift; GKE's are hand-curated from Google's schedule. |
TestValidateProvider TestLoadProvidersFS TestEmbeddedProviders TestProvidersDirHoldsOnlyYAMLEntries TestComputeWindows TestRunSyncsProviders ci:registry |
| SUP-04 | support-lifecycle is keyed support-lifecycle/<provider>/<minor>/<phase> (ending, extended, ended), so an ignore rule, a baseline or a notification for one phase never covers a later one (a rule with the old three-segment key matches nothing and scan warns); it is a warning from 90 days before the day a managed cluster's minor enters extended support, and a blocker from that day and once support has ended. It states the date, never a day count, and is judged with the scan's clock. It is the cluster's, not a PR's: /gate?cluster= tags it source: cluster and leaves the verdict ready for a manifest that adds nothing at fault. |
TestEvalSupportPhases TestEvaluateGolden TestGateTagsSupportLifecycleAsCluster TestSupportKeyNamesThePhase TestSupportRulesAcceptOnePhase TestSupportBaselineDoesNotHoldALaterPhase TestLegacySupportKeyRuleMatchesNothingAndSaysSo TestComputeDeltaAtTheEndOfExtendedSupport |
| SUP-05 | The cost line is the annual list-price delta, (extended − standard) × 8760, labelled with the day the price was read. It is absent when the provider's price is not cited (AKS), the provider is absent (an older agent's inventory, whatever its version string looks like), unknown or other, the minor has no dates, or extended support has ended. |
TestEvalSupportCost TestEvalSupportWithoutPrice TestEvalSupportUnknown TestEvalSupportPriceNote TestEvaluateGolden |
| SUP-06 | scan prints the date and the cost on a Support: line, --output json carries support, and the ClusterReadiness status carries supportPhase, extendedSupportFrom, extendedSupportEnds (where the provider offers extended support), and with a price annualCostDelta, currency, priceAsOf and annualCostNote, plus extendedSupportCondition where extended support is opt-in; the CRD schema lists every status field, so the apiserver does not prune them. |
TestWriteTableSupportLine TestSupportSummary TestJSONReportMatchesSchema TestStatusFromReportsSupport TestManifestStatusListsEveryStatusField |
| SUP-07 | Where a provider's extended support is opt-in (GKE: Extended release channel; AKS: Long Term Support), the finding, the scan line, the report and the ClusterReadiness status state the provider's window conditionally (extendedSupportCondition) and never that the cluster is enrolled; GKE's price carries its "charged only for clusters on the Extended release channel" caveat (annualCostNote) wherever the figure is shown. EKS, where extended support is the default, is stated unconditionally. AKS's note says platform support applies only while the minor is N-3 (one behind the oldest minor in community support) and that older minors are out of support; it never says a cluster past community support is in platform support unconditionally. |
TestEvalSupportOptInProvidersRealDataset TestSupportSummary TestEvaluateGolden TestStatusFromReportsSupport TestEmbeddedProviders TestValidateProvider |
| SUP-08 | The examples on the support-lifecycle concepts page (EKS 1.34 on 2026-10-15, AKS 1.33 on 2026-10-04) are the finding's title, detail and fix line exactly as scan prints them from the embedded knowledge base, apart from line wrapping. |
TestSupportLifecycleDocExamplesMatchScanOutput |
Build, CI and release¶
| ID | Claim | Proven by |
|---|---|---|
| IR-03 | Unit tests, golden tests, lint and the build need only Go: the Go tests that render the chart (deploy/chart, tools/promrule-test) skip when helm is absent, and the package passes. |
ci:test ci:lint ci:build hack/test-nohelm_test.sh |
| IR-04 | Integration tests are env-gated; the kind e2e covers scan, agent, chart, CRD, server and the audit log, on 1.31 and 1.37 for every PR that changes code (IR-24) and 1.29 to 1.37 weekly. | ci:kube hack/e2e_test.sh hack/kind-images_test.sh (the weekly matrix ran green by workflow_dispatch, run 36965517376: #128) |
| IR-04b | Kubernetes 1.24 to 1.28, which kind has no node images for, are scanned against a real kube-apiserver and etcd (envtest): 1.24 and 1.28 on every PR that changes code, every minor weekly (the rows of hack/envtest-versions.txt marked pr and weekly): a cluster holding only GA objects has no removed-API finding and no blocker, a second scan of it is identical, an object written through a beta API that minor still serves is a blocker at its removal minor (and, on 1.27 only, a warning one minor before, so only in the weekly set: on the other minors the removal is the next minor), and a capability that comes back unavailable or partial is listed as not assessed. envtest has no nodes, pods, agent or chart, so only the live collector and engine are tested on those minors (#135, #69; tested range). |
TestEnvtestMatrix ci:envtest hack/envtest_test.sh |
| IR-05 | A new push supersedes a PR's run; main, schedule, dispatch and tag runs of different commits, refs or kinds never cancel each other. Repeat runs of one kind, ref and commit share a group, where GitHub keeps one pending run: a third dispatch of the same commit cancels the pending second, as CONTRIBUTING.md says. | hack/ci-concurrency_test.sh TestDocsRound2Gaps |
| IR-06 | Release notes are generated from conventional commits, grouped by prefix with a Breaking changes group (type!:); docs, chore, test, ci, build and style commits are left out unless marked breaking. They are built from the subject alone, so a BREAKING CHANGE: footer without ! would land under Features or Bug fixes; the pr-lint / breaking-change check (hack/check-breaking.sh, #198) fails a PR whose title, description or commit has the footer without a lowercase type!: subject, and (#390) a PR title or commit whose subject has an uppercase or mixed-case type (Feat:, Chore!:, FIX(scope):), footer or not, because that would land under Other changes; a subject with no type (Update the docs) and every lowercase type behave as before. The release notes open with a header (release.header in .goreleaser.yml, #390) that links CHANGELOG.md at the release tag (the file, not a heading anchor, which would need the release date) and says it is the reviewed record and lists every breaking change, because the notes list only the type!: commits under Breaking changes (rc.3: 2) while CHANGELOG.md's **Breaking entries since v0.1.1 number 6. The check is a required status check on main: ruleset 24343445 (applied 2026-10-11) requires ci-ok and breaking-change, the job's check-run name from the GitHub Actions app (the pull request page shows it as pr-lint / breaking-change), so a failing run blocks a merge (.github/rulesets/main.json; the repository's admin role can bypass the ruleset), and hack/rulesets-live.sh fails CI on every PR and push when the applied ruleset differs from that file. hack/release-footer_test.sh renders the header offline, and v0.2.2 (published 2026-10-11, release run 38164870338) is the first published release whose notes open with it, linking CHANGELOG.md at v0.2.2. v0.2.0-rc.3 (published 2026-10-11) is the first release with the grouping, its Breaking changes group (fix(server)!:, refactor!:) and the check. |
hack/check-breaking_test.sh hack/release-footer_test.sh .github/workflows/pr-lint.yml .github/rulesets/main.json hack/rulesets-live.sh hack/rulesets-live_test.sh |
| IR-10 | Every package compiles for linux, darwin and windows (amd64 and arm64) on every PR that can break a compile (a Go file, a module file, GoReleaser's config, the Makefile, hack/cross-build.sh or ci.yml changes; IR-24), and on every push to main, dispatch and release, so a platform-only break fails its PR, not a tag. The three OS legs together are exactly the platforms hack/cross-build.sh and .goreleaser.yml build. |
make cross-build hack/cross-build_test.sh hack/ci-gates_test.sh ci:cross-build |
| IR-11 | Releases are cut from CI, never a laptop; a re-run overwrites assets; they are reproducible (file times come from the commit); release tags are immutable. The preflight refuses a tag whose commit is not an ancestor of main (hack/check-release-ancestry.sh, #198), the tag ruleset (.github/rulesets/release-tags.json) forbids deleting or moving a release tag (every vX.Y.Z[-pre], refs/tags/v*.*.*, and never a floating major tag such as v0 or v1, which major-tag must move: #244) once it exists, and the rpm packages no longer carry the build host (#206). v0.2.0-rc.3's release run (38108851717) is the first to run the ancestor check (the preflight step "the tagged commit is on main" succeeded); whether its rpm packages carry the build host was not checked in round 3. The applied tag ruleset (24349764) equals release-tags.json: it was re-applied on 2026-10-11 (refs/tags/v*.*.*, no exclude, no bypass actors), and hack/rulesets-live.sh fails CI on every PR and push when it stops equalling the file. CI's token cannot see bypass actors, so the empty list was checked with the owner's token on 2026-10-11, and a later change to it is seen only by running make rulesets-check as the owner. GitHub's immutable releases setting stays off (a documented trade-off), and a build tree whose file times are older than the commit's (a tarball export, not a CI checkout) does not give identical image layers. |
hack/check-release-ancestry_test.sh hack/release-preflight_test.sh hack/ruleset-tags_test.sh .github/rulesets/release-tags.json hack/rulesets-live.sh hack/rulesets-live_test.sh |
| IR-15 | Every behaviour change since v0.1.1 is listed in the changelog, and 0.1.1 changed only packaging. Round 3 found 0.1.1 changed only packaging and the JSON contract changes listed (VS-15), and one gap, since closed: the Ready printer column of kubectl get ucr changed from the boolean .status.targets[0].ready (v0.1.1) to the string .status.targets[0].verdict, which CHANGELOG.md mentioned only as an Added entry and now lists under Changed (#366). The check was by hand; the release preflight only refuses an empty section. |
hack/check-changelog.sh; the printer-column diff: not automated, #99 |
| IR-16 | One required check, ci-ok, fails if any job failed or was cancelled, or was skipped for a reason that does not hold. A job a pull request gates off (IR-24) passes only when the changes job ran, succeeded and said it is not needed: a changes job that failed, was cancelled or was skipped on a pull request fails ci-ok, and so does a gated job skipped while changes said it is needed. Off a pull request ci-ok does not lean on the changes outputs: on a push to main it accepts only the docs-only skips that push always had (kube and envtest on e2e=false, release-check on release=false) and fails any other skipped gated job whatever changes said, and on a dispatch or release, where changes does not run, every gated job must run (the weekly schedule skips the jobs that run on pull requests and pushes by design). A changes job that did not succeed makes every gated job run, so a failed one on main costs no test signal. |
ci:ci-ok hack/ci-ok_test.sh hack/ci-gates_test.sh |
| IR-17 | Only the highest stable release moves what users follow without pinning: GitHub's Latest release (version: latest and @v0 install through it), the image's :latest and the floating major tag. A release run on any other tag (a re-run or re-dispatch of an older release, or the slower of two racing) publishes its own assets and leaves them; GoReleaser never sets GitHub's Latest (make_latest is the literal "false", so neither a release's publication nor a re-run of the job can make a release without an attestation Latest), the goreleaser job's last step does, and a run moves it only to its own release, only when that is the highest published stable release (decided by hack/release-newest.sh's version comparison, as in every other step, not by comparing tag strings), and only after its own actions/attest-build-provenance succeeded (a failed attestation leaves Latest on the previous release, #304; a run on a lower release never moves Latest to a higher one that is published but whose own run has not attested it yet, so the race cannot name an unattested release Latest; a Latest moved by hand is put back only by a re-run of the highest release's own run, not by a run on a lower tag; a re-run of the goreleaser job for the release that is already Latest sends make_latest=false for it again, which GitHub's REST reference for updating a release does not define for a release that is already Latest, so the closing step, reached only after that re-run's attestation succeeded, sets Latest back to the release if GitHub demoted it), while the image's :latest is still set by GoReleaser from the same highest-release check; the major tag (v0) follows the newest release of its own major, checked again inside the major-tag job under one repository-wide concurrency group; and after publishing, a stable run points :latest at the highest published stable release (and Latest at itself when it is that release), so two racing runs end on the higher tag (#244). A failed release listing (an API error, a rate limit) fails the step that asked, never reads as "no higher release": each step writes the list to a file before deciding, since a run: step runs under bash -e without pipefail. Not yet run by a real release (#127). |
hack/release-newest_test.sh |
| IR-18 | The weekly kb-refresh runs the freshly bumped third-party Go modules (go get of the newest k8s.io modules, make gen-kb, go test) and make eol-sync in jobs with contents: read only, no token kept in the checkout and no Go cache; the jobs that push the bot branch, open the PR and approve its runs run no repository or dependency code, and apply only a patch of their PR's paths (no other path, rename, copy or symlink) (#244). |
hack/kb-refresh-ci_test.sh |
| IR-19 | Every service and container image CI pulls is pinned by digest: pg-conformance's Postgres majors, in the release-gating run too, come from hack/pg-images.txt (#244). zizmor cannot see a digest through that job-output matrix, so its service image line carries the workflows' only zizmor ignore (unpinned-images, with the reason); zizmor 1.14.2 --offline reports no findings on .github/workflows (run 2026-10-10 at commit 7c009545). |
hack/pg-images_test.sh ci:pg-matrix ci:pg-conformance |
| IR-20 | The CI-only Renovate validator's lockfile (hack/renovate/package-lock.json) is under a fail-closed advisory gate: any advisory npm reports there that hack/renovate/osv-scanner.toml does not accept with a reason and an unexpired date, or one that gained a fix the lockfile can take, fails CI's vuln job, weekly too (#244). |
hack/renovate-audit_test.sh make renovate-audit ci:vuln |
| IR-21 | CI uploads the Action's fixture SARIF (two blockers on purpose) to code scanning only from a dispatched run on a branch other than the default, never from main, a pull request or a release (#244). | hack/ci-sarif_test.sh |
| IR-22 | No job on the release path restores a GitHub Actions cache, which any job can write with the runner's token whatever its permissions: (a run on a tag restores what main saved; kb-refresh's build of unreviewed modules used to run on main, see IR-23): release.yml's jobs use none (setup-go cache: false, no setup-node cache, setup-buildx cache-binary: false, no actions/cache, no type=gha), and every ci.yml job, when release.yml calls it (inputs.release) or on any tag ref, restores none; every action either workflow uses is on a list checked to keep no Actions cache of its own. Pull requests, main and the schedule keep their caches; no job that runs unreviewed code writes one in main's scope (IR-23). |
hack/release-caches_test.sh |
| IR-23 | kb-refresh's freshly bumped, unreviewed modules (go get of the newest k8s.io ones, make gen-kb, go test, make eol-sync) are built and run only by kb-refresh-build.yml, in workflow_dispatch runs on the branch bot/kb-refresh-build that kb-refresh.yml (on main) creates at main's head, and nothing in kb-refresh.yml builds. Every build job there is guarded to that ref and to its own pipeline input, holds contents: read, and configures no cache (any entry its code writes through the runner token lands in that branch's cache scope, which main's CI and a tag run never restore); the seal job is guarded to that ref and to a build job's success, holds contents: read and runs no repository code. Each pipeline has a run of its own, so a run's artifact store holds that pipeline's one artifact and the other pipeline's unreviewed code cannot upload, replace or delete it. Before it downloads anything, a <pipeline>-verify job (actions: read and contents: read only, no repository code) refuses any run unless, through the API, it is this repository's own (not a fork's) kb-refresh-build.yml run, with the id it dispatched, on bot/kb-refresh-build, at the commit the schedule ran at, by workflow_dispatch, named for this pipeline and the request name its -run job fixed and handed on as an output (so re-running a failed -verify job alone checks the run that was dispatched), completed with conclusion success; unless only this pipeline's job and the seal job ran, both successfully; and unless its artifacts are exactly this pipeline's, not expired. It reads the patch's sha256 from the seal job's log (that job runs after the build job, runs no repository code, and prints the digest where the build job cannot write; the log is fetched with curl, not gh api, because the runner's gh refuses to print a response that holds terminal escape sequences and a job log is full of them (run 38058033971, 2026-10-10); curl follows the redirect to the blob store without --location-trusted, so the token is not forwarded to the second host; the fetch is tried up to three times, and one that keeps failing is refused as a fetch failure), downloads by run id with its read-only token, recomputes the sha256 and refuses a mismatch, and uploads the checked patch again in its own run. The *-pr job, which holds the writes, downloads that copy from its own run, naming no run id and no token, and applies it only after the path check (own paths only, no renames, copies or symlinks). A build that fails, never appears or does not finish fails the refresh and opens the "kb-refresh is failing" issue. Not covered: the unreviewed job decides the patch's content, which only the path check and the reviewed PR confine; the digest binds the download to the artifact as the seal job hashed it, it is not a signature; a maintainer with write access can still push to or dispatch on the branch (a hand dispatch is not adopted by accident: the run is found by its run name and commit; but a maintainer with that same write access who sends the same run name at the same commit would be); the checks are pinned by the stubbed API in hack/, and the stub curl serves the log, so that curl follows the redirect and drops the token across hosts is curl's documented behaviour, not run here, and a dry run on GitHub proves the field formats (path, the log endpoint). |
hack/kb-refresh-scope_test.sh hack/kb-refresh-ci_test.sh |
| IR-24 | A pull request runs a Go-heavy job only for a change it can see, and never passes ci-ok with a needed job skipped. test-heap, build, examples, images, vuln, pg-conformance, the kind e2e and envtest run when a changed file is Go code or one a gated job reads (the embedded data and dashboard, the chart, the examples, the Dockerfiles, the Makefile, the hack/ scripts and data those jobs reach, ci.yml itself); cross-build when a Go file, module file, GoReleaser config or the Makefile changes. The unit tests are never skipped: in full for those changes, and for a change to no Go code the packages whose tests read the repository outside Go code (hack/test-readers.txt: the docs-drift tests, which fail on a stale page, and the two that walk the whole checkout, internal/server decoding every YAML file and internal/crd/apigroup reading every tracked file, which a change to a workflow, an issue template or a script can fail). A pull request of 3000 or more changed files, more than the path filters are shown, runs everything. lint, repo-checks (check-toolchain, every hack/ self-test, the claims ledger), web, helm, notices and kb-freshness run on every pull request. A push to main, the schedule, a dispatch and a release keep what they ran before. Every tracked file is judged: matched by the go filter, or on a short list of files no Go-heavy job reads (not the same as files no Go test reads, which is why the unit tests are not gated; deleting files to see what breaks cannot find a test that skips a missing file, so hack/test_test.sh instead fails when a package that names a docs page or walks the tree from outside its directory is not on hack/test-readers.txt), and every hack/ script a gated job reaches is matched. |
hack/ci-gates_test.sh hack/ci-ok_test.sh hack/test_test.sh ci:changes ci:repo-checks ci:ci-ok |
| IR-25 | The two slowest PR jobs run as shards side by side, and no test is lost or run twice by the split. test-heap is four shards of the heap-bound tests and test two shards of the unit packages (hack/test-heap.sh --shard i/n, hack/test.sh --shard i/n): every discovered test, or package, is in exactly one shard, by a greedy longest-first split of a committed duration table (hack/shard.sh; an entry the table lacks counts 30 s for a heap test and 10 s for a package, so a new one is placed, not dropped), the matrix names every shard of its n, and ci-ok needs the job, which is all of its shards. A shard that runs no test because there are more shards than tests passes; one with a heap test that skipped itself fails. |
hack/shard_test.sh hack/test-heap_test.sh hack/test_test.sh ci:test ci:test-heap |
| IR-26 | The documented cosign verify of the image and of the chart (docs/operations/install.md, the chart README, the release notes' Verify block) states the minimum cosign, 3.x or 2.6.3 or later, because the signatures are cosign 3 bundles and 2.6.2 and older exit 10 with no signatures found (#377). make claims-check fails a section of those two files that shows cosign verify without it; the release workflow's verify job runs the documented commands for the index, each platform digest and the chart with cosign v2.6.3 and, for the chart, with the default cosign, and hack/release-footer_test.sh pins the footer. Those release legs run only on a tag (the next release run is their observation). cosign verify-blob of checksums.txt is not affected and works from cosign 2.4.3 (the oldest the issue tested). |
hack/docs-cosign-minimum.sh hack/docs-cosign-minimum_test.sh hack/release-footer_test.sh make claims-check |
| DO-03 | One binary runs scan, agent and serve. |
TestRootHasAgentSubcommand TestRootRegistersServe e2e:eol_ingress_nginx_blocks |
| DO-03b | The README no longer counts subcommands; the CLI reference has a page for every command the binary has (scan, agent, serve, tokens, clusters, version, completion) and every flag, rendered from its command tree, and CI fails when a page is stale. |
TestReferenceIsFresh TestGenReferenceCLI make docs-check |
| DO-04 | engine, inventory, kb and registry have no client-go dependency; the engine is a pure function with no Kubernetes or network dependency. |
TestEvaluateGolden; the dependency set: not automated, #99 (verified by hand in #131: go list -deps and a wasm build) |
| DO-07 | Golden files cover every finding category and the score formula. | TestEvaluateGolden |
| DO-08 | make lint is CI's lint (pinned staticcheck), and gofmt covers every module. |
ci:lint hack/test.sh |
| CL-01 | Every test this ledger names exists. | make claims-check hack/claims-check_test.sh |
| PF-01, PF-03, PF-04 | Every number in the README's Measured table says what, how, where and at which commit: scan against kind 1.37 from #130's audit at main 8a951dd (median 0.49 s), scan --files at main 9d0b161, the steady agent tick at 0b6fec28 (PF-18, with the two volume lists #362 added since given as computed), and the binary and archive sizes of the published v0.2.0 assets (commit 62603159), measured on 2026-10-11. The sizes in the README and on the Install page (every platform) stay within 2% of the release build: the release check compares each with dist/ and fails beyond that. The README gives no image size; the Install page gives the published v0.2.0 image's. |
hack/check-doc-sizes.sh hack/check-doc-sizes_test.sh ci:release-check (the timings are not automated: #99) |
| RB-08, IR-01, IR-02, IR-07 to IR-09, IR-12 to IR-14, DB-08, SE-15 | The published stable release is current: GitHub's Latest is the newest stable release, and go install …@latest resolves to it (v0.2.0, commit 62603159, when this was re-checked). Every archive matches the keyless-signed checksums.txt, every archive and package carries build provenance, and the image (ghcr.io/abd-ulbasit/upgradescope:v0.2.0, amd64 and arm64) and the OCI chart pull anonymously and verify with the documented cosign verify, on cosign 3.x and 2.6.3. The chart pins the image by its index digest, and govulncheck finds no reachable vulnerability in the release binaries. Release run 38136096281's verify job checks this on every release, and it was re-checked from outside on 2026-10-11. |
hack/release-footer_test.sh hack/docs-cosign-minimum_test.sh hack/release-ci-permissions_test.sh (not automated outside a release run: #99) |
Docs, references and published schemas¶
The documentation site (docs/, built by mkdocs build --strict in
.github/workflows/docs.yml) and the machine-readable contracts in api/.
These are claims the docs make about themselves, kept true by tests that
compare them with the code; the compatibility policy
(docs/compatibility-policy.md) says what may change within a version.
| ID | Claim | Proven by |
|---|---|---|
| DS-01 | api/openapi.yaml documents every route the server registers, and the responses real handlers give for every documented operation validate against it, carrying no field it does not list. Unknown paths answer with its JSON error. |
TestOpenAPICoversEveryRoute TestOpenAPIResponsesMatchSpec TestOpenAPIUnknownPathIsJSONError TestUnlistedFields |
| DS-02 | The published response schemas stay open (no additionalProperties: false), so a field added within a version, as the compatibility policy allows, does not fail a client that validates. |
TestOpenAPIResponseSchemasAreOpen |
| DS-03 | Real scan --output json reports (findings, folded callers, suppressions, a baseline, gaps, unrecognized images, a live cluster's serverVersion, kubeContext and apiServer, a --plan report's hops) validate against api/report.schema.json and carry no field it does not list. |
TestJSONReportMatchesSchema TestScanPlanJSON |
| DS-03b | The report-shaped /api/v1 responses, a cluster's report (stored or what-if) and the gate's JSON answer (with and without ?cluster=), lead with schemaVersion and toolVersion as scan --output json does, from the same envelope, and the OpenAPI document requires both (#60). |
TestReportResponsesLeadWithVersions TestOpenAPIResponsesMatchSpec |
| DS-04 | Real webhook deliveries validate against api/webhook.schema.json, formats included. |
TestWebhookPayloadMatchesSchema |
| DS-05 | Every finding category the engine defines is described in both schemas, which do not close the list (a report with a category added later still validates), has a row on the verdict page and in the architecture, and is listed in, and accepted by, the config reference's ignore rules. | TestDocsListEveryCategory TestDocsConfigReference |
| DS-06 | The generated references are fresh: the CLI, CRD and REST API pages match what tools/gen-docs renders from the code and api/openapi.yaml, and the Helm values tables match helm-docs; CI fails on drift. |
TestReferenceIsFresh TestGenAPIMarkdown TestGenCRDMarkdown make docs-check hack/helm-docs.sh |
| DS-07 | The metrics reference lists every Prometheus metric the agent and the server export. | TestMetricsReferenceListsEveryAgentMetric TestMetricsReferenceListsEveryServerMetric |
| DS-08 | The add-on counts in the README, on the registry page and in ROADMAP.md match registry/data (34 add-ons, 15 synced from endoflife.date, 15 with Kubernetes ranges, 4 with no lifecycle data), and the registry page lists every entry. |
TestDocsRegistryCounts TestRoadmapRegistryCounts |
| DS-09 | The config reference lists every field an .upgradescope.yaml accepts, and its example loads. |
TestDocsConfigReference |
| DS-10 | The GitLab CI template (ci/gitlab/upgradescope.gitlab-ci.yml, included on the other-CI page) avoids each way the job broke before: the image's entrypoint is cleared, no line needs GNU-only flags or a pipeline's exit status, the archive is checked against checksums.txt, and the last line, unpiped, is the gate; its artifacts:reports name the Code Quality and JUnit files the scans write. No GitLab runner executes it. |
TestDocsGitLabJob |
| DS-11 | The site builds with mkdocs build --strict (no broken link, anchor or page missing from the nav) on every pull request, from hash-pinned requirements. |
.github/workflows/docs.yml hack/docs/requirements.txt |
| DS-12 | The Jenkins and Azure Pipelines templates (ci/jenkins/Jenkinsfile, ci/azure/azure-pipelines.yml) pin the release the docs install, check it against checksums.txt, run an unpiped gate that writes JUnit, and publish it in a step that runs when the gate fails. Neither CI system runs them. |
TestCITemplateJenkins TestCITemplateAzure |
| DS-13 | Each release attaches the published contracts, report.schema.json, webhook.schema.json and openapi.yaml (every file in api/), and lists them in its signed checksums.txt; the compatibility policy says where to download them (#60). |
TestDocsContractsAreReleaseAssets hack/release-check.sh ci:release-check |
| DS-14 | No copy-paste expires: example in the docs, README, Action, chart or examples is earlier than the test clock plus a year (they use 2099-12-31 and say to pick a date in your own window), so an example does not expire under the reader; UPGRADESCOPE_TEST_NOW shifts the clock. The agent test that runs the embedded KB against GKE's calendar injects its clock on both sides of the boundary instead of using the wall clock (#269). |
TestDocsExpiresExamplesAreFarInTheFuture TestExpiredExamplesFindsPastDates TestTickVendorServerVersionWritesScoreAndVerdict |
| DS-15 | The add-on changes in the webhook payload example are what the engine prints for the registry's entries at the example's timestamp (key, severity, title, detail), so the page shows no finding the engine cannot produce (#269). | TestWebhookDocExampleAddOnChangesMatchTheEngine |
| DS-16 | Every upgradescope command line in the repository's tracked Markdown (README, docs/, examples/, deploy/, action/, CONTRIBUTING; the changelogs and the maintainer's unpublished design notes excluded; untracked files such as a docs virtualenv do not count; when git lists nothing the tree is walked instead) names only flags its command defines, so a documented scan --format json cannot outlive the flag. The check stops at the first | or || (also one glued to a word, json|jq), &&, &, ;, redirect (>, >>, <, 2>, 1>, &>), -- or # comment, because what follows belongs to another program; a quoted word is a value, not a flag (#307). |
TestDocsFlagsExist TestDocsFlagScannerFindsDrift TestListMarkdownIgnoresUntracked |
| DS-17 | The contributor map and the architecture page match the code: CONTRIBUTING's repository map has a line for every directory under internal/, every top-level directory with Go packages and every command the binary registers, and the component table in docs/architecture.md has a row for every package under internal/ (only testdata and the test-helper packages, <pkg>test inside <pkg> such as junit/junittest, are exempt) whose "Depends on" column is exactly the module packages that package imports (#306). |
TestContributingRepositoryMap TestContributingMapNamesTopLevelPackages TestArchitectureComponentTable TestGoPackageDirsSkipsOnlyTestHelpers |
| DO-01 | Every number in the README is dated and says how it was measured. Round 3 reported a pass, by hand (the live-scan time is still the first audit's, at main 8a951dd). | the Measured table, PF-01 to PF-04, says so since #130; checked by hand: not automated, #99 |
| DO-02 | The comparison with pluto, kubent, kubepug, Nova and the managed platforms' checks (docs/comparison.md), and the evidence in docs/research.md, passed the round-3 audit, which was by hand. |
not automated: #99 |
| DO-05 | The README and the FAQ say the same, checkable thing about who wrote the code: most of it was written by coding agents (the Co-Authored-By trailers in the git history show it: git log -i --grep='Co-Authored-By: Claude' --oneline lists 125 of the 282 commits at f8a6ff9b, 11 October 2026, a lower bound, since a missing trailer does not show a human wrote the commit), working from specs and plans drafted with them and approved by the author; the planning notes are not published. They no longer say the author wrote the specs and plans, which a clone cannot show. |
TestAuthorshipNoteConsistent |
| DO-09 | The README is a summary of under 1,200 words (as wc -w counts them, code and badges included; 1,181 on 2026-10-12, from 1,422 at the v0.3.0 integration and 3,171 before #404): the long material (scale and cost and the Measured rows in full, the agent's RBAC in detail, the verification of every channel, the CI gate's fork and Dependabot cases, kube-upgrade-check) is on the docs pages it links, each of which exists, every figure and commit of a README Measured row is in Scale and cost's copy of the table, and its Project section links CONTRIBUTING, GOVERNANCE, MAINTAINERS, ADOPTERS, ROADMAP, SECURITY, the claims ledger and the Code of Conduct, each of which exists. |
TestReadmeStaysASummary TestReadmeLinksTheMovedDetail TestReadmeMeasuredRowsAreOnTheScalePage TestReadmeLinksTheProjectFiles |
| DO-10 | The project's governance is written down: MAINTAINERS.md has the Name, GitHub, Company and Email table and links GOVERNANCE.md for how maintainers are added; CONTRIBUTING.md links GOVERNANCE.md and MAINTAINERS.md in its opening; ROADMAP.md links the GitHub milestones. |
TestMaintainersTable TestContributingLinksGovernance TestRoadmapLinksMilestones |
| DO-11 | The README's first screen leads, in this order, with add-on end of life and compatibility from the cited registry, custom resources at CRD versions no longer served (status.storedVersions), stored Helm release manifests and chart kubeVersion, and pull-request and CI gating with --files and SARIF, JUnit and GitLab output; managedFields and the deprecated-API metric are features further down, since kube-upgrade-check reads both too (#404). How it compares says kube-upgrade-check covers similar live-cluster ground and adds control-plane and kubelet config, release-note checks and upgrade preflight that upgradescope does not do yet (#397), and a "Use something else when" note points to it. |
TestReadmeLeadsWithTheWedge |
Scale and cost¶
Measured on 4 October 2026 (04:30 to 05:00 in UTC+5, which is 3 October in
UTC), for Scale and cost (#71; the README quotes one steady tick, PF-18). The
numbers come from manual runs of the harness in
hack/bench/: they take minutes and need a lab cluster or a Docker engine, so
CI does not run them, and a number here is not re-proven by a build. What CI
does keep true is the harness's safety rules and report scripts, and the
behaviour the numbers motivated.
| ID | Claim | Proven by |
|---|---|---|
| PF-09 | At 2,001 nodes, about 14,000 pods and 1,000 Helm releases, a steady agent tick made 53 API requests on main 735751d and read 50 MiB (3.4 MiB on the wire) in 4.1 s and 3.0 CPU-seconds (the median of the 4 ticks after the first, the mean of the middle two), with a peak live heap of 28 MiB and a peak RSS of 56 MiB; the first tick after a start made 1,053 requests and took 36.7 s and 23.5 CPU-seconds (main 735751d, 4 October 2026, stamped 03:53 UTC; the first session's run on a3e72ea made 61 requests in 5.6 s and 3.6 CPU-seconds, before #232 read each pod once). At the code of #228 (9810fb6, 4 October 2026, 06:55 to 07:03 UTC; lab B idle, 13 to 15% of a core) it made 31 requests (#228 cut them: pod and node pages sized by their largest object, at most 1,000, API discovery kept between ticks, the agent's object read once; 4 more on a tick that asks discovery again; the drafts' 24, with pages of up to 2,000, had a worst case past the memory limit, PF-02; the target of under 25 is not met by the tick that lists every pod (31); the ticks that reuse the pod pass make 20 (measured) and a cycle of 3 averages 23.7 (computed), PF-18: at 0b6fec28, 10 October 2026, 02:56 to 03:12 UTC, lab B idle, the 5 ticks of 8 that reused the pass made 20 requests, read 27.8 MiB (1.5 MiB on the wire) in 2.4 s and 1.8 CPU-seconds, and the 2 that listed every pod 31, 50.4 MiB, 4.4 s and 3.4 CPU-seconds, peak heap 30.5 MiB; medians; that run was made at a host load average of 8.5 to 11.9, so its times and CPU are not comparable with the 9810fb6 run's below, and the wire bytes, 3.3 MiB against 3.1, differ by fill, not by load: a second run on the same fill read 3.3 MiB too) and read 50.4 MiB (3.1 MiB on the wire) in 3.7 s and 3.0 CPU-seconds: the same bytes and CPU, with a peak heap of 39.2 MiB and a peak RSS of 65.0 MiB (29.3 and 57.1 in the #226/#228 runs of 735751d (with the recorder, 70d16ab): the larger pod pages); its first tick made 1,037 requests and took 42.4 s and 22.0 CPU-seconds on a host whose load average rose to 15.4 (27.1 s and 22.8 at 59d8561, at a load of 7 to 12, where 735751d took 36.1 s and 23.1). Requests grow linearly (ceil(N / 500) per list, pods and nodes up to 1,000 a page since #228; about 15 constant before #228, 4 of them API discovery, and 9 on a steady tick since). Measured against a kind 1.37.0 control plane with KWOK fake nodes on a 2-core i3 that also ran the agent, so with no kubelet load and no network latency, in the benchmark process (outside a CPU quota, PF-14); the Argo CD and Flux lists are PF-13, measured on 735751d only. |
TestBenchAgentTick TestGrowingTheFillAddsExactlyTheDifference hack/bench/agent.sh hack/bench/agent-report.sh make bench-agent; the numbers: manual run, not automated: #99 (the runs are in #71 and #228) |
| PF-10 | The agent fetches a Helm release's Secret or ConfigMap only when the storage object is new or changed (keyed by namespace, name, UID and resourceVersion): a steady tick makes no release GETs. The cache holds a few fields per release, never a payload; drops an entry when its object stops being the installed revision; remembers no failed fetch; keeps a driver's entries when only that driver's list failed; reports an undecodable release or an unparsed manifest in the same words every tick without fetching it again; keeps what a Helm step decoded before its deadline, so the next tick fetches only the releases it left unread (as a pod at 200m the next tick's release GETs were exactly the releases left unread, 43, 51 and 83 in three runs of da90a86e and 9687cffc, 9 October 2026; it used 6.1 to 8.5 CPU-seconds more than a steady tick, which is those releases' own: they are the largest of the bench fill, which sort last, about 8 times a mean release's CPU to decode, and in the profiled run decoding, fetching and collecting the garbage of the 51 were 97% of the excess; nothing was decoded twice, #247); is emptied when the knowledge base flags other APIs; hands out copies; and a one-shot scan has none. |
TestHelmCacheSkipsReleasesItAlreadyDecoded TestHelmCacheRefetchesWhatChanged TestHelmCacheKeepsOnlyWhatIsStillInstalled TestHelmCacheDoesNotRememberAFailedFetch TestHelmCacheKeepsReportingWhatItCouldNotRead TestHelmCacheIsInvalidatedByADifferentKnowledgeBase TestHelmCacheHandsOutCopies TestHelmCacheNeedsAnIdentityToKeyBy TestHelmCacheCoversTheConfigMapDriver TestHelmCacheKeepsADriversEntriesWhenItsListFails TestHelmCacheKeepsWhatAPartialStepDecoded TestTicksFetchHelmReleasesOnlyWhenTheyChange TestLargeReleasesAreTheLastFiveNamespaces; the pod numbers: manual run, not automated, TestHelmDecodeCostByRelease (opt-in), #247 |
| PF-11 | serve ingested 200 clusters x 3 targets, all pushing at once, on SQLite and on Postgres 17 without a failed or retried push: 56 and 40 new snapshots a second, p50 and p99 1.6 and 3.5 s (SQLite) and 2.3 and 4.8 s (Postgres), about 25 CPU-ms per new snapshot, and database growth of at most about 100 KiB a snapshot on SQLite (WAL and page overhead included) and 19 KiB on Postgres (three evaluations each). Generated inventories averaging 28 KiB, on a 2-core i3 that also ran the database. |
TestBenchServeIngest hack/bench/serve.sh hack/bench/serve-report.sh make bench-ingest; the numbers: manual run, not automated: #99 (the run is in #71) |
| PF-12 | The agent benchmark reads only the kubeconfig it is given, never the default one or the current context, and refuses a cluster with more than 3 real nodes or a namespace beyond kind's and its own; KWOK, and with BENCH_GITOPS=1 the Argo CD and Flux CRDs, are pinned by version and sha256 and installed into that cluster only (only the one OCIRepository CRD is cut out of source-controller's six; any other BENCH_GITOPS value is refused before a build, and without it nothing of the GitOps fill runs); the pod sampler reads the one kubeconfig it is given, never the default, and samples the newest running pod that is not terminating; make bench-agent refuses to run without KUBECONFIG on its command line; the server benchmark refuses an unknown backend or a missing Docker engine, and its throwaway Postgres has a random password that is on no command line, is probed over TCP for readiness, and is removed when the script ends, on success and on failure. |
hack/bench_test.sh (stubbed kubectl, go and docker: the make bench-agent guard, the container's removal on both exits, the DSN absent from every command line) hack/bench/agent.sh hack/bench/serve.sh hack/bench/pod-sample.sh |
| PF-13 | With Argo CD and Flux filled (1,000 Applications, half with spec.sources, and 1,000 HelmReleases, half with a chartRef to an OCIRepository of their own) a steady tick at full size made 81 requests since #248, 50 of them GitOps (20 + 20 + 10 paged lists of 50: Applications, HelmReleases and the OCIRepositories the chartRefs name, one cluster-wide list), with all 2,000 charts resolved, read 59.8 MiB (9.0 MiB GitOps) and took 7.5 s and 5.3 CPU-seconds (the median of 4 ticks at da90a86e, 9 October 2026, on a host at a load average of 20 to 28 with another session's lab up and using 10 to 90% of a core); before #248, on main 735751d (4 October 2026), it made 596 requests, 540 of them GitOps (500 of them OCIRepository GETs, which are sequential), read 63 MiB (12 MiB GitOps) and took 12.8 s and 6.2 CPU-seconds, against 53 requests, 4.1 s and 3.0 without GitOps (two runs under different host load). The OCIRepositories are listed of the one namespace the chartRefs point into, else cluster-wide; per namespace when the cluster-wide list is forbidden; a GET by name only in a namespace whose list is forbidden too; the agent remembers a refused list across ticks and asks for it again only once an hour (collect.ForbiddenListRecheck), so a role with get alone does not repeat those refused requests on every tick. The seeded objects are the ones the collector reads: the benchmark fails unless the inventory holds every seeded chart, and a test runs the real collection against a fake apiserver holding them. The status of the objects is generated, so the bytes scale with real ones; the request counts do not. |
TestArgoApplicationsAreSingleAndMultiSource TestFluxHelmReleasesUseChartSourcesAndChartRefs TestGitOpsCounts TestGitOpsFillIsOffByDefault TestSeedGitOpsCreatesEveryObjectAndIsRerunnable TestSeedGitOpsStopsAtTheFirstFailure TestSeedGitOpsRetriesTransientFailures TestTheCollectorReadsEveryGitOpsObject TestGitOpsFluxOCIRepositoriesAreListed TestGitOpsRefusedOCIRepositoryListsAreRememberedAcrossTicks TestBenchCapabilityOK TestRequestRecorderCountsRequestsAndBodyBytes hack/bench/agent-report.sh; the numbers: manual run, not automated: #99 (the runs are in #233 and #248) |
| PF-14 | As a pod under the chart's CPU limit (a cgroup quota) against 2,001 nodes and 1,000 Helm releases, the first tick took 73 s at 200m and stopped at its Helm step's deadline with 132 of 1,001 releases unread (1,000 and the chart's own), 48 s at 500m and 35 s at 1 CPU, both complete; a steady tick took 21 s at 200m (throttled in 58 to 80% of its CFS periods) and 7 s at 1 CPU (not throttled). The agent's peak RSS was 66 to 71 MiB. Hence the chart's default agent.resources.limits.cpu is 1 CPU. A few ticks per limit on a loaded host whose lab control plane restarted once during the 1 CPU run (main 735751d, 4 October 2026). Re-measured at 200m at da90a86e (with #226 and #228; 9 October 2026): the first tick took 75.5 s and 15.0 CPU-seconds and stopped at the Helm step's deadline with 43 of 1,001 releases unread, the next took 51.9 s and 9.0 CPU-seconds, fetching only those 43, and a steady tick 29.2 s and 2.9; peak RSS 81 MiB. At 9687cffc (PR #271's tip; 9 October 2026, fresh pods with the profiler on and off): the first tick 67.8 and 67.3 s and 12.9 and 13.1 CPU-seconds, stopped at the deadline with 51 and 83 releases unread; the next 57.4 and 68.5 s and 10.0 and 11.3 CPU-seconds, fetching only those; a steady tick 16.2 and 17.8 s and 2.5 and 2.8 (peak RSS 83 MiB). |
TestAgentDefaultResources hack/bench/pod-sample.sh hack/bench_test.sh; the numbers: manual run, not automated: #99 (the runs are in #229 and #247) |
| PF-15 | A cold Helm step (the agent's first tick, every scan) fetches the releases it has not decoded on 8 workers and decodes them one at a time in their order, so the inventory, the reasons and their order, and the cache are those of fetching one at a time; a canceled context stops the workers, none outlives the step, and once the step's context is done a payload fetched ahead is not decoded but counted unread, as fetching one at a time would have failed it, with the error fetching it one at a time would have given (client-go's Get …: context deadline exceeded for a GET in flight at the deadline, the context's own error for one begun after it). At a 60 ms round trip a cold step of 1,000 releases took 65.04 s fetching one at a time and 14.27 s on 8 workers (4.56x; the client's rate limit of 50 a second after a burst of 300 is the floor: 16 workers took 14.27 s too) against a fake apiserver on an Apple M1 Pro at a load average of about 6, at commit 5da764e; on the scale lab with 60 ms added by netem, main's first tick (735751d with the recorder, 70d16ab) reached the Helm step's deadline after 67.6 s with 231 of 1,000 releases unread, and with 8 workers the whole first tick took 30.4 s at 06cdf7a (29.6 s at 5da764e) and read all of them (4 October 2026; lab B up and holding no fill, 11 to 22% of a core). It does not make the decoding cheaper (22.0 to 22.8 CPU-seconds for a first tick at its commits, 23.1 to 23.5 on 735751d), so under a CPU quota, where the decoding is the bound, it is not expected to help: at 200m main's first tick stopped at the Helm step's deadline with 132 of 1,001 releases unread, and decoding 1,000 releases takes more CPU than 0.2 CPU gives in the step's 59 s (at most 54 s since the tick reserve, #238); at 1 CPU it completed in 35.2 s, and #226 could save at most the time the GETs wait, which for main's 1,000 sequential GETs beside the apiserver was 13.4 s in all, in the benchmark (PF-14; computed). A pod at da90a86e (with #226 and #228) at 200m still stopped at that deadline, with 43 of 1,001 unread, and at 9687cffc with 51 and 83 (PF-14, 9 October 2026; a different host load, and #228 besides #226). |
TestCollectHelmConcurrentMatchesSequential TestStartFetchingHandsOverInOrderWithinItsBound TestStartFetchingWithOneWorkerFetchesOnNext TestStartFetchingStopsPromptlyOnCancel TestStartFetchingHandsOverNothingOnceDone TestStartFetchingFailsAsTheLoopDoesOnceDone TestCollectHelmFetchesConcurrentlyWithinItsBound TestCollectHelmColdFetchAtRoundTrip; the lab numbers: manual run, not automated: #99 (the runs are in #226) |
| PF-16 | The agent asks for API discovery's groups and resources only when its last complete answer may be stale: the server version (read every tick) changed, which drops it before any step reads it; the CRDs' groups, kinds or served versions changed or the CRD list failed (dropped for the next tick; custom resources that could not be listed, which make the crds capability partial on every cluster with a CRD that deprecates a version, are not a reason, since the CRDs were read); it is an hour old; or it has none, since an answer with any error is not kept. The CRDs are read after discovery is asked, by the later crds step, so a CRD changed between those two steps of the tick that asks is taken as what discovery saw and is seen only when the answer is dropped for another reason, at the latest an hour later. A one-shot scan keeps nothing. A refresh costs 4 requests (/api, /apis, /api and /apis again), plus one for each GitOps group-version the cluster serves that the GitOps detection asks about: a refresh with the Argo CD and Flux CRDs served made 6 (round 3, tick 26 of a source build of rc.3, from the API server's audit log, #99). |
TestDiscoveryCacheServesSteadyTicks TestDiscoveryCacheStaleness TestDiscoveryCacheKeepsNoFailure TestTicksAskForDiscoveryOnce |
| PF-17 | A tick reads its ClusterReadiness once and writes the status over what it read, so a steady tick makes one GET and one status UPDATE; it creates the object only when the read finds it missing, and a write that conflicts (an edit since the read) reads it again. |
TestTickReadsTheClusterReadinessOnce TestWriteStatusOverReadsOnlyOnConflict TestTickStampsTheGenerationItEvaluated |
| PF-18 | The agent lists the pods outside kube-system once per --pod-pass-every ticks (default 3, the full pass being the first of them) and, in between, detects add-ons from the images and labels that pass read; --pod-pass-max-age (default 1h) forces a pass sooner. So an add-on installed or upgraded right after a pass, other than through Helm or a GitOps chart reference, is reported as it was for at most --pod-pass-every minus 1 ticks, and a tick does not reuse a pass that is the maximum age old or more (the age is that of the tick which reuses it, whose report stays up until the next tick). The registry is applied to the reused evidence on every tick, and the kube-system pods, Helm releases, GitOps resources and IngressClasses are read on every tick. A pass that failed or read only some of its pages is never kept, drops the evidence of an earlier one and forces a full pass on the next tick (a tick that lists every pod drops the held pass before its first list, so the old and the new evidence are never held together, and a pass that fails at any page leaves nothing held), as does a tick on which versions did not list kube-system, and a tick on which the Helm releases or GitOps chart references that name a registry add-on (chart, chart version, appVersion, revision, status, target) differ from those the pass was taken with: they are read on every tick, so the trigger costs no request, and a release upgraded across a release line is not reported beside a phantom install from the old pods; releases of no add-on do not count; the first tick, --pod-pass-every=1, a nil cache and scan read every pod. A --pod-pass-max-age that is not above 1.1 times --interval times (--pod-pass-every minus one), the longest spacing the tick jitter draws for each of the ticks that would reuse the pass, cannot reuse it on all of them, and the agent logs a warning at start with that floor (mustExceed; the product saturates at the largest duration for a --pod-pass-every that would overflow it) and the largest --pod-pass-every that fits the maximum age (podPassEveryThatFits): at or below --interval every pod is listed on every tick except one the jitter brings early, and above the floor the last reuse can still fail, because each spacing also holds the tick's run time. The detection of a reusing tick is that of a full one over the same cluster. The age of the reused evidence is addOnEvidenceAgeSeconds in the inventory and report (an optional field, absent when every pod was read), status.addOnEvidenceAgeSeconds, the tick log line and upgradescope_addon_evidence_age_seconds, and is left out of the snapshot hash on the agent and the server, so a reusing tick pushes nothing new. No gap is raised for a reused pass. Measured at the 2,001-node fill (0b6fec28, 10 October 2026, 02:56 to 03:12 UTC, lab B idle, in the benchmark process): 20 requests on the ticks that reused the pass and 31 on those that listed every pod, a cycle of 3 averaging 23.7 (computed); those figures stand at daf513a4, whose commits since 0b6fec28 that change a tick are 8ca7e786 (no add-on chart in the fill, so no change) and 81761afd (the held pass, a few tens of KiB at this fill, is dropped before the first pod list instead of after: checked from the seed, not re-measured); a metadata-only trigger from workloads was weighed and not built (computed, at the metadata list page size of 500: 10 requests more on every tick, 8 pages of the 4,000 Deployments and 1 each of the DaemonSets and StatefulSets, so a cycle of 3 would average 33.7). |
TestPodPassReusedBetweenFullPasses TestPodPassStalenessIsBoundedByTheCount TestPodPassAppliesTheCurrentRegistryToReusedEvidence TestPodPassMaxAgeForcesAFullPass TestPodPassFailedOrPartialPassForcesTheNextOne TestPodPassFailureDropsTheEarlierPass TestPodPassOldPassIsDroppedBeforeTheNewOneIsRead TestPodPassNeedsTheKubeSystemListOfThisCollection TestPodPassHelmUpgradeForcesAFullPass TestPodPassIgnoresReleasesOfNoAddOn TestPodPassGitOpsChartChangeForcesAFullPass TestPodPassEveryOneReadsEveryPod TestPodPassRecordKeepsDistinctEvidenceOutsideKubeSystem TestConfigPodPassDefaultsAndLimits TestAgentPodPassFlags TestRunWarnsWhenPodPassMaxAgeIsTooShortForPodPassEvery TestPodPassMaxAgeFloorFollowsTheJitterBound TestPodPassMaxAgeFloorSaturates TestPodPassEveryThatFits TestTickReusesThePodPassAndRecordsItsAge TestSnapshotHashIgnoresAddOnEvidenceAge TestAddOnEvidenceAgeIsNotANewSnapshot TestCanonicalDropsOnlyTheCollectionsOwnFields TestEvaluateCarriesAddOnEvidenceAge TestBenchAgentTick hack/bench/agent-report.sh; the numbers: manual run, not automated: #99 (the runs are in #228) |
Under repair¶
These claims did not hold, wholly or in part, when audited. Each issue carries the reproduction; the claim moves into a table above when its fix lands with a test.
| ID | What did not hold | Proven by |
|---|---|---|
| PF-06b | The server's RSS under adversarial /gate and ingest load was re-measured by hand on macOS (the table in Running the server), and no gate measures it in the chart's server pod on kind (hack/e2e.sh has no memory step). The round-3 audit saw the bound hold in that pod under combined /gate and ingest load, without an OOM kill (#99); that run is not automated. |
not automated yet: #99 (#121 closed without a gate) |
| SE-09b | The chart's server TLS (agent pushing over HTTPS with the Secret's CA, probes and ServiceMonitor over HTTPS, a cert-manager-issued certificate) is proven by rendered-template tests only, not on a kind cluster. Round 3 confirmed the rendering (helm template with server.tls.secretName, server.sharedIngestToken=false and agent.serverToken gives an https --server-url, the CA volume, --tls-cert-file, HTTPS probes and the token under data:, SE-09); the live run, an agent pushing over HTTPS, was not exercised. |
not automated yet: #99 (#126 closed without a kind step) |
Not yet audited¶
Round 2 of the audit (#99: the missions on the Action, the agent's footprint, the knowledge-base pipeline, performance and the docs) ran and filed #165 to #169 and #189 to #200, and round 3 ran against v0.2.0-rc.3. What is left are the claims no mission has confirmed in full, or that passed in part with the rest named in the row. Each lists the tests that already cover part of it; none is fully pinned. A claim moves into a table above, or under repair, when a mission confirms it or a test pins it.
| ID | Claim | Tests so far |
|---|---|---|
| AC-01 | The Action (composite, in action/) scans a directory of rendered manifests for removed and deprecated APIs and emits SARIF. |
ci:action hack/action_test.sh TestWriteGitHubAcceptable; not audited: #99 |
| AC-04 | The workflow on the CI gate page, in action/README.md and in the README (abd-ulbasit/upgradescope@v0.3.0, security-events: write, and an upload-sarif step that runs when the step did not cancel, wrote a SARIF file, and the event is not a pull request from a fork) works as written on a push and on a same-repository pull request that is not Dependabot's (its token is read-only). On a fork pull request the gate runs and the job fails only when the gate fails; the SARIF upload is skipped, because the fork's token is read-only, and the step summary and annotations are the report (#357). |
hack/docs-sarif-guard_test.sh (the guard in all three copies, evaluated for a push, a same-repository and a fork pull request); not audited: #99 (it needs the v0.2.0 release, #127); the fork behaviour is read from GitHub's token rules, not from a fork run |
| AG-01 | The agent uses no controller-runtime, informer cache, leader election, webhooks, finalizers or owner references. Its pod reads stay periodic lists: since #228 it lists the pods outside kube-system every --pod-pass-every ticks and reuses that pass's images and labels in between (PF-18), and the chart's role still grants no watch. |
e2e:no_webhooks_or_finalizers e2e:audit_agent_writes_only_its_cr TestPodPassReusedBetweenFullPasses; the dependency set and watch/lease traffic: not audited, #99 |
| AG-14 | The knowledge base moves only with the binary and the target only with a new upstream minor, with no Kubernetes event. | TestResolveTargetsDefaultNextMinor TestDatasetVersion; not audited: #99 |
| AO-06 | Everything but control-plane skew works identically on managed and self-managed clusters (managed add-on images included). | TestMatchAddOnsRealWorldImages TestCollectVersionsManagedClusterEmptyControlPlane; not audited: #99 |
| DO-06 | Apache-2.0; third-party licenses in NOTICE; endoflife.date data under MIT (registry/DATA-LICENSE.md). |
not audited: #99 |
| KB-03 | CI reruns the API-lifecycle generator on every push and fails when the committed copy drifts; upstream drift is a warning. | ci:kb-freshness TestDatasetSanity; not audited: #99 |
| KB-07 | CI validates the registry and runs eol-sync -check on pull requests that touch registry/. |
ci:registry TestValidate; not audited: #99 (a pre-audit read of PR #82 found both steps skipped: "no merge base") |
| KB-08 | The weekly kb-refresh opens a reviewable PR, CI re-validates it, and a broken refresh is never silent: a job that fails or hits its timeout (every job has timeout-minutes) opens or updates the kb-refresh is failing issue, a person cancelling the run does not. |
hack/kb-refresh-scope_test.sh (the failure and timeout reporting, evaluated offline: that a job hitting its timeout-minutes leaves job-level cancelled() false while the run is not cancelled is read from GitHub's documentation and not yet seen on GitHub) hack/kb-refresh-ci_test.sh (the workflow's shape). The PR path on a non-empty patch ran in the dispatched forced-diff run 38138195137 on 2026-10-11 (main plus a revert of #277): bot PR #385 carried exactly the four regenerated files, and its held CI runs were approved and passed before it was closed. The first scheduled run, 38270110924 for the Mon 2026-10-12 slot, started at 15:25 UTC, about nine hours after its 06:17 UTC cron time, since GitHub can delay a scheduled run. It opened bot PR #432, a one-line registry/data/karpenter.yaml sync, whose held CI was approved and passed, and the maintainer merged it on 2026-10-12. Not audited: a real timeout: #99 |
| KB-11 | kb.Load fails loudly on an empty or corrupt dataset, never yielding a silently empty knowledge base. A lifecycle file that parses but has fewer than 150 entries or fewer than 100 with a removal is refused as corrupt. |
TestParseLifecycleFile TestLoadRefusesHollowLifecycleData TestCheckLifecycleFloors TestLoadFS TestLoad; an empty registry: not audited, #99 |
| KB-12 | Clean-room: no proprietary code, data, schemas or documents, and no other scanner's dataset copied. | not audited: #99 (KB-05's citations are the automated part) |
| PF-05 | The agent's default requests and limits (50m/64Mi, 1 CPU/256Mi) are enough for a cluster of 2,001 nodes and 1,000 Helm releases, whose pods are each under about 60 KiB encoded. | measured as a pod (PF-14, main 735751d): a peak RSS of 66 to 71 MiB under the 256Mi limit (above the 64Mi request, which only informs scheduling), and a first tick that completes its Helm step at 1 CPU, which it did not at the former 200m; not shown for larger clusters (the about 1,900 releases at 1 CPU, about 1,600 with the GitOps reads that share the Helm step's deadline, are arithmetic from the first tick's wall time and the at most 54 s the Helm step has at the default interval since the tick reserve, not measured) or with the GitOps lists in the pod. With the pod pass reused (PF-18) the benchmark's peak heap was 30.5 MiB on a tick that listed every pod and up to 36.4 MiB on one that reused the pass, and its peak RSS 63.6 MiB (0b6fec28, 10 October 2026, in the benchmark process, not as a pod; at daf513a4 these stand by estimate, not re-measured: 81761afd releases the held pass, a few tens of KiB at this fill, before the first pod list rather than after, which moves no peak the 5 ms sampler could see). Since #228's larger pod pages the benchmark's peak heap was 39.2 MiB and its peak RSS 65.0 MiB (at commit 9810fb6, against 29.3 and 57.1 MiB in the #226/#228 runs of 735751d (with the recorder, 70d16ab); as a pod at that code, da90a86e at 200m, the agent's peak RSS was 81 MiB and the cgroup's peak memory 107 MiB, 9 October 2026, PF-14), and a pod page is bounded by its count only, so its worst case is 1,000 times the largest pod, whatever that pod's size (PF-02): 1,000 pods of about 40 KiB after a page of small ones held up to 125.3 MiB of live heap in a test on GitHub's ubuntu-latest runner (linux/amd64; four runs at 23f8739 and 04c39fa (CI runs 37950213827 and 37964795504), 9 October 2026), about 3.2 bytes of live heap per encoded byte (125.3 MiB for 39.4 MiB); at that rate pods of about 60 KiB after a page of small ones take one page past the agent's GOMEMLIMIT (90% of 256Mi, 230.4 MiB) beside the 39.2 MiB the rest of the agent's heap peaked at in the benchmark ((230.4 − 39.2) MiB / 3.18 / 1,000 is 61.6 KiB; 74.2 KiB with nothing else held) and likely past the 256Mi limit, where pages of at most 500, before #228, would have reached it at about 120 KiB (123.1 KiB; computed, not measured), so the 256Mi limit would not be enough for a cluster whose pods are that large, which Argo Workflows pods, carrying their template, can be; TestBenchAgentTick TestAgentDefaultResources TestPodPagePeakHeapIsBounded; not audited: #99 |
| PF-07 | Each tick issues paged lists (ceil(N / 500) requests per resource type, fewer for pods and nodes, PF-02; since #362 that includes one list of the PersistentVolumes and one of the StorageClasses on every tick, a tick that reuses the pod pass included, which the round-3 audit log shows on 48 of 48 ticks) and one GET per Helm release the agent has not decoded yet, with no watch, so the agent's cost is predictable: it grows with the cluster's pages, and with its releases only on the first tick after a start or when a release changes, which Architecture states. What is bounded is the work held at once, not the request count. | TestCollectAPIUsageFollowsListPagination TestCollectHelmPeakHeapIsBoundedByOneRelease TestCollectReadsEachPodOncePerTick TestTicksFetchHelmReleasesOnlyWhenTheyChange TestTicksAskForDiscoveryOnce TestTickReadsTheClusterReadinessOnce TestCollectLiveVolumesTwoExtraRequests; the request counts at scale: PF-09 (a manual run, not automated: #99; the run is in #71, and it predates the two volume lists) |
| SE-11 | CI jobs run with least privilege, actions are pinned by full commit SHA, and no release job restores a cache. | hack/release-ci-permissions_test.sh hack/kb-refresh-ci_test.sh (kb-refresh, IR-18) hack/pg-images_test.sh (images, IR-19) hack/release-caches_test.sh (caches, IR-22) hack/kb-refresh-scope_test.sh (unreviewed build off main, IR-23); not audited: #99 |
| SE-13 | Vulnerabilities can be reported privately through GitHub's private vulnerability reporting. | not audited: #99 |
| SK-03 | kubectl client skew is out of scope: only audit logs reveal clients (README, version-skew page). | not audited: #99 (the README no longer also lists it among the questions answered) |