Azure Blob ETag mismatch causing significant churn

[JonathonAnderson]

Bucket (provider: azure) re-downloads the whole container on every reconcile because the listing and download ETags never match

Describe the bug

For a Bucket with provider: azure, source-controller treats the container as changed on every reconcile, even when nothing in it has changed.

The Azure provider returns an ETag in one format when it lists the container (VisitObjects) and in a different format when it downloads a blob (FGetObject). The reconciler decides whether the container has changed by comparing two digests:

• one built from the listing ETags;
• the stored artifact revision, which was built from the download ETags.

These two digests never match. So on every interval:

1. syncBucketArtifacts reports changed, and every blob in the container is downloaded again.
2. The Bucket's status goes through Ready=Unknown, Reconciling=True and ArtifactOutdated=True (reason NewRevision), with the message building artifact: new upstream revision '<rev>'. <rev> is the revision the Bucket already has.
3. After the download, every index entry holds the download ETag. The digest then matches the stored revision again, reconcileArtifact logs artifact up-to-date with remote revision, and the status goes back to Ready=True.

No new artifact is written, so the result is correct. But every Azure Bucket repeats the full download and the status change on every reconcile.

The two ETag forms

	Listing (VisitObjects)	Download (FGetObject)
Azure REST operation	List Blobs	Get Blob
Where Azure returns it	<Etag> element in the XML response body	ETag response header
Azure's format	Unquoted: 0x8D52D5C4A4C96B0 (see Microsoft's sample List Blobs response)	Quoted: "0x8D52D5C4A4C96B0". The docs say: "If the request version is 2011-08-18 or later, the ETag value is enclosed in quotation marks" (Get Blob)
azblob v1.7.0 field (the value is copied through unchanged)	BlobProperties.ETag *azcore.ETag from xml:"Etag" (zz_models.go#L97)	BlobClientDownloadResponse.ETag from resp.Header.Get("ETag") (zz_blob_client.go#L1100)
What source-controller does with it	fmt.Sprintf("%x", *blob.Properties.ETag)	string(*res.ETag)
Value that reaches the index	3078384435324435433441344339364230	"0x8D52D5C4A4C96B0"

azcore.ETag is type ETag string. So %x doesn't format a number: it hex-encodes each byte of the string, turning 17 characters into 34 hex digits. Even without the hex-encoding the two values would differ, because Azure quotes one and not the other.

Where the ETags are acquired (internal/bucket/azure/blob.go)

The permalinks below point to main at 143c11a.

Listing: VisitObjects, blob.go#L372-L375

var etag string
if blob.Properties != nil && blob.Properties.ETag != nil {
	etag = fmt.Sprintf("%x", *blob.Properties.ETag)
}

Download: FGetObject, blob.go#L338-L344

var etag string
if res.ETag != nil {
	etag = string(*res.ETag)
}
return etag, nil

Where they are consumed for comparison

Each step is listed in the order the reconciler runs it.

1. The index is filled with listing ETags. fetchEtagIndex calls index.Add(key, etag) with the hex-encoded values from VisitObjects (bucket_controller.go#L723).
2. The digest covers the exact ETag string. Digester.Digest hashes one "<key> <etag>\n" line per key, in sorted order (digest.go#L173-L187, writeLine digest.go#L219-L221). So any difference in how an ETag is written produces a different revision.
3. The change check always fails. syncBucketArtifacts compares the stored revision, which was built from download ETags, with the digest of the listing ETags (bucket_controller.go#L978):

   changed = curRev.Validate() != nil || curRev != index.Digest(curRev.Algorithm())

   This is always true, so the function calls fetchIndexFiles and returns true (#L982-L986).
4. Every entry is overwritten with the download ETag. In fetchIndexFiles, t is the hex listing ETag and etag is the quoted download ETag. They always differ (bucket_controller.go#L774-L776):

   if t != etag {
   	index.Add(k, etag)
   }

5. The status reports a new revision. Because changed is true, reconcileSource marks ArtifactOutdated=True with reason NewRevision and sets the progressing message building artifact: new upstream revision '<rev>' (bucket_controller.go#L479-L485). <rev> is computed from the index, which now holds download ETags, so it's the revision already stored.
6. The artifact check then succeeds. reconcileArtifact compares index.Digest(curRev.Algorithm()) == curRev, and it now matches. It logs artifact up-to-date and clears ArtifactOutdated (#L516, #L527-L528).

In short, the stored revision is always a digest of download ETags, but the change check in step 3 always uses a digest of listing ETags.

The other providers already match

GCP and MinIO read both ETags from the same SDK field, unchanged. Only the Azure provider transforms one side.

Provider	Listing	Download
GCP	object.Etag (gcp.go#L323)	objAttr.Etag (gcp.go#L303)
MinIO	object.ETag (minio.go#L354)	stat.ETag (minio.go#L336)

History

• Introduced with the Azure provider in ec5bc1a (2022-03-01, "Implement Azure Blob BucketProvider"): fmt.Sprintf("%x", *blob.Properties.Etag) for listing, return *res.ETag, nil for download.
• Kept in 754b20b (2022-10-07, "Update Azure Blob Storage SDK to v0.5.0").
• Kept in f6d176b (2026-08-04, GitHub repository_dispatch event trigger Provider #2122, "fix: harden Bucket reconciliation error paths"), which added nil checks around both lines.

The existing tests don't catch it. TestBlobClient_VisitObjects_Prefix checks only paths, and TestBlobClient_VisitObjects_MissingFields covers only missing ETags. No test compares the listing and download ETags for the same blob.

Steps to reproduce

1. Create an Azure Blob container with at least one blob, and a Bucket that points at it. Any authentication method works.

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: Bucket
   metadata:
     name: repro
     namespace: flux-system
   spec:
     provider: azure
     bucketName: <container>
     endpoint: https://<account>.blob.core.windows.net
     interval: 1m

2. Wait for Ready=True, then leave the container alone.

3. Watch the conditions:

   kubectl get bucket repro -n flux-system -w \
     -o jsonpath='{.metadata.resourceVersion}{range .status.conditions[*]} {.type}={.status}/{.reason}{end}{"\n"}'

4. On every interval you'll see:

   • ArtifactOutdated=True/NewRevision appear and disappear, with the same revision each time;
   • source-controller log artifact up-to-date with remote revision with that same revision.

   In the same reconcile, fetchIndexFiles downloads every blob in the container.

Expected behavior

If the container hasn't changed:

• the listing digest equals the stored revision, so changed is false;
• nothing is downloaded;
• the Bucket stays Ready=True, without ArtifactOutdated or the "new upstream revision" message.

Suggested fix

Make VisitObjects return the same string that FGetObject returns for the same blob: the quoted header form.

This is better than stripping the quotes from both sides. Every existing artifact revision was computed from download ETags (step 4 above), so matching the listing to the download form keeps those revisions valid. Stripping quotes from both would change the revision of every Azure Bucket once, which makes every Kustomization that uses one re-apply.

// quoteETag returns etag in the quoted form that the Get Blob ETag response
// header uses for API version 2011-08-18 and later, so that values read from
// List Blobs and from Get Blob compare equal.
func quoteETag(etag string) string {
	if etag == "" || strings.HasPrefix(etag, `"`) {
		return etag
	}
	return `"` + etag + `"`
}

// VisitObjects
if blob.Properties != nil && blob.Properties.ETag != nil {
	etag = quoteETag(string(*blob.Properties.ETag))
}

For a regression test, the mock server in blob_test.go already serves <Etag>0x8D9B2A2A2A2A2A2</Etag> in its list XML. It could also send ETag: "0x8D9B2A2A2A2A2A2" on the blob download, and the test would assert that VisitObjects and FGetObject return equal ETags for the same blob.

Screenshots and recordings

These come from an affected cluster, with names removed. They show the Bucket's conditions during one reconcile of an unchanged container. The revision is identical throughout.

# before
Ready=True              Succeeded    stored artifact: revision 'sha256:10380817…'

# one status write, during the reconcile
Reconciling=True        Progressing  building artifact: new upstream revision 'sha256:10380817…'
Ready=Unknown           Progressing  building artifact: new upstream revision 'sha256:10380817…'
ArtifactInStorage=True  Succeeded    stored artifact: revision 'sha256:10380817…'
ArtifactOutdated=True   NewRevision  new upstream revision 'sha256:10380817…'

# the next status write
Ready=True              Succeeded    stored artifact: revision 'sha256:10380817…'
ArtifactInStorage=True  Succeeded    stored artifact: revision 'sha256:10380817…'

source-controller logs one of these lines per interval:

{"level":"info","ts":"2026-09-29T16:14:57.886Z","msg":"artifact up-to-date with remote revision: 'sha256:10380817…'","controller":"bucket",...}
{"level":"info","ts":"2026-09-29T16:15:56.548Z","msg":"artifact up-to-date with remote revision: 'sha256:10380817…'","controller":"bucket",...}
{"level":"info","ts":"2026-09-29T16:16:59.628Z","msg":"artifact up-to-date with remote revision: 'sha256:10380817…'","controller":"bucket",...}

OS / Distro

N/A (in-cluster controller on AKS)

Flux version

Flux v2.8.8 / source-controller v1.8.5, as shipped in the AKS microsoft.flux extension 1.25.1 (image mcr.microsoft.com/oss/v2/fluxcd/source-controller:v1.8.5-6). The code paths above are unchanged on main at 143c11a.

Flux check

N/A (Flux is managed by the AKS extension)

Git provider

N/A (Azure Blob Storage Bucket source)

Container Registry provider

N/A

Additional context

On AKS, the microsoft.flux extension copies Bucket conditions into its FluxConfig status. So this per-interval status change also causes FluxConfig status writes and a large volume of fluxconfig-agent log lines and an even more significant volume of StorageBlobLogs.GetBlob, which is how we found it.

The unquoted listing format above comes from Microsoft's sample response. Either way, the %x alone guarantees the two values differ, and the per-interval NewRevision → artifact up-to-date cycle shows they do in practice.