feat(blob): blob lease support for the Azure Functions storage backplane

[cmcconomyfwig]

Closes #136

⚠️ Stacked on #182 — this branch includes the Table OData error-envelope commit from #182 (fix(table): wrap error responses in the OData error envelope), because the Durable Functions scenario needs both fixes to work end to end. Please merge #182 first; I'll rebase this branch onto main afterwards so it shows only the blob-lease commit. (If you'd rather take both in one go, merging this PR as-is also lands #182's change verbatim.)

What this is

The storage-binding half of #136. That issue's full ask — durable/timer triggers inside floci-az's own hosted-function runtime — is an architectural feature for the maintainers. But the reason fleets like ours can't use floci-az at all for durable/timer workloads today is narrower: the Azure Functions host runs fine in our own containers, pointed at floci-az as its AzureWebJobsStorage backplane — except that backplane is missing its coordination primitive. Blob leases are not implemented (comp=lease → 501), and Get Blob hardcodes x-ms-lease-state: available.

Blob leases are what the Functions host runs on:

• WebJobs singleton locks (every timer trigger takes one) are blob lease acquire/renew/release cycles on azure-webjobs-hosts/locks/...
• Durable Functions partition management acquires, renews, steals (break + acquire), and hands off (change) blob leases
• All lease-holder writes are lease-conditioned (x-ms-lease-id), which only works if the emulator enforces the guards

Change

• BlobLease — one lease's state as a pure function of (lease, now): LEASED / EXPIRED / BREAKING / BROKEN. Expiry and break timing never mutate anything, so the state machine is unit-testable with synthetic clocks.
• BlobLeaseService (@ApplicationScoped, same pattern as UserDelegationKeyService) — Lease Blob dispatch and the write guard. Lease state is an in-memory map: like a real lease it's transient runtime state, and an emulator restart is equivalent to every lease having expired.
• BlobServiceHandler — routes PUT ?comp=lease (blob-level), enforces the lease guard on every write path (PutBlob, SetBlobMetadata, DeleteBlob, PutBlock, PutBlockList), reports real lease state in Get Blob/Properties, and drops lease state when the blob/container is deleted or reset.

Status/error codes follow the Lease Blob contract:

Scenario	Response
acquire on leased blob	409 LeaseAlreadyPresent
re-acquire with same proposed id	201 (idempotent, SDK retry path)
renew/change/release with wrong id	409 LeaseIdMismatchWithLeaseOperation
lease op with no lease	409 LeaseNotPresentWithLeaseOperation
renew after break	409 LeaseIsBrokenAndCannotBeRenewed
acquire duration outside −1 / 15–60	400 InvalidHeaderValue
write to leased blob, no lease id	412 LeaseIdMissing
write to leased blob, wrong id	412 LeaseIdMismatchWithBlobOperation
write with lease id, no lease	412 LeaseNotPresentWithBlobOperation

Deliberately out of scope: container leases (nothing in the Functions host path needs them) — restype=container&comp=lease still 501s honestly rather than half-working.

Tests

• Unit: BlobLeaseTest (15 HTTP-level tests: lifecycle, error codes, write guards, lease-dies-with-blob) and BlobLeaseStateTest (6 pure state-machine tests: expiry, renew-resets, break windows). All watched fail against main (501s / hardcoded headers) before implementing. Full suite: 575 tests green.
• Compat (sdk-test-java): 4 new BlobCompatibilityTest cases using the real BlobLeaseClient — lifecycle, write guards, steal-via-break, change handoff. All 4 error against the released 0.10.0 image and pass against this branch.
• BlobCompDispatchTest: lease removed from the unimplemented-comp list (that guard's job — don't corrupt blobs via putBlob fallthrough — is now covered by the real implementation).

The Java compat pass caught a real cross-SDK bug in my first cut: the acquire response carried an ISO-8601 Last-Modified, which .NET tolerates but the Java SDK parses strictly (RFC1123) and crashes on. Lease responses now echo the blob's actual ETag/Last-Modified in wire format.

Verification against real SDKs

A .NET Azure.Storage.Blobs 12.24 harness (the SDK family the Functions host uses) runs a 12-step sequence — acquire-infinite, props, write-no-lease, write-wrong-lease, write-with-lease, renew, competing acquire, change, break, steal-after-break, props, content — against Azurite 3.35.0 and against this branch. Output is line-for-line identical:

acquire-infinite: 201 ok
props-leased: Locked/Leased/Infinite
write-no-lease: 412 LeaseIdMissing
write-wrong-lease: 412 LeaseIdMismatchWithBlobOperation
write-with-lease: 201 ok
renew: 200 ok
second-acquire-conflict: 409 LeaseAlreadyPresent
change: 200 proposed-id
break: 202 remaining=0
steal-after-break: 201-then-release ok
props-released: Unlocked/Available
content: renewed-content

Next step on our side once this and #182 ship: run a real Durable Functions fleet (fresh start and restart-with-existing-state) against floci-az as AzureWebJobsStorage and report back on #136 with whatever the host surfaces next, if anything.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

The PR adds in-memory Azure Blob lease lifecycle support and lease-conditioned writes for the Azure Functions storage backplane. The latest follow-up commits also serialize related blob and container mutations to close the previously reported races.

• Implements acquire, renew, change, release, and break operations with lease-state response headers.
• Enforces lease IDs across blob writes, metadata updates, deletion, block staging, and block commits.
• Adds state-machine, HTTP-level, and Java SDK compatibility coverage.
• Moves container and blob mutation preconditions into a shared exclusive section.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported validation, lease atomicity, container lifecycle, and PutBlob authorization races are addressed in the current code.

Important Files Changed

Filename	Overview
src/main/java/io/floci/az/services/blob/BlobLease.java	Defines the immutable lease state machine and correctly caps break completion at a fixed lease’s natural expiry.
src/main/java/io/floci/az/services/blob/BlobLeaseService.java	Implements validated lease transitions, write guards, response headers, and the shared reentrant monitor used to linearize lease and blob mutations.
src/main/java/io/floci/az/services/blob/BlobServiceHandler.java	Routes blob lease requests, enforces lease guards, reports lease state, and synchronizes blob/container mutation preconditions with their store updates.
src/test/java/io/floci/az/services/BlobLeaseTest.java	Adds HTTP-level coverage for lease lifecycle, protocol errors, guarded writes, and lease cleanup.
src/test/java/io/floci/az/services/blob/BlobLeaseStateTest.java	Covers expiry, renewal, break timing, and fixed-lease break bounds with synthetic clocks.
compatibility-tests/sdk-test-java/src/test/java/io/floci/az/compat/BlobCompatibilityTest.java	Adds real Azure Java SDK compatibility coverage for lifecycle, write guards, lease stealing, and ownership handoff.

Sequence Diagram

sequenceDiagram
    participant SDK as Azure SDK / Functions Host
    participant Handler as BlobServiceHandler
    participant Lease as BlobLeaseService
    participant Store as StorageBackend
    SDK->>Handler: "PUT blob?comp=lease"
    Handler->>Lease: exclusively()
    Lease->>Store: Verify blob exists
    Lease->>Lease: Validate and transition lease
    Lease-->>SDK: Lease response and headers
    SDK->>Handler: Lease-conditioned blob mutation
    Handler->>Lease: exclusively()
    Lease->>Lease: Validate current lease ID
    Lease->>Store: Apply mutation atomically
    Handler-->>SDK: Azure-compatible response

Loading

_{Reviews (9): Last reviewed commit: "fix(blob): classify PutBlob authorizatio..." | Re-trigger Greptile}

[cmcconomyfwig]

All three findings verified against the code and fixed in f8f336c:

1. Guard/mutation race (TOCTOU). Confirmed: handleLeaseOp was synchronized but the write guard wasn't, so an acquire could land between validateWrite and the store mutation — and the Durable partition-steal path (break + acquire racing the old holder's lease-conditioned writes) is precisely that interleaving. Fixed by replacing the check-then-act pattern with BlobLeaseService.guardedWrite(request, key, mutation), which runs the guard and the mutation under the same monitor as handleLeaseOp. All five write paths (PutBlob, SetBlobMetadata, DeleteBlob, PutBlock, PutBlockList) now go through it; validateWrite is no longer reachable on its own (made private). On the test-coverage note: a deterministic unit test for the interleaving isn't practical without injecting pause points into production code — the fix removes the race by construction (single monitor), which the guardedWrite javadoc states as the invariant.

2. Malformed change/missing headers. Confirmed, and it was worse than flagged: change without x-ms-proposed-lease-id stored a lease with a null id, and the next renew NPE'd into a 500 with the lease permanently stuck. Now: renew/change/release without x-ms-lease-id → 400 MissingRequiredHeader; change without a proposed id → 400 MissingRequiredHeader; non-GUID proposed ids (acquire and change) → 400 InvalidHeaderValue — so a malformed id can never enter the lease store.

3. Break period validation. Confirmed: negative values were clamped and large values accepted, allowing a lease stuck in breaking for arbitrary time. Now x-ms-lease-break-period outside 0–60 → 400 InvalidHeaderValue, per the Lease Blob contract.

New regression tests (all watched fail before the fix): rejected change leaves the lease intact and renewable, missing-header 400s for all three ops, break period 99999/-1 rejected with the lease still active, malformed proposed id rejected on acquire and change. Suite: 579 tests green; the Java BlobLeaseClient compat tests and the .NET Azure.Storage.Blobs harness (line-for-line vs Azurite 3.35.0) both re-verified against the rebuilt build.

[hectorvent]

Thank you for building this out, and for flagging the stack on #182 in the description yourself, that made the shared history easy to follow.

(blocking) Commits db24941, b6e0788, f8f336c and 7ca231f carry an AI co-author trailer; this repo keeps commit attribution with human contributors (see AGENTS.md). Would you amend or squash those and force push so the attribution is yours alone?

(blocking) This PR also edits CHANGELOG.md, which became generated by semantic release shortly after you opened it, so hand edits conflict with the release commit. Would you drop that hunk? Your change is still credited automatically.

Heads up before you rebase: we merged #166 and #134 today, so this now conflicts on BlobServiceHandler.java as well as two test files. That is our doing, sorry for the churn. Also the description says "Refs #136", so the issue will not auto close; "Closes #136" would wire that up.

[cmcconomyfwig]

Both blocking items addressed, and the stack is rebased onto current main:

• The AI co-author trailers are removed from all four commits — the stack was rewritten and force-pushed, so attribution is with the human contributor. (fix(table): wrap error responses in the OData error envelope #182's branch was updated to the rewritten table commit at the same time, so the stacked history still matches.)
• The CHANGELOG.md hunks are dropped from both commits that carried them; the diff no longer touches that file.
• Conflicts with fix(blob): persist upload HTTP properties #166 and fix(storage): serve service-properties XML for table, queue, and blob #134 resolved. The substantive one was BlobServiceHandler.java: fix(blob): persist upload HTTP properties #166's addBlobHttpProperties now runs inside the lease-guarded mutation in putBlob/putBlockList (replacing this branch's older inline Content-Type handling), and Get Blob emits both the persisted HTTP-property headers from fix(blob): persist upload HTTP properties #166 and the lease headers from this PR. The test-file conflicts were additive on both sides. Full unit suite re-run green after the rebase: 673 tests, 0 failures.

On Refs #136 vs Closes #136: that was deliberate rather than an oversight — this PR is only the storage-binding half. #136's actual ask (durable/timer triggers inside floci-az's own hosted-function runtime) is still open even with leases in place, and the plan in the description is to run a real Durable fleet against this backplane and report findings back on that issue. If you'd rather treat BYO-host-against-the-backplane as resolving #136, say the word and I'll flip it to Closes.

[hectorvent]

Thank you, and apologies this took me a while to review properly.

I have now gone through the lease behaviour against the blob spec and it checks out across the board: acquire returns 201, renew, change and release return 200, and break returns 202, which is exactly what the spec declares for each. The duration validation matches too, 15 to 60 seconds or -1 for infinite, and rejecting outside that with InvalidHeaderValue. Modelling the state machine explicitly, including refusing a renew after an explicit break, is the part I would most expect to be missed.

Two mechanical things left, neither about the lease work:

(blocking) We merged #185 today and it conflicts this branch on TableServiceTest.java, one file. That churn is ours.

The red check is the same node table test as #182, which resolves once that one is updated. Small thing: the description says "Refs #136", so the issue will not auto close; "Closes #136" would wire it up.

[cmcconomyfwig]

Rebased onto current main and force-pushed. Two things worth flagging from the rebase:

• The TableServiceTest.java conflict with fix(core): omit Content-Type on bodyless error responses #185 resolves in the stacked table commit (same resolution as fix(table): wrap error responses in the OData error envelope #182: both sides kept).
• feat: enforce scoped ADLS SAS authorization #158's scoped-SAS authorization also landed since the last rebase and touched every blob write path this PR restructures. The authorize* checks now run ahead of the lease monitor — they read only the SAS token, not blob or lease state, so they don't need to be inside exclusively() — and all of feat: enforce scoped ADLS SAS authorization #158's checks are preserved across the five guarded write paths plus deleteContainer.

Full unit suite green after the rebase: 748 tests, 0 failures. The red node check resolves via #182's test update, verified 11/11 locally against a running emulator.

The description now says Closes #136.

[hectorvent]

Thank you, everything from my last note is addressed: the rebase over #185 and #158, the Closes wiring, and the stack on #182 staying aligned. I also went through your three new commits and they hold up; capping the break period at a fixed lease's natural expiry is exactly what the spec prescribes, and running the authorization checks ahead of the lease monitor matches real Azure's error precedence (403 before 412).

One thing on my side, and it is ours not yours: the #198 merge (ADLS list paths) landed in BlobServiceHandler.java, which conflicts with this branch. That is the third time our merges have broken you here, sorry. The checks on your current head are running now and everything reviewed looks right, so after a rebase onto main and green checks this merges.

[cmcconomyfwig]

Rebased onto current main and force-pushed. With #182 merged, the table commit dropped out of the stack, so the PR now shows only the four blob-lease commits. The #198 conflict was in putBlob — its DataLakeResourceType metadata now rides inside the lease-guarded mutation — and the new resource=filesystem list route is untouched by this branch. The lease code and tests are byte-identical to what you reviewed; the only movement was weaving #198 in.

Full unit suite green on the rebased head: 789 tests, 0 failures.

[hectorvent]

Thank you for the quick turnaround, and for owning the two Greptile findings rather than arguing them. I checked both fixes and they are right: creating the container and classifying the PutBlob authorization now read the same snapshot the mutation uses, under the same monitor, and the body read stays outside the lock.

I also confirmed your claim the easy way: the four lease files are byte identical to the head I reviewed, so the earlier grounding against the blob spec (201 acquire, 200 renew, change and release, 202 break, 15 to 60 second durations, 0 to 60 second break period) still stands, and the #198 metadata hunk now rides inside the guarded write as it should.

All eleven checks are green on this head, including every compat suite. No blockers from my side, merging now. Thank you for seeing this one through three rebases that were our fault, not yours.

[cmcconomyfwig]

Excited to try the new release once it's ready, so I can remove some external container dependencies for azure!

[hectorvent]

🎉 This PR is included in version 0.12.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀