Storage - STG104 Feature Branch by ibrandes · Pull Request #49611 · Azure/azure-sdk-for-java

ibrandes · 2026-06-23T19:41:57Z

No description provided.

* add 103 version * add 104 version

* generate swagger changes * addBlobDownloadHeaders wrapper support, add tests * add recordings * added missing file updates to BlobApiTests * remove added getProperties tests, add check tierChangeTime * remove unused imports * rewrite tests, add recording, add custom check for null for access-tier-inferred

* container feed changes * code cleanup * code cleanup * code cleanup * addressing review comments --------- Co-authored-by: Local Merge <merge@local.tmp>

* Crc polynomial (#43752) * Message encoder (#43803) * wip * basic message encoder logic working * removing print statements and making slightly more readable * fixing several bugs * adding more tests * adding comments for building purposes and to branch off for service testing * wip * wip * working encode that takes in bytebuffer * removing redundant 'this' and making small readability changes * fixing flag bug, adding more comments, adjusting incorrect test case * adding structuredmessage package in implemenation and addressing other comments * refactoring * adding more tests, adjusting test encoder, adding more validation * re-ordering empty buffer return * adding correct comments * addressing comments and fixing ci issues * trying to resolve spotbugs issue * formatting * Extracting non-decoder changes from the decoder PR (#45546) * creating class * replacing encoder constants with new class constants * adding fromValue to StructuredMessageFlags * updating module info to contain more packages * style * undoing module info change * Storage Content Validation - Encoder Performance Improvements (#47531) * adjusting encoder logic * adjusting tests to work with encoder * addressing copilot comments * adding more documentation * Storage - Content Validation Public Interface (#48074) * public facing interface wip * adjusting output stream constructors * removing redundant apis * fixing pageblob api * bloboutputstream adjustment * wip * wip * importing things correctly * formatting * throwing error with client logger * fixing revapi visibility increased error * overload adjustments * adding final to options bags and hopefully resolving revapi error * more edits * more edits * more edits * spotless * spotless * more edits * removing javadoc again * Storage - Content Validation Message Encoder Perf Updates (#47158) * all perf adjustments * addressing copilot comments * Consolidate content validation types under contentvalidation package (#48272) * Storage - Content Validation Encoder Pipeline Implementation (#48354) * adding perf to message encoder tests * changing package name * changing package name * moving checksum algorithm to new package * wip * moving storageschecksumalgorithm out of impl so it can actually be used * removing unused file * bruh * small cleanups * implementation for other APIs * test updates * undoing silly agent changes * async tests * perf tests for all APIs * new path for crc64s without accessible array * md5 compatibility tests * progress reporter tests * changing upload file test to stream file * consolidating conflicting transactional checksum logic * addressing copilot comments * changing policy to compute crc64 header in a non-blocking manner * fixing crc64 policy * addressing copilot comments * deleting perf tests * more tests * addressing more copilot comments * adding recordings and marking multi part tests as live only due to the random block ids * removing unused tests * updating assets * addressing github comments * addressing comments * renaming missed setters * idk * mode resolver cleanup * adjusting mode resolver tests * un-deprecating compute md5 * addressing API view comments * addressing comments * swapping test behavior for new md5 compatibility util * wip * adding validation to prevent progress listener being used with content val * adding comment about flux.generate in encoder class * adding recordings * removing unused imports * Storage - Content Validation Decoder Implementation (#47016) * adding the StructuredMessageDecoder * adding the pipeline policy changes * smart retry changes * fixing smart retry impl * smart retry changes * smart retry changes * smart retry changes * smart retry changes * adding content validation tests * code refactoring * fixing errors i introduced :( * addressing review comments * code refactoring based on latest review comments * code refactoring based on latest review comments * simplifying retry mechanism * removing dead code * addressing Kyle's review comments * addressing latest review comments * refactoring based on latest review comments * refactoring based on latest review comments * refactoring based on latest review comments * refactoring based on latest review comments from kyle * expanding test coverage * removing unused imports * recordings * adding documentation to decoder classes * small fixes and failure path tests * addressing context comment * analyze error * removing close override from decodedresponse * line removal --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage Content Validation - Defer MD5 + CRC64 Val Conflict to Service (#48999) * impl and tests * cleanup * updating assets * strengthening sync assertations * Storage Content Validation - adding failure tests for content validation decoder (#49073) * adding failure tests for content validation decoder * adding Decoder Random Byte Failure Case Test * rearrange code to make it a little more human readable * fix linting --------- Co-authored-by: browndav <browndav@microsoft.com> * Storage Content Validation - Audit DecodedResponse (#49147) * Audit DecodedResponse: tighten override surface, pin UTF-8, add unit tests * Reduce DecodedResponse to minimal override surface Strip overrides and tests that aren't required for transparent HttpResponse behavior, leaving exactly the compiler-required abstract methods plus one discretionary override that fixes a proven base-class bug. DecodedResponse.java: - Remove close() override. Current consumers fully drain the body Flux, so Reactor's onComplete signals release the underlying transport; the explicit close-forwarding was defensive against a try-with-resources / status-only pattern that no caller in the codebase actually uses. - Collapse no-arg getBodyAsString() to delegate to the charset overload, pinning UTF-8 in one place instead of duplicating the lambda. - Final override surface (8 methods): the 7 abstract methods on HttpResponse in azure-core 1.57.1 (compiler-required) plus getBodyAsBinaryData, which is concrete in the base but seeds BinaryData with the wire Content-Length header, making BinaryData.getLength() return the encoded size (frames + CRC trailers) instead of the decoded payload size. DecodedResponseTests.java: - Remove closeDelegatesToWrappedResponse and closeDoesNotSubscribeToDecodedBody (no override left to validate). - Drop the close-counting assertion from the writeBodyTo test and rename it to inheritedWriteBodyToWritesDecodedBytes; it now uses MockHttpResponse directly. - Strengthen getBodyAsBinaryDataReportsDecodedSizeNotContentLength: assert data.getLength() equals the decoded size (post-consumption). Without the override this returns 71 instead of 7, empirically reproduced. - Drop now-unused trackingResponse helper, AtomicInteger import. Tests: 10 of 10 passing. Each test maps 1:1 to one override or one inherited integration; no test is redundant. * some code cleanup * Address Copilot PR #49147 review comments DecodedResponse.java (comment r3220562536): - Restore HttpResponse#getBodyAsString() base contract by switching from unconditional UTF-8 to CoreUtils.bomAwareToString(bytes, contentType), matching BufferedHttpResponse's idiom. The previous UTF-8 pin silently dropped BOM detection and Content-Type charset honoring, which callers viewing this as a generic HttpResponse expect. DecodedResponseTests.java (comment r3220562553): - Add inheritedGetBodyAsBinaryDataReturnsDecodedBytes: exercises the inherited HttpResponse#getBodyAsBinaryData() and asserts the bytes match the decoded payload. Uses a divergent Content-Length header to make the wire vs decoded distinction explicit and guard against header-forwarding regressions. - Add getBodyAsStringHonorsCharsetFromContentTypeHeader: proves the bom-aware fix actually honors a charset declared in Content-Type (would fail if the previous UTF-8 pinning were still in place). - Add getBodyAsStringDetectsUtf8BomAndStripsIt: pins the BOM-detection arm of CoreUtils.bomAwareToString. - Update getBodyAsStringDefaultsToUtf8WhenNoCharsetSpecified docstring to describe the new bom-aware fallback semantics rather than the old unconditional UTF-8 behavior. Tests: 12 of 12 passing. * some code cleanup * analyze error --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage Content Validation - Assert Request and Response Headers for Decoder Tests (#49075) * add checks for response headers * fix linting * refactor async, change messageDownloadRequestHeaders signatures * fix methods with wrong method names * refactor sync * remove unnecessary helper in BlobTestBase * reuse hasStructuredMessageDownloadRequestHeaders with List<HttpHeaders> * remove unused imports * apply copilot suggestions * fix linting * inline local variables * remove try-finally, replace with @tempdir * add checks for response headers * add recording for downloadStreamWithResponseContentValidationRange * readd recording * rerecord recordings * Storage Content Validation - Verify Download Progress Listener Support (#49157) * tests and blob client changes * addressing copilot comments * updating assets * Storage Content Validation - Fuzzy Upload Tests (#49031) * tests and assets * addressing copilot comments * making things actually fuzzy * cleanup * cleanup * addressing copilot comments * recordings * resolving conflicts * updating assets * Storage Content Validation - adding fuzzy tests for content validation decoder (#49058) * adding fuzzy tests for content validation decoder * adding recordings * adding few fuzzy tests * randomizing fuzzy test payloads * randomizing fuzzy test payloads * fixing code review comments * fix spotless formatting violation in BlobContentValidationAsyncDownloadTests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Storage - Content Validation Decoder Dynamic Segment Size Tests and Large GET Behavior Notes (#49172) * tests for dynamic segment size encoding and decoding * cleanup * cleanup * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * style * cant change pr name while ci is running i guess. no-op change * cleaning up test file --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Storage - Decoder Perf Improvements (#49205) * refactoring * removing unused code * reverting unecessary changes * reverting unecessary changes * reverting unecessary changes * not pre-allocating entire byte array of segment size * renaming variable * adding back unecessary removals * renaming variable * addressing copilot comments * Storage - Content validation content-length override (#49226) * content length override changes * content length override changes * removing stuff * addressing review comments * addressing review comments * addressing review comments * resolving merge conflicts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixing spotbugs * fixing spellchecks --------- Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Storage - Content Validation Decoder Stress Tests (#49255) * adding stress tests for content validation decoder * fix TelemetryHelper, add tru to registerOberservers() * Storage - Fix Flaky Stress Tests (#48359) * removed enableDeterministic * change .delete() to .deleteIfExists() * remove Sinks.EmitFailureHandler.FAIL_FAST from CrcInputStream - read functions had FAIL_FAST which would throw an error when the stream had reached then end and we wanted to read from the stream again. So we removed from both reads. - refactor code so that the exit criteria is a tthe beginning - refactor the emitContentInfo for dry * prevent crashes on reattempted close on stream - changed emitValue to tryEmitValue - remove Sinks.EmitFailureHandler.FAIL_FAST so that multiple closes does not cause an error to be thrown * fix telemetry so that it doesnt swallow errors * roll back two deps because they were causing failures in the containers - opentelemetry-runtime-telemetry-java8 from 2.24.0-alpha -> 2.15.0-alpha - opentelemetry-logback-appender-1.0 from 2.24.0-alpha -> 2.15.0-alpha * rollback azure-client-sdk-parent linting extensions from 1.0.0-beta.2 t0 beta.1 * revert linting extensions to beta2 * remove comments with old code * add logging for errors * remove catches for double close issue and okay status * recursively delete files then delete the directory * change to sync deletes, refactor for easier reading * restructing share clean up so super calls only once * incorporate copilot suggestions * incorporate copilot suggestions * incorporate copilot suggestions * incorporate copilot suggestions * fix all deletes to make sync and wrap in try-catch * fix tests so that super.globalCleanupAsync() is only called once * change telemetry to loggin only returns final state instead of failed retries when ultimately successful * undo versio downgrade for linting-extensions * Fixing spacing in error messages Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * refactor datalake delete all so that it is easier to read * refactor runAsync in ShareScenarioBase so retry failures dont show as failures upon success --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * changes to cvdownload content, lazyload versus eagerloading * update to TelemetryHelper based on previous cherry picked stress tests fixes * Fix storage stress fault injector certificate trust Export the fault injector certificate as PEM and wait for it before importing it into the Java truststore so storage stress tests fail fast instead of hitting PKIX errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * bump opentelemetry version * add cv tests to App.class * add script to delete resource groups * update scenarios matrix with tests for cv * fixing stress tests * stageblocksmall scenario adjustment * wip * avoid depending on raw short baseName being available globally * dep pinning * reversion * adding back * wip * storageseekablebytechannel change * bare minimum changes * removing perf core change * removing redundant test file and non-blob package changes * removing redundant test file and non-blob package changes * renaming files and hardcoding crc64 option * removing unused test * removing unused imports --------- Co-authored-by: browndav <browndav@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Isabelle <ibrandes@microsoft.com> * Storage - Content Validation Encoder Stress Tests (#48979) * Add encoder stress tests * enabling fault injector * restoring commitblocklist * renaming files and hardcoding crc64 option * adding scenarios back to app.java --------- Co-authored-by: Rabab Ibrahim <ibrahimr@microsoft.com> Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Isabelle <ibrandes@microsoft.com> Co-authored-by: browndav <browndav@microsoft.com> Co-authored-by: Local Merge <merge@local.tmp> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…dy replayable on retry (#49612) * Make synchronous stageBlock request body replayable on retry * updating assets (unrelated to the changes in this PR) * updating recordings

…to feature/storage/stg104base

This reverts commit b3eaac3.

* adjusting test to use override * adjusting pom

* generate swagger changes * regenerate swagger based on pattern matched swagger doc change * add delete customizaiton back to swagger doc * update sharefile sync and async to be paginated by passing the marker and maxResultsPerPage * finish listRangesDiffWithResponse * readd finalReqeustConsidtions to ShareFileClient * add listRangesPaged and listRangesAccessCOnditions sync * add tests for listRangesAccessConditionFailed and listRangesPagedByPage * add recordings for listRanges tests * create diff tests * add recordinds for list range diff tests * add async tests, refactor helpers to FireShareTestHelper * add async recordings * move null checks into private helper * remove extraneous null checks from private helper * update fail checks for access conditions sync+async, plus recordings * add ShareFileRangeItem and ShareFileRangeOptions for new pageability * remove maxResultsFromPage from SFListRangeOptions * create pageable apis for sharefileclient sync and async for listAllRanges and listAllRangesDiff * restore deprecated apis and recordings * clean up diff in sharefileclient so that old api doesn't show unnecessary changes * deprecate old apis for listRanges* and listRangesDiff* sync and async * create new api tests, add recordings * rename vars to easier understanding, remove unused import * remove unnecessary try-catch blocks * rename function and bifunction vars, reorg sfc and sfac classes * signature changes * cleanup * addressing copilot comments * add recording for listAllRangesDiff --------- Co-authored-by: Isabelle <ibrandes@microsoft.com>

* generate swagger changes * addBlobDownloadHeaders wrapper support, add tests * add recordings * added missing file updates to BlobApiTests * remove added getProperties tests, add check tierChangeTime * remove unused imports * rewrite tests, add recording, add custom check for null for access-tier-inferred --------- Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>

* update swagger file * fix linting * add crc64 support for AppendBlock * add crc64 support for PageBlob * add class header comment, add javadocs to constructorproxy * add support for crc64 for url upload, add tests * add support for crc64 for blockblob * add service versions for tests, add recordings * remove added folder * remove changelog entry * add tests for uploadUrl to BlockBlob and AppendBlob * add recordings for uploadUrl for appendblob and blockblob * change appendblob tests * change blockblobapi tests * change pageblob api tests * add recordings for new tests * add separate upload from url tests for block blob api * add recordings for uploadFromUrlMaxReturnsCrc64 * add missing recordings * remove unneeded service version restrictions in test blockblob api test * formatting --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Isabelle <ibrandes@microsoft.com>

* generate swagger changes * add private props and setters and getters to ShareListFilesAndDirevtoriesoptions * finish implementation * add tests, pending testing on live account * implement accessor pattern in ShareFileItem * add javadocs for SFLADOptions, removed unused imports * create helper to add options to list * add recordings for smb tests * add com.azure.storage.common.implementation.contentvalidation.ContentValidationModeResolver export to module-info.java to fix failing test: OAuthCopySourceTests.copyFromURLWithOAuthSourceInvalidCredential * add recordinds for failing tests * update based on feedback from pr * add nfs recordings * update model helper to make filetype REGULAR if null * hard code DIRECTORY and REGULAR for file types, update tests to remove checks for checks for FileType for SMB * add javadocs for NfsFileType and getFileType * add javadocs for NfsFileType and getFileType * remove unnecessary if-accessor null check * fixing module-info --------- Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com> Co-authored-by: Isabelle <ibrandes@microsoft.com>

#49744) * file share gen * blob generation * test recording update * removing unused imports and adding missing share recordings

* generate new api based on swagger docs * generate code to maintain current xml without arrow * add apache arrow dep 19.0.0 * fix pom dependencies * change pom to bom 18.1.0 * test direct call to arrow endpoint * hard code "preprod." into blob endpoint * fix preprod endpoint * add java.nio=ALL-UNNAMED to blob pom Apache Arrow uses off-heap memory via DirectByteBuffer and needs reflective access to internal NIO classes (like sun.misc.Unsafe or DirectByteBuffer.cleaner()) for memory management. Without this flag, you'll get InaccessibleObjectException at runtime. * addressing illegal dependencies * add ArrowBlobListDeserializer and diagnostic tests for Arrow ListBlobs - Add ArrowBlobListDeserializer that parses Arrow IPC streams into BlobItemInternal objects with safe null-checking helpers - Add diagnostic test to verify Arrow schema from live service - Add deserializer test to validate end-to-end parsing of Arrow response - Add Arrow module requires to module-info.java - Add checker-qual dependency for Arrow annotation compatibility - Add --add-opens java.base/java.nio=ALL-UNNAMED for Arrow memory * wire Arrow path in BlobContainerClient.listBlobs with XML fallback * add Arrow listBlobs integration tests with recordings * update based on new swagger file - Needed to update the README based on the following: adding custom endpoints (the Arrow variants) that deviate from the standard pattern — the swagger defines them as pageable (likely copied from the original list blobs operations), but the response format doesn't match what autorest expects for pageable ops. So the directive is needed to patch the swaggeat codegen time. - This is a common pattern in the Azure SDK — the README.md swagger config files are full of directive blocks that fixup mismatches between what the swagger says and what the generated code actually needs. It's the standard escape hatch when the swagger doesn't perfectly represent the desired codegen behavior. * change method signatures based on updates from swagger * add Arrow hierarchy schema discovery test * wire Arrow path in listBlobsByHierarchy with integration tests * complete Arrow ListBlobs gap remediation: missing fields, NumberOfRecords, and async client Add 6 missing schema field mappings to ArrowBlobListDeserializer (Content-CRC64, Encrypted, OrsPolicySourceBlob, AffinityId, SmartAccessTier, Tags) with supporting model fields and public delegates. Expose NumberOfRecords from Arrow schema metadata. Wire Arrow path into BlobContainerAsyncClient for both flat and hierarchy listings with XML content-type fallback. Add listBlobsArrowWithTags integration test. * add javadocs to new methods in listblobsoptions * changed liveonly tests to recorded tests, minus cpk * remove local blob file and redirect to Nick's repo * rename useArrow to isApacheArrowEnabled * update dependency files * add service minimums for tests * remove tests for schema discovery * remove todo for nonphoton account * add rehydrate tests with recordings * update listblowsarrow pagination test to include calling specific page sizes * add async Arrow listBlobs tests for Photon Mirror the sync Arrow ListBlobs tests from ContainerApiTests.java into ContainerAsyncApiTests.java so CI exercises both code paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix async tests * fix listBLobsArrowDeserializer async test * add recordinds for async tests * replace all references for boolean with StorageResponseSerializationFormat * fix linting issues * remove bom, add versions to pom * change hardcoded serviceVersion in listBlobsArrowDeserializer * swagger - add Accept header for Arrow,Xml * Replace Apache Arrow vector reader with custom ListBlobs Arrow stream reader Swap the official arrow-vector based parser for a self-contained BlobListArrowStreamReader plus internally-implemented column/batch data structures (Batch, Column, StringColumn, IntColumn, BoolColumn, TimestampColumn, MapColumn). The reader decodes the ListBlobs Arrow IPC stream using only the arrow-format flatbuffer definitions. Dependency footprint: arrow-format becomes the sole compile-scoped Arrow dependency; arrow-vector / arrow-memory-core / arrow-memory-unsafe / checker-qual move to test scope (they back the parity test that validates the custom reader against the official Arrow writer); module-info drops requires for org.apache.arrow.vector and org.apache.arrow.memory.core, keeping org.apache.arrow.format. * Revert "Storage - STG104 Add Blob Access Tier to Get Blob Response (#49219)" This reverts commit ce83bc7. * wip - clean up var names * remove getContentCrc64, getOrsPolicySourceBlob, getAffinityId * add external dependencies to external_dependencies.txt, add google buffers to azure-client-sdk-parent and azure-storage-blob xml files * remove arrow-format for java 8 failures, vendor, create parity tests * add dependency exclusion for tests that require arrow deps that cause java 8 deprecation warning * update dependency exclusion so that it works * bump arrow deps to 19.0 for tests * remove arrow deps from banndedDeps includes block * update comments in blob pom.xml to explicitly state workaround for base java version * remove unnecessary arrow parity tests, add remaining parity tests to excludes for java 8 in pom * lint files in arrow package * add arrow golden decode tests and rejection tests, add test for all known columns * change arrow classes to be more human readable * change DecodedArrowStream to record * Revert "change DecodedArrowStream to record" This reverts commit 8d4bd47. * add changes based on feedback * update recording * rename arrow to apachearrow * add apache licensing * add apache share profile to pom * add versions to pom shade profiling, add "Accept" header to match only available * add arrow 19 dep to external_dependencies.txt file * fix linting issues * fix more linting errors with spotless * remove unused imports * revert add "Accept" to blobbasetest, add recordings --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…49770) * Revert "Storage - STG104 NFS List V2 Files and Directories (#49246)" This reverts commit b75c5ff. * add missing deps in ShareDirectory sync and async clients * remove generated code from NfsFileType

* refactoring by removing data from options bags for append block and upload pages * deprecating flattened client methods * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * adjusting javadoc * renaming * adding recordings * test cleanup to resolve transient errors in playback * reducing the size of payload used in some content validation tests --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Reverts the addition of the 2026-12-06 (STG104) service version so that 2026-10-06 (STG103) remains the latest supported service version across Blob, DataLake, Share, and Queue.

ibrandes · 2026-07-22T15:52:25Z

/azp run java - storage - tests

azure-pipelines · 2026-07-22T15:52:45Z