[Storage][Blob] Test Arrow 15.0.2 for Java 8 compatibility#49616
Closed
browndav-msft wants to merge 47 commits into
Closed
[Storage][Blob] Test Arrow 15.0.2 for Java 8 compatibility#49616browndav-msft wants to merge 47 commits into
browndav-msft wants to merge 47 commits into
Conversation
Apache Arrow uses off-heap memory via DirectByteBuffer and needs reflective access to internal NIO classes (like sun.misc.Unsafe or DirectByteBuffer.cleaner()) for memory management. Without this flag, you'll get InaccessibleObjectException at runtime.
- Add ArrowBlobListDeserializer that parses Arrow IPC streams into BlobItemInternal objects with safe null-checking helpers - Add diagnostic test to verify Arrow schema from live service - Add deserializer test to validate end-to-end parsing of Arrow response - Add Arrow module requires to module-info.java - Add checker-qual dependency for Arrow annotation compatibility - Add --add-opens java.base/java.nio=ALL-UNNAMED for Arrow memory
- Needed to update the README based on the following: adding custom endpoints (the Arrow variants) that deviate from the standard pattern — the swagger defines them as pageable (likely copied from the original list blobs operations), but the response format doesn't match what autorest expects for pageable ops. So the directive is needed to patch the swaggeat codegen time. - This is a common pattern in the Azure SDK — the README.md swagger config files are full of directive blocks that fixup mismatches between what the swagger says and what the generated code actually needs. It's the standard escape hatch when the swagger doesn't perfectly represent the desired codegen behavior.
…ords, and async client Add 6 missing schema field mappings to ArrowBlobListDeserializer (Content-CRC64, Encrypted, OrsPolicySourceBlob, AffinityId, SmartAccessTier, Tags) with supporting model fields and public delegates. Expose NumberOfRecords from Arrow schema metadata. Wire Arrow path into BlobContainerAsyncClient for both flat and hierarchy listings with XML content-type fallback. Add listBlobsArrowWithTags integration test.
Mirror the sync Arrow ListBlobs tests from ContainerApiTests.java into ContainerAsyncApiTests.java so CI exercises both code paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reader Swap the official arrow-vector based parser for a self-contained BlobListArrowStreamReader plus internally-implemented column/batch data structures (Batch, Column, StringColumn, IntColumn, BoolColumn, TimestampColumn, MapColumn). The reader decodes the ListBlobs Arrow IPC stream using only the arrow-format flatbuffer definitions. Dependency footprint: arrow-format becomes the sole compile-scoped Arrow dependency; arrow-vector / arrow-memory-core / arrow-memory-unsafe / checker-qual move to test scope (they back the parity test that validates the custom reader against the official Arrow writer); module-info drops requires for org.apache.arrow.vector and org.apache.arrow.memory.core, keeping org.apache.arrow.format.
…zure#49219)" This reverts commit ce83bc7.
…ffers to azure-client-sdk-parent and azure-storage-blob xml files
github-actions
Bot
added
Azure.Core
browndav-msft
force-pushed
the
browndav-arrow-15-java8-test
branch
from
d702604 to
7972050
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
browndav-msft
force-pushed
the
browndav-arrow-15-java8-test
branch
from
6bafef1 to
1b5f899
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n- Downgrade Arrow dependencies used by azure-storage-blob from 18.1.0 to 15.0.2\n- Update versioning metadata and parent allowlist entries accordingly\n\n## Why\n- Validate whether Arrow 15.0.2 is a viable Java 8-compatible option\n\n## Validation intent\n- Let CI run Java 8 compile/install and blob test matrix for this version set\n\n## Notes\n- This PR is for compatibility validation.