Skip to content

Commit 35bbb94

Browse files
browndav-msftibrandesCopilot
authored
Storage - STG 104 Photon (#49365)
* generate new api based on swagger docs * generate code to maintain current xml without arrow * add apache arrow dep 19.0.0 * fix pom dependencies * change pom to bom 18.1.0 * test direct call to arrow endpoint * hard code "preprod." into blob endpoint * fix preprod endpoint * add java.nio=ALL-UNNAMED to blob pom Apache Arrow uses off-heap memory via DirectByteBuffer and needs reflective access to internal NIO classes (like sun.misc.Unsafe or DirectByteBuffer.cleaner()) for memory management. Without this flag, you'll get InaccessibleObjectException at runtime. * addressing illegal dependencies * add ArrowBlobListDeserializer and diagnostic tests for Arrow ListBlobs - Add ArrowBlobListDeserializer that parses Arrow IPC streams into BlobItemInternal objects with safe null-checking helpers - Add diagnostic test to verify Arrow schema from live service - Add deserializer test to validate end-to-end parsing of Arrow response - Add Arrow module requires to module-info.java - Add checker-qual dependency for Arrow annotation compatibility - Add --add-opens java.base/java.nio=ALL-UNNAMED for Arrow memory * wire Arrow path in BlobContainerClient.listBlobs with XML fallback * add Arrow listBlobs integration tests with recordings * update based on new swagger file - Needed to update the README based on the following: adding custom endpoints (the Arrow variants) that deviate from the standard pattern — the swagger defines them as pageable (likely copied from the original list blobs operations), but the response format doesn't match what autorest expects for pageable ops. So the directive is needed to patch the swaggeat codegen time. - This is a common pattern in the Azure SDK — the README.md swagger config files are full of directive blocks that fixup mismatches between what the swagger says and what the generated code actually needs. It's the standard escape hatch when the swagger doesn't perfectly represent the desired codegen behavior. * change method signatures based on updates from swagger * add Arrow hierarchy schema discovery test * wire Arrow path in listBlobsByHierarchy with integration tests * complete Arrow ListBlobs gap remediation: missing fields, NumberOfRecords, and async client Add 6 missing schema field mappings to ArrowBlobListDeserializer (Content-CRC64, Encrypted, OrsPolicySourceBlob, AffinityId, SmartAccessTier, Tags) with supporting model fields and public delegates. Expose NumberOfRecords from Arrow schema metadata. Wire Arrow path into BlobContainerAsyncClient for both flat and hierarchy listings with XML content-type fallback. Add listBlobsArrowWithTags integration test. * add javadocs to new methods in listblobsoptions * changed liveonly tests to recorded tests, minus cpk * remove local blob file and redirect to Nick's repo * rename useArrow to isApacheArrowEnabled * update dependency files * add service minimums for tests * remove tests for schema discovery * remove todo for nonphoton account * add rehydrate tests with recordings * update listblowsarrow pagination test to include calling specific page sizes * add async Arrow listBlobs tests for Photon Mirror the sync Arrow ListBlobs tests from ContainerApiTests.java into ContainerAsyncApiTests.java so CI exercises both code paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix async tests * fix listBLobsArrowDeserializer async test * add recordinds for async tests * replace all references for boolean with StorageResponseSerializationFormat * fix linting issues * remove bom, add versions to pom * change hardcoded serviceVersion in listBlobsArrowDeserializer * swagger - add Accept header for Arrow,Xml * Replace Apache Arrow vector reader with custom ListBlobs Arrow stream reader Swap the official arrow-vector based parser for a self-contained BlobListArrowStreamReader plus internally-implemented column/batch data structures (Batch, Column, StringColumn, IntColumn, BoolColumn, TimestampColumn, MapColumn). The reader decodes the ListBlobs Arrow IPC stream using only the arrow-format flatbuffer definitions. Dependency footprint: arrow-format becomes the sole compile-scoped Arrow dependency; arrow-vector / arrow-memory-core / arrow-memory-unsafe / checker-qual move to test scope (they back the parity test that validates the custom reader against the official Arrow writer); module-info drops requires for org.apache.arrow.vector and org.apache.arrow.memory.core, keeping org.apache.arrow.format. * Revert "Storage - STG104 Add Blob Access Tier to Get Blob Response (#49219)" This reverts commit ce83bc7. * wip - clean up var names * remove getContentCrc64, getOrsPolicySourceBlob, getAffinityId * add external dependencies to external_dependencies.txt, add google buffers to azure-client-sdk-parent and azure-storage-blob xml files * remove arrow-format for java 8 failures, vendor, create parity tests * add dependency exclusion for tests that require arrow deps that cause java 8 deprecation warning * update dependency exclusion so that it works * bump arrow deps to 19.0 for tests * remove arrow deps from banndedDeps includes block * update comments in blob pom.xml to explicitly state workaround for base java version * remove unnecessary arrow parity tests, add remaining parity tests to excludes for java 8 in pom * lint files in arrow package * add arrow golden decode tests and rejection tests, add test for all known columns * change arrow classes to be more human readable * change DecodedArrowStream to record * Revert "change DecodedArrowStream to record" This reverts commit 8d4bd47. * add changes based on feedback * update recording * rename arrow to apachearrow * add apache licensing * add apache share profile to pom * add versions to pom shade profiling, add "Accept" header to match only available * add arrow 19 dep to external_dependencies.txt file * fix linting issues * fix more linting errors with spotless * remove unused imports * revert add "Accept" to blobbasetest, add recordings --------- Co-authored-by: Isabelle <ibrandes@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent cfa35fe commit 35bbb94

48 files changed

Lines changed: 3647 additions & 333 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎NOTICE.txt‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -500,6 +500,23 @@ See the License for the specific language governing permissions and limitations
500500

501501
-------------------------------------------------------------------------------------------------
502502

503+
License notice for Apache Arrow
504+
-------------------------------------------------------------------------------
505+
506+
apache-arrow-java (https://github.com/apache/arrow-java)
507+
Copyright 2015-2024 The Apache Software Foundation
508+
509+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
510+
in compliance with the License. You may obtain a copy of the License at
511+
512+
http://www.apache.org/licenses/LICENSE-2.0
513+
514+
Unless required by applicable law or agreed to in writing, software distributed under the License
515+
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
516+
See the License for the specific language governing permissions and limitations under the License.
517+
518+
-------------------------------------------------------------------------------------------------
519+
503520
License notice for GraalVM
504521
------------------------------------------------------------------------------
505522
org.graalvm.sdk:graal-sdk - https://github.com/graalvm/native-build-tools/blob/master/common/junit-platform-native/LICENSE
@@ -609,4 +626,4 @@ compliance with the License. You may obtain a copy of the License at:
609626

610627
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
611628
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
612-
language governing permissions and limitations under the License.
629+
language governing permissions and limitations under the License.

‎eng/versioning/external_dependencies.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ com.github.spotbugs:spotbugs;4.8.3
2525
com.github.spotbugs:spotbugs-annotations;4.8.3
2626
com.github.spotbugs:spotbugs-maven-plugin;4.8.3.1
2727
com.google.code.gson:gson;2.14.0
28+
com.google.flatbuffers:flatbuffers-java;24.3.25
2829
com.google.guava:guava;33.6.0-jre
2930
com.h2database:h2;2.2.220
3031
com.h3xstream.findsecbugs:findsecbugs-plugin;1.9.0
@@ -63,6 +64,7 @@ io.vertx:vertx-codegen;4.5.27
6364
io.vertx:vertx-core;4.5.27
6465
javax.websocket:javax.websocket-api;1.1
6566
org.apache.ant:ant;1.10.15
67+
org.apache.arrow:arrow-format;19.0.0
6668
org.apache.avro:avro;1.11.4
6769
org.apache.avro:avro-maven-plugin;1.11.4
6870
org.apache.commons:commons-lang3;3.18.0

‎sdk/parents/azure-client-sdk-parent/pom.xml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -602,6 +602,10 @@
602602

603603
<!-- Used by many libraries to bring in annotations used by Reactor -->
604604
<include>com.google.code.findbugs:jsr305:[3.0.2]</include> <!-- {x-include-update;com.google.code.findbugs:jsr305;external_dependency} -->
605+
606+
<!-- Apache Arrow format parsing. flatbuffers-java is the sole (compile-scoped) dependency used by
607+
azure-storage-blob's vendored Arrow FlatBuffer accessors -->
608+
<include>com.google.flatbuffers:flatbuffers-java:[24.3.25]</include> <!-- {x-include-update;com.google.flatbuffers:flatbuffers-java;external_dependency} -->
605609
</includes>
606610
</bannedDependencies>
607611

‎sdk/storage/azure-storage-blob/assets.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
"AssetsRepo": "Azure/azure-sdk-assets",
33
"AssetsRepoPrefixPath": "java",
44
"TagPrefix": "java/storage/azure-storage-blob",
5-
"Tag": "java/storage/azure-storage-blob_802f7f8947"
5+
"Tag": "java/storage/azure-storage-blob_13b4ad4e16"
66
}

‎sdk/storage/azure-storage-blob/pom.xml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,10 @@
5959
</javaModulesSurefireArgLine>
6060
<parallelizeLiveTests>concurrent</parallelizeLiveTests>
6161

62+
<!-- Exclude vendored Apache Arrow code from linting checks -->
63+
<checkstyle.excludes>**/implementation/util/apachearrow/**/*.java</checkstyle.excludes>
64+
<spotbugs.excludes>**/implementation/util/apachearrow/**/*.java</spotbugs.excludes>
65+
6266
<spotbugs.skip>false</spotbugs.skip>
6367
</properties>
6468

@@ -138,6 +142,20 @@
138142
<version>1.17.7</version> <!-- {x-version-update;testdep_net.bytebuddy:byte-buddy-agent;external_dependency} -->
139143
<scope>test</scope>
140144
</dependency>
145+
146+
<!-- Arrow dependency -->
147+
<!-- The internal ListBlobs Arrow reader decodes the IPC stream using vendored FlatBuffer metadata accessors
148+
(com.azure.storage.blob.implementation.util.apachearrow) backed by flatbuffers-java, so flatbuffers-java is the
149+
sole dependency needed to parse the Arrow format. The arrow-format artifact is deliberately NOT a dependency:
150+
it ships Java 11 bytecode and would break the Java 8 baseline build. The reader's behavior is validated against
151+
committed golden base64 fixtures (src/test/resources/arrow) rather than the official arrow-vector writer, so no
152+
Apache Arrow test dependency is required. -->
153+
<dependency>
154+
<groupId>com.google.flatbuffers</groupId>
155+
<artifactId>flatbuffers-java</artifactId>
156+
<version>24.3.25</version> <!-- {x-version-update;com.google.flatbuffers:flatbuffers-java;external_dependency} -->
157+
</dependency>
158+
141159
</dependencies>
142160

143161
<profiles>
@@ -182,5 +200,84 @@
182200
</plugins>
183201
</build>
184202
</profile>
203+
204+
<!-- This profile is used to generate a sources JAR with arrow-format shaded into com.azure.storage.blob.implementation.util.apachearrow. -->
205+
<!-- DO NOT run this profile with release, it is only meant to be run locally when the shaded Apache Arrow code needs to be updated. -->
206+
<!-- -->
207+
<!-- To update the shaded Apache Arrow code, run Maven build with '-Dshade-apache-arrow-format' included in the command. -->
208+
<!-- In the target folder, extract the sources JAR. The path com/azure/storage/blob/implementation/util/apachearrow will contain the relocated Apache Arrow source code. -->
209+
<!-- Copy this code into the com.azure.storage.blob.implementation.util.apachearrow package in azure-storage-blob and overwrite the existing files. -->
210+
<profile>
211+
<id>shade-apache-arrow-format</id>
212+
<activation>
213+
<property>
214+
<name>shade-apache-arrow-format</name>
215+
</property>
216+
</activation>
217+
218+
<dependencies>
219+
<dependency>
220+
<groupId>org.apache.arrow</groupId>
221+
<artifactId>arrow-format</artifactId>
222+
<version>19.0.0</version> <!-- {x-version-update;org.apache.arrow:arrow-format;external_dependency} -->
223+
</dependency>
224+
</dependencies>
225+
226+
<build>
227+
<plugins>
228+
<plugin>
229+
<groupId>org.apache.maven.plugins</groupId>
230+
<artifactId>maven-shade-plugin</artifactId>
231+
<version>3.6.0</version> <!-- {x-version-update;org.apache.maven.plugins:maven-shade-plugin;external_dependency} -->
232+
233+
<executions>
234+
<execution>
235+
<id>shade</id>
236+
<phase>package</phase>
237+
<goals>
238+
<goal>shade</goal>
239+
</goals>
240+
</execution>
241+
</executions>
242+
243+
<configuration>
244+
<createSourcesJar>true</createSourcesJar>
245+
<shadeSourcesContent>true</shadeSourcesContent>
246+
<shadedArtifactAttached>true</shadedArtifactAttached>
247+
<shadedClassifierName>shade-apache-arrow-format</shadedClassifierName>
248+
<createDependencyReducedPom>false</createDependencyReducedPom>
249+
250+
<artifactSet>
251+
<includes>
252+
<include>org.apache.arrow:arrow-format</include> <!-- {x-include-update;org.apache.arrow:arrow-format;external_dependency} -->
253+
</includes>
254+
</artifactSet>
255+
256+
<relocations>
257+
<relocation>
258+
<pattern>org.apache.arrow.flatbuf</pattern>
259+
<shadedPattern>com.azure.storage.blob.implementation.util.apachearrow</shadedPattern>
260+
</relocation>
261+
</relocations>
262+
</configuration>
263+
</plugin>
264+
265+
<plugin>
266+
<groupId>org.apache.maven.plugins</groupId>
267+
<artifactId>maven-enforcer-plugin</artifactId>
268+
<version>3.6.1</version> <!-- {x-version-update;org.apache.maven.plugins:maven-enforcer-plugin;external_dependency} -->
269+
<configuration>
270+
<rules>
271+
<bannedDependencies>
272+
<includes>
273+
<include>org.apache.arrow:arrow-format:[19.0.0]</include> <!-- {x-include-update;org.apache.arrow:arrow-format;external_dependency} -->
274+
</includes>
275+
</bannedDependencies>
276+
</rules>
277+
</configuration>
278+
</plugin>
279+
</plugins>
280+
</build>
281+
</profile>
185282
</profiles>
186283
</project>

0 commit comments

Comments
 (0)