Skip to content

Add all streaming operation to s3 checksums tests #5998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
4ff551c
add test configs for all streaming type
L-Applin Mar 25, 2025
c80568e
buildspec fix
L-Applin Mar 25, 2025
53b55b3
buildspec fix
L-Applin Mar 25, 2025
d892595
check if tests hand without multipart
L-Applin Mar 26, 2025
bd21f3a
add http-auth-aws-crt for crt sigv4a tests
L-Applin Mar 26, 2025
bbb5d1e
fix request body methods with 'remaining'
L-Applin Mar 26, 2025
c889f6e
test 2MiB instead of 200MiB
L-Applin Mar 26, 2025
89c5edc
fix for 'remaining' test type
L-Applin Mar 26, 2025
3737d0a
fix for fromString, reduce logs
L-Applin Mar 26, 2025
633e916
fix for string content length
L-Applin Mar 26, 2025
203b0ed
fix for string content length
L-Applin Mar 27, 2025
babbbdd
test 200MiB large size
L-Applin Mar 27, 2025
422c858
use 80MiB for large files
L-Applin Mar 27, 2025
25e3666
Fix for TM_JAVA
L-Applin Mar 27, 2025
67842a3
Fix for TM_JAVA
L-Applin Mar 27, 2025
764629a
Fix for TM_JAVA
L-Applin Mar 27, 2025
4c13ffb
Merge remote-tracking branch 'origin/olapplin/s3-checksums-tests' int…
L-Applin Mar 27, 2025
2f01bc1
need more memory
L-Applin Mar 27, 2025
3ba77d2
more fix for TM
L-Applin Mar 27, 2025
c07dab3
some more dl stuff
L-Applin Mar 31, 2025
14c46de
logs
L-Applin Mar 31, 2025
efb4dc8
fix multi
L-Applin Mar 31, 2025
51950db
fix multi bucket
L-Applin Mar 27, 2025
692fbe0
fix multi bucket
L-Applin Mar 31, 2025
3af2ee0
part index starts at 1
L-Applin Mar 31, 2025
29136b4
complete multipart fix
L-Applin Mar 31, 2025
994bbc5
complete multipart fix
L-Applin Mar 31, 2025
7d2f2fb
complete multipart fix
L-Applin Mar 31, 2025
bf8dd35
complete multipart fix
L-Applin Mar 31, 2025
0e70032
Merge branch 'master' into olapplin/s3-checksums-tests
L-Applin Mar 31, 2025
50324d1
fix for parts
L-Applin Apr 1, 2025
2f70826
adding logs
L-Applin Apr 1, 2025
cde3548
more logs, no crc
L-Applin Apr 1, 2025
59c705b
improved message for failed assertion
L-Applin Apr 1, 2025
eba7670
fix crc check for multi and cleanup
L-Applin Apr 1, 2025
9ec5867
fix memory issues
L-Applin Apr 1, 2025
e687a77
Merge branch 'master' into olapplin/s3-checksums-tests
L-Applin Apr 1, 2025
29c32ed
add s3 checksum validation
L-Applin Apr 1, 2025
5c06fe3
avoid using AwsChunkedV4aPayloadSigner
L-Applin Apr 1, 2025
92ea909
avoid using AwsChunkedV4aPayloadSigner
L-Applin Apr 2, 2025
b0b984e
DownloadConfig toString
L-Applin Apr 2, 2025
10c337e
fix Unmanaged
L-Applin Apr 2, 2025
78e6b5b
fix for crc32
L-Applin Apr 3, 2025
0084b1f
rename tp regression testing
L-Applin Apr 3, 2025
880c1c8
fix logs
L-Applin Apr 3, 2025
d99d6c1
added README, verify crc32 of uploaded object with an additional get …
L-Applin Apr 4, 2025
ea4b87f
fix npe
L-Applin Apr 4, 2025
3bc6cdf
junit parallel execution
L-Applin Apr 4, 2025
11b0bcb
multipart with no-length
L-Applin Apr 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions buildspecs/s3-regression-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
version: 0.2

phases:
build:
commands:
- mvn clean install -P s3-regression-tests -pl :s3-tests -am -T1C $MAVEN_OPTIONS
- echo $MAVEN_OPTIONS
finally:
- mkdir -p codebuild-test-reports
- find ./ -name 'TEST-*.xml' -type f -exec cp {} codebuild-test-reports/ \;
reports:
ChecksumsTests:
files:
- 'codebuild-test-reports/**/*'
Original file line number Diff line number Diff line change
Expand Up @@ -393,8 +393,8 @@ static AsyncRequestBody fromInputStream(Consumer<AsyncRequestBodyFromInputStream
* S3AsyncClient s3 = S3AsyncClient.create(); // Use one client for your whole application!
*
* byte[] dataToSend = "Hello".getBytes(StandardCharsets.UTF_8);
* InputStream streamToSend = new ByteArrayInputStream();
* long streamToSendLength = dataToSend.length();
* InputStream streamToSend = new ByteArrayInputStream(dataToSend);
* long streamToSendLength = dataToSend.length;
*
* // Start the operation
* BlockingInputStreamAsyncRequestBody body =
Expand Down
46 changes: 46 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -876,6 +876,52 @@
</build>
</profile>

<profile>
<id>s3-regression-tests</id>
<activation>
<property>
<name>doRelease</name>
</property>
</activation>

<properties>
<checkstyle.skip>true</checkstyle.skip>
<spotbugs.skip>true</spotbugs.skip>
<skip.unit.tests>true</skip.unit.tests>
<mdep.analyze.skip>true</mdep.analyze.skip>
<japicmp.skip>true</japicmp.skip>
<maven.javadoc.skip>true</maven.javadoc.skip>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-failsafe-plugin.version}</version>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<!-- Sets the VM argument line used when integration tests are run. -->
<!-- this test needs a lot of memory, we need to increase java heap size or we get OOM in codebuild -->
<argLine>-Xmx12g -Xms4g -XX:+AllowRedefinitionToAddDeleteMethods</argLine>
<includes>
<include>**/*RegressionTesting.java</include>
</includes>
<trimStackTrace>false</trimStackTrace>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</profile>

<profile>
<id>endpoint-tests</id>
<activation>
Expand Down
31 changes: 31 additions & 0 deletions test/s3-tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# SDK Regression Tests for Amazon S3

## Description
This module contains SDK regression tests for Amazon S3 streaming operations with various SDK configurations.


## How to run

### Credentials

The tests require valid AWS credentials to be available in the default credential file under the `aws-test-account` profile.

### Run the tests

- Run from your IDE

- Run from maven command line

```
mvn clean install -P s3-regression-tests -pl :stability-tests
```

## Adding New Tests

- The tests are built using [JUnit 5](https://junit.org/junit5/). Make sure you are using the correct APIs and mixing of
Junit 4 and Junit 5 APIs on the same test can have unexpected results.

- All tests should have the suffix of `RegressionTesting`, eg: ``



17 changes: 17 additions & 0 deletions test/s3-tests/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,23 @@
<version>${awsjavasdk.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>http-auth-aws-crt</artifactId>
<version>${awsjavasdk.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
<version>${awsjavasdk.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>test-utils</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
Loading
Loading