Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build fs-storage and publish Java package #94

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6af1614
Build and release `fs-storage` JNI libraries and publish Java library
oluiscabral Oct 24, 2024
571aab6
Create `cache.yml` workflow
oluiscabral Dec 2, 2024
560fca7
WIP - start transforming it into AAR
oluiscabral Dec 10, 2024
8ccb90e
WIP - set up library as AAR
oluiscabral Dec 11, 2024
0a76e96
Finish AAR setup
oluiscabral Dec 12, 2024
475a94e
Set up NDK to GitHub Actions
oluiscabral Dec 12, 2024
f5f4420
Fix `Set up NDK` step
oluiscabral Dec 12, 2024
85a00ca
Set up Android SDK
oluiscabral Dec 12, 2024
57a28c1
Set `ANDROID_NDK_HOME` env var
oluiscabral Dec 12, 2024
06db31b
Set specific ndk version
oluiscabral Dec 12, 2024
6730418
Fix NDK version
oluiscabral Dec 12, 2024
68f478f
Bring back Android SDK setup
oluiscabral Dec 12, 2024
c80086b
Final touches on AAR build and release workflow
oluiscabral Dec 12, 2024
2e77dfe
Fix: speedup `build.yml` and add targets to `release.yml`
oluiscabral Dec 12, 2024
3c572a6
Define `GITHUB_ACTOR` as well
oluiscabral Dec 12, 2024
5cec424
Wrap GitHubPackages URI in `uri(string)`
oluiscabral Dec 12, 2024
b6c4d2c
Copy old `publications`
oluiscabral Dec 12, 2024
5b1ea61
Update permissions and publications
oluiscabral Dec 12, 2024
229b48f
Use tag name as version and clean publication
oluiscabral Dec 12, 2024
78d058d
Remove local configuration
oluiscabral Dec 14, 2024
7e2bd8b
add `rustfmt` formatting check
oluiscabral Dec 14, 2024
54d9665
Try using regex as released tag evaluator
oluiscabral Dec 14, 2024
736a206
Try new release regex condition
oluiscabral Dec 14, 2024
ceef135
Fix: Properly remove first character of released tag
oluiscabral Dec 15, 2024
9cd65f3
Fix: Try using `GITHUB_REF_NAME` to remove first char of released tag
oluiscabral Dec 15, 2024
0c1fa09
Fix: export `RELEASE_VERSION` instead of setting it as `env` in yml
oluiscabral Dec 15, 2024
61cd3f5
Update Android NDK and `java/README.md` instructions
oluiscabral Dec 16, 2024
14b2fff
CI Workflow Refactor (#96)
tareknaser Jan 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 45 additions & 17 deletions .github/workflows/build.yml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

 getting this warning locally, and I can also see it in the CI:

> Task :lib:compileReleaseJavaWithJavac
Java compiler version 22 has deprecated support for compiling with source/target version 8.
warning: [options] source value 8 is obsolete and will be removed in a future release
Try one of the following options:
    1. [Recommended] Use Java toolchain with a lower language version
    2. Set a higher source/target version
    3. Use a lower version of the JDK running the build (if you're not using Java toolchain)
For more details on how to configure these settings, see https://developer.android.com/build/jdks.
To suppress this warning, set android.javaCompile.suppressSourceTargetDeprecationWarning=true in gradle.properties.
warning: [options] target value 8 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
3 warnings
warning: [options] source value 8 is obsolete and will be removed in a future release

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I see, I'll investigate it

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aaahh, that's why would be great to move all the Java code to ark-android.. especially after we switched from JAR to AAR, so we don't do "plain Java" anymore and really only use Java for Android..

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Maybe we could even move the JNI Rust related code to ark-android then, since it is specifically related to Java as well

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oluiscabral exactly! do you think it's reasonable amount of work for the current pair of PRs, or should we merge these first, and proceed with refactoring in new PRs?

Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,11 @@ jobs:

- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
toolchain: nightly # nightly is required for fmt
components: rustfmt, clippy

- name: Check
run: cargo check

- name: Format
run: |
cargo fmt --all -- --check
cargo clippy --workspace --bins -- -D warnings

- name: Build Debug
run: cargo build --verbose
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pushkarm029 @kirillt I removed those steps for now, because the behavior of the Format was not being 100% predictable. Since I started this PR, around a month ago, the Format step changed its results multiple times, even though the Rust code remained exactly the same

As example, you can check that from time to time some old and recent jobs didn't fully complete because formatting errors were randomly being thrown.

Most recent occurrence:

I judge as a bad practice using a nightly or any other "dynamic" version to format a project, since it can introduce errors from newly added formatting features 🤔

Curious to know what you guys think about it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting point. Actually, I prefer setting up cargo fmt everywhere, because it helps to avoid inconsistent code style (and standard Rust code style looks pretty fine). But I've never encountered cargo fmt check rejecting a bit outdated codebase.

My guess is that the code style used by cargo fmt develops including new decisions made by Rust community. And out code is modified too slowly so we don't include these new updates. If this is true, then we could keep cargo fmt check and actually reformat the code every time we encounter such failures on CI, but only manually.

@Pushkarm029 what do you think?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use nightly for formatting because some of the rules we define in rustfmt.toml are only supported in the nightly version. Rust nightly is updated frequently, that’s not a problem — it’s the pre-release version. We have two options:
• We can abandon the rules we’ve defined (I would like to keep them)
• We can check for formatting in a separate job that runs nightly, within the same workflow.
For example:

jobs:
  format:
    name: Code Formatting Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Nightly Rust with rustfmt
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: nightly
          components: rustfmt

      - name: Run rustfmt
        run: cargo fmt --all -- --check

  build_and_test:
    name: Build and Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Stable Rust
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable

      ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually care about formatting only during merging, so we probably can configure GitHub rules and prevent merge if cargo fmt job fails specifically. This would allow see that the build itself works.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oluiscabral I think we don't have cargo clippy --workspace --bins -- -D warnings anymore..

Not sure if it's really any valuable though, does anyone looks into CI warnings? 🤔

cc @tareknaser @Pushkarm029

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think cargo clippy --workspace --bins -- -D warnings is necessary, It helps us catch common mistakes, anti-patterns, and potential improvements that the compiler doesn't check for.

So, I suggest we should add it between the format and build-and-test job.

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: false

- name: Run tests
run: cargo test --verbose
Expand All @@ -54,6 +45,15 @@ jobs:
- name: Set up Gradle
uses: gradle/actions/setup-gradle@v3

- name: Set up Android SDK
uses: android-actions/setup-android@v3

- name: Set up Android NDK
uses: nttld/setup-ndk@v1
with:
link-to-sdk: true
ndk-version: r28-beta1

- name: Java tests
run: gradle test
working-directory: ./java
Expand All @@ -68,12 +68,17 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Build Release
run: cargo build --verbose --release
- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: false

- name: Run tests
run: cargo test --workspace --verbose

- name: Build Release
run: cargo build --verbose --release

- name: Run `ark-cli watch` test
run: ./integration/ark-cli-watch.sh

Expand All @@ -86,6 +91,15 @@ jobs:
- name: Set up Gradle
uses: gradle/actions/setup-gradle@v3

- name: Set up Android SDK
uses: android-actions/setup-android@v3

- name: Set up Android NDK
uses: nttld/setup-ndk@v1
with:
link-to-sdk: true
ndk-version: r28-beta1

- name: Java tests
run: gradle test
working-directory: ./java
Expand All @@ -100,12 +114,17 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Build Release
run: cargo build --verbose --release
- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: false

- name: Run tests
run: cargo test --workspace --verbose

- name: Build Release
run: cargo build --verbose --release

- name: Run `ark-cli watch` test
run: ./integration/ark-cli-watch.sh

Expand All @@ -118,6 +137,15 @@ jobs:
- name: Set up Gradle
uses: gradle/actions/setup-gradle@v3

- name: Set up Android SDK
uses: android-actions/setup-android@v3

- name: Set up Android NDK
uses: nttld/setup-ndk@v1
with:
link-to-sdk: true
ndk-version: r28-beta1

- name: Java tests
run: gradle test
working-directory: ./java
82 changes: 82 additions & 0 deletions .github/workflows/cache.yml
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pushkarm029 feel free to review it 😀

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we really need a separate workflow just for the cache (.github/workflows/cache.yml). I see that it only runs on push/PR to the main branch. Can we modify the save condition in the main workflow to:

save-if: ${{ github.ref == 'refs/heads/main' }}

and remove the redundant workflow?

Copy link
Member

@kirillt kirillt Dec 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether we keep this workflow or not, I don’t think it’s worth running it on macOS ARM (macos-13-xlarge) with every push to main. We only use the macOS ARM workflow in the weekly build since it’s a paid runner.
Or, we could increase the retention period for this specific GitHub Actions artifact.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and remove the redundant workflow?

Agreed

Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: Cache build

on:
push:
branches:
- main
pull_request:
branches:
- main

jobs:
linux:
name: Linux
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

windows:
name: Windows
runs-on: windows-latest

steps:
- uses: actions/checkout@v4

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

mac-intel:
name: MacOS Intel
runs-on: macos-14

steps:
- uses: actions/checkout@v4

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

mac-arm:
name: MacOS ARM
runs-on: macos-13-xlarge

steps:
- uses: actions/checkout@v4

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release
57 changes: 57 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Release

on:
push:
tags:
- "*"

jobs:
release:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4

- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
targets: aarch64-linux-android armv7-linux-androideabi i686-linux-android x86_64-linux-android

Comment on lines +20 to +21
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add these targets to the instructions in java/README.md, specifically the rustup command?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, do you mean to add a step-by-step on how to locally cross-compile the library?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I tried running this locally, it failed because I didn’t have these targets set up, and I had to debug it myself. It would be helpful to include a command for adding these targets as a perquisite, something like:

rustup target add \
    aarch64-linux-android \
    armv7-linux-androideabi \
    ...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, let’s include a note about the supported ndkVersion, because running this with a different version also causes it to completely fail locally for me (I had version 28.0.12674087. This is why I suggested #94 (comment))

with an annoying error as well (I think it said something like ndk not installed)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tareknaser this one seems to be resolved, right?

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: false

- name: Run tests
run: cargo test --workspace --verbose

- name: Install JDK
uses: actions/[email protected]
with:
distribution: "temurin"
java-version: "22"

- name: Set up Gradle
uses: gradle/actions/setup-gradle@v3

- name: Set up Android SDK
uses: android-actions/setup-android@v3

- name: Set up Android NDK
uses: nttld/setup-ndk@v1
with:
link-to-sdk: true
ndk-version: r28-beta1

- name: Java tests
run: gradle test
working-directory: ./java

- name: Publish Java release
run: gradle publish
working-directory: ./java
env:
RELEASE_VERSION: ${{ github.ref_name }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
20 changes: 20 additions & 0 deletions .github/workflows/weekly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

Expand All @@ -32,6 +37,11 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

Expand All @@ -45,6 +55,11 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release

Expand All @@ -58,5 +73,10 @@ jobs:
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Cargo Cache
uses: Swatinem/rust-cache@v2
with:
save-if: true

- name: Run tests
run: cargo test --workspace --verbose --release
27 changes: 0 additions & 27 deletions ark-cli/README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,5 @@
# Ark-CLI

### Installation

To compile you will need openssl libraries and headers:

```shell
# macOS (Homebrew)
$ brew install openssl@3

# macOS (MacPorts)
$ sudo port install openssl

# macOS (pkgsrc)
$ sudo pkgin install openssl

# Arch Linux
$ sudo pacman -S pkg-config openssl

# Debian and Ubuntu
$ sudo apt-get install pkg-config libssl-dev

# Fedora
$ sudo dnf install pkg-config perl-FindBin openssl-devel

# Alpine Linux
$ apk add pkgconfig openssl-dev
```

### Usage

```shell
Expand Down
2 changes: 1 addition & 1 deletion data-error/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ bench = false

[dependencies]
thiserror = "1"
reqwest = "0.11.11"
reqwest = { version = "0.11.11", features = [ "json", "rustls-tls" ], default-features = false }
serde_json = "1.0.82"
anyhow = "1"
url = { version = "2.2.2", features = ["serde"] }
2 changes: 1 addition & 1 deletion data-link/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ log = { version = "0.4.17", features = ["release_max_level_off"] }
serde_json = "1.0.82"
serde = { version = "1.0.138", features = ["derive"] }
url = { version = "2.2.2", features = ["serde"] }
reqwest = "0.11.11"
reqwest = { version = "0.11.11", features = [ "json", "rustls-tls" ], default-features = false }
scraper = "0.13.0"
tokio = { version = "1", features = ["full"] }

Expand Down
20 changes: 20 additions & 0 deletions java/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
buildscript {
repositories {
google()
gradlePluginPortal()
maven { url "https://plugins.gradle.org/m2/" }
}
dependencies {
classpath libs.gradle
classpath libs.rust.android
}
}

allprojects {
repositories {
google()
gradlePluginPortal()
maven { url "https://jitpack.io" }
maven { url "https://plugins.gradle.org/m2/" }
}
}
16 changes: 7 additions & 9 deletions java/gradle/libs.versions.toml
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
# This file was generated by the Gradle 'init' task.
# https://docs.gradle.org/current/userguide/platforms.html#sub::toml-dependencies-format

[versions]
commons-math3 = "3.6.1"
guava = "33.0.0-jre"
junit-jupiter = "5.10.2"
gradle = "8.7.3"
rustAndroid = "0.9.4"
junitJupiterApi = "5.11.3"

[libraries]
commons-math3 = { module = "org.apache.commons:commons-math3", version.ref = "commons-math3" }
guava = { module = "com.google.guava:guava", version.ref = "guava" }
junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit-jupiter" }
gradle = { module = "com.android.tools.build:gradle", version.ref = "gradle" }
rust-android = { module = "org.mozilla.rust-android-gradle:plugin", version.ref = "rustAndroid" }
junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junitJupiterApi" }
junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junitJupiterApi" }
Loading
Loading