Skip to content

GH-46336: [Release][Packaging] Add support for Reproducible Builds for source archive #46342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

kou
Copy link
Member

@kou kou commented May 7, 2025

Rationale for this change

See https://reproducible-builds.org/ for Reproducible Builds.

Automated Release Signing requires this:
https://infra.apache.org/release-signing.html#automated-release-signing

What changes are included in this PR?

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

@kou kou requested review from assignUser, jonkeane and raulcd as code owners May 7, 2025 07:28
Copy link

github-actions bot commented May 7, 2025

⚠️ GitHub issue #46336 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions bot added the awaiting committer review Awaiting committer review label May 7, 2025
@kou
Copy link
Member Author

kou commented May 7, 2025

Ah, https://infra.apache.org/release-signing.html#automated-release-signing includes the following:

The release procedure contains a validation step where all artifacts are reproduced on trusted hardware before publication to pages intended for end users

We need to add a reproducible check to dev/release/verify-release-candidate.sh for it.

@kou
Copy link
Member Author

kou commented May 7, 2025

We need to add a reproducible check to dev/release/verify-release-candidate.sh for it.

Implemented.

@kou
Copy link
Member Author

kou commented May 8, 2025

@assignUser @raulcd Do you want to review this before we request a review from INFRA?

Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks great to me, thanks @kou ! And thanks for adding the possibility of running this not only for tags. Just a minor comment about something I don't understand

@github-actions github-actions bot added awaiting merge Awaiting merge awaiting review Awaiting review and removed awaiting committer review Awaiting committer review labels May 8, 2025
@github-actions github-actions bot added awaiting review Awaiting review awaiting changes Awaiting changes and removed awaiting review Awaiting review awaiting merge Awaiting merge labels May 8, 2025
@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels May 8, 2025
@kou
Copy link
Member Author

kou commented May 9, 2025

I asked INFRA to review and enable automatic release signing for source arhicve: https://issues.apache.org/jira/browse/INFRA-26808

@assignUser
Copy link
Member

Sorry, I didn't have time to have a look so far, but this is great!

tar czf ${tarball} ${root_folder}
# Create new tarball from modified source directory.
#
# We need GNU tar for Reproducible Builds. We want to use the same
Copy link
Member

@pitrou pitrou May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping we can use something like https://salsa.debian.org/reproducible-builds/strip-nondeterminism but it might not have support for tar files.

Copy link
Member Author

@kou kou May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, I didn't know the tool. It's introduced in https://reproducible-builds.org/docs/stripping-unreproducible-information/ .

The approaches used here are listed in https://reproducible-builds.org/docs/archives/ .

Let's use GNU tar for now because strip-nondeterminism doesn't have tar support.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I've added.

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels May 15, 2025
# * https://github.com/apache/arrow/pull/4488
#
# We need to set constant timestamp for a dummy .git/ directory for
# Reproducible Builds. We use mtime of csharp/ for it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we use SOURCE_DATE_EPOCH?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know SOURCE_DATE_EPOCH.
I used it with tar not here.

@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting changes Awaiting changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants