feat: Add nix flake for ffi #1319

maru-ava · 2025-09-26T20:08:39Z

Why this should be merged

A nix flake for the firewood ffi library enables deterministic builds both for firewood-go-ethhash and a future nix build of avalanchego.

How this works

Adds flake for ffi crate
Updates attach-static-libs workflow to build static libs with the flake

How was this tested

CI (new ffi-nix job)
Locally
- Install nix:
  - curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
- cd ffi
- Build:
  - nix build .#firewood-ffi
- Test (assuming build step perfomed) outside of a nix dev shell:
  - GOLANG="nix run $PWD#go"
    - Runs the version of golang referenced by the ffi flake
    - Need to capture this before changing directories to result/ffi because result is a nix store symlink so ../../ won't resolve to the ffi path containing the flake
    - Running golang this way instead of with nix develop validates usage of the build output won't require nix develop's shell magic
  - cd result/ffi
  - GOEXPERIMENT=cgocheck2 TEST_FIREWOOD_HASH_MODE=ethhash ${GOLANG} test ./... -v

rkuris

Definitely don' tknow enough about nix to consider reviewing this, but willing to stamp it after these questions are answered:

Is there any change to the release process? The RELEASE.md file wasn't changed if so.
When dependency updates happen, are any additional steps required?
I noticed all the files in the /nix result directory are dated 1969. Is this expected?
The object files are about half as big, nearly identical text area bug the "others" section is much smaller. Any idea why?

➜ ffi git:(main) ✗ size firewood_ffi*

__TEXT	__DATA	__OBJC	others	dec	hex
827635	38472	0	44544	910651	de53b	firewood_ffi-ac2de0665df5119e.firewood_ffi.36d0102cbf9e788e-cgu.0.rcgu.o
835697	38464	0	1815990	2690151	290c67	firewood_ffi-df1593dabf0edbed.firewood_ffi.eccd115242f4cb72-cgu.0.rcgu.o

I noticed the relocation table is smaller (25899 entries vs 54092). Maybe some new optimization is being done?

maru-ava · 2025-09-30T17:47:48Z

Is there any change to the release process? The RELEASE.md file wasn't changed if so.

afaik the only change is to the CI job that publishes to firewood-go-ethhash, and I don't see mention of that in RELEASE.md

When dependency updates happen, are any additional steps required?

The only requirement is to keep Cargo.lock up-to-date, and Brandon already added a CI check to ensure PRs won't merge if it's out-of-sync.

I noticed all the files in the /nix result directory are dated 1969. Is this expected?

That is expected - it's the unix epoch (i.e. time=0). Nix sets all built outputs to this time to enable reproducibility, content-addressibility and build caching.

The object files are about half as big, nearly identical text area bug the "others" section is much smaller. Any idea why?

Assuming this is an apples-to-apples comparison (i.e. maxperf and same build args), I would guess there is a difference in the build tooling that nix uses?

rkuris · 2025-09-30T20:56:07Z

Assuming this is an apples-to-apples comparison (i.e. maxperf and same build args), I would guess there is a difference in the build tooling that nix uses?

Yes, it was apples to apples. One concern is that it may affect performance. I'll bring this up at standup.

maru-ava · 2025-10-01T00:47:38Z

Assuming this is an apples-to-apples comparison (i.e. maxperf and same build args), I would guess there is a difference in the build tooling that nix uses?

Yes, it was apples to apples. One concern is that it may affect performance. I'll bring this up at standup.

Would it make sense to remove the update to the CI job pending qualification of the build artifact with an appropriate performance test?

rkuris · 2025-10-01T01:15:33Z

Yes, that's a good idea.

…

On Tue, Sep 30, 2025, 5:48 PM maru ***@***.***> wrote: *maru-ava* left a comment (ava-labs/firewood#1319) <#1319 (comment)> Assuming this is an apples-to-apples comparison (i.e. maxperf and same build args), I would guess there is a difference in the build tooling that nix uses? Yes, it was apples to apples. One concern is that it may affect performance. I'll bring this up at standup. Would it make sense to remove the update to the CI job pending qualification of the build artifact with an appropriate performance test? — Reply to this email directly, view it on GitHub <#1319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYLR3H7WEIXGJDINVKM7DL3VMQEBAVCNFSM6AAAAACHT3JLC6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNJUGI3TMOBWGQ> . You are receiving this because your review was requested.Message ID: ***@***.***>

maru-ava · 2025-10-01T03:49:58Z

Removed changes to attach-static-libs workflow and added new nix-specific ffi-nix job to ensure coverage.

maru-ava · 2025-10-01T04:49:55Z

ffi/flake.nix

+      commonArgs = {
+        inherit src;
+        strictDeps = true;
+        dontStrip = true;


Nix strips debugging details by default, which was resulting in the .a file being ~6MB instead of ~50MB.

demosdemon · 2025-10-01T15:57:34Z

ffi/flake.nix

+        pname = ffiCargoToml.package.name;
+        version = workspaceCargoToml.workspace.package.version;
+
+        CARGO_PROFILE = "maxperf";


this doesn't appear to be working. the size reduction @rkuris noticed is from fewer inlined methods which tells me there's less LTO.

~~I haven't determined the correct way to set this with rust-overlay, but cargo will use the PROFILE environment variable if --profile is not provided on the command line.~~

EDIT: that appears to be incorrect. looks like there is no environment variable that cargo will use to infer the profile and it must be provided in the cli invocation. https://doc.rust-lang.org/cargo/reference/profiles.html#profile-selection

I see CARGO_PROFILE is actually a parameter to crane's mkCargoDerivation which is wrapped by buildDepsOnly and buildPackage. So, I'm not entirely sure why this profile doesn't appear to work.

As per the most recent code comment, the reason for the reduction in size was due to nix automatically stripping debugging symbols (which is the default for nix). I've corrected that, and the .a files are now [edit: almost] identical in size.

I also believe that CARGO_PROFILE is being respected. This var is for configuring crane as per its api docs (search for CARGO_PROFILE) rather than intended for direct consumption by cargo. This can be verified by the following invocation:

cd ffi && nix build .#firewood-ffi --print-build-logs --rebuild 2>&1 | grep -i "cargo build"

Ah, I didn't realize the change to not strip came after Ron's comment.

demosdemon · 2025-10-01T18:04:35Z

ffi/flake.nix

+        # Install the static library and header
+        postInstall = ''
+          mkdir -p $out/lib $out/include
+          cp target/*/libfirewood_ffi.a $out/lib/ || cp target/release/libfirewood_ffi.a $out/lib/


the * should probably be replaced with maxperf as the target directory is deterministic on the profile used. If the resulting artifact isn't in the maxperf directory, then we're using the wrong profile.

maru-ava · 2025-10-01T18:17:58Z

fwiw I added a script to validate build equivalency. Not sure whether it makes sense to merge it, I just wanted to have a common reference point of comparison.

maru-ava · 2025-10-01T18:24:04Z

I tried to run the build equivalency script on macos but there are some issues with the macos support that I think should be corrected before merge.

maru-ava · 2025-10-15T14:06:40Z

jemalloc appears to be introducing non-deterministic build behavior in CI but not locally. Updating to run make invocations serially appears to resolve this.

…ergence

Set MAKEFLAGS=-j1 to force sequential build of vendored jemalloc in tikv-jemallocator. The vendored jemalloc has race conditions during parallel builds causing non-deterministic symbol generation on x86_64 (NixOS/nixpkgs#380852). Using MAKEFLAGS instead of NUM_JOBS ensures only the make invocation for jemalloc is affected, while Cargo continues to build Rust crates in parallel. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

rkuris

Good to go, please rebase and merge. Maintenance question should just get answered here. Other stuff are nits that don't need fixing.

rkuris · 2025-10-17T23:43:31Z

ffi/test-build-equivalency.sh

+# Use grep with -E for better portability (avoid -P which isn't available on macOS)
+if [[ "$OSTYPE" == "darwin"* ]]; then
+    grep -Eo "$RELOC_PATTERN" "$TMPDIR/nix-relocs.txt" | sort | uniq -c > "$TMPDIR/nix-reloc-types.txt"
+    grep -Eo "$RELOC_PATTERN" "$TMPDIR/cargo-relocs.txt" | sort | uniq -c > "$TMPDIR/cargo-reloc-types.txt"
+else
+    grep -oP "$RELOC_PATTERN" "$TMPDIR/nix-relocs.txt" | sort | uniq -c > "$TMPDIR/nix-reloc-types.txt"
+    grep -oP "$RELOC_PATTERN" "$TMPDIR/cargo-relocs.txt" | sort | uniq -c > "$TMPDIR/cargo-reloc-types.txt"
+fi


nit: Aren't there some grep options that work on both platforms?

otherwise, I would have preferred GREPFLAGS=-Eo and GREPFLAGS=-Op based on $OSTYPE and then combining all those into single statements.

Updated to minimize duplication: 15742d2

rkuris · 2025-10-17T23:45:36Z

.github/workflows/ci.yaml

+      - uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 #v20
+      - uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 #v13


Questions: How and when are these hardcoded shas updated? I'm sure dependabot won't tell us when they update. Should we add something to the release checklist to see if they should be updated?

What happens if someone finds a security vulnerability in these packages?

dependabot can, in fact, tell you to update. I just checked, though, and it's not yet configured for this repo. This is what needs to be in .github/dependabot.yml:

- package-ecosystem: github-actions directory: "/" schedule: interval: weekly open-pull-requests-limit: 0 # Disable non-security version updates

As per ava-labs/avalanchego#3822, specifying SHAs for 3rd party actions (i.e. not provided by github) is best practice to avoid supply chain attack. A high-profile incident earlier this year galvanized awareness of the problem.

Updated dependabot config: dc0f60b

maru-ava self-assigned this Sep 26, 2025

maru-ava force-pushed the maru/nix-flake branch 6 times, most recently from 1bf2a49 to 4d41947 Compare September 30, 2025 06:55

maru-ava marked this pull request as ready for review September 30, 2025 07:15

maru-ava requested review from aaronbuchwald, alarso16, demosdemon and rkuris as code owners September 30, 2025 07:15

rkuris reviewed Sep 30, 2025

View reviewed changes

rkuris self-requested a review September 30, 2025 20:56

rkuris approved these changes Sep 30, 2025

View reviewed changes

maru-ava force-pushed the maru/nix-flake branch from ea5ffaa to 237d67e Compare October 1, 2025 03:48

maru-ava force-pushed the maru/nix-flake branch from 237d67e to b1a0a39 Compare October 1, 2025 04:28

maru-ava commented Oct 1, 2025

View reviewed changes

demosdemon requested changes Oct 1, 2025

View reviewed changes

demosdemon approved these changes Oct 1, 2025

View reviewed changes

rkuris self-requested a review October 1, 2025 18:00

maru-ava force-pushed the maru/nix-flake branch from 4f04b6d to 76e2985 Compare October 1, 2025 18:02

demosdemon reviewed Oct 1, 2025

View reviewed changes

maru-ava mentioned this pull request Oct 9, 2025

feat: Add just task runner to ensure reproducibility of CI #1345

Draft

maru-ava force-pushed the maru/nix-flake branch 2 times, most recently from 6ea2831 to dbd034b Compare October 15, 2025 02:47

maru-ava requested a review from rkuris October 15, 2025 15:34

maru-ava and others added 21 commits October 17, 2025 09:19

feat: Add nix flake for ffi

93f4979

fixup: Add crane for dep artifact caching and update static lib action

92d7781

fixup: Document flake usage

1f2d487

fixup: Revert ci changes

5461e71

fixup: Add ffi jobs for nix

f65aa82

fixup: Use dontStrip in the nix flake to match regular cargo build

b4ed030

fixup: Limit ffi-nix job to ubuntu to minimize runtime

ee73111

fixup: Add script to test for build equivalency

f637e29

fixup: Update to copy only the maxperf lib

a8856ce

fixup: Remove deprecated darwin stubs

6d8dc01

fixup: Fix macos target to apply to both buildDepsOnly and buildPackage

d77b0e5

fixup: Add build equivalency check to ffi-nix job

8418c92

fixup: Update script to exit non zero if builds differ

0ee0167

fixup: Clarify comment justifying test mode

55c520e

fixup: Add build-static-ffi cargo alias to minimize the change of div…

3eaf74d

…ergence

fixup: Ensure flake output equivalent to firewood-go-ethhash

f846af1

fixup: Avoid unnecessary rebuilding in build equivalency test

3f33ee6

fixup: Update to use avalanchego's golang flake for 1.24.8

5b29b0d

fixup: Configure dependabot for security updates for github actions

dc0f60b

fixup: Minimize duplication in build equivalency script

15742d2

rkuris approved these changes Oct 17, 2025

View reviewed changes

maru-ava force-pushed the maru/nix-flake branch from 7cd05cc to 0cf4a26 Compare October 18, 2025 19:07

maru-ava requested a review from rkuris October 18, 2025 19:08

maru-ava force-pushed the maru/nix-flake branch from 0cf4a26 to 15742d2 Compare October 18, 2025 19:10

		- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 #v20
		- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 #v13

feat: Add nix flake for ffi #1319

Are you sure you want to change the base?

feat: Add nix flake for ffi #1319

Conversation

maru-ava commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How was this tested

Uh oh!

rkuris left a comment

Choose a reason for hiding this comment

Uh oh!

maru-ava commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rkuris commented Sep 30, 2025

Uh oh!

maru-ava commented Oct 1, 2025

Uh oh!

rkuris commented Oct 1, 2025 via email

Uh oh!

maru-ava commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maru-ava Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

demosdemon Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demosdemon Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maru-ava Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demosdemon Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

demosdemon Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava commented Oct 1, 2025

Uh oh!

maru-ava commented Oct 1, 2025

Uh oh!

maru-ava commented Oct 15, 2025

Uh oh!

rkuris left a comment

Choose a reason for hiding this comment

Uh oh!

rkuris Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rkuris Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

maru-ava Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

maru-ava commented Sep 26, 2025 •

edited

Loading

maru-ava commented Sep 30, 2025 •

edited

Loading

maru-ava commented Oct 1, 2025 •

edited

Loading

demosdemon Oct 1, 2025 •

edited

Loading

demosdemon Oct 1, 2025 •

edited

Loading

maru-ava Oct 1, 2025 •

edited

Loading

maru-ava Oct 18, 2025 •

edited

Loading

maru-ava Oct 18, 2025 •

edited

Loading