Skip to content

contribute-nearcore: add info on ROCKSDB_LIB_DIR and other minor updates #1076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 3, 2022
Merged

contribute-nearcore: add info on ROCKSDB_LIB_DIR and other minor updates #1076

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b395f1
contribute-nearcore: add info on ROCKSDB_LIB_DIR and other minor updates
mina86 Jun 3, 2022
6a15f20
time env
mina86 Jun 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 71 additions & 33 deletions docs/community/contribute/nearcore.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ cd nearcore
Navigate to the root of the repository, and run:

```bash
cargo run --package neard --bin neard -- init
cargo run --package neard --bin neard -- run
cargo run -p neard -- init
cargo run -p neard -- run
```

This will setup a local chain with `init` and will run the node.
Expand All @@ -63,10 +63,11 @@ If you are using CLion IDE you can configure it to run rustfmt automatically eve

## Testing {#testing}

To run NEARCore node in the testing mode, for example to test it or for development of `near-api-js` or `near-cli` you can use scripts that sets up special tests-only local testnet:
To run NEARCore node in the testing mode, for example to test it or for development of `near-api-js` or `near-cli` you can use commands that sets up special tests-only local testnet:

```bash
./scripts/start_localnet.py
cargo run -p neard -- localnet -v1
cargo run -p neard -- run
```

This sets up a new single node testnet, with predetermined private key of the validator and turns on "fast" mode of block production to make development and testing easy.
Expand All @@ -76,7 +77,7 @@ This sets up a new single node testnet, with predetermined private key of the va
Many times in development of the node it's useful to see detailed logs about what is happening. `neard` binary has `--verbose` mode to show more details about what is happening:

```bash
cargo run --package neard --bin neard -- --verbose=true run
cargo run -p neard -- --verbose= run
```

You can also use the `RUST_LOG` environment variable, with `env_logger` [semantics](https://docs.rs/env_logger/0.6.0/env_logger/#enabling-logging) to override the log level for specific targets. `RUST_LOG` can also be used in integration tests which spawn runnable apps.
Expand Down Expand Up @@ -115,7 +116,7 @@ Official image is published at `nearprotocol/nearcore`
> <a href="https://stackoverflow.com/questions/tagged/nearprotocol">
> <h8>Ask it on StackOverflow!</h8></a>

# How to speed up rust compilation time
## How to speed up rust compilation time

This describes on how to improve nearcore compilation time without having to
modify Cargo.toml. It's possible to override Cargo.toml setting by setting
Expand All @@ -124,14 +125,14 @@ modify Cargo.toml. It's possible to override Cargo.toml setting by setting

### Default build {#default-build}

```bash
# cargo clean
# time RUSTFLAGS= RUSTC_WRAPPER= cargo build -p neard --release
```console
$ cargo clean
$ time env RUSTFLAGS= RUSTC_WRAPPER= cargo build -p neard --release
real 2m13.773s
user 44m49.204s
sys 1m22.583s
# touch */*/*/*.rs
# time RUSTFLAGS= RUSTC_WRAPPER= cargo build -p neard --release
$ touch */*/*/*.rs
$ time env RUSTFLAGS= RUSTC_WRAPPER= cargo build -p neard --release
real 0m38.021s
user 9m37.045s
sys 0m5.302s
Expand All @@ -149,14 +150,14 @@ lto=off`.

Requires installing lld linker.

```bash
# cargo clean
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld' RUSTC_WRAPPER= cargo build -p neard --release
```console
$ cargo clean
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld' RUSTC_WRAPPER= cargo build -p neard --release
real 1m50.802s
user 36m50.251s
sys 1m19.069s
# touch */*/*/*.rs
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld' RUSTC_WRAPPER= cargo build -p neard --release
$ touch */*/*/*.rs
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld' RUSTC_WRAPPER= cargo build -p neard --release
real 0m28.951s
user 6m56.670s
sys 0m4.307s
Expand All @@ -166,32 +167,69 @@ sys 0m4.307s

Works only with nightly compiler.

```bash
# cargo clean
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER= cargo build -p neard --release
```console
$ cargo clean
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER= cargo build -p neard --release
real 1m42.999s
user 33m31.341s
sys 1m25.773s
# touch */*/*/*.rs
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER= cargo build -p neard --release
$ touch */*/*/*.rs
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER= cargo build -p neard --release
real 0m23.501s
user 4m30.597s
sys 0m3.804s
```

### Use system-provided rocksdb

By default, building `neard` will result in compiling `librocksdb`
which takes up some non-negligible amount of memory especially when
building on a system with many cores. It’s possible to configure
build process to provide system-provided library instead. This is
done by setting `ROCKSDB_LIB_DIR` environment variable to location
where `librocksdb.a` file is installed:

```console
$ cargo clean
$ time env ROCKSDB_LIB_DIR=/usr/lib/x86_64-linux-gnu cargo build -p neard --release
real 1m31.014s
user 30m46.544s
sys 1m26.932s
$ touch */*/*/*.rs
$ time env ROCKSDB_LIB_DIR=/usr/lib/x86_64-linux-gnu cargo build -p neard --release
real 0m35.061s
user 9m7.968s
sys 0m10.486s
```

Note that the system must provide a recent version of the library
which, depending on operating system you’re using, may require
installing packages from testing branches. For example, on Debian it
requires installing `librocksdb-dev` from `experimental` version:

```bash
echo 'deb http://ftp.debian.org/debian experimental main contrib non-free' |
sudo tee -a /etc/apt/sources.list
sudo apt update
sudo apt -t experimental install librocksdb-dev

ROCKSDB_LIB_DIR=/usr/lib/x86_64-linux-gnu
export ROCKSDB_LIB_DIR
```

### Cache results from previous compilations using sccache {#cache-results-from-previous-compilations-using-sccache}

Requires installing sscache. If you want to compile sccache with `cargo install sccache` make sure you use stable version of Rust.

```bash
# cargo clean
# rm -rf ~/.cache/sccache/
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard --release
```console
$ cargo clean
$ rm -rf ~/.cache/sccache/
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard --release
real 2m6.452s
user 3m24.797s
sys 0m30.919s
# cargo clean
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard --release
$ cargo clean
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard --release
real 0m24.292s
user 3m3.627s
sys 0m27.619s
Expand All @@ -201,9 +239,9 @@ sys 0m27.619s

### Setting for building release binary with debug symbols and reduced inlining {#setting-for-building-release-binary-with-debug-symbols-and-reduced-inlining}

```bash
# cargo clean
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y -C inline-threshold=25 -C debuginfo=2' RUSTC_WRAPPER=sccache cargo build -p neard --release
```console
$ cargo clean
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y -C inline-threshold=25 -C debuginfo=2' RUSTC_WRAPPER=sccache cargo build -p neard --release
real 1m50.521s
user 3m39.398s
sys 0m32.069s
Expand All @@ -214,9 +252,9 @@ sys 0m32.069s
Building this way cuts down build time, but the node will likely be too slow to run. This is useful in case you need
to build package to run tests or do build within CLion/Intellij.

```bash
# cargo clean
# time RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard
```console
$ cargo clean
$ time env RUSTFLAGS='-C lto=off -C link-arg=-fuse-ld=lld -Zshare-generics=y' RUSTC_WRAPPER=sccache cargo build -p neard
real 1m18.198s
user 4m35.409s
sys 0m32.220s
Expand Down