Added Testnet4 Support

[takinbo]

As part of the work to get Dojo packaged for StartOS, the need to provide testnet4 support in electrs has emerged.

This pull request adds support for testnet4 and restores support for using btc-rpc-proxy in the electrs wrapper.

[gStart9]

I could be wrong but I don't think Dojo is coming to StartOS v0351 -- even if you did manage to pull it off, you're going to have to completely retranslate your service for v036. As such you'll need to base your electrs changes on the 036 branch

[kn0wmad]

Dojo is in the works, submission should come soon, followed by fulcrum

[kn0wmad]

Last I heard 036 was backward compatible with 035 packages - is that still accurate?

[linkinparkrulz]

Yeah, why would it not be backwards compatible?

[kn0wmad]

Sounds like that is no longer the case, and after 036 all wrappers need re-packaged

[gStart9]

I'm having trouble building this:

user@desktop:~/Dev/Services/electrs-startos-takinbo$ make
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "/home/user/Dev/Services/electrs-startos-takinbo"/configurator:/home/rust/src messense/rust-musl-cross:aarch64-musl cargo build --release
warning: `/root/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
error: failed to download `base64 v0.13.0`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/base64-0.13.0/Cargo.toml`

Caused by:
  no targets specified in the manifest
  either src/lib.rs, src/main.rs, a [lib] section, or [[bin]] section must be present
make: *** [Makefile:42: configurator/target/aarch64-unknown-linux-musl/release/configurator] Error 101

electrs-startos' current master builds fine. Any ideas what the issue is?

[linkinparkrulz]

I cloned @takinbo 's electrs repo. Ran:

git submodule update --init --recursive
make

and this was the result:

=> [final  9/12] RUN chmod a+x /usr/local/bin/check-electrum.sh                                                                                     0.1s
 => [final 10/12] ADD ./check-synced.sh /usr/local/bin/check-synced.sh                                                                               0.0s
 => [final 11/12] RUN chmod a+x /usr/local/bin/check-synced.sh                                                                                       0.1s
 => [final 12/12] WORKDIR /data                                                                                                                      0.0s
 => exporting to docker image format                                                                                                                 1.6s
 => => exporting layers                                                                                                                              1.3s
 => => exporting manifest sha256:ec33768fd863047664c5e295c0e862619926d2abf6589cc7b257aa6dcc4f563f                                                    0.0s
 => => exporting config sha256:1f0316794fd9fa8ffae8631db5516b026f443f348362082f02ee6c33c568f1f2                                                      0.0s
 => => sending tarball                                                                                                                               0.3s
start-sdk pack
start-sdk verify s9pk electrs.s9pk
linkinparkrulz@pop-os:~/electrs-startos$ 

[takinbo]

I'm having trouble building this:

error: failed to download `base64 v0.13.0`



Caused by:

  unable to get packages from source

electrs-startos' current master builds fine. Any ideas what the issue is?

This error appears to be associated with the build process for the configurator. It seems the download of the base64 crate fails at this step. Considering none of the dependencies for the rust project were changed from the original wrapper, I'm unable to determine why it fails on your build machine. I'll suggest running the build again and if possible clearing your rust crates cache.

[gStart9]

clearing your rust crates cache

That was it. Thanks so much!

I ran your build and it's looking pretty good -- no errors or anything. When I select Bitcoin Core (proxy), it tells me I don't have the required btc-rpc-proxy service:

Which got me to thinking, you're probably not aware, but our bitcoind-startos package includes the btc-rpc-proxy rust crate, and anything that connects to bitcoin's rpc port 8332, is actually connecting to btc-rpc-proxy. This is so that if the user is running in pruned mode, btc-rpc-proxy will fetch the blocks from the network and return them as if it were a full archival node, which irons out any issues with other services that require a full archival node. Is that why you were attempting to include an option for bitcoin rpc proxy? If so, packaging a separate btc-rpc-proxy service entirely seems unnecessary.

[linkinparkrulz]

Woah that's interesting.

Dojo requires a full archival node to work. But, based on what you're saying their may be an opportunity to run pruned via this built in proxy package.

My understanding is that we were making everything standalone because thats how the applications were constructed from what we forked. I think if you look at the fulcrum package, the architecture will be the same.

[takinbo]

I ran your build and it's looking pretty good -- no errors or anything. When I select Bitcoin Core (proxy), it tells me I don't have the required btc-rpc-proxy service:

This is actually in reference to the btc-rpc-proxy startOs package which allows for fine-grained permissions for rpc users. It's an optional dependency so it's not a requirement to use electrs. It's there if the user has it and has a configuration for electrs. This is also specified as an optional dependency in the fulcrum startOs package.

[gStart9]

This is actually in reference to the btc-rpc-proxy startOs package which allows for fine-grained permissions for rpc users

That package hasn't been updated in >3 years, and is no longer published even on our community registry -- we discontinued it because it had bugs that were disrupting some dependent services. So it's not available to anyone, except for people who installed it 3+ years ago. We plan to bring that functionality back, but more likely via Bitcoin Core's build-in capability, and definitely after packages have transitioned to StartOS v036.

[linkinparkrulz]

Is there an action you are recommending @gStart9?

[kn0wmad]

Probably just rip out the optional deps, since the package is deprecated and unmaintained

[gStart9]

I wanted to wait for this but we need to get v0.10.9 moving. I was going to rip out the btc-rpc-proxy stuff from this PR and merge the testnet4 stuff from this PR, and I thought, "cool, people can switch from testnet4 to mainnet and back", but the more I think about it, I think you need to fork electrs-startos and make electrs-testnet4-startos (or maybe even more preferably for future naming purposes, romanz-electrs-testnet4-startos (because with 036 I think people are going to want to also use the much more powerful options that will no doubt be available at some point: blockstream-electrs-startos and/or mempool-electrs-startos)).

Correct me if I'm wrong, but romanz electrs can only look at one DB at a time. We can't have it constantly build the mainnet db and testnet4 db simultaneously, can we? I think it would be more advantageous to be able to run them both at the same time, and if you want whatever client to look at the different db, you just switch the .embassy/.onion:50001 URL you look at. Does that rational for forking this to (romanz-)electrs-testnet4-startos make sense to y'all?

[elvece]

Sounds like that is no longer the case, and after 036 all wrappers need re-packaged

0351 packages can run on 036. We are recommending that all packages be converted (eventually) to make use of performance enhancements and the incredibly featureful new packaging SDK, which greatly enhances the package development and maintenance experience.

[kn0wmad]

Sounds like that is no longer the case, and after 036 all wrappers need re-packaged

0351 packages can run on 036. We are recommending that all packages be converted (eventually) to make use of performance enhancements and the incredibly featureful new packaging SDK, which greatly enhances the package development and maintenance experience.

Oh, awesome!

[takinbo]

Using a hard-coded path to the log file makes it impossible to parse the log file for the testnet4 instance which lives in a different path.

[gStart9]

Thanks for the updates. Btw I believe I was thinking about this totally wrong before -- testnet4 should not be a fork, we should just make the service reflect the software's possibilities to the extent that they exist. So, ultimately, not only should electrs-startos support mainnet and testnet4, it should also support testnet, regtest, and signet.

After this PR gets merged I'll gladly add those other networks.

In that vein, althought testnet4 doesn't actually seem to be formally specified in electrs' config spec:
https://github.com/romanz/electrs/blob/v0.10.9/internal/config_specification.toml#L71-L87
It seems 40001 is the default port for testnet4. We'll want to reflect that in the service's properties page by adding the port to the /data/start9/stats.yaml (as set in) docker_entrypoint.sh if they select testnet4.