Drop Template Distribution Protocol

[TheBlueMatt]

Looks like Bitcoin Core folks have decided to NIH the template distribution protocol in-house, so we should probably follow suit and directly support their (cap'nproto-based local-sockets) API. Once we have that, there's not really any reason to have the template distribution protocol at all, it'll just be an unnecessary proxy, but we should maybe support piping their local-sockets-only stuff over TCP for completeness.

[GitGab19]

Is there a discussion regarding this? I would like to better understand what are their plans

[TheBlueMatt]

They had a private discussion and decided it. bitcoin/bitcoin#31098

[TheBlueMatt]

I'll probably join one of their meetings and try to convince them to at least document the protocol they're building with a BIP, but it seems like they're kinda YOLOing this whole thing :(.

[plebhash]

yeah a BIP would be great, otherwise we're likely going to end up chasing a moving target

[Sjors]

They had a private discussion and decided it.

Not really. There were private discussions and I concluded there was no support for it. It's not a group decision, and who knows maybe people change their minds later.

[plebhash]

it looks like sv2-ipc branch of @Sjors fork will be our north star for this journey

Sjors/bitcoin#48

[TheBlueMatt]

It sounds like (based on the comments at bitcoin/bitcoin#31098) Bitcoin Core wants the IPC interface to actually be a public interface, rather than an internal one. Given this, there's zero reason to have a proxy between Bitcoin Core and the Sv2 proxy, we should just speak directly to Bitcoin Core. Whatever Bitcoin Core does here will become a de-facto standard for work generation, and presumably the various MEV bidding things will speak it, so trying to fight the current and define our own protocol would be a waste of time.

[plebhash]

the first key difference (compared to what we're currently used to) that I notice on Sjors/bitcoin#48 is the existance of two separate binaries:

• bitcoin-node: which seems to be just the usual bitcoind with a new name under the new CMake build system
• bitcoin-mine: which I assume is a sidecar doing IPC with bitcoin-node over UNIX socket and exposing the mining interface over TCP?

[plebhash]

Bitcoin Core wants the IPC interface to actually be a public interface, rather than an internal one. Given this, there's zero reason to have a proxy between Bitcoin Core and the Sv2 proxy, we should just speak directly to Bitcoin Core.

so in other words bitcoin-mine is superfluous and JDS, JDC and Pool should connect directly to bitcoin-node?

over TCP sockets? or are we assuming they're on the same system and this will happen over UNIX sockets?

[TheBlueMatt]

the first key difference (compared to what we're currently used to) that I notice on Sjors/bitcoin#48 is the existance of two separate binaries:

Yea, last I'd heard all the Bitcoin Core IPC things were intended as an internal API for Bitcoin Core to ship several daemons and then they'd handle all the overhead of making it look like a single daemon, ie just an internal thing that doesn't really impact us. Now it sounds like instead the IPC is the public interface and we need to adapt to that.

over TCP sockets? or are we assuming they're on the same system and this will happen over UNIX sockets?

Sounds like they're only going to support UNIX sockets of some form.

[Sjors]

Afaik multiprocess works via TCP too, but that hasn't been merged /PR'd yet.

[Fi3]

NACK, there are at least 2 valid reasons to not drop it:

• I can deploy a TP as sidecar, that will let me does things like safely connect multiple client to a remote TP.
• bitcoind can adopt it in the future, or as also suggested in the spec an alternative implementation can adopt it.

Below other 3 reason that can or can not be valid:

• is already implemented and it work well, all the code assume that there is a TDP removing it could or could not result in a problematic refactor.
• other implementation like demand use it, removing it will likely means do a big refactor
• we spent a lot of energy educating people about sv2, removing a very important part as the TDP and TP: (1) will invalidate most of our education material; (2) will likely create a lot of confusion in the people that are trying to understand what sv2 is and how it works

[Fi3]

btw completly dropping support for the TP seems like a lot of work, I would rather (if we really want to go that way) deprecate it.

[TheBlueMatt]

My point is really that piping this new protocol over netcat is simpler and not logically different from having a "sidecar", its just instead of SRI having to maintain two protocol implementation clients plus some protocol translation proxy we only have to maintain one. We could bundle netcat+a-cryptographic-wrapper as "the sidecar", which would be much, much simpler than trying to convert to a totally unrelated protocol.

[Sjors]

Just repeating my point below that the node fully trusts the IPC client, unlike the current sidecar (modulo bugs).

[Fi3]

You can do that, but keep in mind that the node blindly trusts the application connecting to it. You can't use this interface for offering template generation as a public service. That requires a Template Provider (ish) thing in between.

if that is the case I think that netcat+xyz won't work

[TheBlueMatt]

We're getting our discussions crossed. Mentioned in the other thread

For unrelated reasons (eg bitcoin/bitcoin#31109) I imagine this will (basically) change. Maybe to make it safe you need some proxy that filters certain messages but ideally Bitcoin Core can do that eventually.

Even if it doesn't fully change there's still no reason for a separate protocol, we'd instead have a filtering proxy to make it safe.

[Sjors]

I don't think you should drop the Template Distribution Protocol. It's still useful to have a minimal application that sits between Bitcoin Core and the rest of the ecosystem. The Template Provider code I use still works in the new (sidecar) setup, see Sjors/bitcoin#48. I'd rather not have to expand that c++ code to do even more stratum stuff.

[TheBlueMatt]

maybe to do something like that you should have an agreement with some big miner that trust you

Sadly doing MEVil that way is a great way for only the largest miners to survive and Bitcoin's decentralization to suffer :(. Luckily this isn't an issue we have today, but I imagine it will be one we will have over the next handful of years.

I agree this isn't exactly trivial, but at a protocol level its not that insane - the pool has to be active in the same (trusted) marketplace that the miner is using to and both have to accept transactions in a template which appear in the orderbook. If the miner finds a block, they can submit a solution to the pool and the pool can go to the marketplace and get the transaction data at that point to propagate the block.

None of this is a problem for today, in any case, I use this as an example of something that will probably happen on the horizon at some point that we should consider at the protocol layer, but doesn't need to be seriously considered at the implementation layer today.

[Fi3]

wouldn't be better having some open source thing that do this MEV(il) so each miner can just use it? The value will be extracted directly from the miner or it will be the pool to extract it and the redistribute it to the miners? Is there some resource about MEV(il) in bitcoin?

[TheBlueMatt]

Yes, I imagine the client software for such a marketplace should be open source. Sadly, I think it really has to be done wherever the template is built, and if that's on the pool we're back to square one with centralization :(. I don't see a way to build this in a way that doesn't have a trusted marketplace to provide transaction blinding/hiding, but maybe that part of the marketplace isn't big enough I dunno. I don't know of great resources because its all really early, it really requires understanding the MEV world in ethereum and then trying to squint and map that onto what people are talking about doing in Bitcoin. I did write a rant a while ago at https://bluematt.bitcoin.ninja/2024/04/16/stop-calling-it-mev/ but it doesn't really explain it in detail.

[Fi3]

ok I will look into it when I have more time ty

[TheBlueMatt]

I'm also being somewhat optimistic in how these marketplaces develop (right now they're really painfully naive). They don't exist yet, and they're quite a ways from existing. I'm push people to move in that direction though.

[plebhash]

after some reflection, my current stance on this discussion is NACK

this move feels heavily centered around Bitcoin Core politics

IMO Sv2 spec should be implementation agnostic, and that includes the Template Provider

assuming that only Bitcoin Core could act as TP is a heavily centralizing design decision, which would ossify the protocol around Bitcoin Core

with that said, @Sjors work on maintaing such sidecar implementation alongside a Bitcoin Core fork remains foundational for the progress of SRI

this strategy will allow us to stabilize Sv2 Spec sooner rather than later