Re-enable LES as option

[oskarth]

(Skeleton issue)

Problem

Currently LES on-device node is disabled. This means we are relying on either Infura or custom RPC. Even if we can't get great perf it is probably worth it to have as an option, especially as ULC is still being developed. Otherwise we literally don't have a light client in the app and we are relying on trusted third parties.

Implementation

We need to figure out the following things:

• 1. Write a piece of code that can generate new CHT roots,
• 2. Provide a secure and distributed way of broadcasting found CHT roots (to make sync as fast as possible),
• 3. Run some LES servers in our DC (probably need a combination of an LES server connected to the Ethereum network + local LES server to provide connection slots for the Status app clients).
• 4. Update status-react with it new status-go version and as an option

Acceptance Criteria

Points above addressed.

Notes

https://github.com/status-im/status-react/blob/develop/doc/decisions/0003-geth-node.md

https://nakamotoinstitute.org/trusted-third-parties/

https://github.com/status-im/status-go/wiki/Updating-CHT-BloomTrie

#320

ethereum/go-ethereum#15673

Future Steps

[oskarth]

Additionally, consider if we can have support for all of the following options:

• Infura
• Custom RPC
• LES
• ULC

Until ULC or whatever tech turns out to have the sweet spot of properties in terms of security and perf.

[oskarth]

Inspiration from Vitalik, 100% agree: https://twitter.com/vitalikbuterin/status/1010504159211282433?s=21

Twitter

Vitalik "Not giving away ETH" Buterin on Twitter

“Wishlist: a blockchain node that only syncs when either (i) the app was used in the last N minutes or (ii) the laptop/phone's battery is charging. Would substantially improve the user experience of using a partially or fully verifying node for one's day-to-day transacting.”

[shazow]

Even if we can't get great perf it is probably worth it to have as an option

What kind of perf goals are you looking for?

I'm working on vipnode as part an EF grant and with whitelisted static nodes I'm getting very good light client connection times (well under a second).

This could be a good compromise between relying on centralized providers vs going through the full dicsv5 dance which can take anywhere between minutes-to-hours to find an open spot.

[adambabik]

@shazow that's one issue. Another thing is data volume needed to be synced. One solution is to provide a secure way of updating CHT roots so that the sync is minimal. Another interesting case is what Vitalik said in the tweet posted above by @oskarth. Actually, that sounds like a great phase 1.

vipnode as an option also sounds good. We will definitely consider it.

[shazow]

Another thing is data volume needed to be synced.

Right. IIRC that initial sync data is on the order of ~200mb? In the grand scheme of things, that's on the order of a small-medium sized mobile game these days.

One solution is to provide a secure way of updating CHT roots so that the sync is minimal.

That is an interesting avenue! Not sure how much work that would be.

[divan]

There is a neat idea posted one day by @dshulyak: run trusted servers (we have them in our cluster) and broadcast new CHT/BloomTrie roots via Whisper, encoded by private key(s). So the client just have to subscribe to the given topic and have the corresponding public key(s) hardcoded to decrypt the CHT broadcast.
I think it's worth to evaluate this approach.

[dshulyak]

that is certainly possible, however it makes our clients even more reliant on our own infra, which is probably not a good thing (especially for the chain).

i would start by outlining actual problems:

1. it was pointed our that ethhash nonce verification is cpu consuming. however benchmarks in geth are showing the following:

BenchmarkHashimotoLight-8   	    1000	   2053521 ns/op

and for example encryption that we use in whisper

BenchmarkEncryptionSym-8    	     200	   8640843 ns/op
BenchmarkEncryptionAsym-8   	     200	   5550396 ns/op

i might be missing something (or benchmarks are not correct), but according to those benchmarks - nonce verification is not a burden at all in comparison to whisper usage

so, i would measure what is actual battery impact, and maybe we can use les as is without ulc and thus avoiding any centralization costs

2. storing chain headers

what is the amount of new headers generated every week? do we have pruning implemented in geth client?
we might not even need to update cht dynamically if pruning is implemented and we release new client quite often

3. finding free slots for les connection

3.1. as it was pointed out it is impractical to use discovery v5 for les peers, as it can take long time, which is obviously won't work for mobile. we will have alternative - rendevouz discovery, but we need to understand how to register les servers that use original geth client

3.2. free slots for les peers

vipnode might be an answer, but i don't know any details

[shazow]

3.1. as it was pointed out it is impractical to use discovery v5 for les peers, as it can take long time, which is obviously won't work for mobile. we will have alternative - rendevouz discovery, but we need to understand how to register les servers that use original geth client

Even if you start a light client with some static node(s), the client will still use discv5 to find additional nodes after the fact (unless run with --nodiscover). If it's desirable to mix trusted nodes with random peers (to reduce trust and allow consensus confirmation), then that already works out of the box. If not, then it's easy enough to disable or limit the number of peers to something small.

vipnode might be an answer, but i don't know any details

Happy to share details! I wrote a high-level post about the goals/roadmap here: https://medium.com/@shazow/an-economic-incentive-for-running-ethereum-full-nodes-ecc0c9ebe22

It's all open source being built in the open here: https://github.com/vipnode

My current timeline is end of August to have a working end-to-end implementation of the pool design that the blog post talks about. I'm happy to expand if you have any questions!

Medium

An economic incentive for running Ethereum full nodes

This is a story about vipnode, an Ethereum Foundation Grant recipient.

GitHub

vipnode

Economic incentive for running an Ethereum full node.

[adambabik]

@dshulyak that's a great approach!

it was pointed our that ethhash nonce verification is cpu consuming.

I think it was a guess. Also, LES received many updates since then. However, as a combination of many operations and large number of packets, it still may perform poorly in terms of battery consumption, at least during the initial catch-up.

we might not even need to update cht dynamically if pruning is implemented and we release new client quite often

Pruning does not work this way that it removes old data? So still it would start syncing from CHT Tn until Tm but it would prune old states [Tn, ..., Tm-k]. The stored data volume should be smaller but transferred number of bytes will stay the same. Or am I confusing how pruning works?

From my perspective, only 3. finding free slots for les connection might be a hard blocker for releasing LES in phase 1. CPU usage or network data transfer might not be a problem if we properly inform the user about pending sync and recommend doing that only when connected to a WiFi. After the initial sync is done, the performance should stabilize (all that is a guess :)).

In the next phase, we can try to provide a dynamic way of updating CHT roots.

[adambabik]

@shazow in our setup, we may take an advantage of finding the random LES peers using server nodes and Discovery V5 and pass found peers to mobile clients using more efficient rendevouz peer discovery protocol. Still, the problem will be whether that found LES server has free slots and will accept our connection...

Which brings us to reserving LES server slots. Thanks for sharing the blog post about vipnode! It's very interesting. As I said before we will definitely evaluate it when the project kicks off which should happen pretty soon.

[shazow]

Which brings us to reserving LES server slots.

If you're already running your own full nodes, you could whitelist all Status client node IDs on your cluster. I have a PR pending for geth that exposes an RPC to do this without restarting the server: ethereum/go-ethereum#16333

If you wanted to roll your own, the flow would be: User opens Status client for the first time, node keypair gets generated, which pings your coordinator API to instruct your cluster of private nodes to whitelist the nodeID. The API could also return an updated list of static nodes that the client does admin_addPeer to some random subset.

In vipnode, a "pool" is the coordinator API, and clients call vipnode_register (associates a nodeID to an account, which you could largely skip) and vipnode_connect (triggers whitelist on a subset of nodes and returns the first N nodes that succeed).

I'm definitely happy to ping again once everything is functional but if you're seriously considering using this design then I'd love to coordinate on the design to make sure your use cases are fully covered and that the API is not too burdensome. There is some discussion here: vipnode/vipnode.org#10

[dshulyak]

hey @shazow . are you still around?

I wanted to discuss a couple of concerns:

1. how to discover vipnodes? and when they are discovered how to understand if the given vipnode requires a payment?

current documentation seems to assume that every vipnode must be known in advance. it means that either we will have to hardcode such nodes or offload node selection to a user.

even if it will be discovered - there is no way to understand if it requires payment or it is just
a regular node that is out of slots for connections. current disconnect reason doesn't provide enough information to make an informed choice.

I see that current solution (run.js script) runs alongside ethereum node (requires only admin API). but it seems that ideally more changes in the client itself required. because there should be a way to negotiate that a node requires payment using extended handshake or another dev-p2p subprotocol.

was there any plan on making vipnodes discoverable?

2. vipnode is prone to fraudulent behavior.

one simple scheme that came to my mind, if i am running vipnode and i received enough payments - i can simply create a new identity and stop serving clients that already paid me. this way i will keep tokens that i already earned and can earn even more from an exact same set of clients.

of course, this is not that big of a problem if every vipnode needs to be known in advance (effectively it means that node must be trusted, which is one more thing that we want to avoid).

ideally, there should be a way to make fraudulent behavior verifiable and require a stake from an identity that provides LES service, so that in case of fraud was proven - stake will be slashed.
but there is also simpler way, make identity non-zero cost by requiring a non-redeemable stake. which will make scenario from above less likely.

what is your opinion on this matter?

[shazow]

@dshulyak

1. how to discover vipnodes? and when they are discovered how to understand if the given vipnode requires a payment?

Your understanding is correct.

Quick distinction: Let's call these Vipnode Pools. The nomenclature I've been using is Clients -> Pools <- Hosts, where clients are routed to hosts via pools.

The goal is not to have vipnode pools be homogeneous. In fact, part of the desired outcomes is to have a variety of pools out there with a variety of properties/guarantees/pricing schemes/etc.

As far as user experience goes, I imagine different wallets will choose which vipnode pools to provide by default, but it would be nice if the user was able to override it with their own pool of choice. Payment/membership can be managed out of band for non-default choices.

current disconnect reason doesn't provide enough information to make an informed choice.

This is true. It's on my TODO to investigate patching geth to provide better disconnect reasons. When using the vipnode client handshake to connect, the whitelist preparation happens separately from geth/parity so you will get better error messages in that case at least. (There's a sketch of an example of what that might look like here)

because there should be a way to negotiate that a node requires payment using extended handshake or another dev-p2p subprotocol.

I think it's premature to agree on a payment handshake (or even a payment scheme). One of the goals of vipnode is to help flesh out the possibilities here, and if usage starts to converge then we can draft an EIP to formalize it. I can imagine very many payment schemes here and it's not obvious to me which will win (e.g. per time period, per transaction, per rpc operation, per cpu time used, per bandwidth used, per DApp interacted with, etc).

2. vipnode is prone to fraudulent behavior.

That is also correct. I acknowledge in the design that this is a trusted model similar to how mining pools work. You can join a mining pool, but ultimately you're trusting the pool operator not to run away with your money.

I want to explore using micropayments in a future version, but unfortunately many-to-many micropayments are very expensive (settlement fees often exceed the outstanding balance or you need collateralized hubs like Lightning Network).

The current design I'm working from (v2, the draft contract is here), it's kind of a trusted payment channel that allows the pool to aggregate payments efficiently. The cap of the damage here is the channel balances that are unsettled.

An incremental improvement would be to do proper microtransactions that bounce from the client -> host -> pool, which would reduce the damage cap to the used channel balances that are unsettled and would provide a verifiable "receipt" by hosts.

ideally, there should be a way to make fraudulent behavior verifiable and require a stake from an identity that provides LES service, so that in case of fraud was proven - stake will be slashed.
but there is also simpler way, make identity non-zero cost by requiring a non-redeemable stake. which will make scenario from above less likely.

That might be the way forward in the future, if we can't make trustless p2p payments efficient enough.

I want to emphasize that part of the design is to allow different pools use their own arbitrary payment schemes--not necessarily by smart contract. You can imagine a scenario where people pay with credit card for a subscription, for example. I also imagine there will be many pools that are largely free within some quota (kind of like Infura today, but many mini-Infuras instead) or within some authorized user set (e.g. major wallet could be funding their own pools and giving their own users priority access).

Let me know if I missed anything. :)

[dshulyak]

@shazow

thank you for clarifying design decisions, to repeat some points:

1. vipnode pools will act as a discovery entrypoint, and as long as there are enough capacity in the pool - single failure of the any vipnode won't have a significant impact on the client that paid tokens to the pool

also it will be more efficient then discovery v5. we recently added simple registry for ethereum nodes, because it was taking too long to find nodes with rare topics in ethereum discovery network.

actually information can be advertised using whisper (what we use at the moment) or stored in swarm, but this is also not for now.

2. every vipnode pool will have implicit/community reputation, and client/wallet that makes a choice to use a certain pool will have to be aware of that reputation.

it is desirable to make vipnode pool behavior verifiable and punishable, but i agree that it makes sense to postpone such, non-obvious, things.

I think it's premature to agree on a payment handshake (or even a payment scheme).

if it is possible to negotiate that payment is required using single message delivered when node was disconnected, then certainly sub-protocol/extended handshake is not required for that.

but there are also other benefits for adding payments sub-protocol:

An incremental improvement would be to do proper microtransactions that bounce from the client -> host -> pool, which would reduce the damage cap to the used channel balances that are unsettled and would provide a verifiable "receipt" by hosts.

such receipts can be delivered, using discussed sub-protocol.

i like receipts idea, and it might be even highly necessary for good UX. currently payment, that includes client identity, must be committed before initiating a connection. it is even bigger problem if mobile device is used. one way to solve it is to pay for connection using infura and then switch to vipnode. but this is definitely not optimal and may not work at all in certain cases.

so, if we can deliver a payment using off-chain channel then the problem is solved. maybe it can even go directly to a vipnode pool service.

I want to explore using micropayments in a future version, but unfortunately many-to-many micropayments are very expensive (settlement fees often exceed the outstanding balance or you need collateralized hubs like Lightning Network).

did you have a chance to evaluate probabilistic micro-payments?
they are going to be used in orchid network for quite similar purpose (paying for bandwidth) and authors claim that they are way more efficient than using state channels for discussed purpose.

https://www.orchid.com/whitepaper.pdf
https://medium.com/@gustav.simonsson/ethereum-probabilistic-micropayments-ae6e6cd85a06

is there a vipnode-pool API implementation that can be used? i can make a PoC integration. ideally i also need a solution for delivering a payment off-chain, but for the start infura can be used. any downsides if it will be done using client -> pool ? in brief - client will sign a receipt and send it to pool API, which if claimed successfully will guarantee access to nodes managed by that pool.

UPD:
i should have notice earlier that pool implementation is already available (https://github.com/vipnode/vipnode). I will try to play with it this weekend. Another thing that i will add, just for experiment for now, is similar to https://github.com/vipnode/vipnode-contract/blob/master/contracts/VipnodePool.sol#L73 but one that can be offloaded to vipnode pool to execute. Effectively it will just derive msg.sender from signature over hashed value and nodeid.

Medium

Ethereum Probabilistic Micropayments – Gustav Simonsson – Medium

Most Ethereum projects in need of micropayments have, so far, focused on payment channels. Payment channels allow for the sending of…

[shazow]

currently payment, that includes client identity, must be committed before initiating a connection. it is even bigger problem if mobile device is used. one way to solve it is to pay for connection using infura and then switch to vipnode. but this is definitely not optimal and may not work at all in certain cases.

Indeed. Another option is pools can offer a brief trial per nodeID (maybe something like 10-30 minutes, enough to sign up).

did you have a chance to evaluate probabilistic micro-payments?

I have looked at that, it does look interesting! Looking forward to seeing more fleshed out implementations.

is there a vipnode-pool API implementation that can be used? i can make a PoC integration. ideally i also need a solution for delivering a payment off-chain, but for the start infura can be used. any downsides if it will be done using client -> pool ? in brief - client will sign a receipt and send it to pool API, which if claimed successfully will guarantee access to nodes managed by that pool.

i should have notice earlier that pool implementation is already available (vipnode/vipnode).

That code is still early/not fully functional yet. I'm aiming to have a mostly functional implementation by the end of the month.

I'm still fleshing the finer details of the protocol (currently JSONRPC2.0 but considering switching to gRPC) but the high-level design is here:

From the client's perspective, there are only two simple calls that are of concern:

• vipnode_connect which prompts the pool to allocate some hosts and return them. This can also be used to negotiate payment details, if the existing balance is insufficient.
• vipnode_update which is a periodic keep-alive that helps the pool keep track of which clients are speaking to which hosts (and bill them accordingly).

All requests are signed by the client node's nodeID private key, to avoid forgery. Details of the current signing scheme are here: vipnode/vipnode.org#10 (comment) -- if you're using Go then you can use vipnode's code to do all the requests and signing:

• https://godoc.org/github.com/vipnode/vipnode/request for the lower-level signing
• The code for the client dialing to the pool is still incomplete (working on it this week) but it will ultimately live in main.subcommand(..) somewhere.

request - GoDoc

[dshulyak]

Indeed. Another option is pools can offer a brief trial per nodeID (maybe something like 10-30 minutes, enough to sign up).

there is an additional risk in doing so. any client (status included) will be able to abuse this property, and create new identity every 10-30 minutes.

i have a following proposal in mind:

Vipnode pool will act as a proxy to ethereum for deposits (addBalance). I sketched how it will look like both on client and server here: gist.

It will proxy only specific transactions that match following criteria:

1. it must be sent to a vipnode pool contract
2. message must be signed with key that owns node id that passed to addBalance
3. deposit must be greater than zero

Request itself will have 2 or 3 fields. Depends how signature is verified.

• MethodName - vipnode_bootstrap
• TX - signed ethereum transaction that can be sent to tx pool using sendRawTransaction method
• NodeID is optional, cause it can be derived from signature, but i can also stick with the method you are currently using

In general vipnode pool should be interested in such bootstrap procedure, cause it is safe, and have explicit financial incentives. What do you think about this proposal?
I could add this to vipnode code, but looks like it lacks connection to an ethereum node at the moment. so I will wait for that.

[shazow]

there is an additional risk in doing so. any client (status included) will be able to abuse this property, and create new identity every 10-30 minutes.

Indeed but I suspect that's a relatively low risk/low damage. I imagine it like the Netflix model--convenience from paying a small fee outweighs the savings of piracy for many people.

i have a following proposal in mind: ...

That sounds reasonable also, though I suspect just sending a blind transaction won't be sufficient for some edge cases. Clients would need to sync their state first (e.g. at least to figure out which nonce to use).

In general vipnode pool should be interested in such bootstrap procedure, cause it is safe, and have explicit financial incentives. What do you think about this proposal?

I am not opposed! :) It is definitely more complex than a brief trial period, code-wise, but I'm happy to support this usecase and I'd be happy to work with you on this!

I haven't gotten to the payment piece yet but my goal is to make it modular so I'll definitely keep supporting this scenario in mind. I opened an issue to track it here: vipnode/vipnode.org#14

I could add this to vipnode code, but looks like it lacks connection to an ethereum node at the moment. so I will wait for that.

The code for the host and client establishing local node connections is there. findRPC is what does it: https://github.com/vipnode/vipnode/blob/master/main.go#L138

It sits behind https://godoc.org/github.com/vipnode/vipnode/ethnode#EthNode to normalize between Geth and Parity.

But I haven't finished wiring up all the pieces beyond that yet. I'm hoping to have a partial working demo by the end of this week (probably skipping payment/balance tracking for this milestone).

GitHub

vipnode/vipnode

Command-line pool server, host server, and client for vipnode.

ethnode - GoDoc

[shazow]

@dshulyak Hey there, quick update: vipnode v2 is close to getting finalized. Bulk of the work is complete, just the payment mechanism is still missing. I have a binary pre-release here if you'd like to give it a try (feedback appreciated!): https://github.com/vipnode/vipnode/releases

I'm in the process of documenting some stand-alone examples of how to implement a third-party client in a language-agnostic way. The Go code for it lives here: https://github.com/vipnode/vipnode/blob/master/client.go

GitHub

vipnode/vipnode

Command-line pool server, host server, and client for vipnode. - vipnode/vipnode

GitHub

vipnode/vipnode

Command-line pool server, host server, and client for vipnode. - vipnode/vipnode

[oskarth]

Also see https://slock.it/incubed.html and https://hackmd.io/6IhsF6zRSqmtggvNnp1sHQ?both re ULC economic model - we have a lot of interesting paths forward here. After messaging reliability, getting solid light client support (regardless specific manifestation(s)) is likely to be a top core priority.

Slock.it - Client

Slock.it

On ULC incentives - HackMD

[ghost]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oskarth]

keealive

(backlog of core team iirc @mandrigin)

[oskarth]

Slock it would be useful as another iteration of network incentivization (along with PN work), once research for initial messaging/offline is done.

cc @cammellos @mandrigin

[ghost]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oskarth]

@mandrigin how does this currently fit into roadmap, especially with respect to bounty etc? I still think it'd be great to have a battery-eating heater in your pocket as an option. I don't understand why this isn't a bounty yet either.

[mandrigin]

@oskarth we have it enabled in one of the fleets, but the problem was usually that the initial sync never completes... So we can't call it "working". :(

[mandrigin]

and there used to be not enough server nodes to support LES in Ropsten, and that requires our infra access idk how to make a bounty out of it
granted, it was a while ago

[oskarth]

@oskarth we have it enabled in one of the fleets, but the problem was usually that the initial sync never completes... So we can't call it "working". :(

Isn't that a bounty? We don't need to know exactly how to solve it to state a problem, acceptance criteria and put some $ on it for community people.

and that requires our infra access idk how to make a bounty out of it

Maybe we can see how people creatively solve this, then discussion can continue as people hone in on solution space. Doesn't have to come from us, but we have to reach out a hand.

[mandrigin]

@oskarth yeah, I think you are right, I’ll put it on my TODO list to create a bounty issue for LES and I’ll link them to this issue and close this one afterwards.

[oskarth]

^ fyi @andremedeiros @rachelhamlin @adambabik

This was completely dropped, while it likely isn't top 1 or 2 prio for core app for the next few months, it's quite embarrassing how much we have regressed with this in the last 2 years. At a minimum having a bounty up for this would be useful.

Also even more relevant now than before considering Infura soon starting to charge money.

[oskarth]

Also see latest geth release perf wrt to this, plus new iterations of VIPNode, etc.

[oskarth]

^ @cammellos just a fyi ping and putting this on your radar as VIPNode is still an option here, especially considering recent Infura limits.

[shazow]

For what it's worth, Vipnode development is paused for now. I'm trying to decide on what the future of the project will be, input is very welcome.

My main concerns is that LES is still not enabled by default in any of the major node implementations, and doing many-to-many micropayments is still a difficult problem (though zksync is getting closer).

[ghost]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oskarth]

I guess this issue now should evolve into supporting the Portal Network that Nimbus is contributing to in collaboration with other Ethereum teams.