Filecoin Proofs C++ Implementation

[nemothenoone]

Hello! This is a discussion topic to support the Slack thread in here: https://filecoinproject.slack.com/archives/CEGB67XJ8/p1614080944002500. For the start, I'ill just copy an introduction message from there to here.

What would you say about us open sourcing alternative C++ Filecoin proofs implementation done from scratch? Well, not actually open sourcing, but assembling the suitable rust-fil-proofs/bellman/all the other Rust-y parts C++ replacement from the proprietary code we helped to run mining operations with for the last half of a year? Like a special open-source version of a proprietary product. Performance will be kept exactly the same with the PL’s implementation (that is discussable, but I needed an initial point to start the discussion from).

What is that about. Initially alternative protocol implementations were supposed to diversify the risks of something being wrong inside some particular single protocol implementation. Almost half of a year passed, we have several implementations in development, but they all rely on the same Rust-y proofs implementation. Probably, the time has come to get the second proofs implementation to have completely independent protocol implementations? And there is no cheaper/faster way to do that, but to opensource something. So here I’am. Asking if there is any interest in that. I’m especially interested in Protocol Labs folks opinion and in cpp-filecoin implementers opinion. Having an independent second community-based proofs team would be nice I think (just wondering). Tag anyone you think related to this discussion. Thanks.

[porcuquine]

I think this is a good idea in principle — for several reasons. The devil would be in the details, though. We would need to take more than usual care to ensure the implementations are consistent. We would at least need some process for dealing with security fixes. It also seems unrealistic to expect a second implementation to always keep parity with rust-fil-proofs, as things like proof aggregation, new proving systems, and eventually entirely new proofs are released. The problems trying to make this work would force us to solve are probably good ones to have answers to. However, it would probably make sense to ensure users of an alternate library also had a path to fall back to rust-fil-proofs when either the alternate implementation had not yet implemented something (as above), or if for whatever reason some divergence is discovered which threatens the consistency of the chain. We should think hard and in advance about how we would both detect and deal with such an event.

[nemothenoone]

@porcuquine Luckily most of these questions already have their answers developed over time.

1. Keeping up with the initial implementation updates. Well, right now we're a little bit back on proof aggregation implementation (have not started it yet actually), but since components we want to assemble the C++ proofs implementation from are being used for actual mining operations, we have to keep them updated. It is hard I gotta say, sometimes delays happen, but doable.

2. The fallback option. Initially we maintained the Lotus fork of our own (we even called it POTUS, yeah), but then the overall solution architecture changed (to the distributed one) so we gave up on maintaining that fork and ended up with providing the FFI submodule with the ABI compatible with an initial one. So the whole fallback option (for Lotus, Forest and go-filecoin) can be done exactly like that. With providing the way to simply select different FFI submodule for the compilation.

   Other way to do that - we could revive our Lotus fork, make it public (with no changes in comparison to the official one) do the same with other implementations. It makes sense to do so with Golang-based implementations, (Rust-based implementation would be more comfortable to stay with rust-fil-proofs I think, but FFI-based integration can be done for it as well). So there could exist at least two versions of Golang-based implementations. Doable as well.

   The whole fallback process is going to be different with cpp-filecoin implementation. We would like to make its integration more tight, not via the FFI-compatible interface (because it would bring more troubles than benefits in case of cpp-filecoin). This will definitely require a tighter collaboration. But that works for us as well. The second version could be introduced in case of need.

3. Troubles which can threat the consistency. There was no such case yet, but in general I see no other option, but to perform extensive testing with bigger tests set.

4. Security reports. We can set up a public PGP email address for the community to be able to report vulnerabilities directly to us in case it is known the trouble is about C++ proofs. I also think more closely-tight security-related collaboration will be probably required within the Filecoin Security group in general to figure out IF the particular reported issue is connected to C++ proofs implementation.