replace websocket-star rendezvous with a better system

[fazo96]

The websocket-star rendezvous is a service with no optimization, which is necessary for every other service to work, and it can't be scaled because you can't have multiple instances. This is a problem when going in prod.

We need to replace this with another system: the best solution that we can implement now is having multiple collectors and have them reachable via TCP and Secure Websockets. The browser will have these in the bootstrap list via websocket and the backend services will have them via TCP.

This way, we can just add collectors to scale. However, no DHT means that the nodes won't connect to each other automatically and we might need to manually configure them to talk to each other.

[kulpreet]

If we use go-ipfs for collector, will this help at least the collectors discover each other over DHT?

Won't solve the problem completely, just thinking of options.

[fazo96]

yes that is another possible solution. If we use go-ipfs on the backend we need to make sure it works right with our application and that it interoperates well with js-ipfs

We also need to be super sure that we pin everything we need because go-ipfs has the garbage collector

[fazo96]

also if we use go-ipfs we can leverage the prefetch option of js-ipfs in the browser to make it even more guaranteed that IPFS data created in the browser gets cached in our backend.

[kulpreet]

I like the sound of the prefetch option.

Can't we disable GC in go-ipfs? Will a go-ipfs node gc data it has itself pinned? I would guess not, right?

[fazo96]

No, everything that is read by a node is automatically cached (always).

Then, when the cache gets too big the GC runs (only in go-ipfs for now because js-ipfs doesn't have the GC yet) and deletes some old data. The GC never deletes pinned data, that is the whole point of pinning. If there is no GC then pinning does nothing because it's like everything is automatically pinned