Sync Server Scalability Issues

[jericopulvera]

1. Data size and server capacity: When syncing large amounts of data, there is a risk that the sync server may struggle to handle the load efficiently. This can result in slow sync times, timeouts, or even server crashes. Additionally, the cost of running and maintaining a server capable of handling large data sets can be significant.

2. Lack of data deletion API: Without a dedicated API for deleting data, managing and removing unwanted or outdated data becomes a challenge. This can lead to unnecessary data accumulation and potential performance issues.

3. Lack of sync status indication: The absence of clear indications regarding the success or failure of sync operations poses a risk of data loss. Users who are not familiar with web technology may not be aware of potential issues during the sync process, leading to data discrepancies or unintentional deletions.

---

For Data Size issues, is it possible for us to send only the changes made for syncing, as to not overwhelm the server when data is too big? It might also be good to have an import-export data functionality, as to avoid going through the server at all and just go straight peer to peer.

[steida]

Data size and server capacity

Evolu.world server will handle a lot of data, and we will use rate limiting ofc.

Lack of data deletion API

That's planed feature. TLDR: There will be hashes and hashes of hashes to detect and delete data.

Lack of sync status indication

There is SyncState.

[ariofrio]

As for

For Data Size issues, is it possible for us to send only the changes made for syncing, as to not overwhelm the server when data is too big?

I think what the Merkle tree does here, right?

[steida]

For Data Size issues, is it possible for us to send only the changes made for syncing, as to not overwhelm the server when data is too big?

This is going to be improved a lot.