Repository navigation
DSS: Heavy NRID subscription and eviction operations cause SCD latency spikes #1730
Description
What are the risks associated with hosting NRID and SCD within the same DSS? Industry has observed issues with high NRID subscription creation and NRID evict tool impacting latency for SCD. Any hypothesis here are welcome.
Activity
BenjaminPelletier commentedon Sep 29, 2026
Using the same CockroachDB-DAR-store DSS pool to provide both NetRID and SCD means the following resources are being shared (and perhaps more I haven't identified explicitly):
Any time a resource is shared, there is some potential for one service to impact the other. That could look like:
That said, I don't think I have ever seen a deployment where any of the limits above were hit, except possibly database connections with certain settings. I believe the binding constraint I have seen has always been contention between database transactions, and that type of constraint should not apply to NetRID and SCD interacting since both use entirely separate database tables.
So, I would be surprised if NetRID interactions impacted SCD latency (though of course I wouldn't rule it out!). But, it seems like a diagnostic test should not be too difficult: apply a steady-state SCD load (perhaps with Locust via loadtest, or benchmarker) and then perform a NetRID eviction half way through observing the load. If there is a change in SCD performance matching the timing of the NetRID eviction, then that is a strong candidate -- repeat once or twice, and then file an Issue with reproduction instructions.
Apart from NetRID eviction, are there any other demands on the system that might trigger on a schedule similar to NetRID eviction that would perhaps cause someone to confuse them with NetRID eviction?
utmverse commentedon Sep 29, 2026
Would it be reasonable to investigate this from the backup angle as well? It looks like hourly backups are running, so it may be worth checking whether the memory increases correlate with the backup windows.
github-actions commentedon Oct 13, 2026
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.
wing-utm-sharing-airspace commentedon Oct 28, 2026
One example concern here is that large NRID evict runs generate many CRDB write intents whose cleanup can spike heap usage and trip the Go GC on shared nodes. Industry ended up setting
kv.intent_resolver.batcher.in_flight_backpressure_limit.enabled = trueto mitigate some of this.In theory, NRID and SCD shouldn't impact each other, but that all relies on carefully choosing the right deployment setup and settings, which is where the risk comes from.
The ask here is to better understand the overall risks and potentially provide better guardrails. For the evict tools example, there are already useful discussions happening in #1717.
barroco commentedon Nov 2, 2026
We are looking to follow up on this issue and #1833 this week.
One of the main benefits of the raftstore implementation is gaining deterministic control and predictability over how the DSS processes requests. Relying on opaque underlying database heuristics and fine-tuning always introduces a degree of uncertainty and side effects. Without strict control over request processing, it is very difficult to guarantee workload isolation other than separating services on different DSS pools.