Discussion: signal/noise ratio too low?

[djc]

From the rkyv maintainer:

I think the advisory database has a signal-to-noise ratio that is too low. While there are many legitimate advisories, there are also plenty of advisories that are just pedestrian bugs without a specific impact on security. If codebases have an "attention budget" for preventing vulnerability exploitation, advisories act like a heuristic for where to allocate that attention. The bar for moving that attention to specific issues should be relatively high because we already have good general-purpose heuristics to follow. Using this issue as an example:

• The threat model is not well-defined. An attacker would have to be able to OOM a process as this specific subroutine is being executed. OOM-ing before would most likely cause the process to safely panic, and rkyv itself does not provide any kind of mechanism for causing an OOM.

• The actual outcome is undefined behavior. UB can lead to a vulnerability alone, but it's a very broad class of errors and many of them do not have that kind of potential. Here, a null pointer gets turned into an out pointer and written to. In most situations, that will become a segfault. rkyv doesn't guarantee panic-freedom, so just halting isn't a security issue. Obviously there's no guarantee that the compiler will actually produce code that does this, but...

• There is no proof of concept. This would be a very difficult bug to trigger even under controlled circumstances, and if you can't cause this bug even with everything going your way then I don't think it's worth bringing a lot of attention to.

Overall, I'd consider this the equivalent of a "bogus CVE" for RUSTSEC (as RUSTSEC is not a CNA and does not issue CVEs). The incentives leading to bogus CVEs are systemic, and I won't waste time fighting them when it doesn't make a difference. I take security issues seriously and I do file CVEs to notify users of serious issues when warranted so they can protect themselves. But it's not my job to keep other people's vulnerability databases clean - if RUSTSEC wants people to use and rely on their advisories then the onus is on them.

There have been a bunch of high-volume lower-quality submitters active and I've been facilitating that a little bit (mainly because reviewing advisories is volunteer work and so I'm trying to limit the amount of time spent on it).

Should we establish/document some clearer guidelines to increase the bar, especially for higher volume advisory submitters?

cc @alex @tarcieri @Shnatsel

[alex]

I think some clearer guidelines are good. That said, something that triggers UB seems like it'll always quality for a advisory.

Stuff I'd trim down: panics and other forms of "pure DoS" never get an advisory unless the crate explicitly documents that's part of their threat model.

[djc]

Agreed that triggering UB should always be in scope.

For panics (not sure what other common forms there are of "pure DoS") I think explicit documentation of a threat model is too high a bar and instead maybe come up with some heuristics? Maybe how complicated/likely a triggering program would have to be, and whether triggering input is likely to be from outside the program (like a file or the network).

[alex]

OOM is another pure DoS category.

Basically what I'd say is: for stuff like these, it should only get an advisory if the maintainer says/documents that its a security issue. I think this is the best way to keep the SnR high.

[tarcieri]

I'm sad that people feel this way about RUSTSEC, as we have strived for a high SNR in the past.

For things that are just library-level advisories about unsoundness that lack a particular threat model, in the past we've added informational = "unsound" attributes which hide them from cargo audit unless explicitly enabled which should probably go on these. That lets us still track them, lets people who are interested consume them, but cuts down on alert fatigue.

Regarding panics/DoS, my personal heuristic has been is it something that sits at the network edge and is expected to constantly be interacting with untrusted data, e.g. format parsers. Things like that shouldn't panic because of the contents of the untrusted data they're interacting with, and should ideally be heavily fuzzed.

[shinmao]

Hi everyone, I am the one made the report and hope to join the discussion (where I am the culprit:). Thank you for the open discussion regarding the signal-to-noise ratio. As the reporter of the advisory in question, I’d like to share my original motivation and address the concerns raised.

My primary intent was based on a core tenet of Rust: Safe APIs should not trigger Undefined Behavior (UB) under any circumstances. I believe that even in edge cases like Out-of-Memory (OOM), a Safe API must maintain its safety contract. In languages like C, OOM often leads to unpredictable states, but the value proposition of Rust is that it provides a hard guarantee: even if the system fails or panics, it should do so safely without violating memory safety. If a Safe API can transition from a panic into UB due to the timing of an OOM, I believe that is a technical unsoundness that warrants reporting.

However, I take your feedback regarding "noise" and "alert fatigue" seriously. In hindsight, I should be responsible for explicitly labeling this as unsoundness to distinguish it from a vulnerability with a high-impact threat model.

I fully support @tarcieri’s suggestion to use the informational = "unsound" attribute for these types of issues. I will be more mindful of this distinction in future reports to help maintain a high SNR for the database while still contributing to the overall soundness of the ecosystem.

[djc]

@shinmao to be clear: you're not "the culprit". There have been a number of advisory authors recently that have submitted a larger number of advisories, some of which might not have been particularly interesting from a security perspective. The UB from the rkyv issue has definitely earned an advisory, as mentioned in earlier comments. Thanks for contributing to the RustSec project, and I'm looking forward to your future contributions.

[tbu-]

The UB from the rkyv issue has definitely earned an advisory, as mentioned in earlier comments.

I think it should be considered to assign issues like these an advisory that doesn't trigger issues in reverse-dependencies, e.g. informational = "unsound". As is, it seems unlikely that this is a security vulnerability, at most a DoS, if an attacker can cause the allocation to fail. But that would still be a DoS in the new code, so upgrading doesn't help here.

I assume there's no finding of a miscompilation caused by this missing null check.

[alamb]

From my perspective, the RustSec database is super valuable resource (thank you @djc and everyone else who maintains it) for helping keep the security of Rust based projects high.

TLDR I recommend only adding items to the the RUSTSEC database where there is a clear description of the vulnerability: what the result is and what circumstances are required to trigger the issue.

I think many projects use the rustsec database is as a sort of "blacklist" for crates with vulnerabilities that need to be updated ASAP. There are typically CI jobs setup that run cargo audit and when new entries are added to the rustsec database due to new vulnerabilities, CI starts failing and the engineers rush to respond.

For alerts where there is real potential safety concern, causing this downstream action is warranted.

However, if the 'advisory' is simply for a bug which isn't exploitable (for example, perhaps #2576) this causes lots of seemingly unnecessary downstream work, and I think that is where the source of the noise comes from.

Basically as a user, I only care about things in the RUSTSEC database that potentially compromise my applications security and that I need to take action to address.

[alamb]

I fully support @tarcieri’s suggestion to use the informational = "unsound" attribute for these types of issues. I will be more mindful of this distinction in future reports to help maintain a high SNR for the database while still contributing to the overall soundness of the ecosystem.

@shinmao and @tarcieri can you explain what outcome you hope to achieve by adding these advisories?

The primary result at the moment seems to be that it highly encourages / forces all downstream crates (including all transitive dependencies) to upgrade from the given crate. Given that the cost is high, I think we should only be adding such advisories where the old code is really causing a potential problem

[tarcieri]

informational = "unsound" advisories are not surfaced by cargo-audit by default. They require explicit configuration in audit.toml to opt into them

[djc]

cargo-deny still surfaces them unconditionally in modern versions, AFAICT.

[shinmao]

Hi @alamb , thanks for the question. My core belief is that Rust’s value proposition lies in proactive, compile-time prevention of memory safety bugs through the soundness guarantees of safe APIs, rather than reactive, post-hoc fixes.

In traditional languages like C, the workflow often relies on discovering an exploit after the fact, followed by a discussion between developers and users to determine who is responsible for the fix. In safe Rust, however, the "contract" is much stronger: as long as an API is sound, a user should be able to expect zero undefined behavior (UB) under any circumstances.

A great example of this is how Rust handles null pointers—by using the Option type to force explicit error handling at compile-time. This "proactive defense at compile-time" is exactly what I value in Rust, and I believe maintaining the integrity of that defense is why these soundness issues are worth documenting, even if they seem like edge cases. By documenting these soundness issues and pursuing patches, we provide a safety net for the ecosystem. Even if downstream maintainers initially defer an upgrade to avoid CI noise, they will have a clear reference and a ready-made fix available if they ever encounter bugs stemming from this unsoundness in the future.

[alamb]

Hi @alamb , thanks for the question. My core belief is that Rust’s value proposition lies in proactive, compile-time prevention of memory safety bugs through the soundness guarantees of safe APIs, rather than reactive, post-hoc fixes.

I agree (and it is one of the many reasons I enjoy working in Rust)

By documenting these soundness issues and pursuing patches, we provide a safety net for the ecosystem.

I also agree here that documenting the issue and pursuing patches are great (thank you for one you made in arrow-rs recently, for example)

Even if downstream maintainers initially defer an upgrade to avoid CI noise, they will have a clear reference and a ready-made fix available if they ever encounter bugs stemming from this unsoundness in the future.

I don't think the inclusion of information in the rustsec database always results in a "clear reference" (see for example on #2576 where it is not clear there is even any actual problem).

In my mind, entries in the rustsec database are most useful when someone actually encounters such a bug or there is a reasonable likelihood they would, or the symptoms if they were to do so are sufficiently bad.

The judgement of what constitutes a "reasonable likelihood" and "sufficiently bad" will always be a subjective and subject to debate, but I think the whole community would benefit if the threshold is higher than it is today.