Cargo RFC for min publish age

[tmccombs]

View all comments

Important

When responding to RFCs, try to use inline review comments (it is possible to leave an inline review comment for the entire file at the top) instead of direct comments for normal comments and keep normal comments for procedural matters like starting FCPs.

This keeps the discussion more organized.

Summary

This RFC proposes adding options to Cargo that allow specifying a minimum age for published versions to use.

See rust-lang/cargo#15973

Rendered

[Noratrieb]

The RFC doesn't clearly lay out the expected way to deal with urgent security updates.
From my reading of it, I would deal with those by using cargo update --precise or putting it in Cargo.toml, getting a warning, and then after having the new version in the lockfile, everything works normally.
But this would mean that when regenerating the lockfile (when update --precise was used instead of Cargo.toml), this security update would be lost silently, which sounds suboptimal. Having an exclude array like pnpm does not have this problem.
Can you add why you decided against package exclusion configuration and whether this threat of regenerating the lockfile after a security update is important?

[epage]

But this would mean that when regenerating the lockfile (when update --precise was used instead of Cargo.toml), this security update would be lost silently, which sounds suboptimal

If there is something you depend on, you should raise your version requirement.

Having an exclude array like pnpm does not have this problem.

But that is a heavy hammer. It doesn't record why it was excluded and you can easily forget to un-exclude when it is no longer applicable. With raising a version requirement, you automatically start using the feature again with the dependency.

[tmccombs]

Having an exclude array like pnpm does not have this problem.

I did include that in the Future Possibiliites, but I could probably expand on that more.

[Noratrieb]

It is easier, yes, but more importantly having this in Cargo is the only way to have it actually be secure. As soon as you run cargo build after a malicious package has been resolved, the build is compromised.
This can't be anywhere but in Cargo's resolver to achieve the security benefits.

[tmccombs]

I think that is only true if you don't have a Cargo.lock file, or your Cargo.lock file is missing something from your Cargo.toml file.

But I did try to mention that in d78605c

[Noratrieb]

And how would it interact with important urgent security updates?

[tmccombs]

I've removed this from open questions, and left it as just a Future possibility for now.

[Noratrieb]

Another question worth pointing out: "Are we comfortable making the security guarantees that the build remains secure when a not-yet-of-age malicious update has been released?". I think the answer here ought to be yes, but that's a decision that should be made deliberately and knowingly.

[epage]

I worry when "making [a] security guarantee" on whether we are actually committing to that level (ie a report needs to be issued and a hotfix released).

[tmccombs]

Any such guarantee would need to be predicated on some conditions, including:

• The build was possible before the malicious release on the target platform and compiler version. I.E. there are no cases that require a fallback (a "deny" policy could potentially prevent a compromised build if that isn't the case, but that is currently put off as a future possibility)
• The malicious release happened more recently than the minimum publish age.
• Either none of the versions needed for a secure build have been yanked OR there a lock file is used.

[Noratrieb]

@rust-lang/wg-secure-code @rust-lang/security @walterhpearce y'all may be interested in this too

[clarfonthey]

(going to leave as a detached "review" since GH's system is bad, and I thought this would allow replies)

[Shnatsel]

My take from the wg-secure-code perspective is that the only solution to supply chain attacks is cargo vet/cargo crev, and everything else is snake oil.

The way forward is to stop whining and start using cargo vet instead of trying to pile on ever-increasing amounts of questionable heuristics.

[djc]

FWIW, I found the linked blog post to be fairly convincing that something in this direction makes sense. There is a very wide gap between (a) issuing cargo update sight unseen and (z) making sure all your dependencies are trusted by cargo-vet, and this seems like a decent middle ground.

[epage]

@Shnatsel

The way forward is to stop whining and start using cargo vet instead of trying to pile on ever-increasing amounts of questionable heuristics.

Please note that this is not a constructive way to engage with others on this topic.

[epage]

MSRV resolver doesn't apply to cargo install, so I assume it shouldn't apply here

Suggested change

	* `cargo install`
	* If a specific version is not specified by the user, respect `registries.min-publish-age` for the version of the crate itself,
	as well as transitive dependencies when possible.
	* `cargo install`
	* Unchanged to match the behavior of the MSRV-aware resolver

[epage]

Added a suggestion

Granted, rust-lang/cargo#16694 might muddy the waters here

[tmccombs]

Personally, I think that having this apply to cargo install as a way of preventing installing an executable package that has been published very recently is also very useful. Although perhaps users may want a different policy for cargo install than for updating libraries?

There are also some significant differences between MSRV and this:

• For MSRV, the minimum rust version comes from the package Manifest, whereas for the minimum publish age, it comes from Carog configuration, which may be set by the user themselves. It seems odd to me that setting global-min-publish-age in ~/.cargo/config.toml wouldn't apply to crates installed with cargo install and their dependencies.
• For MSRV and cargo install, generally the user doesn't care if crates are compatible with the minimum version in the package they are installing, so much as that they are compatible with the version of rust they are using to compile it? Does the resolver at least prefer crates that are compatible with the version of rust that cargo is using?

That said, I do think it would be surprising for a .cargo/config.toml in the package itself to impact the resolver when running cargo install.

So maybe we should specify that configuration from $CARGO_HOME/config.toml is used for checking minimum publish age in cargo install but not configuration from the package being installed itself?

[epage]

Personally, I think that having this apply to cargo install as a way of preventing installing an executable package that has been published very recently is also very useful. Although perhaps users may want a different policy for cargo install than for updating libraries?

We've already started down the path of documenting resolver as not applying to cargo install and I think it would be a problem to be inconsistent on that.

We can separately look into an install.resolver table.

Or we could look into more general fixes for cargo install so that the installed package is also checksumed, like installing dependencies.

That said, I do think it would be surprising for a .cargo/config.toml in the package itself to impact the resolver when running cargo install.

cargo install re-reloads the config from only CARGO_HOME

[epage]

I feel like there is content from the issue that isn't here, for example...

we should talk about the reason we are going with fallback and not deny is as a way to allow people to override this when there is a fix or feature they need. We've been experimenting with ways of doing this with cargo update --precise but if a newer transitive dependency is needed, you can't do that.

I think the thread also covered that fallback for this doesn't have the same problems as incompatible-rust-version because that suffers from a lack of data while crates.io exhaustively sets pubtime and so can any other registry that supports it.

We also had conversions on naming that isn't here. e.g. publish is intentionally in the name to help connect it to this only working on some deps.

[epage]

Also, I think we talked about the use case for exclude being lessened because

• per-registry support
• ability to override

[tmccombs]

I attempted to address this in 51e0a7c

[epage]

imo unspecified though I would put incompatible-rust-version at a higher precedence in the implementation so that people are more likely to have successful builds.

[clarfonthey]

Moving discussion from longer/less coherent thread: https://github.com/rust-lang/rfcs/pull/3923/changes#r2841703101

Rewording the arguments, I don't really think that this constitutes an empirical basis for the benefit of these delays. What would constitute a valid empirical basis is vulnerabilities that have been caught by this system after it's been implemented in other places. These should exist in practice since deployment of these mitigations is slow at best, and there should be cases where some people have the mitigations and some don't, and are thus vulnerable. But I haven't seen any so far.

The only argument listed is that the time between publication and mitigation is short, and so, if we delay deployment until a certain time after publication, then deployment won't happen before mitigation.

This really doesn't feel valid for one primary reason: people need to notice that there's a problem in order to mitigate it, and delaying mitigation also delays the time that people actually encounter an issue.

Like, my argument is that the vulnerabilities we should be worrying about are ones like the xz vulnerability, which fall cleanly into this class: they require an analysis of multiple moving parts and are only going to be noticed once someone actually has them running.

There are plenty of notable crates where an exploit would easily be noticed immediately and delaying dependency updates would catch them, in most cases. For example, if someone tried to release a malicious version of syn, we would definitely know almost immediately.

However, there are plenty of other situations where a vulnerability would not be noticed, because Rust's trait system lets you hide code in the most unsuspecting of places. Would you notice if one crate you used suddenly started relying on a particular hasher implementation that was promising but not as scrutinized, and then that hasher implementation was running malicious code? Would you notice if a reasonable-looking trait import also happened to silently change the behaviour of a method that looked fine?

The xz example is such a good example because it fits so well into so many examples. You don't have to be a bad actor hired by a nation-state to exploit a codebase; you just need to get your code in front of one tired maintainer who says "looks good to me" when you've snuck something suspicious in. Or, even just one compromised account: how many repos do you know are big enough to have several PRs per day, and how many of those do you scrutinize every PR?

Simply assuming that a timer starts ticking from release until someone notices a vulnerability is very naïve, because while that is definitely the case for obvious exploits like adding bitcoin_miner::run() to the top of a function, all it takes is one layer of indirection to hide that, and then it's not obvious at all.

There are definitely quick "hit-and-run" vulnerabilities that involve a compromised account that might be stopped by this, but so far we haven't seen any of those in the Rust ecosystem, and it seems like there are plenty of other low-hanging fruit with dependency updating that could be chosen instead of a simple time limit, especially considering how it can make security updates challenging to deploy.

There are a few other things discussed in other threads as ways of further improving the dependency-updating and dependency-locking systems, but here I wanted to specifically hone in on why I don't think that the data here creates a valid argument in this context.

[epage]

There are plenty of notable crates where an exploit would easily be noticed immediately and delaying dependency updates would catch them, in most cases. For example, if someone tried to release a malicious version of syn, we would definitely know almost immediately.

However, there are plenty of other situations where a vulnerability would not be noticed, because Rust's trait system lets you hide code in the most unsuspecting of places.

I'm a bit confused by this. As we said in the other thread, this isn't meant to be an exhaustive solution but one part of improving the whole and one we can deploy rather cheaply / quick for a quick improvement while we continue to work towards the larger improvements.

[woodruffw]

This really doesn't feel valid for one primary reason: people need to notice that there's a problem in order to mitigate it, and delaying mitigation also delays the time that people actually encounter an issue.

FWIW, the core thesis is that the relevant parties here are security scanners, not early victims. In the Python ecosystem, for example, the overwhelming majority of malicious package reports comes from automated static analysis, not from users who have already been victimized.

(And therefore it's not that people need to notice a problem, it's that systems need to be in place to monitor for problems and people should generally wait for that monitoring rather than acting as ecosystem canaries themselves.)

[clarfonthey]

That was not clear from the article, at least, although I definitely was biased against the article despite the fact that you could not have more clearly stated that the problem itself is overhyped. That definitely changes the situation in a way that should be reflected more clearly in the RFC IMHO.

[woodruffw]

That was not clear from the article, at least, although I definitely was biased against the article despite the fact that you could not have more clearly stated that the problem itself is overhyped.

I'm open to suggestions on how to phrase or edit the blog post, I thought I had covered that with this paragraph:

Cooldowns enforce positive behavior from supply chain security vendors: vendors are still incentivized to discover and report attacks quickly, but are not as incentivized to emit volumes of blogspam about “critical” attacks on largely underfunded open source ecosystems.

(But I freely admit that the post is terse, and I'm always open to feedback on clarifying my language.)

[tmccombs]

It's also worth pointing out that many of these attacks have involved a developer's credentials to the repository and/or package registry getting compromised, and in such cases, the developer often eventually realizes their credentials have been compromised. But that isn't always immediate, and it can take time for them to realize what happened, restore their account, and yank the malicious release.

[clarfonthey]

One thing that's particularly relevant IMHO is how npm treats updating package.json and package.lock, which is different from how cargo updates Cargo.lock (and not Cargo.toml, usually).

Right now, Cargo.lock is the brute-force solution of ensuring deterministic builds: every single dependency is locked to an exact version, no exceptions, until the lockfile is updated. Except, there really isn't a way to update the lockfile except updating everything, or updating a single crate. And there isn't really much control over the lockfile except a few key configuration options like MSRV or choosing the minimum valid option.

There are plenty of tools that other projects use to help mitigate some dependency issues that could definitely be helpful if they were part of Cargo. For example, the rust-lang repo has a large list of all crates that can ever be dependencies, ever, to ensure that all crates are at minimum vetted to be from trusted sources. And other tools like cargo vet exist to manage this trust process in a more
configurable way.

It would be nice if you could have an intermediary between Cargo.toml and Cargo.lock that let you add a few more limits on dependencies, like:

• Only allow certain transitive dependencies (of specified versions) and warn or error otherwise
• Distinguishing between "minimum" versions and "preferred" versions: for example, your code might work with version 1.0.0 of a crate but you've vetted version 1.1.2 and you haven't checked newer versions yet (NPM does this by just updating package.json directly, lower versions be damned, and this could be a reasonable option for Cargo too potentially for binary or internal library crates)

It's not clear whether all of these changes should be part of Cargo, but at least some of them could be, and we should make it easier to understand how things are being updated to avoid cases where someone accidentally updates to a vulnerable version. Minimum release time could still be one of the constraints added to packages under this system, but it wouldn't be the only one.

[woodruffw]

Yeah, I think minimum release ages are only a (small) part of a larger toolbox here. I think the larger toolbox includes things like trusted packages/reviews as well as capability analysis (e.g. foo v1.0.0 could only do local I/O, but foo v1.0.1 can do networked I/O and thus merits more attention).

[epage]

Minimum release time could still be one of the constraints added to packages under this system, but it wouldn't be the only one.

If thats the case, I'm not sure I understand the concern. Yes, there are more things that we can be doing. Is it that the system for these rules isn't generalized like rust-lang/cargo#7193 ?

Only allow certain transitive dependencies (of specified versions) and warn or error otherwise

We have several different issues exploring different angles on this:

• Dependency Resolution Hooks cargo#7193
• [feat] ability to blacklist package from user side cargo#15824
• Build script allowlist mode cargo#13681
• Warn when new duplicate dependencies are introduced cargo#13899

Distinguishing between "minimum" versions and "preferred" versions: for example, your code might work with version 1.0.0 of a crate but you've vetted version 1.1.2 and you haven't checked newer versions yet (NPM does this by just updating package.json directly, lower versions be damned, and this could be a reasonable option for Cargo too potentially for binary or internal library crates)

In my opinion, we should work to having vetting minimum dependencies. rust-lang/cargo#5657 includes two different ways to resolve to it to check it but I have concerns about each approach and I wonder if we should instead be more aggressive with Cargo.toml, like with rust-lang/cargo#15583

[epage]

(I almost deleted my last two replies so we could focus on the bigger topic in the first but didn't want to lose track of the links)

[Mark-Simulacrum]

I'm trying to understand when I would want to set this. Is there perhaps an example we can provide of typical use-cases? Isn't setting this to allow the equivalent of removing the registry.global-min-publish-age configuration?

Or does registry.global-min-publish-age apply always (e.g., even with a lock file?) whereas resolver.incompatible-publish-age only applies when we're actively searching for new versions?

[epage]

Is the question "why set to allow when you can set the time to 0?"

It would be good to clarify this. One controls time, the other behavior but both have a way to turn it off due td their design,

I'd say turning off the behavior is for tranisently turning it off for all registries rather than setting it for each registry.

[Mark-Simulacrum]

I think the common case for most users is that they only have one registry, right? In that case just commenting out or setting the time to zero feels better to me than complicating the configuration out of the gate with two options that interact.

(It seems like we can always add the extra configuration to temporarily opt out later).

[Mark-Simulacrum]

I'm wondering how/if we expect this to interact with alternative date sources (some future, some partially already visible today). For example, if we implement something like TUF or a mirror of crates.io, the "age" here might want to be relative to the index snapshot we're using, rather than to absolute local time. Maybe there's a question worth adding about exploring what "now" is?

As a concrete example of that, mirrors today can implement a version of this RFC by delaying imports of new versions by the period into the index they expose, rather than implementing that in Cargo. Fully replacing that with this RFC would require that the now timestamp is kept deterministic over time without a lock file checked in, which could be done if the index 'checkout' itself had the now timestamp stored when it was fetched. That would also help users opt-in more eagerly without needing that controlled at the mirror level, while still staying behind for most versions.

[epage]

We should include that as an alternative.

Note that pubtime is already stabilized in the registry and so for a TUF mirror of crates.io, they would have to preserve pubtime or else they would be invalid.

If we don't, for the unaware, we should call out that this relies on the already stable pubtime field.

[Mark-Simulacrum]

Putting it more concisely: the RFC as written I think assuming the relative age is relative to wall-clock now. I think registries should be able to tell Cargo when now is.

For example, in offline mode, I think cargo ought to use the last time it fetched from the network, not continue ratcheting forward. (Or at least we should consider that...)

[epage]

Could you help me understand what the use case is for this?

[Mark-Simulacrum]

If I'm on a flight, I don't really want Cargo suddenly realizing I have new crates "available" because they have matured in my index copy: I want to keep my index static and without the last N days as of when it was fetched. I think Cargo partially supports this with offline mode, but not completely: even if some newer crate is available locally, I probably don't want to start using it, because without an up to date index I can't know if it was yanked in the period of time since I fetched the index.

My understanding is that our model is that automated scanners take 24-72 hours to pick up bad code in lots of cases. If I believe that I want to use the index as of -72 hours from when I fetched it, not -72 hours from whatever my local time is.

[epage]

I think anything like this could potentially be added later and feel like it should then be best left to a future possibility to keep this more narrowly scoped (for faster resolving of discussion and implementation).

There also isn't a coherent snapshot of the index that we are operating on because of the sparse index design. We only update the entries we request and not all of those that have been downloaded. We'd have to keep per-entry "last update" times and make this relative to those. This is also adding a lot of writes even when the cache is unchanged which seems like it could affect performance and health of the drive (granted, target-dirs are even worse there) for something that won't be used in most cases.

I'm also not aware of any prior art that tracks it this carefully. Some allow the user to set a date for comparison. cargo generate-lockfile --publish-time will limit packages according to a specific date. As a future possibility, we could have a config for people to override "now".

[swarnimarun]

Can we get a minimum version of this through the RFC process? Given the recent rise in supply chain attacks and given all/most used package managers across other major language ecosystems already support this, with Cargo being the only major outlier, putting a significant target on Rust ecosystem.

For exceptional cases maybe we should just allow users to pin versions and cargo can ask the user if they are sure they want to use a "recent" version that may have security concerns, and if they select yes we update lock file and not worry about it.

Just my 2 cents; don't want to create noise, or pressure though.

Just having the ability to set min-publish-age: 30 days will give me quite a bit of piece of mind.

Also just to make it clear, I do also like that the time unit is included in the string, currently npm, bun, and pnpm all 3 have different time units which is kind of insane.

uv does it right, imho.

~/.config/uv/uv.toml
exclude-newer = "7 days"

ignore the name of the variable.

Again feel free to ask me any questions about any doubts, or consistency stuff, but otherwise I don't think I can add much here.

Thanks a bunch for all the awesome work folks.

[ChrisAntaki]

This is shaping up nicely, really looking forward to this feature!

[est31]

FTR, with such a default choice this feature would only be minimally useful for security.

I can have foobar = "0.1" in my Cargo.toml. Currently, there are foobar 0.1.1,0.1.2,0.1.3 releases, I am locked to 0.1.2.

Then a hacker gets hold of foobar publishing credentials and publishes foobar 0.1.4, yanking all of the existing foobar versions. Cargo starts warning that the dependency has been yanked, and suggests you to use a newer version. A cargo update later, and you are compromised. Worse if you are doing cargo install.

I say minimally useful because fallback does protect you from hacked publish-only credentials.

A good solution includes both having an excludes array that you manually vet and maintain for security updates, as well as defaulting to deny.

[tmccombs]

hmm, that's a fair point that an attacker might yank older, stable versions.

One possible way that could be mitigated is that if:

• The version in the Cargo.lock file is yanked
• There is no non-yanked version newer than the current version in the Cargo.lock and older than the minimum age
• And there is a version newer than the minimum age that isn't yanked

Then cargo will prefer the yanked version that has already been selected in Cargo.lock over the new version that isn't yanked.

That still has a weakness if there isn't a Cargo.lock file though.

I'm not sure that even if we supported deny that we would need an exclude list, as long as versions that are already in Cargo.lock are considered safe, even if they are newer than the minimum age and the policy is deny. Or said another way, the Cargo.lock file could itself act as an excludes list.

[epage]

A good solution includes both having an excludes array that you manually vet and maintain for security updates, as well as defaulting to deny.

I consider an exclude list to be a similarly bad solution as there is no way to know when to remove items from it and that then opens you up to attack.

[est31]

Then cargo will prefer the yanked version that has already been selected in Cargo.lock over the new version that isn't yanked.

I'm not sure this is easy to represent in the resolver algorithm, there might be edge cases. I'd say it's easier to reason about if we just treat all crates newer than the configured exclusion period as if they were yanked.

I consider an exclude list to be a similarly bad solution as there is no way to know when to remove items from it and that then opens you up to attack.

looking at the pnpm exclusion design, you apparently put package names there, applying to every version of the package. Where yes, it's harder to remove items as you don't have a good criterion when to remove them.

Most crate authors publish only one security fix release anyway, I've seen it rarely that they also publish security fixes for older semver incompatible branches.

If we require the full version to be spelled out, for example with replace section syntax, then cargo can start emitting warnings the moment those versions are older than the exclusion period.

[epage]

If we require the full version to be spelled out, for example with replace section syntax, then cargo can start emitting warnings the moment those versions are older than the exclusion period.

If we went with PackageIdSpec syntax, I'd want this to start with

• error for git, path sources
  • Not relevant
• error for name-only syntax
  • avoid people under-specifying and getting too many updates
• error if the version is incomplete
  • avoid people under-specifying and getting too many updates
  • only registry sources are supported so version is always relevant

An annoyance with an exclude list and deny would be transitive dependencies. You will need to exclude every single one where the version requirement asks for too-new of a version. Say you do this with clap. That is really clap, clap_derive, clap_builder, and clap_lex. We can't make exceptions because that would prevent the backtracking that would be expected with a deny mode. At least if we had #3826, we would drop the coupled proc-macro / macro-exporter package split.

If we go with deny, we'd probably always respect the lockfile even if it has too-new packages, like with yank.

There may also be a desire to still support cargo update --precise <too-new>. We've been experimenting with precise-pre-release and talked about extending that to yanked as well. The problem with that approach is it is shallow: you can't "bless" a transitive yanked dependency at the same time. That can be even more of a problem for this "too new" case because they are likelier to have "too new" deps (e.g. the clap example) while an entire dependency tree doesn't normally get yanked. I've been considering proposing a fallback mode for yanked packages because of this. At least if we have both fallback and deny, users can temporarily switch to change their lockfile.

[epage]

To be clear, if we deny on this and if something is both yanked and too new, then cargo generate-lockfile will fail.

[mkj]

Reading the RFC I found myself wondering what "global-" meant here, since in the rest of Cargo config docs "global" refers to user settings ("global cache", "global git configuration", "globally change profile settings"). So I don't think the precedent is great?

[epage]

Unfortunately, there is already precedence that [registry] is for crates.io and we have already set the precedence that to use a global- prefix to disambiguate it.