Skip to content
Open
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
80b9ddf
Create 0000-nested-publish.md
kpreid Jul 1, 2023
126b2b4
Clarify some terms and discuss workspaces and normalization
kpreid Jul 1, 2023
ef9a157
Discuss trait coherence relaxation
kpreid Jul 1, 2023
a1d4425
Reposition footnote-links
kpreid Jul 1, 2023
35184e0
Add PR number
kpreid Jul 1, 2023
c4dd372
Clarify `path` behavior and existing dev-dependency behavior
kpreid Jul 1, 2023
e71a953
Avoid "lockstep" and discuss a version duplication hazard
kpreid Jul 1, 2023
3da9eeb
Add Definitions section.
kpreid Jul 6, 2023
7f9c09f
Discuss more alternatives for marking packages; define "nested package"
kpreid Jul 6, 2023
130ceb0
Clarify “is private”
kpreid Jul 6, 2023
42bdb61
Require dependencies to be explicitly nested.
kpreid Feb 5, 2024
25d86b1
More motivation and drawbacks.
kpreid Feb 5, 2024
743d531
Define “nested publishing”.
kpreid Feb 5, 2024
34a19dc
Discuss postponed RFC 2224 as prior art.
kpreid Feb 5, 2024
b75a519
Mention vendoring.
kpreid Feb 5, 2024
d7e8dea
Discuss “subcrate dependencies” in prior art.
kpreid Feb 5, 2024
6670428
Discuss “Inline crates” in prior art.
kpreid Feb 5, 2024
da547de
Rewrite reference-level explanation to focus more on effects than cha…
kpreid Feb 5, 2024
b4923ac
Expand alternatives and move inline-crates discussion there.
kpreid Feb 5, 2024
68ad634
Move license and version ideas.
kpreid Feb 11, 2024
33d1c6e
Specify that package names must be unique.
kpreid Feb 11, 2024
680709c
Typo
kpreid Feb 11, 2024
2af4921
Mention feature flattening.
kpreid Feb 11, 2024
0deba10
Discuss workspace inheritance.
kpreid Feb 11, 2024
b999976
Move `dependencies.*.publish = false` to future possibilities.
kpreid Feb 11, 2024
3edb308
Replace `package.publish = "nested"` with `package.publish.nested = t…
kpreid Feb 11, 2024
dd90f20
Refine explanation of `package.publish` being a table.
kpreid Feb 11, 2024
07b058d
Rephrase name conflict rule to avoid "transitive closure".
kpreid Mar 11, 2024
2c220ee
Always error on `workspace.dependencies.*.publish`.
kpreid Mar 11, 2024
4bc77cb
Rewrite feature flattening section.
kpreid Mar 11, 2024
e470314
Rationale for name uniqueness.
kpreid Mar 13, 2024
2a5474e
Polishing.
kpreid Mar 13, 2024
db9e7fa
Update comparison with packages-as-namespaces given that that RFC has…
kpreid Mar 13, 2024
4817861
Explicitly state that nested names are non-unique *outside* of the pa…
kpreid Mar 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Expand alternatives and move inline-crates discussion there.
  • Loading branch information
kpreid committed Feb 5, 2024
commit b4923aca0dae16f9dfc1c4b7d5a42e6f19a33a9a
36 changes: 22 additions & 14 deletions text/0000-nested-publish.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor question: If a Git repository contains a package with nested packages, can the other package depends on any of those nested packages as a Git dependency? Currently Git dependency searches packages whose name matches recursively inside the repository.

Original file line number Diff line number Diff line change
Expand Up @@ -198,15 +198,24 @@ The reason for doing anything at all in this area is that publishing multiple pa
* When packages are implementation details, it makes a permanent mark on the `crates.io` registry even if the implementation of the parent package stops needing that particular subdivision. By allowing sub-packages we can allow package authors to create whatever sub-packages they imagine might be useful, and delete them in later versions with no consequences.
* It is possible to depend on a published package that is intended as an implementation detail. Ideally, library authors would document this clearly and library users would obey the documentation, but that doesn't always happen. By allowing nested packages, we introduce a simple “visibility” system that is useful in the same way that `pub` and `pub(crate)` are useful within Rust crates.

The alternative to nested packages that I have heard of as a possibility would be to support multiple library targets per package. That would be arguably cleaner, but has these disadvantages:
## Alternatives to nested packages
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another alternative is to add a form of access control related to open namespaces so you can have published packages with APIs scoped to the namespace

  • Perform this access control at the cargo/crates.io level by allowing a published package to be marked as private, disallowing any package to depend on it unless its participating in the namespace.
    • However, namespace membership control is only enforced at the registry level, so for any local development, you can access the private packages by putting yourself in the namespace
  • Language level support with a pub(namespace) (name to be bike-shedded)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, neither helps with the publishing overhead.


* It would require new manifest syntax, not just for declaring the multiple libraries, but for referring to them, and for making per-target dependencies (e.g. only a proc-macro lib should depend on `proc-macro2`+`quote`+`syn`, not the rest of the libraries in the package).
* It would require many new mechanisms in Cargo.
* It might have unforeseen problems; by contrast, nested packages are compiled exactly the same way `path` dependencies currently are, and the only new element is the ability to publish them, so the risk of surprises is lower.
* Support multiple library targets per package. That would be arguably cleaner, but has these disadvantages:

Also, nested packages enables nesting *anything* that Cargo packages can express now and in the future; it is composable with other Cargo functionality.
* It would require new manifest syntax, not just for declaring the multiple libraries, but for referring to them, and for making per-target dependencies (e.g. only a proc-macro lib should depend on `proc-macro2`+`quote`+`syn`, not the rest of the libraries in the package).
* It would require many new mechanisms in Cargo.
* It might have unforeseen problems; by contrast, nested packages are compiled exactly the same way `path` dependencies currently are, and the only new element is the ability to publish them, so the risk of surprises is lower.

We could also do nothing, except for warning the authors of paired macro crates that they should use exact version dependencies. The consequence of this will be continued hassle for developers; it might even be that useful proc-macro features might not be written simply because the author does not want to manage a second package.
Also, nested packages enables nesting *anything* that Cargo packages can express now and in the future; it is composable with other Cargo functionality.

* [Extend the language and `rustc` to support “inline crates”][inline-crates]; support triggering the compilation of child crates declared within the source code of the parent. This would be extremely convenient for proc-macros or when simply wants to cause fully parallel and independently-cached compilation of subsets of their code. However:

* It does not permit specifying different dependencies for each crate (includes false dependency edges).
* The compilations cannot be started until the dependent crate has gotten past the macro expansion phase to discover crate declarations.
* Does not naturally contain a way to define binary crates.
* I expect it would also require significant changes to the Rust language and `rustc`, reaching beyond just “spawn another `rustc`” and including problems like computing the implied dependencies among inline crates that depend on other inline crates (unless inline crates are only allowed to depend on external dependencies and their own child inline crates, which is likely undesirable because it prohibits establishing common core vocabulary among a set of crates to be compiled in parallel).

* Do nothing, except for warning the authors of paired macro crates that they should use exact version dependencies. The consequence of this will be continued hassle for developers; it might even be that useful proc-macro features might not be written simply because the author does not want to manage a second package.

## Details within this proposal

Expand All @@ -232,12 +241,10 @@ There are several ways we could mark packages for nested publishing, rather than
[prior-art]: #prior-art

* Postponed [RFC 2224] (2017) is broadly similar to this RFC, and proposed using `publish = false` to mean what we mean by `publish = "nested"`.

This RFC is more detailed and addresses the questions that were raised in discussion of 2224.

* Blog post [Inline crates, by Yoshua Wuyts (2022)](https://blog.yoshuawuyts.com/inline-crates/) proposes that additional library crates can be declared using Rust syntax in the manner `crate foo;` or `crate foo {}`, like modules.

Compared to this proposal, its advantage is that it requires no new Cargo concepts and fits right in to the source-code-traversal-driven nature of existing ways to define Rust package and crate contents. However, it forces additional edges in the dependency graph (since the dependent's compilation must be started first to discover the inline crate, and there is no place to declare different dependencies for different crates). I expect it would also require significant changes to the Rust language and `rustc`, reaching beyond just “spawn another `rustc`” and including problems like computing the implied dependencies among inline crates that depend on other inline crates (unless inline crates are only allowed to depend on external dependencies and their own child inline crates, which is likely undesirable because it prohibits establishing common core vocabulary among a set of crates to be compiled in parallel).
* Blog post [Inline crates, by Yoshua Wuyts (2022)][inline-crates] proposes that additional library crates can be declared using Rust syntax in the manner `crate foo;` or `crate foo {}`, like modules.
This is discussed above in the alternatives section.

* Blog post [Rust 2030 Christmas list: Subcrate dependencies, by Olivier Faure (2023)](https://poignardazur.github.io/2023/01/24/subcrates/) proposes a mechanism of declaring nested dependencies similar to this RFC, but instead of embedding the files in one package, the “subcrates” are packaged separately on crates.io, but published as a single command and are not usable by other packages. Thus, it is similar to a combination of the also-desired features of “publish an entire workspace” and “namespacing on crates.io”, plus the subcrates being private on crates.io.

Expand Down Expand Up @@ -286,10 +293,11 @@ This RFC does not propose implementing a dependency declared as `{ git = "...",
* Exactly what set of files should be copied?
* It would become possible to depend on (and thus copy during publication) a nested package someone else wrote for their own packages' use, which creates hazards for versioning and for non-compliance with source code licenses; while these are already possible, now Cargo would be doing it invisibly for you, which seems risky.

## Testing

A noteworthy benefit of nesting over separately-published packages is that the entire package can be verified to build outside its development repository/workspace by running `cargo publish --dry-run` or `cargo package`. It might be interesting to add a flag which does not just build the package, but also test it; while this is not at all related to nested packages *per se*, it might be a particular benefit to the kind of large project which currently uses multiple packages.

[artifact dependencies]: https://github.com/rust-lang/rfcs/pull/3028
[#3243 packages as namespaces]: https://github.com/rust-lang/rfcs/pull/3243
[RFC 2224]: https://github.com/rust-lang/rfcs/pull/2224

## Testing

A noteworthy benefit of nesting over separately-published packages is that the entire package can be verified to build outside its development repository/workspace by running `cargo publish --dry-run` or `cargo package`. It might be interesting to add a flag which does not just build the package, but also test it; while this is not at all related to nested packages *per se*, it might be a particular benefit to the kind of large project which currently uses multiple packages.
[inline-crates]: https://blog.yoshuawuyts.com/inline-crates/