Move providers out of rules_pkg

[malt3]

PackageFilesInfo is a very useful construct that many codebases use to define how exactly files should be packaged.
I would like for rules_img to consume this provider as well to build layers (in the image_layer rule).
However, I don't want to depend on the whole transitive closure of rules_pkg (including rules_python).
Could we agree to move providers to a neutral Bazel module that doesn't have any dependencies?
This way, producers and consumers can easily depend on the provider module.

Tagging @aiuto for comment.

[tonyaiuto]

A year ago I would have whined about this. But, that's what we had to end up with for supply_chain, so I'm happy to try it. I want to keep the same physical repo, so that it is easy to develop. My strawman proposal is the strategy we are doing for supply_chain.

• move the providers to /providers
  • add a MODULE.bazel there,
  • publish those as releases named providers-VERSION
• in the main MODULE, refer to the published providers-version, but use local_overried to point locally
• continue to publish this with the big version numbers.

We'll need to pick a new module name, perhaps rules_pkg_providers.

Is that along the lines of what you were thinking?

[malt3]

That's exactly what I was thinking about. The plan sounds good to me.
To avoid breaking users, we should probably still re-export the provider in the old locations in rules_pkg (but still ensure they all refer to the new definition).
Otherwise people will need to update their existing usage of rules_pkg.

[tonyaiuto]

cc: @cgrindel I would like to move this forward. Any thoughts?

[cgrindel]

Sounds reasonable to me. I know of another effort to move the toolchain definitions (e.g., types and providers) to a separate module. Should we move the toolchain type definitions, as well?

[FaBrand]

@cgrindel Maybe not related. There is also a use case for exetending rules_pkg. e.g having a specialized pkg_install or anything else that depends on the structure behind these providers.

However we ended up having to depend on @rules_pkg//pkg/private:pkg_files.bzl to properly destructure the information behind the providers without duplicating the logic elsewhere

In particular the create_mapping_context_from_ctx as well as add_label_list in order to do something with the data of the provider.
This would look like a fitting continuation of this effort.

@tonyaiuto Any thoughts?

[FaBrand]

Another option would be to simply expose those methods in a public context instead of moving them:

Something like this

# pkg/provider_api.bzl
load(
    "//pkg/private:pkg_files.bzl",
    _add_label_list = "add_label_list",
    _create_mapping_context_from_ctx = "create_mapping_context_from_ctx",
    _process_src = "process_src",
    _write_manifest = "write_manifest",
)

create_mapping_context_from_ctx = _create_mapping_context_from_ctx
process_src = _process_src
write_manifest = _write_manifest
add_label_list = _add_label_list

[tonyaiuto]

I think a specialized pkg_install is perfectly reasonable. Some features are going to be general enough to put in the reference implementation, but I'm sure someone will want use-case-specific installers.
So yes, making that private stuff public is worthwhile - with the warning that we may need to break the api periodically and do a major version bump. Especially if we integrate more tightly with rules_img.

Making the toolchains seperate might be reasonable too. That might let us decouple RPM, which seems to need a special variation for every linux distro, so testing will always be spotty. sort of a use-at-your-own-risk feature.

[FaBrand]

@tonyaiuto I pushed a first draft:
#1030

[tonyaiuto]

I saw the draft. If we are going to expose the methods, we should also make sure the documentation build picks them up, and they probably should move up from pkg/private to pkg.