config: specify submodules to clone in Manifest

CHANGELOG.md

@@ -6,6 +6,21 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ## Unreleased
 
+### Breaking Changes
+- **Submodules of a dependency are no longer cloned by default.** A package now declares which of its submodules it needs with a `git_submodules` list in its own `Bender.yml`, and only those are cloned. Bender emits warning `W37` for each dependency that carries submodules without declaring any.
+  - As a package maintainer, add the submodules your sources actually need to your manifest, or `git_submodules: []` to state that none are needed and silence the warning.
+  - As a consumer of a package that has not declared its submodules yet, run with `--git-submodules all` (or set `git_submodules: all` in your configuration) to restore the previous behaviour of cloning every submodule recursively.
+- **`--git-submodules` and the `git_submodules` config field take a mode instead of a boolean.** The env variable `BENDER_GIT_SUBMODULES` takes the same values. Booleans are no longer accepted, so replace them:
+  - `git_submodules: true` -> `git_submodules: all`, cloning all submodules of every dependency recursively, ignoring the per-package lists.
+  - `git_submodules: false` -> `git_submodules: none`, cloning no submodules at all, ignoring the per-package lists.
+  - Not setting it at all is now `manifest`, cloning what each package selects in its manifest.
+
+### Added
+- Add a per-dependency `git_submodules` list to the manifest (`Bender.yml`) that selects which of a package's submodules are cloned (with optional per-entry `recursive` and `shallow` flags, both defaulting to `true`).
+
+### Changed
+- Bender clones submodules shallowly by default (`--depth 1`), unless a package disables it explicitly by specifying `shallow: false` in its `git_submodules` list.
+
 ## 0.32.1 - 2026-07-07
 ### Added
 - Add `git_submodules` config field and `--git-submodules <true|false>` flag (env `BENDER_GIT_SUBMODULES`) to control cloning of dependency submodules; defaults to `true`, the flag overrides the configured value in either direction (https://github.com/pulp-platform/bender/pull/314).

book/src/configuration.md

@@ -57,12 +57,12 @@ Enable or disable Git Large File Storage (LFS) support. Requires `git-lfs` to be
 - **Example:** `git_lfs: false`
 
 ### `git_submodules`
-Clone the git submodules of dependencies. Disabling this skips `git submodule update` during checkout, which can substantially speed up the initial dependency fetch when dependencies carry submodules (e.g. software or tooling) that are not needed for the hardware build. Disable it only when none of your dependencies reference sources that live inside a submodule.
+Selects which git submodules are cloned for dependencies. `manifest` (the default) clones the submodules that each dependency selects via the [`git_submodules`](./dependencies.md#selecting-submodules-per-dependency) list in its own manifest. The other two modes ignore those lists: `none` skips `git submodule update` during checkout entirely, which can substantially speed up the initial dependency fetch when dependencies carry submodules (e.g. software or tooling) that are not needed for the hardware build; `all` clones all submodules of every dependency recursively, which helps when a dependency has not declared a submodule that you need. Only set it to `none` when none of your dependencies reference sources that live inside a submodule.
 - **Config Key:** `git_submodules`
-- **CLI Flag:** `--git-submodules <true|false>` (overrides the configured value in either direction)
+- **CLI Flag:** `--git-submodules <all|none|manifest>` (takes precedence over the configured value)
 - **Env Var:** `BENDER_GIT_SUBMODULES`
-- **Default:** `true`
-- **Example:** `git_submodules: false`
+- **Default:** `manifest`
+- **Example:** `git_submodules: none`
 
 ### `overrides`
 Forces specific dependencies to use a particular version or local path. This is primarily used in [`Bender.local`](./local.md) for development.

book/src/dependencies.md

@@ -109,16 +109,33 @@ Bender detects whether a dependency uses **Git Large File Storage (LFS)** via it
 
 ## Submodules
 
-If a dependency contains a `.gitmodules` file, Bender initializes and updates its Git submodules recursively after checkout by default.
+If a dependency contains a `.gitmodules` file, Bender only checks out the submodules that the dependency explicitly asks for in its own manifest. A dependency that does not list any submodules gets none of them checked out, and Bender emits warning `W37` listing the submodules it skipped.
 
-Cloning submodules is often the slowest part of fetching dependencies, and submodules frequently hold software or tooling that is irrelevant to the hardware build. You can therefore disable submodule cloning:
+Cloning submodules is often the slowest part of fetching dependencies, and submodules frequently hold software or tooling that is irrelevant to the hardware build. As a consumer of a dependency, you can override its selection in either direction, with `git_submodules` in your [configuration](./configuration.md#git_submodules) or with `--git-submodules <MODE>` (env `BENDER_GIT_SUBMODULES`) for a single invocation. The flag takes precedence over the configuration files.
 
-- Set `git_submodules: false` in your [configuration](./configuration.md#git_submodules) to skip submodules persistently for a project.
-- Pass `--git-submodules false` (or set `BENDER_GIT_SUBMODULES=false`) to skip them for a single invocation. The flag overrides the configured value in either direction.
+- **`manifest`** (the default): honor each dependency's own selection, as described below.
+- **`none`**: clone no submodules at all, ignoring what the dependencies select. Use this when none of your dependencies reference sources that live inside a submodule, or when the submodule remotes are not reachable from your machine. Bender emits warning `W36` for each dependency that *selects* submodules, listing the paths it skipped; a dependency that selects none is not reported, since nothing is missing from it that the default mode would have cloned.
+- **`all`**: clone all submodules of every dependency recursively, ignoring what the dependencies select. Use this when a dependency has not declared a submodule that you need — for instance because it does not use `git_submodules` yet.
 
-Only disable submodules when none of your dependencies reference sources that live inside a submodule.
+### Selecting submodules per dependency
 
-When submodules are disabled, Bender emits a warning for each dependency that carries submodules, listing the unchecked-out submodule paths and the `git submodule update --init --recursive` command to fetch them back into its checkout.
+A package maintainer declares which submodules the package actually needs with a `git_submodules` list in the package's own `Bender.yml`:
+
+```yaml
+git_submodules:
+  - sw/deps/printf            # string form: clone this submodule
+  - submodule: sw/deps/cva6-sdk  # map form
+    recursive: false          # skip the submodule's own submodules (default: true)
+    shallow: false            # fetch full history (default: true)
+  # pd/deps/ihp-130-pdk -> omitted, so it is not cloned
+```
+
+- **`git_submodules` present**: only the listed submodules are cloned.
+- **No `git_submodules` field** (the default): no submodules are cloned, and Bender warns (`W37`) with the list of submodules it skipped. Add the submodules you need to the manifest to silence the warning, or use `git_submodules: []` to state explicitly that none are needed.
+- **`recursive`** (default `true`): also update the submodule's own nested submodules. Set it to `false` to fetch only the top-level submodule.
+- **`shallow`** (default `true`): fetch the submodule with `--depth 1`. Set it to `false` to clone the full submodule history.
+
+Note that the list only takes effect for a package that is checked out *as a dependency*; the one in your own root manifest is not applied to your working copy. A `git_submodules` setting in the consumer's configuration or on the command line overrides the list, as described above.
 
 ## Version Resolution and the Lockfile
 

src/cli.rs

@@ -25,7 +25,7 @@ use crate::Result;
 use crate::cmd;
 use crate::cmd::fusesoc::FusesocArgs;
 use crate::config::{
-    Config, Manifest, Merge, PartialConfig, PrefixPaths, Validate, ValidationContext,
+    Config, Manifest, Merge, PartialConfig, PrefixPaths, SubmoduleMode, Validate, ValidationContext,
 };
 use crate::diagnostic::{Diagnostics, Warnings};
 use crate::lockfile::*;
@@ -61,14 +61,15 @@ struct Cli {
     )]
     git_throttle: Option<usize>,
 
-    /// Clone the git submodules of dependencies [default: true]
+    /// Override which submodules are cloned for dependencies
     #[arg(
         long,
         global = true,
+        value_name = "MODE",
         help_heading = "Global Options",
         env = "BENDER_GIT_SUBMODULES"
     )]
-    git_submodules: Option<bool>,
+    git_submodules: Option<SubmoduleMode>,
 
     /// Suppresses specific warnings. Use `all` to suppress all warnings.
     #[arg(long, global = true, action = ArgAction::Append, help_heading = "Global Options", env = "BENDER_SUPPRESS_WARNINGS")]
@@ -210,10 +211,8 @@ pub fn main() -> Result<()> {
     let mut config = load_config(&root_dir, matches!(cli.command, Commands::Update(_)))?;
 
     // The `--git-submodules` CLI flag (or `BENDER_GIT_SUBMODULES`) takes
-    // precedence over the configuration files in both directions.
-    if let Some(v) = cli.git_submodules {
-        config.git_submodules = v;
-    }
+    // precedence over the configuration files.
+    config.git_submodules = cli.git_submodules.unwrap_or(config.git_submodules);
     log::debug!("{:#?}", config);
 
     // Determine git throttle. The precedence is: CLI argument, env variable, config file, default (4).

src/config.rs

@@ -53,6 +53,12 @@ pub struct Manifest {
     pub workspace: Workspace,
     /// Vendorized dependencies
     pub vendor_package: Vec<VendorPackage>,
+    /// Per-dependency submodule selection.
+    ///
+    /// `None` means clone all submodules recursively (the default). `Some`
+    /// restricts cloning to the listed submodules only (an empty list clones
+    /// none).
+    pub git_submodules: Option<Vec<Submodule>>,
 }
 
 impl PrefixPaths for Manifest {
@@ -455,6 +461,8 @@ pub struct PartialManifest {
     pub workspace: Option<PartialWorkspace>,
     /// External Import dependencies
     pub vendor_package: Option<Vec<PartialVendorPackage>>,
+    /// Per-dependency submodule selection.
+    pub git_submodules: Option<Vec<StringOrStruct<PartialSubmodule>>>,
     /// Unknown extra fields
     #[serde(flatten)]
     extra: HashMap<String, Value>,
@@ -604,6 +612,11 @@ impl Validate for PartialManifest {
                 .wrap_err("Unable to parse vendor_package.")?,
             None => Vec::new(),
         };
+        let git_submodules = self
+            .git_submodules
+            .map(|subs| subs.validate(vctx))
+            .transpose()
+            .wrap_err(format!("In git_submodules of package `{}`:", pkg.name))?;
         if !vctx.pre_output {
             self.extra.iter().for_each(|(k, _)| {
                 Warnings::IgnoreUnknownField {
@@ -643,6 +656,7 @@ impl Validate for PartialManifest {
             frozen,
             workspace,
             vendor_package,
+            git_submodules,
         })
     }
 }
@@ -895,6 +909,64 @@ impl Validate for PartialIncludeDir {
     }
 }
 
+/// A submodule of a dependency that should be cloned.
+#[derive(Clone, Debug, Serialize)]
+pub struct Submodule {
+    /// The path of the submodule within the dependency repository.
+    pub path: String,
+    /// Whether to also update the submodule's own submodules (`--recursive`).
+    pub recursive: bool,
+    /// Whether to fetch the submodule shallowly (`--depth 1`).
+    pub shallow: bool,
+}
+
+/// A partial submodule selection entry.
+#[derive(Serialize, Deserialize, Debug)]
+pub struct PartialSubmodule {
+    /// The path of the submodule within the dependency repository.
+    pub submodule: String,
+    /// Whether to also update the submodule's own submodules (default: true).
+    pub recursive: Option<bool>,
+    /// Whether to fetch the submodule shallowly (default: true).
+    pub shallow: Option<bool>,
+    /// Unknown extra fields
+    #[serde(flatten)]
+    extra: HashMap<String, Value>,
+}
+
+impl FromStr for PartialSubmodule {
+    type Err = Void;
+    fn from_str(s: &str) -> std::result::Result<Self, Void> {
+        Ok(PartialSubmodule {
+            submodule: s.into(),
+            recursive: None,
+            shallow: None,
+            extra: HashMap::new(),
+        })
+    }
+}
+
+impl Validate for PartialSubmodule {
+    type Output = Submodule;
+    type Error = Error;
+    fn validate(self, vctx: &ValidationContext) -> Result<Submodule> {
+        if !vctx.pre_output {
+            self.extra.iter().for_each(|(k, _)| {
+                Warnings::IgnoreUnknownField {
+                    field: k.clone(),
+                    pkg: vctx.package_name.to_string(),
+                }
+                .emit();
+            });
+        }
+        Ok(Submodule {
+            path: self.submodule,
+            recursive: self.recursive.unwrap_or(true),
+            shallow: self.shallow.unwrap_or(true),
+        })
+    }
+}
+
 /// A partial filtered preprocessor define
 #[derive(Serialize, Deserialize, Debug, Default)]
 pub struct PartialDefine {
@@ -1607,6 +1679,21 @@ where
     }
 }
 
+/// Which submodules of a dependency are cloned.
+///
+/// The variant docs double as the `--git-submodules` help text.
+#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug, Default, clap::ValueEnum)]
+#[serde(rename_all = "lowercase")]
+pub enum SubmoduleMode {
+    /// Clone all submodules of every dependency, recursively
+    All,
+    /// Clone no submodules at all
+    None,
+    /// Clone the submodules each dependency selects in its own manifest [default]
+    #[default]
+    Manifest,
+}
+
 /// A configuration.
 ///
 /// This struct encapsulates every setting of the tool that can be changed by
@@ -1630,8 +1717,9 @@ pub struct Config {
     pub git_throttle: Option<usize>,
     /// Enable git LFS support, requires git-lfs (default: true)
     pub git_lfs: bool,
-    /// Clone the git submodules of dependencies (default: true)
-    pub git_submodules: bool,
+    /// Which submodules of a dependency are cloned (default: the selection in
+    /// each dependency's own manifest)
+    pub git_submodules: SubmoduleMode,
 }
 
 /// A partial configuration.
@@ -1651,8 +1739,9 @@ pub struct PartialConfig {
     pub git_throttle: Option<usize>,
     /// Enable git LFS support, requires git-lfs (default: true)
     pub git_lfs: Option<bool>,
-    /// Clone the git submodules of dependencies (default: true)
-    pub git_submodules: Option<bool>,
+    /// Which submodules of a dependency are cloned (default: the selection in
+    /// each dependency's own manifest)
+    pub git_submodules: Option<SubmoduleMode>,
 }
 
 impl PartialConfig {
@@ -1702,11 +1791,10 @@ impl Merge for PartialConfig {
                 (Some(v1), Some(v2)) => Some(v1 | v2),
                 (None, None) => None,
             },
-            git_submodules: match (self.git_submodules, other.git_submodules) {
-                (Some(v), None) | (None, Some(v)) => Some(v),
-                (Some(v1), Some(v2)) => Some(v1 | v2),
-                (None, None) => None,
-            },
+            // Unlike `git_lfs`, this is not an "either side may enable it"
+            // switch but a mode selection, so the higher-precedence side wins
+            // outright.
+            git_submodules: self.git_submodules.or(other.git_submodules),
         }
     }
 }
@@ -1742,7 +1830,7 @@ impl Validate for PartialConfig {
             },
             git_throttle: self.git_throttle,
             git_lfs: self.git_lfs.unwrap_or(true),
-            git_submodules: self.git_submodules.unwrap_or(true),
+            git_submodules: self.git_submodules.unwrap_or_default(),
         })
     }
 }

src/diagnostic.rs

@@ -434,18 +434,34 @@ pub enum Warnings {
     OverrideFilesIgnored(String),
 
     #[error(
-        "Git submodules are disabled, dependency {} has submodules that were not checked out:\n{}",
+        "Git submodules are disabled, dependency {} selects submodules that were not checked out:\n{}",
         fmt_pkg!(.0),
         .1.iter().map(|p| format!("  - {}", fmt_path!(p))).collect::<Vec<_>>().join("\n")
     )]
     #[diagnostic(
         code(W36),
         help(
-            "Re-run with `--git-submodules true` (or set `git_submodules: true` in the configuration), or run `git -C \"$(bender path {})\" submodule update --init --recursive` to fetch them manually.",
+            "Drop the `git_submodules: none` setting to clone the submodules {} selects, or use `all` to clone all of them.\nTo fetch them manually, run `git -C \"$(bender path {})\" submodule update --init -- <path>`.",
+            fmt_pkg!(.0),
             fmt_pkg!(.0)
         )
     )]
     SubmodulesDisabled(String, Vec<String>),
+
+    #[error(
+        "Dependency {} has submodules but does not declare which of them to clone, so none were checked out:\n{}",
+        fmt_pkg!(.0),
+        .1.iter().map(|p| format!("  - {}", fmt_path!(p))).collect::<Vec<_>>().join("\n")
+    )]
+    #[diagnostic(
+        code(W37),
+        help(
+            "Add a `git_submodules` list to the {} manifest to select the submodules it needs, or `git_submodules: []` if none are needed to silence this warning.\nTo fetch them anyway, re-run with `--git-submodules all` or run `git -C \"$(bender path {})\" submodule update --init --recursive`.",
+            fmt_pkg!(.0),
+            fmt_pkg!(.0)
+        )
+    )]
+    SubmodulesUnspecified(String, Vec<String>),
 }
 
 #[derive(Error, Diagnostic, Debug, Clone)]