Skip to content

fix: balance conditional nesting when removing spec sections#190

Merged
reubeno merged 1 commit into
microsoft:mainfrom
trungams:tvuong/fix/spec-conditional-balance
May 20, 2026
Merged

fix: balance conditional nesting when removing spec sections#190
reubeno merged 1 commit into
microsoft:mainfrom
trungams:tvuong/fix/spec-conditional-balance

Conversation

@trungams
Copy link
Copy Markdown
Member

collectSectionRanges now adjusts each section range to maintain balanced %if/%endif nesting before removal. This prevents producing invalid specs when sections are adjacent to or wrapped in conditional blocks.

Three cases are handled automatically:

  • Balanced conditionals inside a section (no change needed)
  • Trailing %if openers belonging to the next section (trimmed)
  • Trailing %endif from a wrapping conditional block (trimmed, leaving an empty but valid conditional wrapper)

Conditionals interleaved with section content (spanning case) are detected and reported with an [ErrConditionalSpansSections] error.

Fixes #144

Copilot AI review requested due to automatic review settings May 14, 2026 02:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates RPM spec section/subpackage removal to preserve balanced %if/%endif nesting when removing section ranges, addressing invalid specs caused by boundary conditionals.

Changes:

  • Adds conditional pair collection and range balancing before section removal.
  • Updates RemoveSubpackage documentation and overlay docs for conditional handling.
  • Adds tests for conditionalized subpackage removal scenarios.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
internal/rpm/spec/edit.go Adds conditional balancing logic for section range removal.
internal/rpm/spec/edit_test.go Adds coverage for conditional removal cases.
docs/user/reference/config/overlays.md Documents conditional behavior for subpackage removal overlays.
Comments suppressed due to low confidence (1)

internal/rpm/spec/edit.go:1004

  • This validation only handles straddling %if lines. When the straddling line is a %endif and there is real content later in the same section range, balanceRange trims at the %endif and leaves that later section content behind even though the caller requested the whole section/subpackage be removed. Either treat content after a straddling %endif as ErrConditionalSpansSections, or use a removal representation that can preserve the %endif while still deleting the remaining section body.
		if ifInside {
			// The %if is inside our range but %endif is outside. Check if there's
			// real content between the %if and the range end — that would mean the
			// conditional protects section content that spans into the next section.
			for i := p.ifLine + 1; i < r.end; i++ {
				if !isBlankOrComment(rawLines[i]) && conditionalDepthChange(rawLines[i]) == 0 {
					return r, fmt.Errorf(

Comment thread docs/user/reference/config/overlays.md
Comment thread internal/rpm/spec/edit.go
Comment thread internal/rpm/spec/edit_test.go Outdated
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch 2 times, most recently from c20b949 to d7b1b2e Compare May 14, 2026 03:04
Copilot AI review requested due to automatic review settings May 14, 2026 03:04
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread internal/rpm/spec/edit.go
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from d7b1b2e to 26e0634 Compare May 14, 2026 03:37
@trungams trungams requested a review from Copilot May 14, 2026 03:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comment thread internal/rpm/spec/edit.go Outdated
Comment thread internal/rpm/spec/edit.go
Comment thread internal/rpm/spec/edit.go
// spec's conditional structure. If a conditional block is interleaved with section
// content in a way that cannot be resolved by trimming, an [ErrConditionalSpansSections]
// error is returned.
func (s *Spec) collectSectionRanges(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you tell if spec-append-lines et al. suffer from the same problem? I'm assuming we may need to update them to use this code path too. Otherwise, the lines might get appended after an %ifdef that wraps a succeeding section.

IIRC, that was one of the reasons that Tobias had added spec-insert-tag as a workaround.

I'm okay if you want to split that off as a separate PR, but since you're already in this code path / logic would be great if you could sort that out too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does, since the limitation initially comes from Spec.Visit not accounting for conditionals in section ranges. I need a bit more time to think of a direct improvement to Spec.Visit, so I'd prefer to do it in a follow-up PR. The solution I'm adding here is meant to go on top of collectSectionRanges to have as little impact on existing spec files as possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I'd like to see you continue working in this area after this PR -- to clean it up a bit. If we get more tests in place, we can have the confidence to better unify/factor the logic so it doesn't continue to accrete complexity.

Comment thread internal/rpm/spec/edit.go
// If the straddling line is an %if (opener inside, closer outside),
// check for real content between the %if and the range end. Such content
// would belong to this section but span into the next via the conditional.
if ifInside {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm missing something but don't we want to use the same logic here when %endif was inside but its starting %if wasn't? I'd only expect that to be okay if there's no content after.

Copy link
Copy Markdown
Member Author

@trungams trungams May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already handled later in the function, on line 1035

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already handled later in the function, on line 1035

Understood, but isn't the conditional unneeded complexity? Or is there some other case I'm not thinking of?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to cover the general cases so yes, it's very unlikely this happens and implies a spec is badly written, so we need to throw an error.

Comment thread internal/rpm/spec/edit.go Outdated
)
}

// Trim trailing blank/comment lines for clean output.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually do like the trimming, but for consistency's sake can we keep this trimming consistent across both the case where we need to rebalance and we don't? (My preference would be to do it in both places.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to add a post-processing step that cleans up the specs after all overlays are applied

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kind of post processing?

Copy link
Copy Markdown
Member Author

@trungams trungams May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-read the code and realized "for clean output" was a bit misleading and that trimming logic didn't actually do anything useful, so I removed it. Output should look consistent now

I think it's better to remove the trimming at least for this PR, it's sending me down a rabbit hole trying to properly define section boundaries and accounting for comments/whitespaces. I'll try to keep this PR simple for now

@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from 26e0634 to b1b2aeb Compare May 14, 2026 08:18
Copilot AI review requested due to automatic review settings May 14, 2026 08:49
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from b1b2aeb to bd57684 Compare May 14, 2026 08:49
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from bd57684 to 88e41c2 Compare May 14, 2026 08:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comment thread internal/app/azldev/core/sources/sourceprep.go Outdated
Comment thread internal/app/azldev/core/sources/overlays.go Outdated
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from 88e41c2 to f579aa7 Compare May 14, 2026 09:01
@trungams trungams requested a review from Copilot May 14, 2026 09:04
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comment thread internal/app/azldev/core/sources/overlays.go Outdated
@trungams trungams marked this pull request as draft May 14, 2026 09:10
@trungams trungams marked this pull request as ready for review May 14, 2026 09:13
// collapseSpecBlankLines opens a spec file, collapses consecutive blank lines, and
// writes it back. This is a post-processing step intended to clean up gaps left by
// section or tag removal overlays.
func collapseSpecBlankLines(fs opctx.FS, specPath string) error {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will introduce more deviations from input upstream specs than strictly needed; is this necessary? I don't think we should be in the business of rewriting the specs for readability, unless it's scoped to the specific overlay changes we're making.

I think the "trim" you had been doing for section collection was fine because it didn't actually remove any content, it just considered the section to end a bit earlier so that appended lines / removed sections preserve the space between the sections.

(Minor note: it's also quite inconsistent that this logic gets applied unilaterally to all parts of the spec, including sections not overlaid -- but is only triggered if a certain subset of overlays are requested.)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was overthinking it - 2am thoughts :(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries. I think there's a compromise somewhere in there that's reasonable -- and fine for a subsequent PR.

@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from f579aa7 to 86aaa42 Compare May 14, 2026 17:50
Copilot AI review requested due to automatic review settings May 14, 2026 19:35
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from 86aaa42 to d476ff9 Compare May 14, 2026 19:35
@trungams trungams requested a review from reubeno May 14, 2026 20:01
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from d476ff9 to a2d96f6 Compare May 14, 2026 20:21
@reubeno
Copy link
Copy Markdown
Member

reubeno commented May 15, 2026

Changes are looking good; primarily waiting on microsoft/azurelinux#17259 to go in first so this won't automatically be pulled in. (Looks like that would result in 2 specs rendering differently and resulting errors. We'll want to ingest this change and update those 2 rendered specs in a single PR to microsoft/azurelinux.)

collectSectionRanges now adjusts each section range to maintain balanced
%if/%endif nesting before removal. This prevents producing invalid specs
when sections are adjacent to or wrapped in conditional blocks.

Three cases are handled automatically:

- Balanced conditionals inside a section (no change needed)
- Trailing %if openers belonging to the next section (trimmed)
- Trailing %endif from a wrapping conditional block (trimmed, leaving an
  empty but valid conditional wrapper)

Conditionals interleaved with section content (spanning case) are detected
and reported with an ErrConditionalSpansSections error.

After each range removal, consecutive blank lines at the splice boundary
are collapsed to prevent multi-blank gaps in the output.

Fixes microsoft#144
@trungams trungams force-pushed the tvuong/fix/spec-conditional-balance branch from a2d96f6 to 4a03139 Compare May 16, 2026 00:02
@reubeno reubeno merged commit 503c237 into microsoft:main May 20, 2026
15 checks passed
PawelWMS added a commit to microsoft/azurelinux that referenced this pull request May 22, 2026
…entions

Captures the conventions developed during the multi-day session of
remediating package-signing/scan failures at the component level
(no allow-listing) and folds them into permanent Copilot guidance.
No new tooling -- this is purely instructional / documentation
content to make future agent sessions discover and follow the
patterns we now use across the distro.

Changes
-------

1. `.github/instructions/comp-toml.instructions.md` (+69 lines).
   New "replace-upstream source override" section documenting the
   single-step pattern for serving a locally-modified upstream
   tarball under the same filename: one
   `[[components.<name>.source-files]]` block with
   `replace-upstream = true` and a `replace-reason = "..."` string.
   No separate `file-remove` overlay against `sources` is needed;
   `azldev`'s render step emits an audit WARN log naming the
   override and the from/to SHA-512 pair.

   Covers: field semantics (`filename`, `hash`, `hash-type`,
   `replace-upstream`, `replace-reason`, `origin.uri`); the
   lookaside URL pattern; the requirement that `hash` and the
   `$hash` segment of `origin.uri` stay in sync; the
   `replace-reason` style rules (single TOML string, no
   triple-quoted strings, self-contained motivation); and a
   "Public-content hygiene" subsection establishing that
   motivations be described technically and neutrally (no specific
   internal infrastructure names).

   Also documents the `components.toml` file-organization rule:
   when moving a component to a dedicated `*.comp.toml`, delete
   the inline entry; when removing a component, delete the inline
   entry. No tombstone comments.

2. `.github/skills/skill-modify-source/SKILL.md` (new, 267 lines).
   New skill documenting the canonical `modify_source.sh` pattern
   for byte-deterministic Source0 repacks. The lookaside URL
   embeds the SHA-512 of the served file, so the repack MUST be
   byte-deterministic: same upstream input ⇒ byte-identical
   output ⇒ identical SHA-512, across machines and re-runs.

   Sections:
     (a) Anatomy of `modify_source.sh` (scratch dir under
         `base/build/work/scratch/<name>/`, `BASH_SOURCE[0]`
         root resolution, `set -euo pipefail`).
     (b) The byte-deterministic repack contract -- required `tar`
         flags (`--sort=name`, `--owner=0 --group=0`,
         `--numeric-owner`, `--mtime='@<epoch>'`, `--format=gnu`,
         `LC_ALL=C`) and `xz` flags (`-T 1`, `-9`), plus the
         forbidden set.
     (c) Mandatory upstream SHA-512 verification before extract.
     (d) Strip-list vs keep-list strategies with list-discipline
         rules.
     (e) Output-and-reporting contract (path, hash, ready-to-paste
         `az storage blob upload` command, comp.toml update
         reminder).
     (f) Determinism self-test recipe (two consecutive runs must
         produce byte-identical output).
     (g) Anti-patterns (`mktemp` for output filename, `$$` / `$RANDOM`
         / `$(date)` intermediates, `gzip` without `-n`, multi-
         threaded `xz` without `--block-size`, `find -newer`, etc.).
     (h) Canonical ~40-line bash skeleton showing the whole pattern
         end-to-end.

3. `.github/skills/skill-signing-failure-remediation/SKILL.md`
   (new, 98 lines). New skill documenting the decision tree for
   handling pipeline scan failures at the component level:
     (a) No reverse dependencies → drop the component outright
         (separate PR).
     (b) Offending artefact lives in a test-only / optional
         subpackage → use `spec-remove-subpackage` (or
         `build.without` if a clean bcond exists).
     (c) Reverse deps require the component to keep building →
         strip subtrees from Source0 via the `replace-upstream`
         pattern with a `modify_source.sh` script
         (see `skill-modify-source`).
   Always run a reverse-dependency scan before choosing (a) or (b).

4. `.github/copilot-instructions.md` (+3 lines). Three new
   repository-wide rules at the bottom of "Repository Hygiene
   Rules":
     6. Commit signing -- every commit MUST be GPG-signed;
        `git log -1 --show-signature` must show "Good signature".
     7. Public-content hygiene -- describe motivations in overlay
        descriptions, replace-reason strings, comp.toml comments,
        spec files, modify_source.sh echo lines, commit messages,
        and PR descriptions *technically and neutrally*. Use
        neutral phrasing such as "automated package-signing
        pipeline", "FS-aware deep scanner", "automated malware
        scan".
     8. Component-modification order of operations -- canonical
        sequence is `edit comp.toml → azldev comp update -p <name>
        → commit (toml + lock) → azldev comp render -p <name> →
        amend (rendered spec)`. Keeps the lock fingerprint,
        %changelog, and Release: in sync within a single commit.

5. `.github/skills/skill-update-component/SKILL.md` (+20/-4 lines).
   Clarified the commit ordering with a worked example. The
   canonical sequence is: edit `comp.toml`, `azldev comp update -p
   <name>` to refresh the lock, commit the comp.toml + lock,
   `azldev comp render -p <name>`, amend in the rendered spec.
   Spelled out *why* the second render-and-amend is needed
   (`%changelog` / `Release:` are derived from `git log`, not the
   working tree).

6. `AGENTS.md` (+2 lines). Added two rows to the skill index:
     | Byte-deterministic Source0 repack (`modify_source.sh` +
       `replace-upstream`) | `skill-modify-source` |
     | Remediate component-level signing/scan failures           |
       `skill-signing-failure-remediation` |

7. `base/comps/AGENTS.md` (+4 lines). Mirrored the
   `components.toml` hygiene rule at the directory-level guide:
   when moving a component to a dedicated file, delete the inline
   entry (no tombstone comment); when removing a component,
   delete the inline entry.

8. `.github/copilot-instructions.md` (+1 line, follow-up). Added
   rule 11 to "Repository Hygiene Rules": fix render / lock
   drift locally by re-running `azldev comp render -p <name>` /
   `azldev comp update -p <name>` and amending, never by
   `gh run download`-ing the `rendered-specs-patch` /
   `locks-patch` CI artifact and `git apply`-ing it. The
   artifact is a reviewer convenience; using it as a fix path
   skips local reproduction and masks any local/CI environment
   skew.

9. `.github/skills/skill-update-component/SKILL.md` (+2 lines,
   follow-up). Added a "Drift-fix discipline" paragraph under the
   existing "CI gotcha" section, cross-referencing rule 11.

10. `.github/copilot-instructions.md` + `.github/instructions/comp-toml.instructions.md`
    (follow-up). Encode the "descriptions over comments" policy that
    emerged from the TOML cleanup work in this session:
      - Rule 1 of "Repository Hygiene Rules" reworded from "Overlay
        descriptions" to "Descriptions over comments". Establishes that
        schema-provided fields (`description`, `replace-reason`,
        `skip_reason`, etc.) are preferred over TOML comments, and that
        every rationale string MUST be brief -- one short line,
        motivational (the *why*), not implementation detail.
      - Rule 9 updated to cite `CONTRIBUTING.md` -- Pull Request
        Workflow as the authoritative source for the
        single-commit-per-PR requirement.
      - `comp-toml.instructions.md` section renamed from "Adding
        Description Comments" to "Descriptions over comments".
        Rewritten to lead with the structured-field preference and to
        give DO / DO NOT examples for both descriptions and (sparing)
        TOML comments; the "References" subsection is retained.

11. `.github/skills/skill-fix-overlay/SKILL.md` (follow-up). Refresh
    the `spec-remove-subpackage` / `spec-remove-section` failure-mode
    list. The engine now balances surrounding `%if` / `%endif`
    wrappers automatically (microsoft/azure-linux-dev-tools#190), so
    the "greedy consumption of `%if` guards" entry was dropped along
    with its manual `spec-append-lines` workaround. The two remaining
    failure modes — `%define` / `%global` inside the removed body
    vanishes (tracked in microsoft/azure-linux-dev-tools#203) and
    stray buildroot files — stay as-is, with a cross-link to the
    upstream tracking issue for the macro case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spec-remove-section / spec-remove-subpackage produce invalid specs when sections are inside %if/%endif

3 participants