feat: upgrade existing filesystem by upils · Pull Request #264 · canonical/chisel

upils · 2026-01-29T16:24:00Z

Have you signed the CLA?

This commit enables Chisel upgrading an existing rootfs.
It detects the target directory contains the result of a previous execution
to then operate an upgrade of the content. This initial simple implementation
has the following limitationg:

entries in the rootfs is not verified against the existing manifest.
entries (files, symlinks) are systematically replaced, even if identical to new ones.

As a consequences:

User modifications on chisel-managed content are overridden.
In the context of an OCI image build, this "new" content is identified as
different and thus the new OCI layer contains duplicated files.

Signed-off-by: Paul Mars <paul.mars@canonical.com>

github-actions · 2026-01-29T16:28:45Z

Command	Mean [s]	Min [s]	Max [s]	Relative
`BASE`	13.357 ± 0.114	13.191	13.522	1.01 ± 0.01
`HEAD`	13.280 ± 0.109	13.173	13.509	1.00

letFunny

First review. I am missing a couple of files but I wanted to publish it as quickly as possible so we have more time to iterate. This is looking great Paul, many things are looking dead simple which is always an extremely good sign. I have written some comments about how to make the PR shorter and how to make it, hopefully, even simpler. Let me know what you think.

I see some of the comments are similar to past discussions we had in upils#1. We spent some effort there, so please make sure all the comments were carried over to this one.

letFunny

Thanks Paul, I like the new comments. And thank you for postponing the previous change, this PR is long enough as it is.

niemeyer

Here is a first pass.

For future PRs, let's please try to break it down into smaller parts so it's lighter on you and on the reviewer to get it in quickly.

niemeyer · 2026-03-30T15:59:10Z

+		if err != nil {
+			return err
+		}
+		if mfest != nil {


This must never ever happen if err != nil. This is an important invariant to keep in mind in every Go project we work on. If we skip this rule, we're straight into billion Euro mistake territory. If you asked to select a valid manifest, and it did not return an error, it must have selected a valid manifest!

Let's please fix the function implementation and drop this verification from every call site.

cc @letFunny

I completely agree. I brought the same point in the past out of muscle memory but I couldn't find any document to justify it properly. Maybe we should include it in the go style document that Ed wrote here.

I reworked it and added a custom error type. In cmd_cut.go the error message is not used but I wonder if it should be logged or not. If we are not planning on doing anything with these messages the implementation could be simplified by removing the message field or even by replacing the custom type by a named error.

As a rule of thumb, if we don't have a use case for the error type we don't introduce the error type, but I'm not sure I understand what you're asking, as it sounds like you're both saying "I added a custom error type" and "I don't need to add an error type".

niemeyer · 2026-03-30T16:12:09Z

+			for path, info := range slice.Contents {
+				if info.Generate == setup.GenerateManifest {
+					dir := strings.TrimSuffix(path, "**")
+					path = filepath.Join(dir, DefaultFilename)


This is too loose. Imagine what happens if path is "/hmm/oops**", for instance, or just "/oops".

We need to be strict about how a manifest path looks like. Hopefully this is already the case elsewhere so there's already a pattern to follow. If not, please let me know.

Following our discussion offline, see 85ea2b6. The validation function is heavily inspired from validateGeneratePath in internal/setup/yaml.go.

Can you please outline what was the outcome here? Many comments here saying "See XYZ.", but it's not practical to go to several different links trying to find out how you handled the issue that was outlined here.

I added a validateManifestPath function verifying that the received path matches our definition of a valid manifest path before doing any manipulation on it. This function is now used in FindPaths and FindPathsInRelease. I also added a couple of tests to make sure the issue you pointed out was handled properly.

niemeyer · 2026-03-30T16:16:23Z

+		defer func() {
+			os.RemoveAll(tmpWorkDir)
+		}()
+	}


Can we please try harder to make this logic look nice?

As a hint, it often pays off to resist the temptation of manipulating important core functions, or long functions, and sprinkling it with previously missing needs of new functionality. Instead, it's worth considering what it is that you're adding new, and then consider which specific abstractions are needed for it.

I did this change trying to follow the principle you mention. I isolated the "core" feature of Run in the dedicated cut function specifically to not change its behavior, except to return the report, then added the additional step of applying the recut. It also felt consistent with how it is done in Extract in extract.go. Looking at the diff again, it is a bit misleading and gives the impression of a bigger change than what it really is.

Am I missing your point? Is your comment about another aspect of the function? Would you mind giving more details on why or what seems off here?

Was anything done here or is it still the same logic? My request was for you both to discuss to see if there was an opportunity to make this look nicer before I jump in and help out with specific details if possible. The comment above seems to be justifying what exists instead of looking for improvements. Was there an actual conversation about how to make this nice?

The current implementation is the result of discussions and several iterations to separate concerns and make it consistent with the rest of the code. We discussed it with Alberto and I did not change anything since your first comment because I cannot see is not nice from your point of view.

niemeyer · 2026-03-30T17:13:46Z

+	defer f.Close()
+
+	h := sha256.New()
+	if _, err := io.Copy(h, f); err != nil {


There are more convenient APIs inside that package.

For the sole purpose of getting the hash of the manifest this could be simplified using os.ReadFile and sha256.Sum256. However, in the next step I plan on using this helper to get hashes of the existing content in the rootfs when verifying the rootfs against the manifest. In that situation, this function will handle arbitrarily big files that may not fit in memory. So I thought it safer to use io.Copy.

Were you suggestion using readerProxy from fsutil/create.go? If so, I fail to see how more convenient that would be to use it.

niemeyer · 2026-03-30T17:17:51Z

 	mfestSchema := db.Schema()
 	if mfestSchema != Schema {
-		return nil, fmt.Errorf("unknown schema version %q", mfestSchema)
+		return nil, fmt.Errorf("%w %q", ErrUnknownSchema, mfestSchema)


As Alberto says, we never promise to keep error messages intact. With that said, we can do better here because I still want to know what the unsupported schema was.

Let's please have an actual error type called UnknownSchemaError that contains the wrong schema version as a public Version field, and reports the message "unknown manifest schema version %q".

This reverts commit 5cda705.

upils · 2026-04-03T12:27:01Z

+// process assumes the targetDir was verified with prevManifest, and so
+// prevManifest is an accurate representation of files and directories
+// previously installed by Chisel in the targetDir.
+func applyRecut(targetDir string, tempDir string, report *manifestutil.Report, prevManifest *manifest.Manifest) error {


[Note to reviewer]: I have reworked the whole approach in a way that is much more in line with the agreement, and should be much safer for user content. This is a 3-steps process, relying on the fact that the rootfs will be verified with the previous manifest.

This approach is being very careful of user content by:

only removing what was known to Chisel (using the previous manifest)

checking all new paths do not collide with user content.

The rootfs verification will be added in a follow up PR, as an independent operation executed before cutting the new rootfs.

Finally this approach might also set us in a better position to recover from crashes as a future improvement. As we rely on the previous manifest and the new one, both of these could be stored in the .chisel dir before doing any destructive operation. They could then be used to build a clear representation of what happened before the crash.

niemeyer

Paul, this PR is relatively large, is sitting here for a while, and the replies to comments made are quite unclear in terms of where we are reaching agreements, or disagreements, or when things are getting changed, or need discussinon.

By now I'm seriously tempted to ask for the PR to be broken down a little bit further into easier to agree pieces that we can get merged more easily, so we can make more steady progress.

upils · 2026-04-20T13:17:03Z

This PR is being split in smaller ones to ease review. First step #288.

upils added 2 commits January 27, 2026 11:28

feat: search, load and validates manifests

491cd0c

Signed-off-by: Paul Mars <paul.mars@canonical.com>

feat: adapt to revised strategy

a6e57f7

upils added 6 commits January 30, 2026 14:26

feat: implement upgrade

5da0450

fix: deletion

5703751

fix: revert inadvertent change

26bb2b3

fix: check slice collecting error

4a14230

fix: simplify upgrade and improve deletion

865c1b8

refactor: cleaning

0343ddc

upils changed the title ~~feat: update existing filesystem~~ feat: upgrade existing filesystem Feb 2, 2026

refactor: refine upgrade

e116424

upils requested a review from letFunny February 2, 2026 14:35

upils marked this pull request as ready for review February 2, 2026 14:35

upils added 8 commits February 3, 2026 09:31

refactor: fsutil handles moving and removing

5b9d408

refactor: improve consistency

7861c8d

style: fix lint error

27a0d22

tests: Move and Remove

5f377b0

tests: FindPathsInRelease

f74265c

tests: more test cases for Move

799373c

tests: add spread test

dbe3f30

tests: fix recut spread test

becfe42

upils mentioned this pull request Feb 4, 2026

feat: search, load and validates manifests upils/chisel#1

Closed

1 task

upils added 5 commits February 4, 2026 09:44

tests: SelectValidManifest

3c994b8

fix: add missing deps

6ebb893

tests: recut feature

ceac217

fix: avoid duplicates in FindPathsInRelease

7e34e32

ci: rerun

e1d4e17

letFunny suggested changes Feb 4, 2026

View reviewed changes

upils added 2 commits February 5, 2026 08:46

fix: apply PR suggestions

c3db9d4

refactor: simplify FindPaths and FindPathsInRelease

e1e9c43

letFunny approved these changes Mar 11, 2026

View reviewed changes

upils added 2 commits March 23, 2026 17:03

style: fix typo in comment

bdbba3c

fix: properly return manifest

b12185b

niemeyer requested changes Mar 30, 2026

View reviewed changes

upils and others added 20 commits March 31, 2026 07:46

Merge branch 'canonical:main' into recut

17d7935

fix: track manifest path with bools

0c9b7bd

style: rename previous manifest

c5fa3b8

fix: simplify usage of contentHash

059422c

fix: custom error type for unknown manifest schema

4f868da

style: respect naming convention on manifest vars

b44ef37

feat: stricter manifest path validation

85ea2b6

fix: allocate missingPaths slice at once

91780f4

feat: log when keeping non empty dir

7bd6655

fix: always return an error or non-nil valid manifest

908b49b

style: formating

e68f1f6

refactor: custom error type for no manifest errors

3f19c2c

docs: improve comment on SelectValidManifest

59a4d7f

style: be more consistent with existing terminology

d1c3919

fix: harmonize error message when applying recut

ac1be1b

fix: simplify contentHash

5cda705

Revert "fix: simplify contentHash"

7dd133d

This reverts commit 5cda705.

fix: rework strategy to apply recut

0e892f7

feat: refine upgrade process

18a7b22

fix: lint

4550747

upils commented Apr 3, 2026

View reviewed changes

upils requested a review from niemeyer April 7, 2026 12:43

niemeyer requested changes Apr 13, 2026

View reviewed changes

upils mentioned this pull request Apr 20, 2026

feat: select and validate manifest #288

Open

1 task

upils marked this pull request as draft April 23, 2026 12:06

Conversation

upils commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

letFunny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

letFunny left a comment

Choose a reason for hiding this comment

Uh oh!

niemeyer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upils Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

upils Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niemeyer left a comment

Choose a reason for hiding this comment

Uh oh!

upils commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

upils commented Jan 29, 2026 •

edited

Loading

github-actions Bot commented Jan 29, 2026 •

edited

Loading

upils Apr 1, 2026 •

edited

Loading

upils Apr 3, 2026 •

edited

Loading