package hash differs between operating systems due to file system case sensitivity

[mitchellh]

Zig Version

0.12.0-dev.1647+325e0f5f0

Steps to Reproduce and Observed Behavior

.{
    .name = "test",
    .version = "0.1.0",
    .paths = .{""},
    .dependencies = .{
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
        },
    },
}

On macOS: 12201fff83e91996f5c6abf51c8d0a54630926d81564f75681e1972575058d6476fc
On Linux: 12202a04dfdfc563f0c74a61559a549c194405f0ef562e5c20e32066bde0bbcd1020

Expected Behavior

Hashes should match. I'm not sure what is causing the problem yet.

[andrewrk]

First step to troubleshooting this will be to run zig fetch --debug-hash https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz on each OS, and then diff the two outputted text files.

[oldendick]

diff -u iterm-debug-hash-macos.txt iterm-debug-hash-linux.txt 
--- iterm-debug-hash-macos.txt	2023-11-23 02:23:17
+++ iterm-debug-hash-linux.txt	2023-11-23 02:24:46
@@ -576,6 +576,8 @@
 file: b12ae24abcc570ee232cb3f3b52e329d7031128a7ad062c9590fd897da01d827: alacritty/catppuccin-mocha.yml
 file: e9b349dc1f185b091aedeef71918d4686e11fef7c97fa7f1999da607126aa3f9: alacritty/coffee_theme.yml
 file: 4b3227494faae53ae31d79a47808f0bd99eed99727417794a008383fbf2e8665: alacritty/cyberpunk.yml
+file: a2cb474b347a6adf5a3aeb8cbc0a0ba2ea0b1f7774188634ff531433434e57e8: alacritty/darkermatrix.yml
+file: 522281b9703073d6cdfa91048151cfec3607aa25224d007e669b05af05463206: alacritty/darkmatrix.yml
 file: 17b6b72f117e5ab584ee10e4117d154ccbb88b7385f0c5eccbfc067dd72c08c4: alacritty/deep.yml
 file: d8c24d86c8102b03864fd75e5b109fd62503af27325a6afa545b5cef18fbddc4: alacritty/duckbones.yml
 file: bc74eda647449e0effb0600193648a9d8c6c92123053c140e5f3f3c66d594963: alacritty/flexoki-dark.yml
@@ -596,6 +598,7 @@
 file: f9460bee42703abee47a4315cd5267caccc105087282aea2e2df7a1052cbd701: alacritty/jubi.yml
 file: 25165b9ed2bfdb6594aa12d2cd758177280a44891eaabeae6aa9768a5e497460: alacritty/kanagawabones.yml
 file: f2ded6bdd2ee86ae91b8669a7f7fa1960f7e30869e06ac828f5aca9ae03cc38e: alacritty/lovelace.yml
+file: 05b5ad88a0a4309806776f106bfcaea4fb3a6af2e6af477a80b8fc1e4a2186c7: alacritty/matrix.yml
 file: 688838a8c20bd2a504d7c97f5a5964a69c2fe483c4962a54baf82bef70af3d63: alacritty/midnight-in-mojave.yml
 file: 703030abe167129b2a306680b7592b771cf7e444c6ea0ea1f6be82e9866e3a44: alacritty/neobones_dark.yml
 file: b16fd79bb0d599fd24bd70a0c17e5f38a2f2e5002b36ad05eb14a1f22b57956c: alacritty/neobones_light.yml
@@ -8001,6 +8004,8 @@
 file: 723825d2f244f3a7e2a6e89dd3560573c74bf1863677d29017a8c7a30631ba70: vscode/catppuccin-mocha.json
 file: 080dce3c0e2db8deff9e2188c61eccc319adf70f0b651a622f5d4870b3363d82: vscode/coffee_theme.json
 file: 7fae2360d810859ce6e439d81dfa335e5c56e39b3893892f60404f09ee177eda: vscode/cyberpunk.json
+file: 107b278672c8062fbdc99395e088f02f7cf0947a352d6846831e80fe574c71c7: vscode/darkermatrix.json
+file: 1a59da5a1ed1bc8bf396307f000ffa25b59902a20b29cdff956c3110fcbf411f: vscode/darkmatrix.json
 file: e28045bf63466ca7d64678543614f69c1cd0dec68100c67713e470f4e13e31f0: vscode/deep.json
 file: d1dfa07bb7ba6b7ac65f4f99d55d16f5f34a926455dee4f0c10014428e6b2347: vscode/duckbones.json
 file: 388bb8062225896879405b359decb62981cb5c0babee070a51744d5d72669b65: vscode/flexoki-dark.json
@@ -8021,6 +8026,7 @@
 file: 9cdf866d0c174010b652c46ce86a91e790bde54d0e6ef1346c3f92b123ceead1: vscode/jubi.json
 file: 717e9e69d64962908337c827b0c2c1f21fcffefc27f87516fd8d7642dcc425c7: vscode/kanagawabones.json
 file: 788f321a66832d07097642d22d3e49b9c0d346f1b167c8c8f86591144dd1f832: vscode/lovelace.json
+file: 415d3b9aa10d1e175ae72c7966be0b609446b094201267c49ebd492674192a63: vscode/matrix.json
 file: 4c955ab8ea102dfddef6345c572620abcbe330fe94cee74426510a9510961064: vscode/midnight-in-mojave.json
 file: d654d67d2e224296a347f3ec70b960d7976c0537309deb27032875db054ddd0f: vscode/neobones_dark.json
 file: 0ddb80da587ca8d107609f8c4d8b59e84238317f01b4583f0cf8bd209df55b48: vscode/neobones_light.json
@@ -9289,4 +9295,4 @@
 file: 0ff2794fecfc090fe98c4e879bb86b1b2ab578055828ba9a513d15cbcaea62c7: xrdb/zenburned.xrdb
 file: ef008b8cb4f23378482040df8bffbc4ae30d77472992805a00287b9bd05a31aa: xrdb/zenwritten_dark.xrdb
 file: a9948901a3da22b27ab71f132a2f8aec8b573199efda01deb048e86c04b406e3: xrdb/zenwritten_light.xrdb
-12201fff83e91996f5c6abf51c8d0a54630926d81564f75681e1972575058d6476fc
+12202a04dfdfc563f0c74a61559a549c194405f0ef562e5c20e32066bde0bbcd1020

The mac side seems to show 6 fewer files. Upon looking at the tar, these files are included in the tar twice with different case sensitivity in the filename.

i.e.:

tar ztvf aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz | egrep -i "alacritty/darkermatrix.yml|alacritty/darkmatrix.yml|alacritty/matrix.yml|vscode/darkermatrix.json|vscode/darkmatrix.json|vscode/matrix.json" | sort -k5
-rw-rw-r--  0 root   root      554 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Matrix.yml
-rw-rw-r--  0 root   root      554 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/matrix.yml
-rw-rw-r--  0 root   root      558 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Darkmatrix.yml
-rw-rw-r--  0 root   root      558 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/darkmatrix.yml
-rw-rw-r--  0 root   root      560 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/Darkermatrix.yml
-rw-rw-r--  0 root   root      560 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/alacritty/darkermatrix.yml
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Darkermatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Darkmatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/Matrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/darkermatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/darkmatrix.json
-rw-rw-r--  0 root   root      933 Oct 11 09:37 iTerm2-Color-Schemes-aed2a8f0565a51913e8ddc1731f25bc71978a1e0/vscode/matrix.json

[der-teufel-programming]

I've found the same comparing Windows and Linux. Windows gives the same hash as MacOS and the missing files are the same as listed above

[slonik-az]

Should one allow OS specific hashes in zon dependency description? Otherwise one will be fighting these corner cases forever.

[der-teufel-programming]

This would be problematic if the files with the same case insensitive name, but different case sensitive name actually contained different data

[marler8997]

Oooof...differences in case-sensitivity of file names strikes again.

My suggestion would be for Zig to detect and by default reject archives that aren't usable on all supported platforms. So in this case you'd get:

$ zig build
error: dependency `iterm2_themes` contains the following filename conflicts:

alacritty/Darkermatrix.yml
alacritty/darkermatrix.yml

alacritty/Darkmatrix.yml
alacritty/darkmatrix.yml

...

This solution might be "good enough" and will encourage developers to create archives that are more portable. That being said...if we do want to support these types of archives (not saying we should), we have some options. We could allow projects to opt-in to allowing these conflicts, i.e.

.{
    // ...
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
            .allow_filename_casing_conflicts = true,
        }
}

Note this would mean the package becomes usable when building on a system where zig's package cache is on a case-insensitive filesystem. This would break cross-compilation to/from particular targets which is unfortunate. A better option could be to provide some sort of way to modify how the archive is extracted to resolve the filename conflicts, i.e.

.{
    // ...
        .iterm2_themes = .{
            .url = "https://github.com/mbadolato/iTerm2-Color-Schemes/archive/aed2a8f0565a51913e8ddc1731f25bc71978a1e0.tar.gz",
            .renames = &.{
                &.{ "alacritty/Darkmatrix.yml", "alacritty/darkmatrix2.yml" },
            },
        }
}

This means no longer having a requirement on a case-sensitive filesystem and the dependency is uniform across all platforms.

This being said, I think it's reasonable to start with simply rejecting these types of archives and allow authors to repackage them in a more portable form, after all they'd be the best ones to know the correct changes. Enabling new exotic cross-compilation can be used as a motivator for these kinds of changes. We can think about whether we'd like to make it easy to patch archives separately.

P.S. It seems weird that Windows/macOS, the 2 big OS's backed by large companies happen to be the problem children that don't use case-sensitive filesystems by default. I expect this from Windows but macOS?

[xdBronch]

once #14597 is solved one could conditionally allow packages with conflicts like these on platforms where its allowed

[mitchellh]

I've also made an upstream PR to fix the source repo: mbadolato/iTerm2-Color-Schemes#425

We probably should still figure out the proper behavior for Zig though in these cases.