Conda install fails on Hera if installation directory is more than 90 characters in length

[mkavulich]

Expected behavior

Conda should install successfully

Current behavior

On Hera, if your working directory that you install the app in is longer than 90 characters, the conda install fails.

I swear this is not a joke.

Machines affected

Hera, potentially others but I doubt it? 90 characters doesn't seem all that long...

Steps To Reproduce

1. On Hera, attempt to install the ufs-srweather-app in a working directory that has a path length of 91 or more characters including slashes
2. Observe that the conda environment fails in a strange way when attempting to build new libraries:

/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app$ ./devbuild.sh -p=hera
PLATFORM(MACHINE)=hera
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 82.9M  100 82.9M    0     0  2617k      0  0:00:32  0:00:32 --:--:-- 1869k
PREFIX=/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda
Unpacking payload ...
Extracting _libgcc_mutex-0.1-conda_forge.tar.bz2
Extracting ca-certificates-2023.7.22-hbcca054_0.conda
Extracting ld_impl_linux-64-2.40-h41732ed_0.conda

...
...

Linking conda-23.3.1-py310hff52083_0
Linking conda-libmamba-solver-23.3.0-pyhd8ed1ab_0
Linking mamba-1.4.2-py310h51d5547_0
Transaction finished
installation finished.
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest11111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest11111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest11111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'

The build then continues as normal, but the conda environments were not built and so can not be activated:

/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app$ module use modulefiles/
/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app$ module load wflow_hera
Please do the following to activate conda:
       > conda activate srw_app
/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app$ conda activate srw_app

EnvironmentNameNotFound: Could not find conda environment: srw_app
You can list all discoverable environments with `conda info --envs`.

When I build in a path with one fewer character length (/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest111/ufs-srweather-app), it is successful.

Detailed Description of Fix (optional)

🤷

This feels like the case of the 500 mile email. Why on earth is there a character limit on path lengths somewhere deep in this pile of spaghetti, and why is manifesting in such an obscure way???

[MichaelLueken]

@mkavulich -

On Hera, I created the over-the-top directory - /scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app and encountered the same issue that you did:

Linking conda-libmamba-solver-23.3.0-pyhd8ed1ab_0
Linking mamba-1.4.2-py310h51d5547_0
Transaction finished
installation finished.
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/scratch3/NCEPDEV/stmp/Michael.Lueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'

The maximum path length for a Linux platform should be set in the /usr/include/linux/limits.h file:

#define NAME_MAX         255	/* # chars in a file name */
#define PATH_MAX        4096	/* # chars in a path name including nul */

I'll try this on other machines and report back what I see.

[MichaelLueken]

Gaea C6, Hercules, Orion, and Ursa suffer a different, but similar, issue:

Linking conda-libmamba-solver-23.3.0-pyhd8ed1ab_0
Linking mamba-1.4.2-py310h51d5547_0
Transaction finished
installation finished.
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory
/usr/bin/env: ‘python’: No such file or directory

Derecho has the identical failure to Hera:

Linking conda-libmamba-solver-23.3.0-pyhd8ed1ab_0
Linking mamba-1.4.2-py310h51d5547_0
Transaction finished
installation finished.
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
  File "/glade/derecho/scratch/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba", line 7, in <module>
    from mamba.mamba import main
ModuleNotFoundError: No module named 'mamba'

[MichaelLueken]

@mkavulich -

The issue is the shebang in conda and mamba when the path to the installation directory has a large number of characters.

On Hercules, when using /work/noaa/epic/mlueken/ufs-srweather-app/hercules, the shebang in conda/bin/conda looks like:

#!/work/noaa/epic/mlueken/ufs-srweather-app/hercules/conda/bin/python

Using the long directory, /work/noaa/epic/mlueken/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app, the shebang looks like:

#!/usr/bin/env python

It appears as though there is a system level limitation on shebangs, between 80 and 127 characters.

[MichaelLueken]

@mkavulich -

To correct this issue, please try manually replacing

#!/usr/bin/env python

in

/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/conda

and

/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/condabin/mamba

with

#!/scratch4/BMC/gsd-fv3-dev/Michael.Kavulich/UFS/workdir/FIRE/openmp_changes/pathnametest1111/ufs-srweather-app/conda/bin/python

The srw_app, srw_graphics, and srw_sd conda environments successfully build following these two modifications.

The fundamental WE2E tests were successfully generated and all passed successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              11.29
custom_ESGgrid_Central_Asia_3km_20251103084511                     COMPLETE              56.84
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              13.40
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2025110  COMPLETE              32.30
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20251103084  COMPLETE              25.61
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2025110308462  COMPLETE              22.82
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             162.26

[mkavulich]

@MichaelLueken I did confirm that the manual fix worked, but is there any chance for an automated fix for this? It looks like @gsketefian also ran into this independently (#1350).

[mkavulich]

@MichaelLueken Upon some more investigation, it appears as if the fix is to update to a newer version of Miniforge that uses mamba 2+. I have implemented this in another project (with the latest Miniforge, 26.1.0-0) and it appears to work. I'm running a test with SRW now as well.

Nevermind, I was mistaken, this does not appear to fix the issue.

[zhanglin2603]

Hey, I went over #1329 and can take this on.

• My read: this is primarily a UI/UX task tied to "Conda install fails on Hera if installation directory is more than 90 characters in length".
• I will focus on these issue-specific points first: conda, install, fails.
• First I will reproduce and confirm expected behavior for the UI/UX path.
• Then I will implement the fix with targeted tests and keep scope tight (good confidence).
• I will share a working PR update fast (24h for first patch, 48h for final polish) with verification notes.
  If this approach works for you, I can start now.