[WIP][Fix] Fix lld invocation failed when ROCm is not at the baked-in path - #987
Open
jli-melchior wants to merge 2 commits into
Open
[WIP][Fix] Fix lld invocation failed when ROCm is not at the baked-in path#987jli-melchior wants to merge 2 commits into
lld invocation failed when ROCm is not at the baked-in path#987jli-melchior wants to merge 2 commits into
Conversation
jli-melchior
changed the title
lld invocation failed when ROCm is not at the baked-in pathlld invocation failed when ROCm is not at the baked-in path
`gpu-module-to-binary` spawns `<toolkit>/llvm/bin/ld.lld`, where `<toolkit>` comes from ROCM_PATH/ROCM_ROOT/ROCM_HOME or from `__DEFAULT_ROCM_PATH__`, a path baked into the LLVM build. Any container that installs ROCm somewhere else fails, and upstream reports only `lld invocation failed` without naming the path it tried. Our own LLVM bakes in the build machine's ROCm wheel path, so an image switch is enough to break it. Add `fly-emit-gpu-binary`, a wrapper that runs upstream only as far as `format=isa` -- which returns before the toolkit is consulted -- then finishes in process: `mlir::ROCDL::assembleIsa` for the AMDGPU MC assembler and the LLD ELF driver for the link. That removes the lookup entirely and pins the linker to the LLVM revision that produced the ISA. LLD is resolved from the same install that provides MLIR, with NO_DEFAULT_PATH so a mismatched system LLD can never be picked up, and is linked statically into the shared library. An LLVM built without the lld project keeps the upstream behavior via FLYDSL_HAS_LLD_LIBRARY. Two details are carried over from Triton's equivalent change (#7548): `--threads=1`, because LLD's use of the LLVM thread pool deadlocks in `~TaskGroup()` inside a forked child, and checking `canRunAgain`, because a JIT links thousands of times per process and must not continue on corrupted linker state. The FLYDSL_COMPILE_LLVM_DIR path still uses `gpu-module-to-binary`: it drives an upstream mlir-opt that does not know FlyDSL passes. Verified on the same IR with ROCM_PATH pointing at a nonexistent directory: upstream reports `lld invocation failed`, the new pass emits a valid ET_DYN/EM_AMDGPU `gpu.binary`. Full tests/unit + tests/system swept per file against the upstream pass: identical results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Boss2002n
force-pushed
the
jli/fix-lld-issue
branch
from
August 10, 2026 02:34
ee696ee to
5bb47b5
Compare
…ntainer Linking in process removed the ld.lld lookup, but `appendStandardLibs()` still reads `<toolkit>/amdgcn/bitcode` whenever a module references `__ocml_*` or `__ockl_*`, and errors out if that directory is absent. FlyDSL reaches it: `convert-gpu-to-rocdl` pulls in MathToROCDL, so scalar math ops with no LLVM intrinsic lower to ocml calls -- `fx.erfc` becomes `__ocml_erfc_f32`. So the per-image path problem survived in the device-library half. Bundle the bitcode the way Triton does. CMake locates ocml/ockl/hip/opencl at configure time (ROCM_PATH/ROCM_ROOT/ROCM_HOME or /opt/rocm, overridable with -DFLYDSL_ROCM_BITCODE_DIR) and copies them into the package; those four are the complete set appendStandardLibs() can request, since the oclc_* control variables are synthesized in-module rather than read from disk. They land under `_mlir/`, which is already the packaged build-output subtree, so the editable symlink and the wheel's package_dir mapping carry them with no new plumbing. `rocm_toolkit_path()` then resolves FLYDSL_COMPILE_ROCM_PATH, the bundled tree, and ROCM_PATH/ROCM_ROOT/ROCM_HOME in that order and passes the winner as `toolkit=`. Finding nothing passes no option at all, leaving upstream's lookup untouched rather than forcing a known-bad path on it. `RocmBackend.hash()` folds the bitcode digest into the JIT cache key: swapping ocml.bc changes generated code without changing any FlyDSL shared library, so the native-library hashes alone would serve stale artifacts. Verified with ROCM_PATH/ROCM_ROOT/ROCM_HOME all unset: a kernel calling fx.erfc compiles and runs. Hiding the bundled ocml.bc makes that same compilation fail on the bundled path, confirming which copy is consumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Boss2002n
force-pushed
the
jli/fix-lld-issue
branch
from
August 10, 2026 06:00
5bb47b5 to
e1e46e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist