[MUSA] Add shell script to generate requirements-musa.txt and update doc #1175
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,3 +6,4 @@ dist | |
| .idea | ||
| .vscode | ||
| tmp/ | ||
| requirements-musa.txt | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,105 @@ | ||||||
| #!/bin/bash | ||||||
| # Script to generate requirements-musa.txt from requirements.txt | ||||||
| # MUSA is not compatible with CUDA packages, so they need to be removed | ||||||
| # Torch-related packages are pre-installed in the MUSA docker container | ||||||
|
|
||||||
| set -e | ||||||
|
|
||||||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||||||
| INPUT_FILE="${SCRIPT_DIR}/requirements.txt" | ||||||
| OUTPUT_FILE="${SCRIPT_DIR}/requirements-musa.txt" | ||||||
|
|
||||||
| if [ ! -f "$INPUT_FILE" ]; then | ||||||
| echo "Error: requirements.txt not found at $INPUT_FILE" | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| echo "Generating requirements-musa.txt from requirements.txt..." | ||||||
|
|
||||||
| # Define patterns to remove (CUDA-specific packages) | ||||||
| # These packages are not compatible with MUSA | ||||||
| CUDA_PACKAGES=( | ||||||
| "^cupy" # cupy-cuda12x and similar | ||||||
| "^cuda_bindings" # CUDA bindings | ||||||
| "^nixl" # NIXL (NVIDIA Inter-node eXchange Library) | ||||||
| "^flashinfer" # flashinfer-python (CUDA-specific attention kernel) | ||||||
| "^sgl-kernel" # SGL kernel (CUDA-specific) | ||||||
| ) | ||||||
|
|
||||||
| # Define torch-related packages (pre-installed in MUSA container, remove version pins) | ||||||
| TORCH_PACKAGES=( | ||||||
| "^torch==" | ||||||
| "^torchvision==" | ||||||
| ) | ||||||
|
|
||||||
| # Create the output file with a header comment | ||||||
| cat > "$OUTPUT_FILE" << 'EOF' | ||||||
| # Requirements for MUSA (Moore Threads GPU) | ||||||
| # Auto-generated from requirements.txt by generate_requirements_musa.sh | ||||||
| # CUDA-specific packages have been removed | ||||||
| # Torch-related packages have version pins removed (pre-installed in MUSA container) | ||||||
|
|
||||||
| EOF | ||||||
|
|
||||||
| # Process the requirements file | ||||||
| while IFS= read -r line || [ -n "$line" ]; do | ||||||
| # Skip empty lines and comments (but keep them in output) | ||||||
| if [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]]; then | ||||||
| echo "$line" >> "$OUTPUT_FILE" | ||||||
| continue | ||||||
| fi | ||||||
|
|
||||||
| # Extract package name (before ==, >=, <=, ~=, etc.) | ||||||
| pkg_name=$(echo "$line" | sed -E 's/^([a-zA-Z0-9_-]+).*/\1/') | ||||||
|
Contributor
There was a problem hiding this comment. The regular expression used to extract the package name is not robust. It doesn't account for dots (
Suggested change
|
||||||
|
|
||||||
| # Check if this is a CUDA package to skip | ||||||
| skip=false | ||||||
| for pattern in "${CUDA_PACKAGES[@]}"; do | ||||||
| if [[ "$pkg_name" =~ $pattern ]]; then | ||||||
| echo " Removing CUDA package: $line" | ||||||
| skip=true | ||||||
| break | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| if $skip; then | ||||||
| continue | ||||||
| fi | ||||||
|
|
||||||
| # Check if this is a torch-related package (remove version pin) | ||||||
| for pattern in "${TORCH_PACKAGES[@]}"; do | ||||||
| if [[ "$line" =~ $pattern ]]; then | ||||||
| # Remove version pin, keep just the package name | ||||||
| pkg_only=$(echo "$line" | sed -E 's/==.*//') | ||||||
| echo " Unpinning version for: $pkg_only (pre-installed in MUSA container)" | ||||||
| echo "$pkg_only" >> "$OUTPUT_FILE" | ||||||
| skip=true | ||||||
| break | ||||||
| fi | ||||||
| done | ||||||
|
|
||||||
| if $skip; then | ||||||
| continue | ||||||
| fi | ||||||
|
|
||||||
| # Keep the package as-is | ||||||
| echo "$line" >> "$OUTPUT_FILE" | ||||||
|
|
||||||
| done < "$INPUT_FILE" | ||||||
|
Comment on lines
+45
to
+88
Contributor
There was a problem hiding this comment. Throughout this loop, For example:
|
||||||
|
|
||||||
| # Add MUSA-specific packages at the end | ||||||
| cat >> "$OUTPUT_FILE" << 'EOF' | ||||||
|
|
||||||
| # MUSA-specific packages | ||||||
| torch_musa | ||||||
| torchada | ||||||
| EOF | ||||||
|
|
||||||
| echo "" | ||||||
| echo "Successfully generated: $OUTPUT_FILE" | ||||||
| echo "" | ||||||
| echo "Summary of changes:" | ||||||
| echo " - Removed CUDA-specific packages: cupy-cuda12x, cuda_bindings, nixl, flashinfer-python, sgl-kernel" | ||||||
| echo " - Unpinned torch-related packages: torch, torchvision (pre-installed in MUSA container)" | ||||||
| echo " - Added MUSA-specific packages: torch_musa, torchada" | ||||||
|
|
||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The command
./generate_requirements_musa.shrequires the script to have execute permissions. A user cloning the repository might not have this permission set by default, forcing them to runchmod +xfirst. To provide a smoother experience, you can invoke the script withbashdirectly, which doesn't require the execute bit.