Add SAI GPU validation and multinode cuSolverMp smoke

[Stardust0831]

Reminder

• I have read AGENTS.md and docs/developers_guide/agent_governance.md.
• I have linked an issue or explained why this PR does not need one.
• I have added adequate unit tests and case tests.
• I have listed the exact verification commands run and their results.
• I have described user-visible behavior changes, including INPUT parameter changes.
• I have explained core-module impact.
• I have requested any needed governance exception below.

Motivation and context

I first appreciated the practical value of a continuously running daily regression service after contributing code to CP2K. Its dashboard sent an automated notification that one of my recent commits might have introduced a failure on a specific compiler configuration. The message identified the affected tester, the tested commit, a concise result summary, the last known-good commit, and a link to the archived report.

That feedback made the role of this kind of CI concrete for me. A useful HPC regression service does more than run checks at merge time: it continuously tests the default branch on environments that ordinary GitHub-hosted runners cannot reproduce, retains enough evidence to diagnose failures, and gives maintainers a controlled way to test selected pull-request commits on scarce or privileged hardware.

CP2K keeps site-oriented CI definitions for CSCS Daint and Eiger under ci/, while its public Dashboard aggregates the reports. Quantum ESPRESSO keeps site-oriented configuration under .ci/; its cineca.yml records module-based CPU/GPU jobs for CINECA systems. This PR follows the same broad pattern: cluster-specific orchestration is reviewable with the code, while credentials and site services remain outside the repository.

Linked issue

Related to #7657. This PR adds SAI regression coverage and does not close the NVIDIA library issue. The direct include fix from #7666 and the cuSolverMp version policy from #7658 are already present in the current develop base; this PR no longer depends on another open PR.

What changed?

• Add a daily, approval-free run of the repository default branch and a protected manual trigger for an exact commit reviewed by an authorized maintainer. There is no automatic pull_request trigger.
• Keep the SSH private key in GitHub Environment secrets, pin the SAI host key, pin privileged third-party Actions by commit SHA, and take the control plane from the trusted default branch. Approved candidate code still executes unsandboxed as the configured SAI user; the documentation states this trust boundary explicitly.
• Add the same control path as a local launcher for developers using their own OpenSSH Host alias and SAI account. Usernames and identity files remain in local SSH configuration rather than repository files.
• Use the site-managed abacus/develop-git-079fd0c-260724-sm70-auto module as the dependency-stack anchor. It supplies the validated Open MPI 5.0.10, CUDA 12.9.1, SAI NCCL 2.29.3, cuSolverMp 0.9.0, cuBLASMp 0.9.1, ELPA, and SAIBLAS environment. The requested SOURCE_SHA is still rebuilt into an isolated run directory; the module's installed ABACUS binary is never used as the candidate binary.
• Verify module identity, header and runtime library versions, resolved cuSolverMp/cuBLASMp/NCCL/MPI/CUDA runtime paths, and the SAI NCCL rail policy before accepting a build. The workflow does not download NVIDIA libraries or modify /opt, module files, or system configuration.
• Transfer source as a gzip-compressed full snapshot or Git delta. GitHub runs store the payload in a one-day Actions artifact and give SAI only a short-lived Azure Blob URL. SAI downloads eight bounded ranges in parallel, reconstructs the ZIP, validates its exact entries and size, and verifies the resulting Git tree. The GitHub token never leaves the runner.
• Isolate concurrent scheduled, manual, and local runs by namespace. Manual candidates may reuse a verified daily baseline but cannot advance it.
• Submit all 48 existing GPU cases as resource-homogeneous Slurm arrays with concurrency caps 2/8/8. A narrow PMIx startup signature retries once; numerical failures and timeouts do not retry. No job pins a node or explicitly requests CPU resources.
• Add a reconstructed Si48 solid case and run it as a separate 2-node, 16-rank/16-GPU cuSolverMp RT-TDDFT functional smoke. It checks successful completion and that both requested propagation steps were reached; it is not a numerical or performance reference.
• Materialize both .ci/sai and the Si48 fixture from the trusted CONTROL_SHA. A candidate commit can therefore predate this PR and still run the multinode smoke. The candidate supplies only the shared tests/PP_ORB data.
• Add bounded Slurm-accounting failure handling, whitelisted artifact collection, and a user-level cleanup installer that removes uploaded run data after 72 hours.

Repository layout

GitHub entry points remain under .github/workflows/, the Si48 scientific fixture remains under the existing tests/ tree, and reusable GitHub/local/SSH/Slurm controls live under .ci/sai/. This adds one hidden top-level .ci/ directory and does not add a parallel test hierarchy.

Verification

The refreshed branch is based on the current upstream develop. The following commands were run on commit 014c62d3ed12eb24b385becf9a94d4a580801ee6:

bash .ci/sai/tests/test_sai_ci.sh
find .ci/sai -type f \( -name "*.sh" -o -name "*.sbatch" -o -name "*.env.example" \) -print0 | xargs -0 bash -n
shellcheck -S error .ci/sai/*.sh .ci/sai/*.sbatch .ci/sai/tests/test_sai_ci.sh .ci/sai/toolchains/*.env.example
actionlint .github/workflows/sai-bootstrap.yml .github/workflows/sai-gpu-full.yml
python3 tools/03_code_analysis/agent_governance_check.py --base upstream/develop --head HEAD --format text
git diff --check upstream/develop...HEAD

Results:

• SAI control regression suite: 22/22 passed.
• Bash syntax, ShellCheck 0.9.0, actionlint 1.7.7, workflow YAML parsing, governance, and range diff checks passed.
• The generated C runtime-version probe compiled with -std=c11 -Wall -Wextra -Werror.
• End-to-end fork run 30114759440 passed all 48 GPU matrix cases and the 2-node/16-GPU Si48 cuSolverMp smoke.
• A later clean-upstream candidate run passed all 48 matrix cases but exposed that the old control script expected the candidate itself to contain the Si48 fixture. The refreshed control snapshot removes that coupling and has a focused local regression test. A new full SAI run of this refreshed head has not yet completed.

Governance notes

• INPUT/docs changes: no ABACUS INPUT behavior changes. The new INPUT belongs only to the Si48 smoke fixture and is documented in its README and .ci/sai/README.md.
• Core module impact: none; this PR does not modify source/.
• Existing numerical references and thresholds are not relaxed.
• Exceptions requested: none.
• Operator setup: before enablement, maintainers must configure the sai-ssh-scheduled and protected sai-ssh-manual Environments, variables, and Environment-only SSH secret as documented in .ci/sai/README.md.

[AsTonyshment]

I will convert this PR into draft temporarily since it depends on the merge of #7658.

[Stardust0831]

@AsTonyshment Thanks.
I moved the change that PR #7665 actually depends on into the standalone PR #7666, which adds the required direct includes and normalizes the formatting of diago_cusolvermp.cpp; this decouples #7665 from #7658 and its separate discussion about whether cuSolverMp versions below 0.9.0 should be blocked entirely.

[Copilot]

Pull request overview

Adds an SSH-based, GitHub-hosted control-plane workflow to run ABACUS GPU validation on the SAI Slurm cluster (including a new multinode cuSolverMp RT-TDDFT smoke), plus supporting scripts for secure source transfer, job submission, artifact collection, and automated retention cleanup.

Changes:

• Introduce GitHub Actions workflows for scheduled and manually-approved SAI GPU validation runs over pinned-host-key SSH.
• Add a comprehensive ci/sai/ control suite (toolchain setup, Slurm orchestration, source snapshot/delta transfer cache, artifact collection, cleanup cron installation, and regression tests).
• Add a reconstructed Si48 RT-TDDFT cuSolverMp smoke fixture case for 2-node / 16-GPU functional completion checks.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/STRU	New Si48 structure fixture for multinode cuSolverMp smoke.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/README	Documents provenance and non-reference intent of the Si48 smoke case.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/KPT	Gamma-only KPT for the Si48 smoke case.
tests/15_rtTDDFT_GPU/19_NO_Si48_CUSOLVERMP_TDDFT_GPU/INPUT	RT-TDDFT GPU INPUT selecting ks_solver cusolvermp.
ci/sai/toolchains/archive-mp09-sai-nccl2293.env.example	Example SAI module/toolchain loader for CUDA/NVHPC/OpenMPI + MP/NCCL selection.
ci/sai/tests/test_sai_ci.sh	Local regression tests for the SAI control scripts (security and policy checks).
ci/sai/test_gpu.sbatch	2-node Slurm job script for the multinode cuSolverMp RT-TDDFT smoke run.
ci/sai/test_gpu_case.sh	Per-array-task launcher for running one GPU test case with retry metadata.
ci/sai/summarize_gpu_case_matrix.sh	Produces a markdown summary and CI annotations for the 48-case matrix.
ci/sai/source_transfer_cache.sh	Implements locked baseline/candidate source snapshot + delta transfer and verification.
ci/sai/run_slurm_job.sh	Submits and monitors Slurm jobs with cancellation on signal/exit.
ci/sai/run_remote_ci.sh	Remote coordinator entrypoint for preparing deps, building, and running validations.
ci/sai/run_gpu_validation.sh	Coordinates running the GPU arrays and multinode smoke concurrently.
ci/sai/run_gpu_case_matrix.sh	Submits the GPU case matrix as Slurm arrays and records accounting state.
ci/sai/run_gpu_case_attempts.sh	Runs a case with a single allowed PMIx-startup retry and writes TSV metadata.
ci/sai/README.md	Operator/user documentation for trust boundary, triggers, caching, artifacts, cleanup.
ci/sai/probe_remote_sai.sh	Remote capability probe used by workflows before launching the run.
ci/sai/prepare_remote_run.sh	Creates an isolated run directory on SAI with containment/collision checks.
ci/sai/prepare_nvidia_mp.sh	Fetches/verifies/caches NVIDIA MP archives with SHA256 pinning and atomic install.
ci/sai/prepare_gpu_case_matrix.sh	Builds the manifest inventory for the 48 existing GPU cases.
ci/sai/prepare_cusolvermp_smoke.sh	Stages a fixed-input cuSolverMp smoke case with file hash validation.
ci/sai/prepare_cleanup_install.sh	Stages cleanup installer artifacts in a diagnostic directory for bootstrap.
ci/sai/mpirun_with_mapping.sh	Wrapper to apply SAI partition mapping options to mpirun.
ci/sai/mark_artifacts_uploaded.sh	Atomically writes a marker used by retention cleanup after artifact upload.
ci/sai/known_hosts	Pinned SAI SSH host key material for StrictHostKeyChecking.
ci/sai/install_cleanup_cron.sh	Installs cleanup script into user libexec and manages a marked crontab block.
ci/sai/configure_ssh_client.sh	Creates a locked-down SSH config/key directory from an Environment secret.
ci/sai/collect_remote_artifacts.sh	Whitelist-based artifact collector producing a tar.gz stream from the run root.
ci/sai/cleanup_sai_runs.sh	Retention cleanup for uploaded/incomplete runs, diagnostics, and old transfers.
ci/sai/build_gpu.sh	Builds ABACUS with selected CUDA/NCCL/cuSolverMp/cuBLASMp stack and records linkage.
ci/sai/build_gpu.sbatch	Slurm submission script for the build step.
.github/workflows/sai-gpu-full.yml	Main scheduled/manual workflow driving SSH-based rebuild + validation + artifacts.
.github/workflows/sai-bootstrap.yml	Manual bootstrap workflow for probing SAI and installing cleanup cron.