Gkarch/update 1314

.claude/skills/common/slurm-setup.md

@@ -192,3 +192,128 @@ chmod -R g+rwX /path/to/.hf_cache/
 ```
 
 Scope `chmod` to only the directories the job needs — avoid world-writable paths on shared clusters.
+
+---
+
+## 6. Container Registry Authentication
+
+**Before submitting any SLURM job that pulls a container image**, check that the cluster has credentials for the image's registry. Missing auth causes jobs to fail after waiting in the queue — a costly mistake.
+
+### Step 1: Detect the container runtime
+
+Different clusters use different container runtimes. Detect which is available:
+
+```bash
+# On the cluster (or via ssh):
+which enroot 2>/dev/null && echo "RUNTIME=enroot"
+which docker 2>/dev/null && echo "RUNTIME=docker"
+```
+
+| Runtime | Typical clusters | SLURM integration |
+| --- | --- | --- |
+| **enroot/pyxis** | NVIDIA internal (DGX Cloud, EOS, Selene, GCP-NRT) | `srun --container-image` |
+| **Docker** | Bare-metal / on-prem with GPU | `docker run` inside job script |
+
+### Step 2: Check credentials for the image's registry
+
+Determine the registry from the image URI:
+
+| Image pattern | Registry |
+| --- | --- |
+| `nvcr.io/nvidia/...` | NGC |
+| `vllm/vllm-openai:...`, `lmsysorg/sglang:...`, or no registry prefix | DockerHub |
+| `ghcr.io/...` | GitHub Container Registry |
+| `docker.io/...` | DockerHub (explicit) |
+
+Then check credentials based on the runtime:
+
+#### enroot/pyxis
+
+```bash
+grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null
+```
+
+Look for `machine <registry>` lines:
+- NGC → `machine nvcr.io`
+- DockerHub → `machine auth.docker.io`
+- GHCR → `machine ghcr.io`
+
+#### Docker
+
+```bash
+cat ~/.docker/config.json 2>/dev/null | python3 -c "import json,sys; print('\n'.join(json.load(sys.stdin).get('auths', {}).keys()))"
+```
+
+Look for registry keys (`https://index.docker.io/v1/`, `nvcr.io`, `ghcr.io`).
+
+### Step 3: If credentials are missing
+
+**Do not submit the job.** Instead:
+
+1. Tell the user which registry and runtime need authentication
+2. Show the fix for their runtime:
+
+**enroot/pyxis:**
+
+```bash
+mkdir -p ~/.config/enroot
+
+# DockerHub (get token from https://hub.docker.com/settings/security)
+cat >> ~/.config/enroot/.credentials << 'EOF'
+machine auth.docker.io
+  login <dockerhub_username>
+  password <access_token>
+EOF
+
+# NGC (get API key from https://org.ngc.nvidia.com/setup/api-keys)
+cat >> ~/.config/enroot/.credentials << 'EOF'
+machine nvcr.io
+  login $oauthtoken
+  password <ngc_api_key>
+EOF
+```
+
+**Docker:**
+
+```bash
+# DockerHub (interactive prompt)
+docker login
+
+# NGC (use --password-stdin to avoid exposing secrets in process list)
+echo "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin
+```
+
+3. **Suggest an alternative image** on an authenticated registry. NVIDIA clusters typically have NGC auth pre-configured, so prefer NGC-hosted images:
+
+| DockerHub image | NGC alternative |
+| --- | --- |
+| `vllm/vllm-openai:latest` | `nvcr.io/nvidia/vllm:<YY.MM>-py3` (check [NGC catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm) for latest tag) |
+| `nvcr.io/nvidia/tensorrt-llm/release:<tag>` | Already NGC |
+
+> **Note:** NGC image tags follow `YY.MM-py3` format (e.g., `26.03-py3`). Not all DockerHub images have NGC equivalents. If no NGC alternative exists and DockerHub auth is missing, the user must add DockerHub credentials or pre-cache the image as a `.sqsh` file.
+
+4. After the user fixes auth or switches images, verify the image is **actually pullable** before submitting (credentials alone don't guarantee the image exists):
+
+```bash
+# enroot — test pull (aborts after manifest fetch)
+enroot import --output /dev/null docker://<registry>#<image> 2>&1 | head -10
+# Success: shows "Fetching image manifest" + layer info
+# Failure: shows "401 Unauthorized" or "404 Not Found"
+
+# docker
+docker manifest inspect <image> 2>&1 | head -5
+
+# singularity
+singularity pull --dry-run docker://<image> 2>&1 | head -5
+```
+
+> **Important**: Credentials existing for a registry does NOT mean a specific image is accessible. The image may not exist, or the credentials may lack permissions for that repository. Always verify the specific image before submitting.
+
+### Common failure modes
+
+| Symptom | Runtime | Cause | Fix |
+| --- | --- | --- | --- |
+| `curl: (22) ... error: 401` | enroot | No credentials for registry | Add to `~/.config/enroot/.credentials` |
+| `pyxis: failed to import docker image` | enroot | Auth failed or rate limit | Check credentials; DockerHub free: 100 pulls/6h per IP |
+| `unauthorized: authentication required` | docker | No `docker login` | Run `docker login [registry]` |
+| Image pulls on some nodes but not others | any | Cached on one node only | Pre-cache image or ensure auth on all nodes |

.claude/skills/deployment/SKILL.md

@@ -174,6 +174,8 @@ All checks must pass before reporting success to the user.
 
 If a cluster config exists (`~/.config/modelopt/clusters.yaml` or `.claude/clusters.yaml`), or the user mentions running on a remote machine:
 
+0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per `skills/common/slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**
+
 1. **Source remote utilities:**
 
    ```bash
@@ -193,7 +195,7 @@ If a cluster config exists (`~/.config/modelopt/clusters.yaml` or `.claude/clust
 
 3. **Deploy based on remote environment:**
 
-   - **SLURM** — see `skills/common/slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). Use `remote_submit_job` and `remote_poll_job` to manage the job. Get the node hostname from `squeue -j $JOBID -o %N`.
+   - **SLURM** — see `skills/common/slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). After submitting, register the job and set up monitoring per the **monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.
 
    - **Bare metal / Docker** — use `remote_run` to start the server directly:
 

.claude/skills/evaluation/SKILL.md

@@ -28,6 +28,7 @@ Config Generation Progress:
 - [ ] Step 5: Confirm tasks (iterative)
 - [ ] Step 6: Advanced - Multi-node (Data Parallel)
 - [ ] Step 7: Advanced - Interceptors
+- [ ] Step 7.5: Check container registry auth (SLURM only)
 - [ ] Step 8: Run the evaluation
 ```
 
@@ -74,9 +75,9 @@ Prompt the user with "I'll ask you 5 questions to build the base config we'll ad
 4. Safety & Security (like Garak and Safety Harness)
 5. Multilingual (like MMATH, Global MMLU, MMLU-Prox)
 
-DON'T ALLOW FOR ANY OTHER OPTIONS, only the ones listed above under each category (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config.
+Only accept options from the categories listed above (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config.
 
-> **Note:** These categories come from NEL's `build-config` CLI. If `nel skills build-config --help` shows different options than listed above, use the CLI's current options instead.
+> **Note:** These categories come from NEL's `build-config` CLI. **Always run `nel skills build-config --help` first** to get the current options — they may differ from this list (e.g., `chat_reasoning` instead of separate `chat`/`reasoning`, `general_knowledge` instead of `standard`). When the CLI's current options differ from this list, prefer the CLI's options.
 
 When you have all the answers, run the script to build the base config:
 
@@ -181,6 +182,36 @@ If the user needs multi-node evaluation (model >120B, or more throughput), read
 
 - The docs may show incorrect parameter names for logging. Use `max_logged_requests` and `max_logged_responses` (NOT `max_saved_*` or `max_*`).
 
+**Step 7.5: Check container registry authentication (SLURM only)**
+
+NEL's default deployment images by framework:
+
+| Framework | Default image | Registry |
+| --- | --- | --- |
+| vLLM | `vllm/vllm-openai:latest` | DockerHub |
+| SGLang | `lmsysorg/sglang:latest` | DockerHub |
+| TRT-LLM | `nvcr.io/nvidia/tensorrt-llm/release:...` | NGC |
+| Evaluation tasks | `nvcr.io/nvidia/eval-factory/*:26.03` | NGC |
+
+Before submitting, verify the cluster has credentials for the deployment image. See `skills/common/slurm-setup.md` section 6 for the full procedure.
+
+```bash
+ssh <host> "grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null"
+```
+
+**Decision flow (check before submitting):**
+1. Check if the cluster has credentials for the default DockerHub image (see command above)
+2. If DockerHub credentials exist → use the default image and submit
+3. If DockerHub credentials are missing but can be added → add them (see `slurm-setup.md` section 6), then submit
+4. If DockerHub credentials cannot be added → override `deployment.image` to the NGC alternative and submit:
+
+   ```yaml
+   deployment:
+     image: nvcr.io/nvidia/vllm:<YY.MM>-py3  # check https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm for latest tag
+   ```
+
+5. **Do not retry more than once** without fixing the auth issue
+
 **Step 8: Run the evaluation**
 
 Print the following commands to the user. Propose to execute them in order to confirm the config works as expected before the full run.
@@ -225,64 +256,24 @@ After the dry-run, check the output from `nel` for any problems with the config.
 
 **Monitoring Progress**
 
-After job submission, you can monitor progress using:
+After job submission, register the job per the **monitor skill** for durable cross-session tracking. For one-off queries (live status, debugging a failed run, analyzing results) use the **launching-evals skill**; for querying past runs in MLflow use **accessing-mlflow**.
 
-1. **Check job status:**
+**NEL-specific diagnostics** (for debugging failures):
 
-   ```bash
-   nel status <invocation_id>
-   nel info <invocation_id>
-   ```
-
-2. **Stream logs** (Local execution only):
-
-   ```bash
-   nel logs <invocation_id>
-   ```
-
-   Note: `nel logs` is not supported for SLURM execution.
-
-3. **Inspect logs via SSH** (SLURM workaround):
-
-   When `nel logs` is unavailable (SLURM), use SSH to inspect logs directly:
-
-   First, get log locations:
-
-   ```bash
-   nel info <invocation_id> --logs
-   ```
-
-   Then, use SSH to view logs:
-
-   **Check server deployment logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/server-<slurm_job_id>-*.log"
-   ```
-
-   Shows vLLM server startup, model loading, and deployment errors (e.g., missing wget/curl).
-
-   **Check evaluation client logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/client-<slurm_job_id>.log"
-   ```
-
-   Shows evaluation progress, task execution, and results.
-
-   **Check SLURM scheduler logs:**
-
-   ```bash
-   ssh <username>@<hostname> "tail -100 <log path from `nel info <invocation_id> --logs`>/slurm-<slurm_job_id>.log"
-   ```
-
-   Shows job scheduling, health checks, and overall execution flow.
-
-   **Search for errors:**
-
-   ```bash
-   ssh <username>@<hostname> "grep -i 'error\|warning\|failed' <log path from `nel info <invocation_id> --logs`>/*.log"
-   ```
+```bash
+# Quick status check
+nel status <invocation_id>
+nel info <invocation_id>
+
+# Get log paths
+nel info <invocation_id> --logs
+
+# Inspect logs via SSH
+ssh <user>@<host> "tail -100 <log_path>/server-<slurm_job_id>-*.log"   # deployment errors
+ssh <user>@<host> "tail -100 <log_path>/client-<slurm_job_id>.log"     # evaluation errors
+ssh <user>@<host> "tail -100 <log_path>/slurm-<slurm_job_id>.log"      # scheduling/walltime
+ssh <user>@<host> "grep -i 'error\|failed' <log_path>/*.log"           # search all logs
+```
 
 ---
 
@@ -303,5 +294,6 @@ Config Generation Progress:
 - [ ] Step 5: Confirm tasks (iterative)
 - [ ] Step 6: Advanced - Multi-node (Data Parallel)
 - [ ] Step 7: Advanced - Interceptors
+- [ ] Step 7.5: Check container registry auth (SLURM only)
 - [ ] Step 8: Run the evaluation
 ```

.claude/skills/monitor/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: monitor
+description: Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job <slurm_job_id>", or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.
+---
+
+# Job Monitor
+
+Monitor jobs submitted to SLURM clusters — PTQ quantization, NEL evaluation, model deployment, or raw SLURM jobs.
+
+## When to use
+
+1. **Auto-monitor** — another skill (PTQ, evaluation, deployment) just submitted a job. Register the job and set up monitoring immediately.
+2. **User-initiated** — user asks about a job status, possibly in a new conversation. Check the registry, identify the job, and report.
+
+---
+
+## Job Registry
+
+All active jobs are tracked in `.claude/active_jobs.json`. This file is the single source of truth for what's being monitored.
+
+```json
+[
+  {
+    "type": "nel",
+    "id": "<invocation_id or slurm_job_id>",
+    "host": "<cluster_hostname>",
+    "user": "<ssh_user>",
+    "submitted": "YYYY-MM-DD HH:MM",
+    "description": "<what this job does>",
+    "last_status": "<last known status>"
+  }
+]
+```
+
+`type` is one of: `nel`, `slurm`, `launcher`.
+
+---
+
+## On Job Submission
+
+Every time a job is submitted (by any skill or manually):
+
+1. **Add an entry** to `.claude/active_jobs.json`. Create the file if it doesn't exist.
+2. **Set up a durable recurring cron** (if one isn't already running) that polls all registered jobs every 15 minutes. The cron prompt should: read the registry, check each job, report state changes to the user, remove completed jobs, and delete itself when the registry is empty.
+
+Always do both steps. Don't try to predict job duration.
+
+---
+
+## On Cron Fire / Status Check
+
+Whether triggered by the cron or by the user asking "check status":
+
+1. **Read the registry** from `.claude/active_jobs.json`
+2. **Check each job** using the appropriate method (see below)
+3. **Report only state changes** — compare against `last_status` in registry
+4. **Update `last_status`** in the registry
+5. **Remove completed jobs** — any job in a terminal state (COMPLETED, FAILED, CANCELLED, KILLED)
+6. **If registry is empty** — delete the recurring cron
+
+---
+
+## How to Check Each Job Type
+
+### NEL jobs (`type: nel`)
+
+- **Check:** `nel status <id>`
+- **On completion:** `nel info <id>` to fetch results
+- **On failure:** `nel info <id> --logs` then inspect server/client/SLURM logs via SSH
+
+### Launcher jobs (`type: launcher`)
+
+- **Check:** Tail the launcher's background output file for key events
+- **Key events:** experiment ID, SLURM job ID, container import, calibration progress, export path, final status
+- **On failure:** Look for `Traceback`, `Error`, or `FAILED` in the output
+
+### Raw SLURM jobs (`type: slurm`)
+
+- **Check:** `ssh <host> "squeue -j <id> -h -o '%T %M %R'"` — if empty, job left the queue
+- **On completion:** `ssh <host> "sacct -j <id> --format=State,ExitCode,Elapsed -n"`
+- **On failure:** Check the job's output log file
+
+---
+
+## Identifying Jobs (user-initiated, no ID given)
+
+When the user asks about a job without specifying an ID, check in order:
+
+1. `.claude/active_jobs.json` — most reliable, has context
+2. `nel ls runs --since 1d` — recent NEL runs
+3. `ssh <host> "squeue -u <user>"` — active SLURM jobs
+4. `ls -lt tools/launcher/experiments/cicd/ | head -10` — recent launcher experiments
+
+---
+
+## Reporting Guidelines
+
+- **Report state changes proactively** — PENDING → RUNNING, or job completes
+- **Aggregate multiple jobs** — "2 of 4 completed (MMLU-Pro: 42.3%, GSM8K: 67.1%), 1 running, 1 pending"
+- **Summarize, don't echo** — interpret events ("Calibration complete, exporting checkpoint") not raw logs
+- **On failure, diagnose immediately** — check logs and report root cause without waiting for user to ask
+- **Minimize noise** — don't report "still running" unless the user is actively asking