Skip to content

Confirm gpt-4.1 invalid model issue already resolved - no changes needed#2236

Closed
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-invalid-model-issue
Closed

Confirm gpt-4.1 invalid model issue already resolved - no changes needed#2236
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-invalid-model-issue

Conversation

Copilot AI commented Oct 23, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR confirms that the issue reported in the smoke detector investigation has already been resolved. The invalid OpenAI model gpt-4.1 was previously changed to the valid openai:gpt-4o model in commit 64a7abcc on Oct 23, 2025.

Issue Background

The smoke detector workflow created an issue reporting that .github/workflows/shared/genaiscript.md was using an invalid model name openai:gpt-4.1, which does not exist in OpenAI's model catalog. This was causing GenAIScript workflows to fail with:

TypeError: Cannot read properties of undefined (reading 'text')

Current State

After investigation, the configuration file already contains the correct model:

File: .github/workflows/shared/genaiscript.md

GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4o"  

Verification

  • ✅ No references to gpt-4.1 exist in the repository
  • ✅ All compiled workflow files (*.lock.yml) use openai:gpt-4o
  • ✅ All unit tests pass successfully
  • ✅ Remote GitHub repository confirmed to have the fix

Timeline

  1. Oct 22, 2025: PR Update smoke genaiscript workflow to use openai:gpt-4.1 model #2135 introduced gpt-4.1 (invalid model)
  2. Oct 23, 2025 13:26 UTC: Commit 64a7abcc fixed it to gpt-4o
  3. Oct 23, 2025 18:06 UTC: Workflow run #18757658104 failed
  4. Issue created: Smoke detector reported the failure from the already-fixed run

The issue was created by analyzing a failed workflow run that occurred before the fix was applied. By the time the issue was filed, the configuration had already been corrected.

Changes in This PR

The only changes are workflow lock file regenerations (*.lock.yml) that incorporate updates from PR #2230 (secret redaction feature). These are standard recompilations to keep lock files synchronized with the current main branch state.

No code changes were needed to fix the reported issue - it was already resolved.

Original prompt

This section details on the original issue you should resolve

<issue_title>[smoke-detector] 🚨 CRITICAL: GenAIScript Invalid Model (gpt-4.1) - 5th Consecutive Failure Post-v0.24.0</issue_title>
<issue_description># 🚨 CRITICAL RECURRING FAILURE - 5th Consecutive Occurrence

Summary

The Smoke GenAIScript workflow has FAILED AGAIN after the v0.24.0 release with the EXACT SAME ROOT CAUSE that has been reported in THREE previous issues (#2157, #2204, #2207). This is the 5th consecutive failure of this smoke test since 2025-10-22. Despite multiple investigations and issue reports, the configuration has never been corrected.

Failure Details

  • Run: #18757658104
  • Commit: 8993988 - "Release v0.24.0"
  • Trigger: schedule (automated smoke test)
  • Duration: 3.5 minutes
  • Failed Job: detection (1.2 minutes)
  • Status: ❌ FAILED

Root Cause Analysis

The Problem Persists UNCHANGED

The GenAIScript configuration STILL uses an invalid OpenAI model name:

Location: .github/workflows/shared/genaiscript.md line 6

GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"

Problem: gpt-4.1 DOES NOT EXIST in OpenAI's model catalog.

Valid OpenAI models:

  • gpt-4o ✅ (recommended)
  • gpt-4-turbo
  • gpt-4
  • gpt-3.5-turbo

Error Chain (Identical to All Previous Occurrences)

  1. GenAIScript attempts to resolve and use model openai:gpt-4.1
  2. OpenAI API rejects the request (invalid model)
  3. GenAIScript receives undefined/null response
  4. GenAIScript crashes: TypeError: Cannot read properties of undefined (reading 'text')
  5. Detection job fails with exit code 255
  6. Smoke test marked as failed

Stack Trace

2025-10-23T18:10:09.4293104Z 2025-10-23T18:10:09.429Z genaiscript:error {
2025-10-23T18:10:09.4293428Z   name: 'TypeError',
2025-10-23T18:10:09.4293872Z   message: "Cannot read properties of undefined (reading 'text')",
2025-10-23T18:10:09.4294339Z   stack: "TypeError: Cannot read properties of undefined (reading 'text')\n" +
2025-10-23T18:10:09.4295107Z     '    at githubActionSetOutputs ((redacted))\n' +
2025-10-23T18:10:09.4296330Z     '    at async Command.runScriptWithExitCode ((redacted))'
2025-10-23T18:10:09.4297303Z }

Failed Jobs and Errors

Job Execution Summary

  1. activation - succeeded (2s)
  2. agent - succeeded (1.6m) - Agent completed successfully
  3. detection - FAILED (1.2m) - Threat detection crashed
  4. create_issue - succeeded (5s)
  5. ⏭️ missing_tool - skipped

Investigation Findings

Complete Failure Timeline

# Run ID Date/Time (UTC) Trigger Issue Created Issue Status
1 18727962258 2025-10-22 19:45:52 workflow_dispatch #2157 Closed as "not_planned"
2 18733557489 2025-10-23 00:19:22 schedule - Covered by #2157
3 18739169072 2025-10-23 06:07:04 schedule #2204 Closed as "completed"
4 18747816413 2025-10-23 12:08:41 schedule #2207 Closed as "completed"
5 18757658104 2025-10-23 18:06:57 schedule This issue Open

Pattern: Failing every ~6 hours on scheduled runs
Duration: Over 22 hours of continuous failures
Failure Rate: 100% since first occurrence

Why This Is Critical NOW

  1. Post-Release Failure: This failure occurred immediately after the v0.24.0 release, indicating the configuration issue persists across releases
  2. Multiple Closed Issues: Three separate issues ([smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157, [smoke-detector] 🚨 CRITICAL RECURRING: GenAIScript Invalid Model (gpt-4.1) - 3rd Occurrence #2204, [smoke-detector] Comment on #2157 #2207) have been created and closed without fixing the root cause
  3. Wasted Resources: Every scheduled run (every ~6 hours) consumes CI minutes while producing no value
  4. Security Gap: Threat detection has been non-functional for over 22 hours
  5. False Confidence: The team may not realize smoke tests are failing continuously

Recommended Actions

🔴 CRITICAL - Immediate Fix (1 minute)

Update .github/workflows/shared/genaiscript.md line 6:

- GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"
+ GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4o"

That's it. One line change. Will fix all 5 failures instantly.

🟡 Alternative: Disable Scheduled Workflow

If GenAIScript smoke tests are not being maintained, disable the scheduled trigger to stop generating fai...

Fixes #2227


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix GenAIScript recurring failure after v0.24.0 release Confirm gpt-4.1 invalid model issue already resolved - no changes needed Oct 23, 2025
Copilot AI requested a review from pelikhan October 23, 2025 19:59
@pelikhan pelikhan closed this Oct 23, 2025
@pelikhan pelikhan deleted the copilot/fix-invalid-model-issue branch October 23, 2025 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[smoke-detector] 🚨 CRITICAL: GenAIScript Invalid Model (gpt-4.1) - 5th Consecutive Failure Post-v0.24.0

2 participants