Wrong behaviour when combining 'continue-on-error' and 'failure()' in subsequent steps

[jorgesolebur]

Describe the bug
I found an issue when combining continue-on-error in a step 1 and failure()' in a step 2. It is not working as expected.
In my scenario I have a step 1 that in case if it fails, I would like to run step 2 instead. If step 1 fails, I do not want to job to show as failed. This is why I am using a combination of continue-on-error and failure(). However I do not think it is working as intended.

To Reproduce
Prepare a YAML file like the following:

-name: Step 1  
continue-on-error: true. 
run:  'blabalbalba'  
-name: Step 2  
if: ${{ failure() }}  
run:  'bloblboblo'  

When running the workflow in GitHub Actions, if step 1 fails then step 2 is not executed, which does not make sense because step2 is defined to be executed in case step 1 fails.

Expected behavior
If Step 1 does not fail -> Step 2 is not executed (this is working already as expected)
If Step 1 fails -> Step 2 should be triggered AND the job should show success and not failure because I am using continue-on-error

[jorgesolebur]

Hello ? Any update there ?

[henrygriffiths]

@jorgesolebur I believe this is intentional behavior. By setting continue-on-error to true, you are saying 'pass this step even if it fails'. The if failure specifically looks for steps that have failed, eg, to run any cleanup actions, but since the step didn't fail (as continue-on-error is true), it does not run.

In your case, I would use the outcome attribute of the step to achieve your intended behavior:

-name: Step 1  
id: step1
continue-on-error: true. 
run:  'blabalbalba'  
-name: Step 2  
if: steps.step1.outcome != 'success'
run:  'bloblboblo'  

[emirkmo]

Thanks @henrygriffiths ,

This works but the documentation does not reflect it being intentional, although your example made me understand it.

The documentation for continue-on-error needs to be updated for the steps context. As setting continue on error at the job context allows the workflow from the initial example, with the only caveat being that the job will look like a failure in the github PR UI, which is a well known and unaddressed issue (actions/runner#2347).

Asked in a different way, If continue on error actually made the step not fail, so failure() does not work, then why is GitHub action runner reporting a failure for the job?

Since the step actually failed, I would expect failure to trigger from if: context to work. But it is not a big deal. It would just be nice to have it listed under continue-on-error that in a steps context, the individual step failure and the overall job failure is not resolved until all steps are done, while in a job context steps failure happen as normal and can be looked for with if: in subsequent steps...

Just to be crystal clear:

jobx:
 continue-on-error: true
   steps:
   -name: Step 1  
   run:  'blabalbalba'  
   -name: Step 2  
   if: ${{ failure() }}  
   run:  'bloblboblo'  

works and Step 2 executes only if step 1 fails. But jobx will be marked as an overall failure().

[pkit]

@jorgesolebur I believe this is intentional behavior. By setting continue-on-error to true, you are saying 'pass this step even if it fails'. The if failure specifically looks for steps that have failed, eg, to run any cleanup actions, but since the step didn't fail (as continue-on-error is true), it does not run.

In your case, I would use the outcome attribute of the step to achieve your intended behavior:

-name: Step 1  
id: step1
continue-on-error: true. 
run:  'blabalbalba'  
-name: Step 2  
if: steps.step1.outcome != 'success'
run:  'bloblboblo'  

This leads to fake "success" of the pipeline. As of today. Yes, it worked before. But does not work anymore.

[henrygriffiths]

@jorgesolebur I believe this is intentional behavior. By setting continue-on-error to true, you are saying 'pass this step even if it fails'. The if failure specifically looks for steps that have failed, eg, to run any cleanup actions, but since the step didn't fail (as continue-on-error is true), it does not run.
In your case, I would use the outcome attribute of the step to achieve your intended behavior:

-name: Step 1  
id: step1
continue-on-error: true. 
run:  'blabalbalba'  
-name: Step 2  
if: steps.step1.outcome != 'success'
run:  'bloblboblo'  

This leads to fake "success" of the pipeline. As of today. Yes, it worked before. But does not work anymore.

Hi @pkit,

What specifically does not work anymore? I tried the above code and it worked, Step 1 failed but Step 2 still ran. See https://github.com/henrygriffiths/test-actionstoolkit1034/actions/runs/8240794844/job/22536864238

[pkit]

What specifically does not work anymore?

continue-on-error: true leads to "green" test.
Maybe it's a bug in github actions for some runners.
But still, my tests silently failed for week because of that.
Obviously nothing changed in the workflow.

Runner version: 2.314.1
Fortunately, if: failure() still works. I.e. does not make the previous result "green".

[henrygriffiths]

continue-on-error: true leads to "green" test. Maybe it's a bug in github actions for some runners. But still, my tests silently failed for week because of that. Obviously nothing changed in the workflow.

Runner version: 2.314.1 Fortunately, if: failure() still works. I.e. does not make the previous result "green".

I'm seeing the same in my test workflow. Another workaround for your specific requirements could be to manually call the failure in a subsequent step, eg

-name: Step 1  
id: step1
continue-on-error: true
run:  'command that may fail'

- Additional Steps

-name: Check Step 1
if: steps.step1.outcome != 'success'
run:  echo 'Step 1 Failed'; exit 1

[pkit]

Yeah, but I don't really want to use some special calling conventions for the "on failure" steps.

[lululeon]

Thanks @henrygriffiths . This is unexpected:

...but since the step didn't fail (as continue-on-error is true) ...

Ppl are bound to interpret "on-error" as meaning "on error" => "when an error has occurred". As in, "the step DID fail".
So that an if: failure() in a subsequent step, would be expected to be truthy.