Queries fail to run in full refresh mode

[taneli-salonen1]

How to reproduce:

1. Start a new execution in a Dataform (Cloud) workspace by specifying a tag.
2. Select run with full refresh and include dependendents
3. Start execution

What happens:

The queries included in the workflow all run without errors. However, the incremental table queries are executed in incremental mode instead of full refresh mode.

In my case, the workflow included both regular tables and incremental tables.

How to fix it:

The issue happens with Dataform core versions starting from 3.0.10.

Downgrading the Dataform core version back to 3.0.9 resolved the issue.

[bigsmoke]

I can confirm that this problem still exists on Dataform core 3.0.13.

[bigsmoke]

And in 3.0.16 as well.

[nybach]

I can also confirm this error, I've tested every version between 3.0.9 & 3.0.16. You can see it quite easily via the execution logs, in version 3.0.9 with full refresh option the incremental Dataform script is compiled as a CREATE OR REPLACE TABLE statement in BigQuery, whereas as of version 3.0.10 it gets compiled as a MERGE statement, which only updates or adds data, but does not delete rows that should not be there.

According to a Google Partner, this is linked to the new config parameter 'protected', concerning table deletion protection, which should by default be set to false to allow table recreation by full refresh, but currently defaults to true, blocking this action. The problem can be circumvented by adding the parameter and setting it explicitly to false in the config block of the script (or just use version 3.0.9 for now). A ticket has been logged regarding this wrong default.

[Ceridan]

Version 3.0.17 should fix this. Brings back the default behavior for protected field like it was before.