bug: convert_vertical_obs DARTQC_FAILED_VERT_CONVERT

[hkershaw-brown]

🐛 🐳 🐼

User reported,
convert_vertical_obs
DARTQC_FAILED_VERT_CONVERT is not ending up in the obs_seq.final

will update with reproducer shortly.

Describe the bug

1. List the steps someone needs to take to reproduce the bug.
   Run filter with cam-fv but force convert_vertical_obs to return vstatus /= 0

2. What was the expected outcome?
   The obs_seq.final should have DARTQC=8 for every obs.

3. What actually happened?
   No 8s

Error Message

No error messages

Which model(s) are you working with?

cam-fv but I think applicable to any model that returns a non-zero convert_vertical_obs status.

Screenshots

Here is obs_qc = 8 across all tasks at `cycle SEQUENTIAL_OBS

Here is task 0 writing the obs sequence

Here is obs_seq.final:

Version of DART

Which version of DART are you using?
v10.5.1-1-gf37ba3ccb

Have you modified the DART code?

Yes - to force the failure and also a bunch of print statements.

If your code changes are available on GitHub, please provide the repository:
Forcing vertical fail for cam-fv
https://github.com/NCAR/DART/tree/bug-vertical

Test code here:
/glade/scratch/hkershaw/DART/Bugs/bug_401/

note you will have to grab
/glade/scratch/hkershaw/DART/Bugs/bug_401/test_cam-fv.main
and
/glade/scratch/hkershaw/DART/Bugs/bug_401/in_files/
to run filter in test_cam-fv.main

Build information

Please describe:

1. Cheyenne
2. intel

[hkershaw-brown]

confirmed

[kdraeder]

While trying to trace the evolution of QC values I believe that I found
that many non-0 QC values are converted to 4, which is what should appear in the obs_seq_final.
So we should not expect to see 8s. But I think that some obs that fail the vertical conversion
end up with QC = 0 in the obs_seq_final. Here's some pseudocode outlining what I'm describing.

   forward_operator_mod: get_obs_ens_distrib_state
      call get_dart_qc(..., global_qc_value)
      |    subroutine get_dart_qc (...,dart_qc)
      |       if (maxval(istatus(:)) > 0) then                      ! ensemble members for 1 ob
      |          dart_qc = DARTQC_FAILED_FOP   = 4   ! regardless of istatus actual max or min
      v
      obs_fwd_op_ens_handle%copies(OBS_GLOBAL_QC_COPY, j) = global_qc_value

[kdraeder]

I also see QC values of 5,6, and 7 in the obs_seq_finals,
so those must be defined after get_obs_ens_distrib_state.

[nancycollins]

what fortran code are you making that pseudocode from? if it's the get_dart_qc() routine in quality_control_mod.f90, then there is lots of logic there to avoid overwriting existing QCs and it does different things for the prior and posterior.

if the forward operator returns a non-zero istatus from interpolate() for any ensemble member, then it failed the forward operator. is that what you're questioning?

[nancycollins]

QC values 2 and 3 are "failed the posterior FO" and "failed the posterior FO for eval-only obs". those are certainly possible.

[kdraeder]

The only get_dart_qc that I see is in quality_control_mod.
I listed the source of the pseudocode: forward_operator_mod: get_obs_ens_distrib_state
but should have added that it's from the "else ! distributed state" section,
which seemed relevant because I do ask for distributed state.
There's another call down in the
"if (get_allow_transpose(ens_handle)) then"
section, but that seems to be entered only if the "else ! distributed state" section is not entered
(if I understand the controls here).

@nancycollins, if you're describing the logic in just get_dart_qc, then I'll point out
that not much of it comes before the assignment of DARTQC_FAILED_FOP (4):

if(isprior) then
   if((.not. assimilate_this_ob) .and. (.not. evaluate_this_ob)) then
      dart_qc = DARTQC_NOT_IN_NAMELIST             ! QC=5
   else if(failed_fop) then            ! At least one forward operator failed
      dart_qc = DARTQC_FAILED_FOP                        ! QC=4

It appears to me that it doesn't matter why a forward operator failed,
e.g. ob too high, failed vertical conversion, ...,
all of those reasons trigger the same assignment of DARTQC_FAILED_FOP in dart_qc.

I haven't expected to see QC=2 or 3, so I wasn't looking for them.
I also haven't seen them, so it's possible that they've been reset to something else, like 8 seems to be.

[nancycollins]

@kdraeder you're totally correct that it doesn't matter why the forward operator failed. a QC=4 says interpolate() returned an error for at least one ensemble member. that istatus array is returned from the interpolate routine, which is only called when computing the forward operators for prior or posterior, not from the call to vertical conversion in the assim_tools() routine.

there may a bug there, but the forward operators are called first for all obs to compute the priors, then filter_assim() runs, then the forward operators are called to compute the posteriors. do you suspect the posterior forward operator QC code is overwriting the QC if it's already set to 8? or that a QC=8 isn't getting set in the main assimilation loop code? the filter_assim() code doesn't use get_dart_qc() or get_obs_ens_distrib_state().

could it be that where you added code to return a failed vertical conversion somehow is affecting the interpolate call? if the interpolate routine calls something that calls something that fails a vertical conversion, the interpolate routine will return an istatus /= 0 and it will set a QC=4. an obs that failed the forward operator won't be processed in the main assimilation loop, so the vert conversion won't be called there.

sorry - i forget what you said you saw. did you get QC=4 when you tried making it fail the vertical conversion in filter_assim()? or did you get QC=0?

[hkershaw-brown]

@nancycollins I edited the bug report so you might have not got the edits by email.
The screen shots in the issue show QC 8 being set (with a forced vertical conversion), still 8 when sequential_obs does its qc check, but not 8 anymore when write_obs_sequence is called.

[nancycollins]

ah- it's getting a QC=7 which means it failed the outlier threshold test. so the code shouldn't be testing for an outlier threshold if the QC is 8.

[hkershaw-brown]

yup, & obs 3 has a qc of 0 in the obs_seq.final

[kdraeder]

You two seem to have advanced beyond Nancy's questions, so I won't answer all of them,
unless that would be helpful.

In one GPS profile in the case where I excluded all obs in the N. hemisphere,
obs_seq_final I see QC = 4, 7, and 0 at various levels.

Is there a problem with forward_op for the prior using the ob's native vertical coordinate,
then converting those to scale height, then passing those to the forward op for the posterior?
It seems like it can't be, because the vertical interpolation would fail with that vertical coordinate.

[nancycollins]

Is there a problem with forward_op for the prior using the ob's native vertical coordinate,
then converting those to scale height, then passing those to the forward op for the posterior?
It seems like it can't be, because the vertical interpolation would fail with that vertical coordinate.

the filter_assim() subroutine makes a copy of the obs location while it is doing the assimilation, so changing the vertical in that copy doesn't affect the original location which is used in both calls to the forward operator code.