PatternJsonProvider: omitEmptyFields doesn't work with %property(ENV_VAR)

[brenuart]

Discussed in #760

^{Excerpt from original post by ziomill February 16, 2022}
I'm trying to use the "omitEmptyFields" to print/omit an ENV variable, without success. If the ENV_VAR 'CONTAINER_NAMESPACE' is not populated, the logger prints "null" as the field's value rather than omit the field.
...
This is my configuration:

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
    <providers>
        ...
        <pattern>
             <omitEmptyFields>true</omitEmptyFields>
             <pattern>
                 {"container-namespace":"%property{CONTAINER_NAMESPACE}"}
             </pattern>
        </pattern>
        ...
    </providers>
</encoder>

And this is the printed log record:

{
   ...
   "container-namespace":"null"
   ...
}

Root Cause

The Pattern json provider is using a standard Logback PatternLayout under the hood to resolve placeholders and converters in the pattern string. The resulting string is then converted into JSON and added to the final result. The "omitEmptyFields" property works by telling Jackson to omit fields whose value is considered empty.

When a property does not exist, the %property{} converter returns null and this value is unfortunately added "asis" to the internal StringBuilder. In this example, the original pattern is converted into the string {"container-namespace":"null"} before it is transformed into a valid JSON structure by Jackson. Since the property is not null but is assigned a String value, it is not filtered out by Jackson's "empty field" filtering mechanism.