tutorial/handling null values in protobuf and avro#36
tutorial/handling null values in protobuf and avro#36Lucia Cerchie (Cerchie) wants to merge 16 commits intoconfluentinc:masterfrom
Conversation
Dave Troiano (davetroiano)
left a comment
Dave Troiano (davetroiano)
left a comment
There was a problem hiding this comment.
LG so far Lucia Cerchie (@Cerchie)!
| .setTotalCost(random.nextDouble() * random.nextInt(100)); | ||
| ``` | ||
|
|
||
| In this case, you'll receive a NullPointer error. No newline at end of file |
There was a problem hiding this comment.
whats going to come next? the field being oneof where one is a boolean to signify null?
There was a problem hiding this comment.
It has to be a protobuf schema addition... if I were to show it I think it'd be a oneof, but also there's a google wrapper option. unfortunately because of the way the API is defined with protocol buffers and kafka there are two methods with the same name and java chokes on the protocol buffer... not sure how to fix here
There was a problem hiding this comment.
Does optional do the trick? i.e. does everything just work if you do that? I'm looking at the guidance here:
If not using optional would add complexity or ambiguity, then use optional primitive fields. Wrapper types must not be used going forward.
which makes me think we might want to avoid the wrapper pattern... I'm not sure why this is the guidance, e.g., will wrapper types eventually be deprecated?
I'm thinking that if optional works, go with that, and we'd also want to update this docs section to recommend optional rather than use a wrapper
wdyt?
There was a problem hiding this comment.
so with this protobuf definition:
message Purchase {
optional string item = 1;
double total_cost = 2;
string customer_id = 3;
}
you still end up with a NullPointerException ¯_(ツ)_/¯
but that's a good point about the guidance, I'll pass it on to docs.
Co-authored-by: Dave Troiano <david.a.troiano@gmail.com>
| # How to allow `null` field values in Avro and Protobuf | ||
| Let's say you're using an Avro or Protobuf schema, and sometimes you want to set a field named `item` to null. Say it's a pipeline that takes both donations and purchases to be filtered later, and the donations are processed as purchases with null items. How to adjust the schema to allow for a null value? | ||
|
|
||
| Avro natively supports null fields with the 'null' type. In the above example, in order to make the `item` field nullable, you can allow the type to be "string" or "null" in the following manner: |
There was a problem hiding this comment.
| Avro natively supports null fields with the 'null' type. In the above example, in order to make the `item` field nullable, you can allow the type to be "string" or "null" in the following manner: | |
| Avro natively supports null fields with the `null` type. In the above example, in order to make the `item` field nullable, you can allow the type to be "string" or "null" in the following manner: |
| From the top-level directory: | ||
|
|
||
| ``` | ||
| ./gradlew clean :handling-null-values-in-avro-and-protobuf:kafka:test --info |
There was a problem hiding this comment.
this didn't work for me on a fresh clone:
* What went wrong:
A problem occurred evaluating project ':handling-null-values-in-avro-and-protobuf:kafka'.
> /Users/dtroiano/Desktop/tutorials/handling-null-values-in-avro-and-protobuf/kafka/src/main/resources/confluent.properties (No such file or directory)
| .setTotalCost(random.nextDouble() * random.nextInt(100)); | ||
| ``` | ||
|
|
||
| In this case, you'll receive a NullPointer error. You can allow null values to be explicitly set with a [protocol wrapper type](https://protobuf.dev/reference/protobuf/google.protobuf/https://protobuf.dev/reference/protobuf/google.protobuf/). No newline at end of file |
There was a problem hiding this comment.
| In this case, you'll receive a NullPointer error. You can allow null values to be explicitly set with a [protocol wrapper type](https://protobuf.dev/reference/protobuf/google.protobuf/https://protobuf.dev/reference/protobuf/google.protobuf/). | |
| In this case, you'll receive a NullPointer error. You can allow null values to be explicitly set with a [protocol wrapper type](https://protobuf.dev/reference/protobuf/google.protobuf/). |
| .setTotalCost(random.nextDouble() * random.nextInt(100)); | ||
| ``` | ||
|
|
||
| In this case, you'll receive a NullPointer error. You can allow null values to be explicitly set with a [protocol wrapper type](https://protobuf.dev/reference/protobuf/google.protobuf/https://protobuf.dev/reference/protobuf/google.protobuf/). No newline at end of file |
There was a problem hiding this comment.
Are you planning to have an example with wrapper? or leave it as an exercise to the reader
| .setTotalCost(random.nextDouble() * random.nextInt(100)); | ||
| ``` | ||
|
|
||
| In this case, you'll receive a NullPointer error. You can allow null values to be explicitly set with a [protocol wrapper type](https://protobuf.dev/reference/protobuf/google.protobuf/https://protobuf.dev/reference/protobuf/google.protobuf/). No newline at end of file |
There was a problem hiding this comment.
suggest adding a clean up step that directs the user to delete the env created for this
Lucia Cerchie (Cerchie)
changed the title
2 participants
need to figure out a smooth way to handle testing and a way to teach the reader about what happens when null values are handled/not. tagging Dave Troiano (@davetroiano)