[Bug]: Change reflection-based schema generation to generate deterministic schemas

[scwhittle]

What happened?

Currently schema fields are assigned ordering based upon ordering of ReflectUtils.getMethods and ReflectUtils.getFields.

ReflectUtils.getMethods ordering is based upon Class.getDeclaredMethods which has no specified ordering and may differ across JVMs. It appears that it must be somewhat consistent ordering in our test environment because changing the ordering to sort breaks tests relying on the current ordering.

ReflectUtils.getFields returns ordering based upon HashMap iteration over field names. There are tests that depend on this hash ordering as well (which might not be consistent if hashes are randomized as in some JVMs).

We should modify these methods to return a consistent ordering so that generated schemas for unmodified classes are consistent and not depending on unspecified ordering.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[scwhittle]

.take-issue

[scwhittle]

Another solution to the problem would be to determine where the non-deterministic ordering used between instances causes issues. My suspicion is that they are embedded in the toRow/fromRow functions. A different approach to sorting fields would be to randomize the indices and see what breaks.

[kennknowles]

Well I think Beam Row in general has its origins in SQL and also is a compact format with unlabeled fields so it is inherently order-sensitive. Anything generating a Row from a POJO is where I expect we have the most problems. Note that any change to the ordering could be update-incompatible.