Clarification of "two implementations" requirement for ratification

[javagl]

Summary (and Disclaimer)

There is a requirement for ratification that is sometimes informally summarized as ~"there must be two implementations (of that 'thing' that should be ratified)". In the context of glTF, this is mainly relevant for extensions. But it is not clear what this should mean.

This is, in many ways, not really important for me. I'm certainly not a fan of ~"over-formalizing processes", to the point where "following the process" implies more work than just doing it. But these questions are not so much about the process (how things are done), but rather about the tasks (what has to be done). Given the implications for a formal ratification, there should be clear answers to that.

Context

I brought up the question of what "an implementation" should mean, recently in comment in the KHR_gaussian_splatting PR, and a comment in the KHR_meshopt_compression PR. It is somehow related to (not so recent) attempts to clarify some aspects of the extension development process, even though that PR is closed in the meantime, and I omitted formal details about the ratification process there.

I think it could make sense to funnel these considerations and questions into a dedicated issue.

The Core

The ratification requirement that "there must be two (independent) implementations" probably has its roots in the Khronos Group Operational Guidelines (PDF), which lists...

3. Specification Development
   3.1. Ratification Criteria and Process
   [...]
   Ideally, when a specification is submitted for ratification, the Working Group will be aware of at least two independent conformant implementations of the specification
   [...]
   Pragmatically, final conformant implementations and conformance tests are interdependent and insisting on conformant implementations and proven tests before ratification could delay the specification to the extent of damaging market acceptance. In which case, the Board typically look for the following milestones to be met before ratification:
   • The Working Group is confident that all areas of the specification have been independently implemented twice so that ‘specification bugs and flaws that become obvious through implementation’ have been discovered. The implementations do not have to be complete in themselves - multiple partial implementations from different members can be used to generate specification coverage, additionally, the implementations do not have to be released, or announced or conformant – or even available to the Working Group – working on trust to take implementers at their word.

The vagueness and non-binding nature of these "requirements" justify the informal nature of the summary. With words like "ideally", "confident", "partial", and "trust" (!), there are no explicit, actual "requirements" listed for these "implementations".

Some of that vagueness can be justified, given how broadly this should be applicable. For something like an OpenGL extension, saying "It works on AMD and NVIDIA cards" may be enough. But for things like glTF extensions, there are more cases to consider.

Cases

A glTF extension can just add a single value like emissiveStrength: number to some PBR material property. Parsing the JSON is beyond trivial. For renderers, this might boil down to inserting some ... * emissiveStrength in some shader code. (Yes, it may be more complicated, but conversely, it could often really just mean that an existing configuration option of a renderer is filled with the value from that glTF extension).

A glTF extension that defines a specific binary bitstream is an entirely different category. Examples here are the extensions for Draco- or meshopt compression. The specification of the bitstream itself is already a challenge. It is virtually impossible to look at a specification and say with absolute certainty that it is possible to implement a decoder for this bitstream, from scratch, just based on this description. And implementing a decoder is usually still far easier than implementing an encoder.

There are many in-between cases. For example, I brought up the question about what "an implementation" is in the KHR_gaussian_splatting PR. Should it be a reader or a writer or a renderer? Every existing glTF loader already is an implementation for that, assuming that every loader just loads the accessors with their data in a generic form. It doesn't really have to do anything with this data. Similarly, most writers can probably already write generic accessors, and the question is rather where the splat data is coming from, and who is responsible for bringing it into the exact shape that is required for that extension. (But once it is in that shape, it can trivially and generically be written). The JSON part is negligible here. But there are many really tricky questions about how a renderer should render these splats in order to be a "conformant" implementation. (These questions are probably too specific to be considered here).

Details

The final (nearly philosophical) consideration could be on the level of the language. The linked guidelines talk about "conformant" implementations (but also "partial" implementations), and about "independent" implementations. And I think that the term "independent" could be important.

Emphasizing the point from the guidelines document:

The Working Group is confident that all areas of the specification have been independently implemented twice so that ‘specification bugs and flaws that become obvious through implementation’ have been discovered.

The goal of finding specification bugs (mainly inconsistencies and ambiguities) can only be accomplished when the implementations are really independent - roughly meaning that they have been implemented ~"by two different people, only based on the specification". Just having implementations (by the same person) in two different programming languages is certainly not enough. And porting an existing implementation from one language to another also does not help to identify bugs and shortcomings of the specification document itself - it may not even have been consulted during that porting process. (The case where the code was ported with any form of automated tool obviously falls into the same category).

[lexaknyazev]

Crafting a solid generic definition of what an implementation is seems non-trivial but we can certainly define that for specific extensions on a case-by-case basis, likely with some buckets. A few initial thoughts.

If a glTF library only provides some read/write functions, i.e., it does not render anything, then by definition it only needs to support new JSON schemas and/or stream formats to claim extension support. However, it would be unreasonable to count such a library as an implementation for the purpose of, e.g., ratifying a PBR extension. On the other hand, it may count for an extension like XMP that does not affect rendering at all.

If a glTF extension heavily depends on some pre-existing standard, e.g., WebP, then the glTF implementation mostly consists of JSON parsing and following the additional rules that the extension may define. As long as the external format is officially standardized, RFC 9649 for WebP in this example, we don't need to require multiple independent decoders for it because we essentially delegate external spec issues to the corresponding standards body. On the other hand, if a glTF extension defines both new JSON schemas and a custom stream format, then we have to require at least two independent implementations of all parts; there are some historical exceptions though.

[DRx3D]

For glTF I see three capabilities that need implementation - reading, writing, and rendering. As @lexaknyazev indicated, not all extensions need implementations covering all three capabilities. Implementations do not need to cover all three capabilities, even if the extension is renderable (e.g., a PBR extension).

An exporter (e.g., from Blender) only needs to write. I read the requirements are needing independent implementations that provide double coverage for each capability. For example, the Blender importer/exporter covers reading & writing. SampleViewer covers reading & rendering. That means there are 2 readers, 1 writer, 1 renderer. A third implementation that covered at least writing and rendering is needed to meet the Khronos ratification process requirements.

[lexaknyazev]

I don't see writing as a strict requirement. A file format spec defines only how to read/interpret data, not how to create it.

[javagl]

Crafting a solid generic definition of what an implementation is seems non-trivial but we can certainly define that for specific extensions on a case-by-case basis, likely with some buckets.

A one-fits-all definition could hardly be more specific than what's currently written in the Guidelines document. But the goal could be to get a clearer idea about how to better capture the intention behind this requirement, which was to ensure that
"specification bugs and flaws that become obvious through implementation"
are caught before the ratification.

There certainly are buckets/categories of extensions where similar criteria could be applied. For example, I see the PBR extensions as one of these buckets. Very roughly speaking: Those that define one or more additional properties and textures. Implementations of these in the SampleViewer, three.js and babylon.js are likely created independently, and should have uncovered any flaws. One could make a case to go further, and argue that these are all rasterizers, and there should also be an implementation in a raytracer, but some details here may still depend on the exact nature of the extension.

For glTF I see three capabilities that need implementation - reading, writing, and rendering.

This is true, and one way of slicing it. But with glTF and its extensions usually revolving around JSON, the readers and writers are very often trivial. The possible "bugs and flaws" of the specification will rather refer to the rendering part (or whether the information itself is sufficient and well-defined, regardless of its exact in-file representation).

In some way, the requirements for "implementations" can nearly be disjoint. A bit imprecise, but to illustrate the point: For a PBR extension, it hardly matters whether there's a reader or writer - that emissiveStrength value can be added to the JSON with a text editor. Conversely when a bitstream-based extension like meshopt can be read and written in a "roundtrip" fashion, where the decompressed asset is ~"epsilon-equal" to an asset that was never compressed to begin with, then it's not necessary to renderi it - because it's clear that this decompressed asset could also be rendered as if the extension wasn't present.

---

An aside: It might very well be that all these questions are already considered in the ratification process. Maybe the actual ratification involves more scrutiny than I expect. But if it is rather a formal vote, and not a technical analysis, then this analysis has to happen elsewhere. We are in the lucky situation of having a reviewer who is able to ask questions, seemingly by just looking at the specs, that otherwise would only come up in the fifth implementation (if at all). But I still think that as we straighten out the extension development process description, a short elaboration of the ratification criteria (including the "two implementations" clause) could be worthwhile.

[lexaknyazev]

it hardly matters whether there's a reader or writer - that emissiveStrength value can be added to the JSON with a text editor

Even in that case, all readers and writers must agree on the exact spelling and data type of the property.

all these questions are already considered in the ratification process

Not quite. Ratification is mostly concerned about IP, rather than various edge cases or spec gaps.

[javagl]

Not quite. Ratification is mostly concerned about IP, rather than various edge cases or spec gaps.

I would have expected some form of "checklist" - even though, some of the points on that list might get their ✔️ somewhat graciously, assuming that an extension is only recommended/suggested for the ratification after it has been reviewed from a technical perspective to begin with.

Again: I don't want want to introduce unnecessary overhead at this point - just get a better idea about the requirements.