Make Numeric read/write locale independent

[hodoulp]

For the last few years, the locale introduced some challenges to any OpenColorIO integration as the library does not recommend or enforce any locale requirement(s). So, the float number read could fail if the floats are saved like 10,0 instead of 10.0 (i.e. French versus English notations). For example using a French localized DCC to save a LUT file and/or a config file, and later use an English localized DCC to read one of them is the use case to correctly handle.

Following the discussion at a TSC meeting, the proposal is then to at least enforce an English float notation independently from the host locale, to read/write all numerics.

Over the years, several pull requests were created around that problem: #1089, #297, etc.

[enkore]

A slightly "heretic" idea, but as I understand things SPI1D and SPI3D are Sony inventions which more or less are only used by OCIO, correct? Developing or using a binary format would solve both performance (loading a bunch of LUTs is still kinda slow, even with the sscanf code; initializing OCIO takes a second or so in my app, even though it's effectively only handling a few MB) and interoperability issues, as well as approximately halving file size. This of course doesn't solve the problem for existing deployments and files.

I briefly looked into using one of the "faster-than-libc" float parsing libraries that are out there (and generally completely ignore locale for performance) for #1089 but in part due to the CLA stuff concluded using such a thing would be infeasible in OCIO. That's pretty much were I got off the train.

[Dithermaster]

We had also seen some slow startup times (yes, we should load on a background thread, etc.) and I solved it two ways:

1. First way was to replace the decimal string parsing in the .spi3d reader with something faster, but then later and now instead
2. I extended the .spi1d and .spi3d to read a hexadecimal format that represents in-memory float format, so no conversion is necessary. This helped a lot. I hadn't designed the format to be something we present back, but would be happy to do so (it's really simple, here is a sample:)

From 0.0 1.0
Length 4096
Components 1
{
a7218fbe6d138fbe33058fbef9f68ebebfe88ebe85da8ebe4bcc8ebe11be8ebed7af8ebe9da18ebe63938ebe29858ebeef768ebeb4688ebe7a5a8ebe404c8ebe
063e8ebecc2f8ebe92218ebe58138ebe1e058ebee4f68dbeaae88dbe70da8dbe36cc8dbefcbd8dbec1af8dbe87a18dbe4d938dbe13858dbed9768dbe9f688dbe
655a8dbe2b4c8dbef13d8dbeb72f8dbe7d218dbe43138dbe09058dbecef68cbe94e88cbe5ada8cbe20cc8cbee6bd8cbeacaf8cbe72a18cbe38938cbefe848cbe
...
fd0c164384061743aa01184371fe1843ddfc1943f0fc1a43adfe1b4316021d432f071e43fa0d1f437a162043b1202143a32c2243533a2343c4492443f75a2543
f16d2643b582274344992843a3b12943d3cb2a43d9e72b43b8052d4372252e430a472f43836a3043e28f314328b7324359e03343790b3543893836438f673743
8c98384384cb39437b003b4374373c4372703d4378ab3e438ae83f43ab274143df68424329ac43438df144430d394643af82474374ce4843611c4a43796c4b43
}

[Dithermaster]

FWIW, I had investigated using hex float representation (e.g., "0x1.921FB4D12D84Ap+1"), but oddly they parsed slower than decimal.

[enkore]

Hexfloats aren't used a lot, so I wouldn't be surprised if they're not well optimized in standard libraries. I mean... you can't even read them using C++ streams.

[KevinJW]

The most obvious problem with outputting binary/binary equivalents is that you are adding a requirement for equivalence of representation across platforms - now this may not be a problem as most platforms probably use IEEE 754 but there is at least an endian related problem. At which point I'd certainly look for a library solution for handling this serialisation e.g protobuf, thrift, or maybe just look for some other lightweight bson library rather than rolling our own routines.

Kevin

[hodoulp]

Thanks for all the great ideas but the focus for short-term is to support existing LUT file formats independently from the locale and without any performance hit.

On the other hand developing a fast binary LUT file format is a great idea. But I think that it should not be an OCIO specific format. I would suggest to bring that discussion to the ACES CLF working group. If accepted, that approach would then provide standardisation (i.e. under the ACES CLF umbrella with OCIO as a reference implementation) and performance.

[hodoulp]

The pull request #1351 contains a discussion about the locale agnostic feature plus some hints from OpenImageIO on how to make it happen.

[amyspark]

Hi @hodoulp,

I just closed bug 407921 in Krita which is a direct repro of #297. The only way I could get rid of it (short of employing a sstream-based parser) is by NatronGitHub/openfx-io@27a13e0.

Is there any news regarding this?

[hodoulp]

@amyspark no significant progress on that front. It would be great if you can add make a pull request with the Natron improvement so we can resume the discussion.

[amyspark]

if you can add make a pull request with the Natron improvement

I can't do it against OCIO, as it's BSD3 and Natron is GPL2.

[lgritz]

Ugh, this is why I can't stand GPL. But I digress.

There are two ways to skin this cat:

First, unless the technique is patented, you (or somebody else) ought to be able to rewrite the code from scratch, an independent implementation of the idea of saving/restoring the locale in any OCIO function that does I/O. It's just not that complicated.

Second, whoever wrote this code originally still owns it. They may have submitted it to Natron to be redistributed under GPL2, but the original author still owns the copyright and so I believe they are free to submit it to a different project under a different license. (Unless Natron requires copyright assignment of all submissions, not merely licensing. VERY few projects require copyright assignment. But if that is the case, then forget what I said about relicensing, you would need to re-implement.)

Though personally, although I do appreciate this clever approach and like how unobtrusive it is to the surrounding code, I think it's a little silly to constantly get, save, change, and restore the locale EVERY time you enter and exit certain functions. It seems like a better approach is to just use locale-independent I/O, and then it just doesn't matter what locale the surrounding app is using. Furthermore, locale-independent I/O will tend to be much faster.

My favourite choice these days is to base all text I/O on the fmt library. It's locale-independent, uses the new C++20 std::format syntax (but also has printf-like versions), has hands-down the fastest int and float <-> string implementations, and a bunch of other goodies.

[Dithermaster]

I wrote a C++ RAII wrapper that saves and restores locale on Windows, Mac, and Linux. I'd be happy to contribute it. We use it to wrap our calls into OCIO v1, so it's OCIO-proven already.

[amyspark]

My favourite choice these days is to base all text I/O on the fmt library. It's locale-independent, uses the new C++20 std::format syntax (but also has printf-like versions), has hands-down the fastest int and float <-> string implementations, and a bunch of other goodies.

I'd refrained from suggesting charconv's from_chars because OCIO requires C++11, but if fmt can be used, 🎉 !