Guid.CreateVersion7() - bug?

[sdrapkin]

Description

Guid.CreateVersion7() explicitly states that it "Creates a new Guid according to RFC 9562, following the Version 7 format." (RFC link).

The RFC states that the v7-UUID uses a Unix millisecond-timestamp in the most significant 48 bits (i.e. first 6 bytes in network-order).
Quoting from the RFC:

unix_ts_ms:
48-bit big-endian unsigned number of the Unix Epoch timestamp in milliseconds as per Section 6.1. Occupies bits 0 through 47 (octets 0-5).

The current .NET implementation CreateVersion7() does the following:

• Gets UtcNow timestamp. Ex. 1737048697655 as an integer, or 37-AB-2B-70-94-01-00-00 as little-endian hex bytes.
• Ignores the 2 right-most bytes (which makes sense).
• Writes a little-endian 4-byte int32 as 2B-70-94-01, then writes a little-endian int16 as 37-AB.
• The final v7 generated guid is 2B-70-94-01-37-AB-F0-75-AD-26-BD-59-2A-6E-DB-FE (but lets focus on the left-most 6 bytes of timestamp).

It seems to me that according to the RFC the correct v7 timestamp encoding should be big-endian, and thus the left-most 6 bytes should be:
01-94-70-2B-AB-37.

The .NET string representation of that generated v7 guid is "0194702b-ab37-75f0-ad26-bd592a6edbfe". Ie. the big-endian timestamp bytes appear in the string representation, but not in the byte representation.

Since .NET guid string conversion is the same for all guids (ie. uses the same logic to create a string from guid's' internal .NET field-structure), the .NET implementation must choose to either be big-endian-timestamp compliant in v7-guid's byte-order (which IMHO is what the RFC actually specifies), or be big-endian-timestamp compliant in the .NET's string representation -- but it cannot do both. The current implementation has chosen the string-order approach, which, IMHO, is not what the RFC requires.

Consequences of the current (IMHO incorrect) v7-guid implementation:

• I've seen community guidance to use CreateVersion7() guids for PostgreSQL UUIDs (primary keys and indexed columns) to avoid uuid-fragmentation in PostgreSQL. This is wrong, because with the current CreateVersion7() implementation the 1st byte of that guid increments ~once per minute, and will rotate through 256 values after ~4+ hours (PostgreSQL compares uuids as byte-arrays, left-to-right).

Reproduction Steps

Sharplab.io reproduction

using System;
using System.Runtime.CompilerServices;

Span<byte> bytes = stackalloc byte[8];
var timestamp = DateTimeOffset.UtcNow;

long timestamp_millisec = timestamp.ToUnixTimeMilliseconds();
timestamp_millisec.Dump();
Unsafe.WriteUnaligned(ref bytes[0], timestamp_millisec);

BitConverter.ToString(bytes.ToArray()).Dump();
var guid_v7 = Guid.CreateVersion7(timestamp);
BitConverter.ToString(guid_v7.ToByteArray()).Dump();
guid_v7.ToString().Dump();

Expected behavior

v7 guid bytes should start with big-endian 6-byte Unix timestamp. See description for an example of expected behavior.

Actual behavior

v7 guid bytes currently do not start with a big-endian 6-byte Unix timestamp, as prescribed by RFC. See description for an example of actual behavior.

Regression?

No response

Known Workarounds

No response

Configuration

.NET 9.0.1 on x64 Windows 10 (Intel).

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

[sdrapkin]

Additional information - this is how Golang does it.
I assume Golang's implementation is RFC-compliant. Byte-sequence and string-sequence in Golang match.

[huoyaoyuan]

For RFC compatible binary representation, specify bigendian: true in ToByteArray. The default array representation uses MS OLE byte order, which is incompatible.

[sdrapkin]

IMHO it's not about what conversions are possible and how compatible or easy-to-use they are, but about the fundamental documented API contracts in .NET and corresponding user expectations.
Guid.CreateVersion7() contract says it returns a v7-guid as documented in the RFC. The RFC specifies a byte-layout of v7-guid, and it's quite explicit. What .NET currently expects everyone to infer is the following:

If you generate a guid via CreateVersion7(), you can convert it into RFC-compliant v7 bytes via ToByteArray(bigEndian:true).

So at the very least this needs to be clearly documented (i.e. it is the above algorithm that produces RFC-compliant v7 byte-format). The current return value of CreateVersion7() is documented as:

A new Guid according to RFC 9562, following the Version 7 format.

That is technically incorrect (not only the best kind of incorrect, but misleading to most users and their expectations). The most that can be said about the current return value is that ".ToString() conversion will produce an RFC-compliant string representation, and .ToByteArray(bigEndian:true) will produce an RFC-compliant byte-sequence." But the the returned v7 guid value itself is IMHO not RFC-compliant.

I believe the clear compliance distinction between "the guid value itself" and "a conversion of the guid" is important to .NET users (ex. different memory and computational ramifications that callers should be aware of).

[tannergooding]

The RFC specifies a byte-layout of v7-guid, and it's quite explicit

The RFC specifies a storage layout which is not strictly the same as the its in memory representation.

The potential difference between a storage format and in memory representation is a basic of computer science. It is not a fault in .NET, it is not an incompatibility with the RFC, etc.

Because it is a fundamental, it means that any time you serialize bytes for storage or deserialize bytes from storage you may need to consider the endianness. This is true for any type including primitives like int, long, float, double, char or for more complex types like Guid, Int128, Decimal, and more. This also means it is almost never safe to assume the raw byte layout of structs; because that will vary based on machine, internal implementation details of the runtime/environment, and other factors. This is not unique to .NET either, it also applies to C, C++, Java, Swift, Rust, and beyond.

Golang is a specific example where they opted to define the type as type UUID [16]byte and cement in the contract that it is always 16 bytes in big endian format. This has both pros and cons, it will lead to its own incompatibilities with some of the other storage layouts described in RFC 4122, and will also lead to additional work being required if they want to optimize any functionality internally for additional performance. .NET instead hides the in memory representation and requires you to use explicit serialization/deserialization APIs such as ToByteArray in order to store the data. This then requires you specify which endianness format is expected for your storage layout, as is typical for such APIs.

That is technically incorrect (not only the best kind of incorrect, but misleading to most users and their expectations)

It is not technically incorrect, as per the above reasons. You must use the correct parameters when serializing or deserializing your data in order to ensure that it correctly roundtrips across the wide variety of machines, hardware, domains, and scenarios that exist in the real world. You might end up running on a big endian machine, you might be running on a much more common little endian machine, you might be in a networking scenario where packets are required to be sent as big endian, you might be reading a PE or ELF file where the data is always stored as little endian, etc.

The Guid that is created by CreateVersion7 is an RFC compliant value that will correctly be compared, formatted, etc as such an RFC compliant value. In the same way that 2 is the number 2, regardless of whether it is in memory as 0x00, 0x00, 0x00, 0x02 (big endian) or 0x02, 0x00, 0x00, 0x00 (little endian). It is only when you serialize/deserialize the value that such endianness becomes important because the serializer and deserializer not agreeing on the format means you will get back the number 33,554,432 instead.

[sdrapkin]

The problem is that the documentation claims that Guid.CreateVersion7() creates an RFC 9562 guid which follows the "Version 7 format". It doesn't. That v7 format is specified in the RFC as:

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           unix_ts_ms                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          unix_ts_ms           |  ver  |       rand_a          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|var|                        rand_b                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            rand_b                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
[Figure 11]: UUIDv7 Field and Bit Layout

That "field and bit layout" (if .NET claims to implement it) must be either:

• the structure's in-memory byte representation
• the structure's byte-serialization (where the structure itself has no committed/guaranteed in-memory representation)
• ..or both

.NET's approach is clearly not "both", so let's assume that .NET's approach is "byte-serialization" that follows the "Version 7 format". If that's the case then it is the byte-serialization API that must be documented as following v7 format.

There are 2 provided APIs to byte-serialize a guid: ToByteArray(...bool bigEndian...) and TryWriteBytes(...bool bigEndian...). It so happens (based on internal specifics of .NET's implementation that we're not supposed to care about) that only when these byte-serialization APIs are called with bigEndian = true do they produce the correct "Version 7 format" serialized output.

Nowhere in the documentation does it say that

1. the "Version 7 format" is only produced by calling one of these APIs...
2. ...and only when bigEndian = true

As I mentioned earlier, at the very least the documentation must be updated to explain the exact steps to obtain the v7 format.

[tannergooding]

which follows the "Version 7 format". It doesn't.

You have to take the entirety of the RFC into context, not just parts. Section 4. UUID Format explicitly covers this, gives allowances for deviations, and explicitly calls out that there is a little-endian format which also exists and isn't discussed in detail in the specification.

What you're actually referring to is the UUID Version 7 layout which is part of Section 5. UUID Layout. The layout describes the interpretation of the bits assuming the default format, but would be adjusted to fit within the actual format in use; such as a little-endian representation.

Nowhere in the documentation does it say that

The remarks section of the ToByteArray APIs clearly covers this as

Note that the order of bytes in the returned byte array is different from the string representation of a Guid value. The order of the beginning four-byte group and the next two two-byte groups is reversed, whereas the order of the last two-byte group and the closing six-byte group is the same. The example provides an illustration.

Particularly with the added context of the RFC, its allowances for formats, explanation that alternative formats such as little-endian exist, and that the default described format is a big-endian/network order format, the need to use bigEndian: true becomes obvious.

The documentation would be incorrect to explicitly call out v7 here; as it is far from the only relevant consideration. The consideration remains simply that the default is little-endian, which is used because that is what is used by COM and is the historical default for .NET. It is completely valid to use a UUIDv7 with COM scenarios and to store/retrieve such v7 UUIDs using the little-endian format. It is simply that some targets will more frequently expect the UUID (regardless of version) to instead be serialized as big-endian format.

We could potentially add an additional remark giving UUIDv7 as a common example of a case where a user might want to use bigEndian: true; but it is not strictly correct to always do that and ultimately depends on the consumer.