Add endianness as one of the pointer properties

[andrewrk]

A pointer is a memory address with metadata.

Here's the metadata a pointer currently has:

• type
• const or mutable
• volatile or no-side-effects for load/store
• align(x) - guaranteed alignment of the address.
  if unspecified it is the ABI alignment of the type.
• :a:b - indicates that the value is a bits offset from the address.
  I think we can remove b because it should always be @bitSizeOf(T)
  If :a:b is omitted, a is 0 and b is @bitSizeOf(T).

Here is metadata we plan on adding in accepted proposals:

• null or 0 to indicate that the pointer is null or 0 terminated (see proposal: type for null terminated pointer #265)

This proposal is to add yet another piece of metadata to pointers, which is endianness.

• &.Endian.Little u32
• &.Endian.Big u32

A target has a native endianness. When pointer endianness is unspecified, it
means the native endianness. So on x86_64, &.Endian.Little u32 is the same
as &u32.

The value Endian here can be obtained from @import("builtin").Endian.
We may decide to automatically import builtin into the global namespace,
so it would become builtin.Endian.

Just like the type of a pointer, endianness can be a comptime value:

const E = if (some_comptime_value) Endian.Little else Endian.Big;

fn read(ptr: &.E u32) -> u32 {
    return *ptr;
}

• A load from a foreign endian pointer performs byte swapping.
• A store to a foreign endian pointer performs byte swapping.

These pointer concepts can be combined, and make sense together:

&.Endian.Big const volatile :2 u4

Here we have a memory address that

• const we should not write through
• volatile there are side effects from reading from
• :2 we must bit shift the loaded value
• u4 we must mask only 4 bits from the loaded value
• .Endian.Big bit shift and mask assuming the loaded value is big endian

So how does this work with packed structs? (See #307)

const BitField = packed(Endian.Big) struct {
    a: u32,
    b: u32,
    c: u4,
    d: u4,
    e: u4,
    f: u4,
};

Here, if you take the address of each field, you get respectively:

• a - &.Endian.Big u32.
• b - &.Endian.Big u32.
• c - &.Endian.Big :0 u4.
• d - &.Endian.Big :4 u4.
• e - &.Endian.Big :0 u4.
• f - &.Endian.Big :4 u4.

What happened here is that the sub-byte fields have a parent integer, which
zig automatically determines based on byte boundaries.

const BitField = packed(Endian.Big) struct {
    a: u32,
    b: u32,
    data: packed(Endian.Big, u16) struct {
        c: u4,
        d: u4,
        e: u4,
        f: u4,
    },
};

Now we have explicitly decided the parent integer.

• data.c - &.Endian.Big :0 u4.
• data.d - &.Endian.Big :4 u4.
• data.e - &.Endian.Big :8 u4.
• data.f - &.Endian.Big :12 u4.

[PavelVozenilek]

Wouldn't it be easier to provide handful of functions converting basic datatypes back and forth? Perhaps also function which does it for structs and arrays, with support from the compiler.

[andrewrk]

How does that address bit fields?

[PavelVozenilek]

AFAIK endianness cares only about 16, 32 and 64 bit sized integer data (not sure about 128 bit and floats). Bytes and byte streams are the same everywhere. Unless Zig is doing something really really strange bitfields could be transferred over the wire "as is".

I am not sure about the "parent integer" concept, it feels as a bad idea. Since bitfields are what one really wants here it probably should be treated as bytestream.

[kyle-github]

It is common to think that there are only two ways of representing multibyte values, big endian and little endian. However, that is not the case. There are definitely others. I would request that this be rethought to allow it to be more powerful to allow any arbitrary byte order. I deal with alternate byte orders in the embedded/automation field.

@PavelVozenilek, it is not that simple in a lot of areas. Floats are definitely subject to byte ordering. However, they are not always subject in the same ways. I have heard of (a common manufacturer) of programmable logic controllers that use a byte order of 1032 for 32-bit ints (two big-endian 16-bit words with the words themselves in little-endian order!) and 3210 for 32-bit floats (big-endian). The numbers are the byte offsets from the start of the value in memory.

I like the ability to add these onto pointer values (that makes more sense than what I was thinking a few months ago with the integers themselves having byte order).

const BitField = packed(byteOrder(u32, 3,2,1,0)) struct {
    a: u32,
    b: u32,
    data: packed(byteOrder(u16, 1,0)) struct {
        c: u4,
        d: u4,
        e: u4,
        f: u4,
    },
};

I have been playing with ideas for bit structs where all the fields are specified in bits.

const controlReg = bitstruct {
     a: bitField(u32, 24,25,26,27,28,29,30,31,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7);
     b: bitField(u32, 56,57,58,59,60,61,62,63,48,49,50...);
     c: bitField(u4, 72,73,74,75);
     d: bitField(u4, 76,77,78,79);
     e: bitField(u4, 64,65,66,67);
     f: bitField(u4, 68,69,70,71);
};

I am not very happy about the syntax. What I am trying to achieve is the ability to state exactly which bits belong to each field. In the above example, you can see that the a field is a big-endian 32-bit field. Bits in a bitstruct are considered to be like a bit array. They start with offset 0 and increase. Each field is thus given a discrete set of the bits in the bitstruct (there are some nice tricks you could do with fields that overlap and that are not that uncommon in hardware). Note that the c, d, e and f fields are ordered as they would be in a bit stream, so their bits look out of order.