wrong order of bits in bit fields

[AndreaOrru]

const VGAEntry = packed struct {
    c:  u8,
    fg: u4,
    bg: u4,
};

Expected bit layout as implemented by C compilers (c = char bits, b = bg bits, f = fg bits):
cccccccc bbbbffff

Zig produces this instead:
cccccccc ffffbbbb

Is this expected behavior?

[thejoshwolfe]

here's some relevant discussion: http://stackoverflow.com/questions/19376426/order-of-fields-when-using-a-bit-field-in-c

i didn't cross check the claims there, but i expect they're right that the C spec doesn't specify the order at all. so "as implemented by C compilers" means "the precedent established by popular C compilers", not "as defined by the C spec".

In practice though, that might be irrelevant. If code is already written in C, it might be nice to do simple syntactic updates to translate it to Zig rather than carefully moving bitfields around.

This brings up a serious question: what happens when two different drivers require two different implementations of how to layout sub-byte bits? for example, a {u12,u20} has one byte that's shared between the two fields. does the u12 get the top half or the bottom half of that byte? You can't just swap the order of the fields to resolve this, because the other bytes in the fields would move too. Are all drivers designed to expect the "precedent established by popular C compilers", or do some of them require the opposite decision?

This issue looks exactly analogous to endianness, except at the bit level. are bits specified from lowest to highest or highest to lowest? I would have expected highest to lowest, like big endian, because that's how humans think about it (as shown in your diagram). Does the behavior of popular C compilers actually depend on byte endianness, or is ot always "little endian" for bit order?

Proposal: allow specifying the endianness of a packed struct, and allow specifying the byte order and bit order separately.

[andrewrk]

Is this expected behavior?

Yes. The idea is that you are specifying the exact order of fields in the struct. The order you define in the struct is c, fg, bg, and that matches the bits.

A couple other things to note:

• If a field greater than 8 bits is byte-aligned, it is represented in memory with the endianness of the host.
• If the field is not byte-aligned, it is represented in memory in big-endian.

for example, a {u12,u20} has one byte that's shared between the two fields.

Are you talking about a union?

[AndreaOrru]

As long as this is expected and consistent, the current behavior is OK for me.

+1 for customizable endianness at bit and byte level though.

[andrewrk]

Turns out in addition to byte-level endianness systems also have bit-level endianness. Current implementation of bit fields assumes big endian bits. Instead, we should have the same bit endianness as the byte endianness of the host. Then byte-alignment no longer makes a difference, and everything is as expected.

[AndreaOrru]

Here is a test case for x86(-64):

const io = @import("std").io;

const u4 = @intType(false, 4);

const Bitfields = packed struct {
    f1: u16,
    f2: u16,
    f3: u8,
    f4: u8,
    f5: u4,
    f6: u4,
    f7: u8,
};

pub fn main() -> %void {
    var all: u64 = 0x7765443322221111;
    var bitfields = *@ptrcast(&Bitfields, &all);

    %%io.stdout.printf("f1: {X}\n", u64(bitfields.f1));
    %%io.stdout.printf("f2: {X}\n", u64(bitfields.f2));
    %%io.stdout.printf("f3: {X}\n", u64(bitfields.f3));
    %%io.stdout.printf("f4: {X}\n", u64(bitfields.f4));
    %%io.stdout.printf("f5: {X}\n", u64(bitfields.f5));
    %%io.stdout.printf("f6: {X}\n", u64(bitfields.f6));
    %%io.stdout.printf("f7: {X}\n", u64(bitfields.f7));
}

Output:

f1: 1111
f2: 2222
f3: 33
f4: 44
f5: 6     // Wrong
f6: 5     // Wrong
f7: 77

[thejoshwolfe]

we should have the same bit endianness as the byte endianness of the host.

wikipedia disagrees. it looks any of the 4 combinations are possible, except maybe little byte big bit. I think we need two separate endiannesses.

the proposal for configuring the endianness(es) of a packed struct can be a separate issue. we still need to get the default native behavior correct regardless.

[andrewrk]

The whole point of bit fields is to have a well defined in-memory layout. So we're going to have a way to specify byte endianness as well as bit-endianness. Here's the proposed syntax:

const Endian = @import("builtin").Endian;
const Bitfields = packed(Endian.Little, Endian.Big) struct {
    f1: u16,
    f2: u16,
    f3: u8,
    f4: u8,
    f5: u4,
    f6: u4,
    f7: u8,
}

So the packed keyword takes 2 arguments which is the byte endianness and the bit endianness, respectively.

You can leave off the bit endianness if it doesn't matter, e.g. if you don't have any sub-8-bit fields. You get a compile error if bit endianness is unspecified and it does matter.

You can leave off both the byte endianness and the bit endianness if both don't matter, e.g. if you don't have any sub-8-bit fields and you also don't have any fields greater than 1 byte. You get a compile error if the byte endianness is unspecified and it does matter.

Now consider 2 cases for bit fields:

• Serialization / deserialization, e.g. with a network stream or a file on disk
• OS dev, where bit/byte endianness depends on the architecture

In the first case you would hard code the parameters to the packed keyword, probably Endian.Big for both.

In the second case you would do something like this:

const builtin = @import("builtin");
const Bitfields = packed(builtin.byte_endian, builtin.bit_endian) struct {
    // ...
}

Pointers to fields with foreign endianness are not normal pointers, and cannot be used where normal pointers are expected.

[AndreaOrru]

packed(builtin.byte_endian, builtin.bit_endian) wouldn't this be the default?