Need a Boxed instance for Unboxed (to be able to unbox user data types with some boxed fields)

[dcoutts]

It may sound counter-intuitive, but it would be really useful for unboxing structures that contain some boxed data.

For example, the docs for As give this little example record data type:

data Foo a = Foo Int a
  deriving Show

It defines a relation between this and a tuple (for which there is an existing Unbox instance)

instance VU.IsoUnbox (Foo a) (Int,a) where
  toURepr (Foo i a) = (i,a)
  fromURepr (i,a) = Foo i a

and end up defining an instance

instance VU.Unbox a => VU.Unbox (Foo a)

But this requires the a to be unboxable too. What if I have a record member that really does need to be boxed, such as a reference, another array, a string, etc etc etc.

I should instead be able to define the relation like this

instance VU.IsoUnbox (Foo a) (Int, Boxed a) where
  toURepr (Foo i a) = (i,a)
  fromURepr (i,a) = Foo i a

where Boxed is a provided newtype that is an instance of Unbox already, and thus the tuple (Int, Boxed a) is therefore an instance of Unbox.

Obviously, the representation for a vector of Boxed is just an ordinary boxed vector!

newtype Boxed a = Boxed a

newtype instance VU.MVector s (Boxed a) = MV_Boxed (V.MVector s a)
newtype instance VU.Vector    (Boxed a) = V_Boxed  (V.Vector    a)

instance VGM.MVector VUM.MVector (Boxed a) where
  basicLength (MV_Boxed v) = VGM.basicLength v
  -- ... etc, for all the methods
  
instance VG.Vector VU.Vector (Boxed a) where
  basicUnsafeFreeze (MV_Boxed v) = V_Boxed <$> VG.basicUnsafeFreeze v
  -- ... etc, for all the methods

instance VU.Unbox (Boxed a)

And that's it. Now we can unbox any record, even if some fields are still boxes.

The above code type checks, it's just missing the impl for all the methods, but that's just doing the obvious boring thing. A PR would be easy.

[dcoutts]

Related: it'd also be nice to have specific Unbox instances for types like ByteArray and MutableByteArray that are (morally) unlifed but boxed. This could take advantage of recent GHC's ability to have levity-polymorphic arrays (which ideally should be exposed via Primitive.Array).

And further refinements: we could also have BoxedStrict for cases where we want a vector of boxed data, but want the values in the array to be always evaluated to WHNF. Currently it's actually quite hard to reliably have vectors of boxed but WHNF data.

[Shimuuar]

Yes. I would say that Unbox is misnomer. It about selecting representation of an array by element type. And it need not to be necessarily unboxed.

There's also precedent UnboxViaPrim uses primitive vector as a representation. It could be used as a template. I would gladly accept PR with such addition. Only question is naming. I think it would be nice to be explicit about strictness of underlying array.

I can see two possible names: AsLazyBoxed/AsStrictBoxed using As as reference. Or UnboxVia{Strict,Lazy}Boxed which is based on UnboxViaPrim but it's long and weird. So former it probably better

[lehins]

It may sound counter-intuitive, but it would be really useful for unboxing structures that contain some boxed data.

I had similar thoughts a while ago, but never really had a use case for it. So, I do agree that it would be a good idea to add a type like that. However, with respect to Unbox instance for the Boxed type, I think it would be useful to enforce NFData on the a in Boxed a, because one thing Unboxed vectors gives us is the guarantee that all its elements are in NF. In other words, I do agree with @Shimuuar that "it[Unbox] need not to be necessarily unboxed.", but Normal form is a very nice property of unboxed vectors. Others might have different opinion on the subject.

we could also have BoxedStrict for cases where we want a vector of boxed data, but want the values in the array to be always evaluated to WHNF. Currently it's actually quite hard to reliably have vectors of boxed but WHNF data.

Fairly recently new interface was added for strict boxed vectors where elements are always in WHNF, it just hasn't been released yet: #488 So, I am not really sure of usefullness of a specialized BoxedStrict newtype for elements.

[Shimuuar]

So question is really about semantics. What are unboxed vector? What do we expect from implementations? Documentation say little:

The implementation is based on type families and picks an efficient, specialised representation for every element type.

One possibility is to think that Unbox only provide some "efficient" representation of vector without making any particular promises.

Another one which I haven't considered is to expect that elements are reduced to NF. By enforcing NFData you mean rnfing elements on write, correct?

[ruifengx]

By analogy, I believe there should also be an AsStorable wrapper (or UnboxViaStorable, depending on the naming scheme). In my use case, I get two parallel arrays from C API, and I think the most efficient high-level abstraction is to wrap both into Storable vectors (via ForeignPtrs), and then combine them just like unboxed vectors for pairs. Elements in Storable vectors are effectively forced to NF, so strictness should not be a problem.

[recursion-ninja]

I'm willing to take a stab at this. If unsuccessful, I'll at least report the problematic pain points for a future contributor.

[recursion-ninja]

Another one which I haven't considered is to expect that elements are reduced to NF. By enforcing NFData you mean rnfing elements on write, correct?

The vector package already depends on deepseq so that seems like a palatable constraint. Perhaps that means for strict, compact layouts of unboxed values in an Unboxed vector, they require the following constraint:

-- General constraint
NFData a => Unbox (AsBoxedStrictly a)


-- Example case with this constraint
instance NFData a => VU.IsoUnbox (Foo a) (Int, AsBoxedStrictly a) where
...

[Shimuuar]

So we need to make a decision what sort of instances do we allow. I'm more of anything goes persuasion and @lehins thinks that unboxed vectors should reduce stored element to normal form. It's nice property and all existing instances has it.

If this discussion haven't started I'd just used boxed vector. But reduction ot NF is nice and in line with naming. So I waekly in favor of requiring it. @lehins , @Bodigrim what's your opinion?

[dcoutts]

Lets go back to the use case, it's to be able to have vectors of types like this:

data Foo a = Foo !Int a

or

data Bar a = Foo !Int !a

So we may want strict or lazy fields, but either way the semantics of a constructor are only to force arguments to WHNF. There's no notion of normal form fields (nor should there be).

So that would argue for having BoxedStrict or BoxedLazy to be explicit about the above choice. And the strict one should be WHNF only, not NF. It would not be useful for the intended use case if we required NF. It would also be very expensive to deepseq on each array write.

[dcoutts]

@lehins says:

However, with respect to Unbox instance for the Boxed type, I think it would be useful to enforce NFData on the a in Boxed a, because one thing Unboxed vectors gives us is the guarantee that all its elements are in NF.

but it's also true that unboxed vectors give us the guarantee that all elements are in WHNF, simply because for all the existing unbox instances WHNF == NF. So it's perfectly reasonable for a generalisation to be to WHNF.

And as I note above, the main use case is to unbox constructors with a mixture of field types, including unlifted.

Let me give a concrete use case. I want an unboxed vector of this type:

data IOOp m = IOOpRead  !Fd !FileOffset !(MutableByteArray (PrimState m)) !Int !ByteCount
            | IOOpWrite !Fd !FileOffset !(MutableByteArray (PrimState m)) !Int !ByteCount

This obviously wants to be represented by 5 vectors of bool/int/word etc, and one vector of boxed things. (Of course, even better in this use case is boxed but unlifted, but that's a separate feature).

[dcoutts]

@Shimuuar writes:

I can see two possible names: AsLazyBoxed/AsStrictBoxed using As as reference. Or UnboxVia{Strict,Lazy}Boxed which is based on UnboxViaPrim but it's long and weird. So former it probably better

I like it. In fact I like the latter ones, UnboxVia{Strict,Lazy}Boxed. I think long names here are fine because they're not used often, just in deriving via declarations. And having "UnboxVia..." as a name prefix makes their use case nice and clear, and makes them relatively discoverable. The "As" type while cute and short is quite hard to discover, precisely because of its short generic name. I only stumbled across it because it was next to UnboxViaPrim in the haddock docs.

My personal favourite:

• UnboxViaBoxed{Lifted,Unlifted}

This follows the naming convention of GHC's RuntimeRep with BoxedRep Levity and levity being Lifted or Unlifted.

[Shimuuar]

because for all the existing unbox instances WHNF == NF

But that's only true for primitive types, not for tuples and records with lazy fields which use standard instances.

[dcoutts]

What records with lazy fields? Yes, the tuples are being used as an intermediate type for the IsoUnbox for user records, so that doesn't reflect any "original" lazy fields from the user type. But even there, there's no NF, it's just Unbox again.