Scala 3 typeclass derivation

[keynmol]

(This is very early, Opening it to check the approach and to use the CI)

Goals:

scala> import cats.kernel._

scala> import cats.syntax.all._

scala> case class Hello(b: Int, c: String) derives Eq,Order,Semigroup
// defined case class Hello

scala> (Hello(25, "whaaat") |+| Hello(11, "yo")) > Hello(35, "h")
val res3: Boolean = true

And making sure to achieve parity with kittens.

TODO:

• Verify this is binary compatible
• Do a more efficient version (without zips/allocations)
• Verify that instance construction fromProduct is the only way to do so
• See if there's a way to not re-derive Semigroup's combine for Monoid (and all similar things)

[ybasket]

Isn't that scala.compileTime.summonAll[Tuple.Map[T, F]].toList?

I believe you could also drop the toList, tuples are pretty powerful in Scala 3, even zipping is supported.

Anyway, I very much like the initiative of adding this to cats and will watch this PR (and stop nitpicking now 😉)!

[keynmol]

You're probably right :) I've copied my older code verbatim to test if this is binary compatible and if it works, so now I'm leisurely researching all the ways to make this more concise and faster :)

[keynmol]

I managed to remove the lists and iterators and instead only use the tuples (plus inlining the whole thing).

Now there's a concern about inlining the derived call when deriving the coproduct, not sure if this can have any repercussions

[johnynek]

I think there are a few cases we can handle:

1. F is Contravariant and Semigroupal
2. F is (Covariant) Monoidal
3. F is Invariant Monoidal

It would be cool to have derivation methods for those cases, and the majority of our type classes (Order, Monoid, Eq, ... ) are one of these cases.

[keynmol]

Now that I had more thoughts about this:

1. Hand-rolling efficient derivators is cool and fun (for me), but the low-level API is untyped and products are index based, so making a mistake here is easier than ever

2. shapeless 3 will be available for Scala 3, bringing a generic version for derivation which can satisfy all sorts of usecases

3. My initial idea was to be able to provide the derives UX which is very nice, and I thought that it required the derived method to be defined on the companion object.

   Well I've just made a terrifying discovery :D

   object MyStuff:
     def derived[T](using m: Mirror.Of[T]) = new Hash[T] {
       def hash(x: T) = -1
       def eqv(x: T, y: T) = hash(x) == hash(y)
     }
   
   extension (comp: cats.kernel.Hash.type)
     inline def derived[T](using m: Mirror.Of[T]) = MyStuff.derived[T]
   
   case class Hello(a: Int) derives Hash // works

4. All this means that if kittens does eventually provide derivation through shapeless, it can both be generic enough to not handroll these things, and provide a import cats.derived.given UX to use the derives keyword with the same results.

I'd love to hear maintainers' thoughts (and going to bring in @joroKr21 into the discussion), and may be this PR can be closed.

[joroKr21]

I was just going to suggest giving shapeless 3 a try (I haven't yet myself 😄).
But yeah the low level API offers very little in terms of type safety.
And I assume eventually we will also want to support the Functor hierarchy as well.

One edge case I was thinking about - there are actually more ways to one to derive Functor.
Check out cats-tagless for example which provides derivation for traits and probably won't work with mirrors.
However if we put the "blessed" derived method in the companion object we wouldn't be able to use derives in cats-tagless.
Not terribly important at this point but something to keep in mind.

[joroKr21]

Here is how it could look like in kittens with shapeless 3:

• adding derived as an extension method works
• shapeless 3 is pretty neat - almost no boilerplate for regular instances
• we can use inheritance to mirror the cats hierarchy
• I haven't figured out yet how irregular instances would look like (e.g. Reducible)

package cats.derived

import alleycats._
import cats._
import shapeless3.deriving.K0

object empty extends EmptyDerivation
object semigroup extends SemigroupDerivation
object monoid extends MonoidDerivation

object all extends
  EmptyDerivation,
  SemigroupDerivation,
  MonoidDerivation

class ProductEmpty[F[x] <: Empty[x], A](
  using inst: K0.ProductInstances[F, A]
) extends Empty[A]:
  val empty: A = inst.construct([A] => (F: F[A]) => F.empty)

trait EmptyDerivation:
  extension (E: Empty.type)
    inline def derived[A](using gen: K0.ProductGeneric[A]): Empty[A] =
      ProductEmpty(using K0.mkProductInstances)

class ProductSemigroup[F[x] <: Semigroup[x], A](
  using inst: K0.ProductInstances[F, A]
) extends Semigroup[A]:
  def combine(x: A, y: A): A =
    inst.map2(x, y)([A] => (F: F[A], x: A, y: A) => F.combine(x, y))

trait SemigroupDerivation:
  extension (S: Semigroup.type)
    inline def derived[A](using gen: K0.ProductGeneric[A]): Semigroup[A] =
      ProductSemigroup(using K0.mkProductInstances)

class ProductMonoid[F[x] <: Monoid[x], A](
  using inst: K0.ProductInstances[F, A]
) extends ProductSemigroup[F, A], Monoid[A]:
  val empty: A = inst.construct([A] => (F: F[A]) => F.empty)

trait MonoidDerivation:
  extension (M: Monoid.type)
    inline def derived[A](using gen: K0.ProductGeneric[A]): Monoid[A] =
      ProductMonoid(using K0.mkProductInstances)

[ChristopherDavenport]

Just to make my opinion clear, I'd prefer if we could get something for this for scala 3 in cats. Kittens is great, but another dependency is another moment to get buy-in to commit. I think if we can offer this in vanilla scala 3 in cats we should do so.

[joroKr21]

We could if we accept Shapeless 3 as a dependency.

[mpilquist]

Or we could shade shapeless 3 to avoid dependency clashes

[joroKr21]

Or we could shade shapeless 3 to avoid dependency clashes

Do you mean clashes with Shapeless 2? I think actually there should be no clash.

[mpilquist]

No, I mean clashes against future versions of Shapeless 3. The binary compatibility story of Shapeless 2 caused lots of pain for certain folks -- in particular, some Cats maintainers who were stuck on an old version of Shapeless due to Spark pulling it in. If we were to have cats-kernel and cats-core depend on Shapeless 3, I'd expect a number of folks to object due to those experiences.

[joroKr21]

I guess it's not a problem to shade shapeless3-deriving - it's very light. But in general I would advocate to move past such experiences and look towards the future adopting semantic versioning as the Scala ecosystem matures.

[mpilquist]

It's not a question of semantic versioning but rather stability of the dependency. If shapeless3-deriving isn't going to break binary compatibility until / coincident with Cats 3, then the dependency isn't a concern. Would need to hear more from you or @milessabin w.r.t. long term binary stability.

[joroKr21]

I didn't work on Shapeless 3 but I would expect once 3.0 is out for it to remain stable for a long time. But Miles knows better 😛

[milessabin]

shapeless has maintained patch version number bincompat for years now. To the best of my knowledge the only issues have been things like, eg., Spark depending on ancient versions.

[milessabin]

I would expect once 3.0 is out for it to remain stable for a long time.

I think so too. shapeless 3.0 is modularized, and I think cats would only need to depend on shapeless-3 deriving, which I wouldn't expect to evolve any faster than cats.

But I think it would be a shame to duplicate the fantastic work that @joroKr21 and @kailuowang have done on kittens. If people don't want to add a dependency then I think it might make more sense to make kittens a cats module.

[joroKr21]

Right now we are collaborating in Kittens with @TimWSpence on Scala 3 derivation.
I'd be more than happy if this work lands in cats core.

[milessabin]

Apropos @joroKr21's reference to the work going on in kittens, see here: typelevel/kittens#330.

[TimWSpence]

Just wondering if we're any closer to a concensus on this?

[joroKr21]

Any thoughts on this question:

One edge case I was thinking about - there are actually more ways to one to derive Functor.
Check out cats-tagless for example which provides derivation for traits and probably won't work with mirrors.
However if we put the "blessed" derived method in the companion object we wouldn't be able to use derives in cats-tagless.

[armanbilge]

I think derivation out-of-the-box in Cats would be fantastic, particularly for Scala 3 with derives syntax. At this point it's more likely to happen by way of Kittens than this PR, I reckon :)

[keynmol]

What is it with you, Arman, and closing my PRs :D