The Next API

[mikeal]

After @vmx and I had a talk about what the next iteration of the format API should look like it occurred to me that it’s about time for us to start writing all of this out and discussing it.

Here’s an outline of my ideas for the next API iteration.

serialize(native Object)

Returns a Promise that resolves to an IPLD Block instance.

toBlock(binary Blob)

Returns a Promise that resolve to an IPLD Block instance.

reader(IPLD Block Instance)

Returns an implementation of the Reader interface 👇🏼

Reader.get(path)

Returns a Promise that resolves to an object in the following format.

{ value: ‘some value’ }
// or
{ link: CID Instance,
  remaining: ‘some/path’
}
// or
{ link: CID Instance }

Reader.links()

Returns an async iterator of all the links in the block as instances of CID.

Reader.tree()

Returns an async iterator of all the paths available in the block.

[mikeal]

A few things about this interface.

1. We’re fully embracing the IPLD Block API.
2. By putting all the read operations in an API that is scoped to a block we save on lots of duplicate deserialize steps we hit now by always passing in the binary representation.
3. We don’t need the cid() method anymore because the block instance always has the CID and there’s practically no additional overhead in just creating block instances.
4. The links() API is something we’ve been missing that is currently solved by higher level libraries. The problem with doing this at a higher level is that you end up having to iterate over every property recursively to find links. Some formats have faster ways to pull out just the links, I believe this is one of the reasons we use tags in CBOR, so this needs to be part of the format in order to work efficiently.

[rvagg]

What does toBlock() do that's format-specific? Is it await format.toBlock(buf) -> IPLD Block { CID, buf }?

[vmx]

Where would you get the "native Object" that you can pass into serialize() from?

[vmx]

/cc @Gozala, you probably will enjoy the discussions on this issue.

[Gozala]

@vmx I would love to provide feedback, however I'm not exactly sure I understand what is is being proposed here is this alternative API for implementing IPLD codec ?

Aside: I would also very much love if PL folks embraced flow / typescript syntax for interface definitions, not only they will more precisely capture API but would also allow users to utilize them when working with IPFS. Happy to be of help there as well.

1. We’re fully embracing the IPLD Block API.

I love this this, I always found it annoying that most IPFS APIs attempt to take huge variety of inputs attempting to normalize internally, that promotes incidental complexity - reading code and knowing API is not enough to tell what are the things being passed around. Monophonic APIs + casting / transform functions as an alternative make it easier to reason - Reading code and knowing API is enough to know what is being passed around.

2. We don’t need the cid() method anymore because the block instance always has the CID and there’s practically no additional overhead in just creating block instances.

While I do like uniformity API, I'm less eager to wrap things when unnecessary. Specifically I do have nodes that are encoded multiple times before creating a block. I only need CID for the last encoding and it appears to me that proposed API will need to create [CID, Uin8tArray] each time.

Maybe Block class could be provided that will take care of lazy CID creation, but even then you'd be still boxing actual buffers into Block instances.

reader(IPLD Block Instance)
Returns an implementation of the Reader interface 👇🏼

I'm failing to see how adding Reader is helpful ? Why not just keep things simple and define interface for IPLD codec implementation as follows:

// a is type / shape of the node this codec implements
interface IPLDCodec<a> {
  serialize(a):Promise<IPLDBlock<a>>;
  toBlock(BinaryEncoded<a>):Promise<IPLDBlock<a>>;
  get <b>(IPLDBlock<a>, path:string):Promise<GetResult<b>>;
  links <b>(IPLDBlock<a>):AsyncIterable<CID<b>>;
  tree(IPLDBlock<a>):AsyncIterable<string>;
}

type GetResult <a> =
  | {type:"value", value:a}
  | {type:"link", link: CID<a>, remaining?: string }

interface IPLDBlock<a> {
  cid:CID<a>;
  data:BinaryEncoded<a>;
}

export opaque type BinaryEncoded<a>:Uint8Array = Uint8Array

Above accomplishes the same except it removes need for wrapping IPLDBlock into Reader and implementing a codec is also straight forward. Only reason I can imagine wanting Reader instance if it provided a way for implementer to use instance as cache placeholder is that relevant ? If it is than I'd suggest slightly modified API:

interface IPLDCodec<data, reader> {
  serialize(data):Promise<IPLDBlock<data>>;
  toBlock(BinaryEncoded<data>):Promise<IPLDBlock<data>>;

  reader(IPLDBlock<data>):reader;
  get <b>(reader, path:string):Promise<GetResult<b>>;
  links <b>(reader):AsyncIterable<CID<b>>;
  tree(reader):AsyncIterable<string>;
}

type GetResult <a> =
  | {type:"value", value:a}
  | {type:"link", link: CID<a>, remaining?: string }

interface IPLDBlock<a> {
  cid:CID<a>;
  data:BinaryEncoded<a>;
}

Above would allow implementer to wrap IPLDBlock into some wrapper when it makes sense (for caching purposes) or just return IPLDBlock instead.

Note that this also allows Reader class to be implemented in the next layer, it will just need to keep reference for the IPLDBlock and codec.

[rvagg]

Yeah @mikeal is going to have to define some types and whether we're talking about instance methods or factories.

I think I'm seeing something like:

const FormatX = { // not `class`, these are all static factories
  serialize: async (o) => {
    // `o` is anything inheriting from `Object` that can be serialized
    // returns `IPLDBlock{CID, UInt8Array}` serialization
  },

  toBlock: async (buf) => {
    // `buf` is a `UInt8Array`-like
    // returns `IPLDBlock{CID, UInt8Array}` canonical form
  },

  reader: (block) => return new FormatXReader(block)
}

class FormatXReader {
  constructor (block) {}
  async get (path) {
    // `path` is a string
    // returns a `{ value, link, remaining }` thing that needs clearer specification
  }
  *links () {
    // returns async iterator of CIDs in this block
  }
  *tree () {
    // returns async iterator of strings representing paths in this block
  }
}

Rationale for the detached Reader would be helpful. As would the distinction between serialize and toBlock, is toBlock just responsible for appropriate CID generation and returning a well-formatted IPLDBlock?

[mikeal]

rationale for toBlock

It’s really just to cover the CID creation. The old interface will handle CID creation for you. It’s not strictly necessary, the consumer could create the CID given just the codec string, but it’s rather convenient so I didn’t want to lose it in the transition. We can consider getting rid of it tho as it isn’t completely necessary.

80% of the time that I need a CID I also need a Block instance, and creating the Block instance is very little overhead even if you only want the CID. It also is just nice to have symmetry with the serialize function.

rationale for detached Reader

This is so that you can cache the decoding. The current interface requires you to pass the binary blob into every call, example:

resolve(binary, ‘/some/key’)
resolve(binary, ‘/some/other/key’)

You’re going to do a full decode of the binary blob into native types for every property resolution. This is what pushes people to just decode the full block and do property access themselves, which we specifically want to get away from (which is why there’s no full block decoder in this API).

[vmx]

which we specifically want to get away from (which is why there’s no full block decoder in this API).

I don't :) I prefer accessing the data of block directly rather then "querying" them for data. It kind of reminds me of the days where Web Frameworks needed .get() and .set() rather then accessing things directly. I found those clunky to use.

[vmx]

@Gozala We should really use Flow types. I'm in favour of that and I'd like to use the inline comment style even for code. I hopefully will find the time to actually do that.

[mikeal]

The important ideas from this ended up in https://github.com/ipld/js-ipld-stack

[Gozala]

@mikeal are you still interested in feedback ? If so, would that repo be a place to provide it ?

[mikeal]

@Gozala yes, definitely :) especially since now we can call out specific lines :)