IPNS mount write support

[renne]

Version information:

go-ipfs version: 0.4.17-
Repo version: 7
System version: amd64/linux
Golang version: go1.10.3

Type:

Enhancement

Description:

Using the ipfs add -r and ipfs name publish commands and copy and paste hashes around is quite inconvenient.

So I suggest to make IPFS mounts writable by allowing to directly create directories and files and update them in /ipns/<hash-id>/.

[overbool]

@renne maybe you can try ipfs add -r -q <dir> | tail -n1 | ipfs name publish

[renne]

@overbool
An uppercase Q saves the tail command: ipfs add -r -Q <dir> | ipfs name publish

It doesn't solve the additional manual operation when e.g. sharing the directory via Samba. It is not possible to run ipfs add while /ipfs is mounted read-only.

[djdv]

I am working on an implementation of mount that should be more flexible.
#5003
The current implementation has read only support for IPFS, IPNS, and MFS.
But the goal is to have writable IPNS for keys you own, and writable MFS, at least.

[renne]

I'm using Linux. So this should be platform independent.

[djdv]

I'm using Linux. So this should be platform independent.

We've decided to target cgo-fuse which currently covers macOS, FreeBSD, NetBSD, OpenBSD, Linux, and Windows.

[renne]

I always prefer OS-independent implementations. A drawback I see with cgo-fuse is the lack of go-toolchains for embedded devices (e.g. OpenWRT).

[bam80]

Was it progressed since then in any way?

[djdv]

@bam80
tl;dr Not in any meaningful way. :^(

There were some coordination issues which stalled things for a while, and then the requirements being requested kept/keep changing, which is pushing us farther from this rather than closer.
Originally the task was to make a go-ipfs plugin which added this support, and that worked well enough to demo but it was later requested that this work be a direct component of go-ipfs. (That change required a lot of cleanup to code that hasn't been touched since some of the initial releases)
Now it's being asked to move this effort into a standalone project.

Breaking out of go-ipfs took some effort itself, but a lot is going to have to be (re)written. Before the writable portions can be reimplemented, we'll need to implement some APIs to replace the functionality that was provided by the ones I was using before (when I had direct access to the IpfsNode struct), with the addition of working remotely (like how the CoreAPI works).
This is complicated in its own right but will also require coordination with the project too.
Considering the type of requests we're making it may be unreasonable to rely on the existing HTTP transports too, meaning implementing the CoreAPI over another (local) IPC.
(This remains to be seen, HTTP might be okay, but before we were just doing everything in the same process space)

I shared a status update via IRC and plan to have a repo up for this eventually where progress can be more easily tracked, but am currently in the middle of a mental breakdown so I'm not sure when I'll be able to get to it. (；´∀｀)
Progress has been slow without motivation, I'm sorry for that and hope it turns around.

Anyway, here's the other longwinded status update: https://pastebin.com/2W3RpViA
Contains within it 2 demos of the current implementation which still need work before they're actually useful.
https://www.youtube.com/watch?v=NeuWm8fJoGc
https://www.youtube.com/watch?v=n5iU4R4HxXU
The standalone code for them (temporarily) exists here: https://github.com/djdv/go-ipfs/tree/fs-manager
(the interesting thing is under cmd/fs)
but isn't really useful for practical use yet. I don't know if I still have the old implementations from above but they weren't exactly stable either.

If I make progress on this, I'll try to ping you back here if you'd like.
Spanning multiple years for this feels unacceptable... I'll try not to add another one on it.

@renne

I always prefer OS-independent implementations. A drawback I see with cgo-fuse is the lack of go-toolchains for embedded devices (e.g. OpenWRT).

In addition to Fuse this implementation aims to be agnostic in a lot of ways. It's not tied to go-ipfs specifically, nor is it tied to cgofuse either. The existing implementation allows you to mount an ipfs node (remotley) via bazil Fuse, cgofuse, and expose 9P listening servers. Other file system libraries could be swaped in for this if they're needed.
However, I think in this particular case it'd probably make more sense to patch cgofuse to support this. But if it's easier, and something exists, we could just use it and expose it as an option in addition to the rest.

[bam80]

@djdv Thank you.
I wish you soonest recovering and hope you'll find your way to return to the project.
I'm sure it's highly anticipated peace of technology for many.
So good luck with your endeavor and hope we all get more updates from you soon :)

[bam80]

that worked well enough to demo https://www.youtube.com/watch?v=OX0vM0Ay9Z0

I don't quite understand - we have IPNS already writable, isn't it? (sorry I may be missing things since I am new to all this)

[djdv]

It does in fact work, but it doesn't work reliably yet. Good enough to demo, but not good enough to use.
Same is true for MFS+the FilesAPI.
You can mount them, and reads and writes will work most of the time, but not always.
(specifically lots of concurrent requests at once are not being handled properly somewhere)

---

Deeper technical portion (ignorable):

Those implementations were just experiments though.
The interfaces responsible for this are going to be reworked to use the new Go fs interface (which didn't exist when this was authored, but is very similar), and will be reviewed and better tested in the process.

There's also issues with coordination with things like ipfs name publish, and ipfs files * which are going to have to be worked out.
I wrote a bit about this here but that's out of date now.
Speaking with people from the project, we highlighted some issues around how easy it would be to misuse a (remote) locking API.

There are other ideas around this but it's not been fully thought out yet as some of the other things need to be taken care of first.
But the current idea in mind is to have the file system service expose some interface that lets you query its file table. This way the IPFS node can simply ask it if the key in question is in use. If not the publish can proceed, but otherwise it will error out saying the key is in use by the mount service.

For MFS this doesn't matter at all, but for the FileAPI (the node's own/specific MFS root) we could do either the same thing, or sidestep this entirely by relaying all the operations/data to the node itself. As if you were just doing ipfs files * locally on that node.
We're going to first have to see where the chance for conflicts are, and then figure out how best to deal with them.

IIRC the current implementations are just "last writer wins", so whoever saves last will be reflected on the node's key or Files root. But I haven't looked at it in a while.

[bam80]

I mean, plain go-ipfs seems has IPNS mount writable. Not sure how useful it is, and how it compares with your work.
Are we talking about the same thing?

[djdv]

Short answer is that these are 2 different implementations which use 2 different underlying libraries.

Long answer:
The existing go-ipfs implementation uses a Fuse library (from Bazil) which supports Linux and FreeBSD. I haven't checked on this but I heard that they recently dropped support for macOS, or at least it was causing issues when building go-ipfs on macOS.

One of the libraries I'm targeting is cgo-fuse, which supports more platforms, one of which includes Windows. And could potentially be extended in the future to support even more platforms.

The Bazil Fuse implementations of IPFS and IPNS are very old, and while I don't have any metrics for this, many users have been complaining that they're not practical for use. Using a lot of system resources even when idle, and even more when in use. Grinding the system to a halt unless they're artificially constrained by the OS (which only sidesteps the underlying problem).
Due to the fact that these implementations are bound to fewer platforms already, I never intended on fixing them up. Rather it made more sense to just re-implement them for a new library, which supported the same platforms and more, and likely should be more performant.

So the existing implementations are likely more correct, but not performant enough for actual use as far as I can tell.
The new implementations have given me no problems so far in terms of resource usage or perceived latency. But also, isn't fully correct yet.

---

Rambling on more differences:

Furthermore, there are other issues around the old implementations, such as the fact that it uses more layers of indirection than is necessary. The IPNS implementation utilizes the MFS library, and indirectly (sym)links into the IPFS mountpoint rather than just doing IPNS operations directly. I'm sure this contributes a lot of overheard, specifically all the locking that takes place within MFS, and jumping between process and kernel space via the host FS.
I haven't looked at the implementation too deeply because it's likely going to be only for legacy or deleted outright.
Also, some of that might be incorrect since I just kind of skimmed the code enough to port it to my interfaces and am trying to recall it now.

The cgo-fuse implementations I have stands alone, and just uses IPNS and IPFS methods of the CoreAPI directly, rather than linking into an external mountpoint and indirecting through the filesystem multiple times.
To put that another way, the existing implementation requires IPFS be mounted for IPNS to function fully, where the new IPNS can be mounted independently. Only using IPFS internally.

Most important but hardest to express is how these implementations are managed and used at an API level by the host program. The existing implementation's interfaces are somewhat rigid and have heavy connections to the IPFS node with some preconceived expectations.
The implementation I have exposes a bunch of constructors and management interfaces which abstract a lot of the details away, and allow for some greater flexibility at multiple layers. This helps separate platform differences rather than just expecting that the host is a POSIX-like system. And also should make it easier to implement a new file system (like "PinFS") or host file system API (like Fuse, 9P, NFS) that uses the existing file system implementations without needing any code changes.

I highlighted some of these things in a video , but it's not super interesting.
It's a lot of little things like allowing you to mount IPFS and/or IPNS rather than forcing both, allowing IPFS in offline mode, multiple mount instances rather than just 1, and lot of other little things around sending errors back to the daemon and requester, formatting, and other things I don't remember lol

[bam80]

Thank you for the introspection.
I'm just trying to understand, given the fact legacy implementation has IPNS mount writable, does it mean it actually supports writable MFS mount?
Documentation says nothing about it (or I read it wrong).

[RubenKelevra]

How about this - create a small service that converts NFS server commands to an MFS path by sending the requests to the go-ipfs daemon?

ipfs/roadmap#83

NFS is basically supported on all platforms, even as boot mediums. :)