Windows mount support

[djdv]

Tracking/discussion issue for the topic of implementing ipfs mount on Windows.
This is a broad topic, all comments and criticisms are welcome here as long as they help drive us towards a solution. Nothing is set in stone yet and we'd like to implement this correctly.

Currently we have no first party support for this:

Error: Mount isn't compatible with Windows yet

For third party, I'm aware of these projects:
https://github.com/alexpmorris/dipfs
https://github.com/richardschneider/net-ipfs-mount
both utilize the IPFS API and Dokany(a Windows FUSE alternative).

dipfs does not appear to be maintained.
In my experience it will succeed in mounting IPFS as a drive letter, and allows you to traverse IPFS paths via CLI, however it hangs in explorer (likely from it trying to access metadata), or when trying to read data through any means.

net-ipfs-mount is currently being maintained.
This is our best contender, everything appears to work as intended unless you have a non-tiny pinset. When trying to traverse IPFS paths with a large enough pinset, passing the list of pins from the API to net-ipfs-mount can take long enough for Windows to deem /ipfs inaccessible.
For local testing you can run ipfs pins ls and see how long it takes to return.

---

I would like to start an initiative, aimed at getting Windows mounting on par with the other platforms.
That is to say, at the very least, exposing read only access to /ipfs and /ipns constructs.
If possible, it would be nice to extend the feature set to expose a writable MFS root as well, similar to this https://github.com/tableflip/ipfs-fuse

Most likely, this will mean implementing first party support for mount in go-ipfs, utilizing core APIs where possible.

Mention of https://github.com/billziss-gh/winfsp has come up as an alternative to Dokany. It's likely that winfsp's native API will be our target, however this is not locked down yet. If you have opinions on Windows filesystem APIs, positive or negative, please post them here.

cc:
@mgoelzer, @alanshaw, @dryajov, @mrlindblom, @Kubuxu, @whyrusleeping

Edit: forgot to cc: @alexpmorris, @richardschneider

[xelra]

My vote is for WinFsp.

It has an active developer and in my opinion is the best FUSE alternative for Windows. I'm using it daily in conjunction with SSHFS.

[billziss-gh]

Adding myself here so that I can receive notifications and provide help.

[billziss-gh]

I add here some comments I made over private email regarding the task of creating a Go shim for WinFsp.

Some initial thoughts on the kind of tasks you will have to handle for a Go shim. The WinFsp Tutorial may also be useful to you as it lists the major tasks needed for a successful file system using the native API.

• Find and load the WinFsp DLL.
  • This is normally done using the inline C function FspLoad, which looks into the registry to find where the WinFsp DLL is installed and loads it. This will have to be rewritten in Go.
• WinFsp includes API’s for creating Windows services, etc. You do not need to use those if you do not plan to run IPFS as a Windows service, but if you do they are named FspService*.
• You create a file system by using FspFileSystemCreate and you mount it using FspFileSystemSetMountPoint. You can mount on a drive or directory, but mounting on a directory requires a case-insensitive file system under Windows.
• The file system starts handling operations when you call FspFileSystemStartDispatcher. This will start a number of threads and will call your file system operations on those threads. The default number of threads is the same as the number of processors (with a minimum of 2).
• Most of the file system operations should be relatively easy to understand, especially if you have experience with FUSE. Of notable difference is how files are deleted on Windows and the security model (which uses security descriptors and not POSIX permissions).

I am happy to elaborate on any of this.

[djdv]

Much appreciated @billziss-gh. Very glad to have your support with this, and a clear outline of what's necessary. 👍

WinFsp includes API’s for creating Windows services

I have considered packaging go-ipfs in an MSI that optionally installs a service for the ipfs daemon. There seems to be some support for this in golang already but it's nice to know about interactions with services and WinFSP. We may be able to take advantage of it.

[billziss-gh]

NP. Feel free to ping anytime and I will continue monitoring this issue.

WinFsp includes API’s for creating Windows services

I have considered packaging go-ipfs in an MSI that optionally installs a service for the ipfs daemon.

While I do not understand the internals of IPFS, in general file systems benefit from being run as Services under Windows. I outline some of the reasons here:

https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture

[djdv]

No going back now 👀

[BillDStrong]

https://github.com/billziss-gh/cgofuse is a go library for fuse maintained by the winfsp maintainer. It might be best in the long run to use it to create one ipfs-mount that works on all platforms. winfsp has a fuse compatability mode, when used with a cygwin dll. This would also allow decoupling somewhat the mount functionality.

[djdv]

I should mention I'll be dumping my work here from time to time
https://github.com/djdv/go-ipfs/tree/feat/win-mount
It will probably be ugly for a while. I'll divvy up the source files and refactor after things have at least a basic implementation. That is to say, create the binds, and rework them to be more go-like where/when possible.

@BillDStrong
I agree that it would be better to have a unified mount implementation, however, I don't want to introduce a C compiler as a build dependency if it can be avoided.
We can write an independent Windows variant now, wait for the *nix version to be refactored, and maybe derive an interface from commonality between them if it's desirable. We'll have to see how redundant it would be and if there'd be any benefit in making porters implement another abstraction layer or just be independent.

For the moment, I'm dumping this in /mount/winfsp, we may move /fuse into there and have a structure like this:

mount/
    fuse/
    interface/
    winfsp/

or

mount/
    fuse/
    winfsp/
    mount_interface_and_friends.go

We'll have to see in time.

[BillDStrong]

I looks like cgofuse has a cross compilation option based on xgo. It looks to include the necessary windows winfsp libraries setup. This would still add a dependency on the c compiler, but xgo's ease of use should be enough to offset that downside. It is contained in a docker container, so compilation shouldn't require to much setup over the current workflow.

This is just my opinion, though.

Prerequisites: docker, xgo
Build:

$ docker pull billziss/xgo-cgofuse
$ go get -u github.com/karalabe/xgo
$ cd YOUR-PROJECT-THAT-USES-CGOFUSE
$ xgo --image=billziss/xgo-cgofuse \
    --targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 .

[Kubuxu]

@BillDStrong thanks for showing me xgo. Few blockers: 1. teaching https://github.com/ipfs/distributions how to use xgo 2. making sure that Jenkins can handle windows cgo on Windows.

[djdv]

Excluding the build dependencies, I'm concerned with some of the API differences
http://www.secfs.net/winfsp/develop/native-api-vs-fuse/

I'd rather not target FUSE on Windows if it means making compromises. Since we already have FUSE for the other platforms, it feels like we're not restricted to it (on Windows) now.
It may still be best to have separate, but native, implementations. Giving us as much compatibility/(flexibility later) with the host as we can get.

Thoughts on this? @BillDStrong @Kubuxu @billziss-gh

[billziss-gh]

TLDR

Cgofuse gets you nice cross-platform compatibility (on the 3 OS'es) but at the cost of having some major external dependencies (native toolchain or docker+xgo). These may or may not be appropriate for a large project with an established community and expectations.

In detail

I can see both sides of the argument.

I originally created cgofuse to support rclone. As I was new with Golang at the time, I followed @ ncw 's pointers and used cgo to create a single layer that would run on the 3 OS'es.

I find that cgofuse has been a success as it allows the creation of a file system that runs on the 3 OS'es with relative ease. For example, I use cgofuse in my fledgling project objfs (currently pre-alpha) and I do not have to think very much about cross-platform issues. This benefit outweighs for me any potential negatives.

Now for those negatives:

• Since cgofuse is a cgo project, the native toolchain is required for every user of cgofuse. @ ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

• Xgo works, but is not the same as the familiar go build. So I find that in practice I do not use it as much as I thought I would. (I also tend to have the right compilers at the ready on all my machines/VM's, so go build is easier for me.) And of course xgo requires Docker and the xgo-cgofuse docker image.

• Cgofuse currently works on "the 3 OS'es" (Windows, macOS and Linux). I am hoping to eventually extend to at least FreeBSD, but this does limit the target platforms.

Additional considerations for go-ipfs

• An additional negative for go-ipfs is that it appears to have a large community, which might balk at the idea of introducing a native toolchain requirement in order to support "Windoze" :-) But I am new to IPFS, so I cannot speak on that.

Native API vs FUSE.

Given that IPFS is likely closer to the POSIX file system model rather than the Windows file system model, the differences between the two API's might not mean much for IPFS. For example, if IPFS has no need for and does not use "security descriptors" (Windows ACL's), the ability to use the native GetSecurity and SetSecurity might not be very important.

[djdv]

@billziss-gh
Thanks for sharing the experience! It's very helpful to hear.

ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

I'm curious to hear more about this, since that seems to be the current task at hand for me.
Wrapping the DLL and making Go and C play nice with each other at runtime, and trying to make a sane api around it that allows us to implement an FSP filesystem in Go.

I'm wondering if something couldn't be done to extend cgo-fuse to add dynamic-link support via build tags. However this still restricts us to FUSE.

---

On the topic of FUSE vs Native.
While IPFS itself has POSIX constructs, there's nothing preventing us from reading and storing metadata like ADS, ACL, etc. alongside normal data.
The practical utility of this is unknown though. It would be interesting to see a single file hash that contains file data as well as multiple sets of platform specific metadata. At the same time though, non-portable platform metadata should probably be avoided altogether by now. Hard to say if we should strive to support or ignore these things when dealing with a bridge between multiple systems.

I'm being extra cautious to harp on these details now, only to make sure we don't lock ourselves into something that will cause problems later. It seems like FUSE at least provides the necessities, but I want to make sure this is talked about first.

While not critical at the moment, I should also bring up performance as something to consider as well here. Will the lack of async i/o bite us?

---

On the topic of more build dependencies.
I'd rather decrease the requirements to build on Windows rather than increase them, but if we decide cgo-fuse is the way to go, I won't be the one to object. Not to mention the possibility of a dynamic cgo-fuse varient.

[billziss-gh]

ncw and I tried quite hard to work with precompiled binaries so that rclone would not need to include a native toolchain in its build process. Unfortunately this did not seem possible (at least as of Go 1.8, which was current at the time of the experimentation).

I'm curious to hear more about this...

My recollection is that we were trying to create what golang calls "binary only packages", which were implemented in Go 1.7 (I think). While this experiment was successful the native toolchain was still required to build the final executable of a project that uses cgofuse (perhaps only the native linker was required).

Ping @ncw who may remember more on this. Nick, the question is if you recall what the issues with using cgofuse as a binary only package were.

I'm wondering if something couldn't be done to extend cgo-fuse to add dynamic-link support via build tags. However this still restricts us to FUSE.

I have considered before the possibility of modifying cgofuse to use the syscall.LoadLibrary approach on Windows to hook into the WinFsp-FUSE interface and avoid using the C compiler on that platform. I did not follow through because the Go libraries lack good dlopen support for non-Windows platforms, which made the use of cgo mandatory.

It may still be worthwhile to do this on the Windows platform only.

Another approach (and perhaps the one you allude to @djdv) is to build cgofuse as a shared library or a plugin. Unfortunately buildmode=shared and buildmode=plugin are still not supported on Windows (I think).

While IPFS itself has POSIX constructs, there's nothing preventing us from reading and storing metadata like ADS, ACL, etc. alongside normal data.

This is actually related to something I deeply care about: how to streamline the differences between the 2 major approaches to file systems (POSIX and Windows). ACL's are a big part of that.

For example, I would love to be able to set ACL's on Windows and have them easily translate into ACL's on my Mac, especially because the ACL systems are conceptually compatible. The biggest hurdle I usually face is the need for a user name/id mapping service, but it sounds like IPFS may already have that problem solved.

While not critical at the moment, I should also bring up performance as something to consider as well here. Will the lack of async i/o bite us?

In general I have not found the lack of async I/O in the FUSE interface to be a performance problem.

One complaint from some users with heavy-weight scenarios, is that their file system may sometimes stall while doing too many blocking operations, because they run out of threads. The remedy is usually to increase the number of threads; some even go as far as to implement asynchronous I/O. Only a small number of commercial users with highly parallel file systems have had the need to implement asynchronous I/O.

Increasing the number of OS threads is not very costly in WinFsp (in terms of context-switching), because it uses IOCP scheduling under the hood. More details in this document.