Skip to content

[WasmFS] Create public wasmfs_mount function #23827

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/lib/libwasmfs.js
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ addToLibrary({
}
#endif
var backendPointer = type.createBackend(opts);
return FS.handleError(withStackSave(() => __wasmfs_mount(stringToUTF8OnStack(mountpoint), backendPointer)));
return FS.handleError(withStackSave(() => _wasmfs_mount(stringToUTF8OnStack(mountpoint), backendPointer)));
},
unmount: (mountpoint) => (
FS.handleError(withStackSave(() => _wasmfs_unmount(stringToUTF8OnStack(mountpoint))))
Expand Down
8 changes: 5 additions & 3 deletions system/include/emscripten/wasmfs.h
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,13 @@ backend_t wasmfs_get_backend_by_fd(int fd);
// TODO: Remove this function so that only directories can be mounted.
int wasmfs_create_file(const char* pathname __attribute__((nonnull)), mode_t mode, backend_t backend);

// Creates a new directory using a specific backend.
// Returns 0 on success like `mkdir`, or a negative value on error.
// TODO: Add an alias with wasmfs_mount.
// Legacy function. This function works like `mkdir` + `wasmfs_mount`.
int wasmfs_create_directory(const char* path __attribute__((nonnull)), mode_t mode, backend_t backend);

// Mount a backend at a given location in the filesystem
// `path` must be an existing directory.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? Does it delete the existing directory if it exists? Or if the directory exists and has contents, are they converted to the new backend somehow?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The child directory gets replaced. At least that is the idea.

Anything that was there before is no longer accessible. Ideally it would become visibile again if/when the filesystem is unmounted, but I guess that would only really work for things like OPFS that are persistent.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the "process" stays alive and the mounted filesystem is unmounted, then even if the parent backend is something ephemeral like MEMFS, the previous contents should become visible again, right? As long as the parent backend is concerned, those contents were there the whole time.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree that would be ideal. Is that how it works today though? I think we would need to add more tests to confirm.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not how wasmfs_create_directory works, no, because it's not allowed to overwrite or shadow an existing directory. But if we're adding a proper mount, then that's the only reasonable behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it may be simpler to just leave aside the concept of "mounting".

Starting from scratch: all we have here is a system with files and directories, and each file and directory is implemented in some manner. For example, the root is typically implemented using a simple in-memory approach. Some files may be enormous, larger than can fit in wasm memory, so we want to implement them on the JS side (using a Typed Array or Blob or such), so we also have JS-backed files, and so forth.

And, when one creates a file or directory, the default is to create it using the same implementation as the parent. To change that one must override.

So if you have a game with a folder of files, some of which need to use JS, you would createFile most normally, but createFile the JS-backed ones by specifying the JSFile implementation.

Does that make sense? It seems unambiguous to me. This isn't posix.

For example, having wasmfs_create_file sometimes create a file and sometimes not, depending on the backend seems odd to me.

createFile always creates a file, I think?

Its was not clear to me if the file or directory being created lives in backend itself, or in the FS into which the backend is being mounted.

I wouldn't think in terms of things living in backends or being mounted. Instead, all the files lives as files in WasmFS. The technical details of how they are implemented can vary between files (is their data in wasm memory, or in JS), but WasmFS sees them all as a whole. WasmFS will do operations like read() which are then virtual methods that implement the right behavior for each file.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it may be simpler to just leave aside the concept of "mounting".

Starting from scratch: all we have here is a system with files and directories, and each file and directory is implemented in some manner. For example, the root is typically implemented using a simple in-memory approach. Some files may be enormous, larger than can fit in wasm memory, so we want to implement them on the JS side (using a Typed Array or Blob or such), so we also have JS-backed files, and so forth.

This makes some sense to me when you are talking about creating new, ephemeral files.

It makes a little less sense when you are dealing with external, pre-existing files, such as the case for hostfs/nodefs or OPFS. For example, can I decide that I want certain specific files to be implemented using nodefs or OPFS?

And, when one creates a file or directory, the default is to create it using the same implementation as the parent. To change that one must override.

So if you have a game with a folder of files, some of which need to use JS, you would createFile most normally, but createFile the JS-backed ones by specifying the JSFile implementation.

Does that make sense? It seems unambiguous to me. This isn't posix.

Why stray from POSIX though? Its something that is well understood by many folks. The concept of mounting directories that is. I'm not sure I see the benefit of straying. I think you are saying that fine grained mounting of files is the benefit but I'm not sure I see that. Do we have game developers (or anyone) asking for per-file backends?

For example, having wasmfs_create_file sometimes create a file and sometimes not, depending on the backend seems odd to me.

createFile always creates a file, I think?

So I can't use createFile to add an existing file from nodefs or wasmfs? I agree it would be confusing to do this, but its seems like the API wants to allow this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes a little less sense when you are dealing with external, pre-existing files, such as the case for hostfs/nodefs or OPFS. For example, can I decide that I want certain specific files to be implemented using nodefs or OPFS?

Yes, you can. And what that means is left to the backend.

So if you have a normal root and then add a NODEFS directory under /my_nodefs then accessing /my_nodefs will go through that backend, which will then use node in some manner. Typically when creating such a directory you tell the backend what to do, in this case, which real files to refer to. Say, /my_nodefs in WasmFS may map to /home/foo/sandbox/ in the outer filesystem. So accessing /my_nodefs/file.txt will go through NODEFS to access /home/foo/sandbox/file.txt.

Why stray from POSIX though? Its something that is well understood by many folks. The concept of mounting directories that is. I'm not sure I see the benefit of straying.

I would say that the model of WasmFS is simpler than posix... if you are not very used to posix, I suppose. I have really summarized all of it in that one paragraph before, while posix has a lot more detail to it.

I am not opposed to adding full posix support on top, if we want that. But I think the basic model of "WasmFS has files and directories; each of those is implemented in a customizable manner" is a really simple baseline.

I think you are saying that fine grained mounting of files is the benefit but I'm not sure I see that. Do we have game developers (or anyone) asking for per-file backends?

Yes, that feature request came from @juj .

And it is simple in the WasmFS model. Any file and directory can be created in any backend. That is, disallowing files would be more complex.

So I can't use createFile to add an existing file from nodefs or wasmfs? I agree it would be confusing to do this, but its seems like the API wants to allow this.

I think you can use createFile to add an existing file from nodefs. Why would you not be able to?

Perhaps we could decide that nodefs can only create directories and not files, if that feels simpler. I don't recall if we discussed that. But in principle there is no reason not to create files as well. The file would just track the underlying file it is backed by, just like a JSFile tracks the JS side data it is backed by.

Put another way, why would nodefs be different than JSFile? That nodefs files are persistent is a difference, but not a problem (when creating the file as backed by nodefs, you can tell it what it is backed by).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can use createFile to add an existing file from nodefs. Why would you not be able to?

Doesn't that directly contradict what you just said about "createFile always creates a file".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I think I see our misunderstanding.

createFile always creates a WasmFS file.

And a WasmFS file may be backed in various ways. For example, if we create a WasmFS file that is backed by memory, then we allocate memory for it. Or, if we back it by JS, we call into JS to prepare the data structures there. In each case the backend decides what to do.

And what the backend does, can include persistent storage. createFile might create a WasmFS file that is backed by nodefs, and nodefs might be given a path that already exists. In that case we create a WasmFS file but not a file in the underlying host filesystem.

In other words, whether the backend's implementation also creates anything is left to the backend. Likewise, whether it persists anything afterwards, etc. From the WasmFS perspective all those behaviors are abstracted away and left to the backends, leaving WasmFS simpler.

int wasmfs_mount(const char* path __attribute__((nonnull)), backend_t backend);

// Unmounts the directory (Which must be a valid mountpoint) at a specific path.
// Returns 0 on success, or a negative value on error.
int wasmfs_unmount(const char* path __attribute__((nonnull)));
Expand Down
15 changes: 0 additions & 15 deletions system/lib/wasmfs/js_api.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -277,21 +277,6 @@ int _wasmfs_lstat(const char* path, struct stat* statBuf) {
return __syscall_lstat64((intptr_t)path, (intptr_t)statBuf);
}

// The legacy JS API requires a mountpoint to already exist, so WasmFS will
// attempt to remove the target directory if it exists before replacing it with
// a mounted directory.
int _wasmfs_mount(const char* path, ::backend_t created_backend) {
int err = __syscall_rmdir((intptr_t)path);

// The legacy JS API mount requires the directory to already exist, but we
// will also allow it to be missing.
if (err && err != -ENOENT) {
return err;
}

return wasmfs_create_directory(path, 0777, created_backend);
}

// Helper method that identifies what a path is:
// ENOENT - if nothing exists there
// EISDIR - if it is a directory
Expand Down
71 changes: 43 additions & 28 deletions system/lib/wasmfs/syscalls.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -597,8 +597,7 @@ int __syscall_mknodat(int dirfd, intptr_t path, int mode, int dev) {
OpenReturnMode::Nothing);
}

static int
doMkdir(path::ParsedParent parsed, int mode, backend_t backend = NullBackend) {
static int doMkdir(path::ParsedParent parsed, int mode) {
if (auto err = parsed.getError()) {
return err;
}
Expand All @@ -624,41 +623,57 @@ doMkdir(path::ParsedParent parsed, int mode, backend_t backend = NullBackend) {
return -EACCES;
}

// By default, the backend that the directory is created in is the same as
// the parent directory. However, if a backend is passed as a parameter,
// then that backend is used.
if (!backend) {
backend = parent->getBackend();
if (!lockedParent.insertDirectory(childName, mode)) {
// TODO Receive a specific error code, and report it here. For now, report
// a generic error.
return -EIO;
}

if (backend == parent->getBackend()) {
if (!lockedParent.insertDirectory(childName, mode)) {
// TODO Receive a specific error code, and report it here. For now, report
// a generic error.
return -EIO;
}
} else {
auto created = backend->createDirectory(mode);
if (!created) {
// TODO Receive a specific error code, and report it here. For now, report
// a generic error.
return -EIO;
}
[[maybe_unused]] bool mounted = lockedParent.mountChild(childName, created);
assert(mounted);
// TODO: Check that the insertion is successful.

return 0;
}

int wasmfs_mount(const char* path, backend_t backend) {
path::ParsedParent parsed = path::parseParent(path);
if (auto err = parsed.getError()) {
return err;
}
auto& [parent, childNameView] = parsed.getParentChild();
auto lockedParent = parent->locked();
std::string childName(childNameView);

// TODO: Check that the insertion is successful.
// Child must exist and must be directory
auto child = lockedParent.getChild(childName);
if (!child) {
return -EEXIST;
}
if (!child->dynCast<Directory>()) {
return -ENOTDIR;
}

auto created = backend->createDirectory(0777);
if (!created) {
// TODO Receive a specific error code, and report it here. For now, report
// a generic error.
return -EIO;
}
[[maybe_unused]] bool mounted = lockedParent.mountChild(childName, created);
assert(mounted);

return 0;
}

// This function is exposed to users and allows users to specify a particular
// backend that a directory should be created within.
int wasmfs_create_directory(char* path, int mode, backend_t backend) {
static_assert(std::is_same_v<decltype(doMkdir(0, 0, 0)), int>,
// Legacy function, use wasmfs_mount instead.
int wasmfs_create_directory(const char* path, int mode, backend_t backend) {
static_assert(std::is_same_v<decltype(doMkdir(0, 0)), int>,
"unexpected conversion from result of doMkdir to int");
return doMkdir(path::parseParent(path), mode, backend);
int rtn = doMkdir(path::parseParent(path), mode);
if (rtn != 0) {
return rtn;
}

return wasmfs_mount(path, backend);
}

// TODO: Test this.
Expand Down
2 changes: 1 addition & 1 deletion tools/link.py
Original file line number Diff line number Diff line change
Expand Up @@ -1265,7 +1265,7 @@ def phase_linker_setup(options, linker_args): # noqa: C901, PLR0912, PLR0915
'emscripten_builtin_memalign',
'wasmfs_create_file',
'wasmfs_unmount',
'_wasmfs_mount',
'wasmfs_mount',
'_wasmfs_read_file',
'_wasmfs_write_file',
'_wasmfs_open',
Expand Down