Allowing asynchronous 'stdin' streams via Emterpreter

[joesavage]

Though Emscripten allows users to specify custom 'stdin' streams to their
applications through Module.stdin, it seems that the value of this is
significantly diminished by its requirement to be synchronous (see: #23, #1849).

A potential solution to this would be to allow for some kind of synchronisation
construct via Emterpreter, such that prior to blocking I/O calls in the source
program, the program waits for a specific signal before continuing. This way,
the compiled JS (which is, say, running in a web worker), can pause until the
main thread signals its readiness to provide an input stream - solving the
fundamental sync/async mismatch in this problem.

I'm not totally sure if this is best implemented in Emscripten itself (such that
the Emterpreter can tell where blocking calls are automatically and perform the
described task if requested), or whether this is something that programs should
implement on their own - either way, any push in the right direction would be
appreciated as I am personally in need of this functionality for my own project.

[kripken]

See also #3714 which turned out a little too complicated. Perhaps there is a simpler way. The problem there was the integration deep into libc and syscalls, but if instead we provided emscripten_read_sync() or something like that, that did an async read, that should be pretty simple to add.

Basically, look in src/library_async.js at emscripten_sleep_with_yield. That calls resume after a fixed timeout. But we could make a new method that resumes based on any user input that happens later.

[joesavage]

I see. So if I'm understanding you correctly, something like the following should work:

emscripten_wait_until_signal__deps: ['$EmterpreterAsync'],
emscripten_wait_until_signal: function(some_parameters) {
  EmterpreterAsync.handle(function(resume) {
    read_signal(some_parameters, resume);
  }, true);
}

Where it's up to users (e.g. me) to modify their source programs such that emscripten_wait_until_signal is called prior to blocking synchronous calls that wish to accept asynchronous input.

What synchronisation mechanism do you think might be best suited for read_signal in the above? The file system seems like an obvious candidate (reading particular contents from a particular file, or seeing whether a particular file exists), but perhaps there is something more suitable?

[kripken]

Yes, exactly, something like that. And read_signal would need to call resume later at the right time. I'm not sure what a nice JS API for this would look like, that's the hard part here ;)

In other words, read_signal would be a user-defined thing, somehow. So anyone could add a custom "finish an async operation from JS" handler themselves.

[joesavage]

I've been toying around with some ideas in my local copy of Emscripten and it seems that a system in which suspensions are categorised by identifiers works pretty well. For example, say that Module.suspensions is an object which holds the data related to various program suspensions, then the following calls to emscripten_suspend by emterpreter and to Module.resume by the main thread would modify the state as commented:

emscripten_suspend("test"); // Module.suspensions: { "test": { paused: 1, resume_count: 0 } }
Module.resume("test");      // Module.suspensions: { "test": { paused: 0, resume_count: 0 } }
Module.resume("test");      // Module.suspensions: { "test": { paused: 0, resume_count: 1 } }
Module.resume("test");      // Module.suspensions: { "test": { paused: 0, resume_count: 2 } }
emscripten_suspend("test"); // Module.suspensions: { "test": { paused: 0, resume_count: 1 } }
emscripten_suspend("test"); // Module.suspensions: { "test": { paused: 0, resume_count: 0 } }
emscripten_suspend("test"); // Module.suspensions: { "test": { paused: 1, resume_count: 0 } }

I'm having some problems setting this up to work flawlessly though, probably because I just don't know enough about how Emscripten is structured. I can't figure out how to get emscripten_suspend to take a string properly from C, for example. I'm also getting a weird error in the console that I can't figure out - invalid worker function to call: undefined despite the fact that all the functionality seems to work correctly. This is in addition to the errors I get in the console from simply using already-implemented emterpreter functions such as emscripten_sleep_with_yield of exit(1169448) implicitly called by end of main(), but noExitRuntime, so not exiting the runtime and Calling stub instead of signal().

Perhaps this would work better if implemented by somebody who actually knows what the heck they're doing - the actual code in emscripten_suspend is fairly trivial, but I'm not really sure how to get it running smoothly.

[kripken]

is emscripten_suspend implement in a JS library file? then just calling Pointer_stringify on the argument should convert a C string (a pointer) into a JS string.

[joesavage]

Yeah, it is - Pointer_stringify works great, thanks. The only problems I have left now are how I should go about making Module.resume available before emscripten_suspend is necessarily called (currently I initialise the function when this is first called), and the unexplained errors.

EDIT: Alright, I think I've figured out where default Module stuff should go (in shell.js), so I'm left only with the odd errors now. I've committed my changes so far here.

[kripken]

How about opening a PR with this? Then we can discuss on the diff, which is usually convenient.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.