Formats/ergonomics of importing JS into a Pyret program.

[jpolitz]

@shriram and I talked briefly the other day about quickly documenting what it takes to import a JS file into CPO.

Like, say you wanted to use https://simple-statistics.github.io/

I revisited it, and it's pretty bad.

There are a bunch of things going on. I figure it's worth documenting them.

Pyret modules, after compilation, have a specific shape.

({
  requires: [ ... ],
  nativeRequires: [ ... ],
  provides: { ... },
  theMap: "... a source map ...", // optional
  theModule: function(runtime, _, uri) { ... }
})

There are also “native JS files” that Pyret is aware of, which are RequireJS-style modules that look like

define("module-name", [... list of dependencies ...], function(... dependencies ...) { ... }

Crucially, the define style modules are all resolved when the runtime is built, and it's not particularly straightforward to add more after the fact (well, you can, because actually it's a dumb dictionary behind the scenes. But you need somewhere to run JS code to add to it. which is a chicken-and-egg problem to get that JS running in the first place). Indeed, the only way to get from a Pyret file into one of the raw JS modules is to first import a compiled Pyret file (in the first format), which lists it in its nativeRequires.

The structure of Pyret modules is not just for eval-ing and JSON parsing. It is also used to build “standalone files”, which have a big concatenated dictionary of all the modules, and Pyret expects that the static source of the module is amenable to sticking in the standalones in this format.

The upshot of that is that you can't do something like:

... my cool js code here ...

({
  requires: [ ... ],
  nativeRequires: [ ... ],
  provides: { ... },
  theMap: "... a source map ...", // optional
  theModule: function(runtime, _, uri) { ... }
})

because that code needs to be able to go directly into a context like { ... "jsfile://my-module": ... direct text of module file ... }

Next up would be “I know, use a thunky thing":

(() => {

... my cool js code here ...

({
  requires: [ ... ],
  nativeRequires: [ ... ],
  provides: { ... },
  theMap: "... a source map ...", // optional
  theModule: function(runtime, _, uri) { ... }
})

})

However, this does not work, because other parts of the runtime rely on evaling these strings (or the moral equivalent, like inserting a script tag), and getting the result. If you write a function like this, it can correctly go in function position, but it doesn't return a value. Figuring out where to insert the return is of course intractable.

So, to use the stats library above, here's the best I figured out:

({
	requires: [],
	nativeRequires: [],
	theModule: function(runtime, _, uri) {

		// In this case I noticed the format of the library used exports.foo = val, so I added
		const exports = {};

	... copy paste all of https://unpkg.com/simple-statistics@7.8.5/dist/simple-statistics.js here ...

		// Here's the issue: there's no good way to
		// *synchronously fetch and run that code* with browser APIs.
		// Could do it async with fetch! But then need a pauseStack
		// Could append a script tag with that URL, but that is massively gross
		// and the script tag waits to run, so immediate callers may see
		// uninitialized state.
		// So it has to be pasted in.


	    function sumUsingLib(rawArray) {
	        return sum(rawArray); 
	    }
	    return runtime.makeModuleReturn({
	      "sum-using-library": runtime.makeFunction(sumUsingLib)
	    }, {});

	},
	provides: {
	    values: { "sum-using-library": ["arrow", [["RawArray", "Number"]], "Number"] }
    },

})

Here it is: https://drive.google.com/file/d/1IOVJM6sRndHoE6F2kc35m511aMRLjyzM/view

This program uses it:

import gdrive-js("jsfile.arr", "1IOVJM6sRndHoE6F2kc35m511aMRLjyzM") as J
J.sum-using-library([raw-array: 4, 5, 6])

Note, though, that if you try that Pyret just copy-pasting into code.pyret.org it probably won't work for you. You'd have to open lib.js.arr with Pyret (right-click and open in Pyret via the GDrive interface – which is why it has to have a .arr extension!) to bless it as openable by you, and then try the program again.

That file is only 4000 lines long, but if you do something like D3 all of a sudden its 10s or 100s of thousands of lines, putting editors into modes where autocomplete, syntax highlighting, etc don't really work.

So there needs to be some kind of separate build step to bundle this up for real use, though the above is what I'd document to get someone started today.

It's definitely possible to massage formats and contexts to accept more “normal looking” JS files that happen to return stuff on module.exports, etc. It's just a project to do it, hence this issue documenting where it's at.