Skip to content

Latest commit

 

History

History
652 lines (437 loc) · 31.9 KB

explainer.md

File metadata and controls

652 lines (437 loc) · 31.9 KB

Realms Explainer

Introduction

The Realms proposal provides a new mechanism to execute JavaScript code within the context of a new global object and set of JavaScript built-ins.

The API enables control over the execution of different programs within a Realm, providing a proper mechanism for virtualization. This is not possible in the Web Platform today and the proposed API is aimed to a seamless solution for all JS enviroments.

There are various examples where Realms can be well applied to:

  • Web-based IDEs or any kind of 3rd party code execution using same origin evaluation policies.
  • DOM Virtualization (e.g.: Google AMP)
  • Test frameworks and reporters (in-browser tests, but also in node using vm).
  • testing/mocking (e.g.: jsdom)
  • Most plugin mechanism for the web (e.g., spreadsheet functions).
  • Sandboxing (e.g.: Oasis Project)
  • Server side rendering (to avoid collision and data leakage)
  • in-browser code editors
  • in-browser transpilation

This document expands a list of some of these use cases with examples.

API (TypeScript Format)

This is The Realms API description in TypeScript format:

declare class Realm {
    constructor();
    readonly globalThis: typeof globalThis;
    import(specifier: string): Promise<Namespace>;
}

The proposed specification defines:

Quick API Usage Example

const realm = new Realm();
// realms can import module that will execute within it's own environment.
realm.import('./file.js');
// realms exposes access to its ordinary global object as a communication
// channel between the incubator realm and the newly created realm.
realm.globalThis;

Motivations

It's quite common for applications to contain programs from multiple sources, whether from different teams, vendors, package managers, etc, or just programs with different set of requirements from the environment.

These programs must currently contend for the global shared resources, specifically, the shared global object, and the side effect of executing those programs are often hard to observe, causing conflicts between the different programs, and potentially affecting the integrity of the app itself.

Attempting to solve these problems with existing DOM APIs will require to implement an asynchronous communication protocol, which is often a deal-breaker for many use cases. It usually just adds complexity for cases where a same-process Realm is sufficient. It's also very important that values can be immediately shared. Other communications require data to be serialized before it's sent back and forth.

The primary goal of this proposal is to provide a proper mechanism to control the execution of a program, providing a new global object, a new set of intrinsics, no default access to the incubator realm's global object, a separate module graph and synchronous communication with the incubator realm.

In addition to the motivations given above, another commonly-cited motivation is virtualization and portability. Some of the functionalities of the VM module in Node can also be standardized, providing the infrastructure for virtualization of JavaScript programs in all environments.

Finally, a distinct but related problem this proposal could solve is the current inability to completely virtualize the environment where the program should be executed. With this proposal, we are taking a giant step toward that missing feature of the language.

Realms is an often-requested feature from developers, directly or indirectly. It was an original part of the ES6 spec, but it didn't make to the initial cut. This proposal attempts to resolve prior objections and get to a solution that all implementers can agree upon.

How does Realms operate?

Realms execute code with the same JavaScript heap as the surrounding context where the Realm is created. Code runs synchronously in the same thread. Note: The surrounding context is often referenced as the incubator realm within this proposal.

Same-origin iframes also create a new global object which is synchronously accessible. Realms differ from same-origin iframes by omitting Web APIs such as the DOM.

Sites like Salesforce.com make extensive use of same-origin iframes to create such global objects. Our experience with same-origin iframes motivated us to steer this proposal forward, which has the following advantages:

  • Frameworks would be able to better craft the available API within the global object of the Realm, aiming for what is necessary to evaluate the program.
  • Tailoring up the exposed set of APIs into the code within the Realm provides a better developer experience for a less expensive work compared to tailoring down a full set of exposed APIs - e.g. iframes - that includes handling presence of [LegacyUnforgeable] attributes like window.top.
  • We hope the usage of Realms will be somewhat lighter weight (both in terms of memory and CPU) for the browser if compared to iframes, especially when frameworks rely on several Realms in the same application.
  • Realms are not accessible from by traversing the DOM of the incubator realm. This will ideal and/or better approach compared to attaching iframes elements and their contentWindow to the DOM. Detaching iframes would even add a new own set of problems.
  • A newly created realm does not have immediate access to any object from the incubator realm and won't have access to window.top as iframes would.

Realms are complementary to stronger isolation mechanisms such as Workers and cross-origin iframes. They are useful for contexts where synchronous execution is an essential requirement, e.g., emulating the DOM for integration with third-party code. Realms avoid often-prohibitive serialization overhead by using a common heap to the surrounding context.

The Realms API does not introduce a new evaluation mechanism. The code evaluation is subject to the same restrictions of the incubator realm via CSP, or any other restriction in Node.

JavaScript modules are associated with a global object and set of built-ins. Realms contain their own separate module graph which runs in the context of that Realm, so that a full JavaScript development experience is available.

Clarifications

Terminology

In the Web Platform, both Realm and Global Object are usually associated to Window, Worker, and Worklets semantics. They are also associated to their detachable nature, where they can be pulled out from their parent DOM tree.

This proposal is limited to the semantics specified by ECMA-262 with no extra requirements from the web counterparts.

The Realm's Global Object

Each Realm's Global Object is an Ordinary Object. It does not require exotic internals or new primitives.

Instances of Realm Objects and their Global Objects have their lifeline to their incubator Realm, they are not detachable from it. Instead, they work as a group, sharing the settings of their incubator Realm. In other words, they act as encapsulation boundaries, they are analogous to a closure or a private field.

Evaluation

The Realms API does not introduce a new way to evaluate code, it is subject to the existing evaluation mechanisms such as the Content-Security-Policy (CSP).

If the CSP directive from a page disallows unsafe-eval, it prevents synchronous evaluation in the Realm. E.g.: Realm#globalThis.eval, Realm#globalThis.Function.

The CSP of a page can also set directives like the default-src to prevent a Realm from using Realm#import().

Module Graph

Each instance of Realms must have its own Module Graph.

const realm = new Realm();

// imports code that executes within its own environment.
const { doSomething } = await realm.import('./file.js');

doSomething();

Compartments

This proposal does not define any virtualization mechanism for host behavior. Therefore, it distinguishes itself from the current existing Compartments proposal.

A new Compartment provides a new Realm constructor. A Realm object from a Compartment is subject to the Compartment's host virtualization mechanism.

const compartment = new Compartment(options);
const VirtualizedRealm = compartment.globalThis.Realm;
const realm = new VirtualizedRealm();
const { doSomething } = await realm.import('./file.js');

The Compartments proposal offers a more complex API that offers tailoring over aspects beyond the global APIs but with modifications to internal structure such as module graph. The Realms API just offers immediate access to what is already specified in ECMAScript as it's already structured to distinguish different references from realms.

Why not separate processes?

Creating a Realm that runs in a separate process is another alternative, while allowing users to define and create their own protocol of communication between these processes.

This alternative was discarded for two main reasons:

  1. There are existing mechanism to achieve this today in both browsers, and nodejs. E.g.: cross domain iframes, workers, etc. They seem to be good enough when asynchronous communication is sufficient to implement the feature.
  2. Asynchronous communication is a deal-breaker for many use-cases, specially when security is not an issue, and sometimes it just added complexity for cases where a same-process Realm is sufficient.

E.g. Google AMP run in a cross domain iframe, and just want more control about what code they executed in that cross domain application.

There are some identified challenges explained within the current use cases for Realms such as the WorkerDOM Virtualization challenge for Google AMP and the current use of JSDOM and Node VM modules that would be better placed using an interoperable Realms API as presented by this proposal.

Use Cases

These are some of the key use cases where The Realms API becomes very useful and important:

  • Third Party Scripts
  • Code Testing
  • Codebase segmentation
  • Template libraries
  • DOM Virtualization

Trusted Third Party Scripts

We acknowledge that applications need a quick and simple execution of Third Party Scripts. There are cases where many scripts are executed for the same application. There isn't a need for a new host or agent. This is also not aiming for prevention over non-Trusted Third Party Scripts like malicious code or xss injections. Our focus is on multi libraries and building blocks from different authors.

The Realms API provides integrity preserving semantics - including built-ins - of root and incubator Realms, setting specific boundaries for the Environment Records.

Third Party Scripts can be executed in a non-blocking asynchronous evaluation through the Realm#import().

There is no need for immediate access to the application globals - e.g. window, document. This comes as a convenience for the application that can provide - or not - values and API in different ways, like frozen properties set in the Realm#globalThis. This also creates several opportunities for customization with the Realm Globals and prevent collision with other global values and other third party scripts.

import { fmw } from 'pluginFramework';
const realm = new Realm();

// fmw becomes available in the Realm
realm.globalThis.fmw = fmw;

// The Plugin Script will execute within the Realm
await realm.import('./pluginScript.js');

Code Testing

While multi-threading is useful for testing, the layering enabled from Realms is also great. Test frameworks can use Realms to inject code and also control the order of the injections if necessary.

Testing code can run autonomously within the boundaries set from the Realm object without immediately conflicting with other tests.

Running tests in a Realm

import { test } from 'testFramework';
const realm = new Realm();

realm.globalThis.test = test;
await realm.import('./main-spec.js');

test.report();

Test FWs + Tooling to run tests in a realm

const realm = new Realm();
const [ framework, { tap } ] = await Promise.all([
 realm.import('testFramework'),
 realm.import('reporters')
]);

framework.use(tap);
await realm.import('./main-spec.js');

Codebase segmentation

A big codebase tend to evolve slowly and soon becomes legacy code. Old code vs new code is a constant struggle for developers.

Modifying code to resolve a conflict (e.g.: global variables) is non-trivial, specially in big codebases.

The Realms API can provide a lightweight mechanism to preserve the integrity of the intrinsics.0 Therefore, it could isolate libraries or logical pieces of the codebase per Realm.

Template libraries

Code generation should not be subject to pollution (global, prototype, etc.). E.g.: Lodash's _.template() uses Function(...) to create a compiled template function, instead it could use a Realm to avoid leaking global variables. The same Realm could be reused multiple times.

var compiled = _.template(
'<% _.forEach(users, function(user) { %><li><%- user %></li><% }); %>'
);

compiled({ users: ['user1', 'user2'] });

DOM Virtualization

We still want things to interact with the DOM without spending any excessive amount of resources.

It is important for applications to emulate the DOM as best as possible. Requiring authors to change their code to run in our virtualized environment is difficult. Specially if they are using third party libraries.

import virtualDocument from 'virtual-document';

const realm = new Realm();

realm.globalThis.document = virtualDocument;

await realm.import('./publisher-amin.js');

DOM Virtualization: AMP WorkerDOM Challenge

Problem: Element.getBoundingClientRect() doesn't work over async comm channels (i.e. worker-dom).

AMP WorkerDOM Challenge diagram

The communication is also limited by serialization aspects of transferable objects, e.g.: functions or Proxy objects are not transferable.

JSDOM + vm Modules

JSDOM relies on VM functionality to emulate the HTMLScriptElement and maintains a shim of the vm module when it is bundled to run in a webpage where it doesn’t have access to the Node's vm module.

The Realms API provides a single API for this virtualization in both browsers and NodeJS.

Virtualized Environment

The usage of different realms allow customized access to the global environment. To start, The global object could be immediately frozen.

let realm = new Realm();

Object.freeze(realm.globalThis);

In web browsers, this is currently not possible. The way to get manage new Realms would be through iframes, but they also share a window proxy object.

let iframe = document.createElement('iframe');
document.body.appendChild(iframe);
const rGlobal = iframe.contentWindow; // same as iframe.contentWindow.globalThis

Object.freeze(rGlobal); // TypeError, cannot freeze window proxy

The same iframe approach won't also have a direct access to import modules dynamically. The usage of realm.import('./file.js'); is possible instead of roughly using eval functions or setting script type module in the iframe, if available.

DOM mocking

The Realms API allows a much smarter approach for DOM mocking, where the globalThis can be setup in userland:

const realm = new Realm();

await installFakeDOM(realm.globalThis);

// Custom properties are now added to the Realm such as
realm.globalThis.document;
realm.globalThis.top;
realm.globalThis.location;

This code allows a customized set of properties to each new Realm - e.g. document - and avoid issues on handling immutable accessors/properties from the Window proxy. e.g.: window.top, window.location, etc..

This explainer document speculates a installFakeDOM API to set up a proper frame emulation. We understand there might be many ways to explore how to emulate frames with plenty of room for improvement, as seen in some previous discussions, as in the following pseudo-code:

function installFakeDOM(realmGlobalThis) {
    const someRealmIntrinsicsNeededForWrappers = extractIntrinsicsFromGlobal(realmGlobalThis);
    Object.defineProperties({
         document: createFakeDocumentDescriptor(someRealmIntrinsicsNeededForWrappers),
         Element: createFakeElementDescriptor(someRealmIntrinsicsNeededForWrappers),
         Node: ...
         ... // all necessary DOM related globals should be defined here
    });
}

function createFakeDocumentDescriptor(someIntrinsics) {
     return {
          enumerable: true,
          configurable: false,
          get: new someIntrinsics.Proxy(document, createHandlerWithDistortionsForDocument(someIntrinsics));
     };
}

function extractIntrinsicsFromGlobal(realmGlobalThis) {
   return {
       Proxy: realmGlobalThis.Proxy,
       ObjectPrototype: Object.prototype,
       create: Object.create,
       ... // whatever you need to facilitate the creation of proper identities for the fake DOM
   };
}

That's one option to use proxies, notice that the Proxy constructor used is from the Realm, same for any proxy handler, or object/function/array accessible from inside the realm (e.g.: the one produced by createHandlerWithDistortionsForDocument should be an object with __proto__ set to realmGlobalThis.Object.prototype, this helps with the errors identity). Again, identity issues are hard to solve when multiple realms are playing together, but libraries and frameworks can tackle that at the lower level.

Modules

In principle, the Realm proposal does not provide the controls for the module graphs. Every new Realm initializes its own module graph, while any invocation to Realm.prototype.import() method, or by using import() when evaluating code inside the realm, will populate this module graph. This is analogous to same-domain iframes, and VM in nodejs.

However, the Compartments proposal plans to provide the low level hooks to control the module graph per Realm. This is one of the intersection semantics between the two proposals.

Integrity

We believe that realms can be a good complement to integrity mechanisms by providing ways to evaluate code who access different object graphs (different global objects) while maintaining the integrity of the outer realm. A concrete example of this is the Google's AMP current mechanism:

  • Google News App creates multiples sub-apps that can be presented to the user.
  • Each sub-app runs in a cross-domain iframe (communicating with the main app via post-message).
  • Each vendor (one per app) can attempt to enhance their sub-app that display their content by executing their code in a realm that provide access to a well defined set of APIs to preserve the integrity of the sub-app.

There are many examples like this for the web: Google Sheets, Figma's plugins, or Salesforce's Locker Service for Web Components.

Security vs Integrity

There are also other more exotic cases in which measuring of time ("security") is not a concern, especially in IOT where many devices might not have process boundaries at all. Or examples where security is not a concern, e.g.: test runners like jest (from facebook) that relies on nodejs, JSDOM and VM contexts to execute individual tests while sharing a segment of the object graph to achieve the desired performance budget. No doubts that this type of tools are extremely popular these days, e.g.: JSDOM has 10M installs per week according to NPM's registry.

More Examples

Example: Importing Module

let r = new Realm();
const { x } = await r.import('/path/to/foo.js');

In this example, the new realm will fetch, and evaluate the module, and extract the x named export from that module namespace. Realm.prototype.import is equivalent to the dynamic import syntax (e.g.: const { x } = await import('/path/to/foo.js'); from within the realm. In some cases, evaluation will not be available (e.g.: in browsers, CSP might block unsafe-eval), while importing from module is still possible.

Example: Virtualized Contexts

Importing modules allow us to run asynchronous executions with set boundaries for access to global environment contexts.

  • main.js:
globalThis.DATA = "a global value";

let r = new Realm();

// r.import is equivalent to the dynamic import expression
// It provides asynchronous execution, without creating or relying in a
// different thread or process.
r.import("./sandbox.js").then(({test}) => {

  // globals in this root realm are not leaked
  test("DATA"); // undefined

  let desc = test("Array"); // {writable: true, enumerable: false, configurable: true, value: ƒ}
  let Arr = desc.value;

  Arr === r.globalThis.Array; // true
  Arr === Array; // false

  // foo and bar are immediately visible as globals here.
});
  • sandbox.js:
// DATA is not available as a global name here

// Names here are not leaked to the root realm
var foo = 42;
globalThis.bar = 39;

export function test(property) {

  // Built-ins like `Object` are included.
  return Object.getPropertyDescriptor(globalThis, property);
}

Example: Simple Subclass

class EmptyRealm extends Realm {
  constructor(...args) {
    super(...args);
    let globalThis = this.globalThis;

    // delete global descriptors:
    delete globalThis.Math;
    ...
  }
}

In the example above, the global object of a newly created realm of EmptyRealm will have no global Math available.

Example: DOM Mocking

const realm = new Realm();

await installFakeDOM(realm.globalThis);

// Custom properties are now added to the Realm such as
realm.globalThis.document;
realm.globalThis.top;
realm.globalThis.location;

Example: iframes vs Realms

If you're using anonymous iframes today to "evaluate" javascript code in a different realm, you can replace it with a new Realm, as a more performant option, e.g.:

const globalOne = window;
let iframe = document.createElement('iframe');
document.body.appendChild(iframe);
const globalTwo = iframe.contentWindow;

will become:

const globalOne = window;
const globalTwo = new Realm().globalThis;

Example: Indirect Evaluation

This operation should be equivalent, in both scenarios:

globalOne.eval('1 + 2'); // yield 3
globalTwo.eval('1 + 2'); // yield 3

Example: Direct Evaluation

This operation should be equivalent, in both scenarios:

globalOne.eval('eval("1 + 2")'); // yield 3
globalTwo.eval('eval("1 + 2")'); // yield 3

Example: Identity Discontinuity

Considering that you're creating a brand new realm, with its brand new global variable, the identity discontinuity is still present, just like in the iframe example:

let a1 = globalOne.eval('[1,2,3]');
let a2 = globalTwo.eval('[1,2,3]');
a1.prototype === a2.prototype; // yield false
a1 instanceof globalTwo.Array; // yield false
a2 instanceof globalOne.Array; // yield false

Example: Node's vm objects vs Realms

If you're using node's vm module today to "evaluate" javascript code in a different realm, you can replace it with a new Realm, e.g.:

const vm = require('vm');
const script = new vm.Script('this');
const globalOne = globalThis;
const globalTwo = script.runInContext(new vm.createContext());

will become:

const globalOne = globalThis;
const globalTwo = new Realm().globalThis;

Note: these two are equivalent in functionality.

Status Quo

Using VM module in nodejs, and same-domain iframes in browsers. Although, VM modules in node is a very good approximation to this proposal, iframes are problematic.

Iframes

Developers can technically already create a new Realm by creating a new same-domain iframe, but there are a few impediments to using this as a reliable mechanism:

  • the global object of the iframe is a window proxy, which implements a bizarre behavior, including its unforgeable proto chain.
  • There are multiple unforgeable unvirtualizable objects due to the DOM semantics, this makes it almost impossible to eliminate certain capabilities while downgrading the window to a brand new global without DOM.
  • The global top reference cannot be redefined and leaks a reference to another global object. The only way to null out this behavior is to detach the iframe, which imposes other problems, the more relevant is dynamic import() calls.

Detachable

For clarifications, the term detachable means an iframe pulled out from the DOM tree:

var iframe = document.createElement("iframe");

 // attaching the iframe to the DOM tree
document.body.appendChild(iframe);

var iframeWindow = iframe.contentWindow;

// Get accessor that returns the topmost window.
iframeWindow.top; // Cannot be properly redefined/virtualized: { get: top(), set: undefined, enumerable: true, configurable: false }

// **detaching** the iframe
document.body.removeChild(iframe);

// get accessor still exists, now returns null
iframeWindow.top;

FAQ

So do Realms only have the ECMAScript APIs available?

Yes! It only exposes a new copy of the built-ins from ECMAScript, but it allows extensions defined by each host.

// Proposal:

Realm ()

...
11. Perform ? SetDefaultGlobalBindings(O.[[Realm]]).
...

// ECMAScript

SetDefaultGlobalBindings ( realmRec )

1. Let global be realmRec.[[GlobalObject]].
2. For each property of the Global Object specified in clause 18, do
...

18 The Global Object

...
- may have host defined properties in addition to the properties defined in this specification. This may include a property whose value is the global object itself.

Most libraries won't work unless they add dependencies manually

Doesn't this mean that most libraries won't work unless to add its dependencies manually, like realm.globalThis.fetch = fetch. Like we could see people using this even to isolate WebAssembly code, thought that requires you adding the methods needed for that.

Absolutely, this is equivalent to what happens to Node VM today as a low level API prior art. As a developer you need to setup the environment to execute code.

Ideally the Realms would arrive a clean state, allowing tailoring for what is necessary to be added. This contrasts with the tailoring over unforgeables. e.g. window.top, window.location, etc

Considering all the trade offs, the clean state seems the best option, in our opinion. It allows tailoring for multiple purposes and comprehends more use cases.

The top-level Realm cannot be accessed

We are also a bit afraid that regular developers will have a hard time understanding all these concepts (realms, globals, this) and how they relate to each other: realms, like what is a realm really, especially since the top-level realm (like the one with window === globalThis) cannot be accessed as a Realm object.

Executed code doesn't need to know it's in a realm, this is designed to be a concern for those setting the realm up. Ideally, code executed in a realm would run seamlessly. There is prior art for this (iframes, Workers, node.vm).

Additional feedback from @littledan:

One thing to understand here is that Realms are generally intended to be a sort of metaprogramming construct, which would be used by frameworks and libraries to build emulated JS environments for developers. I understand the feedback that this concept may be difficult for JS developers to understand; probably an introduction in the explainer to show how múltiple globals in JS already work would help make this document more accessible. Either way, it is an underlying primitive in the platform.

Realm MVP or max/min

Maybe for consistency sake it would make sense to have an accessor to expose it as a realm, thought currently the only thing exposed is globalThis and import - but we assume that could be extended in the future.

The initial Realms proposal had more content and more ways to access things. We tried to build a MVP and hope we can explore expansions of the API in the future.