API changes proposal

[pietrovismara]

This is a draft of a proposal with the goal of solving the following issues:

• excessive API verbosity and boilerplate needed
• lack of control over the lifecycle of components
• assumptions over the shape of a component store (like expecting SoA in recursivelyInherit, or all the complexity added by assuming SoA typed arrays)
• limitations with Prefabs (not being able to set default values and override them)
• global state (worlds sharing entity cursor, stores being shared by world)

While I try to preserve compatibility with the current API as much as possible, there are some fundamental changes breaking it. What you don't see presented here (e.g. queries, etc) is implied to need no changes.

Entities

addEntity takes any number of components after the world argument:

const scene = addEntity(world, Scene, Transform);

const player = addEntity(
  world,
  IsA(Player),
  withParams(Transform, { translation: [10, 5, 2] }),
  withParams(Contains(gold), 100),
  ChildOf(scene)
);

addComponents works similarly:

addComponents(
  world, 
  player, 
  ComponentA, 
  withParams(ComponentB, ...),
  ...
);

Components and stores

A store factory is a function called whenever a component is registered with a world, creating a unique store for that world.

A store can be anything, indexable by entity ID or not.

const transformSoAStore = () => ({
  translation: [] as [number, number, number][],
});


const transformAoSStore = () => ([] as { translation: [number, number, number] }[]);

defineComponent can be optionally used to assign a generic type argument, used by the onAdd lifecycle hook. (More on hooks later)

type TransformParams = {
  translation: readonly [number, number, number];
};

type TransformStore = [number, number, number][];

const Transform = defineComponent<TransformStore, TransformParams>(transformSoAStore);

onAdd(Transform, (world, store, eid, params) => {
  store.translation[eid] = params.translation; // typed
});

Store factories add a level of indirection, but practically that will only cause a small amount of added verbosity.

A big portion of the most verbose interactions with store can happen in hooks, where the store is passed as argument, while for systems it will just require adding a line before we iterate over entities:

const system = (world) => {
  const transform = getStore(world, Transform);

  for (const eid of query(world, [Transform]) {
    transform.translation[eid]...
  }
}

Prefabs

definePrefab takes any number of components as arguments, and an optional type argument used by the onAdd hook.

The withParams function can be used to type-safely pass parameters to that component, allowing users to set default values.

type MeshParams = {
  vertexShader: string, 
  fragmentShader: string
};

const MeshPrefab = definePrefab<MeshParams>(
  Material, 
  Geometry, 
  withParams(Transform, { translation: [1, 5, 2] })
);

withParams also allows prefab defaults to be overridden:

addEntity(
  world, 
  IsA(MeshPrefab), 
  withParams(Transform, { translation: [4, 4, 4] }) // overrides MeshPrefab's transform defaults
)

Nested prefabs

Prefabs can have children through the ChildOf relation, now provided by bitECS:

// The bitECS equivalent of https://ajmmertens.medium.com/deconstructing-flecs-prefabs-d604b5ba0fcc
type Vec3 = [number, number, number];
const Box = defineComponent<Vec3[], Vec3>(() => []);
const Position = defineComponent<Vec3[], Vec3>(() => []);
const Color = defineComponent<Vec3[], Vec3>(() => []);

const Tree = definePrefab(Position);
const Trunk = definePrefab(
	ChildOf(Tree),
	withParams(Position, [0, 0.25, 0]),
	withParams(Box, [0.4, 0.5, 0.4]),
	withParams(Color, [0.25, 0.2, 0.1])
);
const Canopy = definePrefab(
	ChildOf(Tree),
	withParams(Position, [0, 0.9, 0]),
	withParams(Box, [0.8, 0.8, 0.8]),
	withParams(Color, [0.2, 0.3, 0.15])
);

const world = createWorld();

const tree = addEntity(world, IsA(Tree));
const trunks = query(world, [ChildOf(tree), IsA(Trunk)]);
const canopies = query(world, [ChildOf(tree), IsA(Canopy)]);
assert(trunks.length === 1);
assert(canopies.length === 1);

Relationships

Relationships optionally wrap a component, so that each time a relation is assigned to a target a new component of that type is created.

Relationship components are also unique per world.

// Relations create a new component for each new target
const Contains = defineRelation({
  component: () =>
    defineComponent<number>(() => ({
      amount: [] as number,
    })),
  exclusive: false,
  autoRemoveSubject: false,
});

const ChildOf = defineRelation();

Lifecycle hooks

Lifecycle hooks can be used to initialize and reset store values.

Hooks are independent from worlds, as they are registered to a world together with the component they refer to.

// onAdd, onRemove and similar store the callback in a hidden field on the component
onAdd(Transform, (world, store, eid, params) => {
  store.translation[eid] = params.translation; // typed
});

onRemove(Transform, (world, store, eid, reset) => {
  if (reset) {
    delete store.translation[eid];
  }
});

Hooks work on relationships too:

// Called when Contains(target) is added to an entity
onAdd(Contains, (world, store, eid, amount) => {
  store.amount[eid] = amount; // typed
});

// Called when Contains(target) is removed from an entity
onRemove(Contains, (world, store, eid, reset) => {
  delete store.amount[eid];
});

// Called when an entity targeted by Contains is removed
onTargetRemoved(Contains, (world, eid) => {});

And on prefabs:

onAdd(MeshPrefab, (world, eid, shaders) => {
  const material = getStore(world, Material);
  material.vertexShader[eid] = shaders.vertexShader;
  material.fragmentShader[eid] = shaders.fragmentShader;
});

Performance

I expect these changes to have no negative impact on performance per-se.

I implemented as a PoC most of the features presented here and updated the benches, comparing it with the results from the next branch shows the PoC to actually improve performance by ~10%.

I suspect that is caused by me removing Storage, Serialization, the concept of world size with related checks, and buffered queries, but I haven't tested this assumption.

Removed features

• Storage
• serialization (can't serialize when we make no assumptions on what a store is)
• Changed modifier, as it's basically only useful for serialization
• the concept of world size, as it's there just to decide how much memory to preallocate for typed arrays
• automated recycling of entities: manual recycling as a default with flushRemovedEntities is more than enough, and it's much simpler to reason about
• defineSystem
• buffered queries

The concept is that bitecs should make no assumptions over what a store is and should not be responsible for managing it, providing instead full access to the lifecycle of stores for users to manage it themselves.

Without all the logic stemming from Storage it also becomes much simpler to remove global state and have isolated worlds.
And stuff like prefab inheritance is simpler as well: as of now it works only with SoA and would require a dedicated code path for AoS, with my proposal this is solved at zero cost with lifecycle hooks and parameters.

Following this I also removed buffered queries as they seem to be needed for multi-threading and it wasn't a direct concern of mine. I figured it would still be possible for users to wrap queries results in a typed array if needed.

[krispya]

This is very well conceived. Thank you for taking the time to put it together. Some initial thoughts and questions:

• How is withParams meant to be used? Is it in conjunction with onAdd or part of the defineComponent? In other words, does passing in values to withParams do anything on its own?
• I believe store indirection is fine with getStore and I already use such a pattern in Sweet ECS. Another benefit to using getStore that might not be obvious is that you can type dynamically generated stores with a function call. It also won't impact perf as the main bottleneck by far is iterating over the store, not getting it pointer.
• Can you have multiple callbacks subscribed to lifecycle hooks? Can we also unsubscribe them? I favor the unsub pattern when the onAdd returns an unsub function.
• I think it is fine to break compatibility with bitECS v3.4 and that we should start a migration guide as we do make breaking changes. The principles we follow at Poimandres are stable, pragmatic and simple. Stable means keeping breaking changes to an absolute minimum and pragmatic means having a real world use case that necessitates the breaking change. Simple means it should be easy to understand the new API and its motivation. I think all three tests pass here.

[pietrovismara]

Thanks! Your feedback on having a more functional approach and the discussion around observers is what brought me to this proposal.

How is withParams meant to be used? Is it in conjunction with onAdd or part of the defineComponent? In other words, does passing in values to withParams do anything on its own?

withParams is just a typescript helper function, allowing users to type-safely pass parameters for the onAdd hook. Vanilla JS users could just write [Component, params] instead. This is how it's implemented:

export function withParams<P extends PrefabToken>(
	prefab: P,
	params: P extends PrefabToken<infer Params> ? Params : never
): [P, P extends PrefabToken<infer Params> ? Params : never];
export function withParams<C extends Component>(
	component: C,
	params: C extends Component<any, infer Params> ? Params : never
): [C, C extends Component<any, infer Params> ? Params : never];
export function withParams(a: any, b: any) {
	return [a, b];
}

I believe store indirection is fine with getStore and I already use such a pattern in Sweet ECS. Another benefit to using getStore that might not be obvious is that you can type dynamically generated stores with a function call. It also won't impact perf as the main bottleneck by far is iterating over the store, not getting it pointer.

I agree. Another little bonus of decoupling stores from components is that you could reuse store factories for different components. Say you have an EightBitColor and a FloatColor component, they could both share the same { r: number, g: number, b: number } store but operate differently.

Can you have multiple callbacks subscribed to lifecycle hooks? Can we also unsubscribe them? I favor the unsub pattern when the onAdd returns an unsub function.

There should be a way to unsubscribe and I also favor the unsub pattern.
Wrt multiple callbacks I haven't considered it yet, but if I think of subscribing to changes for reactive UIs it would definitely make sense.

One small concern: iterating over an array of callbacks on each addComponent / removeComponent call could impact performance as those functions are already the main bottlenecks, even though it's up to users how many times they should subscribe.

I'm also interested in the reactive UI use case so we should find a way to make that work. I wonder if a deferred approach could work, where you "drain" a list of lifecycle events from a world once per frame.

[pietrovismara]

Oh and btw I named the hooks onAdd onRemove etc but if there's a preference for going with the observers API that works as well for me. The thing I care about is keeping them world-independent.

[krispya]

Makes sense. I tried a couple ideas out off the top of my head but I think this works the best. We can add some ergonomics by allowing to optionally pass in an onAdd and onRemove directly into defineComponent like this:

const Transform = defineComponent<TransformParams>(
	transformSoAStore,
	(world, store, eid, params) => {
		store.transform[eid] = params.transform;
	},
	(world, store, eid, reset) => {
		if (reset) {
			delete store.translation[eid];
		}
	}
);

Would this make defineComponent required? We have cases where we add components that are strings, numbers or even sometimes refs to stores that manage their own lifecycles. I suppose if you don't use defineComponent then we can simply pass through the component as the store, which is how we handle tags right now.

[pietrovismara]

We can add some ergonomics by allowing to optionally pass in an onAdd and onRemove directly into defineComponent

Makes sense, I didn't consider it but I don't see why not

I suppose if you don't use defineComponent then we can simply pass through the component as the store, which is how we handle tags right now.

Yes that's the idea, tags and whatever else components are useful and should still be supported.

And if you need external references to be unique per world you'll be able to do something like this:

const sceneStore = () => new Three.Scene();

defineComponent(sceneStore, (...) => {
  // init scene
}, (...) => {
 // remove scene
});