ArrayBuffer and Binary json encoding #111

yourWaifu · 2023-04-05T21:30:01Z

I needed this for a project, and so I implemented them for myself. However, I noticed that others asked about it here, #68, so I decided to share them.

Example:

socket.on("data", (chunk) => {
  // imagine IPC between a program that sends their chunks with a 4 byte length little endian prefix
  const lengthLength = 4;
  let length = 0;
  for (var i = 0; i < lengthLength; i += 1) {
	  length = (length * 256) + chunk[i];
  }
  
  // create QuickJS ArrayBuffer
  let bufferStart = lengthLength;
  let bufferEnd = length + lengthLength;
  this._contextJS.newArrayBuffer(new Uint8Array(chunk.slice(bufferStart, bufferEnd)))
  .consume((handle) => {
	  // decode data inside the buffer
	  this._contextJS?.decodeBinaryJSON(handle)
		  .consume((handle) => console.log(this._contextJS?.dump(handle)));
  });

justjake

Thanks for the contribution! Your changes look good. I have a suggestion for improving the efficiency of copying a ArrayBuffer into QuickJS, and it would also be nice to see some tests.

justjake · 2023-04-06T23:24:01Z

ts/context.ts

+   * 
+   * **WARNING**: QuickJS's binary JSON doesn't have a standard so expect it to change between version
+   */
+  encodeBinaryJSON(handle: QuickJSHandle): QuickJSHandle {


It would be nice to include a comment in the code showing how to use this that shows up in the documentation. How would I persist or transfer this value over the network? What's the use-case?

I mostly have been using decodeBinaryJSON to handle IPC between an app that uses quickJS and one that uses node.js. However, I realized that encodeBinaryJSON isn't all that useful without some sort of getArrayBuffer function. I've been trying to write one, but it seems to not want to work.

justjake · 2023-04-06T23:24:26Z

c/interface.c

+}
+
+// --------------------
+// Debugging QuickJS values 


This comment doesn't make sense to me

justjake · 2023-04-06T23:53:40Z

c/interface.c

@@ -301,6 +301,10 @@ JSValue *QTS_NewArray(JSContext *ctx) {
  return jsvalue_to_heap(JS_NewArray(ctx));
 }

+JSValue *QTS_NewArrayBuffer(JSContext *ctx, JSVoid *buffer, size_t length) {
+  return jsvalue_to_heap(JS_NewArrayBufferCopy(ctx, (uint8_t*)buffer, length));


This method always does an extra copy:

Copy from source ArrayBuffer to Emscripten memory at the _malloc pointer

Copy from the _malloc pointer to the quickjs ArrayBuffer

We can eliminate copy 2 by using JS_NewArrayBuffer to take ownership of the already malloc'd WebAssembly memory. You'd need to define a JSFreeArrayBufferDataFunc, but it'd be super simple, something like void qts_free_buffer(JSRuntime *unused_rt, void *unused_opaque, void *ptr) { free(ptr) }. Then skip wrapping the malloc pointer in a lifetime and you're good to go.

yourWaifu · 2023-04-13T23:17:47Z

I added a getArrayBuffer function, but the values outputed by it seem to be incorrect for unknown reasons, so we should put this into draft until that can be fixed

justjake · 2023-04-14T00:41:52Z

If the array buffer creation bit works well, can we merge that without the JSONB parts?

yourWaifu · 2023-04-16T18:59:45Z

After some experiments, I noticed that NewArrayBufferCopy and NewArrayBuffer creates different outputs when using GetArrayBuffer. I'm guessing you are supposed to read the buffer differently but not sure how the code knows the difference and how it reads NewArrayBuffer. Using NewArrayBufferCopy fixes all the issues with GetArrayBuffer.

yourWaifu · 2023-04-24T14:39:26Z

ts/context.ts

      .newHeapBufferPointer(array)
-      .consume((bufferHandle) => this.ffi.QTS_NewArrayBuffer(this.ctx.value, bufferHandle.value.pointer, array.length))


removing this line seemed to fix the issue. Not sure if it'll cause more issues, but it seems to be fine

justjake

I would love to see some tests for this stuff. Other than that, it looks ready to merge.

ts/memory.ts

ts/context.ts

justjake · 2023-04-25T02:00:17Z

c/interface.c

+// I don't know how to return two values in C, maybe allocate memory in stack?
+size_t QTS_GetArrayBufferLength(JSContext *ctx, JSValueConst *data) {


I think we could add stackAlloc and use it to pass in a *size_t argument like the &length arguments in QuickJS's own API, but it's not document anywhere so I think your current approach is fine.

Looking online, the best solutions I could find is to use Emscripten's embind API and a separate cpp file to use the API. However, I'm not familiar with Makefile, so I don't really know how to get it to compile the separate cpp file.

I can find what the command should look like:

$ emcc --bind -O3 --std=c++11 a_c_file.c another_c_file.c -x c++ your_cpp_file.cpp

But not how to do it using Make because I get this error

emcc: error: cannot specify -o with -c/-S/-E/-M and multiple source files

Removing -c gets me this error

wasm-ld: error: unknown file type: build/wrapper/interface.WASM_DEBUG_SYNC.o

Also updated the docs files

yourWaifu · 2023-06-02T21:53:59Z

It still says awaiting for approval

sgrove · 2023-07-19T05:38:17Z

I also came across a use case where I need to share an ArrayBuffer between host and client, it'd be great to see this merged in!

stickmy · 2023-08-28T11:00:46Z

I has the same scenario, but i went some issues.

javascript code executed in quickjs

console.log(new Float32Array([-1, -1, 1, -1, -1, 1, 1, 1]));

When i use vm.getArrayBuffer() in console.log implementation, i got a error:
Couldn't allocate memory to get ArrayBuffer at QuickJSContext.getArrayBuffer

yourWaifu changed the title ~~Binary json encoding~~ ArrayBuffer and Binary json encoding Apr 5, 2023

justjake approved these changes Apr 7, 2023

View reviewed changes

yourWaifu commented Apr 24, 2023

View reviewed changes

yourWaifu requested a review from justjake April 24, 2023 15:27

justjake approved these changes Apr 25, 2023

View reviewed changes

Hao Qi Wu added 12 commits May 5, 2023 17:59

Add binary JSON encoding

acc6171

Add comments about binary json encoding

7caa3d7

Run Typedoc

03339ba

Add warning about binary JSON in docs

b39dfe8

Undo link changes to yourWaifu

1d751e4

undo accidental changes

6797fa4

change JS_NewArrayBufferCopy to JS_NewArrayBuffer

f09e3f3

Add getArrayBuffer function

bb300c7

GetArrayBuffer seems to work now

dc36718

Add example code to encode and decode Q.JS JSON B.

e42c728

Also updated the docs files

Small name and type changes

4682fce

fix merge conflicts

b6a8a04

yourWaifu force-pushed the binary-json-encoding branch from 56146eb to b6a8a04 Compare May 5, 2023 22:14

run prettier

3bf4084

justjake merged commit 24c340c into justjake:main Jul 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ArrayBuffer and Binary json encoding #111

ArrayBuffer and Binary json encoding #111

yourWaifu commented Apr 5, 2023 •

edited

Loading

justjake left a comment

justjake Apr 6, 2023

yourWaifu Apr 13, 2023

justjake Apr 6, 2023

justjake Apr 6, 2023

yourWaifu commented Apr 13, 2023 •

edited

Loading

justjake commented Apr 14, 2023

yourWaifu commented Apr 16, 2023 •

edited

Loading

yourWaifu Apr 24, 2023

justjake left a comment

justjake Apr 25, 2023

yourWaifu Apr 30, 2023 •

edited

Loading

yourWaifu commented Jun 2, 2023

sgrove commented Jul 19, 2023

stickmy commented Aug 28, 2023

		.newHeapBufferPointer(array)
		.consume((bufferHandle) => this.ffi.QTS_NewArrayBuffer(this.ctx.value, bufferHandle.value.pointer, array.length))

		// I don't know how to return two values in C, maybe allocate memory in stack?
		size_t QTS_GetArrayBufferLength(JSContext ctx, JSValueConst data) {

ArrayBuffer and Binary json encoding #111

ArrayBuffer and Binary json encoding #111

Conversation

yourWaifu commented Apr 5, 2023 • edited Loading

justjake left a comment

Choose a reason for hiding this comment

justjake Apr 6, 2023

Choose a reason for hiding this comment

yourWaifu Apr 13, 2023

Choose a reason for hiding this comment

justjake Apr 6, 2023

Choose a reason for hiding this comment

justjake Apr 6, 2023

Choose a reason for hiding this comment

yourWaifu commented Apr 13, 2023 • edited Loading

justjake commented Apr 14, 2023

yourWaifu commented Apr 16, 2023 • edited Loading

yourWaifu Apr 24, 2023

Choose a reason for hiding this comment

justjake left a comment

Choose a reason for hiding this comment

justjake Apr 25, 2023

Choose a reason for hiding this comment

yourWaifu Apr 30, 2023 • edited Loading

Choose a reason for hiding this comment

yourWaifu commented Jun 2, 2023

sgrove commented Jul 19, 2023

stickmy commented Aug 28, 2023

yourWaifu commented Apr 5, 2023 •

edited

Loading

yourWaifu commented Apr 13, 2023 •

edited

Loading

yourWaifu commented Apr 16, 2023 •

edited

Loading

yourWaifu Apr 30, 2023 •

edited

Loading