Hermes or Static Hermes as Native Messaging host

[guest271314]

I'm trying to integrate Hermes or Static Hermes into my Native Messaging host tests. Here are the Native Messaging protocol documentation for Chrome, Safari, Microsoft Edge, and on MDN https://github.com/guest271314/NativeMessagingHosts?tab=readme-ov-file#native-messaging-documentation and here is the protocol as described by Chrome Developers

Chrome starts each native messaging host in a separate process and communicates with it using standard input (stdin) and standard output (stdout). The same format is used to send messages in both directions; each message is serialized using JSON, UTF-8 encoded and is preceded with 32-bit message length in native byte order. The maximum size of a single message from the native messaging host is 1 MB, mainly to protect Chrome from misbehaving native applications. The maximum size of the message sent to the native messaging host is 4 GB.

I have an existing C implementation that I'm testing using with first, that doesn't work when used with shermes -typed ./fopen-dev-stdin.ts. Here's the working C code

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
uint8_t* getMessage(size_t *inputLength) {
  // Read the first 4 bytes (32 bits) into the memory allocated for the variable 
  // `inputLength`. 
  uint32_t messageLength = 0;
  // The operator `&` gives a pointer to the variable, so `fread()` writes directly
  // into `inputLength`. 
  size_t result = fread(&messageLength, sizeof(messageLength), 1, stdin);
  // Allocate a buffer that is 'messageLength' elements long. `calloc() and 
  // `malloc()` return a pointer to memory on the heap.
  // The operator `*` dereferences a pointer (`*message' is equivalent to 
  // `message[0]`).
  // `sizeof(*message)` will hence return the size in bytes of the type 
  // `message` points at, which means it's equivalent to `sizeof(uint_8)`.
  // To ensure that the message ends with a null terminator
  // rather than whatever garbage is left over after the allocated buffer add
  // 1 to the message length when allocating the message buffer.
  uint8_t *message = calloc(messageLength+1, sizeof(*message));
  result = fread(message, sizeof(*message), messageLength, stdin);
  // `inputLength` is a pointer, so we store the length at the memory address it 
  // points at. This way we return 2 values at once from a function!
  *inputLength = messageLength;
  return message;
}

void sendMessage(uint8_t *response /*, int32_t responseLength */) {
  const uint32_t responseLength = strlen(response);
  fwrite(&responseLength, sizeof responseLength, 1, stdout);
  fwrite(response, responseLength, 1, stdout);
  fflush(stdout);
}

int nmhost(void) {
  while (1) {
    // `messageLength`: Variable to store the message's length.
    // We pass the address of 'messageLength' to the function, and `message` 
    // stores the allocated buffer which we have to `free()`.
    size_t messageLength = 0;
    uint8_t *const message = getMessage(&messageLength);
    // Do things with `message`, but don't change the pointer value, for 
    // example:
    // message[5] = 'a'; // Ok, pointer still points to start of message.
    // message = NULL; // Bad! Now we can't `free()` the message.
    sendMessage(message /*, messageLength */);
    free(message);
  }
}

Here's how I try to test shermes compiled executable using deno to emulate a browser processing connecting and sending and receiving messages https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_standalone_test.js

"use strict";

// Emulate a module scope, since global scope is unsound.
(function (exports) {
  const _getchar = $SHBuiltin.extern_c(
    { include: "stdio.h" },
    function getchar(): c_int {
      throw 0;
    },
  );

  const _nmhost = $SHBuiltin.extern_c(
    { include: "/home/user/bin/nm_c.c" },
    function nmhost(): c_int {
      throw 0;
    },
  );

  _nmhost();
  
  // Optionally force some methods to be emitted for debugging.
  // exports.foo = foo;
})({});

deno -A nm_standalone_test.js ./a.out

Result

deno -A nm_standalone_test.js /home/user/bin/a.out
Testing /home/user/bin/a.out Native Messaging host

a.out: Unknown command line argument 'undefined'.  Try: '/home/user/bin/a.out -help'
error: Uncaught (in promise) BrokenPipe: Broken pipe (os error 32)
    at async Object.write (ext:deno_web/06_streams.js:1181:9)

Expected result using QuickJS

deno -A nm_standalone_test.js /home/user/native-messaging-quickjs/nm_qjs.js
Testing /home/user/native-messaging-quickjs/nm_qjs.js Native Messaging host

{ messageLength: 1048576 }
[
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  null, null, null, null, null, null, null, null, null, null,
  ... 209615 more items
]
{ messageLength: 6 }
test
{ messageLength: 2 }

{ messageLength: 1 }
1
{ messageLength: 8 }
{ "0": 97 }

I'm trying to ultimately compare Static Hermes to SpiderMonkey, V8 (d8), QuickJS, and further to other programming languages, e.g.,

0	'nm_rust'	0.12170000004768372
1	'nm_qjs'	0.126
2	'nm_c'	0.12630000001192093
3	'nm_cpp'	0.135
4	'nm_wasm'	0.18980000001192093
5	'nm_bash'	0.2947000000476837
6	'nm_deno'	0.32139999997615815
7	'nm_bun'	0.3305999999642372
8	'nm_typescript'	0.3315
9	'nm_spidermonkey'	0.3645
10	'nm_nodejs'	0.408
11	'nm_python'	0.4667999999523163
12	'nm_d8'	0.7491000000238418
13	'nm_tjs'	0.798
14	'nm_llrt'	0.8235999999642372

[tmikov]

I am sorry, I am not sure what you are asking. You are invoking the compiled binary with an invalid command line argument and it is terminating. It clearly says so:

a.out: Unknown command line argument 'undefined'.  Try: '/home/user/bin/a.out -help'

You already asked how to access command line arguments in SH compiled binaries and I responded: #1591

[guest271314]

Lemme revist that and see if I can make it do something,

[guest271314]

What I was trying to do is just include working C code into the script. So the idea was that since I'm importing C code that does what I want it to do, invoking that included function would achieve the same result in Shermes environment.

[guest271314]

Actually it's this #1469 I'm trying to expand on to read "32-bit message length in native byte order." from Chrome and Firefox and other browsers, and up to 1 MB message encoded as JSON-like data from host to client, up to 4 GB message length followed by message(s) from browser to host (Static Hermes). So I can compare the speed of Static Hermes to do this to other JavaScript engines, runtimes, and programming languages.

I was using getchar() there https://gist.github.com/guest271314/fec412134903e50521c12370edc88ef3.

[guest271314]

Alright, I somehow got it working by importing a slightly modified working C version of a Native Messaging host.

The C code in OP with this change after reading this https://stackoverflow.com/a/13817070

int nmhost(void) {
  while (1) {
    // `messageLength`: Variable to store the message's length.
    // We pass the address of 'messageLength' to the function, and `message` 
    // stores the allocated buffer which we have to `free()`.
    size_t messageLength = 0;
    uint8_t *const message = getMessage(&messageLength);
    // Do things with `message`, but don't change the pointer value, for 
    // example:
    // message[5] = 'a'; // Ok, pointer still points to start of message.
    // message = NULL; // Bad! Now we can't `free()` the message.
    sendMessage(message /*, messageLength */);
    free(message);
  }
}

int _start() {
  exit(nmhost());
}

JavaScript

const _nmhost = $SHBuiltin.extern_c(
    { include: "/home/user/nm-shermes/nm_c_nomain.c" },
    function nmhost(): c_int {
      throw 0;
    },
  );

  _nmhost();

Compile

shermes -typed -Wc,-nostartfiles ./fopen-dev-stdin.ts -o nm_shermes

Web extension manifest

{
  "name": "nm_shermes",
  "description": "Shermes Native Messaging host",
  "path": "/home/user/nm-shermes/nm_shermes",
  "type": "stdio",
  "allowed_origins": [
    "chrome-extension://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/"
  ]
}

Test

var runtimes = new Map([
  ["nm_bash", 0],
  ["nm_nodejs", 0],
  ["nm_deno", 0],
  ["nm_bun", 0],
  ["nm_tjs", 0],
  ["nm_qjs", 0],
  ["nm_spidermonkey", 0],
  ["nm_d8", 0],
  ["nm_typescript", 0],
  ["nm_llrt", 0],
  ["nm_rust", 0],
  ["nm_cpp", 0],
  ["nm_c", 0],
  ["nm_python", 0],
  ["nm_wasm", 0],
  ["nm_shermes", 0]
]);
for (const [runtime] of runtimes) {
  try {
    const {
      resolve,
      reject,
      promise
    } = Promise.withResolvers();
    const now = performance.now();
    const port = chrome.runtime.connectNative(runtime);
    port.onMessage.addListener((message) => {
      console.assert(message.length === 209715, {message, runtime});
      runtimes.set(runtime, (performance.now() - now) / 1000);
      port.disconnect();
      resolve();
    });
    port.onDisconnect.addListener(() => reject(chrome.runtime.lastError));
    port.postMessage(new Array(209715));
    if (runtime === "nm_spidermonkey") {
      port.postMessage("\r\n\r\n");
    }
    await promise;
  } catch (e) {
    console.log(e, runtime);
    continue;
  }
}
var sorted = [...runtimes].sort(([, a], [, b]) => a < b ? -1 : a === b ? 0 : 1);
console.table(sorted);

First run

0	'nm_shermes'	0.15810000002384186
1	'nm_qjs'	0.16909999990463256
2	'nm_c'	0.177
3	'nm_cpp'	0.20260000002384185
4	'nm_wasm'	0.208
5	'nm_rust'	0.21559999990463258
6	'nm_bash'	0.3835
7	'nm_typescript'	0.40589999997615817
8	'nm_bun'	0.4258000000715256
9	'nm_deno'	0.4443000000715256
10	'nm_spidermonkey'	0.5596999999284744
11	'nm_nodejs'	0.5797999999523162
12	'nm_llrt'	0.9931999999284744
13	'nm_d8'	1.0072999999523162
14	'nm_python'	2.609299999952316
15	'nm_tjs'	2.8229000000953675

Second run

0	'nm_c'	0.148
1	'nm_shermes'	0.1605
2	'nm_qjs'	0.16130000007152556
3	'nm_cpp'	0.20110000002384185
4	'nm_rust'	0.2125
5	'nm_wasm'	0.2645
6	'nm_bash'	0.33380000007152555
7	'nm_bun'	0.3465
8	'nm_deno'	0.40030000007152555
9	'nm_python'	0.4175
10	'nm_spidermonkey'	0.44139999997615814
11	'nm_typescript'	0.44660000002384187
12	'nm_nodejs'	0.5177999999523163
13	'nm_d8'	0.7926999999284744
14	'nm_tjs'	0.9755
15	'nm_llrt'	1.1092999999523163

Third run

0	'nm_shermes'	0.12979999995231628
1	'nm_qjs'	0.138
2	'nm_cpp'	0.13919999992847443
3	'nm_c'	0.15
4	'nm_rust'	0.15310000002384186
5	'nm_wasm'	0.21020000004768372
6	'nm_python'	0.24020000004768371
7	'nm_tjs'	0.24980000007152559
8	'nm_bash'	0.2691000000238419
9	'nm_deno'	0.37010000002384186
10	'nm_bun'	0.38589999997615815
11	'nm_nodejs'	0.39559999990463257
12	'nm_typescript'	0.4165999999046326
13	'nm_spidermonkey'	0.4695
14	'nm_d8'	0.8675
15	'nm_llrt'	1.0377999999523162

Standalone

globalThis.name = chrome.runtime.getManifest().short_name;

globalThis.port = chrome.runtime.connectNative(globalThis.name);
port.onMessage.addListener((message) => console.log(message));
port.onDisconnect.addListener((p) => console.log(chrome.runtime.lastError));
port.postMessage(new Array(209715));

chrome.runtime.onInstalled.addListener((reason) => {
  console.log(reason);
});
undefined
VM13:4 (209715) [null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, …]
globalThis.name

I guess I'm trying to figure out the importance of -nostartfile, and if main() can be used there. Ah, main() is expecting arguments, I guess?