N-API: An api for embedding Node in applications

[empyrical]

Is your feature request related to a problem? Please describe.
Right now there isn't a documented/stable way to use Node as a shared library inside of an application. Were one to be made using N-API, this would open up using Chakra in addition to V8 in an application.

Describe the solution you'd like
I would like for there to be stable APIs in node_api.h for creating/managing a Node environment.

A function that does this could hypothetically look like:

NAPI_EXTERN napi_status napi_create_env(int* argc, const char** argv, napi_env* env);
// Start the node event loop
NAPI_EXTERN napi_status napi_run_env(napi_env env);
// Cleanup (e.g. FreeIsolateData, FreeEnvironment and whatever else needs to be ran on teardown)
NAPI_EXTERN napi_status napi_free_env(napi_env env);

The embedder could get this environment's libuv loop using napi_get_uv_event_loop. But I would also like to have open the possibility of providing my own libuv loop that I have control over to help integrate with other event loops (e.g. Qt's event loop). This could look like:

NAPI_EXTERN napi_status napi_create_env_from_loop(int* argc, const char** argv,
  napi_env* env, struct uv_loop_s* loop);

Keeping the event loop going (using uv_run on the env's loop) would then be the embedder's responsibility.

Also, right now methods like node::CreateEnvironment seem to always jump into a REPL, unless you provide a string to evaluate or a file to run. Tweaks to help make this nicer to use for embedding will have to be made.

These APIs are just hypothetical, and will probably change when an actual attempt to implement them is made.

I am up to trying to implement this, but I would like to see what kind of discussion happens first and what other ideas people have before I start.

Implementation Progress

• Create a clean non-NAPI way to use Node embedded
• Create NAPI functions for creating and managing environments
• Create a NAPI function for evaluating a string (exists in NAPI v1: napi_run_script)
• Create a NAPI function for running a script from file
• Investigate if this can play nicely with worker_threads.

Describe alternatives you've considered
I've tried using the unstable APIs, and they aren't fun to keep up with 😅

For discussions on how the shared library can be distributed, see this issue: #24028

[addaleax]

I've tried using the unstable APIs, and they aren't fun to keep up with 😅

A big part of that is that they haven’t ever been designed as a coherent API (or designed at all, really), and we would likely need to iterate on them a bit more before they are stable – which is probably also the point where we can start to talk about enabling N-API versions of them.

If you want to work on this, good starting points might be #21653 (comment), or splitting CreateEnvironment() into a function that, well creates the Environment, and one that calls Environment::Start() under the hood?

[empyrical]

Thanks for helping point where to start! I also noticed some relevant TODOs in 'node_worker' that would get resolved by more stable apis for this.

[mhdawson]

@rubys another person we should loop into discussions/team about use cases/testing/api for using Node.js as a shared library.

[rubys]

First observation: we should plan to move to having --shared as the default for both CI and releases. This would make releases include a shared library that could be used by third parties. Unscientific comparison of results on Mac OS/X, the combine executable + dynamic library would be a total of 0.1% bigger than a standalone executable.

Second, I would suggest that one of the goals be to allow electron to be built using exclusively NAPI interfaces. See electron/atom/app/node_main.cc.

This means that in addition to Create and Destroy environments, there would need to be an interface to execute a script in an environment, and to evaluate an expression in that environment.

[empyrical]

Like - making the node command basically just be node_main.cc that links against libnode? Would be very nice! And would be nice to include CMake, pkgconfig modules for finding libnode that would ship with it while we're at it too.

[rubys]

@empyrical today if you do the following on Mac or Linux:

./configure --shared
make -j4

You end up with out/Release/node and out/Release/libnode.67.dynlib or out/Release/lib.target/libnode.so.67. Adding additional NAPI apis would be straightforward; I'm merely stating that it should be goal to add enough APIs to make electron's node_main.cc not need to depend on any other APIs.

But again, we would either need to include these libraries in the existing releases or have separate releases.

[empyrical]

Oh - I misunderstood. I thought you meant only building --shared version of Node, and making the node executable you use from the cli just very small executable that links against libnode

[rubys]

@empyrical that's actually what --shared does. Here are the sizes of the output files on Mac OS/X:

$ ls -l out/Release/node out/Release/libnode.67.dylib 
-rwxr-xr-x  1 rubys  staff  40410544 Sep 29 16:33 out/Release/libnode.67.dylib
-rwxr-xr-x  1 rubys  staff      9208 Sep 29 16:33 out/Release/node

[addaleax]

Just two quick things to note:

• I don’t know if that’s implied here, but I don’t think we can get away with a default where people have only a libnode + wrapper available as part of the release tarballs
• Using --shared is definitely something that embedders will tend to do more often than others, but it’s orthogonal to the Embedder API by itself

[empyrical]

Curious for some thoughts with regards to worker_threads: If you create multiple envs, should they all be "main threads" with a threadid of 0 and workers for envs would be created with a separate hypothetical API, or should the first one created be the "main thread", and subsequent ones be considered "workers" with incrementing threadids?

And should the "main thread" only be allowed to be made in the process' main thread? JS code that checks worker_threads.isMainThread to see if it's safe to do something, e.g. call functions in a GUI binding (which typically only work in the main thread) may have issues if a "main" js thread isn't truly in the process' main thread.

Maybe there should be a NAPI function for creating a "main" env, and then a different one for subsequent ones?

Basically:

// Any more than one invocation per process would result in an error napi_status
NAPI_EXTERN napi_status napi_create_main_env(int* argc, const char** argv, napi_env* env);

// Parent env should also show up as parentPort on worker_threads
NAPI_EXTERN napi_status napi_create_env(napi_env parent_env, napi_env* env);

[rubys]

I don’t think we can get away with a default where people have only a libnode + wrapper available as part of the release tarballs

Why not?