/usr/bin/node: '/usr/bin/node' is not an ELF executable for ARM

[lal12]

I cross compiled NodeJS for ARM Linux, and it seems to work flawlessly. Except it prints the error message /usr/bin/node: '/usr/bin/node' is not an ELF executable for ARM. It only prints it when actually executing something, e.g. node --version works fine, without an error message.

I ran strace node dummy-script.js and could see that NodeJS does some kind of fork then opens itself (/usr/bin/node) and afterwards is printing the mentioned error, but afterwards it keeps running and seems to work flawlessly. I guess node tries to do some kind of self exec?

Any ideas on that? Or maybe a hint where the code with the described behaviour can be found, yet I could not find it myself.

[sam-github]

/usr/bin/node and node --version differ in two ways... the arg, but also one has an exact path. Are you sure you are running the same executable both times? I would suggest supplying an absolute path every time. Also type node might be informative. And node doesn't fork itself, that makes it seem like you have some inadvertent wrapper.

[lal12]

No it is definetly the same executable, I double checked that, and it is also not a wrapper. It is the only thing I copied from the build (a 20Mb binary). So I am also pretty sure that the process is cloned by node or one of its libraries. I appended the strace output. In line 192, 216, 228, 238, 247 are calls to clone. In line 334 there is the call open("/usr/bin/node") and some lines below (339+) the error message is printed.

Btw. the called script just contains process.exit(0). But the behaviour is pretty much the same (as described above), when I just call node, just that the Javascript CLI starts.

strace-node.log

[lal12]

I also called strace -f node --version, in the appended log you can see that there aren't any clones or forks.
strace-node-version.log

[bnoordhuis]

I think you've stumbled upon a bug in your copy of uclibc or the toolchain you used to cross-compile node.

uclibc tries to open the binary as a shared object (legal) but then apparently fails to parse the header. That "not an ELF executable" error message comes from uclibc.

Looking at itsldso/ldso/dl-elf.c, it seems uclibc must be built with LDSO_STANDALONE_SUPPORT enabled for exe-as-so to work. I assume it is.

It also expects e_machine to be 40 (for ARM) but from the strace log, it looks like it's either 1 or 257 (I don't know if it's a LE or BE binary ^^):

[pid  2112] read(17, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\2\0(\0\1\0\0\0\00069\0004\0\0\0"..., 4096) = 4096

I'll go ahead and close this out because it's clearly not a Node.js bug. readelf(1) might shed some more light. I'd be interested to know what the root cause was when you manage to track it down!

[lal12]

It is Little Endian (as you can btw tell from the 5th byte).
The 40 for ARM is actually the (\0 = x28 00 = 40, the display format is not really suitable here ^^.

I already assumed that it might not be a NodeJS bug, but thought that node (indirectly) would do the clone and the open, since this does not happen when being called with --version, and I don't expect the loader to behave differently depending on the program arguments...

But I will look further into it, and tell you when I find something.

[lal12]

First of all you were right about the uclibc part: LDSO_STANDALONE_SUPPORT is in fact not enabled during build, therefore the ld-uClibc does not accept to load an executable elf, which leads to the error. This is either a bug or at least a confusing name, since I would expect it to only disable using the ld from the command line, but not disabling dlopen of executables.

So as I expected it is a dlopen call from within node which triggers this behaviour. Didn't get a working nodejs debug build yet, but gdb claims it comes from jitted code. So it seems that node (or I guess maybe v8) is generating code which tries to load itself as a library. I guess the return value of dlopen is not checked and therefore node continues to run, however since the binary (itself) is already loaded, all symbols are there and no problems occur later on. Maybe this is done by nodes internal js, like a require('node') or something.

This is the part I looked for all along and hoped for somebody here to know: which part of node is actually doing the dlopen (or probably generating the jitted code which does the dlopen) and also maybe why it does that. Also loading itself as library seems a bit odd to me, while it probably is not wrong to do so. I compared it with node on my Debian PC, which tries to dlopen 'libnode.so' but also fails.

[bnoordhuis]

There are a couple of places where node and its dependencies call:

addr = dlsym(RTLD_DEFAULT, "some_libc_function_that_may_be_missing");

I speculate uclibc substitutes RTLD_DEFAULT with a dlopen() call (or uclib's internal equivalent) of the main binary.

That gdb points to jitted code probably means that the JS code calls out to the C++ runtime, which calls out to dlsym().

gdb tends to get confused by the non-standard calling convention used by V8's generated code. You may be able to obtain a more legible backtrace when you put a breakpoint on dlsym.

[lal12]

@bnoordhuis thanks, I looked into this and it seems that openssl is actually doing a dlopen. It seems that it tries to reload itself to:

Deliberately leak a reference to ourselves. This will force the library to remain loaded until the atexit() handler is run at process exit.

(as a comment states), which I don't fully understand yet. It uses dladdr to find out from where a openssl global variable has been loaded, since it is statically linked it gets the own binary path as result. Afterwards it will do an dlopen.

I am currently investigating the following: Openssl calls their lib loading mechanism DSO which is used as wrapper for dladdr and dload, I suspect that maybe either a wrong DSO implementation is chosen at compile time or openssl should not be compiled with DSO at all, since it is statically linked.

FYI: I also assumed you can open a binary as a shared object, however this only is true for special -fPIC -pie compiled binaries, however those also will have the ET_DYN and not the ET_EXEC type even though they are executables.

[richardlau]

@bnoordhuis thanks, I looked into this and it seems that openssl is actually doing a dlopen. It seems that it tries to reload itself to Deliberately leak a reference to ourselves. This will force the library to remain loaded until the atexit() handler is run at process exit. (as a comment states), which I don't fully understand yet. It uses dladdr to find out from where a openssl global variable has been loaded, since it is statically linked it gets the own binary path as result. Afterwards it will do an dlopen.

I am currently investigating the following: Openssl calls their lib loading mechanism DSO which is used for as wrapper for dladdr and dload, I suspect that maybe either a wrong DSO implementation is chosen at compile time or openssl should not be compiled with DSO at all, since it is statically linked.

FYI: I also assumed you can open a binary as a shared object, however this only is true for special -fPIC -pie compiled binaries, however those also will have the ET_DYN and not the ET_EXEC type even though they are executables.

Some investigation into that was done in #21845 with an upstream PR to OpenSSL: openssl/openssl#6725

[lal12]

@richardlau that indeed seems to be my problem thanks!