segfault on node v6.3.1 on Ubuntu 14.04.4

[msakrejda]

• Version: v6.3.1
• Platform: Ubuntu 14.04.4 LTS (Heroku)
• Subsystem: ?

Hello,

I seem to be able to fairly reliably (maybe 50% of the time) reproduce a segfault with node v6.3.1 in my express application. I'm not using any native modules (except for segfault-handler to get the trace output below, and the error of course also occurs without segfault-handler).

This is the output I get:

PID 17 received SIGSEGV for address: 0x753760f
/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1b04)[0x7f1fd8283b04]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f1fdce8b330]
node(_ZN4node7TLSWrap7IsAliveEv+0x16)[0x1026586]
node(_ZN4node10StreamBase5GetFDINS_7TLSWrapEEEvN2v85LocalINS3_6StringEEERKNS3_20PropertyCallbackInfoINS3_5ValueEEE+0x92)[0x102b192]
node(_ZN2v88internal25PropertyCallbackArguments4CallEPFvNS_5LocalINS_4NameEEERKNS_20PropertyCallbackInfoINS_5ValueEEEES4_+0xd2)[0x981c12]
node(_ZN2v88internal6Object23GetPropertyWithAccessorEPNS0_14LookupIteratorE+0x1bd)[0xd16c1d]
node(_ZN2v88internal6Object11GetPropertyEPNS0_14LookupIteratorE+0x13b)[0xd51c2b]
node(_ZN2v88internal6LoadIC4LoadENS0_6HandleINS0_6ObjectEEENS2_INS0_4NameEEE+0x145)[0xcbc1c5]
node[0xcbcfdc]
node(_ZN2v88internal24Runtime_KeyedLoadIC_MissEiPPNS0_6ObjectEPNS0_7IsolateE+0x112)[0xcbfef2]
[0x1367f3c0961b]

The error seems to happen right after I send a message to a client via socket.io, but the message itself seems to send correctly (that is, I see a log line after the .emit).

I looked at the other open issues mentioning segfaults, but I did not see anything relevant.

Any ideas?

[mscdex]

Do you have a minimal example that reproduces it?

[msakrejda]

No, unfortunately I have not been able to minimize this.

I noticed the TLS tag: the application does use TLS with Postgres and for external http requests via axios. For incoming requests, though, TLS termination happens outside of the node process.

[mscdex]

/cc @indutny

[addaleax]

Yeah, the segfault occurs in the second part of this line, meaning that stream_ is invalid for some reason.

[mscdex]

@uhoh-itsmaciek Have you tried with different node versions (e.g. v6.0.0, v5.x, v4.x) to find when this problem started?

[msakrejda]

I'll try these out and report back.

[msakrejda]

It appears that on 5.12, we don't get the axios 401s (from another service we run) that trigger this problem in the first place. I had not yet investigated why these 401s were happening and thought that they were legitimate, but maybe we're somehow corrupting the credentials we send with these requests in node 6?

[msakrejda]

Ok, on 5.12 if I force an error response from axios by intentionally sending the wrong credentials, I still get a segfault with a very similar trace, although it seems to consistently be for address 0x0 (which was not the case with 6):

PID 18 received SIGSEGV for address: 0x0
/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1b04)[0x7f659c349b04]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f65a1f52330]
node(_ZN4node7TLSWrap7IsAliveEv+0x16)[0xe800f6]
node(_ZN4node10StreamBase5GetFDINS_7TLSWrapEEEvN2v85LocalINS3_6StringEEERKNS3_20PropertyCallbackInfoINS3_5ValueEEE+0x4c)[0xe84a6c]
node(_ZN2v88internal25PropertyCallbackArguments4CallEPFvNS_5LocalINS_4NameEEERKNS_20PropertyCallbackInfoINS_5ValueEEEES4_+0x80)[0x9153b0]
node(_ZN2v88internal6Object23GetPropertyWithAccessorEPNS0_14LookupIteratorENS0_12LanguageModeE+0x1cd)[0xc04f0d]
node(_ZN2v88internal6Object11GetPropertyEPNS0_14LookupIteratorENS0_12LanguageModeE+0x9e)[0xc0d68e]
node(_ZN2v88internal6LoadIC4LoadENS0_6HandleINS0_6ObjectEEENS2_INS0_4NameEEE+0x17d)[0xbba51d]
node[0xbbb0a8]
node(_ZN2v88internal24Runtime_KeyedLoadIC_MissEiPPNS0_6ObjectEPNS0_7IsolateE+0xd5)[0xbbbcc5]
[0x1908e700839b]

[msakrejda]

On 4.4.7, the asset precompilation fails for my app, so I can't easily check that. On 6.0.0, I get the segfault.

[msakrejda]

A colleague suggested that this seems similar to an issue fixed recently in Postgres--we looked at the node code you linked briefly @addaleax and it's not obviously the same issue, but perhaps it's related? In both cases, two independent clients are managing their own TLS connections in the same process, and at least in the Postgres case, the OpenSSL API made it difficult to do this safely.

[addaleax]

@uhoh-itsmaciek Yeah, I mostly linked that code to explain where the label comes from – unfortunately, I can’t tell from the stack trace itself or the code that it points to why the stream_ field contains invalid data.

Something you can try (if that’s possible) is running your app with valgrind. It’s going to be terribly slow but it’s the best idea I have for now.

[msakrejda]

I'll see if I can set that up--it might be tricky to do on Heroku (and I have not been able to reproduce locally).

In the meantime, I noticed that the endpoint I call with axios does not actually respond with 401s: according to the heroku router logs, my endpoint responds with 301s, but somehow axios sees a 401 response. I'm guessing this is related to the memory corruption, but I suppose it doesn't really tell us much.

[indutny]

@uhoh-itsmaciek may I ask you to update a core dump of this crash? It can be inspected with llnode later on, and we should be able to find much more info from there!