npm --version segfaults on v7.7.3 linux ppc64 (BE) only ON RHEL 7

[mattcolegate]

• Version: 7.7.3
• Platform: linux ppc64 (BE)
• Subsystem: node

Running node --version works on this platform and version, but running npm --version causes a segmentation fault and core dump.

cc/ @gibfahn who helped with initial diagnosis

[gibfahn]

To add some more info, the version downloaded was https://nodejs.org/dist/latest-v7.x/node-v7.7.3-linux-ppc64.tar.gz.

Running node -e 'console.log("hi")' also segfaults.

[bnoordhuis]

Running node -e 'console.log("hi")' also segfaults.

Can you obtain a stack trace?

[sxa]

Ouch - this from a RHEL70 box (EDIT: Also occurs on 7.1)

#0  0x00003fffb7aaa4e0 in .__memset_power7 () from /lib64/power8/libc.so.6
#1  0x0000000010f69094 in ._ZN2v88internal18RegExpResultsCache5ClearEPNS0_10FixedArrayE ()
#2  0x0000000010cace24 in ._ZN2v88internal4Heap19MarkCompactPrologueEv ()
#3  0x0000000010cbf964 in ._ZN2v88internal4Heap11MarkCompactEv ()
#4  0x0000000010cca010 in ._ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE ()
#5  0x0000000010cca4f8 in ._ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorENS0_23GarbageCollectionReasonEPKcNS_15GCCallbackFlagsE ()
#6  0x0000000010ccce30 in ._ZN2v88internal4Heap12ReserveSpaceEPNS0_4ListINS1_5ChunkENS0_25FreeStoreAllocationPolicyEEEPNS2_IPhS4_EE ()
#7  0x000000001104a598 in ._ZN2v88internal12Deserializer11DeserializeEPNS0_7IsolateE ()
#8  0x0000000010dcea9c in ._ZN2v88internal7Isolate4InitEPNS0_12DeserializerE
    ()
#9  0x00000000110560b4 in ._ZN2v88internal8Snapshot10InitializeEPNS0_7IsolateE
    ()
#10 0x000000001075cca8 in ._ZN2v87Isolate3NewERKNS0_12CreateParamsE ()
#11 0x00000000111f6f04 in ._ZN4node5StartEP9uv_loop_siPKPKciS5_ ()
#12 0x00000000111f650c in ._ZN4node5StartEiPPc ()
#13 0x000000001052b720 in .main ()

[bnoordhuis]

@sxa555 Can you post the output of disassemble and info registers?

[sxa]

FYI I've built 7.7.3 from commit 9c68a69 locally and it doesn't fail in the same way. This is with this version of gcc (have the build boxes been upgraded recently to 4.9? I haven't used that yet)

gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)

[sxa]

@bnoordhuis (For reference this output was just running "node" without any parameters)

(gdb) bt
#0  0x00003fffb7aaa4e0 in .__memset_power7 () from /lib64/power8/libc.so.6
#1  0x0000000010f69094 in ._ZN2v88internal18RegExpResultsCache5ClearEPNS0_10FixedArrayE ()
#2  0x0000000010cace24 in ._ZN2v88internal4Heap19MarkCompactPrologueEv ()
#3  0x0000000010cbf964 in ._ZN2v88internal4Heap11MarkCompactEv ()
#4  0x0000000010cca010 in ._ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE ()
#5  0x0000000010cca4f8 in ._ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorENS0_23GarbageCollectionReasonEPKcNS_15GCCallbackFlagsE ()
#6  0x0000000010ccce30 in ._ZN2v88internal4Heap12ReserveSpaceEPNS0_4ListINS1_5ChunkENS0_25FreeStoreAllocationPolicyEEEPNS2_IPhS4_EE ()
#7  0x000000001104a598 in ._ZN2v88internal12Deserializer11DeserializeEPNS0_7IsolateE ()
#8  0x0000000010dcea9c in ._ZN2v88internal7Isolate4InitEPNS0_12DeserializerE ()
#9  0x00000000110560b4 in ._ZN2v88internal8Snapshot10InitializeEPNS0_7IsolateE ()
#10 0x000000001075cca8 in ._ZN2v87Isolate3NewERKNS0_12CreateParamsE ()
#11 0x00000000111f6f04 in ._ZN4node5StartEP9uv_loop_siPKPKciS5_ ()
#12 0x00000000111f650c in ._ZN4node5StartEiPPc ()
#13 0x000000001052b720 in .main ()
(gdb) info registers
r0             0x1	1
r1             0x3fffffffda00	70368744167936
r2             0x3fffb7be4410	70367531910160
r3             0xf	15
r4             0x0	0
r5             0x7ff	2047
r6             0x0	0
r7             0x30	48
r8             0x40	64
r9             0x400	1024
r10            0xf	15
r11            0x7	7
r12            0x800	2048
r13            0x3fffb7ffe190	70367536210320
r14            0x3fffffffe400	70368744170496
r15            0xffffffffba2e8ba3	18446744072538196899
r16            0x11fa4ab0	301615792
r17            0x7a940	502080
r18            0x7a940	502080
r19            0x0	0
r20            0x0	0
r21            0x1	1
r22            0x11f63a80	301349504
r23            0x2	2
r24            0x11f51e0e	301276686
r25            0x0	0
r26            0x0	0
r27            0x56f50	356176
r28            0x0	0
r29            0x11f63aa0	301349536
r30            0x11e1ce00	300011008
r31            0x3fffffffda00	70368744167936
pc             0x3fffb7aaa4e0	0x3fffb7aaa4e0 <.__memset_power7+64>
msr            0x800000010000d032	9223372041149796402
cr             0x44044841	1141131329
lr             0x10f69094	0x10f69094 <._ZN2v88internal18RegExpResultsCache5ClearEPNS0_10FixedArrayE+36>
ctr            0x3fffb7aaa4a0	70367530624160
xer            0x0	0
orig_r3        0xc00000000000908c	-4611686018427350900
trap           0x300	768
(gdb) disassemble
Dump of assembler code for function .__memset_power7:
   0x00003fffb7aaa4a0 <+0>:	cmpldi  cr7,r5,31
   0x00003fffb7aaa4a4 <+4>:	cmpldi  cr6,r5,8
   0x00003fffb7aaa4a8 <+8>:	mr      r10,r3
   0x00003fffb7aaa4ac <+12>:	rlwimi  r4,r4,8,16,23
   0x00003fffb7aaa4b0 <+16>:	rlwimi  r4,r4,16,0,15
   0x00003fffb7aaa4b4 <+20>:	ble     cr6,0x3fffb7aaa830 <.__memset_power7+912>
   0x00003fffb7aaa4b8 <+24>:	neg     r0,r3
   0x00003fffb7aaa4bc <+28>:	ble     cr7,0x3fffb7aaa7a0 <.__memset_power7+768>
   0x00003fffb7aaa4c0 <+32>:	andi.   r11,r10,7
   0x00003fffb7aaa4c4 <+36>:	rldimi  r4,r4,32,0
   0x00003fffb7aaa4c8 <+40>:	mr      r12,r5
   0x00003fffb7aaa4cc <+44>:	beq     0x3fffb7aaa500 <.__memset_power7+96>
   0x00003fffb7aaa4d0 <+48>:	clrldi  r0,r0,61
   0x00003fffb7aaa4d4 <+52>:	mtocrf  1,r0
   0x00003fffb7aaa4d8 <+56>:	subf    r5,r0,r5
   0x00003fffb7aaa4dc <+60>:	bns     cr7,0x3fffb7aaa4e8 <.__memset_power7+72>
=> 0x00003fffb7aaa4e0 <+64>:	stb     r4,0(r10)
   0x00003fffb7aaa4e4 <+68>:	addi    r10,r10,1
   0x00003fffb7aaa4e8 <+72>:	bne     cr7,0x3fffb7aaa4f4 <.__memset_power7+84>
   0x00003fffb7aaa4ec <+76>:	sth     r4,0(r10)
   0x00003fffb7aaa4f0 <+80>:	addi    r10,r10,2
   0x00003fffb7aaa4f4 <+84>:	ble     cr7,0x3fffb7aaa500 <.__memset_power7+96>
   0x00003fffb7aaa4f8 <+88>:	stw     r4,0(r10)
   0x00003fffb7aaa4fc <+92>:	addi    r10,r10,4
   0x00003fffb7aaa500 <+96>:	cmpldi  cr5,r5,255
   0x00003fffb7aaa504 <+100>:	li      r0,32
   0x00003fffb7aaa508 <+104>:	dcbtst  0,r10
   0x00003fffb7aaa50c <+108>:	cmpldi  cr6,r4,0
   0x00003fffb7aaa510 <+112>:	rldicl  r9,r5,61,3
   0x00003fffb7aaa514 <+116>:	crand   4*cr6+so,4*cr6+eq,4*cr5+gt
   0x00003fffb7aaa518 <+120>:	mtocrf  1,r9
   0x00003fffb7aaa51c <+124>:	bso     cr6,0x3fffb7aaa5e0 <.__memset_power7+320>
   0x00003fffb7aaa520 <+128>:	rldicl  r8,r5,59,5
   0x00003fffb7aaa524 <+132>:	clrldi  r11,r5,61
   0x00003fffb7aaa528 <+136>:	cmpldi  cr6,r11,0
   0x00003fffb7aaa52c <+140>:	cmpldi  cr1,r9,4
   0x00003fffb7aaa530 <+144>:	mtctr   r8
   0x00003fffb7aaa534 <+148>:	bne     cr7,0x3fffb7aaa560 <.__memset_power7+192>
   0x00003fffb7aaa538 <+152>:	std     r4,0(r10)
   0x00003fffb7aaa53c <+156>:	std     r4,8(r10)
   0x00003fffb7aaa540 <+160>:	addi    r10,r10,16
   0x00003fffb7aaa544 <+164>:	bns     cr7,0x3fffb7aaa570 <.__memset_power7+208>
   0x00003fffb7aaa548 <+168>:	std     r4,0(r10)
   0x00003fffb7aaa54c <+172>:	addi    r10,r10,8
   0x00003fffb7aaa550 <+176>:	mr      r12,r10
   0x00003fffb7aaa554 <+180>:	blt     cr1,0x3fffb7aaa5b0 <.__memset_power7+272>
   0x00003fffb7aaa558 <+184>:	b       0x3fffb7aaa570 <.__memset_power7+208>
   0x00003fffb7aaa55c <+188>:	ori     r2,r2,0
   0x00003fffb7aaa560 <+192>:	bns     cr7,0x3fffb7aaa570 <.__memset_power7+208>
   0x00003fffb7aaa564 <+196>:	std     r4,0(r10)
   0x00003fffb7aaa568 <+200>:	addi    r10,r10,8
   0x00003fffb7aaa56c <+204>:	ori     r2,r2,0
   0x00003fffb7aaa570 <+208>:	addi    r12,r10,32
   0x00003fffb7aaa574 <+212>:	std     r4,0(r10)
   0x00003fffb7aaa578 <+216>:	std     r4,8(r10)
   0x00003fffb7aaa57c <+220>:	std     r4,16(r10)
   0x00003fffb7aaa580 <+224>:	std     r4,24(r10)
   0x00003fffb7aaa584 <+228>:	bdz     0x3fffb7aaa5b0 <.__memset_power7+272>

[bnoordhuis]

Looks like an almost-nullptr bug. It tries to store a byte at the address in r10, which is 0xf:

0x00003fffb7aaa4e0 <+64>: stb r4,0(r10) # r10 == 0xf

r3 (first function argument) is moved into r10 a few lines up so it would seem Heap::MarkCompactPrologue() is calling RegExpResultsCache::Clear() with a bad FixedArray pointer. That's about all I can glean from it though, the root cause is probably elsewhere.

[mhdawson]

The compiler version on the test/build boxes is:

root@test-osuosl-ubuntu14-ppc64-be-3:~# gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[mhdawson]

The binaries seem to run ok on the machines on which they were built:

iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$ uname -a
Linux test-osuosl-ubuntu14-ppc64-be-3 4.2.0-27-powerpc64-smp #32~14.04.1-Ubuntu SMP Fri Jan 22 15:47:25 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux

[mhdawson]

iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$ node -e 'console.log("hi")';
hi
iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$ node --version
v7.7.3-nightly20170309c62798034a
iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$ node -e 'console.log("hi")';
hi
iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$

[mhdawson]

output just before crash with LD_DEBUG=all

    23377:     symbol=munmap;  lookup in file=/lib64/power8/libc.so.6 [0]
     23377:     binding file ./node [0] to /lib64/power8/libc.so.6 [0]: normal symbol `munmap' [GLIBC_2.3]
     23377:     symbol=mprotect;  lookup in file=./node [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/libdl.so.2 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/power8/librt.so.1 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/libstdc++.so.6 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/power8/libm.so.6 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/libgcc_s.so.1 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/power8/libpthread.so.0 [0]
     23377:     symbol=mprotect;  lookup in file=/lib64/power8/libc.so.6 [0]
     23377:     binding file ./node [0] to /lib64/power8/libc.so.6 [0]: normal symbol `mprotect' [GLIBC_2.3]
Segmentation fault (core dumped)

[mhdawson]

On RHEL machine:

-sh-4.2$ /lib64/power8/libc.so.6
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.3 20140911 (Red Hat 4.8.3-7).
Compiled on a Linux 3.10.0 system on 2015-01-19.
Available extensions:
        The C stubs add-on version 2.1.2.
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
        RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

On community machine

iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/mtest/node-v7.7.3-nightly20170309c62798034a-linux-ppc64/bin$ /lib64/libc.so.6
GNU C Library (Ubuntu EGLIBC 2.19-0ubuntu6.9) stable release version 2.19, by Roland McGrath et al.
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.4.
Compiled on a Linux 3.13.11 system on 2016-05-26.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/eglibc/+bugs>.