FYI: powerpc64 headbuilt via devel/powerpc64-xtoolchain-gcc and C++ exceptions for user code built by system-clang or devel/powerpc64-gcc (as of head -r339076 and ports -r480180)
(too old to reply)
Mark Millard via freebsd-hackers
2018-10-16 04:00:17 UTC
[I've found the problem at the low level for
my context of using WITH_LIBCPLUSPLUS= with
the likes of devel/powerpc64-gcc but I do
not have a solution for WITH_LIBCPLUSPLUS=
so far. I give details of what I found.]
I built a powerpc64 head -r339076 via devel/powerpc64-gcc
and the like that built clang as cc as well (and
WITHOUT_LIB32). This included use of base/binutils to
the the powerpc64 set up. The system and kernel are
non-debug builds (but with symbols). [system-clang is not
used for buildworld buildkernel because of known
issues (last I tried).]
booting, buildworld, buildkernel, poudriere building
what totaled to be somewhat under 400 ports all seem
to work. But . . .
It been a long time since I've done something analogous
and a significant item in the result is different than in
the past once I started testing the throwing of C++
exceptions in code produced by system-clang or by
Such code ends up stuck using around 100% of a CPU.
# more exception_test.cpp
#include <exception>
int main(void)
try { throw std::exception(); }
catch (std::exception& e) {}
return 0;
# ldd a.out
libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
libm.so.5 => /lib/libm.so.5 (0x8101ab000)
libc.so.7 => /lib/libc.so.7 (0x8101eb000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x810554000)
That program goes into an possibly unbounded execution.
(Historically when this program had problems it would
stop and produce a core file.)
When compiled by devel/powerpc64-gcc the a.out that results
does the same thing. ( /usr/local/bin/powerpc64-unknown-freebsd12.0-c++
as the compiler path ) So this is not really clang specific
# ldd a.out
libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
libm.so.5 => /lib/libm.so.5 (0x8101ab000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8101eb000)
libc.so.7 => /lib/libc.so.7 (0x810211000)
(That should not have involved clang or llvm at all.)
But compiled by lang/gcc8's g++8 the a.out that results works
# ldd a.out
libstdc++.so.6 => /usr/local/lib/gcc8/libstdc++.so.6 (0x81006e000)
libm.so.5 => /lib/libm.so.5 (0x8102c7000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x810307000)
libc.so.7 => /lib/libc.so.7 (0x81032d000)
It is not clear if using base/gcc as system cc
would do any better than using system-clang does
or than devel/powerpc64-gcc does: it is sort of
a variant of devel/powerpc64-gcc .
It will probably be some time before I figure out
much about what is going on.
libc++.so.1 => /usr/lib/libc++.so.1 (0x81006d000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x810184000)
lang/gcc8 avoids those being involved.
. . .
WITHOUT_LIB32= is because, for every post-gcc 4.2.1
that I've tried, the lib32 produced misuses R30 in
crtbeginS code (vs. the ABI for FreeBSD) and 32-bit
code just produces core files from the bad so-called
address dereference that results.
I'd rather have throwing C++ exceptions working and
lack of lib32 than have lib32 but not have throwing
C++ exceptions working. But at the moment how to have
such is not obvious when fairly modern compilers
and toolchains are involved.
Here is what I've found so far.
The code is looping in the following routine.
(I've inserted 2 NOTE: lines for what the
sustained looping is like.)
. . . (the routine was _Unwind_RaiseException) . . .

So far I've found that the following in
_Unwind_RaiseException stays invariant once
initialized --despite the uw_frame_state_for
and uw_update_context calls in
_Unwind_RaiseException 's loop that normally
update fs:

(gdb) print fs
$15 = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 31 times>, {loc = {reg = 18446744073709551608, offset = -8,
exp = 0xfffffffffffffff8 <Error reading address 0xfffffffffffffff8: Bad address>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 33 times>, {
loc = {reg = 16, offset = 16, exp = 0x10 <Error reading address 0x10: Bad address>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 80 times>},
prev = 0x0}, cfa_offset = 0, cfa_reg = 1, cfa_exp = 0x0, cfa_how = CFA_REG_OFFSET, pc = 0x8101999f8, personality = 0, data_align = -8, code_align = 4, retaddr_column = 65,
fde_encoding = 27 '\033', lsda_encoding = 255 '?', saw_z = 1 '\001', signal_frame = 0 '\0', eh_ptr = 0x0}

It turns out that pc value 0x8101999f8 is a little after the libcxxrt.so
call to _Unwind_RaiseException that is in throw_exception. But
_Unwind_RaiseException returning would be a failure and would end up
in a non-returning, error-reporting code path.

In other words: this is not an appropriate context for following the
return path to unwind out of _Unwind_RaiseException and its internal
caller (throw_exception).

It got to that address from lr containing the address of the
instruction after the one that does bl to the
_Unwind_RaiseException routine.

Overall it needs to unwind past this in the normal case but is stuck
handling the error/no-return case as "the" case.

Supporting details follow.

What lead up to 0x8101999f8 for initialization was
the lr value related to calling _Unwind_RaiseException
(see the address in lr below, also copied to r5):

. . .
(gdb) c

Breakpoint 9, <signal handler called>
1: x/i $pc 0x8101f35d8 <_Unwind_RaiseException+216>: bl 0x8101f2bc0 <uw_init_context_1>
Current language: auto; currently minimal

where the register values being supplied are (see
r5 and lr):

(gdb) info reg
r0 0x8101999f0 34629851632
r1 0x3fffffffffffc320 4611686018427372320
r2 0x810217900 34630367488
r3 0x3fffffffffffd280 4611686018427376256
r4 0x3fffffffffffd930 4611686018427377968
r5 0x8101999f0 34629851632
r6 0xa0 160
r7 0x0 0
r8 0x1 1
r9 0x8101aac10 34629921808
r10 0x1 1
r11 0x28 40
r12 0x28000282 671089282
r13 0x81005d020 34628554784
r14 0x0 0
r15 0x0 0
r16 0x0 0
r17 0x0 0
r18 0x0 0
r19 0x0 0
r20 0x0 0
r21 0x0 0
r22 0x0 0
r23 0x0 0
r24 0x0 0
r25 0x0 0
r26 0x0 0
r27 0x3fffffffffffd280 4611686018427376256
r28 0x810041060 34628440160
r29 0x3fffffffffffc390 4611686018427372432
r30 0x3fffffffffffcd10 4611686018427374864
r31 0x810041008 34628440072
pc 0x8101f35d8 34630219224
ps 0x0 0
cr 0x0 0
lr 0x8101999f0 34629851632
ctr 0x8101f3500 34630219008
xer 0x0 0
fpscr 0xfff80000 -524288
vscr 0x0 0
vrsave 0x0 0

The pc listed in print fs (0x8101999f8) is in the
following from libcxxrt, as is the value in r5 and
lr (0x8101999f0): (some blank lines inserted to
help identify the area and some related material)

(gdb) disass throw_exception
Dump of assembler code for function throw_exception:
0x0000000810199960 <throw_exception+0>: mflr r0
0x0000000810199964 <throw_exception+4>: std r31,-8(r1)
0x0000000810199968 <throw_exception+8>: mr r31,r3
0x000000081019996c <throw_exception+12>: std r0,16(r1)
0x0000000810199970 <throw_exception+16>: stdu r1,-128(r1)
0x0000000810199974 <throw_exception+20>: bl 0x810197ab0 <thread_info>
0x0000000810199978 <throw_exception+24>: ld r10,8(r3)
0x000000081019997c <throw_exception+28>: mr r9,r3
0x0000000810199980 <throw_exception+32>: cmpdi cr7,r10,0
0x0000000810199984 <throw_exception+36>: std r10,24(r31)
0x0000000810199988 <throw_exception+40>: beq- cr7,0x810199a10 <throw_exception+176>
0x000000081019998c <throw_exception+44>: ld r10,0(r9)
0x0000000810199990 <throw_exception+48>: cmpdi cr7,r10,0
0x0000000810199994 <throw_exception+52>: std r10,32(r31)
0x0000000810199998 <throw_exception+56>: beq- cr7,0x8101999d0 <throw_exception+112>
0x000000081019999c <throw_exception+60>: lwz r10,48(r9)
0x00000008101999a0 <throw_exception+64>: addi r3,r31,88
0x00000008101999a4 <throw_exception+68>: addi r10,r10,1
0x00000008101999a8 <throw_exception+72>: stw r10,48(r9)
0x00000008101999ac <throw_exception+76>: bl 0x81018e500 <00000018.plt_call._Unwind_RaiseException@@GCC_3.0>
0x00000008101999b0 <throw_exception+80>: ld r2,40(r1)
0x00000008101999b4 <throw_exception+84>: addi r1,r1,128
0x00000008101999b8 <throw_exception+88>: mr r4,r31
0x00000008101999bc <throw_exception+92>: ld r0,16(r1)
0x00000008101999c0 <throw_exception+96>: ld r31,-8(r1)
0x00000008101999c4 <throw_exception+100>: mtlr r0
0x00000008101999c8 <throw_exception+104>: b 0x8101996b0 <report_failure>
0x00000008101999cc <throw_exception+108>: nop
0x00000008101999d0 <throw_exception+112>: nop
0x00000008101999d4 <throw_exception+116>: addi r3,r31,88
0x00000008101999d8 <throw_exception+120>: ld r10,-30008(r2)
0x00000008101999dc <throw_exception+124>: std r10,32(r31)
0x00000008101999e0 <throw_exception+128>: lwz r10,48(r9)
0x00000008101999e4 <throw_exception+132>: addi r10,r10,1
0x00000008101999e8 <throw_exception+136>: stw r10,48(r9)

0x00000008101999ec <throw_exception+140>: bl 0x81018e500 <00000018.plt_call._Unwind_RaiseException@@GCC_3.0>

0x00000008101999f0 <throw_exception+144>: ld r2,40(r1)
0x00000008101999f4 <throw_exception+148>: addi r1,r1,128

0x00000008101999f8 <throw_exception+152>: mr r4,r31

0x00000008101999fc <throw_exception+156>: ld r0,16(r1)
0x0000000810199a00 <throw_exception+160>: ld r31,-8(r1)
0x0000000810199a04 <throw_exception+164>: mtlr r0
0x0000000810199a08 <throw_exception+168>: b 0x8101996b0 <report_failure>

0x0000000810199a0c <throw_exception+172>: nop
0x0000000810199a10 <throw_exception+176>: nop
0x0000000810199a14 <throw_exception+180>: ld r10,-30000(r2)
0x0000000810199a18 <throw_exception+184>: std r10,24(r31)
0x0000000810199a1c <throw_exception+188>: b 0x81019998c <throw_exception+44>
0x0000000810199a20 <throw_exception+192>: .long 0x0
0x0000000810199a24 <throw_exception+196>: .long 0x90001
0x0000000810199a28 <throw_exception+200>: lwz r0,0(r1)
End of assembler dump.

For: 0x00000008101999f8 (above)
objdump shows: 00000000000159f8 (below):

0000000000015960 <.__cxa_end_catch+0x460> mflr r0
0000000000015964 <.__cxa_end_catch+0x464> std r31,-8(r1)
0000000000015968 <.__cxa_end_catch+0x468> mr r31,r3
000000000001596c <.__cxa_end_catch+0x46c> std r0,16(r1)
0000000000015970 <.__cxa_end_catch+0x470> stdu r1,-128(r1)
0000000000015974 <.__cxa_end_catch+0x474> bl 0000000000013ab0 <._ZdaPv+0x590>
0000000000015978 <.__cxa_end_catch+0x478> ld r10,8(r3)
000000000001597c <.__cxa_end_catch+0x47c> mr r9,r3
0000000000015980 <.__cxa_end_catch+0x480> cmpdi cr7,r10,0
0000000000015984 <.__cxa_end_catch+0x484> std r10,24(r31)
0000000000015988 <.__cxa_end_catch+0x488> beq cr7,0000000000015a10 <.__cxa_end_catch+0x510>
000000000001598c <.__cxa_end_catch+0x48c> ld r10,0(r9)
0000000000015990 <.__cxa_end_catch+0x490> cmpdi cr7,r10,0
0000000000015994 <.__cxa_end_catch+0x494> std r10,32(r31)
0000000000015998 <.__cxa_end_catch+0x498> beq cr7,00000000000159d0 <.__cxa_end_catch+0x4d0>
000000000001599c <.__cxa_end_catch+0x49c> lwz r10,48(r9)
00000000000159a0 <.__cxa_end_catch+0x4a0> addi r3,r31,88
00000000000159a4 <.__cxa_end_catch+0x4a4> addi r10,r10,1
00000000000159a8 <.__cxa_end_catch+0x4a8> stw r10,48(r9)
00000000000159ac <.__cxa_end_catch+0x4ac> bl 000000000000a500 <CXXABI_1.3@@CXXABI_1.3+0xa500>
00000000000159b0 <.__cxa_end_catch+0x4b0> ld r2,40(r1)
00000000000159b4 <.__cxa_end_catch+0x4b4> addi r1,r1,128
00000000000159b8 <.__cxa_end_catch+0x4b8> mr r4,r31
00000000000159bc <.__cxa_end_catch+0x4bc> ld r0,16(r1)
00000000000159c0 <.__cxa_end_catch+0x4c0> ld r31,-8(r1)
00000000000159c4 <.__cxa_end_catch+0x4c4> mtlr r0
00000000000159c8 <.__cxa_end_catch+0x4c8> b 00000000000156b0 <.__cxa_end_catch+0x1b0>
00000000000159cc <.__cxa_end_catch+0x4cc> nop
00000000000159d0 <.__cxa_end_catch+0x4d0> nop
00000000000159d4 <.__cxa_end_catch+0x4d4> addi r3,r31,88
00000000000159d8 <.__cxa_end_catch+0x4d8> ld r10,-30008(r2)
00000000000159dc <.__cxa_end_catch+0x4dc> std r10,32(r31)
00000000000159e0 <.__cxa_end_catch+0x4e0> lwz r10,48(r9)
00000000000159e4 <.__cxa_end_catch+0x4e4> addi r10,r10,1
00000000000159e8 <.__cxa_end_catch+0x4e8> stw r10,48(r9)

00000000000159ec <.__cxa_end_catch+0x4ec> bl 000000000000a500 <CXXABI_1.3@@CXXABI_1.3+0xa500>

00000000000159f0 <.__cxa_end_catch+0x4f0> ld r2,40(r1)
00000000000159f4 <.__cxa_end_catch+0x4f4> addi r1,r1,128

00000000000159f8 <.__cxa_end_catch+0x4f8> mr r4,r31

00000000000159fc <.__cxa_end_catch+0x4fc> ld r0,16(r1)
0000000000015a00 <.__cxa_end_catch+0x500> ld r31,-8(r1)
0000000000015a04 <.__cxa_end_catch+0x504> mtlr r0
0000000000015a08 <.__cxa_end_catch+0x508> b 00000000000156b0 <.__cxa_end_catch+0x1b0>

0000000000015a0c <.__cxa_end_catch+0x50c> nop
0000000000015a10 <.__cxa_end_catch+0x510> nop
0000000000015a14 <.__cxa_end_catch+0x514> ld r10,-30000(r2)
0000000000015a18 <.__cxa_end_catch+0x518> std r10,24(r31)
0000000000015a1c <.__cxa_end_catch+0x51c> b 000000000001598c <.__cxa_end_catch+0x48c>
0000000000015a20 <.__cxa_end_catch+0x520> .long 0x0
0000000000015a24 <.__cxa_end_catch+0x524> .long 0x90001
0000000000015a28 <.__cxa_end_catch+0x528> lwz r0,0(r1)

And dwarfdump shows starting at 0x00015960 as:

< 117><0x00015960:0x00015a2c><><cie offset 0x0000101c::cie index 0><fde offset 0x00001018 length: 0x00000030>
<eh aug data len 0x0>
0x00015960: <off cfa=00(r1) >
0x00015968: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=r0 >
0x00015974: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x000159b8: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x000159c8: <off cfa=00(r1) >

0x000159d0: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >

0x000159f8: <off cfa=00(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >

0x00015a08: <off cfa=00(r1) >
0x00015a10: <off cfa=128(r1) > <off r31=-8(cfa) > <off r65=16(cfa) >
fde section offset 4120 0x00001018 cie offset for fde: 4124 0x0000101c
0 DW_CFA_advance_loc 8 (2 * 4)
1 DW_CFA_register r65 = r0
4 DW_CFA_offset r31 -8 (1 * -8)
6 DW_CFA_advance_loc 12 (3 * 4)
7 DW_CFA_def_cfa_offset 128
10 DW_CFA_offset_extended_sf r65 16 (-2 * -8)
13 DW_CFA_advance_loc 68 (17 * 4)
14 DW_CFA_remember_state
15 DW_CFA_def_cfa_offset 0
17 DW_CFA_advance_loc 16 (4 * 4)
18 DW_CFA_restore_extended r65
20 DW_CFA_restore r31
21 DW_CFA_advance_loc 8 (2 * 4)
22 DW_CFA_restore_state
23 DW_CFA_advance_loc 40 (10 * 4)
24 DW_CFA_remember_state
25 DW_CFA_def_cfa_offset 0
27 DW_CFA_advance_loc 16 (4 * 4)
28 DW_CFA_restore_extended r65
30 DW_CFA_restore r31
31 DW_CFA_advance_loc 8 (2 * 4)
32 DW_CFA_restore_state
33 DW_CFA_nop
34 DW_CFA_nop

/usr/src/contrib/libstdc++/libsupc++/eh_throw.cc has something
that /usr/src/contrib/libcxxrt/exception.cc does not have in
its error handling code path: a use of __cxa_begin_catch
in __cxxabiv1::__cxa_throw :

_Unwind_SjLj_RaiseException (&header->unwindHeader);
_Unwind_RaiseException (&header->unwindHeader);

// Some sort of unwinding error. Note that terminate is a handler.
__cxa_begin_catch (&header->unwindHeader);
std::terminate ();

It looks to me like __cxa_begin_catch might do either of
terminate or _Unwind_Resume and that the (conditional)
_Unwind_Resume case is possibly needed here for the
normal execution.

There are also no other calls (bl's) before the terminate:
more directly indicated to not return.

I do not know if one or both of those points helped the code
unwind correctly or not. But it seems suggestive.

Other notes:

I've demonstrated the problem on my prior head
-r333594 environment that had been build via a
6.3 vintage of devel/powerpc64-gcc (but that was
then updated to ports -r469844 and so had 6.4 of
devel/powerpc64-gcc installed). Also: base/binutils
was of a 2.30 vintage and base/gcc was of a 6.3
vintage. (system-clang was not cc, base/gcc
provided system-cc.)

Compiling to produce the a.out via:


(and so via a 6.3 vintage g++) made no
difference. It still had the problem.

I have taken to having buildworld buildkernel
use -gdwarf-2 so that /usr/libexec/gdb has
access to the information in a format it is
(mostly) designed for. (/usr/local/bin/gdb is
broken by the thrown-C++-exception problem that
I'm investigating: gdb internally throws
exceptions in normal operation.)

Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)