How to get devel/powerpc-gcc based WITH_LIBCPLUSPLUS= buildworld to have some throwing of C++ exceptions work (patch)
(too old to reply)
Mark Millard via freebsd-hackers
2018-10-16 22:29:45 UTC
I now have a patch that gets some basic C++ exception throwing
going for WITH_LIBCPLUSPLUS= use when building via
devel/powerpc64-gcc . But the overall mechanism seems to mess
up the handling of powerpc64's "red zone" style of stack
processing in various cases.

I've had recent list submittals reporting that buildworld
using WITH_LIBCPLUSPLUS= based on devel/powerpc64-gcc would
get stuck looping in _Unwind_RaiseException. This prevented
use of devel/gdb --which makes extensive use of throwing C++
exceptions in normal operation.

Well, I now have a patch that avoids the problem in
libcxxrt's throw_exception itself that made all throws
get stuck --and so allows some C++ exceptions to be thrown.
See below. (I'm not sure leading spaces will all be
preserved.) Most of text is commentary, not code.

# svnlite diff /usr/src/contrib/libcxxrt/
Index: /usr/src/contrib/libcxxrt/exception.cc
--- /usr/src/contrib/libcxxrt/exception.cc (revision 339076)
+++ /usr/src/contrib/libcxxrt/exception.cc (working copy)
@@ -772,10 +772,71 @@

_Unwind_Reason_Code err = _Unwind_RaiseException(&ex->unwindHeader);
+#if !defined(__powerpc64__) && !defined(__ppc64__)
// The _Unwind_RaiseException() function should not return, it should
// unwind the stack past this function. If it does return, then something
// has gone wrong.
report_failure(err, ex);
+// NOTE: Only tested for devel/powerpc64-gcc based buildworld
+// because clang still silently ignores
+// __builtin_eh_return(offset,handler) for powerpc64
+// (and powerpc), thus not generating correct output.
+// NOTE: I've no clue if other archtiectures might have
+// analogous issues to powerpc64. I'm not sure
+// about powerpc because of it still being stuck
+// at gcc 4.2.1 . (clang problems and no devel/powerpc-gcc .)
+// The above/normal code produced the following sort of structure
+// for throw_exception. r1 is the stack pointer, note its adjustments
+// via stdu r1,-128(r1) and via addi r1,r1,128 .
+// <throw_exception+0>: mflr r0
+// <throw_exception+4>: std r31,-8(r1)
+// <throw_exception+8>: mr r31,r3
+// <throw_exception+12>: std r0,16(r1)
+// <throw_exception+16>: stdu r1,-128(r1)
+// . . .
+// <throw_exception+140>: bl <00000018.plt_call._Unwind_RaiseException@@GCC_3.0>
+// <throw_exception+144>: ld r2,40(r1)
+// <throw_exception+148>: addi r1,r1,128
+// <throw_exception+152>: mr r4,r31
+// <throw_exception+156>: ld r0,16(r1)
+// <throw_exception+160>: ld r31,-8(r1)
+// <throw_exception+164>: mtlr r0
+// <throw_exception+168>: b <report_failure>
+// The loop in __Unwind_RaiseException had its "fs"
+// used with uw_frame_state_for and uw_update_context get
+// stuck with the pc field having the address for
+// throw_exception+152 (just after the stack adjustment
+// addi r1,r1,128). Effectively, throw_exception unwinds
+// its stack use before calling report_failure in a
+// way that throw_exception is no longer on the stack.
+// The exception unwinding logic did not handle this
+// correctly and got stuck looping.
+// The below avoids having any such stack adjustment here
+// by avoiding the report_failure call and directly doing
+// what case _URC_END_OF_STACK in report_failure does for
+// its first couple of lines. (It is also the kind of
+// thing that src/contrib/libstdc++/libsupc++/eh_throw.cc
+// has in its __cxxabiv1::__cxa_throw after the
+// _Unwind_RaiseException call.)
+// Another option could be to turn report_failure into
+// a macro so that no subroutine call could be involved.
+// That should avoid the early stack pointer kadjsutment.
+// Also: For the other archtiectures that I looked at, no
+// such stack adjsutments were involved in the code
+// generated (or the matching dwarfdump output).
+// But I did not look at many.
+ __cxa_begin_catch (&(ex->unwindHeader));
+ std::terminate();

However, code such as the following from devel/kyua leads to
other examples of _Unwind_RaiseException looping without
making progress. Note the stdu r1,-368(r1) and the
addi r1,r1,368 stack pointer adjustments and their timing
relative to stack usage (the "red zone" style used for
FreeBSD's powerpc64 ABI):

(gdb) x/64i 0x100a8528-88
0x100a84d0 <utils::fs::mkdir(utils::fs::path const&, int)>: mflr r0
0x100a84d4 <utils::fs::mkdir(utils::fs::path const&, int)+4>: std r30,-16(r1)
0x100a84d8 <utils::fs::mkdir(utils::fs::path const&, int)+8>: std r31,-8(r1)
0x100a84dc <utils::fs::mkdir(utils::fs::path const&, int)+12>: std r29,-24(r1)
0x100a84e0 <utils::fs::mkdir(utils::fs::path const&, int)+16>: mr r31,r4
0x100a84e4 <utils::fs::mkdir(utils::fs::path const&, int)+20>: std r0,16(r1)
0x100a84e8 <utils::fs::mkdir(utils::fs::path const&, int)+24>: stdu r1,-368(r1)
0x100a84ec <utils::fs::mkdir(utils::fs::path const&, int)+28>: mr r30,r3
0x100a84f0 <utils::fs::mkdir(utils::fs::path const&, int)+32>: bl 0x100abab0 <utils::fs::path::c_str() const>
0x100a84f4 <utils::fs::mkdir(utils::fs::path const&, int)+36>: nop
0x100a84f8 <utils::fs::mkdir(utils::fs::path const&, int)+40>: clrlwi r4,r31,16
0x100a84fc <utils::fs::mkdir(utils::fs::path const&, int)+44>: bl 0x10009fc0 <000000af.plt_call.mkdir@@FBSD_1.0>
0x100a8500 <utils::fs::mkdir(utils::fs::path const&, int)+48>: ld r2,40(r1)
0x100a8504 <utils::fs::mkdir(utils::fs::path const&, int)+52>: cmpwi cr7,r3,-1
0x100a8508 <utils::fs::mkdir(utils::fs::path const&, int)+56>: beq cr7,0x100a8528 <utils::fs::mkdir(utils::fs::path const&, int)+88>
0x100a850c <utils::fs::mkdir(utils::fs::path const&, int)+60>: addi r1,r1,368
0x100a8510 <utils::fs::mkdir(utils::fs::path const&, int)+64>: ld r0,16(r1)
0x100a8514 <utils::fs::mkdir(utils::fs::path const&, int)+68>: ld r29,-24(r1)
0x100a8518 <utils::fs::mkdir(utils::fs::path const&, int)+72>: ld r30,-16(r1)
0x100a851c <utils::fs::mkdir(utils::fs::path const&, int)+76>: ld r31,-8(r1)
0x100a8520 <utils::fs::mkdir(utils::fs::path const&, int)+80>: mtlr r0
0x100a8524 <utils::fs::mkdir(utils::fs::path const&, int)+84>: blr
0x100a8528 <utils::fs::mkdir(utils::fs::path const&, int)+88>: bl 0x10009d40 <000000af.plt_call.__error@@FBSD_1.0>
. . . (more not shown here) . . .

This goes along with (darfdump -v -v -F output):

< 1323><0x100a84d0:0x100a8670><mkdir><cie offset 0x0000f100::cie index 1><fde offset 0x0000f3d8 length: 0x00000034>
<eh aug data len 0x8 bytes 0x00 00 00 00 00 00 f3 c7 >
0x100a84d0: <off cfa=00(r1) >
0x100a84e0: <off cfa=00(r1) > <off r29=-24(cfa) > <off r30=-16(cfa) > <off r31=-8(cfa) > <off r65=r0 >
0x100a84ec: <off cfa=368(r1) > <off r29=-24(cfa) > <off r30=-16(cfa) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x100a8510: <off cfa=00(r1) > <off r29=-24(cfa) > <off r30=-16(cfa) > <off r31=-8(cfa) > <off r65=16(cfa) >
0x100a8524: <off cfa=00(r1) >
0x100a8528: <off cfa=368(r1) > <off r29=-24(cfa) > <off r30=-16(cfa) > <off r31=-8(cfa) > <off r65=16(cfa) >
fde section offset 62424 0x0000f3d8 cie offset for fde: 61696 0x0000f100
0 DW_CFA_advance_loc 16 (4 * 4)
1 DW_CFA_register r65 = r0
4 DW_CFA_offset r30 -16 (2 * -8)
6 DW_CFA_offset r31 -8 (1 * -8)
8 DW_CFA_offset r29 -24 (3 * -8)
10 DW_CFA_advance_loc 12 (3 * 4)
11 DW_CFA_def_cfa_offset 368
14 DW_CFA_offset_extended_sf r65 16 (-2 * -8)
17 DW_CFA_advance_loc 36 (9 * 4)
18 DW_CFA_remember_state
19 DW_CFA_def_cfa_offset 0
21 DW_CFA_advance_loc 20 (5 * 4)
22 DW_CFA_restore_extended r65
24 DW_CFA_restore r31
25 DW_CFA_restore r30
26 DW_CFA_restore r29
27 DW_CFA_advance_loc 4 (1 * 4)
28 DW_CFA_restore_state
29 DW_CFA_nop
30 DW_CFA_nop

The _Unwind_RaiseException's fs ends up reaching and the
holding the following value in my attempted devel/kyua
run for FreeBSD's test suite:

(gdb) print fs
$9 = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 29 times>, {loc = {reg = 18446744073709551592, offset = -24,
exp = 0xffffffffffffffe8 <error: Cannot access memory at address 0xffffffffffffffe8>}, how = REG_SAVED_OFFSET}, {loc = {reg = 18446744073709551600, offset = -16,
exp = 0xfffffffffffffff0 <error: Cannot access memory at address 0xfffffffffffffff0>}, how = REG_SAVED_OFFSET}, {loc = {reg = 18446744073709551608, offset = -8,
exp = 0xfffffffffffffff8 <error: Cannot access memory at address 0xfffffffffffffff8>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0},
how = REG_UNSAVED} <repeats 33 times>, {loc = {reg = 16, offset = 16, exp = 0x10 <error: Cannot access memory at address 0x10>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0,
exp = 0x0}, how = REG_UNSAVED} <repeats 80 times>}, prev = 0x0}, cfa_offset = 0, cfa_reg = 1, cfa_exp = 0x0, cfa_how = CFA_REG_OFFSET,
pc = 0x100a8528 <utils::fs::mkdir(utils::fs::path const&, int)+88>,
personality = @0x810549c40: 0x81053c900 <__gxx_personality_v0(int, _Unwind_Action, uint64_t, _Unwind_Exception*, _Unwind_Context*)>, data_align = -8, code_align = 4, retaddr_column = 65,
fde_encoding = 27 '\033', lsda_encoding = 20 '\024', saw_z = 1 '\001', signal_frame = 0 '\000', eh_ptr = 0x0}

Note the invariant value it is looping with (fs.pc value):

pc = 0x100a8528 <utils::fs::mkdir(utils::fs::path const&, int)+88>

But the routine is at 0x00000000100a85ec :

(gdb) bt
#0 _Unwind_RaiseException (exc=0x8109582e0) at /usr/src/gnu/lib/libgcc/../../../contrib/gcc/unwind.inc:103
#1 0x000000081053d704 in throw_exception (ex=0x810958288) at /usr/src/contrib/libcxxrt/exception.cc:774
#2 0x00000000100a85ec in utils::fs::mkdir (dir=..., mode=***@entry=493) at utils/fs/operations.cpp:484
#3 0x00000000100a8694 in utils::fs::mkdir_p (dir=..., mode=<optimized out>) at utils/fs/operations.cpp:502
#4 0x000000001000cbf4 in (anonymous namespace)::safe_main (mock_command=..., argv=0x3fffffffffffdb88, argc=4, ui=0x3fffffffffffd960, this=<optimized out>, this=<optimized out>) at cli/main.cpp:207
#5 cli::main (ui=***@entry=0x3fffffffffffd960, argc=***@entry=4, argv=***@entry=0x3fffffffffffdb88, mock_command=...) at cli/main.cpp:280
#6 0x000000001000e9dc in cli::main (argc=<optimized out>, argv=0x3fffffffffffdb88) at cli/main.cpp:353
#7 0x000000001000c570 in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:49

where utils::fs::mkdir:

0x00000000100a85d0 <+256>: bne 0x100a85f0 <utils::fs::mkdir(utils::fs::path const&, int)+288>
0x00000000100a85d4 <+260>: addis r9,r2,-1
0x00000000100a85d8 <+264>: addis r10,r2,-1
0x00000000100a85dc <+268>: mr r3,r29
0x00000000100a85e0 <+272>: addi r5,r9,20864
0x00000000100a85e4 <+276>: addi r4,r10,-12216
0x00000000100a85e8 <+280>: bl 0x1000a140 <000000af.plt_call.__cxa_throw@@CXXABI_1.3>
=> 0x00000000100a85ec <+284>: ld r2,40(r1)
0x00000000100a85f0 <+288>: ld r3,320(r1)
0x00000000100a85f4 <+292>: bl 0x1000a8c0 <000000af.plt_call._ZdlPv>
0x00000000100a85f8 <+296>: ld r2,40(r1)
0x00000000100a85fc <+300>: b 0x100a85d4 <utils::fs::mkdir(utils::fs::path const&, int)+260>

This is for the devel/kyua code:

fs::mkdir(const fs::path& dir, const int mode)
if (::mkdir(dir.c_str(), static_cast< mode_t >(mode)) == -1) {
const int original_errno = errno;
throw fs::system_error(F("Failed to create directory %s") % dir,

I've not figured out how to make general throwing of C++
exceptions avoid this _Unwind_RaiseException unbounded
looping with a fixed fs.pc value.

But at least now I can use devel/gdb for some of the
investigation's activity.

Other notes:

WITH_LLVM_LIBUNWIND= does not even compile for targeting
powerpc64 --so that is not currently a way around the

WITHOUT_LIB32= yse is because, for every post-gcc 4.2.1
that I've tried, the lib32 produced misuses R30 in
crtbeginS code (vs. the ABI for FreeBSD) and 32-bit
code just produces core files from a bad so-called
address that is dereferenced via R30 content.

Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)