GH-126910: Add gdb support for unwinding JIT frames by diegorusso · Pull Request #146071 · python/cpython

diegorusso · 2026-03-17T16:36:52Z

The PR adds the support to GDB for unwinding JIT frames by emitting eh frames.
It reuses part of the existent infrastructure for the perf_jit from @pablogsal.

This is part of the overall plan laid out here: #126910 (comment)

The output in GDB looks like:

Program received signal SIGINT, Interrupt.
0x0000fffff7fb50f8 in py::jit_entry:<jit> ()
(gdb) bt
#0  0x0000fffff7fb50f8 in py::jit_entry:<jit> ()
#2  0x0000aaaaaad5e314 in _PyEval_EvalFrameDefault (tstate=0xfffff7fb80f0, frame=0xfffff774bab0, throwflag=6, throwflag@entry=0)
    at ../../Python/generated_cases.c.h:5711
#3  0x0000aaaaaad61350 in _PyEval_EvalFrame (tstate=0xaaaaab1d57b0 <_PyRuntime+344632>, frame=0xfffff7fb8020, throwflag=0)
    at ../../Include/internal/pycore_ceval.h:122
...

Issue: Supporting stack unwinding in the JIT compiler #126910

bedevere-bot · 2026-03-18T11:15:27Z

🤖 New build scheduled with the buildbot fleet by @diegorusso for commit ac018d6 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146071%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

pablogsal · 2026-03-20T00:06:25Z

I have some questions about the EH frame generation and how it applies to the different code regions.

Looking at jit_record_code, it's called in two places:

For jit_shim (line 811): the entry shim compiled from Tools/jit/shim.c
For jit_executor (line 757): the full executor code region (code_size + state.trampolines.size)

Both end up calling _PyJitUnwind_GdbRegisterCode, which builds the same EH frame via _PyJitUnwind_BuildEhFrame.

The EH frame in elf_init_ehframe describes a specific prologue/epilogue sequence. On x86_64 for example:

push %rbp          (1 byte)
mov %rsp, %rbp     (3 bytes)
call *%rcx         (2 bytes)
pop %rbp           (1 byte)
ret

I understand how this is correct for jit_shim. Looking at Tools/jit/shim.c, it's a normal C function that calls into the executor:

_Py_CODEUNIT *
_JIT_ENTRY(...) {
    jit_func_preserve_none jitted = (jit_func_preserve_none)exec->jit_code;
    return jitted(exec, frame, stack_pointer, tstate, ...);
}

The compiler will emit exactly the prologue/epilogue the EH frame describes.

But I don't understand how the same EH frame is correct for jit_executor. The executor code region is a concatenation of many stencils, each compiled from Tools/jit/template.c with __attribute__((preserve_none)), chaining together via __attribute__((musttail)) tail calls. These stencils don't have the push rbp / mov rsp,rbp prologue that the EH frame describes. They use a completely different calling convention.

The FDE covers the full code_size + trampolines.size range but the CFI instructions only describe ~7 bytes of prologue/epilogue. DWARF will apply the last rule (CFA = RSP + 8 on x86_64) to all remaining addresses in the range. I don't understand why that rule would be correct at arbitrary points within the stencil code. Is it guaranteed that preserve_none stencils never modify RSP? Or is there something else going on that makes this work?

The test (test_jit.py) sets a breakpoint at id(42) which hits in the interpreter, not in the middle of a stencil. So the test verifies that the symbols appear in GDB's backtrace, but I don't think it exercises unwinding from an arbitrary point within the executor code region. Could we add a test that triggers unwinding from inside JIT code (e.g., via a signal or Ctrl+C while executing JIT code)?

Am I missing something about how the stencils interact with the stack, or is the EH frame intentionally approximate for the executor region?

pablogsal

A bunch of questions I have from reading the code so far

pablogsal · 2026-03-20T00:36:53Z

+    struct jit_code_entry *first_entry;
+};
+
+static volatile struct jit_descriptor __jit_debug_descriptor = {


Should these be non-static? The GDB JIT interface spec says GDB locates __jit_debug_descriptor and __jit_debug_register_code by name in the symbol table. With static linkage they would be invisible in .dynsym on stripped builds and when CPython is loaded as a shared library via dlopen. Am I missing something, or would this silently break in release/packaged builds where .symtab is stripped?

Maybe also worth adding __attribute__((used)) to prevent the linker from eliding them?

Yes, you are right. Instead of removing the static I've exported with the macro Py_EXPORTED_SYMBOL

pablogsal · 2026-03-20T00:36:53Z

+        id(42)
+        return
+
+    warming_up = True


Could this loop hang? When warming_up=True, the call passes warming_up_caller=True which returns immediately at line 8, so the recursive body never actually executes. If the JIT does not activate via some other path, would this not spin forever until the timeout kills it? Should there be a max iteration count as a safety net?

Also, line 16 uses bitwise & instead of and. Was that intentional? It means is_active() is always evaluated even when is_enabled() is False.

I've simplified the test, the loop is not more controlled and deterministic.

pablogsal · 2026-03-20T00:36:53Z

+        return;
+    }
+    _PyJitUnwind_GdbRegisterCode(
+        code_addr, (unsigned int)code_size, entry, filename);


code_size comes in as size_t but gets cast to unsigned int here. I know JIT regions will not be 4GB, but should the API just take size_t throughout for consistency?

This is now done.

diegorusso · 2026-03-25T15:55:53Z

I have some questions about the EH frame generation and how it applies to the different code regions.

Looking at jit_record_code, it's called in two places:

For jit_shim (line 811): the entry shim compiled from Tools/jit/shim.c

For jit_executor (line 757): the full executor code region (code_size + state.trampolines.size)

Both end up calling _PyJitUnwind_GdbRegisterCode, which builds the same EH frame via _PyJitUnwind_BuildEhFrame.

The EH frame in elf_init_ehframe describes a specific prologue/epilogue sequence. On x86_64 for example:
push %rbp          (1 byte)
mov %rsp, %rbp     (3 bytes)
call *%rcx         (2 bytes)
pop %rbp           (1 byte)
ret
I understand how this is correct for jit_shim. Looking at Tools/jit/shim.c, it's a normal C function that calls into the executor:
_Py_CODEUNIT *
_JIT_ENTRY(...) {
    jit_func_preserve_none jitted = (jit_func_preserve_none)exec->jit_code;
    return jitted(exec, frame, stack_pointer, tstate, ...);
}
The compiler will emit exactly the prologue/epilogue the EH frame describes.

But I don't understand how the same EH frame is correct for jit_executor. The executor code region is a concatenation of many stencils, each compiled from Tools/jit/template.c with __attribute__((preserve_none)), chaining together via __attribute__((musttail)) tail calls. These stencils don't have the push rbp / mov rsp,rbp prologue that the EH frame describes. They use a completely different calling convention.

The FDE covers the full code_size + trampolines.size range but the CFI instructions only describe ~7 bytes of prologue/epilogue. DWARF will apply the last rule (CFA = RSP + 8 on x86_64) to all remaining addresses in the range. I don't understand why that rule would be correct at arbitrary points within the stencil code. Is it guaranteed that preserve_none stencils never modify RSP? Or is there something else going on that makes this work?

The test (test_jit.py) sets a breakpoint at id(42) which hits in the interpreter, not in the middle of a stencil. So the test verifies that the symbols appear in GDB's backtrace, but I don't think it exercises unwinding from an arbitrary point within the executor code region. Could we add a test that triggers unwinding from inside JIT code (e.g., via a signal or Ctrl+C while executing JIT code)?

Am I missing something about how the stencils interact with the stack, or is the EH frame intentionally approximate for the executor region?

What this change synthesises for jit_executor is one unwind description for the executor as a whole, not compiler-emitted per-stencil CFI. Because the stencils are musttail-chained, the jumps between stencils do not add extra native call frames. The unwind job here is just to recover the caller of the executor frame. We don't want to describe each stencil as its own frame.

When GDB stops at a PC inside py::jit_executor:<jit>:

it finds the FDE whose range covers that PC
takes the CFI row for that PC,
computes the CFA from that row
uses the CFA rules to recover the caller registers and return PC.

On AArch64, for most of the covered executor range, the synthetic CFI says:

CFA = x29 + 16
saved x29 at CFA - 16
saved x30 at CFA - 8.
That is enough for GDB to recover the caller frame in py::jit_shim:<jit>, and then continue unwinding into _PyEval_*.

Good catch for the testing gap. I’ve now added a new test that breaks inside the jit executor. It sill breaks at the builtin_id but GDB then finishes out through the C helper frames until the selected frame is py::jit_executor:<jit> (thanks to some GDB-python scripting), single-steps twice inside the executor, and only then runs bt.
The backtrace is now taken with the current PC in executor code itself, and it unwinds through py::jit_shim:<jit> and then back into _PyEval_*.

Fidget-Spinner · 2026-03-25T20:13:31Z

@diegorusso @pablogsal I think I may have come up with a solution that works.

EDIT: I think I gdb doesn't only use backtrace. So we're still stuck. Sorry for the noise!

Background info (skip if not interested):

glibc seems to call out to libgcc on linux when needing to unwind in backtrace from execinfo.h.
libgcc does not seem to implement proper frame pointer backchaining for x86_64 and AArch64, only PPC 1.
So we need eh_frames it seems for backtrace.

The current issue:

In DWARF, you can specify in the eh_frame the canonical frame address (CFA). Traditionally, it's defined as rsp + offset.
The problem with the current PR however, is that each stencil changes rsp. That means the CFA is usually wrong for the executor.
A possible solution is to generate DWARF opcodes for each stencil that say "bump rsp", but that's slow and complicated.

The solution:

Notice: we already have frame pointers in prologue and preserve them! That means you can just say it's tied to rbp at fixed offset of rbp + 16 all the time instead of tying it to an rsp that changes. I got this idea, and also checked with cranelift's Chris Fallin, who said they do it. Thanks a lot Chris!
So that means, with frame pointers, eh frame creation is a lot simpler. You can have just one eh frame for the whole JIT code still, we just need the eh_frame to point to rbp + 16.
This ensures correctness while also allowing for a simple implementation.

This should work with backtrace from execinfo.h. without issues. It should even work with gdb step debugging/bt, with the exception that at the function prologue, it might be broken. However the most important thing is that we unbreak all C extension code that uses backtrace! Also, backtrace should be fast as our DWARF would be tiny and simple.

TLDR: frame pointers = eh_frame is simple.

pablogsal · 2026-03-25T22:44:40Z

I have some questions about the EH frame generation and how it applies to the different code regions.
Looking at jit_record_code, it's called in two places:

For jit_shim (line 811): the entry shim compiled from Tools/jit/shim.c

For jit_executor (line 757): the full executor code region (code_size + state.trampolines.size)

Both end up calling _PyJitUnwind_GdbRegisterCode, which builds the same EH frame via _PyJitUnwind_BuildEhFrame.
The EH frame in elf_init_ehframe describes a specific prologue/epilogue sequence. On x86_64 for example:
push %rbp          (1 byte)
mov %rsp, %rbp     (3 bytes)
call *%rcx         (2 bytes)
pop %rbp           (1 byte)
ret
I understand how this is correct for jit_shim. Looking at Tools/jit/shim.c, it's a normal C function that calls into the executor:
_Py_CODEUNIT *
_JIT_ENTRY(...) {
    jit_func_preserve_none jitted = (jit_func_preserve_none)exec->jit_code;
    return jitted(exec, frame, stack_pointer, tstate, ...);
}
The compiler will emit exactly the prologue/epilogue the EH frame describes.
But I don't understand how the same EH frame is correct for jit_executor. The executor code region is a concatenation of many stencils, each compiled from Tools/jit/template.c with __attribute__((preserve_none)), chaining together via __attribute__((musttail)) tail calls. These stencils don't have the push rbp / mov rsp,rbp prologue that the EH frame describes. They use a completely different calling convention.
The FDE covers the full code_size + trampolines.size range but the CFI instructions only describe ~7 bytes of prologue/epilogue. DWARF will apply the last rule (CFA = RSP + 8 on x86_64) to all remaining addresses in the range. I don't understand why that rule would be correct at arbitrary points within the stencil code. Is it guaranteed that preserve_none stencils never modify RSP? Or is there something else going on that makes this work?
The test (test_jit.py) sets a breakpoint at id(42) which hits in the interpreter, not in the middle of a stencil. So the test verifies that the symbols appear in GDB's backtrace, but I don't think it exercises unwinding from an arbitrary point within the executor code region. Could we add a test that triggers unwinding from inside JIT code (e.g., via a signal or Ctrl+C while executing JIT code)?
Am I missing something about how the stencils interact with the stack, or is the EH frame intentionally approximate for the executor region?
What this change synthesises for jit_executor is one unwind description for the executor as a whole, not compiler-emitted per-stencil CFI. Because the stencils are musttail-chained, the jumps between stencils do not add extra native call frames. The unwind job here is just to recover the caller of the executor frame. We don't want to describe each stencil as its own frame.

When GDB stops at a PC inside py::jit_executor:<jit>:
* it finds the FDE whose range covers that PC

* takes the CFI row for that PC,

* computes the CFA from that row

* uses the CFA rules to recover the caller registers and return PC.
On AArch64, for most of the covered executor range, the synthetic CFI says:
* CFA = x29 + 16

* saved x29 at CFA - 16

* saved x30 at CFA - 8.
  That is enough for GDB to recover the caller frame in `py::jit_shim:<jit>`, and then continue unwinding into `_PyEval_*`.
Good catch for the testing gap. I’ve now added a new test that breaks inside the jit executor. It sill breaks at the builtin_id but GDB then finishes out through the C helper frames until the selected frame is py::jit_executor:<jit> (thanks to some GDB-python scripting), single-steps twice inside the executor, and only then runs bt. The backtrace is now taken with the current PC in executor code itself, and it unwinds through py::jit_shim:<jit> and then back into _PyEval_*.

@diegorusso I have to say that I am tremendously confused here.

If GDB or backtrace() stops at an arbitrary PC inside py::jit_executor:<jit>, the unwind info for that exact PC should let the unwinder reconstruct the caller frame (py::jit_shim:<jit>) and then continue into _PyEval_*.

So the real question is not “does the FDE cover the address range?” and it is not “do the stencils form one logical frame?”. The real question is: does the CFI row that applies at that PC actually describe the machine state there?

That is the part I do not think has been explained.

I agree with the narrow musttail point: tail-chaining the stencils means you do not accumulate one native call frame per stencil. Fine. But that only tells us that we want to unwind the executor as one logical frame. It does not tell us that one fixed synthetic unwind recipe is valid everywhere inside the executor blob.

And that is exactly where I think the argument goes off the rails.

jit_executor is not one ordinary C function with one stable prologue/epilogue. It is a concatenation of many preserve_none stencils, glued together with musttail. For a single synthetic FDE to be correct across the whole region, there has to be some invariant that says “for any PC in executor code, the CFA and saved return state look like this”. I do not see that invariant stated anywhere, and the current explanation seems to jump from “musttail” straight to “the unwind is correct”, which are not the same thing.

A concrete x86_64 example of why this seems wrong to me:

with the same sort of flags used for executor stencils, a preserve_none + musttail function can compile to something as trivial as

jmp callee

or, if it needs temporary stack space / spills, something more like

subq $24, %rsp
...
addq $24, %rsp
jmp callee

In the first case there is no %rbp frame at all. In the second case the CFA is temporarily %rsp-relative and changes inside the body. So I do not understand how one synthetic %rbp-based description for the entire covered executor range is supposed to be generally correct.

For jit_shim I can at least see the intended story, because it is one ordinary non-tail C function that calls into JIT code. For jit_executor, I still do not see what makes the unwind recipe valid for arbitrary PCs inside the blob.

Also, I rebuilt the branch locally and tried the exact “finish to py::jit_executor:<jit>, step twice, then bt” flow. On x86_64 I still get:

#0  py::jit_executor:<jit> ()
#1  ?? ()
...
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

So this is not just a theoretical concern for me. I still do not understand why the model being described here is supposed to work.I am of course not objecting to the goal. I am saying I still do not see the correctness argument. If the claim is that this is actually a correct unwind description for jit_executor as a whole, then I think what is missing from the discussion is the key invariant: what exactly is guaranteed to be true about the CFA / saved FP / saved return address at an arbitrary PC inside executor code that makes this one synthetic FDE valid?

diegorusso · 2026-03-25T22:56:32Z

@diegorusso @pablogsal I think I may have come up with a solution that works.

EDIT: I think I gdb doesn't only use backtrace. So we're still stuck. Sorry for the noise!

Background info (skip if not interested):

glibc seems to call out to libgcc on linux when needing to unwind in backtrace from execinfo.h.

libgcc does not seem to implement proper frame pointer backchaining for x86_64 and AArch64, only PPC 1.

So we need eh_frames it seems for backtrace.

The current issue:

In DWARF, you can specify in the eh_frame the canonical frame address (CFA). Traditionally, it's defined as rsp + offset.

The problem with the current PR however, is that each stencil changes rsp. That means the CFA is usually wrong for the executor.

A possible solution is to generate DWARF opcodes for each stencil that say "bump rsp", but that's slow and complicated.

The solution:

Notice: we already have frame pointers in prologue and preserve them! That means you can just say it's tied to rbp at fixed offset of rbp + 16 all the time instead of tying it to an rsp that changes. I got this idea, and also checked with cranelift's Chris Fallin, who said they do it. Thanks a lot Chris!

So that means, with frame pointers, eh frame creation is a lot simpler. You can have just one eh frame for the whole JIT code still, we just need the eh_frame to point to rbp + 16.

This ensures correctness while also allowing for a simple implementation.

This should work with backtrace from execinfo.h. without issues. It should even work with gdb step debugging/bt, with the exception that at the function prologue, it might be broken. However the most important thing is that we unbreak all C extension code that uses backtrace! Also, backtrace should be fast as our DWARF would be tiny and simple.

TLDR: frame pointers = eh_frame is simple.

Thanks, thanks for the comment. I regenerated the x86_64 and AArch64 stencils after the recent frame-pointer changes. What we have today is that shim gets a real frame-pointer prologue, but the executor stencils still are not uniformly rbp/x29-framed, so I don’t think the current generated code is enough to justify a single executor-wide CFA = rbp + 16 / x29 + const rule for arbitrary PCs in the blob.
If we want to go to that direction, we would need to force frame pointers for the executor stencils too, not just for the shim (which it doesn't make any sense as we moved away from them!)

The current implementation is still one synthetic executor-wide FDE. The unwinder uses the current PC to select that FDE and apply its CFI to recover the caller frame. That works where the actual machine state matches the synthetic rule at the stop PC, but it is still approximate executor-wide unwind metadata, not exact per-stencil CFI.

Separately, once this PR lands, wiring up libgcc-backed backtrace should be fairly easy. We already synthesise .eh_frame; the remaining work is to call the appropriate __register_frame* and deregistration API for that blob so the unwinder can see it.

diegorusso · 2026-03-25T23:48:03Z

I have some questions about the EH frame generation and how it applies to the different code regions.
Looking at jit_record_code, it's called in two places:

For jit_shim (line 811): the entry shim compiled from Tools/jit/shim.c

For jit_executor (line 757): the full executor code region (code_size + state.trampolines.size)

Both end up calling _PyJitUnwind_GdbRegisterCode, which builds the same EH frame via _PyJitUnwind_BuildEhFrame.
The EH frame in elf_init_ehframe describes a specific prologue/epilogue sequence. On x86_64 for example:
push %rbp          (1 byte)
mov %rsp, %rbp     (3 bytes)
call *%rcx         (2 bytes)
pop %rbp           (1 byte)
ret
I understand how this is correct for jit_shim. Looking at Tools/jit/shim.c, it's a normal C function that calls into the executor:
_Py_CODEUNIT *
_JIT_ENTRY(...) {
    jit_func_preserve_none jitted = (jit_func_preserve_none)exec->jit_code;
    return jitted(exec, frame, stack_pointer, tstate, ...);
}
The compiler will emit exactly the prologue/epilogue the EH frame describes.
But I don't understand how the same EH frame is correct for jit_executor. The executor code region is a concatenation of many stencils, each compiled from Tools/jit/template.c with __attribute__((preserve_none)), chaining together via __attribute__((musttail)) tail calls. These stencils don't have the push rbp / mov rsp,rbp prologue that the EH frame describes. They use a completely different calling convention.
The FDE covers the full code_size + trampolines.size range but the CFI instructions only describe ~7 bytes of prologue/epilogue. DWARF will apply the last rule (CFA = RSP + 8 on x86_64) to all remaining addresses in the range. I don't understand why that rule would be correct at arbitrary points within the stencil code. Is it guaranteed that preserve_none stencils never modify RSP? Or is there something else going on that makes this work?
The test (test_jit.py) sets a breakpoint at id(42) which hits in the interpreter, not in the middle of a stencil. So the test verifies that the symbols appear in GDB's backtrace, but I don't think it exercises unwinding from an arbitrary point within the executor code region. Could we add a test that triggers unwinding from inside JIT code (e.g., via a signal or Ctrl+C while executing JIT code)?
Am I missing something about how the stencils interact with the stack, or is the EH frame intentionally approximate for the executor region?
What this change synthesises for jit_executor is one unwind description for the executor as a whole, not compiler-emitted per-stencil CFI. Because the stencils are musttail-chained, the jumps between stencils do not add extra native call frames. The unwind job here is just to recover the caller of the executor frame. We don't want to describe each stencil as its own frame.
When GDB stops at a PC inside py::jit_executor:<jit>:
* it finds the FDE whose range covers that PC

* takes the CFI row for that PC,

* computes the CFA from that row

* uses the CFA rules to recover the caller registers and return PC.
On AArch64, for most of the covered executor range, the synthetic CFI says:
* CFA = x29 + 16

* saved x29 at CFA - 16

* saved x30 at CFA - 8.
  That is enough for GDB to recover the caller frame in `py::jit_shim:<jit>`, and then continue unwinding into `_PyEval_*`.
Good catch for the testing gap. I’ve now added a new test that breaks inside the jit executor. It sill breaks at the builtin_id but GDB then finishes out through the C helper frames until the selected frame is py::jit_executor:<jit> (thanks to some GDB-python scripting), single-steps twice inside the executor, and only then runs bt. The backtrace is now taken with the current PC in executor code itself, and it unwinds through py::jit_shim:<jit> and then back into _PyEval_*.
@diegorusso I have to say that I am tremendously confused here.

My understanding of what this code is supposed to do is pretty simple: if GDB or backtrace() stops at an arbitrary PC inside py::jit_executor:<jit>, the unwind info for that exact PC should let the unwinder reconstruct the caller frame (py::jit_shim:<jit>) and then continue into _PyEval_*.

So the real question is not “does the FDE cover the address range?” and it is not “do the stencils form one logical frame?”. The real question is: does the CFI row that applies at that PC actually describe the machine state there?

That is the part I do not think has been explained.

I agree with the narrow musttail point: tail-chaining the stencils means you do not accumulate one native call frame per stencil. Fine. But that only tells us that we want to unwind the executor as one logical frame. It does not tell us that one fixed synthetic unwind recipe is valid everywhere inside the executor blob.

And that is exactly where I think the argument goes off the rails.

jit_executor is not one ordinary C function with one stable prologue/epilogue. It is a concatenation of many preserve_none stencils, glued together with musttail. For a single synthetic FDE to be correct across the whole region, there has to be some invariant that says “for any PC in executor code, the CFA and saved return state look like this”. I do not see that invariant stated anywhere, and the current explanation seems to jump from “musttail” straight to “the unwind is correct”, which are not the same thing.

A concrete x86_64 example of why this seems wrong to me:

with the same sort of flags used for executor stencils, a preserve_none + musttail function can compile to something as trivial as
jmp callee
or, if it needs temporary stack space / spills, something more like
subq $24, %rsp
...
addq $24, %rsp
jmp callee
In the first case there is no %rbp frame at all. In the second case the CFA is temporarily %rsp-relative and changes inside the body. So I do not understand how one synthetic %rbp-based description for the entire covered executor range is supposed to be generally correct.

For jit_shim I can at least see the intended story, because it is one ordinary non-tail C function that calls into JIT code. For jit_executor, I still do not see what makes the unwind recipe valid for arbitrary PCs inside the blob.

Also, I rebuilt the branch locally and tried the exact “finish to py::jit_executor:<jit>, step twice, then bt” flow. On x86_64 I still get:
#0  py::jit_executor:<jit> ()
#1  ?? ()
...
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
So this is not just a theoretical concern for me. I still do not understand why the model being described here is supposed to work.I am of course not objecting to the goal. I am saying I still do not see the correctness argument. If the claim is that this is actually a correct unwind description for jit_executor as a whole, then I think what is missing from the discussion is the key invariant: what exactly is guaranteed to be true about the CFA / saved FP / saved return address at an arbitrary PC inside executor code that makes this one synthetic FDE valid?

Ok, I think now I understand. After re-checking the generated stencils I agree the current explanation was too bold.

musttail only establishes the narrow point that the stencil-to-stencil transitions do not accumulate one native call frame per stencil and it does not by itself establish the stronger property needed for unwinding: that for an arbitrary PC inside jit_executor, the CFA and saved return state always have a shape described by one executor-wide FDE.

That stronger property is the missing invariant here.

After looking again at the regenerated x86_64 and AArch64 stencils, I don't think we have that invariant today:

only jit_shim gets a guaranteed frame-pointer-based prologue
executor stencils are not uniform
- on x86_64, many executor stencils are frameless and/or adjust rsp
- on AArch64, many executor stencils just save x30 and adjust sp, without establishing x29
- only a small subset of executor stencils actually materialise a conventional rbp/x29 frame

I cannot justify the current synthetic executor-wide FDE as being correct for arbitrary PCs in the executor blob. The new test I added is still useful, but it proves something narrower: that the synthetic FDE works for the exercised in-executor stop. It does not prove that the same CFI is exact for every interior PC in the region (like you did in your example)

I think the real options are:

Make the invariant true in codegen/stencils by forcing all executor stencils to follow one documented frame-layout rule, so one FDE is actually justified.
For example:

x86_64: every executor stencil establishes the same rbp-based frame shape
AArch64: every executor stencil establishes the same x29/x30-based frame shape, or at least guarantees that x29 is stable and the original return-to-shim state is always recoverable in one fixed way

Emit finer-grained unwind metadata: keep the current mixed stencil shapes, but stop pretending one unwind recipe covers the whole executor. That means multiple FDEs or per-range metadata.
Narrow the claim: keep executor symbolisation, but do not claim a correct executor-wide unwind description until we have either (1) or (2).
Something else?

The current implementation does not yet have the invariant needed to justify one executor-wide FDE for jit_executor but at the same time I don't really like the suggestions above.

Let me think about it

Fidget-Spinner · 2026-03-26T05:12:49Z

the executor stencils still are not uniformly rbp/x29-framed,

The current generation reserve the rbp. So all current stencils assume an rbp. Do you think it would fix it if we emitted our own prologue for the very first JIT executor uop ie (push %rbp; movq %rsp, %rbp) , and teardown (popq %rbp) at all rets ? I have a working branch that does that. FWIW, it can be done quite easily using the assembly manipulator we have in the JIT. Will that make it appropiate rbp/x29-framed?

call the appropriate __register_frame* and deregistration API for that blob so the unwinder can see it.

Unfortunately, it seems you're right here. I dug around libgcc a little more and that's the only interface I see that intercepts _Unwind_Find_FDE. The function is public but undocumented, which is annoying. I'm just shocked that libgcc does not seem to use frame pointers as a fallback for x86_64 or AArch64 when I looked around it.

diegorusso · 2026-03-26T10:24:26Z

The current generation reserve the rbp. So all current stencils assume an rbp.

Not all of them. See _SET_IP family. But you can see others as well. On AArch64 if we reserve the frame pointer, it will be barely touched (just a few uops set it). If we don't reserve it, then we have the standard prologue/epilogue for the majority of the uops.

I'm not entirely sure your statement is true.

Fidget-Spinner · 2026-03-26T10:34:10Z

_SET_IP

Huh that's surprising! On x86_64, the current main produces code that doesn't touch rbp at all (from manual inspection at least). I wonder why it's different on AArch64, thanks for reporting back.

Fidget-Spinner · 2026-03-26T10:36:28Z

_SET_IP

Huh that's surprising! On x86_64, the current main produces code that doesn't touch rbp at all (from manual inspection at least). I wonder why it's different on AArch64, thanks for reporting back.

Oh sorry I'm wrong, not main, my branch, I had to pass the usual:

            "-fno-omit-frame-pointer",
            "-mno-omit-leaf-frame-pointer",

to it to get things like that.

diegorusso · 2026-03-27T14:47:08Z

I've just raised 2 PR that will get the invariant that we need for:

@Fidget-Spinner will implement a simple assembly verifier pass in the assembly optimizer that will check on x64 there's no push rbp and on AArch64 there's no stp x29

Fidget-Spinner · 2026-03-31T08:41:10Z

This works on x86_64 on my system now on 17be0a2:

With gdb 17:

#0  get_stack_gnu (self=0x7ffff719a840, args=0x0) at stackunwind.c:73
#1  0x00005555556767d5 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffff712b240, 
    tstate=0x555555c3e968 <_PyRuntime+346248>) at ./Include/internal/pycore_call.h:144
#2  PyObject_Vectorcall (callable=callable@entry=0x7ffff712b240, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:327
#3  0x00005555557f4ce2 in _Py_VectorCallInstrumentation_StackRefSteal (callable=..., arguments=0x7ffff7fb27e8, total_args=0, kwnames=..., 
    call_instrumentation=false, frame=0x7ffff7fb2760, this_instr=0x7ffff7b2b32a, tstate=0x555555c3e968 <_PyRuntime+346248>) at Python/ceval.c:775
#4  0x00005555556043e9 in _PyEval_EvalFrameDefault (tstate=0x7ffff719a840, frame=0x7ffff7fb2760, throwflag=0) at Python/generated_cases.c.h:1841
#5  0x00005555557faa98 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555c3e968 <_PyRuntime+346248>) at ./Include/internal/pycore_ceval.h:122
#6  _PyEval_Vector (tstate=0x555555c3e968 <_PyRuntime+346248>, func=0x7ffff71aa560, locals=0x0, args=<optimized out>, argcount=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:2179
#7  0x00005555556767d5 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7ffff71aa560, 
    tstate=0x555555c3e968 <_PyRuntime+346248>) at ./Include/internal/pycore_call.h:144
#8  PyObject_Vectorcall (callable=0x7ffff71aa560, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:327
#9  0x00005555557f51f3 in _Py_BuiltinCallFastWithKeywords_StackRefSteal (callable=..., arguments=0x7ffff7fb2740, total_args=4) at Python/ceval.c:859
#10 0x00007ffff7fa87ae in py::jit_executor:<jit> ()
#11 0x000055555560ad97 in _PyEval_EvalFrameDefault (tstate=0x7ffff719a840, frame=0xfffffffffffffffa, throwflag=0) at Python/generated_cases.c.h:5825
...
#90 0x00007ffff7c2a28b in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#91 0x000055555560e745 in _start ()

On lldb-21

* thread #1, name = 'python', stop reason = breakpoint 1.1
  * frame #0: 0x00007ffff7fad4c6 stackunwind.cpython-315-x86_64-linux-gnu.so`get_stack_gnu(self=0x00007ffff719a840, args=0x0000000000000000) at stackunwind.c:73:11
    frame #1: 0x00005555556767d5 python`_PyObject_VectorcallTstate(kwnames=<unavailable>, nargsf=<unavailable>, args=<unavailable>, callable=0x00007ffff712b1f0, tstate=0x0000555555c3e968) at pycore_call.h:144:11
    frame #2: 0x00005555557f4ce2 python`_Py_VectorCallInstrumentation_StackRefSteal(callable=<unavailable>, arguments=0x00007ffff7fb27e8, total_args=0, kwnames=<unavailable>, call_instrumentation=false, frame=0x00007ffff7fb2760, this_instr=0x00007ffff7b2b32a, tstate=0x0000555555c3e968) at ceval.c:775:11
    frame #3: 0x00005555556043e9 python`_PyEval_EvalFrameDefault(tstate=<unavailable>, frame=0x00007ffff7fb2760, throwflag=<unavailable>) at generated_cases.c.h:1841:35
    frame #4: 0x00005555557faa98 python`_PyEval_EvalFrame(throwflag=0, frame=<unavailable>, tstate=0x0000555555c3e968) at pycore_ceval.h:122:16 [inlined]
    frame #5: 0x00005555557faa93 python`_PyEval_Vector(tstate=0x0000555555c3e968, func=0x00007ffff71aa560, locals=0x0000000000000000, args=<unavailable>, argcount=<unavailable>, kwnames=<unavailable>) at ceval.c:2179:12
    frame #6: 0x00005555556767d5 python`_PyObject_VectorcallTstate(kwnames=<unavailable>, nargsf=<unavailable>, args=<unavailable>, callable=0x00007ffff71aa560, tstate=0x0000555555c3e968) at pycore_call.h:144:11
    frame #7: 0x00005555557f51f3 python`_Py_BuiltinCallFastWithKeywords_StackRefSteal(callable=<unavailable>, arguments=0x00007ffff7fb2740, total_args=4) at ceval.c:859:11
    frame #8: 0x00007ffff7fa87ae JIT(0x555555e0bdd0)`py::jit_executor:<jit> + 1966
    frame #9: 0x000055555560ad97 python`_PyEval_EvalFrameDefault(tstate=<unavailable>, frame=0xfffffffffffffffa, throwflag=<unavailable>) at generated_cases.c.h:5825:13
...
    frame #78: 0x00007ffff7c2a1ca libc.so.6`__libc_start_call_main(main=(python`main at python.c:14:1), argc=2, argv=0x00007fffffffda38) at libc_start_call_main.h:58:16
    frame #79: 0x00007ffff7c2a28b libc.so.6`__libc_start_main_impl(main=(python`main at python.c:14:1), argc=2, argv=0x00007fffffffda38, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=
0x00007fffffffda28) at libc-start.c:360:3
    frame #80: 0x000055555560e745 python`_start + 37

Fidget-Spinner · 2026-03-31T16:10:15Z

This works on x86_64 on my system now on 17be0a2:

On latest commit, I once again get practically the same result again on gdb-15+llvm-21, this time I tested both with CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" and also without it. They both work. It seems as long as the JIT itself reserves the frame pointer, even the rest of CPython doesn't need to be built with frame pointer and it will still work.

Fidget-Spinner

From my very rudimentary understanding of DWARF. This PR works as long as we maintain the frame pointer invariant (frame pointer-relative CFA). I think it's good to go, but I'll defer approval to @pablogsal as he's the expert here.

This because the two JIT memory regions are visible as unique frame from the GDB point of view.

pablogsal

One clarification here: the reason this works is not “the synthetic EH frame describes every stencil exactly.” The sound argument is “the whole executor region can be unwound as one logical frame because the caller state stays recoverable in one stable place.” That holds if:

the only real frame record comes from the shim
executor stencils tail-chain rather than creating extra frames
the JIT preserves rbp/x29 across the executor region via Tools/jit/_targets.py and the stencil validator
the synthetic CFI we create is understood as describing that steady-state caller layout, not the transient internal stack motion of each stencil

I have tested this on my x86_64 machine and couldn't make it break so I think we are good now appart from these comments but someone should stress test this on an aarch64 machine ;)

pablogsal · 2026-04-04T23:47:14Z

+    shdrs[SH_SYMTAB].sh_addralign = 8;
+    shdrs[SH_SYMTAB].sh_entsize = sizeof(Elf64_Sym);
+
+    struct jit_code_entry *entry = PyMem_RawMalloc(sizeof(*entry));


hen we free an executor's JIT code in _PyJIT_Free, aren't we leaking the corresponding GDB JIT entries? The in-memory ELF and the jit_code_entry node are never unregistered or freed, so sh_addr and st_value end up pointing at unmapped memory. If that address range gets reused by a later JIT compilation, wouldn't GDB resolve the new code to the stale old symbol?

This is a good catch indeed. We should free GDB memory whenever we free executors. I'm stitching something up.

pablogsal · 2026-04-04T23:49:50Z

+    entry->symfile_addr = (const char *)buf;
+    entry->symfile_size = total_size;
+    entry->prev = NULL;
+    entry->next = __jit_debug_descriptor.first_entry;


I am assuming this is all made in a thread-safe context, otherwise multiple threads can corrupt this linked list.

Another good catch. I don't think at the moment is a bid deal as at the moment the JIT works only in single thread mode but this doesn't meant it won't be in the future. I'm adding some mutex around the list modification.

pablogsal · 2026-04-04T23:59:35Z

+        DWRF_U8(DWRF_CFA_def_cfa_offset);         // CFA = SP + 0 (stack restored)
+        DWRF_UV(0);                               // Back to original stack position
+
+        if (absolute_addr) {


On AArch64, when absolute_addr=1, aren't the stuff for the epilogue restore+def_cfa_offset immediately overridden by the rest of the lines? Unless i am missing siomething on x86_64 we skip the epilogue with if !absolute_addr) so should we do the same here to avoid emitting dead CFI bytes? Or is there a reason the AArch64 path needs them?

I'm reworking the AArch64 CFI to follow the same structure as the x86_64 path. So, like x86_64, the common path now establishes the steady-state FP-based rule, and only the !absolute_addr path appends the epilogue row.

pablogsal · 2026-04-12T01:39:54Z

+    size_t code_size;
    const char *entry_name;

    if (!PyArg_ParseTuple(args, "OIs", &code_addr_v, &code_size, &entry_name))


PyArg_ParseTuple format "I" writes unsigned int (4 bytes), but code_size is now size_t (8 bytes on 64-bit).

ok, after some thinking on how to solve this I decided to change all the api to use size_t, parse code_size as Python object and then use PyLong_AsSize_t

pablogsal · 2026-04-12T01:40:55Z

        executor->jit_code = NULL;
        executor->jit_size = 0;
+#ifdef PY_HAVE_PERF_TRAMPOLINE
+        _PyJitUnwind_GdbUnregisterCode(memory);


Registration is conditional but unregistration is not. jit_record_code skips GDB registration when perf callbacks are active, but _PyJIT_Free/_PyJIT_Fini always call _PyJitUnwind_GdbUnregisterCode. This can start a party that has a cost of a mutex acquire + linked-list scan on every executor free under perf. Should mirror the registration condition no?

yes, you are right. I've refactored a bit the whole logic now and this should fix also the issue of O(n) scan of the linked list.

pablogsal · 2026-04-12T01:42:02Z

 }

+static void
+jit_record_code(const void *code_addr, size_t code_size,


I will leave this for the future but as this is unconditionally active I assume will have a perf cost we probably want top measure

I'm measuring it.. it might take some time.

pablogsal · 2026-04-12T01:43:01Z

+    entry->symfile_size = total_size;
+    entry->code_addr = code_addr;
+
+    PyMutex_Lock((PyMutex *)&__jit_debug_descriptor.mutex);


Casting away volatile from __jit_debug_descriptor.mutex is technically UB per C11. The mutex doesn't need to be visible to GDB so consider moving it out of the volatile struct into a separate static PyMutex jit_debug_mutex;.

pablogsal · 2026-04-12T01:44:05Z

+#endif
+}
+
+void


The lookup is O(n) under the mutex. The list is already doubly-linked so unlink is O(1) if you store a pointer to the jit_code_entry alongside the executor, you skip thes earch entirely.

I've changed a bit the way I get the memory to free.

pablogsal · 2026-04-12T01:44:56Z

+_Static_assert(sizeof(EhFrameHeader) == 20, "EhFrameHeader layout mismatch");
+
+/* DWARF encoding constants used in EH frame headers */
+static const uint8_t DwarfUData4 = 0x03;


These DWARF encoding are duplicates of the DWRF_EH_PE_* enum in jit_unwind.c. Since the refactoring's goal was to share DWARF code, consider exposing those constants from the shared header.

pablogsal · 2026-04-12T01:45:51Z

+        DWRF_ALIGNNOP(sizeof(uintptr_t));     // Align to pointer boundary
+    )
+
+    ctx->eh_frame_p = p;  // Remember start of FDE data


I think this is dead code as this field doesn't seem to be read after the refactoring.

thanks! I didn't notice

pablogsal · 2026-04-12T01:46:36Z

@@ -0,0 +1,30 @@
+#ifndef Py_CORE_JIT_UNWIND_H


should be Py_INTERNAL_JIT_UNWIND_H

pablogsal · 2026-04-12T01:46:51Z

@@ -0,0 +1,30 @@
+#ifndef Py_CORE_JIT_UNWIND_H
+#define Py_CORE_JIT_UNWIND_H
+


this is missing Py_BUILD_CORE guard no?

yes, I've seen now the other headers files.

pablogsal · 2026-04-12T01:47:05Z

+#ifndef Py_CORE_JIT_UNWIND_H
+#define Py_CORE_JIT_UNWIND_H
+
+#ifdef PY_HAVE_PERF_TRAMPOLINE


The entire file is gated on PY_HAVE_PERF_TRAMPOLINE, but the GDB JIT interface is conceptually independent of perf no?

Oops yea you're right.

for now, I'll add the bare minimum to address this but I already in mind some refactoring to do with another PR. Let's land this first and then I will refactor the code in light of adding libcc (for gnu backtrace)

this solution won't be the best, but it will be improved in subsequent PRs. I don't want to keep changing this PR.

diegorusso · 2026-04-16T16:10:38Z

The failure seems to be an infra issue (cannot to apt-get update)

The FDE described a push/mov prologue that executor stencils (-mframe-pointer=reserved) never execute, corrupting unwind at the first few bytes of every region. Move the steady-state CFI into the CIE and split the emitter into perf (unchanged) and gdb helpers.

Require linux + x86_64/aarch64 + sys._jit.is_enabled() so unsupported platforms, arches, and interpreter-only tier-2 builds skip cleanly instead of hanging or failing noisily.

Replace the fixed JIT_ENTRY_SINGLE_STEPS=2 loop with a helper that verifies frame.name() after every si, so a stencil or toolchain change that drifts the PC fails loudly instead of matching the tolerant final regex by accident.

pablogsal · 2026-04-18T01:26:02Z

Hey, I did another pass and found something important in the DWARF/GDB unwind info. As I did not want to keep you pushing again and again, I pushed some commits, please check them out.

The issue is that the old GDB CFI was describing the JIT executor like a normal function with a real prologue. On x86_64 it was effectively telling GDB to unwind as if the code began like this:

push %rbp
mov  %rsp, %rbp

and the equivalent DWARF rule was basically:

DW_CFA_advance_loc 1          # after push %rbp
DW_CFA_def_cfa_offset 16
DW_CFA_offset rbp, -16
DW_CFA_advance_loc 3          # after mov %rsp,%rbp
DW_CFA_def_cfa_register rbp

That is a valid description for a normal C-style entry sequence, but it is not what the executor stencils actually do. The executor code keeps the frame pointer pinned across the whole region, so there is no real prologue for GDB to “walk through” instruction by instruction. If GDB stops near the start of the executor and the DWARF says “pretend the prologue already happened”, it computes the CFA from the wrong place and can read the saved frame pointer / return address from the wrong stack slots.

The new GDB-only path fixes that by describing the frame layout that is actually true while the executor is running. On x86_64 the equivalent rule is now just:

DW_CFA_def_cfa rbp, 16
DW_CFA_offset rip, -8
DW_CFA_offset rbp, -16

with no per-PC prologue simulation in the FDE. In other words, instead of telling GDB “watch a fake prologue happen”, we now tell it “this is the frame layout for this JIT region, unwind from that”. The same idea is used on AArch64 with x29 / x30.

I also validated this by hand in GDB instead of trusting only the Python test harness. I built a clean JIT-enabled tree, broke in builtin_id, finished until the selected frame was py::jit_entry:<jit>, single-stepped inside the JIT blob, and checked info frame / bt. With the new DWARF the unwind chain stayed sane and went back into _PyEval_* and PyEval_EvalCode as expected.

I also did a negative check by forcing the wrong unwind mode on purpose, and the backtrace immediately turned into garbage which is a strong sign that this change is fixing a real mismatch between the unwind metadata and the code we actually emit.

Add a shared helper that asserts exactly one py::jit_entry frame above at least one eval frame, so regressions producing duplicate JIT frames or JIT-below-eval can't pass the old tolerant regex.

diegorusso added 2 commits March 17, 2026 01:32

pythonGH-126910 jit_unwind: refactor EH frame generation

5cd7ade

pythonGH-126910 jit_unwind: add GDB JIT interface and test

669dfb9

diegorusso requested review from AA-Turner, brandtbucher, emmatyping, erlend-aasland and savannahostrowski as code owners March 17, 2026 16:36

bedevere-app bot mentioned this pull request Mar 17, 2026

Supporting stack unwinding in the JIT compiler #126910

Open

bedevere-app bot added the awaiting core review label Mar 17, 2026

diegorusso requested a review from pablogsal March 17, 2026 16:40

diegorusso and others added 4 commits March 17, 2026 18:09

Fix make smelly

255c0b3

Add __jit_debug_descriptor to ignored.tsv

ac018d6

📜🤖 Added by blurb_it.

b0bab8c

Fix check C globals

a0dff1f

diegorusso requested a review from ericsnowcurrently as a code owner March 17, 2026 20:56

diegorusso added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 18, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 18, 2026

pablogsal reviewed Mar 20, 2026

View reviewed changes

Merge branch 'main' into add-gdb-support

2b52588

diegorusso added 5 commits March 27, 2026 17:19

Address Pablo's feedback

e44170e

Fix smelly

2e40f1d

Merge branch 'main' into add-gdb-support

d890add

Strengthen JIT GDB backtrace tests

965a543

Fix x86_64 unwind

17be0a2

Add comment for inviariant and fix CFI instructions

f47d763

Fidget-Spinner reviewed Mar 31, 2026

View reviewed changes

Rename jit_executor/jit_shim to just jit_entry

67ae6cb

This because the two JIT memory regions are visible as unique frame from the GDB point of view.

pablogsal approved these changes Apr 5, 2026

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Apr 5, 2026

Address Pablo's feedback

a18cb96

Fidget-Spinner approved these changes Apr 8, 2026

View reviewed changes

Comment thread Python/jit_unwind.c Outdated

Make the mutex private

bdc8d12

pablogsal reviewed Apr 12, 2026

View reviewed changes

Address Pablo's feedback

6357698

diegorusso requested review from ZeroIntensity and markshannon as code owners April 16, 2026 13:39

pablogsal added 3 commits April 18, 2026 02:21

Tighten skip guards on test_gdb.test_jit

93bbf99

Require linux + x86_64/aarch64 + sys._jit.is_enabled() so unsupported platforms, arches, and interpreter-only tier-2 builds skip cleanly instead of hanging or failing noisily.

pablogsal force-pushed the add-gdb-support branch from 621e29a to 77777a6 Compare April 18, 2026 01:22

pablogsal approved these changes Apr 18, 2026

View reviewed changes

Assert backtrace shape in test_gdb.test_jit

a9c6315

Add a shared helper that asserts exactly one py::jit_entry frame above at least one eval frame, so regressions producing duplicate JIT frames or JIT-below-eval can't pass the old tolerant regex.

pablogsal force-pushed the add-gdb-support branch from 77777a6 to a9c6315 Compare April 18, 2026 01:34

+              #endif
+              }
+              void

		@@ -0,0 +1,30 @@
		#ifndef Py_CORE_JIT_UNWIND_H
		#define Py_CORE_JIT_UNWIND_H

Uh oh!

Conversation

diegorusso commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Mar 18, 2026

Uh oh!

pablogsal commented Mar 20, 2026

Uh oh!

pablogsal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diegorusso commented Mar 25, 2026

Uh oh!

Fidget-Spinner commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

diegorusso commented Mar 25, 2026

Uh oh!

diegorusso commented Mar 25, 2026

Uh oh!

Fidget-Spinner commented Mar 26, 2026

Uh oh!

diegorusso commented Mar 26, 2026

Uh oh!

Fidget-Spinner commented Mar 26, 2026

Uh oh!

Fidget-Spinner commented Mar 26, 2026

Uh oh!

diegorusso commented Mar 27, 2026

Uh oh!

Fidget-Spinner commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-Spinner commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-Spinner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablogsal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diegorusso commented Mar 17, 2026 •

edited

Loading

pablogsal left a comment •

edited

Loading

Fidget-Spinner commented Mar 25, 2026 •

edited

Loading

pablogsal commented Mar 25, 2026 •

edited

Loading

Fidget-Spinner commented Mar 31, 2026 •

edited

Loading

Fidget-Spinner commented Mar 31, 2026 •

edited

Loading

Fidget-Spinner left a comment •

edited

Loading

pablogsal left a comment •

edited

Loading