gh-148105: _pyrepl: switch console refresh to structured rendered screens

[pablogsal]

• Issue: _pyrepl: switch console refresh to structured rendered screens #148105

[pablogsal]

Crap I activated this copilot nonsense by mistake ...

[Copilot]

Pull request overview

This PR refactors _pyrepl screen refresh to operate on structured “rendered screens” (cells/lines + cursor) rather than raw string lists, with corresponding updates to console backends and tests.

Changes:

• Introduces structured rendering/layout primitives (RenderCell/RenderLine/RenderedScreen, LayoutMap, content fragments) and updates Reader to produce/compose them.
• Rewrites Unix/Windows console refresh() paths to diff/apply RenderedScreen updates via explicit refresh plans.
• Updates tests and ancillary behavior (e.g., control-char display escaping and history-file NUL sanitization) to match the new rendering model.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Lib/_pyrepl/render.py	New rendering model (cells/lines/screens), diffing, and line-update primitives.
Lib/_pyrepl/layout.py	New layout mapping between buffer positions and rendered rows/columns.
Lib/_pyrepl/content.py	New prompt/body content modeling and buffer-to-fragment conversion.
Lib/_pyrepl/reader.py	Major refactor: invalidation model, rendered screen composition, and layout-driven cursor mapping.
Lib/_pyrepl/unix_console.py	Refresh rewritten to plan/apply structured line updates; fixes raw accumulation bug.
Lib/_pyrepl/windows_console.py	Refresh rewritten to plan/apply structured line updates with rendered state sync.
Lib/_pyrepl/console.py	Console API now refreshes with RenderedScreen; adds rendered-state sync + redraw visualization.
Lib/_pyrepl/utils.py	Adds display-char iterator/control escaping helpers; refactors disp_str; adds StyleRef.
Lib/_pyrepl/types.py	Expands shared type aliases (cursor/dimensions/keymap/event data).
Lib/_pyrepl/trace.py	Adds trace_text() helper for safer trace output.
Lib/_pyrepl/readline.py	Sanitizes embedded NULs when reading history; updates overlay invalidation hooks.
Lib/_pyrepl/commands.py	Converts “dirty” flag usage to targeted invalidations; updates screen sync behavior.
Lib/_pyrepl/completing_reader.py	Moves completions menu to overlay-based composition; updates invalidations.
Lib/_pyrepl/historical_reader.py	Updates history operations to invalidate buffer/prompt appropriately.
Lib/_pyrepl/simple_interact.py	Switches interrupt path to full invalidation.
Lib/_pyrepl/input.py	Cleans up TYPE_CHECKING guard for type-only imports.
Lib/test/test_pyrepl/support.py	Test utilities updated for rendered-line equality and new console refresh signature.
Lib/test/test_pyrepl/test_reader.py	Adds/adjusts tests for control-byte escaping, zero-width behavior, and layout mapping.
Lib/test/test_pyrepl/test_unix_console.py	Adds regression test for colorized multiline typing redraw behavior.
Lib/test/test_pyrepl/test_windows_console.py	Updates expected cursor movement/write patterns due to new refresh diffing.
Lib/test/test_pyrepl/test_pyrepl.py	Adds history/NUL sanitization test and multiline history navigation behavior updates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[johnslavik]

I like the new easter egg rainbow mode 🌈.

I'll be spelunking for a while, but here's what I found on the spot.

[chris-eibl]

See also #145886:

Event.raw isn't read anywhere and is untested.

[johnslavik]

@pablogsal

[chris-eibl]

Yupp, it is dead code AFAICT - see the linked issue.

[chris-eibl]

Great work - Thanks!

Wasn't able to do a detailed review, yet, but played a little bit with it.

When pasting on Windows, the lines are postfixed with ^M for me.
Example:

class foo:
    pass

Pasting result using Windows terminal

>>> class foo:^M
...     pass^M
...

Happens also using the legacy console in paste mode.

[johnslavik]

I'll check the Windows issue. Filed pablogsal#123 for more docs, lmk if you'd like this!

[johnslavik]

On Windows I found a problem with history paging exit:
https://github.com/user-attachments/assets/5d0960dc-c2ff-4f30-9f64-063aa908df88

Clearing seems to be a fix on Windows, not sure it makes sense on Unix yet. NVM, this breaks Unix.

Patch

diff --git a/Lib/_pyrepl/commands.py b/Lib/_pyrepl/commands.py
index 90bafa288c0..744a1be71f6 100644
--- a/Lib/_pyrepl/commands.py
+++ b/Lib/_pyrepl/commands.py
@@ -493,9 +493,10 @@ def do(self) -> None:
 
         # We need to copy over the state so that it's consistent between
         # console and reader, and console does not overwrite/append stuff
+        self.reader.invalidate_prompt()
         trace("command.show_history sync_rendered_screen")
         self.reader.console.sync_rendered_screen(
-            self.reader.rendered_screen,
+            RenderedScreen.empty(),
             self.reader.cxy,
         )

[pablogsal]

i pt a short commit with fixes. I think this should fix both Windows issues:

• Paste ^M: ch += "\n" → ch = "\n" to replace \r instead of concatenating. Thanks @johnslavik and @chris-eibl.

• I replaced sync_rendered_screen(old_state) with console.clear() + invalidate_full(). Uses clear() instead of syncing with empty screen to avoid the grow_lines issue on Unix that @johnslavik found.

Con you pls check this fixed the issue?

[johnslavik]

Great! CRLF fixed.

To fix later

Do we want to make sure leading lines stay rendered in history search mode?
Same for paste mode

Screen.Recording.2026-04-08.at.13.17.46.mov

I think in get_prompt we'd have to examine ps1 again for leading lines?
Hmm, maybe this can be fixed later, I think it might be out of scope

[johnslavik]

• I replaced sync_rendered_screen(old_state) with console.clear() + invalidate_full(). Uses clear() instead of syncing with empty screen to avoid the grow_lines issue on Unix that @johnslavik found.

I tested it on Windows and ran into some other, new issues (discussed privately), I gave it a shot and history mode exiting should be fully fixed by pablogsal#124.

[johnslavik]

LGTM. Thanks everyone for working together through this!

[chris-eibl]

Great! CRLF fixed.

Yupp, works for me, too 👍

[pablogsal]

Thanks a lot everyone for the help and the reviews. You rock 🤘