Split large Objects/unicodeobject.c file into smaller files

[vstinner]

The Objects/unicodeobject.c file is made of 16,683 lines of C code. It's the 4th largest file in the Python code base:

   38334 ./Parser/parser.c
   18485 ./Python/Python-ast.c
   18304 ./Modules/posixmodule.c
   16683 ./Objects/unicodeobject.c

parser.c and Python-ast.c are generated C files.

I propose to split this file into smaller files to ease its maintenance and to allow building it in parallel.

Linked PRs

• gh-139353: Split unicodeobject.c into smaller files #139354
• gh-139353: Add Objects/unicode_format.c file #139491
• gh-139353: Rename formatter_unicode.c to unicode_formatter.c #139723
• gh-139353: Add Objects/unicode_writer.c file #139911
• gh-139353: Add Objects/unicode_codecs.c file #141469
• gh-139353: Add Objects/unicode_codecs_utf.c file #142190

[ZeroIntensity]

I think it would make sense to do this for some other objects too. For example, dictobject.c (~8k lines) and typeobject.c (~12k lines) are both pretty huge and would benefit from being split up.

[vstinner]

unicodeobject.c is a good candidate since there are functions which can be grouped together and make sense as groups (IMO).

[serhiy-storchaka]

You already proposed this idea several years ago, and I like it, but it was not generally supported. It was considered a code churn. So you need to bring this idea up for public discussion again to overturn the old consensus.

Semantically, it could be split on the three parts -- low-level internals, codecs and the str methods. The problem is that they are not independent. The higher level code needs access to low level code, implemented as inline functions or macros, for performance. PyUnicode_FromString() shares code with the UTF-8 codec, PyUnicode_AsUCS4(), PyUnicode_FromKindAndData(), etc share code with UTF-16/32 codecs. Having them in a single file helps to generate optimal code.

Extracting the UTF-7, "unicode-escape" and "raw-unicode-escape" codecs in separate files is the safest action, as they are not performance critical, but this will not bring much.

[vstinner]

Semantically, it could be split on the three parts -- low-level internals, codecs and the str methods. The problem is that they are not independent. The higher level code needs access to low level code, implemented as inline functions or macros, for performance. PyUnicode_FromString() shares code with the UTF-8 codec, PyUnicode_AsUCS4(), PyUnicode_FromKindAndData(), etc share code with UTF-16/32 codecs. Having them in a single file helps to generate optimal code.

LTO+PGO compilation should avoid or reduce this problem. How can I check if the function are affected by a split? Run micro-benchmarks?

[StanFromIreland]

I came across this today, splitting will also allow for GitHub search to search it, see: https://discuss.python.org/t/github-code-search-skips-9-cpython-source-files-because-they-are-large/69933

[vstinner]

You already proposed this idea several years ago, and I like it, but it was not generally supported. It was considered a code churn. So you need to bring this idea up for public discussion again to overturn the old consensus.

After many attempts, I found the old discussion! A python-dev email thread of 2012: https://mail.python.org/archives/list/python-dev@python.org/thread/GV2I2W6MBRNAQP3H2GWD7L4X4UWMBF3X/ Summary of this email thread.

Arguments to split:

• Hard to navigate.
• Maintenance burden: hard to add new code and to fix bugs.
• Opening the file in a web browser does crash the browser.

Arguments against splitting:

• Backports need more work.
• Unknown impact on performance: a large single C file helps inlining. Is it an issue when using LTO+PGO?
• Harder to maintain (according to Martin).
• Harder to navigate: Martin prefers to have all functions in a single C file.

[vstinner]

So you need to bring this idea up for public discussion again to overturn the old consensus.

Ok, I created https://discuss.python.org/t/split-large-objects-unicodeobject-c-file-into-smaller-files/104086 to discuss this split.