Skip to content

[Documentation] Add test coverage guidance for integration tests #967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jan-janssen merged 1 commit into from
Apr 16, 2026
Merged

[Documentation] Add test coverage guidance for integration tests #967

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ as hierarchical job scheduler within the allocations.
* [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
* [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)
* [Message Passing Interface](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#message-passing-interface)
* [Test Coverage for Integration Tests](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#test-coverage-for-integration-tests)
* [Python Version](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#python-version)
* [Resource Dictionary](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#resource-dictionary)
* [SSH Connection](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#ssh-connection)
Expand Down
19 changes: 19 additions & 0 deletions docs/trouble_shooting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,25 @@ Typical error messages related to missing dependencies are `ModuleNotFoundError`
* `ModuleNotFoundError: No module named 'h5py'` - Install [h5py](https://www.h5py.org/) as explained in the [Caching](https://executorlib.readthedocs.io/en/latest/installation.html#caching) section of the installation.
* `ModuleNotFoundError: No module named 'networkx'` - Install [networkx](https://networkx.org/) as explained in the [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation) section of the installation.

## Test Coverage for Integration Tests
When Python functions are executed with executorlib, they run in subprocesses started via `sys.executable`. As a result, `coverage run`
only tracks the main test process by default and can miss function execution inside executorlib workers.

To collect coverage from both the main process and executorlib subprocesses, enable the subprocess patch in your project configuration:
```toml
[tool.coverage.run]
Comment on lines +38 to +40
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The coverage config snippet enables the subprocess patch, but it doesn’t mention enabling parallel data files. Without parallel = true (or an equivalent mechanism), multiple worker processes can end up writing to the same .coverage data file and coverage combine may be ineffective. Consider adding parallel = true to the [tool.coverage.run] example alongside patch = ["subprocess"].

Suggested change
To collect coverage from both the main process and executorlib subprocesses, enable the subprocess patch in your project configuration:
```toml
[tool.coverage.run]
To collect coverage from both the main process and executorlib subprocesses, enable the subprocess patch and parallel data files in your project configuration:
```toml
[tool.coverage.run]
parallel = true

Copilot uses AI. Check for mistakes.
patch = ["subprocess"]
```
Comment on lines +39 to +42
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Does coverage.py support a "patch" configuration option for subprocess tracking? What is the correct configuration to track subprocess execution in coverage.py?

💡 Result:

Yes, coverage.py supports a "patch" configuration option under the [run] section for subprocess tracking. The correct configuration to track subprocess execution is: [patch] subprocess = true This automatically sets parallel = true, so use coverage combine before reporting. It works for processes created with subprocess, os.system, or execv/spawnv functions (ensuring COVERAGE_PROCESS_START is passed for env variants). Example .coveragerc: [run] patch = subprocess Then: coverage run ..., coverage combine, coverage report. For multiprocessing, use concurrency = multiprocessing instead.

Citations:

Fix TOML syntax for subprocess tracking configuration.

The correct coverage.py configuration is patch = subprocess (not a list). Change line 41 to:

[tool.coverage.run]
patch = subprocess

The patch option should be a string value, not an array. This enables coverage tracking for subprocesses created via the subprocess module.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/trouble_shooting.md` around lines 39 - 42, In the [tool.coverage.run]
block update the patch setting to a string instead of an array: replace the
current patch = ["subprocess"] with patch = subprocess so the coverage
configuration uses the string-valued patch key (i.e., change the patch entry
under [tool.coverage.run] from an array to the bare subprocess value).


Then execute:
```bash
coverage run -m unittest discover
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command example uses unittest discover without specifying a start directory. In this repo’s own coverage configuration (pyproject.toml), discovery is run with an explicit tests start directory; making the docs example similarly explicit (e.g., unittest discover tests / -s tests) avoids accidentally discovering non-test modules when invoked from a different working directory.

Suggested change
coverage run -m unittest discover
coverage run -m unittest discover tests

Copilot uses AI. Check for mistakes.
coverage combine
coverage report
```

The `coverage combine` command merges the data from the main process and subprocesses.

## Python Version
Executorlib supports all current Python version ranging from 3.9 to 3.13. Still some of the dependencies and especially
the [flux](http://flux-framework.org) job scheduler are currently limited to Python 3.12 and below. Consequently for high
Expand Down
Loading