[Documentation] test coverage for integration tests that use `executorlib`

[ltalirz]

Hi, in glasagent/amorphouspy#292 I am running an integration test that uses executorlib and I am trying to figure out how I can get it to record also the coverage of the function executed via executorlib, i.e. after going through pickling/unpickling etc. (so far unsuccessfully).

Is there some mechanism (e.g. some special executor class) that I can use to achieve this?

[jan-janssen]

Inside the executorlib package we achieve the coverage by testing the corresponding functions individually.

Internally, executorlib calls sys.executable to start the subprocesses which returns the Python executable which was used to start the test but ignores the coverage command before:
https://github.com/pyiron/executorlib/blob/main/src/executorlib/standalone/command.py

Did you try the coverage patch for subprocesses: https://coverage.readthedocs.io/en/latest/config.html#config-run-patch ? As far as I understand this is exactly designed to address this issue.

[jan-janssen]

Let's take the following example, starting with mymodule.py:

def sum_function(a, b):
    return a + b

And a test for this module with executorlib named test.py:

import unittest
import executorlib

from mymodule import sum_function


class TestSumFunction(unittest.TestCase):
    def test_sum_function(self):
        with executorlib.SingleNodeExecutor(hostname_localhost=True) as exe:
            future = exe.submit(sum_function, 2, 3)
        self.assertEqual(future.result(), 5)

When this is evaluated directly:

coverage run -m unittest discover
coverage report

The test coverage for mymodule.py is reported at 50%. Basically because the function is only imported but the evaluation of the function is not tracked.

To address this limitation we add the following line in the pyproject.toml:

[tool.coverage.run]
patch = ["subprocess"]

Afterwards we can execute the tests again using:

coverage run -m unittest discover
coverage combine
coverage report

The combine command is required to merge the coverage from both the main process and the subprocess. Afterwards the coverage is correctly reported as 100% for the Python module mymodule.py.

[ltalirz]

Thanks a lot for the quick feedback - it was very helpful to know that it should work with the SingleNodeExecutor. I'll ask another question in the PR

[jan-janssen]

Let's keep the issue open, just as a reminder for myself to add a corresponding section to the documentation.

[jan-janssen]

This was added in #967