Ability to ignore type dependencies in `poly check`

[davclark]

Is your feature request related to a problem? Please describe.
I would ideally not need to customize the poly check command to ignore type dependencies. These are easy to fence in regular code like so, which allows installation in your test env, but not your production / default env:

if TYPE_CHECKING:
    from mypy_boto3_s3.service_resource import Object as S3Object

However, with that code in my default build, poly check yields 🤔 Cannot locate mypy_boto3_s3 in my_project.

Describe the solution you'd like
While I can use an alias as described in #206, it would be nice if we could ignore dependencies that are guarded by if typing.TYPE_CHECKING

I'd be happy to chip in on this, but I'd love a brief pointer to understanding how the existing code parses out imports. I'm somehow not finding it, but it's rather cool and I'd love to understand it better.

Describe alternatives you've considered
poly check --alias boto3=mypy_boto3_s3 works, it's just not as nice of an OOtB experience.

Obviously this is not hard - but as a tool to help drive large projects towards better SW eng practices, it would be great for polylith to have a seamless interaction with common patterns for mypy (or perhaps ty if you're up on the latest, which I most certainly am not).

[ferreirapmartin]

Is your feature request related to a problem? Please describe. I would ideally not need to customize the poly check command to ignore type dependencies. These are easy to fence in regular code like so, which allows installation in your test env, but not your production / default env:

if TYPE_CHECKING:
from mypy_boto3_s3.service_resource import Object as S3Object
However, with that code in my default build, poly check yields 🤔 Cannot locate mypy_boto3_s3 in my_project.

Describe the solution you'd like While I can use an alias as described in #206, it would be nice if we could ignore dependencies that are guarded by if typing.TYPE_CHECKING

I'd be happy to chip in on this, but I'd love a brief pointer to understanding how the existing code parses out imports. I'm somehow not finding it, but it's rather cool and I'd love to understand it better.

Describe alternatives you've considered poly check --alias boto3=mypy_boto3_s3 works, it's just not as nice of an OOtB experience.

Obviously this is not hard - but as a tool to help drive large projects towards better SW eng practices, it would be great for polylith to have a seamless interaction with common patterns for mypy (or perhaps ty if you're up on the latest, which I most certainly am not).

Same issue here.

[DavidVujic]

Is this because the mypy_boto3_s3 is not listed in the dependencies of the project?

I'm thinking that it could be solved by adding it to the dev dependencies of the relevant project(s) (just as the root-pyproject adds it). If you use lock-files in the projects, make sure they are updated accordingly too. The lock-file will tag the dependency in a "dev" group.

I think that the poly check command does the correct thing here, since it cannot find the dependency. Maybe the command could lookup if the TYPE_CHECKING is wrapping the dependency, but I think that requires some more advanced AST parsing. In this case, I would recommend adding dev-dependencies to the projects. Would that work? @davclark @ferreirapmartin

[davclark]

Maybe I am setting up the project wrong...

in the project's pyproject.toml, I have the following, with boto3-stubs[s3] in tool.hatch.envs.test. Is this different than what you are proposing? Like, do I just need to call them "dev" instead of "test"?

dependencies = [
    "boto3==1.34.101",
    ...
]

[tool.hatch.envs.default]
dependencies = [
    "hatch-polylith-bricks==1.2.6",
    "polylith-cli==1.17.0"
]

[tool.hatch.envs.test]
dependencies = [
    "hatch-polylith-bricks==1.2.6",
    "polylith-cli==1.17.0",

    "boto3-stubs[s3]==1.38.18",
    ...
]

I likewise have the stubs included in the top level (development) pyproject.toml, i.e. just:

dependencies = [
    "boto3-stubs[s3]==1.38.18",
    ...
]

[ferreirapmartin]

Hi @DavidVujic,

In my case, using UV, adding a dependency to dependency-groups.dev only works correctly if I generate the uv.lock file from the project. According to the documentation, this shouldn't be necessary with UV (only with Poetry), but otherwise the dependency is reported as missing.

Although this might not be the ideal thread to mention it, I’ve also encountered a similar issue with optional dependencies, specifically in the case of OpenTelemetry. For example:

def get_resource() -> Resource:
    attrs = {ResourceAttributes.SERVICE_NAME: os.getenv("OTEL_SERVICE_NAME", "default")}

    try:
        from openinference.semconv.resource import ResourceAttributes as OpenInferenceResourceAttributes

        attrs[OpenInferenceResourceAttributes.PROJECT_NAME] = os.getenv("OTEL_SERVICE_NAME", "default")
    except ImportError:
        pass
    return Resource.create(attrs)

In this case, it still reports openinference as a required dependency.

It would be really helpful to have a way to configure exclusions or ignored dependencies at the component/base level, similar to how aliases are configured.

Best regards,
Martín.

[DavidVujic]

Maybe there's something not working as expected in the poly command, I think your setup looks correct @davclark. I'll have a look!

[DavidVujic]

I will have a look at your use case too, it should pick up those dependencies but maybe there's something missing in the poly tool @ferreirapmartin

[DavidVujic]

I'm sorry, I mislead you with the dev-dependency suggestion! I forgot that the tool is parsing the dependendencies (and not the dev dependencies) by intention - to not have false-positives when adding a dependency.

I'll give this some thought and will try some ideas out!

[DavidVujic]

There's a way today to resolve this by using optional-dependencies in the projects that use the component:

# add this for every project that need the dependency
[project.optional-dependencies]
local-dev = ["boto3-stubs[s3]>=1.38.20"]

Maybe not optimal, but a good enough workaround and alternative to the alias option?

[DavidVujic]

Is the above a good-enough solution @davclark @ferreirapmartin?

[DavidVujic]

I have added a new section to the docs - Tips & Tricks - with the suggested solution for the TYPE_CHECKING constant (or any other optional dependency): https://davidvujic.github.io/python-polylith-docs/tips/

[ferreirapmartin]

Is the above a good-enough solution @davclark @ferreirapmartin?

Hi, in my case the workaround I used was to add:

[dependency-groups]
polylith-workaround = ["types_aiobotocore_sqs"]

Additionally, I had to generate the uv.lock file for each project; otherwise, the error persisted in the same way.

Thanks, best regards.

[DavidVujic]

Thanks for letting me know @ferreirapmartin!

I am curious if the suggested solution would make it not necessary to re-generate the lock files?

# add this for every project that need the dependency
[project.optional-dependencies]
local-dev = ["types_aiobotocore_sqs"]

EDIT: maybe it will need to, when having a setup with lock files in the projects (and not only the "uv workspace" root-level lock file only).

[davclark]

You've been so responsive, and I've taken so long to get back that I wanted to drop a quick note and let you know that this is in my queue, but just haven't had a chance to get to it - as you might guess, we are in the midst of a big refactor :)

[DavidVujic]

No worries @davclark! I would happily help you if it's something related to Polylith.

[DavidVujic]

I'll close this one, please reopen or create a new issue if there's things missing in Polylith 🙏