[Guidance] A way to specify SAAS provided dependencies

[Paulskit]

Hello @DavidVujic,
Thank you for your library, it seems to tick all the boxes for python monorepo I'm trying to build!

I have the following use-case which I'm not sure how to handle properly:
I've got a set of AWS Glue jobs that use the custom python library (built in python-polylith) to unify some common operations. This library has some dependencies that are available natively only in the execution environment, but is not distributed as a package.
When project from my bricks and run poly check command is run, I get the following error:
🤔 Cannot locate awsglue in <project_name>
I understand that in order to avoid the error I have to specify an implicit dependency, which exists only in the actual execution environment. However, looking at the documentation, I can't really find a way to do this.
Can you please advise what would be the best way to solve this?

[DavidVujic]

Hello @Paulskit! I'm happy to hear that you like Polylith 😄

This is a scenario that I haven't thought of before. I think what would be needed in the check command is an --ignore option. I think it should be a fairly straight forward thing to implement, and I will have a closer look at it.

Meanwhile, if it is a package that can be installed from PyPI, I think a workaround could be to add it as a dev-dependency.

Otherwise, there is a hack you could do and that is telling the check command that awsglue is an alias for an existing package already. I know, that's ugly, but it should prevent the check command from exiting with an error code, if it's blocking you in a CI environment.

Example poly check --alias an-existing-dependency=aws_glue.

[DavidVujic]

A follow-up question: how do you install the aws glue library? I had a quick look, but didn't understand it fully. It looked like it can be added as a dependency via github + and then have the OS-specifics also added. If I got that right, would it be added as a Python depencency, from github instead of pypi? Otherwise, wouldn't your code editor and linter(s) also highlight about the unknown dependency?

[Paulskit]

Hey @DavidVujic.

I'm using one of the third-party packages of the mentioned repository

[tool.hatch.envs.default]
dependencies = [
  "awsglue-local-dev==1.0.0",
]

Right now, I have applied the workaround in the project as

It did the trick, but can't say I'm happy with it.
It would be great if implicit dependencies can be somehow put into the pyproject.toml file. Everything in 1 place and a reusable command for all builds out there, just hatch build -t wheel

[DavidVujic]

Ok, that makes sense!

I got an idea that might solve this: having the poly check command take an optional dependency into account, such as dependencies specified in tool.hatch.envs.local. If I understand Hatch correctly, you could pick a specific environment to create in a production environment, and then skip the optionals.

I will give this a try.

[DavidVujic]

Just to double check: for production, you aren't installing any python package for this library? Even if there is an import awsgluein there, right? I wonder how that works, this scenario I haven't experienced yet.

[Paulskit]

Right, for AWS Glue there is a way to provide dependency as a package.
‘--additional-python-modules s3://aws-glue-native-spark/tests/j4.2/ephem-3.7.7.1-cp37-cp37m-linux_x86_64.whl’
See here: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#addl-python-modules-support

If I had to install it the normal way, obviously my workaround wouldn’t work because of missing package. That’s why I don’t like it.

[Paulskit]

I might have misunderstood the question.
‘awsglue’ lib is provided by the runtime. You don’t have to do anything to install it.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python.html

[DavidVujic]

It did the trick, but can't say I'm happy with it. It would be great if implicit dependencies can be somehow put into the pyproject.toml file. Everything in 1 place and a reusable command for all builds out there, just hatch build -t wheel

Will hatch build a proper wheel with that added? I would expect a build failure.

Edit: tried it out and it will actually build a wheel. But the install of that wheel will fail.

My suggestion would to use the --alias feature to workaround this. The option is available for both poly check and poly libs.

Example workaround with the tool as-is today:

hatch run poly check --alias boto3=awsglue

hatch run poly libs --alias boto3=awsglue

I will try out the different possibilities that has been mentioned here: optional dependencies, ignore options or being able to tell the tool about the implicit dependencies.

[DavidVujic]

Hello again @Paulskit!

I have published new versions of the Polylith tools (the CLI version is 1.6.0), that will include any optional dependencies added in the projects. I think this could be a solution for your use case.

If you would specify the awsglue-local-dev dependency there, it will be included when running the check and libs command.

Example:

# add this for every project that need the dependency
[project.optional-dependencies]
local-dev = ["awsglue-local-dev==1.0.0"]

Since the "cutoff" to match dependencies is fairly strict, you still would need to use the --alias option. But now it will make sense, and you won't need to workaround it as before using some other dependency.

Usage:

# check
hatch run poly check --alias awsglue-local-dev=awsglue

# libs
hatch run poly libs --alias awsglue-local-dev=awsglue

I will close this issue as completed. Please reopen, or create a new issue if you have any issues! 🙏

Thank you again for reporting about this thing. By doing that, you have helped me developing a better tool - much appreciated!

[Paulskit]

This is fast! Thank you for such a rapid turnaround, I didn't expect it.