All google-cloud-* packages broke due to poor dependency management

[remcohaszing]

All google-cloud-* packages broke due to poor dependency management when version 0.21.0 was released.

All google-cloud-* packages depend on google-cloud-core >= <current_version>. Version 0.21.0 was a breaking change, compared to 0.20.0.

As a result, a dependency on a pinned down version is automatically broken as soon as a new version is released.

The workaround is fairly simple. If one depends on google-cloud-*, one must explicitly specify the version of google-cloud-core to depend on.

The real issue lies at dependency management of the google-cloud-* packages. Dependencies should be locked down in the packages that use them. So the requirements should be changed from google-cloud-core >= <current_version> to google-cloud-core ~= <current_version> (assuming semver is used and patch versions are non-breaking).

1. OS type and version
   Ubuntu 16.0.4
2. Python version and virtual environment information python --version

$ python --version
Python 3.5.1+

3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

$ pip list
Package                  Version    
------------------------ -----------
...
google-cloud-core        0.21.0     
google-cloud-datastore   0.20.1     
google-cloud-storage     0.20.0
...

4. Stacktrace if available

      File ".tox/docs/lib/python3.5/site-packages/google/cloud/datastore/__init__.py", line 55, in <module>
        from google.cloud.datastore.connection import Connection
      File ".tox/docs/lib/python3.5/site-packages/google/cloud/datastore/connection.py", line 23, in <module>
        from google.cloud import connection as connection_module
    ImportError: cannot import name 'connection'

5. Steps to reproduce

pip install google-cloud-datastore~=0.20.0

[daspecster]

Thanks for reporting @remcohaszing!

@dhermes can we do a 0.20.2 release that fixes this as well?

[dhermes]

Thanks for filing @remcohaszing, it was also reported yesterday (#2739).

This was an oversight on my part not putting upper bounds on the dependencies.

@daspecster I don't see much value in releasing a 0.20.x fix. The correct fix is to upgrade to 0.21.0. Maybe I am missing something here? Maybe @jonparrott can enlighten me on the dos-and-donts of fixing one's packaging mistakes after they've already hit PyPI.

---

@remcohaszing I am curious how you ended up in state where google-cloud-core was upgraded but the other packages were not.

[daspecster]

@dhermes if you make a clean venv and install pip install google-cloud-datastore~=0.20.0.
Then

from google.cloud import storage

That's how you get the error that's mentioned.
So if you're app is locked to a version and you rebuild it, it will break.

[remcohaszing]

@dhermes This state can be achieved exactly as @daspecster said.

I assume most packages use semver nowadays. This means a dependency locked as package-name ~= major.minor.patch shouldn't be breaking. This is technically handled correctly, except that the google-cloud-core dependency should be locked using google-cloud-core ~= 0.21.0 instead of google-cloud-core >= 0.21.0.

Just for clarity, this is the same as google-cloud-core >= 0.21.0, < 0.22.0.

[daspecster]

Per @dhermes,

@remcohaszing I am curious how you ended up in state where google-cloud-core was upgraded but the other packages were not?

Is there something special about your environment that keeps you from upgrading all packages?

[remcohaszing]

I don't think it's necessarily specific to my environment. I think it makes sense to lock down versions. Actually I still think this is a good idea. #2706 is a breaking change, an automated update to the new version including it has been prevented.

I have upgraded by now, but I had to do some refactors. It had become top priority, because tests were failing for unrelated patches in CI.