blob.upload_from_filename blocks for upwards of an hour.

[tweeter0830]

I am using the following code to upload files to GCS.

blob = bucket.blob(gcs_filepath)
logging.info('Uploading file "%s" to "%s".', filepath, gcs_filepath)
blob.upload_from_filename(filepath)
logging.info('Uploaded file "%s" to "%s".', filepath, gcs_filepath)

This works most of the time. However, it has been stuck uploading for the last hour and has not raised an exception yet. It seems like this method should have a timeout or something for situations like this. It would also be helpful if it had the option to log the progress of the method for further diagnosis.

[tseaver]

Duplicate of #3442, which has more information.

[tseaver]

Hmm, not a duplicate.

[dhermes]

@tseaver @tweeter0830 It may be somewhat of a duplicate if the real answer is that httplib2 + threading == a bad time.

[tweeter0830]

I am using this in a thread.

[tseaver]

@tweeter0830 Or switch to using a separate Client instance per thread (they should be relatively cheap to construct).

[dhermes]

@tweeter0830 I'm closing this out as that is the culprit. As @tseaver said, you'll want Client instances to be thread local (for now). We will soon be dumping httplib2 altogether (for requests).

[tweeter0830]

I moved the client into a single thread and I am still having this issue.

[dhermes]

@tweeter0830 Can you provide some example code so we can discuss the implementation?

[GEverding]

I'm seeing similar issues. I'm creating new Client Objects each time I want to upload a file.

client = storage.Client()
bucket = client.get_bucket(config.STORAGE_BUCKET)
blob = bucket.blob(out_file)
blob.upload_from_filename(in_file)
blob.acl.all().grant_read()
blob.acl.save()
blob.content_disposition = 'attachment; filename="' + name + '.mp4"'
blob.patch()
public_url = blob.public_url

This is running in a Celery Task.

[dhermes]

Thanks for the snippet @GEverding. I'm not a regular user of Celery so I'm not sure what the implications are here. Are you saying you put this in a Python module at global scope and then point Celery at that module?