Tell if blob is a directory?

[jagooding]

How can you tell if a blob is a directory or file? I know with the old cloud storage API you could just do stat.is_dir, but I don't see any equivalent functionality here?

[tseaver]

There is no notion of a "directory": a blob/object is always a file. The back-end API does not permit creating "folder"/"directory" objects within a bucket: you and emulate (somewhat) a hierarchy by using a separator in object names (see the "Object names" section of https://cloud.google.com/storage/docs/concepts-techniques#concepts).

[jagooding]

I sorely beg to differ that there is no notion of a directory. The google developer console itself allows you to create "folders", and the app-engine-gcs library which is also maintained by google provides directory emulation.

I wanted to switch over to this library because it seems to be what the documentation is pushing and I prefer the way it handles files compared to the gcs-client library. It would be nice if this library incorporated some directory listing / stat abilities of the old library.

My end goal is to use gcloud-python to write a custom GoogleStorage class for Django. This was easy to do with the old library because of the .is_dir function.

In fact there are many python standard library functions and functions of many third party libraries that require the notion of directories. It would seem to make sense to me that the ability is incorporated into the library.

What is the end goal for google api with python for cloud storage? Is the old library going to be maintained? If so, should I just continue using that one?

[jgeewax]

From the perspective of the API

The UI "does" folders in the sense that it splits filenames on / characters (and has the concept of "creating a folder"), but it's really "folder fiction" in the sense that the API itself doesn't actually have any concept of a "folder".

The definition of is_dir would be:

def is_dir():
  for f in get_files(prefix='/myfoldername/'):
    return True
  return False

That is... if there is an object called "/home/jj/file.txt" then "/home/jj" is a "folder". But you could create an object named "/home/jj/file.txt" wihtout creating the "folders" called "/home" or "/home/jj" -- which doesn't quite line up with the way Unix thinks of "directories" (/"folders")

With a bit of magic

This rule-based approach stands in contrast to the way many tools work, which create objects to mark the existence of folders (such as “dir_$folder$”)

• https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork

In other words, we may just be able to check for the existence of this -- which would correspond to "the UI created this folder".

[tseaver]

An app can emulate folders by creating blobs with embedded separators (e.g., /) in their names, and search for blobs "within" the emulated folder using the prefix argument to Bucket.list_blobs.

However, just like the little man upon the stair, there isn't any "folder" object in the back-end whose name is just the prefix, terminated by the /.

[jgeewax]

It would be nice if this library incorporated some directory listing / stat abilities of the old library

I think we can probably help with this but need a bit more detail. The basic gist here is "GCS technically does not support folders, and a library that did that was 'faking it' for you."

The "faking" is effectively:

• A path is a folder if it ends in /
• You can ls anything (worst case you'll get nothing back)
• ls'ing a directory is basically a prefix search (give me everything that starts with "/foo/bar/")
• ls'ing via prefix actually includes subdirectories as well! (ie, /foo/bar/baz/file.txt will be in the ls for /foo/bar)
  • If you want just the immediate directory, you need to exclude anything containing more slashes (I forgot how to do this one, I think there's a separator attribute?)

So -- if you give us some more detail about what exactly we're shooting for here, maybe we can figure out what methods make sense and how to fake this folder stuff in a consistent way (it'd be bad if we did it a way that wasn't quite right).

[jagooding]

@jgeewax thanks for considering this.

In my mind the most straightforward thing to shoot for here is to duplicate the is_dir property from the old GCSFileStat class onto the blob / object of gcloud-python.

I'll give you an example of how I would use it, so that you get an idea of intent behind my meaning.

Here is the default FileSystemStorage class of django based on the abstract Storage class:

def listdir(self, path):
        path = self.path(path)
        directories, files = [], []
        for entry in os.listdir(path):
            if os.path.isdir(os.path.join(path, entry)):
                directories.append(entry)
            else:
                files.append(entry)
        return directories, files

And here is a custom GoogleStorage class, again based on the default Storage using the old library emulating the same functionality.

def listdir(self, path):
        directories, files = [], []

        for stat in cloudstorage.listbucket(self.bucket + '/' + path, delimiter='/'):
            logger.debug(stat)
            if stat.is_dir:
                directories.append(stat.filename)
            else:
                files.append(stat.filename)

        return directories, files

Now, I think that the .is_dir functionality assumes that if it ends in a / then it is a directory. I think this would fit the majority of use cases. I can also imagine some use cases where it would be nice to have a generator yield the directories so that you can recursively walk through the "file system".

maybe we can figure out what methods make sense and how to fake this folder stuff in a consistent way (it'd be bad if we did it a way that wasn't quite right).

I agree with it being consistent, I'm also a proponent of Kenneth Reitz's spirit of API's should be simple and fit 95% of use cases, instead of being overtly technical to fit 100% of use cases. The requests library vs the python standard library for http requests as a case in point.

[akashgangil]

It makes a lot of sense to have this in the gcs client library. The concept of folders is important as the information within a bucket is organized in folder structure. Yes, the underlying implementing might lay it down as a flat model, but this doesn't help from a API consumer point of view. Please consider raising its priority.

thanks!

[lukesneeringer]

I am going to close this as a duplicate of #920 (which I am keeping open). They are not identical but cover the same territory.

[debuggger]

Not sure if this has been already discussed, but I do see that creating a "folder" from the developer console actually makes a difference, in that there is a blob object corresponding the folder created.
For eg: I start with a bucket with no files/object and I do the following:
gsutil cp file gs://my-bucket/dir1/
No when I list recursively with blank prefix, the following objects is/are listed:
dir1/file1
However when I use the Storage.BlobListOption.currentDirectory option(this is non-recursive listing) with blank prefix, I get:
dir/ (The result is returned as a Blob, but when I try to explicitly get bucket.get("dir1/") I get a null).
Once I use create folder option from the console and create a directory dir1 (mind you dir1/file1 already exists at this point) and when I list , I get the following:
dir1/
dir1/file1
Though I haven't found any other way of creating a folder outside of console, including gsutil
I thought it is worth to point this out here.