KeyProperties returned from single entity GET have bad data

[epluntze]

I'm seeing this in my flask application and ipython when looking up entities by their Key or id

the MailboxxOrder model has 2 KeyProperties: user and recipient both defined as User model entites

class MailboxxOrder(ndb.Model):
    """ Model for mailboxx orders. """

    _use_cache = True
    _use_global_cache = True

    user = ndb.KeyProperty(kind='User', required=True)
    recipient = ndb.KeyProperty(kind='User', required=True)

When querying for the entities by id, NDB returns None in many cases for both properties
When values are returned both properties have the same value, even though the entity actually has different keys for both properties

shown here using NDB python 2.7

Code example python 2.7

In [3]: entity = MailboxxOrder.get_by_id(6678619968503808)

In [4]: entity.user
Out[4]: Key('User', 5129160508833792)

In [5]: entity.recipient
Out[5]: Key('User', 5127556472766464)

Code example python 3.7

In [5]: for x in range(0, 20):
   ...:     print('------')
   ...:     with client.context():
   ...:         order = MailboxxOrder.get_by_id(6678619968503808, use_cache=False, use_memcache=False)
   ...:         print(f'user: {order.user}')
   ...:         print(f'recipient: {order.recipient}')
   ...:         print('-----')
   ...:
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: None
recipient: None
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: None
recipient: None
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: None
recipient: None
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: None
recipient: None
-----
------
user: Key('User', 5127556472766464)
recipient: Key('User', 5127556472766464)
-----
------
user: None
recipient: None
-----

[chrisrossi]

Thanks for reporting this. I'm taking a look. I've done a test and if I create the entity in NDB Python 3.7, I don't see this problem. I suspect it is something to do with data written by the old NDB. Have you noticed whether this is a problem only for data written by the legacy NDB or if it can also happen when data is written by this version of NDB?

[epluntze]

I'm still seeing issues with this but I do have more info:

In [1]: from google.cloud import ndb

In [2]: client = ndb.Client()

In [3]: class TestUserModel(ndb.Model):
   ...:
   ...:     _use_cache = True
   ...:     _use_global_cache = True
   ...:
   ...:     stuff = ndb.StringProperty(required=True)
   ...:

In [4]: new_user_1 = TestUserModel(stuff='stuff')

In [5]: new_user_2 = TestUserModel(stuff='stuff2')

In [6]: class ModelTest(ndb.Model):
   ...:
   ...:     _use_cache = True
   ...:     _use_global_cache = True
   ...:
   ...:     thing_1 = ndb.KeyProperty(kind='TestUserModel', required=True)
   ...:     thing_2 = ndb.KeyProperty(kind='TestUserModel', required=True)
   ...:

In [7]: with client.context():
   ...:     test_user_key1 = new_user_1.put()
   ...:     test_user_key2 = new_user_2.put()
   ...:     print(test_user_key1)
   ...:     print(test_user_key2)
   ...:
Key('TestUserModel', 5555382068969472)
Key('TestUserModel', 5516255185338368)

In [8]: with client.context():
   ...:     new_test = ModelTest(thing_1=ndb.Key('TestUserModel', 5555382068969472), thing_2=ndb.Key('TestUserModel',
   ...: 5516255185338368))
   ...:     new_key = new_test.put()
   ...:     print(new_key)
   ...:
Key('ModelTest', 5568786443796480)

In [11]: with client.context():
    ...:     this = ModelTest.get_by_id(5568786443796480)
    ...:     print(this)
    ...:
    ...:
ModelTest(key=Key('ModelTest', 5568786443796480), thing_2=Key('TestUserModel', 5516255185338368))

it seems that when I create a model with 2 key properties of the same Kind the datastore is only saving one value

and when the entity is viewed in the cloud console the name properties are not present, there is only one property with the name of the Kind saved into it

All the entities in this example were newly created in my ipython session with NDB 3.7 built off the master branch

[epluntze]

I looked into this a bit more this morning and it seems the issue is with passing the Kind as a String vs passing the Kind as a static class in the model definition

In [1]: from google.cloud import ndb

In [2]: client = ndb.Client()

In [3]: class TestUserModel(ndb.Model):
   ...:
   ...:     _use_cache = True
   ...:     _use_global_cache = True
   ...:
   ...:     stuff = ndb.StringProperty(required=True)
   ...:

In [4]: new_user_1 = TestUserModel(stuff='stuff')

In [5]: new_user_2 = TestUserModel(stuff='stuff2')

In [6]: class ModelTest(ndb.Model):
   ...:
   ...:     _use_cache = True
   ...:     _use_global_cache = True
   ...:
   ...:     thing_1 = ndb.KeyProperty(kind=TestUserModel, required=True)
   ...:     thing_2 = ndb.KeyProperty(kind=TestUserModel, required=True)
   ...:

In [7]: with client.context():
   ...:     test_user_key1 = new_user_1.put()
   ...:     test_user_key2 = new_user_2.put()
   ...:     print(test_user_key1)
   ...:     print(test_user_key2)
   ...:
Key('TestUserModel', 5541136585195520)
Key('TestUserModel', 5553588249034752)

In [10]: with client.context():
    ...:     new_test = ModelTest(thing_1=ndb.Key(TestUserModel, 5541136585195520), thing_2=ndb.Key(TestUserModel, 5553588249034752))
    ...:     new_key = new_test.put()
    ...:     print(new_key)
    ...:
    ...:
Key('ModelTest', 5556146304712704)

In [11]: with client.context():
    ...:     this = ModelTest.get_by_id(5556146304712704)
    ...:     print(this)
    ...:
ModelTest(key=Key('ModelTest', 5556146304712704), thing_1=Key('TestUserModel', 5541136585195520), thing_2=Key('TestUserModel', 5553588249034752))

This worked correctly

[epluntze]

I opened a PR against my fork that should fix this: #242