Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

cursor_after() doesn't exist at end of iteration #292

Closed
oliverchang opened this issue Jan 15, 2020 · 1 comment · Fixed by #296
Closed

cursor_after() doesn't exist at end of iteration #292

oliverchang opened this issue Jan 15, 2020 · 1 comment · Fixed by #296
Assignees
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@oliverchang
Copy link

There is a behaviour difference with the old NDB that we're relying on wrt to cursor_after() at the end of iterating through a query.

Given the following:

      class Model(ndb.Model):                 
        a = ndb.IntegerProperty()                 
                                      
      Model(a=1).put()   
      Model(a=2).put()
      Model(a=3).put()   
                 
      result = Model.query().iter(limit=2)
      print(list(result))                                                                               
      print(list(Model.query().iter(start_cursor=result.cursor_after())))

The old NDB gives:

[Model(key=Key('Model', 31), a=1), Model(key=Key('Model', 32), a=2)]
[Model(key=Key('Model', 33), a=3)]

The new NDB gives:

[Model(key=Key(u'Model', 31L), a=1L), Model(key=Key(u'Model', 32L), a=2L)]
BadArgumentError: There is no cursor currently
@oliverchang
Copy link
Author

This works as expected if the following line is removed:

self._cursor_before = self._cursor_after = None

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Jan 15, 2020
@chrisrossi chrisrossi self-assigned this Jan 15, 2020
@chrisrossi chrisrossi added 🚨 This issue needs some love. labels Jan 15, 2020
@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed triage me I really want to be triaged. 🚨 This issue needs some love. 🚨 labels Jan 15, 2020
chrisrossi pushed a commit to chrisrossi/python-ndb that referenced this issue Jan 22, 2020
In an overabundance of caution, we were deleting `cursor_after` from
QueryIterator after exhausting the iterator. This was found to differ
from the original implementation, however, and break some customer's
code.

Fixes googleapis#292
chrisrossi pushed a commit that referenced this issue Jan 22, 2020
In an overabundance of caution, we were deleting `cursor_after` from
QueryIterator after exhausting the iterator. This was found to differ
from the original implementation, however, and break some customer's
code.

Fixes #292
oliverchang added a commit to google/clusterfuzz that referenced this issue Jan 29, 2020
Part of migration to Python 3. This is our last major blocker and
dependency on the Python 2 App Engine SDK.

Also remove ndb_patcher, as that is no longer required.

Notable changes:
- indexed=False is no longer supported for StringProperty. Use TextProperty instead, which is
  equivalent.
- batch_size query keyword arg is no longer supported (Datastore backend change).
- fetch_page() is no longer supported for complex queries with !=, OR etc.
- Many import hacks/workarounds to make things work for App Engine.
  - Notable: add `grpcio` to app.yaml, cron-service.yaml
- Add workaround for googleapis/python-ndb#292 and googleapis/python-ndb#295
- Threads may be created by the new NDB implementation, which are only guaranteed to be 
  stopped after a GC. GAE doesn't like it when there are stray threads at the end of a request.
- Stub out `ndb_utils.get_all_from_query`. This is no longer required, and callers will be cleaned up in a future PR.
- All datastore operations now require:
```python
from datastore import ndb_init
with ndb_init.context():
  ...
```
This is automatically set up in tests, as well as in run_bot, run, heartbeat, run_heartbeat, and butler run, as well as for each individual request.
@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Apr 6, 2020
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants