Links

• Overview
  • Life of a Datastore Write
  • Transaction Isolation in App Engine
  • How Index Building Works
  • How Entities and Indexes are Stored
  • Modeling Entity Relationships
  • Updating Your Model's Schema
  • Handling Datastore Errors
  • Index Selection and Advanced Search
• Python Docs
  • NDB API
    • NDB (Next DB module) is the new datastore API by Guido.
    • http://code.google.com/p/appengine-ndb-experiment/
    • NDB vs old db api Cheat Sheet
    • A Hitchhiker's Guide to Upgrading App Engine Models to NDB
    • Overview
    • Keys
      • Using Numeric Key IDs
      • Key class
    • Functions
    • Async
      • Using Tasklets
      • Future
        • wait_all and wait_any
    • PolyModel
  • Datastore
  • Entities, Properties, and Keys
    • Unindexed Properties
      • e.g. age = db.IntegerProperty(indexed=False)
  • Properties and Value Types
  • db.get()
  • Key.from_path()
  • Model.get_or_insert()
  • Index Selection and Advanced Search
  • Cross-Group Transactions
  • counter-sharding
  • Understanding Datastore Writes: Commit, Apply, and Data Visibility
  • Datastore Statistics in Python
  • Metadata
    • get_entity_group_version(entity_or_key)
  • Async API
  • Queries
    • Ancestor query
    • Setting the Read Policy and Datastore Call Deadline
    • Query Cursors
    • Async Queries
      • query.fetch() is synchronous. query.run() asynchronously fill in the results.
      • You can replace query.fetch(10) with query.run(config=datastore_query.QueryOptions(limit=10))
• High replication datastore vs. master/slave replication can only be selected at application creation time and cannot be changed later.
• Copying entities to a new application
• Transaction Isolation in App Engine

Snippets

It's "generally" cheaper to fetch all the keys from a query (i.e. classifies as a small datastore op) and then a ndb.get_multi() to get those entities

From: Guido's comment on fetching from cache

q = MyModel.query(...)

results = ndb.get_multi(q.fetch(keys_only=True))

# Only get those entities that are in the cache.
results = ndb.get_multi(q.fetch(keys_only=True), use_datastore=False)


# Async version.
def cb(key):
    ent = key.get(use_datastore=False)  # or True
    if ent is not None:
      # ...do what you want with ent...

# and pass this to map, as follows:

yield q.map(cb, keys_only=True)

NDB examples

The Article class

Instances of class Article are used in the examples for Querying for Repeated Properties.

class Article(ndb.Model):
  title = ndb.StringProperty()
  stars = ndb.IntegerProperty()
  tags = ndb.StringProperty(repeated=True)

article = Article(title='Python versus Ruby',
                  stars=3,
                  tags=['python', 'ruby'])
article.put()

ANDs and ORs

Ref: Combining AND and OR Operations

All the filters are normalized into DNF form so this might cause a combinatorial explosion.

Beware that if your filters is mostly CNF like with lots of conjunctions, the transformation can be hude.  e.g. (A1∨B1)∧(A2∨B2)∧...∧(An∨Bn) → 2ⁿ entries.

Limits

• All queries time out in 60 seconds (even if run from a task queue.)