Implementation of getCounter logic for ZCatalog

[andbag]

Add getCounter logic (incl. tests) in order to support cache key generation.

[hannosch]

@andbag Thx, this looks good. I'm only wondering to what degree we need the catalog wide counter, since we also have per-index counters now. If a query can reliably determine what indices it is based on, couldn't it use a combination of the per-index counters to get a cache key? This of course assumes all indices support getCounter.

Using per-index counters might have a slight advantage, as changes in unrelated indices wouldn't invalidate the cache keys.

[andbag]

@hannosch I thought about this too. But a catalog wide counter has AFAICS an advantage for the transaction logic. If another thread or zeo-worker has changed the meta_data of the catalog, a ConflictError would occur before any expensive calculation of meta_data of current thread is running, isn't it? I will move the getCounter logic to Catalog.py (updateMetadata) - if my assumption is correct - and add a getCounter Interface to PluginIndexes in order to support a more sophisticated cache key generation.

[hannosch]

I don't think ConflictError's work as you describe them. To my knowledge a ConflictError can only be raised during the final transaction.commit() call when data is actually send to the ZEO/ZODB storage. All work that was supposed to be done by the current request will have been done before that. If a ConflictError occurs, the ZPublisher will by default retry the request up to three times, each time redoing all the work starting with parsing the HTTP request into a request object and figuring out what object to call and everything after that.

For the catalog we don't get these ConflictErrors as often, as all BTree data structures including Length support storage-side conflict resolution via the _p_resolveConflict API. So the transaction contents get unpickled inside the storage and a new object is created as the result of merging the last committed object with the changes desired from the conflicting one. For BTree.Length this is pretty easy (https://github.com/zopefoundation/BTrees/blob/master/BTrees/Length.py#L48) and for buckets rather hard. This conflict resolution can still raise a ConflictError if it cannot combine the objects, for example because all contents from inside a bucket were deleted or a bucket grew too large and was split into two.

[andbag]

Ok, I understand.

[andbag]

Idea and implementation of current conversation are moved to #17 .