WIP {!join cacheEventually=true ...}... #623
Conversation
| // make cross core joins time-agnostic | ||
| // it should be ruled by param probably | ||
| boolean crossCoreCache = false; | ||
| if(localParams.getBool("cacheEventually", false)) { |
There was a problem hiding this comment.
introducing {!join cacheEventually=true ...}...
| // it should be ruled by param probably | ||
| boolean crossCoreCache = false; | ||
| if(localParams.getBool("cacheEventually", false)) { | ||
| if (query instanceof JoinQuery) { |
There was a problem hiding this comment.
two avoid those ugly instanceofs I'd like to unify existing JoinQP routines to unify and extract common ancestor.
| if(localParams.getBool("cacheEventually", false)) { | ||
| if (query instanceof JoinQuery) { | ||
| if (((JoinQuery) query).fromCoreOpenTime != 0L) { | ||
| ((JoinQuery) query).fromCoreOpenTime = Long.MIN_VALUE; |
There was a problem hiding this comment.
I need these queries match to each other for cache hit regardless of commit in fromCore. It can be done by setting some additional flag and changing equals&hashCode. WDYT?
| WrappedQuery wrap = new WrappedQuery(query); | ||
| wrap.setCache(false); //bypassing searcher cache | ||
| @SuppressWarnings("unchecked") | ||
| final SolrCache<Query,DocSet> rightSideCache = (SolrCache<Query,DocSet>) req.getSearcher().getCache(fromIndex); |
There was a problem hiding this comment.
It uses left side core user cache by right side (fromCore) name. I suppose it's a clear convention.
| final Filter topFilter = docset.getTopFilter(); | ||
|
|
||
| WrappedQuery wrappedCache = new WrappedQuery(topFilter); | ||
| wrappedCache.setCache(false); |
There was a problem hiding this comment.
return non-cached query backed on doc set cached in user cache
| </commitWithin> | ||
| </updateHandler> | ||
|
|
||
| <updateProcessor name="refresh-join-caches" class="solr.RefreshCrossCoreJoinCacheFactory"/> |
There was a problem hiding this comment.
This guy from the right side (fromCore) refreshes left side (to) caches.
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| public class RefreshCrossCoreJoinCacheFactory extends UpdateRequestProcessorFactory { |
| String rightSideCore = req.getCore().getName(); | ||
| for (String leftCoreName: loadedCoreNames){ | ||
| if (!leftCoreName.equals(rightSideCore)) { | ||
| final SolrCore core = coreContainer.getCore(leftCoreName); |
There was a problem hiding this comment.
Enumerates all possible "to" cores, find user cache with name "fromCore".
| SolrQueryResponse rsp = new SolrQueryResponse(); | ||
| SolrRequestInfo.setRequestInfo(new SolrRequestInfo(leftReq, rsp)); | ||
| try { | ||
| joinCache.warm(leftSearcher.get(), joinCache); |
There was a problem hiding this comment.
Self warm left side caches picking fresh "fromCore" searcher.
| } else { // commit into main index refreshes cache as well | ||
| assertU(add(doc("id", "99999999"))); | ||
| assertU(commit()); // TODO this causes an error. | ||
| // cached join leaks as query result key, |
There was a problem hiding this comment.
This is a problem so far. Doc set from old searcher breaks warming of fresh searcher. Have no solution.
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
| @Override public void close() { } | ||
| }; | ||
| SolrQueryResponse rsp = new SolrQueryResponse(); | ||
| SolrRequestInfo.setRequestInfo(new SolrRequestInfo(leftReq, rsp)); |
There was a problem hiding this comment.
RESOURCE_LEAK: resource of type org.apache.solr.search.RefreshCrossCoreJoinCacheFactory$1$1 acquired by call to RefreshCrossCoreJoinCacheFactory$1$1(...) at line 51 is not released after line 56.
Note: potential exception at line 56
(at-me in a reply with help or ignore)
Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]
mkhludnev
force-pushed
the
background-join-warmer
branch
from
b291101 to
8f2f333
Compare
No description provided.