Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternative sources of taxonomies #4381

Open
ymarcon opened this issue Dec 11, 2022 · 1 comment
Open

Alternative sources of taxonomies #4381

ymarcon opened this issue Dec 11, 2022 · 1 comment
Assignees
@ymarcon
Labels
new feature

Comments

@ymarcon
Copy link
Member

ymarcon commented Dec 11, 2022

Just like #4376 for the value tables, this feature proposes to bypass usage of an Opal server for providing the taxonomies. The taxonomies could be read from a local configuration file, or an external server source. The latter implies that a plugin, of type mica-taxonomies, is defined so that one can extend the Mica with custom taxonomies.
@ymarcon ymarcon self-assigned this Dec 11, 2022
@ymarcon ymarcon moved this to Todo in NFDI4Health Dec 11, 2022
@ymarcon ymarcon moved this from Todo to In Progress in NFDI4Health Dec 11, 2022
ymarcon added a commit that referenced this issue Dec 12, 2022
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
a8c0a06 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 11, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
c2025e5 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 19, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
067ef8d 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 19, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
a134398 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
98989dc 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
2b019e5 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 23, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
5fb41ee 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 24, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
a02dc30 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 26, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
78322d9 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 27, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
bd8a4e0 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 30, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
033e77c 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
0dddfbf 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
d3f0154 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Jan 31, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
ba9f06b 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Feb 1, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
f5f4d54 
…roviders, in addition to the opal one
ymarcon added a commit that referenced this issue Feb 1, 2023
@ymarcon
#4381 Variable taxonomies service added to support other taxonomies p…
2ce571d 
…roviders, in addition to the opal one
@ymarcon ymarcon mentioned this issue Feb 1, 2023
Merged
meek0 pushed a commit that referenced this issue Feb 1, 2023
@ymarcon
Mica without Opal (#4393)
2101d87 
* #4376 WIP. StudyTableSource interface added. New property sourceURN replaces opal project/table in dataset's study tables and variables. Requires ES index config update.

* sourceURN property/getter/setter renamed to source

* source package added, with unit tests

* Excel table source added, reads excel from Mica's file system

* Edit/create study tables with opal or file source. WIP.

* File path can be copied. Study table's file source can be relative to the dataset's folder

* Study table context added. Opal dtos (aggregations) are not exposed in the source plugin interface.

* Deferred init of excel datasource. Less verbose log messages when study table source operation is not supported

* SPI mica-source testing

* Deferred attachment stream extraction

* Handle disposable table sources

* Mica source plugin could be an Initialisable

* #4381 Variable taxonomies service added to support other taxonomies providers, in addition to the opal one

* mica-source plugin type renamed to mica-tables. Ne plugin type: mica-taxonomies for loading taxonomies provider services.

* Taxonomy of taxonomies is configurable. Added attribute to have a taxonomy declared but not visible.

* Variable taxonomies can be read from local folder: MICA_HOME/conf/taxonomies/variable

* AbstractTaxonomiesProviderService added

* Caching of taxonomies refactored, not functional yet

* Variable taxonomies are cached

* Fix dataset taxonomy init

* Opal taxonomies cache replaced by generic variable taxonomies cache. Variable taxonomies are decorated with variable attributes

* Code cleaning: Opal service helper merged into opal service

* classifications administration

* make sure variable taxo title/description in the meta taxo matches the one of the original taxo

* hide instead of remove a target vocabulary if type is disabled

* Plugins REST API added

* Plugin REST API added

* Service Plugin REST API added

* Update the search plugin in the upgrade procedure

* ng-obiba-mica version

* No server error when opal projects cannot be retrieved

* Fix some wording and showVariableStatistics template setting added

* Fix some wording and showVariableStatistics template setting added

* Fix harmonized variable id encoding for url

* Fix harmonized variable id encoding for url for reverse proxy

* NoSuchTableSourceException added with their web service mapper

* ng-obiba-mica latest release

* prepare for upgrade to 5.2

* StudyTableSource does not need to return the table name, this is implementation specific

* Modifying the meta txonomy requires admin role

* Plugin management requires admin role

* Clean Opal projects web services

* Helper class to read a taxonomy in YAML format from a stream

* Other study table source handled in the admin
@ymarcon ymarcon moved this from In Progress to Done in NFDI4Health Feb 16, 2023
@johannes-darms
Copy link
Contributor

johannes-darms commented Sep 13, 2023 •
edited
Loading

I believe there is some kind of loop. The following lines indicated that the indexing fails, which triggers a reindex. Sadly, the process never stops.

mica-mica-1   | 2023-09-13 08:34:11.556 DEBUG 22 --- [         task-1] org.obiba.es.mica.ESIndexer              : Indexing all indexables for indexName [term] persistableObjectNumber [2]
mica-mica-1   | 2023-09-13 08:34:12.332  INFO 22 --- [        task-10] org.obiba.mica.taxonomy.TaxonomyIndexer  : All taxonomies were updated
mica-mica-1   | 2023-09-13 08:34:12.333 DEBUG 22 --- [        task-13] org.obiba.es.mica.ESIndexer              : Indexing for indexName [taxonomy] indexableObject [org.obiba.mica.taxonomy.TaxonomyIndexable@1ce983d0]
mica-mica-1   | 2023-09-13 08:34:12.354 ERROR 22 --- [        task-11] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void org.obiba.mica.taxonomy.TaxonomyIndexer.taxonomiesUpdated(org.obiba.mica.micaConfig.event.TaxonomiesUpdatedEvent)
mica-mica-1   | 
mica-mica-1   | org.elasticsearch.index.IndexNotFoundException: no such index
mica-mica-1   | 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:581)
mica-mica-1   | 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
mica-mica-1   | 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
mica-mica-1   | 	at org.elasticsearch.action.admin.indices.mapping.put.TransportPutMappingAction.checkBlock(TransportPutMappingAction.java:66)
mica-mica-1   | 	at org.elasticsearch.action.admin.indices.mapping.put.TransportPutMappingAction.checkBlock(TransportPutMappingAction.java:41)
mica-mica-1   | 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:156)
mica-mica-1   | 	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:148)
mica-mica-1   | 	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:115)
mica-mica-1   | 	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:50)
mica-mica-1   | 	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
mica-mica-1   | 	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
mica-mica-1   | 	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
mica-mica-1   | 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
mica-mica-1   | 	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
mica-mica-1   | 	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
mica-mica-1   | 	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
mica-mica-1   | 	at org.obiba.es.mica.mapping.TaxonomyIndexConfiguration.onIndexCreated(TaxonomyIndexConfiguration.java:57)
mica-mica-1   | 	at org.obiba.es.mica.ESIndexer.lambda$createIndexIfNeeded$2(ESIndexer.java:250)
mica-mica-1   | 	at java.lang.Iterable.forEach(Iterable.java:75)
mica-mica-1   | 	at org.obiba.es.mica.ESIndexer.createIndexIfNeeded(ESIndexer.java:250)
mica-mica-1   | 	at org.obiba.es.mica.ESIndexer.index(ESIndexer.java:79)
mica-mica-1   | 	at org.obiba.es.mica.ESIndexer.index(ESIndexer.java:73)
mica-mica-1   | 	at org.obiba.mica.search.SearchEngineIndexer.index(SearchEngineIndexer.java:46)
mica-mica-1   | 	at org.obiba.mica.taxonomy.TaxonomyIndexer.lambda$index$5(TaxonomyIndexer.java:119)
mica-mica-1   | 	at java.lang.Iterable.forEach(Iterable.java:75)
mica-mica-1   | 	at org.obiba.mica.taxonomy.TaxonomyIndexer.index(TaxonomyIndexer.java:118)
mica-mica-1   | 	at org.obiba.mica.taxonomy.TaxonomyIndexer.taxonomiesUpdated(TaxonomyIndexer.java:75)
mica-mica-1   | 	at org.obiba.mica.taxonomy.TaxonomyIndexer$$FastClassBySpringCGLIB$$1c4955a2.invoke(<generated>)
mica-mica-1   | 	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
mica-mica-1   | 	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
mica-mica-1   | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
mica-mica-1   | 	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
mica-mica-1   | 	at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
mica-mica-1   | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
mica-mica-1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
mica-mica-1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
mica-mica-1   | 	at java.lang.Thread.run(Thread.java:750)
mica-mica-1   | 
mica-mica-1   | 2023-09-13 08:34:12.354  INFO 22 --- [         task-9] org.obiba.es.mica.ESIndexer              : Creating index taxonomy
mica-mica-1   | 
mica-mica-1   | 2023-09-13 08:34:12.362 DEBUG 22 --- [        task-10] org.obiba.es.mica.ESIndexer              : Indexing for indexName [taxonomy] indexableObject [org.obiba.mica.taxonomy.TaxonomyIndexable@3abf1694]
mica-mica-1   | 2023-09-13 08:34:12.362  INF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ymarcon ymarcon

Labels
new feature
Projects
NFDI4Health
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ymarcon @johannes-darms