[SDESK-7278] New Elasticsearch datalayer (#2624)

* Add es async resources

* Add resources system

* Update es commands

* Add 'set_frozeon' to ConfigModel

* Add/update tests

* Update docs

* Update docs

* Use latest Elasticsearch 7.x client lib

* Improve readability for get_core_schema_from_type function

* Rename get_current_app to get_current_async_app

docker-compose.yml

@@ -6,7 +6,7 @@ version: "3.2"
 
 services:
     elastic:
-        image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
+        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.22
         ports:
             - "9200:9200"
         environment:

docs/core/app.rst

@@ -7,18 +7,18 @@ Getting App Instance
 --------------------
 
 The :class:`SuperdeskAsyncApp <app.SuperdeskAsyncApp>` app is built for us automatically in the `SuperdeskEve` app.
-You can retrieve an instance of this app from the :meth:`get_current_app <app.get_current_app>` method.::
+You can retrieve an instance of this app from the :meth:`get_current_async_app <app.get_current_async_app>` method.::
 
-    from superdesk.core.app import get_current_app
+    from superdesk.core.app import get_current_async_app
 
     def test():
-        get_current_app().get_module_list()
+        get_current_async_app().get_module_list()
 
 
 APP References
 --------------
 
-.. autofunction:: superdesk.core.app.get_current_app
+.. autofunction:: superdesk.core.app.get_current_async_app
 
 .. autoclass:: superdesk.core.app.SuperdeskAsyncApp
     :member-order: bysource

docs/core/config.rst

@@ -64,11 +64,11 @@ The following are examples on how to load a config::
 
 You can also load the config from the currently running app config::
 
-    from superdesk.core.app import get_current_app
+    from superdesk.core.app import get_current_async_app
 
     def test_config_from_app():
         config = MyConfig.create_from_dict(
-            get_current_app().wsgi.config,
+            get_current_async_app().wsgi.config,
             prefix="MY"
         )
 

docs/core/elastic.rst

@@ -0,0 +1,69 @@
+.. core_elastic:
+
+Elasticsearch
+=============
+
+.. module:: superdesk.core.elastic
+
+ElasticResources Instance
+-------------------------
+The :class:`ElasticResources <resources.ElasticResources>` instance provides access to the Elasticsearch clients.
+This instance is available under the :attr:`SuperdeskAsyncApp.elastic <superdesk.core.app.SuperdeskAsyncApp.elastic>`.
+
+There are two types of clients you can use: one for standard synchronous connections and another for asynchronous
+connections. The functions ending in ``_async`` provide the asynchronous version.
+
+For example::
+
+    from superdesk.core.app import get_current_async_app
+
+    def test():
+        client = get_current_async_app().elastic.get_client("users")
+        users = client.search(
+            {"query": {"term": {"first_name": "monkey"}}}
+        )
+
+    async def test_async():
+        client = get_current_async_app().elastic.get_client_async("users")
+        users = await client.search(
+            {"query": {"term": {"first_name": "monkey"}}}
+        )
+
+The :attr:`get_client <resources.ElasticResources.get_client>` method returns an instance of
+:class:`ElasticResourceClient <sync_client.ElasticResourceClient>`.
+
+The :attr:`get_client_async <resources.ElasticResources.get_client>` method returns an instance of
+:class:`ElasticResourceAsyncClient <async_client.ElasticResourceAsyncClient>`.
+
+
+Elastic References
+------------------
+.. autoclass:: superdesk.core.elastic.common.SearchArgs
+    :member-order: bysource
+    :members:
+
+.. autoclass:: superdesk.core.elastic.common.SearchRequest
+    :member-order: bysource
+    :members:
+
+.. autoclass:: superdesk.core.elastic.common.ElasticResourceConfig
+    :member-order: bysource
+    :members:
+
+.. autodata:: superdesk.core.elastic.common.ProjectedFieldArg
+
+.. autoclass:: superdesk.core.elastic.resources.ElasticResources
+    :member-order: bysource
+    :members:
+
+.. autoclass:: superdesk.core.elastic.sync_client.ElasticResourceClient
+    :member-order: bysource
+    :members:
+
+.. autoclass:: superdesk.core.elastic.async_client.ElasticResourceAsyncClient
+    :member-order: bysource
+    :members:
+
+.. autofunction:: superdesk.core.elastic.mapping.json_schema_to_elastic_mapping
+
+.. autofunction:: superdesk.core.elastic.mapping.get_elastic_mapping_from_model

docs/core/mongo.rst

@@ -3,68 +3,35 @@
 MongoDB
 =======
 
+.. module:: superdesk.core.mongo
+
 MongoResources Instance
 -----------------------
-The :class:`MongoResources <mongo.MongoResources>` instance provides access to the MongoDB resource configs,
-client and database connections. This instance is available under the :attr:`SuperdeskAsyncApp.mongo <app.SuperdeskAsyncApp.mongo>`
-attribute.
+The :class:`MongoResources` instance provides access to the MongoDB resource configs,
+client and database connections. This instance is available under the
+:attr:`SuperdeskAsyncApp.mongo <superdesk.core.app.SuperdeskAsyncApp.mongo>` attribute.
 
-There are 2 groups of functions to use. 1 for standard synchronous connections and another for asynchronous connections.
-The functions ending in ``_async`` provide the asynchronous version.
+There are two groups of functions to use: one for standard synchronous connections and another for asynchronous
+connections. The functions ending in ``_async`` provide the asynchronous version.
 
 For example::
 
-    from superdesk.core.app import get_current_app
+    from superdesk.core.app import get_current_async_app
 
     def test():
-        users = get_current_app().mongo.get_db("users")
+        users = get_current_async_app().mongo.get_db("users")
         user = users.find_one({"_id": "abcd123"})
 
     async def test_async()
-        users = get_current_app().mongo.get_db_async("users")
+        users = get_current_async_app().mongo.get_db_async("users")
         user = await users.find_one({"_id": "abcd123"})
 
 
-Registering Resources
----------------------
-The :meth:`MongoResources.register_resource_config <mongo.MongoResources.register_resource_config>` method provides a way to
-register resources for use with MongoDB, using :class:`MongoResourceConfig <mongo.MongoResourceConfig>` and
-:class:`MongoIndexOptions <mongo.MongoIndexOptions>` classes.
-
-The details of the registered resource is later used to create client connections,
-create indexes etc for the specific resource. The :attr:`prefix <mongo.MongoResourceConfig.prefix>` config provides the same
-``prefix`` capabilities to older style resources.
-
-Example resource registration::
-
-    from superdesk.core.module import Module, SuperdeskAsyncApp
-    from superdesk.core.mongo import MongoResourceConfig, MongoIndexOptions
-
-    user_mongo_resource = MongoResourceConfig(
-        name="users",
-        indexes=[
-            MongoIndexOptions(
-                name="users_name_1",
-                keys=[("first_name", 1)],
-            ),
-            MongoIndexOptions(
-                name="combined_name_1",
-                keys=[("first_name", 1), ("last_name", -1)],
-                background=False,
-                unique=False,
-                sparse=False,
-                collation={"locale": "en", "strength": 1},
-            ),
-        ],
-    )
-
-
-    def init(app: SuperdeskAsyncApp):
-        app.mongo.register_resource_config(user_mongo_resource)
-
-
-    module = Module(name="tests.users", init=init)
+The :attr:`get_db <MongoResources.get_db>` method returns an instance of a
+`PyMongo Database <https://pymongo.readthedocs.io/en/stable/api/pymongo/database.html>`_.
 
+The :attr:`get_db_async <MongoResources.get_db_async>` method returns an instance of a
+`Motor AsyncIOMotorDatabase <https://motor.readthedocs.io/en/stable/api-asyncio/asyncio_motor_database.html>`_.
 
 Mongo References
 ----------------