Refreshed docs for v5.0.0 (#185)

Co-authored-by: Michał Praszmo <[email protected]>

dev/karton.ini.dev

@@ -1,9 +1,8 @@
-[minio]
+[s3]
 access_key = karton-test-access
 secret_key = karton-test-key
-address = localhost:9000
+address = http://localhost:9000
 bucket = karton
-secure = 0
 
 [redis]
 host=localhost

docs/what_new.rst → docs/breaking_changes.rst

@@ -3,8 +3,63 @@ Breaking changes
 
 This chapter will describe significant changes introduced in major version releases of Karton. Versions before 4.0.0 were not officially released, so they have value only for internal purposes. Don't worry about it if you are a new user.
 
-What changed in Karton 4.0.0
-----------------------------
+What is changed in Karton 5.0.0
+-------------------------------
+
+Karton-System and core services are still able to communicate with previous versions.
+
+* Changed name of ``karton.ini`` section that contains S3 client configuration from ``[minio]`` to ``[s3]``.
+
+  In addition to this, you need to add a URI scheme to the ``address`` field and remove the ``secure`` field.
+  If ``secure`` was 0, correct scheme is ``http://``. If ``secure`` was 1, use ``https://``.
+
+  .. code-block:: diff
+
+    - [minio]
+    + [s3]
+      access_key = karton-test-access
+      secret_key = karton-test-key
+    - address = localhost:9000
+    + address = http://localhost:9000
+      bucket = karton
+    - secure = 0
+
+  v5.0.0 maps ``[minio]`` configuration to correct ``[s3]`` configuration internally, but ``[minio]`` scheme
+  is considered deprecated and can be removed in further major release.
+
+* Karton library uses `Boto3 <https://github.com/boto/boto3>`_ library as a S3 client instead of `Minio-Py <https://github.com/minio/minio-py>`_ underneath.
+  You may want to check if your code relies on exceptions thrown by previous S3 client.
+
+* :class:`karton.core.Config` interface is changed. ``config``, ``minio_config`` and ``redis_config`` attributes are no longer available.
+
+* We noticed lots of issues caused by calling factory method ``main()`` on instance instead of class, which can be misleading (:py:meth:``karton.core.base.KartonBase.main``
+  actually creates own instance of Karton service internally, so the initialization is doubled). To notice these errors more quickly, we prevented ``main()`` call on ``KartonBase`` instance
+
+  .. code-block:: python
+
+    if __name__ == "__main__":
+        MyConsumer.main()  # correct
+
+    if __name__ == "__main__":
+        MyConsumer().main()  # throws TypeError
+
+* ``karton.core.Consumer.process`` no longer accepts no arguments. First argument of this method is the incoming task.
+
+.. code-block:: python
+
+    # Correct
+    class MyConsumer(Karton):
+        def process(self, task: Task) -> None:
+            ...
+
+    # Wrong from v5.0.0
+    class MyConsumer(Karton):
+        def process(self) -> None:
+            ...
+
+
+What is changed in Karton 4.0.0
+-------------------------------
 
 Karton-System and core services are still compatible with both 3.x and 2.x versions.
 
@@ -21,8 +76,8 @@ Karton-System and core services are still compatible with both 3.x and 2.x versi
 
 * All crashed tasks are preserved in ``Crashed`` state until they are removed by Karton-System (default is 72 hours) or retried by user. Keep in mind that they hold all the referenced resources, so keep an eye on that queue.
 
-What changed in Karton 3.0.0
-----------------------------
+What is changed in Karton 3.0.0
+-------------------------------
 
 Karton-System and other core services in 3.x are compatible with 2.x. But if you want to use 3.x in Karton service code, all core services need to be upgraded first.
 

docs/getting_started.rst

@@ -4,34 +4,33 @@ Getting started
 Installation
 ------------
 
-You can get the karton framework from pip:
+You can get the Karton framework from pip:
 
 .. code-block:: console
 
     python -m pip install karton-core
 
 Or, if you're feeling adventurous, download the sources using git and install them manually.
 
-In addition to karton, you'll also need to setup `MinIO <https://docs.min.io/docs/minio-quickstart-guide.html>`_ and `Redis-server <https://redis.io/topics/quickstart>`_.
+In addition to Karton core library, you'll also need to setup `MinIO <https://docs.min.io/docs/minio-quickstart-guide.html>`_ and `Redis server <https://redis.io/topics/quickstart>`_.
 
 
 Configuration
 -------------
 
 Each Karton subsystem needs a ``karton.ini`` file that contains the connection parameters for Redis and MinIO.
 
-You can also use this file to store custom fields and use them e.g. by :ref:`example-overriding-config`.
+You can also use this file to store custom fields and use them e.g. by :ref:`extending-config`.
 
-By default, the config class will look for the config file in several places, but let's start by placing one in the root of our new karton subsystem.
+By default, the config class will look for the config file in several places, but let's start by placing one in the root of our new Karton subsystem.
 
 .. code-block:: ini
 
-    [minio]
+    [s3]
     secret_key = minioadmin
     access_key = minioadmin
-    address = localhost:9000
+    address = http://localhost:9000
     bucket = karton
-    secure = 0
 
     [redis]
     host=localhost
@@ -40,10 +39,13 @@ By default, the config class will look for the config file in several places, bu
 If everything was configured correctly, you should now be able to run the ``karton-system`` broker and get ``"Manager karton.system started"`` signaling that it was able to connect to Redis and MinIO correctly.
 
 
-Docker-compose development setup
+Docker Compose development setup
 --------------------------------
 
-If you're just trying Karton out or you want a quick & easy development environment setup check out the ``dev`` folder in the karton root directory.
+Check out repository called `Karton playground <github.com/CERT-Polska/karton-playground/>`_ that provides similar setup coupled with MWDB Core
+and few open-source Karton services.
+
+If you're just trying Karton out or you want a mimimal, quick & easy development environment setup, check out the ``dev`` folder in the Karton root directory.
 
 It contains a small docker-compose setup that will setup the minimal development environment for you.
 
@@ -53,8 +55,11 @@ All you have to do is run
 
     docker-compose up --build
 
-And then connect additional karton systems using the ``karton.ini.dev`` config file.
+And then connect additional Karton systems using the ``karton.ini.dev`` config file.
+
+.. code-block:: console
 
+   karton-classifier --config-file dev/karton.ini.dev
 
 Writing your first Producer and Consumer
 ----------------------------------------
@@ -95,7 +100,7 @@ That was pretty short! Now for a bit longer consumer:
                 print("Bar")
 
     if __name__ == "__main__":
-        FooBarConsumer().loop()
+        FooBarConsumer.main()
 
 If we now run the consumer and spawn a few "foobar" tasks we should get a few foobars logs in return:
 
@@ -127,6 +132,7 @@ If we now run the consumer and spawn a few "foobar" tasks we should get a few fo
     Bar
     [INFO] Task done - d3a39940-d64c-4033-a7da-80eae9786631
 
+Check :ref:`service-examples` for more details.
 
 Command-line interface (CLI)
 ----------------------------------------

docs/index.rst

@@ -14,7 +14,8 @@ Karton ecosystem consists of:
 
 - `Redis <https://www.redis.io/>`_ - store used for message exchange between Karton subsystems
 
-- `MinIO <https://github.com/minio/minio>`_ - temporary object storage compatible with Amazon S3 API, holds all the heavy objects (aka Resources) like samples, analyses or dumps.
+- Temporary object storage compatible with Amazon S3 API, holds all the heavy objects (aka Resources) like samples, analyses or memory dumps.
+  The recommended one is `MinIO <https://github.com/minio/minio>`_.
 
 
 Task routing and data exchange is achieved with the help of **Karton-System** - core of the Karton, which routes the tasks and keeps everything in order (task lifecycle, garbage collection etc.)
@@ -55,17 +56,18 @@ Task routing and data exchange is achieved with the help of **Karton-System** -
 
    if __name__ == "__main__":
        # Here comes the main loop
-       GenericUnpacker().loop()
+       GenericUnpacker.main()
 
 
 .. toctree::
    :maxdepth: 2
    :caption: Karton reference:
 
-   what_new
+   breaking_changes
    getting_started
-   examples
+   service_examples
    task_headers_payloads
+   service_configuration
    advanced_concepts
    unit_tests
    karton_api

docs/karton_api.rst

@@ -51,3 +51,4 @@ karton.core.Config
 ------------------
 
 .. autoclass:: karton.core.config.Config
+   :members: