From f15e4c705ba1e21ac878be401fe60fb6eaf5fb4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marco=20Vidal=20Garc=C3=ADa?= Date: Thu, 23 Jul 2020 15:06:03 +0200 Subject: [PATCH] docs: remove non-autogenerated sections --- AUTHORS.rst | 1 + CONTRIBUTING.rst | 7 - docs/conf.py | 2 +- docs/developerguide.rst | 19 -- docs/gettingstarted.rst | 189 ------------------ docs/index.rst | 3 - docs/introduction.rst | 2 +- docs/userguide.rst | 428 ---------------------------------------- 8 files changed, 3 insertions(+), 648 deletions(-) delete mode 100644 docs/developerguide.rst delete mode 100644 docs/gettingstarted.rst delete mode 100644 docs/userguide.rst diff --git a/AUTHORS.rst b/AUTHORS.rst index 3194b6d7..17c71168 100644 --- a/AUTHORS.rst +++ b/AUTHORS.rst @@ -11,6 +11,7 @@ The list of contributors in alphabetical order: - `Harri Hirvonsalo `_ - `Jan Okraska `_ - `Leticia Wanderley `_ +- `Marco Vidal `_ - `Rokas Maciulaitis `_ - `Sinclert Perez `_ - `Tibor Simko `_ diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 8542b944..c2a6dbbe 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -24,10 +24,3 @@ is operating correctly: $ ./run-tests.sh Each pull request should preserve or increase code coverage. - -Kanban ------- - -We are using Kanban technique for keeping track of ongoing tasks. Please see -our `Kanban boards `_ and look for -issues that are "ready for work". diff --git a/docs/conf.py b/docs/conf.py index f1d191d5..6cecfd25 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -63,7 +63,7 @@ # General information about the project. project = "reana" -copyright = "2017-2019 info@reana.io" +copyright = "2017-2020 info@reana.io" author = "info@reana.io" # The version info for the project you're documenting, acts as replacement for diff --git a/docs/developerguide.rst b/docs/developerguide.rst deleted file mode 100644 index dd0523ab..00000000 --- a/docs/developerguide.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _developerguide: - -Developer guide -=============== - -Using latest ``reana-client`` version -------------------------------------- - -If you want to use the latest bleeding-edge version of ``reana-client``, without -cloning it from GitHub, you can use: - -.. code-block:: console - - $ # create new virtual environment - $ virtualenv ~/.virtualenvs/myreana - $ source ~/.virtualenvs/myreana/bin/activate - $ # install reana-commons and reana-client - $ pip install git+git://github.com/reanahub/reana-commons.git@master#egg=reana-commons - $ pip install git+git://github.com/reanahub/reana-client.git@master#egg=reana-client diff --git a/docs/gettingstarted.rst b/docs/gettingstarted.rst deleted file mode 100644 index 00f5468c..00000000 --- a/docs/gettingstarted.rst +++ /dev/null @@ -1,189 +0,0 @@ -.. _gettingstarted: - -Getting Started -=============== - -Install REANA client --------------------- - -If you are a researcher that is interested in running analyses on the REANA -cloud, all you need to install is the ``reana-client``, ideally in a new virtual -environment: - -.. code-block:: console - - $ # create new virtual environment - $ virtualenv ~/.virtualenvs/myreana - $ source ~/.virtualenvs/myreana/bin/activate - $ # install reana-client - $ pip install reana-client - -Select REANA cloud ------------------- - -You can now select the REANA cloud instance where to run your analyses by -setting the ``REANA_SERVER_URL`` variable appropriately. You also have to -provide a valid access token through the environment variable -``REANA_ACCESS_TOKEN``. For example: - -.. code-block:: console - - $ export REANA_SERVER_URL=https://reana.cern.ch - $ export REANA_ACCESS_TOKEN=XXXXXXX - -The access token should have been given to you by the administrators of the -REANA cloud instance you are using. - -.. admonition:: Hint for developers - - Note that if you are running REANA cluster locally on your laptop -- i.e. - you are yourself the administrator of your REANA cloud instance -- then you - can use: - - .. code-block:: console - - $ eval $(reana-dev setup-environment) - - which will set both environment variables for you. - -Run example analysis --------------------- - -Let us take `reana-demo-helloworld -`_ as a simple example -analysis to run on our REANA cloud. - -Please familiarise yourself with the structure of ``reana-demo-helloworld`` -GitHub repository and how it specifies the analysis code, data, environment, and -the computation workflow to produce the analysis output. The ``reana-client`` -usage scenario will be identical in submitting any complex research data -computational workflows. - -Let us start by testing connection to the REANA cloud: - -.. code-block:: console - - $ reana-client ping - Server is running. - -We can now create a new computational workflow: - -.. code-block:: console - - $ reana-client create - workflow.1 - -This created a workflow with the default name "workflow" and run number "1". - -Note that if you would like to give your workflow a different name, you can use -the ``-n`` argument: - -.. code-block:: console - - $ reana-client create -n myfirstdemo - myfirstdemo.1 - -We can check the status of our previously created workflow: - -.. code-block:: console - - $ reana-client status -w workflow.1 - NAME RUN_NUMBER CREATED STATUS PROGRESS - workflow 1 2018-08-10T07:27:15 created -/- - -Note that instead of passing ``-w`` argument with the workflow name every time, -we can define a new environment variable ``REANA_WORKON`` which specifies the -workflow we would like to work on: - -.. code-block:: console - - $ export REANA_WORKON=workflow.1 - -Let us upload our code: - -.. code-block:: console - - $ reana-client upload ./code/helloworld.py - File code/helloworld.py was successfully uploaded. - -and check whether it indeed appears seeded in our workspace: - -.. code-block:: console - - $ reana-client ls - NAME SIZE LAST-MODIFIED - code/helloworld.py 2905 2018-08-10T07:29:54 - -Similarly, let us now upload the input data file: - -.. code-block:: console - - $ reana-client upload ./data/names.txt - File data/names.txt was successfully uploaded. - -and check whether it was well seeded into our input workspace: - -.. code-block:: console - - $ reana-client ls - NAME SIZE LAST-MODIFIED - data/names.txt 18 2018-08-10T07:31:15 - code/helloworld.py 2905 2018-08-10T07:29:54 - -Now that the input data and code was uploaded, we can start the workflow execution: - -.. code-block:: console - - $ reana-client start - workflow.1 has been started. - -Let us enquire about its running status; we may see that it is still in the -"running" state: - -.. code-block:: console - - $ reana-client status - NAME RUN_NUMBER CREATED STATUS PROGRESS - workflow 1 2018-08-10T07:27:15 running 0/1 - -After a few minutes, the workflow should be finished: - -.. code-block:: console - - $ reana-client status - NAME RUN_NUMBER CREATED STATUS PROGRESS - workflow 1 2018-08-10T07:27:15 finished 1/1 - -We can now check the list of output files: - -.. code-block:: console - - $ reana-client ls - NAME SIZE LAST-MODIFIED - code/helloworld.py 2905 2018-08-06T13:58:21 - data/names.txt 18 2018-08-06T13:59:59 - results/greetings.txt 32 2018-08-06T14:01:02 - -and retrieve the resulting output file: - -.. code-block:: console - - $ reana-client download results/greetings.txt - File results/greetings.txt downloaded to /home/reana/reanahub/reana-demo-helloworld. - -Let us see whether we got the expected output: - -.. code-block:: console - - $ cat helloworld/greetings.txt - Hello Jane Doe! - Hello John Doe! - -Next steps ----------- - -For more information, please see: - -- Looking for a more comprehensive REANA client user manual? See :ref:`userguide` -- Looking for tips how to develop REANA client component? See :ref:`developerguide` -- Looking for REANA client command-line API reference? See :ref:`cliapi` diff --git a/docs/index.rst b/docs/index.rst index 49ac7142..3822771b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -9,9 +9,6 @@ :numbered: introduction - gettingstarted - userguide - developerguide cliapi apidocs contributing diff --git a/docs/introduction.rst b/docs/introduction.rst index 27571930..4a0e8c4a 100644 --- a/docs/introduction.rst +++ b/docs/introduction.rst @@ -4,4 +4,4 @@ Introduction ============ .. include:: ../README.rst - :start-line: 22 + :start-line: 25 diff --git a/docs/userguide.rst b/docs/userguide.rst deleted file mode 100644 index a8c08af6..00000000 --- a/docs/userguide.rst +++ /dev/null @@ -1,428 +0,0 @@ -.. _userguide: - -User guide -========== - -Environment variables ---------------------- - -REANA_SERVER_URL -~~~~~~~~~~~~~~~~ - -You can set this environment variable in order to specify to which REANA cloud -instance your client should connect. For example: - -.. code-block:: console - - $ export REANA_SERVER_URL=https://reana.cern.ch - -REANA_ACCESS_TOKEN -~~~~~~~~~~~~~~~~~~ - -You should specify valid access token for the REANA cloud instance you would -like to use. For example: - -.. code-block:: console - - $ export REANA_ACCESS_TOKEN=XXXXXXX - -The token should be provided to you by the REANA cluster administrators. - -REANA_WORKON -~~~~~~~~~~~~ - -You can set this environment variable in order to specify a concrete workflow -you would like to work on. (As an alternative to specifying ``--workflow`` -option in commands.) For example: - -.. code-block:: console - - $ export REANA_WORKON=myfirstanalysis - -will work on the latest run of your "myfirstanalysis" workflow. - -Note that you can also specify a concrete run number: - -.. code-block:: console - - $ export REANA_WORKON=myfirstanalysis.42 - -which will permit to work on the third run of the "myfirstanalysis" workflow, -for example to check out past input and output files. - -You can list all your workflow runs and their statuses by doing: - -.. code-block:: console - - $ reana-client list - -and set ``REANA_WORKON`` to the one you would like to work on. - -Usage ------ - -Overview -~~~~~~~~ - -Please see the :ref:`gettingstarted` section for a basic ``reana-client`` use -case scenario. - -Uploading analysis assets -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Uploading files or directories to the analysis workspace is simple as: - -.. code-block:: console - - $ reana-client upload mydata.csv mycode.C mytmp - File mydata.csv was successfully uploaded. - File mycode.C was successfully uploaded. - File mytmp/myfiltereddata.csv was successfully uploaded. - -If you want to upload all input files defined in the reana.yaml of the analysis, -you can just run: - -.. code-block:: console - - $ reana-client upload - File mydata.csv was successfully uploaded. - File mycode.C was successfully uploaded. - -Directory structures are maintained, i.e. -mytmp exists in the workspace. - -Note that symbolic links are resolved at the moment of upload -so that a hard copy of the link target is uploaded to the cloud -storage workspace. The link is not maintained throughout the -workflow execution. - - -Deleting analysis assets -~~~~~~~~~~~~~~~~~~~~~~~~ - -The deletion of files contained in the analysis workspace is possible through -the ``remove`` command: - -.. code-block:: console - - $ reana-client rm mydata.csv - File mydata.csv was successfully deleted. - 25356 bytes freed up. - -If you want to delete more than one file at once it is possible to use -globbing: - -.. code-block:: console - - $ reana-client rm '**/*.csv' - File mydata.csv was successfully deleted. - File mytmp/myfiltereddata.csv was successfully deleted. - 79736 bytes freed up. - - -Moving analysis assets -~~~~~~~~~~~~~~~~~~~~~~ - -The movement of file(s) or folders within the analysis workspace is -possible through the ``mv`` command: - -.. code-block:: console - - $ reana-client mv data/mydata.csv mydata.csv - File mydata.csv was successfully deleted. - - -Adding secrets -~~~~~~~~~~~~~~ - -If you need to use secrets in your workflow you can add them with -the ``secrets-add`` command: - -.. code-block:: console - - $ # You can upload secrets from literal strings: - $ reana-client secrets-add --env CERN_USER=johndoe - --env CERN_KEYTAB=.keytab - Secrets CERN_USER, CERN_KEYTAB were successfully uploaded. - - $ # ...and from files: - $ reana-client secrets-add --file ~/.keytab - Secrets .keytab were successfully uploaded. - - $ # ...you can also combine two options in one command: - $ reana-client secrets-add --env CERN_USER=johndoe - --env CERN_KEYTAB=.keytab - --file ~/.keytab - Secrets .keytab, CERN_USER, CERN_KEYTAB were successfully uploaded. - - $ # Trying to add a secret that is already added - $ # will result in a warning and no action will be taken: - $ reana-client secrets-add --env CERN_USER=johndoe - One of the secrets already exists. No secrets were added. - - $ # If you are sure that you want to overwrite it you can use - $ # the ``--overwrite`` option: - $ reana-client secrets-add --env CERN_USER=janedoe - --overwrite - Secrets CERN_USER were successfully uploaded. - $ # Note that the ``--overwrite`` option will aply to - $ # all of secrets passed next to it. - - -The added secrets will be available in your workflow execution container either -as environment variables (see example ``CERN_USER`` above) or as -mounted files (see ``.keytab`` example above) in the ``/etc/reana/secrets/`` -directory. - -Listing secrets -~~~~~~~~~~~~~~~ - -To list all the secrets that you have uploaded you can use -the ``secrets-list`` command: - -.. code-block:: console - - $ reana-client secrets-list - NAME TYPE - .keytab file - CERN_KEYTAB env - CERN_USER env - - -Deleting secrets -~~~~~~~~~~~~~~~~ - -If you want to delete some of the secrets that you have uploaded you can use -the ``secrets-delete`` command: - -.. code-block:: console - - $ reana-client secrets-delete CERN_USER, CERN_KEYTAB - Secrets CERN_USER, CERN_KEYTAB were successfully deleted. - - -Overriding default input parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you want to run a workflow with different input parameters than the ones in -``reana.yaml``, you can do it by running `reana-client start` with flag ``-p`` -and specifying parameters that you want to override. - -Note that parameters passed with ``-p`` flag must exist in reana.yaml. -Non-existing parameters will be skipped. - -.. code-block:: console - - $ reana-client start -p myparam1=myval1 -p myparam2=myval2 - myanalysis.42 has been started. - - -Running analysis -~~~~~~~~~~~~~~~~ - -If you have fully working analysis with ``reana.yaml``, you can run the workflow -easily via the ``run`` wrapper command, which will create a new workflow, upload -analysis inputs, and start the workflow run. - -.. code-block:: console - - $ vim reana.yaml - $ reana-client run -n myanalysis - [INFO] Creating a workflow... - myanalysis.42 - [INFO] Uploading files... - File code/helloworld.py was successfully uploaded. - File data/names.txt was successfully uploaded. - [INFO] Starting workflow... - myanalysis.42 has been started. - $ export REANA_WORKON=myanalysis - $ reana-client status - NAME RUN_NUMBER CREATED STATUS PROGRESS - myanalysis 1 2018-11-07T12:45:18 running 1/1 - $ reana-client download results/plot.png - -Running specific parts of analysis -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Serial* - -Serial workflows can be executed partially until the step specified by the -user. To do so, you need to provide the target step name as an operational -option to the ``reana-client start`` or ``reana-client run`` commands. - -.. code-block:: console - - $ reana-client start -w myanalysis.42 -o TARGET=gendata - # or - $ reana-client run -w myanalysis.42 -o TARGET=gendata - -If you want to start workflow execution from a certain step you should use -operational option ``FROM`` with a desired step name. - -.. code-block:: console - - $ reana-client start -w myanalysis.42 -o FROM=fitdata - # or - $ reana-client run -w myanalysis.42 -o FROM=fitdata - -If you want to restart a workflow on the same workspace you should use a -``restart`` command. - -Note that workflow restarting can be used in a combination with operational -options ``FROM`` and ``TARGET``. You can also pass a modified workflow -specification with ``-f`` or `--file`` flag. - -.. code-block:: console - - # Restarts workflow on the same workspace from step named fitdata - $ reana-client restart -w myanalysis.42 -o FROM=fitdata - # Restarts workflow on the same workspace and starts execution from mystep3 to mystep7 - $ reana-client restart -w myanalysis.42 -o FROM=mystep3 -o TARGET=mystep7 - # Restarts workflow on the same workspace with a modified workflow specification - $ reana-client restart -w myanalysis.42 -f reana.yaml - -*CWL* - -CWL allows `executing workflows partially `_. -To do so, you need to provide the specific target as an operational option for -the ``reana-client start`` or ``reana-client run`` commands. - -.. code-block:: console - - $ reana-client start -w myanalysis.42 -o --target=gendata - # or - $ reana-client run -w myanalysis.42 -o --target=gendata - -*Yadage* - -Not implemented yet. - -Downloading outputs -~~~~~~~~~~~~~~~~~~~ - -Downloading files from an analysis workspace: - -.. code-block:: console - - $ reana-client download result.png - File plot.png downloaded to /myfirstanalysis. - -In the same way you can download all outputs defined in the reana.yaml -file of the analysis, by just running: - -.. code-block:: console - - $ reana-client download - -Note that downloading directories is not yet supported. - - -Opening interactive sessions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -While your analysis workflows are running, you may want to open interactive -session processes on the workspace, such as Jupyter notebooks, via the `open` -command. This will allow to quickly explore the generated data while the -analysis is in progress, or even run your analyses from within the notebook -environment spawned on the remote containerised platform. - -.. code-block:: console - - $ reana-client open -w myanalysis.42 jupyter - https://reana.cern.ch/7cd4d23e-48d1-4f7f-8a3c-3a6d256fb8bc?token=P-IkL_7w25IDHhes8I7DtICWLNQm2WAZ9gkoKC2vq10 - It could take several minutes to start the interactive session. - -Open the link returned by the command in order to access the interactive -notebook session. - -.. image:: /_static/interactive-session.png - -REANA currently supports `Jupyter `_ notebooks. Note that -you can pass any notebook image you are interested to run on the workspace, -such as PySpark, or even your own image, by using the `--image` option. - - -Closing interactive sessions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once you finished working on your interactive session notebook, you can close it -via ``close`` command. - -.. code-block:: console - - $ reana-client close -w myanalysis.42 - Interactive session for workflow myanalysis.42 was successfully closed - -Deleting workflows -~~~~~~~~~~~~~~~~~~ - -You can mark a workflow as deleted with: - -.. code-block:: console - - $ reana-client delete - -Passing no argument will mark the workflow selected by your REANA_WORKON -variable as deleted. To specify a different workflow than your -currently selected one use the -w/--workflow flag and set the workflow name -and run number. -If e.g. workflow run number 123 of your analysis failed, you can delete it -as follows: - -.. code-block:: console - - $ reana-client delete --workflow=myanalysis.42 - -After simple deletion the workspace can be accessed to retrieve files uploaded -there. If you are sure the workspace can also be deleted pass the ---include-workspace flag. - -.. code-block:: console - - $ reana-client delete --workflow=myanalysis.42 --include-workspace - -To delete all runs of a given workflow, pass the --include-all-runs flag and -run: - -.. code-block:: console - - $ reana-client delete --workflow=myanalysis --include-all-runs - -and to completely remove a workflow run and its workspace from REANA -pass the --include-records flag: - -.. code-block:: console - - $ reana-client delete --workflow=myanalysis.42 --include-records - -Stopping workflows -~~~~~~~~~~~~~~~~~~ - -You can stop a workflow with: - -.. code-block:: console - - $ reana-client stop --force - -The workflow assigned to REANA_WORKON variable will be stopped. To specify a -different workflow than your currently selected one use the -w/--workflow flag -and set the workflow name or UUID. - -.. code-block:: console - - $ reana-client stop --force --workflow=otherworkflow.1 - -Note that currently only force stop is implemented. - -Examples --------- - -You can get inspiration on how to structure your REANA-compatible research data -analysis from several ``reana-demo-...`` examples -`provided on GitHub `_. - -Commands --------- - -The full list of ``reana-client`` commands with their documented options is -available in the :ref:`cliapi` documentation.