Add Developer guide for further development. (#147)

Co-authored-by: Nabil Freij <[email protected]>

changelog/147.doc.rst

@@ -0,0 +1 @@
+Added developer guide with information about the internal logic of sunpy-soar.

docs/dev_guide/index.rst

@@ -0,0 +1,14 @@
+.. _sunpy-soar-dev-guide-index:
+
+***************
+Developer Guide
+***************
+
+This guide will explain the internals of ``sunpy-soar`` and how it interacts with `sunpy.net.Fido`.
+
+.. toctree::
+   :maxdepth: 1
+
+   working
+   tables
+   query

docs/dev_guide/query.rst

@@ -0,0 +1,68 @@
+.. _sunpy-soar-dev-guide-query:
+
+***************************************
+Request methods ``sunpy-soar`` supports
+***************************************
+
+sunpy-soar currently supports two REQUEST methods: ``doQuery`` and ``doQueryFilteredByDistance``.
+
+``doQuery``: This is the standard method used when no specific distance attribute is included in the search query.
+It performs a general query based on the provided parameters, retrieving the data that matches the criteria specified.
+
+``doQueryFilteredByDistance``: This method is employed when a distance parameter is included in the search query.
+Unlike ``doQuery``, this method filters the entire database based on the specified distance value.
+The time attribute is not necessarily required when using ``doQueryFilteredByDistance``.
+The distance range of values is appended to the end of the query using ``&DISTANCE(dmin, dmax)``, where ``dmin`` and ``dmax`` are Astropy quantities representing distances.
+These values must fall within the range of 0.28 AU to 1.02 AU; otherwise, the query will not return any results.
+
+Using the example below,
+
+.. code-block:: python
+
+    >>> import astropy.units as u
+    >>> import sunpy.net.attrs as a
+    >>> from sunpy.net import Fido
+
+    >>> instrument = a.Instrument("RPW")
+    >>> level = a.Level(2)
+    >>> distance = a.soar.Distance(0.28 * u.AU, 0.30 * u.AU)
+    >>> result = Fido.search(instrument & level & distance) # doctest: +REMOTE_DATA
+    >>> result # doctest: +REMOTE_DATA
+    <sunpy.net.fido_factory.UnifiedResponse object at ...>
+    Results from 1 Provider:
+    <BLANKLINE>
+    357 Results from the SOARClient:
+    <BLANKLINE>
+    Instrument     Data product    Level        Start time               End time        Filesize SOOP Name
+                                                                                          Mbyte
+    ---------- ------------------- ----- ----------------------- ----------------------- -------- ---------
+           RPW rpw-tds-surv-tswf-b    L2 2022-10-09 00:00:00.000 2022-10-10 00:00:00.000   13.748      None
+           RPW    rpw-lfr-surv-bp1    L2 2022-10-09 00:00:00.000 2022-10-10 00:00:00.000   61.818      None
+           RPW rpw-tds-surv-tswf-e    L2 2022-10-09 00:00:00.000 2022-10-10 00:00:00.000    95.65      None
+           RPW    rpw-lfr-surv-bp2    L2 2022-10-09 00:00:00.000 2022-10-10 00:00:00.000  145.551      None
+           RPW  rpw-lfr-surv-swf-e    L2 2022-10-09 00:00:00.000 2022-10-10 00:00:00.000   47.452      None
+           ...                 ...   ...                     ...                     ...      ...       ...
+           RPW   rpw-tds-surv-stat    L2 2023-10-10 00:00:00.000 2023-10-11 00:00:00.000    0.531      None
+           RPW rpw-tds-surv-hist2d    L2 2023-10-10 00:00:00.000 2023-10-11 00:00:00.000    3.382      None
+           RPW        rpw-tnr-surv    L2 2023-10-10 00:00:00.000 2023-10-11 00:00:00.000  341.182      None
+           RPW  rpw-lfr-surv-swf-e    L2 2023-10-10 00:00:00.000 2023-10-11 00:00:00.000  200.206      None
+           RPW rpw-tds-surv-tswf-e    L2 2023-10-10 00:00:00.000 2023-10-11 00:00:00.000  173.184      None
+    Length = 357 rows
+    <BLANKLINE>
+    <BLANKLINE>
+
+Here the query's "REQUEST" type to "doQueryFilteredByDistance", which is a special method that filters the entire database based on the specified distance value.
+
+The actual query this example produces is,
+
+.. code-block:: python
+
+   "SELECT+*+FROM+v_sc_data_item+WHERE+instrument='RPW'+AND+level='L2'&DISTANCE(0.28,0.30)"
+
+How can other request methods be added?
+=======================================
+
+To add support for additional request methods, you'll need to consider the impact they will have on the query structure.
+The REQUEST parameter in the query must be updated accordingly, and any necessary modifications to the query string should be made to accommodate the new method.
+If the new request method requires a specific attribute, you may need to add it as a class in the attrs.py file (if it doesn't already exist in (if it doesn't already exist in `sunpy.net.attrs <https://github.com/sunpy/sunpy/blob/main/sunpy/net/attrs.py/>`__).
+Additionally, a walker for this attribute will need to be created.

docs/dev_guide/tables.rst

@@ -0,0 +1,78 @@
+.. _sunpy-soar-dev-guide-tables:
+
+**********************************
+Tables supported in ``sunpy-soar``
+**********************************
+
+The ``sunpy-soar`` library currently supports data retrieval from both science and low-latency data tables, such as ``v_sc_data_item`` and ``v_ll_data_item``.
+Additionally, it provides support for join tables associated with remote instruments, such as ``v_eui_sc_fits``.
+These tables and their columns are described in the `Tables, Views, and Columns <https://www.cosmos.esa.int/web/soar/tables-views-and-columns/>`__.
+
+In the context of ``sunpy-soar``, data tables contain columns related to scientific measurements, while instrument tables contain metadata specific to particular instruments.
+The ``sunpy-soar`` library specifically supports the wavelength and detector columns within these tables.
+These columns are linked to the data columns in the instrument tables through a join operation, using the ``data_item_oid`` as the key.
+
+How can a new column be added?
+==============================
+
+If the new column you wish to add is part of an existing data or instrument FITS table, you can extend the corresponding ADQL query to include this column.
+For example, if the new column is in the instrument table, it should be included in the "SELECT" clause of the query for that table.
+
+Moreover, if one needs to enable filtering by this new column, you must consider adding it as an attribute in the ``attrs.py`` file.
+If the column already exists within `sunpy.net.attrs`, you should add a corresponding walker in the ``attrs.py`` file.
+The ``_can_handle_query`` method in the ``client.py`` file should also be updated to ensure proper support for this new column in queries.
+
+How can a new table be added?
+=============================
+
+When adding support for a new table in ``sunpy-soar``, it is essential to fully understand the context and requirements.
+This includes identifying the key that will be used for join operations and determining what columns should be returned to the user.
+
+For direct joins, the key relationship between the tables should be carefully identified.
+Once the joins are established, the necessary columns can be included in the query, following the process outlined for adding columns.
+
+Depending on the use case, you might need to determine the type of join (e.g., inner join, outer join, left join) and what specific tables or columns should be displayed to the user.
+
+Finally, ensure that the ``_do_search`` method is updated to reflect these changes.
+This method should handle any additional columns or conditional logic for attribute-specific queries, ensuring that the appropriate columns is returned to the user.
+
+Allowing an attribute to work in the search query
+=================================================
+
+For any attribute to work in the search query, we can divide the situation in two cases:
+
+Case 1 - It is already there in `sunpy.net.attrs`, in this situation just a walker needs to be created.
+
+Example on its implementation:
+
+.. code-block:: python
+
+    # Adding a walker applier for that attribute.
+    @walker.add_applier(a.AttrName)
+    def _(wlk, attr, params):
+        params.append(f"Detector='{attr.value}'")
+
+    # Depending upon the attribute the value being added could be a range of values also.
+    @walker.add_applier(a.AttrName)
+    def _(wlk, attr, params)
+        attrmin = attr.min.value
+        attrmax = attr.max.value
+        # appending the attribute depending upon how it should be used in the query
+        params.append(f"Attrmin='{attrmin}'+AND+Attrmax='{attrmax}'")
+
+Case 2 - It is not already there in `sunpy.net.attrs`, in this situation we have to introduce the attribute and a walker needs to be created.
+
+Example on its implementation:
+
+.. code-block:: python
+
+    class Attrname("Type of attribute: SimpleAttr, Range"):
+    """
+    Description of attribute
+    """
+    # Depending on the attribute, this would include methods to make it easy to use in the search query
+    # for example in case of Product, we convert it to lower case for it to be used.
+    def __init__(self, value):
+        self.value = value.lower()
+
+    # After this a walker applier need to be added, which have been shown above

docs/dev_guide/working.rst

@@ -0,0 +1,45 @@
+.. _sunpy-soar-dev-guide-working:
+
+
+How data is retrieved from the SOAR
+===================================
+
+To retrieve data from SOAR, you first use the `sunpy.net.Fido` object from `sunpy` and specify the desired attributes using `sunpy.net.attrs`.
+These attributes define the criteria for the data you want to retrieve, such as the time range, instrument, or wavelength.
+
+Here is an example of how to specify the time range for the data you want to retrieve:
+
+.. code-block:: python
+
+    import sunpy.net.attrs as a
+    from sunpy.net import Fido
+    import sunpy_soar
+
+    instrument = a.Instrument("EUI")
+    time = a.Time("2021-02-01", "2021-02-02")
+    level = a.Level(1)
+    product = a.soar.Product("EUI-FSI174-IMAGE")
+
+    result = Fido.search(instrument & time & level & product)
+
+
+``sunpy-soar`` constructs the query based on the specified criteria, then generates a URL to interact with the SOAR API.
+This is done by using the Table Access Protocol (TAP), a widely adopted standard in the astronomical community for accessing large datasets.
+The queries are formulated in Astronomical Data Query Language (`ADQL <https://www.ivoa.net/documents/ADQL/>`__), which allows for flexible querying of astronomical data.
+The results are returned in the form of an Astropy table, providing a structured and efficient format for further analysis and visualization within the Python environment.
+
+A generated query looks like:
+
+.. code-block:: SQL
+
+    SELECT * FROM v_sc_data_item WHERE instrument='EPD' AND begin_time>='2021-02-01 00:00:00' AND begin_time<='2021-02-02 00:00:00' AND level='L1' AND descriptor='epd-epthet2-nom-close'
+
+Or with a JOIN
+
+.. code-block:: SQL
+
+    SELECT h1.instrument, h1.descriptor, h1.level, h1.begin_time, h1.end_time, h1.data_item_id, h1.filesize, h1.filename, h1.soop_name, h2.detector, h2.wavelength, h2.dimension_index
+    FROM v_sc_data_item AS h1 JOIN v_eui_sc_fits AS h2 USING (data_item_oid) WHERE h1.instrument='EUI' AND h1.begin_time>='2021-02-01 00:00:00' AND h1.begin_time<='2021-02-02 00:00:00' AND
+    h2.dimension_index='1' AND h1.level='L1' AND h1.descriptor='eui-fsi174-image'
+
+The URL is generated with the query formed based on the parameters, then Fido is used to search and download the data.

docs/index.rst

@@ -50,6 +50,7 @@ This will walk you through getting set up for contributing.
 
    generated/gallery/index
    how_to/index
+   dev_guide/index
    api
    whatsnew/index
    coc