Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Developer guide for further development. #147

Merged
merged 16 commits into from
Jan 20, 2025
Merged
14 changes: 14 additions & 0 deletions docs/dev_guide/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. _sunpy-soar-dev-guide-index:
nabobalis marked this conversation as resolved.
Show resolved Hide resolved

***************
Developer Guide
***************

This guide will explain the internals of ``sunpy-soar`` and how it interacts with `sunpy.net.Fido`.

.. toctree::
:maxdepth: 1

working
tables
query
47 changes: 47 additions & 0 deletions docs/dev_guide/query.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
.. _sunpy-soar-dev-guide-query:

***************************************
Request methods ``sunpy-soar`` supports
***************************************

sunpy-soar currently supports two REQUEST methods: ``doQuery`` and ``doQueryFilteredByDistance``.

``doQuery``: This is the standard method used when no specific distance attribute is included in the search query.
It performs a general query based on the provided parameters, retrieving the data that matches the criteria specified.

``doQueryFilteredByDistance``: This method is employed when a distance parameter is included in the search query.
Unlike ``doQuery``, this method filters the entire query based on the specified distance value.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what is different. What does "entire query" mean? Or is that the entire database?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 54 it reads "filters the entire database"

The time attribute is not required when using ``doQueryFilteredByDistance``, as the time is internally calculated based on the provided distance.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implies that there is a computation of time ranges corresponding to a distance range. I am not sure the query works like this internally at SOAR, and in any case the way it works internally at SOAR is not relevant for the sunpy-soar user or developer.

The distance value is appended to the end of the query using ``&DISTANCE(dmin, dmax)``, where ``dmin`` and ``dmax`` are Astropy units of distance.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion:

The distance range of values is appended to the end of the query using ``&DISTANCE(dmin, dmax)``, where ``dmin`` and ``dmax`` are Astropy quantities representing distances.

These values must fall within the range of 0.28 AU to 1.0 AU; otherwise, the query will not return any results.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For DISTANCE(0 * u.au, .5 * u.au) for example (or even a negative value for dmin...), I would expect the filtering to be done on everything below 0.5AU, I don't expect the query to systematically give no result. So if this true, this could be a SOAR misfeature that we can submit to SOAR team.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to this comment SOAR does indeed not behave as I expect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nabobalis I think this was the only comment left to be addressed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a range of "outdated" comments from Eric but they don't seem to be addressed in the text?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this comment the range is up to 1.02 AU


Using the example below,

.. code-block:: python

import astropy.units as u
import sunpy.net.attrs as a
from sunpy.net import Fido

instrument = a.Instrument("RPW")
level = a.Level(2)
distance = a.soar.Distance(0.28 * u.AU, 0.30 * u.AU)

result = Fido.search(instrument & level & distance)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check this result is output.


Here the query's "REQUEST" type to "doQueryFilteredByDistance", which is a special method that filters the entire query based on the specified distance value.
This method is used when a distance parameter is included in the search query, and the time attribute is not required.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second sentence and the end of the first sentence duplicate something that has already been written.


The actual query this example produces is,

.. code-block:: python

"SELECT+*+FROM+v_sc_data_item+WHERE+instrument='RPW'+AND+level='L2'&DISTANCE(0.28,0.30)"

How can other request methods be added?
=======================================

To add support for additional request methods, you'll need to consider the impact they will have on the query structure.
The REQUEST parameter in the query must be updated accordingly, and any necessary modifications to the query string should be made to accommodate the new method.
If the new request method requires a specific attribute, you may need to add it as a class in the attrs.py file (if it doesn't already exist in `sunpy.net.attrs`).
nabobalis marked this conversation as resolved.
Show resolved Hide resolved
Additionally, a walker for this attribute will need to be created.
38 changes: 38 additions & 0 deletions docs/dev_guide/tables.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
.. _sunpy-soar-dev-guide-tables:

**********************************
Tables supported in ``sunpy-soar``
**********************************

The ``sunpy-soar`` library currently supports data retrieval from both science and low-latency data tables, such as ``v_sc_data_item`` and ``v_ll_data_item``.
Additionally, it provides support for join tables associated with remote instruments, such as ``v_eui_sc_fits``.
nabobalis marked this conversation as resolved.
Show resolved Hide resolved

In the context of ``sunpy-soar``, data tables contain columns related to scientific measurements, while instrument tables contain metadata specific to particular instruments.
The ``sunpy-soar`` library specifically supports the wavelength and detector columns within these tables.
These columns are linked to the data columns in the instrument tables through a join operation, using the ``data_item_oid`` as the key.

How can a new column be added?
==============================

If the new column you wish to add is part of an existing data or instrument FITS table, you can extend the corresponding ADQL query to include this column.
For example, if the new column is in the instrument table, it should be included in the "SELECT" clause of the query for that table.

Moreover, if one needs to enable filtering by this new column, you must consider adding it as an attribute in the ``attrs.py`` file.
If the column already exists within `sunpy.net.attrs`, you should add a corresponding walker in the ``attrs.py`` file.
The ``_can_handle_query`` method in the ``client.py`` file should also be updated to ensure proper support for this new column in queries.

How can a new table be added?
=============================

When adding support for a new table in ``sunpy-soar``, it is essential to fully understand the context and requirements.
This includes identifying the key that will be used for join operations and determining what data should be returned to the user.

For direct joins, the key relationship between the tables should be carefully identified.
Once the joins are established, the necessary columns can be included in the query, following the process outlined for adding columns.

Depending on the use case, you might need to determine the type of join (e.g., inner join, outer join, left join) and what specific tables or columns should be displayed to the user.
In some cases, conditional logic may be required, where certain attributes in the search query trigger specific join operations or column selections.

Finally, ensure that the ``_do_search`` method is updated to reflect these changes.
This method should handle any additional columns or conditional logic for attribute-specific queries, ensuring that the appropriate data is returned to the user.

47 changes: 47 additions & 0 deletions docs/dev_guide/working.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
.. _sunpy-soar-dev-guide-working:

****************
Newcomer's Guide
****************

``sunpy-soar`` is a Python library developed to interact with the SOAR (Solar Orbiter Archive) data
hosted by ESA (European Space Agency). It provides tools and utilities to access, search, download and
analyze data collected by the Solar Orbiter mission.
NucleonGodX marked this conversation as resolved.
Show resolved Hide resolved

How data is retrieved from the SOAR
===================================

To retrieve data from SOAR, you first use the `sunpy.net.Fido` object from `sunpy` and specify the desired attributes using `sunpy.net.attrs`.
These attributes define the criteria for the data you want to retrieve, such as the time range, instrument, or wavelength.

Here is an example of how to specify the time range for the data you want to retrieve:

.. code-block:: python

import sunpy.net.attrs as a
from sunpy.net import Fido
import sunpy_soar

instrument = a.Instrument("EUI")
time = a.Time("2021-02-01", "2021-02-02")
level = a.Level(1)
product = a.soar.Product("EUI-FSI174-IMAGE")

result = Fido.search(instrument & time & level & product)


``sunpy-soar`` constructs the query based on the specified criteria, then generates a URL to interact with the SOAR API.
This is done by using the Table Access Protocol (TAP), a widely adopted standard in the astronomical community for accessing large datasets.
nabobalis marked this conversation as resolved.
Show resolved Hide resolved
The results are returned in the form of an Astropy table, providing a structured and efficient format for further analysis and visualization within the Python environment.

A generated query looks like:

.. code-block:: python

"SELECT+*+FROM+v_sc_data_item+WHERE+instrument='EPD'+AND+begin_time>='2021-02-01+00:00:00'+AND+begin_time<='2021-02-02+00:00:00'+AND+level='L1'+AND+descriptor='epd-epthet2-nom-close'"
# Or with a Join
"SELECT+h1.instrument, h1.descriptor, h1.level, h1.begin_time, h1.end_time, h1.data_item_id, h1.filesize, h1.filename, h1.soop_name, h2.detector, h2.wavelength, h2.dimension_index+"
"FROM+v_sc_data_item AS h1 JOIN v_eui_sc_fits AS h2 USING (data_item_oid)+WHERE+h1.instrument='EUI'+AND+h1.begin_time>='2021-02-01+00:00:00'+AND+h1.begin_time<='2021-02-02+00:00:00'+AND+""
"h2.dimension_index='1'+AND+h1.level='L1'+AND+h1.descriptor='eui-fsi174-image'"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would split the code block in 2, write the queries as basic ADQL (no URL-encoding of spaces to pluses, no double quotes in each line; code block language could be SQL), and transform the comment into regular text in between both code blocks.


The URL is generated with the query formed based on the parameters, then Fido is used to print or download the data.
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ This will walk you through getting set up for contributing.

generated/gallery/index
how_to/index
dev_guide/index
api
changelog
coc