add documentation for the preprocessor

This defines what the preprocessor aims to do, why and what it's
intentional limitations are.

Examples on how to use it and how to use the "Defines DB" are also
provided

Author: wnienhaus

README.rst

@@ -24,6 +24,11 @@ Constants defined with ``.set`` are supported in expressions.
 We have some unit tests and also compatibility tests that compare the output
 whether it is identical with binutils-esp32ulp output.
 
+There is a simple preprocessor that understands just enough to allow assembling
+ULP source files containing convenience macros such as WRITE_RTC_REG. The
+preprocessor and how to use it is documented here:
+`Preprocessor support <docs/preprocess.rst>`_.
+
 There might be some stuff missing, some bugs and other symptoms of alpha
 software. Also, error and exception handling is rather rough yet.
 

docs/preprocess.rst

@@ -0,0 +1,139 @@
+Preprocessor
+---------------------
+
+py-esp32-ulp contains a small preprocessor, which aims to fulfill one goal:
+facilitate assembling of ULP code from Espressif and other open-source
+projects to loadable/executable machine code without modification.
+
+Such code uses convenience macros (``READ_RTC_*`` and ``WRITE_RTC_*``)
+provided by the ESP-IDF framework, along with constants defined in the
+framework's include files (such as ``RTC_GPIO_IN_REG``), to make reading
+and writing from/to peripheral registers much easier.
+
+In order to do this the preprocessor has two capabilities:
+
+1. Parse and replace identifiers defined with ``#define``
+2. Recognise the ``WRITE_RTC_*`` and ``READ_RTC_*`` macros and expand
+   them in a way that mirrors what the real ESP-IDF macros do.
+
+
+Usage
+------------------------
+
+Normally the assembler is called as follows
+
+.. code-block:: python
+
+    src = "..full assembler file contents"
+    assembler = Assembler()
+    assembler.assemble(src)
+    ...
+
+With the preprocessor, simply pass the source code via the preprocessor first:
+
+.. code-block:: python
+
+    from preprocess import Preprocessor
+
+    src = "..full assembler file contents"
+    preprocessor = Preprocessor()
+    src = preprocessor.preprocess(src)
+    assembler = Assembler()
+    assembler.assemble(src)
+    ...
+
+
+Using a "Defines Database"
+--------------------------
+
+Because the py-esp32-ulp assembler was built for running on the ESP32
+microcontroller with limited RAM, the preprocessor aims to work there too.
+
+To handle large number of defined constants (such as the ``RTC_*`` constants from
+the ESP-IDF) the preprocessor can use a database (based on BerkleyDB) stored on the
+device's filesystem for looking up defines.
+
+The database needs to be populated before preprocessing. (Usually, when only using
+constants from the ESP-IDF, this is a one-time step, because the include files
+don't change.) The database can be reused for all subsequent preprocessor runs.
+
+(The database can also be generated on a PC and then deployed to the ESP32, to
+save processing effort on the device. In that case the include files themselves
+are not needed on the device either.)
+
+1. Build the defines database
+
+  The ``esp32_ulp.parse_to_db`` tool can be used to generate the defines
+  database from include files. The resulting file will be called
+  ``defines.db``.
+
+  (The following assume running on a PC. To do this on device, refer to the
+  `esp32_ulp/parse_to_db.py <../esp32_ulp/parse_to_db.py>`_ file.)
+
+  .. code-block:: bash
+
+     # general command
+     micropython -m esp32_ulp.parse_to_db path/to/include.h
+
+     # loading specific ESP-IDF include files
+     micropython -m esp32_ulp.parse_to_db esp-idf/components/soc/esp32/include/soc/soc_ulp.h
+
+     # loading multiple files at once
+     micropython -m esp32_ulp.parse_to_db esp-idf/components/soc/esp32/include/soc/*.h
+
+     # if file system space is not a concern, the following can be convenient
+     # by including all relevant include files from the ESP-IDF framework
+     micropython -m esp32_ulp.parse_to_db \
+       esp-idf/components/soc/esp32/include/soc/*.h \
+       esp-idf/components/esp_common/include/*.h
+
+2. Tell the Preprocessor to use the database
+
+  Once the defines database has been created (i.e. you have a populated
+  ``defines.db`` file), you can instruct the preprocessor to use this
+  database.
+
+  .. code-block:: python
+
+     from preprocess import Preprocessor
+     from definesdb import DefinesDB
+
+     preprocessor = Preprocessor()
+     db = DefinesDB()
+     preprocessor.use_db(db)
+     src = preprocessor.preprocess(src)
+     ...
+
+  Note: You can also use the above approach by default, because the preprocessor
+  will treat an absent database like an empty database (and take care not to
+  create an empty database file cluttering up the filesystem).
+
+Design choices
+--------------
+
+The preprocessor does not support:
+
+1. Function style macros such as :code:`#define f(a,b) (a+b)`
+
+   This is not important, because there are only few RTC macros that need
+   to be supported and they are simply implemented as Python functions.
+
+   Since the preprocessor will understand ``#define`` directives directly in the
+   assembler source file, include mechanisms are not needed in some cases
+   (simply copying the needed ``#define`` statements from include files into the
+   assembler source will work).
+
+2. ``#include`` directives
+
+   The preprocessor does not currently follow ``#include`` directives. To
+   limit space requirements (both in memory and on disk), the preprocessor
+   relies on a database of defines (key/value pairs). This database should be
+   populated before using the preprocessor, by using the ``esp32_ulp.parse_to_db``
+   tool (see section below), which parses include files for identifiers
+   defined therein.
+
+3. Preserving comments
+
+   The assumption is that the output will almost always go into the
+   assembler directly, so preserving comments is not very useful and
+   would a lot of complexity.