Update of the documentation to prepare for the old website to be offl…

…ine.

README.md

@@ -14,10 +14,9 @@
   </em>
   <br />
   <em>
-    <a href="https://whotracks.me/trackers.html" target="_blank" rel="noopener noreferrer">Trackers</a>
-    · <a href="https://whotracks.me/websites.html" target="_blank" rel="noopener noreferrer">Websites</a>
-    · <a href="https://whotracks.me/blog.html" target="_blank" rel="noopener noreferrer">Blog</a>
-    · <a href="https://whotracks.me/explorer.html" target="_blank" rel="noopener noreferrer">Explorer</a>
+    <a href="https://www.ghostery.com/whotracksme/trackers" target="_blank" rel="noopener noreferrer">Trackers</a>
+    · <a href="https://www.ghostery.com/whotracksme/websites" target="_blank" rel="noopener noreferrer">Websites</a>
+    · <a href="https://www.ghostery.com/whotracksme/explorer" target="_blank" rel="noopener noreferrer">Explorer</a>
   </em>
 </p>
 
@@ -29,96 +28,36 @@
     <img alt="License Badge" src="https://img.shields.io/github/license/ghostery/whotracks.me?style=flat-square"></a>
 </p>
 
-This repository contains:
-
-- data on trackers and websites as shown on [whotracks.me](https://whotracks.me/) (WTM)
-- database mapping tracker domains to companies
-- code to render the [whotracks.me](https://whotracks.me/) site
-
----
-
-> :warning: **Upcoming changes**
->
-> We are in the process of migrating the website to [ghostery.com/whotracksme](https://ghostery.com/whotracksme):
->
-> * https://www.ghostery.com/whotracksme
-> * https://www.ghostery.com/whotracksme/trackers
-> * https://www.ghostery.com/whotracksme/websites
-> * https://www.ghostery.com/whotracksme/explorer
->
-> The following documentation has not been updated yet and still points to the old website.
-> You can already try out the new website with the links above. If you have any feedback,
-> about missing functionality or if you spot inconsistencies, feel free to create a
-> [Github issue](https://github.com/whotracksme/whotracks.me/issues).
->
-> Monthly data dumps will not be affected by these changes. The licensing also remains unchanged.
->
-> For more information, see https://github.com/whotracksme/whotracks.me/issues/367
->
-> For now, if you came from the Ghostery website and have question on the data itself
-> (e.g. how to download or looking for a technical documentation of the fields),
-> it is best to start here:
->
-> https://github.com/whotracksme/whotracks.me/blob/master/whotracksme/data/Readme.md
->
-> If it does not answer your questions, also feel free to create a
-> [Github issue](https://github.com/whotracksme/whotracks.me/issues).
-> It helps us to improve the missing parts of the documentation.
-
----
-
-# Installation
-
-Python 3.11 is needed to build the site. We recommend creating a
-[virtualenv](http://docs.python-guide.org/en/latest/dev/virtualenvs/)
-to install the dependencies, or use `pipenv` or )
-
-to
-
-```sh
-python -m venv venv
-. venv/bin/activate
-```
-
-After the initial setup, you can proceed with installing whotracks.me.
+# Downloading the data
 
-For nushell:
+Each month, we release a new version of the web site. The data from the last month can be directly [accessed through the website](https://www.ghostery.com/whotracksme/explorer).
 
-```nushell
-python -m virtualenv venv
-overlay use venv/bin/activate.nu
-```
+The raw data, from which the graphs have been computed, is also available as an open data set (updated every month). You can also
+download historical data. More information on the raw data can be found [here]](`whotracksme/data/Readme.md`).
 
-## With Pip
+WhoTracks.me also builts heavily on another open source project called [TrackerDB](https://github.com/ghostery/trackerdb);
+all meta data (e.g. company descriptions) is maintained there.
 
-```sh
-$ python -m pip install git+https://github.com/ghostery/whotracks.me.git
-```
-
-## From source
+# Using the data
 
-After cloning the repository:
+You can directly use the [raw data](`whotracksme/data/Readme.md`), which are all text files. As an alternative, you an also
+download it locally and use the Python API:
 
-```sh
-$ python -m pip install -r requirements.txt
-$ python -m pip install -e .
+```
+python3.11 -m venv venv
+. venv/bin/activate
+pip install git+https://github.com/ghostery/whotracks.me.git
 ```
 
-That’s all you need to get started\!
-
-# Downloading the data
-
-Each month, we release a new version of the web site. The raw data, from which the
-graphs have been computed, are also available as an open data set (updated every month).
-
-The data from month can be also directly [accessed through the website](https://whotracks.me/explorer.html).
-
-More information on the raw data can be found in `whotracksme/data/Readme.md`.
+... or if you have locally checked it out:
 
-# Using the data
+```
+python3.11 -m venv venv
+. venv/bin/activate
+pip install -r requirements.txt
+```
 
-To get started with the data, everything you need can be found in
-`whotracksme.data`:
+The Python API can now be accessed as follows (make sure you have already downloaded data):
 
 ```python
 from whotracksme.data.loader import DataSource
@@ -131,7 +70,7 @@ data.companies
 data.sites
 ```
 
-A whitepaper for whotracks.me is available at https://arxiv.org/abs/1804.08959, and here's a BibTeX entry that you can use to cite it in a publication:
+A whitepaper for WhoTracks.me is available at https://arxiv.org/abs/1804.08959, and here's a BibTeX entry that you can use to cite it in a publication:
 
 ```
 @misc{whotracksme,
@@ -144,56 +83,24 @@ A whitepaper for whotracks.me is available at https://arxiv.org/abs/1804.08959,
 }
 ```
 
-# Building the site
-
-Building the site requires a few extra dependencies, not installed by
-default to not make the installation heavier than it needs to be. You
-will need to install `whotracksme` from the repository, because not all
-assets are packaged with `whotracksme` released on pypi:
-
-```sh
-$ python -m pip install -r requirements-dev.txt
-$ python -m pip install -e '.[dev]'
-```
-
-Once this is done, you will have access to a `whotracksme` entry point
-that can be used this way:
-
-```sh
-$ whotracksme website [serve]
-```
-
-The `serve` part is optional and can be used while making changes on the
-website.
-
-All generated artifacts can be found in the `_site/` folder.
-
-> If you debug the website generator, the parallel execution can be
-> disabled by setting the environment variable DEBUG=1.
-
-## Tests
-
-To run tests, you will need `pytest`, or simply install `whotracksme`
-with the `dev` extra:
-
-```sh
-$ python -m pip install -e '.[dev]'
-$ pytest
-```
-
 # Contributing
 
-We are happy to take contributions on:
+We rely on contributions for the community to keep the quality of this project high. If you want, you can support us in multiple ways:
+* Do you see inconsistencies in the data? Please open a Github issue [here](https://github.com/whotracksme/whotracks.me/issues). We will have a look!
+* Do you see wrong company descriptions? Did we put something in the category? Please check out the [TrackerDB project](https://github.com/ghostery/trackerdb), where all the meta data is kept, and open an [issue](https://github.com/ghostery/trackerdb/issues), or send us a pull request.
+* Do you have any feedback on the [WhoTracks.me homepage](https://www.ghostery.com/whotracksme) or about the documentation? Please, let us know, so we can improve.
 
-- Guest articles for our blog in the topics of tracking, privacy and security. Feel free to use the data in this repository if you need inspiration.
-- Feature requests that are doable using the WTM database.
-- Curating our database of tracker profiles. Open an issue if you spot anything odd.
+You can also contact us via email at [[email protected]](mailto:[email protected])
 
 # Right to Amend
 
 Please read our [Guideline for 3rd parties](https://github.com/ghostery/whotracks.me/blob/master/RIGHT_TO_AMEND.md) wanting to suggest
 corrections to their data.
 
+# Local builds
+
+[Readme on local builds](docs/local-build.md) (this is mostly relevant for the maintainer of this project)
+
 # License
 
 The content of this project itself is licensed under the [Creative

docs/local-build.md

@@ -0,0 +1,40 @@
+# Local development
+
+## Building the website (the static HTML based version) and the internal API
+
+The code to build the website on https://www.ghostery.com/whotracksme is not public;
+but for local testing, you can still use the code for the [previous version](https://web.archive.org/web/20240501140903/https://whotracks.me/).
+
+Python 3.11 is needed to build the site (Python 3.12 is currently not supported):
+
+```sh
+python3.11 -m venv venv
+. venv/bin/activate
+pip -r requirements-dev.txt
+pip install -e '.[dev]'
+```
+
+If you have not done so, make sure that you have downloaded data (see [Data Readme](../whotracksme/data/Readme.md)).
+
+```
+whotracksme website
+```
+
+It will generate static HTML files in the `_site` directory. Plus, it will also create a JSON files
+in the `_site/api/` directory. Use them at your own risk, since the format is expected to change over
+time. If there is interest to stabilize the API files, let us know. Currently, it is only used internally
+within Ghostery to power the new website.
+
+> Hint: if you debug the website generator, the parallel execution can be
+> disabled by setting the environment variable DEBUG=1.
+
+## Tests
+
+To run the unit tests:
+
+```sh
+python3.11 -m venv venv
+. venv/bin/activate
+python -m pip install -e '.[dev]'
+pytest
+```