OPTIMAP

Geospatial discovery of research articles based on open metadata. The OPTIMETA Portal is part of the OPTIMETA project (https://projects.tib.eu/optimeta) and relies on the spatial and temporal metadata collected for scientific papers with the OPTIMETA Geo Plugin for Open Journal Systems (OJS) published at https://github.com/TIBHannover/optimetaGeo. The product name of the portal is OPTIMAP. The development is continued in the project KOMET (https://projects.tib.eu/komet).

The OPTIMAP has the following features:

• Start page with a full screen map (showing geometries and metadata) and a time line of the areas and time periods of interest for scientific publications
• Passwordless login via email
• RESTful API at /api
• Crowdsourced metadata contribution: Logged-in users can contribute spatial and temporal extent data for publications
• Publication workflow: Harvested → Contributed → Published status transitions with full provenance tracking
• Admin controls: Publish/unpublish functionality with audit trails

Publication Status Workflow

Publications in OPTIMAP follow a status-based workflow with six possible states:

Status Definitions

• Draft (d): Internal draft state. Not visible to public. Can be edited by admins. Created when unpublishing a published work.
• Harvested (h): Automatically harvested from OAI-PMH or RSS feeds. May or may not have spatial/temporal extent. Not publicly visible.
• Contributed (c): User has contributed spatial and/or temporal extent. Awaits admin review. Not publicly visible.
• Published (p): Public-facing works visible to all users via website, map, API, and feeds.
• Testing (t): Reserved for testing purposes. Not publicly visible. Admin access only.
• Withdrawn (w): Publication has been withdrawn or retracted. Not publicly visible.

Workflow Transitions

Harvesting → Publishing:

1. Publication harvested from external source → Status: Harvested (h)
2. User contributes spatial/temporal extent → Status: Contributed (c)
3. Admin reviews and approves → Status: Published (p)
4. If needed, admin can unpublish → Status: Draft (d)

Direct Publishing (Skip Contribution):

• Harvested publications with at least one extent type (spatial OR temporal) can be published directly by admins without user contribution

Contribution Requirements:

• Users can only contribute to publications with Harvested (h) status
• Harvested publications without any extent require user contribution before publishing
• Contributed publications can always be published after admin review

Visibility Rules:

• Only Published (p) status is visible to non-admin users
• All other statuses require admin privileges to view
• Published works appear in: main map, work list, API responses, RSS/Atom feeds

Extent Contribution:

• Users can contribute spatial extent (geographic location) via interactive map with drawing tools
• Users can contribute temporal extent (time period) via date form (formats: YYYY, YYYY-MM, YYYY-MM-DD)
• Both extent types can be contributed separately or together in a single submission
• Publications without DOI are supported via ID-based URLs (/work/<id>/)
• All contributions are tracked with full provenance (user, timestamp, changes)
• Contribute page lists publications missing either spatial OR temporal extent

OPTIMAP is based on Django (with GeoDjango and Django REST framework) with a PostgreSQL/PostGIS database backend.

The development of OPTIMAP was and is supported by the projects OPTIMETA and KOMET.

Configuration

All configuration is done via the file optimap/settings.py. Configurations that need to be changed for different installations and for deployment are also exposed as environment variables. The names of these environment variables start with OPTIMAP_. The settings files loads these from a file .env stored in the same location as settings.py, or from the environment the server is run it. A complete list of existing parameters is provided in the file optimap/.env.example.

Run with Docker

The project is containerized using Docker, with services defined in docker-compose.(deploy.)yml. To start all services, run:

docker compose up

docker compose run --entrypoint python app manage.py loaddata fixtures/test_data.json

The database migrations are applied as part of the startup script, see file etc/manage-and-run.sh. You can still run the commands below manually if need be, e.g., during development.

docker compose run --entrypoint python app manage.py makemigrations # should not detect and changes, otherwise your local config might be outdated
docker compose run --entrypoint python app manage.py migrate
docker compose run --entrypoint python app manage.py collectstatic --noinput

Now open a browser at http://localhost:80/.

Services Overview

• db: Runs a PostgreSQL database with PostGIS extensions. Data is persisted in a Docker volume named db_data.
• app: Our primary Django web application.
• webserver: An Nginx server for serving static files and test files.

Ports

Not all of these ports are exposed by default, but they are available for local development - just uncomment the matching lines in the docker-compose.yml file.

• 5432: Database (PostgreSQL/PostGIS)
• 8000: App (Django server)
• 80: Webserver (Nginx)

Development

Test Data

The folder /fixtures contains some test data, which can be used to populate the database during development. These data are provided either as SQL commands for insertion into the database or as database dumps that can be loaded using django-admin.

Managing Test Data

Creating Test Data Dumps

To create a data dump after generating or harvesting test data, use the following command:

python manage.py dumpdata --exclude=auth --exclude=contenttypes | jq > fixtures/test_data.json

Loading Test Data

To load the test data into your database, run the following command choosing one of the existing fixtures:

python manage.py loaddata fixtures/test_data_{optimap, partners, global_feeds}.json

Adding New Test Data

If additional test data is required, you can:

• Copy and paste existing records.
• Add new records via the Django admin interface (preferred method).
• Manually modify the test data JSON file.

Tools for Geometries For creating or editing geometries, consider using the tool WKTMap, which provides an easy way to generate Well-Known Text (WKT) representations of spatial data.

Run Locally

1. Create a .env file based on .env.example in the same directory where settings.py resides, and fill in the configuration settings as needed.
2. Run the commands below to set up and run the project locally. This setup uses the built-in Python venv. Refer to this tutorial for more details.

# Create a virtual environment (once only)
python -m venv .venv

# Activate the virtual environment
source .venv/bin/activate

# Confirm Python path
which python

# Install GDAL
gdalinfo --version

# Install gdal Pyhton library matching your GDAL version
pip install gdal=="$(gdal-config --version).*"

# Install Python dependencies
pip install -r requirements.txt

# create local DB container (once)
# docker run --name optimapDB -p 5432:5432 -e POSTGRES_USER=optimap -e POSTGRES_PASSWORD=optimap -e POSTGRES_DB=optimap -d postgis/postgis:14-3.3
# get a clean one later: docker rm -f optimapDB

# Start the database container
docker start optimapDB

# Apply database migrations
python manage.py makemigrations
python manage.py migrate

# Create a cache table
python manage.py createcachetable

# Collect static files
python manage.py collectstatic --noinput

# If you need to run tasks (harvesting, data export) then start a cluster in a separate shell
python manage.py qcluster

# If you want to use the predefined feeds for continents and oceans we need to load the geometries for global regions
python manage.py load_global_regions

# Harvest publications from real OAI-PMH journal sources
python manage.py harvest_journals --list  # List available journals
python manage.py harvest_journals --all --max-records 20  # Harvest all journals (limited to 20 records each)
python manage.py harvest_journals --journal essd --journal geo-leo  # Harvest specific journals

# Start the Django development server
python manage.py runserver

# Start the app with specific configurations for development
OPTIMAP_CACHE=dummy OPTIMAP_DEBUG=True python manage.py runserver

# Manually regenerating data export files (GeoJSON / GeoPackage cache)
## Via Django-Q cluster: if you already have a Q cluster running (e.g. `python manage.py qcluster`), you can simply add the job to the schedule table (once) by running:
python manage.py schedule_geojson
## One‐off via the Django shell: if you just want a “right‐now” rebuild (without waiting for the next 6-hour tick), drop into a one-liner:
python manage.py shell -c "from publications.tasks import regenerate_geojson_cache; regenerate_geojson_cache()"

Now open a browser at http://127.0.0.1:8000/.

Additional Setup

• Creating a Superuser: Refer to the instructions below to create a superuser for accessing the admin interface.
• Loading Test Data: Refer to the instructions above to load test data into the database.

Shutting Down

When finished, deactivate the virtual environment and stop the database container:

deactivate

docker stop optimapDB

Debug Mode Configuration

By default, OPTIMAP_DEBUG is now set to False to ensure a secure and stable production environment. If you need to enable debug mode for development purposes, explicitly set the environment variable in your .env file or pass it as an argument when running the server.

Enable Debug Mode for Development

To enable debug mode, add the following to your .env file:

OPTIMAP_DEBUG=True

Debug with VS Code

Select the Python interpreter created above (optimap environment), see instructions at https://code.visualstudio.com/docs/python/tutorial-django and https://code.visualstudio.com/docs/python/environments.

Note: The docker-compose.yml file is intended for development, not deployment.

Configuration for debugging with VS Code:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Django Run",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/manage.py",
      "args": ["runserver"],
      "env": {
        "OPTIMAP_DEBUG": "True",
        "OPTIMAP_CACHE": "dummy"
      },
      "django": true,
      "justMyCode": true
    }
  ]
}

Debug email sending

Add EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend to the .env file to have emails printed to the console instead of sent via SMTP.

Alternatively, you can run a local STMP server with the following command and configuration:

python -m smtpd -c DebuggingServer -n localhost:5587

OPTIMAP_EMAIL_HOST=localhost
OPTIMAP_EMAIL_PORT=5587

Accessing list of works

Visit the URL - http://127.0.0.1:8000/works/

Harvest Publications from real journals

The harvest_journals management command allows you to harvest publications from real OAI-PMH journal sources directly into your database. This is useful for:

• Populating your database with real data for testing and development
• Testing harvesting functionality against live endpoints
• Initial data loading for production deployment

List available journals:

python manage.py harvest_journals --list

Harvest all configured journals (with record limit):

python manage.py harvest_journals --all --max-records 50

Harvest specific journals:

# Single journal
python manage.py harvest_journals --journal essd --max-records 100

# Multiple journals
python manage.py harvest_journals --journal essd --journal geo-leo --journal agile-giss

Create source entries automatically:

python manage.py harvest_journals --journal essd --create-sources

Associate with specific user:

python manage.py harvest_journals --all --user-email [email protected]

Currently configured journals:

• essd - Earth System Science Data (OAI-PMH) (Issue #59)
• agile-giss - AGILE-GISS conference series (OAI-PMH) (Issue #60)
• geo-leo - GEO-LEO e-docs repository (OAI-PMH) (Issue #13)
• eartharxiv - EarthArXiv preprint repository (OAI-PMH, ~6,000+ preprints)
• scientific-data - Scientific Data (RSS/Atom) (Issue #58)

The command supports both OAI-PMH and RSS/Atom feeds, automatically detecting the feed type for each journal.

Harvesting EarthArxiv preprints:

EarthArxiv is a preprint server for Earth Sciences hosted by the California Digital Library. All harvested articles automatically receive metadata enrichment from OpenAlex, including author names, keywords, and topics.

# Harvest first 100 preprints for testing
python manage.py harvest_journals --journal eartharxiv --max-records 100 --create-sources

# Harvest all EarthArxiv preprints (6,000+)
python manage.py harvest_journals --journal eartharxiv --create-sources

# Harvest EarthArxiv along with other journals
python manage.py harvest_journals --journal eartharxiv --journal essd --journal geo-leo

EarthArxiv provides comprehensive coverage of Earth Science preprints via its OAI-PMH API endpoint. Each publication is automatically matched with OpenAlex to retrieve:

• Author information
• Keywords and subject classification
• Citation data
• Open access status
• Publication topics

The command provides detailed progress reporting including:

• Number of publications harvested
• Harvesting duration
• Spatial and temporal metadata statistics
• Success/failure status for each journal

When the command runs mutiple times, it will only add new publications that are not already in the database as part of the regular harvesting process.

Create Superusers/Admin

Superusers or administrators can be created using the createsuperuser command. This user will have access to the Django admin interface.

python manage.py createsuperuser --username=optimap [email protected]

You will be prompted for a password. After entering one, the superuser will be created immediately. If you omit the --username or --email options, the command will prompt you for those values interactively.

Access the admin interface at http://127.0.0.1:8000/admin/.

Running in a Dockerized App

To create a superuser within the containerized application, use the following command:

docker-compose run web python manage.py createsuperuser

This will run the same process as above but within the Docker environment. Ensure the container is running and accessible before executing this command

Run tests

See https://docs.djangoproject.com/en/4.1/topics/testing/overview/ for testing Django apps.

UI tests are based on Helium (because Pylenium would need pytest in addition).

pip install -r requirements-dev.txt

Unit Tests

Run all unit tests:

python manage.py test tests

# show deprecation warnings
python -Wa manage.py test

# configure logging level for cleaner test progress output
OPTIMAP_LOGGING_LEVEL=WARNING python manage.py test tests

Integration Tests (Real Harvesting)

Integration tests that harvest from live OAI-PMH endpoints are disabled by default to avoid network dependencies and slow test execution. These tests verify harvesting from real journal sources.

Run all integration tests:

# Enable real harvesting tests
SKIP_REAL_HARVESTING=0 python manage.py test tests.test_real_harvesting

Run a specific journal test:

# Test ESSD harvesting
SKIP_REAL_HARVESTING=0 python manage.py test tests.test_real_harvesting.RealHarvestingTest.test_harvest_essd

# Test GEO-LEO harvesting
SKIP_REAL_HARVESTING=0 python manage.py test tests.test_real_harvesting.RealHarvestingTest.test_harvest_geo_leo

Show skipped tests (these are skipped by default):

# Run with verbose output to see skip reasons
python manage.py test tests.test_real_harvesting -v 2

Supported journals:

• Earth System Science Data (ESSD) - Issue #59
• AGILE-GISS - Issue #60
• GEO-LEO e-docs - Issue #13
• ESS Open Archive (EssOAr) - Issue #99 (endpoint needs confirmation)

Run UI tests

Running UI tests needs either compose configuration or a manage.py runserver in a seperate shell.

docker-compose up --build

python -Wa manage.py test tests-ui

Check test coverage

# run the tests and capture coverage
coverage run --source='publications' --omit='*/migrations/**' manage.py test tests

# show coverage report
coverage report --show-missing --fail-under=70

# save the reports
coverage html
coverage xml

Develop tests

For developing the UI tests, you can remove the headless=True in the statements for starting the browsers so you can "watch along" and inspect the HTML when a breakpoint is hit as the tests are executed.

Debug tests with VS Code

A configuration to debug the test code and also print deprecation warnings:

{
  "name": "Python: Django Test",
  "type": "python",
  "request": "launch",
  "pythonArgs": ["-Wa"],
  "program": "${workspaceFolder}/manage.py",
  "args": ["test", "tests"],
  "env": {
    "OPTIMAP_DEBUG": "True"
  },
  "django": true,
  "justMyCode": true
}

Change the argument tests to tests-ui to run the UI tests.

See also documentation at https://code.visualstudio.com/docs/python/tutorial-django.

Issues during development

• If you get a message during login that there is an issue with the CSRF token, e.g. WARNING:django.security.csrf:Forbidden (CSRF token from POST incorrect.): /loginres/ in the log and also i nthe UI, then switch to using localhost:8000 as the domain, not the localhost IP used in the examples in this README file.

Contributing

All contributions are welcome! We appreciate any feedback, suggestions, or code contributions to improve the project. Please follow the contribution guidelines for more details.

Changelog

We operate a changelog to keep track of changes and updates to the project. The changelog follows the Keep a Changelog format and is versioned according to Semantic Versioning.

The version is managed in optimap/__init__.py.

Design colours and logos

Optimeta colour = primary colour: #158F9B

Complimentary colour for warnings, errors: #9B2115

Colours for highlighting (split complimentary): #3C159B #9B7115

For future use, optional colours for variation, e.g., for different map features: #158F9B #159B71 #159B8C #158F9B #15749B #15599B

The logos and favicon are in the repository in the folder publications/static/.

Deploy

The app is deployed in the TUD Enterprise Cloud. HTTPS certificate is retrieved via certbot, see docker-compose.deploy.yml for the configuration and documentation links.

Operation

Block Emails/Domains

What It Does

• Blocks specific emails and entire domains from registering.
• Prevents login attempts from blocked users.
• Admin can delete users and instantly block their email/domain.

How to Use in Django Admin

1. Manually Add Blocked Emails/Domains
   • Go to /admin/
   • Add emails in Blocked Emails or domains in Blocked Domains.
2. Block Users via Admin Action
   • Go to /admin/auth/user/
   • Select users → Choose "Delete user and block email/domain" → Click Go.

Tasks

We use Django Q2 for scheduling (repeated) tasks.

Run the cluster

python manage.py qcluster

Monitor

Details: https://django-q2.readthedocs.io/en/master/monitor.html

tl;dr:

python manage.py qmonitorq

python manage.py qinfo

License

This software is published under the GNU General Public License v3.0 (see file LICENSE). For licenses of used libraries and dependencies, e.g., scripts and CSS files in publications/static/, see respective files and projects.