Erin dev

.github/workflows/generate_relation_data.yml

@@ -0,0 +1,78 @@
+name: Generate Relation Data
+
+on:
+  workflow_dispatch:  # Manual trigger
+  schedule:
+    - cron: '0 0 * * 0'  # Weekly on Sunday at midnight
+
+jobs:
+  generate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18'
+
+      - name: Install jq
+        run: sudo apt-get update && sudo apt-get install -y jq
+
+      - name: Get relationship types
+        working-directory: relation
+        run: |
+          curl -G "https://sparql.vanderbilt.edu/sparql" \
+            --data-urlencode 'query=
+          PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+
+          SELECT DISTINCT ?subject ?label
+          WHERE {
+            VALUES (?collection) {
+              (<http://syriaca.org/taxonomy/directed-relations-collection>)
+              (<http://syriaca.org/taxonomy/mutual-relations-collection>)
+            }
+            ?collection skos:member ?subject .
+            ?subject skos:prefLabel ?label .
+          }
+          ORDER BY ?label
+          ' \
+          -H "Accept: application/sparql-results+json" > relationship_types.json
+
+          echo "✓ Relationship types retrieved"
+          jq '.results.bindings | length' relationship_types.json
+
+      - name: Query all relation factoids
+        working-directory: relation
+        run: |
+          chmod +x batch_query_relationships.sh
+          ./batch_query_relationships.sh
+
+          echo "✓ All relation factoids generated"
+          ls -lh all_relation_factoids.json
+
+      - name: Filter relation factoids
+        working-directory: relation
+        run: |
+          node filter_factoids.js
+
+          echo "✓ Filtered relation factoids generated"
+          ls -lh filtered_relation_factoids.json
+
+      - name: Query person relation factoids
+        working-directory: relation
+        run: |
+          chmod +x batch_query_person_relationship.sh
+          ./batch_query_person_relationship.sh
+
+          echo "✓ Person relation factoids generated"
+          ls -lh all_person_relation_factoids.json
+
+      - name: Commit and push changes
+        run: |
+          git config --local user.email "github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
+          git add relation/*.json
+          git diff --staged --quiet || git commit -m "chore: update relation data files [skip ci]"
+          git push

README.md

@@ -1,124 +1,160 @@
-# SPEAR - Syriaca Prosopographical Event Analysis and Research
+# SPEAR — Syriaca Prosopographical Event Analysis and Research
 
-A modern web application for exploring prosopographical data from the Syriaca.org project, built with vanilla JavaScript and SPARQL queries.
+A web application for exploring prosopographical data from the Syriaca.org
+project, built with vanilla JavaScript (ES modules) and SPARQL queries against
+an AWS Neptune triplestore.
+
+> **Developers:** for detailed, code-accurate technical documentation
+> (architecture, boot lifecycle, SPARQL query construction, caching,
+> Neptune/API Gateway backend, local development, and operations), see
+> **[`TECHNICAL.md`](./TECHNICAL.md)**.
 
 ## Overview
 
-SPEAR provides an interactive interface for searching and analyzing historical persons, events, and relationships from Syriac sources. The application uses SPARQL queries against a Neptune database with a publicly accessible API to enable complex multi-faceted searches.
+SPEAR provides an interactive interface for searching and analyzing historical
+persons, events, and relationships from Syriac sources. Search runs against a
+public SPARQL endpoint backed by a Neptune database. To reduce load, default and
+simple views are served from bundled JSON snapshots, and only complex facet
+combinations issue live SPARQL queries.
 
 ## Features
 
-### Multi-Modal Search
-- **Person Search**: Find individuals by name, occupation, gender, relationships, and associated places
-- **Event Search**: Explore historical events with filtering by participants, locations, and keywords
-
-### Advanced Filtering
-- **Multi-select filters**: Combine multiple criteria (events, relationships, places, occupations)
-- **Source filtering**: Filter by specific historical sources (Letters of Severus, Lives of Eastern Saints, Chronicle of Edessa)
-- **Uncertainty analysis**: Filter by certainty levels of historical claims
-- **Geographic filtering**: Search by birth places, death places, residences, and event locations
-
-### Performance Optimizations
-- **Query timeout handling**: 15-second timeout prevents hanging requests
-- **Result limiting**: Maximum 500 results per query for optimal performance
-- **POST requests**: Eliminates URL length restrictions for complex queries
-- **Debounced search**: Reduces server load during typing
-
-## Technical Architecture
+### Multi-modal search
+- **Person search** — find individuals by name, occupation, gender,
+  relationships, and associated places.
+- **Event search** — explore historical events, filtering by participants,
+  places, event keywords, and sources.
+- **Relation search** — explore documented relationships, filtering by source
+  and relationship type.
+
+### Filtering
+- **Multi-select facets** — combine events, relationships, places, occupations,
+  gender, and sources.
+- **Source filtering** — Letters of Severus, Lives of the Eastern Saints,
+  Chronicle of Edessa. (Selecting all three is treated as no source filter.)
+- **Uncertainty filter** — exclude claims by certainty level.
+- **Geographic filtering** — birth place, death place, residence, and event
+  place.
+- **Shareable URLs** — active mode and facet selections are encoded in the URL.
+
+### Performance
+- **JSON-first** — default/simple views read from bundled JSON; live SPARQL is a
+  fallback for complex facet combinations.
+- **Client-side caching** — facet menus and event-factoid results are cached in
+  `localStorage` (24h TTL) to reduce endpoint traffic.
+- **Backend safeguards** — API Gateway throttling plus a Neptune query timeout
+  (`neptune_query_timeout`) protect the triplestore.
+
+## Technical architecture
 
 ### Frontend
-- **Vanilla JavaScript**: No framework dependencies
-- **Modular design**: Separate modules for persons, events, and factoids
-- **Responsive UI**: Bootstrap-based interface with collapsible filters
-- **URL state management**: Shareable URLs with filter parameters
-
-### Backend Integration
-- **SPARQL Protocol**: Standard SPARQL queries via POST requests
-- **Neptune compatibility**: Optimized for AWS Neptune SPARQL endpoint
-- **Content-Type**: `application/sparql-query` for proper SPARQL handling
-- **Error handling**: Graceful degradation on query failures or timeouts
-
-## Getting Started
+- **Vanilla JavaScript, no build step** — plain ES modules, no bundler or
+  `package.json`.
+- **Mode router** (`mode.js`) — routes between `person`, `event`, and `relation`
+  modes based on the `?type=` URL parameter; manages mount lifecycle and
+  browser history.
+- **Per-mode modules** (`modes/*.js`) — each exports `sidebar`, `bind`, `fetch`,
+  and `render`.
+- **Query builders** (`{person,event,relation}/search.js`) — construct SPARQL
+  from facet state and fetch results.
+- **Responsive UI** — Bootstrap-based sidebar with collapsible filter sections.
+
+### Backend
+- **SPARQL over HTTP GET** — requests use `?query=<url-encoded>` with
+  `Accept: application/sparql-results+json`.
+- **AWS Neptune** — cluster (writer + reader) in a VPC, reached via API Gateway
+  → VPC Link → NLB, targeting the reader endpoint.
+- **Error handling** — failed/timed-out queries degrade to empty results.
+
+See [`TECHNICAL.md`](./TECHNICAL.md) for the full backend topology, security
+posture, and tuning guidance.
+
+## Getting started
 
 ### Prerequisites
-- Modern web browser with JavaScript enabled
-- Access to SPARQL endpoint (configured in `person/search.js`)
+- A modern browser with JavaScript enabled.
+- Access to the configured SPARQL endpoint.
 
-### Installation
-1. Clone the repository
-2. Update `SPARQL_ENDPOINT` in search modules to point to your endpoint
-3. Serve files via HTTP server (required for CORS)
+### Run locally
+No build step. Serve the folder over HTTP (ES modules and `localStorage` do not
+work under `file://`):
 
-### Usage
-1. Open `browse.html` in your browser
-2. Select search mode (Person, Event, or Factoid)
-3. Apply filters using the sidebar controls
-4. View results in the main content area
-5. Share searches via URL parameters
+```bash
+python3 -m http.server 8000
+# then open http://localhost:8000/browse.html
+```
 
-## Configuration
+Alternatives: `npx serve`, or the VS Code "Live Server" extension.
 
-### SPARQL Endpoint
-Update the endpoint URL in:
-- `person/search.js`
-- `event/search.js` 
-- `factoid/search.js`
+> **CORS:** the SPARQL endpoint must allow your local origin
+> (e.g. `http://localhost:8000`); otherwise browser fetches fail with a CORS
+> error even when the query is valid.
 
-### Query Limits
-Adjust performance settings:
-- `LIMIT 500`: Maximum results per query
-- `15000ms`: Query timeout duration
-- Filter size limits for complex queries
+### Usage
+1. Open `browse.html`.
+2. Choose a search mode (Person, Event, or Relation).
+3. Apply filters in the sidebar.
+4. View results in the main panel.
+5. Share the search via the URL.
 
-## Data Sources
+## Configuration
 
-The application queries data from:
-- **SPEAR Prosopography Graph**: `https://spear-prosop.org`
-- **Syriaca Persons Graph**: `http://syriaca.org/persons#graph`
-- **Source collections**: Letters of Severus, Lives of Eastern Saints, Chronicle of Edessa
+### SPARQL endpoint
+`SPARQL_ENDPOINT` is defined in each search/menu module:
+- `menu.js`, `list.js`
+- `person/search.js`, `event/search.js`, `relation/search.js`
 
-## Browser Support
+### Caching (client-side)
+- 24h TTL in `localStorage`; shared helper `fetchWithCache` in `menu.js`.
+- Cache key prefixes: `dropdown_`, `keywordListV2_`, `keywordPrettyV1_`,
+  `eventFactoidsV1_`.
+- Call `clearMenuCache()` (in `menu.js`) to purge all cached menu/result data —
+  e.g. after a Neptune data refresh.
 
-- Chrome/Edge 88+
-- Firefox 85+
-- Safari 14+
+## Data sources
 
-Requires support for:
-- Fetch API with AbortController
-- ES6 modules
-- CSS Grid/Flexbox
+- **SPEAR prosopography graph:** `https://spear-prosop.org`
+- **Syriaca persons graph:** `http://syriaca.org/persons#graph`
+- **Place labels graph:** `http://syriaca.org/geo#graph`
+- **Source collections:** Letters of Severus, Lives of the Eastern Saints,
+  Chronicle of Edessa
 
-## Performance Notes
+## Browser support
 
-- Queries are limited to prevent timeouts
-- Complex multi-filter searches may take 5-15 seconds
-- Source filters are limited to prevent overly broad queries
-- Results are paginated at 500 items maximum
+Requires ES6 modules, the Fetch API, `localStorage`, and CSS Grid/Flexbox.
+Recent Chrome/Edge, Firefox, and Safari are supported.
 
 ## Development
 
-### File Structure
+### File structure (top level)
 ```
 spear/
-├── browse.html          # Main application entry point
+├── browse.html            # production entry
+├── dev.html               # development entry
+├── mode.js                # mode router / boot lifecycle
+├── filter.js              # shared filter state + URL sync
+├── menu.js                # facet menu queries + caching (fetchWithCache, clearMenuCache)
+├── list.js                # event-concept / ethnicity menu loader + caching
 ├── modes/
-│   ├── person.js        # Person search interface
-│   ├── event.js         # Event search interface
-│   └── factoid.js       # Factoid search interface
-├── person/
-│   └── search.js        # Person SPARQL queries
-├── event/
-│   └── search.js        # Event SPARQL queries
-└── factoid/
-    └── search.js        # Factoid SPARQL queries
+│   ├── person.js
+│   ├── event.js
+│   └── relation.js
+├── person/   { search.js, person.json, README.md }
+├── event/    { search.js, person.json, README.md }
+├── relation/ { search.js, *.json, batch_query_*.sh }
+├── utils/    { cleanUi.js, url.js }
+├── aggregate/             # static person/place/taxonomy HTML pages
+└── CTS/                   # CTS resolver (XQuery) — separate subsystem
 ```
 
-### Adding New Filters
-1. Add UI elements to the appropriate mode file
-2. Update the filter state object
-3. Modify the SPARQL query builder
-4. Add URL parameter handling
+
+### Adding a new filter
+1. Add the UI control to the relevant mode's `sidebar`/`bind` (`modes/*.js`).
+2. Add the facet to the filter state and to `FILTER_MAP` in `filter.js`.
+3. Extend the SPARQL query builder in the relevant `*/search.js`.
+4. Add URL parameter handling so the facet is shareable.
 
 ## License
 
-Open source - see original Srophé Application license terms.
+Open source — see the original Srophé Application license terms.
+See [`LICENSE`](./LICENSE) §3.