diff --git a/_quarto.yml b/_quarto.yml index 2ce91d41..1500b32e 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -63,6 +63,15 @@ website: - section: Boilerplate ingest example contents: - user-guide/content-curation/dataset-ingestion/example-template/example-geoglam-ingest.ipynb + - section: user-guide/content-curation/external-collection-indexing/index.qmd + contents: + - user-guide/content-curation/external-collection-indexing/titiler-cmr.qmd + - section: user-guide/content-curation/external-collection-indexing/arcgis-server-integration.qmd + contents: + - user-guide/content-curation/external-collection-indexing/imageserver-integration.qmd + - user-guide/content-curation/external-collection-indexing/mapserver-integration.qmd + - user-guide/content-curation/external-collection-indexing/featureserver-integration.qmd + - user-guide/content-curation/external-collection-indexing/examples/README.md - section: user-guide/content-curation/dashboard-configuration/index.qmd contents: - user-guide/content-curation/dashboard-configuration/dataset-configuration.qmd diff --git a/user-guide/content-curation/external-collection-indexing/arcgis-server-integration.qmd b/user-guide/content-curation/external-collection-indexing/arcgis-server-integration.qmd new file mode 100644 index 00000000..f7670921 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/arcgis-server-integration.qmd @@ -0,0 +1,244 @@ +--- +title: ArcGIS Server Integration +subtitle: Indexing ArcGIS Server services using pyarc2stac for STAC collection creation +--- + +The [pyarc2stac](https://github.com/NASA-IMPACT/pyarc2stac) library provides seamless integration between ArcGIS Server services and STAC collections, enabling VEDA to index and access geospatial data from ArcGIS Server endpoints without requiring data migration. + +## Overview + +ArcGIS Server hosts various types of geospatial services that can be converted to STAC collections for integration with VEDA. The pyarc2stac library automatically extracts metadata from ArcGIS services and generates STAC-compliant collection documents. + +## Supported ArcGIS Services + +| Service Type | Use Cases | +|--------------|-----------| +| **ImageServer** | Satellite imagery, raster analysis, multidimensional data | +| **MapServer** | Visualized map services, layer-based data | +| **FeatureServer** | Vector data, administrative boundaries, point datasets | + +## Key Features + +- **Automatic Metadata Extraction**: Pulls spatial extent, temporal coverage, and descriptive metadata +- **WMS Integration**: Generates WMS links for visualization in VEDA Dashboard +- **Datacube Support**: Handles multidimensional ImageServer data with proper datacube extensions +- **STAC Compliance**: Generates valid STAC collections following specification standards +- **Render Configuration**: Automatically configures visualization parameters + +## Architecture + +```{mermaid} +graph LR + A[ArcGIS Server] --> B[pyarc2stac Library] + B --> C[STAC Collection] + C --> D[VEDA Catalog] + style A fill:#e1f5fe + style B fill:#f3e5f5 + style C fill:#fff3e0 + subgraph "ArcGIS Services" + E[ImageServer] + F[MapServer] + G[FeatureServer] + end + + E --> B + F --> B + G --> B + + subgraph "VEDA Integration" + H[Dashboard Visualization] + I[API Access] + J[Data Discovery] + end + + D --> H + D --> I + D --> J +``` + +## Installation and Setup + +### Prerequisites + +```bash +# Install pyarc2stac +pip install git+https://github.com/NASA-IMPACT/pyarc2stac.git@main + +# Required dependencies +pip install pystac requests pyproj beautifulsoup4 +``` + +### Basic Usage + +```python +from pyarc2stac.ArcReader import ArcReader + +# Initialize reader with ArcGIS service URL +reader = ArcReader(server_url="https://example.com/arcgis/rest/services/ServiceName/ImageServer") + +# Generate STAC collection +collection = reader.generate_stac() + +# Save to JSON file +reader.save_collection_to_json("my_collection.json") +``` + +## Service-Specific Integration + +### ImageServer Integration +Ideal for raster datasets, satellite imagery, and multidimensional scientific data. + +[Learn more about ImageServer Integration →](./imageserver-integration.qmd) + +### MapServer Integration +Perfect for visualized map services with multiple layers and styling. + +[Learn more about MapServer Integration →](./mapserver-integration.qmd) + +### FeatureServer Integration +Designed for vector datasets, administrative boundaries, and point-based data. + +[Learn more about FeatureServer Integration →](./featureserver-integration.qmd) + +## Configuration Examples + +### Basic Collection Structure + +All pyarc2stac-generated collections follow this basic structure: + +```json +{ + "type": "Collection", + "id": "service_name", + "stac_version": "1.1.0", + "description": "Extracted from ArcGIS service metadata", + "extent": { + "spatial": {"bbox": [...]}, + "temporal": {"interval": [...]} + }, + "links": [...], + "license": "not-applicable", + "item_assets": {} +} +``` + +### Service-Specific Extensions + +Each service type adds specific extensions and metadata: + +#### ImageServer +```json +{ + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json", + "https://stac-extensions.github.io/datacube/v2.2.0/schema.json" + ], + "cube:variables": {...}, + "cube:dimensions": {...}, + "renders": {...} +} +``` + +#### MapServer +```json +{ + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json" + ], + "renders": {...}, + "links": [{ + "rel": "wms", + "href": "...", + "wms:layers": [...], + "wms:styles": ["default"] + }] +} +``` + +#### FeatureServer +```json +{ + "links": [{ + "rel": "featureserver", + "href": "...", + "featureserver:layers": {...} + }] +} +``` + +## Implementation Workflow + +### 1. Service Discovery +Identify ArcGIS services suitable for VEDA integration: + +```python +# Validate service accessibility +import requests + +service_url = "https://example.com/arcgis/rest/services/ServiceName/ImageServer" +response = requests.get(f"{service_url}?f=pjson") + +if response.status_code == 200: + service_info = response.json() + print(f"Service: {service_info.get('name', 'Unknown')}") + print(f"Type: {service_info.get('serviceDataType', 'Unknown')}") +else: + print("Service not accessible") +``` + +### 2. Metadata Extraction +Generate STAC collection with automatic metadata extraction: + +```python +from pyarc2stac.ArcReader import ArcReader + +reader = ArcReader(server_url=service_url) +collection = reader.generate_stac() + +# Inspect generated metadata +print(f"Collection ID: {collection.id}") +print(f"Spatial Extent: {collection.extent.spatial.bboxes}") +print(f"Temporal Extent: {collection.extent.temporal.intervals}") +``` + +### 3. Configuration Validation +Validate the generated STAC collection: + +```python +# Validate STAC compliance +try: + collection.validate() + print("✅ Collection is valid STAC") +except Exception as e: + print(f"❌ Validation error: {e}") +``` + +### 4. Integration Testing +Test integration with VEDA components: + +```python +# Test WMS links (for ImageServer/MapServer) +wms_links = [link for link in collection.links if link.rel == "wms"] +if wms_links: + print(f"WMS endpoint: {wms_links[0].href}") + +# Test render configuration +if hasattr(collection, 'extra_fields') and 'renders' in collection.extra_fields: + print(f"Render configs: {list(collection.extra_fields['renders'].keys())}") +``` + +## Related Resources + +- [pyarc2stac GitHub Repository](https://github.com/NASA-IMPACT/pyarc2stac) +- [ArcGIS REST API Documentation](https://developers.arcgis.com/rest/) +- [STAC Specification](https://stacspec.org/) +- [VEDA Dataset Configuration Guide](../dashboard-configuration/dataset-configuration.qmd) + +## Support + +For assistance with ArcGIS Server integration: + +1. Review the [pyarc2stac examples](https://github.com/NASA-IMPACT/pyarc2stac/tree/main/examples) +2. Check [existing issues](https://github.com/NASA-IMPACT/pyarc2stac/issues) for known problems +3. Open a new issue in the [pyarc2stac repository](https://github.com/NASA-IMPACT/pyarc2stac/issues/new) +4. Consult the [VEDA data repository](https://github.com/NASA-IMPACT/veda-data) for similar configurations \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/examples/README.md b/user-guide/content-curation/external-collection-indexing/examples/README.md new file mode 100644 index 00000000..61d65804 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/examples/README.md @@ -0,0 +1,125 @@ +# External Collection Configuration Examples + +This directory contains example configuration files for indexing external datasets in VEDA using both Titiler-CMR and pyarc2stac. + +## Titiler-CMR Examples + +### GPM Precipitation (gpm-precipitation-config.json) + +A complete STAC collection configuration for the GPM IMERG Final Daily precipitation dataset: + +- **Dataset**: GPM Level 3 IMERG Final Daily 10 x 10 km +- **CMR Concept ID**: C2723754864-GES_DISC +- **Format**: NetCDF +- **Backend**: xarray +- **Temporal Coverage**: 1998-present (daily) +- **Spatial Resolution**: 0.1° × 0.1° + +This example demonstrates: +- Multi-variable configuration with different visualization settings +- Proper backend selection for NetCDF data +- Dimension and coordinate system definitions +- Rendering configuration for precipitation data +- Metadata structure following STAC conventions + +## ArcGIS Server Examples + +### ESI MapServer (esi-mapserver-config.json) + +SERVIR Global Evaporative Stress Index 4-week MapServer service: + +- **Service**: Global ESI 4-Week MapServer +- **Format**: Styled map layers +- **Temporal Coverage**: 2001-present (weekly) +- **Spatial Resolution**: 5 km +- **Update Frequency**: Weekly + +This example demonstrates: +- Multi-layer MapServer configuration +- WMS integration for visualization +- Time-enabled service handling +- Layer-based rendering configuration + +### Soil Moisture ImageServer (soil-moisture-imageserver-config.json) + +NASA disasters test soil moisture percentile ImageServer: + +- **Service**: LIS VSM Percentile 10cm ImageServer +- **Format**: Multidimensional raster data +- **Temporal Coverage**: Daily time series +- **Spatial Coverage**: Continental United States +- **Variables**: Soil moisture percentiles + +This example demonstrates: +- Datacube extension for multidimensional data +- Variable and dimension definitions +- Temporal data handling +- ImageServer-specific configuration + +### Climate Projections FeatureServer (climate-projections-featureserver-config.json) + +Climate resilience and adaptation projections FeatureServer: + +- **Service**: CMRA Climate and Coastal Inundation Projections +- **Format**: Vector features (polygons) +- **Coverage**: Counties, tracts, and tribal areas +- **Data Type**: Climate projection and vulnerability data + +This example demonstrates: +- Multi-layer FeatureServer configuration +- Timeless data handling +- Feature layer metadata +- Administrative boundary data structure + +## Using These Examples + +1. **Copy Configuration**: Start with an example configuration closest to your dataset +2. **Update Metadata**: Modify collection ID, concept ID, and descriptive fields +3. **Adjust Dimensions**: Update spatial and temporal extents for your dataset +4. **Configure Variables**: Add/remove variables based on your dataset structure +5. **Set Rendering**: Customize visualization parameters (colormaps, rescaling) +6. **Test Configuration**: Validate using Titiler-CMR endpoints before deployment + +## Testing Configurations + +### Testing Titiler-CMR Configurations + +Before deploying to VEDA, test your configuration using the Titiler-CMR API: + +```bash +# Test tile generation +curl "https://staging.openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=YOUR_CONCEPT_ID&datetime=2024-01-15&backend=xarray&variable=your_variable" + +# Test info endpoint +curl "https://staging.openveda.cloud/api/titiler-cmr/info?concept_id=YOUR_CONCEPT_ID&datetime=2024-01-15&backend=xarray" +``` + +### Testing ArcGIS Server Configurations + +For ArcGIS Server integrations, test using pyarc2stac: + +```python +from pyarc2stac.ArcReader import ArcReader + +# Test service accessibility +service_url = "https://your-arcgis-server.com/rest/services/YourService/ImageServer" +reader = ArcReader(server_url=service_url) + +# Generate and validate collection +collection = reader.generate_stac() +collection.validate() + +print(f"✅ Generated collection: {collection.id}") +``` + +## Contributing Examples + +To contribute additional examples: + +1. Create a new JSON configuration file +2. Follow the naming convention: `{dataset-name}-config.json` +3. Include a brief description in this README +4. Test the configuration thoroughly +5. Submit a pull request + +For more detailed information, see the [Titiler-CMR Integration Guide](../titiler-cmr.qmd). \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/examples/climate-projections-featureserver-config.json b/user-guide/content-curation/external-collection-indexing/examples/climate-projections-featureserver-config.json new file mode 100644 index 00000000..181a6703 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/examples/climate-projections-featureserver-config.json @@ -0,0 +1,66 @@ +{ + "type": "Collection", + "id": "climate_mapping_resilience_and_adaptation_projections", + "stac_version": "1.1.0", + "description": "Climate Mapping for Resilience and Adaptation (CMRA) provides climate and coastal inundation projections for vulnerability assessment and adaptation planning. The dataset includes projections for counties, tracts, and tribal areas.", + "links": [ + { + "rel": "featureserver", + "href": "https://services3.arcgis.com/0Fs3HcaFfvzXvm7w/ArcGIS/rest/services/Climate_Mapping_Resilience_and_Adaptation_(CMRA)_Climate_and_Coastal_Inundation_Projections/FeatureServer", + "type": "application/json", + "title": "ArcGIS FeatureServer", + "featureserver:layers": { + "0": "Counties", + "1": "Tracts", + "2": "American Indian/Alaska Native/Native Hawaiian Areas" + } + }, + { + "rel": "via", + "href": "https://services3.arcgis.com/0Fs3HcaFfvzXvm7w/ArcGIS/rest/services/Climate_Mapping_Resilience_and_Adaptation_(CMRA)_Climate_and_Coastal_Inundation_Projections/FeatureServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "dashboard:is_timeless": true, + "title": "Climate Mapping Resilience and Adaptation Projections", + "extent": { + "spatial": { + "bbox": [ + [ + -179.23110125118885, + -14.601806016382575, + 179.8596657289939, + 71.43978570694523 + ] + ] + }, + "temporal": { + "interval": [ + [ + null, + null + ] + ] + } + }, + "license": "not-applicable", + "item_assets": {}, + "keywords": [ + "climate", + "adaptation", + "resilience", + "projections", + "coastal inundation", + "vulnerability assessment" + ], + "providers": [ + { + "name": "U.S. Climate Resilience Toolkit", + "roles": [ + "producer" + ], + "url": "https://toolkit.climate.gov" + } + ] +} \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/examples/esi-mapserver-config.json b/user-guide/content-curation/external-collection-indexing/examples/esi-mapserver-config.json new file mode 100644 index 00000000..806c0eb0 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/examples/esi-mapserver-config.json @@ -0,0 +1,68 @@ +{ + "type": "Collection", + "id": "global_esi_4wk", + "stac_version": "1.1.0", + "description": "The Evaporative Stress Index (ESI) reveals regions of drought where vegetation is stressed due to lack of water. The ESI can capture early signals of flash drought and is produced weekly at 5-kilometer resolution for the entire globe.", + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json" + ], + "links": [ + { + "rel": "wms", + "href": "https://gis1.servirglobal.net/arcgis/services/Global/ESI_4WK/MapServer/WMSServer?request=GetCapabilities&service=WMS", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": [ + "1", + "2", + "3" + ], + "wms:styles": [ + "default" + ] + }, + { + "rel": "via", + "href": "https://gis1.servirglobal.net/arcgis/rest/services/Global/ESI_4WK/MapServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "renders": { + "image": { + "layers": "1" + }, + "footprint": { + "layers": "2" + }, + "boundary": { + "layers": "3" + } + }, + "dashboard:is_periodic": true, + "dashboard:time_density": "day", + "dashboard:time_interval": "P7D", + "title": "Global ESI 4-Week", + "extent": { + "spatial": { + "bbox": [ + [ + -180, + -60, + 180.0000000000001, + 90.00000000000009 + ] + ] + }, + "temporal": { + "interval": [ + [ + "2001-01-02T00:00:00Z", + "2025-02-19T00:00:00Z" + ] + ] + } + }, + "license": "not-applicable", + "item_assets": {} +} \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/examples/gpm-precipitation-config.json b/user-guide/content-curation/external-collection-indexing/examples/gpm-precipitation-config.json new file mode 100644 index 00000000..24b1eeec --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/examples/gpm-precipitation-config.json @@ -0,0 +1,214 @@ +{ + "type": "Collection", + "id": "GPM_3IMERGDF", + "stac_version": "1.0.0", + "description": "Version 07 is the current version of the data set. The Integrated Multi-satellitE Retrievals for GPM (IMERG) is a NASA product estimating global surface precipitation rates at a high resolution of 0.1° every half-hour beginning 2000. This dataset is the GPM Level 3 IMERG Final Daily 10 x 10 km (GPM_3IMERGDF) derived from the half-hourly GPM_3IMERGHH.", + "links": [], + "stac_extensions": [ + "https://stac-extensions.github.io/datacube/v2.2.0/schema.json" + ], + "collection_concept_id": "C2723754864-GES_DISC", + "cube:dimensions": { + "time": { + "extent": [ + "1998-01-01T00:00:00Z", + null + ], + "description": "time", + "step": "P1D", + "type": "temporal" + }, + "lon": { + "axis": "x", + "extent": [ + -179.9499969482422, + 179.9499969482422 + ], + "description": "Longitude", + "reference_system": 4326, + "type": "spatial" + }, + "lat": { + "axis": "y", + "extent": [ + -89.9499969482422, + 89.9499969482422 + ], + "description": "Latitude", + "reference_system": 4326, + "type": "spatial" + } + }, + "summaries": { + "precipitation": { + "type": "data", + "description": "Daily mean precipitation rate (combined microwave-IR) estimate", + "dimensions": [ + "time", + "lon", + "lat" + ], + "unit": "mm/day", + "attrs": { + "units": "mm/day", + "long_name": "Daily mean precipitation rate (combined microwave-IR) estimate" + }, + "shape": [ + null, + 3600, + 1800 + ], + "chunks": [ + 1, + 3600, + 900 + ], + "renders": "precipitation" + }, + "precipitation_cnt": { + "type": "data", + "description": "Count of all valid half-hourly precipitation retrievals for the day", + "dimensions": [ + "time", + "lon", + "lat" + ], + "unit": "count", + "attrs": { + "units": "count", + "long_name": "Count of all valid half-hourly precipitation retrievals for the day" + }, + "shape": [ + null, + 3600, + 1800 + ], + "chunks": [ + 1, + 3600, + 900 + ] + }, + "MWprecipitation": { + "type": "data", + "description": "Daily mean High Quality precipitation rate from all available microwave sources", + "dimensions": [ + "time", + "lon", + "lat" + ], + "unit": "mm/day", + "attrs": { + "units": "mm/day", + "long_name": "Daily mean High Quality precipitation rate from all available microwave sources" + }, + "shape": [ + null, + 3600, + 1800 + ], + "chunks": [ + 1, + 3600, + 900 + ], + "renders": "MWprecipitation" + }, + "randomError": { + "type": "data", + "description": "Root-mean-square error estimate for combined microwave-IR daily precipitation rate", + "dimensions": [ + "time", + "lon", + "lat" + ], + "unit": "mm/day", + "attrs": { + "units": "mm/day", + "long_name": "Root-mean-square error estimate for combined microwave-IR daily precipitation rate" + }, + "shape": [ + null, + 3600, + 1800 + ], + "chunks": [ + 1, + 3600, + 900 + ], + "renders": "randomError" + }, + "probabilityLiquidPrecipitation": { + "type": "data", + "description": "Probability of liquid precipitation estimated with a diagnostic parameterization using ancillary data", + "dimensions": [ + "time", + "lon", + "lat" + ], + "unit": "percent", + "attrs": { + "units": "percent", + "long_name": "Probability of liquid precipitation", + "description": "Probability of liquid precipitation estimated with a diagnostic parameterization using ancillary data. 0=missing values; 1=likely solid; 100=likely liquid or no precipitation." + }, + "shape": [ + null, + 3600, + 1800 + ], + "chunks": [ + 1, + 3600, + 900 + ] + } + }, + "renders": { + "precipitation": { + "title": "Precipitation Rate", + "resampling": "average", + "colormap_name": "blues", + "rescale": [ + [ + 0, + 48 + ] + ], + "backend": "xarray" + }, + "MWprecipitation": { + "title": "Microwave Precipitation Rate", + "resampling": "average", + "colormap_name": "blues", + "rescale": [ + [ + 0, + 48 + ] + ], + "backend": "xarray" + }, + "randomError": { + "title": "Precipitation Random Error", + "resampling": "average", + "colormap_name": "viridis", + "rescale": [ + [ + 0, + 10 + ] + ], + "backend": "xarray" + } + }, + "providers": [ + { + "name": "NASA Goddard Earth Sciences Data and Information Services Center (GES DISC)", + "roles": [], + "url": "https://disc.gsfc.nasa.gov/" + } + ], + "item_assets": {} +} \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/examples/soil-moisture-imageserver-config.json b/user-guide/content-curation/external-collection-indexing/examples/soil-moisture-imageserver-config.json new file mode 100644 index 00000000..98439635 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/examples/soil-moisture-imageserver-config.json @@ -0,0 +1,111 @@ +{ + "type": "Collection", + "id": "disasters_test_lis_vsm_percentile_10cm", + "stac_version": "1.1.0", + "description": "Soil moisture percentile data from Land Information System (LIS) for disaster monitoring applications. This dataset provides volumetric soil moisture percentiles for the top 10cm of soil.", + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json", + "https://stac-extensions.github.io/datacube/v2.2.0/schema.json" + ], + "links": [ + { + "rel": "wms", + "href": "https://gis.earthdata.nasa.gov/UAT/services/disasters_test/lis_vsm_percentile_10cm/ImageServer/WMSServer", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": [ + "lis_vsm_percentile_10cm", + "VSM_Percentile_0_10cm" + ], + "wms:styles": [ + "default" + ] + }, + { + "rel": "via", + "href": "https://gis.earthdata.nasa.gov/UAT/rest/services/disasters_test/lis_vsm_percentile_10cm/ImageServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "renders": { + "lis_vsm_percentile_10cm": { + "layers": "lis_vsm_percentile_10cm" + }, + "vsm_percentile_0_10cm": { + "layers": "VSM_Percentile_0_10cm" + } + }, + "cube:variables": { + "lis_vsm_percentile_10cm": { + "type": "data", + "attrs": { + "Count": 27, + "Minimum": 45857, + "Maximum": 45886, + "Interval": 1, + "IntervalUnit": "Days", + "HasRegularIntervals": false, + "HasRanges": false + }, + "dimensions": [ + "StdTime" + ] + }, + "VSM_Percentile_0_10cm": { + "type": "data", + "attrs": { + "Count": 31, + "Minimum": 45857, + "Maximum": 45887, + "Interval": 1, + "IntervalUnit": "Days", + "HasRegularIntervals": true + }, + "dimensions": [ + "StdTime" + ] + } + }, + "cube:dimensions": { + "StdTime": { + "type": "temporal", + "step": "P1D", + "extent": [ + "2025-07-19T00:00:00Z", + "2025-08-17T00:00:00Z" + ], + "values": [ + "2025-07-19T00:00:00Z", + "2025-07-20T00:00:00Z", + "2025-07-21T00:00:00Z" + ] + } + }, + "dashboard:is_periodic": true, + "dashboard:time_density": "day", + "dashboard:time_interval": "P1D", + "title": "Disasters Test LIS VSM Percentile 10cm", + "extent": { + "spatial": { + "bbox": [ + [ + -124.93999999989579, + 25.054242740487126, + -67.0699999998994, + 52.930000000213646 + ] + ] + }, + "temporal": { + "interval": [ + [ + "2025-07-19T00:00:00Z", + "2025-08-18T00:00:00Z" + ] + ] + } + }, + "license": "not-applicable", + "item_assets": {} +} \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/featureserver-integration.qmd b/user-guide/content-curation/external-collection-indexing/featureserver-integration.qmd new file mode 100644 index 00000000..1e9136ca --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/featureserver-integration.qmd @@ -0,0 +1,201 @@ +--- +title: FeatureServer Integration +subtitle: Converting ArcGIS FeatureServer services to STAC collections +--- + +ArcGIS FeatureServer services provide access to vector datasets including points, lines, and polygons. This page covers FeatureServer-specific configuration and examples. + +::: {.callout-note} +**Prerequisites**: Before using FeatureServer integration, complete the [ArcGIS Server Integration setup](./arcgis-server-integration.qmd#installation-and-setup) including pyarc2stac installation. +::: + +## FeatureServer Resources + +- [pyarc2stac FeatureServer Example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/FeatureServer_to_STAC.ipynb) +- [GeoJSON Specification](https://geojson.org/) +- [ArcGIS FeatureServer Documentation](https://developers.arcgis.com/rest/services-reference/enterprise/feature-service.htm) + +::: {.callout-tip} +For general ArcGIS Server integration support, troubleshooting, and additional resources, see the [ArcGIS Server Integration](./arcgis-server-integration.qmd#support) page. +::: + +## FeatureServer Specialization + +FeatureServer integration is specifically designed for: + +- **Vector-Based Datasets**: Point, line, and polygon features with rich attribute data +- **Timeless Collections**: Often static or infrequently updated boundary and reference datasets +- **Attribute-Rich Data**: Features with extensive metadata and descriptive attributes requiring preservation + +## Example: Climate Projections FeatureServer + +This example demonstrates FeatureServer integration using a climate and coastal inundation projections service: + +```python +# Climate projections FeatureServer with multiple feature layers +feature_server_url = "https://services3.arcgis.com/0Fs3HcaFfvzXvm7w/ArcGIS/rest/services/Climate_Mapping_Resilience_and_Adaptation_(CMRA)_Climate_and_Coastal_Inundation_Projections/FeatureServer" + +# FeatureServer-specific inspection +collection = reader.generate_stac() + +# Check feature layers and their schemas (FeatureServer specialty) +feature_links = [link for link in collection.links if link.rel == "featureserver"] +if feature_links: + feature_link = feature_links[0] + layers = feature_link.extra_fields.get('featureserver:layers', {}) + print(f"Available feature layers: {list(layers.keys())}") + + # Inspect layer schemas + for layer_name, layer_info in layers.items(): + if 'fields' in layer_info: + field_names = [f['name'] for f in layer_info['fields']] + print(f" {layer_name} fields: {field_names[:5]}...") # Show first 5 fields + +# Check timeless configuration (common for FeatureServer) +is_timeless = collection.extra_fields.get('dashboard:is_timeless', False) +print(f"Timeless dataset: {is_timeless}") + +# Check geometry types +for layer_name, layer_info in layers.items(): + geom_type = layer_info.get('geometryType', 'Unknown') + print(f" {layer_name}: {geom_type}") +``` + +## Configuration Structure + +### Complete Example Configuration + +Based on the Climate Mapping Resilience and Adaptation projections FeatureServer: + +```json +{ + "type": "Collection", + "id": "climate_mapping_resilience_and_adaptation_(cmra)_climate_and_coastal_inundation_projections", + "stac_version": "1.1.0", + "description": "Climate mapping resilience and adaptation climate and coastal inundation projections for vulnerability assessment and planning.", + "links": [ + { + "rel": "featureserver", + "href": "https://services3.arcgis.com/0Fs3HcaFfvzXvm7w/ArcGIS/rest/services/Climate_Mapping_Resilience_and_Adaptation_(CMRA)_Climate_and_Coastal_Inundation_Projections/FeatureServer", + "type": "application/json", + "title": "ArcGIS FeatureServer", + "featureserver:layers": { + "0": "Counties", + "1": "Tracts", + "2": "American Indian/Alaska Native/Native Hawaiian Areas" + } + }, + { + "rel": "via", + "href": "https://services3.arcgis.com/0Fs3HcaFfvzXvm7w/ArcGIS/rest/services/Climate_Mapping_Resilience_and_Adaptation_(CMRA)_Climate_and_Coastal_Inundation_Projections/FeatureServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "dashboard:is_timeless": true, + "extent": { + "spatial": { + "bbox": [[-179.23110125118885, -14.601806016382575, 179.8596657289939, 71.43978570694523]] + }, + "temporal": { + "interval": [[null, null]] + } + }, + "license": "not-applicable", + "item_assets": {} +} +``` + +### Key Configuration Elements + +#### FeatureServer Link +Provides direct access to the original FeatureServer: + +```json +{ + "links": [{ + "rel": "featureserver", + "href": "https://server.com/FeatureServer", + "type": "application/json", + "title": "ArcGIS FeatureServer", + "featureserver:layers": { + "0": "Counties", + "1": "Tracts", + "2": "Tribal Areas" + } + }] +} +``` + +#### Timeless Data Indicator +Most FeatureServer data is timeless (static features): + +```json +{ + "dashboard:is_timeless": true, + "extent": { + "temporal": { + "interval": [[null, null]] + } + } +} +``` + +#### Layer Information +Maps layer IDs to descriptive names: + +```json +{ + "featureserver:layers": { + "0": "Primary Features", + "1": "Reference Data", + "2": "Administrative Boundaries" + } +} +``` + + + +## FeatureServer-Specific Best Practices + +### Attribute and Schema Validation + +1. **Field Schema Verification**: Ensure attribute schemas are complete + ```python + # Check for required fields in feature layers + for layer in service_info.get('layers', []): + layer_info = requests.get(f"{feature_server_url}/{layer['id']}?f=pjson").json() + fields = layer_info.get('fields', []) + + # Check for essential fields + field_names = [f['name'] for f in fields] + if 'OBJECTID' not in field_names: + print(f"⚠️ Layer {layer['name']} missing OBJECTID field") + ``` + +2. **Geometry Type Validation**: Verify feature geometry consistency + ```python + # Ensure geometry types are properly defined + for layer in layers: + geom_type = layer.get('geometryType') + if not geom_type: + print(f"⚠️ Layer {layer['name']} missing geometry type") + elif geom_type not in ['esriGeometryPoint', 'esriGeometryPolyline', 'esriGeometryPolygon']: + print(f"⚠️ Layer {layer['name']} has unexpected geometry type: {geom_type}") + ``` + ``` + +## Related Resources + +- [pyarc2stac FeatureServer Example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/FeatureServer_to_STAC.ipynb) +- [ArcGIS FeatureServer Documentation](https://developers.arcgis.com/rest/services-reference/enterprise/feature-service.htm) +- [STAC Specification](https://stacspec.org/) +- [GeoJSON Specification](https://geojson.org/) + +## Support + +For FeatureServer integration assistance: + +1. Review the [complete working example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/FeatureServer_to_STAC.ipynb) +2. Check the [pyarc2stac test cases](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/tests/test_featureserver.py) +3. Open an issue in the [pyarc2stac repository](https://github.com/NASA-IMPACT/pyarc2stac/issues) \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/imageserver-integration.qmd b/user-guide/content-curation/external-collection-indexing/imageserver-integration.qmd new file mode 100644 index 00000000..799a5fcd --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/imageserver-integration.qmd @@ -0,0 +1,264 @@ +--- +title: ImageServer Integration +subtitle: Converting ArcGIS ImageServer services to STAC collections +--- + +ArcGIS ImageServer services provide access to raster datasets, satellite imagery, and multidimensional scientific data. This page covers ImageServer-specific configuration and examples. + +::: {.callout-note} +**Prerequisites**: Before using ImageServer integration, complete the [ArcGIS Server Integration setup](./arcgis-server-integration.qmd#installation-and-setup) including pyarc2stac installation and basic usage patterns. +::: + +## ImageServer Specialization + +ImageServer integration is specifically designed for: + +- **Multidimensional Data**: Time series datasets with multiple variables and datacube extensions +- **Scientific Rasters**: Climate data with complex temporal dimensions +- **Variable-Rich Datasets**: Services with multiple data variables requiring individual render configurations +- **Temporal Analysis**: Datasets requiring sophisticated time series handling + +## Example: Soil Moisture ImageServer + +This example demonstrates ImageServer integration using a NASA disasters test service with multidimensional soil moisture data: + +```python +# Soil moisture ImageServer with multidimensional data +image_server_url = "https://gis.earthdata.nasa.gov/UAT/rest/services/disasters_test/lis_vsm_percentile_10cm/ImageServer" + +# ImageServer-specific inspection +collection = reader.generate_stac() + +# Check datacube variables (ImageServer specialty) +if 'cube:variables' in collection.extra_fields: + variables = collection.extra_fields['cube:variables'] + print(f"Variables: {list(variables.keys())}") + + for var_name, var_info in variables.items(): + print(f" {var_name}: {var_info.get('dimensions', [])}") + +# Check temporal dimensions (key for ImageServer) +if 'cube:dimensions' in collection.extra_fields: + dimensions = collection.extra_fields['cube:dimensions'] + time_dim = dimensions.get('StdTime', {}) + print(f"Time step: {time_dim.get('step', 'Unknown')}") + print(f"Time extent: {time_dim.get('extent', 'Unknown')}") +``` + +## Configuration Structure + +### Complete Example Configuration + +Based on the disasters test soil moisture ImageServer: + +```json +{ + "type": "Collection", + "id": "disasters_test_lis_vsm_percentile_10cm", + "stac_version": "1.1.0", + "description": "Soil moisture percentile data for disaster monitoring", + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json", + "https://stac-extensions.github.io/datacube/v2.2.0/schema.json" + ], + "links": [ + { + "rel": "wms", + "href": "https://gis.earthdata.nasa.gov/UAT/services/disasters_test/lis_vsm_percentile_10cm/ImageServer/WMSServer", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": [ + "lis_vsm_percentile_10cm", + "VSM_Percentile_0_10cm" + ], + "wms:styles": ["default"] + }, + { + "rel": "via", + "href": "https://gis.earthdata.nasa.gov/UAT/rest/services/disasters_test/lis_vsm_percentile_10cm/ImageServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "renders": { + "lis_vsm_percentile_10cm": { + "layers": "lis_vsm_percentile_10cm" + }, + "vsm_percentile_0_10cm": { + "layers": "VSM_Percentile_0_10cm" + } + }, + "cube:variables": { + "lis_vsm_percentile_10cm": { + "type": "data", + "attrs": { + "Count": 27, + "Minimum": 45857, + "Maximum": 45886, + "Interval": 1, + "IntervalUnit": "Days", + "HasRegularIntervals": False, + "HasRanges": False + }, + "dimensions": ["StdTime"] + }, + "VSM_Percentile_0_10cm": { + "type": "data", + "attrs": { + "Count": 31, + "Minimum": 45857, + "Maximum": 45887, + "Interval": 1, + "IntervalUnit": "Days", + "HasRegularIntervals": True + }, + "dimensions": ["StdTime"] + } + }, + "cube:dimensions": { + "StdTime": { + "type": "temporal", + "step": "P1D", + "extent": ["2025-07-19T00:00:00Z", "2025-08-17T00:00:00Z"], + "values": [ + "2025-07-19T00:00:00Z", + "2025-07-20T00:00:00Z", + "2025-07-21T00:00:00Z" + ] + } + }, + "dashboard:is_periodic": true, + "dashboard:time_density": "day", + "dashboard:time_interval": "P1D", + "extent": { + "spatial": { + "bbox": [[-124.94, 25.05, -67.07, 52.93]] + }, + "temporal": { + "interval": [["2025-07-19T00:00:00Z", "2025-08-18T00:00:00Z"]] + } + }, + "license": "not-applicable", + "item_assets": {} +} +``` + +### Key Configuration Elements + +#### Datacube Extension +The datacube extension describes multidimensional data structure: + +```json +{ + "cube:variables": { + "variable_name": { + "type": "data", + "attrs": { + "Count": 31, + "Minimum": 45857, + "Maximum": 45887, + "Interval": 1, + "IntervalUnit": "Days", + "HasRegularIntervals": true + }, + "dimensions": ["StdTime"] + } + }, + "cube:dimensions": { + "StdTime": { + "type": "temporal", + "step": "P1D", + "extent": ["2025-07-19T00:00:00Z", "2025-08-17T00:00:00Z"], + "values": ["2025-07-19T00:00:00Z", "2025-07-20T00:00:00Z"] + } + } +} +``` + +#### Render Configuration +Defines how variables should be visualized: + +```json +{ + "renders": { + "variable_name": { + "layers": "ArcGIS_Layer_Name" + } + } +} +``` + +#### WMS Integration +Provides Web Map Service endpoints for visualization: + +```json +{ + "links": [{ + "rel": "wms", + "href": "https://server.com/WMSServer", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": ["layer1", "layer2"], + "wms:styles": ["default"] + }] +} +``` + + + + + +## ImageServer-Specific Best Practices + +### Multidimensional Data Handling + +1. **Variable Validation**: Ensure all variables have proper dimensions + ```python + if 'cube:variables' in collection.extra_fields: + variables = collection.extra_fields['cube:variables'] + for var_name, var_info in variables.items(): + if 'dimensions' not in var_info: + print(f"⚠️ Variable {var_name} missing dimensions") + ``` + +2. **Temporal Dimension Verification**: Check time series configuration + ```python + if 'cube:dimensions' in collection.extra_fields: + dimensions = collection.extra_fields['cube:dimensions'] + if 'StdTime' in dimensions: + time_info = dimensions['StdTime'] + if 'step' not in time_info: + print("⚠️ Missing temporal step information") + ``` + +## ImageServer-Specific Troubleshooting + +### Multidimensional Data Issues + +#### Missing Variable Information +``` +Warning: No variables found in multidimensional service +``` +**Solution**: Check multidimensional configuration: +```python +md_info = requests.get(f"{image_server_url}/multidimensionalInfo?f=pjson").json() +variables = md_info.get('multidimensionalInfo', {}).get('variables', []) +if not variables: + print("⚠️ Service may not be properly configured for multidimensional access") +``` + +#### Datacube Extension Issues +``` +Error: Unable to create datacube dimensions +``` +**Solution**: Verify temporal dimension configuration and ensure service has time-enabled data. + +## ImageServer Resources + +- [pyarc2stac ImageServer Example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/ImageServer_to_STAC.ipynb) +- [STAC Datacube Extension](https://github.com/stac-extensions/datacube) +- [ArcGIS ImageServer Documentation](https://developers.arcgis.com/rest/services-reference/enterprise/image-service.htm) + +::: {.callout-tip} +For general ArcGIS Server integration support, troubleshooting, and additional resources, see the [ArcGIS Server Integration](./arcgis-server-integration.qmd#support) page. +::: \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/index.qmd b/user-guide/content-curation/external-collection-indexing/index.qmd new file mode 100644 index 00000000..e3f4b5f7 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/index.qmd @@ -0,0 +1,101 @@ +--- +title: External Collection Indexing +subtitle: Indexing external datasets in the VEDA catalog without data migration +--- + +External collection indexing allows VEDA to provide access to datasets hosted on external systems without requiring data migration to VEDA's data store. This approach is particularly useful for: + +- Large datasets that are already optimized for cloud access +- Datasets with strict data governance requirements +- Collections maintained by external organizations +- Time-sensitive data where migration would introduce delays + +## Overview + +VEDA supports indexing external collections through integration with NASA's Common Metadata Repository (CMR) via [Titiler-CMR](https://developmentseed.org/titiler-cmr/), which provides dynamic tiling and visualization capabilities for CMR-registered datasets. + +## Key Benefits + +- **No Data Duplication**: Access datasets in their original location +- **Real-time Access**: No synchronization delays +- **Reduced Storage Costs**: Leverages existing infrastructure +- **Maintained Provenance**: Data remains with authoritative source +- **API Consistency**: Same VEDA interface for both internal and external data + +## Supported Formats + +External collection indexing supports datasets in: + +- **NetCDF**: Multi-dimensional scientific data (via xarray backend) +- **Cloud-Optimized GeoTIFF (COG)**: Raster datasets optimized for cloud access +- **Zarr**: Chunked, compressed N-dimensional arrays + +## Integration Methods + +### Titiler-CMR Integration + +The primary method for external collection indexing uses Titiler-CMR to provide: + +- Dynamic tile generation from CMR-registered datasets +- Multi-backend support (xarray, rasterio) +- Statistical analysis capabilities +- Time series API support + +[Learn more about Titiler-CMR Integration →](./titiler-cmr.qmd) + +### ArcGIS Server Integration + +The pyarc2stac library enables integration with ArcGIS Server services: + +- ImageServer, MapServer, and FeatureServer support +- Automatic STAC collection generation +- WMS integration for visualization +- Datacube extension support for multidimensional data + +[Learn more about ArcGIS Server Integration →](./arcgis-server-integration.qmd) + +## Requirements + +To index an external collection in VEDA: + +1. **CMR Registration**: Dataset must be registered in NASA's CMR +2. **Cloud-Optimized Format**: Data should be in COG, NetCDF, or Zarr format +3. **Public Access**: Dataset must be publicly accessible or use standard authentication +4. **Metadata Compliance**: Collection metadata should follow STAC conventions where possible + +## Getting Started + +1. **Identify Dataset**: Locate the CMR concept ID for your dataset +2. **Configure Collection**: Create a STAC collection configuration +3. **Test Access**: Verify data accessibility through Titiler-CMR +4. **Submit Configuration**: Add collection to VEDA's staging environment +5. **Review and Deploy**: Test in staging before production deployment + +### Quick Example + +For the GPM precipitation dataset: + +```bash +# 1. Find the dataset in CMR +curl "https://cmr.earthdata.nasa.gov/search/collections.json?short_name=GPM_3IMERGDF" + +# 2. Test Titiler-CMR access +curl "https://staging.openveda.cloud/api/titiler-cmr/info?concept_id=C2723754864-GES_DISC&datetime=2024-01-15&backend=xarray" + +# 3. Generate tiles for visualization +curl "https://staging.openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2723754864-GES_DISC&datetime=2024-01-15&backend=xarray&variable=precipitation&rescale=0,50&colormap_name=blues" +``` + +## Example Use Cases + +### Titiler-CMR Use Cases +- **NASA Earth Science Data**: GPM precipitation, MODIS imagery, MUR SST +- **Interagency Datasets**: NOAA, USGS, and other federal agency data +- **International Collaborations**: ESA, JAXA, and partner organization datasets +- **Real-time Monitoring**: Weather, climate, and environmental monitoring data + +### ArcGIS Server Use Cases +- **State and Local Data**: Regional climate data, land use datasets +- **Commercial Services**: Third-party geospatial service providers +- **Institutional Repositories**: University and research organization data +- **Operational Services**: Emergency management, disaster response data diff --git a/user-guide/content-curation/external-collection-indexing/mapserver-integration.qmd b/user-guide/content-curation/external-collection-indexing/mapserver-integration.qmd new file mode 100644 index 00000000..0a99ad20 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/mapserver-integration.qmd @@ -0,0 +1,178 @@ +--- +title: MapServer Integration +subtitle: Converting ArcGIS MapServer services to STAC collections +--- + +ArcGIS MapServer services provide pre-styled, visualized map layers optimized for display and visualization. This page covers MapServer-specific configuration and examples. + +::: {.callout-note} +**Prerequisites**: Before using MapServer integration, complete the [ArcGIS Server Integration](./arcgis-server-integration.qmd#installation-and-setup) including pyarc2stac installation and basic usage patterns. +::: + +## MapServer Resources + +- [pyarc2stac MapServer Example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/MapServer_to_STAC.ipynb) +- [WMS Specification](https://www.ogc.org/standard/wms/) +- [ArcGIS MapServer Documentation](https://developers.arcgis.com/rest/services-reference/enterprise/map-service.htm) + +::: {.callout-tip} +For general ArcGIS Server integration support, troubleshooting, and additional resources, see the [ArcGIS Server Integration](./arcgis-server-integration.qmd#support) page. +::: + +## MapServer Specialization + +MapServer integration is specifically designed for: + +- **Pre-styled Visualizations**: Services with built-in symbology and rendering +- **Multi-layer Services**: Complex services with multiple thematic layers requiring individual styling +- **WMS-optimized Services**: Services primarily designed for map visualization rather than data analysis + +## Example: SERVIR ESI Drought Monitoring + +This example demonstrates MapServer integration using SERVIR Global's Evapotranspiration Stress Index (ESI) service: + +```python +# ESI MapServer with multiple styled layers +map_server_url = "https://gis1.servirglobal.net/arcgis/rest/services/Global/ESI_4WK/MapServer" + +# MapServer-specific inspection +collection = reader.generate_stac() + +# Check WMS layers (MapServer specialty) +wms_links = [link for link in collection.links if link.rel == "wms"] +if wms_links: + wms_link = wms_links[0] + layers = wms_link.extra_fields.get('wms:layers', []) + print(f"Available WMS layers: {layers}") + + # Check styling options + styles = wms_link.extra_fields.get('wms:styles', []) + print(f"Available styles: {styles}") + +# Check render configurations (key for MapServer visualization) +if 'renders' in collection.extra_fields: + renders = collection.extra_fields['renders'] + print(f"Render configurations: {list(renders.keys())}") + + # Inspect render details + for render_name, render_config in renders.items(): + print(f" {render_name}: layers={render_config.get('layers', 'Unknown')}") +``` + +## Configuration Structure + +### Complete Example Configuration + +Based on the SERVIR Global ESI 4-week service: + +```json +{ + "type": "Collection", + "id": "global_esi_4wk", + "stac_version": "1.1.0", + "description": "The Evaporative Stress Index (ESI) reveals regions of drought where vegetation is stressed due to lack of water. The ESI can capture early signals of flash drought and is produced weekly at 5-kilometer resolution for the entire globe.", + "stac_extensions": [ + "https://stac-extensions.github.io/render/v2.0.0/schema.json" + ], + "links": [ + { + "rel": "wms", + "href": "https://gis1.servirglobal.net/arcgis/services/Global/ESI_4WK/MapServer/WMSServer?request=GetCapabilities&service=WMS", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": ["1", "2", "3"], + "wms:styles": ["default"] + }, + { + "rel": "via", + "href": "https://gis1.servirglobal.net/arcgis/rest/services/Global/ESI_4WK/MapServer", + "type": "text/html", + "title": "Parent ArcGIS server url" + } + ], + "renders": { + "image": { + "layers": "1" + }, + "footprint": { + "layers": "2" + }, + "boundary": { + "layers": "3" + } + }, + "dashboard:is_periodic": true, + "dashboard:time_density": "day", + "dashboard:time_interval": "P7D", + "extent": { + "spatial": { + "bbox": [[-180, -60, 180.0000000000001, 90.00000000000009]] + }, + "temporal": { + "interval": [["2001-01-02T00:00:00Z", "2025-02-19T00:00:00Z"]] + } + }, + "license": "not-applicable", + "item_assets": {} +} +``` + +### Key Configuration Elements + +#### Layer-Based Rendering +MapServer collections use layer IDs for render configuration: + +```json +{ + "renders": { + "primary_data": { + "layers": "1" + }, + "reference_layer": { + "layers": "2" + }, + "boundaries": { + "layers": "3" + } + } +} +``` + +#### WMS Configuration +Provides Web Map Service access for all layers: + +```json +{ + "links": [{ + "rel": "wms", + "href": "https://server.com/MapServer/WMSServer?request=GetCapabilities&service=WMS", + "type": "image/png", + "title": "Visualized through a WMS", + "wms:layers": ["1", "2", "3"], + "wms:styles": ["default"] + }] +} +``` + +#### Time-Enabled Services +For time-enabled MapServer services: + +```json +{ + "dashboard:is_periodic": true, + "dashboard:time_density": "day", + "dashboard:time_interval": "P7D" +} +``` + + + +## MapServer Resources + +- [pyarc2stac MapServer Example](https://github.com/NASA-IMPACT/pyarc2stac/blob/main/examples/MapServer_to_STAC.ipynb) +- [WMS Specification](https://www.ogc.org/standard/wms/) +- [ArcGIS MapServer Documentation](https://developers.arcgis.com/rest/services-reference/enterprise/map-service.htm) + +::: {.callout-tip} +For general ArcGIS Server integration support, troubleshooting, and additional resources, see the [ArcGIS Server Integration](./arcgis-server-integration.qmd#support) page. +::: \ No newline at end of file diff --git a/user-guide/content-curation/external-collection-indexing/titiler-cmr.qmd b/user-guide/content-curation/external-collection-indexing/titiler-cmr.qmd new file mode 100644 index 00000000..6fc814e0 --- /dev/null +++ b/user-guide/content-curation/external-collection-indexing/titiler-cmr.qmd @@ -0,0 +1,175 @@ +--- +title: Titiler-CMR Integration +subtitle: Indexing CMR datasets using Titiler-CMR for dynamic visualization +--- + +[Titiler-CMR](https://developmentseed.org/titiler-cmr/) provides a powerful bridge between NASA's Common Metadata Repository (CMR) and dynamic tile generation for visualization in VEDA. This integration enables access to thousands of CMR-registered datasets without requiring data migration. + +## Overview + +Titiler-CMR extends the TiTiler library to work directly with CMR-registered datasets, providing: + +- **Dynamic Tile Generation**: On-demand tile creation from NetCDF, COG, and Zarr files +- **Multi-Backend Support**: Supports both xarray (NetCDF/Zarr) and rasterio (COG) backends +- **Temporal Queries**: Access time series data across dataset granules +- **Statistical Analysis**: Compute zonal statistics for arbitrary geometries +- **STAC Integration**: Generate STAC-compliant metadata for external datasets + +## Architecture + +```{mermaid} +%%| fig-width: 10 +%%| fig-height: 6 +graph LR + A[VEDA Dashboard] --> B[Titiler-CMR API] + B --> C[NASA CMR] + C --> D[External Data Storage] + D --> E[NetCDF Files] + D --> F[COG Files] + D --> G[Zarr Stores] + B --> H[Dynamic Tiles] + B --> I[Statistics API] + B --> J[STAC Metadata] + + style A fill:#e1f5fe + style B fill:#f3e5f5 + style C fill:#fff3e0 + style D fill:#f1f8e9 +``` + +::: {.callout-note} +**Architecture Overview**: VEDA Dashboard connects to Titiler-CMR API, which interfaces with NASA's CMR to access external data storage containing NetCDF, COG, and Zarr files. Titiler-CMR generates dynamic tiles, provides statistical analysis, and creates STAC-compliant metadata. +::: + +## Configuration Example: GPM Precipitation + +Here's a real-world example using the GPM IMERG Daily precipitation dataset. The complete configuration is available in [`examples/gpm-precipitation-config.json`](./examples/gpm-precipitation-config.json). + +### Collection Configuration + +```json +{ + "type": "Collection", + "id": "GPM_3IMERGDF", + "stac_version": "1.0.0", + "description": "GPM IMERG Final Daily precipitation estimates", + "collection_concept_id": "C2723754864-GES_DISC", + "cube:dimensions": { + "time": { + "extent": ["1998-01-01T00:00:00Z", null], + "description": "time", + "step": "P1D", + "type": "temporal" + }, + "lon": { + "axis": "x", + "extent": [-179.95, 179.95], + "description": "Longitude", + "reference_system": 4326, + "type": "spatial" + }, + "lat": { + "axis": "y", + "extent": [-89.95, 89.95], + "description": "Latitude", + "reference_system": 4326, + "type": "spatial" + } + }, + "summaries": { + "precipitation": { + "type": "data", + "description": "Daily precipitation rate", + "dimensions": ["time", "lon", "lat"], + "unit": "mm/day", + "attrs": { + "units": "mm/day", + "long_name": "Daily mean precipitation rate" + }, + "renders": "precipitation" + } + }, + "renders": { + "precipitation": { + "title": "Precipitation", + "resampling": "average", + "colormap_name": "blues", + "rescale": [[0, 48]], + "backend": "xarray" + } + } +} +``` + +### Key Configuration Elements + +#### Backend Selection +- **`xarray`**: For NetCDF and Zarr datasets with multidimensional data +- **`rasterio`**: For Cloud-Optimized GeoTIFF (COG) files + +#### CMR Integration +- **`collection_concept_id`**: Links to the CMR collection (e.g., "C2723754864-GES_DISC") +- **`cube:dimensions`**: Defines spatial and temporal dimensions +- **`summaries`**: Describes available data variables + +#### Rendering Configuration +- **`renders`**: Defines visualization parameters for each variable +- **`colormap_name`**: Specifies color scheme for visualization +- **`rescale`**: Sets data range for color mapping +- **`resampling`**: Controls how data is aggregated at different zoom levels + +## API Usage Examples + +### Tile Generation + +Generate dynamic tiles for a specific date and variable: + +```bash +GET /WebMercatorQuad/tilejson.json?concept_id=C2723754864-GES_DISC&datetime=2024-01-15&backend=xarray&variable=precipitation&rescale=0,50&colormap_name=blues +``` + +### Statistical Analysis + +Compute statistics for a geographic region: + +```bash +POST /statistics?concept_id=C2723754864-GES_DISC&datetime=2024-01-15&backend=xarray&variable=precipitation +Content-Type: application/json + +{ + "type": "Feature", + "geometry": { + "type": "Polygon", + "coordinates": [[...]] + } +} +``` + +### Time Series Access + +Access temporal data across multiple granules: + +```bash +GET /time-series?concept_id=C2723754864-GES_DISC&datetime=2024-01-01/2024-01-31&backend=xarray&variable=precipitation&lat=40.7&lon=-74.0 +``` + + + + + +## Related Resources + +- [Titiler-CMR Documentation](https://developmentseed.org/titiler-cmr/) +- [NASA CMR API](https://cmr.earthdata.nasa.gov/search/) +- [Earthdata Access](https://github.com/nsidc/earthaccess) +- [STAC Specification](https://stacspec.org/) +- [Xarray Documentation](https://docs.xarray.dev/) + +## Support + +For assistance with Titiler-CMR integration: + +1. Review the [Titiler-CMR examples](https://developmentseed.org/titiler-cmr) +2. Check existing [VEDA data configurations](https://github.com/NASA-IMPACT/veda-data) +3. Open an issue in the [veda-data repository](https://github.com/NASA-IMPACT/veda-data/issues) +4. Contact the VEDA team for complex integration requirements \ No newline at end of file