We recently added the capability to specify regions by passing the name of a shapefile that defines the region in addition to the old method of lat/lon boxes. This is important for global FEDS, and it was tested and working just fine on the ADE and on our local machines.
However, when I started trying to run FEDS for different regions on DPS using this functionality, I saw that no fire pixels were being used, leading to empty outputs as well.
The input data was all the same. I eventually narrowed the issue down to the preprocess.preprocess_region() outputs- for some reason, running the same code on DPS led to a mangled preprocessed region JSON shape, while it worked just fine elsewhere. I was never able to reproduce the issue locally, so all debugging was done on DPS by creating new staging images and submitting test runs with the manual-v3 GitHub Action.
Long story medium, the issue became apparent in FireIO.get_any_shp. This function attempts to read in the user-specified shapefile, then converts it to EPSG:4326, since all initial input data filtering is done in a geographic coordinate system, as that is what the VIIRS data come with. On the ADE, this worked just fine, but on DPS, it didn't. This was hard to trace, because in both cases .to_crs("EPSG:4326") was being called on the input geometry, and in both cases the resulting geodataframe claimed that was its CRS. However, further investigation revealed that while the CRS attribute was changed for both, on DPS the actual input geometry was never reprojected.
Why? Because we were trying to feed in projected region shapes with geojson files. However, the geojson specification does not support CRSs other than "EPSG:4326." But, because it at one point did support this, some versions of GDAL/fiona/etc will pick up on a CRS attribute, while others won't. The library version combination on DPS didn't, and instead silently assumed it was in EPSG:4326- a reasonable assumption given the geojson specification says it must be. So, since the CRS read as 4326 from the beginning, when the reprojection was called, no operation was actually done on the geometry, since it was nominally already in that projection. Obviously, none of the active fire detections were inside the resulting crazy geometry- this is why no pixels were picked up on those DPS runs.
Proposed solution: we should not use geojson files to store projected shapes. I suggest that we use gpkg for region definition shapefiles instead, but other potential solutions would include other formats (parq, fgb) or making it a requirement that any input bounding geometries are specified in EPSG:4326.
TODO in order to close issue:
We recently added the capability to specify regions by passing the name of a shapefile that defines the region in addition to the old method of lat/lon boxes. This is important for global FEDS, and it was tested and working just fine on the ADE and on our local machines.
However, when I started trying to run FEDS for different regions on DPS using this functionality, I saw that no fire pixels were being used, leading to empty outputs as well.
The input data was all the same. I eventually narrowed the issue down to the
preprocess.preprocess_region()outputs- for some reason, running the same code on DPS led to a mangled preprocessed region JSON shape, while it worked just fine elsewhere. I was never able to reproduce the issue locally, so all debugging was done on DPS by creating new staging images and submitting test runs with the manual-v3 GitHub Action.Long story medium, the issue became apparent in
FireIO.get_any_shp. This function attempts to read in the user-specified shapefile, then converts it to EPSG:4326, since all initial input data filtering is done in a geographic coordinate system, as that is what the VIIRS data come with. On the ADE, this worked just fine, but on DPS, it didn't. This was hard to trace, because in both cases.to_crs("EPSG:4326")was being called on the input geometry, and in both cases the resulting geodataframe claimed that was its CRS. However, further investigation revealed that while the CRS attribute was changed for both, on DPS the actual input geometry was never reprojected.Why? Because we were trying to feed in projected region shapes with geojson files. However, the geojson specification does not support CRSs other than "EPSG:4326." But, because it at one point did support this, some versions of GDAL/fiona/etc will pick up on a CRS attribute, while others won't. The library version combination on DPS didn't, and instead silently assumed it was in EPSG:4326- a reasonable assumption given the geojson specification says it must be. So, since the CRS read as 4326 from the beginning, when the reprojection was called, no operation was actually done on the geometry, since it was nominally already in that projection. Obviously, none of the active fire detections were inside the resulting crazy geometry- this is why no pixels were picked up on those DPS runs.
Proposed solution: we should not use geojson files to store projected shapes. I suggest that we use gpkg for region definition shapefiles instead, but other potential solutions would include other formats (parq, fgb) or making it a requirement that any input bounding geometries are specified in EPSG:4326.
TODO in order to close issue: