Skip to content

Commit

Permalink
Merge pull request #19 from evetion/feat/v2
Browse files Browse the repository at this point in the history
  • Loading branch information
evetion authored Sep 22, 2022
2 parents 691d08d + dcfc878 commit 927aa8c
Show file tree
Hide file tree
Showing 41 changed files with 1,087 additions and 744 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,5 @@ scripts
/temp
/output
/data
/src/old
.env
37 changes: 7 additions & 30 deletions Project.toml
Original file line number Diff line number Diff line change
@@ -1,64 +1,41 @@
name = "SpaceLiDAR"
uuid = "29bf0bfc-420f-4fa5-b441-811fd9e6e11d"
authors = ["Maarten Pronk", "Deltares"]
version = "0.1.6"
version = "0.2.0"

[deps]
AWSCore = "4f1ea46c-232b-54a6-9b17-cc2d0f3e6598"
AWSS3 = "1c724243-ef5b-51ab-93f4-b0a88ac62a95"
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Distances = "b4f34e82-e78d-54a5-968a-f98e89d6e8f7"
Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
FillArrays = "1a297f60-69ca-5386-bcde-b61e274b549b"
GeoArrays = "2fb1d81b-e6a0-5fc5-82e6-8e06903437ab"
GeoDataFrames = "62cb38b5-d8d2-4862-a48e-6a340996859f"
GeoInterface = "cf35fbd7-0cd7-5166-be24-54bfbe79505f"
HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
ImageFiltering = "6a3955dd-da59-5b1f-98d4-e7296123deb5"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
LazIO = "c3605908-9f0f-11e8-0a72-0d361c15a277"
NearestNeighbors = "b8a86587-4115-5ab1-83bc-aa920d37bbce"
PolygonOps = "647866c9-e3ac-4575-94e7-e3d426903924"
Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
Proj = "c94c279d-25a6-4763-9509-64d165bea63e"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
StarTIN = "152385ed-0447-47b3-b417-bbbf3d580ece"
StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
TableOperations = "ab02a1b2-a7df-11e8-156e-fb1833f50b87"
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"

[compat]
AWSCore = "^0.6"
AWSS3 = "^0.8, 0.9, 0.10"
CSV = "^0.8, 0.9, 0.10"
CategoricalArrays = "^0.9, 0.10"
DataFrames = "1.3"
Distances = "^0.10"
FillArrays = "^0.11, 0.12, 0.13"
GeoArrays = "^0.7.1"
GeoDataFrames = "^0.2.1, 0.3"
GeoInterface = "1"
HDF5 = "^0.15, 0.16"
HTTP = "0.9, 1"
ImageFiltering = "^0.6, 0.7"
JSON = "^0.21"
LazIO = "^0.3"
NearestNeighbors = "^0.4"
PolygonOps = "^0.1"
ProgressMeter = "^1.4"
Proj = "1"
RecipesBase = "^1.1"
StarTIN = "^0.1"
StaticArrays = "^0.12, 1"
TableOperations = "1.2"
Tables = "^1.4"
julia = "^1.6"

[extras]
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]
test = ["Test", "DataFrames"]
29 changes: 25 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@


# SpaceLiDAR
A Julia toolbox for ICESat-2 and GEDI data.
A Julia toolbox for ICESat, ICESat-2 and GEDI data. Quickly search, download and load filtered point data with relevant attributes from the `.h5` granules of each data product.

*This is a research package now, things are quick to change.*
*This is a research package, things are quick to change.*

Currently supports the following data products:

Expand All @@ -20,7 +20,7 @@ Currently supports the following data products:
|ICESat-2| ATL06 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL03-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL06_ATBD_r005.pdf) |
|ICESat-2| ATL08 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL08-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL08_ATBD_r005.pdf) |
|ICESat-2| ATL12 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL12-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL12_ATBD_r005.pdf) |
|GEDI| L2A v2 | [UG](https://lpdaac.usgs.gov/documents/998/GEDI02_User_Guide_V2.pdf) | [ATBD](https://lpdaac.usgs.gov/documents/581/GEDI_WF_ATBD_v1.0.pdf) |
|GEDI| L2A v2 | [UG](https://lpdaac.usgs.gov/documents/998/GEDI02_UserGuide_V21.pdf) | [ATBD](https://lpdaac.usgs.gov/documents/581/GEDI_WF_ATBD_v1.0.pdf) |

For a quick overview, see the FOSS4G Pluto notebook [here](https://www.evetion.nl/SpaceLiDAR.jl/dev/tutorial/foss4g_2021.jl.html)

Expand Down Expand Up @@ -66,14 +66,35 @@ local_granules = granules_from_folder(<folder>)
local_granules = instantiate(granules, <folder>)
```

Derive points
```julia
using DataFrames
fn = "GEDI02_A_2019242104318_O04046_01_T02343_02_003_02_V002.h5"
g = SpaceLiDAR.granule_from_file(fn)
df = DataFrame(g)
149680×15 DataFrame
Row │ longitude latitude height height_error datetime intensity sensitivity surface quality nmo
│ Float64 Float64 Float32 Float32 DateTime Float32 Float32 Bool Bool UIn
────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1153.855 -47.2772 -13.3536 0.307976 2019-08-30T10:48:21.047 393.969 -0.0671094 true false
2153.855 -47.2769 -11.2522 0.307978 2019-08-30T10:48:21.055 797.26 0.533529 true true
3153.856 -47.2767 -13.775 0.307981 2019-08-30T10:48:21.063 1010.39 0.695938 true true
4153.857 -47.2765 -11.729 0.307983 2019-08-30T10:48:21.071 852.614 0.544849 true true
5153.857 -47.2763 -13.2443 0.307985 2019-08-30T10:48:21.080 980.66 0.620767 true true
6153.858 -47.2761 -12.1813 0.307987 2019-08-30T10:48:21.088 937.441 0.620531 true true
7153.859 -47.2758 -11.9011 0.30799 2019-08-30T10:48:21.096 1235.02 0.73815 true true
8153.859 -47.2756 -12.3796 0.307992 2019-08-30T10:48:21.104 854.127 0.545655 true true
```


Derive linestrings
```julia
using DataFrames
fn = "ATL03_20181110072251_06520101_003_01.h5"
g = SpaceLiDAR.granule_from_file(fn)
tlines = DataFrame(SpaceLiDAR.lines(g, step=10000))
Table with 4 columns and 6 rows:
geom sun_angle track t
geom sun_angle track datetime
┌───────────────────────────────────────────────────────────────────────────
1 │ wkbLineString25D geometry 38.3864 gt1l_weak 2018-11-10T07:28:01.688
2 │ wkbLineString25D geometry 38.375 gt1r_strong 2018-11-10T07:28:02.266
Expand Down
48 changes: 42 additions & 6 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
site_name: SpaceLiDAR.jl
repo_url: https://github.com/evetion/SpaceLiDAR.jl
site_description: Description...
site_author: evetion
site_description: SpaceLiDAR.jl
site_author: Maarten Pronk
site_url: https://www.evetion.nl/SpaceLiDAR.jl/

extra_css:
Expand All @@ -15,34 +15,63 @@ extra_javascript:
docs_dir: "build"

nav:
- Home: "index.md"
- Changelog: "changelog.md"
- Guides:
- "Downloading": "guides/downloads.md"
- "Track filtering": "guides/tracks.md"
- Tutorials:
- "Installation": "tutorial/installation.md"
- "Usage": "tutorial/usage.md"
- Topics:
- "ICESat": "topics/ICESat.md"
- "ICESat-2": "topics/ICESat-2.md"
- "GEDI": "topics/GEDI.md"
- Reference:
- "Search": "reference/search.md"
- "Granules": "reference/granules.md"
- "Data": "reference/data.md"
- "Utilities": "reference/utilities.md"
- "Index": "reference/api.md"

theme:
name: material
icon:
logo: material/satellite-variant
font:
# text: Source Sans Pro
code: Fira Mono
favicon: img/favicon.svg
language: en
custom_dir: overrides
palette:
- media: "(prefers-color-scheme: light)"
scheme: default
primary: blue grey
accent: indigo
primary: lime
accent: deep orange
toggle:
icon: material/lightbulb-outline
name: Switch to dark mode
- media: "(prefers-color-scheme: dark)"
scheme: slate
primary: lime
accent: deep orange
toggle:
icon: material/lightbulb
name: Switch to light mode
features:
- navigation.instant
- navigation.tracking
- navigation.tabs
- navigation.tabs.sticky
- navigation.sections
- navigation.expand
- navigation.top
- toc.follow
- toc.integrate
- announce.dismiss
- search.suggest
- search.highlight
plugins:
- search
- autorefs
Expand All @@ -55,9 +84,13 @@ markdown_extensions:
- tables
- mdx_math
- admonition
- footnotes
- toc:
permalink: true

- attr_list
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji
emoji_generator: !!python/name:materialx.emoji.to_svg
edit_uri: edit/master/docs/
repo_name: evetion/SpaceLiDAR.jl
extra:
Expand All @@ -70,4 +103,7 @@ extra:
- icon: fontawesome/brands/gitter
link: https://gitter.im/evetion/community
name: Chat with me on Gitter
copyright: Copyright &copy; 2021 Maarten Pronk, Deltares
- icon: fontawesome/brands/twitter
link: https://twitter.com/3vetion
name: Twitter
copyright: Copyright &copy; 2022 Maarten Pronk, Deltares
8 changes: 8 additions & 0 deletions docs/overrides/main.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{% extends "base.html" %}

{% block announce %}
<!-- Add announcement here, including arbitrary HTML -->
The newly released v0.2.0 is a *breaking* release. Please see the <a
href="/SpaceLiDAR.jl/stable/changelog/">changelog</a> for
details.
{% endblock %}
21 changes: 19 additions & 2 deletions docs/src/changelog.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,24 @@
## Unreleased
- Documentation improvements
- Extents support
- Search using polygons
- ICESat-2 S3 access
- 20m resolution data in ICESat-2 ATL08 v05

## v0.2.0

!!! danger
This is a **breaking** release

- Many of the column names have changed to be more descriptive.
- Documentation and docstring improvements.
- Tables support, you can now do `DataFrame(granule)`, without having to call `points(granule)`.
- Memory use improvements, by using SentinelArray of FillArray under the hood.
- Dropped S3, GeoArrays and LAS/LAZ support.
- Added GeoInterface support for lines/points and dropped GeoDataFrames
- Expanded test coverage.

## v0.1.6
- Support for ICESat GLAH06 by [alex-s-gardner](https://github.com/alex-s-gardner)
- Support for ICESat GLAH06 by [alex-s-gardner](https://github.com/alex-s-gardner) :+1::tada:

## v0.1.5
- Support for ICESat-2 ATL06
Expand Down
20 changes: 20 additions & 0 deletions docs/src/guides/downloads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Downloading

As shown in [Tutorial: Search for data](../tutorial/usage.md#search-for-data), it is easy to find data. Downloading the data might be harder though,
especially when downloading a large amount of granules or even mirroring a complete DAAC.
Indeed, the Julia [`download!`](@ref) won't work in parallel, nor will it resume downloads or show its progress.
In such cases it's useful to export a list of granules to a text file and use an external download tool:

```julia
granules = find(:ICESat2, "ATL08")
SpaceLiDAR.write_granule_urls!("atl08_world.txt", granules)
```

In my case, I use [aria2c](https://aria2.github.io/manual/en/html/aria2c.html).
Note that downloading from the granule urls require a EarthData login,
normally setup in an .netrc file (also see [`netrc!`](@ref)).
```bash
aria2c -c -i atl08_world.txt
```

Once finished, one can again [`instantiate`](@ref) the list of granules with the folder to which all files have been downloaded.
27 changes: 12 additions & 15 deletions docs/src/index.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,11 @@

[![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://evetion.github.io/SpaceLiDAR.jl/stable)
[![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://evetion.github.io/SpaceLiDAR.jl/dev)
[![CI](https://github.com/evetion/SpaceLiDAR.jl/actions/workflows/CI.yml/badge.svg)](https://github.com/evetion/SpaceLiDAR.jl/actions/workflows/CI.yml)
[![codecov](https://codecov.io/gh/evetion/SpaceLiDAR.jl/branch/master/graph/badge.svg?token=nztwnGtIcY)](https://codecov.io/gh/evetion/SpaceLiDAR.jl)

# SpaceLiDAR
A Julia toolbox for ICESat-2 and GEDI data.
A Julia toolbox for ICESat, ICESat-2 and GEDI data. Quickly [search](tutorial/usage.md#search-for-data), download and [load](tutorial/usage.md#derive-points) filtered point data with relevant attributes from the `.h5` granules of each data product.

*This is a research package now, things are quick to change.*
For a quick overview, see the FOSS4G Pluto notebook [here](tutorial/foss4g_2021.jl.html).

*This is a research package, things are quick to change.*

## Supported data products
Currently supports the following data products:

| mission | data product | User Guide (UG) | Algorithm Theoretical Basis Document (ATBD)|
Expand All @@ -20,16 +16,17 @@ Currently supports the following data products:
|ICESat-2| ATL06 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL03-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL06_ATBD_r005.pdf) |
|ICESat-2| ATL08 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL08-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL08_ATBD_r005.pdf) |
|ICESat-2| ATL12 v5 | [UG](https://nsidc.org/sites/nsidc.org/files/ATL12-V005-UserGuide.pdf) | [ATBD](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL12_ATBD_r005.pdf) |
|GEDI| L2A v2 | [UG](https://lpdaac.usgs.gov/documents/998/GEDI02_User_Guide_V2.pdf) | [ATBD](https://lpdaac.usgs.gov/documents/581/GEDI_WF_ATBD_v1.0.pdf) |

For a quick overview, see the FOSS4G Pluto notebook [here](tutorial/foss4g_2021.jl.html)
|GEDI| L2A v2 | [UG](https://lpdaac.usgs.gov/documents/998/GEDI02_UserGuide_V21.pdf) | [ATBD](https://lpdaac.usgs.gov/documents/581/GEDI_WF_ATBD_v1.0.pdf) |


## Papers
Results are used (indirectly) in the following papers:
## Documentation
This documentation is set up using the [Divio](https://documentation.divio.com/) documentation system and thus split into [Tutorials](tutorial/usage.md), [Guides](guides/downloads.md), [Topics](topics/GEDI.md) and [References](reference/search.md).

Vernimmen, Ronald, Aljosja Hooijer, and Maarten Pronk. 2020. ‘New ICESat-2 Satellite LiDAR Data Allow First Global Lowland DTM Suitable for Accurate Coastal Flood Risk Assessment’. Remote Sensing 12 (17): 2827. [https://doi.org/10/gg9dg6](https://doi.org/10/gg9dg6).
## Publications
The code produced for the following paper was the beginning of this package:

Hooijer, A., and R. Vernimmen. 2021. ‘Global LiDAR Land Elevation Data Reveal Greatest Sea-Level Rise Vulnerability in the Tropics’. Nature Communications 12 (1): 3592. [https://doi.org/10/gkzf49](https://doi.org/10/gkzf49).
> Vernimmen, Ronald, Aljosja Hooijer, and Maarten Pronk. 2020. ‘New ICESat-2 Satellite LiDAR Data Allow First Global Lowland DTM Suitable for Accurate Coastal Flood Risk Assessment’. Remote Sensing 12 (17): 2827. [https://doi.org/10/gg9dg6](https://doi.org/10/gg9dg6).
The DTM produced using ICESat-2 ATL08 data was in turn used for:

> Hooijer, A., and R. Vernimmen. 2021. ‘Global LiDAR Land Elevation Data Reveal Greatest Sea-Level Rise Vulnerability in the Tropics’. Nature Communications 12 (1): 3592. [https://doi.org/10/gkzf49](https://doi.org/10/gkzf49).
4 changes: 0 additions & 4 deletions docs/src/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,3 @@

```@index
```

```@autodocs
Modules = [SpaceLiDAR]
```
22 changes: 22 additions & 0 deletions docs/src/reference/data.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Data

## ICESat
```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["GLAH06.jl", "GLAH14.jl"]
```

## ICESat-2
```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["ATL03.jl", "ATL06.jl", "ATL08.jl", "ATL12.jl"]
```

## GEDI
```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["L2A.jl"]
```
7 changes: 7 additions & 0 deletions docs/src/reference/granules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Granules

```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["granule.jl", "ICESat/ICESat.jl", "ICESat-2/ICESat-2.jl", "GEDI/GEDI.jl"]
```
7 changes: 7 additions & 0 deletions docs/src/reference/search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Search

```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["search.jl"]
```
7 changes: 7 additions & 0 deletions docs/src/reference/utilities.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Utilities

```@autodocs
Private = false
Modules = [SpaceLiDAR]
Pages = ["geom.jl", "geoid.jl", "utils.jl"]
```
Loading

2 comments on commit 927aa8c

@evetion
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/68763

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.2.0 -m "<description of version>" 927aa8ca624e8fef89ab309f3f7baa413438a8ec
git push origin v0.2.0

Please sign in to comment.